Development of Low Cost Machine Vision System for Part Identification using Android Smartphone, a Frugal Engineering

Abstract

In an automobile assembly line, visual inspection is important in order to increase the reliability. In most of the industries, it is carried out manually in all the work stations by the workers. The manual inspection may lead to errors in the assembly line. A machine vision system is very expensive when implemented for all work stations. On the above concern, the objective of this project is to design a low cost machine vision system using a smartphone, nothing but a frugal innovation. A smartphone has all the needed components for a machine vision system which is comparatively cheaper for the intended assembly task.

I. Foreword

Frugal innovation or frugal engineering is the process of reducing the complexity and cost of a good and its production. Jugaad (alternatively Juggaar) is a colloquial Hindi (Devanagiri जुगाड़) and Punjabi word, literally meaning a hack. This meaning is often used to signify creativity to make existing things work or to create new things with meagre resources. Jugaad is increasingly accepted as a management technique and is recognized all over the world as an acceptable form of frugal engineering at peak in India. Companies in India are adopting Jugaad as a practice to reduce research and development costs. Jugaad also applies to any kind of creative and out of the box thinking or life hacking, which maximizes resources for a company and its stakeholders. This inspired to craft this project.

II.Introduction

Inspection has become an essential part of any manufacturing system. It is the means of rejecting nonconformities and assuring good quality products. The advent of technologically updated inspection equipment helped to overcome the problems associated with traditional approaches. Traditional approach used labor-intensive methods that resulted in the increase of manufacturing lead time and production cost. Moreover, there is a significant delay in detecting an out of control limit.

III.Problem description

Over the past several years, many automotive manufacturers have been motivated to increase the reliability of their products. Implementing machine vision for automated inspection helps these manufacturers achieve this reliability.

Table I.Cost estimation of proposed machine vision system

S.No.MACHINE VISION SYSTEM COMPONENTCOST (Rs.)
1Computer system25,000
2Software (MATLAB)1,25,000
3USB-Camera (8.0 MP)2,500
4Speaker system (warning system)1,000
TOTAL COST1,53,500

Other driving forces include improving quality, streamlining production, decreasing scrap rates, and managing inventory and gathering process control data by reading part codes. From the table we infer that the major reason for the high cost of the machine vision system is due to software licensing, followed by the computer system. Hence there is a need for a customized low cost visual inspection system in the industry for part identification.

IV.Proposed methodology

The major reason for the high cost of machine vision system is due to the software licensing and computer system. Hence re-modifying or eliminating these two components would eventually reduce the cost of the machine vision system. A machine vision system consists of a Camera for image acquisition, Processor for image processing, decision making and Output system for displaying the result.

A.    Smartphone:

A smartphone is a mobile phone built on a mobile operating system, with more advanced computing capability and connectivity than a feature phone. They have the functionality of high speed processors, low-end compact digital cameras, pocket video cameras, and GPS navigation units to form one multi-use device. The increased market demand led a fierce competition among manufacturers, which eventually decreased the cost of the smartphone.

Table II. COMPARISON BETWEEN COMPUTER SYSTEM AND SMARTPHONE

S.NoFeatureComputer SystemSmartphone
1In-built CameraNoYes
2ProcessorYesYes
3RAMYesYes
4ROMYesYes
5In-built speaker systemNoYes

B.    Challenges in Using the Smartphone as Machine Vision System

  • Programming complexity: The programming has to be done in either android or IOS, which is very complex.
  • Accessibility: Accessing the camera hardware of the smartphone
  • Developing a automatic capturing system to capture the image of the part to be identified.
  • Developing a warning system to convey the results.

C.    Android (Operating System)

Android is an mobile operating system based on the Linux kernel and designed primarily for touchscreen mobile devices such as smartphones and tablet computers. It was developed by Android, Inc.

D.    Eclipse Ide

In computer programming, eclipse is a multi-language Integrated development environment (IDE) comprising a base workspace and an extensible plug-in system for customizing the environment. It is written mostly in Java. It can be used to develop applications in Java. Released under the terms of the Eclipse Public License, Eclipse SDK is free and open source software

E.    Image Processing Library

OpenCV (Open Source Computer Vision Library) is a library of programming functions mainly aimed at real-time computer vision, developed by Intel, and now supported by Willow Garage. It is free for use under the open source BSD license. The library is cross-platform. It focuses mainly on real-time image processing. If the library finds Intel’s Integrated Performance Primitives on the system, it will use these proprietary optimized routines to accelerate it. It has a huge collection of image processing libraries which can be easily accessed

Table III.COMPARISON BETWEEN COMPUTER SYSTEM AND SMARTPHONE

S.NoFeatureComputer SystemSmartphone
1Processing speedHighLow
2Programming complexityLowHigh
3CostHighLow
4Camera capacityHighMedium
5Programming languageMATLAB, LabVIEW, IMAGEJ, pythonAndroid, IOS

F.    Functions to be Performed by the Smartphone Application (.APK file format)

  • To accept a template containing the ideal image of the object to be inspected.
  • To acquire and digitalize the image using camera and ADC
  • To smoothen the image and reduce noise
  • To detect the edges
  • To match the acquired image with the template
  • To produce pass/fail result

V. ALGORITHM

1

Fig1.Flow chart of the proposed image capturing technique

VI.SAMPLE PROGRAM CODING

The program code is developed using eclipse IDE. Appropriate android plugins were installed in the IDE. The target is created for android 4.1 Jelly Bean OS. The program is constructed above an openCV source file. The source file consists of camera view listener which will access the java camera of the smartphone.

STEP 1: Access the camera on the smartphone

The camera was accessed using the following code;


/*

*/

STEP 2: Capture the image

The image is captured using the /*mOpenCvCameraView.takePicture(fileName)*/ method. Automatic capturing of image of the part to be identified was done using proximity sensor available in the smartphone. With the help of this proximity sensor we can sense when the part arrives near the camera and use that event to take the picture. The method used is /*public void onSensorChanged(SensorEvent event)*/

STEP 3: Assign particular name for the image

The date and time on which the image was taken is being used to name the image. Thus a unique name is given to the captured image.

/*SimpleDateFormat sdf = new SimpleDateFormat(“yyyy-MM-dd_HH-mm-ss”);

String currentDateandTime = sdf.format(new Date());

String fileName = Environment.getExternalStorageDirectory(). getPath() +”/sample_picture_” + currentDateandTime + “.jpg”;*/

STEP 4: Convert JPEG to Bitmap file

The captured image is stored as JPEG by default. Further processing of JPEG file format, because of its nature reduces the clarity of the image being processed. Hence the image must be converted into Bitmap format.

STEP 5: Decode to Bitmap file

/*Bitmap src = BitmapFactory.decodeFile (mPictureFileName);*/

STEP 6: Convert RBG image into greyscale image

RBG image increases the complexity of image processing. Hence it is converted to greyscale image.

/*Bitmap src = BitmapFactory.decodeFile(mPictureFileName);*/

STEP 7: Convert Bitmap file into matrix form

/*Mat mimage = new Mat();

Utils.bitmapToMat(dest, mimage);*/

STEP 8: Apply canny algorithm

/*Imgproc.Canny(mimage, mimage, 80, 90);

Utils.matToBitmap(mimage, dest);*/

STEP 9: Save the image

The image is saved in SD card using the assigned name.

/*mPictureFileName = “/sdcard/sample.jpg”;

File f = new File(mPictureFileName);*/

STEP 10: Access the template image

The template image is the image of the part to be identified. Initially an ideal image which consists of the part to be identified is taken. The region containing the part to be identified is cropped out. It is saved in the SD card as template.jpg. Later the program must access the template.jpg to apply the matching algorithm.

/*String inFile = mPictureFileName, templateFile = “/sdcard/template.jpg”;*/

STEP 11: Apply template matching algorithm to saved image and template image. Template matching is a technique for finding areas of an image that match (are similar) to a template image (patch).

/*int match_method = Imgproc.TM_CCOEFF;

Imgproc.matchTemplate(img, templ, result, match_method);*/

STEP 12: Compare template matching score with threshold score

The threshold score is calculated by trial and error method. It is then compared with template score to produce pass or fail result. If template score is greater than 0 then part is termed as identified. But if it is less than 0 then part is missing.

STEP 13: If template score is less than threshold score produce sound

Accessing the in-built speaker of the smartphone to will produce sound. A comparator is used to compare the values.

/*streamID = soundPool.play(sound, 1.0f, 1.0f, 0, 0, 1.0f);*/

Thus an android application was created using eclipse IDE in the .apk file format.

VII.RESULTS AND DISCUSSION

The created application which is in the .apk file format was installed in GALAXY- Y smartphone. Experiments were conducted to test the matching capability of the algorithm.

2Fig.2.Screenshot of the smartphone containing the installed application

The application was simulated in the smartphone using eclipse so that the template matching scores and location of the match can be viewed. The smartphone was connected to the eclipse via USB with the help of developer options available under the settings menu of the smartphone

3Fig.3.Screenshot of the accessed camera

The camera was successfully accessed by the application. Figure 6.2 shows the screenshot of the accessed camera.

VIII.EXPERIMENTS FOR PART IDENTIFICATION

Experiments were conducted to identify the presence of the nut. The template image of the nut to be identified was initially saved in the smartphone. The template image can be seen in the figure 6.3.

4Fig.4.Template image of the nut to be identified

A.    Results for ‘with part’

Initially the experiment was conducted with the nut.

5The application successfully captured the image of the nut and applied canny algorithm to the captured image which can be seen in figure 6.4

6Fig.5.Edges detected by canny algorithm

The application successfully applied template matching method to the image to find the template matching score and matching location.

7Fig.6.Part successfully identified by the application.

8Fig.7.Scores for of template matching algorithm (positive score)

B.    Results for ‘without part’

The experiment was conducted without the nut.

9Fig.8.Image of part to be identified

The application successfully captured the image and applied canny algorithm to the captured image which can be seen in figure.

10Fig.9.Edges detected by canny algorithm

The application successfully applied template matching method to the image to find the template matching score and matching location. Figure shows the part identified by the application. Figure shows template matching score.

11

Fig.10.Part not identified by the application.

12

Fig.11.Scores for of template matching algorithm (negative score)

C.    Inference:

From the above experiments it was confirmed that if the part was present, the template match score was between 0 and 1 and if the part was not present the score was between 0 and -1.

D.    Results In Different Smartphones

The application was tested in various other smartphones and results are shown in table 6.1.

Table IV.Analysis under different smartphone

NameProcessor speedCameraTime takenFlashCost in Rs.
GALAXY –Y830 MHz2.0 MP58 SNO7000
GALAXY- S21.5 GHz8.0 MP13 SYES21,500
GALAXY -S DUOS1.0 GHz5.0 MP12 SYES

IX.CONCLUSION and FUTURE SCOPE

In this project, a low cost machine vision system was created using a smartphone. An android application was developed under eclipse IDE which identifies parts using template matching algorithm. Experiments were conducted to test the capability of the application. The application could identify simple components (for instance, Bolt). But the application has not been tested in real time assembly process. Since, the illumination, positioning of the camera, human intervention are some of the challenges in implementing this system in the assembly line. The application can further be improved to identify more components within a single capture by using advanced features of a smart phone.

X.REFERENCES

  • Mikell P. Groover- “Automation, Production Systems, and Computer-Integrated Manufacturing” .
  • S.Fu, R.C.Gonzalez, C.S.G.Lee-“ROBOTICS- Control, Sensing, Vision And Intelligence”
  • RAFAEL GONZALEZ- “Digital Image Processing” .
  • “Smartphone Hardware Sensors”- M.Sc. Maximilian Schirmer, Jun.-Prof. Dr.-Ing. Hagen Hopfner
  • John Scott-Thomas TechInsights (2013)- “A Look Inside Smartphone and Tablets” –
  • “Machine Vision Course for Manufacturing Engineering Undergraduate Students” – Jose Macedo, Kurt Colvin, and Daniel Waldorf, Industrial and Manufacturing Engineering, California Polytechnic State University, San Luis Obispo, California, USA
  • “The mechanical assembly dimensional measurements with the automated visual inspection system” – J. Rejc, F. Kovac, A. Trpin, I. Turk, M. Strus, D. Rejc, P. Obid, M. Munih (2011)
  • “Development of a Low-Cost Machine Vision System and Its Application on Inspection Processes”- Yu-Chieh Chen and Yin-Tien Wang (2001)
  • Ive Billiauws, Kristiaan Bonjean-“Image recognition on an Android mobile phone”
  • “Tutorial on Using Android for Image Processing Projects EE368/CS232 Digital Image Processing”, spring 2013