Development of Low Cost Machine Vision System for Part Identification using Android Smartphone, a Frugal Engineering
In an automobile assembly line, visual inspection is important in order to increase the reliability. In most of the industries, it is carried out manually in all the work stations by the workers. The manual inspection may lead to errors in the assembly line. A machine vision system is very expensive when implemented for all work stations. On the above concern, the objective of this project is to design a low cost machine vision system using a smartphone, nothing but a frugal innovation. A smartphone has all the needed components for a machine vision system which is comparatively cheaper for the intended assembly task.
Frugal innovation or frugal engineering is the process of reducing the complexity and cost of a good and its production. Jugaad (alternatively Juggaar) is a colloquial Hindi (Devanagiri जुगाड़) and Punjabi word, literally meaning a hack. This meaning is often used to signify creativity to make existing things work or to create new things with meagre resources. Jugaad is increasingly accepted as a management technique and is recognized all over the world as an acceptable form of frugal engineering at peak in India. Companies in India are adopting Jugaad as a practice to reduce research and development costs. Jugaad also applies to any kind of creative and out of the box thinking or life hacking, which maximizes resources for a company and its stakeholders. This inspired to craft this project.
Inspection has become an essential part of any manufacturing system. It is the means of rejecting nonconformities and assuring good quality products. The advent of technologically updated inspection equipment helped to overcome the problems associated with traditional approaches. Traditional approach used labor-intensive methods that resulted in the increase of manufacturing lead time and production cost. Moreover, there is a significant delay in detecting an out of control limit.
Over the past several years, many automotive manufacturers have been motivated to increase the reliability of their products. Implementing machine vision for automated inspection helps these manufacturers achieve this reliability.
Table I.Cost estimation of proposed machine vision system
|S.No.||MACHINE VISION SYSTEM COMPONENT||COST (Rs.)|
|3||USB-Camera (8.0 MP)||2,500|
|4||Speaker system (warning system)||1,000|
Other driving forces include improving quality, streamlining production, decreasing scrap rates, and managing inventory and gathering process control data by reading part codes. From the table we infer that the major reason for the high cost of the machine vision system is due to software licensing, followed by the computer system. Hence there is a need for a customized low cost visual inspection system in the industry for part identification.
The major reason for the high cost of machine vision system is due to the software licensing and computer system. Hence re-modifying or eliminating these two components would eventually reduce the cost of the machine vision system. A machine vision system consists of a Camera for image acquisition, Processor for image processing, decision making and Output system for displaying the result.
A smartphone is a mobile phone built on a mobile operating system, with more advanced computing capability and connectivity than a feature phone. They have the functionality of high speed processors, low-end compact digital cameras, pocket video cameras, and GPS navigation units to form one multi-use device. The increased market demand led a fierce competition among manufacturers, which eventually decreased the cost of the smartphone.
Table II. COMPARISON BETWEEN COMPUTER SYSTEM AND SMARTPHONE
|5||In-built speaker system||No||Yes|
B. Challenges in Using the Smartphone as Machine Vision System
- Programming complexity: The programming has to be done in either android or IOS, which is very complex.
- Accessibility: Accessing the camera hardware of the smartphone
- Developing a automatic capturing system to capture the image of the part to be identified.
- Developing a warning system to convey the results.
C. Android (Operating System)
Android is an mobile operating system based on the Linux kernel and designed primarily for touchscreen mobile devices such as smartphones and tablet computers. It was developed by Android, Inc.
D. Eclipse Ide
In computer programming, eclipse is a multi-language Integrated development environment (IDE) comprising a base workspace and an extensible plug-in system for customizing the environment. It is written mostly in Java. It can be used to develop applications in Java. Released under the terms of the Eclipse Public License, Eclipse SDK is free and open source software
E. Image Processing Library
OpenCV (Open Source Computer Vision Library) is a library of programming functions mainly aimed at real-time computer vision, developed by Intel, and now supported by Willow Garage. It is free for use under the open source BSD license. The library is cross-platform. It focuses mainly on real-time image processing. If the library finds Intel’s Integrated Performance Primitives on the system, it will use these proprietary optimized routines to accelerate it. It has a huge collection of image processing libraries which can be easily accessed
Table III.COMPARISON BETWEEN COMPUTER SYSTEM AND SMARTPHONE
|5||Programming language||MATLAB, LabVIEW, IMAGEJ, python||Android, IOS|
F. Functions to be Performed by the Smartphone Application (.APK file format)
- To accept a template containing the ideal image of the object to be inspected.
- To acquire and digitalize the image using camera and ADC
- To smoothen the image and reduce noise
- To detect the edges
- To match the acquired image with the template
- To produce pass/fail result
Fig1.Flow chart of the proposed image capturing technique
VI.SAMPLE PROGRAM CODING
The program code is developed using eclipse IDE. Appropriate android plugins were installed in the IDE. The target is created for android 4.1 Jelly Bean OS. The program is constructed above an openCV source file. The source file consists of camera view listener which will access the java camera of the smartphone.
STEP 1: Access the camera on the smartphone
The camera was accessed using the following code;
STEP 2: Capture the image
The image is captured using the /*mOpenCvCameraView.takePicture(fileName)*/ method. Automatic capturing of image of the part to be identified was done using proximity sensor available in the smartphone. With the help of this proximity sensor we can sense when the part arrives near the camera and use that event to take the picture. The method used is /*public void onSensorChanged(SensorEvent event)*/
STEP 3: Assign particular name for the image
The date and time on which the image was taken is being used to name the image. Thus a unique name is given to the captured image.
/*SimpleDateFormat sdf = new SimpleDateFormat(“yyyy-MM-dd_HH-mm-ss”);
String currentDateandTime = sdf.format(new Date());
String fileName = Environment.getExternalStorageDirectory(). getPath() +”/sample_picture_” + currentDateandTime + “.jpg”;*/
STEP 4: Convert JPEG to Bitmap file
The captured image is stored as JPEG by default. Further processing of JPEG file format, because of its nature reduces the clarity of the image being processed. Hence the image must be converted into Bitmap format.
STEP 5: Decode to Bitmap file
/*Bitmap src = BitmapFactory.decodeFile (mPictureFileName);*/
STEP 6: Convert RBG image into greyscale image
RBG image increases the complexity of image processing. Hence it is converted to greyscale image.
/*Bitmap src = BitmapFactory.decodeFile(mPictureFileName);*/
STEP 7: Convert Bitmap file into matrix form
/*Mat mimage = new Mat();
STEP 8: Apply canny algorithm
/*Imgproc.Canny(mimage, mimage, 80, 90);
STEP 9: Save the image
The image is saved in SD card using the assigned name.
/*mPictureFileName = “/sdcard/sample.jpg”;
File f = new File(mPictureFileName);*/
STEP 10: Access the template image
The template image is the image of the part to be identified. Initially an ideal image which consists of the part to be identified is taken. The region containing the part to be identified is cropped out. It is saved in the SD card as template.jpg. Later the program must access the template.jpg to apply the matching algorithm.
/*String inFile = mPictureFileName, templateFile = “/sdcard/template.jpg”;*/
STEP 11: Apply template matching algorithm to saved image and template image. Template matching is a technique for finding areas of an image that match (are similar) to a template image (patch).
/*int match_method = Imgproc.TM_CCOEFF;
Imgproc.matchTemplate(img, templ, result, match_method);*/
STEP 12: Compare template matching score with threshold score
The threshold score is calculated by trial and error method. It is then compared with template score to produce pass or fail result. If template score is greater than 0 then part is termed as identified. But if it is less than 0 then part is missing.
STEP 13: If template score is less than threshold score produce sound
Accessing the in-built speaker of the smartphone to will produce sound. A comparator is used to compare the values.
/*streamID = soundPool.play(sound, 1.0f, 1.0f, 0, 0, 1.0f);*/
Thus an android application was created using eclipse IDE in the .apk file format.
VII.RESULTS AND DISCUSSION
The created application which is in the .apk file format was installed in GALAXY- Y smartphone. Experiments were conducted to test the matching capability of the algorithm.
Fig.2.Screenshot of the smartphone containing the installed application
The application was simulated in the smartphone using eclipse so that the template matching scores and location of the match can be viewed. The smartphone was connected to the eclipse via USB with the help of developer options available under the settings menu of the smartphone
Fig.3.Screenshot of the accessed camera
The camera was successfully accessed by the application. Figure 6.2 shows the screenshot of the accessed camera.
VIII.EXPERIMENTS FOR PART IDENTIFICATION
Experiments were conducted to identify the presence of the nut. The template image of the nut to be identified was initially saved in the smartphone. The template image can be seen in the figure 6.3.
Fig.4.Template image of the nut to be identified
A. Results for ‘with part’
Initially the experiment was conducted with the nut.
The application successfully captured the image of the nut and applied canny algorithm to the captured image which can be seen in figure 6.4
Fig.5.Edges detected by canny algorithm
The application successfully applied template matching method to the image to find the template matching score and matching location.
Fig.6.Part successfully identified by the application.
Fig.7.Scores for of template matching algorithm (positive score)
B. Results for ‘without part’
The experiment was conducted without the nut.
Fig.8.Image of part to be identified
The application successfully captured the image and applied canny algorithm to the captured image which can be seen in figure.
Fig.9.Edges detected by canny algorithm
The application successfully applied template matching method to the image to find the template matching score and matching location. Figure shows the part identified by the application. Figure shows template matching score.
Fig.10.Part not identified by the application.
Fig.11.Scores for of template matching algorithm (negative score)
From the above experiments it was confirmed that if the part was present, the template match score was between 0 and 1 and if the part was not present the score was between 0 and -1.
D. Results In Different Smartphones
The application was tested in various other smartphones and results are shown in table 6.1.
Table IV.Analysis under different smartphone
|Name||Processor speed||Camera||Time taken||Flash||Cost in Rs.|
|GALAXY –Y||830 MHz||2.0 MP||58 S||NO||7000|
|GALAXY- S2||1.5 GHz||8.0 MP||13 S||YES||21,500|
|GALAXY -S DUOS||1.0 GHz||5.0 MP||12 S||YES|
IX.CONCLUSION and FUTURE SCOPE
In this project, a low cost machine vision system was created using a smartphone. An android application was developed under eclipse IDE which identifies parts using template matching algorithm. Experiments were conducted to test the capability of the application. The application could identify simple components (for instance, Bolt). But the application has not been tested in real time assembly process. Since, the illumination, positioning of the camera, human intervention are some of the challenges in implementing this system in the assembly line. The application can further be improved to identify more components within a single capture by using advanced features of a smart phone.
- Mikell P. Groover- “Automation, Production Systems, and Computer-Integrated Manufacturing” .
- S.Fu, R.C.Gonzalez, C.S.G.Lee-“ROBOTICS- Control, Sensing, Vision And Intelligence”
- RAFAEL GONZALEZ- “Digital Image Processing” .
- “Smartphone Hardware Sensors”- M.Sc. Maximilian Schirmer, Jun.-Prof. Dr.-Ing. Hagen Hopfner
- John Scott-Thomas TechInsights (2013)- “A Look Inside Smartphone and Tablets” –
- “Machine Vision Course for Manufacturing Engineering Undergraduate Students” – Jose Macedo, Kurt Colvin, and Daniel Waldorf, Industrial and Manufacturing Engineering, California Polytechnic State University, San Luis Obispo, California, USA
- “The mechanical assembly dimensional measurements with the automated visual inspection system” – J. Rejc, F. Kovac, A. Trpin, I. Turk, M. Strus, D. Rejc, P. Obid, M. Munih (2011)
- “Development of a Low-Cost Machine Vision System and Its Application on Inspection Processes”- Yu-Chieh Chen and Yin-Tien Wang (2001)
- Ive Billiauws, Kristiaan Bonjean-“Image recognition on an Android mobile phone”
- “Tutorial on Using Android for Image Processing Projects EE368/CS232 Digital Image Processing”, spring 2013