Real-time detection of two dimensional deformable patterns for industrial inspection
The project aims at real-time detection of two dimensional deformable patterns for industrial inspection. The fast, robust and accurate localization of a given 2D object template in images has been a research topic since many decades. The results of this research efforts has enabled numerous different industrial applications, as the detection of the objects pose is the natural prerequisite for any senseful operation like counting, defect detection or pick and place operations. In case that the object to be localized is rigid or articulated, different successful approaches have been proposed and are applied in everyday industrial practice. Various applications domains, which necessitate the detection of non-linearly deformed objects, can still not be comprehensively solved. The goal of current research is to develop a method that can solve these kind of tasks.
It is important to note that we intentionally assume neither spatial nor temporal coherence. The object is found despite partial occlusion, clutter and nonlinear illumination changes. Also significant noise and motion blur does not defeat the method.
For the detection we adapt a robust template matching algorithm. This is in contrast to the mainstream computer vision community that has abandoned this direction, because "it takes forever to exhaustively search an 8 dimensional (homography) search space".
Below you can find preliminary results of our method.
Real-Time Tracking with Initialization
This video shows preliminary results of real-time detection of a perspectively distorted business card. The model and detection result is depicted as the red contour that can be seen on the window of the laptop.
To reach the interactive runtime and to keep processing times down, we automatically switch to a tracking mode once the object was successfully found.
Of course, once tracking is lost, we switch back to detection mode. This is the reason why the business card is constantly moved out of the image and back.
Despite the big scale range, detection needs 100ms, tracking 20ms on a laptop computer.
The application does not make much sense, but this is the kind of video researchers tend to produce to show their work.
However, one could imagine that this kind of functionality is for instance useful if one wants to have a 3D mouse for a human-machine interface but has only his business card at hand.
Please note that we did not use the "book" (and SIFT features) and also not the marker. Both would have made detection much easier.
Note the experimental setup for the articulated 3D detection video in the background.
A screen dump avi sequence can be found here.
Articulated 3D Object Detection
Many objects in the real world are composed of various parts that are rigid, but can move in a defined way with respect to each other.
This movie shows an untextured object that consists of a truck with a wide-bucket excavator. Here, the tilt and streching of the crane can be controlled by
two parameters. The detection of each part theoretically allows us to infer the state of the object and to define e.g. a control law. However, here we
manually controlled the lego truck. Of course estimating the state of the world is of utmost importance to the AI community. One could for instance ask
a PROLOG-based automatic reasoning system what the sense of this storyline is. Or one could train an HMM to learn the structure of the storyline.
For the sake of clarity: The two buttons on the left are unique and the 4 buttons on the crane are also unique. If we would have used a "button detection"-filter (if such a filter exists and works well) we would also find the other buttons on the truck. But then, we would have to write complicated algorithms to detect which button belongs to which button.
ID card identification
This video shows detection results of a perspectively distorted id card of our university library. Some images include multiple instances
of the same object. One possible application could be to unwarp the region of the ID card and perform e.g. a OCR or barcode reading of the information on the card for e.g. identification purpose. This is single image detection but if the application would need tracking that could be added (see first movie). Note that several identical ID cards pose no difficulty for the detection.
Inspection for Semiconductor Industry
In this example the position of an electronic component must be determined (like in an everyday industrial inspection task). However, a typical industrial inspection scenario includes a special setup or a camera calibration such that the object is imaged from directly above. Because we have not used a special fixture a perspective distortion of the object can be seen. We restrict the overall scale/rotation range of the detection, so that we reach very fast processing times.
Inspection for Factory Automation
One interesting application of our research is the detection and inspection of industrial objects. This video shows detection of a car door and estimation of its 3D pose. The first two frames show the image of model generation and the same object with a calibration plate. This
allows us to determine the metric scale factor of the model. In the subsequent images, this model is detected and we estimate the 3D pose of the object. The 3D position of e.g. a borehole that is located at the same plane as the planar model are depicted in the video as a green cross. The last frame
shows compromising footage of a PhD student damaging expensive university property.
Inspection for Mechanical Engineering
The detection of metalic objects is a very difficult computer vision problem. Here we detect an item on loan from the mechanical engineering department. In fact, this part should be familiar to many mechanical engineering students of the TUM, since it is used for many years in the technical drawing classes. A screen dump avi sequence of a motor part can be found here.
A video that shows results for service robotics and industrial automation can be found here.
Industrial Augmented Reality
This is a video clip that shows the augmented reality laser projector, an augmented reality system that was developed in cooperation with Bjoern. What this system does is that it detects the 3D pose of a planar object and augments certain interesting parts with a laser projector. We recently made a demo with the setup at an industrial forum and received positive feedback from diverse companies.
The video clip was made at the beginning of the fair and shows the hardware setup with the camera and the laser projector. The processing is done on a laptop
and we visualized the detection of the model. Here we detect a poster and project the logo of the university and the faculty. Due to the different sampling of
laser and camera you can only see parts of the logo. These parts are integrated into the bigger logo by the human eye. The show effect is at 01:35 were the tripod is moved to a position at the other side of the booth, where the same type of poster is detected again. The detection tolerates scale ranges from 35cm to 4m and more than 45 degrees tilt. If you are interested in the system please contact us. However, we do not recommend to use it as an eye catcher for e.g. a fair, because nobody realy read our poster, but just wanted to see the laser beam or if the system can tolerate e.g. occlusion.
Industrial Endoscopy / Remote Expert
Many industial machines require repeated quality assurance tests. Many check procedures are complicated and cost a lot of time/money. To speed this
up one could imagine that an industrial endoscope is inserted into the machine and special parts are detected (In this demo it is a graphics card inside a packing case). The detection of the part allows inspection applications to check if for instance some parts are damaged or missing. For diffuse illumination we attached a 4 Euro flashlight with plastic foil to the camera. The first frame shows the image that was used for model generation.
Remote Sensing / Marienplatz form above
If the earth is imaged from a satelite/airplane/helicopter and the distance is big enough, one can assume that the surface of the earth is flat. In this video
we detect the inner region of Munich and move the camera into different positions. The model is the red region in the video. The video was generated by screen capturing google earth.
Template matching of multiple models / Marienplatz from the earth
In this video we detect different models of street facades of Marienplatz. The detected models are the green regions in the video.
The application for this technology is video-based tour guide for city travels or museums.
Another useful application is video-based navigation for handicapped people.
3D Computer Vision
The developed algorithm could be of some use in mobile robotics and photogrammetry.
Robust correspondences between images are interesting for self calibration, mosaicing or scene reconstruction. Because we use edge contours we can estimate correspondences in situations in which a corner point extractor fails. This example shows the facade of a building that is used for model generation. Note that the two bars of the fence are included in the model but due to the parallax are treated as occlusion in the subsequent pictures. I am a lousy cameraman so don't watch this movie if you are a bad sailor. The big problem with this sequence is that the camera is shacking and therefore the edges are blurred (in some images more than 5 pixel long smear effects are tolerated).
Deformable Object Detection
This video shows preliminary results of the deformable pattern matching algorithm that is developed in this project. Detection needs around a second.
We only show detection result for the object with the maximal score. The logo is from a small, but very famous and important Munich company.
This video shows real-time tracking and detection of a perspectively distorted business card. Despite the big scale range, detection needs 100ms, tracking 20ms on a laptop computer. The application does not make much sense, but this is the kind of video researchers tend to produce to show their work.
