G. Klinker A Physical Approach to Color Image Understanding Ph.D. thesis, Computer Science Department, Carnegie-Mellon University, May, 1988 (bib)
	It is the goal of computer vision to describe a scene in terms of the intrinsic properties of the objects involved, such as the object shape, their material properties and surface roughness, as well as their position in the scene and optical effects related to their interaction with light and shadow. So far, computer vision has not yet been successful in generating such a descripition from the image of a complex scene. The main reason has been a lack of models rich enough to relate pixels and pixel-aggregates to scene characteristics. Two common optical effects in images are shading and highlights on objects. In particular, the occurence of highlights is a problem to many algorithms in computer vision. Motion analysis and stereo vision methods, for example, are disturbed by highlights because these move on the objects when the camera or the objects move. This thesis uses a physical, intrinsic reflection model to analyze the effects of shading and highlights on dielectric, non-uniform, opaque materials, such as plastics and paints. Since black-and-white images do not provide sufficient information to interpret these effects easily, this work uses color images. The colors seen on a dielectric object are an additive mixture of the object color and the illumination color, i.e.: the colors are a linear combinatin of those two color vectors. They therefore lie in a plane in the color space. Within the plane, they form a cluster that looks like a skewed T, due to the geometric properties of shading and highlight reflection. When combined with a camera model, this description of physically possible color changes on one object can be used to guide an algorithm in interpreting color images. It is the basis of an algorighm that analyzes color variation in real color images. The algorithm looks for characteristic color clusters in the image and relates their shape to hypotheses about the object color and the amount of shading, as well as the position, strength and color of highlights on the object. It verifies the hypotheses by reapplying them back to the image, testing whether and how they apply to the pixels in the local image area from which the respective hypothesis was derived. In this way, it adapts the image interpretation process to local scene characteristics and reacts differently to color and intensity changes at different places in the image. The result is an image segmentation that ignores color changes due to shading and highlights, as well as two intrinsic images, one describing the amount of shading at every pixel in the image, and the other one showing the amount of highlight reflection at every pixel. Such intrinsic images can be a useful tool for other areas of computer vision that are currently distrubed by highlights in images, such as stereo vision and motion analysis. The approach can also be used to determine the illumination color from the highlights and as a preprocessor for methods to determine object shape from shading or highlights. This line of research may lead to physics-based image understanding methods that are both more reliable and more useful than traditional methods.
	This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each authors copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.