Image processing: how to find the principal content automatically

Image processing is a very specific research area in computer science, while it is also a very popular topic due to its magic application in daily life. You might not be an expert on complex image processing algorithms, while you might be very familiar with iPhone’s panorama App, which can help to take an amazing panorama photo when you go around by holding your iPhone.  Isight camera on iPhone will take several photos during the process while image processing algorithms will seamlessly put these photos together automatically and smartly.

In order to fulfill the job like described above, image processing needs to “understand” the content of the photo. Most of time, understanding the principal content will be a practical way to go. Here we don’t depend on Artificial Intelligence, although we mirror the way that human beings interpret an image. Think about what you will notice when seeing a sun rising from sea level. It is “sea level” and the outline of the sun, since these areas contain big changes on color, brightness or shape. These areas are the “principal content” of a photo, which are needed to describe the data in the image economically. Automatically identify the “principal content” of a photo will help us to easily compare two images and find the common-shared part, which is the critical part when stitching photos together.

The first eight eigenfaces abstracted from a principal component analysis (PCA) of the Ekman and Friesen (1976) faces [1]

How to automatically identify the “principal content” is the critical problem if we want machines to help us perform lots of image processing jobs. The basic algorithm to perform this task is Principle Component Analysis (PCA) [2], which was invented in 1901 by Karl Pearson and has been applied in statistics for a long time. I don’t want to discuss about the detailed mathematical content of these algorithms, while I would like to talk a little about the basic idea. Considering the way that used to describe a point in 2-dimension graph, yes, a pair of x and y axes will be enough to fulfill the job and uniquely describe the point without any redundant information. For a 3-dimensional object such as a line, we will naturally use a combination of x, y and z axes to describe. The reason for choosing these two or three mutually perpendicular axes is that they have the best ability to describe the object in the given space. In other word, these perpendicular axes are the “principal component” which can be used to describe the object in the most economically way. PCA is such an algorithm to identify these “principal component” inside a picture. In order to perform this algorithm on an image, we need to digitalize an image to be represented by a matrix first. PCA is to perform a kind of matrix decomposition named SVD (Singular value decomposition) [3] and then determine which set of “axes” (Eigen vector) are the most important to keep in order to describe the photo in a most economically way. 

Reference

[1] The Perception of Facial Expressions, 
http://www.scourge.fr/mathdesc/documents/facerecog/PerceptionFacialExpression.htm

[2] Principal component analysis, http://en.wikipedia.org/wiki/Principal_component_analysis

[3] Singular value decomposition, 
http://en.wikipedia.org/wiki/Singular_value_decomposition