Visual Search Engines: The Future Search Engines

Text based search engines, such as Google, Yahoo, and Microsoft search engines, have become an important tool for us to obtain useful information through Internet. Interestingly, even you want to find a picture on the web, a text description of the picture has to be input to the search box. The search engine is then actually doing a text-context based search.

Imagine the following scenarios. Your girlfriend/wife came across a lady carrying a beautiful handbag on the 5th Ave of New York City. She quickly took a photo of that bag using her iPhone. Can she find similar bags instantly through iPhone? In another scenario when she came across a stylish Chanel handbag, which is too expensive for you to afford. Can you quickly find a cheaper one of similar style online?

Obviously, the current functionalities of major search engines are far from meeting our increasing demand. First of all, “Each picture is worth a thousand of words”, therefore, sometimes it is kind of difficult to describe what you really want using concise phrases. Secondly, even if you can describe the scene, many unrelated pictures are usually returned for you to manually sift what you need page by page.

Is there an efficient and automatic way to solve this problem? The answer is yes, I believe. Visual search will be the future and the ultimate solution for above scenarios.

What is Visual Search?

Visual search is a concept of searching by visual features, such as color, texture (pattern), shape and object (e.g., faces, handbags, cameras, etc.), in contrast to searching by text context. The input to the visual search engine could be a picture, a part of the picture or a manual sketch.

Why Visual Search is a Challenging Problem?

Many computer vision problems share similar challenges due to large variations of object appearance under different image acquisition conditions.

· Different lighting conditions (sunny or cloudy, indoor or outdoor)

· Different viewpoint (angles)

· Different time (young and old faces, day and night)

· Occlusion, some part of the object is not visible

· Environment noise, such as dust

Even for the same object, the images captured under different conditions would have dramatically different appearance. This implies great challenges for computer vision scientists to robustly apply the technology to practical applications. Visual search as a typical computer vision and pattern recognition problem, shares the same flavor of these problems. In addition, online visual search has its own special properties:

· Efficiency: Millions of millions of images are available on the web, and millions of images are uploaded online. There is a high demand of high efficiency indexing and searching algorithm, as well as computational power with large-scale computer clusters.

· Accuracy: unlike security (e.g., face recognition) applications, visual search usually do not require to have four-nine accuracy (99.99%).

In spite of these challenges we discussed above, computer vision engineers and entrepreneurs have made a big stride along the right direction. Many start-up companies prosper in this area, such as:,,

GazoPa TinEye

· You can upload a photo of a predefined category (dresses, shoes, and handbags) and style, will return similar product pictures from the web. The search can be refined by color, shape or patterns.

· GazoPa combines the visual and text based search. You can upload any kind of picture, and also input the keyword for the object you are interested in, to limit the search domain. Color, shape and/or face options are further used to restrict your search.

· You can upload any kind of picture from web or your local computer, the visual search engine can not only tell you where the picture comes from, but also return all the same or similar pictures from web.

We are very happy to see the recent progresses in visual search. However, just like text-based search engines, visual search is far from a solved problem. We will closely follow recent developments of visual search and propose future directions for improvement of visual search.

No Comments

Be the first to comment!

RSS feed for comments on this post. TrackBack URI

Leave a comment