What Search Engine ‘Spiders’ Are And How They Work

March 16th, 2010 |

: Image via Wikipedia

Sometimes referred to as ‘spiders’ or crawlers, automated search engine robots seek out web pages for the user. Just how do they accomplish this and is this of importance? What is the real purpose of these robots?

Robots actually have the same basic functionality that earlier browsers had. Just like these early browsers, search engine robots do not have the ability to do certain things. Robots cannot get past password protected areas. They do not understand frames, Flash movies, nor Images or JavaScript. Even if you use a robot, you have to click the buttons on your website. They can cease to function while using JavaScript navigation or when indexing a dynamically generated URL. A search engine robot retrieves data and finds information and links on the web.

The ‘submit url’ function places the url into a list of urls the robots are going to explore. Even without submitting your url directly, robots will try to find your site by following links. That’s why building visibility through a web of links is important.

Submitting a new URL to a search engine adds this URL to the queue which the spiders are due to ‘crawl’ or visit. However, even if a URL isn’t submitted directly, the spiders usually find it through links from other websites. If you build link popularity, this will help the spiders find you faster. When the robots arrive, they’ll check your site for a file called ‘robots.txt,’ which will tell them what areas of the website they are not allowed to visit. Off-limits files may include things like binaries or other information that the spiders need not report back.

To ensure that searchers get the right results with the most relevant response to their query, quick calculations are done to see that this happens. Server logs and log statistics program results can be checked by the user to see what pages have been visited and how often. Some robots may be easy to identify such as Google‘s ‘Googlebot‘, while less well-known ones such as Inktomi’s ‘Slurp’ are not easily identifiable. Some robots even appear to be human-powered browsers.

A robot ‘reads’ your site by collecting data on any visible text, on tags you may have in the coding of your page, and on any links available. These are the things that determine what the search engines ‘think’ your content is about, so these are the things you really need to pay attention to when building a site that you want to have high visibility in search results.

The search engine sorts the information that has been delivered to the databases which has become a part of the search engine and directory ranking process. This allows it to display the results. Databases are updated periodically. Robots visit you regularly to find any changes to your pages so that the latest information will be available. The way in which the search engine is set up determines how the number of visits you get is calculated. This can vary with different search engines. If your website is down or experiencing a large amount of traffic, the robot may not be able to access the page they are trying to visit. The website may not be re-indexed when this occurs. This depends on how frequently your site is visited by the robot. In the hope that your site will be accessible again, the robot will re-visit your site to see if it has become accessible.

Justin Harrison is an internationally recognised Internet Marketing Consultant who provides world class SEO Services to website owners. For more information visit: http://www.seorankings.co.za

The 6 Key Principles of SEO for Your Blog (socialwebschool.com)
Robots and Stalking 2.0 (petewarden.typepad.com)
The Importance Of Anchor Text (ronmedlin.com)
SEO Health Checks – Regular Housekeeping Tasks for Your Website’s SEO (seomoz.org)

Related Blogs

Shared Post

How To Work Out Your Search Engine Optimization Strategy If you asked the majority of Internet based companies what is the one area of business they would like to improve what do you think that would be? Being further...
Who Will Be The Leading Search Engine At The End Of 2010 It appears as though Google has a fresh contender in the universe of search engines. Have you found that if someone is discussing computers either the name Google or Bing,...
Propositions Imperative To Know About Search Engine Marketing A SEM (Search Engine Marketing) specialist will help you develop a site that can show up very highly in search engine results. There are certain ways which you must do...
A Couple Of General Thoughts On The Importance Of Search Engine Optimization Search engine optimization is something that refers to the entire process of train to insure that your website or your blog is placed and positioned highly when people type particular...
Google Image Search – Its Importance For Your SEO Images can not only make your website visually attractive, they also help you gain more daily search engine traffic. Google Image Search is the version of Google dedicated to images...
Acquiring More Traffic By Employing Search Engine Optimization Strategies Perfecting a websites content, so that you can get yourself a higher search engine ranking is usually what Search Engine Optimization (SEO) is focused on. Presently the word SEO has...
Search Engine Optimization SEO Tutorial — WebBizIdeas Search Engine Optimization SEO Tutorial by WebBizIdeas is for beginners. We will cover SEO techniques that you can use TODAY that will increase your search engine rankings. We will go...