What is a Web Crawler?
A web crawler is a software program that automatically discovers and scans web sites by following one web page to another. Search sites like Google, Yahoo! and Bing use crawlers to discover web pages and the content on those web pages to create an index of information which is stored by the search site and used to produce search results when people enter search terms.
Web crawlers are also called “bots” which is short for robot. Another term for web crawlers is also “spider,” but that term has been falling out of popular usage and Google, which is the most popular search engine, has it's own crawler called the Goolebot. Microsoft has a crawler called the Bing bot and is also used by Yahoo! since Yahoo! sold their search service to Bing. In fact, there are over 300 different crawlers are listed son the website Robotstxt.org. So what do all these crawlers do on the Internet when they creep around?
What a crawler does
A web crawler needs a list of website addresses to start its work. The crawler then visits the website, much like a person would when typing in a website address, but the crawler is a little more detailed. The crawler will identify all of the links to other webpages, adds them to at the list of webpages to visit, and then visits all of those webpages. Think of it as an automated link clicking program that clicks on every single webpage link that it can find. But that's not all a crawler does while it's working.
In addition to visiting webpages and collecting link information, some web crawlers collect information from the webpages they visit and send it back to a database. This function, called archiving, is how Google and other search engine crawlers create a database of information on the Internet and learn what information is on each and every webpage they crawl. Once the information about a webpage is in a database, the search site can use the database to identify useful webpages and websites for people who use the search site.
What to crawlers collect?
Not all crawlers collect all of the information available on a webpage, but crawlers do have the option to collect any information that is publicly available on the webpages they visit. Since webpages are visible to anyone, a crawler can scan all of the visible information on a webpage and return this information to a database. However, there's more information on a webpage than just what appears to you when you see it on your computer or smartphone.
In addition to the visible information, webpages contain a wealth of hidden information that a crawler can scan and send back to its database. Text information such as file names and descriptions are contained in the web page's structure for items on the web page like pictures and graphic images, but don't appear on the visible webpage. This structure, or format, of a web page, is managed by a web standard formatting language called HyperText Markup Language (HTML for short), and you might already be familiar with the term.
Crawlers have priorities too
A 2009 study presented by A. Signorini and A. Gulli at the Association for Computing Machinery estimated that the Internet at that time had over 11.5 billion individual webpages. The study went on to determine that even the largest search engine crawlers probably identify and crawl no more than 40-70% of all webpages on the Internet, and other studies have set the number of webpages crawled at less than 20%.
Even the best and fastest software crawling programs have limits on how many webpages and websites they can reach, and how much information they can send back to their databases. As a result, crawlers have some rules their masters (the programmers who wrote the crawling programs) created to prioritize how they crawl the Internet.
Since we use the work done by search engine crawlers (Googlebot, etc..) I'll focus on how these types of crawlers decide what to do when they crawl the Internet. These crawlers prioritize their effort based on these factors:
- What websites and what webpages to crawl
With the billions of web pages on the Internet, even the best crawlers can't get to every web page. Search Engine crawlers (like Googlebot) focus on crawling websites and the webpages that are most relevant to their users. Frequently visited websites, websites that update information often, websites that include many web pages and websites with links to other webpages are common ways search engine crawlers prioritize which websites to crawl.
- How often to revisit a webpage or website
A crawler will keep track of how often a website updates with “fresh” information and how frequently those updates occur. The crawler then creates a schedule for how often to revisit the website. Crawlers have to balance the need to visit different websites with the need to visit a website that makes frequent updates. For the database of a Search Engine to be useful, it has to have the most current and up-to-date information about a website. For example, a news site like CNN.com needs to be crawled almost continuously to stay current, while a single webpage type of website might only need to be crawled once a month or even less.
- Not overloading any website from crawling
When a crawler visits a website, the website considers the crawler a visitor just like a person visiting the website. Since the crawler “clicks” on links like a person, the website has to respond, which requires some computing effort on the part of the website and the computer that runs the website. Crawlers try to be “polite” not overburden the website and detract from functioning for people who use the website.
- Synchronizes data collected from multiple crawlers
To crawl all the billions of webpages on the Internet search sites have multiple bots out working and sending back information. There's a chance that two bots might crawl the same website or webpages, so the crawler has to understand if the same website was visited by multiple bots and make sure to remove duplicate data.
If you have a website
For website owners, it's important to submit any new website address to Google and other search engines. While the Googlebot might find your website, if you submit your site address to Google, Google will know about it. The other two popular search sites on the Internet are Bin (Microsoft's) and Yahoo!. Yahoo! uses the Bing search site, so you only need to submit your website address to Bing, and you'll have taken care of both Bing and Yahoo!.
Here are the addresses for both Google and Bing submission pages:
Keep in mind that if you have a website, the crawler will try to follow all of the links on your website, so if you have any confidential or private information on your website, work with your website developer or manager to make sure it's blocked from any crawling.
To my way of thinking
Web crawlers are a necessary tool for search engine sites to function and to help people find information on the billions of webpages that exist on the Internet. While crawlers can't crawl and identify all of the webpages that exist on the Internet since there are just too many of them, the work they do to find relevant and useful websites and create the search site databases is helping make the World Wide Web more useful for us every day.
Receive the newsletter by email: