scrapper
Joined: 22 Jan 2010
Posts: 11
|
 Posted: Wed Jun 30, 2010 11:17 am Post subject: About webcrawlers |
 |
|
hi,
when am working on SEO optimazation came to know how the page gets crawled by the google.
The webpages are crawled by web crawlers.Lets see what it is.
Web crawlers :
Webcrawlers is nothing but it is a computer crawler that browses the www in a methodical, automated manner or in an orderly fashion.
Web crawlers are mainly used to create a copy of all the visited pages for later processing.
Crawlers are also used for automating maintainance tasks on a website such as checking link, validating html code.And also these crawlers can be used to gather specific type of information from webpages.
Mainly the webcrawlers starts with a list of URL's to visit which are called as seeds.As the crawler visits these URL's it indentifies all the hyperlinks in the page and adds them to list of URL's to visit called the call frontier.
These web crawlers are also called as bots, indexes,ants,worms or web spiders.
The important characteristics that makes crawling more difficult is
1. Large volume
2. Fast rate of change of crawling
3. Dynamic page generation |
|
