Sunday, 27 March 2011

History

The invention of "robots.txt" is attributed to Martijn Koster, when working for WebCrawler around 1994. "robots.txt" was then popularized with the advent of AltaVista, and other popular search engines, in the following years.
Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)