Robots exclusion standard

Sunday, 27 March 2011

Nonstandard extensions

Several major crawlers support a Crawl-delay parameter, set to the number of seconds to wait between successive requests to the same server:[1][2][3]

User-agent: *
Crawl-delay: 10

Posted by G N at 09:47
Email ThisBlogThis!Share to XShare to FacebookShare to Pinterest

No comments:

Post a Comment

Newer Post Older Post Home
Subscribe to: Post Comments (Atom)
  • Pictures of Bed Bugs
  • steroids
  • casino en ligne
  • mortgages for bad credit
  • the mediterranean diet
  • sydney wedding venue
  • DrPullen.com Health and Medical Blog
  • social bookmarking service
  • Cheap Auto Insurance
  • Home Affordable Modification Program
  • African Mango
  • Instant Income Cash Machine
  • בדיחות
  • סרטים חדשים
  • entrepreneur forum
  • Free MMORPG
  • Get Stuff For Free
  • plumber chicago
  • Preventing Gum Disease
  • Weight Loss Calculator
  • Recruitment Agencies In Qatar
  • Recruitment Agencies In Dubai

Blog Archive

  • ▼  2011 (8)
    • ▼  March (8)
      • Robots exclusion standard
      • History
      • About the standard
      • Disadvantages
      • Automated Content Access Protocol
      • Nonstandard extensions
      • Allow directive
      • Sitemap
Simple theme. Powered by Blogger.