Some major crawlers support an Allow directive which can counteract a following Disallow directive.[4] [5] This is useful when one disallows an entire directory but still wants some HTML documents in that directory crawled and indexed. While by standard implementation the first matching robots.txt pattern always wins, Google's implementation differs in that Allow patterns with equal or more characters in the directive path win over a matching Disallow pattern.[6] Bing uses the Allow or Disallow directive which is the most specific.[7]
In order to be compatible to all robots, if one wants to allow single files inside an otherwise disallowed directory, it is necessary to place the Allow directive(s) first, followed by the Disallow, for example:
Allow: /folder1/myfile.html
Disallow: /folder1/
This example will Disallow anything in /folder1/ except /folder1/myfile.html, since the latter will match first. In case of Google, though, the order is not important
In order to be compatible to all robots, if one wants to allow single files inside an otherwise disallowed directory, it is necessary to place the Allow directive(s) first, followed by the Disallow, for example:
Allow: /folder1/myfile.html
Disallow: /folder1/
This example will Disallow anything in /folder1/ except /folder1/myfile.html, since the latter will match first. In case of Google, though, the order is not important
No comments:
Post a Comment