What robots.txt tells you about corporate culture - the case study of Nike and Adidas

Even if you are not a technical person, you may have heard about the apparently arcane robots.txt file which sits on websites but is seen only by computers, not by human visitors. The file contains instructions for search engines on what they can and cannot index and make searchable.

Note that in the examples below # indicates a comment line that is not read by the computer; the other lines provide specific instructions to allow or disallow access to search engines.

Let's look at the robots.txt file on Nike.com - www.nike.com/robots.txt

# *.nike.com robots.txt -- just crawl it.
User-agent: *
Allow: /
Crawl-delay: 20

In contrast, look at the file for Adidas.com - www.adidas.com/robots.txt:

# go away
User-agent:*
Disallow:/scripts/cud/cud2.asp

In fact this means that the entire Adidas.com site except for one file can be crawled by search engines, but the comment does seems to suggest a difference in corporate culture from Nike :-).

(Hattip to @larsv!)