# /robots.txt file for http://webcrawler.com/ # mail webmaster@webcrawler.com for constructive criticism User-agent: * Disallow: /tmp/ Disallow: /logs/ Disallow: /vsadmin/ Disallow: /GeneratedItems/ Disallow: /cgi-bin/ Disallow: /diets/ Disallow: /forum/admin/ Disallow: /forum/includes/ Disallow: /forum/language/ Disallow: /forum/docs/ Disallow: /forum/db/ Disallow: /baghera1389/ Disallow: /phpMySQLAutoBackup/ Sitemap: http://www.ituneon.com/sitemap.xml