User-agent: Googlebot Crawl-delay: 120 Disallow: /bin/ Disallow: /per/ Disallow: /stats/ Disallow: /images/ Disallow: /feed/ Disallow: /xml/ Disallow: /*.pdf$ Disallow: /*.axd$ User-agent: Slurp Crawl-delay: 3600 Disallow: /bin/ Disallow: /per/ Disallow: /stats/ Disallow: /images/ Disallow: /feed/ Disallow: /xml/ Disallow: /*.pdf$ Disallow: /*.axd$ User-agent: msnbot Crawl-delay: 2400 Disallow: /bin/ Disallow: /per/ Disallow: /stats/ Disallow: /images/ Disallow: /feed/ Disallow: /xml/ Disallow: /*.pdf$ Disallow: /*.axd$ User-agent: Teoma Crawl-delay: 1200 Disallow: /bin/ Disallow: /per/ Disallow: /stats/ Disallow: /images/ Disallow: /feed/ Disallow: /xml/ Disallow: /*.pdf$ Disallow: /*.axd$ User-agent: * Disallow: / #Disallow List User-agent: baiduspider Disallow: / User-agent: ia_archiver Disallow: / User-agent: Yandex Disallow: / User-agent: Spinn3r Disallow: /