User-agent: crawl-it Allow: / User-agent: AboutUsBot disallow: / User-agent: * disallow: /