# Advertising and ad-verification crawlers User-agent: Mediapartners-Google Allow: / User-agent: AdsBot-Google Allow: / User-agent: AdsBot-Google-Mobile Allow: / User-agent: Google-Display-Ads-Bot Allow: / User-agent: CriteoBot/0.1 Disallow: Crawl-delay: 5 User-agent: AmazonAdBot Allow: / User-agent: TTD-Content Allow: / User-agent: Bidswitchbot Allow: / User-agent: OpenXAdsTxtCrawler Allow: / User-agent: PubMatic Crawler Bot Allow: / User-agent: IAS_crawler Allow: / User-agent: IAS_admantx Allow: / User-agent: IAS_wombles Allow: / User-agent: DoubleVerifyBot Allow: / User-agent: DVBot Allow: / User-agent: Leikibot Allow: / User-agent: Peer39_crawler Allow: / User-agent: Proximic Allow: / User-agent: proximic Allow: / User-agent: SinceraSyntheticUser Allow: / User-agent: SlickBot Allow: / User-agent: Gumgum Allow: / User-agent: Grapeshot Allow: / User-agent: Opebot-v Allow: / # Default rules for general crawlers User-agent: * Disallow: /verify/ Disallow: /certificate/ Disallow: /certificates/ Disallow: /glossary/ Disallow: /wp-admin/ Disallow: /search Disallow: /cart Disallow: /checkout Disallow: /admin Disallow: /?s= Disallow: /*?wordfence_ Disallow: /detroitchicago/ Disallow: /porpoiseant/ Disallow: /beardeddragon/ Disallow: /tardisrocinante/ Disallow: /parsonsmaize/ Disallow: /edomontonalberta/ Disallow: /ezais/ Allow: /wp-admin/admin-ajax.php # Block common SEO tool bots User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: BLEXBot Disallow: / User-agent: SEOkicks-Robot Disallow: / User-agent: spbot Disallow: / User-agent: CCBot Disallow: / User-agent: crawler4j Disallow: / User-agent: Rogerbot Disallow: / User-agent: YandexBot Disallow: / User-agent: Sogou Spider Disallow: / User-agent: Baiduspider Disallow: / Sitemap: https://carelearning.org.uk/sitemap.xml