User-agent: * Disallow: /*.pdf$ User-agent: Linguee Disallow: / User-agent: 007ac9 Disallow: / User-agent: 007ac9 Crawler Disallow: / User-agent: 2ADAMbot Disallow: / User-agent: 2ADAMbot/1.0 Disallow: / User-agent: 200PleaseBot Disallow: / User-agent: 360Spider Disallow: / User-agent: &as_qdr=all Disallow: / User-agent: Abonti Disallow: / User-agent: Abonti/0.92 Disallow: / User-agent: aboutthedomain Disallow: / User-agent: Add Catalog Disallow: / User-agent: Add Catalog/2.1 Disallow: / User-agent: AdvBot Disallow: / User-agent: AdvBot/2.0 Disallow: / User-agent: Affectv Robot v1.0/Nutch-1.6 Disallow: / User-agent: Aitellu Disallow: / User-agent: Apache-HttpClient/4.1.1 (java 1.5) Disallow: / User-agent: Baiduspider Disallow: / User-agent: Baidu Spider Disallow: / User-agent: Battleztar Bazinga Disallow: / User-agent: Battleztar Bazinga/0.01 Disallow: / User-agent: betaBot Disallow: / User-agent: Blackboard Safeassign Disallow: / User-agent: Blazer 1.0 Disallow: / User-agent: BLEXBot Disallow: / User-agent: BLEXBot/1.0 Disallow: / User-agent: BLP_bbot Disallow: / User-agent: BLP_bbot/0.1 Disallow: / User-agent: BOIA-Accessibility-agent Disallow: / User-agent: BOIA-Accessibility-agent/PR 1.0 Disallow: / User-agent: BOT for JCE Disallow: / User-agent: BOT/0.1 (BOT for JCE) Disallow: / User-agent: BUbiNG Disallow: / User-agent: ca-crawler Disallow: / User-agent: ca-crawler/1.0 Disallow: / User-agent: CakePHP Disallow: / User-agent: CamelHttpStream/1.0 Evolution/3.2.2 Disallow: / User-agent: CB/Nutch-1.7 Disallow: / User-agent: CCBot Disallow: / User-agent: CCBot/2.0 Disallow: / User-agent: Cliqzbot Disallow: / User-agent: Cliqzbot/0.1 Disallow: / User-agent: CMS Crawler Disallow: / User-agent: coccoc Disallow: / User-agent: Copyscape Disallow: / User-agent: crawler4j Disallow: / User-agent: CrazyWebCrawler Disallow: / User-agent: CrazyWebCrawler-Spider Disallow: / User-agent: curl/7.15.3 Disallow: / User-agent: curl/7.26.0 Disallow: / User-agent: curl/7.32.0 Disallow: / User-agent: curl/7.37.0 Disallow: / User-agent: Curl/PHP 5.3.29 Disallow: / User-agent: cuwhois Disallow: / User-agent: cuwhois/1.0 Disallow: / User-agent: Dispatch/0.11.0 Disallow: / User-agent: DomainAppender Disallow: / User-agent: DomainAppender /1.0 Disallow: / User-agent: DomainSigmaCrawler Disallow: / User-agent: DomainSigmaCrawler/0.1 Disallow: / User-agent: duggmirror Disallow: / User-agent: eCommerceBot Disallow: / User-agent: EPiServer Link Checker Disallow: / User-agent: EuripBot Disallow: / User-agent: EuripBot/2.0 Disallow: / User-agent: Exabot Disallow: / User-agent: Findxbot Disallow: / User-agent: Findxbot/1.0 Disallow: / User-agent: Flamingo_SearchEngine Disallow: / User-agent: Flipboard Robot Disallow: / User-agent: getrequest Disallow: / User-agent: getrequest/1.0 Disallow: / User-agent: GigablastOpenSource Disallow: / User-agent: GigablastOpenSource/1.0 Disallow: / User-agent: Gluten Free Crawler Disallow: / User-agent: Gluten Free Crawler/1.0 Disallow: / User-agent: Go 1.1 package http Disallow: / User-agent: Haiula Disallow: / User-agent: Haiula/1.4 Disallow: / User-agent: HaosouSpider Disallow: / User-agent: heritrix Disallow: / User-agent: heritrix/1.14.4 Disallow: / User-agent: heritrix/3.1.1 Disallow: / User-agent: Hivemind Disallow: / User-agent: http://git.io/tl_S2w Disallow: / User-agent: httpunit/1.x Disallow: / User-agent: HubSpot Links Crawler 1.0 Disallow: / User-agent: HyperCrawl Disallow: / User-agent: HyperCrawl/0.2 Disallow: / User-agent: Ichiro Robot Disallow: / User-agent: image.coccoc/1.0 Disallow: / User-agent: Indy Library Disallow: / User-agent: integrity/5 Disallow: / User-agent: InterNaetBoten Disallow: / User-agent: InterNaetBoten/0.99 Disallow: / User-agent: jakarta Disallow: / User-agent: Jakarta Commons Disallow: / User-agent: Jakarta Commons-HttpClient Disallow: / User-agent: Jakarta Commons-HttpClient/3.0.1 Disallow: / User-agent: Jakarta Commons-HttpClient/3.0-rcx Disallow: / User-agent: Jakarta Commons-HttpClient/3.1 Disallow: / User-agent: JamesBOT Disallow: / User-agent: James BOT Disallow: / User-agent: Java/1.4.1_04 Disallow: / User-agent: Java/1.6.0_04 Disallow: / User-agent: Java/1.6.0_10-rc2 Disallow: / User-agent: Java/1.6.0_15 Disallow: / User-agent: Java/1.6.0_24 Disallow: / User-agent: Java/1.6.0_26 Disallow: / User-agent: Java/1.6.0_32 Disallow: / User-agent: Java/1.7.0_09 Disallow: / User-agent: Java/1.7.0_45 Disallow: / User-agent: Java/1.7.0_55 Disallow: / User-agent: Java/1.7.0_60 Disallow: / User-agent: Java/1.7.0_60-ea Disallow: / User-agent: Java/1.7.0_65 Disallow: / User-agent: Java/1.7.0_67 Disallow: / User-agent: Java/1.7.0_71 Disallow: / User-agent: Java/1.8.0_20 Disallow: / User-agent: Java/1.8.0_25 Disallow: / User-agent: Java/1.8.0_31 Disallow: / User-agent: JetBrains Disallow: / User-agent: JetBrains 5.0 Disallow: / User-agent: Kraken Disallow: / User-agent: Kraken/0.1 Disallow: / User-agent: Kraken/Nutch-2.2.1 Disallow: / User-agent: larbin Disallow: / User-agent: libcurl Disallow: / User-agent: libcurl/7.15.3 Disallow: / User-agent: libcurl/7.19.7 Disallow: / User-agent: libcurl/7.21.0 Disallow: / User-agent: LibreOffice Disallow: / User-agent: Link/1.0 Disallow: / User-agent: Linkdex Disallow: / User-agent: LinkdexBot Disallow: / User-agent: linkdexbot/2.0 Disallow: / User-agent: linkdexbot/2.1 Disallow: / User-agent: linkdex.com/v2.0 Disallow: / User-agent: linkdex.com/v2.1 Disallow: / User-agent: linkdexbot-mobile/2.1 Disallow: / User-agent: LinkpadBot Disallow: / User-agent: LinkpadBot/1.06 Disallow: / User-agent: Lipperhey Disallow: / User-agent: Lipperhey SEO Service Disallow: / User-agent: Lipperhey-Kaus-Australis Disallow: / User-agent: Lipperhey-Kaus-Australis/5.0 Disallow: / User-agent: listicka Disallow: / User-agent: LSSRocketCrawler Disallow: / User-agent: LSSRocketCrawler/1.0 Disallow: / User-agent: LSSRocketCrawler/1.0 LightspeedSystems Disallow: / User-agent: ltx71 Disallow: / User-agent: LWNutch/Nutch-1.4 Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Mail.RU Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: Mail.RU_Bot/2.0 Disallow: / User-agent: Mail.RU_Bot/Fast/2.0 Disallow: / User-agent: MaxPointCrawler Disallow: / User-agent: MaxPointCrawler/Nutch-1.1 Disallow: / User-agent: md5sum Disallow: / User-agent: md5sum\x22 Disallow: / User-agent: meanpathbot Disallow: / User-agent: MegaIndex.ru Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: / User-agent: MixBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: My Nutch Spider/Nutch-1.9 Disallow: / User-agent: NerdyBot Disallow: / User-agent: netseer Disallow: / User-agent: NING/1.0 Disallow: / User-agent: node.io Disallow: / User-agent: node.js Disallow: / User-agent: oBot/2.3.1 Disallow: / User-agent: omgilibot Disallow: / User-agent: omgilibot/0.4 Disallow: / User-agent: Openstat Disallow: / User-agent: Openstat/0.1 Disallow: / User-agent: OptimizationCrawler Disallow: / User-agent: OptimizationCrawler/0.2 Disallow: / User-agent: PageAnalyzer Disallow: / User-agent: PageAnalyzer/1.1 Disallow: / User-agent: PagesInventory Disallow: / User-agent: Pagespeedbot Disallow: / User-agent: Pagespeed/1.1 Disallow: / User-agent: Pagespeed/1.1 Fetcher Disallow: / User-agent: PathDefender Disallow: / User-agent: PathDefender/1.0 Disallow: / User-agent: Perl LWP Disallow: / User-agent: Plukkie Disallow: / User-agent: Plukkie/1.5 Disallow: / User-agent: POGS/2.0 Disallow: / User-agent: PRIVACY_ENFAQ.jsp Disallow: / User-agent: Prlog Disallow: / User-agent: Prlog/1.0 Disallow: / User-agent: psbot Disallow: / User-agent: publiclibraryarchive.org Disallow: / User-agent: publiclibraryarchive.org/1.0 Disallow: / User-agent: pycurl Disallow: / User-agent: Python-urllib Disallow: / User-agent: Python-urllib/1.17 Disallow: / User-agent: Python-urllib/2.5 Disallow: / User-agent: Python-urllib/2.6 Disallow: / User-agent: Python-urllib/2.7 Disallow: / User-agent: R6_FeedFetcher Disallow: / User-agent: R6_CommentReader Disallow: / User-agent: Riddler Disallow: / User-agent: Rome Client Disallow: / User-agent: rootlink Disallow: / User-agent: RU_Bot/2.0 Disallow: / User-agent: Ruby Disallow: / User-agent: Scrapy/0.24.4 Disallow: / User-agent: SeznamBot Disallow: / User-agent: ShowyouBot Disallow: / User-agent: SheerBoredom.Experimental.Robot Disallow: / User-agent: SheerBoredom.Experimental.Robot/0.2 Disallow: / User-agent: sistrix Disallow: / User-agent: SISTRIX Crawler Disallow: / User-agent: SiteExplorer Disallow: / User-agent: SiteExplorer/1.0b Disallow: / User-agent: SkimBot Disallow: / User-agent: SkimBot/1.0 Disallow: / User-agent: SMTBot Disallow: / User-agent: SMTBot/1.0 Disallow: / User-agent: Sogou Pic Spider Disallow: / User-agent: Sogou Pic Spider/3.0 Disallow: / User-agent: Sogou spider Disallow: / User-agent: Sogou web spider Disallow: / User-agent: spbot Disallow: / User-agent: spbot/4.4.2 Disallow: / User-agent: Spiderbot/Nutch-1.7 Disallow: / User-agent: Steeler Disallow: / User-agent: Steeler/3.5 Disallow: / User-agent: StudioFACA Disallow: / User-agent: StudioFACA Search Disallow: / User-agent: sukibot Disallow: / User-agent: sukibot_heritrix Disallow: / User-agent: sukibot_heritrix/3.1.1 Disallow: / User-agent: SurveyBot Disallow: / User-agent: Synthesio Crawler release MonaLisa Disallow: / User-agent: trendictionbot Disallow: / User-agent: TrueBot Disallow: / User-agent: TrueBot/1.0 Disallow: / User-agent: Typhoeus Disallow: / User-agent: uipbot/1.0 Disallow: / User-agent: uMBot-LN Disallow: / User-agent: uMBot-LN/1.0 Disallow: / User-agent: updown_tester Disallow: / User-agent: URLChecker Disallow: / User-agent: WebCookies Disallow: / User-agent: WebCookies/1.0 Disallow: / User-agent: WebCopier vx.xa Disallow: / User-agent: Web corpus crawler Disallow: / User-agent: Web-sniffer Disallow: / User-agent: Web-sniffer/1.1.0 Disallow: / User-agent: Wegtam Disallow: / User-agent: Wegtam Crawler Disallow: / User-agent: WhatWeb Disallow: / User-agent: WhatWeb/0.4.8-dev Disallow: / User-agent: WorldBrewBot Disallow: / User-agent: WorldBrewBot/2.1 Disallow: / User-agent: worldwebheritage.org Disallow: / User-agent: worldwebheritage.org/1.0 Disallow: / User-agent: www.deadlinkchecker.com Disallow: / User-agent: V1.0/1.2 Disallow: / User-agent: vBSEO Disallow: / User-agent: VeriCiteCrawler Disallow: / User-agent: VeriCiteCrawler/Nutch-1.9 Disallow: / User-agent: Voila Robot Disallow: / User-agent: voltron Disallow: / User-agent: Vorboss Web Crawler Disallow: / User-agent: Vorboss Web Crawler/Nutch-2.3 Disallow: / User-agent: Xenu Link Sleuth Disallow: / User-agent: Xenu's Link Sleuth Disallow: / User-agent: XoviBot Disallow: / User-agent: XoviBot/2.0 Disallow: / User-agent: Yandex Disallow: / User-agent: Yandex Robot Disallow: / User-agent: Yetibot Disallow: / User-agent: YisouSpider Disallow: / User-agent: zzabmbot Disallow: / User-agent: zzabmbot/1.0 Disallow: / User-agent: snk Disallow: /