80 - 2 AltaVista-Intranet/V2.3A (www.altavista.co.uk jan.gelin@av.com) 2 appie/1.1 131 ArchitextSpider 10 Atomz/1.0 1 AVSearch-3.0(big/big) 1 beholder 15 Crawler V 0.2.1 admin@crawler.de 4 Crescent Internet ToolPak HTTP OLE Control v.1.0 1 CyberSpyder Link Test/2.1.8 1 DaviesBot 1.7 (Licensed to WholeWeb.net, Inc.) 7 DaviesBot/1.7 (www.wholeweb.net) 1 DIIbot/0.4 1 DIIbot/0.5 46 DIIbot/1.0 7 DIIbot/1.1, www.findsame.com, robot@digital-integrity.com 4 DIIbot/1.2, robot@digital-integrity.com 8 e-sense 1.0 ea(www.vigiltech.com/esensedisclaim.html) 3 Excalibur Internet Spider V6.5.4 3 FAST-WebCrawler/2.0.10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 17 FAST-WebCrawler/2.0.9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 2 FAST-WebCrawler/2.1-pre10 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 5 FAST-WebCrawler/2.1-pre11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 7 FAST-WebCrawler/2.1-pre13 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 1 FAST-WebCrawler/2.1-pre14 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 1 FAST-WebCrawler/2.1-pre5 (oyvinda@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 2 FAST-WebCrawler/2.1-pre7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 3 FAST-WebCrawler/2.2-pre1 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 9 FAST-WebCrawler/2.2-pre10 (Multimedia Search) (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 4 FAST-WebCrawler/2.2-pre11 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 3 FAST-WebCrawler/2.2-pre19 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 1 FAST-WebCrawler/2.2-pre2 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 3 FAST-WebCrawler/2.2-pre20 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 3 FAST-WebCrawler/2.2-pre24 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 5 FAST-WebCrawler/2.2-pre27 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 3 FAST-WebCrawler/2.2-pre30 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 1 FAST-WebCrawler/2.2-pre5 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 2 FAST-WebCrawler/2.2-pre8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 5 FAST-WebCrawler/2.2-pre9 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 3 FAST-WebCrawler/2.2-pre9 (Multimedia Search) (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) 5 FastCrawler 3.0 (crawler@1klik.dk) 1 gazz/1.0 (gazz@nttrd.com) 4 geckobot 2 GentleSpider/0.1 2 GentleSpider/1.0 2 GigaBazVStheWeb crawler@brainbot.com 19 Googlebot/1.0 (googlebot(at)googlebot.com) 5 Googlebot/2.0 (+http://googlebot.com/bot.html) 17 Googlebot/2.0 beta (googlebot(at)googlebot.com) 27 Googlebot/2.1 (+http://googlebot.com/bot.html) 28 Googlebot/2.1 (+http://www.googlebot.com/bot.html) 1 Googlebot/2.1d (+http://googlebot.com/bot.html) 1 Googlebot/2.1d (+http://www.googlebot.com/bot.html) 15 Googlebot/2.1w (+http://googlebot.com/bot.html) 11 griffon/1.0 (griffon@super.navi.ocn.ne.jp) 2 Gulliver/1.2 133 Gulliver/1.3 1 htdig/3.1.5 (unconfigured@htdig.searchengine.maintainer) 7 Hubater (http://www.hubat.com) 4 HyperRobot 66 ia_archiver 166 InfoSeek Sidewinder/0.9 5 Intel ATMis 3 InternetArchive-1.0 23 ip3000.com-crawler 8 Jack 2 Jack 0-0 9 Java1.2.1 59 Java1.2.2 12 JennyBot/0.1 17 Katriona 4 Katriona (compatible; MSIE 4.01; Windows 95) 3 Katriona (compatible; MSIE 4.0; WINDOWS NT) 2 KE_1.0/2.0 libwww/5.2.8 91 KIT-Fireball/2.0 1 KukuVeKiki/1.0 5 larbin ghi@lcs.mit.edu 3 larbin ira@iatlas.com 4 larbin sebastien.ailleret@inria.fr 23 LEIA/2.90 6 libwww-perl/5.33 11 libwww-perl/5.44 4 libwww-perl/5.48 8 Links2Go Similarity Engine 23 LinkWalker 2 logikabot 1 looksmart_urls-1.0 199 Lycos_Spider_(T-Rex) 5 Marvin 134 Mercator-1.0 11 Mercator-1.1 14 Mercator-2.0 2 Modified Links2Go Similarity Engine 11 MOMspider/1.00 libwww-perl/0.40 8 Mothra/126-Paladium 39 Mozilla 4.0/(compatible; MSIE 5.0; Windows 98) 20 Mozilla/2.0 (compatible; EZResult -- Internet Search Engine acurtis@directhit.com) 8 Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) 1 Mozilla/3.0 (compatible; Webinator-cigweb01.ge.com/2.54; "I'll be back") 1 Mozilla/3.0 (compatible; Webinator-nhweb01corpge/2.56; "I'll be back") 18 Mozilla/3.0 (compatible; Win95) 13 Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) 21 Mozilla/4.0 (compatible; MSIE 4.01; MSIECrawler; Windows 95) 20 Mozilla/4.0 (compatible; MSIE 4.01; MSIECrawler; Windows NT) 3 Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Microsoft 1 Mozilla/4.0 (compatible; MSIE 4.02; Windows 98) 51 Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) 1 Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ecom 1 Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) GETIN 1 Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) mmcsb 1 Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Teleservizi S.r.L. 2 Mozilla/4.0 (compatible; MSIE 5.01; MSN 2.5; Windows 98; winusermag39; MSIECrawler) 2 Mozilla/4.0 (compatible; MSIE 5.01; Windows 95; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 95; QXW0332o; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; AO1.0; MSIECrawler) 2 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; FREELER1; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; IDG.pl; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; Mozilla/4.0 (compatible; MSIE 5.0; Windows 95/98/NT; GoConnect Australia Pty Ltd); MSIECrawler) 27 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; MSIECrawler) 2 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; OpsPack Custom IE5, February 2000; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; pi 3.0.0.0; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; QXW0330m; QXW03301; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; QXW0332c; MAX1.0; MSIECrawler) 6 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; QXW0332c; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; QXW0332f; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; QXW0332z; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; QXW03332; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; QXW0333a; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; QXW03342; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows 98; WanadooNL; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; Hotbar 2.0; MSIECrawler) 110 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MSIECrawler) WebWasher 2.0.1 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; NetCaptor 6.02P; MSIECrawler) 2 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; WOL-2.03; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT; Mozilla/4.0 (compatible; MSIE 4.01; Windows NT); MSIECrawler) 26 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT; QXW0332c; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT; QXW0333a; MSIECrawler) 4 Mozilla/4.0 (compatible; MSIE 5.01; Windows NT; TUCOWS Network; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.0; MSN 2.5; Windows 98; DigExt) 2 Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) 8 Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; DigExt) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; DigExt; QXW0331s) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; DT; DigExt) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; QXW03003; QXW03014) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; QXW0300g) 4 Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; QXW0300R; DT; DigExt) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; QXW0330d) 2 Mozilla/4.0 (compatible; MSIE 5.0; Windows 95; QXW0600M; DigExt) 10 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; AT&T WNS IE4.0; DigExt) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; Compaq; DigExt) 203 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt) WebWasher 2.0.1 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; AT&T WNS5.2) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; by IndiaTimes) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; KITV4.6 Wanadoo) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; Madasafish) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; Mannesmann Arcor) 6 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; pi 3.0.0.0) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; QXW0330x) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; Virgo Genie V 1.0; genie1.1) 2 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; ZDPORTUGAL) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; QXW0300R; DT; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; QXW03302) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; QXW0330d) 34 Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; AltaVista 1.01.01) 3 Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; QXW0330v) 2 Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; QXW03314) 1 Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; QXW0332c; DigExt) 1 Mozilla/4.0 (compatible; MSIE 5.5; MSNIA; Windows 98; MSNIA; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; OTCD; MSIECrawler) 3 Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; DT; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Hotbar 2.0; MSIECrawler) 23 Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; QXW03302; pbc4.0.0.0; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; QXW03313; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; QXW0333x; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SurfEU; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; TUCOWS; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; WebRide.com; QXW0333s; MSIECrawler) 10 Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler) 1 Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; Hotbar 2.0; MSIECrawler) 40 Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; MSIECrawler) 9 Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; MSIECrawler) 15 Mozilla/4.0 (compatible; MSIE; ZyBorg; Win32) 18 Mozilla/4.0 (compatible; MuscatFerret/1.7; http://www.webtop.com/) 7 Mozilla/4.0 (compatible; MuscatFerret/2.0; http://www.webtop.com/) 1 Mozilla/4.0 (compatible; www.linkguard.com Linkguard Online 1.0; Windows NT) 21 Mozilla/4.06 [en] (Direct Hit Grabber) 1 Mozilla/4.5 (compatible; HTTrack 2.0x; Windows 98) 1 Mozilla/4.51 [en] (X11; I; Linux 2.2.5-15 i686) 1 Mozilla/4.7 [en] (WinNT; I) 4 Mozilla/5.0 (compatible; MSIE 5.0; Windows 95) 3 MultiText/0.1 3 NetMechanic V2.0 7 NG/1.0 2 NPBot X/0.0 3 null; 5 oBot ((compatible;Win32)) 1 pavuk/0.9pl27 i686-pc-linux-gnu 1 PintaSpider 2 RealNamesBot/2.0 11 roach.smo.av.com-1.0 1 ru-robot/0.1 6 ru-robot/0.1 [hseo@cs.rutgers.edu] 31 ru-robot/0.1_hseo(at)cs.rutgers.edu 13 ru-robot/1.0 [hseo@cs.rutgers.edu] 53 Scooter-1.0 14 Scooter/1.0 93 Scooter/2.0 G.R.A.B. V1.1.0 1 Scooter2_Mercator_1-1.0 4 Scooter2_Mercator_2-1.0 8 Scooter2_Mercator_3-1.0 3 Scooter2_Mercator_4-1.0 2 Scooter2_Mercator_6-1.0 4 Scooter2_Mercator_url-1.0 5 Scooter_Mercator_1-1.0 1 SearchExpress Spider 0.99c obj 1 SiteSweeper 2.0 197 Slurp.so/1.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html) 2 Slurp/2.0-BigOwlWeekly (spider@aeneid.com; http://www.inktomi.com/slurp.html) 8 Slurp/2.0-GreatWhiteCrawl (slurp@inktomi.com; http://www.inktomi.com/slurp.html) 230 Slurp/2.0-KiteHourly (slurp@inktomi.com; http://www.inktomi.com/slurp.html) 129 Slurp/2.0-KiteWeekly (slurp@inktomi.com; http://www.inktomi.com/slurp.html) 1 Slurp/2.0-OwlWeekly (spider@aeneid.com; http://www.inktomi.com/slurp.html) 2218 Slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html) 1 SmartProxy/1.0 (DSLab E.E. NCKU) 23 Speedy Spider ( http://www.entireweb.com/speedy.html ) 7 Spider/MaxBot.com admin@maxbot.com 1 Spider/maxbot.com admin@maxbot.com 13 Spinne/2.0 3 SyncIT/2.0 2 TScooter_ifr2-1.0 1 TV33_Mercator_0-1.0 4 TV33_Mercator_1-1.0 3 TV33_Mercator_2-1.0 2 TV33_Mercator_3-1.0 1 TV33_Mercator_n0b-1.0 4 TV33_Mercator_n1b-1.0 2 TV33_Mercator_n2b-1.0 5 TV33_Mercator_n2s1_A-1.0 3 tv33_Merc_ep-1.0 5 tv33_Merc_tch-1.0 4 TV34_Mercator_4-1.0 4 TV34_Mercator_5-1.0 1 TV34_Mercator_5b-1.0 1 TV34_Mercator_6-1.0 2 TV34_Mercator_7-1.0 2 TV34_Mercator_dmb-1.0 1 TV34_Mercator_n2s2_A-1.0 2 TV34_Mercator_n2s3_A-1.0 4 TV35_Mercator_4444-1.0 3 TV35_Mercator_5-1.0 5 TV35_Mercator_6-1.0 6 TV35_Mercator_7-1.0 1 TV35_Mercator_n0d-1.0 2 TV35_Mercator_n2c-1.0 2 TV35_Mercator_n2s5_A-1.0 1 TV35_Mercator_n3c-1.0 1 TV35_Mercator_n3d-1.0 4 TV36_Mercator_10-1.0 3 TV36_Mercator_9-1.0 1 TV36_Mercator_f5-1.0 4 TV36_Mercator_n1c-1.0 1 TV36_Mercator_n1d-1.0 2 TV36_Mercator_n2s7_A-1.0 3 TV36_Mercator_n6c-1.0 1 TV36_Mercator_n6d-1.0 1 TV36_Mercator_n7b-1.0 1 TV36_Mercator_n7c-1.0 1 TV36_Merc_mv-1.0 1 TV36_Merc_sy-1.0 1 Ultraseek 1 Unlost Web Crawler 2.0.1.4 2 user0.ip3000.com 1 user2.ip3000.com 2 user21.ip3000.com 2 user22.ip3000.com 1 user29.ip3000.com 1 user34.ip3000.com 1 user39.ip3000.com 1 user4.ip3000.com 1 user6.ip3000.com 1 vspider/3.6 39 WebFountain 1.0 (408-927-1838) 5 Webmonkey/0.2 NoComment 2 WebReaper - www.otway.com/webreaper 1 WebReaper v8.1 - www.otway.com/webreaper 10 Webshuttle 3 Wget/1.5.3 1 WhizBang! Lab 6 www.WebWombat.com.au 4 xyro 2 xyro_(xcrawler@cosmos.inria.fr) 19 xyro_(xcrawler@inria.fr) 2 ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com) 22 ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)