February 12, 2004
Weblogs-scanning robot at Deutche Boerse?

It's been a very long time since I did some serious looking at my webserver's log files, but when I finally got around to downloading a batch of them from my ISP and running a bit of webalizer magic on them, I discovered that the last 8 days someone or something running from the IP address 193.29.77.220, disguising as a user of multiple browsers has been pulling down more than 80 Mb of data from my site in more than 5000 hits.

Inspecting the logs more closely, the first hit this month (3rd Feb) is a request for robots.txt. Later the same day, grabbing approximately one page a second, there are requests for an individual archive page, then working the whole way up the directory hierarchy:

GET /anders/blog/archives/2004/02/03/orkut_usability_or_not.html
GET /anders/blog/archives/2004/02/03/
GET /anders/blog/archives/2004/02/
GET /anders/blog/archives/2004/
GET /anders/blog/archives/
GET /anders/blog/
GET /anders/

You get the picture. Not unlike a creative spider looking for content or a user having a look at my site and hence not suspicious in itself.

The WEIRD thing is that each of the requests allegedly come from different browsers off the same IP address. It could be a well-frequented proxy with multiple users running multiple browsers behind it, but seeing that the requests are systematic and in very quick succession, I think someone is trying to camouflage their activity.

The reported browsers from the logs are:
Mozilla/4.7 [en] (X11; I; SunOS 5.8 sun4u)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT)
Mozilla/4.76 [en] (WinNT; U)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Mozilla/5.0 (X11; U; SunOS sun4u; en-US; rv:1.0rc3) Gecko/20020524
Mozilla/4.0 (SunOS 5.8)
Mozilla/4.7 [en] (X11; I; SunOS 5.8 sun4u)
Mozilla/4.61 [en] (OS/2; U)
Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:0.9.8) Gecko/20020204

That's a lot of different browsers and operating systems to use in a 10 second timeframe... (IBM OS/2 (!!), SunOS 5.8 and Windows NT).

It gets weirder: later the same day, exactly the same string of requests as above, same IP, again a whole lot of operating systems and browsers, all in 8 seconds. Further in the day, this repeats again and again. Same files, more and more browsers/OSes:

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.4.1) Gecko/20020508 Netscape6/6.2.3
Opera/6.01 (Windows XP; U) [en]
Mozilla/4.0 (compatible; MSIE 5.0; Windows XP) Opera 6.01 [de]
Mozilla/4.75 [en] (X11; U; SunOS 5.8 sun4u)

All in 4 seconds.

Has anyone seen anything like it? What's going on?

193.29.77.220 is in an IP segment owned by Deutche Boerse, the German stock exchange. ... are they secretly mining blogs for market-relevant data?

If you're the person reading this from 193.29.77.220 / Deutche Börse, please get in touch and explain the activity before I block access to my servers for your user(s) - to me it looks like you're wasting a lot of my bandwith!

Sponsored links
Related Entries
Comments

A lot of browsers and operating systems from one IP address can ofcourse also mean that the IP address is a gateway.

Any of the over 130 employees at Opera Software for example will show up in your logfiles as coming from pat.opera.com (193.69.113.22).

Posted by: Remco on February 12, 2004 06:31 PM

Remco, it's still highly unlikely that visits will come in bursts, where all the Deutsche Boers employees hammer Anders' site within a four second window. :-)

This bot has visited me over the last 8 days as well, but it has behaved much "better", only causing 182 hits. Of which about 70 hits were to my non-existent robots.txt.

Posted by: Arve on February 13, 2004 01:55 AM

I found this entry whilst looking up the same IP hammering my site in the last two weeks or so. The latest is that it now has two IPs doing the same thing - the original 193.29.77.220 and now 193.29.77.221 as well, which comes in 1 second after the former. They only seem to look at the robots.txt file but the annoying thing is that they keep doing it too often. I have since blocked both IPs but they still come anyway.

The reported browsers from the logs in the past two weeks or so has consistently been:

"Mozilla/4.7 [en] (X11; I; SunOS 5.8 sun4u)"

However, today there seems to have been a change on that front... it has now changed to:

"Mozilla/4.0 (Exotic Crawler)"

What on earth is the Deutche Boerse up to?

Posted by: ady on August 26, 2004 06:09 PM

I have the same visitor on my pages for a long time. The headers it sends:

[REMOTE_ADDR: 193.29.77.220]

HTTP-USER-AGENT: Mozilla/4.7 [en] (X11; I; SunOS 5.8 sun4u)
HTTP-ACCEPT: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
HTTP-VIA: ricproxy01.deutsche-boerse.de:8080 (squid/2.5.STABLE6), 1.0 exoprxs1.unix.deutsche-boerse.de:8080 (Squid/2.4.STABLE7)
HTTP-X-FORWARDED-FOR: 172.17.246.217, 193.29.77.204

IP 172.17.246.21 ???

Posted by: Elixon on November 14, 2005 05:01 PM
Post a comment
Name:


Email:
(Will not be displayed if you enter a website below. Otherwise, it will be displayed "spam protected")


Website:
(if you have one)


What do you want to say?
(please don't bother posting "spam" (pornography, viagra-sales etc - I will delete such comments anyway))


Remember info?



Referrers to this page
TrackBack URL for this entry:
http://www.jacobsen.no/cgi-sys/cgiwrap/anders/MT/mt-tb.cgi/731
Das E-Business Weblog: Was sucht die Deutsche Börse im Web? (February 12, 2004 04:12 PM)
"Anders Jacobsen hat etwas Merkwürdiges entdeckt: Anscheinend wandert ein Bot aus dem IP-Raum der Deutschen Börse durchs Web. Auch bei mir ist er aufgetaucht...."
Anders Jacobsen's blog: Deutche Boerse's "Competitive Intelligence" system (February 12, 2004 05:24 PM)
"The previously reported Deutche Börse-originating spider has visited as well, could potentially be a part of their competitive intelligence system ''EXOTIC''. One of Martin Roell's commenters has more info."

33712 visits (84 today, 225 this week)

© Anders Jacobsen
[weblog / photography]