Bad performance today - Please blame Statoil!

Hi everyone.

The server wasn't great today - with quite a bit of downtime. On investigating, I found the server was being tied up by a very poorly executed script run by someone at Statoil (statoil.com) which was trying to download all our mineral pages in quick succession.

Here's a very small segment from the log:

ibv2.statoil.com - - [22/Sep/2011:20:33:22 +0000] "GET /min-16611.html HTTP/1.1" 200 22024 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:33:27 +0000] "GET /min-25531.html HTTP/1.1" 200 22191 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:33:32 +0000] "GET /min-25536.html HTTP/1.1" 200 22916 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:33:38 +0000] "GET /min-25526.html HTTP/1.1" 200 22588 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:33:43 +0000] "GET /min-25533.html HTTP/1.1" 200 22307 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:33:48 +0000] "GET /min-25530.html HTTP/1.1" 200 22129 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:33:53 +0000] "GET /min-25534.html HTTP/1.1" 200 22662 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:33:58 +0000] "GET /min-2860.html HTTP/1.1" 200 70600 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:03 +0000] "GET /min-25538.html HTTP/1.1" 200 23405 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:08 +0000] "GET /min-16610.html HTTP/1.1" 200 22014 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:13 +0000] "GET /min-20171.html HTTP/1.1" 200 22184 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:18 +0000] "GET /min-20170.html HTTP/1.1" 200 22176 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:23 +0000] "GET /min-25537.html HTTP/1.1" 200 22116 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:28 +0000] "GET /min-25535.html HTTP/1.1" 200 22041 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:33 +0000] "GET /min-27028.html HTTP/1.1" 200 22003 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:38 +0000] "GET /min-40144.html HTTP/1.1" 200 48002 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:43 +0000] "GET /min-5304.html HTTP/1.1" 200 22175 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:48 +0000] "GET /min-2861.html HTTP/1.1" 200 107805 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:53 +0000] "GET /min-31206.html HTTP/1.1" 200 22121 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:34:58 +0000] "GET /min-30733.html HTTP/1.1" 200 22697 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:03 +0000] "GET /min-30678.html HTTP/1.1" 200 22961 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:08 +0000] "GET /min-16612.html HTTP/1.1" 200 22147 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:13 +0000] "GET /min-40350.html HTTP/1.1" 200 25908 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:18 +0000] "GET /min-30677.html HTTP/1.1" 200 22961 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:23 +0000] "GET /min-3695.html HTTP/1.1" 200 70189 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:28 +0000] "GET /min-5901.html HTTP/1.1" 200 22079 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:33 +0000] "GET /min-2864.html HTTP/1.1" 200 54065 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:38 +0000] "GET /min-16614.html HTTP/1.1" 200 22044 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:43 +0000] "GET /min-2862.html HTTP/1.1" 200 53391 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:48 +0000] "GET /min-16615.html HTTP/1.1" 200 22101 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:53 +0000] "GET /min-3694.html HTTP/1.1" 200 77867 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:35:58 +0000] "GET /min-30405.html HTTP/1.1" 200 22973 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:36:03 +0000] "GET /min-16616.html HTTP/1.1" 200 22127 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:36:08 +0000] "GET /min-2863.html HTTP/1.1" 200 65462 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:36:13 +0000] "GET /min-31910.html HTTP/1.1" 200 18577 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:36:34 +0000] "GET /min-31669.html HTTP/1.1" 200 18577 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:36:24 +0000] "GET /min-29814.html HTTP/1.1" 200 18577 "-" "MSIE 5.0"
ibv2.statoil.com - - [22/Sep/2011:20:36:18 +0000] "GET /min-16617.html HTTP/1.1" 200 18577 "-" "MSIE 5.0"

Statoil are now banned from mindat.org so hopefully the problem will go away. Sorry for the inconvenience.

Jolyon

I promise, I shal not fill gasoline at statoil tomorrow or next time!

This wasn't the ONLY problem, but it was a contributing factor.

Jolyon

Always nice to know that it's not some malware problem on our computers.

Statoil????? That's interesting......

Let's see here, Statoil gives the University of Texas five million dollars to further education, yet allow their IT gang to pilfer data from mindat on the sly... and they won't purchase a single advert here or make a simple donation to help keep the server up and running?

Scott

Why don't you send them, or rather the president of the company an email and ask him why they were trying to hack data from mindat and inform him that the effort nearly crashed the site and invite him to send some money our way.

Rock Currier
Crystals not pistols.

I have already passed on the information to the network abuse team at Statoil and it's now up to them to investigate and take action internally. Their own firewall logs should be able to identify where this came from within their organization.

There are probably many other companies .........

so much valuable information !

Why make a donation to Mindat, if the information can be stolen?

In the end, nobody carries the responsibility for it.

Homo hominis Lupus

Isn't that "homini" Marcus? Regardless, a fine expression to use.

Why not send them a bill? Disruption of a day's service should be worth 5000 Euros.

Best Wishes, Van King

I am a Statoil employee, and noticed this morning that access to Mindat was forbidden through my job computer. Now I understand why. I am terribly sorry that someone in the company has hacked the site. Hopefully it will never happen again.

Regards,
Alf Olav Larsen

Alf, very sorry you were inconvenienced by this problem - I'm waiting to hear back from Statoil's network people - once they have assured me the problem has been dealt with then access to the statoil network will be restored.

Jolyon

It's very surprising to me that anyone at Statoil has any time to do this - they should normally all be too busy following procedures and management systems! ;)

Lloyd

Norfolk, England

Is this the reason why I'm finding mindat slow to respond these past few nights ?

Here in the Pacific Northwest it was extremely slow until 1500 when it died. Back and not as slow as earlier in the day but not as fast as on a normal day.

Doug

Someone actually wrote this script to do exactly what it did.
And they're using IE 5.0??
Where can I send my resume to these people?
Obviously they need someone in their tech department from this century! >:D<

Corie

the IE5.0 is a fake string designed to hide they are using a robot to grab stuff. Just not hiding it very well.

I didn't hear back from Statoil at all.

As Statoil has been formally notified and there is still recurring denial of service due to running scripts, then their failure to respond indicates at least an unconcern if not a more pernicious attitude. I still suggest that a bill should be sent to Statoil for actions now tolerated by their management. Maybe 5,000 Euoros/day is not enough?

Best Wishes, Van King