Donate now to keep alive!Help|Log In|Register|
Home PageMindat NewsThe Mindat ManualHistory of MindatCopyright StatusManagement TeamContact UsAdvertise on Mindat
Donate to MindatSponsor a PageSponsored PagesTop Available PagesMindat AdvertisersAdvertise on Mindat
What is a mineral?The most common minerals on earthMineral PhotographyThe Elements and their MineralsGeological TimeMineral Evolution
Minerals by PropertiesMinerals by ChemistryAdvanced Locality SearchRandom MineralRandom LocalitySearch by minIDLocalities Near MeSearch ArticlesSearch GlossaryMore Search Options
Search For:
Mineral Name:
Locality Name:
The Mindat ManualAdd a New PhotoRate PhotosLocality Edit ReportCoordinate Completion ReportAdd Glossary Item
StatisticsThe ElementsMember ListBooks & MagazinesMineral MuseumsMineral Shows & EventsThe Mindat DirectoryDevice Settings
Photo SearchPhoto GalleriesNew Photos TodayNew Photos YesterdayMembers' Photo GalleriesPast Photo of the Day Gallery

Bad performance today - Please blame Statoil!

Jolyon & Katya Ralph September 22, 2011 08:59PM
Hi everyone.

The server wasn't great today - with quite a bit of downtime. On investigating, I found the server was being tied up by a very poorly executed script run by someone at Statoil ( which was trying to download all our mineral pages in quick succession.

Here's a very small segment from the log: - - <22/Sep/2011:20:33:22 +0000> "GET /min-16611.html HTTP/1.1" 200 22024 "-" "MSIE 5.0" - - <22/Sep/2011:20:33:27 +0000> "GET /min-25531.html HTTP/1.1" 200 22191 "-" "MSIE 5.0" - - <22/Sep/2011:20:33:32 +0000> "GET /min-25536.html HTTP/1.1" 200 22916 "-" "MSIE 5.0" - - <22/Sep/2011:20:33:38 +0000> "GET /min-25526.html HTTP/1.1" 200 22588 "-" "MSIE 5.0" - - <22/Sep/2011:20:33:43 +0000> "GET /min-25533.html HTTP/1.1" 200 22307 "-" "MSIE 5.0" - - <22/Sep/2011:20:33:48 +0000> "GET /min-25530.html HTTP/1.1" 200 22129 "-" "MSIE 5.0" - - <22/Sep/2011:20:33:53 +0000> "GET /min-25534.html HTTP/1.1" 200 22662 "-" "MSIE 5.0" - - <22/Sep/2011:20:33:58 +0000> "GET /min-2860.html HTTP/1.1" 200 70600 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:03 +0000> "GET /min-25538.html HTTP/1.1" 200 23405 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:08 +0000> "GET /min-16610.html HTTP/1.1" 200 22014 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:13 +0000> "GET /min-20171.html HTTP/1.1" 200 22184 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:18 +0000> "GET /min-20170.html HTTP/1.1" 200 22176 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:23 +0000> "GET /min-25537.html HTTP/1.1" 200 22116 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:28 +0000> "GET /min-25535.html HTTP/1.1" 200 22041 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:33 +0000> "GET /min-27028.html HTTP/1.1" 200 22003 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:38 +0000> "GET /min-40144.html HTTP/1.1" 200 48002 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:43 +0000> "GET /min-5304.html HTTP/1.1" 200 22175 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:48 +0000> "GET /min-2861.html HTTP/1.1" 200 107805 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:53 +0000> "GET /min-31206.html HTTP/1.1" 200 22121 "-" "MSIE 5.0" - - <22/Sep/2011:20:34:58 +0000> "GET /min-30733.html HTTP/1.1" 200 22697 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:03 +0000> "GET /min-30678.html HTTP/1.1" 200 22961 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:08 +0000> "GET /min-16612.html HTTP/1.1" 200 22147 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:13 +0000> "GET /min-40350.html HTTP/1.1" 200 25908 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:18 +0000> "GET /min-30677.html HTTP/1.1" 200 22961 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:23 +0000> "GET /min-3695.html HTTP/1.1" 200 70189 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:28 +0000> "GET /min-5901.html HTTP/1.1" 200 22079 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:33 +0000> "GET /min-2864.html HTTP/1.1" 200 54065 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:38 +0000> "GET /min-16614.html HTTP/1.1" 200 22044 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:43 +0000> "GET /min-2862.html HTTP/1.1" 200 53391 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:48 +0000> "GET /min-16615.html HTTP/1.1" 200 22101 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:53 +0000> "GET /min-3694.html HTTP/1.1" 200 77867 "-" "MSIE 5.0" - - <22/Sep/2011:20:35:58 +0000> "GET /min-30405.html HTTP/1.1" 200 22973 "-" "MSIE 5.0" - - <22/Sep/2011:20:36:03 +0000> "GET /min-16616.html HTTP/1.1" 200 22127 "-" "MSIE 5.0" - - <22/Sep/2011:20:36:08 +0000> "GET /min-2863.html HTTP/1.1" 200 65462 "-" "MSIE 5.0" - - <22/Sep/2011:20:36:13 +0000> "GET /min-31910.html HTTP/1.1" 200 18577 "-" "MSIE 5.0" - - <22/Sep/2011:20:36:34 +0000> "GET /min-31669.html HTTP/1.1" 200 18577 "-" "MSIE 5.0" - - <22/Sep/2011:20:36:24 +0000> "GET /min-29814.html HTTP/1.1" 200 18577 "-" "MSIE 5.0" - - <22/Sep/2011:20:36:18 +0000> "GET /min-16617.html HTTP/1.1" 200 18577 "-" "MSIE 5.0"

Statoil are now banned from so hopefully the problem will go away. Sorry for the inconvenience.

Peter Andresen September 22, 2011 09:25PM
I promise, I shal not fill gasoline at statoil tomorrow or next time!
Jolyon & Katya Ralph September 22, 2011 09:35PM
This wasn't the ONLY problem, but it was a contributing factor.

Robert Knox September 22, 2011 11:44PM
Always nice to know that it's not some malware problem on our computers.
Paul Brandes September 22, 2011 11:44PM
Statoil????? That's interesting......
Scott L. Ritchie September 23, 2011 02:48AM
Let's see here, Statoil gives the University of Texas five million dollars to further education, yet allow their IT gang to pilfer data from mindat on the sly... and they won't purchase a single advert here or make a simple donation to help keep the server up and running?

Rock Currier September 23, 2011 05:23AM
Why don't you send them, or rather the president of the company an email and ask him why they were trying to hack data from mindat and inform him that the effort nearly crashed the site and invite him to send some money our way.

Rock Currier
Crystals not pistols.
Jolyon & Katya Ralph September 23, 2011 07:34AM
I have already passed on the information to the network abuse team at Statoil and it's now up to them to investigate and take action internally. Their own firewall logs should be able to identify where this came from within their organization.
Marcus Voigt September 23, 2011 09:39AM
There are probably many other companies .........

so much valuable information !

Why make a donation to Mindat, if the information can be stolen?

In the end, nobody carries the responsibility for it.

Homo hominis Lupus
David Bernstein September 23, 2011 10:45AM
Isn't that "homini" Marcus? Regardless, a fine expression to use.
Van King September 23, 2011 11:46AM
Why not send them a bill? Disruption of a day's service should be worth 5000 Euros.

Best Wishes, Van King
Alf Olav Larsen September 23, 2011 06:21PM
I am a Statoil employee, and noticed this morning that access to Mindat was forbidden through my job computer. Now I understand why. I am terribly sorry that someone in the company has hacked the site. Hopefully it will never happen again.

Alf Olav Larsen
Jolyon & Katya Ralph September 23, 2011 10:40PM
Alf, very sorry you were inconvenienced by this problem - I'm waiting to hear back from Statoil's network people - once they have assured me the problem has been dealt with then access to the statoil network will be restored.

Lloyd Llewellyn September 30, 2011 04:28PM
It's very surprising to me that anyone at Statoil has any time to do this - they should normally all be too busy following procedures and management systems! ;)


Norfolk, England
Debbie Woolf September 30, 2011 11:44PM
Is this the reason why I'm finding mindat slow to respond these past few nights ?
Douglas Merson October 01, 2011 01:48AM
Here in the Pacific Northwest it was extremely slow until 1500 when it died. Back and not as slow as earlier in the day but not as fast as on a normal day.

Corie Mattar October 01, 2011 04:28AM
Someone actually wrote this script to do exactly what it did.
And they're using IE 5.0??
Where can I send my resume to these people?
Obviously they need someone in their tech department from this century! >:D<

Jolyon & Katya Ralph October 01, 2011 08:15AM
the IE5.0 is a fake string designed to hide they are using a robot to grab stuff. Just not hiding it very well.
Jolyon & Katya Ralph October 01, 2011 09:55AM
I didn't hear back from Statoil at all.
Van King October 01, 2011 11:01PM
As Statoil has been formally notified and there is still recurring denial of service due to running scripts, then their failure to respond indicates at least an unconcern if not a more pernicious attitude. I still suggest that a bill should be sent to Statoil for actions now tolerated by their management. Maybe 5,000 Euoros/day is not enough?

Best Wishes, Van King
Jolyon & Katya Ralph October 02, 2011 09:46AM
The script at statoil stopped running not long after the ban was put in place.

Spencer Ivan Mather October 02, 2011 02:04PM
I agree with Van King, send Statoil a big bill, that should stop whoever it is!
David Bernstein October 02, 2011 02:32PM
It may be an obvious point but without substantiation of damages in place behind a bill of any amount, such a "delivery" would be laughed at. Just sayin' in the event Van's idea gains traction.
Daniel Levesque October 02, 2011 10:35PM
I'd hope the appropriate measures (ban) would remain in place if there was no response from Statoil's IT staff. Obviously they have an internal issue to deal with:
From "Statoil Ethics- Code of Conduct- version 5"
"4.9 ...Any downloading, storing or disseminating in breach of any copyright law or provision is prohibited."

Sounds like they have a rotten apple in the bunch!!!
Tony Albini October 02, 2011 11:02PM

For the past two days, I cannot look at the Today's Mineral Photos. I keep getting a prompt to Register which I have been for a while. Even when I follow the site's prompt, I cannot get in to look at the photos which are so informing. I hope this issue is related to your emails.


Tony Albini
Modris Baum October 03, 2011 12:18AM
Tony - you have to log in as well. According to the Member Directory you last "visited" (i.e. logged in) on Feb 10, 2010)
Rock Currier October 03, 2011 01:22AM
There are still times when Mindat is very slow, and last night there were times when Google chrome could not locate the website at all. Again!

Rock Currier
Crystals not pistols.
Debbie Woolf October 03, 2011 10:18AM
I got this message using chrome the other night "Fatal Error - Unable to connect to database."

Van King October 03, 2011 02:28PM
The real purpose to sending a bill to Statoil is not to get money, but to get their attention. A bill sent to the accounting department and to the president should receive some notice. From the various messages, it is not always clear that the Statoil script remains the only problem. Is it a coincidence or is it the same problem?

Best Wishes, Van King
Jolyon & Katya Ralph October 03, 2011 02:32PM
The statoil script stopped running shortly after I banned them. Statoil are not causing any direct problems to mindat currently
Rick Dalrymple October 10, 2011 06:35PM
I more often than not have a problem getting on Mindat. I get the "Fatal Error-Unable to Connect to Database" a lot. Sometimes man times a day.

I am also getting the You must login to view these pages" which is ok but then I have to log out to go to a different page and then log in again.

I just thought it was my computer/lines/modem.

I know I am in my own little world, but everyone knows me here.
Jolyon & Katya Ralph December 27, 2012 04:24PM
Statoil are back to their bad old tricks and trying again to steal content from mindat with bots.

I have had to block them again.

Dean Allum December 27, 2012 05:31PM
This problem with Statoil appears to be clearly in contradiction to their stated policy shown below. The mindat website audience would normally be one of Statoil's staunchest public supporters, so their actions are mystifying.

Statoil Ethics Code of Conduct Version 2:
"4.9 Information and IT systems
The individual’s use of information, IT systems and,in particular, internet services must be based on the
needs of the business and not by personal interests.
Information produced and stored on Statoil’s IT systems is regarded as the Group’s property.
Statoil therefore reserves the right to access all such information except where limited by law or
agreement. The individual is responsible for maintaining electronic files and archives in an orderly manner.
Private use is only permitted to a limited extent. Information that may be considered illegal, offensive
or inappropriate must not be processed, downloaded, stored or disseminated. Any downloading, storing
or disseminating in breach of any copyright law or provision is prohibited.
Any use of software in breach
of any copyright law or provision is prohibited."
Keith Compton December 27, 2012 11:46PM
Hi J

Perhaps a more permanent ban is required

Jenna Mast December 28, 2012 04:53AM
I guess someone at statoil really wants the entire mindat database for Christmas. I can relate. I often worry "What if someday Mindat should be no more? Then how will I identify my mineral specimens?" May that day never come.
Chris Mavris December 28, 2012 09:56AM
If they keep acting like this, I would consider a legal action.
David DeCourcey December 28, 2012 01:26PM
Rather than banning them or taking legal action, ask them to start sponsering a few pages.
Reiner Mielke December 28, 2012 02:50PM
Hello David,

Why should they sponsor a few pages? You are assuming they have a conscience, which has not been thus far demonstrated.
Branko Rieck December 28, 2012 04:00PM
Why just a few? I'd say they sponsor every page they've downloaded (that is not already sponsored). That would make life easier for Jolyon...
David Von Bargen December 28, 2012 04:20PM
Well, it probably isn't anything that Statoil is officially doing. Probably just a bored employee. If I was on their IT staff, I sure wouldn't want someone loading a spider bot on my network.
Aaron Cross December 31, 2012 07:41PM
Didn't you have a big problem with them earlier??
Sorry, only registered users may post in this forum.

Click here to login

Mineral and/or Locality is an outreach project of the Hudson Institute of Mineralogy, a 501(c)(3) not-for-profit organization.
Copyright © and the Hudson Institute of Mineralogy 1993-2018, except where stated. relies on the contributions of thousands of members and supporters.
Privacy Policy - Terms & Conditions - Contact Us Current server date and time: January 22, 2018 04:07:56
Go to top of page