15 govt websites down after failed IT upgrade

LATEST NEWS

Computerworld is New Zealand's only specialised information systems fortnightly.
Subscribe now for $100 (23 issues) and save more than 37% off the cover price!

Get the latest news from Computerworld delivered via email.
Sign up now

Updated 12pm Tuesday: Parliament website suffers outage Tuesday morning; MED says Personal Property Securities Register closed due to Monday's outage

By Sarah Putt | Auckland | Monday, 26 September, 2011 | 25 Comments

Fifteen government websites - including Companies Office, Patents Office and Consumer Affairs - are down following a failed IT upgrade on Saturday.

A notice on the Ministry of Economic Development website says the outage occured during preparation work to upgrade the servers hosting the websites. It originally contained notice of 14 websites, but an additional website Business Data has been added (see list below).

The notice advises the websites will be unavailable until at least midday Wednesday September 28. It suggests a variety of alternative ways for people to access the services.

The websites are:

business.govt
Companies Office
Ministry of Consumer Affairs
Financial Service Providers Register (FSPR)
G2B
Government Procurement Solutions
Insolvency and Trustee Service
Intellectual Property Office (IPONZ)
Motor Vehicle Traders Register
New Zealand Petroleum & Minerals
Personal Property Securities Register (PPSR)
Radio Spectrum Management
Societies and Trusts
VolunteerNet
Business Data

MED spokesperson Alastair Stewart says the reason for the delay in restoring the websites is that it will involve "rebuilding server systems". New equipment had been ordered to replace the servers. In May an outage of the Companies Office website was put down to a hardware issue.

"The maintenance of our business services websites and registries is contracted to Foster Moore. We own the affected IT hardware," he says, via email.

IT lawyer Rick Shera says of all the website outages, the one that is most concerning is the Personal Property Securities Register as those wanting to lend money to a business can't check to see what securities the company has given to another financier. A freephone enquiry number is supplied but the Register is closed until the website is up and running again.

"The Registrar has temporarily suspended access to the Personal Property Securities Register under section 138 of the Personal Property Securities Act. As soon as the Register is available again, customers should register their security interests and conduct any searches they need to," says Stewart, via email.

"We cannot access the register, so are unable to provide financing statements over the phone."

"We apologise for any inconvenience caused and are working hard to restore our services."

Parliament's website experienced a brief outage [on Tuesday] morning at about 8:30am. The cause is still to be ascertained. "The event caused our backup server to come on line and within a few minutes the website was up again," says Geoff Thorn, general manager at the Parliamentary Service.

When Computerworld first inquired, Parliament's receptionist said "I don't know about the website, but there's been trouble with the phones this morning." Thorn says the phone fault was unrelated to the website interruption and the Parliamentary service is currently looking into the cause of both."

Comments

This just got worser The notification has been updated, they have managed to stand up the sites but you can't search, update or do anything with them. More importantly, the new notification doesn't provide any expected resolution time.
Posted by Worser at 11:13:59 on September 28, 2011

Flag abuse

HP EVA Failure Word is round the lunch room is that MED's HP EVA has failed after a code upgrade. Not the first time this has happened. Mid-range arrays trying to play in the enterprise space.....there is just never enough redundancy in these systems.
Posted by Anonymous at 6:50:36 on September 28, 2011

Flag abuse

HP EVA Failure Shows how little you really understand about storage then doesn't it - listening to lunch room gossip
There isn't much redundnacy in any TWO Controller Array irrespective of vendor.
Posted by Anonymous at 10:40:00 on September 29, 2011

Flag abuse

IT performance standards too low These are important websites and a key part of running NZ's business infrastructure. There should be proper system testing before upgrades, and redundancy of hardware. Backup should be in place after two to four hours and DR restores should be up in 48 hours.
Clearly these sort of standards are either not set in place or are failing to make the mark.
Christchurch has been a solid lesson for us- there were places where the 'alternate site;' is 13 kilometers away = not good enough.

Outsourcing blurs the lines and it becomes harder to identify the owner of the issue.

On the rugby field this event would be called an unforced error. How would these Govt entities perform if there was a real stress test? Poorly I suspect. This does not provide confidence as more and more activity is moved to the internet.

Posted by Steve Casey at 15:03:35 on September 27, 2011

Flag abuse

coming out of the dark ages Maybe they were coming out of the dark ages but the curtain rope failed therfore we are now all in the dark.... or the sever techs were celebrating the rugby win and spilt beer on the new virtual servers...

They gave me a five day extension on the company return so whatever... good to see it made the press...
Posted by Phil at 12:58:44 on September 27, 2011

Flag abuse

Foster Moore "The maintenance of our business services websites and registries is contracted to Foster Moore..."

Maybe a good time to start looking for an alternative?
Posted by Anonymous at 12:23:01 on September 27, 2011

Flag abuse

Foster Moore To be fair, if it's a hardware failure, that's probably not the fault of the software guys: Foster Moore provide and maintain the software systems; according to the article the hardware itself is the Ministry's.
Posted by Anonymous at 9:43:21 on September 28, 2011

Flag abuse

Foster Moore Looks like Foster Moore are after some data migration specialists.. http://www.fostermoore.com/company/careers.html must have been running their B team this weekend :)
Posted by Anonymous at 15:34:35 on September 27, 2011

Flag abuse

Foster Moore It does make you wonder, doesn't it.
Posted by Anonymous at 15:29:43 on September 27, 2011

Flag abuse

Parliament too? Parliament website down this morning (back up now). "Don't know about the website, but we've had rouble with the phones," says the man on reception.

Awaiting clarification.
Posted by Stephen Bell at 9:36:16 on September 27, 2011

Flag abuse