California DMV computer system failure. What happened?

fmitawaps

More Than 102 California DMV Offices Impacted by Major Computer Outage | KTLA

I just saw this. I'd love to know what happened to blow up the entire states' DMV system. What sort of computer failure. The CA DMV was slow enough when it was working, I'd hate to see the long lines out the doors of the DMV offices now!

superbeast

It doesn't say in the article if this is being considered a cyber attack or not. But it's hard to not think that way when just last week we had a DDOS attack on DYN server taking out several websites. I wonder if these small attacks are considered a "test" for something on a much larger scale in the future?

cyberguypr

"The DMV said Tuesday that crews rebuilt the system after the outage began Monday due to a hardware failure. Officials said the computers were not hacked or targeted."

Given how obsolete some .gov systems are, this sounds about right.

Edit to add a funny article I just came across: California DMV thinks "INFOS3C" is a dirty word

fmitawaps

Must have been quite a hardware failure to knock out the whole system. I can't see one router alone doing this. Maybe a storage controller or something?

cyberguypr

More insight:

"ROSEVILLE, Calif. (CBS/AP) – A “perfect storm” of simultaneous hard-drive failures caused a computer outage that crippled two-thirds of California Department of Motor Vehicles offices this week, DMV officials said Wednesday.

The department’s disaster recovery systems were not designed to handle such severe failures over a short period of time, DMV spokesman Jaime Garza said in an email."

TechGromit

I'm not surprised. I used to work for the federal government, the FAA specifically and you be surprised how antiquated some of those systems are. They do of course how newer computer systems too, but the sheer size of the National Airspace System would require a considerable investment to upgrade everything, some of the radar towers and VOR's were built in the 1960's and 70's. Unfortunately the Federal government couldn't get budgets passed on time, in the 7 years I worked there, I think they shut down or scaled back operations four times due to the Federal budget not getting passed in time.

alias454

Like this scenario Failed Windows 3.1 system blamed for shutting down Paris airport | Ars Technica

Matt2

I wouldn't be surprising if it WASN'T simultaneous in the way they present it. Instead a drive(s) may have failed days/weeks ago. But no one in the datacenter was actually running by checking servers for alert lights. Then another drive failed "just now" which resulted in the downtime.

I've personally been at TWO different places where such a thing has happened. People get lazy.

shochan

Don't ya think CHKDSK/FDISK should be ran on systems at least every 90 days or 180 days, just to make sure there are not any bad sectors?? What do you normally do on your personal systems or work systems?

Madmd5

Pardon my ignorance on the topic, but why does it seem most government systems are so outdated? Lack of funding or resistant to change? Should they not be at the forefront of cutting edge technology?

jeremywatts2005

I know one reason the government will not update is not because of budget, but lack of stability. Some of the older mission critical systems run on old VAX mainframe or some other main frame hardware. These systems can run for years and years without ever being touched. As long as they are kept clean and cooled. There was always fear that the newer equipment while faster may not be as stable as the older hardware. I know I worked for one agency and we were out procuring off of ebay and other areas for new old stock. Why because it was stable, proven and the data could not be migrated to another platform without loss. Plus the infrastructure reconfiguration that would go into it.

tpatt100

Not all government agencies have the same budget for technology, I saw a contract that was asking for people to help consolidate a state's disability payment system that was a bunch of different legacy systems.

I know my experience for Federal government work if it's connected to DOD after 9/11 the government purse was wide open so they took advantage of it. Other non military groups were not so lucky.

fmitawaps

I've been in companies where certain computers had to be kept on Windows XP, because the programs they were running were not available on anything newer than XP.

Can't virtual machines be set up to run as XP or other operating systems as needed for whatever programs they run?

TechGromit

Madmd5 wrote: »

Pardon my ignorance on the topic, but why does it seem most government systems are so outdated?

Many systems initially a considerable amount of time and money to create, updating those systems with new systems are horribly expensive as well. Most were designed on propriety hardware, to lock the agencies into one vendor. I'll tell you about the FAA program I worked for. It was the Remote Maintenance Monitoring System (RMMS), this system remotely monitored thousands of Radar towers, VHF Omni Directional Radio Range (VOR), landing systems, etc, and when the RMMS detected problems a Tech was dispatched to resolve the issue. Before the RMMS system someone had to be stationed at all these facilities all the time to ensure they remained operational, some of them were in very remote locations all over the country. Replacing them with RMMS and only having someone visit the location only when there was an issue saved a considerable amount of money. RMMS was designed and programmed on a tandem system, the hardware was replaced at least twice since the 1980's to newer Tandem systems, but the service contracts were very expensive. They were playing over 1 million dollars a year for software called "Goldengate" allowed the tandem to interface with TCP/IP systems alone.

When I was hired as a contractor for the FAA, they were in the process of replacing the tandem system with a server based solution, they paid a government contracting company (Digicon I think) over 100 million dollars, at first to replace the entire system, then only part of the system when the couldn't even do that. In the end the replacement failed to live up to any of it's promised claims and the project was terminated. The decision was made to use the hardware that was already purchased and develop a system inhouse that just to replace the system as is functioned today, instead of trying to add functionality while replacing the system. I took 6 years, but the project was a resounding success, the RMMS system was replaced with the RMLS system, for a mire faction of what they tried to pay a contracting firm to do for them (somewhere around 25 million I think). The system now runs on a server based platform, not reliant on a specific hardware type.

And this is just one of the many systems that make up the US National Air Space system.

fmitawaps wrote: »

I've been in companies where certain computers had to be kept on Windows XP, because the programs they were running were not available on anything newer than XP.

I have the same issue where I work, some programs run in DOS. When we were doing a Windows 7 migration, the respirator testing computer only ran in XP, upgrading the software to run on Windows 7 would have cost 10k, the company opted just to isolated it from the network, instead of upgrading it. After all why replace a perfectly good functioning system just because some vendor decided to stop supporting the operating system? The same thing happens everyday in government agencies and companies all over the world, why should we pay to upgrade when what we have still works perfectly well. So they save money by not upgrading DOS to Windows 3.1, Or Windows 3.1 to Windows 95, or Win 95 to XP and before you know it the systems are horribly outdated with knowledge staff and spare parts in short supply.