Advice about dealing with legacy computer equipment

kotan

Good day TE experts... I'm hoping to gather some advice regarding one of the largest challenges I've faced in IT over the last 6 years.

I began working for a large pharmaceutical company almost 2 years ago. We have come a long ways in regards to modernizing the plant. However, the single largest achilles heel is dealing with legacy equipment.

To provide a high-level summary behind the culture:
As some of you may know, pharma companies are subject to constant scrutiny. Audits are generated almost weekly from customers, the FDA, internal resources, etc. As a result, these companies take change management and validation enforcement very seriously. Entire departments are dedicated to govern changes associated with computer upgrades, projects, simple updates, and most of all... all of the paperwork associated (the sheer volume of paperwork can be staggering). This yields a very interesting contrast between a traditional company and a pharmaceutical company. When you spend a grand total of 8 hours upgrading critical software on a machine, it may take me 3 months (I'm not exaggerating) and bloat the cost of the project tremendously.

To nobodies surprise, this results in application owners never pursuing upgrades because they may not have the money, resources, or time. Fast forward a decade later and you have critical equipment running on Windows 98, NT4, 2000, etc.

Not only is that equipment a security risk (non-supported/non-patched) but that hardware is at risk of crashing every day. If a system dies, you can avoid a mountain of paperwork by replacing like for like parts...however, you can imagine the pain it is ebaying old equipment. If you can't find said equipment and the hardware has to change, not only do you have a crazy situation from a paperwork perspective, but good luck finding drivers.

The obvious answer from an IT perspective is to pursue the educational path to ensure the company understands the risk profile. Educate the business about standardizing equipment and developing a support structure that's both flexible and responsive. However, we haven't yet been able to gain much traction...yet.

So I ask...What have folks done to help streamline support for this kind of equipment? Support meaning flexible with hardware upgrades and in many cases, ensuring backups are taken. Keep in mind a lot of this equipment is tied to physical hardware via ISA cards/Serial connections.

First method that came to mind (VMWare / ESXi):
Try to develop an ESXi host for these devices so you have the flexibility to migrate from hardware to hardware, however, I assume identifying drivers for your ESX host to run an ISA card for a windows 98 virtual machine would be a nightmare.

Second method (network backups / bare metal restores):
Utilize a network backup solution that can perform bare metal restores to identical OR dissimilar hardware. This has been the closet thing to a solution we've been able to identify. My concern is here is that the backup agent can consume a lot of valuable resources on a system with limited resources (128MB of memory in some cases).

CURRENT method (Ghosting):
Today we manually backup critical devices, perhaps once a year or two to a ghost image. This way we at least have the ability to restore to like for like hardware, regardless of the age of the data.

Thanks in advance!

Shdwmage

I work for an archive company, so I feel your pain. We take and hold on to all of your old stuff after you are done with it!

Ghosting is really the only efficient way to backup hardware images of the machines. You can ghost to a virtual image, but if you are connecting to physical hardware, you should really keep it on a physical machine as the virtual machines do not play well with physical hardware.

I would start trying to budget to upgrade the machines, or see if you can find compatible controller cards that work with newer hardware.

joehalford01

If you can't upgrade the hardware, I would proactively purchase spare parts for what you currently have. Old stuff is cheap on ebay, so buy 5-6 servers with spare parts and horde them in a stock room.

Also, maybe start the paperwork before doing an upgrade? Once that's done, do your swap.

petedude

I'll second Joe H.'s input, and add something.

It sounds like this is one of those environments where technological changes moves as slow as molasses for whatever reason. In these situations, it may be helpful to simply accept that direction and orient yourself toward long-term planning. Is there some upgrade that's key but could wait until next year? Start the paperwork now, and hopefully it will clear process before you really need to move forward. Begin planning incrementally with a multi-year window in mind.