HealthCare/Hospital/On Call

pixa241

Anybody on here work at a hospital/healthcare or anyware else where business hours afe 24/7/365? Our department is really bad with On Call and the way we handle some process so would like to see how other in the same en ironment Monitor Critical systems, handle On Call, and maybe some best practices on other stuff.

Queue

Yes at a 24/7/365 hospital. Monitor alerts every 15 minutes that come through Outlook. We have an on-call list on Sharepoint that changes daily. We make mistakes calling the wrong team at times, but it's getting easier. On call people have thirty minutes to call back or it gets escalated to the department manager. We must be very careful to document the entire call such as called xxxxx at xxxx, waited fifteen minutes called back etc. As first point of contact we must ask the user if the issue is patient safety "an emergency" if not it is just assigned for the morning.

We have a 24/7 help desk as first contact. I'm a second shift worker. The department is over 100 people. I believe about six thousand workers here in total.

alias454

We do on-call rotations. Everyone takes a turn in the barrel except admins, who are on-call 24-7 for the apps/servers/hardware we manage. The on-call person takes care of simple things like password resets, initial documentation/troubleshooting, helping users get remote access etc. If an issue is outside the scope for the on-call person, they escalate to an admin or appropriate EMR person.

In total, there are three sys admins, three network/telecom engineers, two EMR admins, one DBA, and one storage admin who never take on-call rotation because we own the infrastructure. There is a bio-med team that rotates once every four weeks to cover their on-call needs. The helpdesk and everyone else gets to rotate through about 2x per year. I should also mention that taking on-call is not optional and does come with extra pay for the weeks someone takes it.

Regards,

pixa241

Well I am the Server/VMWare/Citrix/Storage admin here at my place. We have 5 admins including myself. 3 on the systems sife and 2 network. We rotate once a week and if we need to get other involved we do. We have a 24/7 helpfesk. What do you guys use for monitoring. We currently use WhatsUp gold but it is pretty bad and hard to read so helpdesk always calls us for false positives.

alias454

We use PRTG; you can get a free trial https://www.paessler.com/prtg. After the free trial, you can continue to use it with 100 sensors. Generally spealking we put three sensors on every server for disk ,RAM, and CPU then add appropriate sensors for the role of the machines. We also use lansweeper for inventory Network Inventory - Software Inventory Management for Windows Networks. PRTG can be a little steep on the initial purchase but the yearly maintenance is pretty reasonable after that, it's also agentless and can do checks on VMware, UCS, Cisco, Windows, Printers etc etc. Lansweeper is also very affordable and is agentless. I should also add that our EMR has a system analyitcs server that is seperate from the general server stuff kept in PRTG.

Out of curiosity, how large of an environment are you working in and how much after hours stuff do you get called for?

pixa241

Yes we use Lansweeper in our environment as well but it is kind of bloated. Havent tweaked it out yet. I am looking to replace or redo our WhatsUp Server. Our environemnt consists of about 800 VMs on vCenter 6 with 27 Physical Hosts. And about 30 Physical servers, and our Citrix environment which house 30 servers on XenServer 6.5 with XenApp 7.7.

alias454

Are you doing VDI? Right now we manage about 350 servers with our Citrix farm at about 40-50 VMs for our EMR. the rest of the Citrix farm is for other apps and might equate to another ~15 servers. We are also on 6.5 and planning to upgrade for our next EMR upgrade in 2017. I manage VMware, where 12 servers make up our main prd farm. I also have another 10 standalone VMware servers located out at branches and a few other standalone servers for misc application stacks.

Our Lansweeper install has about 5000 objects ish. The config for it is 2vCPU and 4Gb of RAM. Storage is on an EMC 5300 and we use a SQL express install since the DB is not over 10GB's. it can get pokey when it is rescanning the entire domain but isn't unusable.

Mike-Mike

pixa241 wrote: »

Yes we use Lansweeper in our environment as well but it is kind of bloated. Havent tweaked it out yet. I am looking to replace or redo our WhatsUp Server. Our environemnt consists of about 800 VMs on vCenter 6 with 27 Physical Hosts. And about 30 Physical servers, and our Citrix environment which house 30 servers on XenServer 6.5 with XenApp 7.7.

we use Lansweeper, it has pros and cons, I need to dig more into it

RHEL

I've never had a role in my 6-7 years out of college that wasn't 24x7x365 on-call.

1st job was a team of 6 people in an aerospace/defense environment, rotating one person on a week at a time. Our 24x7 operators would monitor events and would page out if there was an issue with production systems. I was paged a few times per week -- ~400 servers.

2nd job was a team of 2 people in a casino/resort environment, rotating two weeks on two weeks off. Our 24x7 operators would monitor events and page out as needed. I hardly ever got paged -- ~120 servers.

3rd job is a team of 5 people in a healthcare environment, rotating one week per person. We receive automated pages if a production system goes down and also have a 24x7 operations team -- ~500 servers.

ITBot

We do on call rotations. One week at a time. 3 teams, Applications, Desktop and Networking. We also have an outside vendor that is 24/7 doing the front line stuff such as password resets etc. They also handle creating the tickets and escalating the calls. We have a 30 min window to call back or the ticket gets escalated to a manager. We keep a log where we detail all the ticket details including who called the ticket in, what time, what was the problem and what was the resolution. On call is voluntary but it's pretty nice financially so not many people opt out of it.

We're looking into monitoring as well. We just have too many on going projects and not enough time or money. I'm about to start a VDI POC since we haven't been happy with Citrix since implementation. Constant printing problems and issues with our EMR system.

White Wizard

Yes.

I work for an MSP but I work at a hospital.

They use Kaseya so if a server restarts, goes down, HDD gets near capacity, we get alerts for anything.

They also use an after hours answering service which then contacts the on call tech. There are set levels as to what is considered an emergency and on call only gets called for emergencies.