I just learned a very important lesson

Never restart a server that you don't fully understand... I've been doing this MSP/Help desk for about a month now. I've learned some very important lessons. Today topped the cake with restarting a somewhat important server last night and not having it come back up. And I didn't have any physical access to it till this morning. Some dang boot logging garbage that required a key input to continue the boot sequence. Got on site and it took another 2 hours for it to be fully working - DNS crapped, AD was acting goofy... Server network apps weren't responding. Ugg was not a very fun morning. Has anyone ever seen the server restart video with the kid playing Halo? Just was thinking about that lol.

Find more posts tagged with

Free for TechExams community: Cybersecurity salary guide

Compare cert salaries and plan your next career move

Button

Comments

zippie666

themagicone wrote: »

Never restart a server that you don't fully understand...

In the past we have learned some valueable lessons too that are kind of related. We restarted a production critical server but it didn't come back up. We had to run all the way to one of the datacenters to fysically go press a key because a S.M.A.R.T. error came up...

Monkerz

DRAC is a must have. All servers we have, for the most part are DELL and equipped with DRACs. Makes life easier when you work from home during a maintenance window.

themagicone

For the life of me I can't get DRAC to work. Now, the servers I have been installing just have the built-in drac card so that may be why. The server the crapped on my yesterday did have a iLO card but of course it was never set up so it was useless.

Monkerz

What do you mean, you can't get DRAC to work? What issues are you having? The R710s we have been purchasing have onboard controllers, but I haven't seen any issues out of them. I know IE9 spazzes out on me sometimes when I attempt to access the controller.

themagicone

Well I set up the controller with an IP address/subnet/gateway/etc. Turn it on and so on. Go to another computer on the network type in the IP address in an open browser and nothing. I've tried [url]HTTPS://IPaddy[/url] and just the IP but nothing. Tried on 3 different servers so far and same result. Unless I'm setting something up wrong, which could easily be the case.

Forsaken_GA

Monkerz wrote: »

What do you mean, you can't get DRAC to work? What issues are you having? The R710s we have been purchasing have onboard controllers, but I haven't seen any issues out of them. I know IE9 spazzes out on me sometimes when I attempt to access the controller.

Some of the DRAC firmwares are really picky about the browsers they'll actually work with. When I worked for Dell, I occasionally had to play musical browsers between IE and Firefox (I normally use Chrome), and a few times, I had to alter the User Agent to appear to be something else because the web UI itself refused to work with whatever I was running based solely on that string.

HP iLO cards haven't been quite as picky for me, they'll complain, but I've always been able to get them to work with Chrome.

And yeah, always take rebooting a production server with a grain of salt. I get really nervous anytime I have to reboot a box that's had a very long uptime as well. If I don't have physical access to a box, then I want IP KVM access, or terminal server console access

ptilsen

themagicone wrote: »

Never restart a server that you don't fully understand... I've been doing this MSP/Help desk for about a month now. I've learned some very important lessons. Today topped the cake with restarting a somewhat important server last night and not having it come back up. And I didn't have any physical access to it till this morning. Some dang boot logging garbage that required a key input to continue the boot sequence. Got on site and it took another 2 hours for it to be fully working - DNS crapped, AD was acting goofy... Server network apps weren't responding. Ugg was not a very fun morning. Has anyone ever seen the server restart video with the kid playing Halo? Just was thinking about that lol.

Does the server have associated documentation? Does the documentation explain what the server did and outline this caveat?

If so, your mistake was not reading the documentation first. If not, the server was not properly documented.

Monkerz

themagicone wrote: »

Well I set up the controller with an IP address/subnet/gateway/etc. Turn it on and so on. Go to another computer on the network type in the IP address in an open browser and nothing. I've tried [URL]HTTPS://IPaddy[/URL] and just the IP but nothing. Tried on 3 different servers so far and same result. Unless I'm setting something up wrong, which could easily be the case.

So I am guessing the server and DRACs were on the same subnet? I am curious if you are able to ping the DRAC's IP once setup.

Monkerz

Forsaken_GA wrote: »

Some of the DRAC firmwares are really picky about the browsers they'll actually work with. When I worked for Dell, I occasionally had to play musical browsers between IE and Firefox (I normally use Chrome), and a few times, I had to alter the User Agent to appear to be something else because the web UI itself refused to work with whatever I was running based solely on that string.

HP iLO cards haven't been quite as picky for me, they'll complain, but I've always been able to get them to work with Chrome.

And yeah, always take rebooting a production server with a grain of salt. I get really nervous anytime I have to reboot a box that's had a very long uptime as well. If I don't have physical access to a box, then I want IP KVM access, or terminal server console access

I know what you mean. I usually can use Chrome with an IE tab for anything I need. Have to play musical browsers with DRAC, Fluke's Visual Uptime, and Shoretel Director.

rsutton

Most of our servers are virtual but I manage a few physical servers that do not have DRAC/KVM > IP. I make sure that I can get onsite access if needed when rebooting one of those servers. Always have a backup plan.

jamesp1983

Monkerz wrote: »

DRAC is a must have. All servers we have, for the most part are DELL and equipped with DRACs. Makes life easier when you work from home during a maintenance window.

x2. Has saved me more than once.

ChooseLife

"Never restart a server that you don't fully understand" captures it well.

We all get those lessons early in our careers. What distinguishes the good sysadmins from the bad ones is whether we learn from those lessons.

kurosaki00

You know whats even more fun?
A server that needs restart ASAP and when you look @ the info of last time restart it says almost 2 yrs ago.
Now thats something cool

themagicone

The thing I'm coming to grasp with is that I really don't like system admin work. I started my IT career 6 months ago (well I've been off and on for years but now full time after WGU) and really got into networking. I love cisco, networking, etc but damn windows server... Why in #### does there have to be 6 areas to open a port for the firewall? Don't get me started on SBS, that pile of garbage. I've only been doing this system admin/help desk/etc work for a month and all I can think of is when I get my next 'hit' of cisco. But this is paying the bills and building experience so I'll stick at it. I got a new client for myself installing routers in cell sites so that will keep my addiction to networking somewhat satisfied.