New Managed Service Provider brought down our Email Today

genXrcist

Yep, they did and it was 100% their mistake. About 3 weeks ago my company signed up with a managed service provider and this weekend they're migrating all of our servers (about 11) to brand new servers. Going from 2003 FFL/Exchange 2003 Std to 2008R2 FFL and Exchange 2010 Std...

So yesterday they installed a 2008 DC (after forest prep etc) and then Exchange 2010. They also installed SP1 for Exchange and that apparantly blew up. No worries they said as we walked into the server room so they could restore it and reinstall...

Well it turns out to restore from SP1 you have to delete specific Exchange 2010 AD objects and they deleted the wrong ones... in fact they deleted all our Exchange server CN's...

Well after restoring the object tombstones failed they had to perform an authoritative restore on one of our DCs & finally about 5 hours later they got Email back up.

Now here's the thing. My boss, who's a real jerk, specifically championed the MSP and the selection of this provider. My boss is also an IT idiot and I would really love to see him get his just rewards for this.

Any suggestions on how some of you would play this? Office politics is a reality imo, either play the game or just be prepared to lose as the game is played whether you roll the dice or not.

Forsaken_GA

I feel ya man.

I wish I could talk about an incident that we had around the office recently, but all I can say is that when Microsoft says something is a hotfix... don't believe it.

colemic

I would use a snorkel - you still get plenty of air, and you are below the surface out of sight...

If it was me, I would play quiet and if he does get pushed towards the door, to make sure my ducks were in a row so I don't go with. JMO.

Slowhand

Honestly, I'd write down everything that happened and make sure I knew exactly how much time, money, and productivity was lost because of this incident. The direct way to deal with it, compile a report and give it to your boss' boss. The indirect way, and maybe the only way to cover your butt and make sure you don't take the full heat, is to sit on it until someone above him asks what happened and begins pointing fingers. Remember, the more exact the dates, times, and technical information, the better. And be sure to include a preamble about how this project was started and who recommended the provider.

Incident reporting is very handy, and not just for "what not to do".

ssampier

In my short life I seen careers made and lost due to both regional (Politics) and office politics. It's ugly and totally unfair, but it happens.

Best advice: duck-and-cover.

Slowhand is right. Document who signed the contract and whom ultimately worked with the providers to fix the problem.

Since this is not your screw-up make sure you have backers since the jerk boss probably has good, powerful friends.

it_consultant

Something more insidious happened here. I have installed EX 2010 SP1 several times with minimal interruption of service. I suspect they borked the DC and didn't realize it.

EDIT: I accidentally deleted an AD object pertinent to Exchange and the fix was to run setup.exe /DomainPrep from the installation media. Whenever you do a domain prep it checks and fixes any AD object needed for Exchange. I think you need to investigate this further.

blargoe

If you didn't have your hands in the migration I would just sit back and let the boss man take the heat. No way any of this could be pointed back to you, right?

genXrcist

ssampier wrote: »

In my short life I seen careers made and lost due to both regional (Politics) and office politics. It's ugly and totally unfair, but it happens.

Best advice: duck-and-cover.

Slowhand is right. Document who signed the contract and whom ultimately worked with the providers to fix the problem.

Since this is not your screw-up make sure you have backers since the jerk boss probably has good, powerful friends.

Good advice. It's well known that this provider was my boss's pick but yes, he does have a powerful friend... the Executive Director of the organization. Lucky for me that he's my uncle though but only through marriage...

it_consultant wrote: »

Something more insidious happened here. I have installed EX 2010 SP1 several times with minimal interruption of service. I suspect they borked the DC and didn't realize it.

EDIT: I accidentally deleted an AD object pertinent to Exchange and the fix was to run setup.exe /DomainPrep from the installation media. Whenever you do a domain prep it checks and fixes any AD object needed for Exchange. I think you need to investigate this further.

The problem wasn't with the Exchange 2010 objects but in that they deleted the entire Exchange container which included the Backend & Frontend server objects! On an Authoratative directory services restored fixed the problem as only the Infrastructure Master DC was having it's System State data backed up.

blargoe wrote: »

If you didn't have your hands in the migration I would just sit back and let the boss man take the heat. No way any of this could be pointed back to you, right?

Nope, not in the least. I'm so tired of this group and I'm actively looking for a new job. I have to be somewhat picky though as I'm an FTE with great benefits (have family) and they're still paying for me to go to classes, the CBTNuggets subscription not to mention the 10 MS vouchers. Oh well.

But... I think I have my Ace... Monday morning came around and as a result of their moving the File Shares they screwed up by allowing Payroll, budgeting and private personal data to be viewed by EVERYONE. Yep, everyone. We were alerted by one of our clients, the people we exist to serve (we're a non-profit), and not by anything else! Lol

So I discussed this with HR who had not been informed of the Security breech and the wheels have been set in motion....stay tuned...

genXrcist

And the wheels made a screeching halt. This organization is ridiculous, I need to get out soon. The security breach? Brushed under the rug. What a joke.