Active Directory 2003; VANISHING USER OBJECTS!!!

mattipler

Good Evening All,

Active directory is not my area... AT ALL. I carry out administration within AD 2003 and that's where my involvement in my corporate environment starts and ends. Recently, for no apparent reason what-so-ever about 35% of our user accounts vanished from within Active Directory. All within the same domain and from within Mulitple organisational units. The deleted accounts appear completely random and we can't see any pattern! AD is integrated with Exchange and the Exchange accounts were NOT effected. It's also worth mentioning that we recently implemented Cisco Unity Voice Manager solution for user voice mail, that imports user accounts from within AD when first creating the Unity voicemail account. The unity accounts also disappeared for the users who's ad accounts disappeared.

We have 3 domain controllers upon the same network but are geographically separated, that were all effected, (one we even had to completely rebuild) that all sync with one another. In the end we contacted microsoft who talked us through doing a state restore using our backups to the primary domain controller and then we sync’d to the other two. Also, a "foot-print" facility that would have given us a clue as to why this occurred in the first place was not activated and isn't activated by default so we're completely clueless as to what created this issue in the first place.

I’ve searched the internet high and low for possible answers! But found nothing. As I’m the most junior member of the team I’m sure that I’m seen as a possible cause even though I know 110% it had nothing to do with me as I hadn’t access AD or the domain servers for two days prior to the problems and even then it was only to add someone to an organisational unit. There’s nothing obvious in the event logs.

Has anyone experienced anything similar or have a possible answer? Even a link to a website giving me a clue would be greatly appreciated.

The board aren’t impressed that we have no explanation what-so-ever as to why we had 6 hours where 35% of the company lost all access to the network and their email.

Help! 

buttons

I don't know the answer either, however at my work we've seen AD objects vanish. Users
had access to certain shares yesterday but the next day the group was missing. I am just on the helpdesk and do not have access to any logs -- but i wouldnt be surprised if it wasnt someone's
intervention wether it be a script that ran against the AD or something (at least in our case)

bertieb

I've certainly never heard of AD doing that 'on it's own' as it were and there's no reason it should without a little prompt from an external source ......it sounds like a dodgy script or someone's mistake (potentially malicious....)

If you've no audit logging going on for the systems then I'd say it'll be impossible to nail the root cause, though I'd expect one of the AD superheroes on here (hero/royal to name a couple) may chime in with some advice.

Unlucky, I work in Manchester too so it was probably you I heard screaming from my office

Technowiz

Is your exchange server also one of the domain controllers?

Mishra

So you did a restore of the user objects and everything seems to be working just fine now?

Mishra

One place you might want to start looking is in your AD replication.

Were the users accounts that were deleted all "new" (meaning there were no users deleted that are the oldest accounts) users to your corporation?

mattipler

Hi guys,

Many thanks for your responses!

Exchange and DC's are on separate servers.

We restored the objects and all was working fine in that users could connect to the network again. As one of the domain servers had to be completely rebuilt there were other area's that suffered as a consequence... i.e VPN and OWA.

The cisco unity accounts didn't reappear, which my colleague is currently running through with NTL (the providers of our current Cisco telecomm's solution).

Matt

geekie

Also, a "foot-print" facility that would have given us a clue as to why this occurred in the first place was not activated and isn't activated by default so we're completely clueless as to what created this issue in the first place.

I'm assuming your talking about turning on object access auditing in the Domain Controller Security Policy? If the audit Directory Service access option was enabled and the OU SACL in question has 'Delete User Objects' checked for success then I would use

ldifde -f del.txt -d "CN=Deleted Objects,DC=your,DC=domain,DC=com" -r (objectclass=user) -p subtree -x -l DN

to export your deleted items container to a file 'del.txt'. Then search for the DN of one of the users in question and use REPLMON's 'Show attribute meta-data for Active Directory Object' option. This will give you the 'isDeleted' attribute and tell you the originating DC and the date and time as long as it's within the tombstone period. Then check the event logs on that DC. I think the event ID is 630 for account deletions if I remember rightly.

mattipler

Yeah this wasn't activated unfortunately... thanks for the response though!

Technowiz

This may be completely unrelated to what happened at your work but I have experienced something similar where AD user accounts were automatically deleted. My boss in an effort to minimize downtime while replacing a domain controller/exchange server decided to do it with an image. Not sure what imaging tool was used. So the image was created with the DC was running, image was put on another server, original server brought down, new server brought up. Things seemed to go ok for a few hours but then a couple user accounts in one OU disappeared. We had logging turned on and found that the accounts were deleted due to duplicate SIDs. I don't remember the event ID now. A few minutes later another account in the OU disappears. Then another... Long story short we took down the server and recreated the user accounts from scratch.

I never found a definitive answer to what happened but my boss didn't know anything about FSMO roles and didn't transfer them when doing this.

I think what happened may have had something to do with the RID master role but I still don't understand the guts of AD well enough to know what happened here. Or I may be way off. I just know you don't image domain controllers!

Not sure if this sheds any light on what might have happened to you guys. Here is a link from Microsoft discussing why imaging a DC is a bad idea.

http://www.microsoft.com/technet/serviceproviders/wbh4_5/CMSU_CM_Plan_CONC_Domain_Controller_Recovery.mspx?mfr=true