Active Directory go BOOM!!!!

[Jammrock]

The entire M$ active directory forest at work just crashed and BURNED, BABY, BURNED!! This of course means that no one can:

1) Logon if they need to reboot, or just came in.

2) Access email, since we use Exchange.

3) Access any network devices, like shared drives.

4) Use our internal exchange IM service.

5) Can't use various tech tools.

Oh yeah, someone's head will roll over this one.

Jammrock

[Tjalfe]

I hope you have an alibi

[KeiFront]

Backups?

[GT98]

Ouch

But you can still get on MURC to post this

[The PIT]

I guess it was a rain forrest.

Time for a holiday.

[agallag]

Considering my job is "Active Directory Planning and Engineering" for a financial institution with ~40,000 employees, I'd be very interested in more details on this. I certainly hope your backups were up to date...

http://www.microsoft.com/downloads/details.aspx?displaylang=en&familyid=3EDA5A79-C99B-4DF9-823C-933FEBA08CFE

May want to forward that link to the AD guys

[Jammrock]

Rumor has it that they ran a "Microsoft critical update" that the servers didn't like. Or some sort of upgrade because AD servers were randomly rebooting. Well ... the patch made things worse. Within 7 hours of applying the fix the servers began rebooting almost non-stop.

With all the AD servers out of action from reboots, all AD serverices, like AD look up, authentication for network shares and email, etc went bye-bye.

Internet on the other hand was unaffected since it doesn't rely on AD authentication.

It took about 3.5 hours before all services were restored.

Jammrock

[agallag]

If you could find out which update it was, that would be really handy information to have

Good thing they managed to fix it without resorting to restoring the entire forest from backup. That involves shutting down all the domain controllers, restoring one from backup in each domain, then formatting and rebuilding the rest of them from scratch and replicating the restored data from the primary. It could take a few days if there are lots of them (we have a total of 22 DCs under our control (root domain and two subdomains), plus another 100+ in other domains that we don't manage). Very nasty process, luckily it hasn't happened to us...

[The PIT]

Originally posted by agallag
If you could find out which update it was, that would be really handy information to have

Good thing they managed to fix it without resorting to restoring the entire forest from backup. That involves shutting down all the domain controllers, restoring one from backup in each domain, then formatting and rebuilding the rest of them from scratch and replicating the restored data from the primary. It could take a few days if there are lots of them (we have a total of 22 DCs under our control (root domain and two subdomains), plus another 100+ in other domains that we don't manage). Very nasty process, luckily it hasn't happened to us...

You know that sounds like a very nasty weakness of AD. Me thinks Microshaft ought to think of a way of rapidly restoring AD if a normal restore would take days.

[Umfriend]

Possibly this is of a total different order of mnagnitude that such MS product, but when I was a console operator on an Unisys 7200-2 (not sure about the model, 3 yrs ago) mainframe, we had to be able to rebuild from scratch within 60 minutes.... Which we practiced at least quarterly. For critical systems, 3 days downtime is a joke.

[agallag]

That kind of outage doesn't make 3 days downtime. As soon as the first DC is restored, services will continue, although with seriously reduced performance. That could take a maximum 15-30 minutes per domain, since we do hourly backups of the entire directory to the local drive on every single DC (we do daily, weekly, and monthly backups to tape as well). Once the PDC emulator box is restored in a domain, that domain is once again live and providing authentication. Each subsequent DC you rebuild will improve the performance, until you're fully back to normal.

Edit: Also, I haven't heard of this kind of outage on a real production AD forest ever happening outside of a test lab. It's really quite reliable. Our forest has been live for more than two years now with not a minute of downtime. That's right, 100.0% uptime. The only restores we've done so far were as a result of accidental deletion of objects by support people.

[Paulr]

Our AD domain has been more or less reliable too with only one major problem in the 3 years it has been running.
That was when we lost a DC and it just happened to be running all five FSMO roles - the person who planned the network in the first place forgot to distribute those tasks between two DC's.
Oh and it just happened the same server was also running the Global Catalogue for the Exchange Server too.

However, we just thransfered three of the FSMO roles over through the GUI tools, forced the other two with "Seize" commands and created a new Global Catlogue on anotehr server.
Total downtime around 2hrs and we then spent as long as we liked getting the failed server back up and running - new RAID controller required.

Other than that one incident the AD has been absolutely excellent.

[Jammrock]

ROFLMAO!!!!

It turns out that the server reboot was caused by loading some sort of Symantec network security program. After applying the M$ recommeded patches and fixes it only escalated the problem and made things significantly worse.

Removing the Symantec product fixed the reboot problem.

Jammrock

[agallag]

Yeah, that sounds about right Don't they have a test lab for that sort of thing?

[Jammrock]

They do and they did. But the problem took a while before it started, then got progressively worse.