This horror story happened to me a couple of years ago. It’s a great story to be told around the campfire with a flashlight shining from underneath your face.
I was working as a DBA for this company in the financial sector (not my current employer fortunately) and they were working on their new web application. It was something build on homegrown .Net and a (then in the Netherlands often used) content management system. (You can tell from the vagueries that this will not end well.)
At one point the consultant for the content management system appeared before me for the first time in a long time in the project in a alpha male type of way. “Allright. It’s time to get our stuff delivered to production. Make a copy of (the schema in) the acceptance database and import it into production.”
“That’s not a way to deliver your application to production”, I said. So alpha male consultant told me something like: “All right, if you want to block the project, we can take this to the manager.” And so we went to the CIO.
So we told our sides of the story and I told him that if he was going along with this he had a real risk on his hand. But you know, these DBAs and their objections. So the CIO, nervous about delays in the project, responded that we’d better got on with it in the by the consultant suggested method. And so it went. All seemed to work fine.
Fast forward to a couple of months later. The security officer had persuaded management that if we were to become dependent the increasingly complex architecture of the company’s website, perhaps it would be a really good idea to test the disaster recovery scheme. And he eventually got approval for this.
So at one evening there they were: administrators from every part of the architecture and step 1 was to bring the acceptance systems down. Ten minutes later one of the other administrators that was still on site suddenly noted they had a production problem. The website was down. So at one point they decided to bring the acceptance systems up again. And surely enough: there was the production website again.
Turned out that the production website was depending on systems in the acceptance environment. Hmm, I wonder how THAT happend. Amongst which was … the acceptance database. And at this point we started to realize that if the production website was using the acceptance database, then where the <underworld place> was our production data….
It turned out: both in the production AND the acceptance database. It gave the phrase Disaster Recovery a whole new meaning. The lessons in this story: well, do I really have to spell it out?
Okay, let me say this on the matter: when IT administrators come tell you they have objections, as a manager, at least try to find out whether this is a potential small problem or one of those “I have a REAL DARK BAAAAD FEELING about this” types of objection. Ask check questions, like “What will happen if we go along with this?”