IT Horror Stories: Copying schemas from acceptance to production

This horror story happened to me a couple of years ago. It’s a great story to be told around the campfire with a flashlight shining from underneath your face.

I was working as a DBA for this company in the financial sector (not my current employer fortunately) and they were working on their new web application. It was something build on homegrown .Net and a (then in the Netherlands often used) content management system. (You can tell from the vagueries that this will not end well.)

At one point the consultant for the content management system appeared before me for the first time in a long time in the project in a alpha male type of way. “Allright. It’s time to get our stuff delivered to production. Make a copy of (the schema in) the acceptance database and import it into production.”

“That’s not a way to deliver your application to production”, I said. So alpha male consultant told me something like:  “All right, if you want to block the project, we can take this to the manager.” And so we went to the CIO.

So we told our sides of the story and I told him that if he was going along with this he had a real risk on his hand. But you know, these DBAs and their objections. So the CIO, nervous about delays in the project, responded that we’d better got on with it in the by the consultant suggested method. And so it went. All seemed to work fine.

Fast forward to a couple of months later. The security officer had persuaded management that if we were to become dependent the increasingly complex architecture of the company’s website, perhaps it would be a really good idea to test the disaster recovery scheme. And he eventually got approval for this.

So at one evening there they were: administrators from every part of the architecture and step 1 was to bring the acceptance systems down. Ten minutes later one of the other administrators that was still on site suddenly noted they had a production problem. The website was down. So at one point they decided to bring the acceptance systems up again. And surely enough: there was the production website again.

Turned out that the production website was depending on systems in the acceptance environment. Hmm, I wonder how THAT happend. Amongst which was … the acceptance database. And at this point we started to realize that if the production website was using the acceptance database, then where the <underworld place> was our production data….

It turned out: both in the production AND the acceptance database. It gave the phrase Disaster Recovery a whole new meaning. The lessons in this story: well, do I really have to spell it out?

Okay, let me say this on the matter: when IT administrators come tell you they have objections, as a manager, at least try to find out whether this is a potential small problem or one of those “I have a REAL DARK BAAAAD FEELING about this” types of objection. Ask check questions, like “What will happen if we go along with this?”


About Marcel-Jan Krijgsman

Ever since I started working with Oracle, I had an interest in Oracle database performance tuning. This led, eventually, to a four day training I made and gave for customers of Transfer Solutions. Since 2012 I work for Rabobank Nederland. A few years ago I also became interested in Oracle database security. All technology aside, it is my experience that security usually plays out on a political level. I'm a Oracle certified professional for the 8i, 9i, 10g and 11g databases and Oracle Database 11g Performance Tuning Certified Expert.
This entry was posted in IT Operations and tagged , , , , . Bookmark the permalink.

2 Responses to IT Horror Stories: Copying schemas from acceptance to production

  1. B Polarski says:

    I put my bet on DB_LINKS.

    • Not just that: the application saved connection strings in .. the database. So if you copy these from acceptance to production, where does your production application go? To acceptance. True story.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s