You know this saying in ICT: Everyone has a development environment, a lucky few also have a separate production environment.
I witnessed it first hand on IBM, three in the morning, troubleshooting a database problem for a big client. Engineer writes up a script to try and solve the issue, I was the systems operator. Tells me to just run it on the mainframe.
“Wait, was this tested at all?”
“Client authorized it, they just want the downtime gone. Send it.”
So I just ran an untested script that fundamentally changed everything on the production database, written by a sleep deprived engineer that just wanted to go back to sleep. Granted, it worked, that one engineer was an old rockstar who had been with the client for over a decade. But the next three weeks were dedicated to tiptoeing around the changes of this one script and testing everything, in production, to make sure that the solution was viable long term and it didn’t break anything unseen. We all knew better, but everyone agreed and did it anyways.
You know this saying in ICT: Everyone has a development environment, a lucky few also have a separate production environment.
I witnessed it first hand on IBM, three in the morning, troubleshooting a database problem for a big client. Engineer writes up a script to try and solve the issue, I was the systems operator. Tells me to just run it on the mainframe.
“Wait, was this tested at all?”
“Client authorized it, they just want the downtime gone. Send it.”
So I just ran an untested script that fundamentally changed everything on the production database, written by a sleep deprived engineer that just wanted to go back to sleep. Granted, it worked, that one engineer was an old rockstar who had been with the client for over a decade. But the next three weeks were dedicated to tiptoeing around the changes of this one script and testing everything, in production, to make sure that the solution was viable long term and it didn’t break anything unseen. We all knew better, but everyone agreed and did it anyways.