A few weeks ago I was standing in an office when one of the IT people walked through. He was on his regular walk to deliver backup tapes to the "off-site" location, and this also gave him the opportunity see if he could trouble-shoot an issue for a senior exec. And there they were, backup tapes, just sitting unattended on a desk, as Mr IT was at the other end of the floor looking at the laptop. Yes, the data could have been sent across electronically, and certainly, the company's major systems are all automatically backed-up with almost constant checkpoints. Those systems can be flipped from the primary production to the back server farm in a matter of minutes.
But back to the backups; what was on those tapes? What would have been the impact on the company if the tapes disappeared? Source code? Customer data? Test scripts? Downloaded movies? Confidential emails?
After all, the tapes are a standard format, and the operating system is standard, so there would have been a minimal challenge to recover everything from those tapes.
Sometimes we see "non-critical" systems continue to be managed as stand-alone environments, disconnected from the corporate environment, and most especially from the automated backup world.
I was tempted to pocket one of the tapes and see if he noticed it was missing. As there were only two tapes, or the cartridges that replaced "tape" decades ago, I was pretty confident that he'd notice. Cheap joke, not worth the effort.
But it did remind me of a blatant demonstration that I had to give a General Manager almost twenty years ago. Because sometimes a such a demonstration is what is needed to make the point. This is especially true with backup tapes.
We were (internal) auditing a subsidiary of a telephony company, and were in their Auckland office. The subsidiary was building a new product for corporate clients, enabling the clients to take a single file of all telephone data, already data-cubed and with a set of associated software to allow multi-dimensional analysis of that data. All very cool for the time.
The systems ran on a small set of servers, actually just very powerful under-desk PCs, up-configured about as much as was possible in that day. And as this was a subsidiary and not integrated into the telephony company, the servers were of course in a room in their offices in the Auckland office building. And of course, one of their main programmers also worked in that same cubbyhole of an office.
Furthermore, the backup tapes were kept in the drawer of the desk next to the computer. The door to that room was of course locked, I was told, even though the door stayed open all day. At night, when the programmer left, he closed the door behind him, and then unlocked it to get back in the next morning.
By the end of the audit, we had our draft report and findings completed, and we were ready to present these to the General Manager. The only time slot he had was after hours, so we sat down with him at 7pm to go through the draft report and findings.
If there is one thing that I've learned about auditing, it is that the enthusiastic nodding of the executive is as frequently faux-agreement so that you will just go away as it is agreement to fix the issue, no matter how trivial. In fact, the more trivial, the greater the chance that the nodding will actually be a "please go away" message.
We were not getting any of those messages until we got to the IT Audit findings, and then the nodding started. The issues were outside his area, and he didn't understand the issues, their severity or the what effort would actually be required (all issues had been cleared with the IT people first of course). So he entered "thank you for these very helpful findings, please go away" mode.
Being fair, most of the issues were fairly minor, but I knew that a demonstration was needed. Something to make sure that the next morning, he would actually call in the IT people and get the issues, if not resolved, at least on the agenda for resolution.
When it came time to talk about backup security, I stopped, stood up, and excused myself from the room. "Pardon me, I'll be right back", and walked out...
...down the hall to the IT office/cubbyhole...
...Opened the door (that I had unlocked earlier in the day, guessing that the programmer would simply pull the door closed behind him when he left), opened the desk drawer, and walked back down the hall to the General Manager's office...
...where I put a box of tapes/cartridges in the middle of the table and said "these are your backups. If these were to fall into the hands of our competition, this product will no longer be commercially interesting, as our competition will be providing the same service within a month. Furthermore, if there is a fire or other reason this floor is not accessible or damaged, these tapes and your primary server will be lost."
I had to explain to him that the tapes were in a box where almost everyone could get them, and it would have been frighteningly easy for someone to simply replace a tape with an empty one, and walk out with a complete backup of the code, customer data, and effectively the entire product.
They were also kept right next to the machine that they backed-up. Lose the room, lose the computer, and lose the backups at the same time.
Of course, the IT guys weren't thrilled with me, but they suddenly did have the budget to install an electronic back-up to the main telephony company's computer centre and backup environment.
The lead auditor on the job was also shocked, though mostly because I hadn't warned him.
It is easy to become complacent, thinking that technology has come so far, that the silly things we allowed to happen years ago cannot happen today. Yet there is another less we should take; IT governance is first and foremost a people driven set of processes, not technology processes - technology allows us to make those processes work more efficiently, but they remain human processes. And no matter how much technology has progressed in the past twenty years, the human remains fundamentally the same.
But back to the backups; what was on those tapes? What would have been the impact on the company if the tapes disappeared? Source code? Customer data? Test scripts? Downloaded movies? Confidential emails?
After all, the tapes are a standard format, and the operating system is standard, so there would have been a minimal challenge to recover everything from those tapes.
Sometimes we see "non-critical" systems continue to be managed as stand-alone environments, disconnected from the corporate environment, and most especially from the automated backup world.
I was tempted to pocket one of the tapes and see if he noticed it was missing. As there were only two tapes, or the cartridges that replaced "tape" decades ago, I was pretty confident that he'd notice. Cheap joke, not worth the effort.
But it did remind me of a blatant demonstration that I had to give a General Manager almost twenty years ago. Because sometimes a such a demonstration is what is needed to make the point. This is especially true with backup tapes.
We were (internal) auditing a subsidiary of a telephony company, and were in their Auckland office. The subsidiary was building a new product for corporate clients, enabling the clients to take a single file of all telephone data, already data-cubed and with a set of associated software to allow multi-dimensional analysis of that data. All very cool for the time.
The systems ran on a small set of servers, actually just very powerful under-desk PCs, up-configured about as much as was possible in that day. And as this was a subsidiary and not integrated into the telephony company, the servers were of course in a room in their offices in the Auckland office building. And of course, one of their main programmers also worked in that same cubbyhole of an office.
Furthermore, the backup tapes were kept in the drawer of the desk next to the computer. The door to that room was of course locked, I was told, even though the door stayed open all day. At night, when the programmer left, he closed the door behind him, and then unlocked it to get back in the next morning.
By the end of the audit, we had our draft report and findings completed, and we were ready to present these to the General Manager. The only time slot he had was after hours, so we sat down with him at 7pm to go through the draft report and findings.
If there is one thing that I've learned about auditing, it is that the enthusiastic nodding of the executive is as frequently faux-agreement so that you will just go away as it is agreement to fix the issue, no matter how trivial. In fact, the more trivial, the greater the chance that the nodding will actually be a "please go away" message.
We were not getting any of those messages until we got to the IT Audit findings, and then the nodding started. The issues were outside his area, and he didn't understand the issues, their severity or the what effort would actually be required (all issues had been cleared with the IT people first of course). So he entered "thank you for these very helpful findings, please go away" mode.
Being fair, most of the issues were fairly minor, but I knew that a demonstration was needed. Something to make sure that the next morning, he would actually call in the IT people and get the issues, if not resolved, at least on the agenda for resolution.
When it came time to talk about backup security, I stopped, stood up, and excused myself from the room. "Pardon me, I'll be right back", and walked out...
...down the hall to the IT office/cubbyhole...
...Opened the door (that I had unlocked earlier in the day, guessing that the programmer would simply pull the door closed behind him when he left), opened the desk drawer, and walked back down the hall to the General Manager's office...
...where I put a box of tapes/cartridges in the middle of the table and said "these are your backups. If these were to fall into the hands of our competition, this product will no longer be commercially interesting, as our competition will be providing the same service within a month. Furthermore, if there is a fire or other reason this floor is not accessible or damaged, these tapes and your primary server will be lost."
I had to explain to him that the tapes were in a box where almost everyone could get them, and it would have been frighteningly easy for someone to simply replace a tape with an empty one, and walk out with a complete backup of the code, customer data, and effectively the entire product.
They were also kept right next to the machine that they backed-up. Lose the room, lose the computer, and lose the backups at the same time.
Of course, the IT guys weren't thrilled with me, but they suddenly did have the budget to install an electronic back-up to the main telephony company's computer centre and backup environment.
The lead auditor on the job was also shocked, though mostly because I hadn't warned him.
It is easy to become complacent, thinking that technology has come so far, that the silly things we allowed to happen years ago cannot happen today. Yet there is another less we should take; IT governance is first and foremost a people driven set of processes, not technology processes - technology allows us to make those processes work more efficiently, but they remain human processes. And no matter how much technology has progressed in the past twenty years, the human remains fundamentally the same.
