-
entries
30 -
comments
208 -
views
21,772
About this blog
Random stuff from my past
Entries in this blog
Thanks To All My Sal's Friends
I Just Got Burgled
Guest Article Winner - Desireful
Anti-blog Revisited
Power!
Communication with users
Communication with management
Communication with IT helpers (due through the door any minute now)
Monitoring our security systems
Monitoring our network
Acquiring physical maps of site (there were about seven large buildings and about a dozen small buildings on the site)
Within ten minutes I was running a department of around sixty staff. The walls were covered in maps with different coloured pins in them. As our colleagues wandered into the office, they were handed a piece of paper explaining the background of the situation, then asked to report to a scheduler who was putting together site-walking teams. Once sufficient people were in a team they were then allocated a map of one of the buildings, given a list of IP devices on the corresponding segments of network and let loose. We had about 3000 devices live on the network at 10AM when the first team started to walk the site to physically disconnect network cables. There was a wonderful air of focused, intense, busy-ness. Around 11AM my boss (the site one this time) came and joined the party. He asked what he could do to help. I suggested getting pizza in for everyone because no one was going to be going to lunch that day. By 12:30 there were less than fifty network devices still on net. Most of those were our own monitoring systems, but a few were obscure devices like the PC in the groundsman's workshed. These were easily dealt with in the next half hour. Everyone in IT agreed that it had been an exciting, enjoyable and well run operation. Some of the users were less pleased. The site was a pharmaceutical research laboratory. Chemists painstakingly synthesise unique chemicals, then run them through batteries of tests to assess their viability as potential medicines. Several batches of test results were lost as a result of disconnecting the devices, even though we could have isolated their network segments to allow the test instruments to talk to their data gathering devices. So, why the emergency measure? Well, the outbreak had gotten into a manufacturing facility. That facility produced over half of the world's supply of a medicine used to treat a wide-spread chronic condition. The manufacturing plant came within hours of having to shut down as a result and, undoubtedly, would have resulted in severe illness and death for many thousands of people had their medicine supply been interrupted. It was very exciting and very rewarding to run such a large team, if only for half a day. Thanks for reading.