Won’t Fall Down
The Entuity: Out of the Box blog series highlights the efforts, insights, and stories from actual Entuity employees. This week, we interview Larry Balon, our System Engineer & Software Consultant, who discusses his unexpected introduction into network management, working through disasters, and his “childish” first job.
Larry, How did you get into network management?
I worked for a systems integrator that specialized in deploying information security solutions. My company took on a consulting project for one of our service provider customers to design, test and deploy a network management solution for a service provider.
Even though I had little knowledge of network monitoring or SNMP, I was selected as the primary consultant for the job. I successfully designed, tested and deployed a network management system that monitored the service provider’s DSL access multiplexers and their class 4 Voice over IP central office switch.
For my first job, I worked for the Hasbro Toy Company in Pawtucket, RI and worked on the manufacturing line. I worked a machine on the manufacturing line and I made Weebles.
Tell us about a passion of yours.
I do service work for my church by volunteering for week long mission trips. My church works with a national relief organization.
Each trip is one week long and each person on the trip has to raise the money for airfare and lodging. I’ve helped rebuild homes in New Orleans (Hurricane Katrina), Birmingham, AL (tornadoes from 2011) and Shamoakin, PA (floods from 2013). I also did Saturday day trips to Springfield, MA to help clean up from the tornadoes (2011).
What reality TV show would you most like to try out for, and why?
I love to cook and would like to compete on either Master Chef or Chopped. Unfortunately, I am not a chef, but a cook. That means I can follow a recipe but don’t have the skills to “wing it” and create a dish or meal from scratch. But I can dream . . .
What is your favorite network disaster story, and what do you think could have been done to prevent it?
I worked for a network hardware manufacturer and I was the local System Engineer. We had a local supermarket chain as a customer. They had 70 stores and each store had 2 telephones for in-house communications. All 140 voice circuits went through our switch.
I was on-site with the local Field Engineer, and there was a problem with the main processor card in our system. The system was fully redundant and the local Field Engineer and I discussed pulling out the active processor card to force a switch to the redundant card. In theory, this should have worked and not caused a service disruption.
Without completely thinking it through, I told the Field Engineer to pull the active processor card. It took down all 140 voice circuits. Within seconds, the telephone switch’s audible alarms were clanging indicating a major failure. Within 2 minutes, the CIO of the supermarket walked into the data center where the Field Engineer and I stood there with a “deer in the headlights” look and we had to tell him that we took down their entire voice communications network.
Had I looked at the redundant processor card, I would have noticed that there was an LED illuminated that indicated redundancy was not working. Always think things through before making a decision.