A different kind of emergency

I’m used to the pager going off in the middle of the night, but believe it or not I have had “emergencies” in my other line of work (software engineering) as well. One such happened last Wednesday.

In my day job with EmerGeo Solutions Inc, I’m the senior software architect. If someone’s calling me for support, this is a very bad thing because there are usually several lines of support personnel between me and the client. At the same time it wasn’t too surprising to get a call last Wednesday evening from my colleagues, and the City of Vancouver’s (COV) Manager of Emergency Planning since I was scheduled to attend the activation of the Emergency Operations Center (EOC) for the annual Celebration of Light fireworks.

Planned events like the fireworks are the perfect way to exercise emergency planning, and activating the EOC in such an event makes sure that plans and system set up for real emergencies are tested regularly. I’ve attended the EOC several times last year for exercises leading up the Olympics. My company makes emergency management software, and the City of Vancouver is our of our clients; and while my SAR experience, and my professional experience in the Justice and Public Safety industry made me a good fit for my current position, observing an actual EOC activation was a very good way for me to see how things are done from the manager’s chair.

However, this phone call was of the “unplanned emergency” type; several months ago we had installed an addition to our core software package “FusionPoint” and addition to read data out of the city’s 911 CAD (Computer Aided Dispatch) system. It had been working for several months, showing on the status map the location and description of every Fire/Rescue dispatch in the city. In fact the system does a lot more than that (It was used during the Olympics), but you can read about it elsewhere.

Anyway, this particular emergency was that the entire system was no longer working, and even worse, nobody could log into the system at all, and the fireworks event was going to start. Like I said at the start, I’m used to emergencies and gulping down dinner before heading out the door, but this was a little different. When I got into E-Comm, all eyes were on me, and I heard someone say “hooray he’s here!” — of course this is usually a great greeting, but when the software you designed and coded is not working it’s more an admission of failure!

Well, to make a long, technical, and very detailed story short, I fixed it.

The sys-admins at E-Comm had only been allowing our software to see the Vancouver Fire Rescue data; so only those events were being processed. However, just before this event started they “turned on” the feed from the Vancouver Police, and BC Ambulance. This resulted an a lot more data going through the system.

To give you an idea; almost every call that the fire department responds to is a high priority call. In addition to fires, they respond to the highest priority medical calls along with an ambulance, and since fires are fairly rare, most of their calls are medical. The BC Ambulance feed contains all of the same events as the fire feed, and it also contains all of the lower priority calls such as patient discharges and movements. This is roughly 10 times the data of the fire feed.

But it gets better, BC Ambulance covers the entire province of BC, so the feed suddenly contained every ambulance call in BC.

On top of that came the Police calls of all priorities, which gave me a very quick education into what they deal with every day (although for security reasons the description of Police calls are limited to “Police Incident”, and the location with no additional information about priority or incident type).

The system was dealing with 50 times as much data as we had seen throughout testing and deployment, with no filtering. When I got it online, the map looked like a disaster area with about 200 active incidents displayed for Vancouver alone.

It turns out, changing the input parameters for a live system is not such a good idea, but on the up-side, this was a “test event”.

We’ve asked that changes not be made to the live system without proper planning, and that a test system be set up so that we can try out changes there before deploying to the live system. We’ve instituted a way to filter by region so we only get the incidents from Vancouver, and we’re also filtering by priority when available. Chalk it up to experience, but we really should have been given access to all of the data during the design phase so we could get an idea of what we were looking at. I’d estimate at least an order of magnitude change in the volume by the addition of the Police and Ambulance feeds.