Many Everett Emergency Dispatch Issues Related To Hardware, Not New World

October 20, 2016

Everett, Everett Fire

dispatch issues

One year after a new dispatch system went online in Everett the issues appear to be as much hardware related than software.

Back on Sunday October 9th we reported on dispatch issues for Everett Police and Fire after a failure in the Computer Aided Dispatch system or CAD. Commonly referred to as New World, the computer aided dispatch system is actually made up of several parts both software and hardware that all need to function in concert.

New World is the software component of the new CAD system. It’s had an admittedly shaky start but has been in full service now for the past year. In checking on the 5 hour dispatch down time for Sunday October 9th, SNOPAC911 Director Kurt Mills provided MyEverettNews.com the following email sent out to all users of the New World System from their IT Director…

All,
Sunday at 1412 we experienced a system outage which impacted the entire New World Systems suite. The issue has been traced to a network hardware failure. Our technical teams have already taken steps to mitigate the issue and we will be replacing those components with more robust hardware. The loss of network meant that none of the NWS applications, system backups, failover mechanisms and other peripherals could communicate and therefore did not function. Technical teams at both agencies worked until the system was fully restored at 1910.

During the outage backup systems were used and there was no data loss. Manual entry is underway and should be completed NLT tomorrow.

Thank you, Steve Lawlor | Interim IT Manager / Server Administrator

For that outage the dispatch issues were not with the software but a hardware failure. MyEverettNews.com requested copies of the dispatch error logs and emails between users and administrators that spoke of problems with the CAD system over the past several months. The overwhelming majority of those emails and error log entries ended up as hardware or equipment connectivity problems.

Here’s one example of a connection problem…

New World problems all day!

Despite multiple reboots L1 hasn’t shown up on mapping today, also routing hasn’t worked.

For this call L1 was in station available, we added ourselves because we weren’t dispatched and received verbal confirmation from SNOPAC. I rebooted (again) enroute and New World showed us available and shortly thereafter dispatched us on a FAC (121400) while still on the call.

The result after investigation…

L1 GPS plug into dock was loose. COE IT fixed this last Thursday.

Another example of a communications issue…

B1 added self to MedX at 3327 Hoyt, mdc screen was frozen when entered rig, sometime during response it updated to include me. Then when B1 went to go in-service from call 25 minutes later, the mdc screen showed half the city in-service, when in fact they were all on calls. It showed M1 as AIQ while they were transporting the patient. Suddenly it updated showing all rigs on calls (as they were).The lack of reliability of “apparatus status monitor” and mapping screen, which are some of the main sources for the BC to maintain situational awareness has created a very dangerous environment for both the citizens and efd personnel.

An oft cited reply after an investigation…

Frozen MDC’s result from cell connectivity loss. Cell connectivity wanes with bandwidth user saturation and with cellular dead spots.In the near term, GX450 Sierra Wireless combined external celluar and GPS antennae will be added to front-line units with Battalion 1 being first. At the same time, a proprietary Verizon network for public safety is being developed to follow soon after. These are components that should correct connectivity in general. The dead spots should have less impact. In addition, verizon is contemplating an additional cell tower in the middle of the dead spot that impacts reception on evergreen Way from about Madison to about 33rd street.

dispatch issues

The city has had to spend extra money to replace and install new modems in Fire Department vehicles.

MyEverettNews.com asked the City of Everett to explain what had been going on with the dispatch issues and connectivity problems and what kind of fix was being applied. Over the past few months more than $48,000.00 has been spent to outfit Everett Fire rigs with new modems. In addition the city is trying to fine tune the Fire Response Plans (FRPs) which decide what order equipment should be dispatched in based on location of nearest available fire equipment.

Assistant Fire Chief Rick Robinson provided us the following explanation via email…

Effect of new modems on performance…
“Yes, we believe that performance has improved, and that allows us to more accurately isolate any software or other hardware problems we may encounter. We installed 22 modems between September 8th and October 10th. To this point, it looks like the new modems are providing more accurate GPS data and more data throughput between our MDC’s and SNOPAC’s NWS CAD system. I have noticed that there have been fewer on-radio requests by crews asking SNOPAC to repeat info because their MDC is malfunctioning.

Definition of FRP’s: The FRP’s (Fire Response Plans) are logic sequences which determine which unit(s) or vehicle(s) to respond to a 9-11 call in a certain location such as a station area. For example, for a Fire Alarm activation with no fire or smoke noted, at an apartment building in Station 4’s area, the logic sequences like this:

  1. Send Engine 4 to investigate.
  2. If Engine 4 can’t get there in 4 minutes, send the next closest Everett fire engine.
  3. If that Engine can’t get there in 4 minutes, send the next closest Everett ladder apparatus.
  4. If no Everett ladder is available, send the next closest engine in the area (normally either Fire District 1 to the south, or Mukilteo to the west.)

Remarks: The logic sequences provide a logical basis for response, but sometimes intervening conditions such as a human-induced difference between where CAD thinks an apparatus is and where it actually is, missed human keystrokes, unrealistic street attributes (mph, stop intersections, etc) which determine calculated response time for response routes, and misplaced address points on the GIS map, need to be groomed. We have a process in place to pass that info to the SNOPAC GIS expert and adjustments are made as needed.

And, very infrequently, the software does not logically select an apparatus. I know of only one recent example; Engine 5 was dispatched as a support unit for a medic call from its station at Madison Street and Colby Avenue to the 10th Street boat launch on the north end of Everett’s waterfront (Medic 1 was on a different emergency), while Aid 2 was in station and ready to respond from Station 2 at 16th Street and McDougall; an obviously closer location. Investigation at SNOPAC showed that the response would be duplicated if Engine 5 was in Available status at Station 5, and Aid 2 was in Station 2. However, if Engine 5 was actually noted as “in quarters”, even though it was at the same location, Aid 2 was properly sent. This problem is extremely puzzling and was forwarded to Tyler Technologies, and they have worked continuously on it and are reporting they have made progress on it.

The goal is 100% correct dispatches for those apparatus with the new GX450 modems. The only two vehicles which retain the old modems are not subject to GPS-based dispatch, Technical Rescue 1 and HazMat 1. These vehicles are dispatched on an as-needed basis and so require less GPS support.

We will continue to monitor our results and to investigate problems as they come up. I check the tracker sheet daily. Crews are prompted to report problems – that’s really the best way to monitor performance of accuracy of dispatch.”

As an aside, it looks like there are no inexplicably incorrect or questionable dispatches reported since the new modems have been installed. More may be reported and will be addressed as they become known.

MyEverettNews.com asked the city administration if they were satisfied with the progress that’s been made with the New World System. In June of this year Mayor Stephanson wrote a letter critical of the system to Tyler Technologies, owner of New World. On August 3rd, company representatives came out to Everett for a meeting with the Mayor and outlined their plans moving forward. Here’s the response to our inquiry via email from Everett City Public Information Director Meghan Pembroke…

“The City is encouraged by the resources and attention that Tyler Technologies has assigned to our region in order to provide a reliable system for our first responders and residents. The dedicated project manager is now on site and has been working closely with SNOPAC and our staff to understand our concerns and develop solutions. We believe we are moving in the right direction, and will continue to monitor progress on outstanding issues and communicate with the SNOPAC and Tyler teams.”

One other issue on top of connectivity has been the aging radios used by the Everett Police department. We asked about the status of that equipment and received the following in return…

“In response to your question about police radios, at this point Everett PD, as well as the other SERS agencies, will be working with SERS regarding an overall system upgrade that is going to be needed by 2020. Replacement of mobiles and portable radios will likely be tied to this upgrade. Specific technical needs and specifications will be directly related to the project.”

SERS is the agency that is in charge of the actual radios, antennas and equipment emergency agencies use to communicate in Everett and Snohomish County.

MyEverettNews.com has observed dispatch issues caused by both the New World software and the hardware and equipment used by the City of Everett and SNOPAC911. We know both are still a major concern to all of the parties involved. Monitoring and fine tuning are ongoing and we’ll continue to follow the progress that’s being made and report on the issues as they are brought to our attention.


,

About My Everett News Staff

My Everett News is a hyperlocal news website featuring news and events in Everett, Washington. We also cover City of Everett information and items of interest to those who live and work in Everett.

View all posts by My Everett News Staff