Safety Management Systems

Bookmark and Share

Is SMS Worthwhile? Study Explores the Question

April 8, 2011

Safety Event Distribution A recent study of SMS implementation in 58 U.S. flight departments looks into the debate over its value; and the conclusion is…

“It depends,” said Pete Agur, founder and owner of the VanAllen Group of McDonough, GA. “If – and I emphasize ‘if’ – all levels of company management believe in their SMS plan and not only talk the walk but walk the walk, it’s extremely valuable for flight department safety. And we have solid proof.”

Agur said the development of an SMS, or Safety Management System, means building aviation best practices into the company culture and making them stick. And that, said Agur, requires the involvement of all managers, from the company CEO to the pilot in command [PIC] on each trip.

Agur presented his findings at the 19th NBAA Leadership Conference in San Diego last February. His five-year study shows a correlation between management buy-in of the SMS plan and accident or safety event history of their flight department. “It’s up to you,” he told the Leadership Conference attendees. “You must sell safety, both uphill to management above you and downhill to people in your flight department.”

Risk Tolerance Ratings Tell The Tale

The VanAllen Group study of 58 U.S. flight departments assigned a Risk Tolerance Rating (RTR) to four levels of company management. Working from the top down, ratings were assigned to the company CEO, the upper level executive to whom the flight department was responsible, the aviation department manager and the trip commander or pilot in command. Ratings for the aviation department manager and the trip commander or PIC were given double value, in recognition of their direct impact on department safety. Behaviors, as opposed to stated beliefs, were also given double value. “We listened to what they said they believed, and then compared that to what they actually did,” Agur said.

Each company manager’s RTR fell into one of five groups:

  1. World Class: (guarantees outcomes or champions safety)
  2. Best Practices: (assures outcomes or empowers safety)
  3. Standard Practices: (prevents failures or assumes outcomes)
  4. Substandard Practices: (assumes some risks or abdicates)
  5. Unacceptable Practices: (contributes substantial risk).

Over the five years of the VanAllen Group’s research, 37 percent of the companies studied suffered either aviation accidents or safety events.

“Guess which companies were most accident prone?” asked Agur. “Of those with an overall RTR that was substandard or unacceptable, 100 percent suffered accidents or safety events during the study.” He added that of companies with standard practices, only 22 percent suffered a safety event, and of those companies with an overall RTR indicating best practices, only 13 percent had a safety event.

“And of companies with an overall RTR that qualified them as world-class in safety” – he paused for effect – “exactly zero were involved in any safety events. Zero. Does conscientiously applied SMS work? The facts speak for themselves, and the data falls where it will.”

SMS Validation With Before-and-After Studies

As part of the overall study, the VanAllen Group also did “before-and-after” analyses of nine company flight departments. Eight of those flight departments had suffered safety events that forced management to change the operation.

In the “before” state, the four managers in each company averaged RTRs of 60 percent or more, putting them in the lowest RTR categories, substandard or unacceptable. In that “before” phase, 70 percent of companies suffered an aviation safety event.

“But in the ‘after’ phase,” Agur said, “the average RTRs of the managers went down to about 30 percent [equivalent to best practices], and the event rate for the company went down to about 10 percent.”

Although the International Civil Aviation Organization (ICAO) has called for countries to require operators to have an SMS in place, only a handful of countries have adopted such regulations, including Bermuda.

An SMS is not currently mandatory in the U.S. for domestic operations, and NBAA believes FAA imposition of an SMS program for non-commercial operators is unlikely in the near future. NBAA has also consistently cautioned government officials that a paperwork-heavy SMS program suitable for a large company could be a crushing burden and perhaps counterproductive for a small company. “One size does not fit all,” said NBAA’s vice president for safety, security and regulation Doug Carr.

However, operators who fly internationally will likely need to develop their own SMS soon, if they haven’t already.

That’s why the Costco Wholesale flight department, based at King County-Boeing Field (BFI) in Seattle, was an early adopter of SMS, starting in 2004. The registration effort was done largely in-house, and the company is now close to Stage 3 IS-BAO registration, said Corky Culver, the department’s administrative manager and safety officer. “It has made us a much better department,” said Culver, adding that the department’s insurance premiums have decreased over the past three years.

How RTR Was Calculated

“Risk Tolerance Ratings” (RTRs) were developed by the VanAllen Group to measure aviation safety attitudes at the four management levels that have the most influence on a flight department: the CEO, reporting executive, aviation department manager and the PIC on a trip. A total of 58 flight departments with 129 aircraft were involved; the total fleet value was about $2.2 billion, or about $17 million per aircraft, indicating that budget was not a consideration in safety attitudes.

Each manager was surveyed, interviewed and/or observed for stated aviation safety expectations and real-life actions. (“A flight should never go below MDA or DA unless the required visual cues are present.”) Actions counted twice as much as expectations, and the beliefs and actions of the two managers closest to the operations level in each company – the aviation department manager and the PIC on a trip – were given double weight. The results were converted to a scaled percentage for ease of interpretation.

Each of the four managers per company scored an RTR ranging from 0 percent or no risk tolerance (World Class) to 100 percent, or exceptional risk tolerance. (Unacceptable Practices). All four of the individual’s ratings were combined for an aggregate company RTR.

Of the 58 cases, more than half (53 percent) had RTRs corresponding to “Best Practices,” the second highest level of safety consciousness. The next largest group of flight departments rated “Substandard” (17 percent), and 16 percent of departments rated “Compliant,” the middle group. Ten percent of departments were rated “Unacceptable,” and three percent attained “World Class” status.