PERSPECTIVE-Metrics are good. In the right hands.
A well-thought-out set of metrics helps manage performance and can be used to model future expectations.
But just as there must be effective controls in accounting for financial transactions, metrics require equal scrutiny.
Bad or poorly designed controls can bring down a company, as was the case with Enron. They leave the door open to fraud and abuse. Unethical minds can work around them, or even through them.
As we recently learned from the VA scandal, analysis based on metrics in an environment of weak controls can have catastrophic consequences.
There are two key factors that determine whether any metric system is effective:
Relevance
Evaluation
Choosing what to measure can be subject to any number of considerations, not all of them beneficial. For example, unscrupulous managers or officials may favor metrics that are likely to cast them in a good light. Some metrics may inadvertently lead decision-makers down the wrong path.
I recall a professor relating the case of a chandelier factory in the former Soviet Union. The central government planners in Moscow awarded bonuses based on production tonnage. The factory responded by making its chandeliers heavier and heavier. The weight finally reached a point where the product ripped away from ceilings, in some cases causing serious injuries, perhaps a few deaths.
The bonuses were paid, but a few managers probably made a career move to Siberia, with relocation expenses paid for by the government.
After selecting a metric, follow-up is needed to evaluate not only its effectiveness (no chandelier accidents), but the integrity of the data.
The VA was using a reasonable metric – wait time for patient appointments. However, whether it should have been a factor in determining bonuses, or even as a primary measure in evaluating performance is questionable. Furthermore, there was no process in place to assess the accuracy of the data.
Wait time is just one measure of performance. The quality of patient care depends on many other criteria, including outcomes from treatment. Were those considered in the bonus calculations?
Perhaps other meaningful criteria were considered, but for other departments. The tragedy may have been averted to some extent if performance metrics were integrated for the facility as a whole. After all, attainment of an isolated goal is no guarantee of mission success. It was a major achievement to land astronauts on the moon, but I doubt anyone would have been satisfied if they had failed to return.
Accumulating statistical data is a challenge for any organization. There is usually no double-entry (debits equal credits) system to catch entry errors. The independence of the data source is another challenge. If the units of measurement are accumulated internally, it is absolutely essential for auditors to test them for reasonableness.
Comparing results from one organization to another or against industry averages is always a good start. Performance anomalies will stick out, which should force the auditors and managers to drill deeper in the data accumulation process.
Sometimes, cross-checking one department’s performance against a seemingly unrelated area can reveal potential inconsistencies. Is the output consistent with the input? Take the chandelier company. If the units of production were fairly steady, but the raw material usage was increasing, alarms should have activated.
In the case of the VA, did anyone attempt to compare overall patient inquiries with the number of people on the waiting list? Did the call volumes to the Phoenix center correlate with the numbers on the waiting list?
The City of Los Angeles wants to implement performance-based budgeting. The approach is long overdue, but it requires careful judgment in selecting and evaluating metrics.
For example, how should the Bureau of Street Services be funded and evaluated?
The Bureau probably has some of the best statistics in the city, but how will the data be used and interpreted? Certainly, quality of work and degree of difficulty must be used in conjunction with lane miles paved.
Can the miles be fudged? Without controls and audits, yes. Everything from anecdotal observations to material and equipment usage must be compared to reported results.
Will there be rewards or penalties for the managers and workers who either exceed or miss targets?
If the Mayor and City Council are serious about using performance-based budgeting, they better beef up Ron Galperin’s staff or there may be a lot of embarrassing explaining to do down the line.
(Paul Hatfield is a CPA and former NC Valley Village board member and treasurer. He blogs at Village to Village and contributes to CityWatch. He can be reached at: [email protected])
–cw
CityWatch
Vol 12 Issue 45
Pub: June 3, 2014
