Evaluate an Organizational Metric

Measurement of our critical processes is the vital signs of organizational health. Yet many organizations make inferences, which drive decisions on such poorly represented data. Managers and analysts that work often with their enterprise’s outcome data play an integral role in the interpretations of that data and its impact on leadership decisions. Those stakeholders can often provide value to executive leadership when evaluating the assumptions associated with a metric your organization regularly utilizes to evaluate its performance in that they:

 

  • Assist the enterprise in knowing what to collect and what to report
  • Guide decision makers to interpret the measures
  • Strengthen the value of the measures that currently drive operational decisions

Need more information on Patient Safety resources or how to get involved with the community? Connect with Adam Bazer on Twitter @AdamBazer or LinkedIn.

Look at a Specific Metric You Use to Make Decisions

When evaluating the foundations behind a specific metric your organization uses to make strategic or tactical decisions, make sure you can:

 

  • Answer the question “How do we know we got there”?
  • Trace the metric to a specific strategic goal
  • Confirm enterprise-wide reporting standards where utilized in collecting the data
Evaluate the Source & Creation of the Metric

Begin your evaluation of the organizational metric by seeking answers to the following questions focused on the source & creation of the metric:

 

Source

  • Is the data source(s) available/listed with the metric?
  • Who commits to collect and report the measure and at what frequency?
  • Who needs to receive the measure?
  • When is the last time you checked with the ones using the measure as to the value to them?
  • Is confidentiality appropriately secured?
  • Are there other people/departments that might gain from the measure?
  • Do those who impact the measure see the results?

 

Creation

  • If there is not a direct measure, is there an indirect correlative alternate?
  • Is the cost of producing the measure justified?
  • Is the measure clear to all, or does it need an explanation in the graph?
  • Is the metric’s form appropriate?
  • Can you use a smaller set of data with faster feedback instead of larger data with slower feedback?
Evaluate the Accuracy & Efficiency of the Metric

Continue your evaluation of the organizational metric by seeking answers to the following questions focused on the accuracy & efficiency of the data:

 

Accuracy

  • Is it representative of the entire population?
  • Is it statistically sound?
  • Is it placed in the appropriate control chart?
  • Is data graph data over time if possible?
  • Is it sensitive to important changes so they can be acted upon?
  • How can you insure integrity of the measure over time?
  • Does it indicate trends appropriately (six successive data points steadily in an upward or downward constitute a trend)?
  • Does the metric have value Blind Spots (no see) or Dead Zones (no effect)?

 

Efficiency

  • Can it be electronically collected and if so, is it?
  • Can a sampling be representative alternative of 100% collection?
  • Can the measure be more useful as a flag in an exception report?
  • Does it display significant digits: rounded whole numbers, decimal points, etc. so that extra specificity doesn’t get in the way of significance?
Evaluate the Measuring Frequency & Effectiveness of the Metric

Continue your evaluation of the organizational metric by seeking answers to the following questions focused on the measuring frequency & effectiveness of the data:

 

Measuring Frequency

  • How often does it need to re-sampled or rerun?
  • How ‘fresh’ or current does it need to be?
  • When should the measure be discontinued, does it cause action or just provide assurance?
  • Are there parameters for when to stop the measure known?

 

Effectiveness

  • Does it measure the entire process (if appropriate)?
  • Does it measure efficacy, efficiency, effectiveness, or accountability of a system?
  • Is data reporting sensitive to the decisions it’s supposed to drive?
Evaluate the Interpretation of the Metric

Complete your evaluation of the organizational metric by seeking answers to the following questions focused on the interpretation of the metric:

 

  • If metric measures a cycle time, are the start and the stop points clear?
  • Does it require interpretation to have meaning, if so whom do they contact?
  • Is it a process component piece of a larger measure?
  • Are special causes (out of limit points) annotated or fully explained?
  • Do the graphical displays tell the truth?
  • Does it represent significant change, does it infer with integrity, does the scale accurately reflect reality? How do you avoid the manipulation of the data to make it “look good”?