Modified post written to entrepreneurs by Eric Ries "Beware of Vanity Metrics"
The only metrics that physicians should invest energy in collecting are those that help them make decisions. Unfortunately, the majority of data available in off-the-shelf analytics packages are what I call Vanity Metrics. They might make you feel good, but they don’t offer clear guidance for what to do.
To avoid falling into this trap, I recommend you follow the three A's of metrics. All metrics should be: actionable, accessible, and auditable.
Actionable:
When an employee sees a report about a specific metric, it's essential that they have some idea how to replicate the result in the report.
Accessible: Most data warehousing systems provide reports that are too complicated to read and take too long to generate. As a result, many teams don't get any benefit out of them. In order for reports to be worthwhile, it's essential that:
Auditable: The biggest benefit of actionable metrics is that they can be used to reap the biggest savings in all of product development, when they tell you that you don't need to do something. Metrics have to be credible to the people who drive the product vision, including company founders. Even if they understand a report, and know what caused it, that doesn't mean they will actually learn. Is the report accurate? Believe me, when it's your idea on the line, it's much easier to believe the report is the problem rather than the idea.
Thus, it's important that skeptics can audit a report. When possible, this should mean that report generation is simple. Hopefully it can be created with direct access to primary data. For example, a report about revenue that's generated directly from the master orders database is more credible than one that requires several intermediate steps. Most importantly, it should be possible to translate the summary numbers in the report back to the actual customers who generated them.
Remember, metrics are really reports on people. This is where most off-the-shelf metrics packages fail.
To avoid falling into this trap, I recommend you follow the three A's of metrics. All metrics should be: actionable, accessible, and auditable.
Actionable:
When an employee sees a report about a specific metric, it's essential that they have some idea how to replicate the result in the report.
Accessible: Most data warehousing systems provide reports that are too complicated to read and take too long to generate. As a result, many teams don't get any benefit out of them. In order for reports to be worthwhile, it's essential that:
- Everyone in the company understand how to read them
- Everyone in the company has easy access to the latest data
Auditable: The biggest benefit of actionable metrics is that they can be used to reap the biggest savings in all of product development, when they tell you that you don't need to do something. Metrics have to be credible to the people who drive the product vision, including company founders. Even if they understand a report, and know what caused it, that doesn't mean they will actually learn. Is the report accurate? Believe me, when it's your idea on the line, it's much easier to believe the report is the problem rather than the idea.
Thus, it's important that skeptics can audit a report. When possible, this should mean that report generation is simple. Hopefully it can be created with direct access to primary data. For example, a report about revenue that's generated directly from the master orders database is more credible than one that requires several intermediate steps. Most importantly, it should be possible to translate the summary numbers in the report back to the actual customers who generated them.
Remember, metrics are really reports on people. This is where most off-the-shelf metrics packages fail.
