Issue 5  26th May 2017

Metrics Versus Experience – The Year of the Looking Glass

• Gut instinct is no longer enough to create successful products
• Measurement and Analytics allowed us to perform better than Gut Instinct
• Metrics can be manipulated and don’t always correlate with a good experience
• But Metrics and Experience aren’t mutually exclusive
• Metrics (GOOD): a) Give you insight into behavior; b) Help distinguish good from bad; and c) Create tangible output that can be used to solve otherwise subjective opinions.
• Metrics (BAD): a) Only as good as the measuring plan; b) Tell you what not why; c) Easy to manipulate
• Metrics (UGLY): a) Do not provide qualitative insights – Do they trust me? Do they like it? What about competitors? What to change/add/fix? What will happen 1 month ahead?;
• Guidelines:
  • Product-market fit: look at retention, new users are meaningless if they don’t stick
  • Retain by analyzing the AAARRR funnel for churn
  • Plan and measure only the right metrics
  • Find the right metrics by starting with the customer outcome and work your way back to measurable data (e.g. Are recommendations valuable? IF they were, then they’d watch/share them more than other content)
  • Nullius in verba (Take nobody's word for it): question goals and metrics, what could make them fail? Is there an alternative?
  • Question yourself and use countermetrics: If I were wrong about this, how would that look like in metrics?
  • Complement with qualitative research get the full picture

medium.com  

Facebook and Snapchat: metrics versus creation

• You cannot control customers but you can measure them quantitatively at scale
• Customers would choose a linear timeline, Facebook metrics prove personalization works better
• Creation-heavy Snapchat does not rely on metrics to decide where to go next (you can only use quant data to measure success)

A bit too binary for my liking and somewhat confusing post. Quantitative data can absolutely be used to determine where to go next, even if the next is gathering qualitative data to identify unsolved customer jobs, and then move into ideation.

ben-evans.com  

Building something no one else can measure

Healthy skepticism on metrics:

• People anchor a lot on metrics for everything from people's incentives to how quickly to optimize the business.
• Metrics can be the organizational equivalent of Schrödinger’s cat: picking the metric itself can cause weird cultural distortion
  • Campbell's Law: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."
• We cannot measure behavior directly, instead we use metrics as proxies for measuring underlying behavior:
  • e.g. Clicks -> "Did I read this?”
  • e.g. Time spent -> "Do I like this?”
• Metrics turn “this makes people feel warm and fuzzy” into “this improves metric X by 5% week-over-week” which everyone can agree and act upon.
• Metrics always create secondary behavior on some other metric (good science practice suggests creating counter metrics to ensure you can see the whole whack-a-mole system)
  • e.g. “Quality Journalism”: measure by clicks, leads to click-bait content
  • e.g. “Quality Ad”: measure time spent, leads to preventing people from skipping the ad
  • Good Product Managers are aware of systems and how metrics correlate
  • This is a general problem that goes beyond tech, related article on how metrics can be short-sighted, short-term: Metrics Incrementalism and Local Maxima
• Although people are aware of metrics shortfalls, there are no clear better options, so you settle for metrics.
• The problem is that long-term metrics are hard (but possible) to monetize:
  • e.g. Google optimized for sending people away from Search, capitalizing on retention instead of time spend on Search.
  • e.g. Zappos optimized for making the experience great for returning-customers, instead of increasing checkout value.

sriramk.com