Perfect Data. Every Time

How to make your boss happy? Perfect data. ✨

But real-world data is rarely perfect. Biological systems are noisy, and human behavior, being an extension of biology, is similar. Clean data in a complex environment usually means someone filtered out the truth.

In tech, the pressure to deliver perfect data is wrapped in slogans: "Fake it till you make it." "Move fast and break things." "Do now, ask for forgiveness later."

Growth by all means

Data is supposed to show us where we stand. That was the core promise behind OKRs: leadership sets an inspiring vision, teams provide quantitative evidence. But over the past half a century (yes, they've been around since the 70's), they've become an improv theater.

At one point, I was prepping Q3 KRs ahead of a major release. One KR had to be deal conversion rate (a lagging sales metric in summer 🤡). I made a calculation based on historical data, but leadership thought it wasn't ambitious enough and demanded changes.

They expected "Yes, and…", but asked the wrong actor. Reaching their imposed target meant every sales rep needed to triple their prospects and hit 100% conversion. In the summer. It was silly. It appeared leadership was more focused on an optimistic slide for the board than the underlying reality.

Basically, I was mathematically correct, but not strategically. I observed instances where individuals who promised unrealistic targets were promoted, seemingly regardless of whether they actually hit them. It is because OKRs shouldn't link to personal gain. Their team was also uninspired, likely because the targets felt disconnected from reality. However, by the time the results arrived, we were defining the next quarter.

We moved fast, but stopped being agile. We did not learn; we just ran.

Measure as a target

This dynamic is not unique to tech.

Surveys show that ~2% of scientists admit to fabricating or falsifying data at least once, while a third acknowledge using questionable practices (e.g., dropping unwanted data points, altering protocols) because of funding pressure ¹.

As captured by Goodhart's law: when a measure becomes a target, it ceases to be a good measure.

The manipulation of numbers rarely starts as malicious fraud. It begins as a small compromise. Here are some of the most common ways to get perfect data.

Cherry Picking

Cherry picking is selecting only the data points that support the hypothesis and ignoring the rest. In the startup world, it has been shown that Theranos cherry-picked data during quality control checks to make their Edison machines appear accurate.

P-Hacking or Data Dredging

This practice involves analyzing and segmenting a dataset in multiple ways until a statistically significant pattern emerges by pure chance ². A food researcher Brian Wansink had dozens of papers retracted after investigations concluded his lab sliced pizza consumption data by age, gender, and arbitrary traits until they found publishable, yet meaningless, correlations.

Image Manipulation

This is deliberate splicing, duplication, or alteration of images to manufacture evidence of a visual result. It is quite prevalent in biomedicine, where it is suggested that ~4% of published papers contain image duplication ³. A popular case was discovered in 2022, when investigators reported finding evidence of manipulated Western blot images in one of the most important research papers on Alzheimers disease ⁴.

Metric Inflation

This involves altering the parameters to get better results. Facebook reportedly utilized this tactic with video metrics for years, with advertisers alleging in a lawsuit that the company inflated average view times by only counting views that lasted longer than three seconds and ignoring bounces. Advertisers poured money into the platform, reportedly relying on distorted data.

HARKing

The act of Hypothesizing After Results are Known, and presenting it as if it was the original plan. Many tech companies are known to struggle with this during A/B testing. Product launches a feature, fails to see anything significant, but notices a random spike in a non-related vertical. They then rewrite the launch report to claim that the spike was the objective.

Fabrication

This is the outright fiction. Dutch social psychologist Diederik Stapel fabricated entire datasets for dozens of peer-reviewed papers to prove provocative theories about human behavior, such as meat eaters being more selfish.

See your true colors

When your livelihood relies on a number, you must get it right on paper. The pressure to deliver can lead product managers to invent narratives, researchers to trim outliers, and sales teams to discount heavily to hit volume targets. Calling for honesty is naive. In highly pressurized environments, radical honesty can result in being passed over for opportunities.

In science and in tech, negative results are rarely published. If we rewarded failure, we could learn more effectively. But only great data comes with profitability, so this dynamic isn't changing soon. Until we change that, we're left with treating the symptoms, not the root cause.

Unaggregated Data

Instead of pre-aggregated dashboards or rolling averages, inspect the data across standalone time intervals. Authentic data is volatile. A smooth curve or uniform variance should make you want to dive deeper.

Change Conditions

Force teams to run their data under slightly different conditions (e.g., thresholds, smoothing,…). If a breakthrough collapses under a minor parameter tweak, you might be looking at p-hacking.

Mathematical Distribution Laws

Benford's Law dictates that naturally occurring numbers start with a 1 about 30% of the time, and with a 9 less than 5% of the time. Humans are poor at inventing random numbers and will violate this law, usually through an over-representation of zeros and fives.

Image inspection

Use software tools like Error Level Analysis to expose spliced lanes, cloned pixel clusters, and mismatched compression artifacts.

Blinded Analysis

Require analysts to write and commit their scripts before viewing raw data, eliminating post-hoc parameter tuning.

Independent Audits

Employ independent third-party auditors who have no financial stake in whether the numbers look good or bad.

A company must ultimately decide whether it wants a beautiful dashboard or reality. You can rarely have both.