Skip to main content

Measure a launch or patch

The day after a release, someone always asks "how did it go?" and the honest answer is usually a vibe. Someone read a few threads, the loudest reply was negative, and that becomes the team's memory of the launch.

You can do much better, but only if you set it up beforehand. This playbook runs in three phases, and the first one happens before you ship.

Phase 1: Before you ship​

Capture the baseline. You can't measure a change without a starting point, and after release it's too late to get a clean one.

Summarize the state of feedback for the two weeks before today: overall sentiment, the top five complaint themes with counts, and the top five feature requests. This is my pre-2.5 baseline.

Write down what you expect. Ask Kinn to save it as a note so it persists:

Save a note: we ship 2.5 on March 12. It fixes the save-corruption bug and the matchmaking timeouts, and changes the progression curve. We expect complaints about the progression change and expect save-corruption reports to stop.

That note does real work later. It means a question three weeks from now knows what 2.5 was supposed to do. See Notes & memory.

tip

Naming the things you expect to go wrong is the most useful part. When the progression complaints arrive, you'll know whether they're at the level you anticipated or well beyond it — which is a completely different conversation.

Phase 2: The first 72 hours​

This is the highest-value window of the whole release, and it's worth staffing properly. Early feedback arrives fast, it's specific, and it's the only signal you'll get while there's still time to hotfix, roll back, or get ahead of a problem with a post.

The skill here isn't deciding whether to trust it. It's telling an urgent issue apart from a passing trend — and that's a judgment you make from several signals at once, not from volume alone.

Day one — is anything on fire?

What's being reported since the 2.5 release went live? Focus on crashes, blockers, and anything that stops people playing. Flag anything appearing on more than one platform.

Did the fixes actually land? This is the question teams forget, and the one with the clearest answer:

Are people still reporting save corruption since 2.5? Compare the rate to the two weeks before release.

A fix that didn't work shows up here within a day, and it's much cheaper to catch now than in next month's retrospective.

What's new that wasn't there before?

Of the issues reported since 2.5, which were NOT being reported in the two weeks before it? I'm looking for regressions the patch introduced.

Run these at least daily, and twice a day on day one. A scheduled task works well — set it up during release week and pause it after.

Urgent, or just noisy?​

Almost everything you see in the first 48 hours looks alarming. Most of it isn't. Four signals sort it, and you want to read them together rather than individually:

Trajectory. The most useful of the four, and the one you only get by asking twice. Is a theme growing, flat, or already decaying?

Compare what's being reported in the last 6 hours against the 6 hours before that. What's accelerating, and what's dropping off?

A complaint that's flat or falling six hours in is usually a reaction settling. One that's still climbing on day two is a real problem finding more people.

Severity. Volume treats a crash and a color-scheme complaint identically. You shouldn't:

Separate what's being reported into things that stop people using the product, things that degrade it, and things people simply dislike.

Fifteen reports of a progression blocker outrank four hundred posts about a UI change, every time.

Source spread. One platform is a slice; three is your whole audience:

Which of these are being reported on more than one platform, and which are concentrated in one place?

Concentration isn't dismissal — it often means a specific device, region, or storefront, which is a useful lead in itself. Ask what the reporters have in common.

Novelty. Something that was already being reported before the release is not a launch problem, however loud it is now.

Put together: climbing + blocking + multi-platform is a hotfix conversation today. Falling + cosmetic + single-platform is something to note and check again tomorrow.

tip

Early numbers are real, they're just leading indicators rather than final measurements — the people who play in hour one aren't your whole audience. Use them to decide what to do now, and re-check the trajectory before you conclude anything about size. Phase 3 is where the number gets settled.

Phase 3: One to two weeks later​

Now the sample is worth measuring.

Compare against the baseline you captured:

Compare feedback in the two weeks since 2.5 against the two weeks before it. How did overall sentiment change, which complaint themes grew or shrank, and what's new? Chart sentiment across the whole period with the release marked.

Size the things that matter:

How widespread are the complaints about the new progression curve? Break it down by platform and by whether people are new or returning players.

See Size how widespread something is — the progression change is exactly the kind of thing where "loud" and "widespread" diverge.

Check the expected against the actual:

We expected complaints about the progression change. Compare what people are actually saying to that expectation — is it the volume we anticipated, and are they complaining about what we thought they would?

This is where the pre-release note pays off. Teams are usually right that something will be unpopular and wrong about why.

Look for what nobody predicted:

What themes are coming up since 2.5 that weren't in the pre-release feedback at all? I'm not looking for anything specific — show me what I'm not asking about.

Open-ended discovery after a release is consistently where the surprises are. The support counts it returns are a floor, not a measurement — follow anything interesting with a proper sizing question.

Reading launch feedback honestly​

A few patterns worth knowing before you interpret any of this:

  • Negativity front-loads. People who hit problems post immediately; people having a good time post later or not at all. A rough day one doesn't mean a bad launch — but it does mean a real list of things to look at.
  • Change complaints fade; broken complaints don't. Unhappiness about a change usually decays over two to three weeks. Complaints about something broken stay flat or grow. Which one you're looking at is the most useful thing to know, and time is what tells you.
  • Silence on a fix is success. Nobody posts "the save bug is gone." Measure the drop in reports, not the presence of praise.
  • Platform timing differs. Steam reviews arrive within hours; app-store reviews lag by days. Don't compare a source's day-one numbers against another's week-one.

Make it a template​

Once you've done this twice, it's the same three prompts every release. Save them as notes, or keep a scheduled task you enable during release week and pause afterward. The whole routine is maybe twenty minutes of actual attention, spread across two weeks.

Common mistakes​

No baseline. The most common failure, and it's unrecoverable after the fact.

Reading day one once and calling it. The single reading isn't the problem — treating it as final is. Ask again a few hours later; the trajectory is what tells you which way something is going.

Not checking whether fixes worked. Everyone measures the new stuff. Almost nobody verifies the old stuff stopped.

Treating "fewer complaints" as "more happiness." Sometimes it just means fewer people are playing. Check volume alongside sentiment.

Next​