Designing for numbers you cannot picture

The first time I worked on something with a genuinely large audience, the number meant nothing to me. I could say it out loud in a meeting and it connected to nothing. A figure on a slide.

A support thread made it real. We had shipped a change that moved one control to a different position. Small change, well argued, better screen by every measure we had. The thread filled up with people who had been using the old position without looking, for years, and now had to look. Each complaint on its own was trivial. Added together, they came to more human hours of irritation than our whole design process had spent producing the change.

This is what scale does, and it never fully fits in your head. The screen is the small part. What ships is a policy, enforced millions of times, mostly on people who never asked for it and will never tell you anything.

A default is a decision made for someone else

At a small scale, a default is a convenience. At a large scale, the default is the product, because almost nobody changes it.

Whatever you preselect is what happens. The unit, the sort order, the notification frequency, the privacy setting you ship checked. That becomes the behaviour of nearly everybody. The settings screen mostly serves a vocal minority. Everyone else lives with what you preselected.

Which means picking defaults deserves more design attention than most of the visual work around them. It usually gets less. A default is a one line code change and one afternoon of arguing, so it feels small. No other decision on that product pays back as much.

My test for a default: if this were the only option, could I defend it? Because when a setting exists mainly so you can point at it after shipping a bad default, what you designed is an alibi.

Rare events are common events

With a hundred users, a bug that hits one in a thousand sessions does not exist. You will never see it. With millions, one in a thousand is a steady line of people at your door, all day, every day.

That reorders which problems matter. The empty state that only shows for people with no history. The layout that breaks past thirty characters, which sounds exotic until you look at real surnames. The flow that dies on a slow connection at the worst possible step. The date format that half the world reads backwards. You would be right to deprioritise these in a small product. At scale they turn into a full time source of human frustration, and usually nobody owns them.

And here is the trap: your instrumentation hides them by design. Analytics show aggregate behaviour, and the aggregate is dominated by the common path. A problem hitting a small percentage of people is invisible in a funnel chart and enormous in absolute terms. You will only find it if you go looking on purpose, segmenting by the conditions that produce it instead of staring at the total.

The audience itself is also stranger than any research panel. Old devices. Bad connectivity. Screen readers. Borrowed phones. Languages that break your layout in ways you did not know layouts could break. People in a hurry or in real distress, doing all of it on a moving bus. With a small product you can design for a typical user, because your users really are fairly typical. At scale there is no typical user. There is a range, and the edges of that range are full of real people.

Change is billed to the people who learned the old thing

This part clashes hardest with how product teams like to work, so it deserves its own section.

A redesign has a visible cost, the effort of building it. It also has a hidden cost: the relearning you impose on everyone who already internalised the old version. With a small user base the hidden cost rounds to zero. With a large one it can swallow the entire benefit, and the people paying it were perfectly happy with the old version.

Products that stop changing rot, and some improvements are worth real disruption. But the bar sits higher than it feels from inside the team, because the team gets all the upside and does none of the relearning.

What helps, in my experience: move the most-used controls least, and spend the disruption budget on parts people touch rarely. When something has to move, keep a bridge open for a while. Roll structural change out gradually, because a big-bang redesign concentrates all the relearning into one bad week. And ask honestly who benefits. If a change mostly makes the product easier to present in a sales deck, your existing users should not be the ones paying for it.

One more thing I learned to distrust: my own boredom. Teams tire of an interface long before users do. I have stared at some screens for two years. The user glances at them four seconds a week and never thinks about them again. Me wanting it to be different does not mean it needs to be.

What tests measure, and what they miss

Experimentation at scale is genuinely useful. It also gets more trust than it earns.

An experiment measures the days right after a change, on people meeting it for the first time. The effects that matter usually run longer. Novelty inflates the early numbers of anything new and shiny. Habit depresses the early numbers of anything that moved. Both wear off, and most tests get read before that happens.

Tests also measure what is convenient to measure. Clicks, completions, time on task. Whether someone now trusts the product less, whether the completed thing was the thing they actually wanted, whether that completion is a mistake they will reverse next week: none of that shows up unless you specifically instrumented it. A change that lifts completions and lifts reversals is a loss that looks like a win, and you only catch it if the reversal is on the chart.

So before running a test I write down what result would make me kill the change, and which number would reveal harm rather than benefit. If I cannot describe failure in advance, I know I will not recognise it afterwards.

Percentages are for datasets, counts are for people

One habit keeps all of this from turning into pure statistics for me.

A tenth of a percent of a very large audience is a stadium full of people. If a flow fails for them, calling it an acceptable loss rate is a framing trick. That stadium holds more people than most products serve in their lifetime. Written as a percentage it slides past everyone in the room. Counted as people, it is a stadium you have to look at.

So I convert percentages to people before any decision. It happens to work well in meetings too, but that is a side effect. The real reason is accuracy.

What holds up

After years of this, the practices that survived are unglamorous ones.

Spend disproportionate time on defaults, empty states and error states, because scale concentrates exactly there. Touch the most-used things as little as you can. Segment before you conclude, since the aggregate hides whoever you are hurting. Instrument the harm with the same care as the benefit.

And stay in contact with individual users, even when there are millions. Read some support threads yourself, with your own eyes. The aggregate view is what decisions get made from, and it is hopeless at telling you what it feels like to be on the receiving end of them. You need both views. The second one tells you which questions to ask the first.