SEO testing: how to know whether a change worked

Why most SEO changes are never really measured

The usual method is to make a change, wait, and look at the traffic. That compares a page to its own past, and the past is not a control. Everything else that moved in the meantime — demand, competitors, an algorithm update, a reporting change — is inside the number you are reading, and there is no way to separate your change from any of it after the fact.

The fix is not more data. It is a group of comparable pages you deliberately leave alone.

What a control group buys you

A control group does not make the outside world hold still. It makes the outside world apply equally to both halves of your test. If both groups fall, that is the season or the update. If the treatment group falls less, or rises while the control does not, the gap is the part your change is responsible for.

This is the whole idea, and it is why a test with no control is not a test.

The decisions that ruin tests

Most bad results come from three choices made before any data arrives. Groups that are not comparable, so the difference was already there. A duration decided after the numbers started moving, so the test stops on a good day. And a change too small to detect, which produces an inconclusive result that then gets reported as a promising one.

Decide the groups, the duration and what would count as no effect before you touch anything, and write them down where you will have to look at them again.

Reading a result you did not want

Inconclusive is a real answer and the most common one. It means the effect, if there is one, is smaller than the noise you can measure through — which is useful, because it tells you to stop spending on that change. The failure mode is rerunning the analysis until a cut of the data agrees with you.

A test you report honestly is worth more than a test that agrees with you, because you can build the next decision on it.