How to test a VPN fairly

Two tests that differ in more than one respect measure the difference between the tests, not the VPN.

The rule

A comparison is only meaningful if exactly one thing changed between the two measurements. That sounds obvious and is almost never done, which is why published VPN speed comparisons vary so wildly.

What to hold constant

  • The device. An older phone can be the ceiling in both tests, which hides whatever the VPN is doing.
  • The connection.Use a cable if you can. Wi-Fi variation between two tests is frequently larger than the VPN effect you are trying to measure — move a metre and you have changed the experiment.
  • The time. Run them minutes apart, not hours. Congestion changes through the day, on your line and on the VPN server.
  • Everything else using the connection. One console update running during one test and not the other will show up as a VPN effect.
  • The test server. Use the same measurement service both times.

What to expect even when you do it right

Some variation you cannot remove. Run the same test twice with nothing changed and you will get two different numbers. That is why a single pair of tests supports only a moderate conclusion, and why repeating the pair later is the only way to know how much of the difference is consistent.

Things that are not the VPN

  • A slow baseline.If your connection is already poor, percentages become meaningless — ordinary variance is the same size as the effect.
  • Server choice.Comparing your VPN’s Tokyo server against no VPN measures geography, not the provider.
  • Protocol. Two protocols from the same provider can differ substantially. Note which one you used.

What a fair test still cannot tell you

Anything about privacy or security. Speed and trustworthiness are unrelated: a fast VPN is not a private one, and a slow one is not necessarily secure. Those are questions about logging policies, audits and jurisdiction, and no measurement will answer them.

The VPN speed test structures the comparison so the two measurements stay labelled correctly, and reports what the result does and does not establish.