This article compares two approaches to monitoring website performance: lab-based monitoring and collecting field data from real users.
Lab-based monitoring is done by a server regularly loading your page and analyzing its performance. Another common name for this is synthetic monitoring. This is what tools like PageSpeed Insights, WebPageTest, or DebugBear do.
Field data is collected is collected in the browser every time a user loads one of your pages. Code on your website captures performance metrics and sends this data to your server. This is also called real-user monitoring (RUM).
How fast is my site for users?
Field data lets you accurately determine how fast your site loads. You can say things like "The site fully loads within 4s for 95% of users", or see a distribution of page load experiences.
Lab-based testing on the other hand captures data in a set of fixed test environments. There's a risk that these environments are very different from what real users experience.
To get the best results, you need to know what type of device and network connection your users are likely to use, so you can configure the test environments accordingly. Essentially, you measure the experience of a few hypothetical users, masking the complex reality.
Did my site get faster?
So why bother with lab-based testing if it can't tell you how fast your site is? The big advantage is that tests are run in a controlled environment.
Did your load times go down in a lab-based test? Congratulations, you've made your site faster! (Assuming the change was greater than normal metric variance.)
What about real-user metrics? Did your site get faster, or did your users come back from holiday and now have fast wifi again? Did you lose a big customer that happened to use slow devices, bringing up average performance?
Likewise, if you speed up your website you might attract more users on slower connections, who'd otherwise have bounced. These new users who now stick around will drag down your RUM metrics.
With real-user monitoring you need to dig into your metrics and figure out what happened. Here are just a few factors that affect performance:
- Device and browser
- Network connection
- User location
- Content that varies based on account data or ad targeting
- Resource competition between your page and other software on the device
In a lab environment these can be kept constant.
Why did my site get slower?
Here's another aspect where lab-based testing can shine. The test server can capture information about every request, record a filmstrip with screenshots, or collect console messages. There's no need to worry that collecting lots of data will impact page performance for real users. This makes it easier to identify what caused a performance regression.
With RUM metrics this will be much more limited, to reduce data usage and because the page doesn't have access to all the information the browser has for security reasons. If you detect a problem you'll need to investigate what caused it.
Ability to break down the collected data
Real user monitoring can be run on every page that's viewed by your users. You can then break the data down to see what pages are particularly slow. It's also possible to break down the data by country or by device.
With synthetic monitoring you won't collect any data unless you've explicitly set up a monitoring scenario for that page, device, and client location.
If a page is particularly slow in China you'll be able to see that in the broad range of data collected by your real-world data. But synthetic tests will only detect an issue if you're testing with a China-based test client.
What can be monitored?
To collect real user metrics you need to be able to run code on the page, which means you can only do it for your own sites.
If you want to compare your website against the competition you need to use a lab-based tool.
Conclusion
Real-user data allows you to see how many of your users experience slow or fast performance. You can see how urgent performance optimization are, track how different user cohorts are impacted, and monitor real user performance over the long term.
Synthetic testing in the lab limits the impact of external factors on your measurements, helping you isolate how the changes you make affect performance.