Measuring The Washington Post’s Progressive Web App

Yesterday, I examined how The Washington Post’s Progressive Web App performs on iOS. Today, I want to take a look behind the scenes on the difficulties I had conducting that analysis.

Often when organizations make big changes to their site, it is difficult to compare the old site to the new site unless you work for the organization and have access to their development servers.

Fortunately, the Washington Post hasn’t rolled out the Progressive Web App to all of their users yet—it is currently only being served to 10% of visitors. So most people are still seeing their old m-dot site.

You can opt in to the Progressive Web App by visiting washingtonpost.com/pwa/ on a mobile device. This meant that I could compare the behavior on iOS of the Progressive Web App versus the old mobile site.

It can be difficult to tell the difference between the old mobile site and the Progressive Web App because they are visually similar. One tell-tale sign that you’re looking at the Progressive Web App is the load time indicator present under the byline.

If you see the load time, you can be confident that you’re looking at the Progressive Web App. Even if you visit the opt-in URL, if you don’t see the indicator, it may mean that you’ve navigated too quickly and the PWA’s application logic hasn’t taken over navigation between pages yet.

I started digging into the Washington Post’s Progressive Web App where I usually start—using WebPageTest. It is my go-to tool when it comes to measuring performance.

However, I found it nearly impossible to measure the Washington Post using WebPageTest for these reasons:

• Measuring the Progressive Web App’s home page isn’t sufficient. Most people are reading articles which means that we need to look at differences in article loading behavior.
• If we want to measure article pages, we have to visit multiple pages so that we can opt in to the Progressive Web App before viewing the article pages. This requires WebPageTest scripting.
• Simply navigating to an article page doesn’t tell the true experience either. When someone selects a link in the Progressive Web App, the page content is downloaded via XHR instead of a full page load. So to replicate that experience, we need to have WebPageTest “click” the link instead of navigating to a URL.
• We can’t use iOS emulation in WebPageTest because we need to know how iOS devices behave.
• The WebPageTest script that sends click events doesn’t work in Safari. Not to mention the fact that real iOS devices in the WebPageTest suite are notoriously finicky.

In the end, I was forced to give up on WebPageTest. I also tried measuring page size and other behavior using Charles Proxy, but found that ad loads varied so greatly from one page load to another that I couldn’t get any meaningful data.

Some of these challenges were unique to The Washington Post and the fact that I wanted to measure iOS, but some challenges—for example, the need to script multiple pages—will be obstacles anyone wishing to measure the performance of a Progressive Web App will face.

After striking out on my other attempts, I decided to record video of the page loading on my iPhone and then compare the videos. I tethered my phone to my computer and used QuickTime to record the screen.

I also used Charles Proxy to simulate slow networks. Charles has presets for various network speeds, but those presets seemed much too fast (16384 kbps down for 4G?) so I copied the settings from the Chrome Dev Tools throttling settings.

Then I used iMovie to combine the movies and add a timer.

The end result wasn’t ideal, but it was sufficient to be able to see what was happening on iOS and write about it.