Common sense is good; enough of it is genius

Category: performance (page 1 of 1)

Progressive Content Management Systems

Tl;DR

This post summarizes a recent talk Weston Ruter and myself gave at Chrome Dev Summit , 2018:  Content Management Systems (CMS) are software platforms designed to simplify the creation and management of websites and their content. Nowadays, about 50% of websites are powered by some sort of CMS platform. Below, we discuss our experience moving a complex CMS (i.e. WordPress) forward along the Progressive Web Road by integrating modern capabilities (e.g. Service Worker API) and progressive technologies (e.g. AMP) into its platform and ecosystem, and how the lessons learned can be applied to other CMSes.

It is all about UX

As web developers our goal is to maximize the joy our users feel when engaging with the websites we build or the web content we create and publish on the web.  And although User Experience is a complex matter, there are common elements in web experiences that users love.  Specifically,  users love…

  • Sites that load quickly consistently and offer good runtime performance
  • Sites where they feel safe sharing their data and doing transactions on them
  • Sites that are accessible and allow them to take advantage of the full capabilities of their devices, and
  • Sites that offer great content quality

The good news is that we can build experiences that fulfill these pillars if we take advantage of modern capabilities the web has to offer. Doing this is what we call Progressive Web Development: developing user-first experiences using modern development workflows and APIs, following coding and performance best practices, and taking advantage of incentive mechanisms (e.g. engagement increases) and validation structures (e.g. performance budgets and others) which are available to us.

The way we go about taking advantage of Progressive Web Development depends on how we go about our creative process in the web.

Web Development Process

The things we create on the web fall in a general sense into two areas. On one hand, we can build websites or web apps from scratch, in which case we have full control of the creative journey: from the build process, to the functionality, to the look-and-feel, of our website or web app. On the other hand, we can use some kind of Content Management System, or CMS for short, which are Software platforms providing layers of abstraction and capabilities to make easier to create, publish, and consume digital content on the web.

The first route gives us a lot of power and flexibility, but it is also it requires quite a bit of expertise and resources. The second route, allows us to take advantage of the functionality provided by a CMS, but the experience we provide to our users depends a great deal on the specifics of the CMS we choose.

Contemporary Content Creation and Publishing

Nowadays the majority of experiences in the web are created, published, and consumed via some kind of CMS: More than 50% of the web is powered by some form of CMS; and this year there was an 11% YoY growth! [bit.ly/w3tech-stats]. And there are many other platforms that do not consider themselves CMSes but also provide similar capabilities that allow users to create and publish content on the web easily. Therefore, this number is actually much higher in reality.

In practical terms, this means that the CMS space is highly populated and complex. There are many CMS options out there for users to choose from and they come in many shapes and forms, varying across dimensions such as: Static vs. Dynamic, Proprietary vs. Open Source, Special Purpose vs. General Purpose, and others.

But although there are differences between them, they also share lots of commonalities, and many of them face common challenges when it comes to integrating progressive web capabilities into their platforms. To reason about how to tackle such challenges across the CMS space, it is useful to abstract our thinking about these platforms in terms of the fundamental components that make them.

CMS Ecosystems

Each CMS has in one way or another a set of components that form a unique ecosystem. There are usually the following components:

A platform Core: which encompasses the main functionality provided by the platform; a mechanism for extending the core functionality of the platform. Such extensions are known as modules, plugins, and themes; a community of developers who are in charge of implementing and evolving both the core and the extensions for the platform; and a user base who either create content on the platform, or consume content from the platform, or both.

When we talk about progressive CMSes, we are referring to bringing the pillars of PWD to the components of a CMS Ecosystem; specifically, things such as:

  • Bringing coding and performance best practices to the development of both the platform core and the extensions
  • Introducing incentive mechanisms for extension developers to do the right thing
  • Providing tooling to developers to make it easy for them to do the right thing

In order for a CMS to fulfill its potential, all these components need to walk together along the progressive road.

In this post we will describe specific results of efforts our team has been pursuing in the context of a specific CMS, and we will touch upon some of the great work being done by other  players in the CMS space.

Progressive WordPress

Our team has been pursuing efforts to bring progressive web technologies to the WordPress ecosystem. WordPress is certainly one of the most influential CMSes out there, as reflected by the fact that it powers about ? of the web today.  Given the size and complexity of this ecosystem, addressing its challenges will not only have an impact on a very large swath of the web, but it will also help us to shed light on how to address similar challenges in other CMSes.

Our goals are two fold: to tackle the challenges for the WordPress ecosystem, and to distill and generalize the learnings and solutions we gather along the way and bring them over to other CMSes in the form of objective guidance, tutorials, and technology transfersOur approach to achieve these goals consists of two parts:

  • Integrating modern Web APIs into the WordPress ecosystem, either via the core of the platform or via plugin-enabled capabilities.

AMP: A Well-lit Path to Modern Web Development

AMP is a web component library, together with a runtime system and a validation framework. AMP provides, out of the box, a set of capabilities and optimizations addressing essential aspects of high-quality web experiences. These include:

  • Coding and performance best practices, which enable great runtime performance (i.e. no un-responsive content) and great usability (i.e. no content shifting)
  • Performance incentives, such as the ability to achieve near-instant loading times via pre-rendering when AMP content is served from an AMP cache
  • validation framework that guides the development process towards highly performant sites, and makes it easy for developers to keep their sites in that state over time

Enabling AMP content creation in WordPress means that WordPress developers can take advantage of these capabilities without deviating from their standard content creation workflow in the platform. Such integration has been enabled via the official AMP plugin for WordPress, which is currently at version 1.0-RC2 (available on Github), and version 1.0-stable will be released at the end of this month (November, 2018).

The Official AMP Plugin

To enable WordPress developers to take advantage of AMP’s capabilities, AMP content creation must be seamlessly integrated into the standard content creation workflow in WordPress. We can envision a path where AMP-compatible markup is generated by WordPress core itself, as an option, similarly to what happened when HTML5 became a thing. Alternatively, such seamless integration can be achieved by a plugin.

The official AMP Plugin for WordPress is a product designed to fulfill two main roles. First, to enhance the development process of AMP experiences in WordPress, such as AMP-compatible plugins and themes. And second, to ensure that AMP compatibility is maintained as a site evolves, so that no invalid AMP is ever served.

The plugin is not a turn-key solution to the “how to AMPlify my site” problem; instead, the AMP plugin functionality substitutes AMP components for their corresponding HTML ones where required and possible (img => amp-img, video => amp-video, etc.), and it aids the development process by exposing sources of validation errors, providing tools to understand the context of such validation errors (e.g. preview integration), and ensuring that only AMP-valid content is ever served for a given WordPress site. These capabilities make it easier for developers to build high-quality AMP experiences (e.g. full sites) and AMP-related solutions (e.g. editors); and enable the ecosystem of developers making themes (e.g. Genesis) and plugins (e.g. Jetpack), to tackle AMP development in a WordPress way, that is, in a way that is integrated with the regular WordPress development workflow and theme/plugin ecosystem.

In its second role, the plugin functions as key enabling component ensuring a site states AMP compatible by keeping developers, implementers, and site owners, always aware of the compatibility status of their site.

The plugin provides all its functionality in an opt-in manner. And users that do opt-in have the option of enabling AMP only on specific sections of their site, and also disable AMP at a granular level (e.g. on a single post).

Integrating Modern Web APIs into the WordPress

The second part of our approach to bringing Progressive Web Development into WordPress consists of integrating modern web APIs and capabilities into the WordPress core platform. We want the WordPress platform to support natively things such as: Service Workers, Offline access, Streaming, background sync, etc. The goal of such integration is to provide a consistent and unified approach for WordPress developers to easily access these APIs and capabilities as they develop core functionality, as well as themes or plugins.

In addition to our work on AMP for WordPress, our team focused this year on devising and implementing the integration of some of these building blocks into the WordPress platform, and we did this with an emphasis on doing it in a way that is seamlessly integrated to how things work in the WordPress platform.

Service Workers in WordPress, the WordPress Way

The Service Worker API is at the center of many of the advanced capabilities we want to enable natively in WordPress. Nowadays, it is certainly possible to use Service Workers in WordPress, but doing so poses several fundamental challenges that need to be addressed.

First, there needs to be a unified API access for creating and manipulating service workers in WordPress. Without it, if multiple plugins and themes want to leverage the capabilities of this API, they would have to do all the work by themselves, not able to take advantage of any reusability, and running the risk of having collisions among multiple service workers installed on a site, which is not allowed by the API.

Second, providing access to the Service Worker API is just one part of the solution needed; programming the logic of Service Workers is by far not a trivial task. If we want all developers to take advantage of the power offered by Service Workers, an abstraction layer must be added in WordPress that makes doing so easier for developers.

Finally, one of the virtues of WordPress is the flexibility and versatility of its extension mechanism: plugins and themes. The challenge is that these extensions are not sandboxed and consequently, it is very difficult to implement advanced capabilities such as the App Shell model in WordPress because down the road additional plugins can be installed and they can alter/modify the content of a site in ways that breaks these capabilities.

To address these challenges, we have implemented abstractions in WordPress to integrate the Service Worker API into WordPress in a way that is applicable not only to WordPress but also to other CMSes. These abstractions cover three main aspects:

  • A low-level API in WordPress that provides direct access to Service Workers, so that all themes and plugins can each include their own logic in a single installed service worker.
  • An integration of Workbox, which is a library providing additional layers of abstraction to  facilitate developing capabilities on top of the service worker API, eliminating the need of writing of complex JavaScript code that would otherwise need to be written from scratch  over and over again.
  • Since WordPress developers have historically been PHP-first developers, we have implemented an abstraction layer in WordPress that translates PHP API calls into the corresponding Workbox JavaScript code which is then included in the service worker.

Progressive Capabilities in WordPress

Once service workers are accessible in a unified way, many advanced capabilities  are enabled. Some of them are the App Shell model, and offline access capabilities. 

The App Shell Model in WordPress

One of such capabilities is the app shell model. An app shell facilitates features such as seamless page transitions and offline interactions, similar to the experiences provided by native applications. But implementing an app shell is normally very challenging to do in WordPress due to the lack of sandboxing between plugins and themes.

This is where AMP plugin plays a key role because it enables the enforcing of constraints to ensure that the content on a site behaves in a way that doesn’t break the app shell model, while also ensuring the content loads in a performant way. Furthermore, AMP pages are also specifically designed to be embedded in this way via the Shadow DOM API.

In this demo site we have a WordPress site using the default theme and implementing the App Shell model. Pay attention to the fun spinning logo. Notice what happens when we click to navigate to the “About” page and other pages in the nav menu:

  • The logo keeps spinning without resetting.
  • And the nav menu remains expanded with the current page being highlighted.
  • There is no flash of white in the header.
  • Only the actual content of the page is changing, and it does so with fading animation.

To give you a glimpse of how this is accomplished on the user side, here are some snippets of code any WordPress developer would use in order to take advantage of the capabilities provided by the integration of Service Workers into WordPress core.

First of all, to address the problem of no sandboxing, we leveraged the AMP plugin to make all pages of a site available in AMP. This can be done with a standard add_theme_support() call.

Second, we introduced a way of specifying the dynamic part of the template which changes as you navigate the site. Specifically you can see this via these two function calls placed before and after the content.

And lastly, we extended the AMP plugin to add logic to the page as well as the service worker to implement the app shell. All a theme has to do is to opt-in with another add_theme_support() call like you see here.

The bottom line from these examples is that the complexities of dealing with Service Workers are abstracted away from the theme developer and absorbed directly by the platform.

Offline Access

Beyond the app shell model, the service worker is useful for many other advanced capabilities. As mentioned previously, the service worker also enables offline access.

WordPress themes include a 404 template which is served when the site can’t locate the requested content. We’ve extended the WordPress template system to introduce an offline template which is served when the user’s browser can’t access the site. Thus, in addition to a 404.php template, WordPress theme developers just have to add an offline.php like to their site in order to customize the user experience when facing the loss of connectivity.

The service worker is also useful to cache content the user has visited (or may visit) so that if connectivity is lost, they can continue accessing that part of the site’s content when they are offline. In the snippet above, you can see how the stale-while-revalidate serving strategy is added for navigation requests, so that anything previously visited will continue to be available.

When the user wants to provide a comment on an article but are offline when they try to submit it, the service worker can intercept the request and replay it later via background sync when the user is back online. The developer doesn’t have to do anything as all this happens by default.

Progressive CMSes: Beyond WordPress

The CMS space is moving forward steadily along the progressive web road. Google is working and supporting many of the platforms in this space to help them accelerate their pace, and we are seeing great advances by several of them. Including…

  • Drupal is another popular open source CMS. Their developer community is driving the development of AMP and PWA modules for the Drupal platform. Both modules are available and used by Drupal sites today and they are actively working on enhancing them.
  • Magento is one of the leading e-commerce platforms and they have been strong advocates for PWAs on their platform for a while now. They will be releasing Magento 2.3 in November and that release will include the launch of PWA Studio, a set of developer tools and guidelines, to make it easier to build great web experiences on their platform.
  • Adobe Experience Manager is a popular enterprise CMS; they have made it possible to create and publish Single Page Applications directly in the Experience Manager Page Editor and they’re working to provide direct support for Service Workers in the near future.
  • EC-CUBE is a leading eCommerce platform in Japan; they are working, together with a development agency called Sunday Systems, on an AMP plugin, which will enable the creation of AMP content on their platform.

It is exciting to see platforms like these becoming more progressive, and we are looking forward to continued progress by all of them.

The Road Ahead

We have made significant progress on our efforts in the WordPress ecosystem, but there is still some way to go to get to where we really want to be. We are continuing our efforts described in this post, including the following aspects.

The AMP plugin for WordPress is available on Github today. If you are WordPress developer, the plugin gives you the tools and support you need to make your themes and plugins fully AMP compatible. We are working on expanding the integration of AMP content creation in WordPress, especially in the context of Gutenberg, which is the new editing experience in WordPress.

The integration of the Service Worker API and other PWA capabilities into WordPress core is well underway. These capabilities are currently available as a Feature Plugin, which you can download and experiment with today.  We are continuing this effort, and we are working with Automattic and others in the WordPress ecosystem to get the integration merged into core some time not too far in 2019

Finally, we are continuing working with the WordPress ecosystem to help them take advantage of progressive technologies at all levels of their ecosystem.

Progressive WordPress

If you navigate to whatwebcando.today, you will see which Web APIs are currently natively supported by the browser you are using. Although there is some variability between browsers, the important thing to realize is the number and richness of the web APIs that are standard today. With them we can implement awesome and complex web experiences running natively on web technologies: from accessing device features such as vibration signals, the gyroscope and others, to providing native-app-like features such as offline access and add-to-homescreen functionality, to you name it. Even Virtual and Augmented Reality is a thing in the web today.

One drawback of this richness in web capabilities is how complex it has become to  develop web experiences that truly take advantage of what the platform has to offer. As the capabilities and complexity of the web platform have increased, there has been as well an expansion in what we call the Capability/Usage Gap, or C/U Gap for short: the difference between what can be done in the web vs. what is actually done in the Web.

Screen Shot 2018-05-12 at 12.07.26 PM

In the early stages of the history of web development, say circa 1999, arguably this gap was not too big;  we basically needed to master HTML and CSS, we had to develop only for a single target  form factor (Desktop!), and the communication networks, albeit slow, were stable and more predictable. Now here we are in 2018: the web platform has evolved significantly; there has been an explosion on the number of development tools, frameworks, and other technologies; there seems to be a myriad of target device types to develop for; and the underlying networks servicing our web sites and apps are challenged and unpredictable. Sustainably mastering and leveraging this contemporary development landscape is really challenging. 

The Matthew Effect of Accumulated Advantage

The existence of a large C/U gap has serious collateral effects that hinder the mission and success of the web ecosystem. In essence, without exploiting the capabilities of the web platform it is very difficult to sustainably develop and maintain web sites and apps that fulfill the pillars of delightful user experiences.  We see quite a bit of evidence of this in the results of studies showing the prevalence of web experiences that exhibit poor load time performance (e.g. 19 seconds to load on 3G; 75% of mobile sites take 10+ seconds to load), and/or poor runtime performance (e.g. unresponsive pages, content shifting, and other misbehaviors).  And the bad news is that users don’t like these behaviors at all, as evidenced by the results of studies showing an impact on the revenue potential of sites (e.g. high abandonment rates for slow sites, and inverse correlations between conversions and load time, as well as between revenue and load time).

matthew-effect

But perhaps the worst collateral effect of a wide C/U gap is what is known in Sociology as the Matthew Effect of Accumulated Advantage, which in our context essentially means that given the complexity of the development landscape in the web ecosystem, it is implicitly entailed that developers that are super stars, or organizations that can afford large and high-quality engineering teams, are the only ones who can really take advantage of what the web platform has to offer; and that is not what the web is all about.

Closing the C/U Gap

The bottom line is that we must strive to close the C/U gap in every way we can to make our users happy, support developers’ ability to monetize their sites; and democratize the building of delightful user experiences on the web. There seem to be two main roads towards achieving that goal:  the all-developers-do-the-right-thing road; and (2) the progressive-web-ecosystem road.

In a way, the former is the road most traveled:  the web capabilities, tools and frameworks, and coding/performance best practices, are all there for us to take advantage of them, and there are lots of advice and guidance to do so; in principle we all could just “do the right thing”. However, that is way easier said than done. This road is loaded with tons of power and flexibility, but also with tons of complexity and cumbersomeness which are hard to master sustainably. Therefore there should be alternatives in the web ecosystem where doing the right thing and providing delightful experiences is easy.

Progressive Web Ecosystem

The second road to closing the C/U gap is enabled by what we call a progressive web ecosystem; that is, a web ecosystem encompassing the following aspects:

  • An evolving Web platform, where existing Web APIs are progressively improved, and new APIs are integrated into the platform over time
  • Effective tooling to help developers to automate complex tasks; tools like workbox which make it easy to leverage the power of service workers
  • Development frameworks which provide layers of abstraction to facilitate the development of complex experiences
  • A concerted effort for taking the learnings gained from developing web experiences on the current web, with the current tools and frameworks, and integrate them into the web platform itself

In a progressive web ecosystem, doing the right thing is easy because the underlying platform is advanced and ever evolving, there are tools and frameworks that pack a punch of capabilities with high usability and efficiency, and we, as a community, can do what humans do best, learn and improve our platform, tools, and frameworks. After that, repeat.

A Simple Model of the Web

The Web is a complex system composed of many interrelated and interacting parts. To reason about where to focus our efforts to close the C/U gap, it is useful to think about the web ecosystem in terms of a simple model consisting of three layers.

The top layer is composed of a federation of content ecosystems, all interrelated and interacting,  giving us the perception of “the whole world wide web” as a thing. The middle layer is composed by Content Management Systems, or CMSes, which are software platforms designed to simplify the creation and management of sites and their content. Notice that the model’s top layer is divided into two equally sized sets; this is  because, nowadays, about 50% of sites are powered by some sort of CMS platform, and the other 50% is powered by a variety of configurations of tools and frameworks. And the bottom layer, usually referred to as the web platform, encompasses all the capabilities of the web exposed via browser implementations of Web APIs.

Efforts to close the C/U gap must be pursued across all these layers and at Google we are indeed pursuing significant efforts along all dimensions of the web ecosystem. One of the specific efforts we are pursuing focuses on closing the C/U gap on the CMS part of this model.  All CMSes are important and we want all of them to become progressive platforms. In this post we focus specifically on WordPress.

The Progressive WordPress Road

A progressive web ecosystem encompasses the notion of Progressive Web Development: the development of user-first web experiences using modern web capabilities, and coding and performance best practices. The key aspect of this definition is the “user first” part;  the core of progressive web development is all about User eXperience; it is all about how our users feel when navigating and engaging with the content and web experiences we create.

When we talk about progressive WordPress we are referring to a WordPress platform where: modern development workflows, coding and performance best practices, and the use of modern Web APIs are commonplace, standard things; that is, a WordPress platform where doing the right thing, and creating user-first experiences is easy.

Now, given the size and scope of the WordPress ecosystem, how can we advance quickly along the Progressive WordPress road? Here is where the Accelerated Mobile Pages project, or AMP for short, enters the game. AMP is an open source library which makes it easy to build web pages and full sites that offer compelling experiences and load near instantaneously. In a nutshell, what AMP does is to provide, out of the box, a powerful set of capabilities and optimizations that address three essential aspects in the development of delightful experiences: load time performance (i.e. near-instant first visit load times); runtime performance (i.e. no un-responsive content); and usability (i.e. no content shifting). Check the documentation to learn more about the AMP project.

If we could integrate AMP into the WordPress platform, we would put at the fingertips of all WordPress developers the ability to take advantage of the efficiency and speed of AMP with minimal effort. That is, AMP brings home the coding and performance best practices part of the progressive WordPress equation. And this is exactly what we have been working on.

AMP in WordPress

In WordPress everything happens in one of three places: in the WordPress core platform; in Themes, which have to do with the look-and-feel of sites; and in Plugins, which have to do with extending the functionality of the WordPress platform. When we talk about AMP in WordPress, we are referring to the capabilities of an AMP Plugin, and the interaction of that plugin with the other components composing a site, such as the underlying theme and the other plugins installed.

The official WordPress plugin enabling AMP in WordPress was pioneered by an open source AMP Plugin initially developed by Automattic. This plugin was created at a very early stage of AMP, as WordPress was one of the earliest adopters of the project. It provided what we call a paired-mode, where for a given content post there was also an AMP version of it based on simple template parts. This mode did a great job enabling AMP on a large number of sites, but it had the main limitation that without significant engineering efforts, the resulting experience suffered from significant visual gaps between the AMP and non-AMP versions of the content, and significant functional gaps between the two versions (e.g. no hamburger menu, no ‘Load updates” button, etc.).

As the result of a joint effort between Automattic, XWP, and Google, the plugin has evolved significantly and steadily and now there is no need for a paired mode because the plugin generates experiences without neither functional nor visual gaps between the AMP and non-AMP versions. The latest version of the plugin, 1.0, effectively brings the power and efficiency of AMP to WordPress via a seamless integration with the standard WordPress content creation flow.

The AMP plugin will reach version 1.0 with an ETA of December 2018, and this version includes:

  • Automatic conversion of WordPress built-in widgets, embeds, short-codes, etc.
  • Tools to give users the power to control the actions of the plugin during the content creation workflow if needed
  • Integration with Gutenberg, the new and revolutionary content creation workflow in WordPress
  • Automatic optimization of CSS usage via CSS tree-shaking

With v1.0, WordPress developers get significant support making it easier to create native AMP experiences without sacrificing neither the fidelity and functionality of their content, nor the flexibility of the WordPress platform.

Native AMP Themes

The first step towards native AMP WordPress sites is a fully AMP compatible WordPress theme. In our AMP Conference talk, we showed such a theme for a native AMP News site:

Also, in our Google I/O talk you can get a glimpse of an all-AMP WordPress theme, built leveraging the capabilities of the new Gutenberg editing workflow:

And also, we described a summary of the process required to convert the 2015 WordPress theme to become all AMP. In this case, the plugin’s capabilities did most of the work, the whole process entailed only three steps:

  1. Use the add_theme_support( ‘amp’ ) function to register theme support for AMP, effectively telling the plugin to apply its magic and attempt to convert the theme to native AMP
  2. Use the capabilities of AMP bind to fix the menu toggle, which uses custom JS and can’t be converted automatically by the plugin
  3. Use the validation tool in the plugin to ensure the safety of the conversions made automatically by the plugin. This process gives control to the user on the behavior of the plugin, and clearly points developers to the parts of their sites that may need their attention to make them fully AMP compatible.

Without trying hard to tune the performance of this site, we observe good Lighthouse scores for the native AMP News WordPress site, influenced by the coding and performance best practices enabled by AMP.

And here you can see the demonstration of the Adventures Travel site, showing a native AMP theme fully developed using Gutenberg.

Further Along the Progressive WordPress Road

We have made significant progress along the progressive WordPress road with the advances in the plugin and its integration with the platform. Still, there is quite a bit of work ahead of us to further advance along the progressive WordPress road. The main areas we are focusing our efforts are: enabling the AMP Shell Architecture in WordPress to enhance the user experience beyond first visits, on the user’s onwards journey; providing a Progressive Theme Boilerplate/starter kit, to facilitate building progressive themes from scratch; and adding quality incentives to the ecosystem, to empower developers and site owners to develop and choose themes and plugins based on user-centric metrics, rather that just public’s opinion. A little more specifically:

App Shell Architecture in WordPress

The App Shell navigation model is a pattern for architecting sites in a way where the static elements in a page can be cached easily, and therefore loaded reliably and instantly on navigations; and the non-static components of the page can be loaded dynamically.

Enabling the App Shell Architecture in WordPress would enable several benefits and capabilities to sites, such as: reliable performance that is consistently fast, native-like interactions, and a reduction in data consumptionAnd this work involves the integration of the Service Worker API into WordPress core. Very excited about this effort!

Building Progressive Themes from Scratch

One of the key elements in a progressive web is tooling to help developers do the right thing in an easier and faster way. An important tool to have in the toolbox of WordPress developers is one that facilitates the development of Progressive WordPress themes.  Our colleagues at LinkedIn Learning have been busy developing a Progressive WordPress Theme Boilerplate, which is a theme development starter kit that makes it easier for WordPress theme developers to get from 0 to a fully progressive theme. The boilerplate comes out of the box with:

  • A modern development workflow, which is highly-configurable, and uses tools such as Composer, Gulp, and browser sync
  • A range of coding and performance best practices baked in, such as asynchronous loading of resources, and lazy loading of images
  • And it has opt-in integration with the AMP plugin, so that if the developer so choses, a fully AMP compatible WordPress starter theme can be generated from the get go

The main developer of this project, Morten Rand-Hendricksen, who is also a Senior Staff Instructor at LinkedIn learning, will be publishing soon a course on the LinkedIn Learning platform which will teach all about this boilerplate; and that particular course is going to be free. Don’t miss it!

Adding Incentive Mechanisms to the Ecosystem

One of WordPress major virtues is the flexibility of the platform. As a developer you can accomplish basically anything you can imagine; this is demonstrated by the large and ever expanding ecosystem of plugins available to add functionality and customize your website in a myriad of ways. However, this same virtue can also turn into one of the major challenges of the platform. Even taking into account plugins that are very good (i.e. follow coding/performance best practices and coexist nicely with other site components), the likelihood of having plugins colliding in their interactions (or impacting performance) when jointly activated in a site, is definitely not negligible.

We can deal with this challenge by introducing to the WordPress ecosystem tools that facilitate the verification of well-defined quality standards to allow developers to measure how good their products are in terms of UX-related metrics, and empower WordPress site owners to make choices about which plugins and themes to use, based on actual quality as opposed to only on popularity. We are pursuing efforts in this area, for example in the context of projects such as XWP’s Tide and Lighthouse.

Summary

Success on the web is all about building user-first experiences: it is all about UX. WordPress is an amazing platform and we are striving to close the Capabilities/Usage gap and enable content creation and consumption flows satisfying the pillars of delightful experiences.  v1.0, the latest version of the official AMP plugin (currently in alpha), brings the power and efficiency of AMP to WordPress via a seamless integration with the standard WordPress content creation flow effectively enabling native AMP experiences in WordPress without compromising content fidelity or surrendering the flexibility of the WordPress platform.

Stay tuned for new developments in the areas we mention in this post. We will participating in WordCamp EU in Belgrade this year where we will talk and show the latest progress on our efforts. Hope to see you there!

Reasoning about Speed

rail

One challenging aspect of web performance is that it is often addressed once it has become a problem already; this can be referred to as a “reactive approach to handling performance”. A reactive approach is challenging because the symptoms of the problem emerge way after they were introduced early on during the initial design/implementation phases of the a project; and it is often very difficult, if not impossible, to change design/implementation decisions that have pervasively percolated through the whole system.

Another challenging aspect is that we are trying to achieve globally-optimal results (i.e. good overall performance), from a collection of local optimization decisions (e.g. defer the evaluation of a JS file, etc).

Performance models are one approach to mitigate these challenges and enable us to reason about performance. The use of performance models during the process of designing and implementing applications provides guidance to our decision making process; they provide a framework for reasoning about how design/implementation decisions affect performance and user experience and to strike the proper tradeoffs at every step along the way.

Another benefit of defining and using performance models is that they provide a Lingua Franca, a common language for teams to talk about performance and ensure that they are all looking at the reality of the app from the same perspective.

Two specific performance models/patterns relevant for designing/implementing websites are: (1) the Response, Animation, Idle Load (RAIL) model; and (2) the Push, Render, Pre-cache, Lazy-load (PRPL) pattern.

The RAIL Performance Model

The RAIL model seeks to capture the four major areas of a web app life cycle; it stands for: Response, Animation, Idle, Load. If we look at them in chronological order, we would read the acronym backwards:

  • Load. We want our initial load and rendering to happen in about one second.
  • Idle. An app is usually idle while waiting for the user to interact; idle time is divided into chunks of fifty ms each, and they offer a great opportunity to squeeze work that the app needs to do.
  • Animation. This part of the model relates to one of the most challenging aspects of the performance of web apps: the need to achieve animations that render at 60fps (frames per second). This means that we have about 16 ms to render one frame; or even less since in reality, browsers have to squeeze other tasks in that time budget, leaving us with an actual budget of  about 10 ms. Check this reference for more detailed information.
  • Response. This responsiveness corresponds to how quickly the app reacts to user interactions; that is, how quickly the app becomes interactive. Research has shown that there is a limit of about 100 ms from when a user interacts with the application (e.g. tap a button, scroll) and when she gets the desired response.

In practical terms, the RAIL model translates into this :

  • Focus on the user; the ultimate goal is to make users happy
  • Respond to users immediately; acknowledge user input in under 100ms.
  • Animate smoothly. When animating or scrolling, produce a frame in under 10ms.
  • Be efficient. Maximize main thread idle time.
  • Load fast (keep users engaged). Deliver interactive content in under 1000ms.

The PRPL Pattern

The acronym PRLP refers to a pattern for structuring and serving Progressive Web Apps (PWAs), with an emphasis on the performance of app delivery and launching. The pattern comprises the following elements:

  • Push critical resources for the initial route.
  • Render initial route.
  • Pre-cache remaining routes.
  • Lazy-load and create remaining routes on demand.

In a nutshell, PRPL seeks to optimize three aspects of the development, launching, and performance of a web app:

  • Minimize the Time-to-Interactive (TTI) metric: especially on first use (regardless of entry point), and on real-world mobile devices
  • Maximize caching efficiency: Simplify app development and deployment

The PRPL pattern is less about specific technologies or techniques that can be used, and more about a mindset and a long-term vision for improving the performance of the mobile web applications. The goal is to keep the PRPL pattern in mind when planning for performance.

So…

If there is one takeaway from this post, is this: web performance is more than technique; it is a mindset. A mindset that should encompass every aspect involved in the development of our products: designers, developers, decision makers. And for that mindset to be effective, we need to be able to reason about performance. Performance models are your tools to do that and harness control of the performance of your sites and apps.

The Cost of Bad Performance

As we discussed before, users have high expectations with respect to the performance of websites; and when those expectations are not met users react in ways that clearly impact the outcome of key business metrics. There have been a variety of studies aimed and showing this reality. For example, a study made by Double Click and published in September, 2016, results like the following were reported.

User expectations

  • 46% of consumers say that waiting for pages to load is what they dislike the most when browsing the mobile web
  • 50% of people expects a page to load in less than 2 seconds
  • About 53% of all mobile visits are abandoned if the page loads in more than 3 seconds

Load times

  • 77% of mobile sites take longer than 10 seconds to load
  • On 3G networks, the average time to load is 19 seconds
  • On 4G networks, the average time to load is 10 seconds

Site structure

  • The average mobile web page is 2.5MB in size!. This implies that the data alone takes 13 seconds to load on a 3G network (first view).
  • The average mobile page makes 214 server requests; some of those take place simultaneously, while others are serialized.
  • About half of all server requests come from ad-related calls

The same study reports results indicating the positive effects of better performance. Specifically, the study found that sites that load in 5 seconds vs. those loading in 19 seconds exhibit:

  • 25% higher ad viewability
  • 70% longer average sessions
  • 35% lower bounce rates
  • 2X greater mobile ad revenue

So…

These results paint a clear picture of the impact of bad performance on our sites: a dramatic loss of users, inability of delivering our content, and serious monetization challenges. I agree, we can’t afford bad performance.

Why Is Attaining Good Performance Hard?

performance-dog

TL;DR attaining good performance is hard because there has been a drastic change on the dominant platform on which users access the web: the Mobile Web, and that change has been out of sync with the techniques and approaches we use to develop applications that run on that platform.

In the previous post, we discussed what is encompassed when we talk about mobile performance. In this post we discuss why is achieving good performance hard?

In a sense, the Mobile Era, took the underlying computing platform of the web “back in time” (sort of). Mobile devices are significantly more limited than their fellow desktops; or we can argue that today’s mobile devices are as powerful as desktops were several years ago. In a sense, we are trying to squeeze today’s software on yesterday’s computers.

The performance and functionality provided by mobile devices are not in sync with the need for resources of the applications that run on them; the access platform became more limited, but the approach used to develop applications to be run on such limited environments has not changed. One of the issues seems to be how to balance Developer eXperience (DX) and User eXperience (UX). Due to the limitations of mobile devices and the unrelenting demand of speed by users, the balance between these objectives should be drawn closer to the UX side of the picture, which means that websites and mobile web applications must be fast.

The challenges that make high performance hard to achieve on mobile devices and networks are two-fold: (1) Mobile-platform Specific; and (2) Website specific.

Mobile-platform Specific

This Section captures the message in this Chrome Dev Summit talk in a nutshell; in that talk Alex Russell discusses the underlying reasons why mobile devices are challenging from three angles influencing performance: mobile-device CPU, memory and storage, and mobile networks. This article, referenced in the talk, is a useful read to gain more insights in this context.

Mobile device CPUs

CPUs on mobile devices are less powerful than desktop CPU; one of the main reasons for this difference is the constraints imposed by heat dissipation, which is an important factor playing a role in the way mobile devices are built. For example, mobile devices do have multiple cores, but the are not used as Symmetric Multiprocessing (SMP) machines; that is, they asymmetric with respect to their capabilities and use frequency: there are some infrequently used, “big” (high-power) cores, and some aggressively used,  “little” (low power), cores. This construction in turn influences the way the software that runs on them operates.

The OS in mobile devices has what is called a Global Task Scheduler, which uses heuristics to control the way in which the asymmetric cores are used:

  • Touch Boosting: the user touching of the screen signals the scheduler to power the big cores in anticipation of the starting an activity or some other kind of user-triggered task
  • Application launching: the launching of an app signals the scheduler to power the big cores, bring up the app, and then send them back to sleep.

Things get tricky because the way in which mobile devices are designed does not correspond to the way the web works. A good “heuristic” in the web scenarios looks more like Link Clicking: when the user clicks on a link, wait until the associated resources are loaded from the network, and then let the user interacts with it. However, in a mobile device, if the big cores are started when the link is touched, the awakening on the big cores will be wasted while waiting for the data to arrive from the network.

Memory and Storage

Memory pressure and the need for smaller memory footprints on mobile devices do not allow for the trading of space for speed in the same way that can be done on desktops.

The physical limitations of mobile devices also influence the way Storage systems are designed and built. For example, due to space constraints, mobile flash storage is constructed using Multi-level Memory Cells (MLC) as opposed to Single-level Cells (SLC, which are faster, but also require more space and are more expensive.

Mobile Networks

There is a dichotomy between the web’s network infrastructure and the mobile infrastructure, which imposes a variety of challenges to mobile web app performance. A study by Radio Up reports:  “[in mobile networks], last-mile delays add 400% to the total of Internet lag time. In other words, the mobile last mile is the major drag on for the mobile web”.

The web runs on top of TCP/IP protocol suite, which is essentially a control system with feedback loops optimized to operate under certain assumptions such as some level of stability on network conditions, and low error rates. Mobile (cellular) networks often violate such assumptions (i.e. high variability of RRR and error conditions). The result is often degraded performance, which implies that, as Ilya Grigorik points out, a 4G user is not actually a 4G user most of the time. The same type of network may mean different things for different user.

The bottom line is this: cellular networks impose a set of challenges on the performance of mobile web apps. This book is an excellent resource to gain more depth in the area of network (including mobile) performance

Website specific: Webpage bloating

The average size of content for a mobile website is 2.2MB; those who argue that there is a web obesity crisis have a point. This graph (from httparchive.org) shows the trend of the overall transfer size for the top sites in the web. And this graph following graph [httparchive.org] shows the trend of JS requests and transfer sizes for the top 1000 websites since 2011 till now.  More details on the state of the art of page weight can be seen here. The reality is that page bloating contributes to loading delays with the consequent degradation of  UX quality. 

A variety of  experimental evidence exists exposing the problem; for example, here are some stats from a Double Click study:

  • The average mobile page makes 214 requests to servers
  • Half of all server requests come from ad-related calls
  • Average mobile page loading time is 19 seconds
  • 77% of mobile sites take 10+ seconds to load

Overall Performance

The takeaway from this post is this: the constraints of the mobile platform, and the prevalence of certain issues in the development of modern websites, are at the root of most of the performance problems being perceived by users. And the call for action is this: in order to make good, high-performant apps in today’s environment, a change is needed in our outlook, our tools, and our priorities. We (developers, platform, frameworks) must strive to develop lean websites.

What is Mobile Performance?

In the previous post we started a thread aimed at advancing our understanding of the fundamentals and challenges of mobile performance. In this post we continue the discussion, asking: what is performance?

Users have high expectations and demands regarding their website experience; we want simplicity and ease of use (don’t make me think), we want safety and security (don’t betray me), and, most importantly, we want speed (don’t make me wait). When websites are not fast, we get grumpy; every user hates slowness. It is true that performance is only one component of the overall user experience, together with appearance, accessibility, content strategy, usability; but it is a very important one. Some even argue that web performance IS user experience; and in some sense it is, because the lack of good performance often cancels out the role of the other UX components.

Since performance is so closely related to user experience, we can’t really think about one without thinking about the other; thinking about them congruently leads to design and implementation decisions at all levels that strike the best balance between User eXperience (UX), Developer eXperience (DX), and Monetization eXperience (MX). Therefore, it is important to: (1) focus on the right performance metrics; those strongly correlated with an improved UX; and (2) design and implement for performance by making deliberate decisions at every step along the way trying to strike balanced tradeoffs between seemingly conflicting goals.

The performance of a web app, as perceived by users, is influenced by both server-side and client-side components. Steve Sounders’ performance golden rule states that 80% of the end-user response time is spent on the frontend; therefore, our focus is mostly on the frontend side of the equation.  In a nutshell, from receiving the HTML code and the associated resources to rendering the corresponding pixels on the screen, the browser goes through the following steps in order

  1. Parse HTML into DOM
  2. Download resources linked from the HTML
  3. Parse CSS into CSSOM
  4. Parse JavaScript
  5. Construct the render tree by combining the DOM with the CSSOM
  6. Layout (calculate the positions and other attributes of all visible elements)
  7. Paint

These steps are executed for every page rendered by the browser, and there are a number of things that can be optimized in order to enhance the performance as perceived by the users:

  • Requests: Optimize the number and size of requests made by the application
  • Connections: Reduce the number of connections needed to load the app; andKeep in mind that number of requests is not the same as number of connections
  • Page weight: Optimize the size of the resources required by our page: HTML, CSS, JS, Images
  • Critical Rendering Path: Prioritize the display of content that relates to the current user action
  • Geography: Reduce the distance between your users and your content by using some kind of Content Delivery Network

In a following post, we will discuss why is so hard to achieve good performance on mobile devices.

The Perils of Mobile Web Performance

Niklaus Wirth (remember Pascal and Modula?) wrote a paper back in 1995 issuing a Plea for Lean Software. He starts by stating a fact that has remained true in the desktop world since then:  “Memory requirements of today’s workstations typically jump substantially whenever there is a new software release. When demand surpasses capacity, it is time to buy add-on memory; when the system has no more extensibility it is time to buy a new, more powerful workstation”.  And then he poses the question that is discussed in the paper: “Do increased performance and functionality keep pace with the increased demand for resources? Mostly the answer is no”. Today, 22 years later, we have a seemingly completely different reality, but we still can observe the consequences of Wirth’s statement in the harsh challenges involved in attaining highly performant mobile websites; there is a performance problem in the mobile web that needs to be solved.

In essence, we need to solve an optimization problem with three main dimensions:

  1. User Experience (UX): Engagement and delight using apps
  2. Developer Experience (DX): Ability/ease to develop apps
  3. Monetization Experience (MX): Monetization goals.

If we care only about UX, then developers may have to work at the lowest levels of abstractions to achieve the highest speeds, and publishers will make less money by interrupting the user the least with ads and monetization strategies. If we care only about DX, then users will likely  experience slowness from bloated/inefficient websites, and publishers will make less money because a decreased level of user engagement. If we care only about MX, then users will get annoyed with intrusive ads, and developers will be forced to implement aggressive monetization strategies. What we need is, at least, an approximation to an optimal solution to a 3D optimization problem that strikes the proper balance between UX, DX, and MX.

The common factor across these three dimensions is performance. Users want fast websites (UX), which leads to higher engagement and higher ROIs (MX), and developers of apps and frameworks want to design and develop fast products effectively (DX).

This essay argues about the need and approach to building for performance that drives the developing and evolution of mobile websites: the importance of performance in UX and MX, the role DX plays in the overall equation, and a suggested approach for thinking about and analyzing the performance of mobile sites.  The target audience is two-fold: (1) Web Developers; the community who makes design and implementation decisions; and (2) Decision makers; the community who decides what are the most important features and constraints. The goal is to define common grounds to think about performance so that we can reach the best solution to our three-dimensional optimization problem.

Specifically, we want to discuss:

  1. What is performance?
  2. What are the elements of Performance?
  3. Why is attaining high performance hard?
  4. What are the right performance metrics and budgets?
  5. What tools can we use to measure performance?
  6. What is the best performance testing environment?
  7. How do good performance strategies look like? [patterns]
  8. How do I go about analyzing the performance of sites? [methodology]
  9. How can I learn more about this?

Much of the content mentioned, described, and analyzed in elaborating our perspective in this series of articles can be (and has been) found scattered across many resources such as books, online videos, external online courses, internal documents, product documentations, internal performance trainings and presentations, etc. As Wilson Mizner said: When you steal from one author, it is plagiarism; when you steal from many, it is called research. Well, consider this guide the outcome of our research on the performance of mobile websites.

In the next post we ask the question: what is performance?