This post summarizes a recent talk Weston Ruter and myself gave at Chrome Dev Summit , 2018: Content Management Systems (CMS) are software platforms designed to simplify the creation and management of websites and their content. Nowadays, about 50% of websites are powered by some sort of CMS platform. Below, we discuss our experience moving a complex CMS (i.e. WordPress) forward along the Progressive Web Road by integrating modern capabilities (e.g. Service Worker API) and progressive technologies (e.g. AMP) into its platform and ecosystem, and how the lessons learned can be applied to other CMSes.
It is all about UX
As web developers our goal is to maximize the joy our users feel when engaging with the websites we build or the web content we create and publish on the web. And although User Experience is a complex matter, there are common elements in web experiences that users love. Specifically, users love…
- Sites that load quickly consistently and offer good runtime performance
- Sites where they feel safe sharing their data and doing transactions on them
- Sites that are accessible and allow them to take advantage of the full capabilities of their devices, and
- Sites that offer great content quality
The good news is that we can build experiences that fulfill these pillars if we take advantage of modern capabilities the web has to offer. Doing this is what we call Progressive Web Development: developing user-first experiences using modern development workflows and APIs, following coding and performance best practices, and taking advantage of incentive mechanisms (e.g. engagement increases) and validation structures (e.g. performance budgets and others) which are available to us.
The way we go about taking advantage of Progressive Web Development depends on how we go about our creative process in the web.
Web Development Process
The things we create on the web fall in a general sense into two areas. On one hand, we can build websites or web apps from scratch, in which case we have full control of the creative journey: from the build process, to the functionality, to the look-and-feel, of our website or web app. On the other hand, we can use some kind of Content Management System, or CMS for short, which are Software platforms providing layers of abstraction and capabilities to make easier to create, publish, and consume digital content on the web.
The first route gives us a lot of power and flexibility, but it is also it requires quite a bit of expertise and resources. The second route, allows us to take advantage of the functionality provided by a CMS, but the experience we provide to our users depends a great deal on the specifics of the CMS we choose.
Contemporary Content Creation and Publishing
Nowadays the majority of experiences in the web are created, published, and consumed via some kind of CMS: More than 50% of the web is powered by some form of CMS; and this year there was an 11% YoY growth! [bit.ly/w3tech-stats]. And there are many other platforms that do not consider themselves CMSes but also provide similar capabilities that allow users to create and publish content on the web easily. Therefore, this number is actually much higher in reality.
In practical terms, this means that the CMS space is highly populated and complex. There are many CMS options out there for users to choose from and they come in many shapes and forms, varying across dimensions such as: Static vs. Dynamic, Proprietary vs. Open Source, Special Purpose vs. General Purpose, and others.
But although there are differences between them, they also share lots of commonalities, and many of them face common challenges when it comes to integrating progressive web capabilities into their platforms. To reason about how to tackle such challenges across the CMS space, it is useful to abstract our thinking about these platforms in terms of the fundamental components that make them.
Each CMS has in one way or another a set of components that form a unique ecosystem. There are usually the following components:
A platform Core: which encompasses the main functionality provided by the platform; a mechanism for extending the core functionality of the platform. Such extensions are known as modules, plugins, and themes; a community of developers who are in charge of implementing and evolving both the core and the extensions for the platform; and a user base who either create content on the platform, or consume content from the platform, or both.
When we talk about progressive CMSes, we are referring to bringing the pillars of PWD to the components of a CMS Ecosystem; specifically, things such as:
- Bringing coding and performance best practices to the development of both the platform core and the extensions
- Introducing incentive mechanisms for extension developers to do the right thing
- Providing tooling to developers to make it easy for them to do the right thing
In order for a CMS to fulfill its potential, all these components need to walk together along the progressive road.
In this post we will describe specific results of efforts our team has been pursuing in the context of a specific CMS, and we will touch upon some of the great work being done by other players in the CMS space.
Our team has been pursuing efforts to bring progressive web technologies to the WordPress ecosystem. WordPress is certainly one of the most influential CMSes out there, as reflected by the fact that it powers about ? of the web today. Given the size and complexity of this ecosystem, addressing its challenges will not only have an impact on a very large swath of the web, but it will also help us to shed light on how to address similar challenges in other CMSes.
Our goals are two fold: to tackle the challenges for the WordPress ecosystem, and to distill and generalize the learnings and solutions we gather along the way and bring them over to other CMSes in the form of objective guidance, tutorials, and technology transfers. Our approach to achieve these goals consists of two parts:
- Leveraging the capabilities of the Accelerated Mobile Pages project
- Integrating modern Web APIs into the WordPress ecosystem, either via the core of the platform or via plugin-enabled capabilities.
AMP: A Well-lit Path to Modern Web Development
AMP is a web component library, together with a runtime system and a validation framework. AMP provides, out of the box, a set of capabilities and optimizations addressing essential aspects of high-quality web experiences. These include:
- Coding and performance best practices, which enable great runtime performance (i.e. no un-responsive content) and great usability (i.e. no content shifting)
- Performance incentives, such as the ability to achieve near-instant loading times via pre-rendering when AMP content is served from an AMP cache
- A validation framework that guides the development process towards highly performant sites, and makes it easy for developers to keep their sites in that state over time
Enabling AMP content creation in WordPress means that WordPress developers can take advantage of these capabilities without deviating from their standard content creation workflow in the platform. Such integration has been enabled via the official AMP plugin for WordPress, which is currently at version 1.0-RC2 (available on Github), and version 1.0-stable will be released at the end of this month (November, 2018).
The Official AMP Plugin
To enable WordPress developers to take advantage of AMP’s capabilities, AMP content creation must be seamlessly integrated into the standard content creation workflow in WordPress. We can envision a path where AMP-compatible markup is generated by WordPress core itself, as an option, similarly to what happened when HTML5 became a thing. Alternatively, such seamless integration can be achieved by a plugin.
The official AMP Plugin for WordPress is a product designed to fulfill two main roles. First, to enhance the development process of AMP experiences in WordPress, such as AMP-compatible plugins and themes. And second, to ensure that AMP compatibility is maintained as a site evolves, so that no invalid AMP is ever served.
The plugin is not a turn-key solution to the “how to AMPlify my site” problem; instead, the AMP plugin functionality substitutes AMP components for their corresponding HTML ones where required and possible (img => amp-img, video => amp-video, etc.), and it aids the development process by exposing sources of validation errors, providing tools to understand the context of such validation errors (e.g. preview integration), and ensuring that only AMP-valid content is ever served for a given WordPress site. These capabilities make it easier for developers to build high-quality AMP experiences (e.g. full sites) and AMP-related solutions (e.g. editors); and enable the ecosystem of developers making themes (e.g. Genesis) and plugins (e.g. Jetpack), to tackle AMP development in a WordPress way, that is, in a way that is integrated with the regular WordPress development workflow and theme/plugin ecosystem.
In its second role, the plugin functions as key enabling component ensuring a site states AMP compatible by keeping developers, implementers, and site owners, always aware of the compatibility status of their site.
The plugin provides all its functionality in an opt-in manner. And users that do opt-in have the option of enabling AMP only on specific sections of their site, and also disable AMP at a granular level (e.g. on a single post).
Integrating Modern Web APIs into the WordPress
The second part of our approach to bringing Progressive Web Development into WordPress consists of integrating modern web APIs and capabilities into the WordPress core platform. We want the WordPress platform to support natively things such as: Service Workers, Offline access, Streaming, background sync, etc. The goal of such integration is to provide a consistent and unified approach for WordPress developers to easily access these APIs and capabilities as they develop core functionality, as well as themes or plugins.
In addition to our work on AMP for WordPress, our team focused this year on devising and implementing the integration of some of these building blocks into the WordPress platform, and we did this with an emphasis on doing it in a way that is seamlessly integrated to how things work in the WordPress platform.
Service Workers in WordPress, the WordPress Way
The Service Worker API is at the center of many of the advanced capabilities we want to enable natively in WordPress. Nowadays, it is certainly possible to use Service Workers in WordPress, but doing so poses several fundamental challenges that need to be addressed.
First, there needs to be a unified API access for creating and manipulating service workers in WordPress. Without it, if multiple plugins and themes want to leverage the capabilities of this API, they would have to do all the work by themselves, not able to take advantage of any reusability, and running the risk of having collisions among multiple service workers installed on a site, which is not allowed by the API.
Second, providing access to the Service Worker API is just one part of the solution needed; programming the logic of Service Workers is by far not a trivial task. If we want all developers to take advantage of the power offered by Service Workers, an abstraction layer must be added in WordPress that makes doing so easier for developers.
Finally, one of the virtues of WordPress is the flexibility and versatility of its extension mechanism: plugins and themes. The challenge is that these extensions are not sandboxed and consequently, it is very difficult to implement advanced capabilities such as the App Shell model in WordPress because down the road additional plugins can be installed and they can alter/modify the content of a site in ways that breaks these capabilities.
To address these challenges, we have implemented abstractions in WordPress to integrate the Service Worker API into WordPress in a way that is applicable not only to WordPress but also to other CMSes. These abstractions cover three main aspects:
- A low-level API in WordPress that provides direct access to Service Workers, so that all themes and plugins can each include their own logic in a single installed service worker.
Progressive Capabilities in WordPress
Once service workers are accessible in a unified way, many advanced capabilities are enabled. Some of them are the App Shell model, and offline access capabilities.
The App Shell Model in WordPress
One of such capabilities is the app shell model. An app shell facilitates features such as seamless page transitions and offline interactions, similar to the experiences provided by native applications. But implementing an app shell is normally very challenging to do in WordPress due to the lack of sandboxing between plugins and themes.
This is where AMP plugin plays a key role because it enables the enforcing of constraints to ensure that the content on a site behaves in a way that doesn’t break the app shell model, while also ensuring the content loads in a performant way. Furthermore, AMP pages are also specifically designed to be embedded in this way via the Shadow DOM API.
In this demo site we have a WordPress site using the default theme and implementing the App Shell model. Pay attention to the fun spinning logo. Notice what happens when we click to navigate to the “About” page and other pages in the nav menu:
- The logo keeps spinning without resetting.
- And the nav menu remains expanded with the current page being highlighted.
- There is no flash of white in the header.
- Only the actual content of the page is changing, and it does so with fading animation.
To give you a glimpse of how this is accomplished on the user side, here are some snippets of code any WordPress developer would use in order to take advantage of the capabilities provided by the integration of Service Workers into WordPress core.
First of all, to address the problem of no sandboxing, we leveraged the AMP plugin to make all pages of a site available in AMP. This can be done with a standard add_theme_support() call.
Second, we introduced a way of specifying the dynamic part of the template which changes as you navigate the site. Specifically you can see this via these two function calls placed before and after the content.
And lastly, we extended the AMP plugin to add logic to the page as well as the service worker to implement the app shell. All a theme has to do is to opt-in with another add_theme_support() call like you see here.
The bottom line from these examples is that the complexities of dealing with Service Workers are abstracted away from the theme developer and absorbed directly by the platform.
Beyond the app shell model, the service worker is useful for many other advanced capabilities. As mentioned previously, the service worker also enables offline access.
WordPress themes include a 404 template which is served when the site can’t locate the requested content. We’ve extended the WordPress template system to introduce an offline template which is served when the user’s browser can’t access the site. Thus, in addition to a 404.php template, WordPress theme developers just have to add an offline.php like to their site in order to customize the user experience when facing the loss of connectivity.
The service worker is also useful to cache content the user has visited (or may visit) so that if connectivity is lost, they can continue accessing that part of the site’s content when they are offline. In the snippet above, you can see how the stale-while-revalidate serving strategy is added for navigation requests, so that anything previously visited will continue to be available.
When the user wants to provide a comment on an article but are offline when they try to submit it, the service worker can intercept the request and replay it later via background sync when the user is back online. The developer doesn’t have to do anything as all this happens by default.
Progressive CMSes: Beyond WordPress
The CMS space is moving forward steadily along the progressive web road. Google is working and supporting many of the platforms in this space to help them accelerate their pace, and we are seeing great advances by several of them. Including…
- Drupal is another popular open source CMS. Their developer community is driving the development of AMP and PWA modules for the Drupal platform. Both modules are available and used by Drupal sites today and they are actively working on enhancing them.
- Magento is one of the leading e-commerce platforms and they have been strong advocates for PWAs on their platform for a while now. They will be releasing Magento 2.3 in November and that release will include the launch of PWA Studio, a set of developer tools and guidelines, to make it easier to build great web experiences on their platform.
- Adobe Experience Manager is a popular enterprise CMS; they have made it possible to create and publish Single Page Applications directly in the Experience Manager Page Editor and they’re working to provide direct support for Service Workers in the near future.
- EC-CUBE is a leading eCommerce platform in Japan; they are working, together with a development agency called Sunday Systems, on an AMP plugin, which will enable the creation of AMP content on their platform.
It is exciting to see platforms like these becoming more progressive, and we are looking forward to continued progress by all of them.
The Road Ahead
We have made significant progress on our efforts in the WordPress ecosystem, but there is still some way to go to get to where we really want to be. We are continuing our efforts described in this post, including the following aspects.
The AMP plugin for WordPress is available on Github today. If you are WordPress developer, the plugin gives you the tools and support you need to make your themes and plugins fully AMP compatible. We are working on expanding the integration of AMP content creation in WordPress, especially in the context of Gutenberg, which is the new editing experience in WordPress.
The integration of the Service Worker API and other PWA capabilities into WordPress core is well underway. These capabilities are currently available as a Feature Plugin, which you can download and experiment with today. We are continuing this effort, and we are working with Automattic and others in the WordPress ecosystem to get the integration merged into core some time not too far in 2019
Finally, we are continuing working with the WordPress ecosystem to help them take advantage of progressive technologies at all levels of their ecosystem.