The Firefox UI is now built with Web Components

A couple of weeks ago, we landed a commit that took years of effort at Mozilla. It removed “XBL”, which means we’ve completed the process of migrating the Firefox UI to Web Components. It wasn’t easy - but I’ll get to that later.

The Firefox UI can be thought of as a very large single page app. It was built with DOM and JS from the start, which was a bold technology choice for a native application 20+ years ago. Because of this, Mozilla implemented some features necessary to build complex web applications way before the Web platform supported them.

Over time, these features have evolved into specifications like CSS flexbox and Web Components. When this happens, it’s easy to allow the existing codebase to continue to use the original version, and require the platform supports both features. This makes sense - rewriting legacy code that already works is difficult and not always cost effective.

But we made a decision while Web Components were being implemented in Firefox to start a parallel project in which we would migrate our existing UI components to use them. We decided to do this in an incremental way, so that each individual change could land while keeping Firefox in a functional state, as opposed to branching and rewriting the UI from scratch.

It’s taken a couple years of work of remarkably steady progress by a small team of engineers along with the support of the rest of the organization, and I’m happy to report that we’ve now finished. This is a big accomplishment on its own, and also a foundational improvement for Firefox. It allows teams to focus efforts on modern web standards, and means we can remove a whole lot of duplicated and complicated functionality that wasn’t exposed to websites.

What was XBL?

XBL was an XML-based language where you implement “bindings” that get attached to DOM elements. You could then add custom JS properties and anonymous content to a regular element. It was designed and built at Netscape in the late 90s, along with a number of other “XUL” features that allowed you to build desktop web applications at a time long before the Web platform added similar capabilities.

We had around 300 XBL bindings and 50,000 lines of code, which were used for very small widgets (like <toolbarbutton label="Reload" />) and for managing the application (like <tabbrowser />, which controlled much of the state within a browser window by managing tabs, receiving messages from content pages, etc).

XBL has served the project well and has made it possible to build Firefox for nearly two decades. The implementation has needed remarkably few changes over time. But it’s known to cause a lot of problems for us. Specifically:

  • There are hard to debug complications with binding lifecycles in our UI, and very few people know how it works.
  • It adds enormous complexity to our platform in the frame constructor, style system, and the DOM implementation.

Why Web Components?

Because of the problems above, we’ve talked about removing XBL over the years. But it hadn’t gotten much traction - the project just seemed too large and kept looking like it would require rewriting the Firefox UI from scratch.

However, the description of XBL above might sound familiar to you if you’ve worked with Web Components. Instead of “XBL bindings” there are “Custom Elements”, and instead of “anonymous content”, there is “Shadow DOM”. We did a more thorough analysis of the differences between XBL and Web Components if you’re interested in the details, but the gist was that we thought we would be able to migrate our existing components without requiring a total rewrite.

Next we did a design review in which we proposed starting a project to migrate XBL to Web Components. As I mentioned above, we knew this would be a long commitment but we thought the benefits justified the cost. Management agreed, and we kicked off the project.

Because the models are quite similar, we made a deliberate choice to keep API compatibility as much as possible when migrating elements. So we consistently pushed back against requests to refactor individual components as we touched them, unless if doing so made sense for the project. If we included that work the project never would have ended. Although we did find that once elements were implemented in standard JS it became easier to clean up and modularize, so people would start to do it on their own as they worked with them.

“De-XBL”

There are a lot of details I’m going to gloss over here - it’s taken 19 newsletter posts so far to share the work, and has been tracked in a giant metabug that started near the end of 2017.

The easiest way to explain the work is to split it up into phases. There wasn’t a crisp start and end to each of these, but the focus of the team tended to shift over time from one to the other:

  • Planning and tooling. In order to be confident this project would work, I wrote some scripts that could show us which bindings were ready to work on, and help to convert XBL bindings into JS classes. I also put up a website to help show our progress at arewexblstill.com. Seeing continual progress on the site helped to make the project feel achievable for the team working on it, and surfaced our work to people that weren’t involved with it day-to-day.
  • Auditing and removing unused or unnecessary code. Out of the 300 bindings, 77 of them were considered ‘unused’. Some of them were truly unused from the start, but others became unused as other teams shipped things like QuantumBar and the about:addons rewrite. 58 more didn’t need to be applied to DOM elements and were converted to JS modules.
  • Custom Elements. In the meantime, the DOM team was hard at work on shipping Custom Elements. Once this landed we converted our first binding to a Custom Element in May 2018. We started with elements that could be migrated to non-anonymous DOM, since Shadow DOM hadn’t shipped yet. In all, 78 bindings were converted to Custom Elements (though there are many more throughout the codebase today).
  • In-content XBL and UA Widget Shadow DOM. In-content bindings were pieces of UI that ran inside of web pages but aren’t visible to the page. These were extra complicated, and so we couldn’t just migrate them to Custom Elements. We took the chance to rethink how this is done and came up with a simpler design that relies on Shadow DOM called “UA Widgets”. This also took us to some strange places like shipping HTMLMarqueeElement in the year 2018. In total there were 39 in-content bindings.
  • Shadow DOM and Shadow parts. There were many elements that used anonymous content and would have been difficult to implement without it. They would slot content from the page into specific places in their content or style the anonymous content separately from the page. Shadow DOM was a solution for the former and Shadow Parts for the latter. As this feature shipped in the platform we started using them and providing feedback to the engineers working on the implementation.
  • Remove the XBL implementation from the platform. This sounds easy, but it takes some care. We’re currently in the process of doing this. So far we’ve removed around 23,000 LOC and a bunch of complexity in important parts of the engine.
  • Continue on with the rest of the broader XUL replacement program. This project is just part of that work. Thankfully, the biggest part.

It couldn’t have been done without a concentrated effort from the people removing bindings and help from teams across Firefox when we needed questions answered, patches reviewed, and new platform features prioritized. This type of work is often behind the scenes, and I’m grateful to everyone who has helped push this forward over the last two years.

Web Components changes we made in Firefox as a result of using them in the UI

We hoped that because of dogfooding Web Components in our own UI we would be able to fix bugs, improve performance, and request features in Web Components that we wouldn’t have otherwise been able to. Here are some of the things that came of that:

  • New feature: CSS Shadow Parts. This would have been implemented eventually, but we really needed the ::part selector in order to port some of our more complex XBL bindings to Custom Elements without reworking them entirely, so the priority was increased to help.
  • DevTools: Web Components support. The DevTools and DOM teams worked together to build tools to inspect Shadow DOM and to jump directly from a node in the Inspector to the Custom Element class definition in the Debugger. There are more details in the 7th newsletter.
  • Performance improvements. The Firefox UI is heavily performance tested, and as we started using Web Components we noticed some issues with the implementation. The DOM team was always very responsive with getting these fixed. For example, we identified some regressions on the Dromaeo performance tests that were fixed. There was also a case where a user with over 1500 tabs open (scientifically considered a “tab hoarder”) noticed extreme slowness when opening the “all tabs” dropdown. It turned out that he had tripped on an O(N²) edge case, and the issue was fixed.
  • New Firefox UI only features:
    • Lazy defining custom elements. The browser startup time is very closely monitored and in general it’s not acceptable to land things that slow it down. Some elements like <findbar> are actually lazily injected into the DOM and aren’t visible at startup. With XBL it wouldn’t attach until the element became visible, but with Custom Elements we were loading the class definition up front. We saw some performance issues with just loading and parsing the JS files, so we added a mechanism to fire a callback when an element first appears in the DOM to give us a chance to synchronously load the class and register the Custom Element. I don’t think this is suitable for exposing the Web, since there isn’t a way to synchronously load a script in the same manner.
    • Extending customized autonomous elements. We had discussion thread with more details. I did file an issue on the standard to see if there was interest in exposing this behavior, but there wasn’t. Our case was a bit more complex since we had existing JS/C++, along with a bunch of CSS, referencing the overloaded tag names.
    • Allowing non-hyphenated tag names. The standard defines a valid Custom Element name as including a hyphen. After a discussion thread we decided to loosen that restriction in the interest of incrementalism, so that we didn’t need to do the actual conversion and rewrite existing markup at the same time. We’ll probably move to dashed names now that we’ve completed the migration.

XUL/XBL Replacement Newsletter 19

This is the nineteenth edition of the XUL/XBL Replacement Newsletter. Lots to share this time - we’re making progress on the remaining XUL replacement work, while also clearing out the XBL implementation from mozilla-central.

<html> root in browser.xhtml

We had already migrated the main browser window from a XULDocument to HTMLDocument and renamed it from browser.xul to browser.xhtml. But there are still a lot of XUL elements inside of it. Most notably, the documentElement was still a <xul:window>, which meant that we would use a different frame structure than normal webpages.

Brendan Dahl recently landed a change to fix this. Now instead of:

<window title="Firefox">
  <linkset>
     <link rel="localization" />....
  </linkset>
  <script />...
  <the-rest-of-the-content />
</window>

The document looks like:

<html>
  <head>
    <title>Firefox</title>
    <link rel="localization" />....
    <script />...
  </head>
  <body>
    <the-rest-of-the-content />
  </body>
</html>

Ideally, I’d like to do the same thing but for all documents in the tree and then remove nsDocElementBoxFrame (the layout frame used for XUL root elements). This would be a big job, since there are 1200+ <window> tags in tree and the migration likely requires some automatic parsing and rewriting XML. But the browser document is an important first step and gives us confidence that it’ll work with other documents. Also, Kirk Steuber has been taking steps to make the project more manageable by migrating away from the non-“window” XUL root elements <wizard> and <dialog>. I’m also hoping we can use some of the learnings and process from Emma Malysz’ ongoing work to rename all .xul files to .xhtml to help.

L20n

L20n is the code name for an effort that started three years ago to introduce Mozilla’s new localization system (called Project Fluent) into Firefox. It’s an important part of XUL replacement, but it also improves the quality and resilience of localization in Firefox as Zibi Braniecki outlines in a post summarizing the start of the project.

Since then, the work was planned, the core internationalization module in Gecko was refactored, and Fluent was landed and integrated into all major components of the Firefox UI. And recently the old DTD-based localization system for XUL/XHTML has officially been deprecated. So as of now, we are considering the L20n project complete.

The next project is to complete migrating all remaining strings in the Firefox UI to use Fluent. Today around 37% of Firefox l10n strings use Fluent, so there’s still a lot more to do. We are first focusing on removing DTD strings that can cause the “Yellow Screen Of Death” (a startup crash due to an XML parsing error), and on building tooling to help speed the remaining conversions. This work is being tracked at https://arewefluentyet.com/.

AppWindow is the new nsXULWindow/nsWebShellWindow

In the past, top level windows (e.g. the browser UI, places UI, etc) have had an nsXULWindow associated with them for controlling various things related to the OS window. However, as we migrate more of these windows to XHTML and HTML, the “nsXULWindow” name makes less sense. So after discussing with smaug, Brendan changed the name to AppWindow. As part of this cleanup we were also able to merge nsWebShellWindow into nsXULWindow, since it was the only class extending nsXULWindow. We hope the AppWindow name makes the purpose more clear.

Goodbye, XBL

Last newsletter, we were building GeckoView without MOZ_XBL. Since then, Brendan has landed a change to also build desktop without MOZ_XBL and Boris Zbarsky has posted to mozilla.governance about removing the XBL module. We’ve also closed all unnecessary bugs in Core::XBL on Bugzilla, and moved any still relevant bugs into appropriate components.

In the meantime, we’ve started to remove the implementation and clean up related code. Thanks to everyone who’s been pitching in here! Removing the implementation isn’t as easy as it may seem, since it’s wound through many different components and features. If you are aware of anything that can be removed or simplified now that XBL is gone, please file a bug blocking the remove-xbl-implementation metabug.

Here are some of the recent changes:

XUL/XBL Replacement Newsletter 18

This is the eighteenth edition of the XUL/XBL Replacement Newsletter. Even during the excitement of hitting 0 XBL bindings, there’s been a lot of progress and some updates to share.

GeckoView Is Built Without XBL

As we got close to 0 bindings, Brendan Dahl created a build option to disable XBL in Gecko. This wasn’t a simple task due to the amount of places throughout the codebase that reference or interact with XBL. Thanks to Brendan, and to Boris Zbarsky for reviewing this change.

One nice thing about introducing a build option is that we were able to turn off XBL entirely in GeckoView! So this is the first platform where we are shipping Gecko without XBL.

We also have a bug on file to disable XBL in desktop Firefox, which we plan to do after the upcoming merge to 72 on 2019-10-21.

Finally, we are tracking some of the tasks and improvements that can be made once XBL is turned off. This is done in the following metabug: remove XBL support from Gecko. If you know of changes that would be good to make post-XBL, please help us track the work by blocking that bug.

Renaming XUL Files

Emma Malysz has been making quick progress on scripting migrations of .xul files to .xhtml, one top-level directory at a time. As a reminder, we no longer load anything as a XULDocument so chrome xul files are effectively the same as xhtml files. Renaming the files is an important step in making that clear and finishing the “XUL Document Removal” project.

The majority of xul files in tree are tests. One tricky part with them is making sure that we don’t break intermittent test tracking. For instance, if there’s a bug titled:

Intermittent test_largemenu.xul | menu movement (6000, 100) x - got 2554, expected 1530

Then when a failure happens in treeherder with a test called test_largemenu.xul, it will automatically suggest the known intermittent bug based on the summary. However, renaming the file with the .xhtml extension breaks the tracking and the test appears as an unknown orange, which causes extra work for anyone viewing results from pushes to treeherder.

I had previously consulted with the sheriffs and migrated a single test file, confirming that by choosing a summary with both the old and new file name in it it will track intermittents on m-c (where it now has the new name), and beta/release (where it still has the old name). For example:

Intermittent test_largemenu.xhtml,test_largemenu.xul | menu movement (6000, 100) x - got 2554, expected 1530

There are around 100 intermittent bugs that reference xul files, so rather than manually making this change, Emma worked on scripts to:

  • Pull down the existing bugs and modify the summary string
  • Call the Bugzilla “Update Bug” API endpoint for each bug to update the title.

Thanks also to Emma Humphries for helping out with the Bugzilla integration for the above.

Improving the Keyboard Shortcut Handler Implementation

Keyboard shortcut handlers (anything declared with a <xul:key> element and various built in shortcuts like PgUp, PgDown and edit commands) were sharing the same event infrastructure as XBL <handler>. In order to make it possible to build with XBL disabled, Mossop removed the dependency on XBL. This is a followup to moving the definitions out of XBL, and throughout this work he’s been able to make improvements like adding missing test coverage for this core feature.

See the Keybindings in Firefox document for more details and a plan for how we can make it possible for JavaScript to register keyboard shortcuts at the system level so the events correctly wait for the content process to handle them as necessary.

Next up is replacing <xul:key> which has a number of longstanding issues including requiring an oncommand attribute in order to fire, which in turn stops us removing inline event handlers in browser.xhtml, which in turn blocks further improving CSP and eval() defense-in-depth security improvements. The idea will be to switch to JavaScript ways of registering handlers, either through a Custom Element or window initialization code.

XUL/XBL Replacement Newsletter 17

This is the special “0 bindings left” seventeenth edition of the XUL/XBL Replacement Newsletter. I’m proud to announce that as of today, we have no more XBL bindings left in Firefox!

There’s still a lot more to do on the larger XUL Replacement project, and we’re still in the process of removing XBL support in Gecko. I’ll send a follow up post outlining the next steps, but I wanted to take this opportunity to celebrate the accomplishment and acknowledge the efforts over the last two years of steady progress. There have been too many people to list who have been involved with writing and reviewing patches, answering questions, and breaking down bugs. And others who haven’t been directly working on this but have been affected by it being prioritized over other work.

Binding Removals

There are 0 bindings left, compared to 7 from the last update and 300 from the start of the project. Here’s a list of changes:

XUL/XBL Replacement Newsletter 16

This is the sixteenth edition of the XUL/XBL Replacement Newsletter. Since the last edition, we’ve removed the XULDocument implementation and have been helped with XBL removal by two big projects completed by other teams. We’re getting close to finishing the burndown of XBL bindings and are doing some prep work to allow building Firefox without the XBL implementation once that happens.

XULDocument is gone

Over the course of the last year we’ve been working through a plan to remove XUL documents from mozilla-central. We first reviewed all the functionality in XULDocument, then removed obsolete features and migrated ones we wanted to keep into the base Document or into standalone pieces. By the end of July, XULDocument was mostly a skeleton and Brendan Dahl landed a patch that caused all .xul files to be loaded exactly as if they were .xhtml files. After a few weeks with no issues, he removed the XULDocument implementation entirely.

While XULDocument is now gone, there’s still plenty of work to be done to old XUL files:

  • The file extension is still .xul so those files will need to be converted to .xhtml. Since they are already loaded as such, this should involve only renaming files.
  • Most still contain a “XUL like” DOM structure (<window>), whereas we’d rather them be more like regular HTML documents (<html><head><body>). By making our consumers more standardized with the web, they’ll be easier to understand for the uninitiated, allow tooling to work more easily, and let us remove more special case code we have for XUL document elements.

Old about:addons removal

The old about:addons page had the largest block of remaining XBL with 10+ unique bindings. The addons team also wanted to add some new features to about:addons, like abuse reporting and inline recommendations. We met last year to coordinate plans for the page and it became clear that the best way to accomplish both of these goals would be to rewrite the old UI with an updated HTML version.

We knew we’d like it to be done mid-year in order to line up with progress on XBL removal, and I’m happy to report that the addons team has now shipped the new version in Firefox 68, which allowed us to remove the bindings. Thanks to everyone who helped to get the new page shipped on time, especially Mark Striemer, Luca Greco, Rob Wu, and David Durst.

Old URL bar removal

Another project that allowed us to remove a number bindings landed in Firefox 68. QuantumBar is a rewrite of the AwesomeBar which makes experimentation and maintenance easier. It’s also built with HTML/JS instead of XUL/XBL, so after it shipped the team was able to remove quite a few bindings (including the most complex remaining binding: legacy-urlbar). Thanks to everyone who worked on this, especially Marco Bonardo, Dão Gottwald, Drew Willcoxon, Harry Twyford, and Mike de Boer.

Binding Removals

There are 7 bindings left, compared to 25 from the last update and 300 from the start of the project. Here’s a list of changes: