XUL/XBL Replacement Newsletter 20

This is the twentieth edition of the XUL/XBL Replacement Newsletter. As of last week, we have 0 xul files in mozilla-central. We’ve also been scoping out the work remaining in “XUL Replacement”, and have an update on post-XBL cleanups.

No more XUL files in mozilla-central

Back in May, I posted an outline of a plan to remove all XUL documents from mozilla-central to dev-platform. Based on feedback, we adjusted the plan to be smaller in scope and focused on (1) loading all XUL files as if they were HTML and (2) renaming .xul files to .xhtml. The first step was completed by Brendan Dahl in July (see newsletter #16), and Emma Malysz started on the second in October (see newsletter #18).

Emma did a great job breaking down and renaming the ~1500 files. It’s been split up primarily by directory using a script to handle the simpler test file renames, and manually handling the more complicated cases like files loaded over chrome://. This has led to a really nice burndown of xul files in mozilla-central, hitting 0 on Friday.

Another nice result of this project is that eslint operates on .xhtml files but not .xul, so we have increased lint coverage across the tree. Mark Banner also recently improved this support by handling self closing script tags in xhtml files.

XUL document burndown chart, starting with around 1400 documents on 2019-10-02 and ending with 0 documents on 2019-12-13

What’s left to do?

Over the last 2 years, we’ve accomplished a lot in this program. We’ve removed XUL Templates, XUL Overlays, XBL Stylesheets, XUL Documents, XBL, and converted all XUL files to XHTML - migrating to web technologies when possible. So what’s left to do? The three remaining projects we’re tracking are:

1) Complete the DTD conversion to Fluent.
2) Remove the XUL flex model and the other XUL layout frames.
3) Migrate XULElements that have a web equivalent to HTMLElement.

The work in (1) is well scoped and you can expect to see more updates about it early next year. For (2) we’ve been making progress on removing XUL layout frames like stack, grid, and groupbox but still have some performance issues with flexbox along with some functional issues that will be helped by doing (3). So I’d like to talk about more about (3) here.

First, I expect that we will still ship a number of XULElements even after we complete this program. I’ve talked about this before, but things like panel, browser, the menu elements (menu, menupopup, menucaption, menuitem, menulist) don’t have HTML equivalents and are important for our desktop browser experience. While we could invent a way to do this in chrome HTML, I don’t think the cost/benefit justifies doing that ahead of the rest of the things in our list.

So that leaves more “vanilla” XUL elements that are primarily implemented in JS. These cover the majority of our UI and are things like hbox, button, textbox, tab, richlistbox, etc. There are a couple of hurdles here:

a) XULElements act differently from HTMLElements in some ways. For example, getAttribute("not-existent") returns null in HTML and "" in XUL. XUL has some “magic attributes” that dictate layout and other behavior. And XUL supports features like [command] & [tooltip] which don’t have an equivalent in HTML.
b) There’s a long tail of UI migrations that need to happen. These include changing the base Custom Element classes and also updating markup / CSS to use the new element name / namespace.

We’re thinking it makes sense to work on (a) tree-wide first (metabug), and treat (b) as a separate project (metabug). The reason to treat these separately is so that we don’t need to repeat the subset of changes needed for (a) for every single element in (b). In other words, we’d like (b) to be able to be as automated as possible, and selectively rewriting how frontend JS interacts with elements would be a roadblock to doing so.

More XBL cleanup

There have been some really nice simplifications and code removals happening in the remove-xbl-implementation metabug. Lines of code isn’t the best measurement for this type of simplification, but for what it’s worth there’s been a net removal of around 24K LOC so far.

The Firefox UI is now built with Web Components

A couple of weeks ago, we landed a commit that took years of effort at Mozilla. It removed “XBL”, which means we’ve completed the process of migrating the Firefox UI to Web Components. It wasn’t easy - but I’ll get to that later.

The Firefox UI can be thought of as a very large single page app. It was built with DOM and JS from the start, which was a bold technology choice for a native application 20+ years ago. Because of this, Mozilla implemented some features necessary to build complex web applications way before the Web platform supported them.

Over time, these features have evolved into specifications like CSS flexbox and Web Components. When this happens, it’s easy to allow the existing codebase to continue to use the original version, and require the platform supports both features. This makes sense - rewriting legacy code that already works is difficult and not always cost effective.

But we made a decision while Web Components were being implemented in Firefox to start a parallel project in which we would migrate our existing UI components to use them. We decided to do this in an incremental way, so that each individual change could land while keeping Firefox in a functional state, as opposed to branching and rewriting the UI from scratch.

It’s taken a couple years of work of remarkably steady progress by a small team of engineers along with the support of the rest of the organization, and I’m happy to report that we’ve now finished. This is a big accomplishment on its own, and also a foundational improvement for Firefox. It allows teams to focus efforts on modern web standards, and means we can remove a whole lot of duplicated and complicated functionality that wasn’t exposed to websites.

What was XBL?

XBL was an XML-based language where you implement “bindings” that get attached to DOM elements. You could then add custom JS properties and anonymous content to a regular element. It was designed and built at Netscape in the late 90s, along with a number of other “XUL” features that allowed you to build desktop web applications at a time long before the Web platform added similar capabilities.

We had around 300 XBL bindings and 50,000 lines of code, which were used for very small widgets (like <toolbarbutton label="Reload" />) and for managing the application (like <tabbrowser />, which controlled much of the state within a browser window by managing tabs, receiving messages from content pages, etc).

XBL has served the project well and has made it possible to build Firefox for nearly two decades. The implementation has needed remarkably few changes over time. But it’s known to cause a lot of problems for us. Specifically:

  • There are hard to debug complications with binding lifecycles in our UI, and very few people know how it works.
  • It adds enormous complexity to our platform in the frame constructor, style system, and the DOM implementation.

Why Web Components?

Because of the problems above, we’ve talked about removing XBL over the years. But it hadn’t gotten much traction - the project just seemed too large and kept looking like it would require rewriting the Firefox UI from scratch.

However, the description of XBL above might sound familiar to you if you’ve worked with Web Components. Instead of “XBL bindings” there are “Custom Elements”, and instead of “anonymous content”, there is “Shadow DOM”. We did a more thorough analysis of the differences between XBL and Web Components if you’re interested in the details, but the gist was that we thought we would be able to migrate our existing components without requiring a total rewrite.

Next we did a design review in which we proposed starting a project to migrate XBL to Web Components. As I mentioned above, we knew this would be a long commitment but we thought the benefits justified the cost. Management agreed, and we kicked off the project.

Because the models are quite similar, we made a deliberate choice to keep API compatibility as much as possible when migrating elements. So we consistently pushed back against requests to refactor individual components as we touched them, unless if doing so made sense for the project. If we included that work the project never would have ended. Although we, did find that once elements were implemented in standard JS it became easier to clean up and modularize, so people would start to do it on their own as they worked with them.

“De-XBL”

There are a lot of details I’m going to gloss over here - it’s taken 19 newsletter posts so far to share the work, and has been tracked in a giant metabug that started near the end of 2017.

The easiest way to explain the work is to split it up into phases. There wasn’t a crisp start and end to each of these, but the focus of the team tended to shift over time from one to the other:

  • Planning and tooling. In order to be confident this project would work, I wrote some scripts that could show us which bindings were ready to work on, and help to convert XBL bindings into JS classes. I also put up a website to help show our progress at arewexblstill.com. Seeing continual progress on the site helped to make the project feel achievable for the team working on it, and surfaced our work to people that weren’t involved with it day-to-day.
  • Auditing and removing unused or unnecessary code. Out of the 300 bindings, 77 of them were considered ‘unused’. Some of them were truly unused from the start, but others became unused as other teams shipped things like QuantumBar and the about:addons rewrite. 58 more didn’t need to be applied to DOM elements and were converted to JS modules.
  • Custom Elements. In the meantime, the DOM team was hard at work on shipping Custom Elements. Once this landed we converted our first binding to a Custom Element in May 2018. We started with elements that could be migrated to non-anonymous DOM, since Shadow DOM hadn’t shipped yet. In all, 78 bindings were converted to Custom Elements (though there are many more throughout the codebase today).
  • In-content XBL and UA Widget Shadow DOM. In-content bindings were pieces of UI that ran inside of web pages but aren’t visible to the page. These were extra complicated, and so we couldn’t just migrate them to Custom Elements. We took the chance to rethink how this is done and came up with a simpler design that relies on Shadow DOM called “UA Widgets”. This also took us to some strange places like shipping HTMLMarqueeElement in the year 2018. In total there were 39 in-content bindings.
  • Shadow DOM and Shadow parts. There were many elements that used anonymous content and would have been difficult to implement without it. They would slot content from the page into specific places in their content or style the anonymous content separately from the page. Shadow DOM was a solution for the former and Shadow Parts for the latter. As this feature shipped in the platform we started using them and providing feedback to the engineers working on the implementation.
  • Remove the XBL implementation from the platform. This sounds easy, but it takes some care. We’re currently in the process of doing this. So far we’ve removed around 23,000 LOC and a bunch of complexity in important parts of the engine.
  • Continue on with the rest of the broader XUL replacement program. This project is just part of that work. Thankfully, the biggest part.

It couldn’t have been done without a concentrated effort from the people removing bindings and help from teams across Firefox when we needed questions answered, patches reviewed, and new platform features prioritized. This type of work is often behind the scenes, and I’m grateful to everyone who has helped push this forward over the last two years.

Web Components changes we made in Firefox as a result of using them in the UI

We hoped that because of dogfooding Web Components in our own UI we would be able to fix bugs, improve performance, and request features in Web Components that we wouldn’t have otherwise been able to. Here are some of the things that came of that:

  • New feature: CSS Shadow Parts. This would have been implemented eventually, but we really needed the ::part selector in order to port some of our more complex XBL bindings to Custom Elements without reworking them entirely, so the priority was increased to help.
  • DevTools: Web Components support. The DevTools and DOM teams worked together to build tools to inspect Shadow DOM and to jump directly from a node in the Inspector to the Custom Element class definition in the Debugger. There are more details in the 7th newsletter.
  • Performance improvements. The Firefox UI is heavily performance tested, and as we started using Web Components we noticed some issues with the implementation. The DOM team was always very responsive with getting these fixed. For example, we identified some regressions on the Dromaeo performance tests that were fixed. There was also a case where a user with over 1500 tabs open (scientifically considered a “tab hoarder”) noticed extreme slowness when opening the “all tabs” dropdown. It turned out that he had tripped on an O(N²) edge case, and the issue was fixed.
  • New Firefox UI only features:
    • Lazy defining custom elements. The browser startup time is very closely monitored and in general it’s not acceptable to land things that slow it down. Some elements like <findbar> are actually lazily injected into the DOM and aren’t visible at startup. With XBL it wouldn’t attach until the element became visible, but with Custom Elements we were loading the class definition up front. We saw some performance issues with just loading and parsing the JS files, so we added a mechanism to fire a callback when an element first appears in the DOM to give us a chance to synchronously load the class and register the Custom Element. I don’t think this is suitable for exposing the Web, since there isn’t a way to synchronously load a script in the same manner.
    • Extending customized autonomous elements. We had discussion thread with more details. I did file an issue on the standard to see if there was interest in exposing this behavior, but there wasn’t. Our case was a bit more complex since we had existing JS/C++, along with a bunch of CSS, referencing the overloaded tag names.
    • Allowing non-hyphenated tag names. The standard defines a valid Custom Element name as including a hyphen. After a discussion thread we decided to loosen that restriction in the interest of incrementalism, so that we didn’t need to do the actual conversion and rewrite existing markup at the same time. We’ll probably move to dashed names now that we’ve completed the migration.

XUL/XBL Replacement Newsletter 19

This is the nineteenth edition of the XUL/XBL Replacement Newsletter. Lots to share this time - we’re making progress on the remaining XUL replacement work, while also clearing out the XBL implementation from mozilla-central.

<html> root in browser.xhtml

We had already migrated the main browser window from a XULDocument to HTMLDocument and renamed it from browser.xul to browser.xhtml. But there are still a lot of XUL elements inside of it. Most notably, the documentElement was still a <xul:window>, which meant that we would use a different frame structure than normal webpages.

Brendan Dahl recently landed a change to fix this. Now instead of:

<window title="Firefox">
  <linkset>
     <link rel="localization" />....
  </linkset>
  <script />...
  <the-rest-of-the-content />
</window>

The document looks like:

<html>
  <head>
    <title>Firefox</title>
    <link rel="localization" />....
    <script />...
  </head>
  <body>
    <the-rest-of-the-content />
  </body>
</html>

Ideally, I’d like to do the same thing but for all documents in the tree and then remove nsDocElementBoxFrame (the layout frame used for XUL root elements). This would be a big job, since there are 1200+ <window> tags in tree and the migration likely requires some automatic parsing and rewriting XML. But the browser document is an important first step and gives us confidence that it’ll work with other documents. Also, Kirk Steuber has been taking steps to make the project more manageable by migrating away from the non-“window” XUL root elements <wizard> and <dialog>. I’m also hoping we can use some of the learnings and process from Emma Malysz’ ongoing work to rename all .xul files to .xhtml to help.

L20n

L20n is the code name for an effort that started three years ago to introduce Mozilla’s new localization system (called Project Fluent) into Firefox. It’s an important part of XUL replacement, but it also improves the quality and resilience of localization in Firefox as Zibi Braniecki outlines in a post summarizing the start of the project.

Since then, the work was planned, the core internationalization module in Gecko was refactored, and Fluent was landed and integrated into all major components of the Firefox UI. And recently the old DTD-based localization system for XUL/XHTML has officially been deprecated. So as of now, we are considering the L20n project complete.

The next project is to complete migrating all remaining strings in the Firefox UI to use Fluent. Today around 37% of Firefox l10n strings use Fluent, so there’s still a lot more to do. We are first focusing on removing DTD strings that can cause the “Yellow Screen Of Death” (a startup crash due to an XML parsing error), and on building tooling to help speed the remaining conversions. This work is being tracked at https://arewefluentyet.com/.

AppWindow is the new nsXULWindow/nsWebShellWindow

In the past, top level windows (e.g. the browser UI, places UI, etc) have had an nsXULWindow associated with them for controlling various things related to the OS window. However, as we migrate more of these windows to XHTML and HTML, the “nsXULWindow” name makes less sense. So after discussing with smaug, Brendan changed the name to AppWindow. As part of this cleanup we were also able to merge nsWebShellWindow into nsXULWindow, since it was the only class extending nsXULWindow. We hope the AppWindow name makes the purpose more clear.

Goodbye, XBL

Last newsletter, we were building GeckoView without MOZ_XBL. Since then, Brendan has landed a change to also build desktop without MOZ_XBL and Boris Zbarsky has posted to mozilla.governance about removing the XBL module. We’ve also closed all unnecessary bugs in Core::XBL on Bugzilla, and moved any still relevant bugs into appropriate components.

In the meantime, we’ve started to remove the implementation and clean up related code. Thanks to everyone who’s been pitching in here! Removing the implementation isn’t as easy as it may seem, since it’s wound through many different components and features. If you are aware of anything that can be removed or simplified now that XBL is gone, please file a bug blocking the remove-xbl-implementation metabug.

Here are some of the recent changes:

XUL/XBL Replacement Newsletter 18

This is the eighteenth edition of the XUL/XBL Replacement Newsletter. Even during the excitement of hitting 0 XBL bindings, there’s been a lot of progress and some updates to share.

GeckoView Is Built Without XBL

As we got close to 0 bindings, Brendan Dahl created a build option to disable XBL in Gecko. This wasn’t a simple task due to the amount of places throughout the codebase that reference or interact with XBL. Thanks to Brendan, and to Boris Zbarsky for reviewing this change.

One nice thing about introducing a build option is that we were able to turn off XBL entirely in GeckoView! So this is the first platform where we are shipping Gecko without XBL.

We also have a bug on file to disable XBL in desktop Firefox, which we plan to do after the upcoming merge to 72 on 2019-10-21.

Finally, we are tracking some of the tasks and improvements that can be made once XBL is turned off. This is done in the following metabug: remove XBL support from Gecko. If you know of changes that would be good to make post-XBL, please help us track the work by blocking that bug.

Renaming XUL Files

Emma Malysz has been making quick progress on scripting migrations of .xul files to .xhtml, one top-level directory at a time. As a reminder, we no longer load anything as a XULDocument so chrome xul files are effectively the same as xhtml files. Renaming the files is an important step in making that clear and finishing the “XUL Document Removal” project.

The majority of xul files in tree are tests. One tricky part with them is making sure that we don’t break intermittent test tracking. For instance, if there’s a bug titled:

Intermittent test_largemenu.xul | menu movement (6000, 100) x - got 2554, expected 1530

Then when a failure happens in treeherder with a test called test_largemenu.xul, it will automatically suggest the known intermittent bug based on the summary. However, renaming the file with the .xhtml extension breaks the tracking and the test appears as an unknown orange, which causes extra work for anyone viewing results from pushes to treeherder.

I had previously consulted with the sheriffs and migrated a single test file, confirming that by choosing a summary with both the old and new file name in it it will track intermittents on m-c (where it now has the new name), and beta/release (where it still has the old name). For example:

Intermittent test_largemenu.xhtml,test_largemenu.xul | menu movement (6000, 100) x - got 2554, expected 1530

There are around 100 intermittent bugs that reference xul files, so rather than manually making this change, Emma worked on scripts to:

  • Pull down the existing bugs and modify the summary string
  • Call the Bugzilla “Update Bug” API endpoint for each bug to update the title.

Thanks also to Emma Humphries for helping out with the Bugzilla integration for the above.

Improving the Keyboard Shortcut Handler Implementation

Keyboard shortcut handlers (anything declared with a <xul:key> element and various built in shortcuts like PgUp, PgDown and edit commands) were sharing the same event infrastructure as XBL <handler>. In order to make it possible to build with XBL disabled, Mossop removed the dependency on XBL. This is a followup to moving the definitions out of XBL, and throughout this work he’s been able to make improvements like adding missing test coverage for this core feature.

See the Keybindings in Firefox document for more details and a plan for how we can make it possible for JavaScript to register keyboard shortcuts at the system level so the events correctly wait for the content process to handle them as necessary.

Next up is replacing <xul:key> which has a number of longstanding issues including requiring an oncommand attribute in order to fire, which in turn stops us removing inline event handlers in browser.xhtml, which in turn blocks further improving CSP and eval() defense-in-depth security improvements. The idea will be to switch to JavaScript ways of registering handlers, either through a Custom Element or window initialization code.

XUL/XBL Replacement Newsletter 17

This is the special “0 bindings left” seventeenth edition of the XUL/XBL Replacement Newsletter. I’m proud to announce that as of today, we have no more XBL bindings left in Firefox!

There’s still a lot more to do on the larger XUL Replacement project, and we’re still in the process of removing XBL support in Gecko. I’ll send a follow up post outlining the next steps, but I wanted to take this opportunity to celebrate the accomplishment and acknowledge the efforts over the last two years of steady progress. There have been too many people to list who have been involved with writing and reviewing patches, answering questions, and breaking down bugs. And others who haven’t been directly working on this but have been affected by it being prioritized over other work.

Binding Removals

There are 0 bindings left, compared to 7 from the last update and 300 from the start of the project. Here’s a list of changes: