This is the first edition of the XBL Replacement Newsletter. First, as a reminder about the project: we are planning to replace XBL in Firefox (see this blog post for more details). There are 4 main focus areas for the project this quarter, so I’ll have a quick update on each:

1. Migrate a few bindings and update the plan based on what we learn

We’ve already been able to remove some bindings, I’ll highlight each of them at the end of the post. As for updating the plan - we’ve started to look at some of our our options for accessibility when moving away from XBL. At least for <image>, we decided to move the role assignment into C++ based on the tag name, but if we could find a way to expose the role from a Custom Element definition that would allow us to keep the role in the frontend. That part is a bit TBD for now while we are still working on XUL support in Custom Elements.

2. Land XUL support for Custom Elements

There has been some progress on the platform side here, mostly happening in Bug 1404420 and blockers.

Once we have platform support we’ll still need to figure some things out for the frontend, like how to load the scripts and stylesheets needed for the Custom Elements. Since XBL scripts are loaded via CSS and styles are specified via <resources>, we don’t have an exact analog. I’ve been taking a preliminary look into our options in Bug 1411707. We’ll ultimately have to do some performance testing to see what works best.

3. Improve tooling to make converting bindings easier

There’s a new website https://arewexblstill.com that contains tooling and dashboards. I’d like to thank Potch for helping with the site - he helped with the scripts that process XBL and also maintains a node XML parsing library that makes it easy.

This site has a timeline of binding removals and additions, a set of graphs showing various metrics (# of bindings, LOC, LOC per binding, and most used bindings), and a component tree showing inheritance hierarchy of bindings. There’s also an online XBL to Custom Element converter which is useful for prototyping.

4. Begin a bug breakdown for individual bindings

While we are working on Custom Element support, there’s still plenty of work that can be done. Sometimes when looking at a binding we realize that it’s never used or that it’s trivially extending another binding and can be folded into the parent. It’s not an exact science, but I’ve been looking for such bindings in the component tree and filing bugs. If you know of any bindings that can be removed, please file a bug blocking the De-XBL meta.

Bindings that have been removed so far

We have “only” 289 more bindings to go. That’s a lot, but we started with around 300.

We’ve removed 7 bindings so far: “tabbrowser-tabbox”, “image”, “radio” (windows-specific), “radio-with-spacing” (linux-specific), “checkbox-baseline”, and “checkbox-radio”, “panebutton”. There are 4 other bindings that’ve been removed but not specifically as part of the project: “plugin-popupnotification-center-item”, “click-to-play-plugins-notification”, “history-dropmarker”, “menuitem-iconic-tooltip”

OK, now for some of the changes:

• Paolo removed the image binding by moving the small amount of XBL code it had into XUL. This is an important one because although it had very little code, it accounts for around 25% of XBL instances created during a mochitest-browser run:
• Dão removed the Windows and Linux radio overrides (“radio”, “radio-with-spacing”). In the process we found that the only other binding (“toolbarpaletteitem-spacer”) in toolkit/themes/windows/global/globalBindings.xml was unused and we could delete the entire file
• Mossop removed the unused checkbox-baseline and checkbox-radio bindings
• I removed tabbrowser-tabbox in favor of a new attribute on tabbox instead, and also removed the viewbutton binding and the pageInfo.xml file that it was in.

XBL (XML Binding Language) is a markup language used to define appearance and behavior for UI components in Firefox. The implementation of XBL and the set of implemented bindings are deeply ingrained into Firefox, and it's been an influence for Web Components specs like Shadow DOM.

But there are a number of problems with it. And since we're no longer bound to support XUL extensions and the Web Components implementations are progressing, it's a good time to think about what a migration away from XBL would look like.

We've proposed a plan do to just this, so I'll give an overview of the plan and share some of the tools I created to help come up with it. Note that much of this is covered in more detail in this design review document.

Firefox has a lot UI written in XBL - around 300 individual bindings and over 40K LOC with a pretty wide distribution of complexity, so we'll need to take different approaches depending on the binding. Here are some of the approaches we identified:

1. Convert many bindings to Web Components

For a simple case where the binding is only specifying some JS and anonymous content that gets attached to a node, it can be relatively straightforward to transform XBL into a JS class. I've built an online XBL to Custom Element converter that demonstrates this.

Note: there are XBL features like implements that use XPCOM and can make things things significantly more complicated (see "Special Cases" below). But to get a feel for the syntax, here's a simple example:

<?xml version="1.0"?>
<bindings xmlns="http://www.mozilla.org/xbl"
          xmlns:xul="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
  <binding id="okcancel">
    <content>
      <xul:button label="OK"/>
      <xul:button label="Cancel"/>
    </content>
    <handlers>
      <handler event="input"><![CDATA[
        this.onClick(event);
      ]]></handler>
    </handlers>
    <method name="onClick">
      <parameter name="aEvent"/>
      <body><![CDATA[
      console.log("Clicked", aEvent.target);
      ]]></body>
    </method>
  </binding>
</bindings>

Becomes:

class FirefoxOkcancel extends XULElement {
  connectedCallback() {
    this.innerHTML = `
      <xul:button label="OK"></xul:button>
      <xul:button label="Cancel"></xul:button>
    `;

    this.addEventListener("input", event => {
      this.onClick(event);
    });
  }

  onClick(aEvent) {
    console.log("Clicked", aEvent.target);
  }
}
customElements.define("firefox-okcancel", FirefoxOkcancel);

And the markup would change from:

<style type="text/css">
  okcancel { -moz-binding: url(okcancel.xml#okcancel); }
</style>
<okcancel />

to:

<script src="okcancel.js"></script>
<firefox-okcancel />

Notice that we are extending XULElement instead of HTMLElement. We made an intentional decision not to couple replacing XBL with replacing XUL. This is to limit the scope and risk of the project, since XUL provides a lot of features that HTML doesn't. Also, replacing XBL doesn't prevent or add a lot of extra cost to replacing XUL, if we decided to do so in the future.

A few notes about the differences:

• With XBL the binding is attached via CSS (see for example xul.css). A Custom Element is attached via tag name, and is registered in a normal script via customElements.define.
• Because bindings are attached via CSS, it makes migrating to Custom Elements tricky if more than one binding is attached to the same tag based on a CSS selector (i.e. button vs button[type="repeat"]). We'll need to either have two different elements (firefox-button and firefox-repeatbutton), or fold the logic into a single Custom Element definition and branch on the attribute.
• XBL provides a <resources> element which allows you to specify stylesheets that will be scoped to the <content> within the binding. This is useful both as a way to only load styles when a binding is used in a document, and as a way to scope styles. Shadow DOM provides similar a capability, but with just Custom Elements we'll need to work around this.
• Custom Element tag names require a dash by spec, however we could bypass that for chrome documents if we want to avoid a mass find-replace.

2. Convert some bindings to JS modules

There are some bindings that aren't actually widgets - bindings that are only used once, or don’t render any DOM. These can generally be converted to plain JS objects - either inside of a JSM file which is loaded only once per session, or in a script loaded inside of each browser window.

3. Unrefactored XUL/JS/CSS

Bindings that are rarely used and insert relatively little content can be unrefactored into their constituent parts (XUL, JS, CSS) and inserted into each bound element. It would be up to whoever was working on an individual binding to decide if this approach makes sense - we identified some candidates in the review.

4. Flatten the inheritance hierarchy

We use inheritance pretty widely in our bindings. There are likely cases where we don’t need to - the extra inheritance may have been required for XUL extensions, or may have had a purpose at one point but doesn't any longer.

I've created a page to help visualize the inheritance tree. Some potential candidates for this approach can be found in this tree where a binding is the only child of its parent, and the parent is never directly used.

5. Handle special cases

There are plenty of bindings in Firefox that don't fit into any of the approaches above. Generally, bindings that use the [implements] attribute but don’t map pretty closely back to an HTML tag may fall into this category.

Here are a few examples:

• In-content XBL may provide special challenges due to security considerations (Scrollbars, Marquee, Video controls, Click-to-play plugin UI)
• <tree>, which implements nsIDOMXULTreeElement and has a pretty complicated API
• Bindings which take advantage of CSS selector matching to switch between bindings at runtime

What's next?

The next steps are mostly being tracked in this meta bug:

• Migrate a few bindings and update the plan based on what we learn
• Land XUL support for Custom Elements
• Improve tooling to make converting bindings easier
• Begin a bug breakdown for individual bindings

User agent styles refer to the styles loaded by the default stylesheets within a browser. Firefox has some basic style rules that get applied to every page - for instance, links are blue and

tags are big and bold with margins.

Sometimes these styles can interact with the styles you add to a page, which is why it is important that you can see them in devtools. I recently added a feature in Firefox 32 (currently Nightly) that lets you inspect these rules. See Bug 935803 for the implementation details.

Within the inspector, these styles behave pretty much like any other style on the page. It lists rules in the order of application, and you can see which rules have been overridden. There are a few ways that devtools distinguishes between the UA styles and normal styles:

1. They are not editable. Since you can't change these files, you are not able to edit/disable them within the tools.
2. They are shown with a slightly different background color.
3. The links to the stylesheet open a popup window to view the source of a sheet instead of loading the sheet in the Style Editor. There isn't much purpose for loading these in the style editor (see 1), but seeing the source of the sheet can still be nice for the curious.

---

Enabling User Agent Styles

This is turned off by default. To enable user agent styles, select the "Show Browser Styles" checkbox from the options panel:

You can open any page and toggle this feature on to see it. But here's a quick page to try it with:

data:text/html,<blockquote style='color:red;' type=cite><pre _moz_quote=true>fun <a href='foo' style='color:orange'>to</a> inspect</pre></blockquote>

We (Nicole, Aaron, and I) have had a lot of fun doing Node Knockout the past few years. Last year (in 2012), our team got first place in the 'Completeness' category for our app asciigram, which converted a webcam stream into ASCII art.

This year we decided to up the game a little bit, and to try and convert any video file into another video format, while allowing filters to be applied to the video - all inside of the browser, without uploading anything. The result is Video Funhouse (here is our devcomo team page). I'm particularly proud of our project and team this year, since it was a very busy weekend all around.

Obviously, we knew we wouldn't be able to do this in a weekend on our own without relying on great libraries and tools. There is a project called Emscripten, which is an LLVM to JavaScript compiler, so we figured we could try building FFmpeg or avconv into JavaScript to do the heavy lifting for the project.

videoconverter.js

I've open sourced the build utilities and the ffmpeg.js file that we are using in the project as: videoconverter.js.

Building these projects with Emscripten was a little tricky, but actually quite simple when you consider what it was doing for us. What is also amazing about these tools is that I have little experience with build systems like this, and the tools generally just worked after a bunch of trial and error.

Why Would You Compile FFmpeg Into JavaScript?

Mostly just to see if it would work. Also, it seemed like it would be a fun project.

So, How Do You Compile FFmpeg Into JavaScript?

Here are some of notes I made while doing this, for future reference:

Compiling avconv seemed more promising at first, but when running it we bumped into some weird runtime errors with swscaler. Couldn't figure it out quickly, so focused on FFmpeg. We pulled down the latest Emscripten, and the latest FFmpeg:

emconfigure ./configure --cc="emcc" --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic --disable-doc --disable-devices --disable-pthreads --disable-w32threads --disable-network --enable-small --disable-hwaccels --disable-parsers --disable-bsfs --disable-debug --disable-zlib

emconfigure ./configure --cc="emcc" --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic --disable-doc --disable-devices --disable-pthreads --disable-w32threads --disable-network --enable-small --disable-hwaccels --disable-parsers --disable-bsfs --disable-debug --disable-zlib

git clone [email protected]:kripken/emscripten.git
git clone git://source.ffmpeg.org/ffmpeg.git ffmpeg

You may need to also get the SDK to make sure Emscripten will work. The have documentation on their site about getting this to work. Here are the basic configuration options we ended up using. Most of the configuration options are actually disabling features that either don't make sense, or are there to save space. After a lot of trial and error, here is the minimal amount of flags to pass into emconfigure: --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic.

The first time we got a successful build it was a 100MB JS file. By removing a bunch of the other stuff, we get it down to 25MB. It could actually get smaller as well - we were unable to minify the script because the closure compiler process ran out of memory. The following commands are updated after we figured out some better options. The demo as it is right now doesn't run any optimizations on the JS file, and it is about double the size. We can't update the demo, as the weekend is over, but this commit is promising: ffmpeg.js: 190 additions, 1463201 deletions not shown.

emconfigure ./configure --cc="emcc" --disable-ffplay --disable-ffprobe --disable-ffserver --disable-asm --enable-pic --disable-doc --disable-devices --disable-pthreads --disable-w32threads --disable-network --enable-small --disable-hwaccels --disable-parsers --disable-bsfs --disable-debug --disable-zlib

emmake make;

emcc -O2 -s VERBOSE=1 -s ASM_JS= -s ALLOW_MEMORY_GROWTH=1 -v libavutil/*.o libavcodec/*.o libavformat/*.o libavdevice/*.o libswresample/*.o libavfilter/*.o libswscale/*.o *.o -o ../ffmpeg.js --pre-js ../ffmpeg_pre.js --post-js ../ffmpeg_post.js

Peformance

The FFmpeg process runs inside of a web worker to prevent locking up the UI. This was the first optimization, as it was necessary for doing any testing on the project at all. Data is piped back from stdout and stderr to the frontend of the demo app, which is listening for these and logging them to a pre tag. This logging could be limited or removed to limit the impact on the DOM, as a lot of data can be generated by the system.

We experimented with firing up 2 different workers, one for the metadata probe and actual processing, and one for taking screenshots of a video right after receiving it. But we noticed some general instability when both were running at once (many times a tab or browser would crash), so we removed the screenshotting functionality. I'm sure we could have traced it down and improved the performance if we had more than a weekend for the project.

The processing itself is a little slow, but I haven't done benchmarks to compare with native. I generally avoided processing videos of much size. Also, our demo app prints a lot of logs which may not be helping matters. We are looking into how to set up performance tests, but if anyone wants to help with this please let me know or submit a PR on the videoconverter.js project.

I believe this could be quite a bit faster - it couldn't actually use asm.js because it is explicitly disabled when ALLOW_MEMORY_GROWTH is defined. It would be worth experimenting to see if we could set a large max memory size and enable asm to see what kind of speedup we saw.

I'm interested to see if we can get some sort of performance benchmarking set up to compare how fast it runs in different JavaScript environments, and to see how it compares with native performance.

Potential Uses

• This could be extended to bundle other encoding libraries. I wasn't able to figure out how to get the linking to work for libvpx, x264, libtheora, zlib, etc over the weekend, so certain formats cannot be encoded. It would be neat to have this functionality, as it would allow conversion into web friendly video, which could then allow further previewing and processing. As performance improves and FileSystem API support improve it may be possible to build a web-based video or audio editing tool using this.
• Compile the other utilities - there are other programs that ship with FFmpeg that we are excluding right now. ffprobe in particular seems like it could be better for gathering the metadata after the initial load of the file.
• As mentioned previously, this could be useful as a benchmarking tool.
• Probably other things I haven't thought of as well.

Heather and I spent some of our @FirefoxDevTools team meetup working together on an experiment we named the Pixel Inspector. The idea is to focus on an individual region of the page with an eyedropper to pick colors and a zoom feature to debug tricky alignment issues. You can get separate programs that do some of this, such as Digital Color Meter, or addons like ColorZilla, but it would be great to have something built into the developer tools to do this. Here is how it works:

• Open the panel with a keyboard shortcut or by clicking a button in DevTools.
• Move your mouse around the page to the location you want to inspect. Click on the pixel to lock it in place.
• Use the zoom slider to increase the size of each individual pixel.
• Press the 'Copy' button to copy the color to your clipboard, or use the format dropdown to change the color format.

See the screencast and screenshot below to check it out. Would this be a useful feature to add to DevTools?

How does it work?

Pixel Inspector makes use of the drawWindow canvas method (which is only available with "chrome only", meaning a script running with elevated privileges that has access to browser internals). There is an additional flag, named context.mozImageSmoothingEnabled = false; that allows the image to get the pixelated effect we want. Most of the code can be seen in this file: https://github.com/harthur/devtools/blob/magnifier/browser/devtools/magnifier/magnifier.js.