This isn’t an occasional chore. I do it when I’m implementing something new, when I’m working out whether a Gecko bug is a bug in Gecko or a bug in the spec, when I’m reviewing a spec change someone else has proposed, and when I’m trying to design something that has to fit alongside everything that already exists. Four fairly different jobs, and every one of them comes down to the same mechanical work. Start at an algorithm, follow the calls, keep track of where you’ve been.

The specs are built for exactly this, in principle. Every step that invokes another algorithm is a hyperlink, and the whole thing is one large call graph spread across HTML, DOM, Fetch, URL and Infra. The trouble is that walking that graph by hand is slow, and it’s the kind of slow a machine should be doing on your behalf.

Two tools have changed how I do this. One answers questions about the graph, and the other helps me walk it. If you’d rather install them first and read afterwards, they’re both at the end.

webspec-index trace

webspec-index is Jan Jaeschke’s tool for querying web specifications from the command line. It indexes the specs, on demand as it needs them, and lets you ask about definitions, references and IDL. In August it gained a trace subcommand, and that’s the one I keep reaching for.

You give it two anchors and it finds the routes between them. It answers in JSON by default, which is the right choice for feeding it into something else but not for reading, so add --format markdown when the audience is you. Say I want to know how a navigation can end up firing navigateerror:

$ webspec-index trace HTML#navigate HTML#event-navigateerror \
    --format markdown

# trace: `HTML#navigate` -> `HTML#event-navigateerror`

Max depth 6, kind `step`. Found 11 trace(s).

## Trace 1 (5 hop(s))

1) [`HTML#navigate` step 20](…#beginning-navigation:set-the-ongoing-navigation)
   calls `HTML#set-the-ongoing-navigation`
   > Set the ongoing navigation for navigable to navigationId.

2) [`HTML#set-the-ongoing-navigation` step 2](…#aborting-navigation:inform-the-…)
   calls `HTML#inform-the-navigation-api-about-aborting-navigation`
   > Inform the navigation API about aborting navigation given navigable.

3) [`HTML#inform-the-navigation-api-…` step 3.1](…#ongoing-navigation-tracking:…)
   calls `HTML#abort-the-ongoing-navigation`
   - under: While navigation's ongoing navigate event is not null:
   > Abort the ongoing navigation given navigation.

4) [`HTML#abort-the-ongoing-navigation` step 7](…#ongoing-navigation-tracking:…)
   calls `HTML#abort-a-navigateevent`
   > Abort event given error.

5) `HTML#abort-a-navigateevent` step 6 calls `HTML#event-navigateerror`
   > Fire an event named navigateerror at navigation using ErrorEvent, with
     additional attributes initialized according to errorInfo.

I’ve wrapped those lines and shortened the URLs to fit; in the real output each hop is one long line and every link is a full deep link to that exact call site.

Every hop names the exact step number, quotes the step text, and tells you the condition it sits under. That last part matters more than it looks. Step 3.1 only runs while the ongoing navigate event is non-null, and knowing that is the difference between a path that exists on paper and a path your case actually took.

Eleven routes, though¹. Reading eleven of these back to back is not much better than the twenty tabs.

Three zoom levels

This is where webspec-index trace gets genuinely nice, and it’s the part I want to draw attention to, because it’s easy to miss in the help text. The --detail flag changes how much of each route you get, and the three levels turn out to be three different ways of working rather than just three verbosity settings.

Ask the same question again with --detail edges and the prose falls away, leaving the skeleton:

$ webspec-index trace HTML#navigate HTML#event-navigateerror \
    --detail edges --format markdown

# trace: `HTML#navigate` -> `HTML#event-navigateerror`

Max depth 6, kind `step`. Found 11 trace(s).

## Trace 1 (5 hop(s))

1) `HTML#navigate` step 20 calls `HTML#set-the-ongoing-navigation`

2) `HTML#set-the-ongoing-navigation` step 2 calls `HTML#inform-the-navigation-api-about-aborting-navigation`

3) `HTML#inform-the-navigation-api-about-aborting-navigation` step 3.1 calls `HTML#abort-the-ongoing-navigation`

4) `HTML#abort-the-ongoing-navigation` step 7 calls `HTML#abort-a-navigateevent`

5) `HTML#abort-a-navigateevent` step 6 calls `HTML#event-navigateerror`

## Trace 2 (5 hop(s))

1) `HTML#navigate` step 22.4 calls `HTML#fire-a-push/replace/reload-navigate-event`

2) `HTML#fire-a-push/replace/reload-navigate-event` step 2 calls `HTML#inform-the-navigation-api-about-aborting-navigation`

...

Now you can read all eleven at once, and a shape appears that I could not see in the verbose output at all. There are five places inside navigate where a route starts:

And all eleven finish on the same line:

`HTML#abort-a-navigateevent` step 6 calls `HTML#event-navigateerror`

There are exactly two ways into that step, from HTML#abort-the-ongoing-navigation step 7 and from HTML#process-navigate-event-handler-failure step 5. Eleven routes exist because three of the five entry points re-enter the same navigate event firing machinery from different directions, not because there are eleven genuinely different things going on.

That’s a useful thing to know before you start debugging. If a navigation is firing navigateerror and you want to find out why, you don’t instrument eleven places. You instrument the one step they all pass through and work backwards from there.

So the way I use it now is to start at --detail edges to find the shape, then drop back to the default once I know which route I care about and want to read the reasoning. The third level, --detail compact, drops the prose but keeps every anchor as a deep link to the exact call site, which is the form you want when you’re pasting a route into a bug report for someone else to follow.

WebSpec Tracer

webspec-index trace answers a specific kind of question, and it’s worth being precise about which kind. You have to know both ends. That’s fine when you’re asking whether one thing can reach another, but a lot of the time I only know where I’m standing. I have a starting point, a bug in front of me, and no idea yet where the algorithm goes.

There’s also something webspec-index trace can’t do for you in principle. It finds routes through the spec text, statically. It doesn’t know which branch your case took, because that depends on what actually happened at runtime, and only you have that information.

So I wrote WebSpec Tracer, a Firefox extension that walks the specs with you rather than ahead of you. You give it a starting anchor and it shows you that algorithm with every linked term highlighted. Click one and it becomes the next hop in the trace, and the panel moves you into it. Click the wrong one and Back undoes it.

The screenshot is a walk that started at location.assign and has arrived inside navigate. Step 15 is the interesting one. It runs only if four things are all true, that there’s no document resource, no response, the URL matches the current entry’s URL ignoring fragments, and the fragment is non-null. If all four hold, it navigates to a fragment and returns. Otherwise the algorithm keeps going into the full navigation path below.

No tool can decide that for me. Whether my bug went left or right at step 15 depends on the page and on what the user did. But the extension puts the decision in front of me with the alternatives visible, which is precisely what I was failing to do for myself with twenty tabs open.

The trace builds up in the panel underneath as markdown, with every anchor pointing at the exact call site. That box is a display rather than a text field, so there are buttons for getting the trace out of it. Copy does what you’d expect. Insert is the one I actually use, and it’s the reason the extension lives in the sidebar: it writes the markdown straight into whichever field you have focused on the page, so you can have a spec issue open next to the panel and drop the trace into it without going near the clipboard. Reset clears everything and starts over.

Switch the issue over to Preview and there are the same two hops, rendered, each one a link into the spec. That’s most of the reason I built the thing. Explaining a route through an algorithm in prose is miserable, and the version you retype by hand is always slightly wrong. This way what I hand over is exactly what I walked.

One more thing worth knowing, about the anchors themselves. The starting one has to say which spec it’s in, so HTML#dom-location-assign rather than just dom-location-assign, but after that the extension works it out as it goes. Anything that turns up in the trace without a prefix is in the spec you started from, and a prefix appears when the trace crosses a boundary:

That’s the same walk carried a few hops further. Hop 3 is that step 15 decision, taken, and the last line is WEBIDL#wait-for-all. Somewhere in the navigate event firing the trace left HTML entirely, and the prefix is the only thing that tells me so. The panel on the left has stopped showing me HTML too, and is now sitting in WebIDL. That’s easy to lose track of when you’re clicking through by hand, and it’s usually the moment a question stops being about HTML and starts being about something else.

When one call happens twelve times

Sometimes an algorithm invokes another one from many different places, and a trace has to say something sensible about that. This was in the design from the start, and the answer it arrived at is to do what the spec already does. Start at HTML#update-the-image-data and follow it to HTML#abort-the-image-request:

HTML#abort-the-image-request is invoked from twelve separate places inside HTML#update-the-image-data. The extension doesn’t make me choose one, and it doesn’t flatten them into a single link either. It records the hop once and hangs all twelve call sites off it as numbered links, so the line reads #update-the-image-data calls #abort-the-image-request, [2], [3] and onwards to [12], each number pointing at its own anchor.

The numbering isn’t invented. It’s the spec’s own scheme for repeated references: the anchors run #abort-the-image-request, then -2, -3, and so on up to -12. All the trace does is carry that through, so a hop in the output identifies call sites the same way the spec identifies them, and the numbers mean the same thing in both places.

That also keeps the trace honest. The hop happened, and that’s true whichever of the twelve fired, but the fact that there are twelve is worth knowing on its own, and anyone reading can go and check any one of them. Writing this out by hand you would link the first and lose the other eleven without ever noticing. The same thing is visible in the previous screenshot, where hop 3 reads #navigate calls #navigate-fragid, [2], [3].

webspec-index trace deals with the same situation the other way round. Ask it the same question and you get twelve separate one hop traces, each naming its own step number, step 2, then 7.4.2, then 11.1, and so on down the algorithm. Neither is wrong. The command line is telling you which steps, and the extension is keeping your trace to one line while preserving every place it could have come from.

Which one, when

They divide up cleanly enough that I don’t think about it much any more. If I know both ends and want to know what’s between them, or how many different routes there are, that’s webspec-index trace. If I know where I’m starting and want to find out where it goes, that’s the extension.

The one I underused at first was --detail edges. Asking a broad question and then zooming out far enough to see the answer’s shape is a different move from following a single path carefully, and it’s the one that has surprised me most often, usually by showing me that several things I thought were separate all funnel through the same step.

Getting them

webspec-index is on crates.io:

cargo binstall webspec-index

It builds its index as it goes, so the first trace into a corner you haven’t visited before pauses briefly to fetch and index the specs the route touches. After that it’s fast.

WebSpec Tracer is on addons.mozilla.org, and the source is on GitHub. It lives in the sidebar, toggled with Ctrl+Shift+U, so you can keep it open next to whatever you’re reading.

Both are worth a try if you spend any time at all reading specs, whether you’re implementing them, arguing with them, or just trying to find out whether the browser is wrong or you are.

Notes

For years my main tool for understanding that state was reading code and printing things to a log. That works, but it’s slow, and it never quite shows you the shape of the thing. So I built a way to see it: a new DevTools panel in Firefox Nightly called Session History Diagrams.

Enabling it

The panel is available in Firefox today, behind a pref. It’s been there in some form since Firefox 150, growing more stable with each release. To turn it on, set devtools.application.sessionHistory.enabled to true in about:config, then reload DevTools. The new panel lives under the Application tab, next to Service Workers and Manifest, and it draws the browser’s session history as a diagram that updates as you navigate.

Since Firefox 153 it also works over remote debugging. Connect to a device from about:debugging and you can watch the session history of a page running on Android, the same as you would on the desktop.

Jake diagrams

I didn’t invent the idea of drawing this. The HTML spec already has a notation for it, called a Jake diagram after Jake Archibald, and that’s where I started. It’s a tabular notation where columns represent steps in session history, and rows represent navigables (the top-level browsing context plus any iframes). Background colors identify documents, a fresh color marking a new document loaded in that navigable, and the current step is shown in bold. It’s a genuinely useful way to capture multi-navigable interactions that are otherwise hard to describe in prose.

These diagrams don’t have to be drawn by hand. Domenic Denicola, one of the HTML spec editors, built a Jake diagram generator that turns a description of a navigation sequence into a rendered diagram. That’s where I first started playing with a more dynamic approach to the visualization. The thing I missed the most was being able to build a history up step by step rather than describe a finished sequence all at once. So I wrote rejake, a small tool that draws diagrams in the same style¹, but lets you construct the history one step at a time.

But rejake, like the spec’s diagrams and Domenic’s generator before it, was stuck with a limitation the spec itself admits to, that they only work with a single level of nesting. That was exactly my problem. Real pages nest iframes inside iframes, and the bugs I was chasing usually lived down in that deeper nesting, precisely where the diagram stops being able to help. And however I drew them, I was still typing the history out by hand. It’s a short step from there to wanting the diagram to draw itself from the browser’s actual session history instead².

Firefox Session History Diagrams

So the panel extends Jake diagrams to handle arbitrary nesting. Every column is a step in the session history. Every row is a frame, listed in pre-order from the frame tree: top-level document first, then its first iframe, then that iframe’s children, and so on. The current entry is highlighted in blue, and the diagram updates live as you navigate.

The recording above is an ordinary bit of browsing, a handful of pages visited one after another. The top row tracks the page you’re actually looking at, and the current position is the one in blue. The interesting part is everything underneath that top row.

Some of those pages didn’t just load a single document. They pulled in nested frames of their own, and the diagram stacks those below the page that owns them. None of that is visible in the address bar or anywhere in the page chrome, and the frames come and go as you move through the history. Normally you’d have no way of knowing they were ever there. Here you can read straight off the diagram which frames a given step carried, when each one entered, and when it dropped away again.

Who else might want this

I built this for myself, working on Gecko’s session history internals, where being able to watch the diagram change while reproducing a bug turns opaque state into something I can point at. But it turns out I’m not the only one who hits this wall. Plenty of people working elsewhere in Gecko, anywhere near navigation, end up reasoning about the same state, and now we all share one picture of it.

If you build single-page applications, or work with the History API or Navigation API, you’ve probably run into the same kind of confusion from the other side. A push where you expected a replace, a missing history entry, an iframe that accumulated entries unexpectedly. These are hard to reason about without seeing the state directly, and that’s exactly what the diagram gives you.

Session history isn’t a Firefox-specific problem either. Every engine implements the same part of the HTML spec, and Jake diagrams come from that shared spec. The panel only ever shows Firefox’s state, but the rules are the same everywhere, so if you work on another engine it can still be a useful reference for how one implementation behaves. It’s often the only practical way to surface an interoperability difference, which might be a bug in any of the engines, but stays hidden until you can actually see it.

Thanks

A big thanks to Nicolas Chevobbe, whose assistance was invaluable in getting the DevTools integration right. The work, including what’s still to come, is tracked in Bug 2015726. There’s a fair bit still on that list, like marking whether a step was a push or a replace, surfacing back/forward cache state, tying the diagram into the Network and Inspector panels, and more, all heading toward fuller DevTools support for Navigation and Session History.

Notes

What’s the WebIDL step?

When you build Firefox, one of the earlier steps runs python3 -m mozbuild.action.webidl to generate C++ binding code from hundreds of .webidl files. It produces thousands of output files: headers, cpp files, forward declarations, event implementations, and so on. The step isn’t terribly slow on its own, but it runs on every clobber build, and the output is entirely deterministic given the same inputs. That makes it a perfect candidate for caching.

The problem was that the compiler cache was never passed to this step. Buildcache was only wrapping actual compiler invocations, not the Python codegen.

The change

The fix in Bug 2027655 is small. In dom/bindings/Makefile.in, we now conditionally pass $(CCACHE) as a command wrapper to the py_action call:

WEBIDL_CCACHE=
ifdef MOZ_USING_BUILDCACHE
WEBIDL_CCACHE=$(CCACHE)
endif

webidl.stub: $(codegen_dependencies)
	$(call py_action,webidl $(relativesrcdir),$(srcdir),,$(WEBIDL_CCACHE))
	@$(TOUCH) $@

The py_action macro in config/makefiles/functions.mk is what runs Python build actions. The ability to pass a command wrapper as a fourth argument was also introduced in this bug. When buildcache is configured as the compiler cache, this means the webidl action is invoked as buildcache python3 -m mozbuild.action.webidl ... instead of just python3 -m mozbuild.action.webidl .... That’s all buildcache needs to intercept it.

Note the ifdef MOZ_USING_BUILDCACHE guard. This is specific to buildcache because ccache and sccache don’t have a mechanism for caching arbitrary commands. Buildcache does, through its Lua wrappers.

The Lua wrapper

Buildcache’s Lua plugin system lets you write a script that tells it how to handle a program it doesn’t natively understand. The wrapper for WebIDL codegen, webidl.lua, needs to answer a few questions for buildcache:

• Can I handle this command? Match on mozbuild.action.webidl in the argument list.
• What are the inputs? All the .webidl source files, plus the Python codegen scripts. These come from file-lists.json (which mach generates) and codegen.json (which tracks the Python dependencies from the previous run).
• What are the outputs? All the generated binding headers, cpp files, event files, and the codegen state files. Again derived from file-lists.json.

With that information, buildcache can hash the inputs, check the cache, and either replay the cached outputs or run the real command and store the results.

The wrapper uses buildcache’s direct_mode capability, meaning it hashes input files directly rather than relying on preprocessed output. This is the right approach here since we’re not dealing with a C preprocessor but with a Python script that reads .webidl files.

Numbers

Here are build times for ./mach build on Linux, comparing compiler cachers. Each row shows a clobber build with an empty cache (cold), followed by a clobber build with a filled cache (warm):

The “with plugin” column is buildcache with the webidl.lua wrapper active. It shaves another 15 seconds¹, bringing the total down to 1m12s². Not a revolutionary improvement on its own, but it demonstrates the mechanism. The WebIDL step is just the first Python action to get this treatment; there are other codegen steps in the build that could benefit from the same approach.

More broadly, these numbers show buildcache pulling well ahead on warm builds. Going from a 5m35s clean build to a 1m12s cached rebuild is a nice improvement to the edit-compile-test cycle.

These are single runs on one machine, not rigorous benchmarks, but the direction is clear enough.

Setting it up

If you’re already using buildcache with mach, the Makefile change is available when updating to today’s central. To enable the Lua wrapper, clone the buildcache-wrappers repo and point buildcache at it via lua_paths in ~/.buildcache/config.json:

{
"lua_paths": ["/path/to/buildcache-wrappers/mozilla"],
"max_cache_size": 10737418240,
"max_local_entry_size": 2684354560
}

Alternatively, you can set the BUILDCACHE_LUA_PATH environment variable. A convenient place to do that is in your mozconfig:

mk_add_options "export BUILDCACHE_LUA_PATH=/path/to/buildcache-wrappers/mozilla/"

The large max_local_entry_size (2.5 GB) is needed because some Rust crates produce very large cache entries.

What’s next

The Lua plugin system is the interesting part here. The WebIDL wrapper is a proof of concept, but the same technique applies to any deterministic build step that takes known inputs and produces known outputs. There are other codegen actions in the Firefox build that could get the same treatment, and I plan to explore those next.

Notes

For those unfamiliar, buildcache is a compiler cache that can drastically cut down your rebuild times by caching compilation results. It’s similar to ccache, but even more so sccache, in that it supports C/C++ out of the box, as well as Rust. It has some nice unique properties of its own though, which we’ll look at more closely in following posts.

Getting started

Setting it up is straightforward. Just add the following to your mozconfig:

ac_add_options --with-ccache=buildcache

Then build as usual:

./mach build

That’s it.

Give it a try

If you run into any issues, please file a bug and tag me. I’d love to hear how it works out for people, and any rough edges you might hit.