A local model

Our initial approach was just directly sticking an LLM into an application that takes the DOM and spits out a curated semantic outline of the page. While this worked well in our early experiments, it has obvious privacy concerns. Imagine sending all of your page data to a separate company. A screen reader is with you for everything you do on your computer: your email, bank statements, medical portal, private messages, and work documents. Routing each of those pages through a remote model would amount to giving another company a running view of your entire computer, just so you can navigate it.

We wanted to experiment with an alternative approach: could we fine-tune a local language model to produce these semantic outlines while keeping page content on device?

Such an approach also carries other constraints. First, a system running in the background needs to work on a wide variety of hardware and take up minimal space. Secondly, it must do an acceptable job at producing semantic outlines. Thirdly… what does it even mean to produce a good semantic outline?

We’ll spend most of this note on points two and three. After some initial testing on outlines and latency (including Qwen3.5-9B, Qwen3.5-4B, Qwen3-4B-Instruct-2507, and Qwen3.5-0.8B), we selected Qwen3.5-2B for fine-tuning.

Its initial results weren’t great. One issue was the amount of content we were asking it to work through: raw DOM often bloated the context window with wrapper nodes, long text, and repeated list structure. Before fine-tuning, we needed to make that input smaller while keeping the parts someone would need to navigate the page.

We tried many variations of ways to construct this tree, including semantic similarity, templates, and hierarchical clustering. In our pilots, the qualitative results weren’t worth the extra latency and cost. We ended up with a pipeline that compacts the source tree and uses pointer-based output rather than full tree generation.

To do that, we created an intermediate representation (IR) that gives the model a more consistent description of the interface:

Identity

An ID for each element, so the outline can point back to it.

Content

Roles, text, names, and descriptions.

State

Checked, selected, expanded, and other current states.

Relationships

Labels, descriptions, and errors associated with an element.

Structure

Headings, lists, tables, and their relationships.

We compute a shared IR so that the same model can work across platforms: web, macOS, Windows, and others. It is designed to support web accessibility trees, macOS AX, and Windows UIA. The implementation here starts with the web; native platform adapters are a next step.

We also use deterministic compaction to reduce the input: removing wrapper nodes, truncating long text, and summarizing repeated list structure. In this initial experiment, that brought the input down to a median of about 10.6k tokens. The aim is to spend fewer tokens on implementation details while preserving the information needed to navigate.

What makes a good outline?

First we developed a rubric to assess whether a semantic outline was good. We conducted an iterative review of 25 generated outlines, critiquing each one and identifying useful navigational cues. A few qualities seemed to lead to more useful outlines.

Nesting adds semantic grouping. Some outlines mirrored the accessibility tree, producing long chains of groups with only one child each. Every level added another step to navigate without offering a useful distinction. Ideally, each choice should cut the remaining search space substantially, getting us closer to logarithmic navigation as the page gets larger.

Different levels should offer different levels of meaning. Moving down the outline should feel like zooming in: from the page’s broad purpose, to meaningful sections, to individual content and controls. Each level should reveal something more specific. However, this is also branch-specific: level three on one branch might contain an individual control, while level three on another might still be organizing a long list of results. What matters is whether that next level gives someone a useful distinction.

Surface the page’s underlying conceptual structure. We wanted the model to identify what someone might want to understand or do on a page. On a recipe page, for example, “Ingredients,” “Directions,” and “Time and yield” offer useful entry points. They let someone find what they need without first working through the page’s implementation details.

Make the outline shorter while keeping the page within reach. A product card or recipe step can become one meaningful reading unit, with its details grouped together. The semantic outline need not recreate all the details for these objects. It should allow the user to navigate to the object in the DOM, from which they can find the details and interact with it.

Reduce hallucinated page content. Early on, a model generated a group called “Tips: ingredients, technique, storage.” This is a sensible section for a recipe, but the label needs to be supported by content on the page. We need to check what the outline points to, as well as whether its organization sounds plausible.

Given these observations, we created a rubric with four categories: Coverage (how many important elements in the source we cover), Grounding (how well nodes in the semantic tree are linked to real source nodes), Grouping (whether the grouping produces an easily navigable tree), and Labels (whether the labels provide information scent for the elements below them).

We also started testing outlines with computer-use agents: if we provide an agent the semantic outline instead of the raw DOM, does it reduce the token cost or improve task accuracy? We’ll be writing more on this later.

Generating training data

Once we had an input representation and a rubric, we needed examples for the model to learn from. We curated a dataset of Gemini 3.1 Pro–labeled semantic outlines and used those examples for supervised fine-tuning (SFT) of the 2B model.

Each example pairs a compacted semantic IR with an outline: its labels, nesting, and pointers back to source nodes. This gives the small model examples of both the output format and how to organize a page. We focused on grounding during SFT so that the outline would point back to actual elements. These are model-generated training labels, rather than human gold standards.

Results

We ran our base Qwen3.5-2B, an SFT version, and a stronger 27B comparison model on 50 pages each. The base model’s results were lackluster: 49 of 50 outputs failed the validator, and the rubric judge rejected the one that passed. The SFT version dramatically improved results, but it still produced 26 validation failures, compared with 16 for the larger model.

When it does produce valid output, it performs quite well: 19 of 24 outlines were accepted by the automated rubric judge (79%), compared with 31 of 34 (91%) for the larger model. But those percentages leave out the failures. Across all 50 pages, acceptance is 38% and 62%, respectively.

So, is this learnable?

We began this project with the goal of producing a local model capable of running on many devices in the background. These initial results suggest that a small model can learn to read a compacted page and point back at real source nodes. They also show that getting it to do that reliably is still an open problem.

Along the way we developed a compaction algorithm, curated a dataset of model-labeled semantic outlines, and used those to fine-tune a small 2B model. Looking through the traces helped us identify a set of goals for judging the output. The interface design and the model work kept informing each other.

Our next goal is to get closer to the data. Scaling up means having better labels and a clearer understanding of what useful semantic outlines actually are. We’re creating a labeler for manually making and editing these outlines, involving our BLV collaborator.

For me, that’s the interesting part of working across human–AI interaction and the model layer. A choice about how someone should navigate becomes a choice about the representation, the training examples, and the evaluation. Then the model’s mistakes send us back to the design question.