On-Device AI in the Browser¶

On-device AI means running a machine-learning model on the user's own hardware, inside the browser, instead of sending the input to a server. You can do it in two ways: call a model the browser ships (Chrome's Gemini Nano behind the Prompt, Summarizer, Translator and related APIs, or Edge's Phi-4-mini), or bring your own model and run it with WebGPU, WebAssembly or WebNN through libraries such as Transformers.js, ONNX Runtime Web and WebLLM. For a PWA this is a good fit: inference works offline, the data stays on the device, latency doesn't depend on the network, and you pay nothing per request. The costs are multi-gigabyte downloads, hardware requirements that exclude many phones, and APIs that are still Chromium-only or experimental.

Why on-device AI matters for PWAs¶

A PWA already promises to work offline and to feel instant. A feature that calls a cloud model breaks both promises the moment the network drops. On-device inference keeps them.

Offline, privacy, latency and cost¶

Offline. Once the weights are on disk, inference needs no network. A note-taking PWA can summarize, classify or translate notes on a plane, and a field-service app can extract structured data from a technician's free text in a basement. The offline-first architecture you already have for data applies to models too: the model is just another large, versioned asset.

Privacy. Input never leaves the device. Chrome's documentation states that after the initial model download, "No data is sent to Google or any third party when using the model." That changes what you can build: summarizing a user's private messages, proofreading medical notes, or classifying photos without a data-processing agreement for a third-party inference API. It doesn't remove your obligations entirely (see Privacy and security), but it shrinks the data flow you have to document. The privacy page covers the wider rules for PWAs.

Latency. There's no network round trip and no queue on a shared GPU cluster. Small models answer fast enough for interactive features such as autocomplete, inline classification and live translation of chat messages. Large models on weak hardware can be slower than a cloud API, so measure time to first token on your target devices.

Cost. Per-request inference costs move from your cloud bill to the user's battery. For high-volume, low-value calls (tagging every item in a feed, detecting the language of every comment) that difference decides whether you can offer the feature at all.

The limits you have to design around¶

• Model quality. Browser-provided and browser-sized models have a few billion parameters at most. They're good at bounded tasks (summarize, rewrite, classify, extract, translate) and poor at open-ended reasoning and factual recall. Design prompts and UIs around narrow tasks.
• Download size. Gemini Nano and Phi-4-mini are shared by all sites, but the first site to use them pays the wait. Your own models cost from tens of megabytes (embedding models, classifiers) to several gigabytes (chat LLMs). The quantized ONNX build of Qwen2.5-0.5B-Instruct used in the examples below is about 483 MB, according to the Hugging Face Hub's file metadata.
• Hardware. Chrome's built-in foundation model needs more than 4 GB of VRAM, or 16 GB of RAM and 4 CPU cores, on desktop. Phones are excluded from the built-in foundation-model APIs today. Your own models need WebGPU for acceptable speed at LLM scale.
• Browser coverage. Everything built-in is Chromium-only. WebGPU is cross-browser but patchy on Linux and Android Firefox.
• Energy. Sustained GPU inference drains the battery and throttles thermally on laptops and phones.

The practical consequence: on-device AI is an enhancement. The core of the app must work without it, and a cloud path (see Cloud AI APIs) or a non-AI path must exist for devices that can't run a model.

Choosing between built-in, bring-your-own and cloud¶

How the two on-device paths fit together¶

flowchart TD
    A["Feature needs a model"] --> B{"Task-specific built-in API?"}
    B -- "Summarizer / Translator exists" --> C{"availability()"}
    B -- "No" --> F
    C -- "available" --> D["Use built-in API"]
    C -- "downloadable" --> E["Ask user, then create() with monitor"]
    C -- "unavailable" --> F{"WebGPU adapter and enough memory?"}
    E --> D
    F -- "Yes" --> G["Own model in a worker (WebGPU)"]
    F -- "No" --> H{"Small model feasible on WASM?"}
    H -- "Yes" --> I["Own model in a worker (WASM SIMD)"]
    H -- "No" --> J["Cloud fallback or feature hidden"]

Browser built-in AI APIs¶

Chrome's built-in AI is a family of JavaScript APIs backed by models that the browser downloads and manages. The foundation-model APIs (Prompt, Summarizer, Writer, Rewriter, Proofreader) use Gemini Nano in Chrome and Phi-4-mini in Edge. The expert-model APIs (Translator, Language Detector) use smaller task-specific models. The specifications live in the W3C Web Machine Learning Community Group as Draft Community Group Reports: the Prompt API, the Writing Assistance APIs (Summarizer, Writer, Rewriter) and the Translator and Language Detector APIs. None of them is on the W3C Recommendation track.

The shared shape: availability(), create(), monitor and destroy()¶

Every API exposes a global constructor-like object (LanguageModel, Summarizer, Writer, Rewriter, Proofreader, Translator, LanguageDetector) with the same lifecycle:

1. X.availability(options) resolves to one of four strings:
   • "unavailable": the device or the requested options (languages, modalities) aren't supported.
   • "downloadable": supported, but a model (or LoRA weights) must be downloaded first.
   • "downloading": a download is in progress, perhaps started by another site.
   • "available": create() will succeed without downloading.
2. X.create(options) returns a session object. If a download is needed, the page must have user activation. Chrome's docs tell you to check navigator.userActivation.isActive before calling it. The monitor(m) callback receives a CreateMonitor that fires downloadprogress events. Per the explainer, each event is a ProgressEvent whose loaded is a fraction between 0 and 1 and whose total is always 1. At least two events fire (loaded === 0 and loaded === 1) even when nothing needs downloading. If the download fails, events stop and create() rejects with a NetworkError DOMException. Byte counts aren't exposed, by design.
3. Work methods: prompt() / promptStreaming(), summarize() / summarizeStreaming(), write() / writeStreaming(), rewrite() / rewriteStreaming(), translate() / translateStreaming(), detect() and proofread(). Streaming methods return a ReadableStream of string chunks you consume with for await. Every method accepts { signal } for cancellation.
4. session.destroy() frees the session. Passing signal to create() does the same when the signal aborts.

Always pass the same options to availability() that you'll pass to create(). Availability depends on languages and modalities, so a bare availability() can report "available" while create({ expectedInputs: [{ type: "audio" }] }) fails.

This helper wraps the pattern once for every API:

/**
 * Create a built-in AI object (LanguageModel, Summarizer, Translator, ...)
 * with consistent availability checks, user-activation handling and progress.
 *
 * @param {string} apiName  Global name, e.g. "Summarizer".
 * @param {object} options  Options passed to BOTH availability() and create().
 * @param {object} [hooks]
 * @param {(fraction: number) => void} [hooks.onProgress]  0..1 download progress.
 * @param {AbortSignal} [hooks.signal]  Aborts creation (and destroys the object later).
 * @returns {Promise<object|null>}  The session/object, or null when unsupported.
 */
export async function createBuiltIn(apiName, options = {}, { onProgress, signal } = {}) {
  const api = globalThis[apiName];
  if (!api) return null; // Not implemented (Firefox, Safari, older Chrome, workers).

  let availability;
  try {
    availability = await api.availability(options);
  } catch (err) {
    // Invalid option combinations reject (e.g. unsupported language tags).
    console.warn(`${apiName}.availability() failed`, err);
    return null;
  }
  if (availability === "unavailable") return null;

  if (availability !== "available" && !navigator.userActivation?.isActive) {
    // A download is needed and Chrome requires a user gesture to start it.
    // Surface a "Download on-device model" button and call again from its handler.
    const error = new DOMException(
      `${apiName} needs a user gesture to download its model`,
      "NotAllowedError",
    );
    error.availability = availability;
    throw error;
  }

  return api.create({
    ...options,
    signal,
    monitor(m) {
      m.addEventListener("downloadprogress", (e) => onProgress?.(e.loaded));
    },
  });
}

Hardware and platform requirements in Chrome¶

Chrome's get started guide lists these requirements for the APIs that use a foundation model (Prompt, Summarizer, Writer, Rewriter, Proofreader):

The Translator and Language Detector APIs work in Chrome on desktop and not on mobile. The 22 GB figure is a free-space threshold, not the model size. Chrome says the models are "significantly smaller" and shows the current size at chrome://on-device-internals. MDN also notes that availability "may be subject to geographical restrictions". Enterprises can disable the model with the GenAILocalFoundationalModelSettings policy, which makes the APIs report "unavailable".

How Chrome downloads, updates and deletes the model¶

Chrome's model management guide describes behavior that matters for PWA design:

• Variant selection. On the first create() of a Gemini Nano-backed API, Chrome benchmarks the GPU with a representative shader and downloads a larger variant (the guide gives "such as 4B parameters") or a smaller one ("such as 2B parameters"), or falls back to CPU inference if the device meets the static CPU requirements. Output quality can therefore differ between two users of your app.
• Resilient download. Interrupted downloads resume, closing the tab doesn't cancel them, and a browser restart within 30 days resumes them.
• Updates. Chrome checks for model updates at startup, downloads full new versions in the background, then hot-swaps them. A prompt running at the moment of the swap can fail. You can't read the model version from JavaScript.
• Deletion. The model is purged under disk pressure, when an enterprise policy disables it, or when the user hasn't met eligibility criteria for 30 days. It "can be deleted at any time, even mid-session", and isn't re-downloaded until some site calls create() again.

So availability() isn't a one-time check. Re-check it when your feature starts, handle create() or prompt() failing on a session that worked yesterday, and never assume a given model version.

Prompt API (LanguageModel)¶

The Prompt API gives you general prompting of the built-in language model. It shipped for web pages in Chrome 148 on desktop. Chrome Extensions have had it since Chrome 138. Web pages had an origin trial first, which ChromeStatus lists for Chrome 139–144, extended through 147. The global is LanguageModel (earlier drafts used self.ai.languageModel, so ignore old tutorials).

Session creation options:

Session members: prompt(input, options), promptStreaming(input, options), append(messages) (adds context without generating a response), measureContextUsage(input, options), clone({ signal }), destroy(), the attributes contextWindow and contextUsage, and the contextoverflow event. The explainer renamed inputQuota, inputUsage, measureInputUsage() and quotaoverflow to these names. The old names are removed on the web and deprecated in extensions.

Prompt options: signal, responseConstraint (a JSON Schema object or a RegExp) and omitResponseConstraintInput (don't spend context tokens on the schema, so describe the format in the prompt instead). An unsupported JSON Schema feature rejects with NotSupportedError.

Multimodal input: images can be Blob, ImageBitmap, ImageData, VideoFrame, HTMLImageElement, SVGImageElement, HTMLVideoElement (current frame), HTMLCanvasElement or OffscreenCanvas. Audio can be AudioBuffer, ArrayBuffer, ArrayBufferView or Blob. Declare the modality in expectedInputs.

Context window overflow: when a prompt would exceed contextWindow, the oldest user/assistant pairs are dropped (never the initial prompts) and contextoverflow fires. If even that can't make room, the call rejects with a QuotaExceededError and nothing is evicted. Chrome's docs list two properties on the error: requested (the input's token count) and contextWindow (the tokens that were available). Chrome doesn't document a fixed context-window size. Read session.contextWindow at runtime.

The following module implements a production chat session: it streams into the DOM, supports Stop, keeps a meter of context usage, persists history so the conversation survives a reload, and uses a schema-constrained side session for classification.

import { createBuiltIn } from "./built-in-ai.js";

const SYSTEM = {
  role: "system",
  content: "You help users of a note-taking app. Answer briefly. Use Markdown lists when useful.",
};
const OPTIONS = {
  expectedInputs: [{ type: "text", languages: ["en"] }],
  expectedOutputs: [{ type: "text", languages: ["en"] }],
};
const HISTORY_KEY = "assistant-history-v1";

let session = null;
let controller = null;

function loadHistory() {
  try {
    return JSON.parse(localStorage.getItem(HISTORY_KEY)) ?? [];
  } catch {
    return [];
  }
}

function saveHistory(history) {
  try {
    // Keep the stored transcript bounded. The model only sees what fits anyway.
    localStorage.setItem(HISTORY_KEY, JSON.stringify(history.slice(-40)));
  } catch {
    /* Storage may be full or blocked. The chat still works in memory. */
  }
}

export async function ensureSession({ onProgress } = {}) {
  if (session) return session;
  session = await createBuiltIn(
    "LanguageModel",
    { ...OPTIONS, initialPrompts: [SYSTEM, ...loadHistory()] },
    { onProgress },
  );
  if (session) {
    session.addEventListener("contextoverflow", () => {
      // Oldest turns were dropped. Tell the user the assistant "forgot" early context.
      document.querySelector("#context-note").hidden = false;
    });
  }
  return session;
}

export async function ask(question, outputEl, meterEl) {
  const s = await ensureSession();
  if (!s) throw new Error("On-device model unavailable");

  controller?.abort(); // Only one answer in flight at a time.
  controller = new AbortController();

  // Pre-flight the size so you can refuse gracefully instead of evicting history.
  const needed = await s.measureContextUsage(question);
  if (needed > s.contextWindow) {
    throw new RangeError("Question is too long for the on-device model");
  }

  let answer = "";
  outputEl.textContent = "";
  try {
    const stream = s.promptStreaming(question, { signal: controller.signal });
    for await (const chunk of stream) {
      answer += chunk; // Chunks are deltas, not the cumulative text.
      outputEl.textContent = answer; // Render as text. See "Rendering model output safely".
    }
  } catch (err) {
    if (err.name === "AbortError") return answer; // User pressed Stop.
    if (err.name === "QuotaExceededError") {
      // Too big even after evicting history. Start fresh with only the system prompt.
      s.destroy();
      session = null;
      saveHistory([]);
    }
    throw err;
  } finally {
    meterEl.max = s.contextWindow;
    meterEl.value = s.contextUsage;
  }

  const history = loadHistory();
  history.push({ role: "user", content: question }, { role: "assistant", content: answer });
  saveHistory(history);
  return answer;
}

export function stop() {
  controller?.abort();
}

// A separate, short-lived classifier session with its own system prompt.
export async function classifyNote(text) {
  const classifier = await createBuiltIn("LanguageModel", {
    ...OPTIONS,
    initialPrompts: [{ role: "system", content: "Classify notes into one category." }],
  });
  if (!classifier) return null;
  try {
    const schema = {
      type: "object",
      properties: {
        category: { type: "string", enum: ["todo", "idea", "meeting", "reference", "other"] },
        urgent: { type: "boolean" },
      },
      required: ["category", "urgent"],
      additionalProperties: false,
    };
    const json = await classifier.prompt(text, { responseConstraint: schema });
    return JSON.parse(json);
  } finally {
    classifier.destroy(); // Free memory. The model stays loaded while other sessions live.
  }
}

Two session-management details from Chrome's session management guide are worth copying. clone() forks a session with its initial prompts and history, which is cheaper than re-creating and re-processing a long system prompt. And "the model is unloaded after a period of time if there are no living sessions", so keeping one empty session alive keeps the next prompt() fast.

Summarizer API¶

The Summarizer API shipped in Chrome 138 on desktop. Options set at creation (they can't change later, so create a new summarizer to change them):

summarize(input, { context, signal }) and summarizeStreaming(...) produce the output. inputQuota and measureInputUsage(input) tell you whether an input fits. Too-large input rejects with QuotaExceededError. If an implementation chunks internally, inputQuota is Infinity.

import { createBuiltIn } from "./built-in-ai.js";

let summarizerPromise = null;

function getSummarizer(onProgress) {
  summarizerPromise ??= createBuiltIn(
    "Summarizer",
    {
      type: "key-points",
      format: "plain-text",
      length: "short",
      expectedInputLanguages: ["en"],
      outputLanguage: "en",
      sharedContext: "Articles saved for offline reading in a PWA.",
    },
    { onProgress },
  ).catch((err) => {
    summarizerPromise = null; // Allow a retry after a user gesture.
    throw err;
  });
  return summarizerPromise;
}

export async function summarizeArticle(articleEl, outEl, { signal, onProgress } = {}) {
  const summarizer = await getSummarizer(onProgress);
  if (!summarizer) return false; // Caller falls back to cloud or hides the button.

  // innerText, not innerHTML: markup wastes input quota and confuses the model.
  let text = articleEl.innerText;
  if (summarizer.inputQuota !== Infinity) {
    const usage = await summarizer.measureInputUsage(text);
    if (usage > summarizer.inputQuota) {
      // Crude but safe: trim proportionally and summarize the head of the article.
      text = text.slice(0, Math.floor(text.length * (summarizer.inputQuota / usage) * 0.95));
    }
  }

  outEl.textContent = "";
  const stream = summarizer.summarizeStreaming(text, { signal });
  for await (const chunk of stream) outEl.textContent += chunk;
  return true;
}

Writer and Rewriter APIs¶

Shapes, for when they return:

• Writer.create({ tone, format, length, sharedContext, expectedInputLanguages, expectedContextLanguages, outputLanguage }), where tone is "formal", "neutral" (default) or "casual", format is "markdown" (default) or "plain-text", and length is "short" (default), "medium" or "long". Then write(task, { context, signal }) and writeStreaming(...).
• Rewriter.create(...) with tone "more-formal", "as-is" (default) or "more-casual", format "as-is" (default), "markdown" or "plain-text", and length "shorter", "as-is" (default) or "longer". Then rewrite(text, { context, signal }) and rewriteStreaming(...).

Proofreader API¶

Proofreader.create({ expectedInputLanguages }) then proofread(text) resolves to a result with correctedInput and a corrections array whose entries have startIndex, endIndex and correction. Chrome doesn't support the explainer's includeCorrectionTypes and includeCorrectionExplanation options. The Proofreader relies on LoRA weights applied to the base model, which Chrome downloads alongside it.

Translator and Language Detector APIs¶

Both shipped in Chrome 138 on desktop and in Edge 148. They use expert models, not Gemini Nano, so they have lighter hardware needs. They still don't run on Android.

• Translator.availability({ sourceLanguage, targetLanguage }) and Translator.create({ sourceLanguage, targetLanguage, monitor }). Then translate(text) or translateStreaming(text). Chrome processes translations sequentially per translator, so a long request blocks the ones queued behind it. Chunk long documents and show progress.
• LanguageDetector.create({ expectedInputLanguages }). Then detect(text) resolves to an array of { detectedLanguage, confidence } sorted from most to least likely, with confidence between 0 and 1. Per the explainer, the last entry is always "und" (undetermined). Accuracy on single words and very short phrases is low, so apply a confidence threshold.
• Both expose inputQuota and measureInputUsage().

Chrome publishes a static list of supported language codes (about 40 codes, from Arabic to Traditional Chinese) and tracks adding a way to query supported pairs. Always check availability() per pair.

import { createBuiltIn } from "./built-in-ai.js";

const translators = new Map(); // "es→en" → Promise<Translator|null>
let detectorPromise = null;

async function detectLanguage(text) {
  detectorPromise ??= createBuiltIn("LanguageDetector", {}).catch((err) => {
    detectorPromise = null; // e.g. NotAllowedError: retry from a click handler.
    throw err;
  });
  const detector = await detectorPromise;
  if (!detector) return null;
  const [top] = await detector.detect(text);
  // Below ~0.5 the guess is unreliable, especially for short chat messages.
  return top && top.confidence >= 0.5 && top.detectedLanguage !== "und"
    ? top.detectedLanguage
    : null;
}

export async function translateMessage(text, targetLanguage = navigator.language.split("-")[0]) {
  const sourceLanguage = await detectLanguage(text);
  if (!sourceLanguage || sourceLanguage === targetLanguage) return null;

  const key = `${sourceLanguage}→${targetLanguage}`;
  if (!translators.has(key)) {
    translators.set(
      key,
      createBuiltIn("Translator", { sourceLanguage, targetLanguage }).catch((err) => {
        translators.delete(key); // e.g. NotAllowedError: retry from a click handler.
        throw err;
      }),
    );
  }
  const translator = await translators.get(key);
  return translator ? translator.translate(text) : null;
}

Errors you should handle¶

Permissions Policy, iframes and workers¶

All built-in AI APIs are available to top-level windows and same-origin iframes by default. A cross-origin iframe needs delegation through the policy-controlled features language-model, summarizer, writer, rewriter, proofreader, translator and language-detector:

<iframe src="https://widget.example.net/" allow="summarizer; translator; language-detector"></iframe>

They are not exposed in dedicated, shared or service workers. The explainers cite "the complexity of establishing a responsible document for each worker". That means you can't call them from the worker where you run everything else, and you can't use them from a service worker to process push payloads. Keep the calls on the page. Chrome runs the model outside your page's JavaScript, so the main-thread cost is mostly consuming stream chunks and rendering them. Batch that work as shown in Streaming tokens to the UI.

Microsoft Edge: Phi-4-mini and Aion-1.0-Instruct¶

Edge implements the same API surface on its own models. Per Microsoft Learn (updated September 2026):

• The Prompt API and the Writer and Rewriter APIs are a developer preview in Edge Canary and Dev from version 138.0.3309.2, behind flags such as "Prompt API for on-device language model". Microsoft's Writing Assistance APIs page says the Summarizer "has been enabled by default since that version". All of them use Phi-4-mini, which Microsoft's June 2026 blog post describes as a 4B-parameter model.
• Phi-4-mini's requirements: Windows 10 or 11 or macOS 13.3+, at least 20 GB free on the profile volume (deleted below 10 GB), 5.5 GB of VRAM or more, and an unmetered connection. Check edge://on-device-internals: a device performance class of "High" or above is required.
• From Edge 150.0.4070 in Canary and Dev, the pre-release Aion-1.0-Instruct model can replace Phi-4-mini through the "Enable prerelease on-device language model" flag. Microsoft describes it as smaller and faster, supported on less capable GPUs and on CPU-only devices.
• The Translator and Language Detector APIs shipped in Edge 148. The Microsoft Edge Blog says they support 145+ languages.
• The Proofreader API is a developer preview in Canary and Dev from version 142.

The practical difference: the same prompt produces different output on Gemini Nano and Phi-4-mini. Test prompts on both, prefer responseConstraint to parsing free text, and don't hard-code assumptions about tone or length.

Firefox and Safari positions¶

Neither Gecko nor WebKit exposes a built-in model to web content. The standards positions are explicit:

The recurring objection is interoperability. The output depends on whichever model a browser ships, so a prompt tuned for one model behaves differently on another, and sites may end up sniffing browsers. For cross-browser on-device AI today, bring your own model on WebGPU or WebAssembly.

Built-in AI status across browsers (September 2026)¶

Support data as of September 2026. Check MDN's Prompt API page and ChromeStatus for live data.

⚠️ MDN lists the Summarizer in Edge 138, and Microsoft says it has been enabled by default since 138.0.3309.2, but it runs on Phi-4-mini, so it needs Edge's hardware requirements (5.5 GB of VRAM, a "High" device performance class). Feature-detect and call availability() rather than trusting a version number. ChromeStatus has Proposed entries for the Prompt, Summarizer, Translator and Language Detector APIs on Android, with no shipped milestone as of September 2026.

Running your own models: WebGPU, WebNN and WebAssembly¶

When the built-in APIs don't exist (every non-Chromium browser, every phone) or don't fit (embeddings, image segmentation, speech, a specific fine-tuned model), you ship the weights and a runtime. Three compute backends are available.

WebGPU¶

WebGPU exposes compute shaders and storage buffers, which is what matrix multiplication at LLM scale needs. All serious in-browser LLM runtimes target it. Support per MDN's compatibility data:

• Chrome and Edge from 113 on Windows, macOS and ChromeOS. From 144, also on Linux with Intel Gen12+ GPUs.
• Chrome for Android from 121.
• Safari 26 on macOS and iOS/iPadOS.
• Firefox from 141 on Windows, from 145 on macOS Tahoe on Apple silicon, from 147 on older macOS on Apple silicon. Not on Intel Macs, not on Linux and not on Android. Firefox doesn't support it in service workers.
• navigator.gpu is exposed in dedicated workers in all of these, so you can run inference off the main thread.

Two adapter properties matter for models. The shader-f16 feature lets runtimes use half-precision weights and activations, which halves memory for fp16 or q4f16 models, and it's unavailable on some GPUs. The limits maxBufferSize and maxStorageBufferBindingSize cap the largest single tensor. WebLLM's model records include a buffer_size_required_bytes field for this reason.

WebNN¶

WebNN is a graph-level API (navigator.ml.createContext({ deviceType, powerPreference }) then an MLGraphBuilder) that maps onto OS ML stacks such as DirectML, Core ML and TFLite, and can reach NPUs that WebGPU can't use. It's a W3C Candidate Recommendation Draft. Status:

Use WebNN through a library (ONNX Runtime Web's webnn execution provider, or Transformers.js with device: "webnn-npu" / "webnn-gpu") and always list a WebGPU or WASM fallback after it.

WebAssembly with SIMD and threads¶

WebAssembly runs everywhere and is the fallback for devices without a usable GPU. Two features matter:

• SIMD (128-bit) is supported in all current engines and gives large speedups for quantized int8 kernels. Runtimes ship SIMD builds by default.
• Threads need SharedArrayBuffer, which is only available when the page is cross-origin isolated (self.crossOriginIsolated === true). Without isolation, runtimes silently fall back to one thread. ONNX Runtime Web's docs say multi-threading is enabled "only when the browser supports WebAssembly multi-threading and crossOriginIsolated mode is enabled".

WASM memory is also bounded. ONNX Runtime Web's large-model guide notes that "WebAssembly has a memory limit of 4GB" for its 32-bit build, and that Chrome's maximum ArrayBuffer is about 2 GB. Calling response.arrayBuffer() on a larger model file can fail. On CPU, stick to small models: embeddings, classifiers, speech recognition and sub-billion-parameter LLMs.

Cross-origin isolation for threaded WASM¶

Cross-origin isolation needs Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp (or credentialless, which Safari doesn't support) on the document and on worker scripts. In a PWA that has consequences: every cross-origin subresource must opt in with CORP or CORS, OAuth and payment popups lose window.opener, and navigation responses your service worker serves from cache must carry the headers too. The Content Security Policy page covers COOP, COEP, CORP and Chromium's Document-Isolation-Policy in detail, and the OPFS page shows the same setup for SQLite. If isolation is too costly, run WASM single-threaded or rely on WebGPU, which doesn't need it.

Choosing a backend¶

Libraries for in-browser inference¶

Writing WebGPU kernels for transformer models yourself isn't practical. Four library families cover almost all real deployments. Versions below are the latest on npm in September 2026.

Transformers.js: a model worker with progress and streaming¶

Transformers.js runs ONNX models with the same pipeline() API as the Python library. Pass device: "webgpu" to use the GPU (the default in browsers is WASM) and dtype to pick a quantization: "fp32" (default for WebGPU), "fp16", "q8" (default for WASM), "q4", "q4f16" and others. By default it caches downloaded files in Cache Storage under the key transformers-cache (env.useBrowserCache, env.cacheKey), or in any object with match() and put() that you assign to env.customCache.

The worker below loads a small chat model, reports aggregate download progress, warms up, streams tokens, and supports interruption. The page never blocks while the model loads or generates.

import {
  pipeline,
  TextStreamer,
  InterruptableStoppingCriteria,
  env,
} from "@huggingface/transformers";

const MODEL_ID = "onnx-community/Qwen2.5-0.5B-Instruct";

// Serve the ONNX Runtime .wasm files from your own origin so they're cached
// with the app and work offline (the default is a CDN).
env.backends.onnx.wasm.wasmPaths = "/vendor/ort/";

const stopping = new InterruptableStoppingCriteria();
let generatorPromise = null;

async function pickDevice() {
  if (!("gpu" in navigator)) return { device: "wasm", dtype: "q4" };
  const adapter = await navigator.gpu.requestAdapter();
  if (!adapter) return { device: "wasm", dtype: "q4" };
  // q4f16 needs half-precision shaders. Fall back to q4 (fp32 compute) otherwise.
  return {
    device: "webgpu",
    dtype: adapter.features.has("shader-f16") ? "q4f16" : "q4",
  };
}

function load() {
  generatorPromise ??= (async () => {
    const { device, dtype } = await pickDevice();
    const files = new Map(); // file → { loaded, total }

    const generator = await pipeline("text-generation", MODEL_ID, {
      device,
      dtype,
      progress_callback(info) {
        // Per-file events: initiate → download → progress* → done, then "ready".
        if (info.status === "progress") {
          files.set(info.file, { loaded: info.loaded, total: info.total });
          let loaded = 0;
          let total = 0;
          for (const f of files.values()) {
            loaded += f.loaded;
            total += f.total;
          }
          self.postMessage({ type: "progress", loaded, total });
        }
      },
    });

    // Warm-up: compiles WebGPU pipelines and allocates buffers now, not on the
    // user's first question. A single token is enough.
    await generator([{ role: "user", content: "Hi" }], { max_new_tokens: 1 });

    self.postMessage({ type: "ready", device, dtype });
    return generator;
  })().catch((err) => {
    generatorPromise = null; // Allow retry, e.g. after freeing storage.
    throw err;
  });
  return generatorPromise;
}

let runningId = null;
const cancelled = new Set(); // Jobs aborted while still queued.

async function generate({ id, messages, maxNewTokens = 512 }) {
  const generator = await load();
  if (cancelled.delete(id)) {
    self.postMessage({ type: "done", id, text: "" }); // Aborted before it started.
    return;
  }
  stopping.reset();
  runningId = id;

  const streamer = new TextStreamer(generator.tokenizer, {
    skip_prompt: true,
    skip_special_tokens: true,
    callback_function(text) {
      self.postMessage({ type: "token", id, text });
    },
  });

  let output;
  try {
    output = await generator(messages, {
      max_new_tokens: maxNewTokens,
      do_sample: false, // Deterministic output is easier to test and to cache.
      streamer,
      stopping_criteria: stopping,
    });
  } finally {
    runningId = null;
  }

  // For chat input, generated_text is the message list including the new reply.
  const reply = output[0].generated_text.at(-1).content;
  self.postMessage({ type: "done", id, text: reply });
}

// One pipeline, one generation at a time: queue requests instead of running them
// concurrently on the same model (and the same stopping criteria).
let queue = Promise.resolve();

self.addEventListener("message", async ({ data }) => {
  try {
    switch (data.type) {
      case "load":
        await load();
        break;
      case "generate":
        queue = queue.catch(() => {}).then(() => generate(data));
        await queue;
        break;
      case "interrupt":
        // The running job ends at the next token boundary; a queued one is skipped.
        if (data.id === runningId) stopping.interrupt();
        else cancelled.add(data.id);
        break;
    }
  } catch (err) {
    self.postMessage({ type: "error", id: data.id, name: err.name, message: err.message });
  }
});

The page side wraps the worker in a promise-and-callback API:

const worker = new Worker(new URL("./llm-worker.js", import.meta.url), { type: "module" });
const pending = new Map(); // id → { onToken, resolve, reject }
let nextId = 1;

worker.addEventListener("message", ({ data }) => {
  if (data.type === "progress") {
    document.dispatchEvent(new CustomEvent("model-progress", { detail: data }));
    return;
  }
  if (data.type === "ready") {
    document.dispatchEvent(new CustomEvent("model-ready", { detail: data }));
    return;
  }
  const job = pending.get(data.id);
  if (!job) return;
  if (data.type === "token") job.onToken(data.text);
  if (data.type === "done") {
    pending.delete(data.id);
    job.resolve(data.text);
  }
  if (data.type === "error") {
    pending.delete(data.id);
    job.reject(Object.assign(new Error(data.message), { name: data.name }));
  }
});

worker.addEventListener("error", (event) => {
  // Script failed to load or threw at top level. Fail every pending job.
  for (const job of pending.values()) job.reject(new Error(event.message));
  pending.clear();
});

export function preloadModel() {
  worker.postMessage({ type: "load" });
}

export function generate(messages, { onToken = () => {}, signal } = {}) {
  if (signal?.aborted) return Promise.reject(signal.reason);
  const id = nextId++;
  const onAbort = () => worker.postMessage({ type: "interrupt", id });
  signal?.addEventListener("abort", onAbort, { once: true });
  return new Promise((resolve, reject) => {
    pending.set(id, { onToken, resolve, reject });
    worker.postMessage({ type: "generate", id, messages });
  }).finally(() => {
    // Don't let a later abort of a reused signal interrupt another job.
    signal?.removeEventListener("abort", onAbort);
  });
}

ONNX Runtime Web: direct sessions and external data¶

ONNX Runtime Web is the engine under Transformers.js. Use it directly for non-transformer models or when you need tensor-level control. Import paths select bundles: onnxruntime-web (WASM), onnxruntime-web/webgpu (WebGPU), onnxruntime-web/all (adds WebNN). Execution providers are tried in order.

import * as ort from "onnxruntime-web/webgpu";

// Threads only work when cross-origin isolated. 1 disables threading explicitly.
ort.env.wasm.numThreads = self.crossOriginIsolated
  ? Math.min(4, navigator.hardwareConcurrency || 1)
  : 1;
ort.env.wasm.wasmPaths = "/vendor/ort/"; // Self-host for offline use.

let sessionPromise = null;

export function getSession(modelBytes, externalDataBytes) {
  sessionPromise ??= ort.InferenceSession.create(modelBytes, {
    executionProviders: ["webgpu", "wasm"], // First one that initializes wins.
    graphOptimizationLevel: "all",
    // Models > 2 GB store weights outside the protobuf. `path` must match the
    // location recorded in the .onnx file. `data` may be a URL or bytes.
    externalData: externalDataBytes
      ? [{ path: "model.onnx_data", data: externalDataBytes }]
      : undefined,
  });
  return sessionPromise;
}

export async function embed(session, inputIds, attentionMask) {
  const dims = [1, inputIds.length];
  const feeds = {
    input_ids: new ort.Tensor("int64", BigInt64Array.from(inputIds, BigInt), dims),
    attention_mask: new ort.Tensor("int64", BigInt64Array.from(attentionMask, BigInt), dims),
  };
  const results = await session.run(feeds);
  const output = results[session.outputNames[0]];
  const data = await output.getData(); // Downloads from GPU if needed.
  output.dispose?.(); // Release GPU buffers promptly on WebGPU.
  return data;
}

env.wasm.proxy = true moves WASM inference into a library-managed worker, but it doesn't work with the WebGPU EP (GPU buffers aren't transferable) or under a CSP that blocks blob workers. Creating your own module worker, as above, is more predictable. See Content Security Policy for worker-src and wasm-unsafe-eval.

WebLLM: OpenAI-style chat on WebGPU¶

WebLLM runs MLC-compiled LLMs on WebGPU with an OpenAI-compatible engine.chat.completions.create() API. It ships worker and service-worker handlers. Model records in prebuiltAppConfig.model_list carry vram_required_MB, low_resource_required and required_features (such as shader-f16), which you can check before offering a model. For example, Llama-3.2-1B-Instruct-q4f16_1-MLC lists 879 MB of VRAM and a 4,096-token context window in version 0.2.85.

import { WebWorkerMLCEngineHandler } from "@mlc-ai/web-llm";

const handler = new WebWorkerMLCEngineHandler();
self.onmessage = (msg) => handler.onmessage(msg);

import { CreateWebWorkerMLCEngine, prebuiltAppConfig } from "@mlc-ai/web-llm";

const MODEL = "Llama-3.2-1B-Instruct-q4f16_1-MLC";

export async function createEngine(onProgress) {
  const appConfig = {
    ...prebuiltAppConfig,
    cacheBackend: "opfs", // "cache" (default), "indexeddb", "opfs" or "cross-origin"
  };
  return CreateWebWorkerMLCEngine(
    new Worker(new URL("./webllm-worker.js", import.meta.url), { type: "module" }),
    MODEL,
    {
      appConfig,
      // report: { progress: 0..1, timeElapsed: seconds, text: human-readable status }
      initProgressCallback: (report) => onProgress(report.progress, report.text),
    },
  );
}

export async function chat(engine, messages, onDelta) {
  const chunks = await engine.chat.completions.create({
    messages,
    stream: true,
    stream_options: { include_usage: true },
    temperature: 0.3,
  });
  let reply = "";
  for await (const chunk of chunks) {
    const delta = chunk.choices[0]?.delta.content ?? "";
    reply += delta;
    onDelta(delta);
  }
  return reply;
}

WebLLM's cacheBackend accepts "cache" (Cache API, the default), "indexeddb", "opfs" (with opfsAccessMode "async", "sync" or "auto") and "cross-origin". The last one is an experimental backend for the proposed Cross-Origin Storage API, which today requires a browser extension. hasModelInCache() and deleteModelAllInfoInCache() let you build a "Manage downloaded models" screen. WebLLM also offers ServiceWorkerMLCEngineHandler so one engine can outlive page navigations. The README warns that the browser can kill the service worker at any time, so treat that mode as an optimization with a restart path.

MediaPipe LLM Inference and LiteRT-LM¶

MediaPipe's LlmInference task (@mediapipe/tasks-genai) runs Gemma-family models on WebGPU. You load it with FilesetResolver.forGenAiTasks(wasmBaseUrl), then LlmInference.createFromOptions(fileset, { baseOptions: { modelAssetPath }, maxTokens, topK, temperature, randomSeed }), then generateResponse(prompt, (partialResult, done) => ...). Only one generateResponse() can run at a time, and cancelProcessing() stops it. For PWAs, baseOptions.modelAssetBuffer accepts a ReadableStreamDefaultReader, so you can stream a model straight from OPFS without holding it in one ArrayBuffer:

import { FilesetResolver, LlmInference } from "@mediapipe/tasks-genai";

export async function loadGemmaFromOpfs(fileName) {
  const root = await navigator.storage.getDirectory();
  const dir = await root.getDirectoryHandle("models");
  const file = await (await dir.getFileHandle(fileName)).getFile();

  const fileset = await FilesetResolver.forGenAiTasks("/vendor/mediapipe-genai/wasm");
  return LlmInference.createFromOptions(fileset, {
    baseOptions: { modelAssetBuffer: file.stream().getReader() }, // No 2 GB ArrayBuffer.
    maxTokens: 1024, // Input + output tokens combined.
    topK: 40,
    temperature: 0.7,
    randomSeed: 1,
  });
}

Downloading and storing multi-gigabyte models¶

Everything a PWA knows about caching assumes files in the kilobyte-to-megabyte range. Model weights are three orders of magnitude bigger, and the usual tools break in specific ways.

Cache Storage vs OPFS vs IndexedDB¶

All three count toward the same origin quota and are evicted together. None is safer than the others from eviction. Chrome's caching guide also recommends serving weights with Cache-Control: public, max-age=31536000, immutable at a versioned URL, so even the HTTP cache doesn't revalidate them.

A good default for multi-gigabyte files: download into OPFS in chunks, verify, then read with File.stream() or File.slice(). Let libraries that manage their own Cache Storage keep doing so for files under a few hundred megabytes.

Resumable downloads with Range requests into OPFS¶

Hugging Face and most object stores answer Range: bytes=N- with 206 Partial Content and a Content-Range header. That lets you resume from exactly where a broken download stopped. This worker writes with a synchronous access handle (the fastest OPFS path), flushes periodically so a crash loses little data, handles servers that ignore Range, and verifies a SHA-256 before publishing the file under its final name.

import { createSHA256 } from "hash-wasm"; // Incremental SHA-256. Web Crypto can't hash in chunks.

const FLUSH_EVERY = 32 * 1024 * 1024; // Flush to disk every 32 MiB.

/**
 * @param {object} file
 * @param {string} file.url     Immutable, versioned URL.
 * @param {string} file.name    Final file name in OPFS, e.g. "qwen-0.5b-q4f16.onnx".
 * @param {number} file.size    Expected size in bytes (from your model manifest).
 * @param {string} file.sha256  Expected lowercase hex SHA-256.
 */
async function download(file, signal) {
  const root = await navigator.storage.getDirectory();
  const dir = await root.getDirectoryHandle("models", { create: true });

  // Already complete? The final name only exists after verification.
  try {
    const done = await dir.getFileHandle(file.name);
    if ((await done.getFile()).size === file.size) return;
    await dir.removeEntry(file.name); // Stale file from another version: replace it.
  } catch (err) {
    if (err.name !== "NotFoundError") throw err; // NotFoundError: not downloaded yet.
  }

  const partHandle = await dir.getFileHandle(`${file.name}.part`, { create: true });
  const access = await partHandle.createSyncAccessHandle(); // Exclusive lock on the file.
  try {
    let offset = access.getSize();
    if (offset > file.size) {
      access.truncate(0);
      offset = 0;
    }

    if (offset < file.size) {
      const response = await fetch(file.url, {
        headers: offset > 0 ? { Range: `bytes=${offset}-` } : {},
        signal,
        cache: "no-store", // Don't also copy gigabytes into the HTTP cache.
      });

      if (response.status === 200 && offset > 0) {
        access.truncate(0); // Server ignored Range: start over.
        offset = 0;
      } else if (response.status === 206) {
        const start = Number(/bytes (\d+)-/.exec(response.headers.get("Content-Range"))?.[1]);
        if (start !== offset) throw new Error(`Unexpected Content-Range start ${start}`);
      } else if (!response.ok) {
        throw new Error(`HTTP ${response.status} for ${file.url}`);
      }

      const reader = response.body.getReader();
      let sinceFlush = 0;
      for (;;) {
        const { done, value } = await reader.read();
        if (done) break;
        access.write(value, { at: offset });
        offset += value.byteLength;
        sinceFlush += value.byteLength;
        if (sinceFlush >= FLUSH_EVERY) {
          access.flush();
          sinceFlush = 0;
        }
        self.postMessage({ type: "progress", name: file.name, loaded: offset, total: file.size });
      }
      access.flush();
    }

    if (access.getSize() !== file.size) {
      throw new Error(`Size mismatch for ${file.name}: ${access.getSize()} != ${file.size}`);
    }

    // Verify in 16 MiB slices so memory stays flat even for multi-GB files.
    const hasher = await createSHA256();
    hasher.init();
    const buf = new Uint8Array(16 * 1024 * 1024);
    for (let pos = 0; pos < file.size; ) {
      const n = access.read(buf, { at: pos });
      hasher.update(buf.subarray(0, n));
      pos += n;
    }
    const digest = hasher.digest("hex");
    if (digest !== file.sha256) {
      access.truncate(0); // Corrupt or tampered: never keep it.
      throw new Error(`SHA-256 mismatch for ${file.name}`);
    }
  } finally {
    access.close(); // Release the lock even on failure.
  }

  await partHandle.move(file.name); // Publish atomically under the final name.
}

let controller = null;

self.addEventListener("message", async ({ data }) => {
  if (data.type === "cancel") {
    controller?.abort();
    return;
  }
  if (data.type !== "download") return;
  controller = new AbortController();
  try {
    for (const file of data.files) await download(file, controller.signal);
    self.postMessage({ type: "complete" });
  } catch (err) {
    // AbortError, QuotaExceededError, TypeError (network) ... keep the .part file for resume.
    self.postMessage({ type: "error", name: err.name, message: err.message });
  }
});

A few details to get right:

• Cross-origin hosts. A model on a CDN needs CORS. A Range header with a single simple range may still trigger a preflight in some browsers, so the host must allow it. Hugging Face's file endpoints send CORS headers and Accept-Ranges: bytes, and answer ranged requests with 206. Test your own host.
• Where the hash comes from. For files stored with Git LFS, the Hugging Face tree API returns an lfs.oid that is the file's SHA-256. Record it in your own manifest at release time. Don't fetch it at runtime from the same place you fetch the weights.
• Quota errors. A write that exceeds quota throws QuotaExceededError from access.write(). Check navigator.storage.estimate() first, but Chromium intentionally reports a fake quota, so the storage quotas page explains why "usage vs. quota" isn't real headroom.
• Tab contention. createSyncAccessHandle() takes an exclusive lock. A second tab starting the same download gets NoModificationAllowedError. Coordinate with a Web Lock (navigator.locks.request("model-download", ...)) so only one tab downloads.

Background Fetch for very large downloads¶

Background Fetch hands a download to the browser, which shows a system notification with progress, continues after the user closes the tab, and wakes your service worker on backgroundfetchsuccess. For a multi-gigabyte model on Android Chrome that's the most robust option. It's Chromium-only. Chromium now rejects backgroundFetch.fetch() calls made from a service worker, so start it from a page in response to a click. In the success handler, copy each BackgroundFetchRecord's response into Cache Storage or stream it into OPFS, then verify the hash as above. Browsers without Background Fetch fall back to the worker download.

sequenceDiagram
    participant U as User
    participant P as Page
    participant SW as Service worker
    participant B as Browser download manager
    U->>P: Click "Download model (1.2 GB)"
    P->>P: navigator.storage.persist()
    alt Background Fetch supported
        P->>B: registration.backgroundFetch.fetch(id, urls, options)
        B-->>P: progress events (while page open)
        B->>SW: backgroundfetchsuccess
        SW->>SW: Copy records to Cache Storage or OPFS, verify SHA-256
        SW-->>P: postMessage("model-ready")
    else Not supported
        P->>P: Worker download with Range resume into OPFS
    end

Quota, persist() and eviction¶

Model files are the largest thing your origin stores, and eviction is least-recently-used by origin. A best-effort origin holding 2 GB is an attractive target when the disk fills up. Before a large download:

1. Call navigator.storage.persist(). Chromium grants it silently based on engagement (installed PWAs usually qualify). Firefox prompts the user. Safari decides on its own heuristics. Persistence protects against pressure-based eviction, not against the user clearing site data.
2. Check free space with estimate() as a rough guide only, and handle QuotaExceededError anyway.
3. Tell the user the size up front and let them opt in. Respect navigator.connection?.saveData (Chromium only) and never start a multi-gigabyte download on a metered connection without asking.

On iOS and iPadOS, WebKit's quotas are generous since iOS 17 (about 60% of disk for Safari and Home Screen web apps), but Safari deletes all script-writable storage for sites that haven't been used for seven days of browser use, unless the user installed the site to the Home Screen. A model downloaded in a Safari tab can vanish within a week. Encourage installation before offering a large model on iOS, and read Storage Quotas & Eviction and iOS & iPadOS for the exact rules. Always detect a missing model at startup and re-offer the download. No event tells you that eviction happened.

Don't precache models in the service worker¶

A service worker's install step should cache the app shell: a few megabytes that must be consistent with each other. Putting a model there is a mistake:

• install fails as a whole if any file fails, so a flaky connection on a 1 GB file blocks every app update.
• The browser may terminate a long-running install, and there's no progress UI.
• Each new service worker version re-downloads the precache manifest's changed entries. A model with a new hash in the manifest means gigabytes on every deploy.
• Workbox's maximumFileSizeToCacheInBytes (default 2 MiB) already excludes large files from precaching. Don't raise it to include weights, and exclude model directories from globPatterns.

Download models on demand, after the user opts in, from a page or worker. Make sure your fetch handler doesn't interfere: either don't call respondWith() for model URLs, or bypass the worker entirely with static routing. A generic "cache everything" runtime route that clones gigabyte responses into another cache doubles disk use and memory pressure. Handling Fetch Events covers selective interception, and Precaching covers what does belong in the install step.

self.addEventListener("fetch", (event) => {
  const url = new URL(event.request.url);
  // Model weights: let the network stack handle them (Range requests, no cloning).
  if (url.pathname.startsWith("/models/") || url.hostname === "huggingface.co") return;
  // ...app shell and API strategies here...
});

Versioning and updating models¶

Treat models like any other release artifact, but never update them silently:

• Immutable, versioned URLs. /models/summarizer/2026-09-01/model_q4f16.onnx, never /models/latest.onnx.
• A small manifest fetched with your app (and precached with the shell) lists each model's files, sizes and SHA-256 hashes. The app compares it with what's in OPFS.
• Download the new version alongside the old one, verify it, switch a pointer (an IndexedDB record or a small JSON file in OPFS), then delete the old files. The app keeps working with the old model the whole time, as with Chrome's own hot-swap.
• Ask before re-downloading large models, and keep using the old model if the user declines. The update flows in Service Worker Updates apply to the prompt UX.
• Record the model version with every stored output (embeddings in particular). Embeddings from two model versions aren't comparable, so re-embed after switching.

{
  "summarizer": {
    "version": "2026-09-01",
    "files": [
      {
        "url": "/models/summarizer/2026-09-01/model_q4f16.onnx",
        "name": "summarizer-2026-09-01.onnx",
        "size": 483003582,
        "sha256": "b11c1dd99efd57e6c6e5bc4443a019931a5fbd5dd500d48644d8225f5ce0b2cb"
      }
    ]
  }
}

Running inference without hurting INP¶

Inference is the heaviest work a web page will ever do. A 0.5B-parameter model can peg a GPU for seconds, and a WASM fallback can block a CPU core for longer. Interaction to Next Paint suffers when any of that runs on the main thread. See Runtime Performance for the underlying task model.

Workers, messaging and transfer¶

• Run your own models in a dedicated module worker, as all examples above do. WebGPU, WASM threads, OPFS sync handles and fetch() are all available there.
• Load the model once per app, not per tab if you can avoid it. Each tab running its own copy of a 1 GB model means 1 GB of GPU memory per tab. A SharedWorker can host one engine for all tabs where it's supported (MDN lists Chrome for Android only from 148, and Safari from 16). Otherwise elect a leader tab with Web Locks and relay requests over BroadcastChannel.
• Send large inputs (images, audio) as transferables (postMessage(msg, [buffer])) so the structured clone doesn't copy megabytes on the main thread.
• The built-in APIs can't run in workers, but their inference happens outside your JavaScript. Keep your per-chunk work small.

Streaming tokens to the UI without jank¶

Tokens arrive every few milliseconds. Updating the DOM for each one, and re-parsing Markdown each time, creates a continuous stream of small tasks and layout work that competes with input. Coalesce updates into one per frame:

/**
 * Batches streamed text into at most one DOM update per animation frame.
 * Renders plain text. Markdown rendering belongs in a sanitizing step (see below).
 */
export function createTokenRenderer(el) {
  let pending = "";
  let scheduled = false;
  const textNode = document.createTextNode("");
  el.replaceChildren(textNode);

  function flush() {
    scheduled = false;
    if (!pending) return;
    textNode.appendData(pending); // Appends without re-serializing existing text.
    pending = "";
    // Keep the newest text visible without forcing layout on every token.
    el.scrollTop = el.scrollHeight;
  }

  return {
    push(delta) {
      pending += delta;
      if (!scheduled) {
        scheduled = true;
        requestAnimationFrame(flush);
      }
    },
    end() {
      flush();
    },
  };
}

Wire it up with generate(messages, { onToken: renderer.push }) for the worker client, or for await (const chunk of stream) renderer.push(chunk) for built-in APIs. Mark the output region with aria-live="polite" and aria-busy="true" while streaming, and set aria-busy="false" at the end, so screen readers announce the finished answer instead of every token. See Accessibility.

Warm-up and first-token latency¶

The first inference after loading is much slower than the rest. WebGPU compiles shader pipelines and allocates buffers, and WASM compiles and instantiates modules. Hide that cost:

• Preload after idle. Once the page is interactive and the user is likely to use the feature, send { type: "load" } to the worker inside requestIdleCallback or after scheduler.yield(). Never block initial render on it.
• Warm up with a one-token generation, as the Transformers.js worker does.
• Keep a session alive for built-in APIs. Chrome unloads the model when no sessions exist, so create one on feature open and reuse it.
• Measure time to first token and tokens per second on your slowest supported device, and record them per device/dtype in your analytics (see Analytics) to decide which devices get which model.

Memory limits on mobile¶

A quantized model needs roughly its file size in GPU memory, plus the KV cache (which grows with context length), plus activations. Phones and integrated GPUs share that memory with the OS and every other tab. When a mobile browser runs out, it doesn't throw a helpful error: the tab is killed and reloads. On iOS the limits for web content processes aren't documented. Practical rules:

• Offer LLMs only above a device threshold, and prefer sub-1B models on phones. WebLLM's low_resource_required flag marks models its authors consider suitable for limited devices.
• Keep the context short: cap history, and use max_new_tokens / maxTokens.
• Free GPU memory when the feature closes. Call dispose() on Transformers.js models, engine.unload() in WebLLM and session.release() in ONNX Runtime Web, or terminate the worker, which frees everything.
• Terminate the worker on pagehide or when the document has been hidden for a while. The browser may discard hidden tabs holding gigabytes anyway.

A capability probe to decide what to offer:

export async function probeDevice() {
  const result = {
    webgpu: false,
    shaderF16: false,
    maxBufferSize: 0,
    maxStorageBufferBindingSize: 0,
    // Chromium only. Coarse, bucketed value for privacy (from Chrome 147: 1, 2, 4 or 8 on
    // Android; 2, 4, 8, 16 or 32 elsewhere).
    deviceMemoryGB: navigator.deviceMemory ?? null,
    cores: navigator.hardwareConcurrency ?? null,
    crossOriginIsolated: self.crossOriginIsolated === true,
    saveData: navigator.connection?.saveData === true,
    builtIn: {
      prompt: "LanguageModel" in self,
      summarizer: "Summarizer" in self,
      translator: "Translator" in self,
    },
  };

  if ("gpu" in navigator) {
    try {
      const adapter = await navigator.gpu.requestAdapter({ powerPreference: "high-performance" });
      if (adapter) {
        result.webgpu = true;
        result.shaderF16 = adapter.features.has("shader-f16");
        result.maxBufferSize = adapter.limits.maxBufferSize;
        result.maxStorageBufferBindingSize = adapter.limits.maxStorageBufferBindingSize;
        result.gpuVendor = adapter.info?.vendor ?? "";
      }
    } catch {
      /* Blocklisted GPU or driver crash: treat as no WebGPU. */
    }
  }
  return result;
}

export function chooseLocalModelTier(p) {
  if (p.webgpu && p.shaderF16 && (p.deviceMemoryGB ?? 8) >= 8) return "llm-1b";
  if (p.webgpu) return "llm-0.5b";
  if (p.crossOriginIsolated && (p.cores ?? 1) >= 4) return "embeddings-only";
  return "none";
}