Offline-First Data & Sync¶

An offline-first PWA reads and writes application data against a local database on the device and treats the network as a background channel that synchronizes that database with a server when it can. The user never waits for a round trip to see or change data, the app keeps working with no connection, and a sync engine later reconciles local changes with everyone else's. This page covers the whole data layer that makes that possible: where the source of truth lives, how mutations are queued and replayed, how conflicts are resolved (last-write-wins, version vectors, operational transformation and CRDTs), how delta sync with cursors and tombstones works, how to migrate client schemas, how to coordinate several tabs, which sync engines exist in 2026, how to protect data at rest, and a complete end-to-end implementation.

Offline-first, local-first and cache-first are different things¶

The three terms are often used interchangeably, but they describe different architectures with different failure modes.

The term local-first was popularized by Ink & Switch's 2019 essay Local-first software: you own your data, in spite of the cloud, which lists seven ideals: no spinners (work is fast because it is local), work is not trapped on one device, the network is optional, seamless collaboration, "the long now" (data outlives the vendor), security and privacy by default, and ultimate user ownership and control. Most PWAs sit in the middle column: the server stays authoritative for business rules, while the client holds enough data and enough logic to operate without it.

The key shift from online-first is that HTTP responses are no longer the data model. A cached GET /api/tasks response cannot express "the user renamed task 7 while offline, and that change has not been accepted yet". A local database plus a queue of mutations can. The Cache Storage API remains the right tool for the app shell and static assets; application data belongs in IndexedDB or, for SQL workloads, SQLite on the Origin Private File System.

Deciding where the source of truth lives¶

Before writing any sync code, decide, per kind of data, who wins. The answer changes everything downstream: conflict handling, validation, security and how much logic runs on the client.

A single app usually mixes models. A project-management PWA might use server-authoritative mutations for task assignment (only members can be assigned), a Yjs document for the task description (collaborative text), per-field last-write-wins for the user's column widths, and server-only for billing.

Also decide what subset each client replicates. "Sync the whole database" works for a personal notes app with a few megabytes of data; it does not work for a team workspace with millions of rows or for data the user is not allowed to see. Partial replication is covered in Partial replication and authorization.

Anatomy of a sync engine¶

Every offline-first data layer, whether you build it or adopt a library, has the same moving parts.

flowchart LR
    UI["UI components"] -->|"queries"| View["Local view (IndexedDB / SQLite)"]
    UI -->|"mutate(name, args)"| Mut["Mutator"]
    Mut -->|"same transaction"| View
    Mut -->|"same transaction"| Outbox["Outbox (pending mutations)"]
    Outbox -->|"push"| Server["Server: re-run mutators, validate"]
    Server -->|"pull: patch + cursor + lastMutationID"| Base["Confirmed base state"]
    Base -->|"rebase: base + replay pending"| View
    View -->|"change events"| UI

The components and their responsibilities:

Local view

The database the UI queries. It reflects confirmed server state plus the optimistic effect of every pending mutation. Queries never touch the network.

Confirmed base

The last state the server acknowledged, up to a cursor. Some engines store it separately (Replicache keeps the last server snapshot and replays pending mutations over it); others store only the view plus enough information to undo optimistic changes.

Mutators

Named, deterministic functions such as createTask({id, title}) that transform state. The same mutator runs twice: speculatively on the client and authoritatively on the server. Sending intent ("toggle task 7") rather than state ("task 7 is {done: true, title: …}") is what lets the server merge concurrent changes correctly.

Outbox

An ordered, durable queue of mutations not yet acknowledged, each with a client ID and a monotonically increasing mutation ID. It survives reloads, crashes and eviction-free restarts.

Push loop

Sends batches of outbox entries to the server, in order, and retries with backoff. The server records the highest mutation ID processed per client, which makes replays harmless.

Pull loop

Fetches changes since the client's cursor, applies them to the base, drops acknowledged mutations and rebases the view. Triggered by a timer, a server "poke" (SSE, WebSocket, push message) or after a push.

Coordinator

Ensures one push/pull loop per origin across tabs, and broadcasts "data changed" to the others.

Migrator

Upgrades the local schema and reinterprets old outbox entries after an app update.

Reads: query the local store, never the network¶

The rule that makes an app feel instant is that no UI read awaits a fetch. Components subscribe to local queries; when sync changes the local store, the queries re-run and the UI re-renders. Loading states only exist for data the client has never synced, such as the first open of a project, and even then the UI shows a skeleton from local metadata instead of a spinner over nothing. Libraries expose this as reactive queries (db.watch() in PowerSync, observable queries in RxDB, subscribe() in Replicache, useQuery() in Zero); in a hand-written engine you re-run queries when a BroadcastChannel message or an in-tab event says a store changed.

Writes: mutations, not state¶

A write is a call like mutate("toggleTask", { id }). The mutator runs inside one IndexedDB readwrite transaction that both updates the view and appends to the outbox, so the two can never disagree after a crash. Because the outbox stores {name, args} rather than a row snapshot, the server can re-run the logic against its newer state: if another device renamed the task in the meantime, toggling done does not revert the rename.

The sync loop: push, then pull¶

A robust loop always pushes before it pulls. Pushing first means the pull that follows reflects the server's processing of those mutations, so the rebase drops them from the outbox at the same moment their authoritative effect arrives. The reverse order causes a visible flicker: the pull delivers state without the pending change, the rebase re-applies it, and the next pull confirms it.

sequenceDiagram
    participant Tab as Leader tab
    participant IDB as IndexedDB
    participant API as Sync API
    Tab->>IDB: read outbox (mutationID > lastAcked)
    Tab->>API: POST /sync/push {clientID, mutations[]}
    API-->>Tab: 200 (processed up to mutationID 42)
    Tab->>API: GET /sync/pull?cursor=1180
    API-->>Tab: {cursor: 1193, lastMutationID: 42, patch[]}
    Tab->>IDB: tx: apply patch to base, delete outbox <= 42, rebuild view
    Tab-->>Tab: BroadcastChannel "changed"

The outbox pattern for offline writes¶

The outbox (sometimes called a mutation queue or pending-operations log) is the heart of offline writes. The site already has two complete implementations: a raw IndexedDB outbox in IndexedDB and a Background Sync-driven one in Background Sync, which also covers idempotency keys in depth. This section focuses on the design decisions that matter once the outbox feeds a sync engine rather than replaying raw HTTP requests.

What an outbox entry contains¶

const outboxEntry = {
  clientID: "c_7f3b…",          // stable per browser profile + database, generated once
  mutationID: 43,               // strictly increasing per clientID, never reused
  name: "renameTask",           // mutator name the server knows
  args: { id: "0193…", title: "Ship v2" },
  schema: 3,                    // version of the mutator's argument format
  createdAt: 1790000000000,     // for diagnostics and TTL, not for ordering
  attempts: 0,                  // incremented on retryable failures
};

Why each field matters:

• clientID + mutationID form the idempotency key. The server stores last_mutation_id per client. A mutation with an ID at or below it has already been applied and is skipped; an ID more than one above it indicates a gap (a lost mutation) and the server refuses the batch so the client resends from the gap. This is the scheme Replicache uses and it removes the need for a separate idempotency table.
• clientID is per database, not per user. Two tabs sharing one database share a client ID, which is why only one of them may push at a time (see multi-tab coordination). If the database is deleted or evicted, a new client ID is generated, so the server never sees a counter reset.
• schema lets the server accept mutations queued by an older app version after a deploy (see client schema migrations).
• Never order by createdAt. Device clocks jump. Order is the mutation ID.

Ordering, dependencies and batching¶

Mutations from one client must be processed in order: createTask before renameTask for the same ID. Processing them strictly in sequence per client guarantees that. Batching (for example up to 100 mutations per request) cuts round trips without changing semantics, provided the server processes the batch sequentially and reports how far it got.

Across clients there is no order to preserve; the server serializes them as they arrive, which is exactly why mutators must be re-executed against current state rather than replayed as snapshots.

Classifying failures¶

A push can fail in three ways, and each needs a different response:

The second row is the subtle one. If a server rejects a mutation by returning an error without marking it processed, the client retries it forever and every later mutation is stuck behind it, the classic poison-message problem. Treat a rejected mutation as processed-with-no-effect.

When to flush the outbox¶

• Immediately after a local mutation (debounced by a few hundred milliseconds so a burst of keystrokes becomes one push).
• On the online event, and on visibilitychange to visible. navigator.onLine === true only means "there is a network interface", so treat it as a hint and let the request decide.
• On a timer with backoff while the outbox is non-empty.
• On a sync event in Chromium-based browsers, where Background Sync can run the push after the tab closes. Firefox and Safari do not implement it, so it can only ever be an addition.
• When the server pokes the client (SSE, WebSocket, or a push message), which triggers a pull.

Optimistic updates and rebasing¶

An optimistic update shows the result of a mutation before the server confirms it. Doing that naïvely, by editing the local row and hoping, fails as soon as the server's result differs from the client's guess or another device's change arrives first. The robust technique is the rebase:

1. Keep the confirmed server state (the base) as of cursor C.
2. Keep the list of pending mutations P1 … Pn.
3. The visible view is replay(base, [P1 … Pn]).
4. When a pull arrives with changes up to cursor C' and the information that the server processed up to Pk: apply the patch to the base, drop P1 … Pk, and recompute the view as replay(base', [Pk+1 … Pn]).

Because pending mutations are re-executed against the new base, the view always equals "what the server will probably end up with". If the server rejected P2, its effect simply disappears in step 4, with no special rollback code. This is exactly what Replicache's documentation calls rewinding to the last server state, applying the patch and replaying pending mutations, and it is why mutators must be deterministic and re-runnable: no Date.now(), Math.random() or ID generation inside the mutator; pass those in as arguments.

/**
 * Rebuild the optimistic view inside one IndexedDB transaction.
 * `db` is an `idb` (promise-wrapped IndexedDB) connection; `mutators` is the same
 * registry the server uses. Full rebuild is O(rows); see the note below for
 * incremental rebasing.
 */
export async function rebase(db, mutators) {
  const tx = db.transaction(["base", "view", "outbox"], "readwrite");
  const base = tx.objectStore("base");
  const view = tx.objectStore("view");

  await view.clear();
  for (const row of await base.getAll()) await view.put(row);

  const pending = await tx.objectStore("outbox").getAll(); // ordered by mutationID key
  const writer = {
    get: (id) => view.get(id),
    put: (row) => view.put(row),
    del: (id) => view.delete(id),
  };
  for (const m of pending) {
    const fn = mutators[m.name];
    // A mutator that throws on the client is skipped for the view, but kept in
    // the outbox: the server is the one that decides whether it is valid.
    try { await fn(writer, m.args); } catch (err) { console.warn("replay failed", m, err); }
  }
  await tx.done;
}

A full rebuild is fine for thousands of rows. For larger datasets, rebase incrementally: record which keys each pending mutation touched (a write set), and on pull only restore those keys plus the patched keys from the base before replaying. Replicache and Zero do this internally with copy-on-write B-trees; RxDB keeps a "fork" and a "master" state per document and resolves at document granularity.

Client-generated IDs¶

Offline creation requires IDs that are unique without asking the server. Temporary IDs that the server later swaps for real ones force you to rewrite every reference in the outbox and view. Use client-generated identifiers instead:

• crypto.randomUUID() returns an RFC 9562 version 4 UUID. It is available in secure contexts in Chrome 92, Firefox 95 and Safari 15.4 and later.
• UUIDv7, defined in RFC 9562 (May 2024), puts a 48-bit Unix millisecond timestamp in the high bits, so IDs sort roughly by creation time and B-tree indexes on the server stay compact. There is no built-in generator in browsers; generating one takes a few lines.

// RFC 9562 UUIDv7: 48-bit ms timestamp, version 7, 74 random bits, variant 10.
export function uuidv7() {
  const bytes = crypto.getRandomValues(new Uint8Array(16));
  const ts = BigInt(Date.now());
  for (let i = 0; i < 6; i++) bytes[i] = Number((ts >> BigInt(8 * (5 - i))) & 0xffn);
  bytes[6] = (bytes[6] & 0x0f) | 0x70; // version 7
  bytes[8] = (bytes[8] & 0x3f) | 0x80; // RFC 9562 variant
  const hex = [...bytes].map((b) => b.toString(16).padStart(2, "0")).join("");
  return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}`;
}

The timestamp in a UUIDv7 is for index locality only. Do not use it to order conflicting writes; a device with a wrong clock produces IDs "from the future".

Showing pending state honestly¶

Optimistic does not mean silent. Good offline-first UIs expose three states per record or per screen: synced, pending (in the outbox) and failed (rejected by the server, with the reason). Because the outbox is a store, a pending badge is a simple query: "does any outbox entry reference this ID?" Keep a small global indicator ("3 changes waiting to sync") and a way to inspect failures. Offline UX & Fallbacks covers the interaction patterns.

Conflict resolution strategies¶

A conflict exists when two replicas change the same logical data concurrently: neither change was made with knowledge of the other. Sequential changes, where device B edits after having pulled A's edit, are not conflicts, and a correct system must be able to tell the difference. The strategies below differ in how they detect concurrency and what they do about it.

Server-authoritative mutations¶

With the mutation model described above, most "conflicts" dissolve: the server applies each mutation's intent to whatever state it has. Two devices that set different fields of the same task both succeed. Two devices that rename the same task produce "the last one processed wins", which is usually acceptable. For operations that must not blindly overwrite, the mutator itself carries a precondition:

export class MutationError extends Error {
  constructor(code, details = {}) { super(code); this.code = code; this.details = details; }
}

export async function renameTask(tx, { id, title, expectedTitle }) {
  const task = await tx.get(id);
  if (!task || task.deleted) throw new MutationError("not_found");
  // Only rename if nobody changed the title since the user started editing.
  if (expectedTitle !== undefined && task.title !== expectedTitle) {
    throw new MutationError("conflict", { current: task.title });
  }
  await tx.put({ ...task, title });
}

The same idea exists at the HTTP level for REST APIs that are not mutation-based: send If-Match with the entity's ETag and let the server answer 412 Precondition Failed when the resource changed (RFC 9110 conditional requests). The client then refetches and either re-applies the user's change or asks the user.

Last-write-wins and why clocks lie¶

Last-write-wins (LWW) keeps the value with the latest timestamp and discards the other. It is simple, converges, and loses data by design. The hard part is the timestamp:

• Device wall clocks can be minutes or years off and can move backwards (NTP corrections, manual changes, dead RTC batteries on cheap devices). A device whose clock is ahead wins every conflict until real time catches up.
• Server receive time is consistent, but it orders by arrival, not by when the user acted: an edit made offline yesterday and pushed today overwrites an edit made online an hour ago.
• Hybrid logical clocks (HLC) combine a physical timestamp with a logical counter so that timestamps are monotonic per node, respect causality (a write made after seeing another always has a larger timestamp) and stay close to wall time. Each node tracks the maximum timestamp it has seen and never issues a smaller one.

// Timestamp = { wall: ms, counter: int, node: string }, compared lexicographically.
export class HLC {
  constructor(node, maxDriftMs = 60_000) {
    this.node = node; this.wall = 0; this.counter = 0; this.maxDrift = maxDriftMs;
  }
  now() { // local event or send
    const pt = Date.now();
    if (pt > this.wall) { this.wall = pt; this.counter = 0; } else { this.counter++; }
    return { wall: this.wall, counter: this.counter, node: this.node };
  }
  receive(remote) { // merge a timestamp seen on incoming data
    const pt = Date.now();
    if (remote.wall - pt > this.maxDrift) throw new Error("remote clock too far ahead");
    const wall = Math.max(this.wall, remote.wall, pt);
    if (wall === this.wall && wall === remote.wall) this.counter = Math.max(this.counter, remote.counter) + 1;
    else if (wall === this.wall) this.counter++;
    else if (wall === remote.wall) this.counter = remote.counter + 1;
    else this.counter = 0;
    this.wall = wall;
    return { wall: this.wall, counter: this.counter, node: this.node };
  }
}
export const compareHLC = (a, b) =>
  a.wall - b.wall || a.counter - b.counter || (a.node < b.node ? -1 : a.node > b.node ? 1 : 0);

The node ID as the final tiebreaker guarantees a total order, so every replica picks the same winner.

Apply LWW per field, not per row. Row-level LWW turns "A changed the title, B changed the due date" into a lost update. Store a timestamp per field (or per column group) and merge field by field; this is what an LWW-map CRDT does, and it is how TinyBase's MergeableStore and many hand-written engines behave.

Version vectors: detecting concurrency¶

A version vector maps each replica ID to the number of updates from that replica that a copy of the data has seen. Comparing two vectors tells you whether one copy descends from the other (safe to overwrite) or whether they are concurrent (a real conflict).

// vv = { [replicaId]: counter }
export function compareVV(a, b) {
  let aBigger = false, bBigger = false;
  for (const k of new Set([...Object.keys(a), ...Object.keys(b)])) {
    const x = a[k] ?? 0, y = b[k] ?? 0;
    if (x > y) aBigger = true;
    if (y > x) bBigger = true;
  }
  if (aBigger && bBigger) return "concurrent";
  if (aBigger) return "a-descends";
  if (bBigger) return "b-descends";
  return "equal";
}
export const mergeVV = (a, b) => {
  const out = { ...a };
  for (const [k, v] of Object.entries(b)) out[k] = Math.max(out[k] ?? 0, v);
  return out;
};
export const bumpVV = (vv, replica) => ({ ...vv, [replica]: (vv[replica] ?? 0) + 1 });

Version vectors detect conflicts; they do not resolve them. On "concurrent" you still need a policy: LWW on a tiebreaker, a merge function, or keeping both versions as siblings and asking the user (Amazon Dynamo and Riak popularized siblings). CouchDB and PouchDB use a related mechanism, a revision tree per document (_rev values such as 3-a1b2…): concurrent edits create branches, the database deterministically picks a winning revision so every replica agrees, and the losing branches remain available through _conflicts until the application resolves them.

Vectors grow with the number of replicas that ever wrote. In a PWA every browser profile is a replica, so pruning matters: keep vectors per document rather than globally, and let the server collapse entries from clients that have been retired.

Operational transformation¶

Operational transformation (OT) is the technique behind Google Docs-style editing. Each edit is an operation such as insert(pos, text) or delete(pos, len). When two operations are concurrent, a transform function rewrites one against the other so that both orders of application produce the same document. If A inserts "X" at position 5 while B deletes position 2, B's delete is unaffected and A's insert must shift to position 4 when applied after B.

// Transform op `a` so it can be applied after concurrent op `b` (single characters for brevity).
export function transform(a, b) {
  if (a.type === "insert" && b.type === "insert") {
    if (a.pos < b.pos || (a.pos === b.pos && a.site < b.site)) return a;
    return { ...a, pos: a.pos + b.text.length };
  }
  if (a.type === "insert" && b.type === "delete") {
    return a.pos <= b.pos ? a : { ...a, pos: a.pos - 1 };
  }
  if (a.type === "delete" && b.type === "insert") {
    return a.pos < b.pos ? a : { ...a, pos: a.pos + b.text.length };
  }
  // delete vs delete of the same position: the second becomes a no-op
  if (a.pos === b.pos) return { type: "noop" };
  return a.pos < b.pos ? a : { ...a, pos: a.pos - 1 };
}

Practical OT systems rely on a central server that assigns a total order to operations; clients transform their pending operations against everything the server has accepted since their last known revision. That makes OT a good fit for online collaboration with short offline gaps and a poor fit for long offline periods or peer-to-peer sync, where CRDTs are the standard choice. Implementing OT correctly for rich text is notoriously hard; use an established implementation (ShareDB, or an editor with built-in collaboration) rather than writing transform functions for a rich data model.

CRDTs: conflict-free replicated data types¶

A CRDT is a data structure whose merge operation is commutative, associative and idempotent, so any two replicas that have received the same set of updates are in the same state regardless of order or duplication. No central server is needed to decide anything, which is why CRDTs are the foundation of local-first software.

There are two families:

• State-based (convergent) CRDTs ship the whole state (or a delta of it) and merge with a join function, for example element-wise max for a grow-only counter.
• Operation-based (commutative) CRDTs ship operations that commute; they require exactly-once, causally ordered delivery, which the sync layer must provide.

Common building blocks:

Two libraries dominate in JavaScript:

Yjs (yjs 13.6.x as of September 2026) implements shared types (Y.Text, Y.Array, Y.Map, Y.XmlFragment) on an algorithm based on YATA. It is optimized for text editing, has bindings for ProseMirror, TipTap, CodeMirror, Monaco, Quill and Lexical, and a provider ecosystem: y-indexeddb for local persistence, y-websocket and hosted services for relay, y-webrtc for peer-to-peer. Updates are compact binary Uint8Arrays; Y.encodeStateVector() and Y.encodeStateAsUpdate(doc, stateVector) compute exactly the delta another replica is missing.

import * as Y from "yjs";
import { IndexeddbPersistence } from "y-indexeddb";

const fromB64 = (s) => Uint8Array.from(atob(s), (c) => c.charCodeAt(0));

export async function openNote(noteId) {
  const doc = new Y.Doc();
  const local = new IndexeddbPersistence(`note:${noteId}`, doc);
  await local.whenSynced; // local state loaded: render immediately, even offline

  async function syncWithServer() {
    // 1. Send our state vector; the server answers with the update we are missing
    //    (body) and its own state vector (header), in one round trip.
    const res = await fetch(`/api/notes/${noteId}/sync`, {
      method: "POST",
      headers: { "Content-Type": "application/octet-stream" },
      body: Y.encodeStateVector(doc),
    });
    if (!res.ok) throw new Error(`sync failed: ${res.status}`);
    const missing = new Uint8Array(await res.arrayBuffer());
    const svHeader = res.headers.get("X-Yjs-State-Vector");
    if (!svHeader) throw new Error("sync response lacks a state vector");
    const serverSV = fromB64(svHeader);
    Y.applyUpdate(doc, missing, "server"); // origin tag: not echoed back below

    // 2. Send the server exactly what it lacks. Re-sending is harmless: updates are idempotent.
    const ours = Y.encodeStateAsUpdate(doc, serverSV);
    if (ours.byteLength > 2) { // an empty update encodes to 2 bytes
      const put = await fetch(`/api/notes/${noteId}/update`, {
        method: "POST",
        headers: { "Content-Type": "application/octet-stream" },
        body: ours,
      });
      if (!put.ok) throw new Error(`upload failed: ${put.status}`);
    }
  }

  // Local edits are already persisted by y-indexeddb; just schedule a sync.
  let timer;
  doc.on("update", (_update, origin) => {
    if (origin === "server") return;
    clearTimeout(timer);
    timer = setTimeout(() => syncWithServer().catch(() => { /* retried on "online" */ }), 500);
  });
  addEventListener("online", () => syncWithServer().catch(() => {}));

  return { doc, text: doc.getText("body"), syncWithServer };
}

The server side of this endpoint is symmetrical: it loads its copy of the document, answers Y.encodeStateAsUpdate(serverDoc, clientSV) in the body and Y.encodeStateVector(serverDoc) in the header, and applies incoming updates with Y.applyUpdate() after checking that the user may write to that note.

Because Yjs updates are idempotent and commutative, a Yjs document needs no mutation outbox at all: the local document already contains every unsent change, and exchanging state vectors on reconnect recovers exactly what each side lacks. Y.mergeUpdates() compacts queued updates before sending.

Automerge models a document as a JSON-like tree with full change history. Automerge 3.0, released in July 2025, reduced memory usage by more than ten times according to the Automerge team while staying backwards compatible; the JavaScript package is at 3.5.x in September 2026. automerge-repo (2.x) adds storage adapters (including IndexedDB), network adapters and a sync server. Its sync protocol exchanges compact "have/need" messages based on hashes of change heads (generateSyncMessage / receiveSyncMessage with a per-peer sync state), so two peers converge in a few round trips even after long divergence.

import * as A from "@automerge/automerge";

let doc = A.from({ tasks: [] });
doc = A.change(doc, "add task", (d) => { d.tasks.push({ title: "Offline edit", done: false }); });

const bytes = A.save(doc);            // compact binary snapshot for IndexedDB

// Two replicas loaded from the same snapshot. A.load() gives each one its own
// random actor ID, which is what makes their concurrent changes distinguishable.
const left = A.change(A.load(bytes), (d) => { d.tasks[0].done = true; });
const right = A.change(A.load(bytes), (d) => { d.tasks[0].title = "Renamed offline"; });
const merged = A.merge(left, right);  // tasks[0]: { title: "Renamed offline", done: true }

Automerge documents behave as immutable values: A.change() returns a new document, and calling A.change() a second time on the old value throws ("Attempting to change an outdated document"). To fork a replica in memory, use A.clone(doc) or load a saved snapshot, as above; never branch by reusing a stale reference.

What CRDTs do not solve¶

• Invariants. A CRDT guarantees convergence, not correctness. Two offline replicas can each withdraw the last 10 units of stock; both withdrawals survive the merge and the counter goes negative. Anything with a global invariant needs a server-authoritative step.
• Authorization. A merge function accepts any well-formed update. The server (or relay) must check that the sender may write that document, and, for fine-grained permissions within a document, you must inspect updates before applying them, which the CRDT libraries do not do for you.
• Semantic conflicts. Merging "delete the paragraph" with "fix a typo in the paragraph" converges, but the result may not be what either user meant. Surface history and presence in the UI.
• Metadata growth. Sequence CRDTs keep tombstones for deleted elements so that concurrent inserts can still be positioned. Yjs garbage-collects deleted content when possible and Automerge compacts its storage, but documents with long histories still grow; plan for snapshots and size limits.
• Schema evolution. Changing the shape of a CRDT document (renaming a field, splitting a list) is itself a concurrent operation that old clients do not understand. Version documents and migrate by writing a new document when necessary.

Choosing a conflict strategy¶

Delta sync with cursors¶

Pulling the whole dataset on every sync does not scale past a few hundred rows. Delta sync transfers only what changed since the client's last sync, identified by a cursor the server handed out last time. Getting the cursor right is the single most common source of silently missing data in hand-built sync engines.

Why "updated_at > last sync time" loses changes¶

The obvious implementation, SELECT * FROM tasks WHERE updated_at > $lastSync, has three independent bugs:

1. Clock granularity and ties. Two rows updated in the same millisecond straddle the boundary; with > one is skipped, with >= rows are re-sent forever.
2. Clock skew between application servers. If updated_at is set by the app server rather than the database, a server with a slow clock writes timestamps in the past.
3. Commit order is not timestamp order. Transaction T1 sets updated_at = 10:00:00.100 and commits at .900; transaction T2 sets .500 and commits at .600. A pull at .700 sees T2, returns cursor .500, and T1's row, committed later with an earlier timestamp, is never sent.

The third bug also applies to auto-increment sequences (bigserial, IDENTITY): sequence values are assigned when a row is written, not when the transaction commits, so a lower value can become visible after a higher one has already been read.

Commit-ordered versions¶

A correct cursor must be monotonic in commit order. Three approaches work:

Serialized version counter

Keep a version row per sync scope (per user, workspace or "space"). Every writing transaction does UPDATE spaces SET version = version + 1 WHERE id = $1 RETURNING version first and stamps each changed row with that version. The row lock is held until commit, so the next writer cannot obtain version + 1 before this transaction commits: version order equals commit order. Throughput per scope is limited to one writer at a time, which is fine for per-user or per-team scopes. This is the approach Replicache's reference backends use and the one in the end-to-end example.

Transaction IDs with a visibility horizon

In PostgreSQL 13+, record pg_current_xact_id() on each change and, when serving a pull, only return changes from transactions older than pg_snapshot_xmin(pg_current_snapshot()), the oldest transaction still in progress. Anything newer might still be joined by an earlier, uncommitted transaction, so it waits for the next pull.

Logical replication

Read the database's write-ahead log (Postgres logical replication, MySQL binlog, MongoDB change streams) and assign cursors from the log position, which is commit-ordered by construction. This is how Electric, Zero's zero-cache and PowerSync's service ingest changes; it is the most robust option and the most infrastructure.

The pull protocol¶

A minimal but complete pull exchange:

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store

{
  "cursor": 1193,
  "hasMore": false,
  "lastMutationID": 42,
  "patch": [
    { "op": "put", "table": "tasks", "row": { "id": "0193…", "title": "Ship v2", "done": false, "version": 1190 } },
    { "op": "del", "table": "tasks", "id": "0192…" }
  ]
}

Rules that keep it correct:

• The cursor is opaque to the client. Today it is an integer; tomorrow it may be "{shard}:{lsn}". Clients store and echo it, nothing else.
• lastMutationID is returned in the same response, read in the same database snapshot as the patch. If the patch includes the effect of mutation 42, the client must learn in the same step that 42 is processed; otherwise the rebase replays 42 on top of its own result.
• With hasMore: true the client applies the page, stores the new cursor and immediately pulls again. Apply each page atomically (one IndexedDB transaction) so a crash never leaves a half-applied page with an advanced cursor.
• Send latest row state, not every intermediate change. If a row changed five times since the cursor, one put suffices. That is what makes delta sync cheap after long offline periods.
• Cache-Control: no-store keeps the HTTP cache and any service worker runtime caching out of the sync path. Sync responses must never be served stale.

Cursor expiry and full resync¶

A server cannot keep deletion history forever (see tombstones below). When a client presents a cursor older than the retention horizon, or one that belongs to a different shard or schema, respond with an explicit reset instruction, for example {"reset": true} or HTTP 410 Gone. The client then clears the base store, keeps its outbox, pulls from the beginning, and rebases. Electric's shape API has the same concept as a must-refetch control message that tells the client to discard its local shape data and resync from scratch.

Live updates without polling¶

Pull loops can poll (every 30–60 seconds while visible), but a poke channel is better: the server sends a tiny "something changed in scope X" message and the client pulls. Pokes carry no data, so they need no ordering or delivery guarantees; a missed poke only delays the next pull. Options:

• Server-Sent Events (EventSource): one long-lived HTTP response, automatic reconnection, works through most proxies. The simplest choice.
• WebSocket: bidirectional, lets you push mutations on the same connection.
• Long polling: Electric's HTTP API uses live=true requests that the server holds open until the shape changes, which keeps everything cacheable by CDNs.
• Web Push: wakes the service worker even when no tab is open, but every push must show a notification on most platforms (see Push Notifications), so it is not a silent data channel.

Run the poke connection only in the leader tab (next sections) and only while a tab is visible; mobile browsers kill background connections anyway.

Tombstones and deletes¶

A delta sync that only reports rows that exist cannot tell a client that a row stopped existing. Deletes must become data:

• Soft delete with a tombstone. Keep the row with deleted = true and a new version; the pull sends it as {"op": "del"}. RxDB and PouchDB work this way (_deleted: true), and CRDT sequences keep tombstones internally for the same reason.
• A separate deletions log. deleted_rows(table, id, version), when the main table must physically delete rows (for storage or legal reasons).

Tombstones need a retention horizon. Choose it longer than the longest realistic offline period (30–90 days is common), purge older tombstones, and record the purge watermark. Any pull whose cursor is below the watermark gets the full-resync instruction above, because the server can no longer prove which rows that client still holds that were deleted.

Two edge cases to handle deliberately:

• Resurrection. Device A deletes a task while device B, offline, edits it. When B pushes renameTask, the server finds a tombstone. Decide per mutator: reject (not_found, the most common choice), or undelete if the domain allows it. With LWW on a row including its deleted flag, the later timestamp wins and the row may silently come back; make that an explicit decision rather than an accident.
• Cascades. Deleting a project must produce tombstones (or a scope removal) for every task in it, or clients keep orphaned tasks forever. Generate them in the same transaction.

Partial replication and authorization¶

Real apps replicate a subset: the user's own records, the projects they belong to, the last 90 days of messages. The sync layer must enforce that subset on the server, because the client database is fully readable by the user.

• Define scopes (also called shapes, buckets, sync streams or partitions) server-side, from the authenticated user. The pull endpoint must never accept a client-supplied filter as the security boundary; client parameters may only narrow what the server already permits.
• Permission changes are data changes. When a user is removed from a project, their next pull must include removals for every row they lost access to, even though those rows did not change. Engines handle this by recomputing scope membership and emitting deletes (PowerSync's buckets and sync streams, Electric shapes, Zero's permission rules); in a custom engine, bump the user's scope version and send a full resync or a computed removal set.
• Writes are checked independently. The push endpoint re-runs mutators with the server's view of permissions; a client that edits IndexedDB directly and pushes a forged mutation gets rejected like any other invalid request.
• Size the subset for the device. Mobile browsers impose quotas and may evict non-persistent storage; see Storage Quotas & Persistence. Sync metadata and recent data first, fetch large blobs lazily, and request navigator.storage.persist() once the user has meaningful unsynced work.

Client schema migrations¶

An offline-first client carries state across app updates, so it has three things that are versioned independently:

Migrating the local database¶

IndexedDB migrations run in the upgradeneeded event inside a versionchange transaction; they are covered in detail, including the blocked and versionchange events between tabs, in IndexedDB. Two sync-specific rules:

1. Prefer resync over transformation for derived data. The base and view stores can always be rebuilt from the server: in the upgrade, delete and recreate them and reset the cursor. Only the outbox and genuinely local data (drafts never pushed, local settings) need careful, step-by-step transformation.
2. Never drop the outbox in a migration. Transform its entries or keep the old store until the queue drains. Losing the outbox loses user work that exists nowhere else.

import { openDB } from "idb";

export const DB_VERSION = 3;

export function openAppDB(userId) {
  return openDB(`app-${userId}`, DB_VERSION, {
    upgrade(db, oldVersion, _newVersion, tx) {
      if (oldVersion < 1) {
        db.createObjectStore("meta");
        db.createObjectStore("outbox", { keyPath: "mutationID" });
      }
      if (oldVersion < 2) {
        // v2 changed row shapes: derived stores are rebuilt from the server.
        for (const name of ["base", "view"]) {
          if (db.objectStoreNames.contains(name)) db.deleteObjectStore(name);
          const store = db.createObjectStore(name, { keyPath: "id" });
          store.createIndex("byList", "listId");
        }
        tx.objectStore("meta").delete("cursor"); // forces a full pull
      }
      if (oldVersion < 3) {
        // v3 renamed the mutator "setDone" to "setTaskDone": rewrite queued entries in place.
        // Not awaited: `upgrade` must not await anything but requests on `tx`, and the
        // versionchange transaction stays open while these requests are pending.
        (async () => {
          let cursor = await tx.objectStore("outbox").openCursor();
          while (cursor) {
            if (cursor.value.name === "setDone") {
              await cursor.update({ ...cursor.value, name: "setTaskDone", schema: 3 });
            }
            cursor = await cursor.continue();
          }
        })().catch(() => {
          try { tx.abort(); } catch { /* already aborted by the failed request */ }
        }); // a failed rewrite aborts the whole upgrade, and the old database stays intact
      }
    },
    blocking() {
      // Another tab wants a newer version: close so its upgrade can run, then reload.
      location.reload();
    },
  });
}

Old mutations after a deploy¶

Service worker updates mean some tabs run old code for hours or days (see Updating Service Workers), and a device that was offline for a week wakes up with a week-old outbox. The server must therefore accept every mutation schema still in the wild for a support window. Keep old mutator implementations on the server, dispatch on name and schema, and only remove a version after telemetry shows no client still sends it. For breaking protocol changes, have the pull endpoint return a "client too old" response that the page turns into an update prompt, and let the push endpoint keep accepting the old outbox so that updating never discards work.

Page and service worker version skew¶

If the service worker also reads the database (for example to push in a sync event), it may be newer or older than the open pages. Keep the database version and mutator registry in one shared module imported by both, and make the worker refuse to touch a database whose version it does not know: openDB() without a version argument opens the current version, and comparing db.version against the worker's expected version lets it bail out instead of corrupting data.

Coordinating multiple tabs¶

Every tab of the app shares one origin, one IndexedDB database and one client ID. If each tab runs its own sync loop, they push the same outbox entries concurrently, pull redundantly, rebase over each other and multiply server load. The fix is to run one sync engine per origin and let the other tabs be readers and writers of the local database only.

IndexedDB itself is safe across tabs: readwrite transactions with overlapping scopes are serialized, so a mutation in tab B and a rebase in tab A never interleave. What needs coordination is network work and change notifications.

Leader election with the Web Locks API¶

navigator.locks.request(name, callback) grants a named lock to one holder per origin (across tabs, dedicated workers, shared workers and service workers) and holds it until the promise returned by the callback settles. If the holder's tab closes or crashes, the lock is released automatically, which makes it a reliable leader election primitive. Web Locks are supported in Chrome 69, Firefox 96 and Safari 15.4 and later.

/**
 * Resolve `run()` in exactly one context per origin at a time. When the leader
 * tab closes, a waiting tab acquires the lock and becomes the new leader.
 */
export function electLeader(name, run) {
  const controller = new AbortController();
  navigator.locks
    .request(name, { mode: "exclusive", signal: controller.signal }, async () => {
      // We are the leader until the promise returned here settles.
      await run();              // should only return on shutdown
    })
    .catch((err) => {
      if (err.name !== "AbortError") console.error("leader lock failed", err);
    });
  return () => controller.abort(); // stop waiting (e.g. on sign-out)
}

Options worth knowing:

• mode: "shared" allows many holders, useful for readers that must block an exclusive maintenance task (for example a migration that clears stores).
• ifAvailable: true calls the callback with null instead of queueing when the lock is held, which suits "try to flush now, but do not wait".
• steal: true forcibly takes the lock, rejecting the current holder's request() promise with an AbortError. Use it for recovery from a stuck leader, never routinely.
• signal aborts a pending request; it does not release a lock already held.
• navigator.locks.query() returns held and pending locks, useful in diagnostics.

A page frozen in the back/forward cache cannot release a lock it holds, so holding a Web Lock can make a page ineligible for the bfcache in some browsers, much like an open IndexedDB connection (see IndexedDB). If bfcache eligibility matters, abort the leadership on pagehide and request it again on pageshow.

Change notifications with BroadcastChannel¶

After the leader applies a pull, or after any tab writes a mutation, other tabs must re-run their queries. BroadcastChannel delivers a structured-cloned message to every other BroadcastChannel object with the same name in the same origin (and the same storage partition), including objects in workers. The only object that does not receive a message is the one that posted it: a second BroadcastChannel object for the same name in the same tab does receive it, which the end-to-end example below relies on.

const channel = new BroadcastChannel("app-sync");

export function announceChange(stores) {
  channel.postMessage({ type: "changed", stores, at: Date.now() });
}
export function onChange(handler) {
  const listener = (e) => { if (e.data?.type === "changed") handler(e.data.stores); };
  channel.addEventListener("message", listener);
  return () => channel.removeEventListener("message", listener);
}
// Non-leader tabs ask the leader to flush the outbox soon.
export function requestFlush() { channel.postMessage({ type: "flush" }); }

Messages are notifications, not data: send the names of changed stores and let each tab re-query IndexedDB. That keeps messages tiny and makes a missed message harmless, because the next one triggers a full re-query. Also re-query on visibilitychange and pageshow with event.persisted === true, since a tab restored from the bfcache or frozen in the background may have missed messages. Messaging & the Clients API compares BroadcastChannel with postMessage to and from the service worker.

A SharedWorker as the sync hub¶

A SharedWorker is one worker instance shared by all same-origin tabs that construct it with the same URL and name. Hosting the sync engine there removes leader election entirely: the worker owns the network loop and the database connection, and tabs talk to it over MessagePorts.

SharedWorker is supported in desktop Chrome, Edge, Firefox and Safari 16+, in Safari on iOS 16+, and, since Chrome 148 (stable on May 5, 2026), in Chrome for Android, where it had long been disabled because of concerns about unpredictable process lifetimes on Android. Check "SharedWorker" in globalThis and fall back to Web Locks leader election where it is absent. Library SDKs make the same choice: PowerSync's web SDK uses shared workers for multi-tab coordination where available and documents a less reliable broadcast-based fallback elsewhere.

const ports = new Set();
let engine; // created on the first "init" message by createEngine(), your engine factory

self.addEventListener("connect", (event) => {
  const port = event.ports[0];
  ports.add(port);
  port.addEventListener("message", async (e) => {
    const { type, id, payload } = e.data;
    try {
      if (type === "init") engine ??= await createEngine(payload.userId, notifyAll);
      if (type === "mutate") await engine.mutate(payload.name, payload.args);
      if (type === "flush") await engine.sync();
      port.postMessage({ id, ok: true });
    } catch (err) {
      port.postMessage({ id, ok: false, error: String(err) });
    }
  });
  port.start();
});

function notifyAll(stores) {
  for (const port of ports) port.postMessage({ type: "changed", stores });
}

Two caveats:

• Detecting departed tabs. The HTML Standard now defines a close event on MessagePort that fires when the other side is disentangled, and Chromium has enabled it, but MDN's compatibility data does not yet record it as a cross-browser feature. Until it is, have tabs post a goodbye message on pagehide, listen for close as an extra signal where it exists, and treat the port set as best-effort (a port you post to after its tab died simply drops the message).
• Lifetime. A shared worker normally terminates shortly after the last tab that uses it closes, so it cannot finish a sync in the background. Chrome 148 added an extendedLifetime: true option to the SharedWorker constructor that asks the browser to keep the worker alive for a while after all clients unload, which suits "finish pushing the outbox" work. It is Chromium-only and the browser still decides how long the worker may outlive its clients, so treat it as an enhancement; for true background delivery, only the service worker (with Background Sync, also Chromium-only) can help.

Where the service worker fits¶

The service worker is shared by all tabs, but it is a poor host for a long-running sync loop: browsers terminate idle workers after roughly 30 seconds without events and cap how long event handlers may extend their lifetime. Use it for what only it can do:

• Run a push on a sync event after all tabs closed (Chromium; see Background Sync), with the same Web Lock the pages use, so it never pushes concurrently with a leader tab.
• Receive a push message and pull in the background before showing the notification.
• Serve the app shell offline (Precaching).

Choosing a coordination mechanism¶

Support data as of September 2026. See MDN browser compatibility data for live data.

Sync engines and libraries in 2026¶

Building the engine described on this page is a few thousand lines of code plus a server. Several mature options exist; they differ mainly in where the authority lives, which backend they sync with and what the client database is.

Versions come from each project's npm registry entries and release notes as of September 2026; check the linked documentation before you pin a version.

PouchDB and CouchDB¶

PouchDB is a JavaScript implementation of CouchDB's data model and replication protocol: every document has a revision history, db.sync(remote, { live: true, retry: true }) replicates both ways through the _changes feed with a checkpoint per replication, and deletes are tombstones (_deleted: true). Its strengths are maturity and a fully specified protocol that any CouchDB-compatible server can speak; its weaknesses are the document-per-record model (no server-side relational rules), per-database access control in CouchDB (the common pattern is one database per user) and a slower release cadence: the last major release, 9.0.0, was published to npm in June 2024 and is still the latest version in September 2026.

RxDB¶

RxDB is a reactive NoSQL database with JSON-schema collections and observable queries. Its replication protocol is backend-agnostic: you implement a pullHandler(lastCheckpoint, batchSize) that returns { documents, checkpoint }, a pushHandler(rows) where each row has an assumedMasterState and a newDocumentState and which returns the current master state of every conflicting document, and optionally a stream$ that emits changes or 'RESYNC'. This "checkpoint iteration plus event stream" design is essentially the delta sync described above, with conflicts detected by comparing the assumed state with the real one. Check the storage licensing before choosing: some storages and plugins are premium.

Replicache and Zero¶

Replicache introduced many of the ideas on this page to web developers: named mutators run speculatively on the client and authoritatively on the server, a per-client lastMutationID, pulls with an opaque cookie that return a patch, and rebase of pending mutations on top of new server state. Rocicorp has put Replicache in maintenance mode, open-sourced it and made it free, and recommends that existing users migrate to Zero.

Zero keeps the mutator model but moves the server side into infrastructure. zero-cache maintains a read-only replica of your Postgres database; the client keeps a store of recently used rows and runs queries written in ZQL (a streaming, incrementally maintained query language) against it first, returning local results instantly while the server returns authoritative results. The client syncs whatever your queries need rather than a predefined subset. Zero reached 1.0 in March 2026.

Electric¶

Electric (now at electric.ax, formerly ElectricSQL) is a read-path sync engine for Postgres. A client subscribes to a shape (a table plus optional where clause and columns) through a plain HTTP API: GET /v1/shape?table=…&offset=-1 returns the initial snapshot, subsequent requests pass the returned handle and offset, and live=true long-polls (or live_sse=true streams) for changes. Control messages such as up-to-date and must-refetch tell the client when it has caught up or must resync. Because shapes are ordinary HTTP responses, they can be cached by CDNs. Writes are explicitly out of scope ("bring your own writes"): you send mutations to your own API and reconcile optimistic state when the change flows back through the shape. Electric pairs well with a custom outbox like the one on this page.

PowerSync¶

PowerSync runs SQLite in the browser (via wa-sqlite, persisted to IndexedDB with IDBBatchAtomicVFS by default or to OPFS with OPFSCoopSyncVFS, which its docs recommend for Safari and iOS) and streams partitioned data from a PowerSync Service connected to your backend database. Local writes are applied to SQLite immediately and recorded in an upload queue (the ps_crud table); your connector's uploadData() sends them to your own API, and the SDK retries on failure. Server-side configuration (sync streams in the current docs) defines which rows each user receives, and clients subscribe to the streams they need. Multi-tab support uses shared workers where available.

TinyBase¶

TinyBase is a small reactive in-memory store (tables and key-values) with persisters to browser storage, IndexedDB, OPFS, SQLite engines and others, and synchronizers over WebSocket, BroadcastChannel or custom transports. Its MergeableStore is a CRDT with per-cell timestamps, so two stores that exchange changes converge without a server. It suits apps whose working set fits in memory. Version 10.0 was released on September 24, 2026.

Build or adopt¶

Adopt an engine when your backend matches it (Postgres for Zero and Electric; CouchDB for PouchDB), when you need partial replication with permissions, or when real-time collaboration matters. Build your own when your data is small and per-user, your backend is not supported, or you need full control of the protocol. Even then, copy the proven design: named mutators, clientID plus mutationID, commit-ordered cursors, tombstones and rebase.

Securing data on the device¶

Offline-first moves sensitive data out of your data center and into a browser profile. Plan the threat model explicitly.

What each measure actually protects against:

Encrypting records at rest¶

Web Crypto keys can be generated as non-extractable and stored in IndexedDB (a CryptoKey is structured-cloneable), so script can use the key but never read its bytes. That still leaves the key in the same profile as the data, which is why at-rest encryption only adds value when the key is not stored persistently: for example, derived from a user passphrase with PBKDF2 at unlock, or delivered by the server after authentication and kept only in memory. Encrypt values, not keys and indexes, or you lose the ability to query.

let key; // CryptoKey, set after unlock; never persisted

export async function unlock(passphrase, salt /* Uint8Array from server or meta store */) {
  const material = await crypto.subtle.importKey(
    "raw", new TextEncoder().encode(passphrase), "PBKDF2", false, ["deriveKey"]);
  key = await crypto.subtle.deriveKey(
    { name: "PBKDF2", salt, iterations: 600_000, hash: "SHA-256" },
    material, { name: "AES-GCM", length: 256 }, false, ["encrypt", "decrypt"]);
}

export async function seal(value) {
  const iv = crypto.getRandomValues(new Uint8Array(12)); // unique per encryption
  const data = new TextEncoder().encode(JSON.stringify(value));
  const ct = await crypto.subtle.encrypt({ name: "AES-GCM", iv }, key, data);
  return { iv, ct }; // store both; both are structured-cloneable
}

export async function open({ iv, ct }) {
  const pt = await crypto.subtle.decrypt({ name: "AES-GCM", iv }, key, ct);
  return JSON.parse(new TextDecoder().decode(pt));
}

Signing out and multi-user devices¶

• Namespace databases per user and delete the user's databases on sign-out (indexedDB.deleteDatabase(), caches.delete(), OPFS removeEntry()), after warning if the outbox still holds unsynced changes.
• From the server, send Clear-Site-Data: "cache", "storage" on the sign-out response to wipe all origin storage, including service worker registrations, in one step. The header is supported in Chrome 61, Firefox 63 and Safari 17 and later; Chromium's support for the "cache" directive is marked partial in MDN's compatibility data. It clears everything for the origin, including other accounts' data, which is usually what a sign-out on a shared device should do.
• Avoid storing long-lived refresh tokens in IndexedDB; prefer HttpOnly cookies for the session so an XSS cannot exfiltrate them. See Authentication & Passkeys.
• Remember eviction: a best-effort origin can lose its data under storage pressure, and Safari deletes script-writable storage for sites without user interaction for seven days in the browser (installed Home Screen web apps are exempt). An offline-first app must survive losing its database and resync from the server; only the outbox is irreplaceable, which is one more reason to push promptly. Details in Storage Quotas & Persistence and Privacy & Storage Partitioning.

End-to-end example: an offline-first task list¶

This example puts every piece of the page together for a per-user task list: shared mutators, an IndexedDB base/view/outbox, rebase on pull, a commit-ordered cursor, idempotent pushes, tombstones, leader election and cross-tab notifications. It uses the idb wrapper on the client and Express with pg on the server. The architecture is the one Replicache popularized, reduced to about 500 lines you can read in one sitting.

flowchart TB
    subgraph Browser
        T1["Tab 1 (leader)"] --- L[("Web Lock: tasks-sync")]
        T2["Tab 2"] -. "BroadcastChannel" .- T1
        T1 & T2 --> IDB[("IndexedDB: base, view, outbox, meta, failures")]
    end
    T1 -->|"POST /api/sync/push"| API["Express sync routes"]
    T1 -->|"GET /api/sync/pull?cursor="| API
    API -->|"SSE poke"| T1
    API --> PG[("Postgres: spaces, clients, tasks")]

Shared mutators¶

The mutators run on both sides against a tiny transaction interface (get, put, del). They take every non-deterministic input (IDs, timestamps) as arguments.

export class MutationError extends Error {
  constructor(code, details = {}) {
    super(code);
    this.name = "MutationError";
    this.code = code;
    this.details = details;
  }
}

const MAX_TITLE = 500;

function cleanTitle(title) {
  if (typeof title !== "string") throw new MutationError("invalid_title");
  const t = title.trim();
  if (!t || t.length > MAX_TITLE) throw new MutationError("invalid_title");
  return t;
}

export const mutators = {
  async createTask(tx, { id, title, createdAt }) {
    if (await tx.get(id)) return; // idempotent on the ID as well
    await tx.put({ id, title: cleanTitle(title), done: false, createdAt, deleted: false });
  },

  async renameTask(tx, { id, title }) {
    const task = await tx.get(id);
    if (!task || task.deleted) throw new MutationError("not_found");
    await tx.put({ ...task, title: cleanTitle(title) });
  },

  async setDone(tx, { id, done }) {
    // "set" rather than "toggle": two devices ticking the same box converge.
    const task = await tx.get(id);
    if (!task || task.deleted) throw new MutationError("not_found");
    await tx.put({ ...task, done: Boolean(done) });
  },

  async deleteTask(tx, { id }) {
    const task = await tx.get(id);
    if (!task || task.deleted) return; // deleting twice is fine
    await tx.del(id);
  },
};

export const MUTATION_SCHEMA = 1;

Server schema¶

-- One row per sync scope. Updating it first in every write transaction
-- serializes writers, so versions are assigned in commit order.
CREATE TABLE spaces (
  user_id      uuid PRIMARY KEY,
  version      bigint NOT NULL DEFAULT 0,
  min_cursor   bigint NOT NULL DEFAULT 0      -- tombstones older than this were purged
);

CREATE TABLE clients (
  client_id         text PRIMARY KEY,
  user_id           uuid NOT NULL REFERENCES spaces(user_id),
  last_mutation_id  bigint NOT NULL DEFAULT 0,
  last_seen         timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE tasks (
  id          uuid PRIMARY KEY,
  owner_id    uuid NOT NULL REFERENCES spaces(user_id),
  title       text NOT NULL,
  done        boolean NOT NULL DEFAULT false,
  created_at  bigint NOT NULL,
  deleted     boolean NOT NULL DEFAULT false,  -- tombstone
  version     bigint NOT NULL                  -- space version of the last change
);
CREATE INDEX tasks_owner_version ON tasks (owner_id, version);

CREATE TABLE mutation_errors (
  client_id    text NOT NULL,
  mutation_id  bigint NOT NULL,
  code         text NOT NULL,
  PRIMARY KEY (client_id, mutation_id)
);

Server push and pull routes¶

import express from "express";
import pg from "pg";
import { mutators, MutationError } from "../shared/mutators.js";

const pool = new pg.Pool(); // PG* environment variables
const PAGE = 1000;
const MAX_BATCH = 100;
export const router = express.Router();
router.use(express.json({ limit: "1mb" }));

// Adapter that lets the shared mutators write to Postgres for one user and version.
function serverTx(db, userId, version) {
  return {
    async get(id) {
      const { rows } = await db.query(
        `SELECT id, title, done, created_at AS "createdAt", deleted
           FROM tasks WHERE id = $1 AND owner_id = $2`, [id, userId]);
      return rows[0];
    },
    async put(row) {
      const { rowCount } = await db.query(
        `INSERT INTO tasks (id, owner_id, title, done, created_at, deleted, version)
              VALUES ($1, $2, $3, $4, $5, false, $6)
         ON CONFLICT (id) DO UPDATE
               SET title = EXCLUDED.title, done = EXCLUDED.done,
                   deleted = false, version = EXCLUDED.version
             WHERE tasks.owner_id = EXCLUDED.owner_id`,
        [row.id, userId, row.title, row.done, row.createdAt, version]);
      // rowCount 0: the ID exists but belongs to someone else.
      if (rowCount === 0) throw new MutationError("forbidden");
    },
    async del(id) {
      await db.query(
        `UPDATE tasks SET deleted = true, version = $3 WHERE id = $1 AND owner_id = $2`,
        [id, userId, version]);
    },
  };
}

async function ensureClient(db, clientID, userId) {
  await db.query(`INSERT INTO spaces (user_id) VALUES ($1) ON CONFLICT DO NOTHING`, [userId]);
  await db.query(
    `INSERT INTO clients (client_id, user_id) VALUES ($1, $2) ON CONFLICT DO NOTHING`,
    [clientID, userId]);
}

router.post("/sync/push", async (req, res) => {
  const userId = req.user.id; // set by your auth middleware
  const { clientID, mutations } = req.body ?? {};
  const wellFormed = (m) =>
    Number.isSafeInteger(m?.mutationID) && m.mutationID > 0 && typeof m.name === "string";
  if (typeof clientID !== "string" || !Array.isArray(mutations) ||
      mutations.length > MAX_BATCH || !mutations.every(wellFormed)) {
    return res.status(400).json({ error: "bad_request" });
  }
  const rejected = [];
  const db = await pool.connect();
  try {
    await ensureClient(db, clientID, userId);
    for (const m of mutations) {
      await db.query("BEGIN");
      // Lock order: space first, then client, in every transaction (avoids deadlocks).
      const { rows: [space] } = await db.query(
        `UPDATE spaces SET version = version + 1 WHERE user_id = $1 RETURNING version`, [userId]);
      const { rows: [client] } = await db.query(
        `SELECT user_id, last_mutation_id FROM clients WHERE client_id = $1 FOR UPDATE`, [clientID]);
      if (client.user_id !== userId) {
        await db.query("ROLLBACK");
        return res.status(403).json({ error: "client_belongs_to_other_user" });
      }
      const expected = Number(client.last_mutation_id) + 1;
      if (m.mutationID < expected) { await db.query("ROLLBACK"); continue; } // already applied
      if (m.mutationID > expected) {
        await db.query("ROLLBACK");
        return res.status(409).json({ error: "mutation_gap", expected });
      }

      const fn = Object.hasOwn(mutators, m.name) ? mutators[m.name] : null;
      await db.query("SAVEPOINT mutation");
      try {
        if (!fn) throw new MutationError("unknown_mutator");
        await fn(serverTx(db, userId, space.version), m.args ?? {});
      } catch (err) {
        if (!(err instanceof MutationError)) throw err; // a bug: 500, client retries later
        // Rejected: undo its partial effects but still mark it processed.
        await db.query("ROLLBACK TO SAVEPOINT mutation");
        await db.query(
          `INSERT INTO mutation_errors (client_id, mutation_id, code) VALUES ($1, $2, $3)
           ON CONFLICT DO NOTHING`, [clientID, m.mutationID, err.code]);
        rejected.push({ mutationID: m.mutationID, code: err.code });
      }
      await db.query(
        `UPDATE clients SET last_mutation_id = $2, last_seen = now() WHERE client_id = $1`,
        [clientID, m.mutationID]);
      await db.query("COMMIT");
    }
    res.json({ ok: true, rejected });
    poke(userId); // tell this user's other devices to pull
  } catch (err) {
    await db.query("ROLLBACK").catch(() => {});
    console.error("push failed", err);
    res.status(500).json({ error: "internal" });
  } finally {
    db.release();
  }
});

// Cursor format (opaque to the client): "<version>" once caught up, or
// "<version>@<snapshot>" while paging through an initial or reset sync, where
// <snapshot> is the space version when that sync started.
function parseCursor(raw) {
  const m = /^(\d{1,15})(?:@(\d{1,15}))?$/.exec(String(raw ?? "0"));
  if (!m) return null;
  return { pos: Number(m[1]), snapshot: m[2] === undefined ? null : Number(m[2]) };
}

router.get("/sync/pull", async (req, res) => {
  const userId = req.user.id;
  const clientID = String(req.query.clientID ?? "");
  const cursor = parseCursor(req.query.cursor);
  if (!clientID || !cursor) {
    return res.status(400).json({ error: "bad_request" });
  }
  res.set("Cache-Control", "no-store");
  const db = await pool.connect();
  try {
    await ensureClient(db, clientID, userId);
    // One snapshot for the patch, the new cursor and lastMutationID.
    await db.query("BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY");
    const { rows: [space] } = await db.query(
      `SELECT version, min_cursor FROM spaces WHERE user_id = $1`, [userId]);
    const { rows: [client] } = await db.query(
      `SELECT user_id, last_mutation_id FROM clients WHERE client_id = $1`, [clientID]);
    if (client.user_id !== userId) {
      await db.query("ROLLBACK");
      return res.status(403).json({ error: "client_belongs_to_other_user" });
    }
    // What this client is guaranteed to know about deletions: everything up to its
    // cursor, or, mid initial sync, everything up to the snapshot that sync began at.
    // Tombstones below min_cursor are gone, so an older client must start over.
    const known = cursor.snapshot ?? cursor.pos;
    const reset = known > 0 && known < Number(space.min_cursor);
    const from = reset ? 0 : cursor.pos;
    const initial = from === 0 || (!reset && cursor.snapshot !== null);
    const snapshot = from === 0 ? Number(space.version) : cursor.snapshot;

    const cols = `id, title, done, created_at AS "createdAt", deleted, version`;
    const { rows } = await db.query(
      `SELECT ${cols} FROM tasks
        WHERE owner_id = $1 AND version > $2 ${from === 0 ? "AND NOT deleted" : ""}
        ORDER BY version, id LIMIT $3`, [userId, from, PAGE + 1]);

    let page = rows;
    let next = Number(space.version);
    let hasMore = false;
    if (rows.length > PAGE) {
      // Never split one version across pages, or the next pull would skip its tail.
      hasMore = true;
      const boundary = Number(rows[PAGE].version);
      page = rows.filter((r) => Number(r.version) < boundary);
      if (page.length === 0) {
        page = (await db.query(
          `SELECT ${cols} FROM tasks WHERE owner_id = $1 AND version = $2`, [userId, boundary])).rows;
        next = boundary;
      } else {
        next = Number(page[page.length - 1].version);
      }
    }
    await db.query("COMMIT");

    res.json({
      reset,
      // Paging through an initial sync keeps the snapshot marker; afterwards the
      // cursor is a plain version again.
      cursor: hasMore && initial ? `${next}@${snapshot}` : String(next),
      hasMore,
      lastMutationID: Number(client.last_mutation_id),
      patch: page.map((r) => r.deleted
        ? { op: "del", id: r.id }
        : { op: "put", row: { id: r.id, title: r.title, done: r.done,
                              createdAt: Number(r.createdAt), deleted: false } }),
    });
  } catch (err) {
    await db.query("ROLLBACK").catch(() => {});
    console.error("pull failed", err);
    res.status(500).json({ error: "internal" });
  } finally {
    db.release();
  }
});

// Pokes over Server-Sent Events. In-memory: use Postgres LISTEN/NOTIFY or a
// pub/sub service to fan out when you run more than one server instance.
const listeners = new Map(); // userId -> Set<res>
router.get("/sync/poke", (req, res) => {
  const userId = req.user.id;
  res.set({ "Content-Type": "text/event-stream", "Cache-Control": "no-store" });
  res.flushHeaders();
  res.write("retry: 5000\n\n");
  const set = listeners.get(userId) ?? new Set();
  set.add(res);
  listeners.set(userId, set);
  const heartbeat = setInterval(() => res.write(": keep-alive\n\n"), 25_000);
  req.on("close", () => { clearInterval(heartbeat); set.delete(res); });
});
function poke(userId) {
  for (const res of listeners.get(userId) ?? []) res.write("data: poke\n\n");
}