Production Reliability and Deterministic Execution for Visual AI Agents

Aliax is the layer between your AI model and the live browser that stops ghost clicks and silent stalls. The Steady Hand stamps a unique ID onto every clickable element before acting, so clicks always land on the right target even when the page reflows or re-renders. The Compactor strips the page down to just the interactive elements — about 1,500 tokens instead of 100,000 — and paints a numbered screenshot your model can read. The Airbag sends a visual replay to your dashboard the moment an agent stalls — like Sentry, but for agents.

Five steps. You'll have a VLM agent clicking the right element in under five minutes. Code blocks below follow your language pick above — change it any time and every snippet updates in place.

pip install aliax
playwright install chromium

from aliax import Aliax

# Zero-config: works immediately out of the box with no key!
aliax = Aliax()

# Or authenticate with a personal key:
# aliax = Aliax(api_key="sk_live_...")  # or set os.environ["ALIAX_API_KEY"]

from playwright.async_api import async_playwright

async with async_playwright() as p:
    page = await (await p.chromium.launch()).new_page()
    await page.goto("https://shop.example.com/cart")

    # 1. Translate the live DOM → Set-of-Mark image + JSON node map.
    ctx = await aliax.parse_ui(page, render_config={"format": "jpeg", "quality": 70})
    #    ctx.image_bytes  — JPEG (default) with numbered Set-of-Mark boxes.
    #    ctx.elements     — [{element_id, tag, bounds, text, ...}, ...]

    # 2. Hand BOTH to your VLM. It replies with a structured action.
    decision = await ask_llm(image=ctx.image_bytes, map=ctx.elements)
    #   → {"action": "CLICK", "element_id": "el_41"}

    # 3. Aliax executes natively — scroll-into-view, React onChange,
    #    iframe coord math, retina DPR, all handled.
    result = await aliax.execute(page, decision)
    print(result)  # {'ok': True, 'action': 'CLICK', 'coords': [905, 150], ...}

# When the agent loops past recovery, dump the state to the queue:
await aliax.capture_failure(
    page,
    goal="Close newsletter popup",
    thoughts=agent.current_reasoning,
    last_attempted_action={"action": "CLICK", "element_id": "el_41"},
    failure_reason="modal_blocked",
    step=agent.loop_step,
)
# Then open /captures to see it land and tap the right element.

Every call to parse_ui strips a typical page from 100,000 tokens of raw HTML down to around 1,500 tokens — just the interactive elements your model needs to decide what to do next. Aliax is three parts: the SDK (Python or Node) that runs in your agent process, a Cloudflare Worker at the edge, and a Postgres + R2 database for capturing failures and storing screenshots.

The three product pillars

How clicks stay accurate after page changes

When parse_ui() runs, the DOM mapper writes a data-aliax-id attribute on every interactable element before the screenshot is taken:

<button data-aliax-id="el_41" aria-label="Add to cart">Add to cart</button>

execute() resolves the target via document.querySelector('[data-aliax-id="el_41"]') — never by coordinate. This makes every click CLS-immune: React re-renders, lazy hydration, ad reflow, and viewport resize cannot shift the target out from under the executor. The stamp is rewritten on every parse_ui() call so it always reflects the live DOM.

Deployment topology

The DOM scan and screenshot happen entirely inside your browser — nothing leaves the machine on Aliax's critical path. The numbered overlay is drawn by the bundled script and captured by Playwright directly. No extra image processing step. A tiny background ping records usage for billing. Only capture_failure() actually sends a screenshot to the cloud — and if that upload fails, your agent keeps running. Screenshots are served from a CDN; the API server is never in the read path.

The pure, zero-dependency local loop. Import Aliax, call parse_ui(), send the compact payload to your VLM, and call execute(). No frameworks, no orchestrator — just you, Playwright, and your model of choice.

The complete loop

from playwright.async_api import async_playwright
from aliax import Aliax, SYSTEM_INSTRUCTIONS

# Zero-signup: starts keyless via free local sandbox (~/.aliax/credentials).
# Or pass your personal key: Aliax(api_key=os.environ["ALIAX_API_KEY"])
aliax = Aliax()
goal = "Add the cheapest in-stock item to the cart and check out."

async with async_playwright() as p:
    page = await (await p.chromium.launch()).new_page()
    await page.goto("https://shop.example.com/cart")

    while True:
        # 1. SEE — CLS-immune perception: 1.5k visual map + Set-of-Mark image
        ctx = await aliax.parse_ui(page)

        # 2. THINK — your VLM picks an element_id, not a pixel
        decision = await ask_llm(
            image=ctx.image_bytes,
            elements=ctx.elements,
            system=SYSTEM_INSTRUCTIONS,
        )
        if decision["action"] == "DONE":
            break

        # 3. ACT — CLS-immune execution via data-aliax-id (0 ghost clicks)
        result = await aliax.execute(page, decision)
        if not result["ok"]:
            # Aliax checks whether the page is actually stuck before escalating.
            await aliax.report_issue(
                page,
                reason="execute_failed",
                goal=goal,
                last_attempted_action=decision,
            )
            break

Aliax works with any headless cloud browser provider — Browserbase, Steel, Browserless, or your own setup. The DOM scan and screenshot run inside the remote browser. The only thing that travels over the network each step is the compact element list (~1,500 tokens) plus the screenshot — never the full page HTML.

1. Mint a Browserbase session (or Steel)

Both providers expose a REST endpoint that returns a CDP URL. Browserbase: POST https://www.browserbase.com/v1/sessions. Steel: POST https://api.steel.dev/v1/sessions. Same shape, swap the host and the auth header.

import os, httpx

async def mint_browserbase_cdp_url() -> str:
    async with httpx.AsyncClient(timeout=10) as client:
        r = await client.post(
            "https://www.browserbase.com/v1/sessions",
            headers={"x-bb-api-key": os.environ["BROWSERBASE_API_KEY"]},
            json={"projectId": os.environ["BROWSERBASE_PROJECT_ID"]},
        )
        r.raise_for_status()
        return r.json()["connectUrl"]

# Steel variant — identical shape, different host:
#   POST https://api.steel.dev/v1/sessions
#   Header: Steel-Api-Key: <key>
#   Response: { "websocketUrl": "wss://..." }

2. Connect Playwright, drive Aliax

from playwright.async_api import async_playwright
from aliax import Aliax

aliax = Aliax(api_key=os.environ["ALIAX_API_KEY"])
cdp_url = await mint_browserbase_cdp_url()

async with async_playwright() as p:
    browser = await p.chromium.connect_over_cdp(cdp_url)
    try:
        page = await browser.new_page()
        await page.goto("https://target.example.com")

        # The Compactor runs INSIDE the remote browser — no extra round-trip.
        ctx = await aliax.parse_ui(page)
        decision = await ask_llm(image=ctx.image_bytes, elements=ctx.elements)
        await aliax.execute(page, decision)
    finally:
        await browser.close()  # release the Browserbase session

Already using Browser-Use, Stagehand, or LangChain? You don't need to rewrite your agent. Add Aliax as a per-step hook — about 10 lines — to get accurate clicks that survive layout shifts and SPAs, plus visual crash captures when something goes wrong.

Per-step hook (Browser-Use)

Browser-Use exposes lifecycle hooks. Call aliax.parse_ui() before each model decision to pass Aliax's CLS-immune element map and Set-of-Mark image to your agent state, and use aliax.execute() to execute actions deterministically:

import os
from browser_use import Agent, AgentHooks
from aliax import Aliax

aliax = Aliax(api_key=os.environ["ALIAX_API_KEY"])

class AliaxHooks(AgentHooks):
    async def on_step_start(self, agent_state, browser_context):
        page = await browser_context.get_current_page()
        ctx = await aliax.parse_ui(page)
        # Compact node map (~1.5k tokens) + collision-free Set-of-Mark image.
        # Supplies CLS-immune element IDs and shadow DOM resolution.
        agent_state.aliax_ctx = ctx

agent = Agent(
    task="Find the cheapest non-stop flight from JFK → LHR next Friday.",
    llm=my_llm,
    hooks=AliaxHooks(),
)

await agent.run()

Visual telemetry on error (Stagehand / LangChain / custom)

If your framework throws on a failed action, capture_failure() snapshots the live page and sends it straight to your annotation queue — no stall check, fires on every caught error. Use this for the "something broke in production, show me the screenshot" path.

try:
    await stagehand.act(instruction)
except Exception as e:
    # Unconditional capture: snapshots the page, posts to your annotation queue.
    await aliax.capture_failure(
        page,
        goal=current_goal,
        thoughts=agent.last_reasoning,
        last_attempted_action={"action": "CLICK", "element_id": e.target},
        failure_reason="stagehand_error",
    )

Report a stuck loop

If the agent keeps trying the same action and nothing changes, call report_issue(). Aliax checks whether the page has actually stalled before logging the failure — it will only escalate after seeing 3 identical page states or the same action repeated in a cycle. That means it's safe to call on every step; it won't cry wolf. reason is required.

await aliax.report_issue(
    page,
    reason="suspected_stall",
    goal=current_goal,
    thoughts=agent.last_reasoning,
    last_attempted_action=last_decision,
)

You can run Aliax right now — no account needed. Or add an API key to unlock the dashboard, live telemetry, and crash captures.

1. Free sandbox — no signup required

Call Aliax() or new Aliax() with no API key and Aliax works immediately. No account, no credit card. On first use, the SDK quietly creates and saves a free credential to your machine so you don't have to do anything.

2. Personal workspace API key (sk_live_...)

Keys are minted from 32 bytes of crypto.getRandomValues and rendered into a 32-character lowercase Crockford-style alphabet (digits 2-9, letters a-z excluding l and o) — shown as sk_live_<32 chars> (40 total characters including the prefix). We store only the SHA-256 hash; the plaintext is shown exactly once.

A personal key grants access to the web dashboard, live telemetry, and visual crash captures (POST /v1/capture). Every free account includes 1,000 starter credits.

3. Enterprise fail-closed opt-out

For SOC2 compliance or air-gapped CI/CD environments that must never make unauthenticated background requests, pass allow_anonymous=False (Python) or allowAnonymous: false (Node), or set the environment variable ALIAX_DISABLE_ANONYMOUS=1. When opted out, missing keys throw immediately instead of minting a sandbox token.

How the SDK sends credentials

The client adds the key as an Authorization: Bearer <key> header on every request (telemetry and capture). The plaintext token never appears in the request body, never lands in multipart parts, and is never logged at proxies, CDNs, or in Worker request logs. The Worker reads it exclusively from this header — a request that puts the key in a form field will be rejected with 401 missing_api_key.

Rotation & revocation

Revoke from API keys. A revoked key returns 401 revoked_api_key on the next call. Rotate quarterly and on any suspected leak.

Don't write your own system prompt. Aliax ships a ready-made, versioned instruction string — SYSTEM_INSTRUCTIONS — that tells your model exactly how to read the page and what actions are valid.

How to import

Both the Python and Node SDKs export SYSTEM_INSTRUCTIONS and its alias ALIAX_SYSTEM_INSTRUCTIONS directly from the package root:

from aliax import Aliax, SYSTEM_INSTRUCTIONS

# Pass SYSTEM_INSTRUCTIONS directly to your model's system prompt

What the prompt handles automatically

Plugging into popular LLM providers

# OpenAI (GPT-4o)
from openai import AsyncOpenAI
from aliax import SYSTEM_INSTRUCTIONS

openai = AsyncOpenAI()
res = await openai.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM_INSTRUCTIONS},
        {
            "role": "user",
            "content": [
                {"type": "text", "text": f"Goal: {user_goal}\n\nDOM Map:\n{ctx.llm_text_block()}"},
                {"type": "image_url", "image_url": {"url": f"data:{ctx.image_mime};base64,{ctx.image_base64}"}},
            ],
        },
    ],
    response_format={"type": "json_object"},
)
decision = json.loads(res.choices[0].message.content)
await aliax.execute(page, decision)

A single endpoint. Multipart in, JSON out.

cURL

curl -X POST https://api.aliax.xyz/v1/capture \
  -H "Authorization: Bearer sk_live_..." \
  -H "X-Aliax-SDK-Version: 1.0.5" \
  -H "X-Idempotency-Key: $(uuidgen)" \
  -F "goal=Add the Medium size to cart" \
  -F "ai_thoughts=Targeting the green dismiss button at top-right of modal." \
  -F 'last_attempted_action={"action":"CLICK","target_x":905,"target_y":150,"selector":".modal-close","element_id":"el_88"}' \
  -F "failure_reason=element_not_found" \
  -F "step=3" \
  -F 'viewport={"width":1920,"height":1080,"dpr":2}' \
  -F 'spatial_map=[{"element_id":"el_1","role":"button","label":"Close","bbox":[1820,40,80,40]}]' \
  -F "sdk_version=1.0.5" \
  -F "screenshot=@./snag.jpg;type=image/jpeg"

Limits

Every field is a form field. JSON objects (like the spatial map) are passed as a JSON string in that field — the SDK handles this for you.

Send the agent's reasoning and the action it tried, and Aliax turns every failure into a complete training example — the wrong move and the human-corrected right move, side by side.

Why it matters

A screenshot shows you what the page looked like. The attempted action shows you what the model tried to do. Together, the dashboard can pin a pulsing red-X exactly where the agent clicked, look up which element was actually at those coordinates, and tell you at a glance: "agent clicked a wrapper div when it should have clicked <button>Submit</button>."

The attempted-action shape

from dataclasses import dataclass
from typing import Optional

@dataclass
class AttemptedAction:
    action: str                          # "CLICK" | "TYPE" | "SCROLL" | "HOVER" | "NAVIGATE" | ...
    target_x: Optional[int] = None       # CSS pixels, viewport-relative
    target_y: Optional[int] = None
    value: Optional[str] = None          # text typed (TYPE actions)
    selector: Optional[str] = None       # css/xpath the agent THOUGHT it was hitting
    element_id: Optional[str] = None     # the el_<N> id from parse_ui, if any

    def to_dict(self) -> dict: ...       # asdict() with None values stripped

Either pass an AttemptedAction instance or a plain dict/object — the SDK accepts both. Coordinates are CSS pixels in the same coordinate space as spatial_map[].bbox, so the dashboard can do bounding-box lookups directly.

End-to-end example

import os
from aliax import Aliax, AttemptedAction

aliax = Aliax(api_key=os.environ["ALIAX_API_KEY"])

await aliax.capture_failure(
    page=page,
    goal="Close the newsletter modal and continue to checkout.",
    thoughts=agent.current_reasoning,
    last_attempted_action=AttemptedAction(
        action="CLICK",
        target_x=905,
        target_y=150,
        selector=".modal-close",
    ),
    failure_reason="element_not_found",
    step=agent.loop_step,
)

How the dashboard uses it

1. Reads last_attempted_action.target_x / target_y and pins a pulsing red-X marker on the screenshot at the scaled pixel.
2. Runs a point-in-bbox lookup against the spatial map to find which element the agent actually hit (often a wrapper, a backdrop, or empty space).
3. Renders ai_thoughts in the side panel so the annotator sees the model's reasoning, not just the outcome.
4. On annotation, writes back corrected_action alongside the original attempt — a complete (prompt, rejected, chosen) tuple for DPO.

Retina / HiDPI

The SDK reads window.devicePixelRatio inside the same __AliaxCore.mapDOM() call that returns the spatial map, then ships it as viewport.dpr. The dashboard scales the overlay accordingly, so a screenshot taken at 2× on a Retina display still lands the red-X on the right pixel.

201 — success

{
  "status": "success",
  "capture_id": "8f3c2a64-9c2e-4a3b-9d20-1f4e1c6e2b51",
  "screenshot_url": "https://cdn.aliax.xyz/<user_id>/8f3c2a64-….jpg",
  "msg": "Captured."
}

4xx / 5xx — error envelope

{
  "status": "error",
  "code": "invalid_api_key",
  "msg": "Unknown API key"
}

The SDK normalises both shapes into {status, capture_id, msg} so call sites don't need to branch on HTTP status.

The spatial map is the numbered list of every clickable element on the page — buttons, inputs, links — each with its position, label, and role. It's what lets the annotator tap a spot on the screenshot and know exactly which element the agent should have hit.

Envelope shape (v0.2)

{
  "elements": [ /* node[] */ ],
  "viewport": { "dpr": 2, "width": 1920, "height": 1080 }
}

The SDK extracts elements for the spatial_map field and copies viewport.dpr into the top-level viewport object. v0.1 clients that read a bare array still work — the SDK's extractor falls back to result.elements ?? result.

Node shape

{
  "element_id": "el_14",
  "role": "button",
  "label": "Close",
  "bbox": [1820, 40, 80, 40],
  "z": 999,
  "in_viewport": true,
  "frame": "main"
}

Errors are never silent. Every non-2xx response carries a stable code string — log against this, not against the HTTP status.

One API surface, two languages. Every method is identical in behavior — Python uses snake_case, Node/TypeScript uses camelCase. The wire format and the action verbs inside decision objects stay snake_case in both because they are JSON keys, not language identifiers.

pip install aliax
playwright install chromium

from dataclasses import dataclass
from typing import Optional, Union, Mapping, Any, List

@dataclass
class AttemptedAction:
    action: str
    target_x: Optional[int] = None
    target_y: Optional[int] = None
    value: Optional[str] = None
    selector: Optional[str] = None
    element_id: Optional[str] = None

@dataclass
class ParseContext:
    image_bytes: bytes          # JPEG (default) screenshot with numbered SoM boxes
    image_mime: str             # "image/jpeg" or "image/png"
    image_size: tuple           # (width, height)
    elements: List[dict]        # [{element_id, tag, bounds, text, ...}]
    viewport: dict              # {width, height, dpr, scroll_x, scroll_y}
    url: str
    path: str
    title: str
    truncated: bool

class Aliax:
    def __init__(
        self,
        api_key: Optional[str] = None,         # falls back to os.environ["ALIAX_API_KEY"], else free anonymous sandbox
        allow_anonymous: bool = True,          # True = auto-bootstrap sk_anon_...; False = fail-closed
        endpoint: str = "https://api.aliax.xyz/v1",
        debug_mode: bool = False,
        redact_selectors: Optional[list] = None,
        check_for_updates: bool = True,
        max_image_dim: int = 0,                # DEPRECATED — no-op since v1.0
        fallback_endpoint: Optional[str] = None,  # auto-set on the public default
    ): ...

    # Async context manager — deterministic httpx pool teardown.
    async def __aenter__(self) -> "Aliax": ...
    async def __aexit__(self, *exc) -> None: ...
    async def aclose(self) -> None: ...

async def parse_ui(
    self,
    page,
    *,
    render_config: Optional[Mapping[str, Any]] = None,
    #     {"format": "jpeg" | "png",   # default "jpeg"
    #      "quality": 30..100}          # default 80, clamped to floor 30
    draw_overlay: bool = True,
    min_size: int = 12,
    max_elements: int = 200,
    # Legacy kwargs — folded into render_config when not provided.
    max_image_dim: Optional[int] = None,   # DEPRECATED no-op (logs once)
    image_format: Optional[str] = None,    # legacy alias for render_config.format
    image_quality: Optional[int] = None,   # legacy alias for render_config.quality
) -> ParseContext: ...                     # raises AliaxInvalidKeyError / AliaxOutOfCreditsError

# --- Pointer / keyboard ---
await aliax.execute(page, {"action": "CLICK",          "element_id": "el_41"})
await aliax.execute(page, {"action": "HOVER",          "element_id": "el_14"})
await aliax.execute(page, {"action": "TYPE",           "element_id": "el_7",  "value": "Nike Shoes"})
await aliax.execute(page, {"action": "TYPE_AND_ENTER", "element_id": "el_7",  "value": "Nike Shoes"})
await aliax.execute(page, {"action": "PRESS",          "element_id": "el_7",  "key": "Enter"})

# --- Scrolling (directional shortcuts scroll ~80% of viewport / container) ---
await aliax.execute(page, {"action": "SCROLL_DOWN"})
await aliax.execute(page, {"action": "SCROLL_UP",   "element_id": "el_22"})  # scroll inside a container
await aliax.execute(page, {"action": "SCROLL_LEFT"})
await aliax.execute(page, {"action": "SCROLL_RIGHT"})
await aliax.execute(page, {"action": "SCROLL",      "dy": 600})              # explicit delta

# --- Navigation / pacing / completion ---
await aliax.execute(page, {"action": "NAVIGATE", "url": "https://example.com"})  # http(s) only — SSRF guard
await aliax.execute(page, {"action": "WAIT",     "ms": 1500})
await aliax.execute(page, {"action": "DONE"})    # also: "FINISH", "NOOP"

# --- Composite verbs ---
await aliax.execute(page, {
    "action": "BATCH_TYPE",
    "inputs": [
        {"element_id": "el_3", "value": "jane@example.com"},
        {"element_id": "el_4", "value": "hunter2"},
    ],
})
await aliax.execute(page, {
    "action": "COMBO",                    # exactly 2 sub-actions, no nesting
    "actions": [
        {"action": "CLICK", "element_id": "el_8"},
        {"action": "TYPE",  "element_id": "el_9", "value": "Hello"},
    ],
})

# --- Escalation (Gatekeeper-gated; rejected unless stagnation is proven) ---
await aliax.execute(page, {
    "action": "REPORT_ISSUE",
    "reason": "submit_unresponsive",
    "context": {
        "expected_outcome": "Navigate to /confirmation",
        "actual_outcome":   "Still on /checkout after 3 identical parses",
    },
})

# --- Raw-coords escape hatch (when you have your own mapper) ---
await aliax.execute(page, {"action": "CLICK", "x": 905, "y": 150})

# Direct capture — always posts to /v1/capture (or writes to tmp in debug_mode)
cap = await aliax.capture_failure(
    page,
    goal="Complete checkout",
    thoughts="The submit button is visible but clicking does nothing.",
    last_attempted_action={"action": "CLICK", "element_id": "el_42"},
    failure_reason="submit_unresponsive",
    step=7,
    context={"expected_outcome": "/confirmation", "actual_outcome": "still on /checkout"},
)
# cap["status"]: "success" | "error"
# cap["capture_id"]: str | None
# cap["screenshot_url"]: str (when success)

# Gatekeeper-gated escalation — rejected unless either
#   (a) three consecutive parse_ui rounds returned the same state hash, OR
#   (b) a period-2/3/4 action cycle was observed across distinct element_ids.
issue = await aliax.report_issue(
    page,
    reason="submit_button_dead_zone",
    expected_outcome="Page navigates to /confirmation after clicking Submit.",
    actual_outcome="Page stays on /checkout. 3 identical parse states observed.",
    goal="complete checkout",
    step=7,
    # force=True,   # bypass the Gatekeeper (developer assertion)
)
if issue["status"] == "rejected":
    print(issue["msg"])     # explains exactly what proof is still missing

# Pure in-memory snapshot — no network call.
status = aliax.billing_status()
# {"locked": bool, "reason": str | None, "balance": int | None, "credits_exhausted": bool}

# Force a fresh server-side balance check (one 0-cost telemetry ping):
await aliax.refresh_billing_status()

# Deterministic teardown — closes the httpx pool.
await aliax.aclose()

# Or use as an async context manager:
async with Aliax(api_key=os.environ["ALIAX_API_KEY"]) as aliax:
    ctx = await aliax.parse_ui(page)
    # ... aclose() runs at scope exit, even on exception

from aliax import (
    AliaxError,             # base — catch this for any SDK error
    AliaxInvalidKeyError,   # 401 — terminal; SDK locks permanently
    AliaxOutOfCreditsError, # 402 after grace loop; self-heals on top-up
)

try:
    await aliax.parse_ui(page)
except AliaxOutOfCreditsError as e:
    print("balance:", e.balance)        # last known balance (negative or None)
except AliaxInvalidKeyError as e:
    print("reason:", e.reason)          # mint a new key
except AliaxError:
    # any other SDK-originated failure
    raise

Aliax does not auto-detect what counts as sensitive on your app. You tell it which elements to blur — it blurs them before the screenshot and strips their text from the element list. Nothing else leaves the page.

Layer 1 — visual blur

Pass CSS selectors to redact_selectors. Anything matching is given a heavy CSS filter: blur(12px) for the duration of the screenshot, then unblurred.

aliax = Aliax(
    api_key=os.environ["ALIAX_API_KEY"],
    redact_selectors=[
        "[data-pii]",
        ".customer-email",
        "input[name='credit_card']",
    ],
)

Layer 2 — text strip on the spatial map

The DOM mapper runs the same selector list against label values before they leave the page, replacing them with [REDACTED]. Bboxes and roles are preserved so annotators still see the layout.

The DOM mapper is bundled inside the SDK package — it never fetches anything from a CDN at runtime. This is deliberate: a silent update mid-run could change what the element list looks like and corrupt your captures. What you install is what runs.

Active-degradation notice

On construction (when check_for_updates=True), the SDK pings GET /v1/version with a 1 s timeout. If you're behind, a single warning fires — no exceptions, no retry storm.

Pinning

pip install aliax

Pin in production. The mapper output schema is part of the SDK version contract, so upgrading mid-training would invalidate cached spatial maps.

The moment a human annotates a capture, it becomes a complete training example: the wrong action the agent took, and the correct action the human provided — side by side, ready to export.

1. The capture lands in Captures with status: unannotated.
2. The inspector renders the screenshot with a pulsing red-X at the agent's last_attempted_action.target_x/y (scaled by viewport.dpr), the goal, and ai_thoughts in the side panel.
3. The platform cross-references the red-X against the spatial map and surfaces which element the agent actually hit — usually a wrapper, backdrop, or empty space.
4. The annotator taps the element the agent should have hit. We snap by point-in-bbox + z-index and prompt for the action type (CLICK, TYPE, HOVER, SCROLL).
5. A corrected_action JSON blob is written back alongside the original last_attempted_action, and status flips to annotated.

Output shape (DPO-ready)

{
  "task_id": "8f3c2a64-9c2e-4a3b-9d20-1f4e1c6e2b51",
  "original_goal": "Add the Medium size to cart",
  "ai_thoughts": "Targeting the green dismiss button at top-right of modal.",
  "last_attempted_action": {
    "action": "CLICK",
    "target_x": 905,
    "target_y": 150,
    "selector": ".modal-close",
    "element_id": "el_88",
    "failure_reason": "element_not_found",
    "step": 3
  },
  "ai_attempt_hit_element_id": "el_88",
  "corrected_action": {
    "action": "CLICK",
    "target_element_id": "el_14",
    "target_role": "button",
    "target_label": "Close",
    "value": null,
    "annotated_by": "auth0|6710f1c2…",
    "annotated_at": "2026-06-22T14:03:11.482Z"
  },
  "viewport": { "width": 1920, "height": 1080, "dpr": 2 }
}

The wrong action (last_attempted_action) and the correct action (corrected_action) are in the same row. Export this directly into your model fine-tuning pipeline.

Aliax is pay-as-you-go. No plans, no seats, no expiration. One parse_ui() call = one credit.

How the SDK sees your balance

The first parse_ui() after process start carries a telemetry ping; the response includes your current balance and stamps the in-memory BillingStatus the SDK exposes via aliax.billing_status() / aliax.billingStatus().

• While balance > 0 telemetry is fire-and-forget — sub-millisecond overhead per parse.
• When balance ≤ 0 the SDK awaits the next ping so a top-up flips the kill-switch off within one parse, and a still-empty wallet rejects deterministically with AliaxOutOfCreditsError instead of leaking work.
• refresh_billing_status() / refreshBillingStatus() sends a zero-cost ping if you want to re-check without parsing.

• Write a specific goal. "Add the Medium size to cart" beats "shop". The annotator sees this verbatim — a vague goal becomes a vague training row.
• Capture at the snag, not after. The popup or layout shift that broke your agent must still be on screen. Don't page.reload() first.
• Always send thoughts + last_attempted_action. A capture without them is half a training row. The chain-of-thought is the prompt; the attempted action is the rejected response. Skipping either drops the capture out of your DPO pipeline.
• Coordinates are CSS pixels, viewport-relative. Use the same coordinate space as page.mouse.click(x, y). DPR is captured automatically — don't pre-multiply by it.
• Don't dedupe client-side. Two captures of the same popup from different sessions are signal, not noise — a model that fails twice fails consistently.
• Cap your QPS per agent. aliax.capture_failure() uploads a screenshot — if you're firing >5/s per agent process, you're probably looping on a recoverable error.