Step 1/5
Status: READY
Coordinate Resolution & DPI
Retina 2.0x
VLM models infer normalized bounds [0..1000]. On macOS, Cocoa points correspond directly to UI points, whereas Wayland requires surface-local fractional scaling matrices to prevent misclicks.
// macOS Point Mapping:
const ptX = (normX / 1000) * logicalWidth; // 1280 pt
const ptY = (normY / 1000) * logicalHeight; // 800 pt
CGEventPost(kCGHIDEventTap, clickEvent);
Accessibility vs VLM Grounding
AXUIElement API
macOS provides synchronous, high-fidelity AXUIElement trees. Linux AT-SPI over DBus introduces IPC latency, prompting vision-first screenshot grounding via multi-modal LLMs.
// Tree Inspection:
role: "AXButton",
title: "Confirm Merge PR",
frame: { x: 42, y: 198, w: 140, h: 32 },
actionable: true