DSH iOS Simulator

A live, interactive iOS Simulator inside a DeepSeek Harness conversation — plus your real iPhone over USB.
_{22 agent tools • live MJPEG sidebar panel • simulator & real iPhone over USB • list/feed row actions • SwiftUI preview hot reload}

_{npm: @zseven-w/dsh-ios · Current plugin release: 0.1.0-rc.10 · Tested with DSH 0.1.5-rc.1}

English · 简体中文 · 繁體中文 · 日本語 · 한국어 · Français · Español · Deutsch · Português · Русский · हिन्दी · Türkçe · ไทย · Tiếng Việt · Bahasa Indonesia

_{npm: @zseven-w/dsh-ios · Current plugin release: 0.1.0-rc.10 · Tested with DSH 0.1.5-rc.1}

_{A real iPhone driven from inside a DSH conversation — the agent's tool calls on the left, the live device panel on the right}

Why DSH iOS Simulator

DSH iOS Simulator gives the agent a real iOS Simulator inside the conversation — and gives you the pixels. The agent can boot a device, build and run an Xcode project or Swift package, drive the UI by accessibility identity or by OCR text, read unified logs, and inspect processes, backtraces, and leaks, while a live stream of the device renders in a persistent sidebar panel where you can tap, drag, rotate, and press Home directly on the video. The same verbs also work on a real iPhone connected over USB: the plugin builds and launches WebDriverAgent on the phone, tunnels its control and screen ports over loopback, and streams the device into the same panel, cards, and tools. No image blocks, no screen-recording files: visual bytes reach the UI only through signed, expiring URLs served by the DSH webserver.

Tools

All 22 tools are registered on every host and return plain JSON — visual bytes reach the UI only through presentationMeta + signed routes, never as image blocks. Simulator udids route through simctl/serve-sim; physical-device udids route through WebDriverAgent automatically. On non-macOS hosts (or when serve-sim is unresolvable) the tools stay registered but fail with an explanatory error; the one exception is ios_sim_preview status, which truthfully reports { running: false } on any host.

Core simulator tools

UI-tree tools (AXe-backed)

List & feed rows

List/feed apps aggregate each item into one accessibility cell whose label carries the whole summary and all its counters ("57 回复。18 喜欢。592 次查看") — there are no per-control child buttons to match, and the row cells only surface at a deep snapshot. These two tools expose that structure as rows and act inside a row.

OCR tools (Vision)

Logs tool

Preview tool

Debug tools

Display surfaces

• Sidebar panel — “iOS 模拟器”. The live view lives in a persistent right-hand panel (a fixed dock that pushes the conversation aside, or a centered overlay on narrow viewports). It renders the live MJPEG stream and accepts click-to-tap and drag-to-gesture directly on the video, with an icon toolbar (Home, screenshot, rotate, refresh) whose buttons carry hover tooltips. Size controls offer 适应 (fit to panel width), 50–125% zoom of the device's logical width, and S / M / L presets that size the device's short side (portrait width; landscape scales so the device keeps its physical size). Frame styles are 无框 / 边框 / 真机框 (frameless / bezel / realistic device shell) with a proportional corner radius. When the device rotates to landscape the panel auto-widens to a comfortable size and restores your width when it rotates back — a manual drag during the stint always wins. The left-edge handle drags the panel wider/narrower (max 960 px; double-click resets to the default width). When a USB-connected iPhone is the stream target, the same panel shows the phone's WebDriverAgent MJPEG stream with the same controls.
• Compact conversation cards. Tool results render as one-line cards with no inline imagery: the unified “iOS 模拟器” title, an action sub-label (Boot / Screenshot / Interact / Build & Run / Start WebDriverAgent), the device name, a status badge, and an “open in sidebar” cue. Clicking the row opens the panel; clicks on buttons, links, or the live frame itself never trigger it.
• Status capsule above the input. While the panel is closed and a stream is online, a small green-dot pill (<device> · 实时) appears above the composer and opens the panel when clicked. It is session-gated: it renders and polls only while the current conversation has mounted simulator results, and stops when you switch to a session without them.
• Standard mode and Code Mode. Standard sessions use the host-projected presentationMeta. Nested Code Mode (PTC) dispatches never carry meta, so the client reconstructs the identical meta from the durable result JSON — the panel, the cards, and the capsule work in both modes.

Security

• The browser never talks to serve-sim's port. Every byte crosses the DSH webserver origin through plugin-owned /_dsh/dsh-ios/* routes: /stream/<token> (MJPEG proxy), /screenshot/<token> (cached PNG), /ws?token=… (HID control relay), plus /grant, /capture, and /status endpoints.
• Tokens are HMAC-SHA256 capabilities (base64url(payload).base64url(mac)) expiring within 10 minutes, signed with a per-DSH-home key (<DSH_HOME>/cache/dsh-ios/stream-access.key, 0600, created atomically).
• Every route applies a loopback/trusted transport fence before any capability is consulted: loopback peer address, loopback Host (DNS-rebinding rejected), and Fetch-Metadata/Origin checks. The screenshot route serves only files inside the plugin cache directory (symbolic links refused, realpath containment).
• serve-sim runs as a foreground child on loopback only, in a dedicated port range (3181–3244), so a user's own serve-sim on port 3100 is never touched; --host is never used.
• Real-device transport — the WebDriverAgent control (REST, device port 8100) and screen (MJPEG, port 9100) tunnels are loopback usbmux forwards over the USB link; they sit behind the same signed-route fence, and the browser still only ever talks to the DSH webserver origin.
• Orphan adoption/reclaim — if a previous DSH host was killed ungracefully and its serve-sim helper survived, the same device is adopted (the orphan's handshake is authoritative); a stale helper squatting on a slot for a different device is reclaimed via serve-sim -k and relaunched once.
• Keep-alive + idle stop — a crashed stream restarts in the background (~5 s delay); with zero consumers the stream stops automatically after 5 minutes. Intentional stops are never fought. (The real-device runner is exempt from the idle reaping on purpose: restarting it costs a multi-minute xcodebuild rebuild.)

Requirements

• macOS with full Xcode — not just Command Line Tools. xcodebuild, xcrun simctl, and the simulator runtimes all ship with Xcode.
• At least one iOS Simulator runtime installed in Xcode.
• DSH ≥ 0.1.0-rc.6 with the web bundle for the panel. Headless profiles work too: all 22 tools function normally, just without the live view.
• Non-macOS hosts: the plugin loads and all 22 tools register, but every call returns an explanatory error (iOS Simulator requires macOS with Xcode …).
• serve-sim ships as an npm dependency of this plugin, so it resolves locally on real installs; the npx -y serve-sim fallback covers development trees (first use needs network).
• AXe (optional — only the AXe-backed tools need it: ios_sim_ui_tree / ios_sim_tap_element, plus ios_sim_ui_rows / ios_sim_tap_row on a simulator): brew install cameroncooke/axe/axe, or let the plugin auto-download the pinned release (v1.8.0, SHA-256 verified) into ~/Library/Caches/dsh-ios/bin. DSHPLUGIN_IOS_AXE_BIN overrides resolution; DSHPLUGIN_IOS_AXE_OFFLINE=1 disables the download.
• Vision OCR (optional — only ios_sim_find_text / ios_sim_tap_text need it): the plugin compiles its bundled assets/ocr.swift with swiftc on first use into ~/Library/Caches/dsh-ios/bin/ocr (zh-Hans + en-US recognition).
• lldb attach needs macOS Developer Mode: run sudo DevToolsSecurity -enable once. Until then ios_sim_backtrace uses Xcode's sample engine (non-suspending) and ios_sim_leaks degrades with the enable hint.
• Real iPhone — a USB-connected iPhone with the screen unlocked (WebDriverAgent cannot start on a locked screen; consider Auto-Lock: Never), a data-capable USB cable (a Wi-Fi-only pairing cannot carry the port forward), Developer Mode enabled on the device, a WebDriverAgent checkout at ~/Library/Caches/dsh-ios/wda/src (the plugin builds its WebDriverAgentRunner scheme from there — it never downloads or clones anything). The first WDA build installs a signed WebDriverAgentRunner: trust its certificate on the device when prompted, and re-run ios_real_start_wda when the free-team signing profile expires (7-day lifetime). Signing-team order is DSHPLUGIN_IOS_TEAM_ID > a team the Xcode account can provision for (preferring one with a matching Apple Development keychain identity, then the personal team) > the first keychain identity's team > the legacy default. Set DSHPLUGIN_IOS_WDA_BUNDLE_ID to override the runner bundle identifier when WdaOptions.bundleId is not supplied.

Install into DSH

dsh plugin --profile web add @zseven-w/dsh-ios@latest
dsh web

Quick start

A typical first conversation:

1. Discover devices — “List the available simulators.” → ios_sim_devices.
2. Boot — “Boot the iPhone 17 Pro.” → ios_sim_boot. The stream starts and the “iOS 模拟器” panel opens: the device is live in the sidebar. (Click any simulator card row, or the status pill above the input, to reopen it.)
3. Tap on the video — tap or drag directly on the panel; or let the agent drive the UI: “Open Settings, then tap General.” → ios_sim_interact (or ios_sim_ui_tree + ios_sim_tap_element for identity-based taps; ios_sim_find_text + ios_sim_tap_text for text-based taps; ios_sim_ui_rows + ios_sim_tap_row for list/feed apps).
4. Build & run your app — “Build and run /path/to/MyApp.xcodeproj.” → ios_sim_build_run. A full build takes minutes; when it lands, the app launches on the simulator and you watch it live in the panel.
5. Preview hot reload — “Show the SwiftUI previews of /path/to/MyPackage.” → ios_sim_preview start. Edit a source file and the preview hot-swaps in the running simulator within ~2–5 s — no relaunch.
6. Drive a real iPhone — plug the phone in over USB (data cable), unlock it, then “Start WebDriverAgent on the phone.” → ios_real_start_wda. The panel switches to the phone's live stream and every tool accepts its realDevices udid; when a call fails, read the coded reason from the panel's status (device-locked, cert-untrusted, profile-expired, tunnel-failed, device-unplugged).

Troubleshooting

• Backtrace uses sample instead of lldb, or leaks complains about restricted inspection — macOS Developer Mode is off. Run sudo DevToolsSecurity -enable once and retry. The tools degrade cleanly until then: ios_sim_backtrace falls back to Xcode's sample (symbolized, non-suspending) and ios_sim_leaks reports the enable hint.
• ios_sim_ui_tree / ios_sim_tap_element need AXe — install it with brew install cameroncooke/axe/axe, or let the plugin download the pinned release on first use (needs network to github.com). The error message always carries the full install hint; DSHPLUGIN_IOS_AXE_BIN=/path/to/axe overrides resolution. The row tools (ios_sim_ui_rows / ios_sim_tap_row) need AXe on a simulator too.
• ios_sim_find_text / ios_sim_tap_text report the OCR helper is missing — first use compiles the bundled assets/ocr.swift with swiftc (needs Xcode) into ~/Library/Caches/dsh-ios/bin/ocr; the error carries the exact path and hint.
• ios_sim_ui_rows finds no rows — the result says why: depth too shallow (raise max_depth; on a phone each deeper snapshot costs ~15–25 s), not a list screen, or genuinely no accessibility information after a deep read. A shallow read is never misreported as missing accessibility.
• ios_sim_leaks on iOS 26.2 simulators — on iOS 26.2 runtimes, Xcode's leaks can fail to inspect simulator processes with fatal diagnostics such as Failed to get DYLD info or minimal-corpse errors, even with Developer Mode enabled. The tool degrades cleanly: you get the raw diagnostic, the target is always verified resumed, and nothing hangs. There is no plugin-side fix — when it bites, try mode: "memgraph" or a different runtime.
• Real-device calls fail with a coded status — the panel's status names the cause instead of guessing: device-locked (unlock the phone; it recovers by itself), cert-untrusted (trust the WebDriverAgent certificate on the device), profile-expired (free-team signing lasts 7 days — re-run ios_real_start_wda to rebuild), tunnel-failed (check the USB link/usbmuxd), device-unplugged (use a data-capable USB cable — Wi-Fi-only pairing cannot carry the port forward).
• The stream stops by itself — that is the idle policy, not a crash: with zero consumers (panel closed, no cards mounted, no route active) the stream stops after 5 minutes and restarts on the next tool call or panel open. A crashed stream restarts in the background within ~5 seconds.

Development

pnpm install
pnpm run build      # host tsc + client bundle → lib/
pnpm run typecheck

The scripts/ smoke tests exercise the built lib/ (macOS only for the parts that boot a simulator or talk to a USB-connected phone; set DSH_IOS_SMOKE_SKIP_SIM=1 to skip those parts):

Ecosystem

• DSH Android — a live Android emulator or USB device inside the conversation, driven entirely through adb
• DSH Crew — dispatch work to DSH agents from Claude Code / Codex
• DSH Noema — long-term memory for DSH
• DSH OpenPencil — inspect and edit .op design documents inside a conversation

Credits & License

• serve-sim — Evan Bacon — the simulator streaming engine (Apache-2.0; bundled runtime dependency).
• AXe — Cameron Cooke — the accessibility CLI behind the UI-tree tools (MIT).
• WebDriverAgent — the WebDriver server the plugin builds and launches on real devices (BSD-licensed).
• Architecture inspired by Codex's “Build iOS Apps” plugin; the SwiftUI preview engine is a clean-room reimplementation of the publicly documented approach — no Codex code is copied.
• See THIRD_PARTY_NOTICES.md for the full notices.

License: MIT