Orchestral

A TypeScript orchestration layer for media generation — text-to-image, image-to-video, text-to-speech, speech recognition — built for local-first, BYOK apps: capability routing, opt-in semantic fallback, and an asset-handle protocol.

You describe what a step needs (text-to-image, image-to-video, automatic-speech-recognition, …), not which model to call. Orchestral routes that capability to a model you supplied, retries inside the router, knows the semantically equivalent paths a capability without a model could degrade through (reporting them on failure by default; redirecting automatically is opt-in), and passes generated media between steps as opaque handles the host resolves. It ships no provider SDK and no API keys — calling a model is a ~15-line adapter you write, over whichever SDK you already use. Everything runs in your process; there is no hosted control plane.

How it relates to the AI SDK / LangChain

Different layers, and Orchestral expects you to keep using the others:

• A provider SDK (the Vercel AI SDK, an official vendor SDK) owns one model call — auth, request shape, streaming, transport retries. Your Orchestral call adapter is usually a dozen lines over one of these; the examples here use the AI SDK's generateImage.
• An agent framework (LangChain / LangGraph, the AI SDK's own tool loop) owns the generic tool loop — planning, memory, a graph of steps. Orchestral's agent patterns delegate the loop to whichever one you inject.
• Orchestral owns what neither covers for media: routing a capability rather than a model id, declaring semantically equivalent fallback paths for when no model serves that capability, and threading generated assets between steps as handles instead of raw ids.

Quickstart

Not on npm yet — 0.1.0 publishes shortly. Until then, clone this repo and run the examples.

npm install @orchestral/core @orchestral/runtime @orchestral/patterns zod

Two optional packages sit on top: @orchestral/discovery (the BM25 search behind a find_pattern tool) and @orchestral/agent (the two agent patterns). Neither pulls in a provider SDK.

zod v4 (>=4.3 <5) is a peer dependency: pattern input/output schemas are zod schemas on the public API, so your app and Orchestral must share one zod instance.

Two runnable hosts live in this repo:

• examples/atomic-hello-world — one atomic text-to-image dispatch, with the whole provider bridge in src/ai-sdk-wiring.ts.
• examples/agent-hello-world — an agent-kind dispatch: an LLM tool-loop that picks and runs patterns.

pnpm install
pnpm --filter atomic-hello-world start   # needs OPENAI_API_KEY
pnpm --filter atomic-hello-world test    # no key: same wiring, mock model

Minimal example

Registry, model bridge, router, runtime, one dispatch — the whole surface:

import {
  PatternRegistry,
  InMemoryJobStore,
  createDefaultCapabilityRouter,
  type DispatchContext,
  type DispatchResult,
  type ModelCapability,
} from '@orchestral/core'
import { InlineRuntime } from '@orchestral/runtime'
import { createTextToImagePattern } from '@orchestral/patterns'
import { generateImage } from 'ai'
import { openai } from '@ai-sdk/openai'

const registry = new PatternRegistry()
registry.add(createTextToImagePattern())

// The seam you write: your provider SDK behind a ModelCapability envelope.
const model: ModelCapability = {
  capabilities: ['text-to-image'],
  provider: 'openai',
  modelId: 'gpt-image-1',
  inputs: ['text'],
  outputs: ['image'],
  tags: [],
  source: 'user',
  async call<I, O>(input: I, ctx: DispatchContext): Promise<DispatchResult<O>> {
    const startedAt = Date.now()
    const { images } = await generateImage({
      model: openai.image('gpt-image-1'),
      prompt: (input as { prompt: string }).prompt,
      abortSignal: ctx.signal,
    })
    const assets = images.map((img, i) => ({
      assetId: `img-${i}`,
      modality: 'image' as const,
      url: `data:${img.mediaType ?? 'image/png'};base64,${img.base64}`,
    }))
    const output = {
      modality: 'image' as const,
      assets,
      cost: 0,
      latencyMs: Date.now() - startedAt,
      model: 'openai:gpt-image-1',
      provider: 'openai',
    }
    return { output: output as O }
  },
}

const runtime = new InlineRuntime({
  store: new InMemoryJobStore(),
  registry,
  router: createDefaultCapabilityRouter({
    getModels: (cap) => (cap === 'text-to-image' ? [model] : []),
  }),
})

const job = await runtime.submitJob({
  patternId: 'text-to-image',
  input: { prompt: 'a watercolour fox in a misty forest' },
})
console.log(job.status, job.output) // 'done'  { modality: 'image', assets: [...] }

The annotated version of the same wiring is in packages/orchestral-core/README.md.

What's in the box

@orchestral/patterns ships 25 patterns: 10 atomic ones (one per capability — text-to-image, image-to-video, text-to-speech, automatic-speech-recognition, …) and 15 meta pipelines with their prompts inlined (storyboarding, script planning, idea-to-video, best-of-N image selection, …). The 2 agent patterns (an orchestrator and a long-form-video director) live in the optional @orchestral/agent package.

The full table — kind, input slots, outputs, and the host operations each pattern expects you to supply — is generated from the built package: pattern catalog.

The three seams

A host adopts Orchestral by satisfying three injection points. Two are implementations you swap; the third is the call adapter:

Agent patterns add a fourth seam, AgentRunImpl, which drives the inner LLM tool-loop. It is @alpha, and it ships nothing for the same reason ModelCapability.call does not: picking an agent framework is the host's call. examples/agent-hello-world wires one over the AI SDK's tool loop in ~150 lines — copy it and swap in whatever loop you already run.

Packages

All packages are Apache-2.0. The @orchestral/* packages are released together on one version line; @orchestral/dsh-plugin versions independently.

Honest limitations

This is 0.x. Each package README states its own edges rather than hiding them:

• Agent resume is lossy. The transcript stores a step projection, not raw provider messages: tool_use pairing and reasoning blocks are gone on resume — runtime § Resume fidelity.
• No durable queue. InlineRuntime runs a job in the caller's tick, and abandonOrphanedJobs() marks jobs a dead process left behind as stale; ctx.askUser parks in memory only — runtime § Runtime semantics worth knowing.
• Deliverable metas need a multimedia backend you supply. Six MetaCommonDeps operations (ffmpeg-shaped: concat, subtitles, background audio, …) are specified but not implemented here — patterns § Deliverable metas.
• One shipped fallback, and taking it is opt-in. image-to-image → caption → re-render is the only Alternative in the first-party catalog, and InlineRuntime defaults to failing with the applicable paths listed rather than redirecting through them (alternatives: 'auto' turns redirects on) — runtime § Alternative fallback is opt-in, core § Semantic fallback.

Versioning

0.x: minor versions may contain breaking changes, patch versions never do. The @orchestral/* packages share one version line and are published together — pin the exact set you tested against. @orchestral/dsh-plugin versions independently against its dev-preview host. The 1.0 line will follow semver strictly.

Repository layout

packages/orchestral-core/        @orchestral/core
packages/orchestral-discovery/   @orchestral/discovery
packages/orchestral-patterns/    @orchestral/patterns
packages/orchestral-runtime/     @orchestral/runtime
packages/orchestral-agent/       @orchestral/agent
packages/orchestral-dsh-plugin/  @orchestral/dsh-plugin (independent version line)
examples/                        runnable hosts, ~50 lines each
scripts/smoke-dist.mjs           executes the built dist bundles end to end

Development

pnpm install
pnpm build        # tsdown bundle + tsc declarations + api-extractor rollup
pnpm test         # vitest, all packages and examples
pnpm typecheck
pnpm api:check    # public .d.ts surface vs the committed etc/*.api.md report
pnpm smoke:dist   # build, then run the published dist bundles end to end
pnpm docs:catalog # regenerate the pattern catalog table from the built dist

pnpm api:check failing means the public API changed. Review the diff, run pnpm api:update, and commit the updated report alongside the change.

License

Apache-2.0 — see LICENSE and NOTICE.

@orchestral/patterns contains prompt text derived from a third-party MIT project; the affected constants and the upstream license text are listed in packages/orchestral-patterns/CREDITS.md.

Contributing and security

See CONTRIBUTING.md. For vulnerabilities, please use the private channel in SECURITY.md rather than a public issue.