Compare commits

...

8 Commits

Author SHA1 Message Date
realZachi
b38d44a5d4 fix(ai): defer vision fallback, harden image cache keys, and remount model browser per provider
Address CodeRabbit review on #18:
- resolve the OpenCode vision describer only once a prepared step contains
  an image, so text-only models run image-free requests without a fallback
- relay tool preview images whenever the vision fallback is active, not
  only for chat-dialect models
- hash full binary payloads for image description cache keys so distinct
  same-length images are described separately
- build the unavailable-transport error from the selected provider
- remount AiCatalogModelBrowser per provider so catalogs never leak
  between OpenRouter and OpenCode gateways
- bound the OpenCode catalog fetch with a 10s timeout and guard the
  catalog-revision state update after unmount
- correct the worker key-handling doc and the vision-default comment

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 00:52:44 +02:00
realZachi
de3a8a6d77 fix: allow images to be used in devices 2026-08-19 23:21:37 +02:00
realZachi
038aa5bc3b refactor(ai): enhance OpenCode model handling and proxy functionality 2026-08-19 23:20:39 +02:00
realZachi
e47c316e47 fix(ai): surface run status events and cache image descriptions across steps
OpenCode text-only models describe screenshots with a vision model before
generating. The modal dropped 'status' run events, so the band showed
'Starting up …' for the whole describe phase, and each tool-loop step
re-described every image from scratch because the description cache lived
only one prepare call.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-19 21:21:32 +02:00
realZachi
8581cc5e24 Merge branch 'main' into cursor/opencode-provider-c505
# Conflicts:
#	src/ai/runner.ts
2026-08-19 20:36:11 +02:00
Cursor Agent
53bba774c0
chore(deps): bump nanoid override to 3.3.18
Clear GHSA-2v37-7h3g-55p8 so bun audit can pass. The previous 3.3.17 pin is now reported as vulnerable.

Co-authored-by: mahmoud <realZachi@users.noreply.github.com>
2026-08-15 17:35:37 +00:00
Cursor Agent
84f6c8f00f
fix(ai): keep OpenCode Zen/Go path prefixes when proxying
Absolute /v1/... suffixes were resolved against the OpenCode origin and dropped /zen and /zen/go. Join relative to those prefixes in the worker and Vite rewrites.

Co-authored-by: mahmoud <realZachi@users.noreply.github.com>
2026-08-15 17:34:31 +00:00
Cursor Agent
d71b7e0a7d
feat(ai): add OpenCode Zen and Go providers
Add OpenCode as a key-based provider with shared Zen/Go credentials, a same-origin CORS proxy, runtime model catalogs, and a vision-model fallback that describes screenshots for text-only coding models.

Co-authored-by: mahmoud <realZachi@users.noreply.github.com>
2026-08-15 17:33:15 +00:00
38 changed files with 2395 additions and 168 deletions

View File

@ -7,6 +7,7 @@ VITE_GOOGLE_GENERATIVE_AI_API_KEY=
VITE_ALIBABA_API_KEY=
VITE_OPENAI_API_KEY=
VITE_ANTHROPIC_API_KEY=
VITE_OPENCODE_API_KEY=
VITE_XAI_API_KEY=
# Developer-only: write one JSON file per AI run to ./ai-logs/.

View File

@ -27,7 +27,7 @@ Use Bun for all package and script operations. Never use npm or manually edit `b
- Keep editor mutations immutable; undo/redo snapshots depend on it.
- This public repository owns only the editor. The production marketing site is maintained privately and deployed separately; never add its source, styles, assets, or production-only configuration here.
- Never commit secrets, private screenshots, unlicensed assets, PSD/PSB files, or generated `dist/`.
- AI provider keys are entered in the browser (stored in `localStorage`) with optional `.env.local` `VITE_*` fallback for local dev; only Moonshot uses the local CORS proxy. Never commit `.env.local` or deploy a keyed build.
- AI provider keys are entered in the browser (stored in `localStorage`) with optional `.env.local` `VITE_*` fallback for local dev; Moonshot and OpenCode use a same-origin CORS proxy. Never commit `.env.local` or deploy a keyed build.
- Use shadcn/Base UI and Hugeicons for editor chrome. Do not add a parallel UI or token system.
- Update generated `src/components/ui/` files only with the shadcn CLI.
- Use Conventional Commit messages and keep commits independently coherent.

View File

@ -23,7 +23,7 @@ Projects live entirely in your browser — no Shotluma account or project backen
- **AI that designs, not renders** — the agent works through editor tool calls, so its output is a normal project: every element it creates is selectable and editable, and you can revise a single screen with the magic-cursor action instead of regenerating everything.
- **Use your ChatGPT plan** — connect the Codex app or CLI to the hosted editor without giving Shotluma an API key.
- **Bring your own model** — also works with Moonshot, Google, Qwen, OpenAI, Anthropic, xAI, and OpenRouter; pick provider and model per run.
- **Bring your own model** — also works with Moonshot, Google, Qwen, OpenAI, Anthropic, xAI, OpenRouter, and OpenCode Zen/Go; pick provider and model per run.
- **Full screenshot sets** — design a multi-screen story on portrait artboards, not one image at a time.
- **Direct manipulation** — typography, gradients, shapes, images, mockups, alignment, stacking, and spacing are all hand-editable, with or without AI.
- **Perspective device mockups** — drop a raw screenshot into a device frame with correct 3D geometry.
@ -84,6 +84,7 @@ VITE_ALIBABA_API_KEY=
VITE_OPENAI_API_KEY=
VITE_ANTHROPIC_API_KEY=
VITE_XAI_API_KEY=
VITE_OPENCODE_API_KEY=
```
Restart the dev server after changing `.env.local` keys. Pick the provider and model in the generation dialog — models with configurable reasoning also expose their effort levels there. If a selected provider has no key, the dialog offers **Enter API key** and disables generation until one is saved.
@ -91,7 +92,7 @@ Restart the dev server after changing `.env.local` keys. Pick the provider and m
How it works, and what to know:
- Codex runs go from the hosted browser to the paired loopback connector and then to Codex App Server. The connector exposes only the account, model, thread, turn, and Shotluma dynamic-tool RPC surface; Codex gets a read-only empty workspace with network access disabled.
- Google, Qwen, OpenAI, Anthropic, xAI, and OpenRouter are called directly from the browser via the AI SDK. Moonshot works only on localhost through the local `/api/moonshot` CORS proxy.
- Google, Qwen, OpenAI, Anthropic, xAI, and OpenRouter are called directly from the browser via the AI SDK. Moonshot works only on localhost through the local `/api/moonshot` CORS proxy. OpenCode Zen and Go use a same-origin `/api/opencode` proxy (Vite locally, Cloudflare Worker when hosted) because those APIs do not answer browser CORS preflight. Models without native vision send screenshots and previews through a second OpenCode vision model first.
- When you start a run, your description and selected screenshots are sent to the selected provider or through Codex. Normal editing, persistence, and export never call any AI service. Provider limits or charges may apply.
- Browser-stored keys are visible to same-origin JavaScript by design. Optional `VITE_*` keys are exposed only by the local dev server; production builds replace every provider env key with an empty value. Never commit `.env.local` (see [Self-hosting](#self-hosting)).
- Prompts can be written in any language; generated canvas copy and summaries are in English.
@ -109,7 +110,7 @@ For debugging, set `SHOTLUMA_AI_LOGGING=true` in `.env.local` to write one JSON
| `bun run test` | Run the Vitest suite once |
| `bun run test:coverage` | Tests with coverage thresholds |
| `bun run build` | Type-check and build the production bundle |
| `bun run preview` | Serve the production bundle locally, including the Moonshot proxy |
| `bun run preview` | Serve the production bundle locally, including the Moonshot and OpenCode proxies |
| `bun run audit` | Check dependencies for known vulnerabilities |
| `bun run check` | All required local and CI quality gates |
@ -117,13 +118,13 @@ Changes to rendering, export, or AI behavior also need manual verification in th
## Self-hosting
`bun run build` produces a static app in `dist/`, including the downloadable Codex connector at `codex/shotluma-codex-bridge.mjs`. Provider env values are stripped from the bundle, so it can be hosted as a static site while users connect Codex or supply their own direct-provider keys in the browser.
`bun run build` produces a static app in `dist/`, including the downloadable Codex connector at `codex/shotluma-codex-bridge.mjs`. Provider env values are stripped from the bundle, so it can be hosted as a static site while users connect Codex or supply their own direct-provider keys in the browser. OpenCode on a hosted origin also needs the `/api/opencode` Worker in `scripts/shotluma-worker.ts` (the official `app.shotluma.com` deploy includes it).
The official editor is deployed at `https://app.shotluma.com`. The marketing
site at `https://shotluma.com` is maintained and deployed separately; its source
and production configuration are not part of this repository.
Codex connection prompts pair against the deployment's own origin. Google, Qwen, OpenAI, Anthropic, xAI, and OpenRouter work with browser-entered keys. Moonshot
Codex connection prompts pair against the deployment's own origin. Google, Qwen, OpenAI, Anthropic, xAI, OpenRouter, and OpenCode work with browser-entered keys. Moonshot
remains local-only; offering it on a hosted deployment requires an authenticated
proxy. A hosted workflow with shared credentials likewise needs a backend or
short-lived credential exchange.

View File

@ -32,8 +32,8 @@ Maintainers will acknowledge reports and coordinate a fix and disclosure on a be
- `.env.local` is ignored by Git and must remain untracked.
- Production builds replace provider env keys with empty values. Do not remove this build boundary or publish a manually altered keyed bundle.
- Uploaded images and projects stay in IndexedDB during normal editing.
- Screenshots selected for an AI run are sent to the selected provider. Google, Qwen, OpenAI, Anthropic, and xAI are contacted directly; Moonshot works only on localhost through the same-origin local CORS proxy. Opt-in overlay generation contacts OpenAI separately.
- Screenshots selected for an AI run are sent to the selected provider. Google, Qwen, OpenAI, Anthropic, and xAI are contacted directly; Moonshot works only on localhost through the same-origin local CORS proxy. OpenCode Zen and Go use the same-origin `/api/opencode` proxy. Opt-in overlay generation contacts OpenAI separately.
- Users are responsible for protecting provider keys and usage quotas.
- A hosted deployment can use browser-entered keys with the direct providers. A deployment that supplies shared credentials or offers Moonshot requires a separate authenticated backend design.
- A hosted deployment can use browser-entered keys with the direct providers and OpenCode. A deployment that supplies shared credentials or offers Moonshot requires a separate authenticated backend design.
If a key is accidentally exposed, revoke it at the provider immediately, remove it from the current files, and treat it as compromised even if the Git commit is later rewritten.

View File

@ -17,10 +17,11 @@ Optional AI path
Browser → local Shotluma bridge → Codex App Server → ChatGPT plan
Browser → AI SDK provider → Google / Qwen / OpenAI / Anthropic / xAI / OpenRouter
Browser → /api/moonshot/* → Vite proxy → Moonshot API
Browser → /api/opencode/* → Vite or Cloudflare Worker proxy → OpenCode Zen/Go
Browser → /api/ai-run-logs → Vite → ./ai-logs/*.json (developer opt-in)
```
The core editor has no backend requirement. Hosted users can connect a Codex app or CLI already signed in with ChatGPT on the same computer; `app.shotluma.com` remains the application host. The browser talks to a paired loopback bridge, and the bridge talks to Codex App Server over stdio. The alternative AI path uses bring-your-own keys entered in the browser (`localStorage`), with optional `.env.local` `VITE_*` fallback served only during local development. Production builds replace all provider env values with empty strings. Google, Qwen, OpenAI, Anthropic, xAI, and OpenRouter are called directly; Moonshot is enabled only on localhost and uses a same-origin proxy because its API does not support the required browser CORS flow.
The core editor has no backend requirement. Hosted users can connect a Codex app or CLI already signed in with ChatGPT on the same computer; `app.shotluma.com` remains the application host. The browser talks to a paired loopback bridge, and the bridge talks to Codex App Server over stdio. The alternative AI path uses bring-your-own keys entered in the browser (`localStorage`), with optional `.env.local` `VITE_*` fallback served only during local development. Production builds replace all provider env values with empty strings. Google, Qwen, OpenAI, Anthropic, xAI, and OpenRouter are called directly; Moonshot is enabled only on localhost and uses a same-origin proxy because its API does not support the required browser CORS flow. OpenCode Zen and Go share one API key and use the same-origin `/api/opencode` proxy (Vite in local dev, Cloudflare Worker when hosted) for the same CORS reason.
This public repository contains and deploys only the editor at
`app.shotluma.com`. The marketing site at `shotluma.com` has a separate private
@ -139,6 +140,11 @@ The AI feature is split into explicit layers:
| `src/html-slide/device-attributes.ts` | `<shotluma-device>` placeholder parsing |
| `src/ai/prompt-caching.ts` | OpenAI, Anthropic, and Qwen prompt-cache routing and breakpoints |
| `src/ai/chat-tool-images.ts` | Image relay for providers whose chat tool results support text only |
| `src/ai/describe-images.ts` | Replace image parts with text descriptions for models without vision |
| `src/ai/opencode-models.ts` | Runtime OpenCode Zen/Go catalog, vision flags, and fallback picker |
| `src/ai/opencode-client.ts` | OpenAI-compat chat client against the same-origin OpenCode proxy |
| `scripts/opencode-proxy.ts` | URL rewrite and header allowlist for Zen/Go |
| `scripts/shotluma-worker.ts` | Hosted Cloudflare Worker that proxies `/api/opencode/*` |
| `src/ai/tools.ts` | Tool composition and generate/edit tool boundary |
| `src/ai/*-tools.ts`, `src/ai/*-tool.ts` | Focused slide, media, text, update, inspection, and overlay-asset tools |
| `src/ai/chroma-key.ts` | Pure soft chroma-key pixel math for overlay cutouts |
@ -168,13 +174,15 @@ Rich text is built from structured highlight input. The model never writes raw H
The generate dialog covers input only. Once a run starts it closes and `src/components/AiRunBand.tsx` takes over as a band floating over the canvas: assistant prose and reasoning at reading size (accumulated per sentence by `src/ai/run-narration.ts`), a screen rail fed by the model's `declare_plan` call and reconciled against reality by `src/ai/run-plan.ts`, and an action count in place of a tool log. The band is also where a finished or failed run reports its result, so no run hands control back to the centered dialog.
`src/ai/provider-catalog.ts` owns the selectable providers, models, transport metadata, and model-specific reasoning-effort choices. `src/ai/provider-config.ts` resolves keys from browser `localStorage` (API keys dialog) with optional `.env.local` `VITE_*` fallback, and `src/ai/runner.ts` routes either to the Codex bridge or a lazily loaded native AI SDK provider. The runner passes portable efforts through the AI SDK's standardized `reasoning` option so each native provider can map it to its own API; OpenAI GPT-5.6 and Moonshot/Kimi K3 `max` go through OpenAI-compat `providerOptions` because the shared option has no `max`. The generate modal presents one model picker grouped by provider, with reasoning effort as secondary chips when the model supports it, a Codex connection tutorial, and an API keys dialog for key-based providers. Requests go directly from the local browser to Google, Alibaba/Qwen, OpenAI, Anthropic, xAI, or OpenRouter. Moonshot uses the OpenAI chat provider through the only Vite proxy route.
`src/ai/provider-catalog.ts` owns the selectable providers, models, transport metadata, and model-specific reasoning-effort choices. `src/ai/provider-config.ts` resolves keys from browser `localStorage` (API keys dialog) with optional `.env.local` `VITE_*` fallback, and `src/ai/runner.ts` routes either to the Codex bridge or a lazily loaded native AI SDK provider. The runner passes portable efforts through the AI SDK's standardized `reasoning` option so each native provider can map it to its own API; OpenAI GPT-5.6 and Moonshot/Kimi K3 `max` go through OpenAI-compat `providerOptions` because the shared option has no `max`. The generate modal presents one model picker grouped by provider, with reasoning effort as secondary chips when the model supports it, a Codex connection tutorial, and an API keys dialog for key-based providers. Requests go directly from the local browser to Google, Alibaba/Qwen, OpenAI, Anthropic, xAI, or OpenRouter. Moonshot uses the OpenAI chat provider through the local Vite proxy. OpenCode Zen and Go share one API key, call chat-completions through `/api/opencode`, and describe images with a second vision model when the selected model has no native vision.
Codex pairing is deliberately not OAuth inside Shotluma. The setup prompt downloads the connector asset emitted with the hosted app, asks Codex to inspect it, and starts it as a detached user process. The browser stores a random pairing token and the exact app origin. The connector binds only `127.0.0.1:47447`, stores its matching configuration with user-only permissions, requires both the exact `Origin` header and bearer pairing token, and never reads Codex authentication files. It starts `codex app-server` over stdio and allowlists only account/rate-limit/model reads plus ephemeral thread and turn methods. Thread and turn requests are overwritten to use an empty read-only workspace, no approvals, and no network. The model can change the canvas only through the same `createEditorTools` dynamic tools used by direct providers.
The bridge uses Codex App Server's experimental dynamic-tool API. Keep bridge and web-client versions compatible, fail closed on unknown RPC methods, and retain the API-key provider path as a fallback while this integration is experimental.
OpenRouter is the one provider with a runtime model catalog: `src/ai/openrouter-models.ts` fetches the public OpenRouter `/models` endpoint (no key required), keeps only models that accept image input and support tool calling, caches the result in `localStorage` for an hour, and registers it with the static catalog so selections resolve everywhere. The curated OpenRouter shortlist in `provider-catalog.ts` doubles as the offline fallback, and the model picker adds a searchable browser over the fetched catalog. Requests use the OpenAI chat provider against `https://openrouter.ai/api/v1`.
OpenRouter is one runtime model catalog: `src/ai/openrouter-models.ts` fetches the public OpenRouter `/models` endpoint (no key required), keeps only models that accept image input and support tool calling, caches the result in `localStorage` for an hour, and registers it with the static catalog so selections resolve everywhere. The curated OpenRouter shortlist in `provider-catalog.ts` doubles as the offline fallback, and the model picker adds a searchable browser over the fetched catalog. Requests use the OpenAI chat provider against `https://openrouter.ai/api/v1`.
OpenCode Zen and Go are the other runtime catalogs: `src/ai/opencode-models.ts` fetches each gateway's `/v1/models` list through the same-origin proxy, keeps every tool-capable model including those without vision, and caches for an hour. Text-only models stay selectable; `src/ai/describe-images.ts` sends screenshots and slide previews to a vision model from the same gateway (defaulting to a cheap one such as MiMo V2.5 Free on Zen or MiMo V2.5 on Go) and replaces the image parts with the description before the coding model runs.
Keys are intentionally browser-visible and stored unencrypted. Never commit `.env.local` or weaken the production-build stripping of provider env values. Use dedicated keys with restrictive quotas. A hosted deployment that must hide or share credentials needs an authenticated backend.

View File

@ -13,11 +13,24 @@ Generate mode does not spend a model turn re-reading canvas state already presen
Preview images are rendered at the editor's native 330 px width rather than upscaled, use OpenAI's low-detail vision path because exact geometry arrives separately, and are capped at three attempts per generated screen (four for edit mode's before/after checks). Mutations return only warnings involving the changed element; the preview is the one deduplicated slide-wide validation point. Measurement boxes remain rounded to one decimal (`src/ai/measure.ts`).
OpenAI Responses, Anthropic, Google, xAI, and Codex accept image-bearing tool results directly. OpenAI-compatible Chat Completions and Alibaba do not: before Moonshot, OpenRouter, or Qwen steps, `src/ai/chat-tool-images.ts` moves image files out of the tool-result JSON and into a following multimodal user message. Never allow base64 image data to be serialized as model-visible text.
OpenAI Responses, Anthropic, Google, xAI, and Codex accept image-bearing tool results directly. OpenAI-compatible Chat Completions and Alibaba do not: before Moonshot, OpenRouter, Qwen, or chat-completions OpenCode steps, `src/ai/chat-tool-images.ts` moves image files out of the tool-result JSON and into a following multimodal user message. Never allow base64 image data to be serialized as model-visible text.
OpenCode models without native vision then go through `src/ai/describe-images.ts`: each remaining image part is sent to a vision-capable model on the same Zen or Go gateway, and the coding model receives the text description instead. Do not send raw images to a text-only OpenCode model.
## Security and provider boundary
The browser uses the AI SDK's native Google, Alibaba/Qwen, OpenAI, Anthropic, and xAI providers directly. OpenRouter is called directly through the OpenAI chat provider against `https://openrouter.ai/api/v1`. Moonshot uses the OpenAI chat provider through the local `/api/moonshot` CORS proxy.
The browser uses the AI SDK's native Google, Alibaba/Qwen, OpenAI, Anthropic, and xAI providers directly. OpenRouter is called directly through the OpenAI chat provider against `https://openrouter.ai/api/v1`. Moonshot uses the OpenAI chat provider through the local `/api/moonshot` CORS proxy. OpenCode Zen and Go go through `/api/opencode/{zen|go}/v1` because `opencode.ai` does not answer browser CORS preflight.
OpenCode is a multi-dialect gateway, not an OpenAI-compatible one. Each model answers on exactly one endpoint, and calling any other returns a 400 or an `Endpoint is unavailable` 5xx rather than routing itself. `src/ai/opencode-dialects.ts` owns the mapping and `src/ai/opencode-client.ts` builds the matching provider:
| Dialect | Endpoint | Provider | Families |
| --- | --- | --- | --- |
| `responses` | `/v1/responses` | `@ai-sdk/openai` Responses | `gpt-*`, `grok-*`, `muse-spark-*` |
| `messages` | `/v1/messages` | `@ai-sdk/anthropic` | `claude-*`, `qwen*`, `minimax-*` on Go only |
| `google` | `/v1/models/<id>:…` | `@ai-sdk/google` | `gemini-*` |
| `chat` | `/v1/chat/completions` | `@ai-sdk/openai` chat | everything else, including `minimax-*` on Zen |
Match `src/ai/opencode-dialects.ts` against the endpoint tables at <https://opencode.ai/docs/zen> and <https://opencode.ai/docs/go> whenever OpenCode adds a family; prefixes are matched rather than exact ids because `/v1/models` only reports ids and keeps growing. Chat completions is the default for an unrecognised id because every open-weight family lands there. The proxy header allowlist in `scripts/opencode-proxy.ts` must keep carrying `x-api-key`, `anthropic-version`, `anthropic-beta`, and `x-goog-api-key` alongside `authorization`, or the non-OpenAI dialects lose their credentials.
Codex is a separate no-API-key transport for hosted users. The browser pairs with `scripts/shotluma-codex-bridge.ts` on `127.0.0.1`; that process launches `codex app-server` over stdio and uses Codex's existing ChatGPT authentication. The user stays on `app.shotluma.com` and never needs a repository checkout or local Vite server.
@ -29,7 +42,7 @@ Codex is a separate no-API-key transport for hosted users. The browser pairs wit
- `scripts/codex-bridge-asset-plugin.ts` must emit the reviewed connector source at `/codex/shotluma-codex-bridge.mjs` in both development and production builds. The copied setup prompt must use the current browser origin so normal hosted and self-hosted deployments pair only with themselves.
- Open the ChatGPT desktop app with the documented `codex://threads/new?prompt=` deep link and URI-encode the complete setup prompt. Deep links prefill but do not submit the composer, so the tutorial must tell the user to press Send and retain the manual copy fallback.
- Keys are intentionally visible to same-origin browser JavaScript. Never commit `.env.local`, reuse a shared production credential, or weaken the production build boundary. Recommend dedicated keys with restrictive quotas.
- Do not add a proxy for providers whose browser API supports the required CORS flow. Moonshot is available only on localhost through `/api/moonshot`; a hosted deployment that offers Moonshot or must hide keys needs a separate authenticated backend design.
- Do not add a proxy for providers whose browser API supports the required CORS flow. Moonshot is available only on localhost through `/api/moonshot`; a hosted deployment that offers Moonshot or must hide keys needs a separate authenticated backend design. OpenCode is the exception that is proxied in production as well: `scripts/shotluma-worker.ts` forwards `/api/opencode/*` to `opencode.ai`, copying only the allowlisted auth headers upstream without storing or logging the caller's key.
- Never return secrets or raw data URLs in model-visible state.
- Keep uploads browser-local except for screenshots and app logos explicitly included in an AI run.
- In generate mode, collect app name and app logo separately from the app description and screenshots. Pass the name and logo asset id through the user message and attach the logo image. Instruct the model to place the logo with `add_image` in element runs and with an `<img src="asset:…">` tag in HTML screen runs — never as a device screenshot.
@ -108,4 +121,6 @@ Reasoning **visibility** is a separate switch from reasoning **effort**, and pro
This matters more than a missing nicety because `prompt.ts` tells the model not to narrate its work turn by turn. Reasoning is therefore the only live prose during a run, and for a model that streams none, the tool-activity line in the run band is the only remaining signal — it is load-bearing, not decoration.
OpenRouter is the only provider with a runtime model catalog (`src/ai/openrouter-models.ts`): the public `/models` endpoint is fetched without a key, filtered to models that accept image input and support tool calling — both are hard requirements for a Shotluma run — and cached in `localStorage` for an hour. Loaded models are registered with `setDynamicOpenRouterModels` so catalog lookups resolve them; unknown OpenRouter ids synthesize a minimal option instead of throwing, while every other provider keeps failing fast on unknown models. Keep the curated OpenRouter shortlist in `provider-catalog.ts` working as the fetch fallback, keep the vision+tools filter intact, and offer reasoning effort only when the fetched model advertises the `reasoning` parameter.
OpenRouter is a runtime model catalog (`src/ai/openrouter-models.ts`): the public `/models` endpoint is fetched without a key, filtered to models that accept image input and support tool calling — both are hard requirements for a Shotluma run — and cached in `localStorage` for an hour. Loaded models are registered with `setDynamicProviderModels` so catalog lookups resolve them; unknown OpenRouter ids synthesize a minimal option instead of throwing, while every other static provider keeps failing fast on unknown models. Keep the curated OpenRouter shortlist in `provider-catalog.ts` working as the fetch fallback, keep the vision+tools filter intact, and offer reasoning effort only when the fetched model advertises the `reasoning` parameter.
OpenCode Zen and Go are also runtime catalogs (`src/ai/opencode-models.ts`). Fetch `/v1/models` through the same-origin proxy, keep text-only tool models, and mark them `supportsVision: false`. The generate modal exposes a vision-fallback picker for those models. Unknown OpenCode ids synthesize like OpenRouter.

View File

@ -17,6 +17,7 @@ describe('client provider environment', () => {
expect(values.VITE_MOONSHOT_API_KEY).toBe('moonshot-public')
expect(values.VITE_OPENAI_API_KEY).toBe('openai-public')
expect(values.VITE_XAI_API_KEY).toBe('')
expect(values.VITE_OPENCODE_API_KEY).toBe('')
expect(values).not.toHaveProperty('MOONSHOT_API_KEY')
})
@ -28,6 +29,7 @@ describe('client provider environment', () => {
VITE_OPENAI_API_KEY: '',
VITE_ANTHROPIC_API_KEY: '',
VITE_XAI_API_KEY: '',
VITE_OPENCODE_API_KEY: '',
})
expect(createClientProviderDefinitions(environment, 'build')).toMatchObject({
'import.meta.env.VITE_MOONSHOT_API_KEY': '""',

View File

@ -5,6 +5,7 @@ export const AI_PROVIDER_ENV_VARIABLES = [
'VITE_OPENAI_API_KEY',
'VITE_ANTHROPIC_API_KEY',
'VITE_XAI_API_KEY',
'VITE_OPENCODE_API_KEY',
] as const
export type AiProviderEnvironmentVariable = typeof AI_PROVIDER_ENV_VARIABLES[number]

View File

@ -0,0 +1,138 @@
import { describe, expect, it, vi } from 'vitest'
import {
copyOpencodeProxyHeaders,
isOpencodeProxyMethod,
proxyOpencodeRequest,
rewriteOpencodeProxyUrl,
} from './opencode-proxy'
describe('OpenCode CORS proxy rewrite', () => {
it('maps Zen and Go prefixes onto the public OpenCode origins', () => {
expect(rewriteOpencodeProxyUrl(
new URL('https://app.shotluma.com/api/opencode/zen/v1/chat/completions'),
)?.toString()).toBe('https://opencode.ai/zen/v1/chat/completions')
expect(rewriteOpencodeProxyUrl(
new URL('https://app.shotluma.com/api/opencode/go/v1/models?foo=1'),
)?.toString()).toBe('https://opencode.ai/zen/go/v1/models?foo=1')
expect(rewriteOpencodeProxyUrl(
new URL('https://app.shotluma.com/api/opencode/zen'),
)?.toString()).toBe('https://opencode.ai/zen/')
expect(rewriteOpencodeProxyUrl(
new URL('https://app.shotluma.com/api/moonshot/v1/chat/completions'),
)).toBeNull()
expect(rewriteOpencodeProxyUrl(
new URL('https://app.shotluma.com/api/opencode/go'),
)?.toString()).toBe('https://opencode.ai/zen/go/')
})
it('maps every dialect path each gateway serves', () => {
expect(rewriteOpencodeProxyUrl(
new URL('https://app.shotluma.com/api/opencode/go/v1/responses'),
)?.toString()).toBe('https://opencode.ai/zen/go/v1/responses')
expect(rewriteOpencodeProxyUrl(
new URL('https://app.shotluma.com/api/opencode/zen/v1/messages'),
)?.toString()).toBe('https://opencode.ai/zen/v1/messages')
expect(rewriteOpencodeProxyUrl(
new URL('https://app.shotluma.com/api/opencode/zen/v1/models/gemini-3.7-flash:streamGenerateContent'),
)?.toString()).toBe('https://opencode.ai/zen/v1/models/gemini-3.7-flash:streamGenerateContent')
})
it('does not treat a zen path as go', () => {
expect(rewriteOpencodeProxyUrl(
new URL('https://app.shotluma.com/api/opencode/zen/v1/models'),
)?.toString()).toBe('https://opencode.ai/zen/v1/models')
})
})
describe('OpenCode CORS proxy headers and methods', () => {
it('forwards only the allowlisted request headers', () => {
const headers = copyOpencodeProxyHeaders(new Headers({
authorization: 'Bearer test-key',
'content-type': 'application/json',
accept: 'text/event-stream',
cookie: 'session=secret',
host: 'app.shotluma.com',
}))
expect(Object.fromEntries(headers.entries())).toEqual({
authorization: 'Bearer test-key',
'content-type': 'application/json',
accept: 'text/event-stream',
})
})
it('forwards the Anthropic and Google credentials the other dialects use', () => {
const headers = copyOpencodeProxyHeaders(new Headers({
'x-api-key': 'sk-test',
'anthropic-version': '2023-06-01',
'anthropic-beta': 'fine-grained-tool-streaming-2025-05-14',
'x-goog-api-key': 'goog-test',
origin: 'https://evil.example',
}))
expect(Object.fromEntries(headers.entries())).toEqual({
'x-api-key': 'sk-test',
'anthropic-version': '2023-06-01',
'anthropic-beta': 'fine-grained-tool-streaming-2025-05-14',
'x-goog-api-key': 'goog-test',
})
})
it('allows GET, POST, and OPTIONS only', () => {
expect(isOpencodeProxyMethod('get')).toBe(true)
expect(isOpencodeProxyMethod('POST')).toBe(true)
expect(isOpencodeProxyMethod('OPTIONS')).toBe(true)
expect(isOpencodeProxyMethod('PUT')).toBe(false)
})
it('answers OPTIONS without contacting OpenCode', async () => {
const fetchMock = vi.fn()
vi.stubGlobal('fetch', fetchMock)
const response = await proxyOpencodeRequest(new Request(
'https://app.shotluma.com/api/opencode/zen/v1/models',
{ method: 'OPTIONS' },
))
expect(response.status).toBe(204)
expect(fetchMock).not.toHaveBeenCalled()
vi.unstubAllGlobals()
})
it('rejects unsupported methods and unknown paths', async () => {
expect((await proxyOpencodeRequest(new Request(
'https://app.shotluma.com/api/opencode/zen/v1/models',
{ method: 'DELETE' },
))).status).toBe(405)
expect((await proxyOpencodeRequest(new Request(
'https://app.shotluma.com/api/moonshot/v1/models',
{ method: 'GET' },
))).status).toBe(404)
})
it('forwards GET and POST to the rewritten OpenCode origin', async () => {
const fetchMock = vi.fn().mockResolvedValue(new Response('ok', { status: 200 }))
vi.stubGlobal('fetch', fetchMock)
const getResponse = await proxyOpencodeRequest(new Request(
'https://app.shotluma.com/api/opencode/go/v1/models',
{ method: 'GET', headers: { authorization: 'Bearer test-key' } },
))
expect(getResponse.status).toBe(200)
expect(fetchMock).toHaveBeenCalledTimes(1)
const [getUrl, getInit] = fetchMock.mock.calls[0] as [URL, RequestInit]
expect(getUrl.toString()).toBe('https://opencode.ai/zen/go/v1/models')
expect(getInit.method).toBe('GET')
expect(getInit.body).toBeUndefined()
await proxyOpencodeRequest(new Request(
'https://app.shotluma.com/api/opencode/zen/v1/chat/completions',
{
method: 'POST',
headers: { 'content-type': 'application/json' },
body: '{"model":"kimi-k3"}',
},
))
const [, postInit] = fetchMock.mock.calls[1] as [URL, RequestInit & { duplex?: string }]
expect(postInit.method).toBe('POST')
expect(postInit.body).toBeDefined()
expect(postInit.duplex).toBe('half')
vi.unstubAllGlobals()
})
})

89
scripts/opencode-proxy.ts Normal file
View File

@ -0,0 +1,89 @@
export const OPENCODE_ZEN_UPSTREAM = 'https://opencode.ai/zen'
export const OPENCODE_GO_UPSTREAM = 'https://opencode.ai/zen/go'
const GO_PREFIX = '/api/opencode/go'
const ZEN_PREFIX = '/api/opencode/zen'
/**
* OpenCode serves each model on one of four dialects, so the proxy has to carry
* the auth and versioning headers of all of them: `authorization` for the
* OpenAI Responses and Chat Completions endpoints, `x-api-key` plus the
* `anthropic-*` headers for `/v1/messages`, and `x-goog-api-key` for the Gemini
* endpoint. Anything outside this list (cookies, origin, client hints) is
* dropped so the proxy stays a pure credential passthrough.
*/
export const OPENCODE_FORWARDED_HEADERS = [
'authorization',
'content-type',
'accept',
'x-api-key',
'anthropic-version',
'anthropic-beta',
'x-goog-api-key',
] as const
const joinUpstream = (origin: string, suffix: string): URL => {
const base = `${origin.replace(/\/$/u, '')}/`
const relative = suffix.replace(/^\//u, '')
return new URL(relative, base)
}
/**
* Map a same-origin OpenCode CORS-proxy path onto the Zen or Go upstream.
* `/api/opencode/go` is matched before `/api/opencode/zen` so the prefixes
* cannot overlap.
*/
export const rewriteOpencodeProxyUrl = (requestUrl: URL): URL | null => {
const path = requestUrl.pathname
const search = requestUrl.search
if (path === GO_PREFIX || path.startsWith(`${GO_PREFIX}/`)) {
const upstream = joinUpstream(OPENCODE_GO_UPSTREAM, path.slice(GO_PREFIX.length))
upstream.search = search
return upstream
}
if (path === ZEN_PREFIX || path.startsWith(`${ZEN_PREFIX}/`)) {
const upstream = joinUpstream(OPENCODE_ZEN_UPSTREAM, path.slice(ZEN_PREFIX.length))
upstream.search = search
return upstream
}
return null
}
export const copyOpencodeProxyHeaders = (source: Headers): Headers => {
const headers = new Headers()
for (const name of OPENCODE_FORWARDED_HEADERS) {
const value = source.get(name)
if (value) headers.set(name, value)
}
return headers
}
export const isOpencodeProxyMethod = (method: string): boolean => {
const normalized = method.toUpperCase()
return normalized === 'GET' || normalized === 'POST' || normalized === 'OPTIONS'
}
export const proxyOpencodeRequest = async (request: Request): Promise<Response> => {
if (request.method.toUpperCase() === 'OPTIONS') {
return new Response(null, { status: 204 })
}
if (!isOpencodeProxyMethod(request.method)) {
return new Response('Method not allowed', { status: 405 })
}
const upstream = rewriteOpencodeProxyUrl(new URL(request.url))
if (!upstream) {
return new Response('Not found', { status: 404 })
}
const method = request.method.toUpperCase()
// Node's fetch types require `duplex` when forwarding a stream body.
const init: RequestInit & { duplex?: 'half' } = {
method,
headers: copyOpencodeProxyHeaders(request.headers),
redirect: 'manual',
}
if (method !== 'GET' && method !== 'HEAD') {
init.body = request.body
init.duplex = 'half'
}
return fetch(upstream, init)
}

View File

@ -0,0 +1,23 @@
import { proxyOpencodeRequest, rewriteOpencodeProxyUrl } from './opencode-proxy'
type AssetFetcher = {
fetch: (request: Request) => Promise<Response>
}
export type ShotlumaWorkerEnv = {
ASSETS: AssetFetcher
}
/**
* Hosted same-origin CORS proxy for OpenCode Zen/Go. Browser keys stay on the
* request; the worker only rewrites the URL so app.shotluma.com can call an
* API that does not answer OPTIONS preflight.
*/
export default {
async fetch(request: Request, env: ShotlumaWorkerEnv): Promise<Response> {
if (rewriteOpencodeProxyUrl(new URL(request.url))) {
return proxyOpencodeRequest(request)
}
return env.ASSETS.fetch(request)
},
}

View File

@ -0,0 +1,216 @@
import { describe, expect, it } from 'vitest'
import {
collectMessageImages,
describeMessageImages,
imagePartKey,
replaceMessageImages,
toVisionFilePart,
} from './describe-images'
import type { FilePart, ModelMessage, UserModelMessage } from 'ai'
const filePart = (data: string, mediaType = 'image/png'): FilePart => ({
type: 'file',
mediaType,
data,
})
const userImages = (...parts: FilePart[]): UserModelMessage => ({
role: 'user',
content: [{ type: 'text', text: 'Screenshots:' }, ...parts],
})
describe('describeMessageImages', () => {
it('leaves messages without images unchanged', async () => {
const messages: ModelMessage[] = [{ role: 'user', content: 'hello' }]
expect(await describeMessageImages(messages, async () => 'nope')).toBe(messages)
expect(collectMessageImages(messages)).toEqual([])
})
it('replaces image files with text descriptions and skips tool payloads', async () => {
const shot = filePart('data:image/png;base64,abc')
const messages: ModelMessage[] = [
userImages(shot),
{
role: 'tool',
content: [{
type: 'tool-result',
toolCallId: 'call-1',
toolName: 'render_slide_preview',
output: {
type: 'content',
value: [{
type: 'file',
mediaType: 'image/jpeg',
data: { type: 'data', data: 'preview-bytes' },
}],
},
}],
},
]
const described = await describeMessageImages(messages, async (part) => {
return part.mediaType === 'image/jpeg' ? 'preview layout' : 'app screenshot'
})
expect(collectMessageImages(messages)).toHaveLength(1)
expect(described[0]).toMatchObject({
role: 'user',
content: [
{ type: 'text', text: 'Screenshots:' },
{ type: 'text', text: 'Image description:\napp screenshot' },
],
})
expect(described[1]).toBe(messages[1])
})
it('describes identical payloads once and reports failures as text', async () => {
const first = filePart('same-bytes')
const second = filePart('same-bytes')
const broken = filePart('broken')
let calls = 0
const described = await describeMessageImages(
[userImages(first, second, broken)],
async (part) => {
calls += 1
if (imagePartKey(part).includes('broken')) throw new Error('vision timeout')
return ' shared view '
},
)
expect(calls).toBe(2)
const content = (described[0] as UserModelMessage).content
if (!Array.isArray(content)) throw new Error('expected parts')
expect(content.map((part) => 'text' in part ? part.text : '')).toEqual([
'Screenshots:',
'Image description:\nshared view',
'Image description:\nshared view',
'[Image could not be described: vision timeout]',
])
})
it('reuses a shared cache across calls instead of re-describing per step', async () => {
const shot = filePart('step-bytes')
const cache = new Map<string, string>()
let calls = 0
const describeImage = async () => {
calls += 1
return 'described once'
}
const first = await describeMessageImages([userImages(shot)], describeImage, cache)
const second = await describeMessageImages([userImages(shot)], describeImage, cache)
expect(calls).toBe(1)
for (const described of [first, second]) {
const content = (described[0] as UserModelMessage).content
if (!Array.isArray(content)) throw new Error('expected parts')
expect(content[1]).toEqual({
type: 'text',
text: 'Image description:\ndescribed once',
})
}
})
it('keys distinct binary payloads separately, including ArrayBuffers', async () => {
const bytesPart = (bytes: number[]): FilePart => ({
type: 'file',
mediaType: 'image/png',
data: new Uint8Array(bytes),
})
// Same length and first byte must not collide.
expect(imagePartKey(bytesPart([9, 8, 7]))).not.toBe(imagePartKey(bytesPart([9, 0, 0])))
const bufferPart = (bytes: number[]): FilePart => ({
type: 'file',
mediaType: 'image/png',
data: new Uint8Array(bytes).buffer,
})
expect(imagePartKey(bufferPart([1, 2, 3]))).not.toBe(imagePartKey(bufferPart([1, 2, 4])))
// An ArrayBuffer keys by content, matching its Uint8Array view.
expect(imagePartKey(bufferPart([1, 2, 3]))).toBe(imagePartKey(bytesPart([1, 2, 3])))
const descriptions: string[] = []
const described = await describeMessageImages(
[userImages(bytesPart([9, 8, 7]), bytesPart([9, 0, 0]))],
async () => {
descriptions.push(`payload ${descriptions.length + 1}`)
return descriptions[descriptions.length - 1] ?? ''
},
)
expect(descriptions).toHaveLength(2)
expect(described[0]).toMatchObject({
content: [
{ type: 'text', text: 'Screenshots:' },
{ type: 'text', text: 'Image description:\npayload 1' },
{ type: 'text', text: 'Image description:\npayload 2' },
],
})
})
it('converts legacy image parts for the vision request', () => {
const converted = toVisionFilePart({
type: 'image',
image: 'abc',
mediaType: 'image/jpeg',
})
expect(converted).toEqual({
type: 'file',
mediaType: 'image/jpeg',
data: { type: 'data', data: 'abc' },
})
expect(replaceMessageImages([], new Map())).toEqual([])
})
it('covers image-part keys, empty descriptions, and tagged data wrappers', async () => {
const url = new URL('https://example.test/shot.png')
expect(toVisionFilePart({ type: 'file', mediaType: 'image/png', data: 'kept' })).toEqual({
type: 'file',
mediaType: 'image/png',
data: 'kept',
})
expect(toVisionFilePart({
type: 'image',
image: url,
})).toEqual({
type: 'file',
mediaType: 'image/png',
data: { type: 'url', url },
})
expect(imagePartKey({
type: 'file',
mediaType: 'image/webp',
data: { type: 'data', data: 'nested' },
})).toContain('nested')
expect(imagePartKey({
type: 'file',
mediaType: 'image/png',
data: { type: 'url', url },
})).toContain('example.test')
expect(imagePartKey({
type: 'file',
mediaType: 'image/png',
data: new Uint8Array([9, 8, 7]),
})).toContain('bytes:3:')
expect(imagePartKey({
type: 'image',
image: 'abc',
mediaType: 'image/jpeg',
})).toContain('abc')
const described = await describeMessageImages(
[{
role: 'user',
content: [
{ type: 'image', image: 'abc', mediaType: 'image/jpeg' },
{ type: 'file', mediaType: 'application/pdf', data: 'not-an-image' },
],
}],
async () => ' ',
)
expect(described[0]).toMatchObject({
content: [
{ type: 'text', text: '[Image description was empty]' },
{ type: 'file', mediaType: 'application/pdf', data: 'not-an-image' },
],
})
})
})

167
src/ai/describe-images.ts Normal file
View File

@ -0,0 +1,167 @@
import type { FilePart, ModelMessage, TextPart } from 'ai'
type LegacyImagePart = {
type: 'image'
image: unknown
mediaType?: string
}
type DescribeablePart = FilePart | LegacyImagePart
const isRecord = (value: unknown): value is Record<string, unknown> =>
typeof value === 'object' && value !== null
const isFilePart = (part: unknown): part is FilePart =>
isRecord(part) && part['type'] === 'file' && typeof part['mediaType'] === 'string'
const isLegacyImagePart = (part: unknown): part is LegacyImagePart =>
isRecord(part) && part['type'] === 'image'
const isImageMediaType = (mediaType: string | undefined): boolean =>
typeof mediaType === 'string' && mediaType.toLowerCase().startsWith('image/')
export const OPENCODE_VISION_PROMPT = `Describe this image for a designer who cannot see it.
Focus on layout, visual hierarchy, colors, typography, exact visible copy, UI elements, icons, device frames, and spacing.
Be concrete and complete. Do not suggest improvements.`
// Two FNV-1a variants over the full payload keep same-length images from
// sharing a cache key without pulling in a hashing dependency.
const hashBytes = (bytes: Uint8Array): string => {
let a = 0x811c9dc5
let b = 0x811c9dc5
for (const byte of bytes) {
a = Math.imul(a ^ byte, 0x01000193)
b = Math.imul(b ^ byte, 0x85ebca77)
}
return `${(a >>> 0).toString(16)}-${(b >>> 0).toString(16)}`
}
const serializeImageData = (data: unknown): string => {
if (typeof data === 'string') return data
if (data instanceof URL) return data.toString()
if (isRecord(data) && data['type'] === 'data') return serializeImageData(data['data'])
if (isRecord(data) && data['type'] === 'url') return serializeImageData(data['url'])
if (data instanceof Uint8Array) return `bytes:${data.byteLength}:${hashBytes(data)}`
if (data instanceof ArrayBuffer) return serializeImageData(new Uint8Array(data))
return JSON.stringify(data)
}
export const imagePartKey = (part: DescribeablePart): string => {
if (isFilePart(part)) {
return `${part.mediaType}:${serializeImageData(part.data)}`
}
return `image:${part.mediaType ?? 'image'}:${serializeImageData(part.image)}`
}
const isDescribeablePart = (part: unknown): part is DescribeablePart => {
if (isLegacyImagePart(part)) return true
return isFilePart(part) && isImageMediaType(part.mediaType)
}
const collectFromContent = (content: unknown, images: DescribeablePart[]): void => {
if (!Array.isArray(content)) return
for (const part of content as unknown[]) {
if (isDescribeablePart(part)) images.push(part)
}
}
export const collectMessageImages = (
messages: readonly ModelMessage[],
): readonly DescribeablePart[] => {
const images: DescribeablePart[] = []
for (const message of messages) {
if (message.role === 'tool') continue
collectFromContent(message.content, images)
}
return images
}
const toDescriptionPart = (description: string): TextPart => ({
type: 'text',
text: description.startsWith('[Image')
? description
: `Image description:\n${description}`,
})
const replaceContentImages = (
content: unknown,
descriptions: ReadonlyMap<string, string>,
): unknown => {
if (!Array.isArray(content)) return content
const parts = content as unknown[]
const next = parts.map((part) => {
if (!isDescribeablePart(part)) return part
const description = descriptions.get(imagePartKey(part))
if (description === undefined) return part
return toDescriptionPart(description)
})
const changed = next.some((part, index) => part !== parts[index])
return changed ? next : content
}
export const replaceMessageImages = (
messages: ModelMessage[],
descriptions: ReadonlyMap<string, string>,
): ModelMessage[] => {
if (descriptions.size === 0) return messages
const next = messages.map((message) => {
if (message.role === 'tool') return message
const content = replaceContentImages(message.content, descriptions)
if (content === message.content) return message
return { ...message, content } as ModelMessage
})
const changed = next.some((message, index) => message !== messages[index])
return changed ? next : messages
}
/**
* Replace image file parts with text descriptions so a text-only model can
* still use screenshots and slide previews. Descriptions are cached by image
* payload so repeated previews of the same bytes are described once. Pass a
* shared `cache` to reuse descriptions across calls — each tool-loop step
* re-prepares the full message history, and describe calls cost tens of
* seconds each, so re-describing per step would stall every step.
*/
export const describeMessageImages = async (
messages: ModelMessage[],
describeImage: (part: FilePart) => Promise<string>,
cache?: Map<string, string>,
): Promise<ModelMessage[]> => {
const images = collectMessageImages(messages)
if (images.length === 0) return messages
const descriptions = cache ?? new Map<string, string>()
for (const image of images) {
const key = imagePartKey(image)
if (descriptions.has(key)) continue
try {
const description = (await describeImage(toVisionFilePart(image))).trim()
descriptions.set(
key,
description.length > 0 ? description : '[Image description was empty]',
)
} catch (error) {
const detail = error instanceof Error ? error.message : String(error)
descriptions.set(key, `[Image could not be described: ${detail}]`)
}
}
return replaceMessageImages(messages, descriptions)
}
export const toVisionFilePart = (part: DescribeablePart): FilePart => {
if (isFilePart(part)) return part
const mediaType = part.mediaType ?? 'image/png'
const image = part.image
if (image instanceof URL) {
return { type: 'file', mediaType, data: { type: 'url', url: image } }
}
if (
typeof image === 'string'
|| image instanceof Uint8Array
|| image instanceof ArrayBuffer
) {
return { type: 'file', mediaType, data: { type: 'data', data: image } }
}
return { type: 'file', mediaType, data: { type: 'data', data: serializeImageData(image) } }
}

View File

@ -0,0 +1,17 @@
import { describe, expect, it } from 'vitest'
import { opencodeBaseUrl } from './opencode-client'
import { opencodeRequestModelId } from './opencode-models'
describe('OpenCode client routing', () => {
it('builds same-origin proxy base URLs for Zen and Go', () => {
expect(opencodeBaseUrl('opencode-zen', 'https://app.shotluma.com'))
.toBe('https://app.shotluma.com/api/opencode/zen/v1')
expect(opencodeBaseUrl('opencode-go', 'http://127.0.0.1:4173'))
.toBe('http://127.0.0.1:4173/api/opencode/go/v1')
})
it('sends the upstream model id rather than the catalog prefix', () => {
expect(opencodeRequestModelId('opencode-go/glm-5.3', 'glm-5.3')).toBe('glm-5.3')
expect(opencodeRequestModelId('opencode-zen/kimi-k3')).toBe('kimi-k3')
})
})

71
src/ai/opencode-client.ts Normal file
View File

@ -0,0 +1,71 @@
import { generateText, type FilePart } from 'ai'
import { OPENCODE_VISION_PROMPT, toVisionFilePart } from './describe-images'
import { opencodeModelDialect } from './opencode-dialects'
import type { OpencodeProviderId } from './opencode-models'
export const opencodeBaseUrl = (
providerId: OpencodeProviderId,
origin: string,
): string =>
`${origin}/api/opencode/${providerId === 'opencode-go' ? 'go' : 'zen'}/v1`
/**
* Build a language model for one OpenCode model on the dialect that gateway
* serves it on. All four dialects share the same proxied base URL; only the
* provider package and the path it appends differ.
*/
export const createOpencodeModel = async (options: {
providerId: OpencodeProviderId
modelId: string
apiKey: string
origin?: string
}) => {
const { providerId, modelId, apiKey } = options
const baseURL = opencodeBaseUrl(providerId, options.origin ?? window.location.origin)
switch (opencodeModelDialect(providerId, modelId)) {
case 'responses': {
const { createOpenAI } = await import('@ai-sdk/openai')
return createOpenAI({ apiKey, baseURL }).responses(modelId)
}
case 'messages': {
const { createAnthropic } = await import('@ai-sdk/anthropic')
return createAnthropic({ apiKey, baseURL })(modelId)
}
case 'google': {
const { createGoogle } = await import('@ai-sdk/google')
return createGoogle({ apiKey, baseURL })(modelId)
}
case 'chat': {
const { createOpenAI } = await import('@ai-sdk/openai')
return createOpenAI({ apiKey, baseURL }).chat(modelId)
}
}
}
export const createOpencodeImageDescriber = (options: {
providerId: OpencodeProviderId
visionModelId: string
apiKey: string
signal?: AbortSignal
}): ((part: FilePart) => Promise<string>) => {
let modelPromise: ReturnType<typeof createOpencodeModel> | null = null
return async (part) => {
modelPromise ??= createOpencodeModel({
providerId: options.providerId,
modelId: options.visionModelId,
apiKey: options.apiKey,
})
const { text } = await generateText({
model: await modelPromise,
messages: [{
role: 'user',
content: [
{ type: 'text', text: OPENCODE_VISION_PROMPT },
toVisionFilePart(part),
],
}],
...(options.signal ? { abortSignal: options.signal } : {}),
})
return text
}
}

View File

@ -0,0 +1,61 @@
import { describe, expect, it } from 'vitest'
import {
opencodeDialectNeedsChatToolImageRelay,
opencodeModelDialect,
opencodeSelectionDialect,
} from './opencode-dialects'
describe('OpenCode dialect routing', () => {
it('routes GPT, Grok, and Muse Spark to the Responses endpoint', () => {
expect(opencodeModelDialect('opencode-zen', 'gpt-5.6-terra')).toBe('responses')
expect(opencodeModelDialect('opencode-go', 'gpt-5.6-luna')).toBe('responses')
expect(opencodeModelDialect('opencode-go', 'grok-4.5')).toBe('responses')
expect(opencodeModelDialect('opencode-zen', 'grok-4.6')).toBe('responses')
expect(opencodeModelDialect('opencode-go', 'muse-spark-1.2-contributor'))
.toBe('responses')
})
it('routes Claude and Qwen to the Anthropic messages endpoint', () => {
expect(opencodeModelDialect('opencode-zen', 'claude-sonnet-5')).toBe('messages')
expect(opencodeModelDialect('opencode-zen', 'qwen3.7-max')).toBe('messages')
expect(opencodeModelDialect('opencode-go', 'qwen3.8-max')).toBe('messages')
})
it('routes Gemini to the Google endpoint', () => {
expect(opencodeModelDialect('opencode-zen', 'gemini-3.7-flash')).toBe('google')
})
it('keeps the open-weight families and unknown ids on chat completions', () => {
expect(opencodeModelDialect('opencode-zen', 'kimi-k3')).toBe('chat')
expect(opencodeModelDialect('opencode-go', 'mimo-v2.5')).toBe('chat')
expect(opencodeModelDialect('opencode-go', 'glm-5.3')).toBe('chat')
expect(opencodeModelDialect('opencode-zen', 'deepseek-v4-pro')).toBe('chat')
expect(opencodeModelDialect('opencode-zen', 'hy3-free')).toBe('chat')
expect(opencodeModelDialect('opencode-go', 'some-future-open-model')).toBe('chat')
})
it('splits MiniMax by gateway because Zen and Go disagree', () => {
expect(opencodeModelDialect('opencode-zen', 'minimax-m2.5')).toBe('chat')
expect(opencodeModelDialect('opencode-go', 'minimax-m3')).toBe('messages')
})
it('resolves a selection and ignores non-OpenCode providers', () => {
expect(opencodeSelectionDialect({
provider: 'opencode-go',
model: 'opencode-go/gpt-5.6-luna',
})).toBe('responses')
expect(opencodeSelectionDialect({
provider: 'opencode-zen',
model: 'opencode-zen/kimi-k3',
})).toBe('chat')
expect(opencodeSelectionDialect({ provider: 'openai', model: 'gpt-5.6-terra' }))
.toBeNull()
})
it('relays tool images only on the chat-completions dialect', () => {
expect(opencodeDialectNeedsChatToolImageRelay('chat')).toBe(true)
expect(opencodeDialectNeedsChatToolImageRelay('responses')).toBe(false)
expect(opencodeDialectNeedsChatToolImageRelay('messages')).toBe(false)
expect(opencodeDialectNeedsChatToolImageRelay('google')).toBe(false)
})
})

View File

@ -0,0 +1,74 @@
import {
isOpencodeProviderId,
opencodeRequestModelId,
type OpencodeProviderId,
} from './opencode-models'
import { getAiModel, type AiModelSelection } from './provider-catalog'
/**
* OpenCode Zen and Go are multi-dialect gateways: every model answers on exactly
* one endpoint, and the wrong one fails with a 400 or an "Endpoint is
* unavailable" 5xx instead of falling back. The table mirrors the endpoint
* columns published at https://opencode.ai/docs/zen and https://opencode.ai/docs/go.
*/
export type OpencodeDialect = 'responses' | 'messages' | 'google' | 'chat'
const PREFIX_DIALECTS: readonly (readonly [string, OpencodeDialect])[] = [
['gpt-', 'responses'],
['grok-', 'responses'],
['muse-spark-', 'responses'],
['claude-', 'messages'],
['qwen', 'messages'],
['gemini-', 'google'],
]
/** MiniMax is the one family the two gateways route differently. */
const GATEWAY_PREFIX_DIALECTS: Record<
OpencodeProviderId,
readonly (readonly [string, OpencodeDialect])[]
> = {
'opencode-zen': [['minimax-', 'chat']],
'opencode-go': [['minimax-', 'messages']],
}
const matchPrefix = (
modelId: string,
table: readonly (readonly [string, OpencodeDialect])[],
): OpencodeDialect | null => {
for (const [prefix, dialect] of table) {
if (modelId.startsWith(prefix)) return dialect
}
return null
}
/**
* Resolve the endpoint dialect for an upstream OpenCode model id. Prefixes are
* used rather than a fixed id list because `/v1/models` keeps gaining entries;
* the open-weight families (DeepSeek, GLM, Kimi, MiMo, Hy3, Laguna, Nemotron,
* Big Pickle, and the free tier) all answer on chat completions, which makes it
* the correct default for an unrecognised id.
*/
export const opencodeModelDialect = (
providerId: OpencodeProviderId,
modelId: string,
): OpencodeDialect =>
matchPrefix(modelId, GATEWAY_PREFIX_DIALECTS[providerId])
?? matchPrefix(modelId, PREFIX_DIALECTS)
?? 'chat'
/** The dialect a selection resolves to, or `null` for a non-OpenCode provider. */
export const opencodeSelectionDialect = (
selection: AiModelSelection,
): OpencodeDialect | null => {
if (!isOpencodeProviderId(selection.provider)) return null
const model = getAiModel(selection)
return opencodeModelDialect(
selection.provider,
opencodeRequestModelId(model.id, model.providerModelId),
)
}
/** Only the chat-completions dialect rejects images inside tool results. */
export const opencodeDialectNeedsChatToolImageRelay = (
dialect: OpencodeDialect,
): boolean => dialect === 'chat'

View File

@ -0,0 +1,171 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import {
formatOpencodeModelLabel,
loadAllOpencodeModels,
loadOpencodeModels,
opencodeCatalogId,
opencodeModelsUrl,
opencodeModelSupportsVision,
parseOpencodeModels,
pickOpencodeVisionModel,
readCachedOpencodeModels,
toOpencodeModelOption,
} from './opencode-models'
import {
getAiProvider,
getDynamicProviderModels,
setDynamicProviderModels,
} from './provider-catalog'
const createMemoryStorage = (entries: Record<string, string> = {}) => {
const store = new Map(Object.entries(entries))
return {
getItem: (key: string) => store.get(key) ?? null,
setItem: (key: string, value: string) => {
store.set(key, value)
},
}
}
afterEach(() => {
setDynamicProviderModels('opencode-zen', [])
setDynamicProviderModels('opencode-go', [])
vi.unstubAllGlobals()
})
describe('OpenCode model catalog', () => {
it('formats labels and marks known text-only models', () => {
expect(formatOpencodeModelLabel('gpt-5.6-luna')).toBe('GPT 5.6 Luna')
expect(formatOpencodeModelLabel('kimi-k3')).toBe('Kimi K3')
expect(formatOpencodeModelLabel('mimo-v2.5-pro')).toBe('MiMo V2.5 Pro')
expect(formatOpencodeModelLabel('big-pickle')).toBe('Big Pickle')
expect(opencodeModelSupportsVision('kimi-k3')).toBe(true)
expect(opencodeModelSupportsVision('glm-5.3')).toBe(false)
expect(opencodeCatalogId('opencode-go', 'glm-5.3')).toBe('opencode-go/glm-5.3')
})
it('parses the live /models list and keeps text-only tool models', () => {
const models = parseOpencodeModels('opencode-go', {
data: [
{ id: 'glm-5.3' },
{ id: 'kimi-k3' },
{ id: 'kimi-k3' },
{ id: '' },
'nope',
],
})
expect(models.map((model) => model.id)).toEqual([
'opencode-go/glm-5.3',
'opencode-go/kimi-k3',
])
expect(toOpencodeModelOption('opencode-go', 'glm-5.3')).toMatchObject({
providerModelId: 'glm-5.3',
supportsVision: false,
reasoningEfforts: ['low', 'medium', 'high'],
})
expect(toOpencodeModelOption('opencode-zen', 'kimi-k3').supportsVision).toBeUndefined()
expect(parseOpencodeModels('opencode-zen', null)).toEqual([])
})
it('picks a cheap vision fallback and honors an explicit preference', () => {
const models = [
toOpencodeModelOption('opencode-zen', 'glm-5.2'),
toOpencodeModelOption('opencode-zen', 'gpt-5-nano'),
toOpencodeModelOption('opencode-zen', 'claude-sonnet-5'),
]
expect(pickOpencodeVisionModel('opencode-zen', models)?.id).toBe(
'opencode-zen/gpt-5-nano',
)
expect(pickOpencodeVisionModel(
'opencode-zen',
models,
'opencode-zen/claude-sonnet-5',
)?.id).toBe('opencode-zen/claude-sonnet-5')
expect(pickOpencodeVisionModel(
'opencode-go',
[toOpencodeModelOption('opencode-go', 'glm-5.3')],
)).toBeUndefined()
expect(pickOpencodeVisionModel(
'opencode-go',
[toOpencodeModelOption('opencode-go', 'kimi-k3')],
)?.id).toBe('opencode-go/kimi-k3')
expect(formatOpencodeModelLabel('unknown-xyz')).toBe('Unknown Xyz')
expect(parseOpencodeModels('opencode-zen', { data: 'nope' })).toEqual([])
})
})
describe('loadOpencodeModels', () => {
it('serves a fresh cache without fetching', async () => {
const cachedModel = toOpencodeModelOption('opencode-zen', 'kimi-k3')
const storage = createMemoryStorage({
'shotluma-opencode-zen-models': JSON.stringify({
fetchedAt: 500,
models: [cachedModel],
}),
})
const fetchMock = vi.fn()
vi.stubGlobal('fetch', fetchMock)
const result = await loadOpencodeModels('opencode-zen', { now: 600, storage })
expect(result).toEqual({ models: [cachedModel], source: 'cache' })
expect(fetchMock).not.toHaveBeenCalled()
expect(getDynamicProviderModels('opencode-zen')).toEqual([cachedModel])
expect(readCachedOpencodeModels('opencode-zen', 600, storage)).toEqual([cachedModel])
})
it('fetches, caches, and registers the remote catalog', async () => {
const storage = createMemoryStorage()
vi.stubGlobal('fetch', vi.fn().mockResolvedValue({
ok: true,
json: () => Promise.resolve({ data: [{ id: 'glm-5.2' }, { id: 'kimi-k3' }] }),
}))
const result = await loadOpencodeModels('opencode-zen', { now: 42, storage })
expect(result.source).toBe('remote')
expect(result.models.map((model) => model.id)).toEqual([
'opencode-zen/glm-5.2',
'opencode-zen/kimi-k3',
])
expect(getDynamicProviderModels('opencode-zen')).toEqual(result.models)
})
it('falls back to the curated shortlist when the fetch fails', async () => {
vi.stubGlobal('fetch', vi.fn().mockResolvedValue({ ok: false, status: 503 }))
const result = await loadOpencodeModels('opencode-go', { now: 42, storage: null })
expect(result.source).toBe('fallback')
expect(result.models).toBe(getAiProvider('opencode-go').models)
})
it('ignores stale or invalid cache and coalesces remote loads', async () => {
const storage = createMemoryStorage({
'shotluma-opencode-zen-models': '{not-json',
'shotluma-opencode-go-models': JSON.stringify({
fetchedAt: 1,
models: [toOpencodeModelOption('opencode-go', 'kimi-k3')],
}),
})
expect(readCachedOpencodeModels('opencode-zen', 10, storage)).toBeNull()
expect(readCachedOpencodeModels('opencode-go', 60 * 60 * 1000 + 2, storage)).toBeNull()
expect(opencodeModelsUrl('opencode-zen', 'https://app.shotluma.com'))
.toBe('https://app.shotluma.com/api/opencode/zen/v1/models')
expect(opencodeModelsUrl('opencode-go', 'null'))
.toBe('https://opencode.ai/zen/go/v1/models')
const fetchMock = vi.fn().mockResolvedValue({
ok: true,
json: () => Promise.resolve({ data: [{ id: 'kimi-k3' }] }),
})
vi.stubGlobal('fetch', fetchMock)
const [first, second, all] = await Promise.all([
loadOpencodeModels('opencode-zen', { now: 42, storage: null }),
loadOpencodeModels('opencode-zen', { now: 42, storage: null }),
loadAllOpencodeModels({ now: 42, storage: null }),
])
expect(first.source).toBe('remote')
expect(second.source).toBe('remote')
expect(all).toHaveLength(2)
// One request per gateway, not one per caller.
expect(fetchMock).toHaveBeenCalledTimes(2)
})
})

361
src/ai/opencode-models.ts Normal file
View File

@ -0,0 +1,361 @@
import {
OPENROUTER_REASONING_EFFORTS,
getAiProvider,
setDynamicProviderModels,
type AiModelOption,
type AiProviderId,
} from './provider-catalog'
export type OpencodeGateway = 'zen' | 'go'
export const OPENCODE_PROVIDERS = [
'opencode-zen',
'opencode-go',
] as const satisfies readonly AiProviderId[]
export type OpencodeProviderId = (typeof OPENCODE_PROVIDERS)[number]
export const isOpencodeProviderId = (
value: unknown,
): value is OpencodeProviderId =>
value === 'opencode-zen' || value === 'opencode-go'
export const OPENCODE_MODELS_TTL_MS = 60 * 60 * 1000
// Bounds the catalog request so a stalled gateway falls back to the curated
// shortlist instead of leaving the model browser loading forever.
export const OPENCODE_MODELS_REQUEST_TIMEOUT_MS = 10_000
const STORAGE_KEYS: Record<OpencodeProviderId, string> = {
'opencode-zen': 'shotluma-opencode-zen-models',
'opencode-go': 'shotluma-opencode-go-models',
}
const GATEWAY_BY_PROVIDER: Record<OpencodeProviderId, OpencodeGateway> = {
'opencode-zen': 'zen',
'opencode-go': 'go',
}
/**
* Models.dev reports these OpenCode ids as text-only; the runner describes
* images for them instead of attaching them. Unknown live ids default to
* vision-capable so screenshots reach new models untouched — a wrong default
* fails loudly at the provider instead of silently flattening images to text.
*/
const OPENCODE_TEXT_ONLY_MODELS = new Set([
'big-pickle',
'deepseek-v4-flash',
'deepseek-v4-flash-free',
'deepseek-v4-pro',
'glm-5',
'glm-5.1',
'glm-5.2',
'glm-5.3',
'gpt-5.3-codex-spark',
'hy3',
'hy3-free',
'hy3-preview',
'laguna-s-2.1-free',
'minimax-m2.5',
'minimax-m2.7',
'nemotron-3-ultra-free',
'nemotron-3.5-lightning-free',
'qwen3.7-max',
])
const OPENCODE_LABEL_OVERRIDES: Record<string, string> = {
'big-pickle': 'Big Pickle',
hy3: 'Hy3',
'hy3-free': 'Hy3 Free',
'hy3-preview': 'Hy3 Preview',
'mimo-v2.5': 'MiMo V2.5',
'mimo-v2.5-free': 'MiMo V2.5 Free',
'mimo-v2.5-pro': 'MiMo V2.5 Pro',
'mimo-v2-omni': 'MiMo V2 Omni',
'mimo-v2-pro': 'MiMo V2 Pro',
}
const DEFAULT_VISION_MODEL_IDS: Record<OpencodeProviderId, readonly string[]> = {
'opencode-zen': [
'opencode-zen/mimo-v2.5-free',
'opencode-zen/gpt-5-nano',
'opencode-zen/gemini-3.5-flash-lite',
'opencode-zen/gpt-5.4-nano',
],
'opencode-go': [
'opencode-go/mimo-v2.5',
'opencode-go/gpt-5.6-luna',
'opencode-go/minimax-m3',
],
}
const TOKEN_LABELS: Record<string, string> = {
claude: 'Claude',
deepseek: 'DeepSeek',
gemini: 'Gemini',
glm: 'GLM',
gpt: 'GPT',
grok: 'Grok',
kimi: 'Kimi',
laguna: 'Laguna',
minimax: 'MiniMax',
mimo: 'MiMo',
muse: 'Muse',
nemotron: 'Nemotron',
qwen: 'Qwen',
}
export const formatOpencodeModelLabel = (modelId: string): string => {
const override = OPENCODE_LABEL_OVERRIDES[modelId]
if (override) return override
return modelId.split('-').map((token) => {
const mapped = TOKEN_LABELS[token]
if (mapped) return mapped
if (/[0-9]/.test(token)) return token.toUpperCase()
if (token.length === 0) return token
return `${token[0]?.toUpperCase() ?? ''}${token.slice(1)}`
}).join(' ')
}
export const opencodeModelSupportsVision = (modelId: string): boolean =>
!OPENCODE_TEXT_ONLY_MODELS.has(modelId)
export const opencodeCatalogId = (
providerId: OpencodeProviderId,
modelId: string,
): string => `${providerId}/${modelId}`
export const opencodeProviderModelId = (catalogId: string): string =>
catalogId.replace(/^opencode-(?:zen|go)\//, '')
/** The upstream id to send for a catalog entry, preferring the stored mapping. */
export const opencodeRequestModelId = (
catalogId: string,
providerModelId?: string,
): string => providerModelId ?? opencodeProviderModelId(catalogId)
const gatewayLabel = (providerId: OpencodeProviderId): string =>
providerId === 'opencode-go' ? 'OpenCode Go' : 'OpenCode Zen'
const describeOpencodeModel = (
providerId: OpencodeProviderId,
supportsVision: boolean,
): string => {
const gateway = gatewayLabel(providerId)
if (supportsVision) return `${gateway} · vision and tools`
return `${gateway} · no native vision · a vision model describes images first`
}
export const toOpencodeModelOption = (
providerId: OpencodeProviderId,
modelId: string,
): AiModelOption => {
const supportsVision = opencodeModelSupportsVision(modelId)
return {
id: opencodeCatalogId(providerId, modelId),
providerModelId: modelId,
label: formatOpencodeModelLabel(modelId),
description: describeOpencodeModel(providerId, supportsVision),
reasoningEfforts: OPENROUTER_REASONING_EFFORTS,
...(supportsVision ? {} : { supportsVision: false }),
}
}
const parseModelId = (value: unknown): string | null => {
if (!value || typeof value !== 'object') return null
const id = (value as Record<string, unknown>)['id']
return typeof id === 'string' && id.length > 0 ? id : null
}
/** Parse an OpenAI-compatible `/models` list into OpenCode catalog options. */
export const parseOpencodeModels = (
providerId: OpencodeProviderId,
payload: unknown,
): AiModelOption[] => {
if (!payload || typeof payload !== 'object') return []
const data = (payload as Record<string, unknown>)['data']
if (!Array.isArray(data)) return []
const seen = new Set<string>()
const models: AiModelOption[] = []
for (const entry of data) {
const modelId = parseModelId(entry)
if (!modelId || seen.has(modelId)) continue
seen.add(modelId)
models.push(toOpencodeModelOption(providerId, modelId))
}
return models.sort((left, right) => left.label.localeCompare(right.label))
}
export const listOpencodeVisionModels = (
models: readonly AiModelOption[],
): readonly AiModelOption[] =>
models.filter((model) => model.supportsVision !== false)
export const pickOpencodeVisionModel = (
providerId: OpencodeProviderId,
models: readonly AiModelOption[],
preferredId?: string,
): AiModelOption | undefined => {
const visionModels = listOpencodeVisionModels(models)
if (visionModels.length === 0) return undefined
if (preferredId) {
const preferred = visionModels.find((model) => model.id === preferredId)
if (preferred) return preferred
}
for (const candidateId of DEFAULT_VISION_MODEL_IDS[providerId]) {
const match = visionModels.find((model) => model.id === candidateId)
if (match) return match
}
return visionModels[0]
}
type CachedOpencodeModels = {
fetchedAt: number
models: AiModelOption[]
}
const parseCachedModels = (raw: string | null): CachedOpencodeModels | null => {
if (!raw) return null
let parsed: unknown
try {
parsed = JSON.parse(raw)
} catch {
return null
}
if (!parsed || typeof parsed !== 'object') return null
const record = parsed as Record<string, unknown>
if (typeof record['fetchedAt'] !== 'number' || !Array.isArray(record['models'])) {
return null
}
const models = record['models'].filter((entry): entry is AiModelOption => {
if (!entry || typeof entry !== 'object') return false
const model = entry as Record<string, unknown>
return typeof model['id'] === 'string'
&& typeof model['label'] === 'string'
&& typeof model['description'] === 'string'
})
return { fetchedAt: record['fetchedAt'], models }
}
type ModelStorage = Pick<Storage, 'getItem' | 'setItem'>
const getBrowserStorage = (): ModelStorage | null => {
try {
if (typeof localStorage === 'undefined') return null
return localStorage
} catch {
return null
}
}
export const opencodeModelsUrl = (
providerId: OpencodeProviderId,
origin: string = typeof window === 'undefined' ? '' : window.location.origin,
): string => {
const gateway = GATEWAY_BY_PROVIDER[providerId]
if (origin && origin !== 'null') {
return `${origin}/api/opencode/${gateway}/v1/models`
}
return gateway === 'go'
? 'https://opencode.ai/zen/go/v1/models'
: 'https://opencode.ai/zen/v1/models'
}
export const readCachedOpencodeModels = (
providerId: OpencodeProviderId,
now: number,
storage: ModelStorage | null = getBrowserStorage(),
): AiModelOption[] | null => {
if (!storage) return null
let cached: CachedOpencodeModels | null
try {
cached = parseCachedModels(storage.getItem(STORAGE_KEYS[providerId]))
} catch {
return null
}
if (!cached || cached.models.length === 0) return null
if (now - cached.fetchedAt > OPENCODE_MODELS_TTL_MS) return null
return cached.models
}
const writeCachedOpencodeModels = (
providerId: OpencodeProviderId,
models: AiModelOption[],
now: number,
storage: ModelStorage | null,
): void => {
if (!storage) return
try {
storage.setItem(
STORAGE_KEYS[providerId],
JSON.stringify({ fetchedAt: now, models } satisfies CachedOpencodeModels),
)
} catch {
// A full or blocked storage only loses the cache, not the feature.
}
}
export type OpencodeModelsResult = {
models: readonly AiModelOption[]
source: 'remote' | 'cache' | 'fallback'
}
const pendingFetches = new Map<OpencodeProviderId, Promise<OpencodeModelsResult>>()
const fetchRemoteModels = async (
providerId: OpencodeProviderId,
now: number,
storage: ModelStorage | null,
): Promise<OpencodeModelsResult> => {
const response = await fetch(opencodeModelsUrl(providerId), {
signal: AbortSignal.timeout(OPENCODE_MODELS_REQUEST_TIMEOUT_MS),
})
if (!response.ok) {
throw new Error(`OpenCode models request failed: ${response.status}`)
}
const models = parseOpencodeModels(providerId, await response.json() as unknown)
if (models.length === 0) throw new Error('OpenCode models response was empty')
writeCachedOpencodeModels(providerId, models, now, storage)
setDynamicProviderModels(providerId, models)
return { models, source: 'remote' }
}
/**
* Load an OpenCode gateway catalog: fresh cache, then the proxied `/models`
* list, then the curated shortlist. Successful loads register the models so
* selections resolve everywhere. Never rejects.
*/
export const loadOpencodeModels = (
providerId: OpencodeProviderId,
options?: {
now?: number
storage?: ModelStorage | null
},
): Promise<OpencodeModelsResult> => {
const now = options?.now ?? Date.now()
const storage = options?.storage === undefined ? getBrowserStorage() : options.storage
const cached = readCachedOpencodeModels(providerId, now, storage)
if (cached) {
setDynamicProviderModels(providerId, cached)
return Promise.resolve({ models: cached, source: 'cache' })
}
const pending = pendingFetches.get(providerId)
?? fetchRemoteModels(providerId, now, storage)
.catch((): OpencodeModelsResult => ({
models: getAiProvider(providerId).models,
source: 'fallback',
}))
.finally(() => {
pendingFetches.delete(providerId)
})
pendingFetches.set(providerId, pending)
return pending
}
export const loadAllOpencodeModels = (options?: {
now?: number
storage?: ModelStorage | null
}): Promise<readonly OpencodeModelsResult[]> =>
Promise.all(OPENCODE_PROVIDERS.map((providerId) => loadOpencodeModels(providerId, options)))

View File

@ -1,3 +1,4 @@
import { opencodeSelectionDialect } from './opencode-dialects'
import {
getAiStreamReasoningOptions,
type AiModelSelection,
@ -34,12 +35,19 @@ export type AiStreamRequestOptions = {
* still thinks and still bills those tokens while withholding every thought
* summary — the run band then has no live prose to show for the whole run.
*
* OpenCode's Gemini models run on the same Google provider through the gateway,
* so they need the same option to stream thoughts.
*
* Keep this object limited to `includeThoughts`. Adding `thinkingLevel` or
* `thinkingBudget` here would take full precedence over the portable `reasoning`
* option and silently discard the user's effort choice.
*/
const GOOGLE_THOUGHT_OPTIONS = { thinkingConfig: { includeThoughts: true } } as const
const usesGoogleGeneration = (selection: AiModelSelection): boolean =>
selection.provider === 'google'
|| opencodeSelectionDialect(selection) === 'google'
/**
* Combine the reasoning-effort options with OpenAI's request-wide cache
* routing. Implicit mode intentionally stays enabled: its moving breakpoint
@ -57,7 +65,7 @@ export const buildStreamRequestOptions = (
promptCacheKey: string,
): AiStreamRequestOptions => {
const reasoningOptions = getAiStreamReasoningOptions(selection)
if (selection.provider === 'google') {
if (usesGoogleGeneration(selection)) {
return {
...(reasoningOptions && 'reasoning' in reasoningOptions
? { reasoning: reasoningOptions.reasoning }

View File

@ -11,8 +11,11 @@ import {
getAiStreamReasoningOptions,
getDefaultAiModel,
getDynamicOpenRouterModels,
getAiProviderKeyFields,
isAiProviderId,
modelSupportsVision,
setDynamicOpenRouterModels,
setDynamicProviderModels,
toAiSdkReasoningEffort,
} from './provider-catalog'
@ -27,6 +30,8 @@ describe('AI provider catalog', () => {
'anthropic',
'xai',
'openrouter',
'opencode-zen',
'opencode-go',
])
expect(AI_PROVIDERS.map((provider) => provider.envVar)).toEqual([
'',
@ -37,10 +42,12 @@ describe('AI provider catalog', () => {
'VITE_ANTHROPIC_API_KEY',
'VITE_XAI_API_KEY',
'VITE_OPENROUTER_API_KEY',
'VITE_OPENCODE_API_KEY',
'VITE_OPENCODE_API_KEY',
])
expect(AI_PROVIDERS.filter((provider) => provider.transport === 'proxy').map(
(provider) => provider.id,
)).toEqual(['moonshot'])
)).toEqual(['moonshot', 'opencode-zen', 'opencode-go'])
expect(getAiProvider('codex')).toMatchObject({
auth: 'chatgpt',
transport: 'bridge',
@ -219,4 +226,47 @@ describe('AI provider catalog', () => {
providerOptions: { openai: { reasoningEffort: 'max' } },
})
})
it('shares one OpenCode key field and synthesizes runtime catalog models', () => {
const opencodeField = getAiProviderKeyFields().find((field) => field.id === 'opencode')
expect(opencodeField).toMatchObject({
label: 'OpenCode',
envVar: 'VITE_OPENCODE_API_KEY',
providerIds: ['opencode-zen', 'opencode-go'],
})
expect(modelSupportsVision(getAiModel({
provider: 'opencode-go',
model: 'opencode-go/glm-5.3',
}))).toBe(false)
expect(modelSupportsVision(getAiModel({
provider: 'opencode-zen',
model: 'opencode-zen/kimi-k3',
}))).toBe(true)
const synthesized = getAiModel({
provider: 'opencode-go',
model: 'opencode-go/new-model',
})
expect(synthesized).toEqual({
id: 'opencode-go/new-model',
label: 'opencode-go/new-model',
description: 'OpenCode Go model',
})
const dynamicModel = {
id: 'opencode-go/new-model',
label: 'New Go Model',
description: 'fetched',
supportsVision: false,
}
setDynamicProviderModels('opencode-go', [dynamicModel])
expect(getAiModel({
provider: 'opencode-go',
model: 'opencode-go/new-model',
})).toBe(dynamicModel)
expect(findAiModelById('opencode-go/new-model')).toEqual({
provider: getAiProvider('opencode-go'),
model: dynamicModel,
})
setDynamicProviderModels('opencode-go', [])
})
})

View File

@ -7,6 +7,8 @@ export type AiProviderId =
| 'anthropic'
| 'xai'
| 'openrouter'
| 'opencode-zen'
| 'opencode-go'
export type AiReasoningEffort =
| 'minimal'
@ -23,6 +25,7 @@ export type AiModelSelection = {
provider: AiProviderId
model: string
reasoningEffort?: AiReasoningEffort
visionModel?: string
}
export type AiModelOption = {
@ -31,6 +34,13 @@ export type AiModelOption = {
label: string
description: string
reasoningEfforts?: readonly AiReasoningEffort[]
/** Omitted means the model accepts image input natively. */
supportsVision?: boolean
}
export type AiProviderKeyGroup = {
id: string
label: string
}
export type AiProviderOption = {
@ -40,8 +50,12 @@ export type AiProviderOption = {
auth: 'apiKey' | 'chatgpt'
transport: 'bridge' | 'direct' | 'proxy'
models: readonly AiModelOption[]
keyGroup?: AiProviderKeyGroup
}
export const modelSupportsVision = (model: AiModelOption): boolean =>
model.supportsVision !== false
export const DEFAULT_AI_REASONING_EFFORT = 'high' as const satisfies AiReasoningEffort
export const AI_REASONING_EFFORT_LABELS: Record<AiReasoningEffort, string> = {
@ -320,6 +334,86 @@ export const AI_PROVIDERS: readonly AiProviderOption[] = [
},
],
},
{
id: 'opencode-zen',
label: 'OpenCode Zen',
envVar: 'VITE_OPENCODE_API_KEY',
auth: 'apiKey',
transport: 'proxy',
keyGroup: { id: 'opencode', label: 'OpenCode' },
// Curated recommendations; the full tools catalog is fetched at runtime
// (src/ai/opencode-models.ts) and this list doubles as the offline fallback.
models: [
{
id: 'opencode-zen/gpt-5.6-terra',
providerModelId: 'gpt-5.6-terra',
label: 'GPT 5.6 Terra',
description: 'Recommended · vision and tools',
reasoningEfforts: OPENROUTER_REASONING_EFFORTS,
},
{
id: 'opencode-zen/claude-sonnet-5',
providerModelId: 'claude-sonnet-5',
label: 'Claude Sonnet 5',
description: 'Balanced quality and cost',
reasoningEfforts: OPENROUTER_REASONING_EFFORTS,
},
{
id: 'opencode-zen/kimi-k3',
providerModelId: 'kimi-k3',
label: 'Kimi K3',
description: 'Open coding model with vision',
reasoningEfforts: OPENROUTER_REASONING_EFFORTS,
},
{
id: 'opencode-zen/glm-5.2',
providerModelId: 'glm-5.2',
label: 'GLM 5.2',
description: 'No native vision · a vision model describes images first',
reasoningEfforts: OPENROUTER_REASONING_EFFORTS,
supportsVision: false,
},
],
},
{
id: 'opencode-go',
label: 'OpenCode Go',
envVar: 'VITE_OPENCODE_API_KEY',
auth: 'apiKey',
transport: 'proxy',
keyGroup: { id: 'opencode', label: 'OpenCode' },
models: [
{
id: 'opencode-go/gpt-5.6-luna',
providerModelId: 'gpt-5.6-luna',
label: 'GPT 5.6 Luna',
description: 'Recommended · vision and tools',
reasoningEfforts: OPENROUTER_REASONING_EFFORTS,
},
{
id: 'opencode-go/kimi-k3',
providerModelId: 'kimi-k3',
label: 'Kimi K3',
description: 'Open coding model with vision',
reasoningEfforts: OPENROUTER_REASONING_EFFORTS,
},
{
id: 'opencode-go/mimo-v2.5',
providerModelId: 'mimo-v2.5',
label: 'MiMo V2.5',
description: 'Fast open model with vision',
reasoningEfforts: OPENROUTER_REASONING_EFFORTS,
},
{
id: 'opencode-go/glm-5.3',
providerModelId: 'glm-5.3',
label: 'GLM 5.3',
description: 'No native vision · a vision model describes images first',
reasoningEfforts: OPENROUTER_REASONING_EFFORTS,
supportsVision: false,
},
],
},
]
export const DEFAULT_AI_SELECTION: AiModelSelection = {
@ -331,37 +425,93 @@ export const DEFAULT_AI_SELECTION: AiModelSelection = {
export const isAiProviderId = (value: unknown): value is AiProviderId =>
typeof value === 'string' && AI_PROVIDERS.some((provider) => provider.id === value)
export type AiProviderKeyField = {
id: string
label: string
envVar: string
providerIds: readonly AiProviderId[]
}
/** One API-key field per credential, sharing Zen/Go under a single OpenCode key. */
export const getAiProviderKeyFields = (): readonly AiProviderKeyField[] => {
const fields: AiProviderKeyField[] = []
const indexById = new Map<string, number>()
for (const provider of AI_PROVIDERS) {
if (provider.auth !== 'apiKey') continue
const id = provider.keyGroup?.id ?? provider.id
const existingIndex = indexById.get(id)
if (existingIndex !== undefined) {
const existing = fields[existingIndex]
if (!existing) continue
fields[existingIndex] = {
...existing,
providerIds: [...existing.providerIds, provider.id],
}
continue
}
indexById.set(id, fields.length)
fields.push({
id,
label: provider.keyGroup?.label ?? provider.label,
envVar: provider.envVar,
providerIds: [provider.id],
})
}
return fields
}
/**
* OpenRouter models discovered at runtime (src/ai/openrouter-models.ts).
* Registered here so catalog lookups resolve dynamic selections; the static
* catalog stays the source of truth for every other provider.
* Runtime catalogs (OpenRouter, OpenCode Zen/Go) register here so lookups
* resolve dynamic selections. The static catalog stays the fallback.
*/
let dynamicOpenRouterModels: readonly AiModelOption[] = []
const DYNAMIC_CATALOG_PROVIDERS = new Set<AiProviderId>([
'openrouter',
'opencode-zen',
'opencode-go',
])
const dynamicModelsByProvider: Partial<Record<AiProviderId, readonly AiModelOption[]>> = {}
export const setDynamicProviderModels = (
providerId: AiProviderId,
models: readonly AiModelOption[],
): void => {
dynamicModelsByProvider[providerId] = models
}
export const getDynamicProviderModels = (
providerId: AiProviderId,
): readonly AiModelOption[] =>
dynamicModelsByProvider[providerId] ?? []
export const setDynamicOpenRouterModels = (
models: readonly AiModelOption[],
): void => {
dynamicOpenRouterModels = models
setDynamicProviderModels('openrouter', models)
}
export const getDynamicOpenRouterModels = (): readonly AiModelOption[] =>
dynamicOpenRouterModels
getDynamicProviderModels('openrouter')
const findOpenRouterModel = (modelId: string): AiModelOption | undefined =>
getAiProvider('openrouter').models.find((option) => option.id === modelId)
?? dynamicOpenRouterModels.find((option) => option.id === modelId)
const findDynamicCatalogModel = (
providerId: AiProviderId,
modelId: string,
): AiModelOption | undefined =>
getAiProvider(providerId).models.find((option) => option.id === modelId)
?? getDynamicProviderModels(providerId).find((option) => option.id === modelId)
/**
* A dynamic selection may outlive the fetched catalog (e.g. the model list is
* still loading); synthesize a minimal option so lookups never throw for the
* dynamic provider. Reasoning support is unknown, so no effort is offered.
*/
const synthesizeOpenRouterModel = (modelId: string): AiModelOption => ({
const synthesizeDynamicCatalogModel = (
providerId: AiProviderId,
modelId: string,
): AiModelOption => ({
id: modelId,
label: modelId,
description: 'OpenRouter model',
description: `${getAiProvider(providerId).label} model`,
})
export const isDynamicCatalogProvider = (providerId: AiProviderId): boolean =>
DYNAMIC_CATALOG_PROVIDERS.has(providerId)
export const getAiProvider = (providerId: AiProviderId): AiProviderOption => {
const provider = AI_PROVIDERS.find((option) => option.id === providerId)
if (!provider) throw new Error(`Unknown AI provider: ${providerId}`)
@ -375,9 +525,9 @@ export const getDefaultAiModel = (providerId: AiProviderId): AiModelOption => {
}
export const getAiModel = (selection: AiModelSelection): AiModelOption => {
if (selection.provider === 'openrouter') {
return findOpenRouterModel(selection.model)
?? synthesizeOpenRouterModel(selection.model)
if (isDynamicCatalogProvider(selection.provider)) {
return findDynamicCatalogModel(selection.provider, selection.model)
?? synthesizeDynamicCatalogModel(selection.provider, selection.model)
}
const model = getAiProvider(selection.provider).models.find(
(option) => option.id === selection.model,
@ -395,9 +545,13 @@ export const findAiModelById = (
const model = provider.models.find((option) => option.id === modelId)
if (model) return { provider, model }
}
const dynamicModel = dynamicOpenRouterModels.find((option) => option.id === modelId)
if (dynamicModel) {
return { provider: getAiProvider('openrouter'), model: dynamicModel }
for (const providerId of DYNAMIC_CATALOG_PROVIDERS) {
const dynamicModel = getDynamicProviderModels(providerId).find(
(option) => option.id === modelId,
)
if (dynamicModel) {
return { provider: getAiProvider(providerId), model: dynamicModel }
}
}
throw new Error(`Unknown AI model: ${modelId}`)
}

View File

@ -43,8 +43,9 @@ describe('AI provider configuration', () => {
VITE_ANTHROPIC_API_KEY: 'anthropic-key',
VITE_XAI_API_KEY: ' xai-key ',
VITE_OPENROUTER_API_KEY: ' openrouter-key ',
VITE_OPENCODE_API_KEY: ' opencode-key ',
})).toEqual({
codex: '',
...createEmptyAiProviderKeys(),
moonshot: 'moonshot-key',
google: 'google-key',
qwen: 'qwen-key',
@ -52,6 +53,8 @@ describe('AI provider configuration', () => {
anthropic: 'anthropic-key',
xai: 'xai-key',
openrouter: 'openrouter-key',
'opencode-zen': 'opencode-key',
'opencode-go': 'opencode-key',
})
})
@ -63,14 +66,8 @@ describe('AI provider configuration', () => {
})
expect(keys).toEqual({
codex: '',
moonshot: '',
google: '',
qwen: '',
...createEmptyAiProviderKeys(),
openai: 'openai-key',
anthropic: '',
xai: '',
openrouter: '',
})
expect(getAiProviderAvailability(keys)).toEqual({
codex: false,
@ -81,6 +78,8 @@ describe('AI provider configuration', () => {
anthropic: false,
xai: false,
openrouter: false,
'opencode-zen': false,
'opencode-go': false,
})
})
@ -94,6 +93,8 @@ describe('AI provider configuration', () => {
anthropic: true,
xai: true,
openrouter: true,
'opencode-zen': false,
'opencode-go': false,
})).toEqual({
provider: 'openai',
model: 'gpt-5.6-terra',
@ -108,6 +109,8 @@ describe('AI provider configuration', () => {
anthropic: false,
xai: false,
openrouter: true,
'opencode-zen': false,
'opencode-go': false,
})).toEqual({
provider: 'openrouter',
model: 'anthropic/claude-sonnet-5',
@ -122,6 +125,8 @@ describe('AI provider configuration', () => {
anthropic: false,
xai: false,
openrouter: false,
'opencode-zen': false,
'opencode-go': false,
})).toEqual({
provider: 'codex',
model: 'codex/gpt-5.6-terra',
@ -136,6 +141,8 @@ describe('AI provider configuration', () => {
anthropic: false,
xai: false,
openrouter: false,
'opencode-zen': false,
'opencode-go': false,
})).toEqual({
provider: 'moonshot',
model: 'kimi-k3',
@ -154,14 +161,9 @@ describe('AI provider configuration', () => {
}
expect(mergeAiProviderKeys(environment, stored)).toEqual({
codex: '',
moonshot: '',
...createEmptyAiProviderKeys(),
google: 'env-google',
qwen: '',
openai: 'browser-openai',
anthropic: '',
xai: '',
openrouter: '',
})
})
@ -176,6 +178,20 @@ describe('AI provider configuration', () => {
...createEmptyAiProviderKeys(),
openai: 'sk-test',
})
expect(parseStoredAiProviderKeys(JSON.stringify({
opencode: ' zen-go-key ',
}))).toEqual({
...createEmptyAiProviderKeys(),
'opencode-zen': 'zen-go-key',
'opencode-go': 'zen-go-key',
})
expect(parseStoredAiProviderKeys(JSON.stringify({
'opencode-zen': ' zen-only-key ',
}))).toEqual({
...createEmptyAiProviderKeys(),
'opencode-zen': 'zen-only-key',
'opencode-go': 'zen-only-key',
})
const storage = createMemoryStorage()
expect(writeStoredAiProviderKeys({
@ -238,6 +254,8 @@ describe('AI provider configuration', () => {
expect(isLocalAiProxyHostname('[::1]')).toBe(true)
expect(isLocalAiProxyHostname('app.shotluma.com')).toBe(false)
expect(getAiProviderTransportAvailability('app.shotluma.com').moonshot).toBe(false)
expect(getAiProviderTransportAvailability('app.shotluma.com')['opencode-zen']).toBe(true)
expect(getAiProviderTransportAvailability('app.shotluma.com')['opencode-go']).toBe(true)
})
it('does not mark a stored Moonshot key as usable without the local proxy', () => {

View File

@ -22,6 +22,7 @@ type AiProviderEnvironment = {
VITE_ANTHROPIC_API_KEY?: unknown
VITE_XAI_API_KEY?: unknown
VITE_OPENROUTER_API_KEY?: unknown
VITE_OPENCODE_API_KEY?: unknown
}
export const AI_PROVIDER_KEYS_STORAGE_KEY = 'shotluma-ai-provider-keys'
@ -40,20 +41,27 @@ export const createEmptyAiProviderKeys = (): AiProviderKeys => ({
anthropic: '',
xai: '',
openrouter: '',
'opencode-zen': '',
'opencode-go': '',
})
export const createAiProviderKeys = (
environment: AiProviderEnvironment,
): AiProviderKeys => ({
codex: '',
moonshot: readKey(environment.VITE_MOONSHOT_API_KEY),
google: readKey(environment.VITE_GOOGLE_GENERATIVE_AI_API_KEY),
qwen: readKey(environment.VITE_ALIBABA_API_KEY),
openai: readKey(environment.VITE_OPENAI_API_KEY),
anthropic: readKey(environment.VITE_ANTHROPIC_API_KEY),
xai: readKey(environment.VITE_XAI_API_KEY),
openrouter: readKey(environment.VITE_OPENROUTER_API_KEY),
})
): AiProviderKeys => {
const opencode = readKey(environment.VITE_OPENCODE_API_KEY)
return {
codex: '',
moonshot: readKey(environment.VITE_MOONSHOT_API_KEY),
google: readKey(environment.VITE_GOOGLE_GENERATIVE_AI_API_KEY),
qwen: readKey(environment.VITE_ALIBABA_API_KEY),
openai: readKey(environment.VITE_OPENAI_API_KEY),
anthropic: readKey(environment.VITE_ANTHROPIC_API_KEY),
xai: readKey(environment.VITE_XAI_API_KEY),
openrouter: readKey(environment.VITE_OPENROUTER_API_KEY),
'opencode-zen': opencode,
'opencode-go': opencode,
}
}
export const getAiProviderAvailability = (
keys: AiProviderKeys,
@ -66,6 +74,8 @@ export const getAiProviderAvailability = (
anthropic: keys.anthropic.length > 0,
xai: keys.xai.length > 0,
openrouter: keys.openrouter.length > 0,
'opencode-zen': keys['opencode-zen'].length > 0,
'opencode-go': keys['opencode-go'].length > 0,
})
const getBrowserHostname = (): string => {
@ -95,6 +105,8 @@ export const getAiProviderTransportAvailability = (
anthropic: true,
xai: true,
openrouter: true,
'opencode-zen': true,
'opencode-go': true,
})
export const mergeAiProviderKeys = (
@ -125,6 +137,11 @@ export const parseStoredAiProviderKeys = (raw: string | null): AiProviderKeys =>
for (const providerId of PROVIDER_IDS) {
keys[providerId] = readKey(record[providerId])
}
const sharedOpencode = readKey(record['opencode'])
|| keys['opencode-zen']
|| keys['opencode-go']
keys['opencode-zen'] = sharedOpencode
keys['opencode-go'] = sharedOpencode
return keys
}
@ -232,6 +249,7 @@ export const ENVIRONMENT_AI_PROVIDER_KEYS = createAiProviderKeys({
VITE_ANTHROPIC_API_KEY: import.meta.env.VITE_ANTHROPIC_API_KEY,
VITE_XAI_API_KEY: import.meta.env.VITE_XAI_API_KEY,
VITE_OPENROUTER_API_KEY: import.meta.env.VITE_OPENROUTER_API_KEY,
VITE_OPENCODE_API_KEY: import.meta.env.VITE_OPENCODE_API_KEY,
})
export const AI_PROVIDER_AVAILABILITY = getResolvedAiProviderAvailability()

98
src/ai/runner.test.ts Normal file
View File

@ -0,0 +1,98 @@
import { describe, expect, it, vi } from 'vitest'
import { prepareGenerationMessages, type OpencodeVisionFallback } from './runner'
import type { ModelMessage } from 'ai'
const textOnlyMessages: ModelMessage[] = [
{ role: 'user', content: 'Design three slides' },
]
const messagesWithUserImage: ModelMessage[] = [
{
role: 'user',
content: [
{ type: 'text', text: 'Screenshot:' },
{ type: 'file', mediaType: 'image/png', data: 'data:image/png;base64,abc' },
],
},
]
const messagesWithToolPreview: ModelMessage[] = [
{ role: 'user', content: 'Design three slides' },
{
role: 'tool',
content: [{
type: 'tool-result',
toolCallId: 'call-1',
toolName: 'render_slide_preview',
output: {
type: 'content',
value: [{
type: 'file',
mediaType: 'image/png',
data: { type: 'data', data: 'preview-bytes' },
}],
},
}],
},
]
const prepare = (
messages: ModelMessage[],
resolveVisionFallback: (() => OpencodeVisionFallback) | null,
needsChatImageRelay = false,
) => prepareGenerationMessages({
messages,
movingCacheProvider: null,
needsChatImageRelay,
resolveVisionFallback,
visionAnnouncement: { sent: false },
descriptionCache: new Map(),
})
describe('prepareGenerationMessages', () => {
it('never resolves the vision fallback for image-free requests', async () => {
const resolveVisionFallback = vi.fn((): OpencodeVisionFallback => {
throw new Error('no OpenCode vision model is available')
})
const prepared = await prepare(textOnlyMessages, resolveVisionFallback)
expect(prepared).toBe(textOnlyMessages)
expect(resolveVisionFallback).not.toHaveBeenCalled()
})
it('surfaces the missing-vision-model error once an image appears', async () => {
const resolveVisionFallback = (): OpencodeVisionFallback => {
throw new Error('no OpenCode vision model is available')
}
await expect(prepare(messagesWithUserImage, resolveVisionFallback))
.rejects.toThrow('no OpenCode vision model is available')
})
it('describes user images through the resolved fallback', async () => {
const describeImage = vi.fn().mockResolvedValue('a login screen')
const prepared = await prepare(messagesWithUserImage, () => ({
describeImage,
visionLabel: 'MiMo V2.5',
}))
expect(describeImage).toHaveBeenCalledTimes(1)
expect(prepared[0]).toMatchObject({
content: [
{ type: 'text', text: 'Screenshot:' },
{ type: 'text', text: 'Image description:\na login screen' },
],
})
})
it('relays tool preview images for non-chat dialects when the fallback is active', async () => {
const describeImage = vi.fn().mockResolvedValue('slide preview')
const prepared = await prepare(messagesWithToolPreview, () => ({
describeImage,
visionLabel: 'MiMo V2.5',
}), false)
expect(describeImage).toHaveBeenCalledTimes(1)
const relayedUserMessage = prepared[prepared.length - 1]
expect(relayedUserMessage).toMatchObject({
role: 'user',
content: [{ type: 'text', text: 'Image description:\nslide preview' }],
})
})
})

View File

@ -14,7 +14,21 @@ import { relayChatToolImages } from './chat-tool-images'
import { runCodexAppServerGeneration } from './codex-app-server'
import { readCodexConnection } from './codex-connection'
import { scopeAiControllerToSlide, type AiEditorController } from './controller'
import { collectMessageImages, describeMessageImages } from './describe-images'
import { buildHtmlInstructions } from './html-prompt'
import {
createOpencodeImageDescriber,
createOpencodeModel,
} from './opencode-client'
import {
opencodeDialectNeedsChatToolImageRelay,
opencodeSelectionDialect,
} from './opencode-dialects'
import {
isOpencodeProviderId,
opencodeRequestModelId,
pickOpencodeVisionModel,
} from './opencode-models'
import { buildInstructions, buildUserMessage } from './prompt'
import {
buildStreamRequestOptions,
@ -26,6 +40,8 @@ import {
getAiModel,
getAiProvider,
getAiSdkReasoningEffort,
getDynamicProviderModels,
modelSupportsVision,
type AiModelSelection,
} from './provider-catalog'
import {
@ -107,7 +123,9 @@ const buildUserContent = (options: {
const createAiModel = async (selection: AiModelSelection) => {
if (!getAiProviderTransportAvailability()[selection.provider]) {
throw new Error('Moonshot requires the local Shotluma CORS proxy and is unavailable on this host')
throw new Error(
`${getAiProvider(selection.provider).label} requires the local Shotluma CORS proxy and is unavailable on this host`,
)
}
const apiKey = getAiProviderKey(selection.provider)
if (!apiKey) {
@ -162,6 +180,15 @@ const createAiModel = async (selection: AiModelSelection) => {
},
}).chat(selection.model)
}
case 'opencode-zen':
case 'opencode-go': {
const model = getAiModel(selection)
return createOpencodeModel({
providerId: selection.provider,
modelId: opencodeRequestModelId(model.id, model.providerModelId),
apiKey,
})
}
}
}
@ -304,7 +331,99 @@ const collectStreamPart = <TOOLS extends ToolSet>(options: {
*/
export const NARRATION_WORD_DELAY_MS = 18
export const narrationSmoothing = () => smoothStream<ToolSet>({ delayInMs: NARRATION_WORD_DELAY_MS })
export const narrationSmoothing = () =>
smoothStream<ToolSet>({ delayInMs: NARRATION_WORD_DELAY_MS })
/**
* OpenCode serves each model on one of four dialects, and only its
* chat-completions models reject images inside tool results.
*/
const needsChatToolImageRelay = (selection: AiModelSelection): boolean => {
const dialect = opencodeSelectionDialect(selection)
if (dialect) return opencodeDialectNeedsChatToolImageRelay(dialect)
return selection.provider === 'moonshot'
|| selection.provider === 'openrouter'
|| selection.provider === 'qwen'
}
export type OpencodeVisionFallback = {
describeImage: ReturnType<typeof createOpencodeImageDescriber>
visionLabel: string
}
const needsOpencodeVisionFallback = (selection: AiModelSelection): boolean =>
isOpencodeProviderId(selection.provider) && !modelSupportsVision(getAiModel(selection))
const resolveOpencodeVisionDescriber = (options: {
selection: AiModelSelection
apiKey: string
signal?: AbortSignal
}): OpencodeVisionFallback | null => {
if (!isOpencodeProviderId(options.selection.provider)) return null
const model = getAiModel(options.selection)
if (modelSupportsVision(model)) return null
const catalog = [
...getAiProvider(options.selection.provider).models,
...getDynamicProviderModels(options.selection.provider),
]
const vision = pickOpencodeVisionModel(
options.selection.provider,
catalog,
options.selection.visionModel,
)
if (!vision) {
throw new Error(
`${model.label} cannot see images, and no OpenCode vision model is available to describe them.`,
)
}
return {
visionLabel: vision.label,
describeImage: createOpencodeImageDescriber({
providerId: options.selection.provider,
visionModelId: opencodeRequestModelId(vision.id, vision.providerModelId),
apiKey: options.apiKey,
...(options.signal ? { signal: options.signal } : {}),
}),
}
}
/**
* The resolver runs only once a prepared step actually contains an image, so
* a text-only OpenCode model without any vision fallback in the catalog can
* still run image-free requests; the missing-fallback error surfaces the
* moment a screenshot or slide preview enters the conversation.
*/
export const prepareGenerationMessages = async (options: {
messages: ModelMessage[]
movingCacheProvider: 'anthropic' | 'alibaba' | null
needsChatImageRelay: boolean
resolveVisionFallback: (() => OpencodeVisionFallback) | null
onStatus?: (message: string) => void
visionAnnouncement: { sent: boolean }
descriptionCache: Map<string, string>
}): Promise<ModelMessage[]> => {
// Text-only models reject images anywhere, so the vision fallback also
// needs tool-result images relayed into user messages before describing.
const relayed = options.needsChatImageRelay || options.resolveVisionFallback !== null
? relayChatToolImages(options.messages)
: options.messages
let next = relayed
if (options.resolveVisionFallback && collectMessageImages(relayed).length > 0) {
const visionFallback = options.resolveVisionFallback()
if (!options.visionAnnouncement.sent) {
options.visionAnnouncement.sent = true
options.onStatus?.(`Describing images with ${visionFallback.visionLabel}…`)
}
next = await describeMessageImages(
relayed,
visionFallback.describeImage,
options.descriptionCache,
)
}
return options.movingCacheProvider
? withMovingCacheBreakpoint(next, options.movingCacheProvider)
: next
}
const openAiGenerationStream = (options: {
model: Awaited<ReturnType<typeof createAiModel>>
@ -317,6 +436,7 @@ const openAiGenerationStream = (options: {
signal?: AbortSignal
onActivity?: (activity: AiToolActivity) => void
onStepUsage?: (usage: LanguageModelUsage, toolNames: string[]) => void
onStatus?: (message: string) => void
}) => {
const {
model,
@ -329,6 +449,7 @@ const openAiGenerationStream = (options: {
signal,
onActivity,
onStepUsage,
onStatus,
} = options
const runController = targetSlideId
? scopeAiControllerToSlide(controller, targetSlideId)
@ -352,9 +473,23 @@ const openAiGenerationStream = (options: {
const movingCacheProvider = selection.provider === 'anthropic'
? 'anthropic'
: selection.provider === 'qwen' ? 'alibaba' : null
const needsChatImageRelay = selection.provider === 'moonshot'
|| selection.provider === 'openrouter'
|| selection.provider === 'qwen'
const needsChatImageRelay = needsChatToolImageRelay(selection)
let cachedVisionFallback: OpencodeVisionFallback | null = null
const resolveVisionFallback = needsOpencodeVisionFallback(selection)
? (): OpencodeVisionFallback => {
cachedVisionFallback ??= resolveOpencodeVisionDescriber({
selection,
apiKey: getAiProviderKey(selection.provider),
...(signal ? { signal } : {}),
})
if (!cachedVisionFallback) {
throw new Error(`${getAiModel(selection).label} has no vision fallback to describe images.`)
}
return cachedVisionFallback
}
: null
const visionAnnouncement = { sent: false }
const descriptionCache = new Map<string, string>()
return streamText({
model,
@ -375,18 +510,19 @@ const openAiGenerationStream = (options: {
...requestOptions,
// Anthropic and Alibaba require explicit message breakpoints. Keep one
// moving forward so each step can reuse the growing tool-loop prefix.
...(movingCacheProvider || needsChatImageRelay
...(movingCacheProvider || needsChatImageRelay || resolveVisionFallback
? {
prepareStep: ({ messages }: { messages: ModelMessage[] }) => {
const relayed = needsChatImageRelay
? relayChatToolImages(messages)
: messages
return {
messages: movingCacheProvider
? withMovingCacheBreakpoint(relayed, movingCacheProvider)
: relayed,
}
},
prepareStep: async ({ messages }: { messages: ModelMessage[] }) => ({
messages: await prepareGenerationMessages({
messages,
movingCacheProvider,
needsChatImageRelay,
resolveVisionFallback,
...(onStatus ? { onStatus } : {}),
visionAnnouncement,
descriptionCache,
}),
}),
}
: {}),
...(signal ? { abortSignal: signal } : {}),
@ -499,6 +635,9 @@ const runDirectProviderGeneration = async (options: {
...(htmlMode ? { htmlMode: true } : {}),
...(signal ? { signal } : {}),
...(onActivity ? { onActivity } : {}),
onStatus: (message) => {
onEvent({ type: 'status', message })
},
onStepUsage: (usage, toolNames) => {
accumulator.steps.push({
usage: toAiRunTokenUsage(usage),

View File

@ -50,7 +50,8 @@ describe('AI API keys dialog', () => {
expect(markup).toContain('Qwen')
expect(markup).toContain('OpenAI')
expect(markup).toContain('Anthropic')
expect(markup).toContain('xAI')
expect(markup).toContain('OpenRouter')
expect(markup).toContain('OpenCode')
expect(markup).toContain('type="password"')
expect(markup).toContain('<form')
expect(markup).toContain('type="submit"')

View File

@ -5,7 +5,7 @@ import {
type SyntheticEvent,
} from 'react'
import {
AI_PROVIDERS,
getAiProviderKeyFields,
type AiProviderId,
} from '../ai/provider-catalog'
import {
@ -51,9 +51,12 @@ export const AiApiKeysDialog = ({
if (!open) return
const frame = window.requestAnimationFrame(() => {
const targetId = focusProviderId
?? AI_PROVIDERS.find((provider) => provider.auth === 'apiKey')?.id
?? getAiProviderKeyFields()[0]?.providerIds[0]
if (!targetId) return
const node = document.getElementById(`${formId}-${targetId}`)
const field = getAiProviderKeyFields().find((entry) =>
entry.providerIds.includes(targetId),
)
const node = document.getElementById(`${formId}-${field?.id ?? targetId}`)
if (!(node instanceof HTMLInputElement)) return
node.focus()
node.select()
@ -102,18 +105,24 @@ export const AiApiKeysDialog = ({
</summary>
<p>
Keys are stored unencrypted in this browser. AI requests send a key only
to the provider used for that request; Moonshot passes through the local proxy,
and optional overlay assets use OpenAI. Use dedicated keys with strict limits.
to the provider used for that request; Moonshot and OpenCode pass through
a same-origin CORS proxy, and optional overlay assets use OpenAI. Use
dedicated keys with strict limits.
</p>
</details>
<div className="ai-keys-dialog-fields">
{AI_PROVIDERS.filter((provider) => provider.auth === 'apiKey').map((provider) => {
const inputId = `${formId}-${provider.id}`
const hasEnvFallback = envAvailability[provider.id] && !draft[provider.id]
{getAiProviderKeyFields().map((field) => {
const inputId = `${formId}-${field.id}`
const value = field.providerIds
.map((providerId) => draft[providerId])
.find((entry) => entry.length > 0) ?? ''
const hasEnvFallback = field.providerIds.some((providerId) =>
envAvailability[providerId] && !draft[providerId],
)
return (
<label className="ai-keys-dialog-field" htmlFor={inputId} key={provider.id}>
<span>{provider.label}</span>
<label className="ai-keys-dialog-field" htmlFor={inputId} key={field.id}>
<span>{field.label}</span>
<Input
id={inputId}
type="password"
@ -121,16 +130,22 @@ export const AiApiKeysDialog = ({
autoCapitalize="none"
spellCheck={false}
placeholder={hasEnvFallback ? 'Using .env.local' : 'Paste API key'}
value={draft[provider.id]}
value={value}
onKeyDown={(event) => {
if (event.key !== 'Enter' || event.nativeEvent.isComposing) return
event.preventDefault()
saveDraft()
}}
onChange={(event) => {
const value = event.target.value
const nextValue = event.target.value
setSaveError(null)
setDraft((current) => ({ ...current, [provider.id]: value }))
setDraft((current) => {
const next = { ...current }
for (const providerId of field.providerIds) {
next[providerId] = nextValue
}
return next
})
}}
/>
</label>

View File

@ -5,9 +5,16 @@ import {
type DragEvent,
type RefObject,
} from 'react'
import {
isOpencodeProviderId,
pickOpencodeVisionModel,
} from '../ai/opencode-models'
import {
clampAiReasoningEffort,
findAiModelById,
getAiModel,
getAiProvider,
getDynamicProviderModels,
modelSupportsVision,
type AiModelSelection,
type AiProviderId,
} from '../ai/provider-catalog'
@ -54,6 +61,28 @@ const toImageDrafts = async (files: File[], idPrefix: 'logo' | 'shot'): Promise<
dataUrl: await fileToDataUrl(file),
})))
const nextAiSelection = (
provider: AiProviderId,
modelId: string,
current: AiModelSelection,
): AiModelSelection => {
const model = getAiModel({ provider, model: modelId })
const reasoningEffort = clampAiReasoningEffort(model, current.reasoningEffort)
const visionModel = isOpencodeProviderId(provider) && !modelSupportsVision(model)
? pickOpencodeVisionModel(
provider,
[...getAiProvider(provider).models, ...getDynamicProviderModels(provider)],
current.provider === provider ? current.visionModel : undefined,
)?.id
: undefined
return {
provider,
model: modelId,
...(reasoningEffort ? { reasoningEffort } : {}),
...(visionModel ? { visionModel } : {}),
}
}
export type AiGenerateModalProps = {
open: boolean
onClose: () => void
@ -320,6 +349,7 @@ const ModalFooter = ({
transportAvailability,
onModelSelect,
onReasoningEffortChange,
onVisionModelChange,
onManageKeys,
onConnectCodex,
onGenerate,
@ -331,6 +361,7 @@ const ModalFooter = ({
transportAvailability: AiProviderAvailability
onModelSelect: (provider: AiProviderId, modelId: string) => void
onReasoningEffortChange: (reasoningEffort: NonNullable<AiModelSelection['reasoningEffort']>) => void
onVisionModelChange: (visionModel: string) => void
onManageKeys: (providerId: AiProviderId) => void
onConnectCodex: () => void
onGenerate: () => void
@ -342,6 +373,7 @@ const ModalFooter = ({
transportAvailability={transportAvailability}
onModelSelect={onModelSelect}
onReasoningEffortChange={onReasoningEffortChange}
onVisionModelChange={onVisionModelChange}
onManageKeys={onManageKeys}
onConnectCodex={onConnectCodex}
/>
@ -452,19 +484,15 @@ export const AiGenerateModal = ({ open, onClose, controller, targetSlide, onPrep
const removeScreenshot = (id: string) => setScreenshots((current) => current.filter((shot) => shot.id !== id))
const handleModelSelect = (provider: AiProviderId, modelId: string) => {
const { model } = findAiModelById(modelId)
const reasoningEffort = clampAiReasoningEffort(model, selection.reasoningEffort)
setSelection({
provider,
model: modelId,
...(reasoningEffort ? { reasoningEffort } : {}),
})
setSelection(nextAiSelection(provider, modelId, selection))
}
const handleEvent = (event: AiRunEvent) => {
if (cancelledRef.current) return
if (event.type === 'tool') {
setLatestActivity(event.detail)
} else if (event.type === 'status') {
setLatestActivity(event.message)
} else if (event.type === 'plan') setPlan(event.screens)
// `index` is the count of screens finished before this one, which is exactly the
// rail's "built" count while this screen is in progress.
@ -483,7 +511,7 @@ export const AiGenerateModal = ({ open, onClose, controller, targetSlide, onPrep
setPhase('done')
onActivity?.(null)
onFinished(event.slidesCreated)
} else if (event.type === 'error') {
} else {
setNarration(flushNarration)
setErrorMessage(event.message)
setPhase('error')
@ -627,6 +655,9 @@ export const AiGenerateModal = ({ open, onClose, controller, targetSlide, onPrep
onReasoningEffortChange={(reasoningEffort) => {
setSelection((current) => ({ ...current, reasoningEffort }))
}}
onVisionModelChange={(visionModel) => {
setSelection((current) => ({ ...current, visionModel }))
}}
onManageKeys={handleManageKeys}
onConnectCodex={handleConnectCodex}
onGenerate={() => void handleGenerate()}

View File

@ -42,6 +42,8 @@ const availableTransports = {
anthropic: true,
xai: true,
openrouter: true,
'opencode-zen': true,
'opencode-go': true,
}
vi.mock('./ui/button', () => ({
@ -72,10 +74,13 @@ describe('AI provider controls', () => {
anthropic: false,
xai: false,
openrouter: false,
'opencode-zen': false,
'opencode-go': false,
}}
transportAvailability={availableTransports}
onModelSelect={() => undefined}
onReasoningEffortChange={() => undefined}
onVisionModelChange={() => undefined}
onManageKeys={() => undefined}
onConnectCodex={() => undefined}
/>,
@ -100,10 +105,13 @@ describe('AI provider controls', () => {
anthropic: true,
xai: true,
openrouter: true,
'opencode-zen': false,
'opencode-go': false,
}}
transportAvailability={availableTransports}
onModelSelect={() => undefined}
onReasoningEffortChange={() => undefined}
onVisionModelChange={() => undefined}
onManageKeys={() => undefined}
onConnectCodex={() => undefined}
/>,
@ -135,10 +143,13 @@ describe('AI provider controls', () => {
anthropic: false,
xai: false,
openrouter: false,
'opencode-zen': false,
'opencode-go': false,
}}
transportAvailability={availableTransports}
onModelSelect={() => undefined}
onReasoningEffortChange={() => undefined}
onVisionModelChange={() => undefined}
onManageKeys={() => undefined}
onConnectCodex={() => undefined}
/>,
@ -168,10 +179,13 @@ describe('AI provider controls', () => {
anthropic: false,
xai: false,
openrouter: false,
'opencode-zen': false,
'opencode-go': false,
}}
transportAvailability={availableTransports}
onModelSelect={() => undefined}
onReasoningEffortChange={() => undefined}
onVisionModelChange={() => undefined}
onManageKeys={() => undefined}
onConnectCodex={() => undefined}
/>,
@ -202,10 +216,13 @@ describe('AI provider controls', () => {
anthropic: false,
xai: false,
openrouter: false,
'opencode-zen': false,
'opencode-go': false,
}}
transportAvailability={availableTransports}
onModelSelect={() => undefined}
onReasoningEffortChange={() => undefined}
onVisionModelChange={() => undefined}
onManageKeys={() => undefined}
onConnectCodex={() => undefined}
/>,
@ -238,10 +255,13 @@ describe('AI provider controls', () => {
anthropic: false,
xai: false,
openrouter: true,
'opencode-zen': false,
'opencode-go': false,
}}
transportAvailability={availableTransports}
onModelSelect={() => undefined}
onReasoningEffortChange={() => undefined}
onVisionModelChange={() => undefined}
onManageKeys={() => undefined}
onConnectCodex={() => undefined}
/>,
@ -265,10 +285,13 @@ describe('AI provider controls', () => {
anthropic: false,
xai: false,
openrouter: false,
'opencode-zen': false,
'opencode-go': false,
}}
transportAvailability={{ ...availableTransports, moonshot: false }}
onModelSelect={() => undefined}
onReasoningEffortChange={() => undefined}
onVisionModelChange={() => undefined}
onManageKeys={() => undefined}
onConnectCodex={() => undefined}
/>,
@ -278,4 +301,37 @@ describe('AI provider controls', () => {
expect(markup).toContain('Moonshot · local proxy unavailable')
expect(markup).not.toContain('Enter API key')
})
it('offers a vision fallback picker for OpenCode models without native vision', () => {
const markup = renderToStaticMarkup(
<AiProviderControls
selection={{
provider: 'opencode-go',
model: 'opencode-go/glm-5.3',
reasoningEffort: 'high',
}}
availability={{
...availableTransports,
moonshot: false,
google: false,
qwen: false,
openai: false,
anthropic: false,
xai: false,
openrouter: false,
}}
transportAvailability={availableTransports}
onModelSelect={() => undefined}
onReasoningEffortChange={() => undefined}
onVisionModelChange={() => undefined}
onManageKeys={() => undefined}
onConnectCodex={() => undefined}
/>,
)
expect(markup).toContain('OpenCode Go · GLM 5.3')
expect(markup).toContain('Vision fallback')
expect(markup).toContain('cannot see images')
expect(markup).toContain('Other models…')
})
})

View File

@ -1,4 +1,13 @@
import { useState } from 'react'
import { useEffect, useState } from 'react'
import {
isOpencodeProviderId,
listOpencodeVisionModels,
loadAllOpencodeModels,
loadOpencodeModels,
pickOpencodeVisionModel,
type OpencodeProviderId,
} from '../ai/opencode-models'
import { loadOpenRouterModels } from '../ai/openrouter-models'
import {
AI_PROVIDERS,
AI_REASONING_EFFORT_LABELS,
@ -6,11 +15,16 @@ import {
findAiModelById,
getAiModel,
getAiProvider,
getDynamicProviderModels,
modelSupportsVision,
type AiModelOption,
type AiModelSelection,
type AiProviderId,
type AiProviderOption,
type AiReasoningEffort,
} from '../ai/provider-catalog'
import {
AiGenerative,
AiNetwork,
AlertCircle,
ChatGpt,
@ -21,7 +35,7 @@ import {
KimiAi,
Qwen,
} from './icons'
import { OpenRouterModelBrowser } from './OpenRouterModelBrowser'
import { AiCatalogModelBrowser } from './OpenRouterModelBrowser'
import { Button } from './ui/button'
import {
Popover,
@ -45,7 +59,24 @@ import type { ComponentType } from 'react'
* browser instead of selecting a model. Real OpenRouter ids use `vendor/model`,
* so the `:` namespace cannot collide.
*/
const OPENROUTER_BROWSE_VALUE = 'openrouter:browse-all'
const CATALOG_BROWSE_SUFFIX = ':browse-all'
const catalogBrowseValue = (providerId: AiProviderId): string =>
`${providerId}${CATALOG_BROWSE_SUFFIX}`
const catalogProviderFromBrowseValue = (value: string): AiProviderId | null => {
if (!value.endsWith(CATALOG_BROWSE_SUFFIX)) return null
const providerId = value.slice(0, -CATALOG_BROWSE_SUFFIX.length)
return AI_PROVIDERS.some((provider) => provider.id === providerId)
? providerId as AiProviderId
: null
}
const CATALOG_BROWSE_PROVIDERS = new Set<AiProviderId>([
'openrouter',
'opencode-zen',
'opencode-go',
])
const AI_PROVIDER_ICONS: Record<AiProviderId, ComponentType<{ className?: string; size?: number }>> = {
codex: ChatGpt,
@ -56,6 +87,8 @@ const AI_PROVIDER_ICONS: Record<AiProviderId, ComponentType<{ className?: string
anthropic: Claude,
xai: Grok,
openrouter: AiNetwork,
'opencode-zen': AiGenerative,
'opencode-go': AiGenerative,
}
export type AiProviderControlsProps = {
@ -64,6 +97,7 @@ export type AiProviderControlsProps = {
transportAvailability: AiProviderAvailability
onModelSelect: (providerId: AiProviderId, modelId: string) => void
onReasoningEffortChange: (reasoningEffort: AiReasoningEffort) => void
onVisionModelChange: (visionModel: string) => void
onManageKeys: (providerId: AiProviderId) => void
onConnectCodex: () => void
}
@ -76,22 +110,192 @@ const triggerLabel = (
return `${modelLabel} · ${AI_REASONING_EFFORT_LABELS[reasoningEffort]}`
}
const uniqueVisionModels = (providerId: OpencodeProviderId): AiModelOption[] => {
const models = [
...getAiProvider(providerId).models,
...getDynamicProviderModels(providerId),
]
const seen = new Set<string>()
return listOpencodeVisionModels(models).filter((model) => {
if (seen.has(model.id)) return false
seen.add(model.id)
return true
})
}
const providerGroupCaption = (
option: AiProviderOption,
availability: AiProviderAvailability,
transportAvailability: AiProviderAvailability,
): string => {
if (availability[option.id]) return option.label
if (option.id === 'codex') return `${option.label} · connect`
if (transportAvailability[option.id]) return `${option.label} · key missing`
return `${option.label} · local proxy unavailable`
}
const CatalogModelBrowsers = ({
browsingProvider,
selectedModelId,
onModelSelect,
}: {
browsingProvider: AiProviderId | null
selectedModelId: string
onModelSelect: (providerId: AiProviderId, modelId: string) => void
}) => {
// The key remounts the browser per provider — its load-once effect would
// otherwise keep the previous provider's catalog when React reuses the instance.
if (browsingProvider === 'openrouter') {
return (
<AiCatalogModelBrowser
key="openrouter"
heading="All OpenRouter models"
selectedModelId={selectedModelId}
loadModels={loadOpenRouterModels}
onModelSelect={(modelId) => onModelSelect('openrouter', modelId)}
/>
)
}
if (!isOpencodeProviderId(browsingProvider)) return null
return (
<AiCatalogModelBrowser
key={browsingProvider}
heading={`All ${getAiProvider(browsingProvider).label} models`}
selectedModelId={selectedModelId}
loadModels={() => loadOpencodeModels(browsingProvider)}
onModelSelect={(modelId) => onModelSelect(browsingProvider, modelId)}
/>
)
}
const VisionFallbackPicker = ({
codingModelLabel,
visionModel,
visionModels,
onVisionModelChange,
}: {
codingModelLabel: string
visionModel: AiModelOption
visionModels: readonly AiModelOption[]
onVisionModelChange: (visionModel: string) => void
}) => (
<div className="ai-model-picker-section">
<span className="ai-model-picker-section__label" id="ai-vision-label">
Vision fallback
</span>
<Select
value={visionModel.id}
onValueChange={(value) => {
if (typeof value === 'string') onVisionModelChange(value)
}}
>
<SelectTrigger
id="ai-vision-trigger"
className="ai-provider-trigger"
aria-labelledby="ai-vision-label"
aria-label="Vision fallback model"
>
<SelectValue>
<span>{visionModel.label}</span>
</SelectValue>
</SelectTrigger>
<SelectContent align="start" className="ai-modal-select-content">
{visionModels.map((option) => (
<SelectItem key={option.id} value={option.id}>
{option.label}
</SelectItem>
))}
</SelectContent>
</Select>
<small className="ai-provider-description">
{codingModelLabel} cannot see images. {visionModel.label} will describe
screenshots and previews first.
</small>
</div>
)
const ProviderSetupWarning = ({
providerId,
keyLabel,
isTransportAvailable,
onManageKeys,
onConnectCodex,
}: {
providerId: AiProviderId
keyLabel: string
isTransportAvailable: boolean
onManageKeys: (providerId: AiProviderId) => void
onConnectCodex: () => void
}) => {
const isCodex = providerId === 'codex'
return (
<div className="ai-provider-warning" role="alert">
<AlertCircle size={15} />
<div className="ai-provider-warning-body">
<span>
{isCodex
? <><b>Codex isn’t connected.</b> Use your ChatGPT plan without an API key.</>
: isTransportAvailable
? <><b>API key missing.</b> Add your {keyLabel} key to generate with this model.</>
: <><b>Local proxy unavailable.</b> Moonshot can be used only from localhost.</>}
</span>
{(isCodex || isTransportAvailable) && (
<Button
type="button"
size="sm"
variant="outline"
className="ai-provider-warning-action"
onClick={() => {
if (isCodex) onConnectCodex()
else onManageKeys(providerId)
}}
>
{isCodex ? 'Connect Codex' : 'Enter API key'}
</Button>
)}
</div>
</div>
)
}
export const AiProviderControls = ({
selection,
availability,
transportAvailability,
onModelSelect,
onReasoningEffortChange,
onVisionModelChange,
onManageKeys,
onConnectCodex,
}: AiProviderControlsProps) => {
const [isBrowsingOpenRouter, setBrowsingOpenRouter] = useState(false)
const [browsingProvider, setBrowsingProvider] = useState<AiProviderId | null>(null)
const [catalogRevision, setCatalogRevision] = useState(0)
const provider = getAiProvider(selection.provider)
const model = getAiModel(selection)
const isConfigured = availability[selection.provider]
const isTransportAvailable = transportAvailability[selection.provider]
const reasoningEffort = clampAiReasoningEffort(model, selection.reasoningEffort)
const SelectedIcon = AI_PROVIDER_ICONS[selection.provider]
const keyLabel = provider.keyGroup?.label ?? provider.label
const needsVisionFallback = isOpencodeProviderId(selection.provider)
&& !modelSupportsVision(model)
void catalogRevision
const visionModels = isOpencodeProviderId(selection.provider)
? uniqueVisionModels(selection.provider)
: []
const visionModel = needsVisionFallback && isOpencodeProviderId(selection.provider)
? pickOpencodeVisionModel(selection.provider, visionModels, selection.visionModel)
: undefined
useEffect(() => {
let cancelled = false
void loadAllOpencodeModels().then(() => {
if (!cancelled) setCatalogRevision((current) => current + 1)
})
return () => {
cancelled = true
}
}, [])
return (
<Popover>
@ -129,11 +333,12 @@ export const AiProviderControls = ({
value={selection.model}
onValueChange={(value) => {
if (typeof value !== 'string') return
if (value === OPENROUTER_BROWSE_VALUE) {
setBrowsingOpenRouter(true)
const browseProvider = catalogProviderFromBrowseValue(value)
if (browseProvider) {
setBrowsingProvider(browseProvider)
return
}
setBrowsingOpenRouter(false)
setBrowsingProvider(null)
const match = findAiModelById(value)
onModelSelect(match.provider.id, match.model.id)
}}
@ -156,13 +361,7 @@ export const AiProviderControls = ({
<SelectLabel>
<ProviderIcon className="ai-provider-icon" size={14} />
<span>
{availability[option.id]
? option.label
: option.id === 'codex'
? `${option.label} · connect`
: transportAvailability[option.id]
? `${option.label} · key missing`
: `${option.label} · local proxy unavailable`}
{providerGroupCaption(option, availability, transportAvailability)}
</span>
</SelectLabel>
{option.models.map((modelOption) => (
@ -170,8 +369,8 @@ export const AiProviderControls = ({
{modelOption.label}
</SelectItem>
))}
{option.id === 'openrouter' && (
<SelectItem value={OPENROUTER_BROWSE_VALUE}>
{CATALOG_BROWSE_PROVIDERS.has(option.id) && (
<SelectItem value={catalogBrowseValue(option.id)}>
Other models…
</SelectItem>
)}
@ -182,10 +381,18 @@ export const AiProviderControls = ({
</Select>
</div>
{isBrowsingOpenRouter && (
<OpenRouterModelBrowser
selectedModelId={selection.model}
onModelSelect={(modelId) => onModelSelect('openrouter', modelId)}
<CatalogModelBrowsers
browsingProvider={browsingProvider}
selectedModelId={selection.model}
onModelSelect={onModelSelect}
/>
{needsVisionFallback && visionModel && (
<VisionFallbackPicker
codingModelLabel={model.label}
visionModel={visionModel}
visionModels={visionModels}
onVisionModelChange={onVisionModelChange}
/>
)}
@ -230,32 +437,13 @@ export const AiProviderControls = ({
</div>
{!isConfigured && (
<div className="ai-provider-warning" role="alert">
<AlertCircle size={15} />
<div className="ai-provider-warning-body">
<span>
{selection.provider === 'codex'
? <><b>Codex isn’t connected.</b> Use your ChatGPT plan without an API key.</>
: isTransportAvailable
? <><b>API key missing.</b> Add your {provider.label} key to generate with this model.</>
: <><b>Local proxy unavailable.</b> Moonshot can be used only from localhost.</>}
</span>
{(selection.provider === 'codex' || isTransportAvailable) && (
<Button
type="button"
size="sm"
variant="outline"
className="ai-provider-warning-action"
onClick={() => {
if (selection.provider === 'codex') onConnectCodex()
else onManageKeys(selection.provider)
}}
>
{selection.provider === 'codex' ? 'Connect Codex' : 'Enter API key'}
</Button>
)}
</div>
</div>
<ProviderSetupWarning
providerId={selection.provider}
keyLabel={keyLabel}
isTransportAvailable={isTransportAvailable}
onManageKeys={onManageKeys}
onConnectCodex={onConnectCodex}
/>
)}
</PopoverContent>
</Popover>

View File

@ -1,17 +1,21 @@
import { useEffect, useId, useMemo, useState } from 'react'
import {
loadOpenRouterModels,
type OpenRouterModelsResult,
} from '../ai/openrouter-models'
import { Search01 } from './icons'
import { Input } from './ui/input'
import type { AiModelOption } from '../ai/provider-catalog'
const VISIBLE_MODEL_LIMIT = 60
export type OpenRouterModelBrowserProps = {
export type CatalogModelsResult = {
models: readonly AiModelOption[]
source: 'remote' | 'cache' | 'fallback'
}
export type AiCatalogModelBrowserProps = {
heading: string
selectedModelId: string
onModelSelect: (modelId: string) => void
loadModels: () => Promise<CatalogModelsResult>
fallbackNote?: string
}
const filterModels = (
@ -26,26 +30,30 @@ const filterModels = (
}
/**
* Searchable list over the full OpenRouter catalog (vision + tools capable
* models only). Shown in the model picker once the OpenRouter provider is
* selected; the curated shortlist stays in the grouped select above.
* Searchable list over a runtime provider catalog. Shown in the model picker
* once the user chooses "Other models…".
*/
export const OpenRouterModelBrowser = ({
export const AiCatalogModelBrowser = ({
heading,
selectedModelId,
onModelSelect,
}: OpenRouterModelBrowserProps) => {
loadModels,
fallbackNote = 'Model list could not be loaded — showing the built-in shortlist.',
}: AiCatalogModelBrowserProps) => {
const labelId = useId()
const [query, setQuery] = useState('')
const [result, setResult] = useState<OpenRouterModelsResult | null>(null)
const [result, setResult] = useState<CatalogModelsResult | null>(null)
useEffect(() => {
let cancelled = false
void loadOpenRouterModels().then((loaded) => {
void loadModels().then((loaded) => {
if (!cancelled) setResult(loaded)
})
return () => {
cancelled = true
}
// Load once when this catalog panel mounts; the parent remounts it per provider.
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [])
const models = useMemo(
@ -58,7 +66,7 @@ export const OpenRouterModelBrowser = ({
return (
<div className="ai-model-picker-section">
<span className="ai-model-picker-section__label" id={labelId}>
{result ? `All OpenRouter models (${models.length})` : 'All OpenRouter models'}
{result ? `${heading} (${models.length})` : heading}
</span>
<div className="ai-openrouter-search">
<Search01 size={13} aria-hidden="true" />
@ -72,7 +80,7 @@ export const OpenRouterModelBrowser = ({
</div>
{result?.source === 'fallback' && (
<small className="ai-openrouter-note" role="status">
Model list could not be loaded — showing the built-in shortlist.
{fallbackNote}
</small>
)}
<div className="ai-openrouter-list" role="listbox" aria-labelledby={labelId}>

View File

@ -39,6 +39,11 @@ describe('deviceElementFromAttributes', () => {
expect(deviceElementFromAttributes({ screenshot: 'https://evil.example/shot.png' }, 0).screenshot).toBeUndefined()
})
it('accepts an object URL resolved from the Blob asset store', () => {
const screenshot = 'blob:http://127.0.0.1:4173/0b1c2d3e'
expect(deviceElementFromAttributes({ screenshot }, 0).screenshot).toBe(screenshot)
})
it('clamps shadow into the editor range', () => {
expect(deviceElementFromAttributes({ shadow: '400' }, 0).shadow).toBe(100)
expect(deviceElementFromAttributes({ shadow: '-4' }, 0).shadow).toBe(0)

View File

@ -27,9 +27,13 @@ export const deviceElementFromAttributes = (
const mockup = attributes.mockup ?? ''
const theme = attributes.theme ?? ''
const shadow = Number(attributes.shadow ?? '')
// Only an already-resolved upload (data URL) may reach the screen; an
// Only an already-resolved upload may reach the screen: an object URL from
// the Blob asset store, or a data URL from its storage fallback. An
// unresolved asset: reference or anything else falls back to the fake screen.
const screenshot = attributes.screenshot?.startsWith('data:image/') ? attributes.screenshot : undefined
const rawScreenshot = attributes.screenshot
const screenshot = rawScreenshot && (rawScreenshot.startsWith('blob:') || rawScreenshot.startsWith('data:image/'))
? rawScreenshot
: undefined
return {
id: `html-device-${index}`,

1
src/vite-env.d.ts vendored
View File

@ -10,6 +10,7 @@ interface ImportMetaEnv {
readonly VITE_ANTHROPIC_API_KEY?: string
readonly VITE_XAI_API_KEY?: string
readonly VITE_OPENROUTER_API_KEY?: string
readonly VITE_OPENCODE_API_KEY?: string
}
interface ImportMeta {

View File

@ -10,12 +10,24 @@ export default defineConfig(({ command, mode }) => {
const env = loadEnv(mode, process.cwd(), '')
const aiLoggingValue = env['SHOTLUMA_AI_LOGGING'] ?? env['FRAMEFLOW_AI_LOGGING']
const isAiLoggingEnabled = aiLoggingValue?.trim().toLowerCase() === 'true'
const moonshotProxy = {
const moonshotAndOpencodeProxy = {
'/api/moonshot': {
target: 'https://api.moonshot.ai',
changeOrigin: true,
rewrite: (requestPath: string) => requestPath.replace(/^\/api\/moonshot/, ''),
},
'/api/opencode/go': {
target: 'https://opencode.ai',
changeOrigin: true,
rewrite: (requestPath: string) =>
requestPath.replace(/^\/api\/opencode\/go/, '/zen/go') || '/zen/go',
},
'/api/opencode/zen': {
target: 'https://opencode.ai',
changeOrigin: true,
rewrite: (requestPath: string) =>
requestPath.replace(/^\/api\/opencode\/zen/, '/zen') || '/zen',
},
}
return {
@ -40,10 +52,10 @@ export default defineConfig(({ command, mode }) => {
server: {
host: '127.0.0.1',
port: Number(process.env['PORT']) || 4173,
proxy: moonshotProxy,
proxy: moonshotAndOpencodeProxy,
},
preview: {
proxy: moonshotProxy,
proxy: moonshotAndOpencodeProxy,
},
}
})

View File

@ -10,7 +10,9 @@ export default defineConfig({
reporter: ['text', 'lcov'],
include: [
'src/ai/chroma-key.ts',
'src/ai/describe-images.ts',
'src/ai/normalize-copy.ts',
'src/ai/opencode-models.ts',
'src/ai/overlay-asset-prompt.ts',
'src/ai/provider-config.ts',
'src/ai/provider-catalog.ts',

View File

@ -2,9 +2,12 @@
"$schema": "./node_modules/wrangler/config-schema.json",
"name": "shotluma-app",
"compatibility_date": "2026-08-03",
"main": "scripts/shotluma-worker.ts",
"assets": {
"directory": "./dist",
"not_found_handling": "single-page-application"
"binding": "ASSETS",
"not_found_handling": "single-page-application",
"run_worker_first": ["/api/opencode/*"]
},
"routes": [
{