Editing assistant
An agent loop that edits your project.
@miraiclip/assistant runs an LLM agent loop against a project. The user types "add dissolves between the clips", and the edit lands as one undo step. It includes a vendor-neutral model contract, an OpenAI-compatible adapter, editing tools, and a server handler that keeps your API key off the browser.
npm install @miraiclip/assistantimport { createAssistant, openAIChatModel } from "@miraiclip/assistant";
const assistant = createAssistant({
model: openAIChatModel({ apiKey: process.env.OPENAI_API_KEY, model: "gpt-5.4-mini" }),
});
const turn = await assistant.run(project, "start the title at 1 second");
turn.reply; // "Moved the title to 1 s."
turn.changes; // ["Moved clip title to 1s"]
project.undo(); // reverts the whole requestIn a browser app, keep the key on your server: Keys stay on the server. To build your own loop on core instead, see Tools from the command catalog.
How a turn runs
- The assistant copies the project. Tools edit the copy while the model works.
- The model gets a system prompt with a project summary (
describeForAssistant), yourcontext, and the tools. - Each tool call runs against the copy. Failures go back to the model so it can fix its input.
- When the model answers without calling a tool, the commands that changed the copy are replayed onto your project in one transaction.
If the turn fails or is cancelled, your project isn't touched.
createAssistant options
| Option | Default | What it does |
|---|---|---|
model | required | A ChatModel |
tools | editorTools() | The tools the model can call |
instructions | none | Extra system instructions, added to DEFAULT_INSTRUCTIONS |
maxSteps | 12 | Model round-trips per request |
modelId | the adapter's default | Model id, when the adapter serves several |
maxOutputTokens | adapter default | Per model call |
params | none | Vendor request fields, passed through |
fork | createProject(doc) | { createFork?, ensureIds? }. Register custom commands on the copy with createFork |
assistant.run(project, request, options?)
import { createAssistant, type ChatMessage, type ChatModel } from "@miraiclip/assistant";
declare const model: ChatModel;
const assistant = createAssistant({ model });
let history: ChatMessage[] = [];
const controller = new AbortController();
const turn = await assistant.run(project, "make the title pop in", {
history,
context: "Selected: clip title. Playhead: 4.2s.",
signal: controller.signal,
onEvent: (event) => {
if (event.type === "text") console.log(event.delta);
if (event.type === "tool-end") console.log(event.call.name, event.ok, event.changes);
},
});
history = turn.messages; // continue the conversation next time| Option | Default | What it does |
|---|---|---|
history | none | Earlier messages: the last turn's turn.messages |
context | none | What the user is looking at: selection, playhead |
apply | "auto" | "auto" commits when the turn ends. "review" waits for turn.apply() |
signal | none | Cancel the turn. The project isn't changed |
onEvent | none | text (delta), tool-start (call), tool-end (call, ok, changes?, error?), step (index) |
label | "Assistant: <request>" | Undo history label |
The turn
| Field | Meaning |
|---|---|
reply | The model's final answer |
changes | Human-readable lines: what changed |
commands | The commands the turn applies, in order |
status | "applied", "review", "no-changes", "failed" or "canceled" |
error | Set when the turn failed |
preview | The edited document, for previews in review mode |
messages | The conversation without the system prompt. Pass it back as history |
usage | { inputTokens, outputTokens } |
apply() | Commit a reviewed turn. Returns { ok: true, applied } or { ok: false, error } |
discard() | Drop a reviewed turn |
Review mode
import { createAssistant, type ChatModel } from "@miraiclip/assistant";
declare const model: ChatModel;
const assistant = createAssistant({ model });
const turn = await assistant.run(project, "move the title to 2 seconds", { apply: "review" });
turn.status; // "review": the project is unchanged
turn.preview; // the edited document: render it to show the result
turn.changes; // show these with Apply / Discard buttons
const result = turn.apply(); // or turn.discard()
if (!result.ok) console.warn(result.error); // the project changed in a way the edit no longer fitsTools
editorTools() is the default set:
| Tool | What it does |
|---|---|
get_state | The project summary, optionally with the full document JSON |
get_command_schema | One command's payload schema |
apply_commands | A batch of core commands, all or nothing. Fields a command would ignore are errors, and commands that changed nothing are reported |
add_transition | Add, replace or remove transitions on cuts |
animate_clip | In, loop and out animation presets on a visual clip |
add_effect / remove_effects | Add effects from core's catalog, or remove them |
set_effects_enabled | Turn effects off or on without deleting them |
trim_clip | Edge trims in seconds, with optional ripple |
set_keyframes | Keyframes for one property at timeline seconds |
set_background | A solid color background on a bottom track |
close_gaps | Pack a track's clips back to back |
editorTools option | Default | What it does |
|---|---|---|
transitionKinds | the built-ins | Add your custom transition kinds |
commandCatalog | the built-ins | Pass project.commandCatalog() when you register custom commands |
Your own tools
import { createAssistant, defineTool, editorTools, type ChatModel } from "@miraiclip/assistant";
declare const model: ChatModel;
const addEndCard = defineTool({
name: "add_end_card",
description: "Add a 2-second end card with the given text after the last clip.",
inputSchema: {
type: "object",
properties: { text: { type: "string" } },
required: ["text"],
additionalProperties: false,
},
run(input, { project }) {
// `project` is the turn's working copy: edit through it
const clips = Object.values(project.getState().doc.clips);
const endUs = Math.max(0, ...clips.map((clip) => clip.startUs + clip.durationUs));
project.dispatch({ type: "track/add", payload: { id: "end-card", kind: "video" } });
project.dispatch({
type: "clip/add",
payload: {
kind: "text", id: "end-card-text", trackId: "end-card", startUs: endUs, durationUs: 2_000_000,
text: String(input["text"]), fontFamily: "Inter", fontSizePx: 64, color: "#ffffff",
},
});
return { ok: true, changes: ["Added an end card"] };
},
});
const assistant = createAssistant({
model,
tools: [...editorTools(), addEndCard],
instructions: "Keep titles under six words.",
});run returns | Effect |
|---|---|
{ ok: true, result?, changes? } | result goes back to the model as JSON. changes are added to turn.changes |
{ ok: false, error } | The error goes back to the model |
toolsFromDefinitions(definitions, run) adopts tools that come as definitions plus one runner, such as audioToolDefinitions and runAudioTool from @miraiclip/audio-sources. It accepts Anthropic-style (input_schema) and OpenAI function definitions.
Models
A model is any object with this shape:
import type { ChatOptions, ChatRequest, ChatResponse } from "@miraiclip/assistant";
interface ChatModel {
id: string; // "openai"; letters, digits, - or _
label: string;
models?: { id: string; label?: string }[]; // the first is the default
complete(request: ChatRequest, options?: ChatOptions): Promise<ChatResponse>;
}| Type | Fields |
|---|---|
ChatRequest | messages, tools? ({ name, description, inputSchema }), model?, maxOutputTokens?, params? |
ChatMessage | system / user (content), assistant (content, toolCalls?), tool (toolCallId, name, content) |
ChatResponse | content, toolCalls ({ id, name, arguments }), finish, model?, usage? |
ChatOptions | signal?, onText?(delta) for streaming text |
OpenAI and compatible servers
openAIChatModel uses the Chat Completions API with function calling and streaming. Many servers speak it, so baseUrl points it at Azure OpenAI, OpenRouter, Groq, vLLM, Ollama or LM Studio.
import { openAIChatModel } from "@miraiclip/assistant";
const local = openAIChatModel({
id: "ollama",
label: "Ollama",
baseUrl: "http://localhost:11434/v1",
model: "llama3.1",
});| Option | Default | What it does |
|---|---|---|
model | required | Model id |
models | none | Other models the app may pick per request |
apiKey | none | Omit for local servers |
baseUrl | https://api.openai.com/v1 | Any OpenAI-compatible base URL |
organization, project, headers | none | Sent with every request |
params | none | Request fields merged into every call, e.g. { reasoning_effort: "low" } |
stream | true | Stream responses |
maxRetries | 4 | Retries on 429 and 500/502/503/504, waiting as long as the server asks, else 1 s, 2 s, 4 s… up to 30 s. 0 disables |
id, label | "openai", "OpenAI" | Adapter identity |
fetch | global fetch | Custom fetch |
Vendor errors are thrown as ChatModelError with provider and status.
Other vendors
Other vendors need an adapter: map ChatRequest to the vendor's API and the reply back to ChatResponse. defineChatModel checks the shape once.
import { defineChatModel, type ChatRequest, type ChatResponse } from "@miraiclip/assistant";
declare function callMyVendor(request: ChatRequest, signal?: AbortSignal): Promise<ChatResponse>;
const model = defineChatModel({
id: "my-vendor",
label: "My vendor",
async complete(request, options) {
const response = await callMyVendor(request, options?.signal);
options?.onText?.(response.content); // adapters that can't stream call it once
return response;
},
});Rate limits
rateLimitedChatModel makes calls wait for room under a per-minute budget instead of failing with 429s. Use it for batch jobs or a server sharing one key.
import { openAIChatModel, rateLimitedChatModel } from "@miraiclip/assistant";
const model = rateLimitedChatModel(openAIChatModel({ apiKey: process.env.OPENAI_API_KEY, model: "gpt-5.4-mini" }), {
tokensPerMinute: 150_000,
requestsPerMinute: 400,
});Each call reserves an estimate (estimate, default: request JSON length ÷ 4 plus room for the reply), replaced by the real usage when the response reports it.
Keys stay on the server
The agent loop and tools run in the browser, next to the project. Only model calls go over the network: mount createChatHandler on your server, and use remoteChatModel in the browser.
// Server: any Fetch API runtime (Node 18+, edge runtimes, Next, Hono…)
import { createChatHandler, openAIChatModel } from "@miraiclip/assistant";
declare function isSignedIn(request: Request): boolean;
const handle = createChatHandler(openAIChatModel({ apiKey: process.env.OPENAI_API_KEY, model: "gpt-5.4-mini" }), {
basePath: "/api/assistant",
authorize: (request) => (isSignedIn(request) ? undefined : new Response("sign in", { status: 401 })),
prepare: (request) => ({ ...request, maxOutputTokens: 2000 }),
});
export async function fetchHandler(request: Request): Promise<Response> {
return (await handle(request)) ?? new Response("not found", { status: 404 });
}// Browser
import { createAssistant, remoteChatModel } from "@miraiclip/assistant";
const assistant = createAssistant({ model: await remoteChatModel("/api/assistant") });| Handler option | Default | What it does |
|---|---|---|
basePath | none | Where the handler is mounted |
maxBodyBytes | 2 MB | Larger request bodies are rejected |
authorize | none | Return a Response to refuse a request (auth, rate limits), or undefined to allow |
prepare | none | Change each request on the server: pin a model, cap tokens, add instructions |
The handler returns null for paths it doesn't own. It serves:
| Route | Response |
|---|---|
GET {basePath}/ | { id, label, models } |
POST {basePath}/chat with { request: ChatRequest } | NDJSON lines: { "type": "text", "delta" } while streaming, then { "type": "done", "response" } or { "type": "error", "error", "status" } |
Protect the endpoint
The handler sends whatever the browser asks to your key. Put authentication and per-user rate limits in authorize.
remoteChatModel(endpoint, options?) takes fetch, headers, and info (skip the GET for model info).
Tests and demos
scriptedChatModel answers from a script, with no network. A string is a final answer; an object can call tools.
import { createAssistant, scriptedChatModel } from "@miraiclip/assistant";
const model = scriptedChatModel([
{ toolCalls: [{ name: "apply_commands", arguments: { commands: [{ type: "clip/move", payload: { clipId: "title", startUs: 1_000_000 } }] } }] },
"Moved the title to 1 s.",
]);
const turn = await createAssistant({ model }).run(project, "start the title at 1 second");
turn.status; // "applied"
model.requests; // every ChatRequest the assistant sentLower-level pieces
| Export | What it does |
|---|---|
describeForAssistant(doc) | describeProject plus track names, cuts, and each clip's position, size, opacity, text style, volume, effects and animation |
DEFAULT_INSTRUCTIONS | The built-in system instructions |
forkProject(project, options?) / commitFork(project, fork, label?) | The working copy that records commands, and the one-step replay |
withExplicitIds(command) | Gives built-in commands with optional ids an explicit id, so replay produces the ids the model saw |
applyChecked(project, commands) | apply_commands' batch logic: all or nothing, unknown fields are errors, no-op commands reported |
applyCommandsSchema(catalog) | apply_commands' input schema |
describeCommands(commands) | One readable line per command |
toOpenAIMessages, parseToolArguments, retryDelayMs, unknownFields | Helpers for writing adapters and tools |