Miraiclip SDK
AI

Editing assistant

An agent loop that edits your project.

@miraiclip/assistant runs an LLM agent loop against a project. The user types "add dissolves between the clips", and the edit lands as one undo step. It includes a vendor-neutral model contract, an OpenAI-compatible adapter, editing tools, and a server handler that keeps your API key off the browser.

npm install @miraiclip/assistant
import { createAssistant, openAIChatModel } from "@miraiclip/assistant";

const assistant = createAssistant({
  model: openAIChatModel({ apiKey: process.env.OPENAI_API_KEY, model: "gpt-5.4-mini" }),
});

const turn = await assistant.run(project, "start the title at 1 second");
turn.reply;   // "Moved the title to 1 s."
turn.changes; // ["Moved clip title to 1s"]
project.undo(); // reverts the whole request

In a browser app, keep the key on your server: Keys stay on the server. To build your own loop on core instead, see Tools from the command catalog.

How a turn runs

  1. The assistant copies the project. Tools edit the copy while the model works.
  2. The model gets a system prompt with a project summary (describeForAssistant), your context, and the tools.
  3. Each tool call runs against the copy. Failures go back to the model so it can fix its input.
  4. When the model answers without calling a tool, the commands that changed the copy are replayed onto your project in one transaction.

If the turn fails or is cancelled, your project isn't touched.

createAssistant options

OptionDefaultWhat it does
modelrequiredA ChatModel
toolseditorTools()The tools the model can call
instructionsnoneExtra system instructions, added to DEFAULT_INSTRUCTIONS
maxSteps12Model round-trips per request
modelIdthe adapter's defaultModel id, when the adapter serves several
maxOutputTokensadapter defaultPer model call
paramsnoneVendor request fields, passed through
forkcreateProject(doc){ createFork?, ensureIds? }. Register custom commands on the copy with createFork

assistant.run(project, request, options?)

import { createAssistant, type ChatMessage, type ChatModel } from "@miraiclip/assistant";

declare const model: ChatModel;
const assistant = createAssistant({ model });
let history: ChatMessage[] = [];
const controller = new AbortController();

const turn = await assistant.run(project, "make the title pop in", {
  history,
  context: "Selected: clip title. Playhead: 4.2s.",
  signal: controller.signal,
  onEvent: (event) => {
    if (event.type === "text") console.log(event.delta);
    if (event.type === "tool-end") console.log(event.call.name, event.ok, event.changes);
  },
});
history = turn.messages; // continue the conversation next time
OptionDefaultWhat it does
historynoneEarlier messages: the last turn's turn.messages
contextnoneWhat the user is looking at: selection, playhead
apply"auto""auto" commits when the turn ends. "review" waits for turn.apply()
signalnoneCancel the turn. The project isn't changed
onEventnonetext (delta), tool-start (call), tool-end (call, ok, changes?, error?), step (index)
label"Assistant: <request>"Undo history label

The turn

FieldMeaning
replyThe model's final answer
changesHuman-readable lines: what changed
commandsThe commands the turn applies, in order
status"applied", "review", "no-changes", "failed" or "canceled"
errorSet when the turn failed
previewThe edited document, for previews in review mode
messagesThe conversation without the system prompt. Pass it back as history
usage{ inputTokens, outputTokens }
apply()Commit a reviewed turn. Returns { ok: true, applied } or { ok: false, error }
discard()Drop a reviewed turn

Review mode

import { createAssistant, type ChatModel } from "@miraiclip/assistant";

declare const model: ChatModel;
const assistant = createAssistant({ model });

const turn = await assistant.run(project, "move the title to 2 seconds", { apply: "review" });
turn.status;  // "review": the project is unchanged
turn.preview; // the edited document: render it to show the result
turn.changes; // show these with Apply / Discard buttons

const result = turn.apply(); // or turn.discard()
if (!result.ok) console.warn(result.error); // the project changed in a way the edit no longer fits

Tools

editorTools() is the default set:

ToolWhat it does
get_stateThe project summary, optionally with the full document JSON
get_command_schemaOne command's payload schema
apply_commandsA batch of core commands, all or nothing. Fields a command would ignore are errors, and commands that changed nothing are reported
add_transitionAdd, replace or remove transitions on cuts
animate_clipIn, loop and out animation presets on a visual clip
add_effect / remove_effectsAdd effects from core's catalog, or remove them
set_effects_enabledTurn effects off or on without deleting them
trim_clipEdge trims in seconds, with optional ripple
set_keyframesKeyframes for one property at timeline seconds
set_backgroundA solid color background on a bottom track
close_gapsPack a track's clips back to back
editorTools optionDefaultWhat it does
transitionKindsthe built-insAdd your custom transition kinds
commandCatalogthe built-insPass project.commandCatalog() when you register custom commands

Your own tools

import { createAssistant, defineTool, editorTools, type ChatModel } from "@miraiclip/assistant";

declare const model: ChatModel;

const addEndCard = defineTool({
  name: "add_end_card",
  description: "Add a 2-second end card with the given text after the last clip.",
  inputSchema: {
    type: "object",
    properties: { text: { type: "string" } },
    required: ["text"],
    additionalProperties: false,
  },
  run(input, { project }) {
    // `project` is the turn's working copy: edit through it
    const clips = Object.values(project.getState().doc.clips);
    const endUs = Math.max(0, ...clips.map((clip) => clip.startUs + clip.durationUs));
    project.dispatch({ type: "track/add", payload: { id: "end-card", kind: "video" } });
    project.dispatch({
      type: "clip/add",
      payload: {
        kind: "text", id: "end-card-text", trackId: "end-card", startUs: endUs, durationUs: 2_000_000,
        text: String(input["text"]), fontFamily: "Inter", fontSizePx: 64, color: "#ffffff",
      },
    });
    return { ok: true, changes: ["Added an end card"] };
  },
});

const assistant = createAssistant({
  model,
  tools: [...editorTools(), addEndCard],
  instructions: "Keep titles under six words.",
});
run returnsEffect
{ ok: true, result?, changes? }result goes back to the model as JSON. changes are added to turn.changes
{ ok: false, error }The error goes back to the model

toolsFromDefinitions(definitions, run) adopts tools that come as definitions plus one runner, such as audioToolDefinitions and runAudioTool from @miraiclip/audio-sources. It accepts Anthropic-style (input_schema) and OpenAI function definitions.

Models

A model is any object with this shape:

import type { ChatOptions, ChatRequest, ChatResponse } from "@miraiclip/assistant";

interface ChatModel {
  id: string;                                  // "openai"; letters, digits, - or _
  label: string;
  models?: { id: string; label?: string }[];   // the first is the default
  complete(request: ChatRequest, options?: ChatOptions): Promise<ChatResponse>;
}
TypeFields
ChatRequestmessages, tools? ({ name, description, inputSchema }), model?, maxOutputTokens?, params?
ChatMessagesystem / user (content), assistant (content, toolCalls?), tool (toolCallId, name, content)
ChatResponsecontent, toolCalls ({ id, name, arguments }), finish, model?, usage?
ChatOptionssignal?, onText?(delta) for streaming text

OpenAI and compatible servers

openAIChatModel uses the Chat Completions API with function calling and streaming. Many servers speak it, so baseUrl points it at Azure OpenAI, OpenRouter, Groq, vLLM, Ollama or LM Studio.

import { openAIChatModel } from "@miraiclip/assistant";

const local = openAIChatModel({
  id: "ollama",
  label: "Ollama",
  baseUrl: "http://localhost:11434/v1",
  model: "llama3.1",
});
OptionDefaultWhat it does
modelrequiredModel id
modelsnoneOther models the app may pick per request
apiKeynoneOmit for local servers
baseUrlhttps://api.openai.com/v1Any OpenAI-compatible base URL
organization, project, headersnoneSent with every request
paramsnoneRequest fields merged into every call, e.g. { reasoning_effort: "low" }
streamtrueStream responses
maxRetries4Retries on 429 and 500/502/503/504, waiting as long as the server asks, else 1 s, 2 s, 4 s… up to 30 s. 0 disables
id, label"openai", "OpenAI"Adapter identity
fetchglobal fetchCustom fetch

Vendor errors are thrown as ChatModelError with provider and status.

Other vendors

Other vendors need an adapter: map ChatRequest to the vendor's API and the reply back to ChatResponse. defineChatModel checks the shape once.

import { defineChatModel, type ChatRequest, type ChatResponse } from "@miraiclip/assistant";

declare function callMyVendor(request: ChatRequest, signal?: AbortSignal): Promise<ChatResponse>;

const model = defineChatModel({
  id: "my-vendor",
  label: "My vendor",
  async complete(request, options) {
    const response = await callMyVendor(request, options?.signal);
    options?.onText?.(response.content); // adapters that can't stream call it once
    return response;
  },
});

Rate limits

rateLimitedChatModel makes calls wait for room under a per-minute budget instead of failing with 429s. Use it for batch jobs or a server sharing one key.

import { openAIChatModel, rateLimitedChatModel } from "@miraiclip/assistant";

const model = rateLimitedChatModel(openAIChatModel({ apiKey: process.env.OPENAI_API_KEY, model: "gpt-5.4-mini" }), {
  tokensPerMinute: 150_000,
  requestsPerMinute: 400,
});

Each call reserves an estimate (estimate, default: request JSON length ÷ 4 plus room for the reply), replaced by the real usage when the response reports it.

Keys stay on the server

The agent loop and tools run in the browser, next to the project. Only model calls go over the network: mount createChatHandler on your server, and use remoteChatModel in the browser.

// Server: any Fetch API runtime (Node 18+, edge runtimes, Next, Hono…)
import { createChatHandler, openAIChatModel } from "@miraiclip/assistant";

declare function isSignedIn(request: Request): boolean;

const handle = createChatHandler(openAIChatModel({ apiKey: process.env.OPENAI_API_KEY, model: "gpt-5.4-mini" }), {
  basePath: "/api/assistant",
  authorize: (request) => (isSignedIn(request) ? undefined : new Response("sign in", { status: 401 })),
  prepare: (request) => ({ ...request, maxOutputTokens: 2000 }),
});

export async function fetchHandler(request: Request): Promise<Response> {
  return (await handle(request)) ?? new Response("not found", { status: 404 });
}
// Browser
import { createAssistant, remoteChatModel } from "@miraiclip/assistant";

const assistant = createAssistant({ model: await remoteChatModel("/api/assistant") });
Handler optionDefaultWhat it does
basePathnoneWhere the handler is mounted
maxBodyBytes2 MBLarger request bodies are rejected
authorizenoneReturn a Response to refuse a request (auth, rate limits), or undefined to allow
preparenoneChange each request on the server: pin a model, cap tokens, add instructions

The handler returns null for paths it doesn't own. It serves:

RouteResponse
GET {basePath}/{ id, label, models }
POST {basePath}/chat with { request: ChatRequest }NDJSON lines: { "type": "text", "delta" } while streaming, then { "type": "done", "response" } or { "type": "error", "error", "status" }

Protect the endpoint

The handler sends whatever the browser asks to your key. Put authentication and per-user rate limits in authorize.

remoteChatModel(endpoint, options?) takes fetch, headers, and info (skip the GET for model info).

Tests and demos

scriptedChatModel answers from a script, with no network. A string is a final answer; an object can call tools.

import { createAssistant, scriptedChatModel } from "@miraiclip/assistant";

const model = scriptedChatModel([
  { toolCalls: [{ name: "apply_commands", arguments: { commands: [{ type: "clip/move", payload: { clipId: "title", startUs: 1_000_000 } }] } }] },
  "Moved the title to 1 s.",
]);

const turn = await createAssistant({ model }).run(project, "start the title at 1 second");
turn.status;    // "applied"
model.requests; // every ChatRequest the assistant sent

Lower-level pieces

ExportWhat it does
describeForAssistant(doc)describeProject plus track names, cuts, and each clip's position, size, opacity, text style, volume, effects and animation
DEFAULT_INSTRUCTIONSThe built-in system instructions
forkProject(project, options?) / commitFork(project, fork, label?)The working copy that records commands, and the one-step replay
withExplicitIds(command)Gives built-in commands with optional ids an explicit id, so replay produces the ids the model saw
applyChecked(project, commands)apply_commands' batch logic: all or nothing, unknown fields are errors, no-op commands reported
applyCommandsSchema(catalog)apply_commands' input schema
describeCommands(commands)One readable line per command
toOpenAIMessages, parseToolArguments, retryDelayMs, unknownFieldsHelpers for writing adapters and tools

On this page