Tools from the command catalog
Turn commands into LLM tool definitions.
Every edit is a validated JSON command, so an LLM can edit a project by calling commands as tools. @miraiclip/core has the pieces:
| Function | What it does |
|---|---|
toToolDefinitions(catalog, options?) | The command catalog as tool definitions |
commandTypeForTool(name, types) | Tool name back to command type: "clip_add" → "clip/add" |
tryDispatch(project, command) | Run one command; failures come back as data |
applyCommands(project, commands, options?) | Run a batch as one all-or-nothing undo step |
describeProject(doc, options?) | A short text summary of the project for the prompt |
For a ready-made loop with model adapters and higher-level tools, see Editing assistant. For MCP clients such as Claude Desktop, see MCP server.
Tool definitions
import { toToolDefinitions } from "@miraiclip/core";
const tools = toToolDefinitions(project.commandCatalog());
// [{ name: "asset_add", description: "Register a media asset …", input_schema: { … } }, …]Pass project.commandCatalog() so custom commands are included. The built-ins give 26 tools. Tool names replace / with _: clip_add, clip_set-property, keyframe_set.
| Option | Values | Default |
|---|---|---|
style | "anthropic": { name, description, input_schema }. "openai": { type: "function", function: { name, description, parameters } } | "anthropic" |
mode | "per-command": one tool per command. "dispatch": a single miraiclip_dispatch tool taking { type, payload } | "per-command" |
descriptions | { [commandType]: string }: extra or replacement descriptions, e.g. for custom commands | none |
"dispatch" mode suits hosts that limit the number of tools. Its type is an enum of every command, but the payload schema isn't included, so also give the model the payload schemas (project.commandCatalog()).
import { commandTypeForTool, toolNameForCommand } from "@miraiclip/core";
const types = Object.keys(project.commandCatalog());
commandTypeForTool("clip_add", types); // "clip/add"
commandTypeForTool("nope", types); // undefined
toolNameForCommand("keyframe/set"); // "keyframe_set"Run a tool call
tryDispatch returns { ok: true } or { ok: false, error } instead of throwing. Send the result back as the tool result: the error says what to fix.
import { tryDispatch } from "@miraiclip/core";
tryDispatch(project, { type: "clip/move", payload: { clipId: "x", startUs: 1.5 } });
// { ok: false, error: {
// kind: "invalid-payload", commandType: "clip/move",
// message: 'Invalid payload for "clip/move": startUs — Invalid input: expected int, received number',
// issues: [{ path: "startUs", message: "Invalid input: expected int, received number" }] } }error.kind | When | Extra fields |
|---|---|---|
unknown-command | No command with that type | validTypes: every accepted type |
invalid-payload | The payload fails the schema | issues: [{ path, message }] |
rejected | Valid payload, but it can't apply (unknown clip, …) | code, e.g. clip-not-found |
Every failure also has commandType and message. Errors that aren't command failures (bugs) still throw.
Apply a plan
applyCommands runs a list of commands as one transaction. If one fails, none are applied, and failedIndex says which.
import { applyCommands } from "@miraiclip/core";
const result = applyCommands(
project,
[
{ type: "clip/move", payload: { clipId: "clip-1", startUs: 0 } },
{ type: "clip/remove", payload: { clipId: "ghost" } },
],
{ label: "AI edit" },
);
// { ok: false, applied: 0, failedIndex: 1, error: { kind: "rejected", commandType: "clip/remove",
// code: "clip-not-found", message: 'Command "clip/remove" rejected (clip-not-found): no clip "ghost"' } }On success it returns { ok: true, applied }, and one project.undo() reverts the whole batch.
Describe the project
project.toJSON() is long. describeProject lists what a model needs to write commands: ids, kinds and time ranges in microseconds.
import { describeProject } from "@miraiclip/core";
const summary = describeProject(project.toJSON());Miraiclip project — 1920x1080 @ 30fps. Composition length: 8000000us (8.00s). All command times are MICROSECONDS (1 second = 1000000us).
assets:
- intro: video, 12000000us, src "/media/intro.mp4"
tracks (bottom to top):
- video-1 (video), 2 clips:
- clip-1: video[intro] at 0us..5000000us
- clip-2: video[intro] trim 1000000us at 5000000us..8000000us
transitions:
- t1: crossDissolve between clip-1 -> clip-2, 500000usThe output is deterministic. On long tracks the middle clips are left out: maxClipsPerTrack (default 50).
A complete agent loop
This loop uses the Anthropic Messages API shape. The client is declared loosely so the snippet works with any SDK version: swap in your own client.
import { commandTypeForTool, describeProject, toToolDefinitions, tryDispatch } from "@miraiclip/core";
type ContentBlock =
| { type: "text"; text: string }
| { type: "tool_use"; id: string; name: string; input: unknown };
type Message = { role: "user" | "assistant"; content: string | unknown[] };
declare const anthropic: {
messages: {
create(request: {
model: string;
max_tokens: number;
system?: string;
tools: unknown[];
messages: Message[];
}): Promise<{ content: ContentBlock[]; stop_reason: string }>;
};
};
async function edit(request: string, maxSteps = 10): Promise<string> {
const catalog = project.commandCatalog();
const types = Object.keys(catalog);
const tools = toToolDefinitions(catalog);
const messages: Message[] = [{ role: "user", content: request }];
for (let step = 0; step < maxSteps; step++) {
const response = await anthropic.messages.create({
model: "your-model-id",
max_tokens: 2048,
system: `You edit a video project by calling tools.\n\n${describeProject(project.toJSON())}`,
tools,
messages,
});
messages.push({ role: "assistant", content: response.content });
const calls = response.content.filter((block) => block.type === "tool_use");
if (calls.length === 0) {
return response.content.map((block) => (block.type === "text" ? block.text : "")).join("");
}
messages.push({
role: "user",
content: calls.map((call) => {
const type = commandTypeForTool(call.name, types);
const result = type
? tryDispatch(project, { type, payload: call.input })
: { ok: false, error: { kind: "unknown-command", validTypes: types } };
return {
type: "tool_result",
tool_use_id: call.id,
content: JSON.stringify(result),
is_error: !result.ok,
};
}),
});
}
return "Stopped after too many steps.";
}| Step | Why |
|---|---|
| The summary is rebuilt every step | The model sees its own edits |
Failures go back as tool_result with is_error | The model reads issues or code and retries |
maxSteps | Stops a model that keeps failing |
Each tryDispatch is its own undo step. To make the whole request one undo step, collect the commands and use applyCommands, or use @miraiclip/assistant, which edits a copy and commits once.
To fill a template from an LLM instead of editing freely, use toFieldToolDefinition.