Skip to content

AI Assistant — Drive PatchWork by Chat

PatchWork can be driven by the AI assistant in plain language — Claude, connected to your live browser tab, building and editing what's on your screen in real time. Ask for a title on a clip, a new scene, a duplicated project, or "wire a Prompt node into the image node," and it happens on the canvas as you watch.

This is not the same as the pipeline's generation agents

Chapter 1's Agents (generation-runner, prompt-tuning, etc.) are batch automations that run files. This is the chat assistant editing your open PatchWork tab live — a different thing. Think of it as a co-pilot sitting next to you, not a background job.

What it can drive

flowchart LR
    A[You ask<br/>in chat] --> B[AI Assistant]
    B --> C[Node canvas<br/>build / run / approve]
    B --> D[Projects<br/>list / create / organize]
    B --> E[Video Editor<br/>titles / clips / trims]
    B --> F[Scene Builder<br/>scenes / clips]

Five surfaces, all on the project you have open on screen:

Surface The assistant can…
Node canvas Add / connect / edit / remove nodes, run the workflow, approve a candidate on an approval gate, read the current graph
Projects List your projects, create new ones, open / rename / duplicate / delete / file into folders, inspect a closed project, edit a closed project in the background
Video Editor timeline Read the open timeline, add text titles, drop generated clips, move / trim / scale / reposition clips — all as one undo
Scene Builder Add scenes, drop clips into a scene, rename / remove / switch scenes
Introspection Report per-project stats — clip counts, whether the editor has content, how many exports and the last export

Turn it on (one-time)

  1. In PatchWork → Settings → Agent Access → enable. This hosts a private connection for the assistant and shows a key.
  2. Make sure that key is in your Claude Code config (it lives in a local .agent-key file — once it's set you don't touch it again).
  3. Restart Claude Code so it loads the PatchWork tools.
  4. Keep the PatchWork browser tab open. The assistant acts through your live tab — if it's closed, there's nothing to drive.

The golden rule: it acts on what's OPEN

The assistant edits whatever you currently have open. For timeline edits, the Video Editor must be open on the project you mean. For Scene Builder edits, that project must be the one open on your canvas. It won't go hunting for a project by name and open it for you before a timeline/scene edit — open it first, then ask.

The surfaces, with example asks

1. The node canvas — build and run workflows

The assistant can read your graph and edit it live: add a Prompt / Image / Video / Media / Reference / Output / Note node, wire nodes together, change a node's properties, run the workflow, and approve candidates on approval gates.

"Add a Prompt node that says 'close-up, golden hour' and connect it to the image node." "Run the workflow." "Approve the best candidate on the Media node so the video gen fires."

2. Projects — create and organize

"List my projects and tell me which ones have anything in their video editor." "Create a new project called 'Rhodiola — hook tests' in the Salvora folder." "Duplicate this project as 'V2' and open it." "Move these three into a folder called 'Archive'."

It can also edit a closed project in the background (without disturbing the canvas you're on) and inspect any project's structure without opening it.

3. The Video Editor timeline — last-mile edits

With the Video Editor open, the assistant reads the timeline (so it knows your clip ids) and applies a batch of edits as one undo step:

  • Add a text title — text, timing, position, font size, colour
  • Add a clip from a generated asset
  • Move / trim a clip (start time, track, in/out points)
  • Transform — scale, position, opacity, playback speed

"Add a centered title 'SALE ENDS SUNDAY' from 0 to 2 seconds." "Scale the B-roll clip to 0.5 and move it to track 2." "Trim the first clip to start half a second in."

Track hierarchy

Video tracks stack with track 1 at the bottom (higher tracks layer on top); audio tracks run A1 at the top downward. So "move it to track 2" puts a clip one layer above the base video.

4. The Scene Builder — assemble the reel

With a project open, the assistant can build out the Scene Builder reel:

  • Add a scene (a new tab on the reel — up to 10)
  • Drop a clip into a scene, from a generated asset or a media URL
  • Rename / remove / switch scenes

"Add a new scene called 'Hook'." "Drop the latest avatar clip into the Hook scene." "Rename scene 2 to 'CTA' and make it the active one."

A natural flow: "Add three scenes — Hook, Demo, CTA — and put the matching clips into each."

5. Introspection — ask about your library

"Which projects haven't been exported yet?" "How many clips are in this project's editor, and when was it last exported?"

Good habits & limits

  • Open first, then ask — timeline and scene edits act on the open editor / open project. (Canvas and project-management asks don't need anything pre-opened.)
  • It edits what's open, not by name — for timeline/scene work it assumes the open one is the one you mean. If the wrong project is open, it'll edit that. Glance at the screen before a big batch.
  • Clips need real assets — "drop the avatar clip in" works when that clip already exists in your asset library / was just generated. Generate first, then assemble.
  • One undo per batch — a multi-step edit reverses with a single Ctrl/Cmd+Z.
  • Scene cap is 10 — the assistant will tell you if you've hit it.
  • It confirms destructive actions — deleting a project asks first; it won't nuke things silently.

Troubleshooting

Symptom Fix
The assistant says it has no PatchWork tools Restart Claude Code so it loads them (especially right after Agent Access was first enabled, or after an update added new tools like the Scene Builder ones)
"Video editor is not open" Open the project's Video Editor first (CREATE → Video Editor, or the Video Editor button on the project), then ask again
Nothing happens / "can't connect" Agent Access is off, or the PatchWork tab is closed — re-enable it and keep the tab open
A scene clip didn't show up The media wasn't a durable link (a session-only blob: url won't persist) — use a generated asset or an https:// url
It edited the wrong project It acts on whatever's open — open the right one and re-ask

When you're ready

→ Next: Chapter 5 — Quality, Testing & Prompt Tuning. You can now drive PatchWork's creative tools both by hand and by chat — next is judging whether a generation is actually good, and when to iterate.