Prompt Tuning in Practice

The prompt-tuning skill is what you use when a prompt is close but not quite and normal regens aren't fixing it. This page is the operator-side walkthrough — when to reach for it, what the loop looks like in practice, and where it lands its outputs.

When to use it

The decision tree:

flowchart TD
    A[Image candidate isn't right]
    A --> B{All 4 candidates have the<br/>same issue?}
    B -->|No, only some| C[Pick a good candidate<br/>or rerun for new seeds]
    B -->|Yes, structural problem| D{Can a single prompt<br/>edit fix it?}
    D -->|Yes, obvious fix| E[Edit prompt, rerun]
    D -->|No, ambiguous, won't<br/>move with normal edits| F[Use prompt-tuning skill]

Prompt tuning is for the bottom branch — when the prompt is structurally fine but the result is missing something subtle, and you've tried regenerating + small edits.

What the loop produces

For each iteration:

• A fresh image generation under the current prompt
• A visual judgment: what's working, what's not
• A proposed adjustment to the prompt for the next round
• The updated prompt either applied or queued for your approval

Output files land in:

projects/{month}/{brand}/Assets/{workflow}/qa_loop/v0-N/
  scene03-r1.png     iteration 1
  scene03-r2.png     iteration 2
  scene03-r3.png     iteration 3
  ...
  scene03-prompt-final.json   the dialed-in prompt at end

You can scroll back through the iteration chain to see what changed.

How to invoke it

You: tune the prompt at Assets/XYZG3/main-body/scene-03-image-start.json.
     Reference frame is reference_frames/scene_03.jpg. The lighting keeps
     coming out too cold — I want it warmer. Try 5 iterations.

Claude:
  Starting prompt-tuning loop on scene-03-image-start.json.

  Reference: reference_frames/scene_03.jpg
  Iterations: 5
  Target: warmer lighting matching the reference

  [auto-starts cloudflared tunnel since none was passed]

  Round 1: generated. Output saved.
    Issue: lighting still cool, but the kitchen window is rendered
    on the right (reference shows it on the left).
    Adjustment: changing "natural window light" to "natural window light
    from camera-left, warm golden hour quality".

  Round 2: generated.
    Issue: lighting warmth is better, but the wardrobe came out cream
    instead of white.
    Adjustment: "crisp pure white cotton t-shirt, slight texture, no
    off-white tones".

  Round 3: generated.
    Issue: looking good. Lighting, wardrobe, and composition all
    matching reference.
    Adjustment: no change. This iteration's prompt is the dialed-in version.

  Stopping at iteration 3 (early exit — converged). Final prompt saved
  to scene-03-image-start.json.

  Ready to regen Scene 03 in the workflow with the new prompt.

The loop usually converges in 2-4 iterations. Running all 5 isn't required if it's already landed.

What it can fix

Issue	Prompt-tuning can fix it
Lighting tone / direction off	Yes
Composition slightly different from reference	Yes
Wardrobe color or fit wrong	Yes
Camera angle off	Yes
Subject pose slightly wrong	Maybe — depends on how subtle
Hair / styling inconsistent	Yes
Background environment wrong	Yes

What it can't fix

Issue	Why prompt-tuning won't help
Extra fingers / melted face / AI tells	That's anatomy variance — use the Generation Runner's regen attempts instead
Hallucinated text on signs	Low priority, ignore
The reference image is wrong	Fix the reference, not the prompt
Wrong subject in the reference	Same — fix the input, not the prompt
Tuning a video prompt (Veo)	Skill is image-only
Anything requiring more than 8 seconds of motion in a video	That's a Veo limit, not a prompt issue

Budget awareness

Each iteration is a full image generation. 5 iterations = 5 × (4 candidates per gen) = 20 generations.

If you're tuning multiple scenes in parallel, the cost adds up fast. Set iteration counts thoughtfully:

Situation	Iterations
Tuning one stubborn scene that's mostly there	3-5
Tuning a scene that needs significant work	5-8
Exploring how a prompt responds	3-4
Bulk tuning across scenes you haven't seen yet	Don't — get a baseline gen first

After tuning

Once the prompt is dialed in:

1. The final prompt is saved back into the workflow file (or sidecar JSON)
2. Run a fresh Generation Runner pass on the scene with the new prompt
3. The candidates from that pass should match the dialed-in reference closely
4. Pick the best one and continue

Don't forget to bump V0-N if the prompt change is meaningful — the prompt edit counts as a workflow change (see Variants & Versioning).

Example: tuning to match a video-copy reference

This is the most common use case — you have a viral reference video and you're trying to nail a specific scene's look.

You: I'm doing a video copy of @example_creator. Scene 03 in their video
     has a really specific look — soft window light from the left, hand
     resting on a coffee mug, slight forward lean. Our current Scene 03
     gen has the lighting wrong and the pose too neutral. Tune it.
     6 iterations.

Claude:
  Tuning scene-03-image-start.json against reference_frames/scene_03.jpg
  (your screenshot from @example_creator's video).

  Round 1: ...
  Round 2: ...
  Round 3: ...

  Round 4: convergence reached. Hand pose, lighting direction, and
  forward lean all matching reference.

  Final prompt saved. Ready for regen.

When prompt-tuning isn't enough

If after 8+ iterations a scene still won't land, the issue is usually NOT the prompt. Check:

• Is the reference image clear and tight to the target composition? Blurry or wide reference = vague target
• Does the avatar reference sheet match the body / build you're trying to render? Body isn't locked by the face reference
• Is the scene structurally possible? "Two people in tight crop with both faces visible and product centered" might not have a valid composition
• Could a different scene structure solve it? Split into two clips, change the camera, frame differently

When the prompt-tuning loop keeps converging but the output isn't right — the goal itself may be wrong.

When you're ready

→ Next: Cache and Re-generation — what gets cached, when it's wiped, and how to force a fresh roll when the cache is getting in your way.