Section 07

Video As Prompts

Set up a repeatable video-as-prompt run for this project before doing any full video analysis.


setup agent prompt
Set up a repeatable video-as-prompt run for this project before doing any full video analysis.

1. Resolve the `oracle-video` skill first. Check for `skills/oracle-video/SKILL.md`; if it is not bundled, check the common local install path at `.agents/skills/oracle-video/SKILL.md`. Read the skill and any directly referenced repeatable video process file before creating outputs. If the skill is missing, stop and report the missing path checks.

2. Create a run id like `<YYYY-MM-DD>-<short-slug>` and a run folder at `runs/<run-id>/` with this structure: `inputs/`, `receipts/`, `chunks/01/`, `analysis/`, `final/`, `keyframes/`, and `contact-sheets/`.

3. Accept either a local video path or a video URL. For a local file, copy it to `inputs/source-video.<ext>` and keep that copy immutable. For a URL, first save metadata with `yt-dlp --dump-single-json --skip-download --no-check-formats` to `receipts/source.metadata.json`, then download with `yt-dlp -f "bv*+ba/best"` into `inputs/`.

4. Write a source hash with `shasum -a 256` to `inputs/source-video.sha256`. Confirm the file exists and is non-empty.

5. Capture media facts with `ffprobe -hide_banner -show_format -show_streams -of json` and save them to `receipts/ffprobe.source.json`. Confirm the file exists and is non-empty. If `ffprobe`, `ffmpeg`, or `yt-dlp` is missing, stop setup and report the missing tool plus the failed command.

6. Prepare `chunks/01/chunk.mp4`. For a short video, copy the source video into that path. For a long video, create additional numbered chunk folders as needed and save a timestamp mapping note before any analysis. Use absolute paths for every video file that will be attached to Oracle.

7. Write `chunks/01/gemini-prompt.md`. It must request separate lanes for transcript evidence, chat evidence, visual evidence, and inference. It must require preserved timestamps, forbid invented URLs, tools, commands, files, UI labels, and speakers, and end with the exact final line `END_OF_VIDEO_ANALYSIS`.

8. Write `chunks/01/expected-sentinel.txt` containing only `END_OF_VIDEO_ANALYSIS`. Confirm the sentinel appears in `gemini-prompt.md` and that the expected sentinel file is non-empty.

9. Prepare `chunks/01/oracle-call.json` or equivalent setup notes with the required Gemini browser-mode contract: `engine: "browser"`, `model: "gemini-3-pro"`, `browserAttachments: "always"`, `files` containing an absolute path to `chunks/01/chunk.mp4`, and a stable slug. Do not use API mode for video analysis.

10. Generate keyframes and a contact sheet if `ffmpeg` is available. Save frames under `keyframes/` using a low-frequency extraction such as `fps=1/10`, then save a tiled contact sheet under `contact-sheets/`. If generation is skipped, save the exact commands to run later and report why they were not run.

11. Create evidence lane files under `analysis/`: `transcript.md`, `chat.md`, `visual-evidence.md`, and `inference.md`. Keep spoken claims, chat or comments, visible facts, and reasoned conclusions separate. Label inference clearly until code, runtime behavior, or visual checks support it.

12. Create `analysis/omissions-pass-prompt.md`. It should ask only for missing visible UI states, skipped timestamps, unsupported claims, invented details, uncertainty, and patch-style corrections. It must require the response to end with `END_OF_VIDEO_ANALYSIS`.

13. Create downstream artifact stubs under `final/`, such as `implementation-prompt.md`, `qa-checklist.md`, and `variation-matrix.md`. Leave evidence-dependent fields blank until verified analysis exists.

14. Enforce privacy boundaries before any upload or external analysis. Do not print full environment output, tokens, cookies, API keys, private endpoints, resolved secret values, or raw private config. Do not copy secret helper files into the run folder. If the video, metadata, transcript, or chat text contains credentials or private config, stop and report the issue instead of uploading.

15. Run a setup verification gate before Gemini analysis. Report `OK` or `MISSING` for: `oracle-video` skill resolved, source video exists, source hash non-empty, `ffprobe.source.json` non-empty, URL metadata present when the input is a URL, chunk file exists, Gemini prompt exists, sentinel file exists, Oracle call uses browser mode with `gemini-3-pro` and `browserAttachments: "always"`, Oracle file path is absolute, evidence lane files exist, keyframes/contact sheet exist or commands are saved, omissions prompt exists, and downstream stubs exist.

16. Stop before full video analysis unless every required setup check passes. Return the run id, absolute run folder path, commands run, verification report, failed checks, and the smallest next fix if anything is missing. Only say it is ready for Gemini analysis after the setup gate passes.

Workflow route

Video As Prompts

Set up a repeatable video-as-prompt run for this project before doing any full video analysis.

Eggo pulls visible evidence from the video strip into the packet for the Video As Prompts workflow, making the page's operating move visible.
Intent

Name the software-workflow decision before launching work.

Evidence

Inspect the file, command, log, prompt, or recording that proves the state.

Route

Pick the narrow agent, tool, shell, or council path that matches the problem.

Verify

Confirm the result with output, screenshots, receipts, or saved decisions.

  1. Big Idea
  2. Try This Now
  3. Tool Reference
  4. When To Reach For This
  5. How This Used To Work
  6. Core Pattern
the prompt packet gains evidence frames while empty summary bubbles disappear

Big Idea

Video is prompt material.

Try This Now

For a short bug repro video, create a run folder and save a source hash. Write the Gemini prompt with separate lanes: transcript, chat, visual evidence, inference. End the prompt with END_OF_VIDEO_ANALYSIS before analysis starts.

A screen recording can show things a written prompt usually loses:

A transcript can tell you what someone said. A video can show what happened.

Tool Reference

Use oracle-video when the task depends on what is visible or audible in a recording.

Conceptually:

snippet-1
oracle-packx bundles code/text context for ChatGPT.
oracle-video attaches video and sends it to Gemini through Oracle browser mode.

Correct Oracle call shape for video analysis:

js
mcp__oracle__consult({
  prompt: "[video-analysis-slug]\n\n{contents of chunks/01/gemini-prompt.md}",
  files: ["/absolute/path/to/runs/2026-06-27-bug-repro/chunks/01/chunk.mp4"],
  engine: "browser",
  model: "gemini-3-pro",
  browserAttachments: "always",
  slug: "video-analysis-slug"
})

Required:

snippet-3
engine: "browser"
model: "gemini-3-pro"
browserAttachments: "always"
absolute file path

Do not use API mode for video analysis.

When To Reach For This

Use video when the important context is visible, temporal, or hard to describe.

Use Video ForWhy
bug reproduction clipcaptures exact UI state, clicks, errors, timing
design feedback videopreserves visual hierarchy and spoken intent
app walkthroughturns observed workflow into requirements and QA checks
loading/timing bugshows delay, spinner, race, or order of events
terminal/browser workflowpreserves commands, pages, output, and sequence
long tutorial or workshop recordingcan become chapters, exercises, references, and follow-up prompts

Skip video when:

How This Used To Work

What you used to expect:

snippet-4
Watch the video, summarize from memory, then ask the agent to fix the vague thing.

What the workflow gives you now:

snippet-5
Preserve the source, capture hashes and media facts, analyze with Gemini, keep
evidence lanes separate, verify claims, then turn the result into a grounded
prompt.

What you used to expect:

snippet-6
A transcript is the prompt.

What the workflow gives you now:

snippet-7
The transcript is one evidence lane. Visual state, timing, chat, cursor motion,
and inference stay separate until verified.

Core Pattern

snippet-8
source media
-> source receipts
-> Gemini analysis
-> evidence lanes
-> verification
-> downstream prompt

Do not skip the middle.

snippet-9
video without source receipts = hard to trust
analysis without timestamps = hard to verify
transcript without visuals = incomplete
claims without keyframes = risky
prompt without verification = hallucination bait

Short Path vs Long Path

PathUse ForShape
short pathbug repros, design feedback clips, quick app walkthroughspreserve source, one Gemini pass, keyframes, downstream artifact
long pathlong tutorials, workshop recordings, multi-section demossource receipts, segment/chunk, analyze chunks, omissions pass, reconcile, final packet

Short path output examples:

Long path output examples:

Run Folder Shape

Use a run folder so the video stays tied to its analysis.

snippet-10
runs/<date>-<slug>/
  inputs/
    source-video.mp4
    source-video.sha256
  receipts/
    ffprobe.source.json
  chunks/
    01/
      chunk.mp4
      gemini-prompt.md
      expected-sentinel.txt
      output.log
      analysis.md
  analysis/
  final/
  keyframes/
  contact-sheets/

Create the folders:

bash
RUN_ID="2026-06-27-checkout-repro"
RUN_DIR="runs/$RUN_ID"
mkdir -p "$RUN_DIR"/{inputs,receipts,analysis,final,keyframes,contact-sheets}
mkdir -p "$RUN_DIR/chunks/01"

Preserve The Source

Keep the original media immutable.

For a local file:

bash
cp /path/to/source-video.mp4 "$RUN_DIR/inputs/source-video.mp4"
shasum -a 256 "$RUN_DIR/inputs/source-video.mp4" \
  > "$RUN_DIR/inputs/source-video.sha256"

For a URL source, capture metadata first when useful:

bash
yt-dlp --dump-single-json --skip-download --no-check-formats "$URL" \
  > "$RUN_DIR/receipts/source.metadata.json"

Then download:

bash
yt-dlp -f "bv*+ba/best" -o "$RUN_DIR/inputs/source-video.%(ext)s" "$URL"

Capture media facts:

bash
ffprobe -hide_banner -show_format -show_streams -of json \
  "$RUN_DIR/inputs/source-video.mp4" \
  > "$RUN_DIR/receipts/ffprobe.source.json"

Confirm:

bash
test -s "$RUN_DIR/inputs/source-video.sha256" && echo "sha256:OK"
test -s "$RUN_DIR/receipts/ffprobe.source.json" && echo "ffprobe:OK"

Evidence Lanes

Keep these lanes separate:

LaneWhat It MeansExample
transcriptspoken words“The save button disappears.”
chatchat messages or comments“This only happens after refresh.”
visual evidencevisible UI statesave button hidden at 00:01:34
inferencereasoned conclusionlikely state reset after token refresh

Plain rule:

snippet-17
A spoken claim is not a visible fact.
A visible fact is not automatically the cause.
An inference must stay labeled until code or runtime proof confirms it.

Prompt With A Sentinel

Save the Gemini prompt beside the chunk.

bash
cat > "$RUN_DIR/chunks/01/gemini-prompt.md" <<'EOF'
Analyze this video as prompt material for a software project.

Return:

1. short summary
2. timestamped workflow
3. transcript evidence
4. chat evidence
5. visual evidence
6. inferences, clearly labeled
7. likely downstream artifacts
8. omissions or uncertainty

Rules:

- Do not invent URLs, tools, commands, files, UI labels, or speakers.
- Preserve original timestamps.
- Separate visible facts from spoken claims and inferences.
- Include the exact sentinel below as the last line.

END_OF_VIDEO_ANALYSIS
EOF

printf 'END_OF_VIDEO_ANALYSIS\n' \
  > "$RUN_DIR/chunks/01/expected-sentinel.txt"

Confirm:

bash
rg -n "END_OF_VIDEO_ANALYSIS" "$RUN_DIR/chunks/01/gemini-prompt.md"
test -s "$RUN_DIR/chunks/01/expected-sentinel.txt" && echo "sentinel:OK"

Analyze With Gemini

Use Oracle browser mode and attach the video.

js
mcp__oracle__consult({
  prompt: "[checkout-repro-video]\n\n{contents of runs/.../chunks/01/gemini-prompt.md}",
  files: ["/absolute/path/to/runs/2026-06-27-checkout-repro/chunks/01/chunk.mp4"],
  engine: "browser",
  model: "gemini-3-pro",
  browserAttachments: "always",
  slug: "checkout-repro-video"
})

After the run, read the local output:

bash
sed -n '1,220p' ~/.oracle/sessions/checkout-repro-video/output.log
rg -n "END_OF_VIDEO_ANALYSIS" ~/.oracle/sessions/checkout-repro-video/output.log

If the output is clipped in a tool response, treat output.log as the local source to inspect.

Verify The Analysis

Check that the answer is grounded.

CheckCommand Or Evidence
source hash existstest -s "$RUN_DIR/inputs/source-video.sha256"
media facts existtest -s "$RUN_DIR/receipts/ffprobe.source.json"
prompt savedtest -s "$RUN_DIR/chunks/01/gemini-prompt.md"
sentinel presentrg -n "END_OF_VIDEO_ANALYSIS" output.log
timestamps preservedcompare analysis timestamps to source duration
visual claims checkedkeyframes or contact sheet
invented facts removedno unsupported URLs/tools/commands/files
omissions capturedmissing sections are listed as uncertainty

Extract keyframes for visual checks:

bash
mkdir -p "$RUN_DIR/keyframes"
ffmpeg -i "$RUN_DIR/inputs/source-video.mp4" \
  -vf "fps=1/10" \
  "$RUN_DIR/keyframes/frame-%04d.jpg"

Create a contact sheet:

bash
ffmpeg -pattern_type glob -i "$RUN_DIR/keyframes/*.jpg" \
  -vf "scale=320:-1,tile=4x4" \
  "$RUN_DIR/contact-sheets/source-contact-sheet.jpg"

Omissions Pass

Run an omissions pass when coverage is incomplete.

Ask:

snippet-24
Review the prior video analysis for omissions.

Focus only on missing visible UI states, skipped timestamps, unsupported claims,
invented details, and uncertainty. Do not rewrite the whole analysis. Return a
patch-style list of corrections and end with END_OF_VIDEO_ANALYSIS.

Use the omissions pass when:

Turn Video Into Prompt Material

Common downstream artifacts:

Video SourceDownstream Artifact
bug reproductionrepro steps, expected vs actual behavior, implementation prompt
design feedbackvisual acceptance criteria, change brief, QA checklist
app walkthroughrequirements, acceptance criteria, open questions
terminal workflowcommand sequence, failure point, debugging prompt
long recordingchapters, exercises, resources, follow-up packet
UI alternativesSection 6 variation matrix

Bug prompt shape:

markdown
# Implementation Prompt From Video

## Source

- video:
- source hash:
- ffprobe:
- analysis:
- keyframes:

## Observed Failure

- timestamp:
- visual evidence:
- transcript/chat evidence:
- expected behavior:
- actual behavior:

## Request

Implement the smallest safe fix.

## Constraints

- do not change:
- preserve:

## Verification

- command:
- what passing means:

Variation seed shape:

markdown
# Video-Derived Variation Matrix

## Goal

## Evidence From Video

- timestamp:
- observation:
- keyframe:
- why it matters:

## Invariants

- same source evidence:
- same target flow:
- same verification command:
- same review criteria:

## Variable Under Test

- variable:
- why this variable matters:
- what is out of scope:

Ask The Agent This

Analyze a bug reproduction:

snippet-27
Use the video analysis workflow.

Preserve the source video, save a source hash and ffprobe output, write a Gemini
prompt with END_OF_VIDEO_ANALYSIS, analyze the video through Oracle browser mode
with Gemini, then separate transcript evidence, visual evidence, chat evidence,
and inference. Turn the verified result into repro steps and an implementation
prompt for the smallest safe fix.

Review design feedback:

snippet-28
Turn this design feedback video into a change brief.

Keep visible UI facts separate from spoken opinions. Save keyframes for the
visual claims. Produce acceptance criteria, a QA checklist, and open questions.
Do not invent UI labels or routes that are not visible or spoken.

Create a Section 6 seed:

snippet-29
Use this verified video analysis to create a controlled variation matrix.

Choose one variable under test, preserve the same source evidence and
verification command for every candidate, and write three candidate settings.

What To Confirm / How To Know It Worked

GoalConfirmation
source preservedoriginal media exists under inputs/
source hash savedsource-video.sha256 exists and is non-empty
media facts savedffprobe.source.json exists
prompt savedchunks/01/gemini-prompt.md exists
sentinel requiredexpected-sentinel.txt exists
Oracle call shape correctengine: "browser", model: "gemini-3-pro", browserAttachments: "always"
output inspected~/.oracle/sessions/<slug>/output.log read
sentinel presentEND_OF_VIDEO_ANALYSIS found in output
lanes separatedtranscript, chat, visual evidence, inference are separate
visual claims checkedkeyframes/contact sheet exists
unsupported claims removedno invented URLs/tools/commands/files
downstream artifact savedrepro, requirements, QA, brief, lesson, or matrix exists

If It Goes Wrong

No Source Hash

Create it before analysis:

bash
shasum -a 256 "$RUN_DIR/inputs/source-video.mp4" \
  > "$RUN_DIR/inputs/source-video.sha256"

No ffprobe Output

Run:

bash
ffprobe -hide_banner -show_format -show_streams -of json \
  "$RUN_DIR/inputs/source-video.mp4" \
  > "$RUN_DIR/receipts/ffprobe.source.json"

Oracle Call Used API Mode

Rerun with browser mode.

snippet-32
engine: "browser"
model: "gemini-3-pro"
browserAttachments: "always"

File Path Was Relative

Use an absolute video path in files.

bash
realpath "$RUN_DIR/chunks/01/chunk.mp4"

Sentinel Missing

Read output.log. If the sentinel is truly missing, rerun or ask for an omissions pass.

bash
rg -n "END_OF_VIDEO_ANALYSIS" ~/.oracle/sessions/<slug>/output.log

Analysis Only Summarizes Transcript

Run an omissions pass focused on visual evidence:

snippet-35
Find missing visual UI states, cursor actions, timing, browser/terminal state,
and unsupported claims. Preserve timestamps.

Visual Claim Has No Keyframe

Extract keyframes or create a contact sheet. Downgrade unsupported claims to inference.

URL, Tool, Command, Or File Appears Without Evidence

Remove it or label it as inference. Do not let invented details become prompt constraints.

Long Recording Was Not Chunked

Create chunks and analyze them separately. Preserve original timestamps across chunks.

References And Resources