Source agents

Source agents operate on media before or alongside timeline planning. They do not change the live timeline.

source.work(...)

brief = source.work(prompt, tools=(), name="source-work")

Source Work is the single source-analysis entry point. It probes the real media, builds a visual inventory, and authors the run's sole grounded, asset-agnostic SourceBrief v2 — the source_brief.json that planner.compile consumes. In the same pass it runs the safe preparation renderer: the specialist may choose any number of grounded transcode, extract_audio, trim, concat, mux, or extract_frame jobs, then publishes the accepted batch atomically under source/. Its Artifact.result includes output paths and stable job IDs; paired IDs such as kitchen-wide-first and kitchen-wide-last can identify a first/last frame pair.

Opt-in evidence capabilities are granted per call: transcript import and computation with tools=[transcribe], grounded video events with tools=[detect_events], beat/onset analysis with tools=[detect_beats], and reusable matte precomputation with tools=[masking]. See Agent-only capabilities. Generation tools remain exclusive to source.generate_agent.create.

The brief contains schemaVersion: 2 and selections[]. Each selection names a manifest source, its matching kind (video, audio, image, graphic, or text), a free-form editorial role, and an optional span for video/audio. Whole temporal assets use span: null; image, graphic, and text selections always use span: null. Roles such as opener, montage, ender, b_roll, narration, audio_bed, title_card, and overlay are useful conventions, not required values.

A useful source-work prompt says:

• what each important file is for;
• which speech source should be transcribed;
• the roles to find, such as hook, establishing shot, proof, reaction, detail, transition, or closing image;
• selection criteria such as camera stability, subject visibility, continuity, and unwanted content;
• whether source order matters;
• any explicit mechanical derivatives to render (working copies, trims, extracted audio, concatenations), naming inputs, outputs, formats, and ordering.

Keep mechanical preparation requests explicit. Do not ask Source Work to design titles or build the final timeline. Its job is to find, explain, and prepare usable evidence in the source material, then hand a grounded brief to the planner.

Agent-only capabilities

Some evidence work is too expensive or too provider-specific to run on every pass, so it is opt-in per specialist call. You import a capability descriptor from scrambo.tools and grant it explicitly for one call:

from scrambo.tools import transcribe, detect_events, detect_beats, masking

Every grant is fail-closed and per call. It does not persist to later agents. Masking is the only capability accepted by the layer refinement specialists, and generate accepts masking only alongside at least one generation tool. Pass the descriptor again whenever a downstream specialist independently needs it. Calling a capability like transcribe(...), detect_events(...), or masking(...) directly always raises — the specialist decides which evidence is relevant and the host performs the work.

Video masking

Grant masking when an agent needs a subject matte. A Source Work, generate, or Planner Compile turn may precompute and cache the matte without changing the timeline. A later timeline specialist can reuse the exact request without a new remote submission or customer charge:

from scrambo.tools import masking

brief = source.work(
    "Prepare a person mask for presenter.mp4 from 1.0s to 3.5s.",
    tools=[masking],
)

timeline.titles_agent.edit(
    "Put the opening title behind the presenter from 1.0s to 3.5s.",
    tools=[masking],
    name="masked-title",
)

Masking is paid and has no caller-configurable fields. Scrambo enforces fixed operator ceilings, disables automatic retries for the paid turn, records the managed service's returned USD estimate unchanged, and applies no additional markup. A timeline mask is promoted only after its full placement contract and layer ordering pass deterministic validation.

Agent-directed transcripts

Import the agent-only capability descriptor and grant it to the Source Work, Planner Compile, or author that needs transcript evidence:

from scrambo.tools import transcribe

brief = source.work(
    "Find the strongest explanation in interview_maya.mov and the B-roll that "
    "supports it.",
    tools=[transcribe],
    name="interview-brief",
)

Defaults are ElevenLabs Scribe v2, automatic language detection, speaker diarization, and mandatory word timestamps. Configure the descriptor before the agent call when the defaults are not appropriate:

# Pin a language or disable diarization while keeping ElevenLabs Scribe v2.
transcribe.set_config(language="en", diarize=False)

# Select the explicit WhisperX fallback and its default model. Other settings
# retain their current values unless reset explicitly.
transcribe.set_config(provider="whisperx", language=None, diarize=True)

Configuration applies to subsequent grants in the current Python process. Supported keys are provider, model, language, and diarize; word timestamps cannot be disabled. Set every non-default choice before making the agent call so the effective configuration is explicit in that task's cache identity.

Top-level .json files supplied beside media are uploaded as untrusted transcript candidates. They are not automatically associated with a source and are never treated as transcripts merely because of their filename. An enabled agent first inspects the candidates and may ask the host to validate and import one; only if no candidate fits should it compute a transcript for selected audio or video. The importer accepts Scrambo word-timed segments[].words[] JSON and native ElevenLabs words[] JSON, normalizes the selected evidence, preserves the uploaded raw payload, and publishes the working transcript used downstream.

Grant transcript access at the earliest specialist that needs it. For example, grant it to Source Work when quotes determine source selection, to Planner Compile when word timing affects narrative structure, or to a direct author when there is no separate source-work/planner pass. The caption specialist consumes the working transcript established upstream; it cannot create one itself.

Video event detection

Grant detect_events when a specialist should watch a clip and record what happens in it. It works on any footage — screen recordings, browser, mobile, or app captures, games, design tools, or camera footage:

from scrambo.tools import detect_events

brief = source.work(
    "Watch onboarding_screen_capture.mov and mark each step the user completes: "
    "sign-up, workspace creation, and first invite. Note which region of the "
    "frame the action happens in.",
    tools=[detect_events],
    name="onboarding-brief",
)

The capability deterministically samples the named clips into cached, timestamped contact sheets; the selected specialist provider then watches those sheets and authors the events. No separate vision provider is called, and the specialist samples only the clips the request names — not the whole source/ folder — so name the footage you care about.

Each event lands in the SourceBrief with an importance and confidence score (both 0.0–1.0), an optional normalized focusRegion marking where in the frame it happens, and a paceHint recommending how to play that moment back (realtime, speed_up with an optional 1–16 speed, or cut). When your prompt names specific event types, the brief records a request-scoped eventTaxonomy and tags each event with a stable typeId; open-ended discovery requests omit the taxonomy and leave typeId: null. Downstream planner and author passes read these events from the accepted source_brief.json; grant detect_events again if a later pass needs to watch footage the source-work pass did not.

Tune sampling density before the call when the defaults (dense sampling, capped at 900 frames) are not appropriate:

# Coarser sampling with a lower frame ceiling for a long recording.
detect_events.set_config(sample_density="sparse", max_frames=400)

Supported keys are sample_density ("sparse" or "dense") and max_frames (an integer from 1 to 2400). As with transcribe, set every non-default choice before the agent call so the effective configuration is explicit in that task's cache identity.

Beat and onset detection

Grant detect_beats when source selection or pacing should follow the music. It is Source Work-only and takes no configuration — Source Work chooses the beat selection parameters per call:

from scrambo.tools import detect_beats

brief = source.work(
    "Analyze the beats in soundtrack.wav and pick the strongest downbeats to "
    "cut the montage on. Note the tempo and any section changes.",
    tools=[detect_beats],
    name="montage-brief",
)

The capability runs deterministic, cached beat/onset analysis over a manifest audio source (or a video source that carries audio) and preserves the full result at that source's manifest.sources[].beats path so later work can recover the raw numbers. Source Work promotes its interpretation into the SourceBrief as a typed, source-relative beatGrid — a tempoBpm, the selectedBeats it chose, and optional structural sections — which the planner and author consume to align cuts and rhythm.

Deterministic generation tools

Import deterministic generation separately from the source-agent facade:

from scrambo.tools import masking
from scrambo.tools.genAI import img2video, voiceover

Both tools write media into source/, register it in manifest.sources[], and return a normal Artifact. Matching requests reuse content-addressed outputs. Each direct call defaults to a $5 spend cap, and cached work costs nothing. These calls do not require editor.start() unless the program proceeds to timeline editing.

In the thin cloud SDK, vendor credentials stay on the Scrambo server. Local .json, .txt, and .md control files are read by the SDK and canonicalized before submission. Every first/last frame must have been part of the initial editor.open(input=...) upload; cloud generation does not late-upload frames in this MVP. If a generation control .json is also top-level inside the initial input directory, it is uploaded as a transcript candidate as well; keep control files outside that directory when they are not intended as run evidence.

Create voiceover from literal text or a .txt/.md file:

voice = voiceover(
    "A quieter way to start the day.",
    voice="rachel",
    name="brand-voiceover",
    budget_usd=5.0,
)

Create one or more motion clips from text and optional first/last images:

footage = img2video(
    [
        {
            "name": "coffee-pour",
            "seed_image": "./stills/cup.jpg",
            "prompt": "Slow macro push-in as coffee pours into the ceramic cup; "
                      "preserve the cup design and morning window light.",
            "duration": 4.0,
        },
        {
            "name": "pack-shot",
            "seed_image": "./stills/package.jpg",
            "last_frame": "./stills/logo-lockup.jpg",
            "prompt": "Controlled tabletop dolly ending on the supplied logo frame; "
                      "keep all packaging text unchanged.",
            "duration": 4.0,
        },
    ],
    resolution="720p",
    aspect_ratio="16:9",
    name="generated-product-shots",
)

img2video accepts one shot dictionary, a list, or a compatible prompts.json path. Every shot requires a first-frame image; an optional last frame requests interpolation. Its call-level arguments include duration, vendor, resolution, aspect_ratio, name, and budget_usd; a per-shot duration overrides the default. Text-to-video is intentionally outside this MVP.

voiceover accepts literal script text or a .txt/.md path and requires a voice. Its optional arguments are vendor, name, and budget_usd.

Unknown Veo pricing is rejected before vendor submission. ElevenLabs voiceover remains available as usage-only unless the operator configured a plan-rate snapshot; submitted characters remain measurable even when USD is unknown. If a cached artifact points to a missing output, Scrambo runs the deterministic kernel again; a surviving content-addressed file is adopted and re-registered without spending.

source.generate_agent.create(...)

Use the specialist when the prompt should decide what to generate:

from scrambo import planner, source, timeline
from scrambo.tools.genAI import img2video, voiceover

source.generate_agent.create(
    "Create a narrated vertical tour from the listing photos.",
    tools=[img2video, voiceover, masking],
    budget_usd=5.0,
    max_calls=4,
    name="generated-assets",
)

planner.compile("Build a 30-second tour")
timeline.author_agent.edit()

The specialist receives only the registered Scrambo tool callables explicitly listed in tools; arbitrary Python functions are rejected. At least one generation tool is required, while masking may be added for matte precomputation. The specialist inspects seed images, creates only requested assets, inspects generated video contact sheets, and writes one complete SourceBrief v2. It does not create a generation-specific root contract.

budget_usd is cumulative across uncached vendor work in that create() turn. max_calls is the cumulative requested-output cap, not an LLM iteration count: a batch of three video requests consumes three, and a cached output still consumes one slot while spending zero dollars. The default is four outputs. name must contain 1–80 characters. Deployment ceilings are configurable and default to $15 and 15 outputs; requests above the active ceiling fail with generation_limit_exceeded. Budget, output-cap, unknown-price, and generation-tool errors fail the task. There is no automatic specialist revision after the paid turn; successful outputs remain content-addressed and can be adopted free on a rerun.

The portable cloud SDK exposes the same registered tool identities and sends only their stable allowlisted IDs to the server.