snapshot of https://github.com/tldraw/tldraw.git (main, 2026-09-23; blobs > 131072B omitted)
Some checks failed
Deploy analytics worker / Deploy analytics worker to ${{ (github.ref == 'refs/heads/production' && 'production') || (github.ref == 'refs/heads/main' && 'staging') || 'preview' }} (push) Failing after 1s
i18n upload strings / Upload latest strings (push) Failing after 1s
Deploy bemo / Deploy bemo to ${{ (github.ref == 'refs/heads/bemo-production' && 'production') || (github.ref == 'refs/heads/main' && 'staging') || 'preview' }} (push) Failing after 1s
Deploy grafana resources / ${{ github.event_name == 'push' && 'Deploy grafana resources' || 'Validate grafana resources (dry run)' }} (push) Failing after 0s
End to end tests (dotcom) / Check for relevant changes (push) Failing after 0s
End to end tests (examples) / Check for relevant changes (push) Failing after 0s
End to end tests (mermaid readability) / Check for relevant changes (push) Failing after 1s
End to end tests (perf) / Check for relevant changes (push) Failing after 0s
Publish editor extensions / Publish editor extensions (push) Failing after 1s
Publish SDK packages / Remove the publish-packages label (push) Has been skipped
Checks / Tests & checks (push) Has been cancelled
Checks / Build all projects (push) Has been cancelled
Deploy templates / Deploy templates (push) Has been cancelled
End to end tests (dotcom) / End to end tests (dotcom) (push) Has been cancelled
End to end tests (mermaid readability) / End to end tests (mermaid readability) (push) Has been cancelled
End to end tests (perf) / End to end tests (perf) (push) Has been cancelled
Deploy .com / Deploy dotcom to ${{ (github.ref == 'refs/heads/production' && 'production') || (github.ref == 'refs/heads/main' && 'staging') || 'preview' }} (push) Has been cancelled
End to end tests (examples) / End to end tests (examples) (push) Has been cancelled
Publish SDK packages / Publish SDK packages (push) Has been cancelled
Publish SDK packages / Tell user released version (push) Has been cancelled
Publish SDK packages / Publishes code templates to separate repositories (push) Has been cancelled
Publish SDK packages / Deploys the template demos with the new SDK version (push) Has been cancelled

This commit is contained in:
upstream-mirror
2026-09-23 16:12:52 +05:00
commit a774ac673a
4541 changed files with 770499 additions and 0 deletions

View File

@@ -0,0 +1,247 @@
---
name: pr-walkthrough
description: Create a narrated video walkthrough of a pull request with code slides and audio narration. Use when asked to create a PR walkthrough, PR video, or walkthrough video.
---
# PR walkthrough video
Create a narrated walkthrough video for a pull request. This is designed to be an internal artifact, providing the same benefit as would a loom video created by the pull request's author — walking through the code changes, explaining what was done and why, so that anyone watching can understand the PR quickly.
**Input:** A GitHub pull request URL (e.g., `https://github.com/tldraw/tldraw/pull/7924`). If given just a PR number or other description, assume that the PR is on the tldraw/tldraw repository.
**Output:** An MP4 video at 1280×720 (30 fps) with audio narration, whisper-timed yellow-on-black subtitles, and standardized intro / outro slides, saved to `out/pr-<number>-walkthrough.mp4`.
All intermediate files (audio, manifest, scripts) go in `tmp/pr-<number>/` relative to this skill directory. This directory is gitignored. Only the final `.mp4` lives at `out/`.
Run commands that reference `./scripts` or `./video` from this skill directory.
## Philosophy
**This is a walkthrough from the author's perspective.** The goal is the same as if the PR author sat down with someone and walked them through the changes — showing specific code, explaining what changed and why, in an order that builds understanding. The viewer should come away understanding both _what the code does_ and _how to think about the changes_.
This means:
- **The narration drives everything.** Write the walkthrough narration first, as a continuous explanation of the PR. Then figure out what should be on screen at each moment to support what's being said.
- **Show the code.** The default visual is a code diff or source file. Text slides are the exception (intro, brief transitions, outro), not the rule. When the narration talks about a function, the viewer should be looking at that function.
- **Walk through changes in a logical order**, not necessarily file order or commit order — but always anchored to concrete code, not abstract descriptions.
- **Explain the "why", not just the "what".** The code on screen shows what changed. The narration adds the reasoning — why this approach, what problem it solves, what edge cases it handles.
## Workflow
### Step 1: Understand the PR
Read the PR commits, diff, and description. Understand the narrative arc:
- What problem does this solve?
- What's the approach?
- What are the key mechanisms?
```bash
gh pr view <number> --json title,body,commits
git log main..HEAD --oneline
git diff main..HEAD --stat
```
**Skip generated files.** When reading the diff, ignore files that are auto-generated rather than hand-edited. These add noise without informing the walkthrough. Common examples in this repo:
- `**/api-report.md`, `**/api-report.api.md` — generated API surface reports
- `**/*.api.json`, `**/temp/*.api.json` — API extractor output
- `apps/docs/content/reference/**` — generated reference docs
- `yarn.lock`, `package-lock.json` — lockfiles
- `**/CHANGELOG.md` — auto-generated changelogs
- Any file the repo's tooling regenerates (snapshots, schema dumps, bundled assets)
If unsure whether a file is generated, check for a header comment like "DO NOT EDIT" or check whether the repo has a generator command that produces it. Filter these out when picking which files to feature in `diff`/`code` slides.
### Step 2: Write the narration
Write the narration as continuous text, broken into logical segments. Each segment is a beat of the walkthrough — a concept, a change, or a group of related changes. Save this as `tmp/pr-<number>/SCRIPT.md`.
The narration should read like the author explaining the PR to a colleague: "So here's what we're doing... The core problem was X... The approach I took was Y... If you look at this function here..."
Structure: intro → context/problem → code walkthrough → summary. See **Script structure** below.
If the commits are simple and organized well (often on a branch with `-clean` in its name), you can follow their commit messages and descriptions to guide your narration. Otherwise, examine the code and create your own narrative. Introduce concepts in an order that builds on previous ones.
Avoid redundancy, especially between intro and first content segment.
### Step 3: Generate audio and timestamps
Generate per-segment audio clips with one TTS call per segment. This avoids chunking, alignment, and splitting entirely — each segment is short enough for a single reliable TTS call.
Write a `narration.json` file, then run the `generate-audio.sh` CLI tool:
```bash
./scripts/generate-audio.sh narration.json tmp/pr-<number>/
```
**API key:** Sourced automatically from the repo `.env` file (`GEMINI_API_KEY`).
#### Narration JSON format
```json
{
"style": "Read the following walkthrough narration in a calm, steady, professional tone. Speak at a measured pace as if the author of a pull request were walking a colleague through the code changes.",
"voice": "Iapetus",
"slides": [
"This pull request adds group-aware binding resolution to the arrow tool...",
"The core problem was that arrow bindings broke when the target shape...",
"If you look at the getBindingTarget method in ArrowBindingUtil.ts..."
]
}
```
- **`style`** — Voice persona and pacing instructions. Keep it short and specific.
- **`voice`** — Gemini voice name (default: `Iapetus`).
- **`slides`** — Array of narration text, one entry per segment.
#### How it works
1. For each segment, the script builds a prompt: style preamble + segment text.
2. One API call to `gemini-3.1-flash-tts-preview` per segment generates a WAV clip directly.
3. Each clip is validated (duration sanity check vs word count) and retried automatically if the output is bad.
4. Leading/trailing silence is trimmed from each clip.
**Output:** Per-segment audio clips (`audio-00.wav`, ...) and a `durations.json` file mapping each audio filename to its duration in seconds.
**Dependencies:** ffmpeg / ffprobe. No Python packages required beyond the standard library.
**Do NOT use** `[pause long]` or `[pause medium]` markup tags in the narration text — the model may read them aloud literally.
### Step 4: Write the manifest
Read `durations.json` from step 3 for each clip length, then write `manifest.json` alongside the audio files. It lists one entry per slide, in order, each with its `type`, `audio`, and `durationInSeconds`.
See `references/manifest.md` for the slide types and their fields.
### Step 5: Render the video
Run the `render.sh` script:
```bash
./video/render.sh \
tmp/pr-<number>/manifest.json \
out/pr-<number>-walkthrough.mp4
```
The script:
1. Copies referenced audio/image files into `video/assets/`.
2. Runs whisper transcription on each audio file → `video/transcripts/audio-NN.json` (idempotent — only re-transcribes if the audio is newer than the existing transcript).
3. Runs `build.mjs <manifest>` to generate `video/index.html` — a hyperframes composition with timed clips for every slide, GSAP timeline for transitions and code-focus pans, and yellow-on-black caption clips with start/end times derived from the whisper transcripts.
4. Lints the composition and renders 1920×1080 frames via `npx hyperframes render`.
5. Downscales to 1280×720 / 30fps and recompresses with ffmpeg (CRF 26 + AAC 96k) for the final small-but-sharp MP4.
**Dependencies:** Node.js 22+, ffmpeg, Python 3 (used by `render.sh` to parse the manifest). `hyperframes` is invoked via `npx --yes`, so no install step. Whisper runs locally (small.en model, ~150MB on first download).
#### Caption sync via whisper
Captions appear as yellow text on a solid black pill, anchored to the bottom of the frame. Their start/end times come from word-level whisper transcripts grouped into 5–7 word chunks, breaking early on natural pauses (>450ms gaps = sentence boundaries). One implication: whisper transcribes brand/code names phonetically — "tldraw" → "TL Draw", "OverlayUtil" → "overlay util". This is acceptable for captions but could be normalized later via a substitution table in `build.mjs`.
#### File size knobs
The default render targets ~30–60 MB for an 8-minute video. To tune:
- `--crf <n>` in the ffmpeg downscale step inside `render.sh` — 22 is near-lossless, 26 is the default, 30+ is much smaller. CRF 28–30 is a good range for a docs-quality result.
- The 1080p hyperframes render uses `-q draft --crf 30` to keep the intermediate file small (the downscale dominates final size anyway).
## File organization
Final output lives in this skill directory. All intermediate files go in `tmp/` (gitignored):
```
pr-walkthrough/
├── SKILL.md # This file
├── scripts/ # CLI tools (checked in)
│ └── generate-audio.sh # narration.json → per-slide WAVs + durations.json
├── video/ # Hyperframes project (checked in)
│ ├── hyperframes.json # hyperframes config
│ ├── meta.json # project meta
│ ├── build.mjs # manifest.json → index.html composition
│ ├── render.sh # manifest.json → 720p MP4 (full pipeline)
│ ├── assets/ # Auto-populated at render time (gitignored)
│ ├── transcripts/ # Whisper word-level JSON (gitignored, cached)
│ └── renders/ # Intermediate 1080p renders (gitignored)
├── out/ # Final outputs (gitignored)
│ └── pr-XXXX-walkthrough.mp4
└── tmp/ # Intermediate files (gitignored)
└── pr-XXXX/
├── SCRIPT.md # Narration script
├── narration.json # Input to generate-audio.sh
├── durations.json # Audio filename → duration in seconds
├── manifest.json # Input to render.sh
└── audio-XX.wav # Per-segment audio clips
```
## API configuration
- **Gemini API key:** Stored as `GEMINI_API_KEY` in the project root `.env` file. Used for TTS and audio alignment.
- **TTS model:** `gemini-3.1-flash-tts-preview`
- **TTS voice:** `Iapetus` (always)
## Script structure
The walkthrough follows a consistent narrative arc. Not every section needs its own segment — combine or skip sections based on the PR's complexity. The goal is 8-12 segments total, with the vast majority showing code.
### Intro (1 segment)
The intro card: tldraw logo + PR title + date. The narration should be a single sentence that frames what this PR does at a high level. Don't go into detail yet.
Manifest slide type: `intro`.
### Context (0-1 segments)
Brief orientation before diving into code. What was the situation before this PR? What problem or need motivated the work? Keep this short — just enough framing that the code walkthrough makes sense.
- Be concrete: "Arrow bindings broke when the target shape was inside a group" not "There were issues with bindings"
- Name the area of the codebase affected
If the context can be explained while showing the first piece of relevant code, skip the standalone context segment and fold it into the first code segment.
Manifest slide type: `text` or `diff` (if showing the problematic code).
### Code walkthrough (6-10 segments)
The bulk of the video. Walk through the actual code changes, showing specific diffs and files while explaining what was done and why.
**Every segment should show code.** Use `diff` slides for changes and `code` slides for unchanged reference code.
Guidelines:
- **Name files and functions.** Every narrated segment should reference at least one specific file or function.
- **Show the diff.** The visual for each segment should be the actual diff being discussed. Use `git diff main..HEAD -- path/to/file` to get the diff, then extract the relevant hunks.
- **Order by understanding, not by file.** Present changes in the order that builds comprehension. If a new type is defined in one file and consumed in another, show the definition first.
- **Explain the "why", not just the "what".** The diff shows _what_ changed — the narration adds the reasoning, the edge cases it handles, the alternatives that were considered.
- **Skip boilerplate, but mention it.** Don't dedicate a segment to every import change or type export, but do mention in passing: "There are also some type exports added in `index.ts` — those are just re-exports of the new types we'll see next."
- **Group related small changes.** If three files all got the same one-line fix, one segment can cover all three. Mention each file by name.
### Summary (1 segment)
Briefly recap what the PR accomplished. This is a short wrap-up — a sentence or two summarizing the overall change, mentioning any known limitations or follow-up work if relevant.
Manifest slide type: `text`.
### Outro (1 segment, silent)
The tldraw logo, 3 seconds of silence. Always include this as the final slide.
Manifest slide type: `outro` with `durationInSeconds: 3`.
## Narration writing tips
- **Be specific about code.** Say "In `BindingUtil.ts`, the `onAfterChange` handler now checks for group ancestors" — not "The binding system was updated." Name files and functions so the viewer can connect the narration to what's on screen.
- **Each segment = one change or closely related group of changes.** If you can't point to a specific diff for the segment, it's probably too abstract.
- **Write as the author.** The tone should be explanatory and natural — like walking someone through your work. "So the main thing here is..." or "The tricky part was..." are fine.
- **Avoid redundancy** between intro and first content segment.
- **Mention files that aren't shown.** If a PR touches 15 files but only 6 are interesting, briefly acknowledge the others: "The remaining changes are type exports and test fixtures."
- Aim for **5-7 minutes** total narration.
## Checklist
- [ ] Read all PR commits and understand the full diff
- [ ] Write narration in SCRIPT.md (8-12 segments)
- [ ] Generate per-segment audio (Iapetus voice)
- [ ] Read durations.json to get per-segment durations
- [ ] Write manifest.json with slide types, diffs/code, and audio references
- [ ] Render video with render.sh
- [ ] Verify final output: 1280×720 / 30 fps, audio synced, captions readable, outro present

View File

@@ -0,0 +1,138 @@
# Manifest format
The manifest is a JSON file describing every slide in the video. It bridges the narration/audio step and the hyperframes renderer. Referenced from step 4 of `../SKILL.md`.
Read the `durations.json` from step 3 to get the duration (in seconds) for each audio clip. Then write a `manifest.json` alongside the audio files:
```json
{
"pr": 7865,
"slides": [
{
"type": "intro",
"title": "Fix canvas-in-front z-index layering #7865",
"date": "February 14, 2026",
"audio": "audio-00.wav",
"durationInSeconds": 3.2
},
{
"type": "diff",
"filename": "packages/editor/editor.css",
"language": "css",
"diff": "@@ -12,7 +12,7 @@\n --tl-z-canvas: 100;\n- --tl-z-canvas-in-front: 600;\n+ --tl-z-canvas-in-front: 250;\n --tl-z-shapes: 300;",
"audio": "audio-01.wav",
"durationInSeconds": 25.8
},
{
"type": "code",
"filename": "packages/editor/src/lib/Editor.ts",
"language": "typescript",
"code": "function getZIndex() {\n return 250\n}",
"audio": "audio-02.wav",
"durationInSeconds": 13.5
},
{
"type": "text",
"title": "Summary",
"subtitle": "Moved canvas-in-front from z-index 600 to 250.",
"audio": "audio-07.wav",
"durationInSeconds": 7.4
},
{
"type": "list",
"title": "Key changes",
"items": ["Lowered z-index", "Updated tests", "Added migration"],
"audio": "audio-06.wav",
"durationInSeconds": 10.2
},
{
"type": "outro",
"durationInSeconds": 3
}
]
}
```
#### Slide types
| Type | Required fields | Description |
| --------- | ------------------------------------------------------------ | ---------------------------------- |
| `intro` | `title`, `date`, `audio`, `durationInSeconds` | Logo + title + date |
| `diff` | `filename`, `language`, `diff`, `audio`, `durationInSeconds` | Syntax-highlighted unified diff |
| `code` | `filename`, `language`, `code`, `audio`, `durationInSeconds` | Syntax-highlighted source code |
| `text` | `title`, `audio`, `durationInSeconds` | Title + optional `subtitle` |
| `list` | `title`, `items`, `audio`, `durationInSeconds` | Title + numbered items |
| `image` | `src`, `audio`, `durationInSeconds` | Pre-rendered image (fallback) |
| `segment` | `title`, `durationInSeconds` | Silent title card between segments |
| `outro` | `durationInSeconds` | Logo only, no audio |
#### Animated scroll with `focus`
For longer diffs or code (more than ~30 lines), the renderer keeps the font at a readable 16px and uses an animated viewport that scrolls between focus points. Add a `focus` array to `diff` or `code` slides:
```json
{
"type": "diff",
"filename": "packages/editor/src/lib/Editor.ts",
"language": "typescript",
"diff": "... 60-line diff ...",
"focus": [
{ "line": 3, "at": 0 },
{ "line": 25, "at": 0.4 },
{ "line": 50, "at": 0.8 }
],
"audio": "audio-03.wav",
"durationInSeconds": 30
}
```
- **`line`** — The line number (0-indexed into the parsed diff/code lines) to center on screen.
- **`at`** — When to arrive at this position, as a fraction of the slide's duration (0 = start, 1 = end).
The viewport smoothly eases between focus points. Before the first point, it holds at the first position; after the last, it holds there.
**When to use focus:** Any diff or code slide with more than ~30 lines. Without focus, long content starts at the top and stays static — the viewer can't see the bottom. With focus, you guide the viewer's eye to the code being discussed at each moment.
**When to omit focus:** Short diffs (≤30 lines) fit on screen at 16px and don't need scrolling.
#### Writing diff fields
For `diff` slides, paste the **unified diff** for the relevant hunk(s). This is the output of `git diff` for that section of the file — including the `@@` hunk header and `+`/`-`/` ` line prefixes. The renderer parses these prefixes to apply green/red backgrounds and syntax highlighting.
To get a diff for a specific file:
```bash
git diff main..HEAD -- path/to/file.ts
```
Include only the relevant hunks, not the entire file diff. Strip the `diff --git` and `---`/`+++` header lines — start from the `@@` hunk header.
For `code` slides, paste the relevant source code (a function, a class, a section). No diff prefixes needed.
#### Segment title slides
Insert a **`segment` slide** before each content segment to introduce it — except before the intro and context/overview segments. This includes code walkthrough segments and the summary/conclusion. Each segment slide is **3 seconds of silence** with the segment title centered on screen.
```json
{
"type": "segment",
"title": "Zoom state machine",
"durationInSeconds": 3
}
```
These provide clear visual breaks between sections and give the viewer a moment to orient before each new topic.
#### Segment title labels on code/diff slides
Add a `title` field to `code` and `diff` slides to show a small label in the top-left corner identifying which segment the viewer is in. Use the same title as the preceding `segment` slide. This helps orient viewers, especially when a segment spans multiple slides.
```json
{
"type": "diff",
"title": "Zoom state machine",
"filename": "packages/editor/src/lib/ZoomTool.ts",
...
}
```

View File

@@ -0,0 +1,279 @@
#!/bin/bash
# generate-audio.sh — Generate walkthrough narration audio from a JSON script.
#
# Generates one TTS call per segment, producing individual WAV clips directly.
# No chunking, alignment, or splitting needed.
#
# Usage:
# ./generate-audio.sh <script.json> [output-dir]
#
# Input JSON format:
# {
# "style": "Read in a calm, steady, professional tone...",
# "voice": "Iapetus", (optional, default: Iapetus)
# "slides": [
# "Intro narration text...",
# "Problem slide narration...",
# "Approach narration...",
# ...
# ]
# }
#
# Output:
# <output-dir>/audio-00.wav, audio-01.wav, ...
# <output-dir>/durations.json
#
# Dependencies:
# ffmpeg / ffprobe
#
# Environment:
# GEMINI_API_KEY — required. Auto-sourced from .env if not set.
#
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
# --- Args ---
SCRIPT_JSON="${1:?Usage: generate-audio.sh <script.json> [output-dir]}"
OUTPUT_DIR="${2:-.}"
# Resolve relative paths
[[ "$SCRIPT_JSON" != /* ]] && SCRIPT_JSON="$(pwd)/$SCRIPT_JSON"
[[ "$OUTPUT_DIR" != /* ]] && OUTPUT_DIR="$(pwd)/$OUTPUT_DIR"
if [ ! -f "$SCRIPT_JSON" ]; then
echo "Error: ${SCRIPT_JSON} not found"
exit 1
fi
mkdir -p "$OUTPUT_DIR"
PYTHON="python3"
# --- API key ---
REPO_ROOT=$(git rev-parse --show-toplevel 2>/dev/null || echo ".")
if [ -z "${GEMINI_API_KEY:-}" ]; then
if [ -f "${REPO_ROOT}/.env" ]; then
export $(grep '^GEMINI_API_KEY=' "${REPO_ROOT}/.env" | xargs) 2>/dev/null || true
fi
fi
GEMINI_API_KEY="${GEMINI_API_KEY:?Set GEMINI_API_KEY environment variable or add it to .env}"
# --- Config ---
TTS_MODEL="gemini-3.1-flash-tts-preview"
TTS_ENDPOINT="https://generativelanguage.googleapis.com/v1beta/models/${TTS_MODEL}:generateContent"
SPEED=1.2 # Speed up narration (1.0 = no change)
# --- Run everything in Python for reliability ---
"$PYTHON" - "$SCRIPT_JSON" "$OUTPUT_DIR" "$GEMINI_API_KEY" "$TTS_MODEL" "$TTS_ENDPOINT" "$SPEED" <<'PYTHON_SCRIPT'
import json, sys, os, subprocess, base64, urllib.request, re
script_json = sys.argv[1]
output_dir = sys.argv[2]
api_key = sys.argv[3]
tts_model = sys.argv[4]
tts_endpoint = sys.argv[5]
speed = float(sys.argv[6])
MAX_RETRIES = 2
def api_call(endpoint, body_dict):
body = json.dumps(body_dict).encode()
req = urllib.request.Request(
f"{endpoint}?key={api_key}",
data=body,
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(req) as resp:
return json.loads(resp.read())
# --- Load narration ---
with open(script_json) as f:
data = json.load(f)
voice = data.get("voice", "Iapetus")
slides = data["slides"]
style = data.get("style",
"Read the following in a calm, steady, professional tone. "
"Speak at a measured pace.")
word_count = sum(len(s.split()) for s in slides)
print(f"=== Generating narration audio ===")
print(f" Voice: {voice}")
print(f" Slides: {len(slides)}")
print(f" Words: {word_count}")
print()
def call_tts(prompt_text):
"""Make a single TTS API call and return raw PCM bytes."""
response = api_call(tts_endpoint, {
"contents": [{"parts": [{"text": prompt_text}]}],
"generationConfig": {
"responseModalities": ["AUDIO"],
"speechConfig": {
"voiceConfig": {
"prebuiltVoiceConfig": {
"voiceName": voice
}
}
}
}
})
error_msg = response.get("error", {}).get("message", "")
if error_msg:
raise RuntimeError(f"TTS API error: {error_msg}")
return base64.b64decode(response["candidates"][0]["content"]["parts"][0]["inlineData"]["data"])
def pcm_to_wav(pcm_bytes, out_wav):
"""Convert raw 24kHz PCM to 48kHz WAV with speed adjustment."""
pcm_tmp = out_wav + ".pcm"
with open(pcm_tmp, "wb") as f:
f.write(pcm_bytes)
subprocess.run([
"ffmpeg", "-y", "-f", "s16le", "-ar", "24000", "-ac", "1",
"-i", pcm_tmp, "-af", f"atempo={speed}", "-ar", "48000", out_wav
], capture_output=True, check=True)
os.remove(pcm_tmp)
def get_duration(wav_path):
"""Get duration of a WAV file in seconds."""
result = subprocess.run(
["ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", wav_path],
capture_output=True, text=True
)
return float(result.stdout.strip())
def validate_duration(wav_path, word_count):
"""Check if audio duration is reasonable for the word count. Returns (ok, duration)."""
dur = get_duration(wav_path)
# At ~150 wpm before speed adjustment, expect word_count/150*60/speed seconds.
# Allow generous bounds: 0.5x to 3x expected.
expected = word_count / 150 * 60 / speed
lower = expected * 0.3
upper = expected * 3.0
# For very short segments (< 15 words), just check it's under 30s
if word_count < 15:
return dur < 30, dur
return lower <= dur <= upper, dur
# --- Generate one TTS call per segment ---
durations = {}
for i, text in enumerate(slides):
num = f"{i:02d}"
out_path = os.path.join(output_dir, f"audio-{num}.wav")
wc = len(text.split())
prompt = f"{style}\n\n{text}"
ok = False
for attempt in range(MAX_RETRIES + 1):
try:
label = f" [{num}] " + ("" if attempt == 0 else f"(retry {attempt}) ")
print(f"{label}Generating ({wc} words)...", end=" ", flush=True)
pcm_data = call_tts(prompt)
pcm_to_wav(pcm_data, out_path)
ok, dur = validate_duration(out_path, wc)
if ok:
print(f"{dur:.1f}s")
durations[f"audio-{num}.wav"] = round(dur, 2)
break
else:
expected = wc / 150 * 60 / speed
print(f"{dur:.1f}s (expected ~{expected:.0f}s, retrying)")
except (urllib.error.HTTPError, RuntimeError) as e:
print(f"error: {e}")
if attempt == MAX_RETRIES:
print(f" [error] Segment {i} failed after {MAX_RETRIES + 1} attempts")
sys.exit(1)
if not ok:
# Use last attempt even if validation failed — warn but continue
dur = get_duration(out_path)
durations[f"audio-{num}.wav"] = round(dur, 2)
print(f" [warn] Segment {i} audio may be unreliable ({dur:.1f}s for {wc} words)")
# --- Trim silence from each clip ---
MAX_SILENCE = 0.15 # strip nearly all silence — segment title slides handle inter-slide pauses
SILENCE_THRESHOLD = "-40dB"
print()
print("=== Trimming silence ===")
for i in range(len(slides)):
num = f"{i:02d}"
clip_path = os.path.join(output_dir, f"audio-{num}.wav")
# Detect silence at start and end using silencedetect
detect = subprocess.run([
"ffmpeg", "-i", clip_path, "-af",
f"silencedetect=noise={SILENCE_THRESHOLD}:d=0.1",
"-f", "null", "-"
], capture_output=True, text=True)
stderr = detect.stderr
# Get clip duration
clip_dur = get_duration(clip_path)
# Parse silence periods
silence_starts = re.findall(r'silence_start: ([\d.]+)', stderr)
silence_ends = re.findall(r'silence_end: ([\d.]+)', stderr)
# Find leading silence: silence that starts at ~0
trim_start = 0.0
if silence_starts and float(silence_starts[0]) < 0.05:
if silence_ends:
leading_silence = float(silence_ends[0])
if leading_silence > MAX_SILENCE:
trim_start = leading_silence - MAX_SILENCE
# Find trailing silence: silence that starts near the end and extends to clip end.
# Skip trailing trim on last segment — it's followed by the outro and
# the silence detector often clips the final word.
trim_end = clip_dur
is_last = (i == len(slides) - 1)
if not is_last and silence_starts:
last_silence_start = float(silence_starts[-1])
# Verify this silence actually extends to the clip end (not an internal pause).
last_silence_is_trailing = True
for se in silence_ends:
se_val = float(se)
if se_val > last_silence_start and se_val < clip_dur - 0.05:
last_silence_is_trailing = False
break
if last_silence_is_trailing and last_silence_start > 0.05: # not the leading silence
trailing_silence = clip_dur - last_silence_start
if trailing_silence > MAX_SILENCE:
trim_end = last_silence_start + MAX_SILENCE
if trim_start > 0 or trim_end < clip_dur:
trimmed_path = clip_path + ".tmp.wav"
subprocess.run([
"ffmpeg", "-y", "-i", clip_path,
"-ss", str(trim_start), "-to", str(trim_end),
"-c", "copy", trimmed_path
], capture_output=True)
os.replace(trimmed_path, clip_path)
new_dur = trim_end - trim_start
durations[f"audio-{num}.wav"] = round(new_dur, 2)
print(f" audio-{num}.wav: {clip_dur:.1f}s -> {new_dur:.1f}s (trimmed {clip_dur - new_dur:.1f}s)")
else:
print(f" audio-{num}.wav: {clip_dur:.1f}s (no trim needed)")
# --- Write durations.json ---
durations_path = os.path.join(output_dir, "durations.json")
with open(durations_path, "w") as f:
json.dump(durations, f, indent=2)
total_dur = sum(durations.values())
print(f"\n Wrote durations.json ({len(durations)} entries, {total_dur:.1f}s total)")
print()
print("=== Done ===")
PYTHON_SCRIPT
echo ""
echo "Output:"
ls -la "${OUTPUT_DIR}"/audio-*.wav 2>/dev/null || echo " (no files generated)"

View File

@@ -0,0 +1,89 @@
#!/bin/bash
# make-video.sh — Assemble walkthrough slides + audio into a final MP4.
#
# Usage:
# ./make-video.sh <slide-dir> <output.mp4> [outro-duration]
#
# Expects in <slide-dir>:
# slide-00.png, slide-01.png, ... (one per segment, including outro)
# audio-00.wav, audio-01.wav, ... (one per narrated segment)
#
# The last slide PNG without a matching audio WAV is the silent outro.
#
set -euo pipefail
SLIDE_DIR="${1:?Usage: make-video.sh <slide-dir> <output.mp4> [outro-duration]}"
OUTPUT="${2:?Usage: make-video.sh <slide-dir> <output.mp4> [outro-duration]}"
OUTRO_DUR="${3:-3}"
# Resolve relative paths
[[ "$SLIDE_DIR" != /* ]] && SLIDE_DIR="$(pwd)/$SLIDE_DIR"
[[ "$OUTPUT" != /* ]] && OUTPUT="$(pwd)/$OUTPUT"
mkdir -p "$(dirname "$OUTPUT")"
TMPDIR_WORK=$(mktemp -d)
trap "rm -rf $TMPDIR_WORK" EXIT
echo "=== Assembling video ==="
echo " Slides: $SLIDE_DIR"
echo " Output: $OUTPUT"
# Count slides and audio
SLIDE_COUNT=$(ls "$SLIDE_DIR"/slide-*.png 2>/dev/null | wc -l | tr -d ' ')
AUDIO_COUNT=$(ls "$SLIDE_DIR"/audio-*.wav 2>/dev/null | wc -l | tr -d ' ')
echo " Found $SLIDE_COUNT slides, $AUDIO_COUNT audio clips"
echo " Last slide (no audio) = outro (${OUTRO_DUR}s)"
# Create per-segment videos
CONCAT_LIST="$TMPDIR_WORK/concat.txt"
> "$CONCAT_LIST"
for i in $(seq 0 $((SLIDE_COUNT - 1))); do
NUM=$(printf "%02d" $i)
SLIDE="$SLIDE_DIR/slide-${NUM}.png"
AUDIO="$SLIDE_DIR/audio-${NUM}.wav"
SEGMENT="$TMPDIR_WORK/segment-${NUM}.mp4"
if [ -f "$AUDIO" ]; then
# Get audio duration
DUR=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$AUDIO")
echo " segment-${NUM}: slide + audio (${DUR}s)"
# Create video segment: static image for duration of audio, with audio
ffmpeg -y -loop 1 -i "$SLIDE" -i "$AUDIO" \
-c:v libx264 -tune stillimage -pix_fmt yuv420p \
-vf "scale=1600:900:force_original_aspect_ratio=decrease,pad=1600:900:(ow-iw)/2:(oh-ih)/2" \
-c:a aac -b:a 192k -ar 48000 \
-shortest -movflags +faststart \
"$SEGMENT" 2>/dev/null
else
# Silent outro slide
echo " segment-${NUM}: silent outro (${OUTRO_DUR}s)"
ffmpeg -y -loop 1 -i "$SLIDE" -f lavfi -i anullsrc=r=48000:cl=mono \
-c:v libx264 -tune stillimage -pix_fmt yuv420p \
-vf "scale=1600:900:force_original_aspect_ratio=decrease,pad=1600:900:(ow-iw)/2:(oh-ih)/2" \
-c:a aac -b:a 192k -ar 48000 \
-t "$OUTRO_DUR" -movflags +faststart \
"$SEGMENT" 2>/dev/null
fi
echo "file '$SEGMENT'" >> "$CONCAT_LIST"
done
# Concatenate all segments
echo ""
echo " Concatenating ${SLIDE_COUNT} segments..."
ffmpeg -y -f concat -safe 0 -i "$CONCAT_LIST" \
-c copy -movflags +faststart \
"$OUTPUT" 2>/dev/null
# Report
FINAL_DUR=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$OUTPUT")
FINAL_SIZE=$(ls -lh "$OUTPUT" | awk '{print $5}')
echo ""
echo "=== Done ==="
echo " Output: $OUTPUT"
echo " Duration: ${FINAL_DUR}s"
echo " Size: $FINAL_SIZE"

View File

@@ -0,0 +1,6 @@
# Generated by render.sh
index.html
assets/
transcripts/
renders/
node_modules/

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,9 @@
{
"$schema": "https://hyperframes.heygen.com/schema/hyperframes.json",
"registry": "https://raw.githubusercontent.com/heygen-com/hyperframes/main/registry",
"paths": {
"blocks": "compositions",
"components": "compositions/components",
"assets": "assets"
}
}

View File

@@ -0,0 +1,4 @@
{
"id": "pr-walkthrough",
"name": "pr-walkthrough"
}

View File

@@ -0,0 +1,160 @@
#!/bin/bash
# render.sh — Render a pr-walkthrough video from a manifest JSON file using
# hyperframes. The pipeline:
# 1. Copy referenced audio/image files into ./assets/
# 2. Run whisper transcription on each audio file → ./transcripts/
# 3. Run build.mjs to generate index.html
# 4. Lint + render via npx hyperframes (1080p/30fps)
# 5. Downscale + recompress to 1280×720 with ffmpeg → final MP4
#
# Usage:
# ./render.sh <manifest.json> <output.mp4>
#
set -euo pipefail
MANIFEST="${1:?Usage: render.sh <manifest.json> <output.mp4>}"
OUTPUT="${2:?Usage: render.sh <manifest.json> <output.mp4>}"
# Resolve relative paths
[[ "$MANIFEST" != /* ]] && MANIFEST="$(pwd)/$MANIFEST"
[[ "$OUTPUT" != /* ]] && OUTPUT="$(pwd)/$OUTPUT"
if [ ! -f "$MANIFEST" ]; then
echo "Error: Manifest not found: $MANIFEST" >&2
exit 1
fi
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
ASSETS_DIR="$SCRIPT_DIR/assets"
TRANSCRIPTS_DIR="$SCRIPT_DIR/transcripts"
RENDERS_DIR="$SCRIPT_DIR/renders"
MANIFEST_DIR="$(dirname "$MANIFEST")"
mkdir -p "$ASSETS_DIR" "$TRANSCRIPTS_DIR" "$RENDERS_DIR"
mkdir -p "$(dirname "$OUTPUT")"
echo "=== Rendering walkthrough video ==="
echo " Manifest: $MANIFEST"
echo " Output: $OUTPUT"
echo ""
# --- 1. Extract referenced audio/image filenames from the manifest ---------
REFERENCED_FILES=$(python3 -c "
import json, sys
with open(sys.argv[1]) as f:
m = json.load(f)
files = set()
for s in m['slides']:
if 'audio' in s and s['audio']: files.add(s['audio'])
if 'src' in s and s['src']: files.add(s['src'])
print('\n'.join(sorted(files)))
" "$MANIFEST")
# --- 2. Copy referenced files into ./assets/ -------------------------------
echo " [1/5] Copying assets..."
# Clear assets to avoid stale files from previous runs.
rm -rf "$ASSETS_DIR"
mkdir -p "$ASSETS_DIR"
for FILE in $REFERENCED_FILES; do
SRC="$MANIFEST_DIR/$FILE"
if [ -f "$SRC" ]; then
cp "$SRC" "$ASSETS_DIR/$FILE"
else
echo " Warning: Referenced file not found: $SRC" >&2
fi
done
# --- 3. Whisper transcribe each audio file (idempotent per file) ----------
echo " [2/5] Transcribing audio (whisper)..."
for WAV in "$ASSETS_DIR"/*.wav; do
[ -f "$WAV" ] || continue
BASE=$(basename "$WAV" .wav)
OUT_JSON="$TRANSCRIPTS_DIR/$BASE.json"
# Skip if already transcribed and source is older
if [ -f "$OUT_JSON" ] && [ "$OUT_JSON" -nt "$WAV" ]; then
continue
fi
echo " transcribing $BASE..."
(cd "$SCRIPT_DIR" && npx --yes hyperframes transcribe "assets/$BASE.wav" --json >/dev/null)
# hyperframes writes to ./transcript.json — move to per-audio location
if [ -f "$SCRIPT_DIR/transcript.json" ]; then
mv "$SCRIPT_DIR/transcript.json" "$OUT_JSON"
fi
done
# --- 4. Generate index.html ------------------------------------------------
echo " [3/5] Building composition..."
(cd "$SCRIPT_DIR" && node build.mjs "$MANIFEST")
# --- 5. Lint (warn but don't fail) ------------------------------------------
(cd "$SCRIPT_DIR" && npx --yes hyperframes lint) || {
echo " Warning: lint reported issues (continuing)" >&2
}
# --- 6. Render at 1080p/30fps with hyperframes -----------------------------
# hyperframes can hang on telemetry/cleanup after writing the MP4. Run it in
# the background, wait for the file to appear and stop growing, then kill the
# process and continue.
echo " [4/5] Rendering 1080p frames..."
RENDER_NAME="walkthrough-$$"
TEMP_RENDER="$RENDERS_DIR/$RENDER_NAME.mp4"
rm -f "$TEMP_RENDER"
(cd "$SCRIPT_DIR" && npx --yes hyperframes render \
-q draft --crf 30 \
-o "renders/$RENDER_NAME.mp4") >/dev/null 2>&1 &
RENDER_PID=$!
# Poll until the MP4 exists and its size is stable for 3 consecutive checks
# (file fully written), then kill the parent npx and any child processes.
PREV_SIZE=-1
STABLE=0
while kill -0 "$RENDER_PID" 2>/dev/null; do
if [ -f "$TEMP_RENDER" ]; then
SIZE=$(stat -f '%z' "$TEMP_RENDER" 2>/dev/null || echo 0)
if [ "$SIZE" -gt 0 ] && [ "$SIZE" -eq "$PREV_SIZE" ]; then
STABLE=$((STABLE + 1))
if [ "$STABLE" -ge 3 ]; then break; fi
else
STABLE=0
fi
PREV_SIZE=$SIZE
fi
sleep 2
done
# Reap any lingering hyperframes/node children, then the npx wrapper.
pkill -P "$RENDER_PID" 2>/dev/null || true
kill "$RENDER_PID" 2>/dev/null || true
wait "$RENDER_PID" 2>/dev/null || true
if [ ! -f "$TEMP_RENDER" ]; then
echo "Error: hyperframes did not produce $TEMP_RENDER" >&2
exit 1
fi
# --- 7. Downscale 1080p → 720p, recompress for smaller file ---------------
echo " [5/5] Downscaling to 720p / 30fps..."
ffmpeg -y -i "$TEMP_RENDER" \
-vf "scale=1280:720:flags=lanczos,fps=30" \
-c:v libx264 -preset slow -crf 26 -pix_fmt yuv420p \
-c:a aac -b:a 96k -ar 48000 \
-movflags +faststart \
"$OUTPUT" 2>/dev/null
rm -f "$TEMP_RENDER"
# --- Report ------------------------------------------------------------------
if [ -f "$OUTPUT" ]; then
FINAL_DUR=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$OUTPUT" 2>/dev/null || echo "?")
FINAL_SIZE=$(ls -lh "$OUTPUT" | awk '{print $5}')
echo ""
echo "=== Done ==="
echo " Output: $OUTPUT"
echo " Resolution: 1280×720"
echo " FPS: 30"
echo " Duration: ${FINAL_DUR}s"
echo " Size: $FINAL_SIZE"
else
echo "Error: render failed — output file not created" >&2
exit 1
fi