diff --git a/.changes/knowledge/fix-video-digest-relayout-and-caption-dedupe-59495b96.md b/.changes/knowledge/fix-video-digest-relayout-and-caption-dedupe-59495b96.md new file mode 100644 index 00000000000..e71a572a330 --- /dev/null +++ b/.changes/knowledge/fix-video-digest-relayout-and-caption-dedupe-59495b96.md @@ -0,0 +1,11 @@ +--- +bump: minor +--- + +### Added + +- video-digest: `watch/relayout-slice.js` copies a slice into the knowledge-corpus layout (`transcript/`, `metadata/`, `frames/all/`, `frames/key/`, `media/`, `analysis/`, provenance `README.md`), rewrites the slice paths its markdown names through a path map (a pipeline file the layout does not keep becomes plain text saying so), and exits 1 listing every slice-internal path that still does not resolve. It runs on a closed slice, or before `close` on one whose outcome checks pass so the temp media is still there, and refuses a slice whose temp media is gone unless `--no-media`, which takes nothing from the temp session. It builds the layout in a staging directory and moves it into place only when the link check passes; an existing target needs `--replace` ([#6902](https://github.com/melodic-software/claude-code-plugins/issues/6902)). Hand re-layouts had broken relative paths twice: melodic-software/knowledge-corpus#42 needed four review rounds and six threads to clear them, and knowledge-corpus#46 fixed 41 broken paths in the slice knowledge-corpus#37 had landed. + +### Fixed + +- video-digest: an auto-caption transcript paragraph no longer repeats the previous paragraph's closing words that a rolling caption carried over: 2 to 12 words compared case- and punctuation-insensitively, dropped only when the opening cue carries them from the cue before it (the cues overlap in time, or the cue starts with the previous cue's words verbatim), so a phrase the speaker genuinely repeats stays ([#6992](https://github.com/melodic-software/claude-code-plugins/issues/6992)). diff --git a/plugins/knowledge/skills/video-digest/context/output-contract.md b/plugins/knowledge/skills/video-digest/context/output-contract.md index 988b6fa4396..c7af5d6103c 100644 --- a/plugins/knowledge/skills/video-digest/context/output-contract.md +++ b/plugins/knowledge/skills/video-digest/context/output-contract.md @@ -58,6 +58,115 @@ once per process, and the new name wins when both are set. material at a differently-shaped path (for example `sources///`) does not get that shape from this skill: land under `library_dir` as-written and re-lay-out by hand, or fork the sub-path in your own automation. This skill's contract is root relocation only. +For the knowledge-corpus layout there is also `relayout-slice.js`, described next. + +### Re-laying out a slice with `relayout-slice.js` + +Run it on a slice landed under `library_dir`: + +```bash +node "/extraction/run.mjs" --data-dir "" watch/relayout-slice.js "" "" [--no-media] [--replace] +``` + +Re-laying out a slice by hand has repeatedly left broken relative paths in the corpus; the +script's rewrite and link check are what catch them. + +The media, every extracted frame, the served caption tracks and the info JSON come from the temp +session `watch.json` records, which `close` removes. To keep them, run in this order: + +1. `evals/check-watch-outcomes.js ""` passes. +2. `watch/relayout-slice.js "" ""`, with media. +3. `watch/watch-state.js close ""`. + +The script runs on a closed slice, or on an unclosed one whose outcome checks pass (it runs them +itself and refuses when they fail). It refuses any slice whose temp media is gone unless +`--no-media` is passed, which takes nothing from the temp session (no media, frames, caption +tracks or info JSON) even when it still exists; after a normal close that is the only way it runs. +It refuses a `watch.json` that is not an object, whose `videoId` is not a plain id (it names +`media/.`), that lacks a `title` or `sourceUrl`, or whose temp session dirs are not +strings. Without `--no-media` it also refuses a partly removed temp session, naming the missing +piece: the video, an empty video, caption track or frame image, any of those (or the +sidecar) that is not a regular file, the info JSON or +`frame-times.json` (or either one not holding a JSON object, or a `frame-times.json` record +without a finite number or null `timestampSec`), any frame image, a frame +`frame-times.json` lists, a `frame-times.json` entry for a scene or interval frame on disk +(anchor frames carry their time in their name), or the caption tracks. It refuses a temp session +holding several videos (an X post with more than one), naming them, since the watch does not +record which one is primary; a video whose id is not this slice's (for an X post, whose +info JSON does not name this post as its `display_id`); and a symlink in the slice or the temp +session, since a copy would follow it. In every mode, `--no-media` included, it refuses a recorded +work, frames or contact-sheets dir outside the OS temp dir or overlapping the slice, existing or +not, since `close` deletes those. It refuses when the slice is, holds, or sits +inside a path the run writes: the target, `.relayout-backup`, or a +`..relayout-*` staging dir beside the target. In +every mode, `--no-media` included, it refuses when one of those paths is, holds, or sits inside a +temp dir `watch.json` records, existing or not, since `close` deletes those. It refuses an +existing target unless `--replace` is passed, which replaces the target and keeps its +`README.md`. The replaced target is moved to `.relayout-backup` and deleted only once the +new layout is in place. If a run is interrupted mid-swap, the next run restores the target from +that backup when the target is missing (and says so), and refuses, naming the backup, when both +exist; remove whichever one you do not want to keep. + +It writes the knowledge-corpus layout: `transcript/`, `metadata/` (with a trimmed `info.json`, +the companion sources brief, and the fetched `decks/` and `attachments/`), `frames/all/`, +`frames/key/` beside the frame and vision logs in `frames/`, `media/.`, `analysis/` +(`RESEARCH.md`, `research/`, `recommendations/`, the companion digest), and a provenance +`README.md` when the target has none; the trimmed `info.json` also drops the local download +paths yt-dlp records. The script's header comment is the authoritative mapping. Each slice path +the copied markdown names is rewritten through a path map relative to the file's new location: +inline links (balanced or escaped parentheses in the destination, quoted or parenthesized titles), +reference-style definitions whose line holds nothing after the destination but an optional +title, HTML `src`/`href` attributes in any quoting or case (character references decoded), +and backticked paths, with each +segment percent-decoded (and re-encoded on rewrite) and `\` separators read as `/`. +A relative link that leaves the slice keeps pointing at the same file. Fenced code blocks are +left untouched, and link syntax inside an inline code span is not read as a link. Text inside an +HTML tag or comment is data: only a tag's own `src`/`href` attributes are read. +One naming a pipeline file the layout does not keep (such as `run-state/watch.json`, the slice's +own journey `README.md`, or an extracted frame's bare name under `--no-media`) becomes plain text +saying it is not retained; one naming a directory whose files went to several places (`source/`) +names where they went; one naming a lane folder the slice never produced (`source/decks/`) says +so. It builds the layout in a sibling staging directory and link-checks it there; it exits 1 +listing every slice-internal path that still does not resolve, including a bare extracted-frame +name and a path under an original lane whose parent directory the slice had (`source/missing.txt`). +Not checked: paths under `plugins/`, `docs/`, `templates/` and `.claude/`, `AGENTS.md`, paths +that leave the target, and backticked paths that are neither under such a lane directory nor +under a top-level directory of the target or an entry beside the naming file (paths into another +repository, such as `research/deepening/...` from an upstream repo). Fix what the check lists in +the slice and re-run. Only a passing check moves the staging directory into place; on any failure +the staging directory is removed and the target is left as it was. Without `--replace` the +install is a single rename of the staging dir onto the +target path, which fails, and the run refuses, when a target with content appeared during the +run. On POSIX that rename replaces an empty directory someone created meanwhile; that is +accepted, since an empty directory holds no data. A run killed mid-copy or mid-install leaves its +`..relayout-*` staging dir behind; the next run names any such dir in a warning for +deletion by hand and never installs or deletes one, since it may belong to another slice's run +or to a run still going. The warning does not stop the run; delete a named dir once no run is +using it. + +### Known limits of the re-layout + +The reference scanner is not a full CommonMark parser. It does not model indented code blocks, +fenced code blocks that are indented or inside block quotes or list items, HTML blocks, link +labels nested more than one level, or link reference definitions inside containers (lists, block +quotes). References in those constructs may be left unrewritten or read as ordinary text, and the +link check is the backstop: it refuses a dangling slice-internal path it can see. + +It also does not decode named character references beyond `& < > " '` +(numeric references are decoded). A destination holding another one (`a@b.pdf` for the file +`a@b.pdf`) is not mapped to its file, and the link check refuses it, naming the path, so nothing +passes silently. +When a retained file's name literally contains such a reference and the layout leaves its path +unchanged, the link is left as written and is not flagged. + +Link-shaped text inside a reference definition's title is read as a link: the rewrite may update +it, and if it names a missing path the link check refuses the run. + +A `<` right after an unescaped `](` is read as a link destination even when the `]` has no +opening label; an HTML tag there is not masked, and a missing path inside it makes the run refuse. + +Code spans are found before raw HTML, so a backtick inside a quoted HTML attribute can pair with a +later backtick and hide the text between them from the rewrite and the link check. ## Agent-written artifacts share the same root diff --git a/plugins/knowledge/skills/video-digest/extraction/transcript/write-transcript.js b/plugins/knowledge/skills/video-digest/extraction/transcript/write-transcript.js index 1415b26dd74..6e3bd5cac0b 100644 --- a/plugins/knowledge/skills/video-digest/extraction/transcript/write-transcript.js +++ b/plugins/knowledge/skills/video-digest/extraction/transcript/write-transcript.js @@ -11,7 +11,7 @@ import { cleanManualCaptions, stripCaptionHtmlEntities, } from "@melodic/video-digestion/transcript/manual-caption-clean"; -import { formatTranscript } from "@melodic/video-digestion/transcript/vtt-parser"; +import { formatTimestamp, formatTranscript } from "@melodic/video-digestion/transcript/vtt-parser"; import { primaryEntry } from "../adapters/adapter-contract.js"; import { LANES, lanePath } from "../lib/slice-lanes.js"; @@ -34,6 +34,127 @@ function countParagraphs(transcript) { return transcript ? transcript.split("\n\n").length : 0; } +const BOUNDARY_REPEAT_MIN_WORDS = 2; +const BOUNDARY_REPEAT_MAX_WORDS = 12; +const PARAGRAPH_STAMP = /^(\[[^\]]*\]\s*)([\s\S]*)$/; + +/** @param {string} text */ +function tokens(text) { + return text.split(/\s+/).filter(Boolean); +} + +/** @param {string} word */ +function comparableWord(word) { + // Apostrophes in any form (', ’, ʼ) drop out, so don't, don’t and dont match. + // Only sentence punctuation at either end goes, so C++. and C# stay distinct. + return word.toLowerCase().replace(/['\u2019\u02BC]/g, "").replace(EDGE_PUNCTUATION, ""); +} + +/** + * Sentence punctuation (quotes, brackets, ellipses) at a word's ends. Hyphens + * and dashes stay, so C-- and C never compare equal. + */ +const EDGE_PUNCTUATION = + /^[.,;:!?"\u2018\u201a\u201c\u201d\u201e\u2039\u203a\u00ab\u00bb()[\]{}\u2026]+|[.,;:!?"\u2018\u201a\u201c\u201d\u201e\u2039\u203a\u00ab\u00bb()[\]{}\u2026]+$/g; + +/** + * The lexical words of `text`: punctuation-only tokens such as `>>` are not + * words, so they never count toward a run's length. + * + * @param {string} text + */ +function words(text) { + return tokens(text).filter(isWord); +} + +/** @param {string} token - a word holds at least one letter or digit */ +function isWord(token) { + return /[\p{L}\p{N}]/u.test(token); +} + +/** @param {string[]} run */ +function comparableRun(run) { + return run.map(comparableWord).join(" "); +} + +/** + * True when the cue opening the paragraph stamped `stamp` carries `run` over + * from the cue before it: the two cues overlap in time, or the opening cue + * starts with the previous cue's closing words verbatim (a rolling caption). + * False whenever the opening cue cannot be identified exactly. + * + * @param {string} stamp - the paragraph's `[M:SS]` + * @param {string[]} run - the paragraph's leading words + * @param {readonly { startSec: number, endSec: number, text: string }[]} cues + */ +function carriesOverFromPreviousCue(stamp, run, cues) { + // The paragraph stamp floors the opening cue's start to the second, so it + // identifies the opener only when exactly one cue starts in that second. + // With several, the evidence is ambiguous and the words stay. + const openers = cues.flatMap((cue, at) => + `[${formatTimestamp(cue.startSec)}]` === stamp ? [at] : [], + ); + if (openers.length !== 1 || openers[0] === 0) return false; + const cue = cues[openers[0]]; + const previous = cues[openers[0] - 1]; + const head = run.join(" "); + if (words(cue.text).slice(0, run.length).join(" ") !== head) return false; + return ( + previous.endSec > cue.startSec || words(previous.text).slice(-run.length).join(" ") === head + ); +} + +/** + * Drop the words a paragraph opens with when they repeat the previous + * paragraph's closing words, the longest run first: 2 to 12 words compared + * case- and punctuation-insensitively, and only when the opening cue carries + * them over from the cue before it ({@link carriesOverFromPreviousCue}). A + * rolling auto caption carries its tail into the next cue, and the vendor + * formatter only drops overlaps of 3 or more words it matches ignoring case + * alone. A repeat the speaker actually said ("very good." / "Very good + * question", "I don't know." / "I don't know why.") stays. A paragraph always + * keeps at least one word. + * + * @param {string} transcript - paragraphs joined by a blank line, each `[M:SS] text` + * @param {readonly { startSec: number, endSec: number, text: string }[]} cues - the + * cues the transcript was formatted from + * @returns {string} + */ +export function dropParagraphBoundaryRepeats(transcript, cues) { + const paragraphs = transcript.split("\n\n"); + for (let index = 1; index < paragraphs.length; index++) { + const current = PARAGRAPH_STAMP.exec(paragraphs[index]); + const previous = PARAGRAPH_STAMP.exec(paragraphs[index - 1]); + if (!current || !previous) continue; + const previousWords = words(previous[2]); + const currentTokens = tokens(current[2]); + const wordAt = currentTokens.flatMap((token, at) => (isWord(token) ? [at] : [])); + const currentWords = wordAt.map((at) => currentTokens[at]); + const longest = Math.min( + BOUNDARY_REPEAT_MAX_WORDS, + previousWords.length, + currentWords.length - 1, + ); + for (let length = longest; length >= BOUNDARY_REPEAT_MIN_WORDS; length--) { + const run = currentWords.slice(0, length); + if (comparableRun(previousWords.slice(-length)) !== comparableRun(run)) continue; + // Every match, whatever its length, needs cue-level carry-over evidence: + // a speaker may genuinely repeat several words ("I don't know." / "I + // don't know why."). + if (!carriesOverFromPreviousCue(current[1].trim(), run, cues)) continue; + // Keep punctuation-only tokens before the dropped words (a `>>` speaker + // marker); drop any between them. + const kept = [ + ...currentTokens.slice(0, wordAt[0]), + ...currentTokens.slice(wordAt[length - 1] + 1), + ]; + paragraphs[index] = current[1] + kept.join(" "); + break; + } + } + return paragraphs.join("\n\n"); +} + /** * Build transcript text from a caption file. * @@ -66,7 +187,8 @@ export function buildTranscriptText(vttText, isAutoCaption, { repairLexicon = nu repairedTermCount = repaired.replacementCount; } - const transcript = formatTranscript(cues); + const formatted = formatTranscript(cues); + const transcript = isAutoCaption ? dropParagraphBoundaryRepeats(formatted, cues) : formatted; return { transcript, diff --git a/plugins/knowledge/skills/video-digest/extraction/transcript/write-transcript.test.js b/plugins/knowledge/skills/video-digest/extraction/transcript/write-transcript.test.js index b2353bfe5c9..bb2600c46a9 100644 --- a/plugins/knowledge/skills/video-digest/extraction/transcript/write-transcript.test.js +++ b/plugins/knowledge/skills/video-digest/extraction/transcript/write-transcript.test.js @@ -41,6 +41,266 @@ describe("buildTranscriptText", () => { expect(result.transcript).toBe("[0:04] [music] >> Hi everybody."); }); + it("drops two words a rolling caption carries into the next paragraph's cue", () => { + // The 51Eb4EtGqrI shape (#6992): the cue opening the new paragraph starts + // with the previous cue's closing words verbatim. + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +we ran the + +00:00:03.000 --> 00:00:05.000 +numbers on benchmarks, too. + +00:00:05.000 --> 00:00:07.000 +benchmarks, too. And then we moved on. +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe( + "[0:01] we ran the numbers on benchmarks, too.\n\n[0:05] And then we moved on.", + ); + }); + + it("drops two repeated words when the opening cue overlaps the previous cue in time", () => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +we ran the + +00:00:03.000 --> 00:00:05.500 +numbers on benchmarks, too. + +00:00:05.000 --> 00:00:07.000 +Benchmarks too and then we moved on. +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe( + "[0:01] we ran the numbers on benchmarks, too.\n\n[0:05] and then we moved on.", + ); + }); + + it("drops a repeat of three or more words compared case- and punctuation-insensitively when the cues overlap", () => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +we ran the + +00:00:03.000 --> 00:00:05.500 +numbers on benchmarks, too. + +00:00:05.000 --> 00:00:07.000 +On benchmarks too and then we moved on. +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe( + "[0:01] we ran the numbers on benchmarks, too.\n\n[0:05] and then we moved on.", + ); + }); + + it("keeps two repeated words when a later cue in the opener's second carries them", () => { + // The opener (5.0s) does not carry "Very good" over; a later cue in the + // same second (5.4s) overlaps the opener and starts with those words. + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +that answer was + +00:00:03.000 --> 00:00:05.000 +very good. + +00:00:05.000 --> 00:00:05.600 +Very good question + +00:00:05.400 --> 00:00:07.000 +Very good indeed, thanks. +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript.split("\n\n")[1]).toMatch(/^\[0:05\] Very good question/); + }); + + it.each([ + ["a curly apostrophe", "don’t"], + ["a modifier-letter apostrophe", "donʼt"], + ["no apostrophe", "dont"], + ])("matches don't against %s when dropping a 3-word repeat across overlapping cues", (_label, variant) => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +so we + +00:00:03.000 --> 00:00:05.500 +really don't care. + +00:00:05.000 --> 00:00:07.000 +Really ${variant} care about it. +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe("[0:01] so we really don't care.\n\n[0:05] about it."); + }); + + it("keeps an overlapping cue whose words differ only by symbols, like C++ and C#", () => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +these days we + +00:00:03.000 --> 00:00:05.500 +use C++. + +00:00:05.000 --> 00:00:07.000 +use C# today +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe("[0:01] these days we use C++.\n\n[0:05] use C# today"); + }); + + it("keeps an overlapping cue whose word differs only by trailing dashes, like C-- and C", () => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +these days we + +00:00:03.000 --> 00:00:05.500 +use C--. + +00:00:05.000 --> 00:00:07.000 +use C today +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe("[0:01] these days we use C--.\n\n[0:05] use C today"); + }); + + it("drops an overlapping carry-over that differs only by curly single quotes", () => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +she said + +00:00:03.000 --> 00:00:05.500 +‘very good’. + +00:00:05.000 --> 00:00:07.000 +Very good question +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe("[0:01] she said ‘very good’.\n\n[0:05] question"); + }); + + it("drops an overlapping carry-over that differs only by curly quotes", () => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +she said + +00:00:03.000 --> 00:00:05.500 +“very good”. + +00:00:05.000 --> 00:00:07.000 +Very good question +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe("[0:01] she said “very good”.\n\n[0:05] question"); + }); + + it("keeps a three-word repeat the speaker said when no cue carries it over", () => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +honestly + +00:00:03.000 --> 00:00:05.000 +I don't know. + +00:00:05.000 --> 00:00:07.000 +I don't know why. +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe("[0:01] honestly I don't know.\n\n[0:05] I don't know why."); + }); + + it("does not count a punctuation-only token like >> as a repeated word", () => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +that answer was + +00:00:03.000 --> 00:00:05.000 +>> Very good. + +00:00:05.000 --> 00:00:07.000 +>> Very good question, thanks. +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe( + "[0:01] that answer was >> Very good.\n\n[0:05] >> Very good question, thanks.", + ); + }); + + it("drops overlapping cues' carried-over words after a >> marker, keeping the marker", () => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +that answer was + +00:00:03.000 --> 00:00:05.500 +>> Very good. + +00:00:05.000 --> 00:00:07.000 +>> Very good question, thanks. +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe( + "[0:01] that answer was >> Very good.\n\n[0:05] >> question, thanks.", + ); + }); + + it("keeps two words the speaker genuinely repeats across a paragraph boundary", () => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +that answer was + +00:00:03.000 --> 00:00:05.000 +very good. + +00:00:05.000 --> 00:00:07.000 +Very good question, thanks. +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe( + "[0:01] that answer was very good.\n\n[0:05] Very good question, thanks.", + ); + }); + + it("keeps a single repeated word at a paragraph boundary", () => { + const autoVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +we ran the + +00:00:03.000 --> 00:00:05.000 +numbers again. + +00:00:05.000 --> 00:00:07.000 +Again we moved on. +`; + const result = buildTranscriptText(autoVtt, true); + expect(result.transcript).toBe("[0:01] we ran the numbers again.\n\n[0:05] Again we moved on."); + }); + + it("leaves a manual-caption paragraph boundary repeat in place", () => { + const manualVtt = `WEBVTT + +00:00:01.000 --> 00:00:03.000 +We ran the numbers on benchmarks, too. + +00:00:05.000 --> 00:00:07.000 +Benchmarks too, and then we moved on. +`; + const result = buildTranscriptText(manualVtt, false); + expect(result.transcript).toContain("Benchmarks too, and then we moved on."); + }); + it("parses manual captions without auto-clean pass", () => { const manualVtt = `WEBVTT diff --git a/plugins/knowledge/skills/video-digest/extraction/watch/relayout-slice.js b/plugins/knowledge/skills/video-digest/extraction/watch/relayout-slice.js new file mode 100755 index 00000000000..44a1b1eeaad --- /dev/null +++ b/plugins/knowledge/skills/video-digest/extraction/watch/relayout-slice.js @@ -0,0 +1,2131 @@ +#!/usr/bin/env node +/** + * Copy a finished video-digest slice into a target directory in the + * knowledge-corpus layout, rewrite the slice-internal paths its markdown names, + * then link-check the result. A slice is finished when it is closed, or when + * it is not closed yet and its outcome checks pass (the pre-close run, while + * the temp session still holds the media). + * + * Target layout: `transcript/` (transcript text and the served caption tracks), + * `metadata/` (trimmed info JSON, harvested links and repo analysis, deck + * inventory, companion sources), `frames/all/` (every extracted frame plus + * `frame-times.json`), `frames/key/` (promoted key frames) beside the frame + * logs in `frames/`, `media/.`, `analysis/` (`RESEARCH.md`, + * `research/`, `recommendations/`, `companion-digest/`) and a provenance + * `README.md` when the target has none. + * + * Every reference is rewritten through a path map built from the copy plan: + * a reference resolves first against the naming file's own directory, then + * against the slice root. One landing on a copied path is rewritten relative + * to the file's new location; one landing on a slice file or directory the + * layout does not keep becomes plain text naming it as not retained; anything + * else is left alone. The link check then reports what still does not resolve. + * + * The layout is built in a sibling staging directory and moved into place only + * when the link check passes; on any failure the target is left untouched. An + * existing target is replaced only with `--replace`, which keeps its README. + * `--no-media` copies nothing from the temp session. + * + * Usage: node watch/relayout-slice.js [--no-media] [--replace] + * Exit: 0 written and every slice-internal path resolves; 1 refused, failed, + * or unresolved paths (listed on stderr, nothing written); 2 usage. + */ + +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import { isMainModule } from "@melodic/video-digestion/shared/main-module"; +import { writeStderr, writeStdout } from "@melodic/video-digestion/shared/terminal"; + +import { listWorkDirFiles } from "../acquisition/acquire.js"; +import { runCheckWatchOutcomes } from "../evals/check-watch-outcomes.js"; +import { LANES } from "../lib/slice-lanes.js"; +import { resolveTempSession } from "../lib/temp-session-paths.js"; +import { watchStatePath } from "./watch-state.js"; + +/** info.json keys the corpus copy drops: format, caption-URL, heatmap and request lists, and the local download paths yt-dlp records. */ +export const INFO_JSON_DROPPED_KEYS = Object.freeze([ + "formats", + "requested_formats", + "thumbnails", + "automatic_captions", + "subtitles", + "heatmap", + "http_headers", + "requested_subtitles", + "_format_sort_fields", + "_filename", + "filename", + "__files_to_move", + "requested_downloads", +]); + +export const INFO_JSON_COMMENT_CAP = 20; + +/** + * Paths that name files in another repository, never in the slice: a + * directory (trailing `/`) matches any path under it, a file only exactly. + */ +export const ALLOWED_MISS_PREFIXES = Object.freeze([ + "plugins/", + "docs/", + "templates/", + ".claude/", + "AGENTS.md", +]); + +/** + * True when `refPath` (fragment already split off) is an allowed miss: + * under one of the directories, or exactly one of the files, with any + * leading `./` ignored. `AGENTS.md.old` or `docs-old/x` does not match. + * + * @param {string} refPath + */ +function isAllowedMiss(refPath) { + const plain = refPath.replace(/^(?:\.\/)+/, ""); + return ALLOWED_MISS_PREFIXES.some((entry) => + entry.endsWith("/") ? plain.startsWith(entry) : plain === entry, + ); +} + +/** Source-lane folders the layout keeps, and where each one goes. */ +const SOURCE_FOLDERS = new Map([ + ["companion-digest", "analysis/companion-digest"], + ["decks", "metadata/decks"], + ["attachments", "metadata/attachments"], +]); +/** Lane folders the output contract names, present in a slice only when produced. */ +const KNOWN_LANE_FOLDERS = new Set([ + ...[...SOURCE_FOLDERS.keys()].map((folder) => `${LANES.source}/${folder}`), + `${LANES.research}/findings`, + `${LANES.keyFrames}/frames`, +]); +/** The pipeline's extracted frame names: scene, interval and anchor frames. */ +const FRAME_NAME = /^(?:(?:scene|interval)_\d{4}|anchor_\d+_\d{4})\.png$/; +/** Frames scene detection writes and records in `frame-times.json`. */ +const TIMED_FRAME = /^(?:scene|interval)_\d{4}\.png$/; +const KEY_FRAME_LOGS = new Set([ + "key-frames-manifest.md", + "visual-frames.md", + "key-frame-quality-audit.md", + "frame-triage-log.md", + "vision-plan.md", + "visual-gaps.md", +]); +const SOURCE_METADATA = new Set([ + "harvested-links.json", + "harvested-repo-analysis.json", + "deck-inventory.md", + "companion-sources.md", +]); +const TRANSCRIPT_TEXT = /^transcript(?:-\d+)?\.txt$/; + +/** Markdown files, by extension, in any case. */ +const MARKDOWN_FILE = /\.(?:md|markdown)$/i; +/** + * A paragraph boundary a code span cannot cross: a blank line, or a line that + * interrupts or ends a paragraph (ATX or setext heading, block quote, fence, thematic + * break, list item). + */ +const BLOCK_BREAK = + /\n(?:[ \t]*\r?\n| {0,3}(?:#{1,6}(?:[ \t]|\r?$)|>|`{3,}|~{3,}|(?:(?:\*[ \t]*){3,}|(?:-[ \t]*){3,}|(?:_[ \t]*){3,})\r?$|(?:=+|-+)[ \t]*\r?$|[-*+][ \t]+\S|1[.)][ \t]+\S))/m; +const VIDEO_FILE = /\.(?:mp4|mkv|webm)$/; +const FRAME_IMAGE = /\.(?:png|jpe?g)$/i; +/** The slice lane directories, plus the `SLICE/` root alias. */ +const SLICE_LANE_NAMES = new Set([...Object.values(LANES), "SLICE"]); +const URL_SCHEME = /^[a-z][a-z0-9+.-]*:/i; +/** Placeholder syntax, not a real path: ``, `${NAME}`, `{name}`, or a glob segment with `*`. */ +const PLACEHOLDER = /<[^<>]*>|\$\{[^}]*\}|\{[^{}]*\}|(?:^|\/)[^/]*\*/; +/** `:12`, `:12-20`, `:12,15` and `:12:5` (line and column) suffixes. */ +const LINE_SUFFIX = /(?::\d+(?:[-,]\d+)*)+$/; +/** A home path: `~` alone, `~/x` or `~user/x`; `~notes.md` is a relative name. */ +const HOME_PATH = /^~(?:[^/]*\/|$)/; +/** ASCII punctuation: the only characters a CommonMark backslash escapes. */ +const ESCAPABLE = /[!-/:-@[-`{-~]/; +/** + * A backslash and the punctuation it escapes, or a lone literal backslash: + * before anything else, a line ending included, it escapes nothing. + */ +const BACKSLASH = String.raw`\\[!-/:-@[-\x60{-~]|\\(?![!-/:-@[-\x60{-~])`; +/** + * The `[label]:` opening of a reference-style link definition, with + * CommonMark's label rules: up to 999 characters, backslash escapes allowed + * (`[a\]b]`), no unescaped bracket, line endings allowed but not a blank line. + * The destination may follow one line ending; parseDestination reads it. + */ +const REFERENCE_DEFINITION_LABEL = new RegExp( + String.raw`^ {0,3}\[((?:${BACKSLASH}|[^[\]\\\r\n]|\r?\n(?![ \t]*\r?\n)){1,999})\]:[ \t]*(?:\r?\n[ \t]*)?`, + "gm", +); +/** A line ending inside a title, which never spans a blank line (only spaces or tabs). */ +const TITLE_LINE_END = String.raw`\r?\n(?![ \t]*\r?\n)`; +/** What may follow a definition's destination: a line ending, or a title then one. */ +const DEFINITION_TAIL = new RegExp( + String.raw`^(?:[ \t]+(?:"(?:${BACKSLASH}|${TITLE_LINE_END}|[^"\\\n])*"|'(?:${BACKSLASH}|${TITLE_LINE_END}|[^'\\\n])*'|\((?:${BACKSLASH}|${TITLE_LINE_END}|[^()\\\n])*\)))?[ \t]*(?:\r?\n|$)`, +); + +/** + * Every reference-style link definition in `text` whose label holds a + * non-space character (CommonMark rejects an all-whitespace label), its + * destination read by the same parser as inline links and followed, on its + * line, by nothing but whitespace and an optional title. A line such as + * `[note]: source/x.txt is illustrative` is prose, not a definition. + * + * @param {string} text + * @returns {{ from: number, to: number, raw: string, angle: boolean }[]} + */ +function referenceDefinitions(text) { + const found = []; + for (const match of text.matchAll(REFERENCE_DEFINITION_LABEL)) { + if (!/\S/.test(match[1])) continue; + const from = /** @type {number} */ (match.index) + match[0].length; + const parsed = parseDestination(text, from); + if (parsed === null || parsed.raw === "") continue; + if (!DEFINITION_TAIL.test(text.slice(parsed.end))) continue; + found.push({ from, to: parsed.end, raw: parsed.raw, angle: parsed.angle }); + } + return found; +} +/** + * One attribute of an HTML start tag, read from where the previous one ended: + * a name, then an optional double-, single- or unquoted value. Sticky, so a + * `src=` inside another attribute's quoted value is never read as a name. + */ +const HTML_ATTRIBUTE = /[\s/]*([^\s"'>/=]+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s"'=<>`]+)))?/dy; +/** + * A backticked span names a path when it holds a separator or ends in an + * extension, optionally followed by a `?query`, a `#fragment`, or both. + */ +const PATH_LIKE = /[/\\]|\.[A-Za-z0-9]+(?::\d+(?:[-,]\d+)*)*(?:\?[^#]*)?(?:#.*)?$/; + +/** An HTML start tag: ``. */ +const HTML_START_TAG = + /<([A-Za-z][A-Za-z0-9-]*)(?:(?:[ \t]+(?:\r?\n[ \t]*)?|\r?\n[ \t]*)[A-Za-z_:][A-Za-z0-9_.:-]*(?:[ \t]*(?:\r?\n[ \t]*)?=[ \t]*(?:\r?\n[ \t]*)?(?:[^ \t\r\n"'=<>`]+|'[^']*'|"[^"]*"))?)*[ \t]*(?:\r?\n[ \t]*)?\/?>/g; +/** An HTML comment. */ +const HTML_COMMENT = /||/g; +/** Other raw inline HTML: a closing tag, processing instruction, declaration or CDATA section. */ +const HTML_OTHER = + /<\/[A-Za-z][A-Za-z0-9-]*\s*>|<\?[\s\S]*?\?>|]*>|/g; +/** A CommonMark URI or email autolink: https://spec.commonmark.org/0.31.2/#autolinks */ +const URI_AUTOLINK = /[A-Za-z][A-Za-z0-9+.-]{1,31}:[^\x00-\x20<>]*/.source; +const EMAIL_LABEL = /[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?/.source; +const EMAIL_AUTOLINK = `${/[\w.!#$%&'*+/=?^`{|}~-]+@/.source}${EMAIL_LABEL}(?:\\.${EMAIL_LABEL})*`; +const AUTOLINK = new RegExp(`<(?:${URI_AUTOLINK}|${EMAIL_AUTOLINK})>`, "g"); +/** + * A `<` opening a link or definition destination: after `](` or a definition + * label's `]:`, plus optional whitespace with at most one line ending. + */ +const DESTINATION_OPENER = + /(?<=(?:(? match?.[at] !== undefined) ?? 2; + const span = match.indices?.[group]; + if (!span || span[0] === span[1]) continue; + values.push({ from: base + span[0], to: base + span[1], group }); + } + } + return values; +} + +/** + * The URL an HTML `src`/`href` value names: ASCII tab, LF and CR removed, as + * the URL parser removes them, so a value wrapped across lines still names its + * path, then character references decoded. + * https://url.spec.whatwg.org/#concept-basic-url-parser + * + * @param {string} value + */ +function htmlAttributeUrl(value) { + // The HTML parser decodes references first; the URL parser then trims leading and + // trailing C0 controls and spaces and drops tab, LF and CR. + return decodeHtmlReferences(value, { attribute: true }) + .replace(/^[\x00-\x20]+|[\x00-\x20]+$/g, "") + .replace(/[\t\n\r]/g, ""); +} + +/** + * Ranges of `text` holding HTML start tags, comments and autolinks: attribute, + * comment and autolink text is data, so the link, definition and + * backticked-path passes skip it (an autolink's `](` is no link); only + * htmlAttributeValues reads a tag. An angle-bracket link or definition + * destination such as `` is a destination, not a tag or autolink. + * + * @param {string} text + * @returns {[number, number][]} + */ +function htmlRanges(text) { + const isDestination = (/** @type {number} */ at) => { + DESTINATION_OPENER.lastIndex = at; + return DESTINATION_OPENER.test(text); + }; + const tags = [...text.matchAll(HTML_START_TAG), ...text.matchAll(AUTOLINK)].filter( + (tag) => !isDestination(Number(tag.index)), + ); + return [...text.matchAll(HTML_COMMENT), ...text.matchAll(HTML_OTHER), ...tags] + // A `<` after an odd run of backslashes is escaped, so it opens no HTML. + .filter((match) => !isEscapedAt(text, Number(match.index))) + .map((match) => { + const from = /** @type {number} */ (match.index); + return /** @type {[number, number]} */ ([from, from + match[0].length]); + }); +} + +const HTML_NAMED_REFERENCES = /** @type {Record} */ ({ + amp: "&", + lt: "<", + gt: ">", + quot: '"', + apos: "'", +}); + +/** + * HTML's replacements for numeric references to 0x80-0x9F (windows-1252), + * indexed from 0x80; a 0 entry keeps the code point. + * https://html.spec.whatwg.org/multipage/parsing.html#numeric-character-reference-end-state + */ +const WINDOWS_1252 = [ + 0x20ac, 0, 0x201a, 0x0192, 0x201e, 0x2026, 0x2020, 0x2021, + 0x02c6, 0x2030, 0x0160, 0x2039, 0x0152, 0, 0x017d, 0, + 0, 0x2018, 0x2019, 0x201c, 0x201d, 0x2022, 0x2013, 0x2014, + 0x02dc, 0x2122, 0x0161, 0x203a, 0x0153, 0, 0x017e, 0x0178, +]; + +/** + * An HTML attribute value with its character references decoded: the named + * ones a path can hold (`&` `<` `>` `"` `'`) and numeric + * ones (`'`, `'`) under HTML's replacements: 0, a surrogate or a + * value past U+10FFFF is U+FFFD, and 0x80-0x9F map through windows-1252. + * Anything else stays as written. + * + * @param {string} value + */ +function decodeHtmlReferences(value, { attribute = false } = {}) { + // HTML attributes also accept a numeric reference without its semicolon, and a + // legacy named one (amp, lt, gt, quot) unless a letter, digit or `=` follows; + // Markdown accepts neither. + const pattern = attribute + ? /&(?:#(\d+);?|#[xX]([0-9A-Fa-f]+);?|([A-Za-z]+);|(amp|lt|gt|quot)(?![A-Za-z0-9=]))/g + : /&(?:#(\d{1,7})|#[xX]([0-9A-Fa-f]{1,6})|([A-Za-z]+));/g; + return value.replace(pattern, (match, dec, hex, named, legacy) => { + const name = named ?? legacy; + if (dec !== undefined || hex !== undefined) { + const code = dec !== undefined ? Number(dec) : Number.parseInt(hex, 16); + const invalid = code === 0 || code > 0x10ffff || (code >= 0xd800 && code <= 0xdfff); + return String.fromCodePoint(invalid ? 0xfffd : WINDOWS_1252[code - 0x80] || code); + } + return HTML_NAMED_REFERENCES[name] ?? match; + }); +} + +/** + * A code span's content split for path handling: CommonMark strips one space + * on each side when both are there (`` ` x ` ``), and a path may sit in + * quotes. `core` is null when the content cannot be one path. `spaced` marks + * a candidate holding spaces (`My Slides.pdf`, or a command such as + * `node run.mjs x.js --no-media`): it counts as a path only when the whole + * span resolves through the path map, and it is never link-checked. + * + * @param {string} content + * @returns {{ pad: string, quote: string, core: string|null, spaced: boolean }} + */ +function spanPath(content) { + const pad = /^ .*[^ ].* $/.test(content) ? " " : ""; + let core = pad ? content.slice(1, -1) : content; + const quoted = /^(["'])(.+)\1$/.exec(core); + const quote = quoted ? quoted[1] : ""; + if (quoted) core = quoted[2]; + const readsAsPath = + !core.includes("\n") && + (core === "SLICE" || (PATH_LIKE.test(core) && !/^\s|\s$|\s[/\\]|[/\\]\s|\s{2}/.test(core))); + return { pad, quote, core: readsAsPath ? core : null, spaced: /\s/.test(core) }; +} + +/** + * @typedef {Object} CodeSpan + * @property {number} start - offset of the opening backtick run + * @property {number} end - offset just past the closing run + * @property {string} ticks - the backtick run + * @property {string} content - what sits between the runs + */ + +/** + * Every inline code span in `text`, read as CommonMark does: an unescaped run + * of backticks closed by the next run of exactly the same length, possibly on + * a later line of the same paragraph (no blank line between). A run with no + * such closer is literal backticks. + * + * @param {string} text + * @returns {CodeSpan[]} + */ +export function codeSpans(text) { + /** @type {CodeSpan[]} */ + const spans = []; + let at = 0; + while (at < text.length) { + const open = text.indexOf("`", at); + if (open === -1) break; + let runEnd = open; + while (text[runEnd] === "`") runEnd += 1; + const ticks = text.slice(open, runEnd); + if (isEscaped(text, open)) { + at = open + 1; + continue; + } + // Three or more backticks opening a line (after any block-quote or list + // markers) are a fence, not a code span. Fences inside containers are not + // masked (a known limit), so what they hold stays visible to the check. + const lineStart = text.lastIndexOf("\n", open - 1) + 1; + const lead = text.slice(lineStart, open); + if (ticks.length >= 3 && /^[ \t]*(?:>[ \t]*)*(?:(?:[-*+]|\d+[.)])[ \t]+)?[ \t]*$/.test(lead)) { + at = runEnd; + continue; + } + let close = -1; + let scan = runEnd; + while (scan < text.length) { + const next = text.indexOf("`", scan); + if (next === -1) break; + if (BLOCK_BREAK.test(text.slice(runEnd, next))) break; + let nextEnd = next; + while (text[nextEnd] === "`") nextEnd += 1; + if (nextEnd - next === ticks.length) { + close = next; + break; + } + scan = nextEnd; + } + if (close === -1) { + at = runEnd; + continue; + } + spans.push({ + start: open, + end: close + ticks.length, + ticks, + content: text.slice(runEnd, close), + }); + at = close + ticks.length; + } + return spans; +} + +/** + * True when the character at `at` is backslash-escaped: an odd number of + * backslashes sits right before it. + * + * @param {string} text + * @param {number} at + */ +function isEscaped(text, at) { + let count = 0; + for (let back = at - 1; back >= 0 && text[back] === "\\"; back -= 1) count += 1; + return count % 2 === 1; +} + +/** + * Corpus-layout path for a slice-relative file, or null when the corpus keeps + * no copy of it. + * + * @param {string} sliceRel - posix path relative to the slice root + * @returns {string|null} + */ +export function corpusPathForSliceFile(sliceRel) { + const [lane, ...rest] = sliceRel.split("/"); + const name = rest.join("/"); + if (sliceRel === "RESEARCH.md") return "analysis/RESEARCH.md"; + if (lane === LANES.research || lane === LANES.recommendations) return `analysis/${sliceRel}`; + if (lane === LANES.source && SOURCE_FOLDERS.has(rest[0]) && rest.length > 1) { + return `${SOURCE_FOLDERS.get(rest[0])}/${rest.slice(1).join("/")}`; + } + if (lane === LANES.source && rest.length === 1) { + if (TRANSCRIPT_TEXT.test(name)) return `transcript/${name}`; + if (SOURCE_METADATA.has(name)) return `metadata/${name}`; + } + if (lane === LANES.keyFrames) { + if (rest[0] === "frames" && rest.length > 1) return `frames/key/${rest.slice(1).join("/")}`; + if (rest.length === 1 && KEY_FRAME_LOGS.has(name)) return `frames/${name}`; + } + return null; +} + +/** + * @param {string} dir + * @param {string} [prefix] + * @returns {string[]} posix paths of every file under dir, relative to it + */ +function listFiles(dir, prefix = "") { + if (!fs.existsSync(dir)) return []; + return fs.readdirSync(dir, { withFileTypes: true }).flatMap((entry) => { + const rel = prefix ? `${prefix}/${entry.name}` : entry.name; + return entry.isDirectory() ? listFiles(path.join(dir, entry.name), rel) : [rel]; + }); +} + +/** + * Directory mappings implied by the file mappings: a source directory maps to + * a target directory only when every mapped file under it agrees. + * + * @param {Map} fileMap + * @returns {Map} + */ +function deriveDirectoryMap(fileMap) { + /** @type {Map} */ + const candidates = new Map(); + for (const [from, to] of fileMap) { + const fromParts = from.split("/"); + const toParts = to.split("/"); + for (let depth = 1; depth < fromParts.length && depth < toParts.length; depth++) { + if (fromParts.at(-depth) !== toParts.at(-depth)) break; + const fromDir = fromParts.slice(0, -depth).join("/"); + const toDir = toParts.slice(0, -depth).join("/"); + const seen = candidates.get(fromDir); + candidates.set(fromDir, seen === undefined || seen === toDir ? toDir : null); + } + } + return new Map( + [...candidates].filter( + /** @returns {entry is [string, string]} */ (entry) => entry[1] !== null, + ), + ); +} + +/** + * @typedef {Object} PathMap + * @property {Map} files - slice-relative file → target-relative file + * @property {Map} dirs - slice-relative directory → target-relative directory + * @property {Map} tempFrames - temp frame file name → target-relative file + * @property {boolean} framesDropped - the layout keeps no extracted frames (`--no-media`) + * @property {string|null} sliceWorkPath - the slice's `.work//` path under its work root, or null when it sits under none + * @property {Set} sliceEntries - every slice-relative file and directory + * @property {string} sliceAbs - absolute slice directory + * @property {string} targetAbs - absolute final target directory + */ + +/** @typedef {{ target: string, isDir: boolean, guessed?: boolean } | { retired: string, note: string } | { outside: string }} Lookup */ + +/** Stand-ins for a backslash-escaped `#` or `?`, which are part of the path. */ +const LITERAL_HASH = ""; +const LITERAL_QUERY = ""; + +/** Stand-in for a backslash-escaped `&`, which starts no character reference. */ +const LITERAL_AMP = ""; + +/** + * A link or definition destination decoded as CommonMark does: backslash + * escapes of ASCII punctuation, then character references (`&`, + * `&`). An escaped `#` or `?` is literal, not a fragment or query, so it + * becomes a stand-in that splitReference keeps in the path. `escapedChars` + * lists the characters the original escaped, to re-escape on rewrite; + * `references` marks a destination that held a character reference, so a + * rewrite writes `&` as `&`. + * + * @param {string} raw + * @returns {{ dest: string, escapedChars: string, references: boolean }} + */ +function unescapeDestination(raw) { + let escapedChars = ""; + const unescaped = raw.replace(/\\([!-/:-@[-`{-~])/g, (_match, char) => { + if (!escapedChars.includes(char)) escapedChars += char; + if (char === "#") return LITERAL_HASH; + if (char === "?") return LITERAL_QUERY; + if (char === "&") return LITERAL_AMP; + return char; + }); + const decoded = decodeHtmlReferences(unescaped); + return { + dest: decoded.replaceAll(LITERAL_AMP, "&"), + escapedChars, + references: decoded !== unescaped, + }; +} + +/** Text a CommonMark destination would decode as a character reference. */ +const REFERENCE_START = /^&(?:#\d+|#[xX][0-9A-Fa-f]+|[A-Za-z]+);/; + +/** + * Write `newPath` as a Markdown destination that parses back to exactly + * `newPath`, in the original's form. A literal `#` or `?` arrives as its + * stand-in and is written backslash-escaped. Angle form escapes `<`, `>` and + * `\`, and writes a line ending as a numeric reference; bare form escapes `\`, + * a leading `<` and every unbalanced parenthesis, and switches to angle form + * for a space or a control character. Punctuation the original escaped stays + * escaped; an `&` is written `&` where the original used references, and + * escaped where it would otherwise start one. + * + * @param {string} newPath + * @param {{ angle: boolean, escapedChars: string, references: boolean }} original + * @returns {string} the destination as written, angle brackets included + */ +export function encodeDestination(newPath, { angle, escapedChars, references }) { + const chars = [...newPath]; + const asAngle = angle || chars.some((char) => /[\x00-\x20\x7f]/.test(char)); + /** @type {Set} */ + const unbalanced = new Set(); + if (!asAngle) { + const open = []; + chars.forEach((char, at) => { + if (char === "(") open.push(at); + else if (char === ")" && open.pop() === undefined) unbalanced.add(at); + }); + for (const at of open) unbalanced.add(at); + } + const written = chars.map((char, at) => { + if (char === LITERAL_HASH) return "\\#"; + if (char === LITERAL_QUERY) return "\\?"; + if (char === "&") { + if (references) return "&"; + const startsReference = REFERENCE_START.test(chars.slice(at).join("")); + return escapedChars.includes("&") || startsReference ? "\\&" : "&"; + } + if (asAngle && (char === "\n" || char === "\r")) return `&#${char.charCodeAt(0)};`; + const structural = + char === "\\" || + (asAngle ? char === "<" || char === ">" : unbalanced.has(at) || (at === 0 && char === "<")); + // A raw `#` or `?` is a fragment or query; only its stand-in was escaped. + const kept = escapedChars.includes(char) && char !== "#" && char !== "?"; + return structural || kept ? `\\${char}` : char; + }); + return asAngle ? `<${written.join("")}>` : written.join(""); +} + +/** @param {string} value */ +function literalText(value) { + return value.replaceAll(LITERAL_HASH, "#").replaceAll(LITERAL_QUERY, "?"); +} + +/** + * Split a reference into its path and the suffix to carry over unchanged + * (`#anchor`, `?query`, `:line`). With `fragments: false`, a raw `#` or `?` + * stays part of the path (a file named `notes#1.md`); with `lines: false`, so + * does a `:12` or `:12:5` ending (a file named `slide.pdf:12`). + * + * @param {string} ref + * @param {{ fragments?: boolean, lines?: boolean }} [options] + * @returns {{ refPath: string, suffix: string, encoded: boolean }} + */ +function splitReference(ref, { fragments = true, lines = true } = {}) { + const hash = fragments ? ref.search(/[#?]/) : -1; + let refPath = hash === -1 ? ref : ref.slice(0, hash); + let suffix = hash === -1 ? "" : ref.slice(hash); + const line = lines ? LINE_SUFFIX.exec(refPath) : null; + if (line) { + suffix = line[0] + suffix; + refPath = refPath.slice(0, line.index); + } + // Compare paths the way the filesystem sees them: Windows separators as + // `/`, then each segment percent-decoded (`%23` is `#`, `%3F` is `?`). A + // malformed escape, or one decoding to a separator, stays as written. A + // rewrite re-encodes each segment when the original was encoded. + if (!URL_SCHEME.test(refPath)) refPath = refPath.replace(/\\/g, "/"); + let encoded = false; + if (refPath.includes("%")) { + const decoded = refPath + .split("/") + .map((segment) => { + try { + const plain = decodeURIComponent(segment); + return /[/\\]/.test(plain) ? segment : plain; + } catch { + return segment; + } + }) + .join("/"); + encoded = decoded !== refPath; + refPath = decoded; + } + return { refPath: literalText(refPath), suffix: literalText(suffix), encoded }; +} + +/** + * A reference to the slice directory itself: the slice has no single new + * location, so name where its kept files went, as for any split directory. + * + * @param {PathMap} map + * @returns {Lookup} + */ +function sliceRootLookup(map) { + const splitInto = [ + ...new Set([...map.files.values()].map((to) => `${to.split("/")[0]}/`)), + ].sort(); + return { + retired: path.basename(map.sliceAbs), + note: splitInto.length > 0 ? `split across ${splitInto.join(", ")} in this copy` : NOT_RETAINED, + }; +} + +/** + * What a reference names: a copied path (its target), a slice file or + * directory the layout does not keep (`retired`), or null when it names + * neither. + * + * @param {string} refPath + * @param {string} sourceFile - slice-relative path of the file naming it + * @param {PathMap} map + * @returns {Lookup|null} + */ +function lookUpReference(refPath, sourceFile, map) { + const bare = refPath.replace(/\/+$/, ""); + if (!bare || URL_SCHEME.test(bare) || bare.startsWith("/")) { + return null; + } + const lookup = (/** @type {string|null} */ key) => { + if (key === null) return null; + const file = map.files.get(key); + if (file !== undefined) return { target: file, isDir: false }; + const dir = map.dirs.get(key); + if (dir !== undefined) return { target: dir, isDir: true }; + return null; + }; + /** @type {(string|null)[]} */ + let keys; + // `SLICE/` names the slice root explicitly, and so does the work-root + // form `.work///`, but only with this slice's own prefix: + // a path segment that merely equals the slice's name says nothing. The bare + // aliases (`SLICE`, `SLICE/`, `.work///`) name the root itself. + if (bare === "SLICE" || (map.sliceWorkPath !== null && bare === map.sliceWorkPath)) { + return sliceRootLookup(map); + } + if (bare.startsWith("SLICE/")) { + keys = [path.posix.normalize(bare.slice("SLICE/".length))]; + } else if (map.sliceWorkPath && bare.startsWith(`${map.sliceWorkPath}/`)) { + keys = [path.posix.normalize(bare.slice(map.sliceWorkPath.length + 1))]; + } else { + const fromFile = path.posix.normalize(path.posix.join(path.posix.dirname(sourceFile), bare)); + if (fromFile === ".." || fromFile.startsWith("../")) { + // Resolved from the naming file's real location: a path that comes back + // into the slice (`..//x` from the root) is the slice's; any other + // (a sibling slice, the repo) keeps pointing at the same place from the + // file's new, deeper location. + const absolute = path.resolve(map.sliceAbs, ...fromFile.split("/")); + const inside = path.relative(map.sliceAbs, absolute); + if (inside === "") return sliceRootLookup(map); + if (!isSameOrInside(absolute, map.sliceAbs)) return { outside: absolute }; + keys = [inside.split(path.sep).join("/")]; + } else { + keys = [fromFile, bare.startsWith(".") ? null : path.posix.normalize(bare)]; + } + } + if (keys[0] === ".") return sliceRootLookup(map); + for (const key of keys) { + const found = lookup(key); + if (found) return found; + } + const retired = keys.find((key) => key !== null && map.sliceEntries.has(key)); + if (retired) { + // A directory whose kept files went to several places (`source/`) has no + // single new location: name where they went instead. + const splitInto = [ + ...new Set( + [...map.files] + .filter(([from]) => from.startsWith(`${retired}/`)) + .map(([, to]) => `${to.split("/")[0]}/`), + ), + ].sort(); + return { + retired, + note: + splitInto.length > 0 ? `split across ${splitInto.join(", ")} in this copy` : NOT_RETAINED, + }; + } + // A placeholder (`/x`, `${ROOT}/x`, `{tmp}/x`, `research/*.md`) + // names no real file. It counts as one only now, after neither the path map + // nor the slice's own entries named it, so a real `report-{draft}.pdf` is + // still mapped or retired. + if (PLACEHOLDER.test(bare)) return null; + // A lane folder the layout knows but this slice never produced (prose such + // as "no deck in `source/decks/`") names nothing to point at. + // A path below such a folder (`source/decks/slides.pdf`) names nothing either. + const absentFolder = keys.find( + (key) => + key !== null && + [...KNOWN_LANE_FOLDERS].some( + (folder) => + (key === folder || key.startsWith(`${folder}/`)) && !map.sliceEntries.has(folder), + ), + ); + if (absentFolder) return { retired: absentFolder, note: "none in this slice" }; + // An unknown file in a copied directory moves with the directory, so the + // link check flags it at its new location. + for (const key of bare.includes("/") ? keys : []) { + const dir = key === null ? undefined : map.dirs.get(path.posix.dirname(key)); + if (dir !== undefined && key !== null) { + return { target: `${dir}/${path.posix.basename(key)}`, isDir: false, guessed: true }; + } + } + if (bare.includes("/")) return null; + const frame = map.tempFrames.get(bare); + if (frame) return { target: frame, isDir: false }; + // Without the temp frames (`--no-media`) a frame name names nothing kept. + return map.framesDropped && FRAME_NAME.test(bare) ? { retired: bare, note: NOT_RETAINED } : null; +} + +const NOT_RETAINED = "not retained in this copy"; + +/** + * How a code span is read, in order: the whole name, a raw `#` or `?` and a + * `:12` ending included (`notes#1.md`, `slide.pdf:12`), then without its line + * suffix, and only then with the rest read as a fragment. A code span names a + * file literally. + */ +const WHOLE_NAME_FIRST = [{ fragments: false, lines: false }, { fragments: false }, {}]; + +/** + * How a link destination (Markdown or HTML) is read: a raw `#` or `?` always + * starts its fragment or query (an escaped `\#` or `%23` is a name + * character), and the name is tried with its `:12` ending, then without it. + */ +const LINK_READINGS = [{ lines: false }, {}]; + +/** + * What a reference names under the first of `readings` that names anything + * (an earlier reading must name it exactly, not as a guess into a copied + * directory), and that reading's split options (the last when none does). + * + * @param {string} ref + * @param {string} sourceFile + * @param {PathMap} map + * @param {{ fragments?: boolean, lines?: boolean }[]} [readings] + * @returns {{ found: Lookup|null, split: { fragments?: boolean, lines?: boolean } }} + */ +function findReference(ref, sourceFile, map, readings = LINK_READINGS) { + for (const split of readings) { + const found = lookUpReference(splitReference(ref, split).refPath, sourceFile, map); + if (found && !("guessed" in found && split !== readings.at(-1))) { + return { found, split }; + } + } + return { found: null, split: {} }; +} + +/** + * @typedef {Object} InlineLink + * @property {string} dest - the destination, angle brackets removed and backslash escapes decoded + * (an escaped `#` or `?` as a literal stand-in, see unescapeDestination) + * @property {number} destStart - offset of the destination as written (its `<` when angled) + * @property {number} destEnd - offset just past it + * @property {boolean} angle - written as `<...>` + * @property {string} escapedChars - the punctuation the destination backslash-escaped + * @property {boolean} references - the destination held a character reference (`&`) + * @property {string} label - the link text, `!` and brackets removed + * @property {number} labelStart - offset of the link's `[` (or its `!`), -1 when unmatched + * @property {number} labelEnd - offset of the `]` that closes the link text + * @property {number} linkStart - offset where the whole link starts + * @property {number} linkEnd - offset just past the link's closing `)` + */ + +/** + * The offset past spaces and tabs at `at`, including at most one line ending + * (CommonMark allows one between a link's destination and its title). + * + * @param {string} text + * @param {number} at + */ +function skipLinkSpace(text, at) { + let next = at; + while (text[next] === " " || text[next] === "\t") next += 1; + if (text[next] === "\r" && text[next + 1] === "\n") next += 2; + else if (text[next] === "\n" || text[next] === "\r") next += 1; + while (text[next] === " " || text[next] === "\t") next += 1; + return next; +} + +/** + * The link destination starting at `at`, read as CommonMark does for both + * inline links and reference definitions: `<...>` up to the first unescaped + * `>` (an unescaped `<` or a line ending makes it no destination), or a bare + * run with balanced parentheses up to whitespace or an unbalanced `)`, where + * a backslash escapes any ASCII punctuation. `raw` is the destination as + * written, without the angle brackets. + * + * @param {string} text + * @param {number} at + * @returns {{ raw: string, angle: boolean, end: number }|null} + */ +export function parseDestination(text, at) { + if (text[at] === "<") { + for (let scan = at + 1; scan < text.length; scan += 1) { + const char = text[scan]; + if (char === "\\" && ESCAPABLE.test(text[scan + 1] ?? "")) { + scan += 1; + continue; + } + if (char === "\n" || char === "\r" || char === "<") return null; + if (char === ">") return { raw: text.slice(at + 1, scan), angle: true, end: scan + 1 }; + } + return null; + } + let end = at; + let depth = 0; + while (end < text.length) { + const char = text[end]; + if (char === "\\" && ESCAPABLE.test(text[end + 1] ?? "")) { + end += 2; + continue; + } + // CommonMark ends a bare destination only at the ASCII space or an ASCII control. + if (/[\x00-\x20\x7f]/.test(char)) break; + if (char === "(") depth += 1; + else if (char === ")") { + if (depth === 0) break; + depth -= 1; + } + end += 1; + } + if (end === at || depth !== 0) return null; + return { raw: text.slice(at, end), angle: false, end }; +} + +/** + * Every inline link in `text`, scanned the way CommonMark reads a + * destination: `<...>`, or a run with balanced or backslash-escaped + * parentheses up to whitespace or the unbalanced `)`. A regex stops at the + * first `)`, which breaks `slides_(final).pdf`. + * + * @param {string} text + * @returns {InlineLink[]} + */ +export function inlineLinks(text) { + /** @type {InlineLink[]} */ + const links = []; + let from = 0; + for (;;) { + const open = text.indexOf("](", from); + // Links cannot contain links: one whose label holds another (non-image) + // link is literal text. An image in the label leaves it a link. + if (open === -1) { + const isLink = (/** @type {InlineLink} */ link) => text[link.linkStart] !== "!"; + return links.filter( + (outer) => + !isLink(outer) || + !links.some( + (inner) => + isLink(inner) && inner.linkStart > outer.linkStart && inner.linkEnd <= outer.labelEnd, + ), + ); + } + from = open + 2; + let at = open + 2; + at = skipLinkSpace(text, at); + const destStart = at; + const parsed = parseDestination(text, at); + if (parsed === null) continue; + const { raw, angle } = parsed; + at = parsed.end; + const destEnd = at; + // An optional title, then the closing `)`. + // Spaces with at most one line ending may separate the parts (CommonMark). + at = skipLinkSpace(text, at); + // A title is "...", '...' or (...), set off from the destination by space. + const quote = text[at]; + if (at > destEnd && (quote === '"' || quote === "'" || quote === "(")) { + // The closing delimiter, skipping backslash-escaped characters. + const closer = quote === "(" ? ")" : quote; + let closeQuote = -1; + for (let scan = at + 1; scan < text.length; scan += 1) { + if (text[scan] === "\\" && ESCAPABLE.test(text[scan + 1] ?? "")) { + scan += 1; + continue; + } + if (text[scan] === closer) { + closeQuote = scan; + break; + } + // A (...) title holds no unescaped `(`; with one, this is no link. + if (quote === "(" && text[scan] === "(") break; + // A title never spans a blank line (one holding only spaces or tabs). + if (text[scan] === "\n" && /^[ \t]*\r?\n/.test(text.slice(scan + 1))) break; + } + if (closeQuote === -1) continue; + at = closeQuote + 1; + at = skipLinkSpace(text, at); + } + if (text[at] !== ")") continue; + const linkEnd = at + 1; + // The `[` that opens this link's text, matching nested brackets. An + // escaped `]`, or no unescaped opening `[` (`\[watch](...)`), means this + // is not a link. + if (isEscaped(text, open)) continue; + let labelStart = -1; + let depth = 0; + for (let back = open - 1; back >= 0; back -= 1) { + // Link text never crosses a blank line (one holding only spaces counts). + if ( + text[back] === "\n" && + /^[ \t\r]*$/.test(text.slice(text.lastIndexOf("\n", back - 1) + 1, back)) + ) { + break; + } + // An escaped bracket is literal text and takes no part in nesting. + if ((text[back] === "[" || text[back] === "]") && isEscaped(text, back)) continue; + if (text[back] === "]") depth += 1; + else if (text[back] === "[") { + if (depth === 0) { + labelStart = back; + break; + } + depth -= 1; + } + } + if (labelStart === -1) continue; + const label = labelStart === -1 ? "" : text.slice(labelStart + 1, open); + // A `!` right before the label marks an image, unless it is escaped (`\![x](dest)`). + const image = + labelStart > 0 && text[labelStart - 1] === "!" && !isEscaped(text, labelStart - 1); + const linkStart = image ? labelStart - 1 : labelStart; + links.push({ + ...unescapeDestination(raw), + destStart, + destEnd, + angle, + label, + labelStart: linkStart, + labelEnd: open, + linkStart, + linkEnd, + }); + from = linkEnd; + } +} + +/** + * Rewrite every markdown link and backticked path in `text` that names a + * slice path: a copied one relative to the file's target location, one the + * layout does not keep as plain text naming it as not retained. + * + * @param {string} text + * @param {string} sourceFile - slice-relative path of the file + * @param {string} targetFile - target-relative path the file is copied to + * @param {PathMap} map + * @returns {{ text: string, rewrites: number, spacedRefs: string[], retiredRefs: string[] }} - + * `spacedRefs`, the rewritten code spans holding spaces, for the link check to verify; + * `retiredRefs`, the references left unchanged because they name a slice path the + * layout drops, which the link check must flag whatever their name looks like + */ +export function rewriteSliceReferences(text, sourceFile, targetFile, map) { + let rewrites = 0; + /** @type {string[]} */ + const spacedRefs = []; + /** @type {string[]} */ + const retiredRefs = []; + const rewrite = ( + /** @type {string} */ ref, + /** @type {{ fragments?: boolean, lines?: boolean }} */ split = {}, + escapedChars = "", + ) => { + const { refPath, suffix, encoded } = splitReference(ref, split); + const found = lookUpReference(refPath, sourceFile, map); + if (!found || "retired" in found) return ref; + let relative; + let trailing = ""; + if ("outside" in found) { + const fileDir = path.dirname(path.join(map.targetAbs, targetFile)); + relative = path.relative(fileDir, found.outside).split(path.sep).join("/"); + trailing = refPath.endsWith("/") ? "/" : ""; + } else { + relative = path.posix.relative(path.posix.dirname(targetFile), found.target) || "."; + trailing = found.isDir && refPath.endsWith("/") ? "/" : ""; + } + let written = encoded ? relative.split("/").map(encodeURIComponent).join("/") : relative; + // A `#` or `?` the original escaped stays a literal path character: + // its stand-in, which encodeDestination writes escaped. + if (escapedChars.includes("#")) written = written.replaceAll("#", LITERAL_HASH); + if (escapedChars.includes("?")) written = written.replaceAll("?", LITERAL_QUERY); + const next = `${written}${trailing}${suffix}`; + if (next === ref) return ref; + rewrites += 1; + return next; + }; + const { fenceMasked, linkMasked, tagText } = maskCode(text); + // Every pass records edits by offset into the original text; the scans run + // over masked copies, so nothing inside a fenced code block, an HTML tag or + // an HTML comment is touched and no link syntax inside an inline code span + // is read as a link. Inline code spans themselves are read for paths (the + // backticked-path rewrite); tags only for their src/href values. + /** @type {{ from: number, to: number, text: string }[]} */ + const edits = []; + // Inline links: a link naming a file the layout drops keeps its label in + // place and loses only its syntax: the opening `[` (or `![`) goes and + // `](dest)` becomes the note, so an image nested in the label keeps its + // own rewrite. Any other destination takes the path rewrite. + for (const link of inlineLinks(linkMasked)) { + const { found, split } = findReference(link.dest, sourceFile, map); + if (found && "retired" in found && link.labelStart !== -1) { + rewrites += 1; + const bracket = text[link.linkStart] === "!" ? link.linkStart + 2 : link.linkStart + 1; + edits.push({ from: link.linkStart, to: bracket, text: "" }); + edits.push({ + from: link.labelEnd, + to: link.linkEnd, + text: ` (${found.retired}, ${found.note})`, + }); + continue; + } + if (found && "retired" in found) retiredRefs.push(link.dest); + const next = rewrite(link.dest, split, link.escapedChars); + if (next === link.dest) continue; + edits.push({ from: link.destStart, to: link.destEnd, text: encodeDestination(next, link) }); + } + // Reference-style definitions and HTML src/href values take the same path + // rewrite. One naming a file the layout drops is left for the link check to + // flag: a definition or attribute has no prose to rewrite into. + for (const { from, to, angle } of referenceDefinitions(linkMasked)) { + const raw = text.slice(angle ? from + 1 : from, angle ? to - 1 : to); + const original = { ...unescapeDestination(raw), angle }; + const { dest } = original; + const { found, split } = findReference(dest, sourceFile, map); + if (found && "retired" in found) retiredRefs.push(dest); + const rewritten = rewrite(dest, split, original.escapedChars); + if (rewritten === dest) continue; + edits.push({ from, to, text: encodeDestination(rewritten, original) }); + } + for (const { from, to, group } of htmlAttributeValues(tagText)) { + // An attribute value may hold character references (`&`): map the + // decoded path, then escape the new value for its quoting: `&` always, + // the enclosing quote character, and an unquoted value that now needs + // quotes gets double quotes. + const value = text.slice(from, to); + const decoded = htmlAttributeUrl(value); + const { found, split } = findReference(decoded, sourceFile, map); + if (found && "retired" in found) retiredRefs.push(decoded); + const next = rewrite(decoded, split); + if (next === decoded) continue; + let escaped = next.replaceAll("&", "&"); + if (group === 2) escaped = escaped.replaceAll('"', """); + else if (group === 3) escaped = escaped.replaceAll("'", "'"); + else if (/[\s"'=<>`]/.test(escaped)) escaped = `"${escaped.replaceAll('"', """)}"`; + edits.push({ from, to, text: escaped }); + } + for (const span of codeSpans(fenceMasked)) { + const from = span.start; + const whole = text.slice(span.start, span.end); + const { ticks, content } = span; + const { pad, quote, core, spaced } = spanPath(content); + if (core === null) continue; + const { found, split } = findReference(core, sourceFile, map, WHOLE_NAME_FIRST); + // A span with spaces is a path only when it resolves as one whole path. + if (spaced && !found) continue; + let next; + if (found && "retired" in found) { + rewrites += 1; + next = `${found.retired} (${found.note})`; + } else { + const rewritten = rewrite(core, split); + next = `${ticks}${pad}${quote}${rewritten}${quote}${pad}${ticks}`; + // The link check skips spans with spaces (they may be commands), so a + // spaced path this rewrite placed is handed to it explicitly. + if (spaced) spacedRefs.push(rewritten); + } + if (next !== whole) edits.push({ from, to: from + whole.length, text: next }); + } + edits.sort((a, b) => a.from - b.from); + /** @type {typeof edits} */ + const kept = []; + for (const edit of edits) { + if (kept.length === 0 || edit.from >= kept[kept.length - 1].to) kept.push(edit); + } + let rewritten = text; + for (const edit of kept.reverse()) { + rewritten = rewritten.slice(0, edit.from) + edit.text + rewritten.slice(edit.to); + } + return { text: rewritten, rewrites, spacedRefs, retiredRefs }; +} + +/** + * Copies of `text` with code and HTML masked by spaces (line endings kept, so + * every offset still matches the original). All three blank top-level fenced + * code blocks (``` or ~~~ opening at column 0; an unclosed fence runs to the + * end). `fenceMasked` also blanks HTML start tags and comments, whose text is + * data; `linkMasked` blanks those and inline code spans; `tagText` blanks + * inline code spans and HTML comments, keeping start tags for the src/href + * pass. Fences that are indented or inside a block quote or list item, and + * indented code blocks, are not masked: telling them apart needs container + * parsing. What they hold is read as ordinary text, so the link check still + * sees a dangling path. + * + * @param {string} text + * @returns {{ fenceMasked: string, linkMasked: string, tagText: string }} + */ +export function maskCode(text) { + const blank = (/** @type {string} */ source, /** @type {[number, number][]} */ ranges) => { + let out = source; + for (const [from, to] of ranges) { + out = out.slice(0, from) + out.slice(from, to).replace(/[^\n]/g, " ") + out.slice(to); + } + return out; + }; + /** @type {[number, number][]} */ + const fences = []; + let offset = 0; + /** @type {{ char: string, length: number, start: number }|null} */ + let open = null; + for (const line of text.split(/(?<=\n)/)) { + const body = line.replace(/\r?\n$/, ""); + if (open === null) { + const opening = /^(`{3,}|~{3,})(.*)$/.exec(body); + if (opening && !(opening[1][0] === "`" && opening[2].includes("`"))) { + open = { char: opening[1][0], length: opening[1].length, start: offset }; + } + } else { + const closing = /^ {0,3}(`{3,}|~{3,})[ \t]*$/.exec(body); + if (closing && closing[1][0] === open.char && closing[1].length >= open.length) { + fences.push([open.start, offset + line.length]); + open = null; + } + } + offset += line.length; + } + if (open !== null) fences.push([open.start, text.length]); + const fenceMasked = blank(text, fences); + /** @type {[number, number][]} */ + const spans = codeSpans(fenceMasked).map((span) => [span.start, span.end]); + const codeMasked = blank(fenceMasked, spans); + const html = htmlRanges(codeMasked); + return { + fenceMasked: blank(fenceMasked, html), + linkMasked: blank(codeMasked, html), + tagText: codeMasked + .replace(HTML_COMMENT, (comment) => comment.replace(/[^\n]/g, " ")) + .replace(HTML_OTHER, (raw) => raw.replace(/[^\n]/g, " ")), + }; +} + +/** + * True when a backticked span reads as a path into the slice: `SLICE/`, an + * explicit `./` or `../` path, or one whose first segment is a top-level + * entry of the re-laid-out slice or an entry beside the naming file. Any + * other path is one the path map did not know under no slice directory, a + * path into another repository. A bare name reads as a slice path only when + * it is an extracted frame's name, which the rewrite turns into a path or + * not-retained text. + * + * @param {string} refPath + * @param {string} fileDir - absolute directory of the naming file + * @param {Set} topLevel - top-level entries of the re-laid-out slice + * @param {Set} [sliceDirs] - posix directories of the source slice + */ +function looksSliceInternal(refPath, fileDir, topLevel, sliceDirs) { + if (!refPath.includes("/")) return FRAME_NAME.test(refPath); + if (/^(?:\.\.?|SLICE)\//.test(refPath)) return true; + const first = refPath.split("/")[0]; + // A path under an original slice lane name (`source/missing.txt`) is the + // slice's when its parent directory existed in the slice, so a path into + // another repository that shares a lane name (`research/deepening/x.md`) + // stays external even beside `analysis/research/`. Without the slice's + // directories, any lane name counts. + if (SLICE_LANE_NAMES.has(first)) { + return ( + sliceDirs === undefined || sliceDirs.has(path.posix.dirname(refPath.replace(/\/+$/, ""))) + ); + } + return topLevel.has(first) || fs.existsSync(path.join(fileDir, first)); +} + +/** + * Link check over a re-laid-out slice: every markdown link and every + * backticked slice path in its markdown must resolve against the naming + * file's directory. URLs, placeholders, absolute paths, paths that leave the + * slice (another repository) and the allowed-miss prefixes are skipped. + * + * @param {string} rootDir + * @param {{ + * sliceDirs?: Set, + * spacedRefs?: Map, + * retiredRefs?: Map, + * }} [options] + * `sliceDirs`, the source slice's directories, narrows which lane-named paths + * count as the slice's; + * `spacedRefs` maps a target-relative file to the spaced code-span paths a + * rewrite placed in it, which are checked like links; + * `retiredRefs` maps a target-relative file to the references a rewrite left + * naming a dropped slice file, always flagged + * @returns {{ file: string, ref: string }[]} unresolved references + */ +export function checkLayoutLinks( + rootDir, + { sliceDirs, spacedRefs = new Map(), retiredRefs = new Map() } = {}, +) { + const root = path.resolve(rootDir); + const topLevel = new Set(fs.existsSync(root) ? fs.readdirSync(root) : []); + /** @type {{ file: string, ref: string }[]} */ + const unresolved = []; + const markdown = listFiles(root).filter( + (file) => + MARKDOWN_FILE.test(file) && !file.startsWith("frames/all/") && !file.startsWith("media/"), + ); + for (const file of markdown) { + const fileDir = path.dirname(path.join(root, file)); + const text = fs.readFileSync(path.join(root, file), "utf8"); + const retired = new Set(retiredRefs.get(file)); + const check = ( + /** @type {string} */ ref, + /** @type {boolean} */ isLink, + /** @type {boolean} */ knownPath = isLink, + ) => { + // A reference the rewrite found naming a dropped slice file dangles, whatever + // its text looks like (an allowed miss, a placeholder). + if (retired.has(ref)) { + unresolved.push({ file, ref: literalText(ref) }); + return; + } + // A code span may hold a raw `#` or `?` in its name, and any reference a + // `:12` ending: read as WHOLE_NAME_FIRST or LINK_READINGS orders it, the + // first reading that resolves passes; the last one is checked below. + for (const option of (isLink ? LINK_READINGS : WHOLE_NAME_FIRST).slice(0, -1)) { + const whole = splitReference(ref, option).refPath; + if (whole && fs.existsSync(path.resolve(fileDir, whole))) return; + } + const { refPath } = splitReference(ref); + if (!refPath || URL_SCHEME.test(refPath)) return; + if (refPath.startsWith("/") || HOME_PATH.test(refPath)) return; + if (isAllowedMiss(refPath)) return; + if (!knownPath && !looksSliceInternal(refPath, fileDir, topLevel, sliceDirs)) return; + const resolved = path.resolve(fileDir, refPath); + if (!isSameOrInside(resolved, root)) return; + if (fs.existsSync(resolved)) return; + // A placeholder that names no real file is not a dangling path. + if (PLACEHOLDER.test(refPath)) return; + unresolved.push({ file, ref: literalText(ref) }); + }; + const { fenceMasked, linkMasked, tagText } = maskCode(text); + for (const link of inlineLinks(linkMasked)) check(link.dest, true); + // A definition is unambiguous link syntax, so every relative destination is + // checked, an extensionless one (`[guide]: README`) included. + for (const { raw } of referenceDefinitions(linkMasked)) { + check(unescapeDestination(raw).dest, true); + } + for (const { from, to } of htmlAttributeValues(tagText)) { + check(htmlAttributeUrl(text.slice(from, to)), true); + } + for (const span of codeSpans(fenceMasked)) { + // A span with spaces may be a command (`node run.mjs x.js --flag`): only + // single-token spans are link-checked. + const { core, spaced } = spanPath(span.content); + if (core !== null && !spaced) check(core, false); + } + // Spaced spans a rewrite placed (see rewriteSliceReferences) are known to + // be paths: checked by the backticked-path rules, whole span first. + for (const ref of spacedRefs.get(file) ?? []) check(ref, false, true); + } + return unresolved; +} + +/** + * @param {Record} info + * @returns {Record} + */ +export function trimInfoJson(info) { + const trimmed = Object.fromEntries( + Object.entries(info).filter(([key]) => !INFO_JSON_DROPPED_KEYS.includes(key)), + ); + if (Array.isArray(trimmed.comments)) { + trimmed.comments = trimmed.comments.slice(0, INFO_JSON_COMMENT_CAP); + } + return trimmed; +} + +const CONTENTS_LINES = /** @type {const} */ ([ + ["media/", "source video as downloaded"], + ["transcript/", "timestamped transcript, plus the caption tracks as served when kept"], + ["metadata/", "source metadata: trimmed info JSON, harvested links, repo analysis, decks"], + ["frames/all/", "every frame the pipeline extracted, with their times in `frame-times.json`"], + ["frames/key/", "the promoted key frames; the logs beside them in `frames/` describe them"], + ["analysis/", "the digest's research and recommendations"], +]); + +/** + * Plain text for a Markdown line: one line, with every Markdown-significant character + * backslash-escaped, so an upstream title never renders (or is link-checked) as syntax. + * @param {string} text + */ +export function markdownText(text) { + return text.replace(/\s+/g, " ").trim().replace(/[\\`*_{}[\]()<>#+\-.!|~&]/g, "\\$&"); +} + +/** + * @param {{ title: string, sourceUrl: string, videoId: string, acquiredAt?: string }} meta + * @param {Set} presentTopDirs - target directories that received files + */ +function provenanceReadme({ title, sourceUrl, videoId, acquiredAt }, presentTopDirs) { + const fetched = acquiredAt ? ` on ${acquiredAt.slice(0, 10)}` : ""; + const contents = CONTENTS_LINES.filter(([dir]) => presentTopDirs.has(dir)).map( + ([dir, label]) => `- \`${dir}\`: ${label}`, + ); + return [ + `# ${markdownText(title)}`, + "", + "## Provenance", + "", + `- Canonical origin: <${sourceUrl.replace(/[\s<>]/g, encodeURIComponent)}>`, + `- Video id: \`${videoId}\``, + `- Fetched${fetched} through \`/knowledge:video-digest\` and re-laid out from its finished slice.`, + "", + "## Contents", + "", + ...contents, + "", + ].join("\n"); +} + +/** + * The JSON object in `file`, or null when it cannot be read, does not parse, + * or parses to anything but a plain object (null, an array, a scalar). + * + * @param {string} file + * @returns {Record|null} + */ +function readJsonObject(file) { + /** @type {unknown} */ + let parsed; + try { + parsed = JSON.parse(fs.readFileSync(file, "utf8")); + } catch { + return null; + } + return parsed !== null && typeof parsed === "object" && !Array.isArray(parsed) + ? /** @type {Record} */ (parsed) + : null; +} + +/** + * The slice's path under its work root, from the last `.work` segment of its + * absolute path (`.work//`), or null when no `.work` segment holds it. + * + * @param {string} slice - absolute slice directory + * @returns {string|null} + */ +function workRootRelative(slice) { + const segments = slice.split(path.sep); + const at = segments.lastIndexOf(".work"); + return at === -1 || at === segments.length - 1 ? null : segments.slice(at).join("/"); +} + +/** @param {unknown} error */ +function errorMessage(error) { + return error instanceof Error ? error.message : String(error); +} + +/** A video id as yt-dlp's `%(id)s` writes it: no separator, no `..`. */ +const VIDEO_ID = /^[A-Za-z0-9_-]+$/; + +/** + * What is wrong with a parsed watch.json's shape for this script, or null. + * `videoId` names `media/.`, so it must not hold a path; the title + * and source URL fill the provenance README; the temp session dirs are paths. + * + * @param {unknown} state + * @returns {string|null} + */ +function watchStateShapeProblem(state) { + if (!state || typeof state !== "object" || Array.isArray(state)) return "is not a JSON object"; + const record = /** @type {Record} */ (state); + if (typeof record.videoId !== "string" || !VIDEO_ID.test(record.videoId)) { + return `videoId ${JSON.stringify(record.videoId)} is not a plain video id`; + } + for (const field of ["title", "sourceUrl"]) { + if (typeof record[field] !== "string" || record[field] === "") { + return `has no ${field} string`; + } + } + const temp = record.tempSession; + if (temp === undefined || temp === null) return null; + if (typeof temp !== "object" || Array.isArray(temp)) return "tempSession is not an object"; + const session = /** @type {Record} */ (temp); + for (const field of ["workDir", "framesDir", "contactSheetsDir", "acquiredAt"]) { + if (session[field] !== undefined && typeof session[field] !== "string") { + return `tempSession.${field} is not a string`; + } + } + return null; +} + +/** + * What the frames dir lacks for a complete `frames/all/`: `frame-times.json` + * (an object keyed by frame file), at least one frame image, and every frame + * the sidecar lists. Empty when nothing is missing. + * + * @param {string|undefined} framesDir + * @returns {string[]} + */ +function missingFrames(framesDir) { + if (!framesDir) return ["the frames (no tempSession.framesDir)"]; + if (!readableDirectory(framesDir)) { + return [`the frames (${framesDir} is not a readable directory)`]; + } + const names = new Set(fs.readdirSync(framesDir)); + const missing = []; + // A frame image or the sidecar must be a regular file, not just a name. + const notFiles = [...names].filter( + (name) => + (FRAME_IMAGE.test(name) || name === "frame-times.json") && + !isRegularFile(path.join(framesDir, name)), + ); + if (notFiles.length > 0) { + missing.push(`regular files for the frames entries ${notFiles.sort().join(", ")}`); + } + /** @type {unknown} */ + let times = null; + let readable = true; + try { + times = JSON.parse(fs.readFileSync(path.join(framesDir, "frame-times.json"), "utf8")); + } catch { + readable = false; + missing.push(`a readable frame-times.json (${framesDir})`); + } + // Scene detection writes an object keyed by frame file; null, an array or + // a scalar is not one, and `Object.keys` would read it as an empty listing. + const isTimesObject = times !== null && typeof times === "object" && !Array.isArray(times); + if (readable && !isTimesObject) { + missing.push(`a frame-times.json holding a JSON object (${framesDir})`); + } + const emptyImages = [...names].filter( + (name) => FRAME_IMAGE.test(name) && sizeOf(path.join(framesDir, name)) === 0, + ); + if (emptyImages.length > 0) + missing.push(`non-empty frames (empty: ${emptyImages.sort().join(", ")})`); + if (![...names].some((name) => FRAME_IMAGE.test(name))) { + missing.push(`any frame image (${framesDir})`); + } + if (isTimesObject) { + const listing = /** @type {Record} */ (times); + const listed = Object.keys(listing); + // Scene detection's writeFrameTimes writes each record as + // { timestampSec, timestampSource, timestampMethod, timestampErrorSec }, + // with timestampSec a number, or null for an untimed frame. + const malformed = listed.filter((name) => { + const record = listing[name]; + if (!record || typeof record !== "object" || Array.isArray(record)) return true; + const seconds = /** @type {Record} */ (record).timestampSec; + return !(seconds === null || (typeof seconds === "number" && Number.isFinite(seconds))); + }); + if (malformed.length > 0) { + missing.push( + `frame-times.json records with an object and a finite or null timestampSec ` + + `(malformed: ${malformed.sort().join(", ")})`, + ); + } + // Every key, whatever its extension, must name a regular file in the + // frames dir itself: not a directory, a symlink or a missing name. + const gone = listed.filter( + (name) => !names.has(name) || !isRegularFile(path.join(framesDir, name)), + ); + if (gone.length > 0) { + missing.push(`regular files for the frame-times.json keys: ${gone.sort().join(", ")}`); + } + // Scene detection records every scene and interval frame it writes; an + // anchor frame carries its time in its own name and is never listed. + const untimed = [...names].filter( + (name) => TIMED_FRAME.test(name) && !Object.hasOwn(listing, name), + ); + if (untimed.length > 0) { + missing.push(`frame-times.json entries for: ${untimed.sort().join(", ")}`); + } + } + return missing; +} + +/** + * True when `dir` is a directory this run can list. + * + * @param {string} dir + */ +function readableDirectory(dir) { + try { + if (!fs.statSync(dir).isDirectory()) return false; + fs.readdirSync(dir); + return true; + } catch { + return false; + } +} + +/** + * True when `file` is a regular file (not a directory, symlink or device). + * + * @param {string} file + */ +function isRegularFile(file) { + try { + return fs.lstatSync(file).isFile(); + } catch { + return false; + } +} + +/** + * True when `p` itself is a symlink (not followed); false when it is absent. + * + * @param {string} p + */ +function isSymlink(p) { + try { + return fs.lstatSync(p).isSymbolicLink(); + } catch { + return false; + } +} + +/** + * Size of `file` in bytes; 0 when it cannot be read, so an unreadable + * artifact is refused like an empty one. + * + * @param {string} file + */ +function sizeOf(file) { + if (!isRegularFile(file)) return 0; + try { + return fs.statSync(file).size; + } catch { + return 0; + } +} + +/** + * posix paths, relative to `dir`, of every symlink under it. + * + * @param {string} dir + * @param {string} [prefix] + * @returns {string[]} + */ +function findSymlinks(dir, prefix = "") { + let entries; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + // Missing or unreadable: the readability checks name it. + return []; + } + return entries.flatMap((entry) => { + const rel = prefix ? `${prefix}/${entry.name}` : entry.name; + if (entry.isSymbolicLink()) return [rel]; + return entry.isDirectory() ? findSymlinks(path.join(dir, entry.name), rel) : []; + }); +} + +/** + * Real path of `p`, resolving symlinks through its nearest existing ancestor + * when `p` itself does not exist yet. + * + * @param {string} p - absolute path + * @returns {string} + */ +function realPathOf(p) { + let existing = p; + const rest = []; + while (!fs.existsSync(existing) && path.dirname(existing) !== existing) { + rest.unshift(path.basename(existing)); + existing = path.dirname(existing); + } + return path.join(fs.realpathSync.native(existing), ...rest); +} + +/** + * @param {string} inner + * @param {string} outer + */ +function isSameOrInside(inner, outer) { + const rel = path.relative(outer, inner); + return rel === "" || (rel !== ".." && !rel.startsWith(`..${path.sep}`) && !path.isAbsolute(rel)); +} + +/** + * @typedef {Object} RelayoutResult + * @property {number} exitCode + * @property {number} copied - files written to the target + * @property {number} rewrites - references rewritten + * @property {{ file: string, ref: string }[]} unresolved + */ + +/** + * @param {{ + * sliceDir: string, + * targetDir: string, + * noMedia?: boolean, + * replace?: boolean, + * verifyOutcomes?: (sliceDir: string) => number, + * copyFile?: (from: string, to: string) => void, + * }} options - `noMedia` copies nothing from the temp session; `replace` + * allows an existing target, keeping its `README.md`; `verifyOutcomes` gates + * a slice that is not closed yet and returns 0 when its outcome checks pass + * @returns {Promise} + */ +export async function relayoutSlice({ + sliceDir, + targetDir, + noMedia = false, + replace = false, + verifyOutcomes = (dir) => runCheckWatchOutcomes(dir), + copyFile = fs.copyFileSync, +}) { + const slice = path.resolve(sliceDir); + const target = path.resolve(targetDir); + const refuse = (/** @type {string} */ message) => { + writeStderr(`relayout: ${message}`); + return { exitCode: 1, copied: 0, rewrites: 0, unresolved: [] }; + }; + + // Every path this run creates, renames or deletes: the target, its backup + // and its staging dirs (`..relayout-*` beside the target). One that + // is, holds, or sits inside the slice would copy the slice into itself or + // move it away and delete it. Real paths, so a symlink cannot hide overlap. + const backup = `${target}.relayout-backup`; + const realSlice = realPathOf(slice); + const stagingPrefix = `.${path.basename(target)}.relayout-`; + const overlapsSlice = (/** @type {string} */ p) => { + const real = realPathOf(p); + return isSameOrInside(real, realSlice) || isSameOrInside(realSlice, real); + }; + const sliceInStaging = path + .relative(realPathOf(path.dirname(target)), realSlice) + .split(path.sep)[0] + .startsWith(stagingPrefix); + for (const written of [target, backup]) { + if (overlapsSlice(written)) { + return refuse(`${written} overlaps the slice ${slice}; pick a target outside it`); + } + } + if (sliceInStaging) { + return refuse(`the slice ${slice} sits in a relayout staging dir of ${target}`); + } + // Read the whole slice tree once, up front: an unreadable directory in it + // is refused here, naming it, rather than failing mid-copy. + /** @type {string[]} */ + let sliceFiles; + try { + sliceFiles = listFiles(slice); + } catch (error) { + const where = /** @type {NodeJS.ErrnoException} */ (error).path ?? slice; + return refuse(`cannot read ${where} in the slice: ${errorMessage(error)}`); + } + const symlinks = findSymlinks(slice); + if (symlinks.length > 0) { + return refuse(`the slice holds symlinks, which a copy would follow: ${symlinks.join(", ")}`); + } + // The reference scanner reads LF and CRLF line endings only; refuse rather than misread. + const bareCr = sliceFiles.filter( + (file) => + MARKDOWN_FILE.test(file) && /\r(?!\n)/.test(fs.readFileSync(path.join(slice, file), "utf8")), + ); + if (bareCr.length > 0) { + return refuse(`Markdown with bare CR line endings is not supported: ${bareCr.join(", ")}`); + } + + // An interrupted `--replace` swap leaves the previous target at this name. + if (fs.existsSync(backup)) { + if (fs.existsSync(target)) { + return refuse( + `a leftover backup ${backup} sits beside ${target}; keep one of them and remove the other`, + ); + } + try { + fs.renameSync(backup, target); + } catch (error) { + return refuse(`could not restore ${target} from ${backup}: ${errorMessage(error)}`); + } + writeStderr(`relayout: restored ${target} from the backup an interrupted run left`); + } + + // A killed run leaves its staging dir behind. Name any for deletion by + // hand; never install or delete one: it may belong to another slice's run + // or to a run still in progress. + let siblings = []; + try { + siblings = fs.readdirSync(path.dirname(target)); + } catch { + // No parent yet: nothing left behind. + } + const leftovers = siblings.filter((name) => name.startsWith(stagingPrefix)); + if (leftovers.length > 0) { + writeStderr( + `relayout: warning: staging dirs from an earlier run sit beside ${target}; ` + + `delete them by hand when no run is using them: ${leftovers + .map((name) => path.join(path.dirname(target), name)) + .join(", ")}`, + ); + } + + let state; + try { + state = JSON.parse(fs.readFileSync(watchStatePath(slice), "utf8")); + } catch { + return refuse(`no readable run-state/watch.json under ${slice}`); + } + const shapeProblem = watchStateShapeProblem(state); + if (shapeProblem) return refuse(`run-state/watch.json ${shapeProblem}`); + const outcomesPass = () => { + try { + return verifyOutcomes(slice) === 0; + } catch (error) { + writeStderr(`relayout: outcome checks threw: ${/** @type {Error} */ (error).message}`); + return false; + } + }; + if (state.status !== "complete" && !outcomesPass()) { + return refuse( + `slice status is "${state.status}" and its outcome checks fail; ` + + "re-lay out a closed slice, or one whose check-watch-outcomes.js passes", + ); + } + + if (fs.existsSync(target) && !replace) { + return refuse(`${target} exists; pass --replace to replace it`); + } + if (fs.existsSync(target) && !fs.lstatSync(target).isDirectory()) { + return refuse(`${target} exists and is not a directory; --replace only replaces a directory`); + } + + // `--no-media` takes nothing from the temp session, even when it still exists. + // `close` deletes every temp dir watch.json records, so in every mode, + // `--no-media` included and whether or not the dir still exists, nothing + // this run writes may sit in, hold, or be one of them. + const recorded = resolveTempSession(state.tempSession ?? {}); + // A recorded temp dir that is itself a symlink points somewhere the watch + // never made: checked before any containment or readability check follows it. + for (const dir of [recorded.workDir, recorded.framesDir, recorded.contactSheetsDir]) { + if (dir && isSymlink(dir)) { + return refuse(`temp session dir ${dir} is a symlink`); + } + } + for (const dir of [recorded.workDir, recorded.framesDir, recorded.contactSheetsDir]) { + if (!dir) continue; + const realDir = realPathOf(dir); + const clashes = + [target, backup].some((p) => { + const real = realPathOf(p); + return isSameOrInside(real, realDir) || isSameOrInside(realDir, real); + }) || isSameOrInside(realPathOf(path.dirname(target)), realDir); + if (clashes) { + return refuse(`target ${target} overlaps the recorded temp session dir ${dir}`); + } + } + + // watch.json names the temp dirs, and the watch only ever makes them in the + // OS temp dir; anything else (a hand-edited path, the slice, the target) is + // not a temp session, and `close` would delete it. Checked in every mode, + // whether or not the dir still exists. + const realTmp = fs.realpathSync.native(os.tmpdir()); + for (const dir of [recorded.workDir, recorded.framesDir, recorded.contactSheetsDir]) { + if (!dir) continue; + if (!isSameOrInside(realPathOf(dir), realTmp) || realPathOf(dir) === realTmp) { + return refuse(`temp session dir ${dir} is not inside the OS temp dir`); + } + if (overlapsSlice(dir)) { + return refuse(`temp session dir ${dir} overlaps the slice`); + } + } + // `--no-media` reads nothing from the temp session. + const temp = noMedia ? {} : recorded; + const workFiles = temp.workDir ? await listWorkDirFiles(temp.workDir) : []; + // The watch does not record which work file is the primary entry's, so only + // a work dir holding exactly one video is unambiguous (an X post with + // several videos is not). That video's id, the `%(id)s` of yt-dlp's output + // template, picks its info JSON and caption tracks. + const videos = workFiles.filter((file) => VIDEO_FILE.test(file)); + if (!noMedia && videos.length > 1) { + return refuse( + `the temp session holds several videos (${videos.map((file) => path.basename(file)).join(", ")}) ` + + "and the watch did not record which is primary; pass --no-media", + ); + } + const mediaId = videos.length === 1 ? path.basename(videos[0], path.extname(videos[0])) : null; + const ofMedia = (/** @type {string} */ file) => + mediaId !== null && path.basename(file).startsWith(`${mediaId}.`); + const media = { + videoPath: videos[0] ?? "", + captionPaths: workFiles.filter((file) => ofMedia(file) && file.endsWith(".vtt")), + metadataPath: workFiles.find((file) => ofMedia(file) && file.endsWith(".info.json")) ?? "", + }; + // The one video must be this slice's: its yt-dlp id is watch.json's videoId + // (YouTube), or, for an X post, whose media id differs from the post id, its + // info JSON's display_id is the post id the watch recorded. + if (!noMedia && mediaId !== null && mediaId !== state.videoId) { + const displayId = media.metadataPath ? readJsonObject(media.metadataPath)?.display_id : null; + const postIds = [state.videoId, state.sourceMetadata?.["source:displayId"]].filter(Boolean); + if (!(typeof displayId === "string" && postIds.includes(displayId))) { + return refuse( + `the temp session's video is ${mediaId}, not this slice's ${state.videoId}` + + (typeof displayId === "string" ? ` (its info JSON names post ${displayId})` : ""), + ); + } + } + if (!noMedia) { + // A partly removed temp session would yield a layout silently missing a part. + // Captions are required unless the transcript phase records that no + // caption fed the transcript: run-watch.js writes `transcriptStrategy: + // "asr"`, or `skipped: true` when no entry produced one. A + // `transcriptDegradation` can be a caption's provenance note, so it is not + // that evidence. A recovered watch records only `recovered`, and recovery + // itself needs a caption track, so it requires them too. + const transcriptMetrics = state.phases?.transcript?.metrics ?? {}; + const captioned = !( + transcriptMetrics.transcriptStrategy === "asr" || transcriptMetrics.skipped === true + ); + const missing = [ + ...(temp.workDir && !readableDirectory(temp.workDir) + ? [`the work dir (${temp.workDir} is not a readable directory)`] + : media.videoPath + ? [] + : [`the video (${temp.workDir ?? "no tempSession.workDir"})`]), + ...(media.videoPath && sizeOf(media.videoPath) === 0 + ? [`a non-empty video (${path.basename(media.videoPath)} is empty)`] + : []), + ...(media.videoPath && !media.metadataPath ? [`the info JSON (${mediaId}.info.json)`] : []), + ...(media.metadataPath && !readJsonObject(media.metadataPath) + ? [`an info JSON holding a JSON object (${path.basename(media.metadataPath)})`] + : []), + ...missingFrames(temp.framesDir), + ...[temp.workDir, temp.framesDir] + .flatMap((dir) => (dir ? findSymlinks(dir).map((link) => path.join(dir, link)) : [])) + .map((link) => `regular files in place of symlinks (${link})`), + ...(captioned && media.captionPaths.length === 0 ? ["the caption tracks"] : []), + ...media.captionPaths + .filter((caption) => sizeOf(caption) === 0) + .map((caption) => `a non-empty caption track (${path.basename(caption)} is empty)`), + ...[media.videoPath, media.metadataPath, ...media.captionPaths] + .filter((file) => file && fs.existsSync(file) && !isRegularFile(file)) + .map((file) => `a regular file at ${path.basename(file)}`), + ]; + if (missing.length > 0) { + return refuse( + `the temp session is missing ${missing.join(", ")}; ` + + "pass --no-media to re-lay out without media/, frames/all/, the caption tracks and info.json", + ); + } + } + + /** @type {{ from: string, to: string, sourceFile?: string }[]} */ + const plan = []; + + const sliceEntries = new Set( + sliceFiles.flatMap((file) => + file.split("/").map((_part, index, parts) => parts.slice(0, index + 1).join("/")), + ), + ); + for (const file of sliceFiles) { + const to = corpusPathForSliceFile(file); + if (to) plan.push({ from: path.join(slice, file), to, sourceFile: file }); + } + for (const caption of media.captionPaths) { + const name = path.basename(caption).replace(`${mediaId}.`, ""); + plan.push({ from: caption, to: `transcript/${name}` }); + } + /** @type {Map} */ + const tempFrames = new Map(); + if (temp.framesDir) { + for (const name of listFiles(temp.framesDir).filter((file) => !file.includes("/"))) { + plan.push({ from: path.join(temp.framesDir, name), to: `frames/all/${name}` }); + tempFrames.set(name, `frames/all/${name}`); + } + } + if (media.videoPath) { + plan.push({ + from: media.videoPath, + to: `media/${state.videoId}${path.extname(media.videoPath)}`, + }); + } + + // The slice's own README (its journey) is not copied: the target's README is + // a provenance page with other headings, so a reference to the slice README + // becomes not-retained text rather than a link to a different page. + const files = new Map( + plan.flatMap((entry) => + entry.sourceFile ? [/** @type {[string, string]} */ ([entry.sourceFile, entry.to])] : [], + ), + ); + /** @type {PathMap} */ + const map = { + files, + dirs: deriveDirectoryMap(files), + tempFrames, + framesDropped: noMedia, + sliceWorkPath: workRootRelative(slice), + sliceEntries, + sliceAbs: slice, + targetAbs: target, + }; + + // Build the whole layout in a sibling staging directory and move it into + // place only after the link check passes, so a failure never leaves a + // partial target. + let staging; + try { + fs.mkdirSync(path.dirname(target), { recursive: true }); + staging = fs.mkdtempSync(path.join(path.dirname(target), stagingPrefix)); + } catch (error) { + return refuse(`cannot stage beside ${target}: ${errorMessage(error)}`); + } + const discard = (/** @type {string} */ dir) => fs.rmSync(dir, { recursive: true, force: true }); + let rewrites = 0; + /** @type {Map} */ + const spacedRefs = new Map(); + /** @type {Map} */ + const retiredRefs = new Map(); + let copied = 0; + /** @type {{ file: string, ref: string }[]} */ + let unresolved; + try { + const write = (/** @type {string} */ to, /** @type {string} */ body) => { + const dest = path.join(staging, to); + fs.mkdirSync(path.dirname(dest), { recursive: true }); + fs.writeFileSync(dest, body); + copied += 1; + }; + for (const entry of plan) { + if (entry.sourceFile && MARKDOWN_FILE.test(entry.sourceFile)) { + const result = rewriteSliceReferences( + fs.readFileSync(entry.from, "utf8"), + entry.sourceFile, + entry.to, + map, + ); + rewrites += result.rewrites; + if (result.spacedRefs.length > 0) spacedRefs.set(entry.to, result.spacedRefs); + if (result.retiredRefs.length > 0) retiredRefs.set(entry.to, result.retiredRefs); + write(entry.to, result.text); + } else { + fs.mkdirSync(path.dirname(path.join(staging, entry.to)), { recursive: true }); + copyFile(entry.from, path.join(staging, entry.to)); + copied += 1; + } + } + if (media.metadataPath) { + // Checked to be an object before staging; a change since then is a failure. + const info = readJsonObject(media.metadataPath); + if (!info) throw new Error(`${media.metadataPath} no longer holds a JSON object`); + write("metadata/info.json", `${JSON.stringify(trimInfoJson(info), null, 2)}\n`); + } + + const existingReadme = path.join(target, "README.md"); + if (fs.existsSync(existingReadme)) { + write("README.md", fs.readFileSync(existingReadme, "utf8")); + } else { + const present = new Set( + CONTENTS_LINES.map(([dir]) => dir).filter((dir) => fs.existsSync(path.join(staging, dir))), + ); + write( + "README.md", + provenanceReadme( + { + title: state.title, + sourceUrl: state.sourceUrl, + videoId: state.videoId, + acquiredAt: state.tempSession?.acquiredAt, + }, + present, + ), + ); + } + + unresolved = checkLayoutLinks(staging, { + sliceDirs: new Set([...sliceEntries].filter((entry) => !sliceFiles.includes(entry))), + spacedRefs, + retiredRefs, + }); + } catch (error) { + discard(staging); + return refuse(`copy failed, target left untouched: ${/** @type {Error} */ (error).message}`); + } + if (unresolved.length > 0) { + discard(staging); + return { exitCode: 1, copied: 0, rewrites, unresolved }; + } + + // With --replace the old target moves to a fixed-name backup, so a run + // killed between the two renames is recovered at the next start (see the + // backup check above). Without --replace the install is one rename, which + // fails (ENOTEMPTY or EEXIST on POSIX, EPERM on Windows) when a target with + // content appeared during the run. POSIX rename replaces an empty directory + // created meanwhile; that is accepted, since an empty directory holds no data. + const hadTarget = replace && fs.existsSync(target); + try { + if (hadTarget) fs.renameSync(target, backup); + fs.renameSync(staging, target); + } catch (error) { + try { + if (hadTarget && !fs.existsSync(target) && fs.existsSync(backup)) { + fs.renameSync(backup, target); + } + discard(staging); + } catch (cleanupError) { + return refuse( + `could not move the layout into place (${errorMessage(error)}), and cleanup failed ` + + `(${errorMessage(cleanupError)}); check ${target}, ${backup} and ${staging}`, + ); + } + if (!replace && fs.existsSync(target)) { + return refuse(`${target} appeared during the run; pass --replace to replace it`); + } + return refuse(`could not move the layout into place: ${errorMessage(error)}`); + } + try { + if (hadTarget) discard(backup); + } catch (error) { + writeStderr(`relayout: installed ${target}, but cleanup failed: ${errorMessage(error)}`); + } + return { exitCode: 0, copied, rewrites, unresolved }; +} + +/** + * @param {string[]} args - argv after the script path + * @returns {Promise} + */ +export async function runRelayoutCli(args) { + const noMedia = args.includes("--no-media"); + const replace = args.includes("--replace"); + const positional = args.filter((arg) => arg !== "--no-media" && arg !== "--replace"); + if (positional.length !== 2 || positional.some((arg) => arg.startsWith("--"))) { + writeStderr( + "Usage: node watch/relayout-slice.js [--no-media] [--replace]", + ); + return 2; + } + const [sliceDir, targetDir] = positional; + const result = await relayoutSlice({ sliceDir, targetDir, noMedia, replace }); + if (result.copied > 0) { + writeStdout( + `relayout: ${result.copied} files written to ${path.resolve(targetDir)}, ` + + `${result.rewrites} references rewritten`, + ); + } + for (const { file, ref } of result.unresolved) { + writeStderr(`unresolved: ${file} -> ${ref}`); + } + if (result.unresolved.length > 0) { + writeStderr(`relayout: ${result.unresolved.length} slice-internal paths do not resolve`); + } + return result.exitCode; +} + +if (isMainModule(import.meta.url)) { + process.exit(await runRelayoutCli(process.argv.slice(2))); +} diff --git a/plugins/knowledge/skills/video-digest/extraction/watch/relayout-slice.test.js b/plugins/knowledge/skills/video-digest/extraction/watch/relayout-slice.test.js new file mode 100644 index 00000000000..efc513f2e6d --- /dev/null +++ b/plugins/knowledge/skills/video-digest/extraction/watch/relayout-slice.test.js @@ -0,0 +1,2807 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +import { afterEach, describe, expect, it, vi } from "vitest"; + +import { + checkLayoutLinks, + encodeDestination, + inlineLinks, + relayoutSlice, +} from "./relayout-slice.js"; + +/** @type {string[]} */ +const scratch = []; + +function makeDir(prefix) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix)); + scratch.push(dir); + return dir; +} + +/** @param {string} root @param {Record} files */ +function writeTree(root, files) { + for (const [rel, body] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), body); + } +} + +/** @param {string} root @param {string} rel */ +function read(root, rel) { + return fs.readFileSync(path.join(root, rel), "utf8"); +} + +const VIDEO_ID = "abc123XYZ_0"; + +/** + * A small closed slice plus its temp session (media, caption tracks, info JSON + * and two extracted frames). + * + * @param {{ status?: string, withMedia?: boolean, extraFiles?: Record }} [options] + */ +function makeFixture({ status = "complete", withMedia = true, extraFiles = {} } = {}) { + const parent = makeDir("relayout-slice-"); + const sliceDir = path.join(parent, "sample-talk-abc123XYZ_0"); + const workDir = makeDir("relayout-work-"); + const framesDir = makeDir("relayout-frames-"); + if (withMedia) { + writeTree(workDir, { + [`${VIDEO_ID}.mp4`]: "video-bytes", + [`${VIDEO_ID}.en.vtt`]: "WEBVTT\n", + [`${VIDEO_ID}.en-orig.vtt`]: "WEBVTT\n", + [`${VIDEO_ID}.info.json`]: JSON.stringify({ + id: VIDEO_ID, + title: "Sample Talk", + formats: [{}], + requested_formats: [{}], + thumbnails: [{}], + automatic_captions: {}, + subtitles: {}, + heatmap: [], + http_headers: {}, + requested_subtitles: {}, + _format_sort_fields: [], + requested_downloads: [{}], + _filename: "/tmp/video-extraction-x/abc123XYZ_0.mp4", + filename: "/tmp/video-extraction-x/abc123XYZ_0.mp4", + __files_to_move: {}, + comments: Array.from({ length: 25 }, (_, index) => ({ id: `c${index}` })), + }), + }); + writeTree(framesDir, { + "scene_0001.png": "png", + "anchor_00012000_0001.png": "png", + "frame-times.json": JSON.stringify({ "scene_0001.png": { timestampSec: 1 } }), + }); + } else { + fs.rmSync(workDir, { recursive: true, force: true }); + fs.rmSync(framesDir, { recursive: true, force: true }); + } + writeTree(sliceDir, { + "run-state/watch.json": JSON.stringify({ + videoId: VIDEO_ID, + videoSlug: "sample-talk-abc123XYZ_0", + sourceUrl: `https://www.youtube.com/watch?v=${VIDEO_ID}`, + title: "Sample Talk", + status, + phases: { transcript: { metrics: { transcriptStrategy: "captions" } } }, + tempSession: { workDir, framesDir, acquiredAt: "2026-10-10T12:00:00.000Z" }, + }), + "README.md": "# Sample Talk journey\n", + "RESEARCH.md": "Clusters from `research/claim-inventory.md`; shards in `research/findings/`.\n", + "source/transcript.txt": "[0:01] hello\n", + "source/harvested-links.json": "[]", + "key-frames/frames/title-slide.png": "png", + "key-frames/visual-frames.md": + "| `frames/title-slide.png` | `scene_0001.png` |\n\nTiers: [manifest](key-frames-manifest.md)\n", + "key-frames/key-frames-manifest.md": "| title-slide.png |\n", + "key-frames/key-frame-quality-audit.md": "| title-slide.png | yes |\n", + "key-frames/frame-triage-log.md": "| sheet_001 | promote |\n", + "key-frames/selection.json": "{}", + "source/companion-sources.md": "Companion: [digest](companion-digest/README.md).\n", + "source/companion-digest/README.md": "Brief: `source/companion-sources.md`.\n", + "research/claim-inventory.md": "Transcript: `source/transcript.txt`.\n", + "research/findings/topic.md": + "Quote at `SLICE/source/transcript.txt:3`; see [agenda](../claim-inventory.md#sessions).\n", + "recommendations/menu.md": [ + "Evidence: `source/transcript.txt` and ![slide](../key-frames/frames/title-slide.png).", + "Detail: `research/findings/topic.md`, the hub [README](README.md), the [journey](../README.md).", + "Target file: `plugins/knowledge/skills/video-digest/SKILL.md`.", + "Run state: `run-state/watch.json`, [selection](../key-frames/selection.json).", + "Upstream: `research/deepening/html-report.md:92` in mattpocock/skills.", + "", + ].join("\n"), + "recommendations/README.md": "# Hub\n", + ...extraFiles, + }); + return { sliceDir, workDir, framesDir, targetDir: path.join(makeDir("relayout-target-"), "out") }; +} + +afterEach(() => { + for (const dir of scratch.splice(0)) fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe("relayoutSlice", () => { + it("writes an upstream title with Markdown syntax as plain text in the README", async () => { + const { sliceDir, targetDir } = makeFixture(); + const watchPath = path.join(sliceDir, "run-state/watch.json"); + const watch = JSON.parse(fs.readFileSync(watchPath, "utf8")); + fs.writeFileSync(watchPath, JSON.stringify({ ...watch, title: "[Episode](trailer.md)\n#1" })); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + const readme = fs.readFileSync(path.join(targetDir, "README.md"), "utf8"); + expect(readme.split("\n")[0]).toBe("# \\[Episode\\]\\(trailer\\.md\\) \\#1"); + }); + + it("copies a closed slice into the corpus layout", async () => { + const { sliceDir, targetDir } = makeFixture(); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + for (const rel of [ + "transcript/transcript.txt", + "transcript/en.vtt", + "transcript/en-orig.vtt", + "metadata/info.json", + "metadata/harvested-links.json", + "frames/all/scene_0001.png", + "frames/all/anchor_00012000_0001.png", + "frames/all/frame-times.json", + "frames/key/title-slide.png", + "frames/visual-frames.md", + "frames/key-frames-manifest.md", + "frames/key-frame-quality-audit.md", + "frames/frame-triage-log.md", + "metadata/companion-sources.md", + "analysis/companion-digest/README.md", + `media/${VIDEO_ID}.mp4`, + "analysis/RESEARCH.md", + "analysis/research/claim-inventory.md", + "analysis/research/findings/topic.md", + "analysis/recommendations/menu.md", + "analysis/recommendations/README.md", + "README.md", + ]) { + expect(fs.existsSync(path.join(targetDir, rel)), rel).toBe(true); + } + expect(fs.existsSync(path.join(targetDir, "key-frames"))).toBe(false); + expect(fs.existsSync(path.join(targetDir, "run-state"))).toBe(false); + expect(fs.readdirSync(targetDir).filter((name) => name.startsWith("."))).toEqual([]); + }); + + it("trims info.json to the corpus copy", async () => { + const { sliceDir, targetDir } = makeFixture(); + + await relayoutSlice({ sliceDir, targetDir }); + + const info = JSON.parse(read(targetDir, "metadata/info.json")); + expect(Object.keys(info).sort()).toEqual(["comments", "id", "title"]); + expect(info.comments).toHaveLength(20); + }); + + it("rewrites slice-internal references relative to each file's new location", async () => { + const { sliceDir, targetDir } = makeFixture(); + + await relayoutSlice({ sliceDir, targetDir }); + + expect(read(targetDir, "analysis/recommendations/menu.md")).toBe( + [ + "Evidence: `../../transcript/transcript.txt` and ![slide](../../frames/key/title-slide.png).", + "Detail: `../research/findings/topic.md`, the hub [README](README.md), the journey (README.md, not retained in this copy).", + "Target file: `plugins/knowledge/skills/video-digest/SKILL.md`.", + "Run state: run-state/watch.json (not retained in this copy), selection (key-frames/selection.json, not retained in this copy).", + "Upstream: `research/deepening/html-report.md:92` in mattpocock/skills.", + "", + ].join("\n"), + ); + expect(read(targetDir, "metadata/companion-sources.md")).toBe( + "Companion: [digest](../analysis/companion-digest/README.md).\n", + ); + expect(read(targetDir, "analysis/companion-digest/README.md")).toBe( + "Brief: `../../metadata/companion-sources.md`.\n", + ); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + "| `key/title-slide.png` | `all/scene_0001.png` |\n\nTiers: [manifest](key-frames-manifest.md)\n", + ); + expect(read(targetDir, "analysis/research/findings/topic.md")).toBe( + "Quote at `../../../transcript/transcript.txt:3`; see [agenda](../claim-inventory.md#sessions).\n", + ); + expect(read(targetDir, "analysis/research/claim-inventory.md")).toBe( + "Transcript: `../../transcript/transcript.txt`.\n", + ); + expect(read(targetDir, "analysis/RESEARCH.md")).toBe( + "Clusters from `research/claim-inventory.md`; shards in `research/findings/`.\n", + ); + }); + + it("replaces an existing target with --replace, keeping its README", async () => { + const { sliceDir, targetDir } = makeFixture(); + writeTree(targetDir, { "README.md": "# Curated\n", "stale.md": "old\n" }); + + const result = await relayoutSlice({ sliceDir, targetDir, replace: true }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "README.md")).toBe("# Curated\n"); + expect(fs.existsSync(path.join(targetDir, "stale.md"))).toBe(false); + expect(fs.readdirSync(path.dirname(targetDir))).toEqual(["out"]); + }); + + it("refuses --replace on a target that is a regular file and keeps the file", async () => { + const { sliceDir, targetDir } = makeFixture(); + fs.mkdirSync(path.dirname(targetDir), { recursive: true }); + fs.writeFileSync(targetDir, "precious\n"); + + const result = await relayoutSlice({ sliceDir, targetDir, replace: true }); + + expect(result.exitCode).toBe(1); + expect(fs.readFileSync(targetDir, "utf8")).toBe("precious\n"); + }); + + it("refuses a slice whose Markdown uses bare CR line endings", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "research/findings/cr.md": "[deck]: ../../source/transcript.txt\rnext\r" }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("refuses an existing target without --replace and leaves it untouched", async () => { + const { sliceDir, targetDir } = makeFixture(); + writeTree(targetDir, { "stale.md": "old\n" }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.readdirSync(targetDir)).toEqual(["stale.md"]); + }); + + it("leaves the target untouched and no staging behind when a copy fails midway", async () => { + const { sliceDir, targetDir } = makeFixture(); + writeTree(targetDir, { "README.md": "# Curated\n", "stale.md": "old\n" }); + let calls = 0; + + const result = await relayoutSlice({ + sliceDir, + targetDir, + replace: true, + copyFile: (from, to) => { + calls += 1; + if (calls === 3) throw new Error("disk full"); + fs.copyFileSync(from, to); + }, + }); + + expect(result.exitCode).toBe(1); + expect(calls).toBe(3); + expect(fs.readdirSync(targetDir).sort()).toEqual(["README.md", "stale.md"]); + expect(read(targetDir, "stale.md")).toBe("old\n"); + expect(fs.readdirSync(path.dirname(targetDir))).toEqual(["out"]); + }); + + it("takes nothing from a temp session that still exists under --no-media", async () => { + const { sliceDir, targetDir } = makeFixture(); + + const result = await relayoutSlice({ sliceDir, targetDir, noMedia: true }); + + expect(result.exitCode).toBe(0); + for (const rel of ["media", "frames/all", "transcript/en.vtt", "metadata/info.json"]) { + expect(fs.existsSync(path.join(targetDir, rel)), rel).toBe(false); + } + expect(read(targetDir, "transcript/transcript.txt")).toBe("[0:01] hello\n"); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + "| `key/title-slide.png` | scene_0001.png (not retained in this copy) |\n\nTiers: [manifest](key-frames-manifest.md)\n", + ); + }); + + it("copies decks, attachments and the vision logs, rewriting their references", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/session-1/slides.pdf": "pdf", + "source/attachments/pdf/notes.pdf": "pdf", + "source/deck-inventory.md": "Fetched: `decks/session-1/slides.pdf`.\n", + "research/sources.md": "Notes: [pdf](../source/attachments/pdf/notes.pdf).\n", + "key-frames/vision-plan.md": "# Vision plan\n", + "key-frames/visual-gaps.md": "# Visual gaps\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "metadata/decks/session-1/slides.pdf")).toBe("pdf"); + expect(read(targetDir, "metadata/attachments/pdf/notes.pdf")).toBe("pdf"); + expect(read(targetDir, "metadata/deck-inventory.md")).toBe( + "Fetched: `decks/session-1/slides.pdf`.\n", + ); + expect(read(targetDir, "analysis/research/sources.md")).toBe( + "Notes: [pdf](../../metadata/attachments/pdf/notes.pdf).\n", + ); + expect(read(targetDir, "frames/vision-plan.md")).toBe("# Vision plan\n"); + expect(read(targetDir, "frames/visual-gaps.md")).toBe("# Visual gaps\n"); + }); + + it("names where a split directory's files went instead of a single new path", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "recommendations/questions.md": "Inputs: `source/`.\n" }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "analysis/recommendations/questions.md")).toBe( + "Inputs: source (split across analysis/, metadata/, transcript/ in this copy).\n", + ); + }); + + it("fails the link check on a frame name the temp session does not hold", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": "Rejected: `scene_0099.png`.\n" }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([{ file: "frames/visual-frames.md", ref: "scene_0099.png" }]); + }); + + it("warns about leftover staging dirs and leaves them alone", async () => { + const { sliceDir, targetDir } = makeFixture(); + const leftover = path.join(path.dirname(targetDir), ".out.relayout-abc123"); + writeTree(leftover, { "README.md": "# From a killed run\n" }); + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true); + + const result = await relayoutSlice({ sliceDir, targetDir }); + const message = stderr.mock.calls.map(([chunk]) => String(chunk)).join(""); + stderr.mockRestore(); + + expect(result.exitCode).toBe(0); + expect(message).toContain(leftover); + expect(read(leftover, "README.md")).toBe("# From a killed run\n"); + expect(read(targetDir, "README.md")).not.toBe("# From a killed run\n"); + }); + + it.skipIf(process.platform === "win32")( + "installs over an empty target directory created during the run (it holds no data)", + async () => { + const { sliceDir, targetDir } = makeFixture(); + let calls = 0; + + const result = await relayoutSlice({ + sliceDir, + targetDir, + copyFile: (from, to) => { + calls += 1; + if (calls === 1) fs.mkdirSync(targetDir); + fs.copyFileSync(from, to); + }, + }); + + expect(result.exitCode).toBe(0); + expect(fs.existsSync(path.join(targetDir, "analysis/RESEARCH.md"))).toBe(true); + }, + ); + + it.skipIf(process.platform === "win32" || process.getuid?.() === 0)( + "refuses, without throwing, when the backup restore fails", + async () => { + const { sliceDir, targetDir } = makeFixture(); + writeTree(`${targetDir}.relayout-backup`, { "README.md": "# Curated\n" }); + const parent = path.dirname(targetDir); + fs.chmodSync(parent, 0o555); + try { + const result = await relayoutSlice({ sliceDir, targetDir, replace: true }); + + expect(result.exitCode).toBe(1); + expect(read(`${targetDir}.relayout-backup`, "README.md")).toBe("# Curated\n"); + } finally { + fs.chmodSync(parent, 0o755); + } + }, + ); + + it("refuses, without --replace, a target that appears after the early check", async () => { + const { sliceDir, targetDir } = makeFixture(); + let calls = 0; + + const result = await relayoutSlice({ + sliceDir, + targetDir, + copyFile: (from, to) => { + calls += 1; + // Another run installs its layout while this one is staging. + if (calls === 1) writeTree(targetDir, { "theirs.md": "theirs\n" }); + fs.copyFileSync(from, to); + }, + }); + + expect(result.exitCode).toBe(1); + expect(fs.readdirSync(targetDir)).toEqual(["theirs.md"]); + expect(fs.readdirSync(path.dirname(targetDir))).toEqual(["out"]); + }); + + it("fails the link check on a missing file under a slice directory", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "recommendations/questions.md": "Slide: `key-frames/frames/deleted.png`.\n" }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "analysis/recommendations/questions.md", ref: "../../frames/key/deleted.png" }, + ]); + expect(fs.readdirSync(path.dirname(targetDir))).toEqual([]); + }); + + it("re-lays out an unclosed slice, media included, when its outcome checks pass", async () => { + const { sliceDir, targetDir } = makeFixture({ status: "synthesizing" }); + /** @type {string[]} */ + const verified = []; + + const result = await relayoutSlice({ + sliceDir, + targetDir, + verifyOutcomes: (dir) => { + verified.push(dir); + return 0; + }, + }); + + expect(result.exitCode).toBe(0); + expect(verified).toEqual([path.resolve(sliceDir)]); + expect(fs.existsSync(path.join(targetDir, `media/${VIDEO_ID}.mp4`))).toBe(true); + }); + + it("refuses an unclosed slice whose outcome checks fail", async () => { + const { sliceDir, targetDir } = makeFixture({ status: "synthesizing" }); + + const result = await relayoutSlice({ sliceDir, targetDir, verifyOutcomes: () => 1 }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("refuses an unclosed slice that fails the real outcome checks", async () => { + const { sliceDir, targetDir } = makeFixture({ status: "synthesizing" }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("skips the outcome checks for a closed slice", async () => { + const { sliceDir, targetDir } = makeFixture(); + + const result = await relayoutSlice({ sliceDir, targetDir, verifyOutcomes: () => 1 }); + + expect(result.exitCode).toBe(0); + }); + + it("refuses when the temp session's media is gone, unless --no-media", async () => { + const { sliceDir, targetDir } = makeFixture({ withMedia: false }); + + const refused = await relayoutSlice({ sliceDir, targetDir }); + expect(refused.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + + const result = await relayoutSlice({ sliceDir, targetDir, noMedia: true }); + expect(result.exitCode).toBe(0); + expect(fs.existsSync(path.join(targetDir, "media"))).toBe(false); + expect(fs.existsSync(path.join(targetDir, "analysis/recommendations/menu.md"))).toBe(true); + }); + + it.each([ + ["the slice itself", (/** @type {string} */ slice) => slice], + ["an ancestor of the slice", (/** @type {string} */ slice) => path.dirname(slice)], + ["a directory inside the slice", (/** @type {string} */ slice) => path.join(slice, "out")], + ])("refuses a target that is %s, even with --replace", async (_label, toTarget) => { + const { sliceDir } = makeFixture(); + const targetDir = toTarget(sliceDir); + const before = fs.readdirSync(sliceDir).sort(); + + const result = await relayoutSlice({ sliceDir, targetDir, replace: true }); + + expect(result.exitCode).toBe(1); + expect(fs.readdirSync(sliceDir).sort()).toEqual(before); + expect(fs.existsSync(path.join(sliceDir, "out"))).toBe(false); + }); + + it("restores the target from the backup an interrupted --replace left", async () => { + const { sliceDir, targetDir } = makeFixture(); + // Killed between the two renames: the old target sits at the backup name. + writeTree(`${targetDir}.relayout-backup`, { "README.md": "# Curated\n" }); + + const result = await relayoutSlice({ sliceDir, targetDir, replace: true }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "README.md")).toBe("# Curated\n"); + expect(fs.readdirSync(path.dirname(targetDir))).toEqual(["out"]); + }); + + it("refuses when both the target and a leftover backup exist", async () => { + const { sliceDir, targetDir } = makeFixture(); + writeTree(targetDir, { "README.md": "# New\n" }); + writeTree(`${targetDir}.relayout-backup`, { "README.md": "# Old\n" }); + + const result = await relayoutSlice({ sliceDir, targetDir, replace: true }); + + expect(result.exitCode).toBe(1); + expect(read(targetDir, "README.md")).toBe("# New\n"); + expect(read(`${targetDir}.relayout-backup`, "README.md")).toBe("# Old\n"); + }); + + it("refuses a temp session whose frames are gone though the video survives", async () => { + const { sliceDir, targetDir, framesDir } = makeFixture(); + fs.rmSync(framesDir, { recursive: true, force: true }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("refuses a temp session holding several videos, since the primary is not recorded", async () => { + const { sliceDir, targetDir, workDir } = makeFixture(); + // An X post with two videos: yt-dlp writes each as `.`. + writeTree(workDir, { + "1800000000000000002.mp4": "video-2", + "1800000000000000002.info.json": "{}", + "1800000000000000002.en.vtt": "WEBVTT\n", + }); + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true); + + const result = await relayoutSlice({ sliceDir, targetDir }); + const message = stderr.mock.calls.map(([chunk]) => String(chunk)).join(""); + stderr.mockRestore(); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + expect(message).toContain("1800000000000000002.mp4"); + expect(message).toContain(`${VIDEO_ID}.mp4`); + }); + + it("refuses a temp session whose info JSON is gone though the video survives", async () => { + const { sliceDir, targetDir, workDir } = makeFixture(); + fs.rmSync(path.join(workDir, `${VIDEO_ID}.info.json`)); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it.each([ + [ + "is the backup path", + (/** @type {string} */ parent) => path.join(parent, "out.relayout-backup"), + ], + [ + "sits inside the backup path", + (/** @type {string} */ parent) => path.join(parent, "out.relayout-backup", "slice"), + ], + [ + "sits inside a staging dir", + (/** @type {string} */ parent) => path.join(parent, ".out.relayout-abc123", "slice"), + ], + ])("refuses when the slice %s of the target", async (_label, toSlice) => { + const { sliceDir: original } = makeFixture(); + const parent = makeDir("relayout-overlap-"); + const sliceDir = toSlice(parent); + fs.mkdirSync(path.dirname(sliceDir), { recursive: true }); + fs.renameSync(original, sliceDir); + const before = fs.readdirSync(sliceDir).sort(); + + const result = await relayoutSlice({ + sliceDir, + targetDir: path.join(parent, "out"), + replace: true, + }); + + expect(result.exitCode).toBe(1); + expect(fs.readdirSync(sliceDir).sort()).toEqual(before); + expect(fs.existsSync(path.join(parent, "out"))).toBe(false); + }); + + it("refuses a slice holding a symlink, which a copy would follow", async () => { + const { sliceDir, targetDir } = makeFixture(); + const outside = path.join(makeDir("relayout-secret-"), "secret.md"); + fs.writeFileSync(outside, "secret\n"); + fs.symlinkSync(outside, path.join(sliceDir, "research", "leak.md")); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it.each([ + ["outside the OS temp dir", () => path.join(path.parse(os.tmpdir()).root, "relayout-not-tmp")], + ["the slice itself", (/** @type {string} */ slice) => slice], + ])("refuses a recorded temp dir %s", async (_label, toWorkDir) => { + const { sliceDir, targetDir } = makeFixture(); + const statePath = path.join(sliceDir, "run-state", "watch.json"); + const state = JSON.parse(read(sliceDir, "run-state/watch.json")); + state.tempSession.workDir = toWorkDir(sliceDir); + fs.writeFileSync(statePath, JSON.stringify(state)); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it.each([ + [ + "frame-times.json is gone", + (/** @type {string} */ dir) => fs.rmSync(path.join(dir, "frame-times.json")), + ], + [ + "a frame frame-times.json lists is gone", + (/** @type {string} */ dir) => fs.rmSync(path.join(dir, "scene_0001.png")), + ], + [ + "a scene frame has no frame-times.json entry", + (/** @type {string} */ dir) => fs.writeFileSync(path.join(dir, "scene_0002.png"), "png"), + ], + [ + "an interval frame has no frame-times.json entry", + (/** @type {string} */ dir) => fs.writeFileSync(path.join(dir, "interval_0001.png"), "png"), + ], + [ + "frame-times.json holds null", + (/** @type {string} */ dir) => fs.writeFileSync(path.join(dir, "frame-times.json"), "null"), + ], + [ + "frame-times.json holds an array", + (/** @type {string} */ dir) => + fs.writeFileSync(path.join(dir, "frame-times.json"), '["scene_0001.png"]'), + ], + [ + "a frame-times.json record is not an object", + (/** @type {string} */ dir) => + fs.writeFileSync(path.join(dir, "frame-times.json"), '{"scene_0001.png": 1}'), + ], + [ + "a frame-times.json record has no timestampSec", + (/** @type {string} */ dir) => + fs.writeFileSync( + path.join(dir, "frame-times.json"), + '{"scene_0001.png": {"timestampSource": "scene-detection"}}', + ), + ], + [ + "a frame-times.json record has a non-numeric timestampSec", + (/** @type {string} */ dir) => + fs.writeFileSync( + path.join(dir, "frame-times.json"), + '{"scene_0001.png": {"timestampSec": "1.5"}}', + ), + ], + [ + "only the sidecar is left", + (/** @type {string} */ dir) => { + for (const name of fs.readdirSync(dir).filter((file) => file.endsWith(".png"))) { + fs.rmSync(path.join(dir, name)); + } + }, + ], + ])("refuses a temp session where %s", async (_label, damage) => { + const { sliceDir, targetDir, framesDir } = makeFixture(); + damage(framesDir); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("refuses a temp session holding a symlink, which a copy would follow", async () => { + const { sliceDir, targetDir, framesDir } = makeFixture(); + const outside = path.join(makeDir("relayout-secret-"), "secret.png"); + fs.writeFileSync(outside, "secret"); + fs.symlinkSync(outside, path.join(framesDir, "anchor_00030000_0002.png")); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("refuses, without throwing, a target whose parent is a file", async () => { + const { sliceDir } = makeFixture(); + const blocker = path.join(makeDir("relayout-blocker-"), "file"); + fs.writeFileSync(blocker, ""); + + const result = await relayoutSlice({ sliceDir, targetDir: path.join(blocker, "out") }); + + expect(result.exitCode).toBe(1); + }); + + it.each([ + [ + "inside the recorded workDir", + (/** @type {Record} */ dirs) => path.join(dirs.workDir, "out"), + ], + [ + "the recorded framesDir itself", + (/** @type {Record} */ dirs) => dirs.framesDir, + ], + [ + "holding a recorded contactSheetsDir", + (/** @type {Record} */ dirs) => path.dirname(dirs.contactSheetsDir), + ], + [ + "inside a recorded contactSheetsDir that no longer exists", + (/** @type {Record} */ dirs) => path.join(dirs.contactSheetsDir, "out"), + ], + ])("refuses, even under --no-media, a target %s", async (_label, toTarget) => { + const { sliceDir, workDir, framesDir } = makeFixture(); + // `close` already removed this one; its parent holds nothing else. + const contactSheetsDir = path.join(makeDir("relayout-holder-"), "sheets"); + const statePath = path.join(sliceDir, "run-state", "watch.json"); + const state = JSON.parse(fs.readFileSync(statePath, "utf8")); + state.tempSession.contactSheetsDir = contactSheetsDir; + fs.writeFileSync(statePath, JSON.stringify(state)); + const targetDir = toTarget({ workDir, framesDir, contactSheetsDir }); + const before = fs.readdirSync(framesDir).sort(); + + const result = await relayoutSlice({ sliceDir, targetDir, noMedia: true, replace: true }); + + expect(result.exitCode).toBe(1); + expect(fs.readdirSync(framesDir).sort()).toEqual(before); + expect(fs.existsSync(path.join(workDir, "out"))).toBe(false); + expect(fs.existsSync(contactSheetsDir)).toBe(false); + }); + + /** + * Rename the temp session's video, info JSON and caption tracks to another + * yt-dlp id, optionally writing an info JSON with `display_id`. + * + * @param {string} workDir + * @param {string} id + * @param {Record} [info] + */ + function renameMedia(workDir, id, info) { + for (const name of fs.readdirSync(workDir)) { + fs.renameSync(path.join(workDir, name), path.join(workDir, name.replace(VIDEO_ID, id))); + } + if (info) fs.writeFileSync(path.join(workDir, `${id}.info.json`), JSON.stringify(info)); + } + + it("refuses a temp session whose video is another id, naming both", async () => { + const { sliceDir, targetDir, workDir } = makeFixture(); + renameMedia(workDir, "otherVideo1", { id: "otherVideo1", display_id: "otherVideo1" }); + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true); + + const result = await relayoutSlice({ sliceDir, targetDir }); + const message = stderr.mock.calls.map(([chunk]) => String(chunk)).join(""); + stderr.mockRestore(); + + expect(result.exitCode).toBe(1); + expect(message).toContain("otherVideo1"); + expect(message).toContain(VIDEO_ID); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("accepts an X media id whose info JSON names this slice's post as display_id", async () => { + const { sliceDir, targetDir, workDir } = makeFixture(); + renameMedia(workDir, "1800000000000000001", { + id: "1800000000000000001", + display_id: VIDEO_ID, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(fs.existsSync(path.join(targetDir, `media/${VIDEO_ID}.mp4`))).toBe(true); + }); + + it.each([false, true])( + "refuses a recorded contact-sheets dir outside the OS temp dir (noMedia: %s)", + async (noMedia) => { + const { sliceDir, targetDir } = makeFixture(); + const statePath = path.join(sliceDir, "run-state", "watch.json"); + const state = JSON.parse(fs.readFileSync(statePath, "utf8")); + state.tempSession.contactSheetsDir = path.join( + path.parse(os.tmpdir()).root, + "relayout-sheets", + ); + fs.writeFileSync(statePath, JSON.stringify(state)); + + const result = await relayoutSlice({ sliceDir, targetDir, noMedia }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }, + ); + + it.each([false, true])( + "refuses a recorded contact-sheets dir that overlaps the slice (noMedia: %s)", + async (noMedia) => { + const { sliceDir, targetDir } = makeFixture(); + const statePath = path.join(sliceDir, "run-state", "watch.json"); + const state = JSON.parse(fs.readFileSync(statePath, "utf8")); + state.tempSession.contactSheetsDir = path.join(sliceDir, "key-frames"); + fs.writeFileSync(statePath, JSON.stringify(state)); + + const result = await relayoutSlice({ sliceDir, targetDir, noMedia }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }, + ); + + it("reads no link inside a code span that spans lines, and treats a lone backtick as text", async () => { + const body = [ + "Code: `see [gone](../research/gone.md)", + "and more` then [t](../source/transcript.txt).", + "A lone ` backtick and [u](../source/transcript.txt).", + "", + ].join("\n"); + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": body }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + body.replaceAll("](../source/transcript.txt)", "](../transcript/transcript.txt)"), + ); + }); + + it("ends a code span search at a CRLF blank line, so a later link is still checked", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "analysis/notes.md": "A lone ` backtick.\r\n\r\n[gone](missing.md) then ` again.\r\n", + }); + + expect(checkLayoutLinks(root)).toEqual([{ file: "analysis/notes.md", ref: "missing.md" }]); + }); + + it("reads \\![x](dest) as a plain link and keeps the \\! when retiring it", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "key-frames/visual-frames.md": + "Bang: \\![state](../run-state/watch.json) and \\![t](../source/transcript.txt).\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + "Bang: \\!state (run-state/watch.json, not retained in this copy) and \\![t](../transcript/transcript.txt).\n", + ); + }); + + it("catches, through the link check, a dangling path in a fence inside a block quote or list", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "recommendations/questions.md": [ + "> ```", + "> [gone](../research/gone.md)", + "> ```", + "", + "- item", + "", + " ```", + " [also gone](../research/also-gone.md)", + " ```", + "", + ].join("\n"), + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "analysis/recommendations/questions.md", ref: "../research/gone.md" }, + { file: "analysis/recommendations/questions.md", ref: "../research/also-gone.md" }, + ]); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("does not read an escaped \\[label](...) as a link", async () => { + const body = "Literal: \\[watch](../run-state/watch.json) and \\[x\\](../research/gone.md).\n"; + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": body }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe(body); + }); + + it("leaves a command span alone and does not link-check it", async () => { + const body = [ + "Run `node run.mjs source/transcript.txt --no-media` or `source/transcript.txt --verbose`.", + "", + ].join("\n"); + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "recommendations/questions.md": body }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "analysis/recommendations/questions.md")).toBe(body); + }); + + it("leaves fenced code blocks alone and reads no link inside inline code", async () => { + const body = [ + "Outside: [t](../source/transcript.txt).", + "```", + "[t](../source/transcript.txt) `source/transcript.txt` [gone](../research/gone.md)", + "```", + "~~~~ text", + "[t](../source/transcript.txt)", + "~~~~", + "Inline: `[gone](../research/gone.md)`.", + "", + ].join("\n"); + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": body }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + body.replace( + "Outside: [t](../source/transcript.txt)", + "Outside: [t](../transcript/transcript.txt)", + ), + ); + }); + + it.each([ + ["framesDir", "framesDir"], + ["workDir", "workDir"], + ])("refuses, without throwing, a recorded %s that is a file", async (_label, field) => { + const fixture = makeFixture(); + const dir = field === "framesDir" ? fixture.framesDir : fixture.workDir; + fs.rmSync(dir, { recursive: true, force: true }); + fs.writeFileSync(dir, "not a directory"); + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true); + + const result = await relayoutSlice({ + sliceDir: fixture.sliceDir, + targetDir: fixture.targetDir, + }); + const message = stderr.mock.calls.map(([chunk]) => String(chunk)).join(""); + stderr.mockRestore(); + + expect(result.exitCode).toBe(1); + expect(message).toContain(`${dir} is not a readable directory`); + expect(fs.existsSync(fixture.targetDir)).toBe(false); + }); + + it("decodes backslash escapes in link and definition destinations, keeping \\# in the path", async () => { + const body = [ + "[n](../research/findings/notes\\#1.md#intro) [q](../research/findings/what\\?.md)", + "", + "[def]: ../research/findings/notes\\#1.md", + "", + ].join("\n"); + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "research/findings/notes#1.md": "n\n", + "research/findings/what?.md": "w\n", + "key-frames/visual-frames.md": body, + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + [ + "[n](../analysis/research/findings/notes\\#1.md#intro) [q](../analysis/research/findings/what\\?.md)", + "", + "[def]: ../analysis/research/findings/notes\\#1.md", + "", + ].join("\n"), + ); + }); + + it.skipIf(process.platform === "win32" || process.getuid?.() === 0)( + "refuses, naming it, a slice holding an unreadable nested dir", + async () => { + const { sliceDir, targetDir } = makeFixture(); + const locked = path.join(sliceDir, "research", "findings"); + fs.chmodSync(locked, 0o000); + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true); + try { + const result = await relayoutSlice({ sliceDir, targetDir }); + const message = stderr.mock.calls.map(([chunk]) => String(chunk)).join(""); + + expect(result.exitCode).toBe(1); + expect(message).toContain(locked); + expect(fs.readdirSync(path.dirname(targetDir))).toEqual([]); + } finally { + stderr.mockRestore(); + fs.chmodSync(locked, 0o755); + } + }, + ); + + it("matches a backticked path whose name holds # or ? before reading a fragment", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "research/findings/notes#1.md": "n\n", + "recommendations/questions.md": + "Raw: `research/findings/notes#1.md`. Fragment: `research/findings/topic.md#intro`.\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "analysis/recommendations/questions.md")).toBe( + "Raw: `../research/findings/notes#1.md`. Fragment: `../research/findings/topic.md#intro`.\n", + ); + }); + + it("reads a raw # in a link as a fragment, unlike the same backticked path", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "analysis/research/findings/notes#1.md": "n\n", + "analysis/notes.md": + "[notes](research/findings/notes#1.md) `research/findings/notes#1.md`\n", + }); + + expect(checkLayoutLinks(root)).toEqual([ + { file: "analysis/notes.md", ref: "research/findings/notes#1.md" }, + ]); + writeTree(root, { "analysis/research/findings/notes": "n\n" }); + expect(checkLayoutLinks(root)).toEqual([]); + }); + + it("refuses a link whose raw # names a retained file only as part of the name", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "research/findings/notes#1.md": "n\n", + "recommendations/questions.md": "[notes](research/findings/notes#1.md)\n", + }, + }); + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true); + try { + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toHaveLength(1); + expect(result.unresolved[0].ref).toMatch(/research\/findings\/notes#1\.md$/); + } finally { + stderr.mockRestore(); + } + }); + + it.each([ + [ + "a frame name that is a directory", + (/** @type {Record} */ d) => + fs.mkdirSync(path.join(d.framesDir, "anchor_00030000_0002.png")), + ], + [ + "a frame-times.json that is a directory", + (/** @type {Record} */ d) => { + fs.rmSync(path.join(d.framesDir, "frame-times.json")); + fs.mkdirSync(path.join(d.framesDir, "frame-times.json")); + }, + ], + ])("refuses %s, naming it", async (label, damage) => { + const { sliceDir, targetDir, workDir, framesDir } = makeFixture(); + damage({ workDir, framesDir }); + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true); + + const result = await relayoutSlice({ sliceDir, targetDir }); + const message = stderr.mock.calls.map(([chunk]) => String(chunk)).join(""); + stderr.mockRestore(); + + expect(result.exitCode).toBe(1); + expect(message).toContain("regular files for the frames entries"); + expect(message).toContain( + label.includes("frame-times") ? "frame-times.json" : "anchor_00030000_0002.png", + ); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("refuses, naming it, a frame-times.json key that is a directory", async () => { + const { sliceDir, targetDir, framesDir } = makeFixture(); + fs.writeFileSync( + path.join(framesDir, "frame-times.json"), + JSON.stringify({ "scene_0001.png": { timestampSec: 1 }, scene_0002: { timestampSec: 2 } }), + ); + fs.mkdirSync(path.join(framesDir, "scene_0002")); + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true); + + const result = await relayoutSlice({ sliceDir, targetDir }); + const message = stderr.mock.calls.map(([chunk]) => String(chunk)).join(""); + stderr.mockRestore(); + + expect(result.exitCode).toBe(1); + expect(message).toContain("scene_0002"); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("refuses an empty video file", async () => { + const { sliceDir, targetDir, workDir } = makeFixture(); + fs.writeFileSync(path.join(workDir, `${VIDEO_ID}.mp4`), ""); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("refuses a captioned temp session whose caption tracks are gone", async () => { + const { sliceDir, targetDir, workDir } = makeFixture(); + for (const name of fs.readdirSync(workDir).filter((file) => file.endsWith(".vtt"))) { + fs.rmSync(path.join(workDir, name)); + } + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + /** + * @param {string} sliceDir + * @param {Record} transcriptMetrics + * @param {string} workDir + */ + function withTranscriptMetricsAndNoCaptions(sliceDir, transcriptMetrics, workDir) { + const statePath = path.join(sliceDir, "run-state", "watch.json"); + const state = JSON.parse(fs.readFileSync(statePath, "utf8")); + // What recover-watch-bootstrap.js writes: acquire and transcript carry only `recovered`. + state.phases = { + acquire: { metrics: { videoDownloaded: true, recovered: true } }, + transcript: { metrics: transcriptMetrics }, + }; + fs.writeFileSync(statePath, JSON.stringify(state)); + for (const name of fs.readdirSync(workDir).filter((file) => file.endsWith(".vtt"))) { + fs.rmSync(path.join(workDir, name)); + } + } + + it("refuses a bootstrap-recovered session whose caption tracks are gone", async () => { + const { sliceDir, targetDir, workDir } = makeFixture(); + withTranscriptMetricsAndNoCaptions(sliceDir, { recovered: true }, workDir); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("still requires caption tracks when a caption transcript records a degradation", async () => { + const { sliceDir, targetDir, workDir } = makeFixture(); + // write-transcript.js sets transcriptDegradation to the caption's + // provenanceNote even when that caption produced the transcript. + withTranscriptMetricsAndNoCaptions( + sliceDir, + { transcriptStrategy: "captions", transcriptDegradation: "auto captions only" }, + workDir, + ); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + it("needs no caption tracks when no entry produced a transcript", async () => { + const { sliceDir, targetDir, workDir } = makeFixture(); + withTranscriptMetricsAndNoCaptions( + sliceDir, + { skipped: true, reason: "no transcript-bearing entries" }, + workDir, + ); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + }); + + it("needs no caption tracks when the transcript came from ASR", async () => { + const { sliceDir, targetDir, workDir } = makeFixture(); + withTranscriptMetricsAndNoCaptions(sliceDir, { transcriptStrategy: "asr" }, workDir); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(fs.existsSync(path.join(targetDir, "transcript/en.vtt"))).toBe(false); + }); +}); + +describe("relayoutSlice input shapes and reference forms (adversarial sweep)", () => { + /** @param {string} sliceDir @param {(state: Record) => unknown} edit */ + function editState(sliceDir, edit) { + const statePath = path.join(sliceDir, "run-state", "watch.json"); + const next = edit(JSON.parse(fs.readFileSync(statePath, "utf8"))); + fs.writeFileSync(statePath, JSON.stringify(next)); + } + + it.each([ + ["a videoId that is a path", (/** @type {any} */ s) => ({ ...s, videoId: "../../escaped" })], + ["no title", (/** @type {any} */ s) => ({ ...s, title: undefined })], + ["no sourceUrl", (/** @type {any} */ s) => ({ ...s, sourceUrl: "" })], + ["null instead of an object", () => null], + [ + "a temp dir that is not a string", + (/** @type {any} */ s) => ({ ...s, tempSession: { ...s.tempSession, workDir: 123 } }), + ], + ])("refuses, without throwing, a watch.json with %s", async (_label, edit) => { + const { sliceDir, targetDir } = makeFixture(); + editState(sliceDir, edit); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.readdirSync(path.dirname(targetDir))).toEqual([]); + }); + + it("rewrites reference-style definitions, HTML src, encoded, spaced and backslash paths", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "research/findings/my notes.md": "notes\n", + "key-frames/visual-frames.md": [ + '[defined][1] and ', + "[spaced](<../research/findings/my notes.md>) [encoded](../research/findings/my%20notes.md)", + "Windows: `frames\\title-slide.png`.", + "", + "[1]: ../source/transcript.txt", + "", + ].join("\n"), + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + [ + '[defined][1] and ', + "[spaced](<../analysis/research/findings/my notes.md>) [encoded](../analysis/research/findings/my%20notes.md)", + "Windows: `key/title-slide.png`.", + "", + "[1]: ../transcript/transcript.txt", + "", + ].join("\n"), + ); + }); + + it("rewrites link destinations holding balanced or escaped parentheses", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/session-1/slides_(final).pdf": "pdf", + "key-frames/visual-frames.md": [ + "[balanced](../source/decks/session-1/slides_(final).pdf)", + '[escaped](../source/decks/session-1/slides_\\(final\\).pdf "Deck")', + "", + ].join("\n"), + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + [ + "[balanced](../metadata/decks/session-1/slides_(final).pdf)", + '[escaped](../metadata/decks/session-1/slides_\\(final\\).pdf "Deck")', + "", + ].join("\n"), + ); + }); + + // CommonMark: a backslash-escaped bracket is literal link text, so the link + // opens at the unescaped `[` before it. + it("reads a link whose text holds an escaped opening bracket", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": "[see \\[draft](../source/transcript.txt)\n" }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + "[see \\[draft](../transcript/transcript.txt)\n", + ); + }); + + // CommonMark: entity and numeric character references are decoded in link + // destinations, so `a&b.pdf` names the file `a&b.pdf`. + it("decodes character references in link and definition destinations", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/a&b.pdf": "pdf", + "key-frames/visual-frames.md": [ + "[deck](../source/decks/a&b.pdf)", + "[hex][d]", + "", + "[d]: ../source/decks/a&b.pdf", + "", + ].join("\n"), + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + [ + "[deck](../metadata/decks/a&b.pdf)", + "[hex][d]", + "", + "[d]: ../metadata/decks/a&b.pdf", + "", + ].join("\n"), + ); + }); + + it("keeps a link whose destination names a retained file by a windows-1252 reference", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/€.pdf": "pdf", + "key-frames/visual-frames.md": "[deck](../source/decks/€.pdf)\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "metadata/decks/€.pdf")).toBe("pdf"); + }); + + // Processing instructions, declarations, CDATA and closing tags are raw inline HTML, + // and a tag after an escaped `](` is a tag, not a link destination. + it("treats link-shaped text inside other raw HTML forms as data", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/notes.md": [ + "before after", + " ", + '\\]()', + "[real](gone.md)", + "", + ].join("\n"), + }); + + expect(checkLayoutLinks(root)).toEqual([{ file: "metadata/notes.md", ref: "gone.md" }]); + }); + + it("follows CommonMark's raw HTML whitespace, short comments and reference digit limits", () => { + const nbsp = String.fromCodePoint(0xa0); + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/notes.md": [ + ``, + "[t](gone-2.md)-->", + "[t](aA.md)", + "", + ].join("\n"), + }); + + expect(checkLayoutLinks(root)).toEqual([ + { file: "metadata/notes.md", ref: "gone-1.md" }, + { file: "metadata/notes.md", ref: "gone-2.md" }, + { file: "metadata/notes.md", ref: "aA.md" }, + ]); + }); + + it("reads an escaped HTML opener as text and uncapped digits in attributes", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/decks/a&b.pdf": "pdf", + "metadata/notes.md": [ + '\\', + 'deck', + "", + ].join("\n"), + }); + + expect(checkLayoutLinks(root)).toEqual([{ file: "metadata/notes.md", ref: "gone.md" }]); + }); + + it("does not pair code-span backticks across a heading, quote or list item", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/notes.md": [ + "dangling `", + "# [h](gone-1.md) `", + "again `", + "> [q](gone-2.md) `", + "once more `", + "- [l](gone-3.md) `", + "", + "setext `", + "===", + "[s](gone-4.md) `", + "", + ].join("\n"), + }); + + expect(checkLayoutLinks(root)).toEqual([ + { file: "metadata/notes.md", ref: "gone-1.md" }, + { file: "metadata/notes.md", ref: "gone-2.md" }, + { file: "metadata/notes.md", ref: "gone-3.md" }, + { file: "metadata/notes.md", ref: "gone-4.md" }, + ]); + }); + + it("ignores escaped tags in the attribute pass and tags that span a blank line", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/notes.md": [ + '\\', + "", + '', + "", + ].join("\n"), + }); + + expect(checkLayoutLinks(root)).toEqual([{ file: "metadata/notes.md", ref: "gone-2.md" }]); + }); + + it("does not read attributes inside CDATA, declarations or processing instructions", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/notes.md": + ']]> ?> x\n', + }); + + expect(checkLayoutLinks(root)).toEqual([{ file: "metadata/notes.md", ref: "lost.md" }]); + }); + + // The URL parser trims leading and trailing spaces and controls from an attribute URL. + it("trims edge whitespace from an HTML URL", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/ok.md": "ok\n", + "metadata/notes.md": 'ok\n', + }); + + expect(checkLayoutLinks(root)).toEqual([]); + }); + + // Known limit: a definition title is not masked, so link-shaped text in it is checked + // and a missing path there makes the run refuse rather than pass. + it("refuses a missing path written as a link inside a definition title", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/ok.md": "ok\n", + "metadata/notes.md": '[real]: ok.md "see [fake](missing.md)"\n', + }); + + expect(checkLayoutLinks(root)).toEqual([{ file: "metadata/notes.md", ref: "missing.md" }]); + }); + + // The HTML parser decodes references before the URL parser drops tab, LF and CR, + // so an encoded line break in an attribute is removed too. + it("drops a line break written as a reference in an HTML URL", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "transcript/transcript.txt": "[0:01] hi\n", + "metadata/notes.md": 't\n', + }); + + expect(checkLayoutLinks(root)).toEqual([]); + }); + + // HTML attributes accept a numeric reference without its semicolon; Markdown + // destinations do not, so only the attribute is decoded. + it("decodes semicolonless numeric references in HTML attributes only", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/decks/a&b.pdf": "pdf", + "metadata/deck-inventory.md": + 'deck\n[md](decks/c&d.pdf)\n', + }); + + expect(checkLayoutLinks(root)).toEqual([ + { file: "metadata/deck-inventory.md", ref: "decks/c&d.pdf" }, + ]); + }); + + // HTML attributes decode a legacy named reference without its semicolon, + // except before a letter, digit or `=`: `&.` is `&`, `&x` stays. + it("decodes a semicolonless legacy named reference in an HTML attribute", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/decks/a&.pdf": "pdf", + "metadata/decks/a&x.pdf": "pdf", + "metadata/deck-inventory.md": + 'deck\nx\n', + }); + + expect(checkLayoutLinks(root)).toEqual([ + { file: "metadata/deck-inventory.md", ref: "decks/a&x.pdf" }, + ]); + }); + + // HTML numeric character reference replacements: 0x80 is windows-1252's + // euro sign (U+20AC), and 0 becomes U+FFFD. + it("applies HTML's numeric character reference replacements", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/decks/€.pdf": "pdf", + "metadata/deck-inventory.md": "[deck](decks/€.pdf)\n", + }); + + expect(checkLayoutLinks(root)).toEqual([]); + const replacement = String.fromCodePoint(0xfffd); + expect( + inlineLinks("[a](a�b.md) [b](�.md) [c](�.md) [d](.md)").map( + ({ dest }) => dest, + ), + ).toEqual([`a${replacement}b.md`, `${replacement}.md`, `${replacement}.md`, "\u0081.md"]); + }); + + it("resolves a destination holding a character reference in the link check", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/decks/a&b.pdf": "pdf", + "metadata/deck-inventory.md": "[deck](decks/a&b.pdf) [gone](decks/c&d.pdf)\n", + }); + + expect(checkLayoutLinks(root)).toEqual([ + { file: "metadata/deck-inventory.md", ref: "decks/c&d.pdf" }, + ]); + }); + + // Only `& < > " '` and numeric references are decoded + // (a documented limit): any other named reference stays as written, so the + // destination names no file and the run fails closed instead of passing. + it("refuses, naming it, a destination holding a named reference it does not decode", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/a@b.pdf": "pdf", + "key-frames/visual-frames.md": "[deck](../source/decks/a@b.pdf)\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "frames/visual-frames.md", ref: "../metadata/decks/a@b.pdf" }, + ]); + expect(fs.existsSync(targetDir)).toBe(false); + }); + + // `:` is not allowed in a Windows file name. + it.skipIf(process.platform === "win32")( + "maps a code-span path whose real name ends in :12 before reading it as a line suffix", + async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/slide.pdf:12": "pdf", + "key-frames/vision-plan.md": "Deck: `source/decks/slide.pdf:12`.\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "metadata/decks/slide.pdf:12")).toBe("pdf"); + expect(read(targetDir, "frames/vision-plan.md")).toBe( + "Deck: `../metadata/decks/slide.pdf:12`.\n", + ); + }, + ); + + // `:` is not allowed in a Windows file name. + it.skipIf(process.platform === "win32")( + "maps a linked path whose real name ends in :12 before reading it as a line suffix", + async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/slide.pdf:12": "pdf", + "key-frames/vision-plan.md": [ + "Deck: [deck](source/decks/slide.pdf:12).", + 'deck', + "", + "[d]: source/decks/slide.pdf:12", + "", + ].join("\n"), + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/vision-plan.md")).toBe( + [ + "Deck: [deck](../metadata/decks/slide.pdf:12).", + 'deck', + "", + "[d]: ../metadata/decks/slide.pdf:12", + "", + ].join("\n"), + ); + }, + ); + + // Creating a symlink on Windows needs a privilege a test run may lack. + it.skipIf(process.platform === "win32")( + "refuses, naming it, a recorded temp dir that is itself a symlink", + async () => { + const { sliceDir, workDir, targetDir } = makeFixture(); + const link = path.join(makeDir("relayout-link-"), "work"); + fs.symlinkSync(workDir, link); + editState(sliceDir, (s) => ({ ...s, tempSession: { ...s.tempSession, workDir: link } })); + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(() => true); + + const result = await relayoutSlice({ sliceDir, targetDir }); + const message = stderr.mock.calls.map(([chunk]) => String(chunk)).join(""); + stderr.mockRestore(); + + expect(result.exitCode).toBe(1); + expect(message).toContain(`${link} is a symlink`); + expect(fs.existsSync(targetDir)).toBe(false); + }, + ); + + it("flags a missing destination with parentheses rather than a truncated one", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { "metadata/deck-inventory.md": "[deck](decks/slides_(v2).pdf)\n" }); + + expect(checkLayoutLinks(root)).toEqual([ + { file: "metadata/deck-inventory.md", ref: "decks/slides_(v2).pdf" }, + ]); + }); + + it("skips backslash-escaped delimiters inside quoted and parenthesized titles", () => { + expect( + inlineLinks( + '[a](a.md "say \\"hi\\"") [b](b.md \'it\\\'s\') [c](c.md (one \\) two)) [d](d.md)', + ).map(({ dest }) => dest), + ).toEqual(["a.md", "b.md", "c.md", "d.md"]); + }); + + // CommonMark ends a bare destination only at the ASCII space or a control. + it("keeps non-ASCII whitespace inside a bare destination", () => { + const nbsp = String.fromCodePoint(0xa0); + expect(inlineLinks(`[deck](../source/decks/a${nbsp}b.pdf)`).map(({ dest }) => dest)).toEqual([ + `../source/decks/a${nbsp}b.pdf`, + ]); + }); + + // CommonMark: a link title may span lines but not a blank line (one holding + // only spaces or tabs); with such a title the construct is not a link. + it("reads no link or definition whose title spans a blank line", () => { + expect(inlineLinks('[example](missing.md "title\n\ncontinued")')).toEqual([]); + expect(inlineLinks("[example](missing.md (title\n \t\ncontinued))")).toEqual([]); + expect(inlineLinks('[ok](kept.md "title\ncontinued")').map(({ dest }) => dest)).toEqual([ + "kept.md", + ]); + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/notes.md": [ + '[example](missing.md "title', + "", + 'continued")', + "", + "[d]: gone.md 'title", + " ", + "continued'", + "", + '[e]: lost.md "title', + 'continued"', + "", + ].join("\n"), + }); + + expect(checkLayoutLinks(root)).toEqual([{ file: "metadata/notes.md", ref: "lost.md" }]); + }); + + // CommonMark: a (...) title holds no unescaped parenthesis, so this is no link. + it("reads no link whose parenthesized title holds a bare parenthesis", () => { + expect(inlineLinks("[example](missing.md (title(with paren))")).toEqual([]); + }); + + // CommonMark: an autolink binds before link grouping, so its `](` is no link. + it("flags no link inside a URI autolink", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "metadata/notes.md": "[foo\n", + }); + + expect(checkLayoutLinks(root)).toEqual([]); + }); + + it("reads a link with a parenthesized title and rewrites its destination", async () => { + expect(inlineLinks("[x](a.md (Title)) and [y](b.md)").map(({ dest }) => dest)).toEqual([ + "a.md", + "b.md", + ]); + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "key-frames/visual-frames.md": "[t](../source/transcript.txt (The transcript))\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + "[t](../transcript/transcript.txt (The transcript))\n", + ); + }); + + it("reads a title on the next line after the destination", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "key-frames/visual-frames.md": '[t](../source/transcript.txt\n "The transcript")\n', + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + '[t](../transcript/transcript.txt\n "The transcript")\n', + ); + }); + + it("keeps a nested image's rewrite when the outer link names a dropped file", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "key-frames/visual-frames.md": + "[![slide](frames/title-slide.png)](../run-state/watch.json)\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + "![slide](key/title-slide.png) (run-state/watch.json, not retained in this copy)\n", + ); + }); + + it("reads an image nested in a link as two links with the right labels", () => { + expect( + inlineLinks("[![alt](a.png)](b.md)").map(({ dest, label }) => ({ dest, label })), + ).toEqual([ + { dest: "a.png", label: "alt" }, + { dest: "b.md", label: "![alt](a.png)" }, + ]); + }); + + it("reads an outer link whose label holds a link as literal text", () => { + expect(inlineLinks("[outer [real](ok.md)](missing.md)").map(({ dest }) => dest)).toEqual([ + "ok.md", + ]); + }); + + it("leaves the literal outer text of a link holding a link untouched", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "key-frames/visual-frames.md": + "[outer [t](../source/transcript.txt)](missing.md)\n" + + "[outer [t](../source/transcript.txt)](../run-state/watch.json)\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + "[outer [t](../transcript/transcript.txt)](missing.md)\n" + + "[outer [t](../transcript/transcript.txt)](../run-state/watch.json)\n", + ); + }); + + it("rewrites single-quoted and unquoted HTML src and href", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "key-frames/visual-frames.md": + " t\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + " t\n", + ); + }); + + it("rewrites percent-encoded #, ? and space in path segments, keeping the encoding", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "research/findings/notes#1.md": "n\n", + "research/findings/what?.md": "w\n", + "key-frames/visual-frames.md": + "[a](../research/findings/notes%231.md) [b](../research/findings/what%3F.md#top)\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + "[a](../analysis/research/findings/notes%231.md) [b](../analysis/research/findings/what%3F.md#top)\n", + ); + }); + + it("decodes HTML character references in src and href, re-encoding & as the original did", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/s1/a&b.pdf": "pdf", + "key-frames/visual-frames.md": + 'deck \n', + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + 'deck \n', + ); + const root = makeDir("relayout-check-"); + writeTree(root, { "frames/visual-frames.md": '\n' }); + expect(checkLayoutLinks(root)).toEqual([ + { file: "frames/visual-frames.md", ref: "key/x&y.png" }, + ]); + }); + + it.skipIf(process.platform === "win32")( + "escapes a rewritten src/href value for its quoting", + async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + 'source/decks/s1/a"b.pdf': "pdf", + "source/decks/s1/it's.pdf": "pdf", + "source/decks/s1/a&b.pdf": "pdf", + "key-frames/visual-frames.md": [ + 'dq', + "sq", + "bare", + 'raw amp', + "", + ].join("\n"), + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + [ + 'dq', + "sq", + 'bare', + 'raw amp', + "", + ].join("\n"), + ); + }, + ); + + it.skipIf(process.platform === "win32")( + "maps a real file whose name looks like a placeholder before treating it as one", + async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "research/findings/report-{draft}.pdf": "pdf", + "research/findings/report-.html": "html", + "recommendations/questions.md": + "Draft `research/findings/report-{draft}.pdf`, final `research/findings/report-.html`, template `research/findings/report-{name}.pdf`.\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "analysis/recommendations/questions.md")).toBe( + "Draft `../research/findings/report-{draft}.pdf`, final `../research/findings/report-.html`, template `research/findings/report-{name}.pdf`.\n", + ); + }, + ); + + it("flags a definition or src naming a dropped real file whose name looks like a placeholder", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "run-state/report-{draft}.md": "draft", + "research/claim-inventory.md": + '[state]: ../run-state/report-{draft}.md\n\nSee [state] and .\n', + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "analysis/research/claim-inventory.md", ref: "../run-state/report-{draft}.md" }, + { file: "analysis/research/claim-inventory.md", ref: "../run-state/report-{draft}.md" }, + ]); + }); + + it("flags a definition naming a dropped slice file that matches an allowed miss", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "AGENTS.md": "slice notes", + "research/claim-inventory.md": "[rules]: AGENTS.md\n\nSee [rules].\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "analysis/research/claim-inventory.md", ref: "AGENTS.md" }, + ]); + }); + + it("matches the AGENTS.md allowed miss exactly and the directories by whole segment", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "research/claim-inventory.md": + "[a](./AGENTS.md#rules) [b](AGENTS.md) [c](AGENTS.md.old) [d](docs-old/x.md) [e](docs/x.md)\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "analysis/research/claim-inventory.md", ref: "AGENTS.md.old" }, + { file: "analysis/research/claim-inventory.md", ref: "docs-old/x.md" }, + ]); + }); + + it.skipIf(process.platform === "win32")( + "skips backslash-escaped < and > when finding the end of an angle destination", + async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "research/findings/a>b.md": "a\n", + "key-frames/visual-frames.md": "[x](<../research/findings/a\\>b.md>)\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + "[x](<../analysis/research/findings/a\\>b.md>)\n", + ); + }, + ); + + it.skipIf(process.platform === "win32")( + "reads a reference definition's angle destination past a backslash-escaped >", + async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/a>b.pdf": "pdf", + "key-frames/vision-plan.md": "[deck][deck]\n\n[deck]: <../source/decks/a\\>b.pdf>\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/vision-plan.md")).toBe( + "[deck][deck]\n\n[deck]: <../metadata/decks/a\\>b.pdf>\n", + ); + }, + ); + + // CommonMark: `>` decodes to `>`, which written raw would close the + // angle destination early. + it.skipIf(process.platform === "win32")( + "escapes a > decoded from a character reference in a rewritten angle destination", + async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/a>b.pdf": "pdf", + "key-frames/vision-plan.md": "[deck](<../source/decks/a>b.pdf>)\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/vision-plan.md")).toBe( + "[deck](<../metadata/decks/a\\>b.pdf>)\n", + ); + }, + ); + + it("escapes an unbalanced parenthesis in a rewritten bare destination", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/a)b.pdf": "pdf", + "key-frames/vision-plan.md": + "[deck](<../source/decks/a)b.pdf>) [bare](../source/decks/a\\)b.pdf)\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/vision-plan.md")).toBe( + "[deck](<../metadata/decks/a)b.pdf>) [bare](../metadata/decks/a\\)b.pdf)\n", + ); + }); + + // Round trip against the CommonMark reading inlineLinks applies: the written + // destination spans exactly what was written and decodes to the new path. + it.each( + ["a>b.pdf", "a [false, true].map((angle) => [name, angle])) + .flatMap((row) => [false, true].map((references) => [...row, references])), + )( + "writes %s (angle %s, references %s) so it parses back unchanged", + (name, angle, references) => { + const newPath = `../metadata/decks/${name}`; + const written = encodeDestination(newPath, { + angle: Boolean(angle), + escapedChars: "", + references: Boolean(references), + }); + const [link] = inlineLinks(`[x](${written})`); + + expect(link?.dest).toBe(newPath); + expect(link?.destEnd).toBe(4 + written.length); + }, + ); + + it("link-checks a spaced code-span path with a raw # as one whole path", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/My #1.pdf": "pdf", + "key-frames/vision-plan.md": "Deck: `../source/decks/My #1.pdf`.\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.unresolved).toEqual([]); + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/vision-plan.md")).toBe( + "Deck: `../metadata/decks/My #1.pdf`.\n", + ); + }); + + it("maps a real path holding $ and skips only complete placeholder syntax", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/budget$2026.pdf": "pdf", + "key-frames/visual-frames.md": "[b](../source/decks/budget$2026.pdf)\n", + "recommendations/questions.md": [ + "Missing: `research/budget$2025.md`.", + "Placeholders: `/research/x.md` `${ROOT}/research/x.md` `{tmp}/x.png` `research/*.md`.", + "", + ].join("\n"), + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "analysis/recommendations/questions.md", ref: "../research/budget$2025.md" }, + ]); + fs.writeFileSync( + path.join(sliceDir, "recommendations", "questions.md"), + "Placeholders: `/research/x.md` `${ROOT}/research/x.md` `research/*.md`.\n", + ); + const fixed = await relayoutSlice({ sliceDir, targetDir }); + expect(fixed.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + "[b](../metadata/decks/budget$2026.pdf)\n", + ); + }); + + it("link-checks a spaced code span its rewrite placed through the unknown-child fallback", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "recommendations/questions.md": "Notes: `research/findings/My Missing Notes.md`.\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { + file: "analysis/recommendations/questions.md", + ref: "../research/findings/My Missing Notes.md", + }, + ]); + }); + + it("reads src and href only inside HTML tags, never in prose", async () => { + const body = [ + "Set href=../research/gone.md or src='../source/transcript.txt' in the config.", + 't', + "", + ].join("\n"); + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": body }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + body.replace( + '', + '', + ), + ); + }); + + it("does not treat an invalid HTML-looking wrapper as a tag around a real link", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": "\n" }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + "\n", + ); + }); + + it("rewrites and checks upper-case HTML attribute names", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": '\n' }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe('\n'); + const root = makeDir("relayout-check-"); + writeTree(root, { "frames/visual-frames.md": "x\n" }); + expect(checkLayoutLinks(root)).toEqual([ + { file: "frames/visual-frames.md", ref: "key/gone.png" }, + ]); + }); + + it("reads an HTML href wrapped across lines without its tab, LF and CR", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": 't\n' }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + 't\n', + ); + const root = makeDir("relayout-check-"); + writeTree(root, { "frames/visual-frames.md": '\n' }); + expect(checkLayoutLinks(root)).toEqual([ + { file: "frames/visual-frames.md", ref: "key/gone.png" }, + ]); + }); + + it("flags a single-quoted HTML src that does not resolve", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { "frames/visual-frames.md": "\n" }); + + expect(checkLayoutLinks(root)).toEqual([ + { file: "frames/visual-frames.md", ref: "key/gone.png" }, + ]); + }); + + it("reads no link, definition or backticked path inside an HTML tag or comment", async () => { + const body = [ + 'x', + "t", + "", + "", + "", + ].join("\n"); + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": body }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe(body); + const root = makeDir("relayout-check-"); + writeTree(root, { + "frames/visual-frames.md": [ + 'x', + "t", + "", + "", + ].join("\n"), + }); + expect(checkLayoutLinks(root)).toEqual([]); + }); + + it("reads src and href only as attribute names, never inside another attribute's value", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "key-frames/visual-frames.md": + 't\n', + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + 't\n', + ); + const root = makeDir("relayout-check-"); + writeTree(root, { + "frames/visual-frames.md": + "\"href='key/gone.png'\"\n", + }); + expect(checkLayoutLinks(root)).toEqual([ + { file: "frames/visual-frames.md", ref: "key/missing.png" }, + ]); + }); + + it("reads a reference definition only when the rest of its line is blank or a title", async () => { + const body = "[note]: ../source/transcript.txt is illustrative\n"; + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": body }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe(body); + const root = makeDir("relayout-check-"); + writeTree(root, { + "frames/visual-frames.md": [ + "[note]: source/missing.txt is illustrative", + '[t]: key/gone.png "A title" ', + "", + ].join("\n"), + }); + expect(checkLayoutLinks(root)).toEqual([ + { file: "frames/visual-frames.md", ref: "key/gone.png" }, + ]); + }); + + it.each([ + ["null", "null"], + ["an array", "[]"], + ["corrupt JSON", "{"], + ])("refuses, before staging, an info JSON holding %s", async (_label, body) => { + const { sliceDir, targetDir, workDir } = makeFixture(); + fs.writeFileSync(path.join(workDir, `${VIDEO_ID}.info.json`), body); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.readdirSync(path.dirname(targetDir))).toEqual([]); + }); + + it("applies CommonMark's label rules: line endings yes, blank lines, brackets, blanks and 1000 characters no", async () => { + const body = [ + "[deck", + "slides]: ../source/transcript.txt", + "", + "[long]: ../source/transcript.txt", + `[${"x".repeat(999)}]: ../source/transcript.txt`, + `[${"x".repeat(1000)}]: ../research/gone.md`, + "[ ]: ../research/gone.md", + "[a[b]: ../research/gone.md", + "", + "[split", + "", + "label]: ../research/gone.md", + "", + "[text", + " ", + "more](../research/gone.md)", + "", + ].join("\n"); + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": body }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + body.replaceAll("]: ../source/transcript.txt", "]: ../transcript/transcript.txt"), + ); + }); + + it("reads labels holding escaped brackets in definitions and inline links", async () => { + const body = [ + "[deck\\]][d] and [a\\]b](../source/transcript.txt)", + "", + "[deck\\]]: ../source/transcript.txt", + "[x\\[y]: ../research/gone.md", + "", + ].join("\n"); + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/visual-frames.md": body }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "frames/visual-frames.md", ref: "../analysis/research/gone.md" }, + ]); + fs.writeFileSync( + path.join(sliceDir, "key-frames", "visual-frames.md"), + body.replace("[x\\[y]: ../research/gone.md\n", ""), + ); + const fixed = await relayoutSlice({ sliceDir, targetDir }); + expect(fixed.exitCode).toBe(0); + expect(read(targetDir, "frames/visual-frames.md")).toBe( + [ + "[deck\\]][d] and [a\\]b](../transcript/transcript.txt)", + "", + "[deck\\]]: ../transcript/transcript.txt", + "", + ].join("\n"), + ); + }); + + it("reads a reference definition whose destination follows a line ending", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/slides.pdf": "pdf", + "key-frames/vision-plan.md": "[deck][d]\n\n[d]:\n ../source/decks/slides.pdf\n", + "recommendations/questions.md": "[g][g]\n\n[g]:\n ../research/gone.md\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "analysis/recommendations/questions.md", ref: "../research/gone.md" }, + ]); + fs.rmSync(path.join(sliceDir, "recommendations", "questions.md")); + const fixed = await relayoutSlice({ sliceDir, targetDir }); + expect(fixed.exitCode).toBe(0); + expect(read(targetDir, "frames/vision-plan.md")).toBe( + "[deck][d]\n\n[d]:\n ../metadata/decks/slides.pdf\n", + ); + }); + + it("checks a bare-file definition destination carrying a query, a fragment or both", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "research/claim-inventory.md": + "[notes]: missing.md?download=1\n[both]: gone.md?download=1#top\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved.map((entry) => entry.ref)).toEqual([ + "missing.md?download=1", + "gone.md?download=1#top", + ]); + }); + + it("checks an extensionless definition destination", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "analysis/README": "readme", + "analysis/notes.md": "[guide]: missing-thing\n[readme]: README\n", + }); + + expect(checkLayoutLinks(root)).toEqual([{ file: "analysis/notes.md", ref: "missing-thing" }]); + }); + + it("reads no angle destination broken by a backslash before a line ending", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "frames/visual-frames.md": [ + "[example]()", + "", + "[d]: ", + "", + ].join("\n"), + }); + + expect(checkLayoutLinks(root)).toEqual([]); + }); + + it("flags a reference-style definition to a path that does not exist", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "research/claim-inventory.md": "[gone][g]\n\n[g]: findings/gone.md\n" }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "analysis/research/claim-inventory.md", ref: "findings/gone.md" }, + ]); + }); + + it("treats an outside path whose directory shares the slice's name as outside", async () => { + const { sliceDir, targetDir } = makeFixture(); + const slug = path.basename(sliceDir); + const outsideRef = `../../repos/${slug}/source/transcript.txt`; + fs.writeFileSync(path.join(sliceDir, "research", "claim-inventory.md"), `[x](${outsideRef})\n`); + const pointedAt = path.resolve(sliceDir, "research", outsideRef); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + const link = /\]\(([^)]+)\)/.exec(read(targetDir, "analysis/research/claim-inventory.md"))?.[1]; + expect(path.resolve(targetDir, "analysis/research", String(link))).toBe(pointedAt); + }); + + it("accepts an untimed frame-times.json record (timestampSec null, as the writer emits)", async () => { + const { sliceDir, targetDir, framesDir } = makeFixture(); + fs.writeFileSync( + path.join(framesDir, "frame-times.json"), + JSON.stringify({ "scene_0001.png": { timestampSec: null, timestampSource: null } }), + ); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + }); + + it.each([ + [ + "..// from the slice root", + "RESEARCH.md", + (/** @type {string} */ slug) => `../${slug}/`, + ], + ["../ from a lane file", "research/claim-inventory.md", () => "../"], + ])( + "rewrites a reference to the slice dir itself (%s) as split text", + async (_label, file, ref) => { + const { sliceDir, targetDir } = makeFixture(); + const slug = path.basename(sliceDir); + fs.writeFileSync(path.join(sliceDir, file), `Whole slice: [slice](${ref(slug)}).\n`); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + const target = file === "RESEARCH.md" ? "analysis/RESEARCH.md" : `analysis/${file}`; + expect(read(targetDir, target)).toBe( + `Whole slice: slice (${slug}, split across analysis/, frames/, metadata/, transcript/ in this copy).\n`, + ); + }, + ); + + it("reads a path that leaves and re-enters the slice as the slice's", async () => { + const { sliceDir, targetDir } = makeFixture(); + const slug = path.basename(sliceDir); + fs.writeFileSync(path.join(sliceDir, "RESEARCH.md"), `[t](../${slug}/source/transcript.txt)\n`); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "analysis/RESEARCH.md")).toBe("[t](../transcript/transcript.txt)\n"); + }); + + it("rewrites a tag-like angle link destination rather than masking it as HTML", async () => { + const { sliceDir, targetDir } = makeFixture(); + fs.writeFileSync( + path.join(sliceDir, "RESEARCH.md"), + "[sources]() beside tag.\n", + ); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "analysis/RESEARCH.md")).toBe( + "sources (source, split across analysis/, metadata/, transcript/ in this copy)" + + " beside tag.\n", + ); + }); + + it("checks tag-like angle link and definition destinations", async () => { + const { sliceDir, targetDir } = makeFixture(); + fs.writeFileSync( + path.join(sliceDir, "RESEARCH.md"), + "[x]() beside tag.\n\n[s]: \n", + ); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + // A definition has no prose to take the split text, so the check flags it. + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "analysis/RESEARCH.md", ref: "missing" }, + { file: "analysis/RESEARCH.md", ref: "source" }, + ]); + }); + + it("reads the .work// form only with the slice's own work path", async () => { + const { sliceDir: original, targetDir } = makeFixture(); + const slug = path.basename(original); + const sliceDir = path.join(makeDir("relayout-root-"), ".work", "youtube-watch", slug); + fs.mkdirSync(path.dirname(sliceDir), { recursive: true }); + fs.renameSync(original, sliceDir); + fs.writeFileSync( + path.join(sliceDir, "recommendations", "questions.md"), + [ + `Own: \`.work/youtube-watch/${slug}/source/transcript.txt\`.`, + `Other epic: \`.work/other-epic/${slug}/source/transcript.txt\`.`, + "", + ].join("\n"), + ); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "analysis/recommendations/questions.md")).toBe( + [ + "Own: `../../transcript/transcript.txt`.", + `Other epic: \`.work/other-epic/${slug}/source/transcript.txt\`.`, + "", + ].join("\n"), + ); + }); + + it("keeps a link that leaves the slice pointing at the same file from its deeper location", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "research/claim-inventory.md": "[other](../../other-slice/RESEARCH.md)\n" }, + }); + const sibling = path.join(path.dirname(sliceDir), "other-slice", "RESEARCH.md"); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + const link = /\]\(([^)]+)\)/.exec(read(targetDir, "analysis/research/claim-inventory.md"))?.[1]; + expect(path.resolve(targetDir, "analysis/research", String(link))).toBe(sibling); + }); + + it("leaves another repo's lane-named path in RESEARCH.md alone", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "RESEARCH.md": "Upstream: `research/deepening/html-report.md:92` in mattpocock/skills.\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + }); + + it.each([ + [ + "an empty caption track", + (/** @type {Record} */ d) => + fs.writeFileSync(path.join(d.workDir, `${VIDEO_ID}.en.vtt`), ""), + ], + [ + "an empty frame image", + (/** @type {Record} */ d) => + fs.writeFileSync(path.join(d.framesDir, "anchor_00012000_0001.png"), ""), + ], + ])("refuses a temp session with %s", async (_label, damage) => { + const { sliceDir, targetDir, workDir, framesDir } = makeFixture(); + damage({ workDir, framesDir }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(fs.existsSync(targetDir)).toBe(false); + }); +}); + +describe("checkLayoutLinks", () => { + it("stays quiet on URLs, placeholders, other-repository paths and resolvable paths", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "transcript/transcript.txt": "", + "analysis/RESEARCH.md": [ + "[docs](https://code.claude.com/docs/en/hooks) and .", + "Paths: `plugins/knowledge/SKILL.md`, `docs/adr/0001.md`, `templates/x.md`, `.claude/rules/a.md`.", + "Links: [skill](plugins/knowledge/SKILL.md), [adr](docs/adr/0001.md), [rules](AGENTS.md).", + "[agents](../../../AGENTS.md), `AGENTS.md`, `mattpocock/skills`, `/code-review`.", + "Temp: `{tmp}/video-frames-x/scene_0001.png`, `/report-.html`.", + "Outside the slice: `../../other-repo/README.md`. Inside: `../transcript/transcript.txt`.", + "[anchor](#verdicts)", + "", + ].join("\n"), + }); + + expect(checkLayoutLinks(root)).toEqual([]); + }); + + it("checks a ~-prefixed relative name and skips only home paths", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "analysis/RESEARCH.md": "[notes](~notes.md), [home](~/x.md), [user](~someone/x.md).\n", + }); + + expect(checkLayoutLinks(root)).toEqual([{ file: "analysis/RESEARCH.md", ref: "~notes.md" }]); + }); + + it("checks a ..-prefixed name as inside the target", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { "README.md": "[notes](..notes.md)\n" }); + + expect(checkLayoutLinks(root)).toEqual([{ file: "README.md", ref: "..notes.md" }]); + }); + + it("flags an unresolved path under a slice lane whose directory the slice had", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "recommendations/questions.md": + "Typo: `source/missing.txt`. Upstream: `research/deepening/html-report.md:92`.\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(1); + expect(result.unresolved).toEqual([ + { file: "analysis/recommendations/questions.md", ref: "source/missing.txt" }, + ]); + }); + + it("rewrites every backticked path form and leaves prose spans alone", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "source/decks/s1/My Slides.pdf": "pdf", + "recommendations/questions.md": [ + "Spaced: `../source/decks/s1/My Slides.pdf`.", + "Double: `` source/transcript.txt `` and ``a `tick` in source/transcript.txt``.", + 'Quoted: `"source/transcript.txt"`.', + "Line and column: `source/transcript.txt:3:5`.", + "Backslash: `source\\transcript.txt`.", + "Trailing slash: `research/findings/`.", + "Dot prefix: `./menu.md`.", + "Lane only: `research`.", + "Prose: `run the source / research step`.", + "", + ].join("\n"), + "recommendations/menu.md": "# Menu\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "analysis/recommendations/questions.md")).toBe( + [ + "Spaced: `../../metadata/decks/s1/My Slides.pdf`.", + "Double: `` ../../transcript/transcript.txt `` and ``a `tick` in source/transcript.txt``.", + 'Quoted: `"../../transcript/transcript.txt"`.', + "Line and column: `../../transcript/transcript.txt:3:5`.", + "Backslash: `../../transcript/transcript.txt`.", + "Trailing slash: `../research/findings/`.", + "Dot prefix: `menu.md`.", + "Lane only: `research`.", + "Prose: `run the source / research step`.", + "", + ].join("\n"), + ); + }); + + it("rewrites the exact slice-root aliases as split text", async () => { + const { sliceDir: original, targetDir } = makeFixture(); + const slug = path.basename(original); + const sliceDir = path.join(makeDir("relayout-root-"), ".work", "youtube-watch", slug); + fs.mkdirSync(path.dirname(sliceDir), { recursive: true }); + fs.renameSync(original, sliceDir); + fs.writeFileSync( + path.join(sliceDir, "recommendations", "questions.md"), + `\`SLICE\` \`SLICE/\` \`.work/youtube-watch/${slug}\` \`.work/youtube-watch/${slug}/\`\n`, + ); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + const split = `${slug} (split across analysis/, frames/, metadata/, transcript/ in this copy)`; + expect(read(targetDir, "analysis/recommendations/questions.md")).toBe( + `${split} ${split} ${split} ${split}\n`, + ); + }); + + it("does not check a reference definition whose destination is prose", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "research/claim-inventory.md": "[0:33]: implementing a whole spec in one session\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + }); + + it("says so for a path below a lane folder this slice never produced", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/vision-plan.md": "Slides: `source/decks/s1/slides.pdf`.\n" }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/vision-plan.md")).toBe( + "Slides: source/decks/s1/slides.pdf (none in this slice).\n", + ); + }); + + it("rewrites and checks Markdown files whatever the extension's case", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { + "research/notes.MD": "See `source/transcript.txt`.\n", + "research/more.markdown": "See [t](../source/transcript.txt).\n", + }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "analysis/research/notes.MD")).toBe( + "See `../../transcript/transcript.txt`.\n", + ); + expect(read(targetDir, "analysis/research/more.markdown")).toBe( + "See [t](../../transcript/transcript.txt).\n", + ); + const root = makeDir("relayout-check-"); + writeTree(root, { "frames/log.MARKDOWN": "[gone](key/gone.png)\n" }); + expect(checkLayoutLinks(root)).toEqual([{ file: "frames/log.MARKDOWN", ref: "key/gone.png" }]); + }); + + it("says so when prose names a lane folder this slice never produced", async () => { + const { sliceDir, targetDir } = makeFixture({ + extraFiles: { "key-frames/vision-plan.md": "No deck in the `source/decks/` lane.\n" }, + }); + + const result = await relayoutSlice({ sliceDir, targetDir }); + + expect(result.exitCode).toBe(0); + expect(read(targetDir, "frames/vision-plan.md")).toBe( + "No deck in the source/decks (none in this slice) lane.\n", + ); + }); + + it("treats any lane-named path as the slice's when no slice directories are given", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { "analysis/RESEARCH.md": "See `run-state/watch.json`.\n" }); + + expect(checkLayoutLinks(root)).toEqual([ + { file: "analysis/RESEARCH.md", ref: "run-state/watch.json" }, + ]); + }); + + it("flags a bare extracted-frame name but not other bare file names", () => { + const root = makeDir("relayout-check-"); + writeTree(root, { + "frames/all/scene_0001.png": "png", + "frames/visual-frames.md": + "Kept: `all/scene_0001.png`. Dangling: `anchor_00012000_0003.png`. Prose: `SKILL.md`, `README.md`.\n", + }); + + expect(checkLayoutLinks(root)).toEqual([ + { file: "frames/visual-frames.md", ref: "anchor_00012000_0003.png" }, + ]); + }); +});