diff options
Diffstat (limited to 'notes')
| -rw-r--r-- | notes/gpui-port.md | 98 | ||||
| -rw-r--r-- | notes/roadmap.md | 30 |
2 files changed, 118 insertions, 10 deletions
diff --git a/notes/gpui-port.md b/notes/gpui-port.md new file mode 100644 index 00000000..853d65d3 --- /dev/null +++ b/notes/gpui-port.md @@ -0,0 +1,98 @@ +# GPUI desktop port — architecture & plan + +Captured 2026-07-28. Owner decisions this session: + +- **Full 1:1 port** of the editor to GPUI. Linux-first; macOS/Windows not targeted for now, BSD parked (GPUI support unproven). +- **In-process ffmpeg** via `ffmpeg-next` for decode AND encode. No subprocess. +- **Licensing is not a constraint** — x264 (GPL) is the H.264 encoder; quality over AppImage-ability. The documented OpenH264 dead end (ignores bitrate, undershoots ~10x) is bypassed entirely. +- Web version stays and keeps working; the browser editor remains a feature. + +## Why the port + +Browser inconsistency (WebGPU quirks, encoder behavior varying by browser) +keeps producing walls. A native app gives one controlled runtime: wgpu + +ffmpeg, one decoder, one encoder, one compositor. + +## What "backend" means here + +There is no server worth porting: the Next.js backend is an auth scaffold +(no flows implemented), a health route, and a sounds proxy (sounds are +feature-flagged off). The real "backend" is the offline project store +(IndexedDB) — on desktop it becomes a plain filesystem layout per project. +So: no Rust server, no API. Everything lives in the app process. + +## Crate map + +Existing, reused natively as-is (already target-agnostic): + +- `rust/crates/time` — media time, frame rate, timecode +- `rust/crates/compositor` + `effects` + `masks` + `gpu` — wgpu compositor, + WGSL effect pipeline, SDF mask feathering. The web drives this through + wasm; desktop links it directly. + +New crates: + +- `scene` — project/scene graph model, serde, `.ocp` project.json compat + (round-trip proven against the web's SerializedProject shape). +- `timeline` — tracks, elements, placement, snapping, split/trim/resize, + ripple, grouping. Pure logic, no UI. +- `animation` — keyframes, bezier interpolation, channel sampling. +- `commands` — undo/redo command stack (ports `apps/web/src/commands`). +- `media` — ffmpeg-next demux/decode, frame pipeline to GPU textures, + thumbnails, waveform extraction. +- `audio` — decode, mixing, rate stretching, mastering. +- `encode` — export pipeline: compositor frames → x264/VP9, muxing. +- `transfer` — `.ocp` zip container build/parse (cross-compat with the web's + fflate archives proven in both directions). +- `storage` — project dirs on disk, per-project media store, autosave, + `.ocp` import/export (replaces IndexedDB; stays offline-first). +- `text` — text layout/measure/raster via cosmic-text (ports + `apps/web/src/text`). +- `graphics` — gradient parser + shape raster via tiny-skia (ports + `apps/web/src/graphics` + `gradients`). +- `effect-defs` — effect definitions registry: params → WGSL pass + descriptors (ports `apps/web/src/effects/definitions`). +- `mask-defs` — mask shape renderers via tiny-skia (ports + `apps/web/src/masks/builtin` + `freeform`). +- `renderer` — scene → node tree → resolve at time t → compositor + FrameDescriptor (ports `apps/web/src/services/renderer`). Integration hub; + starts once the leaf crates land. +- `playback` — clock, frame scheduling, audio sync. + +`apps/desktop` (GPUI) holds UI only; all logic lives in crates so the web +version can adopt them later via wasm where it makes sense. + +## Compositor → GPUI bridge + +GPUI does not expose its internal GPU device. v1 approach: the compositor +owns its own wgpu device, renders the frame to an offscreen texture, +reads back, and paints it as a `gpui::Image` in a custom element. ~8 MB +per 1080p frame — fine for preview; revisit if profiling says otherwise. + +## Milestones (each runnable) + +1. **Environment** — nix devshell with rust + GPUI native deps + ffmpeg dev. +2. **Domain model** — `scene` crate: full project/scene/element model + ported from `apps/web/src/timeline/types.ts` + scene managers, `.ocp` + round-trip compat proven against web-generated archives. +3. **Timeline core** — `timeline`, `animation`, `commands` crates with + ported unit tests (web test suite is the behavior spec). +4. **Media** — `media`/`audio` crates: decode → texture, waveform, + thumbnails. +5. **Compositor bridge** — scene resolve → wgpu frame → GPUI preview. +6. **GPUI UI** — projects screen, editor layout, timeline UI (drag, trim, + split, snap, ripple), preview overlays/selection, all panels. +7. **Export** — `encode` crate + dialog, x264 quality ladder. +8. **Features** — versions, voiceover (cpal), clipboard, keybindings, + `.ocp` import/export. +9. **Packaging** — .desktop entry, binary distribution. AppImage optional. + +## Port rules + +- Web behavior is the spec. Where the web has unit tests, port the test + with the code. +- Flagged-off web features (sounds, stickers, effects domains) are NOT + ported — they are slated for deletion upstream of us. +- `.ocp` archives must round-trip between web and desktop. +- No parallel divergence in shared crates: `compositor`/`time` changes must + keep compiling for `wasm32`. diff --git a/notes/roadmap.md b/notes/roadmap.md index c886edeb..fc89bb46 100644 --- a/notes/roadmap.md +++ b/notes/roadmap.md @@ -1,10 +1,21 @@ # Roadmap & session notes Working notes for the fork's development direction. Captured 2026-07-26, -updated 2026-07-27 (roadmap feature slices implemented, last commit -`9c3b788b`). +updated 2026-07-28 (GPUI desktop port greenlit — see notes/gpui-port.md). -## Done +## Done (2026-07-28 session) + +- Freeze-duration prompt removed; freeze frame uses a fixed 3s (store, dialog, and test deleted). +- Ripple-extend: dragging the right edge of a static element (image/freeze still/text, i.e. no `sourceDuration`) past its neighbor pushes the whole right-hand chain on that track. Same undo step as the resize; live during drag. Video/audio keep clamping. Core: `timeline/group-resize/compute-resize.ts` + `rightPushChain` from `resize-controller.ts`. +- Timeline filmstrip: video elements render real per-position thumbnails, density adapts to zoom (power-of-two interval snapping). New `services/thumbnail-cache/` (LRU, batched mediabunny decode) + `timeline/components/video-filmstrip.tsx`. Images unchanged. Verified in browser by agent (distinct tiles, zoom density, viewport clipping, cache clear on media delete). +- Export hardening (`services/renderer/scene-exporter.ts`, `export-encoding.ts`): bitrate is fps-aware (`w*h*fps*bpp`, high = 0.2 bpp → ~12.4 Mbps at 1080p30, was ~6 fps-blind); keyframe every 2s; export canvas rounded to even dims. A capture-timing change (wait past WebGPU present before snapshot) was tried and REVERTED the same day — the canvas is blank after present, export came out black. Synchronous same-task snapshot is the correct behavior for WebGPU canvases; the temporal-ghosting hypothesis is dead. Colorspace tagging: no API exists (verified mediabunny + lib.dom) — BT.709 vs 601 is Chrome's encoder's call. +- MP4 export: ffmpeg.wasm + libx264 was implemented (chunked rawvideo → x264 CRF ladder) and then REVERTED at the owner's request after two wasm-side bugs (x264 auto-threading spawns 28 pthreads and hangs the worker — `-threads 4` fixes it; `@ffmpeg/ffmpeg` `writeFile` transfers/detaches the input ArrayBuffer, so chunk buffers must be single-use). The investigation notes stay here in case the idea is revisited; WebM/VP9 export is the owner's daily path and works well. MP4 stays on the mediabunny/WebCodecs avc path (which undershoots bitrate on this machine — OpenH264 ignores bitrateMode, has no quantizer mode, no hardware encoder exists; probe-verified 2026-07-28). +- Encoder investigation (2026-07-28, probe-verified in the user's Helium): NO hardware H.264 encode exists on the machine; Chrome's OpenH264 ignores `bitrateMode` (byte-identical VBR/CBR output), has no AVC quantizer mode (`isConfigSupported` false), and undershoots to ~0.6 Mbps on real content regardless of configured bitrate (6-50 Mbps). WebCodecs H.264 is a dead end for quality export; x264 via wasm is the only in-browser path. The fps-aware `computeVideoBitrate` still serves the WebM path. +- Next dev overlay disabled (`devIndicators: false` in `apps/web/next.config.ts`). +- Gates green: bun test 316 pass / 1 skip, tsc, eslint on touched files, `bun build:web`. +- Human verification still owed: freeze + ripple-extend click-through by owner. + +## Done (2026-07-26/27 session) - Freeze frame at playhead (snowflake button), composite canvas capture. - Freeze frame is a single undo step (`BatchCommand` + `ShiftSplitRemainderCommand`). @@ -69,13 +80,12 @@ work. The GPUI desktop app sidesteps it entirely (native GPU). 5. ~~Voice-over recording~~ done. 6. ~~Git-like project versioning~~ done (v1). v2 candidates: diff view, branches, content-addressed media dedup, shipping history inside `.ocp`. -7. **Self-contained binary, first-class Linux + BSD.** `apps/desktop` (GPUI) - is the vehicle but is a hello-world scaffold today; porting the editor is - a large effort. Caveat: GPUI's Linux support is young; BSD support is - unproven — spike before committing. Browser version stays regardless - (editing in a browser is a feature, not a compromise). Distribution: - AppImage/GitHub releases first; Flathub is effectively closed to us (June - 2026 policy bans AI-assisted code in new submissions). +7. ~~Self-contained binary, first-class Linux + BSD.~~ **GREENLIT + 2026-07-28: full 1:1 GPUI port, Linux-first.** Decisions: in-process + ffmpeg (`ffmpeg-next`) for decode+encode, x264 for H.264 (licensing + waived, quality first), no server backend (project store moves to + filesystem), web version stays. BSD parked (GPUI unproven there). + Full plan: notes/gpui-port.md. ## Deprioritized / parked |