update dotfiles & WORKFLOW.org

1 files changed, 172 insertions, 70 deletions

diff --git a/WORKFLOW.org b/WORKFLOW.org
index 95fdb25..af575e4 100644
--- a/WORKFLOW.org
+++ b/WORKFLOW.org
@@ -3,40 +3,37 @@
 
 * Overview
 
-This repository contains custom FreeBSD ports. The goal is to build and test them in an isolated jail using the standard FreeBSD tooling rather than ad-hoc =make DISTFILES=...= invocations.
+This repository is a custom FreeBSD ports overlay. Ports that live here are built and tested in an isolated poudriere jail; everything they depend on from the official tree is fetched as an official binary package. Only overlay ports are ever compiled from source.
 
-* Tool choice: poudriere
+Tool choice, for the record:
+- *poudriere* :: builds ports in clean jails, supports overlays, bulk builds and single-port testing (=testport=). The standard tool for this job.
+- *synth* :: similar goals, but no overlay support.
+- *portmaster* / plain =make install= :: no build isolation.
 
-After comparing the common FreeBSD port building tools:
-
-- *poudriere* :: The standard tool for building and testing ports in isolated jails. Supports ZFS, overlays, reproducible bulk builds, and =testport= for individual ports. Best fit for this workflow.
-- *synth* :: A lighter alternative, but less commonly used for custom port trees and lacks the overlay/jail flexibility of poudriere.
-- *portmaster* / plain =make= :: Good for installing packages on a live system, but do not provide isolated build environments, so they do not solve the "rebuild everything from source" problem.
+One privilege rule applies everywhere: poudriere always runs via =doas= (it manages jails and mounts; =doas.conf= already permits it without a password), and host-side =make= in a port directory never runs as root.
 
 * Layout
 
 #+begin_example
 ~/ports
-├── devel/
+├── devel/               # custom development ports
 │   ├── bash-language-server/
 │   ├── just-lsp/
 │   ├── nixd/
 │   ├── py-rassumfrassum/
 │   ├── pyrefly/
 │   └── yaml-language-server/
-├── distfiles/          # shared distfiles cache
-├── misc/
-│   └── kimi-code/
-├── security/
-│   └── pinentry-emacs/
-├── WORKFLOW.org        # this file
-└── .agent-shell/
-    └── poudriere.conf  # reference config
+├── misc/kimi-code/
+├── security/pinentry-emacs/
+├── distfiles/           # shared distfiles cache ($DISTDIR)
+├── work/                # host-side build directories ($WRKDIRPREFIX)
+├── pkglist              # all overlay ports; input for poudriere bulk
+└── WORKFLOW.org         # this file
 #+end_example
 
-Each category directory (=devel/=, =security/=, etc.) mirrors the top-level layout of the official Ports Collection.
+Category directories (=devel/=, =misc/=, =security/=) mirror the top-level layout of the official Ports Collection. The poudriere configuration is not stored here — its source of truth is =dotfiles.org= (see step 2 below).
 
-* Setup instructions
+* Setup (once)
 
 ** 1. Install poudriere
 
@@ -46,104 +43,209 @@ doas pkg install -y poudriere
 
 ** 2. Configure poudriere
 
+The authoritative =poudriere.conf= lives in =dotfiles.org= under =Tasks → Ports → poudriere.conf=. Open =dotfiles.org= in Emacs and tangle it (=M-x org-babel-tangle=, or just save — auto-tangle is enabled). The block is tagged =:tangle /doas::/usr/local/etc/poudriere.conf=, so tramp writes the file as root via doas and prompts for the login password.
+
+What the config does, in short:
+- =PACKAGE_FETCH_BRANCH=quarterly= + =PACKAGE_FETCH_URL=pkg+https://pkg.FreeBSD.org/FreeBSD:15:amd64= :: fetch official binary packages for dependencies instead of compiling them.
+- =ZPOOL=zroot=, =ZROOTFS=/poudriere=, =BASEFS=/usr/local/poudriere= :: jails and trees live on the existing ZFS pool.
+- =DISTFILES_CACHE=/home/cashmere/ports/distfiles= :: poudriere reuses the distfiles already collected in this repo.
+- =USE_TMPFS=yes= :: build in memory where possible.
+
+** 3. Export the ports environment
+
+Host-side =make= in a port directory must work as the unprivileged user, without flags. Three variables take care of that; they live in the =.env= block of =dotfiles.org= (tangle, then re-login or source =~/.env=):
+
 #+begin_src sh
-doas cp ~/ports/.agent-shell/poudriere.conf /usr/local/etc/poudriere.conf
+DISTDIR=$HOME/ports/distfiles
+WRKDIRPREFIX=$HOME/ports/work
+PORTSDIR=/usr/local/poudriere/ports/base
+export DISTDIR WRKDIRPREFIX PORTSDIR
 #+end_src
 
-Key settings:
-- =ZPOOL=zroot= :: reuse the existing ZFS pool.
-- =DISTFILES_CACHE=/home/cashmere/ports/distfiles= :: share the distfiles
-  already collected in this repo.
-- =PACKAGE_FETCH_BRANCH=quarterly= and =PACKAGE_FETCH_URL=pkg+https://pkg.FreeBSD.org/FreeBSD:15:amd64= ::
-  tell poudriere to fetch official binary packages for dependencies instead of
-  compiling everything from source.
-
-Important: the official ports tree (=/usr/ports=) must be kept on the same
-quarterly branch as the binary repository. See step 4 below.
+=DISTDIR= keeps distfiles in this repo instead of a root-owned tree; =WRKDIRPREFIX= keeps build directories out of the port directories (a =work= symlink in the port directory still points at the real build directory); =PORTSDIR= points host-side =make= at the poudriere-managed base tree (step 5), so the framework files come from the same tree poudriere builds against and no =/usr/ports= checkout is needed at all. With these set, plain =make fetch= / =makesum= / =stage= / =makeplist= work in any port directory — never with =doas=.
 
-** 3. Create a build jail
+** 4. Create the build jail
 
-Use a jail matching the host version (15.1-RELEASE at the time of writing):
+One jail, named after the OS version and architecture (15.1-RELEASE amd64 at the time of writing):
 
 #+begin_src sh
 doas poudriere jail -c -j 151amd64 -v 15.1-RELEASE -a amd64
 #+end_src
 
-** 4. Create a ports tree overlay
+** 5. Create the ports trees
 
-The base tree must track the same quarterly branch as the binary package
-repository configured in step 2. For FreeBSD 15.1 amd64 the current quarterly
-branch is =2026Q3=.
+Two trees. =base= is a git clone managed by poudriere itself, created pinned to the quarterly branch that matches =PACKAGE_FETCH_BRANCH= (=2026Q3= as of mid-2026). =custom= null-mounts this repository, so edits here are visible to poudriere immediately; poudriere never writes to it.
 
 #+begin_src sh
-  # Base tree from the official ports collection
-doas poudriere ports -c -p base -m null -M /usr/ports
-  # Overlay pointing at this repository
+doas poudriere ports -c -p base -m git+https -B 2026Q3
 doas poudriere ports -c -p custom -m null -M /home/cashmere/ports
 #+end_src
 
-Then tell poudriere to use the overlay:
+A =/usr/ports= checkout is not needed at all — =PORTSDIR= (step 3) points host-side =make= at the poudriere tree. If one exists, delete it; on this system it is its own ZFS dataset, so plain =rm= fails with =Device busy= — destroy the dataset instead:
 
 #+begin_src sh
-doas poudriere ports -u -p base
-doas poudriere ports -u -p custom
+doas zfs destroy zroot/usr/ports
 #+end_src
 
-To switch an existing =/usr/ports= checkout to quarterly:
+** 6. Update the base tree
+
+The tree and the binary package repository (=PACKAGE_FETCH_BRANCH=quarterly=) must stay on the same quarterly branch, or version mismatches force poudriere to rebuild official-tree dependencies from source. Routine updates are a =git pull --rebase= of the pinned branch — poudriere never switches branches:
 
 #+begin_src sh
-cd /usr/ports
-git config --global --add safe.directory /usr/ports
-git fetch origin 2026Q3
-git checkout -B 2026Q3 FETCH_HEAD
+doas poudriere ports -u -p base
 #+end_src
 
-After that, update with:
+When the quarter rolls over (e.g. =2026Q3= → =2026Q4=), switch the tree to the new branch, then update:
 
 #+begin_src sh
-cd /usr/ports && git pull origin 2026Q3
+doas git -C /usr/local/poudriere/ports/base checkout 2026Q4
+doas poudriere ports -u -p base
 #+end_src
 
-* Building a port
+* Building and testing ports
 
-The easiest port to validate the workflow is =devel/py-rassumfrassum= (pure Python, small dependency tree). To build and test it in the isolated jail:
+** A single port: =testport=
 
 #+begin_src sh
 doas poudriere testport -j 151amd64 -p base -O custom -o devel/py-rassumfrassum
 #+end_src
 
-For a bulk build of a list:
+=testport= builds the port in the jail and then runs the full QA cycle: stage, =check-plist=, install, deinstall, reinstall. Dependencies from the official tree are fetched as binary packages; only overlay ports compile.
+
+When a build fails, =testport -i= drops into an interactive shell inside the jail after the attempt (cleaned up on exit), and =-w= preserves the build directory for inspection:
 
 #+begin_src sh
-cat > /tmp/pkglist <<'EOF'
-devel/py-rassumfrassum
-EOF
-doas poudriere bulk -j 151amd64 -p base -O custom -f /tmp/pkglist
+doas poudriere testport -i -j 151amd64 -p base -O custom -o devel/foo
 #+end_src
 
-Once =py-rassumfrassum= builds cleanly, heavier ports like =devel/pyrefly= can be added to the list. Avoid using =security/pinentry-emacs= as a smoke test — it depends on =editors/emacs=, which pulls in a very large dependency tree (X11, GCC, etc.).
+Good smoke tests for a fresh setup:
+- =devel/py-rassumfrassum= :: pure Python, tiny dependency tree.
+- =security/pinentry-emacs= :: a shell script; depends only on =shells/bash= and =security/pinentry-curses=.
 
-* Validated workflow
+** Several ports: =bulk=
 
-The following commands have been executed successfully:
+The tracked =pkglist= file at the repository root lists every port in this overlay, one origin per line. Build them all with:
 
-1. First run (populated the empty jail with Python dependencies):
 #+begin_src sh
-doas poudriere testport -j 151amd64 -p base -O custom -o devel/py-rassumfrassum
+doas poudriere bulk -j 151amd64 -p base -O custom -f /home/cashmere/ports/pkglist
 #+end_src
-Result: success, build time ~6 seconds.
 
-2. Second run (reproducibility check):
-#+begin_src sh 
-doas poudriere testport -j 151amd64 -p base -O custom -o devel/py-rassumfrassum
+=bulk= produces packages for everything on the list. It does not run the interactive install/deinstall tests =testport= runs, so use =testport= when validating a port and =bulk= when you just want the packages.
+
+** Stopping a build
+
+#+begin_src sh
+doas poudriere jail -k -j 151amd64
 #+end_src
-#+RESULT: success, build time ~3 seconds.
 
-3. Node-based port with binary package prefetching:
+** Clearing stale state
+
+After changing the poudriere configuration or switching the base tree's branch, remove the stale package cache metadata or poudriere may refuse to fetch packages:
+
+#+begin_src sh
+doas rm -rf /usr/local/poudriere/data/packages/151amd64-base/.building
+#+end_src
+
+** Pruning old distfiles
+
+=poudriere distclean= cannot be used with this setup: with a shared =DISTFILES_CACHE= it only sees the trees it is passed — =-p base= alone would delete the overlay ports' distfiles (nothing in =base= references them), and =-p custom= fails because the overlay has no =Mk/= directory of its own. Prune by hand instead: after a version upgrade, delete the previous version's distfile from =distfiles/= — the current, referenced file names are listed in the port's =distinfo=.
+
+* Creating a new port
+
+A port is a directory inside a category (=devel/foo=, =misc/bar=, ...) containing at minimum:
+
+- =Makefile= :: port metadata and build logic (see the Porter's Handbook).
+- =pkg-descr= :: a paragraph describing the software.
+- =distinfo= :: checksums of the distfiles, generated by =makesum=.
+- =pkg-plist= :: the packing list. Not needed when the port uses =USE_PYTHON=autoplist= or sets =PLIST_FILES=.
+- =files/= :: patches and extra data files, if any.
+
+Existing ports to copy from: =devel/py-rassumfrassum= for a minimal Python port, =security/pinentry-emacs= for a no-build script port, =misc/kimi-code= for an npm-based port.
+
+Procedure, from the new port's directory. All =make= invocations run as the normal user; the environment from setup step 3 handles the paths.
+
+1. Write the =Makefile= and =pkg-descr=.
+2. Fetch the distfiles and generate =distinfo=:
 #+begin_src sh
-doas poudriere testport -j 151amd64 -p base -O custom -o devel/bash-language-server
+make fetch
+make makesum
 #+end_src
-#+RESULT: success. All official-tree dependencies (=www/node24=, =devel/cmake-core=,
-=devel/icu=, =dns/c-ares=, etc.) were fetched as binary packages; only
-=devel/bash-language-server= from the custom overlay was built from source.
+3. Extract and look around =work/= to verify the build/install assumptions:
+#+begin_src sh
+make extract
+#+end_src
+4. Stage and generate the packing list. The first line of =makeplist= output is a reminder (=/you/have/to/check/what/makeplist/gives/you=), not a plist entry — =tail -n +2= strips it. Review the result:
+#+begin_src sh
+make stage
+make makeplist | tail -n +2 > pkg-plist
+#+end_src
+5. Lint the port (optional but recommended): =portlint -AC= checks against porting conventions, =portclippy= checks variable ordering, =portfmt= rewrites the =Makefile= in canonical format:
+#+begin_src sh
+portlint -AC
+portclippy Makefile
+portfmt Makefile
+#+end_src
+6. Test in the jail until =check-plist= and the install/deinstall cycle are clean:
+#+begin_src sh
+doas poudriere testport -j 151amd64 -p base -O custom -o devel/foo
+#+end_src
+
+* Upgrading a port
+
+** Standard ports
+
+For ports that fetch a normal upstream distfile:
+
+1. Bump =DISTVERSION= (or =PORTVERSION=) in the =Makefile=.
+2. Refresh =distinfo=:
+#+begin_src sh
+make clean
+make fetch
+make makesum
+#+end_src
+3. If the installed file list changed, regenerate =pkg-plist= as in step 4 of "Creating a new port".
+4. Test:
+#+begin_src sh
+doas poudriere testport -j 151amd64 -p base -O custom -o devel/foo
+#+end_src
+5. Prune the previous version's distfile (see "Pruning old distfiles").
+
+** npm-based ports (e.g. =misc/kimi-code=)
+
+These ports build their distfile by running =npm ci=, which requires a =files/package-lock.json= pinning the package to exactly =DISTVERSION=. After a version bump the checked-in lockfile still pins the old version, and =npm ci= aborts with =lock file's X@old does not satisfy X@new=. The lockfile therefore has to be regenerated before anything else.
+
+Procedure, from the port's directory:
+
+1. Bump =DISTVERSION= in the =Makefile=.
+2. Move the stale lockfile aside (in the port directory, not in =/tmp=):
+#+begin_src sh
+mv files/package-lock.json files/package-lock.json.old
+#+end_src
+3. Regenerate the lockfile, the distfile and =distinfo=. The first =fetch= finds no lockfile and writes a fresh one to =work/package-lock.json.generated=; the second rebuilds the distfile reproducibly from that lockfile — the same file poudriere later consumes via =DISTFILES_CACHE=:
+#+begin_src sh
+make clean
+make fetch
+cp work/package-lock.json.generated files/package-lock.json
+make clean
+make fetch
+make makesum
+#+end_src
+4. Extract and verify the upstream layout assumptions used by =do-build= / =do-install= still hold (for kimi-code: =dist/main.mjs=, =node_modules/node-pty=, =native/=):
+#+begin_src sh
+make extract
+#+end_src
+5. Regenerate =pkg-plist= as in step 4 of "Creating a new port".
+6. Test (this also runs =check-plist=):
+#+begin_src sh
+doas poudriere testport -j 151amd64 -p base -O custom -o misc/kimi-code
+#+end_src
+7. Clean up: delete =files/package-lock.json.old= and prune the previous version's distfile (see "Pruning old distfiles").
+
+* Troubleshooting
 
+- Everything compiles instead of being fetched as packages :: check, in order: (1) =/usr/local/etc/poudriere.conf= has =PACKAGE_FETCH_BRANCH=quarterly= (no typos) and a literal =FreeBSD:15:amd64= URL — an unescaped =${ABI}= resolves to an empty string; (2) the =base= tree is on the matching quarterly branch (see setup step 6); (3) the stale =.building= cache was removed (see "Clearing stale state").
+- =npm ci= aborts with =lock file's X@old does not satisfy X@new= :: the lockfile was not regenerated after a version bump; see "npm-based ports".
+- =tar: Failed to open '/usr/ports/distfiles/...'= on the host :: the ports environment from setup step 3 is not active in this shell; re-source =~/.env= or re-login.
+- =check-plist= complains about =/you/have/to/check/what/makeplist/gives/you= :: the reminder line from =make makeplist= ended up in =pkg-plist=; strip the first line.
+- A build hangs or must be aborted :: =doas poudriere jail -k -j 151amd64=.
+- =USES=cargo= port seems to build all of Rust :: it does not; =lang/rust= is installed as a binary package. Only the port itself compiles.