From c35a62bcce922756d107d7b5ca6428fda8a9afda Mon Sep 17 00:00:00 2001 From: cashmere Date: Thu, 16 Jul 2026 18:50:34 +0200 Subject: update dotfiles & WORKFLOW.org --- WORKFLOW.org | 242 ++++++++++++++++++++++++++++++++++++++++++----------------- dotfiles.org | 9 ++- pkglist | 11 +++ 3 files changed, 191 insertions(+), 71 deletions(-) create mode 100644 pkglist diff --git a/WORKFLOW.org b/WORKFLOW.org index 95fdb25..af575e4 100644 --- a/WORKFLOW.org +++ b/WORKFLOW.org @@ -3,40 +3,37 @@ * Overview -This repository contains custom FreeBSD ports. The goal is to build and test them in an isolated jail using the standard FreeBSD tooling rather than ad-hoc =make DISTFILES=...= invocations. +This repository is a custom FreeBSD ports overlay. Ports that live here are built and tested in an isolated poudriere jail; everything they depend on from the official tree is fetched as an official binary package. Only overlay ports are ever compiled from source. -* Tool choice: poudriere +Tool choice, for the record: +- *poudriere* :: builds ports in clean jails, supports overlays, bulk builds and single-port testing (=testport=). The standard tool for this job. +- *synth* :: similar goals, but no overlay support. +- *portmaster* / plain =make install= :: no build isolation. -After comparing the common FreeBSD port building tools: - -- *poudriere* :: The standard tool for building and testing ports in isolated jails. Supports ZFS, overlays, reproducible bulk builds, and =testport= for individual ports. Best fit for this workflow. -- *synth* :: A lighter alternative, but less commonly used for custom port trees and lacks the overlay/jail flexibility of poudriere. -- *portmaster* / plain =make= :: Good for installing packages on a live system, but do not provide isolated build environments, so they do not solve the "rebuild everything from source" problem. +One privilege rule applies everywhere: poudriere always runs via =doas= (it manages jails and mounts; =doas.conf= already permits it without a password), and host-side =make= in a port directory never runs as root. * Layout #+begin_example ~/ports -├── devel/ +├── devel/ # custom development ports │ ├── bash-language-server/ │ ├── just-lsp/ │ ├── nixd/ │ ├── py-rassumfrassum/ │ ├── pyrefly/ │ └── yaml-language-server/ -├── distfiles/ # shared distfiles cache -├── misc/ -│ └── kimi-code/ -├── security/ -│ └── pinentry-emacs/ -├── WORKFLOW.org # this file -└── .agent-shell/ - └── poudriere.conf # reference config +├── misc/kimi-code/ +├── security/pinentry-emacs/ +├── distfiles/ # shared distfiles cache ($DISTDIR) +├── work/ # host-side build directories ($WRKDIRPREFIX) +├── pkglist # all overlay ports; input for poudriere bulk +└── WORKFLOW.org # this file #+end_example -Each category directory (=devel/=, =security/=, etc.) mirrors the top-level layout of the official Ports Collection. +Category directories (=devel/=, =misc/=, =security/=) mirror the top-level layout of the official Ports Collection. The poudriere configuration is not stored here — its source of truth is =dotfiles.org= (see step 2 below). -* Setup instructions +* Setup (once) ** 1. Install poudriere @@ -46,104 +43,209 @@ doas pkg install -y poudriere ** 2. Configure poudriere +The authoritative =poudriere.conf= lives in =dotfiles.org= under =Tasks → Ports → poudriere.conf=. Open =dotfiles.org= in Emacs and tangle it (=M-x org-babel-tangle=, or just save — auto-tangle is enabled). The block is tagged =:tangle /doas::/usr/local/etc/poudriere.conf=, so tramp writes the file as root via doas and prompts for the login password. + +What the config does, in short: +- =PACKAGE_FETCH_BRANCH=quarterly= + =PACKAGE_FETCH_URL=pkg+https://pkg.FreeBSD.org/FreeBSD:15:amd64= :: fetch official binary packages for dependencies instead of compiling them. +- =ZPOOL=zroot=, =ZROOTFS=/poudriere=, =BASEFS=/usr/local/poudriere= :: jails and trees live on the existing ZFS pool. +- =DISTFILES_CACHE=/home/cashmere/ports/distfiles= :: poudriere reuses the distfiles already collected in this repo. +- =USE_TMPFS=yes= :: build in memory where possible. + +** 3. Export the ports environment + +Host-side =make= in a port directory must work as the unprivileged user, without flags. Three variables take care of that; they live in the =.env= block of =dotfiles.org= (tangle, then re-login or source =~/.env=): + #+begin_src sh -doas cp ~/ports/.agent-shell/poudriere.conf /usr/local/etc/poudriere.conf +DISTDIR=$HOME/ports/distfiles +WRKDIRPREFIX=$HOME/ports/work +PORTSDIR=/usr/local/poudriere/ports/base +export DISTDIR WRKDIRPREFIX PORTSDIR #+end_src -Key settings: -- =ZPOOL=zroot= :: reuse the existing ZFS pool. -- =DISTFILES_CACHE=/home/cashmere/ports/distfiles= :: share the distfiles - already collected in this repo. -- =PACKAGE_FETCH_BRANCH=quarterly= and =PACKAGE_FETCH_URL=pkg+https://pkg.FreeBSD.org/FreeBSD:15:amd64= :: - tell poudriere to fetch official binary packages for dependencies instead of - compiling everything from source. - -Important: the official ports tree (=/usr/ports=) must be kept on the same -quarterly branch as the binary repository. See step 4 below. +=DISTDIR= keeps distfiles in this repo instead of a root-owned tree; =WRKDIRPREFIX= keeps build directories out of the port directories (a =work= symlink in the port directory still points at the real build directory); =PORTSDIR= points host-side =make= at the poudriere-managed base tree (step 5), so the framework files come from the same tree poudriere builds against and no =/usr/ports= checkout is needed at all. With these set, plain =make fetch= / =makesum= / =stage= / =makeplist= work in any port directory — never with =doas=. -** 3. Create a build jail +** 4. Create the build jail -Use a jail matching the host version (15.1-RELEASE at the time of writing): +One jail, named after the OS version and architecture (15.1-RELEASE amd64 at the time of writing): #+begin_src sh doas poudriere jail -c -j 151amd64 -v 15.1-RELEASE -a amd64 #+end_src -** 4. Create a ports tree overlay +** 5. Create the ports trees -The base tree must track the same quarterly branch as the binary package -repository configured in step 2. For FreeBSD 15.1 amd64 the current quarterly -branch is =2026Q3=. +Two trees. =base= is a git clone managed by poudriere itself, created pinned to the quarterly branch that matches =PACKAGE_FETCH_BRANCH= (=2026Q3= as of mid-2026). =custom= null-mounts this repository, so edits here are visible to poudriere immediately; poudriere never writes to it. #+begin_src sh - # Base tree from the official ports collection -doas poudriere ports -c -p base -m null -M /usr/ports - # Overlay pointing at this repository +doas poudriere ports -c -p base -m git+https -B 2026Q3 doas poudriere ports -c -p custom -m null -M /home/cashmere/ports #+end_src -Then tell poudriere to use the overlay: +A =/usr/ports= checkout is not needed at all — =PORTSDIR= (step 3) points host-side =make= at the poudriere tree. If one exists, delete it; on this system it is its own ZFS dataset, so plain =rm= fails with =Device busy= — destroy the dataset instead: #+begin_src sh -doas poudriere ports -u -p base -doas poudriere ports -u -p custom +doas zfs destroy zroot/usr/ports #+end_src -To switch an existing =/usr/ports= checkout to quarterly: +** 6. Update the base tree + +The tree and the binary package repository (=PACKAGE_FETCH_BRANCH=quarterly=) must stay on the same quarterly branch, or version mismatches force poudriere to rebuild official-tree dependencies from source. Routine updates are a =git pull --rebase= of the pinned branch — poudriere never switches branches: #+begin_src sh -cd /usr/ports -git config --global --add safe.directory /usr/ports -git fetch origin 2026Q3 -git checkout -B 2026Q3 FETCH_HEAD +doas poudriere ports -u -p base #+end_src -After that, update with: +When the quarter rolls over (e.g. =2026Q3= → =2026Q4=), switch the tree to the new branch, then update: #+begin_src sh -cd /usr/ports && git pull origin 2026Q3 +doas git -C /usr/local/poudriere/ports/base checkout 2026Q4 +doas poudriere ports -u -p base #+end_src -* Building a port +* Building and testing ports -The easiest port to validate the workflow is =devel/py-rassumfrassum= (pure Python, small dependency tree). To build and test it in the isolated jail: +** A single port: =testport= #+begin_src sh doas poudriere testport -j 151amd64 -p base -O custom -o devel/py-rassumfrassum #+end_src -For a bulk build of a list: +=testport= builds the port in the jail and then runs the full QA cycle: stage, =check-plist=, install, deinstall, reinstall. Dependencies from the official tree are fetched as binary packages; only overlay ports compile. + +When a build fails, =testport -i= drops into an interactive shell inside the jail after the attempt (cleaned up on exit), and =-w= preserves the build directory for inspection: #+begin_src sh -cat > /tmp/pkglist <<'EOF' -devel/py-rassumfrassum -EOF -doas poudriere bulk -j 151amd64 -p base -O custom -f /tmp/pkglist +doas poudriere testport -i -j 151amd64 -p base -O custom -o devel/foo #+end_src -Once =py-rassumfrassum= builds cleanly, heavier ports like =devel/pyrefly= can be added to the list. Avoid using =security/pinentry-emacs= as a smoke test — it depends on =editors/emacs=, which pulls in a very large dependency tree (X11, GCC, etc.). +Good smoke tests for a fresh setup: +- =devel/py-rassumfrassum= :: pure Python, tiny dependency tree. +- =security/pinentry-emacs= :: a shell script; depends only on =shells/bash= and =security/pinentry-curses=. -* Validated workflow +** Several ports: =bulk= -The following commands have been executed successfully: +The tracked =pkglist= file at the repository root lists every port in this overlay, one origin per line. Build them all with: -1. First run (populated the empty jail with Python dependencies): #+begin_src sh -doas poudriere testport -j 151amd64 -p base -O custom -o devel/py-rassumfrassum +doas poudriere bulk -j 151amd64 -p base -O custom -f /home/cashmere/ports/pkglist #+end_src -Result: success, build time ~6 seconds. -2. Second run (reproducibility check): -#+begin_src sh -doas poudriere testport -j 151amd64 -p base -O custom -o devel/py-rassumfrassum +=bulk= produces packages for everything on the list. It does not run the interactive install/deinstall tests =testport= runs, so use =testport= when validating a port and =bulk= when you just want the packages. + +** Stopping a build + +#+begin_src sh +doas poudriere jail -k -j 151amd64 #+end_src -#+RESULT: success, build time ~3 seconds. -3. Node-based port with binary package prefetching: +** Clearing stale state + +After changing the poudriere configuration or switching the base tree's branch, remove the stale package cache metadata or poudriere may refuse to fetch packages: + +#+begin_src sh +doas rm -rf /usr/local/poudriere/data/packages/151amd64-base/.building +#+end_src + +** Pruning old distfiles + +=poudriere distclean= cannot be used with this setup: with a shared =DISTFILES_CACHE= it only sees the trees it is passed — =-p base= alone would delete the overlay ports' distfiles (nothing in =base= references them), and =-p custom= fails because the overlay has no =Mk/= directory of its own. Prune by hand instead: after a version upgrade, delete the previous version's distfile from =distfiles/= — the current, referenced file names are listed in the port's =distinfo=. + +* Creating a new port + +A port is a directory inside a category (=devel/foo=, =misc/bar=, ...) containing at minimum: + +- =Makefile= :: port metadata and build logic (see the Porter's Handbook). +- =pkg-descr= :: a paragraph describing the software. +- =distinfo= :: checksums of the distfiles, generated by =makesum=. +- =pkg-plist= :: the packing list. Not needed when the port uses =USE_PYTHON=autoplist= or sets =PLIST_FILES=. +- =files/= :: patches and extra data files, if any. + +Existing ports to copy from: =devel/py-rassumfrassum= for a minimal Python port, =security/pinentry-emacs= for a no-build script port, =misc/kimi-code= for an npm-based port. + +Procedure, from the new port's directory. All =make= invocations run as the normal user; the environment from setup step 3 handles the paths. + +1. Write the =Makefile= and =pkg-descr=. +2. Fetch the distfiles and generate =distinfo=: #+begin_src sh -doas poudriere testport -j 151amd64 -p base -O custom -o devel/bash-language-server +make fetch +make makesum #+end_src -#+RESULT: success. All official-tree dependencies (=www/node24=, =devel/cmake-core=, -=devel/icu=, =dns/c-ares=, etc.) were fetched as binary packages; only -=devel/bash-language-server= from the custom overlay was built from source. +3. Extract and look around =work/= to verify the build/install assumptions: +#+begin_src sh +make extract +#+end_src +4. Stage and generate the packing list. The first line of =makeplist= output is a reminder (=/you/have/to/check/what/makeplist/gives/you=), not a plist entry — =tail -n +2= strips it. Review the result: +#+begin_src sh +make stage +make makeplist | tail -n +2 > pkg-plist +#+end_src +5. Lint the port (optional but recommended): =portlint -AC= checks against porting conventions, =portclippy= checks variable ordering, =portfmt= rewrites the =Makefile= in canonical format: +#+begin_src sh +portlint -AC +portclippy Makefile +portfmt Makefile +#+end_src +6. Test in the jail until =check-plist= and the install/deinstall cycle are clean: +#+begin_src sh +doas poudriere testport -j 151amd64 -p base -O custom -o devel/foo +#+end_src + +* Upgrading a port + +** Standard ports + +For ports that fetch a normal upstream distfile: + +1. Bump =DISTVERSION= (or =PORTVERSION=) in the =Makefile=. +2. Refresh =distinfo=: +#+begin_src sh +make clean +make fetch +make makesum +#+end_src +3. If the installed file list changed, regenerate =pkg-plist= as in step 4 of "Creating a new port". +4. Test: +#+begin_src sh +doas poudriere testport -j 151amd64 -p base -O custom -o devel/foo +#+end_src +5. Prune the previous version's distfile (see "Pruning old distfiles"). + +** npm-based ports (e.g. =misc/kimi-code=) + +These ports build their distfile by running =npm ci=, which requires a =files/package-lock.json= pinning the package to exactly =DISTVERSION=. After a version bump the checked-in lockfile still pins the old version, and =npm ci= aborts with =lock file's X@old does not satisfy X@new=. The lockfile therefore has to be regenerated before anything else. + +Procedure, from the port's directory: + +1. Bump =DISTVERSION= in the =Makefile=. +2. Move the stale lockfile aside (in the port directory, not in =/tmp=): +#+begin_src sh +mv files/package-lock.json files/package-lock.json.old +#+end_src +3. Regenerate the lockfile, the distfile and =distinfo=. The first =fetch= finds no lockfile and writes a fresh one to =work/package-lock.json.generated=; the second rebuilds the distfile reproducibly from that lockfile — the same file poudriere later consumes via =DISTFILES_CACHE=: +#+begin_src sh +make clean +make fetch +cp work/package-lock.json.generated files/package-lock.json +make clean +make fetch +make makesum +#+end_src +4. Extract and verify the upstream layout assumptions used by =do-build= / =do-install= still hold (for kimi-code: =dist/main.mjs=, =node_modules/node-pty=, =native/=): +#+begin_src sh +make extract +#+end_src +5. Regenerate =pkg-plist= as in step 4 of "Creating a new port". +6. Test (this also runs =check-plist=): +#+begin_src sh +doas poudriere testport -j 151amd64 -p base -O custom -o misc/kimi-code +#+end_src +7. Clean up: delete =files/package-lock.json.old= and prune the previous version's distfile (see "Pruning old distfiles"). + +* Troubleshooting +- Everything compiles instead of being fetched as packages :: check, in order: (1) =/usr/local/etc/poudriere.conf= has =PACKAGE_FETCH_BRANCH=quarterly= (no typos) and a literal =FreeBSD:15:amd64= URL — an unescaped =${ABI}= resolves to an empty string; (2) the =base= tree is on the matching quarterly branch (see setup step 6); (3) the stale =.building= cache was removed (see "Clearing stale state"). +- =npm ci= aborts with =lock file's X@old does not satisfy X@new= :: the lockfile was not regenerated after a version bump; see "npm-based ports". +- =tar: Failed to open '/usr/ports/distfiles/...'= on the host :: the ports environment from setup step 3 is not active in this shell; re-source =~/.env= or re-login. +- =check-plist= complains about =/you/have/to/check/what/makeplist/gives/you= :: the reminder line from =make makeplist= ended up in =pkg-plist=; strip the first line. +- A build hangs or must be aborted :: =doas poudriere jail -k -j 151amd64=. +- =USES=cargo= port seems to build all of Rust :: it does not; =lang/rust= is installed as a binary package. Only the port itself compiles. diff --git a/dotfiles.org b/dotfiles.org index 10764c6..8c2bf67 100644 --- a/dotfiles.org +++ b/dotfiles.org @@ -58,6 +58,11 @@ export EDITOR PAGER PASSWORD_STORE_DIR=$HOME/pass export PASSWORD_STORE_DIR +DISTDIR=$HOME/ports/distfiles +WRKDIRPREFIX=$HOME/ports/work +PORTSDIR=/usr/local/poudriere/ports/base +export DISTDIR WRKDIRPREFIX PORTSDIR + # Launch gpg-agent and expose its SSH socket gpgconf --launch gpg-agent export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) @@ -317,7 +322,9 @@ CLOSED: [2026-07-12 Sun 17:07] *** poudriere.conf -#+begin_src conf +Source of truth for =/usr/local/etc/poudriere.conf=. Tangles through tramp's doas method and asks for the login password. See the "Setup" section of WORKFLOW.org for how this file is used. + +#+begin_src conf :tangle /doas::/usr/local/etc/poudriere.conf # /usr/local/etc/poudriere.conf PACKAGE_FETCH_BRANCH=quarterly diff --git a/pkglist b/pkglist new file mode 100644 index 0000000..5a6defa --- /dev/null +++ b/pkglist @@ -0,0 +1,11 @@ +# Every custom port in this overlay, one origin per line. +# Build all: +# doas poudriere bulk -j 151amd64 -p base -O custom -f /home/cashmere/ports/pkglist +devel/bash-language-server +devel/just-lsp +devel/nixd +devel/py-rassumfrassum +devel/pyrefly +devel/yaml-language-server +misc/kimi-code +security/pinentry-emacs -- cgit v1.3.1