path: root/WORKFLOW.org

#+title: Workflow
#+author: cashmere

* Overview

This repository is a custom FreeBSD ports overlay. Ports that live here are built and tested in an isolated poudriere jail; everything they depend on from the official tree is fetched as an official binary package. Only overlay ports are ever compiled from source.

Tool choice, for the record:
- *poudriere* :: builds ports in clean jails, supports overlays, bulk builds and single-port testing (=testport=). The standard tool for this job.
- *synth* :: similar goals, but no overlay support.
- *portmaster* / plain =make install= :: no build isolation.

One privilege rule applies everywhere: poudriere always runs via =doas= (it manages jails and mounts; =doas.conf= already permits it without a password), and host-side =make= in a port directory never runs as root.

* Layout

#+begin_example
~/ports
├── devel/               # custom development ports
│   ├── bash-language-server/
│   ├── just-lsp/
│   ├── nixd/
│   ├── py-rassumfrassum/
│   ├── pyrefly/
│   └── yaml-language-server/
├── misc/kimi-code/
├── security/pinentry-emacs/
├── distfiles/           # shared distfiles cache ($DISTDIR)
├── work/                # host-side build directories ($WRKDIRPREFIX)
├── pkglist              # all overlay ports; input for poudriere bulk
└── WORKFLOW.org         # this file
#+end_example

Category directories (=devel/=, =misc/=, =security/=) mirror the top-level layout of the official Ports Collection. The poudriere configuration is not stored here — its source of truth is =dotfiles.org= (see step 2 below).

* Setup (once)

** 1. Install poudriere

#+begin_src sh
doas pkg install -y poudriere
#+end_src

** 2. Configure poudriere

The authoritative =poudriere.conf= lives in =dotfiles.org= under =Tasks → Ports → poudriere.conf=. Open =dotfiles.org= in Emacs and tangle it (=M-x org-babel-tangle=, or just save — auto-tangle is enabled). The block is tagged =:tangle /doas::/usr/local/etc/poudriere.conf=, so tramp writes the file as root via doas and prompts for the login password.

What the config does, in short:
- =PACKAGE_FETCH_BRANCH=quarterly= + =PACKAGE_FETCH_URL=pkg+https://pkg.FreeBSD.org/FreeBSD:15:amd64= :: fetch official binary packages for dependencies instead of compiling them.
- =ZPOOL=zroot=, =ZROOTFS=/poudriere=, =BASEFS=/usr/local/poudriere= :: jails and trees live on the existing ZFS pool.
- =DISTFILES_CACHE=/home/cashmere/ports/distfiles= :: poudriere reuses the distfiles already collected in this repo.
- =USE_TMPFS=yes= :: build in memory where possible.

** 3. Export the ports environment

Host-side =make= in a port directory must work as the unprivileged user, without flags. Three variables take care of that; they live in the =.env= block of =dotfiles.org= (tangle, then re-login or source =~/.env=):

#+begin_src sh
DISTDIR=$HOME/ports/distfiles
WRKDIRPREFIX=$HOME/ports/work
PORTSDIR=/usr/local/poudriere/ports/base
export DISTDIR WRKDIRPREFIX PORTSDIR
#+end_src

=DISTDIR= keeps distfiles in this repo instead of a root-owned tree; =WRKDIRPREFIX= keeps build directories out of the port directories (a =work= symlink in the port directory still points at the real build directory); =PORTSDIR= points host-side =make= at the poudriere-managed base tree (step 5), so the framework files come from the same tree poudriere builds against and no =/usr/ports= checkout is needed at all. With these set, plain =make fetch= / =makesum= / =stage= / =makeplist= work in any port directory — never with =doas=.

** 4. Create the build jail

One jail, named after the OS version and architecture (15.1-RELEASE amd64 at the time of writing):

#+begin_src sh
doas poudriere jail -c -j 151amd64 -v 15.1-RELEASE -a amd64
#+end_src

** 5. Create the ports trees

Two trees. =base= is a git clone managed by poudriere itself, created pinned to the quarterly branch that matches =PACKAGE_FETCH_BRANCH= (=2026Q3= as of mid-2026). =custom= null-mounts this repository, so edits here are visible to poudriere immediately; poudriere never writes to it.

#+begin_src sh
doas poudriere ports -c -p base -m git+https -B 2026Q3
doas poudriere ports -c -p custom -m null -M /home/cashmere/ports
#+end_src

A =/usr/ports= checkout is not needed at all — =PORTSDIR= (step 3) points host-side =make= at the poudriere tree. If one exists, delete it; on this system it is its own ZFS dataset, so plain =rm= fails with =Device busy= — destroy the dataset instead:

#+begin_src sh
doas zfs destroy zroot/usr/ports
#+end_src

** 6. Update the base tree

The tree and the binary package repository (=PACKAGE_FETCH_BRANCH=quarterly=) must stay on the same quarterly branch, or version mismatches force poudriere to rebuild official-tree dependencies from source. Routine updates are a =git pull --rebase= of the pinned branch — poudriere never switches branches:

#+begin_src sh
doas poudriere ports -u -p base
#+end_src

When the quarter rolls over (e.g. =2026Q3= → =2026Q4=), switch the tree to the new branch, then update:

#+begin_src sh
doas git -C /usr/local/poudriere/ports/base checkout 2026Q4
doas poudriere ports -u -p base
#+end_src

* Building and testing ports

** A single port: =testport=

#+begin_src sh
doas poudriere testport -j 151amd64 -p base -O custom -o devel/py-rassumfrassum
doas poudriere bulk -j 151amd64 -p base -O custom -o misc/kimi-code
#+end_src

=testport= builds the port in the jail and then runs the full QA cycle: stage, =check-plist=, install, deinstall, reinstall. Dependencies from the official tree are fetched as binary packages; only overlay ports compile.

When a build fails, =testport -i= drops into an interactive shell inside the jail after the attempt (cleaned up on exit), and =-w= preserves the build directory for inspection:

#+begin_src sh
doas poudriere testport -i -j 151amd64 -p base -O custom -o devel/foo
#+end_src

Good smoke tests for a fresh setup:
- =devel/py-rassumfrassum= :: pure Python, tiny dependency tree.
- =security/pinentry-emacs= :: a shell script; depends only on =shells/bash= and =security/pinentry-curses=.

** Several ports: =bulk=

The tracked =pkglist= file at the repository root lists every port in this overlay, one origin per line. Build them all with:

#+begin_src sh
doas poudriere bulk -j 151amd64 -p base -O custom -f /home/cashmere/ports/pkglist
#+end_src

=bulk= produces packages for everything on the list. It does not run the interactive install/deinstall tests =testport= runs, so use =testport= when validating a port and =bulk= when you just want the packages.

** Stopping a build

#+begin_src sh
doas poudriere jail -k -j 151amd64
#+end_src

** Clearing stale state

After changing the poudriere configuration or switching the base tree's branch, remove the stale package cache metadata or poudriere may refuse to fetch packages:

#+begin_src sh
doas rm -rf /usr/local/poudriere/data/packages/151amd64-base/.building
#+end_src

** Pruning old distfiles

=poudriere distclean= cannot be used with this setup: with a shared =DISTFILES_CACHE= it only sees the trees it is passed — =-p base= alone would delete the overlay ports' distfiles (nothing in =base= references them), and =-p custom= fails because the overlay has no =Mk/= directory of its own. Prune by hand instead: after a version upgrade, delete the previous version's distfile from =distfiles/= — the current, referenced file names are listed in the port's =distinfo=.

* Creating a new port

A port is a directory inside a category (=devel/foo=, =misc/bar=, ...) containing at minimum:

- =Makefile= :: port metadata and build logic (see the Porter's Handbook).
- =pkg-descr= :: a paragraph describing the software.
- =distinfo= :: checksums of the distfiles, generated by =makesum=.
- =pkg-plist= :: the packing list. Not needed when the port uses =USE_PYTHON=autoplist= or sets =PLIST_FILES=.
- =files/= :: patches and extra data files, if any.

Existing ports to copy from: =devel/py-rassumfrassum= for a minimal Python port, =security/pinentry-emacs= for a no-build script port, =misc/kimi-code= for an npm-based port.

Procedure, from the new port's directory. All =make= invocations run as the normal user; the environment from setup step 3 handles the paths.

1. Write the =Makefile= and =pkg-descr=.
2. Fetch the distfiles and generate =distinfo=:
#+begin_src sh
make fetch
make makesum
#+end_src
3. Extract and look around =work/= to verify the build/install assumptions:
#+begin_src sh
make extract
#+end_src
4. Stage and generate the packing list. The first line of =makeplist= output is a reminder (=/you/have/to/check/what/makeplist/gives/you=), not a plist entry — =tail -n +2= strips it. Review the result:
#+begin_src sh
make stage
make makeplist | tail -n +2 > pkg-plist
#+end_src
5. Lint the port (optional but recommended): =portlint -AC= checks against porting conventions, =portclippy= checks variable ordering, =portfmt= rewrites the =Makefile= in canonical format:
#+begin_src sh
portlint -AC
portclippy Makefile
portfmt Makefile
#+end_src
6. Test in the jail until =check-plist= and the install/deinstall cycle are clean:
#+begin_src sh
doas poudriere testport -j 151amd64 -p base -O custom -o devel/foo
#+end_src

* Upgrading a port

** Standard ports

For ports that fetch a normal upstream distfile:

1. Bump =DISTVERSION= (or =PORTVERSION=) in the =Makefile=.
2. Refresh =distinfo=:
#+begin_src sh
make clean
make fetch
make makesum
#+end_src
3. If the installed file list changed, regenerate =pkg-plist= as in step 4 of "Creating a new port".
4. Test:
#+begin_src sh
doas poudriere testport -j 151amd64 -p base -O custom -o devel/foo
#+end_src
5. Prune the previous version's distfile (see "Pruning old distfiles").

** npm-based ports (e.g. =misc/kimi-code=)

These ports build their distfile by running =npm ci=, which requires a =files/package-lock.json= pinning the package to exactly =DISTVERSION=. After a version bump the checked-in lockfile still pins the old version, and =npm ci= aborts with =lock file's X@old does not satisfy X@new=. The lockfile therefore has to be regenerated before anything else.

Procedure, from the port's directory:

1. Bump =DISTVERSION= in the =Makefile=.
2. Move the stale lockfile aside (in the port directory, not in =/tmp=):
#+begin_src sh
mv files/package-lock.json files/package-lock.json.old
#+end_src
3. Regenerate the lockfile, the distfile and =distinfo=. The first =fetch= finds no lockfile and writes a fresh one to =work/package-lock.json.generated=; the second rebuilds the distfile reproducibly from that lockfile — the same file poudriere later consumes via =DISTFILES_CACHE=:
#+begin_src sh
make clean
make fetch
cp work/package-lock.json.generated files/package-lock.json
make clean
make fetch
make makesum
#+end_src
4. Extract and verify the upstream layout assumptions used by =do-build= / =do-install= still hold (for kimi-code: =dist/main.mjs=, =node_modules/node-pty=, =native/=):
#+begin_src sh
make extract
#+end_src
5. Regenerate =pkg-plist= as in step 4 of "Creating a new port".
6. Test (this also runs =check-plist=):
#+begin_src sh
doas poudriere testport -j 151amd64 -p base -O custom -o misc/kimi-code
#+end_src
7. Clean up: delete =files/package-lock.json.old= and prune the previous version's distfile (see "Pruning old distfiles").

* Troubleshooting

- Everything compiles instead of being fetched as packages :: check, in order: (1) =/usr/local/etc/poudriere.conf= has =PACKAGE_FETCH_BRANCH=quarterly= (no typos) and a literal =FreeBSD:15:amd64= URL — an unescaped =${ABI}= resolves to an empty string; (2) the =base= tree is on the matching quarterly branch (see setup step 6); (3) the stale =.building= cache was removed (see "Clearing stale state").
- =npm ci= aborts with =lock file's X@old does not satisfy X@new= :: the lockfile was not regenerated after a version bump; see "npm-based ports".
- =tar: Failed to open '/usr/ports/distfiles/...'= on the host :: the ports environment from setup step 3 is not active in this shell; re-source =~/.env= or re-login.
- =check-plist= complains about =/you/have/to/check/what/makeplist/gives/you= :: the reminder line from =make makeplist= ended up in =pkg-plist=; strip the first line.
- A build hangs or must be aborted :: =doas poudriere jail -k -j 151amd64=.
- =USES=cargo= port seems to build all of Rust :: it does not; =lang/rust= is installed as a binary package. Only the port itself compiles.