From 4f49b16079b27d7ce633619e4cddfdf2bffbc952 Mon Sep 17 00:00:00 2001 From: cashmere Date: Fri, 4 Sep 2026 09:05:24 +0200 Subject: initial commit --- www/aboutcgi.wiki | 301 +++++ www/aboutdownload.wiki | 101 ++ www/adding_code.wiki | 228 ++++ www/alerts.md | 727 ++++++++++ www/antibot.wiki | 276 ++++ www/apple-touch-icon.png | Bin 0 -> 3997 bytes www/background.jpg | Bin 0 -> 15282 bytes www/backoffice.md | 225 ++++ www/backup.md | 304 +++++ www/blame.wiki | 62 + www/blockchain.md | 478 +++++++ www/branching.wiki | 767 +++++++++++ www/bugtheory.wiki | 124 ++ www/build-icons/linux.gif | Bin 0 -> 3834 bytes www/build-icons/linux64.gif | Bin 0 -> 3994 bytes www/build-icons/mac.gif | Bin 0 -> 3681 bytes www/build-icons/openbsd.gif | Bin 0 -> 5437 bytes www/build-icons/src.gif | Bin 0 -> 4592 bytes www/build-icons/win32.gif | Bin 0 -> 5003 bytes www/build.wiki | 543 ++++++++ www/cap-theorem.md | 173 +++ www/caps/admin-v-setup.md | 461 +++++++ www/caps/impl.md | 111 ++ www/caps/index.md | 405 ++++++ www/caps/login-groups.md | 128 ++ www/caps/ref.html | 454 +++++++ www/cgi.wiki | 210 +++ www/changes.wiki | 2297 ++++++++++++++++++++++++++++++++ www/chat.md | 262 ++++ www/checkin.wiki | 89 ++ www/checkin_names.wiki | 316 +++++ www/childprojects.wiki | 57 + www/chroot.md | 45 + www/ckout-workflows.md | 142 ++ www/co-vs-up.md | 31 + www/collisions.ipynb | 139 ++ www/colordiff.md | 104 ++ www/concepts.wiki | 472 +++++++ www/contact.md | 14 + www/containers.md | 996 ++++++++++++++ www/contribute.wiki | 128 ++ www/copyright-release.html | 106 ++ www/copyright-release.pdf | Bin 0 -> 38316 bytes www/css-tricks.md | 74 + www/css/diff.md | 112 ++ www/css/index.md | 12 + www/custom_ticket.wiki | 153 +++ www/customgraph.md | 228 ++++ www/customskin.md | 598 +++++++++ www/defcsp.md | 490 +++++++ www/delta-manifests.md | 246 ++++ www/delta_encoder_algorithm.wiki | 248 ++++ www/delta_format.wiki | 316 +++++ www/embeddeddoc.wiki | 268 ++++ www/encode1.gif | Bin 0 -> 1377 bytes www/encode2.gif | Bin 0 -> 1918 bytes www/encode3.gif | Bin 0 -> 598 bytes www/encode4.gif | Bin 0 -> 202 bytes www/encode5.gif | Bin 0 -> 262 bytes www/encode6.gif | Bin 0 -> 1189 bytes www/encode7.gif | Bin 0 -> 1642 bytes www/encode8.gif | Bin 0 -> 958 bytes www/encode9.gif | Bin 0 -> 369 bytes www/encryptedrepos.wiki | 82 ++ www/env-opts.md | 509 +++++++ www/event.wiki | 120 ++ www/faq.tcl | 181 +++ www/faq.wiki | 149 +++ www/fileedit-page.md | 317 +++++ www/fileformat.wiki | 995 ++++++++++++++ www/fiveminutes.wiki | 94 ++ www/forum.wiki | 614 +++++++++ www/foss-cklist.wiki | 112 ++ www/fossil-from-msvc.wiki | 51 + www/fossil-is-not-relational.md | 407 ++++++ www/fossil-v-git.wiki | 984 ++++++++++++++ www/fossil.eps | 129 ++ www/fossil.gif | Bin 0 -> 4703 bytes www/fossil2.eps | 215 +++ www/fossil2.gif | Bin 0 -> 5210 bytes www/fossil3.gif | Bin 0 -> 2078 bytes www/fossil_logo_small.gif | Bin 0 -> 932 bytes www/fossil_logo_small2.gif | Bin 0 -> 1478 bytes www/fossil_logo_small3.gif | Bin 0 -> 1148 bytes www/fossil_prompt.sh | 14 + www/fossil_prompt.wiki | 22 + www/gitusers.md | 1335 +++++++++++++++++++ www/globs.md | 618 +++++++++ www/glossary.md | 466 +++++++ www/grep.md | 160 +++ www/gsoc-ideas.md | 177 +++ www/hacker-howto.wiki | 17 + www/hashes.md | 154 +++ www/hashpolicy.wiki | 199 +++ www/hints.wiki | 74 + www/history.md | 95 ++ www/hooks.md | 162 +++ www/hundredandone.md | 1093 +++++++++++++++ www/image-format-vs-repo-size.ipynb | 1719 ++++++++++++++++++++++++ www/image-format-vs-repo-size.md | 302 +++++ www/image-format-vs-repo-size.svg | 1472 ++++++++++++++++++++ www/index.wiki | 108 ++ www/inout.wiki | 141 ++ www/interwiki.md | 117 ++ www/javascript.md | 642 +++++++++ www/json-api/_template.md | 22 + www/json-api/api-artifact.md | 155 +++ www/json-api/api-auth.md | 271 ++++ www/json-api/api-branch.md | 86 ++ www/json-api/api-checkout.md | 54 + www/json-api/api-config.md | 36 + www/json-api/api-diff.md | 44 + www/json-api/api-dir.md | 53 + www/json-api/api-finfo.md | 53 + www/json-api/api-misc.md | 92 ++ www/json-api/api-query.md | 85 ++ www/json-api/api-settings.md | 127 ++ www/json-api/api-stat.md | 59 + www/json-api/api-tag.md | 229 ++++ www/json-api/api-ticket.md | 136 ++ www/json-api/api-timeline.md | 214 +++ www/json-api/api-user.md | 142 ++ www/json-api/api-version.md | 32 + www/json-api/api-wiki.md | 277 ++++ www/json-api/conventions.md | 649 +++++++++ www/json-api/hacking.md | 353 +++++ www/json-api/index.md | 40 + www/json-api/intro.md | 172 +++ www/json-api/tips.md | 102 ++ www/loadmgmt.md | 96 ++ www/makefile.wiki | 327 +++++ www/mdtest/test1.md | 59 + www/mirrorlimitations.md | 112 ++ www/mirrortogithub.md | 166 +++ www/mkindex.tcl | 187 +++ www/newrepo.wiki | 163 +++ www/password.wiki | 143 ++ www/patchcmd.md | 112 ++ www/permutedindex.html | 140 ++ www/pikchr.md | 139 ++ www/pop.wiki | 72 + www/private.wiki | 121 ++ www/qandc.wiki | 154 +++ www/quickstart.wiki | 577 ++++++++ www/quotes.wiki | 171 +++ www/rebaseharm.md | 522 ++++++++ www/relatedwork.md | 93 ++ www/reviews.wiki | 138 ++ www/scgi.wiki | 27 + www/selfcheck.wiki | 107 ++ www/selfhost.wiki | 78 ++ www/server/any/althttpd.md | 42 + www/server/any/cgi.md | 117 ++ www/server/any/http-over-ssh.md | 173 +++ www/server/any/index.md | 11 + www/server/any/inetd.md | 52 + www/server/any/none.md | 56 + www/server/any/scgi.md | 72 + www/server/any/stunnel.md | 81 ++ www/server/any/xinetd.md | 27 + www/server/debian/index.md | 6 + www/server/debian/nginx.md | 395 ++++++ www/server/debian/service.md | 259 ++++ www/server/index.html | 316 +++++ www/server/macos/index.md | 5 + www/server/macos/service.md | 181 +++ www/server/openbsd/fastcgi.md | 302 +++++ www/server/openbsd/index.md | 6 + www/server/openbsd/service.wiki | 118 ++ www/server/whyuseaserver.wiki | 95 ++ www/server/windows/cgi-bin-perm.png | Bin 0 -> 15784 bytes www/server/windows/cgi-exec-perm.png | Bin 0 -> 6789 bytes www/server/windows/cgi-install-iis.png | Bin 0 -> 34422 bytes www/server/windows/cgi-script-map.png | Bin 0 -> 7456 bytes www/server/windows/cgi.md | 115 ++ www/server/windows/iis.md | 134 ++ www/server/windows/index.md | 8 + www/server/windows/none.md | 69 + www/server/windows/service.md | 120 ++ www/server/windows/stunnel.md | 156 +++ www/serverext.wiki | 375 ++++++ www/settings.wiki | 61 + www/shunning.wiki | 163 +++ www/signing.md | 124 ++ www/ssl-server.md | 289 ++++ www/ssl.wiki | 328 +++++ www/stats.wiki | 149 +++ www/style.wiki | 159 +++ www/sync.wiki | 1120 ++++++++++++++++ www/tech_overview.wiki | 378 ++++++ www/th1-hooks.md | 134 ++ www/th1.md | 1024 ++++++++++++++ www/theory1.wiki | 142 ++ www/tickets.wiki | 199 +++ www/title-test.md | 12 + www/title-test.wiki | 14 + www/tls-nginx.md | 3 + www/uitest.html | 129 ++ www/unvers.wiki | 115 ++ www/userlinks.wiki | 52 + www/webpage-ex.md | 127 ++ www/webui.wiki | 181 +++ www/whyallinone.md | 235 ++++ www/whyusefossil.wiki | 196 +++ www/wikitheory.wiki | 90 ++ www/wsl_caveats.wiki | 176 +++ www/xkcd-git.gif | Bin 0 -> 38650 bytes 207 files changed, 44145 insertions(+) create mode 100644 www/aboutcgi.wiki create mode 100644 www/aboutdownload.wiki create mode 100644 www/adding_code.wiki create mode 100644 www/alerts.md create mode 100644 www/antibot.wiki create mode 100644 www/apple-touch-icon.png create mode 100644 www/background.jpg create mode 100644 www/backoffice.md create mode 100644 www/backup.md create mode 100644 www/blame.wiki create mode 100644 www/blockchain.md create mode 100644 www/branching.wiki create mode 100644 www/bugtheory.wiki create mode 100644 www/build-icons/linux.gif create mode 100644 www/build-icons/linux64.gif create mode 100644 www/build-icons/mac.gif create mode 100644 www/build-icons/openbsd.gif create mode 100644 www/build-icons/src.gif create mode 100644 www/build-icons/win32.gif create mode 100644 www/build.wiki create mode 100644 www/cap-theorem.md create mode 100644 www/caps/admin-v-setup.md create mode 100644 www/caps/impl.md create mode 100644 www/caps/index.md create mode 100644 www/caps/login-groups.md create mode 100644 www/caps/ref.html create mode 100644 www/cgi.wiki create mode 100644 www/changes.wiki create mode 100644 www/chat.md create mode 100644 www/checkin.wiki create mode 100644 www/checkin_names.wiki create mode 100644 www/childprojects.wiki create mode 100644 www/chroot.md create mode 100644 www/ckout-workflows.md create mode 100644 www/co-vs-up.md create mode 100755 www/collisions.ipynb create mode 100644 www/colordiff.md create mode 100644 www/concepts.wiki create mode 100644 www/contact.md create mode 100644 www/containers.md create mode 100644 www/contribute.wiki create mode 100644 www/copyright-release.html create mode 100644 www/copyright-release.pdf create mode 100644 www/css-tricks.md create mode 100644 www/css/diff.md create mode 100644 www/css/index.md create mode 100644 www/custom_ticket.wiki create mode 100644 www/customgraph.md create mode 100644 www/customskin.md create mode 100644 www/defcsp.md create mode 100644 www/delta-manifests.md create mode 100644 www/delta_encoder_algorithm.wiki create mode 100644 www/delta_format.wiki create mode 100644 www/embeddeddoc.wiki create mode 100644 www/encode1.gif create mode 100644 www/encode2.gif create mode 100644 www/encode3.gif create mode 100644 www/encode4.gif create mode 100644 www/encode5.gif create mode 100644 www/encode6.gif create mode 100644 www/encode7.gif create mode 100644 www/encode8.gif create mode 100644 www/encode9.gif create mode 100644 www/encryptedrepos.wiki create mode 100644 www/env-opts.md create mode 100644 www/event.wiki create mode 100755 www/faq.tcl create mode 100644 www/faq.wiki create mode 100644 www/fileedit-page.md create mode 100644 www/fileformat.wiki create mode 100644 www/fiveminutes.wiki create mode 100644 www/forum.wiki create mode 100644 www/foss-cklist.wiki create mode 100644 www/fossil-from-msvc.wiki create mode 100644 www/fossil-is-not-relational.md create mode 100644 www/fossil-v-git.wiki create mode 100644 www/fossil.eps create mode 100644 www/fossil.gif create mode 100644 www/fossil2.eps create mode 100644 www/fossil2.gif create mode 100644 www/fossil3.gif create mode 100644 www/fossil_logo_small.gif create mode 100644 www/fossil_logo_small2.gif create mode 100644 www/fossil_logo_small3.gif create mode 100644 www/fossil_prompt.sh create mode 100644 www/fossil_prompt.wiki create mode 100644 www/gitusers.md create mode 100644 www/globs.md create mode 100644 www/glossary.md create mode 100644 www/grep.md create mode 100644 www/gsoc-ideas.md create mode 100644 www/hacker-howto.wiki create mode 100644 www/hashes.md create mode 100644 www/hashpolicy.wiki create mode 100644 www/hints.wiki create mode 100644 www/history.md create mode 100644 www/hooks.md create mode 100644 www/hundredandone.md create mode 100644 www/image-format-vs-repo-size.ipynb create mode 100644 www/image-format-vs-repo-size.md create mode 100644 www/image-format-vs-repo-size.svg create mode 100644 www/index.wiki create mode 100644 www/inout.wiki create mode 100644 www/interwiki.md create mode 100644 www/javascript.md create mode 100644 www/json-api/_template.md create mode 100644 www/json-api/api-artifact.md create mode 100644 www/json-api/api-auth.md create mode 100644 www/json-api/api-branch.md create mode 100644 www/json-api/api-checkout.md create mode 100644 www/json-api/api-config.md create mode 100644 www/json-api/api-diff.md create mode 100644 www/json-api/api-dir.md create mode 100644 www/json-api/api-finfo.md create mode 100644 www/json-api/api-misc.md create mode 100644 www/json-api/api-query.md create mode 100644 www/json-api/api-settings.md create mode 100644 www/json-api/api-stat.md create mode 100644 www/json-api/api-tag.md create mode 100644 www/json-api/api-ticket.md create mode 100644 www/json-api/api-timeline.md create mode 100644 www/json-api/api-user.md create mode 100644 www/json-api/api-version.md create mode 100644 www/json-api/api-wiki.md create mode 100644 www/json-api/conventions.md create mode 100644 www/json-api/hacking.md create mode 100644 www/json-api/index.md create mode 100644 www/json-api/intro.md create mode 100644 www/json-api/tips.md create mode 100644 www/loadmgmt.md create mode 100644 www/makefile.wiki create mode 100644 www/mdtest/test1.md create mode 100644 www/mirrorlimitations.md create mode 100644 www/mirrortogithub.md create mode 100755 www/mkindex.tcl create mode 100644 www/newrepo.wiki create mode 100644 www/password.wiki create mode 100644 www/patchcmd.md create mode 100644 www/permutedindex.html create mode 100644 www/pikchr.md create mode 100644 www/pop.wiki create mode 100644 www/private.wiki create mode 100644 www/qandc.wiki create mode 100644 www/quickstart.wiki create mode 100644 www/quotes.wiki create mode 100644 www/rebaseharm.md create mode 100644 www/relatedwork.md create mode 100644 www/reviews.wiki create mode 100644 www/scgi.wiki create mode 100644 www/selfcheck.wiki create mode 100644 www/selfhost.wiki create mode 100644 www/server/any/althttpd.md create mode 100644 www/server/any/cgi.md create mode 100644 www/server/any/http-over-ssh.md create mode 100644 www/server/any/index.md create mode 100644 www/server/any/inetd.md create mode 100644 www/server/any/none.md create mode 100644 www/server/any/scgi.md create mode 100644 www/server/any/stunnel.md create mode 100644 www/server/any/xinetd.md create mode 100644 www/server/debian/index.md create mode 100644 www/server/debian/nginx.md create mode 100644 www/server/debian/service.md create mode 100644 www/server/index.html create mode 100644 www/server/macos/index.md create mode 100644 www/server/macos/service.md create mode 100644 www/server/openbsd/fastcgi.md create mode 100644 www/server/openbsd/index.md create mode 100644 www/server/openbsd/service.wiki create mode 100644 www/server/whyuseaserver.wiki create mode 100644 www/server/windows/cgi-bin-perm.png create mode 100644 www/server/windows/cgi-exec-perm.png create mode 100644 www/server/windows/cgi-install-iis.png create mode 100644 www/server/windows/cgi-script-map.png create mode 100644 www/server/windows/cgi.md create mode 100644 www/server/windows/iis.md create mode 100644 www/server/windows/index.md create mode 100644 www/server/windows/none.md create mode 100644 www/server/windows/service.md create mode 100644 www/server/windows/stunnel.md create mode 100644 www/serverext.wiki create mode 100644 www/settings.wiki create mode 100644 www/shunning.wiki create mode 100644 www/signing.md create mode 100644 www/ssl-server.md create mode 100644 www/ssl.wiki create mode 100644 www/stats.wiki create mode 100644 www/style.wiki create mode 100644 www/sync.wiki create mode 100644 www/tech_overview.wiki create mode 100644 www/th1-hooks.md create mode 100644 www/th1.md create mode 100644 www/theory1.wiki create mode 100644 www/tickets.wiki create mode 100644 www/title-test.md create mode 100644 www/title-test.wiki create mode 100644 www/tls-nginx.md create mode 100644 www/uitest.html create mode 100644 www/unvers.wiki create mode 100644 www/userlinks.wiki create mode 100644 www/webpage-ex.md create mode 100644 www/webui.wiki create mode 100644 www/whyallinone.md create mode 100644 www/whyusefossil.wiki create mode 100644 www/wikitheory.wiki create mode 100644 www/wsl_caveats.wiki create mode 100644 www/xkcd-git.gif (limited to 'www') diff --git a/www/aboutcgi.wiki b/www/aboutcgi.wiki new file mode 100644 index 0000000..362f536 --- /dev/null +++ b/www/aboutcgi.wiki @@ -0,0 +1,301 @@ +How CGI Works In Fossil + +

Introduction

+ +CGI or "Common Gateway Interface" is a venerable yet reliable technique for +generating dynamic web content. This article gives a quick background on how +CGI works and describes how Fossil can act as a CGI service. + +This is a "how it works" guide. This document provides background +information on the CGI protocol so that you can better understand what +is going on behind the scenes. If you just want to set up Fossil +as a CGI server, see the [./server/ | Fossil Server Setup] page. Or +if you want to develop CGI-based extensions to Fossil, see +the [./serverext.wiki|CGI Server Extensions] page. + +

A Quick Review Of CGI

+ +An HTTP request is a block of text that is sent by a client application +(usually a web browser) and arrives at the web server over a network +connection. The HTTP request contains a URL that describes the information +being requested. The URL in the HTTP request is typically the same URL +that appears in the URL bar at the top of the web browser that is making +the request. The URL might contain a "?" character followed by +query parameters. The HTTP will usually also contain other information +such as the name of the application that made the request, whether or +not the requesting application can accept a compressed reply, POST +parameters from forms, and so forth. + +The job of the web server is to interpret the HTTP request and formulate +an appropriate reply. +The web server is free to interpret the HTTP request in any way it wants, +but most web servers follow a similar pattern, described below. +(Note: details may vary from one web server to another.) + +Suppose the filename component of the URL in the HTTP request looks like this: + +
/one/two/timeline/four
+ +Most web servers will search their content area for files that match +some prefix of the URL. The search starts with /one, then goes to +/one/two, then /one/two/timeline, and finally +/one/two/timeline/four is checked. The search stops at the first +match. + +Suppose the first match is /one/two. If /one/two is an +ordinary file in the content area, then that file is returned as static +content. The "/timeline/four" suffix is silently ignored. + +If /one/two is a CGI script (or program), then the web server +executes the /one/two script. The output generated by +the script is collected and repackaged as the HTTP reply. + +Before executing the CGI script, the web server will set up various +environment variables with information useful to the CGI script: + +
VariableMeaning +
GATEWAY_INTERFACEAlways set to "CGI/1.0" +
REQUEST_URI + The input URL from the HTTP request. +
SCRIPT_NAME + The prefix of the input URL that matches the CGI script name. + In this example: "/one/two". +
PATH_INFO + The suffix of the URL beyond the name of the CGI script. + In this example: "timeline/four". +
QUERY_STRING + The query string that follows the "?" in the URL, if there is one. +
+ +There are other CGI environment variables beyond those listed above. +Many Fossil servers implement the +[https://fossil-scm.org/home/test-env/two/three?abc=xyz|test-env] +webpage that shows some of the CGI environment +variables that Fossil pays attention to. + +In addition to setting various CGI environment variables, if the HTTP +request contains POST content, then the web server relays the POST content +to standard input of the CGI script. + +In summary, the task of the +CGI script is to read the various CGI environment variables and +the POST content on standard input (if any), figure out an appropriate +reply, then write that reply on standard output. +The web server will read the output from the CGI script, reformat it +into an appropriate HTTP reply, and relay the result back to the +requesting application. +The CGI script exits as soon as it generates a single reply. +The web server will (usually) persist and handle multiple HTTP requests, +but a CGI script handles just one HTTP request and then exits. + +The above is a rough outline of how CGI works. +There are many details omitted from this brief discussion. +See other on-line CGI tutorials for further information. + +

How Fossil Acts As A CGI Program

+ +An appropriate CGI script for running Fossil will look something +like the following: + +
+#!/usr/bin/fossil
+repository: /home/www/repos/project.fossil
+
+ +The first line of the script is a +"[https://en.wikipedia.org/wiki/Shebang_%28Unix%29|shebang]" +that tells the operating system what program to use as the interpreter +for this script. On unix, when you execute a script that starts with +a shebang, the operating system runs the program identified by the +shebang with a single argument that is the full pathname of the script +itself. +In our example, the interpreter is Fossil, and the argument might +be something like "/var/www/cgi-bin/one/two" (depending on how your +particular web server is configured). + +The Fossil program that is run as the script interpreter +is the same Fossil that runs when +you type ordinary Fossil commands like "fossil sync" or "fossil commit". +But in this case, as soon as it launches, the Fossil program +recognizes that the GATEWAY_INTERFACE environment variable is +set to "CGI/1.0" and it therefore knows that it is being used as +CGI rather than as an ordinary command-line tool, and behaves accordingly. + +When Fossil recognizes that it is being run as CGI, it opens and reads +the file identified by its sole argument (the file named by +argv[1]). In our example, the second line of that file +tells Fossil the location of the repository it will be serving. +Fossil then starts looking at the CGI environment variables to figure +out what web page is being requested, generates that one web page, +then exits. + +Usually, the webpage being requested is the first term of the +PATH_INFO environment variable. (Exceptions to this rule are noted +in the sequel.) For our example, the first term of PATH_INFO +is "timeline", which means that Fossil will generate +the [/help/www/timeline|/timeline] webpage. + +With Fossil, terms of PATH_INFO beyond the webpage name are converted into +the "name" query parameter. Hence, the following two URLs mean +exactly the same thing to Fossil: +
    +
  1. [https://fossil-scm.org/home/info/c14ecc43] +
  2. [https://fossil-scm.org/home/info?name=c14ecc43] +
+ +In both cases, the CGI script is called "/fossil". For case (A), +the PATH_INFO variable will be "info/c14ecc43" and so the +"[/help/www/info|/info]" webpage will be generated and the suffix of +PATH_INFO will be converted into the "name" query parameter, which +identifies the artifact about which information is requested. +In case (B), the PATH_INFO is just "info", but the same "name" +query parameter is set explicitly by the URL itself. + +

Serving Multiple Fossil Repositories From One CGI Script

+ +The previous example showed how to serve a single Fossil repository +using a single CGI script. +On a website that wants to serve multiple repositories, one could +simply create multiple CGI scripts, one script for each repository. +But it is also possible to serve multiple Fossil repositories from +a single CGI script. + +If the CGI script for Fossil contains a "directory:" line instead of +a "repository:" line, then the argument to "directory:" is the name +of a directory that contains multiple repository files, each ending +with ".fossil". For example: + +
+#!/usr/bin/fossil
+directory: /home/www/repos
+
+ +Suppose the /home/www/repos directory contains files named +one.fossil, two.fossil, and subdir/three.fossil. +Further suppose that the name of the CGI script (relative to the root +of the webserver document area) is "cgis/example2". Then to +see the timeline for the "three.fossil" repository, the URL would be: + +
+http://example.com/cgis/example2/subdir/three/timeline
+
+ +Here is what happens: +
    +
  1. The input URI on the HTTP request is + /cgis/example2/subdir/three/timeline +
  2. The web server searches prefixes of the input URI until it finds + the "cgis/example2" script. The web server then sets + PATH_INFO to the "subdir/three/timeline" suffix and invokes the + "cgis/example2" script. +
  3. Fossil runs and sees the "directory:" line pointing to + "/home/www/repos". Fossil then starts pulling terms off the + front of the PATH_INFO looking for a repository. It first looks + at "/home/www/resps/subdir.fossil" but there is no such repository. + So then it looks at "/home/www/repos/subdir/three.fossil" and finds + a repository. The PATH_INFO is shortened by removing + "subdir/three/" leaving it at just "timeline". +
  4. Fossil looks at the rest of PATH_INFO to see that the webpage + requested is "timeline". +
+ + +The web server sets many environment variables in step 2 in addition +to just PATH_INFO. The following diagram shows a few of these variables +and their relationship to the request URL: + + +charwid = 0.075 +thickness = 0 + +SCHEME: box "https://" mono fit +DOMAIN: box "example.com" mono fit +SCRIPT: box "/cgis/example2" mono fit +PATH: box "/subdir/three/timeline" mono fit +QUERY: box "?c=55d7e1" mono fit + +thickness = 0.01 + +DB: box at 0.3 below DOMAIN "HTTP_HOST" mono fit invis +SB: box at 0.3 below SCRIPT "SCRIPT_NAME" mono fit invis +PB: box at 0.3 below PATH "PATH_INFO" mono fit invis +QB: box at 0.3 below QUERY "QUERY_STRING" mono fit invis +RB: box at 0.5 above PATH "REQUEST_URI" mono fit invis + +color = lightgray + +box at SCHEME width SCHEME.width height SCHEME.height +line fill 0x7799CC behind QUERY \ + from SCRIPT.nw \ + to RB.sw \ + to RB.se \ + to QUERY.ne \ + close +line fill 0x99CCFF behind DOMAIN \ + from DOMAIN.nw \ + to DOMAIN.sw \ + to DB.n \ + to DOMAIN.se \ + to DOMAIN.ne \ + close +line fill 0xCCEEFF behind SCRIPT \ + from SCRIPT.nw \ + to SCRIPT.sw \ + to SB.n \ + to SCRIPT.se \ + to SCRIPT.ne \ + close +line fill 0x99CCFF behind PATH \ + from PATH.nw \ + to PATH.sw \ + to PB.n \ + to PATH.se \ + to PATH.ne \ + close +line fill 0xCCEEFF behind QUERY \ + from QUERY.nw \ + to QUERY.sw \ + to QB.n \ + to QUERY.se \ + to QUERY.ne \ + close + + +

Additional CGI Script Options

+ +The CGI script can have additional options used to fine-tune +Fossil's behavior. See the [./cgi.wiki|CGI script documentation] +for details. + +

Additional Observations

+
    +

  1. +Fossil does not distinguish between the various HTTP methods (GET, PUT, +DELETE, etc). Fossil figures out what it needs to do purely from the +webpage term of the URI.

    2. +

  2. +Fossil does not distinguish between query parameters that are part of the +URI, application/x-www-form-urlencoded or multipart/form-data encoded +parameter that are part of the POST content, and cookies. Each information +source is seen as a space of key/value pairs which are loaded into an +internal property hash table. The code that runs to generate the reply +can then reference various properties values. +Fossil does not care where the value of each property comes from (POST +content, cookies, or query parameters) only that the property exists +and has a value.

    3. +

  3. +The "[/help/ui|fossil ui]" and "[/help/server|fossil server]" commands +are implemented using a simple built-in web server that accepts incoming HTTP +requests, translates each request into a CGI invocation, then creates a +separate child Fossil process to handle each request. In other words, CGI +is used internally to implement "fossil ui/server". +

    +SCGI is processed using the same built-in web server, just modified +to parse SCGI requests instead of HTTP requests. Each SCGI request is +converted into CGI, then Fossil creates a separate child Fossil +process to handle each CGI request.

    4. +

  4. +Fossil is itself often launched using CGI. But Fossil can also then +turn around and launch [./serverext.wiki|sub-CGI scripts to implement +extensions].

    5. +
diff --git a/www/aboutdownload.wiki b/www/aboutdownload.wiki new file mode 100644 index 0000000..4eb09af --- /dev/null +++ b/www/aboutdownload.wiki @@ -0,0 +1,101 @@ +How The Fossil Download Page Works + +

1.0 Overview

+ +The [/uv/download.html|Download] page for the Fossil self-hosting +repository is implemented using [./unvers.wiki|unversioned files]. +The "download.html" screen itself, and the various build products +are all stored as unversioned content. The download.html page +uses XMLHttpRequest() to retrieve the [/help/www/juvlist|/juvlist] webpage +for a list of all unversioned files. Javascript in the +[/uv/download.js?mimetype=text/plain|download.js] file (which is +sourced by "download.html") then figures out which unversioned files are +build products and paints appropriate icons on the displayed +download page. + +Except, the "Source Tarball" download products are not stored as +unversioned files. They are computed on-demand by the +[/help/www/tarball|/tarball web page]. + +When a new version is generated, the developers use the +[/help/uv|fossil uv edit] command to make minor changes +to the "[/uv/download.js?mimetype=text/plain|download.js]" +file so that it knows about the +new version number. Then the developers run +the [/help/uv|fossil uv add] command for each +build product. Finally, the +[/help/uv|fossil uv sync] command is run to push all +the content up to servers. All +[./selfhost.wiki|three self-hosting repositories] for Fossil +are updated automatically. + +

2.0 Details

+ +The current text of the "download.html" and "download.js" files can +be seen at: + + * [/uv/download.html?mimetype=text/plain] + * [/uv/download.js?mimetype=text/plain] + +Notice how the hyperlinks above use the "mimetype=text/plain" +query parameter in order to display the file as plain text +instead of the usual HTML or Javascript. + +The default mimetype for "download.html" is +text/html. But because the entire page is enclosed within + +
...
 + +Fossil knows to add its standard header and footer information to the +document, making it look just like any other page. See +"[./embeddeddoc.wiki|embedded documentation]" for further details on +how this <div class='fossil-doc'> markup works. + +With each new release, the "releases" variable in the javascript on +the [/uv/download.js?mimetype=text/plain|download.js] page is +edited (using "[/help/uv|fossil uv edit download.js]") to add +details of the release. + +When the JavaScript in the "download.js" file runs, it requests +a listing of all unversioned content using the /juvlist URL. +([/juvlist|sample /juvlist output]). The content of the download page is +constructed by matching unversioned files against regular expressions +in the "releases" variable. + +Build products need to be constructed on different machines. The precompiled +binary for Linux is compiled on Linux, the precompiled binary for Windows +is compiled on Windows11, and so forth. After a new release is tagged, +the release manager goes around to each of the target platforms, checks +out the release and compiles it, then runs +[/help/uv|fossil uv add] for the build product followed by +[/help/uv|fossil uv sync] to push the new build product to the +[./selfhost.wiki|various servers]. This process is repeated for +each build product. + +When older builds are retired from the download page, the +[/uv/download.js?mimetype=text/plain|download.js] page is again +edited to remove the corresponding entry from the "release" variable +and the edit is synced using +[/help/uv|fossil uv sync]. This causes the build products to +disappear from the download page immediately. But those build products +are still taking up space in the unversioned content table of the +server repository. To purge the obsolete build products, one or +more [/help/uv|fossil uv rm] commands are run, followed by +another [/help/uv|fossil uv sync]. It is important to purge +obsolete build products since they take up a lot of space. +At [/repo-tabsize] you can see that the unversioned table takes up +a substantial fraction of the repository. + +

3.0 Security

+ +Only users with the [/setup_ulist_notes|"y" permission] are allowed +to push unversioned content up to the servers. Having the ability +to push check-ins (the [/setup_ulist_notes|"i" permission]) is not +sufficient. + +On the Fossil project there are (as of 2023-07-31) 71 people who have +check-in privileges. But only the project lead can push unversioned +content and thus change the build products on the download page. +Minimizing the number of people who can change the build products +helps to ensure that rogue binaries do not slip onto the download page +unnoticed. diff --git a/www/adding_code.wiki b/www/adding_code.wiki new file mode 100644 index 0000000..1192907 --- /dev/null +++ b/www/adding_code.wiki @@ -0,0 +1,228 @@ +Adding Features To Fossil + +

1.0 Introduction

+ +This article provides a brief overview of how to write new C-code code that +extends or enhances the core Fossil binary. + +New features can be added to a Fossil server using +[./serverext.wiki|external CGI programs], +but that is not what this article is about. +This article focuses on how to make changes +to Fossil itself. + +

2.0 Programming Language

+ +Fossil is written in C-89. There are specific [./style.wiki | style guidelines] +that are required for any new code that will be accepted into the Fossil core. +But, of course, if you are writing an extension just for yourself, you can +use any programming style you want. + +The source code for Fossil is not sent directly into the C compiler. +There are three separate code [./makefile.wiki#preprocessing|preprocessors] +that run over the code first. + + 1. The mkindex preprocessor scans all regular source files looking + for special comments that contain "help" text and which identify routines + that implement specific commands or which generate particular web pages. + + 2. The makeheaders preprocessor generates all the ".h" files + automatically. Fossil programmers write ".c" files only and let the + makeheaders preprocessor create the ".h" files. + + 3. The translate preprocessor converts source code lines that + begin with "@" into string literals, or into print statements that + generate web page output, depending on context. + +The [./makefile.wiki|Makefile] for Fossil takes care of running these +preprocessors with all the right arguments and in the right order. So it is +not necessary to understand the details of how these preprocessors work. +(Though, the sources for all three preprocessors are included in the source +tree and are well commented, if you want to dig deeper.) It is only necessary +to know that these preprocessors exist and hence will affect the way you +write code. + +

3.0 Adding New Source Code Files

+ +New source code files are added in the "src/" subdirectory of the Fossil +source tree. Suppose one wants to add a new source code file named +"xyzzy.c". The first step is to add this file to the various makefiles. +Do so by editing the file tools/makemake.tcl and adding "xyzzy" (without +the final ".c") to the list of source modules at the top of that script. +Save the result and then run the makemake.tcl script using a TCL +interpreter. The command to run the makemake.tcl script is: + + +tclsh makemake.tcl + + +The working directory must be src/ when the command above is run. +Note that TCL is not normally required to build Fossil, but +it is required for this step. If you do not have a TCL interpreter on +your system already, they are easy to install. A popular choice is the +[http://www.activestate.com/activetcl|Active Tcl] installation from +ActiveState. + +After the makefiles have been updated, create the xyzzy.c source file +from the following template: + + +/* +** Copyright boilerplate goes here. +***************************************************** +** High-level description of what this module goes +** here. +*/ +#include "config.h" +#include "xyzzy.h" + +#if INTERFACE +/* Exported object (structure) definitions or #defines +** go here */ +#endif /* INTERFACE */ + +/* New code goes here */ + + +Note in particular the #include "xyzzy.h" line near the top. +The "xyzzy.h" file is automatically generated by makeheaders. Every +normal Fossil source file must have a #include at the top that imports +its private header file. (Some source files, such as "sqlite3.c" are +exceptions to this rule. Don't worry about those exceptions. The +files you write will require this #include line.) + +The "#if INTERFACE ... #endif" section is optional and is only needed +if there are structure definitions or typedefs or macros that need to +be used by other source code files. The makeheaders preprocessor +uses definitions in the INTERFACE section to help it generate header +files. See [../tools/makeheaders.html | makeheaders.html] for additional +information. + +After creating a template file such as shown above, and after updating +the makefiles, you should be able to recompile Fossil and have it include +your new source file, even before your source file contains any code. +It is recommended that you try this. + +Be sure to [/help/add|fossil add] your new source file to the self-hosting +Fossil repository and then [/help/commit|commit] your changes! + +

4.0 Creating A New Command

+ +By "commands" we mean the keywords that follow "fossil" when invoking +Fossil from the command-line. So, for example, in + + +fossil diff xyzzy.c + + +The "command" is "diff". Commands may optionally be followed by +arguments and/or options. To create new commands in Fossil, add code +(either to an existing source file, or to a new source file created as +described above) according to the following template: + + +/* +** COMMAND: xyzzy +** +** Help text goes here. Backslashes must be escaped. +*/ +void xyzzy_cmd(void){ + /* Implement the command here */ + fossil_print("Hello, World!\n"); +} + + +The example above creates a new command named "xyzzy" that prints the +message "Hello, World!" on the console. This command is a normal command +that will show up in the list of command from [/help/help|fossil help]. +If you add an asterisk to the end of the command name, like this: + + +** COMMAND: xyzzy* + + +Then the command will only show up if you add the "--all" option to +[/help/help|fossil help]. Or, if the command name starts with +"test" then the command will be considered experimental and will only +show up when the --test option is used with [/help/help|fossil help]. + +The example above is a fully functioning Fossil command. You can add +the text shown to an existing Fossil source file, recompiling then test +it out by typing: + + +./fossil xyzzy +./fossil help xyzzy +./fossil xyzzy --help + + +The name of the C function that implements the command can be anything +you like (as long as it does not collide with some other symbol in the +Fossil code) but it is traditional to name the function +"commandname_cmd", as is done in the example. + +You could also use "printf()" instead of "fossil_print()" to generate +the output text, if desired. But "fossil_print()" is recommended as +it has extra logic to insert \r characters at the right times on +Windows systems. + +Once you have the command running, you can then start adding code to +make it do useful things. There are lots of utility functions in +Fossil for parsing command-line options and for +opening and accessing and manipulating the repository and +the working check-out. Study implementations of existing commands +to get an idea of how things are done. You can easily find the implementations +of existing commands by searching for "COMMAND: name" in the +files of the "src/" directory. + +

5.0 Creating A New Web Page

+ +As with commands, new webpages can be added simply by inserting a function +that generates the webpage together with a special header comment. A +template follows: + + +/* +** WEBPAGE: helloworld +*/ +void helloworld_page(void){ + style_header("Hello World!"); + @

Hello, World!

+ style_footer(); +} + + +Add the code above to a new or existing Fossil source code file, then +recompile fossil and run [/help/ui|fossil ui] then enter +"http://localhost:8080/helloworld" in your web browser and the routine +above will generate a web page that says "Hello World." +It really is that simple. + +The special "WEBPAGE:" comment is picked up by the "mkindex" preprocessor +and used to generate a table that maps the "helloworld" webpage name +into a pointer to the "helloworld_page()" function. The function that +implements a webpage can be named anything you like (as long as it does +not collide with another name) but the traditional name is +"pagename_page". + +HTML pages begin with a call to style_header() and end with the call to +style_footer(). Content is generated by the "@" lines that are translated +(by the "translate" preprocessor) into printf-like code that generates the +content of the webpage. Different techniques are used to generate +non-HTML content. In the unlikely event that you need to generate +non-HTML content, look at existing webpage implementations +(ex: "logo" or "style.css") to see how that is done. + +There are lots of other things that a real web-page implementation will +need to do, such as verifying user credentials, parsing query parameters, +and interacting with the repository. But now that you have the general +idea of how webpages are implemented, you can look at the many other +webpage implementations already built into Fossil to see how all that +works. + +

6.0 See Also

+ + * [./makefile.wiki|The Fossil Build Process] + * [./tech_overview.wiki|A Technical Overview Of Fossil] + * [./contribute.wiki|Contributing To The Fossil Project] + * [./serverext.wiki|Adding CGI Extensions To A Fossil Server] diff --git a/www/alerts.md b/www/alerts.md new file mode 100644 index 0000000..6a516e9 --- /dev/null +++ b/www/alerts.md @@ -0,0 +1,727 @@ +# Email Alerts + +## Overview + +Beginning with version 2.7, Fossil can send email messages to +subscribers to alert them to changes in the repository: + + * New [checkins](/help/ci) + * [Ticket](./tickets.wiki) changes + * [Wiki](./wikitheory.wiki) page changes + * New and edited [forum](./forum.wiki) posts + * Users receiving [new permissions](./caps/index.md) (admins only) + * Announcements + +Subscribers can elect to receive emails as soon as these events happen, +or they can receive a daily digest of the events instead. + +Email alerts are sent by a [Fossil server](./server/), which must be +[set up](#quick) by the Fossil administrator to send email. + +Email alerts do not currently work if you are only using Fossil from the +command line. + +A bit of terminology: Fossil uses the terms "email alerts" and +"notifications" interchangeably. We stick to the former term in this +document except when referring to parts of the Fossil UI still using the +latter term. + + +## Setup Prerequisites + +Much of this document describes how to set up Fossil's email alert +system. To follow this guide, you will need a Fossil UI browser window +open to the [Admin → Notification](/setup_notification) Fossil UI screen +on the Fossil server that will be sending these email alerts, logged +in as a user with [**Admin** capability](./caps/ref.html#a). It is not possible to work on a +clone of the server's repository and push the configuration changes up +to that repo as an Admin user, [on purpose](#backup). + + +You will also need a CLI window open with its working directory changed +to a checkout directory of the Fossil repository you are setting up to +send email. If you don't `cd` to such a checkout directory first, +you'll need to add `-R /path/to/repo.fossil` to each `fossil` command +below to tell Fossil which repository you mean it to apply the command +to. + +There are other prerequisites for email service, but since they vary +depending on the configuration you choose, we'll cover these inline +below. + + + +## Quick Email Service Setup + +If you've already got a working OpenSMTPD, Postfix, Exim, Sendmail, +or similar server on the machine running your Fossil instance(s), +and you aren't using Fossil's [chroot jail feature](./chroot.md) +to wall Fossil off from the rest of the machine, it's +fairly simple to set up email alerts. + +(Otherwise, skip [ahead](#advanced) to the sections on advanced email +service setup.) + +This is our "quick setup" option even though setting up an SMTP mail +server is not trivial, because there are many other reasons to have such +a server set up already: internal project email service, `cron` +notifications, server status monitoring notifications... + +With that out of the way, the Fossil-specific steps are easy: + +1. Go to [Admin → Notification](/setup_notification) and fill out all + of the **Required** fields: + + * **Canonical server URL** — Use the suggested URL + + * **"From" email address** — `forum-bounces@example.com` is + traditional, but suit yourself + + * **Repository nickname** — See the suggested examples on the web page. + +2. Set "Email Send Method" to "Pipe to a command" + +3. Set the "Administrator email address" to a suitable valid email + address on that machine. It could be the same value you used for + the "From" address above, or it could be a different value like + `admin@example.com`. + +Save your changes. + +At the command line, say + + $ fossil set email-send-command + +If that gives a blank value instead of `sendmail -ti`, say + + $ fossil set email-send-command "sendmail -ti" + +to force the setting. That works around a [known +bug](https://fossil-scm.org/forum/forumpost/840b676410) which may be +squished by the time you read this. + +If you're running Postfix or Exim, you might think that command is +wrong, since you aren't running Sendmail. These mail servers provide a +`sendmail` command for compatibility with software like Fossil that has +no good reason to care exactly which SMTP server implementation is +running at a given site. There may be other SMTP servers that also +provide a compatible `sendmail` command, in which case they may work +with Fossil using the same steps as above. + + +If you reload the Admin → Notification page, the Status section at the +top should show: + + Outgoing Email: Piped to command "sendmail -ti" + Pending Alerts: 0 normal, 0 digest + Subscribers: 0 active, 0 total + +Before you move on to the next section, you might like to read up on +[some subtleties](#pipe) with the "pipe to a command" method that we did +not cover above. + + + +## Usage and Testing + +Now that email service from Fossil is set up, you can test it and begin +using it. + + + +### Subscribing to Alerts + +In the Status output above, we saw that there are no subscribers, so the +next step is to add the first one. + +Go to the `/subscribe` page on your Fossil instance to sign up for email +alerts. At the very least, you will need to sign up for "Forum Posts" +and "Announcements" to complete the testing steps below. + +If you're logged in with a Fossil repository user account and put the +same user name and email address into this forum as you used for your +user information under Admin → Users, Fossil will simply tie your alert +preferences to your login record, and the email address in your user's +Contact Info field will be considered already-verified. Otherwise, +Fossil will create an alert-only record, and you will have to verify the +email address before Fossil will send alerts to it. + +This shows a key aspect of the way Fossil's email alerts system works, +by the way: a user can be signed up for email alerts without having a +full-fledged Fossil user account. Only when both user names are the same +are the two records tied together under the hood. For more on this, see +[Users vs Subscribers below](#uvs). + +If you are seeing the following complaint from Fossil: + +> Use a different login with greater privilege than FOO to access /subscribe + +...then the repository's administrator forgot to give the +[**EmailAlert** capability][cap7] +to that user or to a user category that the user is a member of. + +After a subscriber signs up for alerts for the first time, a single +verification email is sent to that subscriber's given email address. +The new subscriber must click a link in that email in order to activate +the subscription. + +Subscription verification emails are only sent once. This is a defense +against malicious robots that try to harass innocent Internet users by +having subscription pages send multiple verification emails. If the +initial subscription verification does not go through correctly, an +administrator must [intervene](#admin) to reset the subscription. + +Every subscriber-only email address has a [long random hexadecimal +security code](#scode) that serves in place of a password. All email +alerts contain a link in their footer back to the Fossil server, +incorporating this security code, which allows the subscriber to adjust +their subscription options. If a user doesn't have any of those emails, +they can request a link via email by visiting the `/alerts` or +`/unsubscribe` page on the repository. + +Those with Fossil repository logins can adjust their email alert +settings by visiting the `/alerts` page on the repository. With the +default skin, you can get there by clicking the "Logout" link in the +upper right corner of any Fossil UI page then clicking the "Email +Alerts" link. That link is also available via the Sitemap (`/sitemap`) +and via the default skin's hamburger menu (☰). + +[cap7]: ./caps/ref.html#7 + + + +### Unsubscribing + +To unsubscribe from alerts, visit the `/alerts` page on the repository, +click the "Unsubscribe" button, then check the "Unsubscribe" checkbox to +verify your action and press the "Unsubscribe" button a second time. + +This interlock is intended to prevent accidental unsubscription. + + + +### Test Email Service + +The easiest way to test email sending from Fossil is via the "[Send +Announcement](/announce)" link at the top of the "Email Notification +Setup" page. Put your email address in the "To:" line and a test +message below, then press "Send Message" to verify that outgoing email +is working. + +Another method is from the command line: + + $ fossil alerts test-message you@example.com --body README.md --subject Test + +That should send you an email with "Test" in the subject line and the +contents of your project's `README.md` file in the body. + +That command assumes that your project contains a "readme" file, but of +course it does, because you have followed the [Programming Style Guide +Checklist][cl], right? Right. + +[cl]: https://sendgrid.com/blog/programming-style-guide-checklist/ + + + +### User Capabilities + +Once email alerts are working, you may need to [adjust the default user +capabilities](./caps/) to give "[Email Alerts][cap7]" capability to any +[user category](./caps/#ucat) or [individual user](./caps/#ucap) that +needs to use the subscription setup pages, `/subscribe` and `/alerts`. +[**Admin**][capa] and [**Setup**][caps] users always have this +capability. + +To allow any passer-by on the Internet to subscribe, give the "Email +Alerts" capability to the "nobody" user category. To require that a +person solve a simple CAPTCHA first, give that capability to the +"anonymous" user category instead. + +[capa]: ./caps/ref.html#a +[caps]: ./caps/ref.html#s + + + +### First Post + +I suggest taking the time to compose a suitable introductory message +especially for your project's forum, one which a new user would find +helpful. + +Wait a few seconds, and you should receive an email alert with the +post's subject and body text in the email. + + + +### Troubleshooting + +If email alerts aren't working, there are several useful commands you +can give to figure out why. + +(Be sure to [`cd` into a repo checkout directory](#cd) first!) + + $ fossil alerts status + +This should give much the same information as you saw [above](#status). +One difference is that, since you've created a forum post, the +`pending-alerts` value should only be zero if you did in fact get the +requested email alert. If it's zero, check your mailer's spam folder. If +it's nonzero, continue with these troubleshooting steps. + + $ fossil backoffice + +That forces Fossil to run its ["back office" process](./backoffice.md). +Its only purpose at the time of this writing is to push out alert +emails, but it might do other things later. Sometimes it can get stuck +and needs to be kicked. For that reason, you might want to set up a +crontab entry to make sure it runs occasionally. + + $ fossil alerts send + +This should also kick off the backoffice processing, if there are any +pending alerts to send out. + + $ fossil alert pending + +Show any pending alerts. The number of lines output here should equal +the [status output above](#status). + + $ fossil test-add-alerts f5900 + $ fossil alert send + +Manually create an email alert and push it out immediately. + +The `f` in the first command's final parameter means you're scheduling a +"forum" alert. The integer is the ID of a forum post, which you can find +by visiting `/timeline?showid` on your Fossil instance. + +The second command above is necessary because the `test-add-alerts` +command doesn't kick off a backoffice run. + + $ fossil ale send + +This only does the same thing as the final command above, rather than +send you an ale, as you might be hoping. Sorry. + + + +## Advanced Email Setups + +Fossil offers several methods of sending email: + + 1. Pipe the email message text into a command. + 2. Store email messages as entries in an SQLite database. + 3. Store email messages as individual files in a directory. + 4. Send emails to an SMTP relay. + 5. Send emails directly to the recipients via SMTP. + +This wide range of options allows Fossil to talk to pretty much any +SMTP setup. + +The first four options let Fossil delegate email handling to an existing +[MTA] so that Fossil does not need to implement the [roughly two +dozen][mprotos] separate [RFCs][rfcs] required in order to properly +support SMTP email in this complex world we've built. As well, this +design choice means you do not need to do duplicate configuration, such +as to point Fossil at your server's TLS certificate in order to support +users behind mail servers that require STARTTLS encryption. + +[mprotos]: http://sqlite.1065341.n5.nabble.com/Many-ML-emails-going-to-GMail-s-SPAM-tp98685p98722.html +[rfcs]: https://en.wikipedia.org/wiki/Request_for_Comments + + + +### Method 1: Pipe to a Command + +This is our ["quick setup" option](#quick) above, but there are some +details we ignored which we'll cover now. + +Fossil pipes the email message in [RFC 822 format][rfc822] to the +standard input of the command you gave as the "Email Send Method", +defaulting to `sendmail -ti`. This constitutes a protocol between Fossil +and the SMTP [message transfer agent (MTA)][MTA]. Any other MTA which +speaks the same protocol can be used in place of the most common +options: Sendmail, Exim, and Postfix. + +The `-t` option tells the command to expect the list of email recipients +in a `To` header in the RFC 822 message presented on its standard input. +Without this option, the `sendmail` command expects to receive the +recipient list on the command line, but that's not possible with the +current design of this email sending method. Therefore, if you're +attempting to use a less common MTA which cannot parse the recipient +list from the `To` header in the email message, you might need to look +for a different MTA. + +The `-i` option is only needed for MTAs that take a dot/period at the +beginning of a line of standard input text as "end of message." Fossil +doesn't attempt to escape such dots, so if the line wrapping happens to +occur such that a dot or period in an alert message is at the beginning +of a line, you'll get a truncated email message without this option. +Statistically, this will happen about once every 70 or so messages, so +it is important to give this option if your MTA treats leading dots on a +line this way. + + +The [`msmtp`][msmtp] SMTP client is compatible with this +protocol if you give it the `-t` option. It’s a useful option on a server +hosting a Fossil repository which doesn't otherwise require a separate +SMTP server for other purposes, such as because you’ve got a separate +provider for your email and merely need a way to let Fossil feed +messages into it. + +It is probably also possible to configure [`procmail`][pmdoc] to work +with this protocol. If you know how to do it, a patch to this document +or a how-to on [the Fossil forum][ff] would be appreciated. + +[ff]: https://fossil-scm.org/forum/ +[msmtp]: https://marlam.de/msmtp/ +[MTA]: https://en.wikipedia.org/wiki/Message_transfer_agent +[pmdoc]: http://pm-doc.sourceforge.net/doc/ +[rfc822]: https://www.w3.org/Protocols/rfc822/ + + + +### Method 2: Store in a Database + +The self-hosting Fossil repository at +currently uses this method rather than [the pipe method](#pipe) because +it is running inside of a restrictive [chroot jail][cj] which is unable +to hand off messages to the local MTA directly. + +When you configure a Fossil server this way, it adds outgoing email +messages to an SQLite database file. A separate daemon process can then +extract those messages for further disposition. + +Fossil uses a short TCL script (seen at [](/file/tools/email-sender.tcl)) +that continuously monitors this database for new messages and hands any that +it finds off to a local MTA using the same [pipe to MTA protocol](#pipe) +as above. + +In this way, outbound email alerts escape the chroot jail without +requiring that we insert a separate MTA configuration inside that jail. +We only need to arrange that the same SQLite DB file be visible both +inside and outside the chroot jail, which we do by naming the database +file in the "Store Emails In This Database" setting under Admin → +Notification. The Tcl script has this path hard-coded as +`/home/www/fossil/emailqueue.db`, but you will probably need to adjust +that for your local purposes. + +This method may work with other similar technologies besides `chroot`: +Docker containers, LXC containers, BSD jails, Solaris zones, etc. + +With suitable file share mappings, this method may even work with +virtual machine or distributed computing setups where the MTA and Fossil +servers are not on the same machine, though beware the [risk of DB +corruption][rdbc] if used with a file sharing technology that doesn't +use proper file locking. + +You can start this Tcl script as a daemon automatically on most Unix and +Unix-like systems by adding the following line to the `/etc/rc.local` +file of the server that hosts the repository sending email alerts: + + /usr/bin/tclsh /home/www/fossil/email-sender.tcl & + +[cj]: https://en.wikipedia.org/wiki/Chroot +[rdbc]: https://www.sqlite.org/howtocorrupt.html#_filesystems_with_broken_or_missing_lock_implementations + + + +### Method 3: Store in a Directory + +This method is functionally very similar to [the DB method](#db), +differing only in that messages are written to a directory in the +filesystem. You should therefore read that section and make the minor +adjustments required by the storage method. + +This method may work over a file sharing mechanism that doesn't do file +locking properly, as long as the reading process is somehow restricted +from reading a message file as it's being written. + +It might be useful in testing and debugging to temporarily switch to +this method, since you can easily read the generated email messages +without needing to involve an [MTA]. + + + +### Method 4: SMTP Relay + +In this configuration, the Fossil server contacts an open SMTP relay and +sends the messages to it. This method is only appropriate when: + +1. You have a local MTA that doesn't accept [the pipe + protocol](#pipe). + +2. The MTA is willing to accept anonymous submissions, since Fossil + currently has no way to authenticate itself to the MTA. This is [an + unsafe configuration][omr] in most cases, but some SMTP servers make + an exception for connections coming from a `localhost` or LAN + address, choosing to accept such submissions as inherently safe. + +If you have a local MTA meeting criterion #1 but not #2, we'd suggest +using a more powerful SMTP client such as [msmtp](#msmtp) along with one +of the other methods above. + +[omr]: https://en.wikipedia.org/wiki/Open_mail_relay + + + +### Method 5: Direct SMTP Send + +As of Fossil 2.7, the code to support this method is incomplete, so you +cannot currently select it as an option in Admin → Notification. + + + +## Users vs Subscribers + +Fossil makes a distinction between "users" and "subscribers". A user is +someone with a username and password: that is, someone who can log into +the Fossil repository. A subscriber is someone who receives email +alerts. Users can also be subscribers and subscribers can be users, but +that does not have to be the case. It is possible to be a user without +being a subscriber and to be a subscriber without being a user. + +In the repository database file, users are tracked with the `user` table +and subscribers are tracked via the `subscriber` table. + + + +## Administrator Activities + +The "[List Subscribers](/subscribers)" button at the top of the Admin → +Notification screen gives a list of subscribers, which gives a Fossil +server administrator a lot of power over those subscriptions. + +Clicking an email address in this subscriber list opens the same +`/alerts` page that the user can see for their own subscription, but +with more information and functionality than normal users get: + +* Subscription creation and modification timestamps. + +* The IP address the user had when they last made a change via either + `/subscribe` or `/alert`. + +* The user's login name, if they are not [a mere subscriber](#uvs). A + Fossil Admin user is allowed to modify this, either to tie a + subscription-only record to an existing Fossil user account or to + break that tie. + +* The "Do not call" checkbox allows a Fossil Admin user to mark a given + email address so that Fossil never sends email to that address. This + is distinct from unsubscribing that email address because it prevents + Fossil from accepting a new subscription for that address. + +* The Verified checkbox is initially unchecked for subscriber-only + email addresses until the user clicks the link in the verification + email. This checkbox lets the Fossil Admin user manually verify the + user, such as in the case where the verification email message got + lost. Unchecking this box does not cause another verification email + to be sent. + +* Admin users (only) may activate the "user elevation" subscription, + which sends a notification when a user is created or is explicitly + assigned permission they did not formerly have. + +This screen also allows a Fossil Admin user to perform other activities +on behalf of a subscriber which they could do themselves, such as to +[unsubscribe](#unsub) them. + + + +## Cloning, Syncing, and Backups + +That’s [covered elsewhere](./backup.md#alerts). + + + +## Controlling the Email Alert System + +This section collects the list of Fossil UI pages and CLI commands that +control the email alert system, some of which have not been mentioned so +far: + +Commands: + + * The [`alerts`](/help/alerts) command + * The [`test-alert`](/help/test-alert) command + * The [`test-add-alerts`](/help/test-add-alerts) command + +Web pages available to users and subscribers: + + * The [`/subscribe`](/help/www/subscribe) page + * The [`/alerts`](/help/www/alerts) page + * The [`/unsubscribe`](/help/www/unsubscribe) page + * The [`/renew`](/help/www/renew) page + * The [`/contact_admin`](/help/www/contact_admin) page + +Administrator-only web pages: + + * The [`/setup_notification`](/help/www/setup_notification) page + * The [`/subscribers`](/help/www/subscribers) page + + + +## Design of Email Alerts + +This section describes the low-level design of the email alert system in +Fossil. This expands on the high-level administration focused material +above with minimal repetition. + +This section assumes expert-level systems knowledge. If the material +above sufficed for your purposes, feel free to skip this section, which +runs to the end of this document. + + + +### Data Design + +There are two new tables in the repository database, starting with +Fossil 2.7. These tables are not created in new repositories by +default. The tables only come into existence as needed when email +alerts are configured and used. + + * SUBSCRIBER → + The subscriber table records the email address for people who + want to receive email notifications. Each subscriber has a + `subscriberCode` which is a random 32-byte blob that uniquely + identifies the subscriber. There are also fields to indicate + what kinds of notifications the subscriber wishes to receive, + whether or not the email address of the subscriber has been + verified, etc. + + * PENDING\_ALERT → + The PENDING\_ALERT table contains records that define events + about which alert emails might need to be sent. + A pending\_alert always refers to an entry in the + EVENT table. The EVENT table is part of the standard schema + and records timeline entries. In other words, there is one + row in the EVENT table for each possible timeline entry. The + PENDING\_ALERT table refers to EVENT table entries for which + we might need to send alert emails. + +As pointed out above, ["subscribers" are distinct from "users"](#uvs). +The SUBSCRIBER.SUNAME field is the optional linkage between users and +subscribers. + + + +### The "stdout" Method + +The [list of mail sending methods](#advanced) above left out an +internal-only method called "stdout" which simply writes the text of the +email message on standard output. The "stdout" method is used for +testing and debugging. If you need something similar and can't modify +your local Fossil instance to use this method, you might temporarily +switch to [the "dir" method](#dir) instead. + + + +### Message Format + +The email messages generated by Fossil have a [well-formed +header][rfc822]. The downstream processing is expected to extract the +"To:", "From:", "Subject:" and whatever other attributes it needs from +the email header text. + +These emails use the `text/plain` MIME type with the UTF-8 character +set. We currently use a transfer encoding of `quoted-printable`, but +there is commented-out code in Fossil to switch to `base64` encoding, +which Fossil used in the early days leading up to the 2.7 release. + +If you switch Fossil back to `base64` mode, you may want to build a +utility program that ships in the Fossil source tree named +["tools/decode-email.c"](/file/tools/decode-email.c) which can decode +these messages into a human-readable format. + + + +### Dealing with Inbound Email + +Inbound email messages — for example, bounces from failed alert emails — +should be relayed to the `fossil email inbound` command. That command +is currently a no-op place-holder. At some point, we will need to +design and write a bounce-message processing system for Fossil. + + + +### Passwords vs Subscriber Codes + +When anonymous passers-by on the Internet sign up for email alerts, +their email address must first be verified. An email message is sent to +the address supplied inviting the user to click on a link. The link +includes a pseudorandom 128-bit blob encoded as 32 hexadecimal digits, +which serves in place of a password for that email address. (This is +stored in the database as `subscriber.subscriberCode`.) If anyone visits +the link, the email address is verified. + +Knowledge of the `subscriberCode` is sufficient to control a +subscription. + +Because this code is included in plain text in email alert messages, it +is not as secure as a separate password, but it has several virtues: + +* It is easier for the average subscriber to deal with in that they + don't have to come up with yet another password and store it safely. + +* If the `subscriberCode` is stolen, the worst that can happen is that + the thief can change that email address's subscription settings. + Contrast a password which may be shared with other services, which + then compromises those other services. + +* No PII other than the subscriber's email address is available to an + attacker with the `subscriberCode`. Nor can knowledge of the + `subscriberCode` lead to an email flood or other annoyance attack, as + far as I can see. + +If the `subscriberCodes` for a Fossil repository are ever compromised, +new ones can be generated as follows: + + UPDATE subscriber SET subscriberCode=randomblob(32); + +Since this then affects all new email alerts going out from Fossil, your +end users may never even realize that they're getting new codes, as long +as they don't click on the URLs in the footer of old alert messages. + +With that in mind, a Fossil server administrator could choose to +randomize the `subscriberCodes` periodically, such as just before the +daily digest emails are sent out each day. + +**Important:** All of the above is distinct from the passwords for users +with a Fossil repository login. Such users also have subscriber codes, +but those codes can only be used to modify the user's email alert +settings. That code cannot allow a user to log into the user's Fossil +repository account. + + + +### Internal Processing Flow + +Almost all of the email alert code is found in the +[`src/alerts.c`](/file/src/alerts.c) source file. + +When email alerts are enabled, a trigger is created in the schema +(`email_trigger1`) that adds a new entry to the `PENDING_ALERT` table +every time a row is added to the `EVENT` table. During a +`fossil rebuild`, the `EVENT` table is rebuilt from scratch; since we do not +want users to get alerts for every historical check-in, the trigger is +disabled during `rebuild`. + +Email alerts are sent out by the `alert_send_alerts()` function, which +is normally called automatically due to the `email-autoexec` setting, +which defaults to enabled. If that setting is disabled or if the user +simply wants to force email alerts to be sent immediately, they can give +a `fossil alert send` command, such as via a `cron` script. Each time +this function is called, the alert messages are moved further down the +chain, so you cannot cause duplicate alerts by calling it too often. + +Digests are handled by recording the time of the last digest in the +`email-last-digest` setting, and only sending a new digest if the +current time is one day or later after the last digest. + +Individual emails are sent to each subscriber. I (drh) ran tests and +found that I could send about 1200 emails/second, which is fast enough +that I do not need to resort to trying to notify multiple subscribers +with a single email. Because each subscriber gets a separate email, the +system can include information in the email that is unique to the +subscriber, such as a link to the page to edit their subscription. That +link includes the `subscriberCode`. diff --git a/www/antibot.wiki b/www/antibot.wiki new file mode 100644 index 0000000..c38cfec --- /dev/null +++ b/www/antibot.wiki @@ -0,0 +1,276 @@ +Defense Against Robots + +A typical Fossil website can have billions and billions of pages, +and many of those pages (for example diffs and annotations and tarballs) +can be expensive to compute. +If a robot walks a Fossil-generated website, +it can present a crippling bandwidth and CPU load. +A "robots.txt" file can help, but in practice, most robots these +days ignore the robots.txt file, so it won't help much. + +A Fossil website is intended to be used +interactively by humans, not walked by robots. This article +describes the techniques used by Fossil to try to welcome human +users while keeping out robots. + +

Defenses Are Enabled By Default

+ +In the latest implementations of Fossil, most robot defenses are +enabled by default. You can probably get by with standing up a +public-facing Fossil instance in the default configuration. But +you can also customize the defenses to serve your particular needs. + +

Customizing Anti-Robot Defenses

+ +Admin users can configure robot defenses on the +"Robot Defense Settings" page (/setup_robot). +That page is accessible (to Admin users) from the default menu bar +by click on the "Admin" menu choice, then selecting the +"Robot-Defense" link from the list. + +

The Hyperlink User Capability

+ +Every Fossil web session has a "user". For random passers-by on the internet +(and for robots) that user is "nobody". The "anonymous" user is also +available for humans who do not wish to identify themselves. The difference +is that "anonymous" requires a login (using a password supplied via +a CAPTCHA) whereas "nobody" does not require a login. +The site administrator can also create logins with +passwords for specific individuals. + +Users without the [./caps/ref.html#h | Hyperlink] capability +do not see most Fossil-generated hyperlinks. This is +a simple defense against robots, since [./caps/#ucat | the "nobody" +user category] does not have this capability by default. +Users must log in (perhaps as +"anonymous") before they can see any of the hyperlinks. A robot +that cannot log into your Fossil repository will be unable to walk +its historical check-ins, create diffs between versions, pull zip +archives, etc. by visiting links, because there are no links. + +A text message appears at the top of each page in this situation to +invite humans to log in as anonymous in order to activate hyperlinks. + +But requiring a login, even an anonymous login, can be annoying. +Fossil provides other techniques for blocking robots which +are less cumbersome to humans. + +

Automatic Hyperlinks Based on UserAgent and Javascript

+ +Fossil has the ability to selectively enable hyperlinks for users +that lack the Hyperlink capability based on their UserAgent string in the +HTTP request header and on the browsers ability to run Javascript. + +The UserAgent string is a text identifier that is included in the header +of most HTTP requests that identifies the specific maker and version of +the browser (or robot) that generated the request. Typical UserAgent +strings look like this: + + + +The first two UserAgent strings above identify Firefox 19 and +Internet Explorer 8.0, both running on Windows NT. The third +example is the robot used by Google to index the internet. +The fourth example is the "wget" utility running on OpenBSD. +Thus the first two UserAgent strings above identify the requester +as human whereas the second two identify the requester as a robot. +Note that the UserAgent string is completely under the control +of the requester and so a malicious robot can forge a UserAgent +string that makes it look like a human. But most robots want +to "play nicely" on the internet and are quite open +about the fact that they are a robot. And so the UserAgent string +provides a good first-guess about whether or not a request originates +from a human or a robot. + +The [/help/auto-hyperlink|auto-hyperlink] setting, shown as +"Enable hyperlinks based on User-Agent and/or Javascript" on +the Robot Defense Settings page, +can be set to "HTTP Header Only" or "HTTP Header and Javascript" or "off". +If the UserAgent string looks like a human and not a robot, then +Fossil will enable hyperlinks even if the Hyperlink capability +is omitted from the user permissions. This setting gives humans easy +access to the hyperlinks while preventing robots +from walking the billions of pages on a typical Fossil site. + +If the setting is "HTTP Header Only" (2), then the hyperlinks are simply +enabled and that is all. But if the setting is "HTTP Header and Javascript" (1), +then the hyperlinks are not enabled directly. +Instead, the HTML code that is generated contains anchor tags ("<a>") +with "href=" attributes that point to [/honeypot] rather than the correct +link. JavaScript code is added to the end of the page that goes back and +fills in the correct "href=" attributes of +the anchor tags with the true hyperlink targets, thus enabling the hyperlinks. +This extra step of using JavaScript to enable the hyperlink targets +is a security measure against robots that forge a human-looking +UserAgent string. Most robots do not bother to run JavaScript and +so to the robot the empty anchor tag will be useless. But all modern +web browsers implement JavaScript, so hyperlinks will show up +normally for human users. +(UPDATE 2026-08-06: The previous two sentences +were true when this document was first written in 2013, but they are +not true any more. Most robots these days have normal-looking UserAgent +strings and do run Javascript.) + +If the [/help/auto-hyperlink|"auto-hyperlink"] setting is (2) +"Enable hyperlinks using HTTP-Header and/or Javascript", +then there are now two additional sub-settings that control when +hyperlinks are enabled. + +The first new sub-setting is a delay (in milliseconds) before setting +the "href=" attributes on anchor tags. The default value for this +delay is 10 milliseconds. The idea here is that a robots will try to +interpret the links on the page immediately, and will not wait for delayed +scripts to be run, and thus will never enable the true links. + +The second sub-setting waits to run the +JavaScript that sets the "href=" attributes on anchor tags until after +at least one "mousedown" or "mousemove" event has been detected on the +<body> element of the page. The thinking here is that robots will not be +simulating mouse motion and so no mouse events will ever occur and +hence the hyperlinks will never become enabled for robots. + +See also [./loadmgmt.md|Managing Server Load] for a description +of how expensive pages can be disabled when the server is under heavy +load. + +

Do Not Allow Robot Access To Certain Pages

+ +The [/help/robot-restrict|robot-restrict setting] is a comma-separated +list of GLOB patterns for pages for which robot access is prohibited. +The default value is: + +
+timelineX,diff,annotate,fileage,file,finfo,reports,tree,hexdump,download
+
+ +Each entry corresponds to the first path element on the URI for a +Fossil-generated page. If Fossil does not know for certain that the +HTTP request is coming from a human, then any attempt to access one of +these pages brings up a javascript-powered captcha. The user has to +click the accept button the captcha once, and that sets a cookie allowing +the user to continue surfing without interruption for 15 minutes or so +before being presented with another captcha. + +To disable this feature (not recommended for public-facing websites) +change this setting to "off". If robots become particularly troublesome, +change this setting to "*" (the GLOB wildcard) which restricts robots +from accessing any page of the Fossil webpage other than "/login". +Robots have gotten so bad lately (as of 2026) that the use of "*" is +advisable for any public-facing website that is running on a low-power +platform such as a RaspberryPI or a $6/month VPS, or any platform with +limited bandwidth. + +Some path elements have special meanings: + + * timelineX → + This means a subset of /timeline/ pages that are considered + "expensive". The exact definition of which timeline pages are + expensive and which are not is still the subject of active + experimentation and is likely to change by the time you read this + text. The idea is that anybody (including robots) can see a timeline + of the most recent changes, but timelines of long-ago change or that + contain lists of file changes or other harder-to-compute values are + prohibited. + + * zip → + The special "zip" keyword also matches "/tarball/" and "/sqlar/". + + * zipX → + This is like "zip" in that it restricts access to "/zip/", "/tarball"/ + and "/sqlar/" but with exceptions:
    +

  1. If the [/help/robot-zip-leaf|robot-zip-leaf] setting is + true, then tarballs of leaf check-ins are allowed. This permits + URLs that attempt to download the latest check-in on trunk or + from a named branch, for example. +

  2. If a check-in has a tag that matches the GLOB list in + [/help/robot-zip-tag|robot-zip-tag], then tarballs of that + check-in are allowed. This allow check-ins tagged with + "release" or "allow-robots" (for example) to be downloaded + without restriction. +

+ The "zipX" restriction is not in the default robot-restrict setting. + This is something you might want to add, depending on your needs. + + * diff → + This matches /vdiff/ and /fdiff/ and /vpatch/ and any other page that + is primarily about showing the difference between two check-ins or two + file versioons. + + * annotate → + This also matches /blame/ and /praise/. + +Other special keywords may be added in the future. + +The default [/help/robot-restrict|robot-restrict] +setting has been shown (circa 2026) to do a reasonable job of keeping +robots from consuming all available CPU and bandwidth while will +still allowing humans access to the full power of the site without +having to be logged in. + +One possible enhancement is to add "zipX" to the +[/help/robot-restrict|robot-restrict] setting, +and enable [help?cmd=robot-zip-leaf|robot-zip-leaf] +and configure [help?cmd=robot-zip-tag|robot-zip-tag]. +Do this if you find that robots downloading lots of +obscure tarballs is causing load issues on your site. + +

Anti-robot Exception RegExps

+ +The [/help/robot-exception|robot-exception setting] under the name +of Exceptions to anti-robot restrictions is a list of +[/re_rules|regular expressions], one per line, that match +URIs that will bypass the captcha and allow robots full access. The +intent of this setting is to allow automated build scripts +to download specific tarballs of project snapshots. + +The recommended value for this setting allows robots to use URIs of the +following form: + +
+https://DOMAIN/tarball/release/HASH/NAME.tar.gz +
+ +The HASH part of this URL can be any valid +[./checkin_names.wiki|check-in name]. The link works as long as that +check-in is tagged with the "release" symbolic tag. In this way, +robots are permitted to download tarballs (and ZIP archives) of official +releases, but not every intermediate check-in between releases. Humans +who are willing to click the captcha can still download whatever they +want, but robots are blocked by the captcha. This prevents aggressive +robots from downloading tarballs of every historical check-in of your +project, once per day, which many robots these days seem eager to do. + +For example, on the Fossil project itself, this URL will work, even for +robots: + +
+https://fossil-scm.org/home/tarball/release/version-2.27/fossil-scm.tar.gz +
+ +But the next URL will not work for robots because check-in 3bbd18a284c8bd6a +is not tagged as a "release": + +
+https://fossil-scm.org/home/tarball/release/3bbd18a284c8bd6a/fossil-scm.tar.gz +
+ +The second URL will work for humans, just not robots. + +

The Ongoing Struggle

+ +Fossil currently does a good job of providing easy access to humans +while keeping out troublesome robots. However, robots +continue to grow more sophisticated, requiring ever more advanced +defenses. This "arms race" is unlikely to end. The developers of +Fossil will continue to try improve the robot defenses of Fossil so +check back from time to time for the latest releases and updates. + +Readers of this page who have suggestions on how to improve the robot +defenses in Fossil are invited to submit your ideas to the Fossil Users +forum: +[https://fossil-scm.org/forum]. diff --git a/www/apple-touch-icon.png b/www/apple-touch-icon.png new file mode 100644 index 0000000..5b9178b Binary files /dev/null and b/www/apple-touch-icon.png differ diff --git a/www/background.jpg b/www/background.jpg new file mode 100644 index 0000000..f788555 Binary files /dev/null and b/www/background.jpg differ diff --git a/www/backoffice.md b/www/backoffice.md new file mode 100644 index 0000000..b49b7f9 --- /dev/null +++ b/www/backoffice.md @@ -0,0 +1,225 @@ +Backoffice +========== + +This is technical documentation about the internal workings of Fossil. +Ordinary Fossil users do not need to know about anything covered by this +document. The information here is intended for people who want to enhance +or extend the Fossil code, or who just want a deeper understanding of +the internal workings of Fossil. + +What Is The Backoffice +---------------------- + +The backoffice is a mechanism used by a +[Fossil server](./server/) to do low-priority +background work that is not directly related to the user interface. Here +are some examples of the kinds of work that backoffice performs: + + 1. Sending email alerts and notifications + 2. Sending out daily digests of email notifications + 3. Other background email handling chores + 4. Automatic syncing of peer repositories + 5. Repository maintenance and optimization + +(As of 2018-08-07, only items 1 and 2 have actually been implemented.) +The idea is that the backoffice handles behind-the-scenes work that does +not have tight latency requirements. + +When Backoffice Runs +-------------------- + +A backoffice process is usually launched automatically by a webpage +request. After each webpage is generated, Fossil checks to see if any +backoffice work needs to be done. If there is work to do, and no other +process is already assigned to do the work, then a new backoffice process +is started to do the work. + +This happens for every webpage, regardless of how that webpage is launched, +and regardless of the purpose of the webpage. This also happens on the +server for "[fossil sync](/help/sync)" and +"[fossil clone](/help/clone)" commands which are implemented as +web requests - albeit requests that the human user never sees. +Web requests can arrive at the Fossil server via direct TCP/IP (for example +when Fossil is started using commands like "[fossil server](/help/server)") +or via [CGI](./server/any/cgi.md) or +[SCGI](./server/any/scgi.md) or via SSH. +A backoffice process might be started regardless of the origin of the +request. + +The backoffice is not a daemon. Each backoffice process runs for a short +while and then exits. This helps keep Fossil easy to manage, since there +are no daemons to start and stop. To upgrade Fossil to a new version, +you simply replace the older "fossil" executable with the newer one, and +the backoffice processes will (within a minute or so) start using the new +one. (Upgrading the executable on Windows is more complicated, since on +Windows it is not possible to replace an executable file that is in active +use. But Windows users probably already know this.) + +The backoffice is serialized and rate limited. No more than a single +backoffice process will be running at once, and backoffice runs will not +occur more frequently than once every 60 seconds. (The 60-second spacing +is controlled by the BKOFCE_LEASE_TIME macro in the +[backoffice.c](/file/src/backoffice.c) source file.) + +If a Fossil server is idle, then no backoffice processes will be running. +That means there are no extra processes sitting around taking up memory +and process table slots for seldom accessed repositories. +The backoffice is an on-demand system. +A busy repository will usually have a backoffice +running at all times. But an infrequently accessed repository will only have +backoffice processes running for a minute or two following the most recent +access. + +Manually Running The Backoffice +------------------------------- + +The automatic backoffice runs are sufficient for most installations. +However, the daily digest of email notifications is handled by the +backoffice. If a Fossil server can sometimes go more than a day without +being accessed, then the automatic backoffice will never run, and the +daily digest might not go out until somebody does visit a webpage. +If this is a problem, an administrator can set up a cron job to +periodically run: + + fossil backoffice _REPOSITORY_ + +That command will cause backoffice processing to occur immediately. +Note that this is almost never necessary for an internet-facing +Fossil repository, since most repositories will get multiple accesses +per day from random robots, which will be sufficient to kick off the +daily digest emails. And even for a private server, if there is very +little traffic, then the daily digests are probably a no-op anyhow +and won't be missed. + +Automatic Backoffice Does Not Work On Some Systems +-------------------------------------------------- + +We have observed that the automatic backoffice does not work on +some systems - OpenBSD in particular. We still do not understand why +this is. (If you have insights, please share them on the +[Fossil Forum](https://fossil-scm.org/forum) so that we can perhaps +fix the problem.) For now, the backoffice must be run manually +on OpenBSD systems. + +To set up a fully-manual backoffice, first disable the automatic backoffice +using the "[backoffice-disable](/help/backoffice-disable)" setting. + + fossil setting backoffice-disable on + +Then arrange to invoke the backoffice separately using a command +like this: + + fossil backoffice --poll 30 _REPOSITORY-LIST_ + +Multiple repositories can be named. This one command will handle +launching the backoffice for all of them. There are additional useful +command-line options. See the "[fossil backoffice](/help/backoffice)" +documentation for details. + +The backoffice processes that are run manually using the "fossil backoffice" +command do not normally use a lease. That means that if you run the +"fossil backoffice" command with --poll and you forget to disable +automatic backoffice by setting the "backoffice-disable" flag, then +you might have one backoffice running due to a command and another due +to a webpage access, both at the same time. This is harmless. The +only downside is that it uses extra CPU time. + +How Backoffice Is Implemented +----------------------------- + +The backoffice is implemented by the +"[backoffice.c](/file/src/backoffice.c)" source file. + +Serialization and rate limiting is handled by a single entry in the +repository database CONFIG table named "backoffice". This entry is +called "the lease". The value of the lease +is a text string representing four integers, which +are respectively: + + 1. The process id of the "current" backoffice process + 2. The lease expiration time of the current backoffice process + 3. The process id of the "next" backoffice process + 4. The lease expiration time for the next backoffice process + +Times are expressed in seconds since 1970. A process id of zero means +"no process". Sometimes the process id will be non-zero even if there +is no corresponding process. Fossil knows how to figure out whether or +not a process still exists. + +You can print out a decoded copy of the current backoffice lease using +this command: + + fossil test-backoffice-lease -R _REPOSITORY_ + +If a system has been idle for a long time, then there will be no +backoffice processes. (Either the process id entries in the lease +will be zero, or there will exist no process associated with the +process id.) When a new web request comes in, the system +sees that no backoffice process is active and so it kicks off a separate +process to run backoffice. + +The new backoffice process becomes the "current" process. It sets a +lease expiration time for itself to be 60 seconds in the future. +Then it does the backoffice processing and exits. Note that usually +the backoffice process will exit long before its lease expires. That +is ok. The lease is there to limit the rate at which backoffice processes +run. + +If a new backoffice process starts up and sees that the "current" lease has +yet to expire, the new process makes itself the "next" backoffice process +and sets its expiration time to be 60 seconds past the expiration time of +the "current" backoffice process. The "next" process then puts itself to +sleep until the "current" lease expires. After the "current" +lease expires and the "current" process has itself exited, then +the "next" process promotes itself to the new "current" process. It +sets the current lease expiration to be 60 seconds in the future, runs +whatever backoffice work is needed, then exits. + +If a new backoffice process starts up and finds that there is already +a "current" lease and a "next" process, it exits without doing anything. +This should happen only rarely, since the lease information is checked +prior to spawning the backoffice process, so a conflict will only happen +in a race. + +Because the "backoffice" entry of the CONFIG table is in the repository +database, access to the lease is serialized. The lease prevents more +than one backoffice process from running at a time. It prevents +backoffice processes from running more frequently than once every 60 seconds. +And, it guarantees (assuming processes are not killed out-of-band) that +every web request will be followed within 60 seconds by a backoffice +run. + +Debugging The Backoffice +------------------------ + +The backoffice should "just work". It should not require administrator +attention. However, if you suspect that something is not working right, +there are some debugging aids. + +We have already mentioned the command that shows the backoffice lease +for a repository: + + fossil test-backoffice-lease -R _REPOSITORY_ + +Running that command every few seconds should show what is going on with +backoffice processing in a particular repository. + +There are also settings that control backoffice behavior. The +"backoffice-nodelay" setting prevents the "next" process from taking a +lease and sleeping. If "backoffice-nodelay" is set, that causes all +backoffice processes to exit either immediately or after doing whatever +backoffice works needs to be done. If something is going wrong and +backoffice leases are causing delays in webpage processing, then setting +"backoffice-nodelay" to true can work around the problem until the bug +can be fixed. The "backoffice-logfile" setting is the name of a log +file onto which is appended a short message every time a backoffice +process actually starts to do the backoffice work. This log file can +be used to verify that backoffice really is running, if there is any +doubt. The "backoffice-disable" setting prevents automatic backoffice +processing, if true. Use this to completely disable backoffice processing +that occurs automatically after each HTTP request. The "backoffice-disable" +setting does not affect the operation of the manual +"fossil backoffice" command. +Most installations should leave "backoffice-nodelay" and "backoffice-disable" +set to their default values of off and +leave "backoffice-logfile" unset or set to an empty string. diff --git a/www/backup.md b/www/backup.md new file mode 100644 index 0000000..198be34 --- /dev/null +++ b/www/backup.md @@ -0,0 +1,304 @@ +# Backing Up a Remote Fossil Repository + +One of the great benefits of Fossil and other [distributed version control systems][dvcs] +is that cloning a repository makes a backup. If you are running a project with multiple +developers who share their work using a [central server][server] and the server hardware +catches fire, the clones of the repository on each developer +workstation *may* serve as a suitable backup. + +[dvcs]: wikipedia:/wiki/Distributed_version_control +[server]: ./server/whyuseaserver.wiki + +We say “may” because +it turns out not everything in a Fossil repository is copied when cloning. You +don’t even always get copies of all historical file artifacts. More than +that, a Fossil repository typically contains +other useful information that is not always shared as part of a clone, which might need +to be backed up separately. To wit: + + +## Sensitive Information + +Fossil purposefully does not clone certain sensitive information unless +you’re logged in as a user with [Setup] capability. As an example, a local clone +may have a different `user` table than the remote, because only a +Setup user is allowed to see the full version for privacy and security +reasons. + + +## Configuration Drift + +Fossil allows the local configuration to differ in several areas from +that of the remote. You get a copy +of *some* of these configuration areas on initial clone — not all! — but after that, +remote configuration changes mostly do not sync down automatically. + + +#### Skin + +Changes to the remote’s skin don’t sync down, on purpose, since you may +want to have a different skin on the local clone than on the remote. You +can ask for updates with [`fossil config pull skin`][cfg], but that does +not happen automatically during the course of normal development. + + +#### Email Alerts + +The Admin → Notification settings do not get copied on clone or sync, +and it is not possible to push such settings from one repository to +another. We did this on purpose because you may have a network of peer +repositories, and you only want one repository sending email alerts. If +Fossil were to automatically replicate the email alert settings to a +separate repository, subscribers would get multiple alerts for each +event, which would be *bad.* + +The only element of the email alert configuration that can be pulled +over the sync protocol on demand is the subscriber list, via +[`fossil config pull subscriber`][cfg]. + + +#### Project Configuration + +This is normally generated once during `fossil init` and never changed, +so Fossil doesn’t pull this information without being forced, on +purpose. You could accidentally merge two separate Fossil repos by +pushing one repo’s project config up to another, for example. + + +#### Others + +A repo’s URL aliases, [interwiki configuration](./interwiki.md), and +[ticket customizations](./custom_tcket.wiki) also do not normally sync. + +[cfg]: /help/configuration + + + +## Private Branches + +The very nature of Fossil’s [private branch feature][pbr] ensures that +remote clones don’t get a copy of those branches. Normally this is +exactly what you want, but in the case of making backups, you probably +want to back up these branches as well. One of the two backup methods below +provides this. + + +## Shunned Artifacts + +Fossil purposefully doesn’t sync [shunned artifacts][shun]. If you want +your local clone to be a precise match to the remote, it needs to track +changes to the shun table as well. + + +## Unversioned Artifacts + +Data in Fossil’s [unversioned artifacts table][uv] doesn’t sync down by +default unless you specifically ask for it. Like local configuration +data, it doesn’t get pulled as part of a normal `fossil sync`, but +*unlike* the config data, you don’t get unversioned files as part of the +initial clone unless you ask for it by passing the `--unversioned/-u` +flag. + + +## Autosync Is Intransitive + +If you’re using Fossil in a truly distributed mode, rather than the +simple central-and-clones model that is more common, there may be no +single source of truth in the network because Fossil’s autosync feature +isn’t transitive. + +That is, if you cloned from server A, and then you stand that up on a +server B, then if I clone from your server as my repository C, your changes to B +autosync up to A, but not down to me on C until I do something locally +that triggers autosync. The inverse is also true: if I commit something +on C, it will autosync up to B, but A won’t get a copy until someone on +B does something to trigger a sync there. + +An easy way to run into this problem is to set up failover servers +`svr1` thru `svr3.example.com`, then set `svr2` and `svr3` up to sync +with the first. If all of the users normally clone from `svr1`, their +commits don’t get to `svr2` and `svr3` until something on one of the +servers pushes or pulls the changes down to the next server in the sync +chain. + +Likewise, if `svr1` falls over and all of the users re-point their local +clones at `svr2`, then `svr1` later reappears, `svr1` is likely to +remain a stale copy of the old version of the repository until someone +causes it to sync with `svr2` or `svr3` to catch up again. And then if +you originally designed the sync scheme to treat `svr1` as the primary +source of truth, those users still syncing with `svr2` won’t have their +commits pushed up to `svr1` unless you’ve set up bidirectional sync, +rather than have the two backup servers do `pull` only. + + +# Solution 1: Explicit Pulls + +The following script solves most of the above problems for the use case +where you want a *nearly-complete* clone of the remote repository using nothing +but the normal Fossil sync protocol. It only does so if you are logged into +the remote as a user with Setup capability, however. + +``` shell +#!/bin/sh +fossil sync --unversioned +fossil configuration pull all +fossil rebuild +``` + +The last step is needed to ensure that shunned artifacts on the remote +are removed from the local clone. The second step includes +`fossil conf pull shun`, but until those artifacts are actually rebuilt +out of existence, your backup will be “more than complete” in the sense +that it will continue to have information that the remote says should +not exist any more. That would be not so much a “backup” as an +“archive,” which might not be what you want. + + +# Solution 2: SQL-Level Backup + +The first method doesn’t get you a copy of the remote’s +[private branches][pbr], on purpose. It may also miss other info on the +remote, such as SQL-level customizations that the sync protocol can’t +see. (Some [ticket system customization][tkt] schemes rely on this ability, for example.) You can +solve such problems if you have access to the remote server, which +allows you to get a SQL-level backup by delegating handling of locking +and transaction isolation to +[the `backup` command][bu], allowing the user to safely back up an in-use +repository. + +If you have SSH access to the remote server, something like this will work: + +``` shell +#!/bin/bash +bf=repo-$(date +%Y-%m-%d).fossil +ssh example.com "cd museum ; fossil backup -R repo.fossil backups/$bf" && + scp example.com:museum/backups/$bf ~/museum/backups +``` + +Beware that this method does not solve [the intransitive sync +problem](#ait), in and of itself: if you do a SQL-level backup of a +stale repo DB, you have a *stale backup!* You should therefore run this +on every node that may need to serve as a backup so that at least *one* +of the backups is also up-to-date. + + +# Encrypted Off-Site Backups + +A useful refinement that you can apply to both methods above is +encrypted off-site backups. You may wish to store backups of your +repositories off-site on a service such as Dropbox, Google Drive, iCloud, +or Microsoft OneDrive, where you don’t fully trust the service not to +leak your information. This addition to the prior scripts will encrypt +the resulting backup in such a way that the cloud copy is a useless blob +of noise to anyone without the key: + +```shell +iter=152830 +pass="h8TixP6Mt6edJ3d6COaexiiFlvAM54auF2AjT7ZYYn" +gd="$HOME/Google Drive/Fossil Backups/$bf.xz.enc" +fossil sql -R ~/museum/backups/"$bf" .dump | xz -9 | + openssl enc -e -aes-256-cbc -pbkdf2 -iter $iter -pass pass:"$pass" -out "$gd" +``` + +If you’re adding this to the first script above, remove the +“`-R repo-name`” bit so you get a dump of the repository backing the +current working directory. + +Change the `pass` value to some other long random string, and change the +`iter` value to something in the hundreds of thousands range. A good source for +the first is [here][grcp], and for the second, [here][rint]. + +You may find posts online written by people recommending millions of +iterations for PBKDF2, but they’re generally talking about this in the +context of memorizable passwords, where adding even one more character +to the password is a significant burden. Given our script’s purely +random maximum-length passphrase, there isn’t much more that increasing +the key derivation iteration count can do for us. + +Conversely, if you were to reduce the passphrase to 41 characters, that +would drop the key strength by roughly 2⁶, being the entropy value per +character for using most of printable ASCII in our passphrase. To make +that lost strength up on the PBKDF2 end, you’d have to multiply your +iterations by 2⁶ = 64 times. It’s easier to use a max-length passphrase +in this situation than get crazy with key derivation iteration counts. + +(This, by the way, is why the example passphrase above is 42 characters: +with 6 bits of entropy per character, that gives you a key size of 252, +as close as we can get to our chosen encryption algorithm’s 256-bit key +size without going over. If it pleases you to give it 43 random +characters for a passphrase in order to pick up those last four bits of +security, you’re welcome to do so.) + +Compressing the data before encrypting it removes redundancies that can +make decryption easier, and it results in a smaller backup than you get +with the previous script alone, at the expense of a lot of CPU time +during the backup. You may wish to switch to a less space-efficient +compression algorithm that takes less CPU power, such as [`lz4`][lz4]. +Changing up the compression algorithm also provides some +security-thru-obscurity, which is useless on its own, but it *is* a +useful adjunct to strong encryption. + +This requires OpenSSL 1.1 or higher. If you’re on 1.0 or older, you +won’t have the `-pbkdf2` and `-iter` options, and you may have to choose +a different cipher algorithm; both changes are likely to weaken the +encryption significantly, so you should install a newer version rather +than work around the lack of these features. + +Beware that macOS ships a fork of OpenSSL called [LibreSSL][lssl] that +lacked this capability until Ventura (13.0). If you’re on Monterey (12) +or older, we recommend use of the [Homebrew][hb] OpenSSL package rather +than give up on the security afforded by use of configurable-iteration +PBKDF2. To avoid a conflict with the platform’s `openssl` binary, +Homebrew’s installation is [unlinked][hbul] by default, so you have to +give an explicit path to it, one of: + + /usr/local/opt/openssl/bin/openssl ... # Intel x86 Macs + /opt/homebrew/opt/openssl/bin/openssl ... # ARM Macs (“Apple silicon”) + +[lssl]: https://www.libressl.org/ + + +## Restoring From An Encrypted Backup + +The “restore” script for the above fragment is basically an inverse of +it, but it’s worth showing it because there are some subtleties to take +care of. If all variables defined in earlier scripts are available, then +restoration is: + +``` +openssl enc -d -aes-256-cbc -pbkdf2 -iter $iter -pass pass:"$pass" -in "$gd" | + xz -d | fossil sql --no-repository ~/museum/restored-repo.fossil +``` + +We changed the `-e` to `-d` on the `openssl` command to get decryption, +and we changed the `-out` to `-in` so it reads from the encrypted backup +file and writes the result to stdout. + +The decompression step is trivial. + +The last change is tricky: we used `fossil sql` above to ensure that +we’re using the same version of SQLite to write the encrypted backup DB +as was used to maintain the repository. We must also do that on +restoration: +Fossil serves as a dogfooding project for SQLite, +often making use of the latest features, so it is quite likely that a given +random `sqlite3` binary in your `PATH` will be unable to understand the +file created by “`fossil sql .dump`”! The tricky bit is, you can’t just +pipe the decrypted SQL dump into `fossil sql`, because on startup, Fossil +normally goes looking for tables created by `fossil init`, and it won’t +find them in a newly-created repo DB. We get around this by passing +the `--no-repository` flag, which suppresses this behavior. Doing it +this way saves you from needing to go and build a matching version of +`sqlite3` just to restore the backup. + +[bu]: /help/backup +[grcp]: https://www.grc.com/passwords.htm +[hb]: https://brew.sh +[hbul]: https://docs.brew.sh/FAQ#what-does-keg-only-mean +[lz4]: https://lz4.github.io/lz4/ +[pbr]: ./private.wiki +[rint]: https://www.random.org/integers/?num=1&min=100000&max=1000000&col=5&base=10&format=html&rnd=new +[Setup]: ./caps/admin-v-setup.md#apsu +[shun]: ./shunning.wiki +[tkt]: ./tickets.wiki +[uv]: ./unvers.wiki diff --git a/www/blame.wiki b/www/blame.wiki new file mode 100644 index 0000000..f3ffb3e --- /dev/null +++ b/www/blame.wiki @@ -0,0 +1,62 @@ +The Annotate Algorithm + +

1.0 Introduction

+ +The [/help/annotate|fossil annotate], +[/help/blame|fossil blame], and +[/help/praise|fossil praise] commands, and the +[/help/www/annotate|/annotate], +[/help/www/blame|/blame], and +[/help/www/praise|/praise] web pages are all used to show the most +recent check-in that modified each line of a particular file. +This article overviews the algorithm used to compute the annotation +for a file in Fossil. + +

2.0 Algorithm

+ +
    +
  1. Locate the check-in that contains the file that is to be + annotated. Call this check-in C0. +
  2. Find all direct ancestors of C0. A direct ancestor is the closure + of the primary parent of C0. Merged in branches are not part of + the direct ancestors of C0. +
  3. Prune the list of ancestors of C0 so that it contains only + check-ins in which the file to be annotated was modified. +
  4. Load the complete text of the file to be annotated from check-in C0. + Call this version of the file F0. +
  5. Parse F0 into lines. Mark each line as "unchanged". +
  6. For each ancestor of C0 on the pruned list (call the ancestor CX), + beginning with the most + recent ancestor and moving toward the oldest ancestor, do the + following steps: +
      +
    1. Load the text for the file to be annotated as it existed in check-in CX. + Call this text FX. +
    2. Compute a diff going from FX to F0. +
    3. For each line of F0 that is changed in the diff and which was previously + marked "unchanged", update the mark to indicated that line + was modified by CX. +
    +
  7. Show each line of F0 together with its change mark, appropriately + formatted. +
+ +

3.0 Discussion and Notes

+ +The time-consuming part of this algorithm is step 6b - computing the +diff from all historical versions of the file to the version of the file +under analysis. For a large file that has many historical changes, this +can take several seconds. For this reason, the default +[/help/www/annotate|/annotate] webpage only shows those lines that were +changed by the 20 most recent modifications to the file. This allows +the loop on step 6 to terminate after only 19 diffs instead of the hundreds +or thousands of diffs that might be required for a frequently modified file. + +As currently implemented (as of 2015-12-12) the annotate algorithm does not +follow files across name changes. File name change information is +available in the database, and so the algorithm could be enhanced to follow +files across name changes by modifications to step 3. + +Step 2 is interesting in that it is +[/artifact/6cb824a0417?ln=196-201 | implemented] using a +[https://www.sqlite.org/lang_with.html#recursivecte|recursive common table expression]. diff --git a/www/blockchain.md b/www/blockchain.md new file mode 100644 index 0000000..af8c613 --- /dev/null +++ b/www/blockchain.md @@ -0,0 +1,478 @@ +# Is Fossil A Blockchain? + +The Fossil version control system shares a lot of similarities with +other blockchain based technologies, but it also differs from the more common +sorts of blockchains. This document will discuss the term’s +applicability, so you can decide whether applying the term to Fossil +makes sense to you. + + +## The Dictionary Argument + +The [Wikipedia definition of "blockchain"][bcwp] begins: + +> + "A blockchain…is a growing list of records, called blocks, which are linked using + cryptography… Each block contains a cryptographic hash of the previous + block, a timestamp, and transaction data (generally represented as a Merkle tree)." + +Point-for-point, Fossil follows this partial definition. +The blocks +are Fossil’s ["manifest" artifacts](./fileformat.wiki#manifest). Each +manifest has a cryptographically-strong [SHA-1] or [SHA-3] hash linking it to +one or more “parent” blocks. The manifest also contains a timestamp and +the transactional data needed to express a commit to the repository. +To traverse the Fossil repository from the tips of its [DAG] to the +root by following the parent hashes in each manifest is to traverse +a Merkle tree. +Every change in Fossil starts by adding one or more manifests to +the repository, extending this Merkle tree. + +[bcwp]: https://en.wikipedia.org/wiki/Blockchain +[DAG]: https://en.wikipedia.org/wiki/Directed_acyclic_graph +[SHA-1]: https://en.wikipedia.org/wiki/SHA-1 +[SHA-3]: https://en.wikipedia.org/wiki/SHA-3 + + + + +## Cryptocurrency + +Because blockchain technology was first popularized as Bitcoin, many +people associate the term with cryptocurrency. Fossil has nothing to do +with cryptocurrency, so a claim that “Fossil is a blockchain” may fail +to communicate the speaker’s concepts clearly due to conflation with +cryptocurrency. + +Cryptocurrency has several features and requirements that Fossil doesn’t +provide, either because it doesn’t need them or because we haven’t +gotten around to creating the feature. Whether these are essential to +the definition of “blockchain” and thus disqualify Fossil as a blockchain +is for you to decide. + +Cryptocurrencies must prevent three separate types of fraud to be useful: + +* **Type 1** is modification of existing currency. To draw an analogy + to paper money, we wish to prevent someone from using green and + black markers to draw extra zeroes on a US $10 bill so that it + claims to be a $100 bill. + +* **Type 2** is creation of new fraudulent currency that will pass + in commerce. To extend our analogy, it is the creation of new + US $10 bills. There are two sub-types to this fraud. In terms of + our analogy, they are: + + * **Type 2a**: copying an existing legitimate $10 bill

+ + * **Type 2b**: printing a new $10 bill that is unlike an existing + legitimate one, yet which will still pass in commerce + +* **Type 3** is double-spending existing legitimate cryptocurrency. + There is no analogy in paper money due to its physical form; it is a + problem unique to digital currency due to its infinitely-copyable + nature. + +How does all of this compare to Fossil? + +1. **Signatures.** Cryptocurrencies use a chain + of [digital signatures][dsig] to prevent Type 1 and Type 3 frauds. This + chain forms an additional link between the blocks, separate from the + hash chain that applies an ordering and lookup scheme to the blocks. + [_Blockchain: Simple Explanation_][bse] explains this “hash chain” + vs. “block chain” distinction in more detail. + + These signatures prevent modification of the face value of each + transaction (Type 1 fraud) by ensuring that only the one signing a + new block has the private signing key that could change an issued + block after the fact. + + The fact that these signatures are also *chained* prevents Type + 3 frauds by making the *prior* owner of a block sign it over to + the new owner. To avoid an O(n²) auditing problem as a result, + cryptocurrencies add a separate chain of hashes to make checking + for double-spending quick and easy. + + Fossil has [a disabled-by-default feature][cs] to call out to an + external copy of [PGP] or [GPG] to sign commit manifests before + inserting them into the repository. You can couple that with + a server-side [after-receive hook][arh] to reject unsigned commits. + + Although there are several distinctions you can draw between the way + Fossil’s commit signing scheme works and the way block signing works + in cryptocurrencies, only one is of material interest for our + purposes here: Fossil commit signatures apply only to a single + commit. Fossil does not sign one commit over to the next “owner” of + that commit in the way that a blockchain-based cryptocurrency must + when transferring currency from one user to another, beacuse there + is no useful analog to the double-spending problem in Fossil. The + closest you can come to this is double-insert of commits into the + blockchain, which we’ll address shortly. + + What Fossil commit signatures actually do is provide in-tree forgery + prevention, both Type 1 and Type 2. You cannot modify existing + commits (Type 1 forgery) because you do not have the original + committer’s private signing key, and you cannot forge new commits + attesting to come from some other trusted committer (Type 2) because + you don’t have any of their private signing keys, either. + Cryptocurrencies use the work problem to prevent Type 2 + forgeries, but the application of that to Fossil is a matter we get + to [later](#work). + + Although you have complete control over the contents of your local + Fossil repository clone, you cannot perform Type 1 forgery on its + contents short of executing a [preimage attack][prei] on the hash + algorithm. ([SHA3-256][SHA-3] by default in the current version of + Fossil.) Even if you could, Fossil’s sync protocol will prevent the + modification from being pushed into another repository: the remote + Fossil instance says, “I’ve already got that one, thanks,” and + ignores the push. Thus, short of breaking into the remote server + and modifying the repository in place, you couldn’t make use of + a preimage attack even if you had that power. Further, that would be an attack on the + server itself, not on Fossil’s data structures, so while it is + useful to think through this problem, it is not helpful in answering + our questions here. + + The Fossil sync protocol’s duplication detection also prevents the closest analog to Type 3 + frauds in Fossil: copying a commit manifest in your local repo clone + won’t result in a double-commit on sync. + + In the absence of digital signatures, Fossil’s [RBAC system][caps] + restricts Type 2 forgery to trusted committers. Thus once again + we’re reduced to an infosec problem, not a data structure design + question. + + (Inversely, enabling commit clearsigning is a good idea + if you have committers on your repo whom you don’t trust not to + commit Type 2 frauds. But let us be clear: your choice of setting + does not answer the question of whether Fossil is a blockchain.) + + If Fossil signatures prevent Type 1 and Type 2 frauds, you + may wonder why they are not enabled by default. It is because + they are defense-in-depth measures, not the minimum sufficient + measures needed to prevent repository fraud, unlike the equivalent + protections in a cryptocurrency blockchain. Fossil provides its + primary protections through other means, so it doesn’t need to + mandate signatures. + + Also, Fossil is not itself a [PKI], and there is no way for regular + users of Fossil to link it to a PKI, since doing so would likely + result in an unwanted [PII] disclosure. There is no email address + in a Fossil commit manifest that you could use to query one of the + public PGP keyservers, for example. It therefore becomes a local + policy matter as to whether you even *want* to have signatures, + because they’re not without their downsides. + +2. **Work Contests.** Cryptocurrencies prevent Type 2b forgeries + by setting up some sort of contest that ensures that new coins can come + into existence only by doing some difficult work task. This “mining” + activity results in a coin that took considerable work to create, + which thus has economic value by being a) difficult to re-create, + and b) resistant to [debasement][dboc]. + + Fossil repositories are most often used to store the work product of + individuals, rather than cryptocoin mining machines. There is + generally no contest in trying to produce the most commits. There + may be an implicit contest to produce the “best” commits, but that + is a matter of project management, not something that can be + automatically mediated through objective measures. + + Incentives to commit to the repository come from outside of Fossil; + they are not inherent to its nature, as with cryptocurrencies. + Moreover, there is no useful sense in which we could say that one + commit “re-creates” another. Commits are generally products of + individual human intellect, thus necessarily unique in all but + trivial cases. This is foundational to copyright law. + +3. **Longest Chain Rule.** Cryptocurrencies generally + need some way to distinguish which blocks are legitimate and which + not. They do this in part by identifying the linear chain with the + greatest cumulative [work time](#work) as the legitimate chain. All + blocks not on that linear chain are considered “orphans” and are + ignored by the cryptocurrency software. + + Its inverse is sometimes called the “51% attack” because a single + actor would have to do slightly more work than the entire rest of + the community using a given cryptocurrency in order for their fork + of the currency to be considered the legitimate fork. This argument + soothes concerns that a single bad actor could take over the + network. + + The closest we can come to that notion in Fossil is the default + “trunk” branch, but there’s nothing in Fossil that delegitimizes + other branches just because they’re shorter, nor is there any way in + Fossil to score the amount of work that went into a commit. Indeed, + [forks and branches][fb] are *valuable and desirable* things in + Fossil. + +This much is certain: Fossil is definitely not a cryptocurrency. Whether +this makes it “not a blockchain” is a subjective matter. + +[arh]: ./hooks.md +[bse]: https://www.researchgate.net/publication/311572122_What_is_Blockchain_a_Gentle_Introduction +[caps]: ./caps/ +[cs]: /help/clearsign +[dboc]: https://en.wikipedia.org/wiki/Debasement +[dsig]: https://en.wikipedia.org/wiki/Digital_signature +[fb]: ./branching.wiki +[GPG]: https://gnupg.org/ +[PGP]: https://www.openpgp.org/ +[PII]: https://en.wikipedia.org/wiki/Personal_data +[PKI]: https://en.wikipedia.org/wiki/Public_key_infrastructure +[pow]: https://en.wikipedia.org/wiki/Proof_of_work +[prei]: https://en.wikipedia.org/wiki/Preimage_attack + + + + +## Distributed Ledgers + +Cryptocurrencies are an instance of [distributed ledger technology][dlt]. If +we can convince ourselves that Fossil is also a distributed +ledger, then we might think of Fossil as a peer technology, +having at least some qualifications toward being considered a blockchain. + +A key tenet of DLT is that records be unmodifiable after they’re +committed to the ledger, which matches quite well with Fossil’s design +and everyday use cases. Fossil puts up multiple barriers to prevent +modification of existing records and injection of incorrect records. + +Yet, Fossil also has [purge] and [shunning][shun]. Doesn’t that mean +Fossil cannot be a distributed ledger? + +These features only remove existing commits from the repository. If you want a +currency analogy, they are ways to burn a paper bill or to melt a [fiat +coin][fc] down to slag. In a cryptocurrency, you can erase your “wallet” +file, effectively destroying money in a similar way. These features +do not permit forgery of either type described above: you can’t use them +to change the value of existing commits (Type 1) or add new commits to +the repository (Type 2). + +What if we removed those features from Fossil, creating an append-only +Fossil variant? Is it a DLT then? Arguably still not, because [today’s Fossil +is an AP-mode system][ctap], which means +there can be no guaranteed consensus on the content of the ledger at any +given time. An AP-mode accounts receivable system would allow +different bottom-line totals at different sites, because you’ve +cast away “C” to get AP-mode operation. (See the prior link or +[Wikipedia’s article on the CAP theorem][cap] if you aren’t following +this terminology.) + +By the same token, you cannot guarantee that the command +“`fossil info tip`” gives the same result everywhere. You would need to +recast Fossil as a CA or CP-mode system to solve that. +(Everyone not +partitioned away from the majority of the network at any rate, in the CP +case.) + +What are the prospects for CA-mode or CP-mode Fossil? [We don’t want +CA-mode Fossil][ctca], but [CP-mode could be useful][ctcp]. Until the latter +exists, this author believes Fossil is not a distributed ledger in a +technologically defensible sense. + +The most common technologies answering to the label “blockchain” are all +DLTs, so if Fossil is not a DLT, then it is not a blockchain in that +sense. + +[ctap]: ./cap-theorem.md#ap +[ctca]: ./cap-theorem.md#ca +[ctcp]: ./cap-theorem.md#cp +[cap]: https://en.wikipedia.org/wiki/CAP_theorem +[dlt]: https://en.wikipedia.org/wiki/Distributed_ledger +[DVCS]: https://en.wikipedia.org/wiki/Distributed_version_control +[fc]: https://en.wikipedia.org/wiki/Fiat_money +[purge]: /help/purge +[shun]: ./shunning.wiki + + + +## Distributed Partial Consensus + +If we can’t get DLT, can we at least get some kind of distributed +consensus at the level of individual Fossil’s commits? + +Many blockchain based technologies have this property: given some +element of the blockchain, you can make certain proofs that it either is +a legitimate part of the whole blockchain, or it is not. + +Unfortunately, this author doesn’t see a way to do that with Fossil. +Given only one “block” in Fossil’s putative “blockchain” — a commit, in +Fossil terminology — all you can prove is whether it is internally +consistent, that it is not corrupt. That then points you at the parent(s) of that +commit, which you can repeat the exercise on, back to the root of the +DAG. This is what the enabled-by-default [`repo-cksum` setting][rcks] +does. + +If cryptocurrencies worked this way, you wouldn’t be able to prove that +a given cryptocoin was legitimate without repeating the proof-of-work +calculations for the entire cryptocurrency scheme! Instead, you only +need to check a certain number of signatures and proofs-of-work in order +to be reasonably certain that you are looking at a legitimate section of +the whole blockchain. + +What would it even mean to prove that a given Fossil commit “*belongs*” +to the repository you’ve extracted it from? For a software project, +isn’t that tantamount to automatic code review, where the server would +be able to reliably accept or reject a commit based solely on its +content? That sounds nice, but this author believes we’ll need to invent +[AGI] first. + +A better method to provide distributed consensus for Fossil would be to +rely on the *natural* intelligence of its users: that is, distributed +commit signing, so that a commit is accepted into the blockchain only +once some number of users countersign it. This amounts to a code review +feature, which Fossil doesn’t currently have. + +Solving that problem basically requires solving the [PKI] problem first, +since you can’t verify the proofs of these signatures if you can’t first +prove that the provided signatures belong to people you trust. This is a +notoriously hard problem in its own right. + +A future version of Fossil could instead provide [consensus in the CAP +sense][ctcp]. For instance, you could say that if a quorum of servers +all have a given commit, it “belongs.” Fossil’s strong hashing tech +would mean that querying whether a given commit is part of the +“blockchain” would be as simple as going down the list of servers and +sending each an HTTP GET `/info` query for the artifact ID, concluding +that the commit is legitimate once you get enough HTTP 200 status codes back. All of this is +hypothetical, because Fossil doesn’t do this today. + +[AGI]: https://en.wikipedia.org/wiki/Artificial_general_intelligence +[rcks]: /help/repo-cksum + + + + +## Anonymity + +Many blockchain based technologies go to extraordinary lengths to +allow anonymous use of their service. + +As typically configured, Fossil does not: commits synced between servers +always at least have a user name associated with them, which the remote +system must accept through its [RBAC system][caps]. That system can run +without having the user’s email address, but it’s needed if [email +alerts][alert] are enabled on the server. The remote server logs the IP +address of the commit for security reasons. That coupled with the +timestamp on the commit could sufficiently deanonymize users in many +common situations. + +It is possible to configure Fossil so it doesn’t do this: + +* You can give [Write capability][capi] to user category “nobody,” so + that anyone that can reach your server can push commits into its + repository. + +* You could give that capability to user category “anonymous” instead, + which requires that the user log in with a CAPTCHA, but which doesn’t + require that the user otherwise identify themselves. + +* You could enable [the `self-register` setting][sreg] and choose not to + enable [commit clear-signing][cs] so that anonymous users could push + commits into your repository under any name they want. + +On the server side, you can also [scrub] the logging that remembers +where each commit came from. + +Commit source info isn’t transmitted from the remote server on clone or pull: +the size of the `rcvfrom` table after initial clone is 1, containing +only the remote server’s IP address. On each pull containing new +artifacts, your local `fossil` instance adds another entry to this +table, likely with the same IP address unless the server has moved or +you’re using [multiple remotes][mrep]. This table is far more +interesting on the server side, containing the IP addresses of all +contentful pushes; thus [the `scrub` command][scrub]. + +Because Fossil doesn’t +remember IP addresses in commit manifests or require commit signing, it +allows at least *pseudonymous* commits. When someone clones a remote +repository, they don’t learn the email address, IP address, or any other +sort of [PII] of prior committers, on purpose. + +Some people say that private, permissioned blockchains (as you may +imagine Fossil to be) are inherently problematic by the very reason that +they don’t bake anonymous contribution into their core. The very +existence of an RBAC is a moving piece that can break. Isn’t it better, +the argument goes, to have a system that works even in the face of +anonymous contribution, so that you don’t need an RBAC? Cryptocurrencies +do this, for example: anyone can “mine” a new coin and push it into the +blockchain, and there is no central authority restricting the transfer +of cryptocurrency from one user to another. + +We can draw an analogy to encryption, where an algorithm is +considered inherently insecure if it depends on keeping any information +from an attacker other than the key. Encryption schemes that do +otherwise are derided as “security through obscurity.” + +You may be wondering what any of this has to do with whether Fossil is a +blockchain, but that is exactly the point: all of this is outside +Fossil’s core hash-chained repository data structure. If you take the +position that you don’t have a “blockchain” unless it allows anonymous +contribution, with any needed restrictions provided only by the very +structure of the managed data, then Fossil does not qualify. + +Why do some people care about this distinction? Consider Bitcoin, +wherein an anonymous user cannot spam the blockchain with bogus coins +because its [proof-of-work][pow] protocol allows such coins to be +rejected immediately. There is no equivalent in Fossil: it has no +technology that allows the receiving server to look at the content of a +commit and automatically judge it to be “good.” Fossil relies on its +RBAC system to provide such distinctions: if you have a commit bit, your +commits are *ipso facto* judged “good,” insofar as any human work +product can be so judged by a blob of compiled C code. This takes us +back to the [digital ledger question](#dlt), where we can talk about +what it means to later correct a bad commit that got through the RBAC +check. + +We may be willing to accept pseudonymity, rather than full anonymity. +If we configure Fossil as above, either bypassing the RBAC or abandoning +human control over it, scrubbing IP addresses, etc., is it then a public +permissionless blockchain in that sense? + +We think not, because there is no [longest chain rule](#lcr) or anything +like it in Fossil. + +For a fair model of how a Fossil repository might behave under such +conditions, consider GitHub: here one user can fork another’s repository +and make an arbitrary number of commits to their public fork. Imagine +this happens 10 times. How does someone come along later and +*automatically* evaluate which of the 11 forks of the code (counting the +original repository among their number) is the “best” one? For a +computer software project, the best we could do to approximate this +devolves to a [software project cost estimation problem][scost]. These +methods are rather questionable in their own right, being mathematical +judgement values on human work products, but even if we accept their +usefulness, then we still cannot say which fork is better based solely +on their scores under these metrics. We may well prefer to use the fork +of a software program that took *less* effort, being smaller, more +self-contained, and with a smaller attack surface. + + +[alert]: ./alerts.md +[capi]: ./caps/ref.html#i +[mrep]: /help/remote +[scost]: https://en.wikipedia.org/wiki/Software_development_effort_estimation +[scrub]: /help/scrub +[sreg]: /help/self-register + + +# Conclusion + +This author believes it is technologically indefensible to call Fossil a +“blockchain” in any sense likely to be understood by a majority of those +you’re communicating with. Using a term in a nonstandard way just because you can +defend it means you’ve failed any goal that requires clear communication. +The people you’re communicating your ideas to must have the +same concept of the terms you use. + +What term should you use instead? Fossil stores a DAG of hash-chained +commits, so an indisputably correct term is a [Merkle tree][mt], named +after [its inventor][drrm]. You could also use the more generic term +“hash tree.” + +Fossil is a technological peer to many common sorts of blockchain +technology. There is a lot of overlap in concepts and implementation +details, but when speaking of what most people understand as +“blockchain,” Fossil is not that. + +[drrm]: https://en.wikipedia.org/wiki/Ralph_Merkle +[mt]: https://en.wikipedia.org/wiki/Merkle_tree diff --git a/www/branching.wiki b/www/branching.wiki new file mode 100644 index 0000000..90eb265 --- /dev/null +++ b/www/branching.wiki @@ -0,0 +1,767 @@ +Branching, Forking, Merging, and Tagging +

Background

+ +In a simple and perfect world, the development of a project would proceed +linearly, as shown in Figure 1. + + +ALL: [circle rad 0.1in thickness 1.5px "1" +arrow right 40% +circle same "2" +arrow same +circle same "3" +arrow same +circle same "4"] +box invis "Figure 1" big fit with .n at .3cm below ALL.s + + +Each circle represents a check-in. For the sake of clarity, the check-ins +are given small consecutive numbers. In a real system, of course, the +check-in numbers would be long hexadecimal hashes since it is not possible +to allocate collision-free sequential numbers in a distributed system. +But as sequential numbers are easier to read, we will substitute them for +the long hashes in this document. + +The arrows in Figure 1 show the evolution of a project. The initial +check-in is 1. Check-in 2 is derived from 1. In other words, check-in 2 +was created by making edits to check-in 1 and then committing those edits. +We say that 2 is a child of 1 +and that 1 is a parent of 2. +Check-in 3 is derived from check-in 2, making +3 a child of 2. We say that 3 is a descendant of both 1 and 2 and that 1 +and 2 are both ancestors of 3. + +

DAGs

+ +The graph of check-ins is a +[http://en.wikipedia.org/wiki/Directed_acyclic_graph | directed acyclic graph], +commonly shortened to DAG. Check-in 1 is the root of the DAG +since it has no ancestors. Check-in 4 is a leaf of the DAG since +it has no descendants. (We will give a more precise definition later of +"leaf.") + +Alas, reality often interferes with the simple linear development of a +project. Suppose two programmers make independent modifications to check-in 2. +After both changes are committed, the check-in graph looks like Figure 2: + + +ALL: [circle rad 0.1in thickness 1.5px "1" +arrow right 40% +circle same "2" +circle same "3" at 2nd circle+(.4,.3) +arrow from 2nd circle to 3rd circle chop +circle same "4" at 2nd circle+(.4,-.3) +arrow from 2nd circle to 4th circle chop] +box invis "Figure 2" big fit with .n at .3cm below ALL.s + + +The graph in Figure 2 has two leaves: check-ins 3 and 4. Check-in 2 has +two children, check-ins 3 and 4. We call this state a fork. + +Fossil tries to prevent forks, primarily through its +"[./concepts.wiki#workflow | autosync]" mechanism. + +Suppose two programmers named Alice and +Bob are each editing check-in 2 separately. Alice finishes her edits +and commits her changes first, resulting in check-in 3. When Bob later +attempts to commit his changes, Fossil verifies that check-in 2 is still +a leaf. Fossil sees that check-in 3 has occurred and aborts Bob's commit +attempt with a message "would fork." This allows Bob to do a "fossil +update" to pull in Alice's changes, merging them into his own +changes. After merging, Bob commits check-in 4 as a child of check-in 3. +The result is a linear graph as shown in Figure 1. This is how CVS +works. This is also how Fossil works in autosync mode. + +But perhaps Bob is off-network when he does his commit, so he has no way +of knowing that Alice has already committed her changes. Or, it could +be that Bob has turned off "autosync" mode in Fossil. Or, maybe Bob +just doesn't want to merge in Alice's changes before he has saved his +own, so he forces the commit to occur using the "--allow-fork" option to +the [/help/commit | fossil commit] command. For any of these +reasons, two commits against check-in 2 have occurred, so the DAG now +has two leaves. + +In such a condition, a person working with this repository has a +dilemma: which version of the project is the "latest" in the sense of +having the most features and the most bug fixes? When there is more +than one leaf in the graph, you don't really know, which is why we +would ideally prefer to have linear check-in graphs. + +Fossil resolves such problems using the check-in time on the leaves to +decide which leaf to use as the parent of new leaves. When a branch is +forked as in Figure 2, Fossil will choose check-in 4 as the parent for a +later check-in 5, but only if it has sync'd that check-in down +into the local repository. If autosync is disabled or the user is +off-network when that fifth check-in occurs so that check-in 3 is the +latest on that branch at the time within that clone of the repository, +Fossil will make check-in 3 the parent of check-in 5! We show practical +consequences of this [#bad-fork | later in this article]. + +Fossil also uses a forked branch's leaf check-in timestamps when +checking out that branch: it gives you the fork with the latest +check-in, which in turn selects which parent your next check-in will be +a child of. This situation means development on that branch can fork +into two independent lines of development, based solely on which branch +tip is newer at the time the next user starts his work on it. + +Because of these potential problems, we strongly recommend that you do +not intentionally create forks on long-lived shared working branches +with "--allow-fork". (Prime example: trunk.) The inverse case — +intentional forks on short-lived single-developer branches — is far +easier to justify, since presumably the lone developer is never confused +about why there are two or more leaves on that branch. Further +justifications for intentional forking are [#forking | given below]. + +Let us return to Figure 2. To resolve such situations before they can +become a real problem, Alice can use the [/help/merge | fossil +merge] command to merge Bob's changes into her local copy of +check-in 3. Without arguments, that command merges all leaves on the +current branch. Alice can then verify that the merge is sensible and if +so, commit the results as check-in 5. This results in a DAG as shown in +Figure 3. + + +ALL: [circle rad 0.1in thickness 1.5px "1" +arrow right 40% +circle same "2" +circle same "3" at 2nd circle+(.4,.3) +arrow from 2nd circle to 3rd circle chop +circle same "4" at 2nd circle+(.4,-.3) +arrow from 2nd circle to 4th circle chop +circle same "5" at 3rd circle+(.4,-.3) +arrow from 3rd circle to 5th circle chop +arrow dashed .03 from 4th circle to 5th circle chop] +box invis "Figure 3" big fit with .n at .2cm below ALL.s + + +Check-in 5 is a child of check-in 3 because it was created by editing +check-in 3, but since check-in 5 also inherits the changes from check-in 4 by +virtue of the merge, we say that check-in 5 is a merge child +of check-in 4 and that it is a direct child of check-in 3. +The graph is now back to a single leaf, check-in 5. + +We have already seen that if Fossil is in autosync mode then Bob would +have been warned about the potential fork the first time he tried to +commit check-in 4. If Bob had updated his local check-out to merge in +Alice's check-in 3 changes, then committed, the fork would have +never occurred. The resulting graph would have been linear, as shown +in Figure 1. + +Realize that the graph of Figure 1 is a subset of Figure 3. If you hold your +hand over the ④ in Figure 3, it looks +exactly like Figure 1 except that the leaf has a different check-in +number. That is just a notational difference: the two check-ins +have exactly the same content. + +Inversely, Figure 3 is a +superset of Figure 1. The check-in 4 of Figure 3 captures additional +state which is omitted from Figure 1. Check-in 4 of Figure 3 holds a +copy of Bob's local checkout before he merged in Alice's changes. That +snapshot of Bob's changes, which is independent of Alice's changes, is +omitted from Figure 1. + +Some people say that the development approach taken in +Figure 3 is better because it preserves this extra intermediate state. +Others say that the approach taken in Figure 1 is better because it is +much easier to visualize linear development and because the +merging happens automatically instead of as a separate manual step. We +will not take sides in that debate. We will simply point out that +Fossil enables you to do it either way. + +

The Alternative to Forking: Branching

+ +Having more than one leaf in the check-in DAG is called a "fork." This +is usually undesirable and either avoided entirely, +as in Figure 1, or else quickly resolved as shown in Figure 3. +But sometimes, one does want to have multiple leaves. For example, a project +might have one leaf that is the latest version of the project under +development and another leaf that is the latest version that has been +tested. +When multiple leaves are desirable, we call this branching +instead of forking: + +Figure 4 shows an example of a project where there are two branches, one +for development work and another for testing. + + +ALL: [circle rad 0.1in thickness 1.5px fill white "1" +arrow 40% +C2: circle same "2" +arrow same +circle same "3" +arrow same +C5: circle same "5" +arrow same +C7: circle same "7" +arrow same +C8: circle same "8" +arrow same +C10: circle same "10" +C4: circle same at 3rd circle-(0,.35) "4" +C6: circle same at (1/2 way between C5 and C7,C4) "6" +C9: circle same at (1/2 way between C8 and C10,C4) "9" +arrow from C2 to C4 chop +arrow from C4 to C6 chop +arrow from C6 to C9 chop +arrow dashed 0.03 from C6 to C7 chop +arrow same from C9 to C10 +layer = 0 +box fill 0x9bcdfc color 0x9bcdfc wid (C10.e.x - C2.w.x) ht C6.height*1.5 at C6.c +box invis "test" fit with .sw at last box.sw] +box invis "Figure 4" big with .n at 0 below ALL.s + + +Figure 4 diagrams the following scenario: the project starts and +progresses to a point where (at check-in 2) +it is ready to enter testing for its first release. +In a real project, of course, there might be hundreds or thousands of +check-ins before a project reaches this point, but for simplicity of +presentation we will say that the project is ready after check-in 2. +The project then splits into two branches that are used by separate +teams. The testing team, using the blue branch, finds and fixes a few +bugs with check-ins 6 and 9. Meanwhile, the development +team, working on the top uncolored branch, +is busy adding features for the second +release. Of course, the development team would like to take advantage of +the bug fixes implemented by the testing team, so periodically the +changes in the test branch are merged into the dev branch. This is +shown by the dashed merge arrows between check-ins 6 and 7 and between +check-ins 9 and 10. + +In both Figures 2 and 4, check-in 2 has two children. In Figure 2, +we call this a "fork." In diagram 4, we call it a "branch." What is +the difference? As far as the internal Fossil data structures are +concerned, there is no difference. The distinction is in the intent. +In Figure 2, the fact that check-in 2 has multiple children is an +accident that stems from concurrent development. In Figure 4, giving +check-in 2 multiple children is a deliberate act. To a good +approximation, we define forking to be by accident and branching to +be by intent. Apart from that, they are the same. + +When the fork is intentional, it helps humans to understand what is +going on if we name the forks. This is not essential to Fossil's +internal data model, but humans have trouble working with long-lived +branches identified only by the commit ID currently at its tip, being a +long string of hex digits. Therefore, Fossil conflates two concepts: +branching as intentional forking and the naming of forks as branches. +They are in fact separate concepts, but since Fossil is intended to be +used primarily by humans, we combine them in Fossil's human user +interfaces. + +

+Key Distinction: A branch is a named, intentional fork. +

+ +Unnamed forks may be intentional, but most of the time, they're +accidental and left unnamed. + +Fossil offers two primary ways to create named, intentional forks, +a.k.a. branches. First: + +
+$ fossil commit --branch my-new-branch-name
+
+ +This is the method we recommend for most cases: it creates a branch as +part of a check-in using the version in the current checkout directory +as its basis. (This is normally the tip of the current branch, though +it doesn't have to be. You can create a branch from an ancestor check-in +on a branch as well.) After making this branch-creating +check-in, your local working directory is switched to that branch, so +that further check-ins occur on that branch as well, as children of the +tip check-in on that branch. + +The second, more complicated option is: + +
+$ fossil branch new my-new-branch-name trunk
+$ fossil update my-new-branch-name
+$ fossil commit
+
+ +Not only is this three commands instead of one, the first of which is +longer than the entire simpler command above, you must give the second command +before creating any check-ins, because until you do, your local working +directory remains on the same branch it was on at the time you issued +the command, so that the commit would otherwise put the new material on +the original branch instead of the new one. + +In addition to those problems, the second method is a violation of the +[https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it|YAGNI +Principle]. We recommend that you wait until you actually need the +branch before you create it using the first command above. + +The "trunk" is just another named branch in Fossil. It is simply +the default branch name for the first check-in and every check-in made as +one of its direct descendants. It is special only in that it is Fossil's +default when it has no better idea of which branch you mean. + + +

Justifications For Forking

+ +The primary cases where forking is justified over branching are all when +it is done purely in software in order to avoid losing information: + +
    +

  1. By Fossil itself when two users check in children to the same + leaf of a branch, as in Figure 2. +

    + If the fork occurs because + autosync is disabled on one or both of the repositories or because + the user doing the check-in has no network connection at the moment + of the commit, Fossil has no way of knowing that it is creating a + fork until the two repositories are later synchronized.

    2. + +

  2. By Fossil when the cloning hierarchy is more + than 2 levels deep. +

    + [./sync.wiki|Fossil's synchronization protocol] is a two-party + negotiation; syncs don't automatically propagate up the clone tree + beyond that. Because of that, if you have a master repository and + Alice clones it, then Bobby clones from Alice's repository, a + check-in by Bobby that autosyncs with Alice's repo will not + also autosync with the master repo. The master doesn't get a copy of + Bobby's check-in until Alice separately syncs with the master. + If Carol cloned from the master repo and checks something in that + creates a fork relative to Bobby's check-in, the master repo won't + know about that fork until Alice syncs her repo with the master. + Even then, realize that Carol still won't know about the fork until + she subsequently syncs with the master repo. +

    + One way to deal with this is to just accept it as a fact of using a + [https://en.wikipedia.org/wiki/Distributed_version_control|Distributed + Version Control System] like Fossil. +

    + Another option, which we recommend you consider carefully, is to + make it a local policy that check-ins be made only directly against the master + repo or one of its immediate child clones so that the autosync + algorithm can do its job most effectively. Any clones deeper than + that should be treated as read-only and thus get a copy of the new + state of the world only once these central repos have negotiated + that new state. This policy avoids a class of inadvertent fork you + might not need to tolerate. Since [#bad-fork|forks on long-lived + shared working branches can end up dividing a team's development + effort], a team may easily justify this restriction on distributed + cloning.

    3. + +

  3. You've automated Fossil, so you use + fossil commit --allow-fork commands to prevent Fossil from + refusing the check-in simply because it would create a fork. +

    + If you are writing such a tool — e.g. a shell script to make + multiple manipulations on a Fossil repo — it's better to make it + smart enough to detect this condition and cope with it, such as + by making a call to [/help/update | fossil update] + and checking for a merge conflict. That said, if the alternative is + losing information, you may feel justified in creating forks that an + interactive user must later manually clean up with fossil merge + commands.

    4. +
+ +That leaves only one case where we can recommend use of "--allow-fork" +by interactive users: when you're working on a personal branch so that +creating a dual-tipped branch isn't going to cause any other user an +inconvenience or risk [#bad-fork | inadvertently forking the development +effort]. In such a case, the lone developer working on that branch is +not confused, since the fork in development is intentional. Sometimes it +simply makes no sense to bother creating a name, cluttering the global +branch namespace, simply to convert an intentional fork into a "branch." +This is especially the case when the fork is short-lived. + +There's a common generalization of that case: you're a solo developer, +so that the problems with branching vs forking simply don't matter. In +that case, feel free to use "--allow-fork" as much as you like. + + +

Fixing Forks

+ +If your local checkout is on a forked branch, you can usually fix a fork +automatically with: + +
+$ fossil merge
+
+ +Normally you need to pass arguments to fossil merge to tell it +what you want to merge into the current basis view of the repository, +but without arguments, the command seeks out and fixes forks. + + +

Tags And Properties

+ +Tags and properties are used in Fossil to help express the intent, and +thus to distinguish between forks and branches. Figure 5 shows the +same scenario as Figure 4 but with tags and properties added: + + +ALL: [arrowht = 0.07 +C1: circle rad 0.1in thickness 1.5px fill white "1" +arrow 40% +C2: circle same "2" +arrow same +circle same "3" +arrow same +C5: circle same "5" +arrow same +C7: circle same "7" +arrow same +C8: circle same "8" +arrow same +C10: circle same "10" +C4: circle same at 3rd circle-(0,.35) "4" +C6: circle same at (1/2 way between C5 and C7,C4) "6" +C9: circle same at (1/2 way between C8 and C10,C4) "9" +arrow from C2 to C4 chop +arrow from C4 to C6 chop +arrow from C6 to C9 chop +arrow dashed 0.03 from C6 to C7 chop +arrow same from C9 to C10 +layer = 0 +box fill 0x9bcdfc color 0x9bcdfc wid (C10.e.x - C2.w.x) ht C6.height*1.5 at C6.c +text " test" above ljust at last box.sw +box fill lightgray "branch=trunk" "sym-trunk" fit with .ne at C1-(0.05,0.3); +line color gray from last box.ne to C1 chop +box same "branch=test" "sym-test" "bgcolor=blue" "cancel=sym-trunk" fit \ + with .n at C4-(0,0.3) +line color gray from last box.n to C4 chop +box same "sym-release-1.0" "closed" fit with .n at C9-(0,0.3) +line color gray from last box.n to C9 chop] +box invis "Figure 5" big fit with .n at 0.2cm below ALL.s + + +A tag is a name that is attached to a check-in. A +property is a name/value pair. Internally, Fossil implements +tags as properties with a NULL value. So, tags and properties really +are much the same thing, and henceforth we will use the word "tag" +to mean either a tag or a property. + +A tag can be a one-time tag, a propagating tag or a cancellation tag. +A one-time tag only applies to the check-in to which it is attached. A +propagating tag applies to the check-in to which it is attached and also +to all direct descendants of that check-in. A direct descendant +is a descendant through direct children. Tag propagation does not +cross merges. Tag propagation also stops as soon +as it encounters another check-in with the same tag. A cancellation tag +is attached to a single check-in in order to either override a one-time +tag that was previously placed on that same check-in, or to block +tag propagation from an ancestor. + +The initial check-in of every repository has two propagating tags. In +Figure 5, that initial check-in is check-in 1. The branch tag +tells (by its value) what branch the check-in is a member of. +The default branch is called "trunk." All tags that begin with "sym-" +are symbolic name tags. When a symbolic name tag is attached to a +check-in, that allows you to refer to that check-in by its symbolic +name rather than by its hexadecimal hash name. When a symbolic name +tag propagates (as does the sym-trunk tag) then referring to that +name is the same as referring to the most recent check-in with that name. +Thus the two tags on check-in 1 cause all descendants to be in the +"trunk" branch and to have the symbolic name "trunk." + +Check-in 4 has a branch tag which changes the name of the branch +to "test." The branch tag on check-in 4 propagates to check-ins 6 and 9. +But because tag propagation does not follow merge links, the branch=test +tag does not propagate to check-ins 7, 8, or 10. Note also that the +branch tag on check-in 4 blocks the propagation of branch=trunk +so that it cannot reach check-ins 6 or 9. This causes check-ins 4, 6, and +9 to be in the "test" branch and all others to be in the "trunk" branch. + +Check-in 4 also has a sym-test tag, which gives the symbolic name +"test" to check-ins 4, 6, and 9. Because tags do not propagate across +merges, check-ins 7, 8, and 10 do not inherit the sym-test tag and +are hence not known by the name "test." +To prevent the sym-trunk tag from propagating from check-in 1 +into check-ins 4, 6, and 9, there is a cancellation tag for +sym-trunk on check-in 4. The net effect is that +check-ins on the trunk go by the symbolic name of "trunk" and check-ins +on the test branch go by the symbolic name "test." + +The bgcolor=blue tag on check-in 4 causes the background color +of timelines to be blue for check-in 4 and its direct descendants. + +Figure 5 also shows two one-time tags on check-in 9. (The diagram does +not make a graphical distinction between one-time and propagating tags.) +The sym-release-1.0 tag means that check-in 9 can be referred to +using the more meaningful name "release-1.0." The closed tag means +that check-in 9 is a "closed leaf." A closed leaf is a leaf that should +never have direct children. + +

How Can Forks Divide Development Effort?

+ +[#dist-clone|Above], we stated that forks carry a risk that development +effort on a branch can be divided among the forks. It might not be +immediately obvious why this is so. To see it, consider this swim lane +diagram: + + + $laneh = 0.75 + +ALL: [ + # Draw the lanes + down + box width 3.5in height $laneh fill 0xacc9e3 + box same fill 0xc5d8ef + box same as first box + box same as 2nd box + line from 1st box.sw+(0.2,0) up until even with 1st box.n \ + "Alan" above aligned + line from 2nd box.sw+(0.2,0) up until even with 2nd box.n \ + "Betty" above aligned + line from 3rd box.sw+(0.2,0) up until even with 3rd box.n \ + "Charlie" above aligned + line from 4th box.sw+(0.2,0) up until even with 4th box.n \ + "Darlene" above aligned + + # fill in content for the Alice lane + right +A1: circle rad 0.1in at end of first line + (0.2,-0.2) \ + fill white thickness 1.5px "1" + arrow right 50% + circle same "2" + arrow right until even with first box.e - (0.65,0.0) + ellipse "future" fit fill white height 0.2 width 0.5 thickness 1.5px +A3: circle same at A1+(0.8,-0.3) "3" fill 0xc0c0c0 + arrow from A1 to last circle chop "fork!" below aligned + + # content for the Betty lane +B1: circle same as A1 at A1-(0,$laneh) "1" + arrow right 50% + circle same "2" + arrow right until even with first ellipse.w + ellipse same "future" +B3: circle same at A3-(0,$laneh) "3" + arrow right 50% + circle same as A3 "4" + arrow from B1 to 2nd last circle chop + + # content for the Charlie lane +C1: circle same as A1 at B1-(0,$laneh) "1" + arrow 50% + circle same "2" + arrow right 0.8in "goes" "offline" +C5: circle same as A3 "5" + arrow right until even with first ellipse.w \ + "back online" above "pushes 5" below "pulls 3 & 4" below + ellipse same "future" + + # content for the Darlene lane +D1: circle same as A1 at C1-(0,$laneh) "1" + arrow 50% + circle same "2" + arrow right until even with C5.w + circle same "5" + arrow 50% + circle same as A3 "6" + arrow right until even with first ellipse.w + ellipse same "future" +D3: circle same as B3 at B3-(0,2*$laneh) "3" + arrow 50% + circle same "4" + arrow from D1 to D3 chop +] +box invis "Figure 6" big fit with .n at 0.2cm below ALL.s + + +This is a happy, cooperating team. That is an important restriction on +our example, because you must understand that this sort of problem can +arise without any malice, selfishness, or willful ignorance in sight. +All users on this diagram start out with the same view of the +repository, cloned from the same master repo, and all of them are +working toward their shared vision of a unified future. + +All users, except possibly Alan, start out with the same two initial +check-ins in their local working clones, 1 & 2. It might be that Alan +starts out with only check-in 1 in his local clone, but we'll deal with +that detail later. + +It doesn't matter which branch this happy team is working on, only that +our example makes the most sense if you think of it as a long-lived shared +working branch like trunk. Each user makes +only one check-in, shaded light gray in the diagram. + +

Step 1: Alan

+ +Alan sets the stage for this problem by creating a +fork from check-in 1 as check-in 3. How and why Alan did this doesn't +affect what happens next, though we will walk through the possible cases +and attempt to assign blame [#post-mortem|in the post mortem]. +For now, you can assume that Alan did this out of unavoidable ignorance. + +

Step 2: Betty

+ +Because Betty's local clone is autosyncing with +the same upstream repository as Alan's clone, there are a number of ways +she can end up seeing Alan's check-in 3 as the latest on that branch: + +
    +

  1. The working check-out directory she's using at the moment was + on a different branch at the time Alan made check-in 3, so Fossil + sees that as the tip at the time she switches her working directory + to that branch with a fossil update $BRANCH command. (There is an + implicit autosync in that command, if the option was enabled at the + time of the update.)

    2. + +

  2. The same thing, only in a fresh checkout directory with a + [/help/open | fossil open $REPO $BRANCH] command.

    3. + +

  3. Alan makes his check-in 3 while Betty has check-in 1 or 2 as + the tip in her local clone, but because she's working with an + autosync'd connection to the same upstream repository as Alan, on + attempting what will become check-in 4, she gets the "would fork" + message from fossil commit, so she dutifully updates her clone + and tries again, moving her work to be a child of the new tip, + check-in 3. (If she doesn't update, she creates a second + fork, which simply complicates matters beyond what we need here for + our illustration.)

    4. +
+ +For our purposes here, it doesn't really matter which one happened. All +that matters is that Alan's check-in 3 becomes the parent of Betty's +check-in 4 because it was the newest tip of the working branch at the +time Betty does her check-in. + +

Step 3: Charlie

+ +Meanwhile, Charlie went offline after syncing +his repo with check-in 2 as the latest on that branch. When he checks +his changes in, it is as a child of 2, not of 4, because Charlie doesn't +know about check-ins 3 & 4 yet. He does this at an absolute wall clock +time after Alan and Betty made their check-ins, so when Charlie +comes back online and pushes his check-in 5 to the master repository and +learns about check-ins 3 and 4 during Fossil sync, Charlie inadvertently +revives the other side of the fork. + +

Step 4: Darlene

+ +Darlene sees all of this, because she joins in +on the work on this branch after Alan, Betty, and Charlie made their +check-ins and pushed them to the master repository. She's taking one of +the same three steps as we [#bf-betty|outlined for Betty above]. +Regardless of her path to this view, it happens after Charlie pushed his +check-in 5 to the master repo, so Darlene sees that as the latest on the +branch, causing her work to be saved as a child of check-in 5, not of +check-in 4, as it would if Charlie didn't come back online and sync +before Darlene started work on that branch. + +

Post Mortem

+ +The end result of all of this is that even though everyone makes only one check-in +and no one disables autosync without genuine need, +half of the check-ins end up on one side of the fork and half on +the other. + +A future user — his mother calls him Edward, but please call him Eddie — +can then join in on the work on this branch and end up on either side of +the fork. If Eddie joins in with the state of the repository as drawn +above, he'll end up on the top side of the fork, because check-in 6 is +the latest, but if Alan or Betty makes a seventh check-in to that branch +first, it will be as a child of check-in 4 since that's the version in +their local check-out directories. Since that check-in 7 will then be the latest, +Eddie will end up on the bottom side of the fork instead. + +In all of this, realize that neither side of the fork is obviously +"correct." Every participant was doing the right thing by their own +lights at the time they made their lone check-in. + +Who, then, is to blame? + +We can only blame the consequences of creating the fork on Alan if he +did so on purpose, as by passing "--allow-fork" when creating a check-in +on a shared working branch. Alan might have created it inadvertently by +going offline while check-in 1 was the tip of the branch in his local +clone, so that by the time he made his check-in 3, check-in 2 had +arrived at the shared parent repository from someone else. (Francine?) +When Alan rejoins the network and does an autosync, he learns about +check-in 2. Since his #3 is already checked into his local clone because +autosync was off or blocked, the sync creates an unavoidable fork. We +can't blame either Alan or Francine here: they were both doing the right +thing given their imperfect view of the state of the global situation. + +The same is true of Betty, Charlie, and Darlene. None of them tried to +create a fork, and none of them chose a side in this fork to participate +in. They just took Fossil's default and assumed it was correct. + +The only blame I can assign here is on any of these users who believed +forks couldn't happen before this did occur, and I blame them only for +their avoidable ignorance. (You, dear reader, have been ejected from +that category by reading this very document.) Any time someone can work +without getting full coordination from every other clone of the repo, +forks are possible. Given enough time, they're all but inevitable. This +is a general property of DVCSes, not just of Fossil. + +This sort of consequence is why forks on shared working branches are +bad, which is why [./concepts.wiki#workflow|Fossil tries so hard to avoid them], why it warns you +about it when they do occur, and why it makes it relatively [#fix|quick and +painless to fix them] when they do occur. + + +

Review Of Terminology

+ +
+
Branch
+

A branch is a set of check-ins with the same value for their +"branch" property.

+
Leaf
+

A leaf is a check-in with no children in the same branch.

+
Closed Leaf
+

A closed leaf is any leaf with the closed tag. These leaves +are intended to never be extended with descendants and hence are omitted +from lists of leaves in the command-line and web interface.

+
Open Leaf
+

A open leaf is a leaf that is not closed.

+
Fork
+

A fork is when a check-in has two or more direct (non-merge) +children in the same branch.

+
Branch Point
+

A branch point occurs when a check-in has two or more direct (non-merge) +children in different branches. A branch point is similar to a fork, +except that the children are in different branches.

+
+ +Check-in 4 of Figure 3 is not a leaf because it has a child (check-in 5) +in the same branch. Check-in 9 of Figure 5 also has a child (check-in 10) +but that child is in a different branch, so check-in 9 is a leaf. Because +of the closed tag on check-in 9, it is a closed leaf. + +Check-in 2 of Figure 3 is considered a "fork" +because it has two children in the same branch. Check-in 2 of Figure 5 +also has two children, but each child is in a different branch, hence in +Figure 5, check-in 2 is considered a "branch point." + +

Differences With Other DVCSes

+ +

Single DAG

+ +Fossil keeps all check-ins on a single DAG. Branches are identified with +tags. This means that check-ins can be freely moved between branches +simply by altering their tags. + +Most other DVCSes maintain a separate DAG for each branch. + +

Branch Names Need Not Be Unique

+ +Fossil does not require that branch names be unique, as in some VCSes, +most notably Git. Just as with unnamed branches (which we call forks) +Fossil resolves such ambiguities using the timestamps on the latest +check-in in each branch. If you have two branches named "foo" and you say +fossil update foo, you get the tip of the "foo" branch with the most +recent check-in. + +This fact is helpful because it means you can reuse branch names, which +is especially useful with utility branches. There are several of these +in the SQLite and Fossil repositories: "broken-build," "declined," +"mistake," etc. As you might guess from these names, such branch names +are used in renaming the tip of one branch to shunt it off away from the +mainline of that branch due to some human error. (See +[/help/amend | fossil +amend] and the Fossil UI check-in amendment features.) This is a +workaround for Fossil's [./shunning.wiki|normal inability to forget +history]: we usually don't want to actually remove history, but +would like to sometimes set some of it aside under a new label. + +Because some VCSes can't cope with duplicate branch names, Fossil +collapses such names down on export using the same time stamp based +arbitration logic, so that only the branch with the newest check-in gets +the branch name in the export. + +All of the above is true of tags in general, not just branches. diff --git a/www/bugtheory.wiki b/www/bugtheory.wiki new file mode 100644 index 0000000..56c2edd --- /dev/null +++ b/www/bugtheory.wiki @@ -0,0 +1,124 @@ +Bug-Tracking In Fossil +

Introduction

+ +A bug-report in fossil is called a "ticket". Tickets are tracked +separately from code check-ins. + +Some other distributed bug-tracking systems store tickets as files within +the source tree and thereby leverage the syncing and merging capabilities +of the versioning system to sync and merge tickets. This approach is +rejected in fossil for three reasons: + + 1. Check-ins in fossil are immutable. So if + tickets were part of the check-in, then there would be no way to add + new tickets to a check-in as new bugs are discovered. + + 2. Any project of reasonable size and complexity will generate thousands + and thousands of tickets, and we do not want all those ticket files + cluttering the source tree. + + 3. We want tickets to be managed from the web interface and to have a + permission system that is distinct from check-in permissions. + In other words, we do not want to restrict the creation and editing + of tickets to developers with check-in privileges and an installed + copy of the fossil executable. Casual passers-by on the internet should + be permitted to create tickets. + +Recall that a fossil repository consists of an +unordered collection of artifacts. (See the +file format document for details.) +Some artifacts have a special format, and among those are +Ticket Change Artifacts. +One or more ticket change artifacts are associated with each +ticket. A ticket is created by a ticket change artifact. +Each subsequent modification of the ticket is a separate artifact. + +The "push", "pull", and "sync" algorithms share ticket change artifacts +between repositories in the same way as every other artifact. In fact, +the sync algorithm has no knowledge of the meaning of the artifacts it +is syncing. As far as the sync algorithm is concerned, all artifacts are +alike. After the sync has occurs, the individual repositories must +make sense of the meaning of the various artifacts for themselves. + +

Interpretation Of Ticket Change Artifacts

+ +Note: The following is implementation detail which can be safely ignored by +casual users of fossil. + +Every ticket change artifact contains (among other things) + + * a timestamp, + * a ticket ID, and + * one or more name/value pairs. + +The current state of a ticket is found by replaying all ticket +change artifacts with the same ticket ID in timestamp order. +For a given ticket, all values are initially NULL. As each +ticket change artifact is encountered, values are either replaced +or appended, according to a flag on the name/value pair. The current +values for the fields of a ticket are the values that remain at the +end of the replay process. + +To create a new ticket, one inserts a ticket change +artifact with a new ID. The ticket ID +is a random 40-character lower-case hexadecimal number. The "tktnew" +page in the fossil web interface creates new ticket IDs +using a good source of randomness to insure uniqueness. +The name/value pairs on the initial ticket change +artifact are the initial values for the fields in the ticket. + +Amending a ticket means simply creating a new artifact with the +same ticket ID and with name/value pairs for those fields which +are changing. Fields of the ticket which are not being modified +should not appear as name/value pairs in the new artifact. + +This approach to storing ticket state means that independently entered +changes are automatically merged together when artifacts are shared +between repositories. Tickets do not branch. This approach also makes +it trivial to track the historic progression of changes to a ticket. + +In order for this scheme to work, the system clocks on machines that +add new ticket changes artifacts have to be set close to reality. +It is OK for a ticket change artifact timestamp to be off by a few +minutes or even a few hours. But if a timestamp on a ticket change +artifact is off by months or years, it can seriously confuse the +replay algorithm for determining the current ticket state. There +are techniques available to cause rogue artifacts to be ignored +by fossil. So if a ticket change artifact with a bad timestamp +does get into a repository, it can be removed by an administrator. +But the best approach is to take steps to insure that timestamps +are approximately correct in the first place. + +

Local Configuration

+ +The ticket change artifacts are part of the global state for a project. +The global state is that which is shared between repositories during +a sync. Each repository also has local state which is not (normally) +shared. The name/value pairs for a ticket are part of global state, +but the interpretation and display of that information is local state. +Hence, each repository is free to set up its own ticket display and +input formats and reporting rules according to its own particular needs. + +Each repository defines its own TICKET table in its database. There is +one row in the TICKET table for each unique ticket ID. The +names of columns in the TICKET table correspond to the names in +the name/value pairs of ticket change artifacts. When running the replay +algorithm, if a name/value pair is seen which has no corresponding column +in the TICKET table, then that name/value pair is ignored. Columns can +be added or removed from the TICKET table at any time. Whenever the +TICKET table is modified, the replay algorithm automatically reruns +to repopulate the table using the new column names. Note that the TICKET +table schema and content is part of the local state of a repository +and is not shared with other repositories during a sync, push, or pull. + +Each repository also defines scripts used to generate web pages for +creating new tickets, viewing existing tickets, and modifying an +existing ticket. These scripts consist of HTML with an embedded +scripts written in a Tcl-like language called "[./th1.md|TH1]". Every +new fossil repository is created with default scripts. Administrators +wishing to customize their ticket entry, viewing, and editing screens +should modify the default scripts to suit their needs. These screen +generator scripts are part of the local state of a repository and are +not shared with other repositories during a sync, push, or pull. + +To be continued... diff --git a/www/build-icons/linux.gif b/www/build-icons/linux.gif new file mode 100644 index 0000000..eaa5cb7 Binary files /dev/null and b/www/build-icons/linux.gif differ diff --git a/www/build-icons/linux64.gif b/www/build-icons/linux64.gif new file mode 100644 index 0000000..b254a5b Binary files /dev/null and b/www/build-icons/linux64.gif differ diff --git a/www/build-icons/mac.gif b/www/build-icons/mac.gif new file mode 100644 index 0000000..30f9538 Binary files /dev/null and b/www/build-icons/mac.gif differ diff --git a/www/build-icons/openbsd.gif b/www/build-icons/openbsd.gif new file mode 100644 index 0000000..88b76e3 Binary files /dev/null and b/www/build-icons/openbsd.gif differ diff --git a/www/build-icons/src.gif b/www/build-icons/src.gif new file mode 100644 index 0000000..810d39b Binary files /dev/null and b/www/build-icons/src.gif differ diff --git a/www/build-icons/win32.gif b/www/build-icons/win32.gif new file mode 100644 index 0000000..9098e86 Binary files /dev/null and b/www/build-icons/win32.gif differ diff --git a/www/build.wiki b/www/build.wiki new file mode 100644 index 0000000..7787461 --- /dev/null +++ b/www/build.wiki @@ -0,0 +1,543 @@ +Compiling and Installing Fossil + +

0.0 Using A Pre-compiled Binary

+ +[/uv/download.html|Pre-compiled binaries] are available for recent +releases. Just download +the appropriate executable for your platform +and put it on your $PATH. +To uninstall, simply delete the executable. +To upgrade from an older release, just overwrite the older binary with +the newer one. + +For details about how those binaries are built, see +[/wiki?name=Release+Build+How-To | the Release Build How-To wiki page]. + + +

0.1 Executive Summary

+ +Building and installing is very simple. Three steps: + +
    +
  1. Download and unpack a source tarball or ZIP. +
  2. ./configure; make +
  3. Move the resulting "fossil" or "fossil.exe" executable to someplace on +your $PATH. +
+ +
+ +

1.0 Obtaining The Source Code

+ +Fossil is self-hosting, so you can obtain a ZIP archive or tarball +containing a snapshot of the latest version directly from +Fossil's own fossil repository. Additionally, source archives of +released versions of +fossil are available from the [/uv/download.html|downloads page]. +To obtain a development version of fossil, follow these steps: + +
    +
  1. Point your web browser to [https://fossil-scm.org/]
    2. + +
  2. Click on the [/timeline|Timeline] +link at the top of the page.
    3. + +
  3. Select a version of Fossil you want to download. The latest +version on the trunk branch is usually a good choice. Click on its +link.
    4. + +
  4. Finally, click on one of the +"Zip Archive" or "Tarball" links, according to your preference. +These links will build a ZIP archive or a gzip-compressed tarball of the +complete source code and download it to your computer.
    5. +
+ +

Aside: Is it really safe to use an unreleased development version of +the Fossil source code?

+ +Yes! Any check-in on the +[/timeline?t=trunk | trunk branch] of the Fossil +[https://fossil-scm.org/ | self-hosting repository] +will work fine. (Dodgy code is always on a branch.) In the unlikely +event that you pick a version with a serious bug, it still won't +clobber your files. Fossil uses several +[./selfcheck.wiki | self-checks] prior to committing any +repository change that prevent loss-of-work due to bugs. + +The Fossil [./selfhost.wiki | self-hosting repositories], especially +the one at [https://fossil-scm.org/home], usually run a version +of trunk that is less than a week or two old. Look at the bottom +left-hand corner of this screen (to the right of "This page was +generated in...") to see exactly which version of Fossil is +rendering this page. It is always safe to use whatever version +of the Fossil code you find running on the main Fossil website. + +

2.0 Compiling

+ +
    +
  1. +Unpack the ZIP or tarball you downloaded then +cd into the directory created.
    2. + +
  2. (Optional, Debian-compatible Linux only) +Make sure you have all the necessary tools and libraries at hand by running: +sudo apt install tcl-dev tk libssl-dev zlib1g-dev. + +
  3. (Optional, Unix only) +Run ./configure to construct a makefile. + +
      +
    1. +The build system for Fossil on Unix-like systems assumes that the +OpenSSL development and runtime files are available on your system, +because unprotected repositories are trivial to attack otherwise. +Indeed, some public Fossil repositories — including Fossil's own — today +run in an HTTPS-only mode, so that you can't even do an anonymous clone +from them without using the TLS features added to Fossil by OpenSSL. To +weaken that stance could allow a +[https://en.wikipedia.org/wiki/Man-in-the-middle_attack|man in the +middle attack], such as one that substitutes malicious code into your +Fossil repository clone. + +You can force the Fossil build system to avoid searching for, building +against, and linking to the OpenSSL library by passing +--with-openssl=none to the configure script. + +If you do not have the OpenSSL development libraries on your system, +we recommend that you install them, typically via your OS's package +manager. The Fossil build system goes to a lot of effort to seek these +out wherever they may be found, so that is typically all you need to +do. + +For more advanced use cases, see the [./ssl.wiki#openssl-bin|OpenSSL +discussion in the "TLS and Fossil" document]. +
      2. + +
    2. +To build a statically linked binary, you can try adding +the --static option, but +[https://stackoverflow.com/questions/3430400/linux-static-linking-is-dead +| it may well not work]. If your platform of choice is affected by this, +the simplest workaround we're aware of is to build a Fossil container, +then [./containers.md#static | extract the static executable from it]. +
      3. + +
    3. +To enable the native [./th1.md#tclEval | Tcl integration feature] feature, +add the --with-tcl=1 and --with-tcl-private-stubs=1 options. +
      4. + +
    4. +Other configuration options can be seen by running +./configure --help +
      5. +
    + +
  4. Run "make" to build the "fossil" or "fossil.exe" executable. +The details depend on your platform and compiler.
    5. + +
      +
    1. Unix → the configure-generated Makefile should work on +all Unix and Unix-like systems. Simply type "make".
      2. + +
    2. Unix without running "configure" → if you prefer to avoid +running configure, you can also use: make -f Makefile.classic. You may +want to make minor edits to Makefile.classic to configure the build for your +system.
      3. + +
    3. MinGW / MinGW-w64 → The best-supported path is to build +via the MinGW specific Makefile under a POSIX build of GNU make: +"make -f win/Makefile.mingw".
      4. + +To enable the native [./th1.md#tclEval | Tcl integration feature], use a +command line like the following (all on one line): + +
      make -f win/Makefile.mingw FOSSIL_ENABLE_TCL=1 FOSSIL_ENABLE_TCL_STUBS=1 FOSSIL_ENABLE_TCL_PRIVATE_STUBS=1
      + +
    4. MSYS2 / Cygwin → This is something of a hybrid between +options "a" and "c" above: it configures and builds +fossil.exe much as on Linux, but you get a native Windows +executable out at the end. The primary downside is that this type of +executable can become confused when attempting to interoperate with +fully native Windows EXEs by making assumptions that only hold true when +all elements are running under the Cygwin/MSYS environment. The MSVC and +MinGW options do not have this limitation. + +Even so, there is value to Linux/Unix natives in having this hybrid +while off in Windows-land. The simpler of the two paths is MSYS2, since +it lets you install the necessary prerequisites in a single command +after installing the base environment: + +
      pacman -sS gcc make openssl-devel zlib-devel
      + +The equivalent in Cygwin's setup.exe requires stepping +through the GUI package chooser, and then if you miss one of the +prereqs, going all the way back through it again until you get it right. + +
    5. MSVC → Use the MSVC makefile.
      6. + +NB: Run the following nmake commands from a "x64 Native +Tools Command Prompt"; buildmsvc.bat is able to automatically load +the build tools (x64 by default, pass "x86" as the first argument to use the +x86 tools), so it can be called from a normal command prompt. + +First, change to the "win/" subdirectory ("cd win"), then run +"nmake /f Makefile.msc".

      Alternatively, the batch +file "win\buildmsvc.bat" may be used and it will attempt to +detect and use the latest installed version of MSVC.

      To enable +the optional OpenSSL support, +first download the official +source code for OpenSSL and extract it to an appropriately named +"openssl" subdirectory within the local +[/tree?ci=trunk&name=compat | compat] directory then make sure that some recent +Perl binaries are installed locally, +and finally run one of the following commands: +
      +nmake /f Makefile.msc FOSSIL_ENABLE_SSL=1 FOSSIL_BUILD_SSL=1 PERLDIR=C:\full\path\to\Perl\bin
+
      +
      +buildmsvc.bat FOSSIL_ENABLE_SSL=1 FOSSIL_BUILD_SSL=1 PERLDIR=C:\full\path\to\Perl\bin
+
      +To enable the optional native [./th1.md#tclEval | Tcl integration feature], +run one of the following commands or add the "FOSSIL_ENABLE_TCL=1" +argument to one of the other NMAKE command lines: +
      +nmake /f Makefile.msc FOSSIL_ENABLE_TCL=1
+
      +
      +buildmsvc.bat FOSSIL_ENABLE_TCL=1
+
      + +
    6. Cygwin → The same as other Unix-like systems. It is +recommended to configure using: "configure --disable-internal-sqlite", +making sure you have the "libsqlite3-devel" , "zlib-devel" and +"openssl-devel" packages installed first.
      7. +
    +
+ +

3.0 Installing

+ +
    +
  1. +The finished binary is named "fossil" (or "fossil.exe" on Windows). +Put this binary in a +directory that is somewhere on your PATH environment variable. +It does not matter where. +
    2. + +
  2. +(Optional:) +To uninstall, just delete the binary. +
    3. +
+ +

4.0 Additional Considerations

+ + + + +

5.0 Building a Docker Container

+ +The information on building Fossil inside an +[https://opencontainers.org/ | OCI container] is now in +[./containers.md | a separate document]. + +This includes the instructions on using the OCI container as an +expedient intermediary for building a statically-linked Fossil binary on +modern Linux platforms, which otherwise make this difficult. + + +

6.0 Building on/for Android

+ +

6.1 Cross-compiling from Linux

+ +The following instructions for building Fossil for Android via Linux, +without requiring a rooted OS, are adapted from +[forum:/forumpost/e0e9de4a7e | a forum post]. + +On the development machine, from the fossil source tree: + +
export CC=$NDK_PATH/toolchains/llvm/prebuilt/linux-x86_64/bin/armv7a-linux-androideabi21-clang
+./configure --with-openssl=none
+make
+
+ + +On the Android device, enable the USB debugging option from +Developer menu in Device Options. Connect the device to the development +system with USB. If it's configured and connected properly, +the device should show up in the output of adb devices: + +
sudo adb devices
+
+ +Copy the resulting fossil binary onto the device... + +
sudo adb push fossil /data/local/tmp
+
+ +And run it from an adb shell: + +
sudo adb shell
+> cd /data/local/tmp
+# Fossil requires a HOME directory to work with:
+> export HOME=$PWD
+> export PATH=$PWD:$PATH
+> fossil version
+This is fossil version 2.11 [e5653a4ceb] 2020-03-26 18:54:02 UTC
+
+ +The output might, or might not, include warnings such as: + +
WARNING: linker: ./fossil: unused DT entry: type 0x6ffffef5 arg 0x1464
+WARNING: linker: ./fossil: unused DT entry: type 0x6ffffffe arg 0x1ba8
+WARNING: linker: ./fossil: unused DT entry: type 0x6fffffff arg 0x2
+
+ +The source of such warnings is not 100% certain. +Some information about these (reportedly harmless) warnings can +be found +[https://stackoverflow.com/a/41900551 | on this StackOverflow post]. + + + +

7.0 Building for Fuzz Testing

+ +This feature is primarily intended for fossil's developers and may +change at any time. It is only known to work on Linux systems and has +been seen to work on x86/64 and ARM. + +Fossil has builtin support for processing specific features using +libfuzzer. The features which can be tested this way are +found in the help text for the [/help/test-fuzz|test-fuzz +command]. + +Fuzzing requires: + + * The clang C compiler. + * libfuzzer. On Ubuntu-derived systems, it can be installed with + apt install libfuzzer-XYZ, where XYZ is a version number + (several versions may be available on any given system) + + +Compile as follows: + +
make clean
+make TCCFLAGS='-DFOSSIL_FUZZ -fsanitize=fuzzer,address,undefined -O0 -g' CC=clang
+
+ +The resulting fossil binary differs from the standard +one primarily in that it runs the test-fuzz command by +default. It needs to be told what to fuzz and needs to be given a +directory of input files to seed the fuzzer with: + + +
$ mkdir cases
+  # Copy input files into ./cases. e.g. when fuzzing the markdown
+  # processor, copy any to-be-tested .md files into that directory.
+  # Then start the fuzzer:
+$ ./fossil-fuzz --fuzztype markdown cases
+
+ +As it works, it writes its mutated test files into the "cases" +directory, each one named in the form of a hash. When it finds a +problem it will produce a stack trace for the offending code, will +output the name of the file which triggered the crash (named +cases/SOME_HASH) and may, depending on the nature of the +problem, produce a file named crash-SOMETHING. In theory the +crash file can be fed directly back into the fuzzer to reproduce the +problem: + +
$ ./fossil-fuzz --fuzztype markdown crash-SOMETHING
+
+ +But whether or not it will genuinely crash may depend on static +app-level state which might not trigger the crash when running an +individual test. + +For a detailed information about the fuzzer's flags and features, see: + + * [https://llvm.org/docs/LibFuzzer.html] + * [https://github.com/google/fuzzing/blob/master/tutorial/libFuzzerTutorial.md] + +Flags for the fuzzer can be passed directly to fossil, +e.g. -jobs=4 to start four fuzzer jobs in parallel, but doing +so may cause the fuzzer to strip the --fuzztype flag, leading +to it testing the wrong thing. When passing on fuzzer-specific flags +along with --fuzztype, be sure to check your system's process +list to ensure that your --fuzztype flag is there. + + + +

8.0 Building WebAssembly Components

+ +Fossil uses one component built as +[https://developer.mozilla.org/en-US/docs/WebAssembly | WebAssembly] +a.k.a. WASM. Because compiling WASM code requires non-trivial +client-side tooling, the repository includes compiled copies of these +pieces. Most Fossil hackers should never need to concern themselves +with the WASM parts, but this section describes how to for those who +want or need to do so. + +The bits described in this section are necessary when updating +extsrc/pikchr.c from the upstream source, or the +fossil binary will use a different version of pikchr than +[/pikchrshow] does (as the latter runs the WASM build of pikchr). + +These instructions have only ever been tested on Linux systems. They +"should" work on any Unix-like system supported by Emscripten. The +fossil makefiles for Windows builds do not include any of the +WASM-related components (patches to add that would be welcomed, of +course). + +The first step is to configure the tree with support for +[https://emscripten.org/|Emscripten]. This requires that the system +has the Emscripten SDK (a.k.a. emsdk) installed, as documented at: + +[https://emscripten.org/docs/getting_started/downloads.html] + +For instructions on keeping the SDK up to date, see: + +[https://emscripten.org/docs/tools_reference/emsdk.html] + +
Getting Emscripten up and running is trivial and +painless, at least on Linux systems, but the installer downloads +many hundreds of megabytes of tools and dependencies, all of which +will be installed under the single SDK directory (as opposed to +being installed at the system level). It does, however, require +that python3 be installed at the system level and it can +optionally make use of a system-level cmake for certain tasks +unrelated to how fossil uses the SDK.
+ +After installing the SDK, configure the fossil tree with emsdk +support: + +
$ ./configure --with-emsdk=/path/to/emsdk \
+  --and-other-options...
+
+ +If the --with-emsdk flag is not provided, the configure +script will check for the environment variable EMSDK, which +is one of the standard variables the SDK environment uses. If that +variable is found, its value will implicitly be used in place of the +missing --with-emsdk flag. Thus, if the emsdk_env.sh +script is sourced into the shell before running the configure script, +the SDK will be detected even without the config flag. + +The configure script installs some makefile variables which tell the +build where to find the SDK and it generates a script named +tools/emcc.sh (from the template file +[/file/tools/emcc.sh.in|/tools/emcc.sh.in]), which is a +wrapper around the Emscripten C compiler (emcc). The wrapper +script uses the configure-time state to attempt to set up the various +environment variables which are required by emcc and will +fail if it cannot do so. Once it's set up the environment, it passes +on all of its arguments to emcc. + +The WASM-related build parts are set up such that none of them should +ever trigger implicitly (e.g. via dependencies resolution) in a normal +build cycle. They are instead explicitly built as described below. + +From the top of the source tree, all WASM-related components can be +built with: + +
$ make wasm
+ +
The file +[/file/extsrc/pikcher-worker.js|extsrc/pikcher-worker.js] +is hand-coded and intended to be loaded as a "Worker" in +JavaScript. That file loads the main module and provides an +interface via which a main JavaScript thread can communicate with +pikchr running in a Worker thread. The file +[/file/src/fossil.page.pikchrshowasm.js|src/fossil.page.pikchrshowasm.js] +implements the [/pikchrshow] app and demonstrates how +pikchr-worker.js is used.
+ +As of this writing, those parts include: + + * extsrc/pikchr.wasm is a WASM-compiled form of + extsrc/pikchr.c. + * extsrc/pikchr.js is JS/WASM glue code generated by Emscripten + to give JS code access to the API exported by the WASM file. + +When a new version of extsrc/pikchr.c is installed, the +files pikchr.{js,wasm} will need to be recompiled to account +for that. Running make wasm will, if the build is set up for +the emsdk, recompile those: + +
$ rm extsrc/pikchr.{js,wasm}
+# ^^^^ that rm has proven necessary in order to ensure rebuilds
+$ make wasm
+./tools/emcc.sh -o extsrc/pikchr.js ...
+$ ls -la extsrc/pikchr.{js,wasm}
+-rw-rw-r-- 1 stephan stephan 17263 Jun  8 03:59 extsrc/pikchr.js
+-rw-rw-r-- 1 stephan stephan 97578 Jun  8 03:59 extsrc/pikchr.wasm
+
+ +
If that fails with a message along the lines of +“setting `EXPORTED_RUNTIME_METHODS` expects `` but +got ``” then the emcc being invoked is too old: emcc +changed the format of list-type arguments at some point. The required +minimum version is unknown, but any SDK version from May 2022 or later +"should" (as of this writing) suffice. Any older version may or may not +work.
+ +After that succeeds, we need to run the normal build so that those +generated files can be compiled in to the fossil binary, accessible +via the [/help/www/builtin|/builtin page]: + +
$ make
+ +Before checking in those newly-built files, they need to be tested by +running the [/pikchrshow] page. If that page loads, the compilation +process fundamentally worked (a load failure will be made obvious to +the viewer). If it fails to load then the browser's dev tools console +likely provides at least a small hint (and sometimes a useful +hint) about the nature of the problem. Don't check those files in +until [/pikchrshow] runs, though! + +Should pikchr's C interface ever change, pikchr-worker.js +will need to be updated to accommodate it, but such modification is +typically trivial. + +

8.1 Solutions other than Emscripten?

+ +Emscripten is not the only option for building C as WASM, but it +provides a complete toolchain which eliminates many other steps which +must otherwise be accounted for on a per-project basis. Despite its +convenience, it behooves us to explore other build options for the +sake of portability and avoiding what amounts to vendor lock-in. + +For later refererence, here are articles specifically covering +building WASM projects without using Emscripten: + + * [https://surma.dev/things/c-to-webassembly/] + * [https://schellcode.github.io/webassembly-without-emscripten] diff --git a/www/cap-theorem.md b/www/cap-theorem.md new file mode 100644 index 0000000..d65a89f --- /dev/null +++ b/www/cap-theorem.md @@ -0,0 +1,173 @@ +# Fossil and the CAP Theorem + +[The CAP theorem][cap] is a fundamental mathematical proof about +distributed systems. A software system can no more get around it than a +physical system can get past *c*, the [speed of light][sol] constant. + +Fossil is a distributed system, so it can be useful to think about it in +terms of the CAP theorem. We won’t discuss the theorem itself or how you +reason using its results here. For that, we recommend [this article][tut]. + +[cap]: https://en.wikipedia.org/wiki/CAP_theorem +[sol]: https://en.wikipedia.org/wiki/Speed_of_light +[tut]: https://www.ibm.com/cloud/learn/cap-theorem + + + +## Fossil Is an AP-Mode System + +As with all common [DVCSes][dvcs], Fossil is an AP-mode system, meaning +that your local clone isn’t necessarily consistent with all other clones +(C), but the system is always available for use (A) and +partition-tolerant (P). This is what allows you to turn off Fossil’s +autosync mode, go off-network, and continue working with Fossil, even +though only a single node (your local repo clone) is accessible at the +time. + +You may consider that going back online restores “C”, because upon sync, +you’re now consistent with the repo you cloned from. But, if another +user has gone offline in the meantime, and they’ve made commits to their +disconnected repo, *you* aren’t consistent with *them.* Besides which, +if another user commits to the central repo, that doesn’t push the +change down to you automatically: even if all users of a Fossil system +are online at the same instant, and they’re all using autosync, Fossil +doesn’t guarantee consistency across the network. + +There’s no getting around the CAP theorem! + +[dvcs]: https://en.wikipedia.org/wiki/Distributed_version_control + + + +## CA-Mode Fossil + +What would it mean to redesign Fossil to be CA-mode? + +It means we get a system that is always consistent (C) and available (A) +as long as there are no partitions (P). + +That’s basically [CVS] and [Subversion][svn]: you can only continue +working with the repository itself as long as your connection to the central repo server functions. + +It’s rather trivial to talk about single-point-of-failure systems like +CVS or Subversion as +CA-mode. Another common example used this way is a classical RDBMS, but +aren’t we here to talk about distributed systems? What’s a good example +of a *distributed* CA-mode system? + +A better example is [Kafka], which in its default configuration assumes +it being run on a corporate LAN in a single data center, so network +partitions are exceedingly rare. It therefore sacrifices partition +tolerance to get the advantages of CA-mode operation. In its particular application of +this mode, a +message isn’t “committed” until all running brokers have a copy of it, +at which point the message becomes visible to the client(s). In that +way, all clients always see the same message store as long as all of the +Kafka servers are up and communicating. + +How would that work in Fossil terms? + +If there is only one central server and I clone it on my local laptop, +then CA mode means I can only commit if the remote Fossil is available, +so in that sense, it devolves to the old CVS model. + +What if there are three clones? Perhaps there is a central server *A*, +the clone *B* on my laptop, and the clone *C* on your laptop. Doesn’t CA +mode now mean that my commit on *B* doesn’t exist after I commit it to +the central repo *A* until you, my coworker, *also* pull down the copy +of that commit to your laptop *C*, validating the commit through the +network? + +That’s one way to design the system, but another way would be to scope +the system to only talk about proper *servers*, not about the clients. +In that model, a CA-mode Fossil alternative might require 2+ servers to +be running for proper replication. When I make a commit, if all of the +configured servers aren’t online, I can’t commit. This is basically CVS +with replication, but without any useful amount of failover. + +[CVS]: https://en.wikipedia.org/wiki/Concurrent_Versions_System +[Kafka]: https://engineering.linkedin.com/kafka/intra-cluster-replication-apache-kafka +[svn]: https://en.wikipedia.org/wiki/Apache_Subversion + + + +## CP-Mode Fossil + +What if we modify our CA-mode system above with “warm spares”? We can +say that commits must go to all of the spares as well as the active +servers, but a loss of one active server requires that one warm spare +come into active state, and all of the clients learn that the spare is +now considered “active.” At this point, you have a CP-mode system, not a +CA-mode system, because it’s now partition-tolerant (P) but it becomes +unavailable when there aren’t enough active servers or warm +spares to promote to active status. + +CP is your classical [BFT] style distributed consensus system, where the +system is available only if the client can contact a *majority* of the +servers. This is a formalization of the warm spare concept above: with +*N* server nodes, you need at least ⌊*N* / 2⌋ + 1 of them to be online +for a commit to succeed. + +Many distributed database systems run in CP mode because consistency (C) and +partition-tolerance (P) is a useful combination. What you lose is +always-available (A) operation: with a suitably bad partition, the +system goes down for users on the small side of that partition. + +An optional CP mode for Fossil would be attractive in some ways since in +some sense Fossil is a distributed DBMS, but in practical terms, it +means Fossil would then not be a [DVCS] in the most useful sense, being +that you could work while your client is disconnected from the remote +Fossil it cloned from. + +A fraught question is whether the non-server Fossil clones count as +“nodes” in this sense. + +If they do count, then if there are only two systems, the central server +and the clone on my laptop, then it stands to reason from the formula +above that I can only commit if the central server is available. In that +scheme, a CP-mode Fossil is basically like CVS. + +But what happens if my company hires a coworker to help me with the +project, and this person makes their own clone of the central repo? The +equation says I still need 2 nodes to be available for a commit, so if +my new coworker goes off-network, that doesn’t affect whether I can make +commits. Likewise, if I go off-network, my coworker can make commits to +the central server. + +But what happens if the central server goes down? The equation says we +still have 2 nodes, so we should be able to commit, right? Sure, but +only if my laptop and communicate directly to my coworker’s laptop! If +it can’t, that’s also a network partition, so *N=1* on both sides in +that case. The implication is that for a true CP-mode Fossil, we’d need +some kind of peer-to-peer networking layer so that our laptops can +accept commits from the other, so that when the central server comes +online, one of us can send the results up to it to get it caught up. + +But doesn’t that then mean there is no security? How does [Fossil’s RBAC +system][caps] work if peer-to-peer commits are allowed? + +You can instead reconceptualize the system as “node” meaning only server +nodes, so that client-only systems don’t count. This allows you to have +an RBAC system again. + +With just one central server, ⌊1/2⌋+1=1, so you get CVS-like behavior: +if the server’s up, you can commit. + +If you set up 2 servers for redundancy, both must be up for commits to +be allowed, since otherwise you could end up with half the commits going +to the server on one side of a network partition, half going to the +other, and no way to arbitrate among the two once the partition is +lifted. + +(Today’s AP-mode Fossil has this capability, but the necessary cost is +“C”, consistency! Once again, you can’t get around the CAP theorem.) + +3 servers is more sensible: any client that can see at least 2 of them +can commit. + +Will there ever be a CP-mode Fossil? This author doubts it, but as I’ve +shown, it would be useful in contexts where you’d rather have a +guarantee of consistency than availability. + +[BFT]: https://en.wikipedia.org/wiki/Byzantine_fault +[caps]: ./caps/ diff --git a/www/caps/admin-v-setup.md b/www/caps/admin-v-setup.md new file mode 100644 index 0000000..db1b412 --- /dev/null +++ b/www/caps/admin-v-setup.md @@ -0,0 +1,461 @@ +# Differences Between Setup and Admin Users + +This document explains the distinction between [Setup users][caps] and +[Admin users][capa]. For other information about use types, see: + +* [Administering User Capabilities](./) +* [How Moderation Works](../forum.wiki#moderation) +* [Users vs Subscribers](../alerts.md#uvs) +* [Defense Against Spiders](../antibot.wiki) + + +## Philosophical Core + +The Setup user "owns" the Fossil repository and may delegate a subset of +that power to one or more Admin users. + +The Setup user can grant Admin capability and take it away, but Admin +users cannot grant themselves Setup capability, either directly via the +Admin → Users UI page or via any indirect means. (If you discover +indirect means to elevate Admin privilege to Setup, it's a bug, so +please [report it][forum]!) + +It is common for the Setup user to have administrative control over the +host system running the Fossil repository, whereas it makes no sense for +Admin users to have that ability. If an Admin-only user had `root` +access on a Linux box running the Fossil instance they are an Admin on, +they could elevate their capability to Setup in several ways. (The +`fossil user` command, the `fossil sql` command, editing the repository +DB file directly, etc.) Therefore, if you wish to grant someone +Setup-like capability on a Fossil repository but you're unwilling to +give them a login on the host system, you probably want to grant +them Admin capability instead. + +Admin power is delegated from Setup. When a Setup user grants Admin +capability, it is an expression of trust in that user's judgement. + +Admin-only users must not fight against the policies of the Setup user. +Such a rift would be just cause for the Setup user to strip the Admin +user's capabilities. This may then create a fork in the project’s +development effort as the ex-Admin takes their clone and stands it up +elsewhere, so they may become that fork’s Setup user. + +A useful rule of thumb here is that Admin users should only change +things that the Setup user has not changed from the stock configuration. +In this way, an Admin-only user can avoid overriding the Setup user's +choices. + +You can also look at the role of Admin from the other direction, up +through the [user power hierarchy][ucap] rather than down from Setup. An +Admin user is usually a “super-developer” role, given full control over +the repository’s managed content: versioned artifacts in [the hash tree][bc], +[unversioned content][uv], forum posts, wiki articles, +tickets, etc. + +We’ll explore these distinctions in the rest of this document. + +[bc]: ../blockchain.md +[ucap]: ./index.md#ucap +[uv]: ../unvers.wiki + + +## No Granularity + +Fossil doesn’t make any distinction between these two user types beyond +this binary choice: Setup or Admin. + +A few features of Fossil are broken down so that only part of the +feature is accessible to Admin, with the rest left only to Setup users, +but for the most part each feature affected by this distinction is +either Admin + Setup or Setup-only. + +We could add more capability letters to break down individual +sub-features, but we’d run out of ASCII alphanumerics pretty quickly, +and we might even run out of ASCII punctuation and symbols. Then would +we need to shift to Unicode? + +Consider the Admin → Settings page, which is currently restricted to +Setup users only: you might imagine breaking this up into several +subsets so that some settings can be changed by Admin users. Is that a +good idea? Maybe, but it should be done only after due consideration. It +would definitely be wrong to assign a user capability bit to *each* +setting on that page. + +Now consider the opposite sort of case, Admin → Skins. Fossil grants +Admin users full access to this page so that the Admins can maintain and +extend the skin as the repository evolves, not so Admins can switch the +entire skin to another without consulting with the Setup user first. How +would Fossil decide, using user capabilities only, which skin changes +the Admin user is allowed to do, and which must be left to Setup? Do we +assign a separate capability letter to each step in `/setup_skin`? Do we +assign one more each to the five sections of a skin? (Header, Footer, +CSS, JavaScript, and Details.) It quickly becomes unmanageable. + + + +## Capability Groups + +We can break up the set of powers the Admin user capability grants into +several groups, then defend each group as a coherent whole. + + +### Security + +While establishing the Fossil repository's security policy is a task for +the Setup user, *maintaining* that policy is something that Fossil +allows a Setup user to delegate to trustworthy users via the Admin user +capability: + +* **Manage users**: The only thing an Admin-only user cannot do on the + Admin → Users page is grant Setup capability, either to themselves + or to other users. The intent is that Admin users be able to take + some of the load of routine user management tasks off the shoulders + of the Setup user: delete accounts created by spammers, fix email + alert subscriptions, reset passwords, etc. + +* **Security audit**: The Admin → Security-Audit page runs several + tests on the Fossil repository's configuration, then reports + potential problems it found and offers canned solutions. Those + canned solutions do not do anything that an Admin-user could not do + via other means, so this page offers the Admin-only user no more + power than they otherwise had. For example, this page's "Take it + Private" feature can also be done manually via Admin → Users. This + page is a convenience, not a grant of new power to the Admin-only + user. + +* **Logging**: Admin-only users get to see the various + Fossil logs in case they need to use them to understand a problem + they're empowered to solve. An obvious example is a spam attack: the + Admin might want to find the user's last-used IP, see if they cloned + the repository, see if they attempted to brute-force an existing + login before self-registering, etc. + +Some security-conscious people might be bothered by the fact that +Admin-only users have these abilities. Think of a large IT organization: +if the CIO hires a [tiger team][tt] to test the company's internal IT +defenses, the line grunts fix the reported problems, not the CIO. + + +### Administrivia + +It is perfectly fine for a Fossil repository to only have Setup users, +no Admin users. The smaller the repository, the more likely the +repository has no Admin-only users. If the Setup user neither needs nor +wants to grant Admin power to others, there is no requirement in Fossil +to do so. [Setup capability is a pure superset of Admin capability.][sia] + +As the number of users on a Fossil repository grows, the value in +delegating administrivia also grows, because the Setup user typically +has other time sinks they consider more important. + +Admin users can take over the following routine tasks on behalf of the +Setup user: + +* **Shunning**: After user management, this is one of the greatest + powers of an Admin-only user. Fossil grants access to the Admin → + Shunned page to Admin users rather than reserve it to Setup users + because one of the primary purposes of [the Fossil shunning + system][shun] is to clean up after a spammer, and that's + exactly the sort of administrivia we wish to delegate to Admin users. + + Coupled with the Rebuild button on the same page, an Admin user has + the power to delete the repository's entire + [hash tree][bc]! This makes this feature a pretty good + razor in deciding whether to grant someone Admin capability: do you + trust that user to shun Fossil artifacts responsibly? + + Realize that shunning is cooperative in Fossil. As long as there are + surviving repository clones, an Admin-only user who deletes the + whole hash tree has merely caused a nuisance. An Admin-only user + cannot permanently destroy the repository unless the Setup user has + been so silly as to have no up-to-date clones. + +* **Moderation**: According to [the user power hierarchy][ucap], + Admins are greater than Moderators, so control over + what Moderators can do clearly belongs to both Admins and to the + Setup user(s). + +* **Status**: Although the Fossil `/stat` page is visible to every + user with Read capability, there are several additional things this + page gives access to when a user also has the Admin capability: + + *

[Email alerts][ale] and [backoffice](../backoffice.md) + status. Admin-only users cannot modify the email alerts setup, + but they can see some details about its configuration and + current status.

+ + *

The `/urllist` page, which is a read-only page showing the + ways the repository can be accessed and how it has been accessed + in the past. Logically, this is an extension to logging, + [covered above](#log).

+ + *

The Fossil repository SQL schema. This is not particularly + sensitive information, since you get more or less the same + information when you clone the repository. It's restricted to + Admin because it's primarily useful in debugging SQL errors, + which happen most often when Fossil itself is in flux and the + schema isn't being automatically updated correctly. That puts + this squarely into the "administrivia" category.

+ + *

Web cache status, environment, and logging: more + administrivia meant to help the Admin debug problems.

+ +* **Configure search** + +[ale]: ../alerts.md +[shun]: ../shunning.wiki + + +### Cosmetics + +While the Setup user is responsible for setting up the initial "look" of +a Fossil repository, the Setup user entrusts Admin users with +*maintaining* that look. An Admin-only user therefore has the following +special abilities: + +* Modify the repository skin + +* Create and modify URL aliases + +* Manage the "ad units" feature, if enabled. + +* Adjust the `/timeline` display preferences. + +* Change the "logo" element displayed by some skins. + +These capabilities allow an Admin-only user to affect the branding and +possibly even the back-end finances of a project. This is why we began +this document with a philosophical discussion: if you cannot entrust a +user with these powers, you should not grant that user Admin capability. + + +## Clones and Backups + +Fossil is a *distributed* version control system, which has direct +effects on the “Setup user” concept in the face of clones. When you +clone a repository, your local user becomes a Setup user on the local +clone even if you are not one on the remote repository. This may be +surprising to you, but it should also be sensible once you realize that +your operating system will generally give you full control over the +local repository file. What use trying to apply remote restrictions on +the local file, then? + +The distinctions above therefore are intransitive: they apply only +within a single repository instance. + +Fossil behaves differently when you do a clone as a user with Setup +capability on the remote repository, which primarily has effects on the +fidelity of clone-as-backup, which we cover [elsewhere](../backup.md). +We strongly encourage you to read that document if you expect to use a +clone as a complete replacement for the remote repository. + + +## The All-Powerful Setup User + +Setup users get [every user capability](./ref.html) of Fossil except for +[two exceptionally dangerous capabilities](#dcap), which they can later +grant to themselves or to others. + +In addition, Setup users can use every feature of the Fossil UI. If Fossil can do a +thing, a Setup user on that repo can make Fossil do it. + +Setup users can do many things that Admin users cannot. They may not +only use all of the Admin UI features, they may also: + +* See record IDs (RIDs) on screens that show them +* See the MIME type of attachments on [`/ainfo` pages](/help/www/ainfo) +* See a remote repo’s HTTP [cache status](/help/www/cachestat) + and [pull cache entries](/help/www/cacheget) +* Edit a Setup user’s account! + +The “Admin” feature of Fossil UI is so-named because Admin users can use +about half of its functions, but only Setup can use these pages: + +* **Access**: This page falls under the [Security](#security) + category above, but like Configuration, it's generally something set + up once and never touched, so only Setup users should change it. + +* **Configuration**: This page nominally falls + under [Cosmetics](#cosmetics) above, but it's such a core part of the Fossil + configuration — something every Setup user is expected to fully + specify on initial repository setup — that we have trouble + justifying any case where an Admin-only user would have good cause + to modify any of it. This page is generally set up once and then + never touched again. + +* **Email-Server**: This is an experimental SMTP server feature which + is currently unused in Fossil. Should we get it working, it will + likely remain Setup-only, since it will likely be used as a + replacement for the platform’s default SMTP server, a powerful + position for a piece of software to take. + +* **Login-Group**: [Login groups][lg] allow one Fossil repository to + delegate user access to another. Since an Admin-only user on one + repo might not have such access to another repo on the same host + system, this must be a Setup-only task. + +* **Notification**: This is the main UI for setting up integration + with a platform’s SMTP service, for use in sending out [email + notifications][ale]. Because this screen can set commands to execute + on the host, and because finishing the configuration requires a + login on the Fossil host system, it is not appropriate to give Admin + users access to it. + +* **Settings**: The [repository settings][rs] available via Admin → + Settings have too wide a range of power to allow modification by + Admin-only users: + + *

Harmless: Admin-only users on a repository may well + have checkin rights on the repository, so the fact that + versionable settings like `crlf-glob` can also be set at the + repository level seems like a thing we might want to allow + Admin-only users the ability to change. Since Fossil currently + has no way to allow only some settings to be changed by + Admin-only users and some not, we can't just show these harmless + settings to Admin-only users.

+ + *

Low-Risk: The admin-log setting controls + whether the Fossil admin log is generated. Since we've already decided that Admin-only users can see + this log, it seems fine that the Admin users can choose whether + this log gets generated in the first place.

+ +

There's a small risk that a rogue Admin user could disable + the log before doing something evil that the log would capture, + so ideally, we'd want to restrict changing this setting from 1 + to 0 to Setup only while allowing Admin-only users to change it + from 0 to 1. Fossil doesn't currently allow that.

+ + *

Risky: The https-login setting falls under + the "Security" section above, but it should probably never be + adjusted by Admin-only users. Sites that want it on will never + want it to be disabled without a very good reason.

+ +

There is also an inverse risk: if the site has a front-end + HTTPS proxy that uses HTTP to communicate over localhost to + Fossil, enabling this setting will create an infinite redirect + loop! (Ask me how I know.)

+ + *

Dangerous: The email-send-command setting + could allow a rogue Admin to run arbitrary commands on the host + system, unless it's prevented via some kind of host-specific + restriction. (chroot, jails, SELinux, VMs, etc.) Since it makes + no sense to trust Admin-only users with root level + access on the host system, we almost certainly don't want to + allow them to change such settings.

+ +* **SQL**: The Admin → SQL feature allows the Setup user to enter raw + SQL queries against the Fossil repository via Fossil UI. This not + only allows arbitrary ability to modify the repository hash tree + and its backing data tables, it can probably also be used to damage + the host such as via `PRAGMA temp_store = FILE`. + +* **Tickets**: This section allows input of arbitrary TH1 code that + runs on the server, affecting the way the Fossil ticketing system + works. The justification in the **TH1** section below therefore + applies. + +* **TH1**: The [TH1 language][th1] is quite restricted relative to the + Tcl language it descends from, so this author does not believe there + is a way to damage the Fossil repository or its host via the Admin → + TH1 feature, which allows execution of arbitrary TH1 code within the + repository's execution context. Nevertheless, interpreters are a + well-known source of security problems, so it seems best to restrict + this feature to Setup-only users as long as we lack a good reason + for Admin-only users to have access to it. + +* **Transfers**: This is for setting up TH1 hooks on various actions, + so the justification in the **TH1** section above applies. + +* **Wiki**: These are mainly cosmetic and usability settings. We might + open this up to Admin users in the future. + +Just remember, [user caps affect Fossil’s web interfaces only][webo]. A +user is a Setup user by default on their local clone of a repo, and +Fossil’s ability to protect itself against malicious (or even simply +incorrect) pushes is limited. Someone with clone and push capability on +your repo could clone it, modify their local repo, and then push the +changes back to your repo. Be careful who you give that combination of +capabilities to! + +When you run [`fossil ui`][fui], you are the Setup user on that repo +through that UI instance, regardless of the capability set defined in +the repo’s user table. This is true even if you cloned a remote repo +where you do not have Setup caps. This is why `ui` always binds to +`localhost` without needing the `--localhost` flag: in this mode, anyone +who can connect to that repo’s web UI has full power over that repo. + + +## Dangerous Capabilities Initially Denied to Everyone + +There are two capabilities that Fossil doesn’t grant by default to Setup +or Admin users automatically. They are exceptionally dangerous, so +Fossil makes these users grant themselves (or others) these capabilities +deliberately, hopefully after careful consideration. + + +### Write Unversioned + +Fossil currently doesn’t distinguish the sub-operations of +[`fossil uv`](/help/uv); they’re all covered by [**WrUnver**][capy] +(“y”) capability. Since some of these operations are unconditionally +destructive due to the nature of unversioned content, and since this +goes against Fossil’s philosophy of immutable history, nobody gets cap +“y” on a Fossil repo by default, not even the Setup or Admin users. A +Setup or Admin user must grant cap “y” to someone — not necessarily +themselves! — before modifications to remote +unversioned content are possible. + +Operations on unversioned content made without this capability affect +your local clone only. In this way, your local unversioned file table +can have different content from that in its parent repo. This state of +affairs will continue until your user either gets cap “y” and syncs that +content with its parent or you say `fossil uv revert` to make your local +unversioned content table match that of its parent repo. + + +### Private Branch Push + +For private branches to remain private, they must never be accidentally +pushed to a public repository. It can be [difficult to impossible][shun] +to recover from such a mistake, so nobody gets [**Private**][capx] (“x”) +capability on a Fossil repo by default, not even Admin or Setup users. + +There are two common uses for private branches. + +One use is part of a local social contract allowing individual +developers to work on some things in private until they’re ready to push +them up to the parent repository. This goes against [a core tenet][fdp] +of Fossil’s design philosophy, but Fossil allows it, so some development +organizations do this. If yours is one of these, you might give cap “x” +to the “developer” category. + +The other use is in development organizations that follow the Fossil +philosophy, where you do not work in private unless you absolutely must. +You may have a public-facing project — let’s call it “SQLite” for the +sake of argument — but then someone comes along and commissions a custom +modification to your project which they wish to keep proprietary. You +do your work on a private branch, which you absolutely must never push +to the public repo, because that would be illegal. (Breach of contract, +copyright violation on a work-for-hire agreement, etc.) If you are using +Fossil in this way, we recommend that you give “x” capability to a +special developer account only, if at all, to minimize the chance of an +accidental push. + + +[capa]: ./ref.html#a +[caps]: ./ref.html#s +[capx]: ./ref.html#x +[capy]: ./ref.html#y + +[fcp]: https://fossil-scm.org/home/help/configuration +[fdp]: ../fossil-v-git.wiki#devorg +[forum]: https://fossil-scm.org/forum/ +[fui]: /help/ui +[lg]: ./login-groups.md +[rs]: https://fossil-scm.org/home/doc/trunk/www/settings.wiki +[sia]: https://fossil-scm.org/home/artifact?ln=1259-1260&name=0fda31b6683c206a +[snoy]: https://fossil-scm.org/forum/forumpost/00e1c4ecff +[th1]: ../th1.md +[tt]: https://en.wikipedia.org/wiki/Tiger_team#Security +[webo]: ./#webonly diff --git a/www/caps/impl.md b/www/caps/impl.md new file mode 100644 index 0000000..1f9b5a4 --- /dev/null +++ b/www/caps/impl.md @@ -0,0 +1,111 @@ +# Implementation Details of User Capabilities + +## Capability Letter Choices + +We [assigned][ref] user capability characters using only lowercase ASCII +letters at first, so those are the most important within Fossil: they +control the functions most core to Fossil’s operation. Once we used up +most of the lowercase letters, we started using uppercase, and then +during the development of the [forum feature][for] we assigned most of +the decimal numerals. All of the lowercase ASCII letters are now +assigned. Eventually, we might have to start using ASCII +punctuation and symbols. We expect to run out of reasons to define new caps before +we’re forced to switch to Unicode, though the possibilities for [mnemonic][mn] +assignments with emoji are intriguing. 😉 + +The existing caps are usually mnemonic, especially among the +earliest and therefore most central assignments, made when we still had +lots of letters to choose from. There is still hope for good future +mnemonic assignments among the uppercase letters, which are mostly still +unused. + + +## Why Not Bitfields? + +Some may question the use of ASCII character strings for [capability +sets][ucap] instead of bitfields, which are more efficient, both in +terms of storage and processing time. + +Fossil handles these character strings in one of two ways. For most HTTP +hits, Fossil [expands][sexp] the string into a [`struct` full of +flags][sff] so that later code can just do simple Boolean tests. In a +minority of cases, where Fossil only needs to check for the presence of +a single flag, it just does a [`strchr()` call][sc] on the string +instead. + +Both methods are slower than bit testing in a bitfield, but keep the +execution context in mind: at the front end of an HTTP request handler, +where the nanosecond differences in such implementation details are +completely swamped by the millisecond scale ping time of that repo’s +network connection, followed by the required I/O to satisfy the request. +Either method is plenty fast in that context. + +In exchange for this immeasurable cost per hit, we get human-readable +capability sets. + + +## Why Doesn’t Fossil Filter “Bad” Artifacts on Sync? + +Fossil is more trusting about the content it receives from a remote +clone during sync than you might expect. Common manifestations of this +design choice are: + +1. A user may be able to impersonate other users. This can be + [accidental](./index.md#defuser) as well as purposeful. + +2. If your local system clock is out-of-sync with absolute time, + artifacts committed to that repo will appear with the “wrong” time + when sync’d. If the time sync error is big enough, it can make + check-ins appear to go back in time and other bad effects. + +3. You can purposely overwrite good timestamps with bad ones and push + those changes up to the remote with no interference, even though + Fossil tries to make that a Setup-only operation. + +All of this falls out of two of Fossil’s design choices: sync is +all-or-nothing, and [the Fossil hash tree][bc] is immutable. Fossil +would have to violate one or both of these principles to filter such +problems out of incoming syncs. + +We have considered auto-[shunning][shun] “bad” content on sync, but this +is [difficult][asd] due to [the design of the sync protocol][dsp]. This +is not an impossible set of circumstances, but implementing a robust +filter on this input path would be roughly as difficult as writing a +basic [inter-frame video codec][ifvc]: do-able, but still a lot of +work. Patches to do this will be thoughtfully considered. + +We can’t simply change content as it arrives. Such manipulations would +change the artifact manifests, which would change the hashes, which +would require rewriting all parts of the block chain from that point out +to the tips of those branches. The local Fossil repo must then go +through the same process as the remote one on subsequent syncs in order +to build up a sync sequence that the remote can understand. Even if +you’re willing to accept all of that, this would break all references to +the old artifact IDs in forum posts, wiki articles, check-in comments, +tickets, etc. + +The bottom line here is that [**Clone**](./ref.html#g) and +[**Write**](./ref.html#i) are a potent combination of user capabilities. +Be careful who you give that pair to! + + +----- + +*[Back to Administering User Capabilities](./)* + + +
+ +[asd]: https://fossil-scm.org/forum/forumpost/ce4a3b5f3e +[bc]: ../blockchain.md +[dsp]: https://fossil-scm.org/fossil/doc/trunk/www/sync.wiki +[for]: ./forum.wiki +[ifvc]: https://en.wikipedia.org/wiki/Inter_frame +[mn]: https://en.wikipedia.org/wiki/Mnemonic +[ref]: ./ref.html +[sexp]: /artifact?ln=1223-1298&name=889d6724 +[sff]: /artifact?ln=80-117&name=52d2860f +[sc]: https://en.cppreference.com/w/c/string/byte/strchr +[shun]: ../shunning.wiki +[ucap]: ./index.md#ucap diff --git a/www/caps/index.md b/www/caps/index.md new file mode 100644 index 0000000..3492e5e --- /dev/null +++ b/www/caps/index.md @@ -0,0 +1,405 @@ +# Administering User Capabilities (a.k.a. Permissions) + +Fossil includes a powerful [role-based access control system][rbac] +which affects which users have which capabilities(^Some parts of the +Fossil code call these “permissions” instead, but since there is [a +clear and present risk of confusion](#webonly) with operating system +level file permissions in this context, we avoid using that term for +Fossil’s RBAC capability flags in these pages.) within a given +[served][svr] Fossil repository. We call this the “caps” system for +short. + +Fossil stores a user’s caps as an unordered string of ASCII characters, +one capability per, [currently](./impl.md#choices) limited to +[alphanumerics][an]. Caps are case-sensitive: “**A**” and “**a**” are +different user capabilities. + +This is a complex topic, so some sub-topics have their own documents: + +1. [Login Groups][lg] +2. [Implementation Details](./impl.md) +3. [User Capability Reference](./ref.html) + +[an]: https://en.wikipedia.org/wiki/Alphanumeric +[avs]: ./admin-v-setup.md +[lg]: ./login-groups.md +[rbac]: https://en.wikipedia.org/wiki/Role-based_access_control + + +## User Categories + +Before we explain individual user capabilities and their proper +administration, we want to talk about an oft-overlooked and +misunderstood feature of Fossil: user categories. + +Fossil defines four user categories. Two of these apply based on the +user’s login status: **nobody** and **anonymous**. The other two act +like Unix or LDAP user groups: **reader** and **developer**. Because we +use the word “group” for [another purpose][lg] in Fossil, we will +avoid using it that way again in this document. The correct term in +Fossil is “category.” + +Fossil user categories give you a way to define capability sets for four +hard-coded situations within the Fossil C source code. Logically +speaking: + +> *(developer* ∨ *reader)* ≥ *anonymous* ≥ *nobody* + +When a user visits a [served Fossil repository][svr] via its web UI, +they initially get the capabilities of the “nobody” user category. This +category would be better named “everybody” because it applies whether +you’re logged in or not. + +When a user logs in as “anonymous” via [`/login`](/help?name=/login) they +get all of the “nobody” category’s caps plus those assigned to the +“anonymous” user category. It would be better named “user” because it +affects all logged-in users, not just those logged in via Fossil’s +anonymous user feature. + +When a user with either the “reader” ([**u**][u]) or “developer” +([**v**][v]) capability letter logs in, they get their [individual user +caps](#ucap) plus those assigned to this special user category. They +also get those assigned to the “anonymous” and “nobody” categories. + +Because “developer” users do not automatically inherit “reader” caps, +it is standard practice to give both letters to your “developer” users: +**uv**. You could instead just assign cap **u** to the “developer” +category. + +Fossil shows how these capabilities apply hierarchically in the user +editing screen (Admin → Users → name) with the `[N]` `[A]` `[D]` `[R]` +tags next to each capability check box. If a user gets a capability from +one of the user categories already assigned to it, there is no value in +redundantly assigning that same cap to the user explicitly. For example, +with the default **ei** cap set for the “developer” category, the cap +set **ve** is redundant because **v** grants **ei**, which includes +**e**. + +We suggest that you lean heavily on these fixed user categories when +setting up new users. Ideally, your users will group neatly into one of +the predefined categories, but if not, you might be able to shoehorn +them into our fixed scheme. For example, the administrator of a +wiki-only Fossil repo for non-developers could treat the “developer” +user category as if it were called “author,” and a forum-only repo could +treat the same category as if it were called “member.” + +There is currently no way to define custom user categories. + +[svr]: ../server/ + + +## Individual User Capabilities + +When one or more users need to be different from the basic capabilities +defined in user categories, you can assign caps to individual users. You +may want to have the [cap reference][ref] open when doing such work. + +It is useful at this time to expand on the logical +expression [above](#cat), which covered only the four fixed user categories. +When we bring the individual user capabilities into it, the complete +expression of the way Fossil implements user power becomes: + +> *setup* ≥ *admin* ≥ *moderator* ≥ *(developer* ∨ *reader)* ≥ *[subscriber]* ≥ *anonymous* ≥ *nobody* + +The two additions at the top are clear: [setup is all-powerful][apsu], +and since admin users have [all capabilities][ref] except for Setup +capability, they are [subordinate only to the setup user(s)][avsp]. + +The moderator insertion could go anywhere from where it’s shown now down +to above the “anonymous” level, depending on what other caps you give to +your moderators. Also, there is not just one type of moderator: Fossil +has [wiki][l], [ticket][q], and [forum][5] moderators, each +independent of the others. Usually your moderators are fairly +high-status users, with developer capabilities or higher, but Fossil +does allow the creation of low-status moderators. + +The placement of “subscriber” in that hierarchy is for the +sort of subscriber who has registered an account on the repository +purely to [receive email alerts and announcements][7]. Users with +additional caps can also be subscribers, but not all users *are* in fact +subscribers, which is why we show it in square brackets. (See [Users vs +Subscribers](../alerts.md#uvs).) + +[apsu]: ./admin-v-setup.md#apsu +[avsp]: ./admin-v-setup.md#philosophy + + +## New Repository Defaults + +Fossil creates one user account in new repos, which is named after your +OS user name [by default](#defuser). + +Fossil gives the initial repository user the [all-powerful Setup +capability][apsu]. + +Users who visit a [served repository][svr] without logging in get the +“nobody” user category’s caps which default to +**[g][g][j][j][o][o][r][r][z][z]**: clone the repo, read the wiki, +check-out files via the web UI, view tickets, and pull version archives. +This default is suited to random passers-by on a typical FOSS project’s +public web site and its code repository. + +Users who [prove they are not a bot][bot] by logging in — even if only +as “anonymous” — get the “nobody” capability set plus +**[h][h][m][m][n][n][c][c]**: see internal hyperlinks, append to +existing wiki articles, file new tickets, and comment on existing +tickets. We chose these additional capabilities as those we don’t want +bots to have, but which a typical small FOSS project would be happy to +give anonymous humans visiting the project site. + +The “reader” user category is typically assigned to users who want to be +identified within the repository but who primarily have a passive role +in the project. The default capability set on a Fossil repo adds +**[k][k][p][p][t][t][w][w]** caps to those granted by “nobody” and +“anonymous”. This category is not well-named, because the default caps +are all about modifying repository content: edit existing wiki pages, +change one’s own password, create new ticket report formats, and modify +existing tickets. This category would be better named “participant”. + +Those in the “developer” category get the “nobody” and “anonymous” cap +sets plus **[e][e][i][i]**: view +sensitive user material and check in changes. + +[bot]: ../antibot.wiki + + +## Consequences of Taking a Repository Private + +When you click Admin → Security-Audit → “Take it private,” one of the +things it does is set the user capabilities for the “nobody” and +“anonymous” user categories to blank, so that users who haven’t logged +in can’t even see your project’s home page, and the option to log in as +“anonymous” isn’t even offered. Until you log in with a user name, all +you see is the repository’s skin and those few UI elements that work +without any user capability checks at all, such as the “Login” link. + +Beware: Fossil does not reassign the capabilities these users had to +other users or to the “reader” or “developer” user category! All users +except those with Setup capability will lose all capabilities they +inherited from “nobody” and “anonymous” categories. Setup is the [lone +exception][apsu]. + +If you will have non-Setup users in your private repo, you should parcel +out some subset of the capability set the “nobody” and “anonymous” +categories had to other categories or to individual users first. + + +## Reading vs. Cloning + +Fossil has two capabilities that are often confused: +[**Read**](./ref.html#o) and [**Clone**](./ref.html#g). + +The **Read** capability has nothing to do with reading data from a local +repository, because [caps affect Fossil’s web interfaces +only](#webonly). Once you’ve cloned a remote repository to your local +machine, you can do any reading you want on that repository irrespective +of whether your local user within that repo has Read capability. +The repo clone is completely under your user’s power at that point, +affected only by OS file permissions and such. If you need to prevent +that, you want to deny **Clone** capability instead. + +Withholding the **Read** capability has a different effect: it +prevents a web client from viewing [embedded documentation][edoc], +using [the file browser](/help?name=/dir), +exploring the [history](/help?name=/timeline) of check-ins, +and pulling file content via the [`/artifact`](/help?name=/artifact), +[`/file`](/help?name=/file), and [`/raw`](/help?name=/raw) URLs. +It is common to withhold **Read** capability from low-status visitors +on private or semi-private repos to prevent them from pulling individual +elements of the repo over the web one at a time, as someone may do when +denied the bulk **Clone** capability. + +[edoc]: ../embeddeddoc.wiki + + +## Default User Name + +By default, Fossil assumes your OS user account name is the same as the +one you use in any Fossil repository. It is the [default for a new +repository](#new), though you can override this with [the `--admin-user` +option][auo]. Fossil has other ways of overriding this in other contexts +such as the `name@` syntax in clone URLs. + +It’s simplest to stick with the default; a mismatch can cause problems. +For example, if you clone someone else’s repo anonymously, turn off +autosync, and make check-ins to that repository, they will be assigned +to your OS user name by default. If you later get a login on the remote +repository under a different name and sync your repo with it, your +earlier “private” check-ins will get synced to the remote under your OS +user name! + +When such problems occur, you can amend the check-in to hide the +incorrect name from Fossil reports, but the original values remain in +the repository [forever][shun]. It is [difficult enough][fos] to fix +such problems automatically during sync that we are unlikely to ever do +so. + +[auo]: /help?name=new +[fos]: ./impl.md#filter +[shun]: ../shunning.wiki + + + +## Cloning the User Table + +When cloning over HTTP, the initial user table in the local clone is set +to its “[new state:](#new)” only one user with Setup capability, named +after either your OS user account, per the default above, or after the +user given in the clone URL. + +There is one exception: if you clone as a named Setup user, you get a +complete copy of the user information. This restriction keeps the user +table private except for the only user allowed to make absolutely +complete clones of a remote repo, such as for failover or backup +purposes. Every other user’s clone is missing this and a few other +items, either for information security or PII privacy reasons. + +When cloning with file system paths, `file://` URLs, or over SSH, you +get a complete clone, including the parent repo’s complete user table. + +All of the above applies to [login groups][lg] as well. + + +## Caps Affect Web Interfaces Only + +Fossil’s user capability system only affects accesses over `http[s]://` +URLs. This includes clone, sync/push/pull, the [UI pages][wp], and [the +JSON API][japi]. For everything else, the user caps aren’t consulted at +all. + +The only checks made when working directly with a local repository are +the operating system’s file system permissions. This should strike you +as sensible, since if you have read access to the repository file, you +can do anything you want to that repo DB including giving your user’s +record the [**Setup**][s] capability, after which Fossil’s user +capability system is effectively bypassed. (Or, create another Setup +user, with the same end effect.) If you’re objecting that you need +*write* access to the DB file to achieve this, realize that you can copy +a read-only file to another location, giving yourself write access to +it. + +This is why the `fossil ui` command +gives you Setup permissions within Fossil UI: it can’t usefully prevent +you from doing anything through the UI since only the local file system +permissions actually matter, and you can’t start `fossil ui` without +having at least read access to that file. + +What may be more surprising to you is that this is also true when +working on a *clone* done over a local file path, except that there are +then two sets of file system permission checks: once to modify the +working check-out’s repo clone DB file, then again on [sync][sync] with +the parent DB file. The Fossil capability checks are effectively +defeated because your user has [**Setup**][s] capability on both sides +of the sync. Be aware that those file checks do still matter, however: +Fossil requires write access to a repo DB while cloning from it, so you +can’t clone from a read-only repo DB file over a local file path. + +Even more surprising to you may be the fact that user caps do not affect +cloning and syncing over SSH! (Not unless you go [out of your way][sshfc] +to patch around it, at any rate.) When you make a change to such a +repository, the stock Fossil behavior is that the change first goes to the +local repo clone where file system +permissions are all that matter, but then upon sync, the situation is +effectively the same as when the parent repo is on the local file +system. The reason behind this is that if you can log into the remote +system over SSH and that user has the necessary file system permissions +on that remote repo DB file to allow clone and sync operations, then +we’re back in the same situation as with local files: there’s no point +trying to enforce the Fossil user capabilities when you can just modify +the remote DB directly, so the operation proceeds unimpeded by any user +capability settings on the remote repo. + +Where this gets confusing is that *all* Fossil syncs are done over the +HTTP protocol, including those done over `file://` and `ssh://` URLs, +not just those done over `http[s]://` URLs: + +* For `ssh://` URLs, Fossil pipes the HTTP conversation through a + local SSH client to a remote instance of Fossil running the + [`test-http`](/help?name=test-http) command to receive the tunneled + HTTP connection. [This interface is intentionally permissionless][sxycap]. + +* For `file://` URLs — as opposed to plain local file paths — + the “sending” Fossil instance writes its side of + the HTTP conversation out to a temporary file in the same directory + as the local repo clone and then calls itself on the “receiving” + repository to read that same HTTP transcript file back in to apply + those changes to that repository. Presumably Fossil does this + instead of using a pipe to ease portability to Windows. + +Despite use of HTTP for these URL types, the fact remains that +checks for capabilities like [**Read**][o] and [**Write**][i] within the +HTTP conversation between two Fossil instances only have a useful effect +when done over an `http[s]://` URL. + +[sshfc]: ../server/any/http-over-ssh.md +[sxycap]: /file?ci=ec5efceb8aac6cb4&name=src/main.c&ln=2748-2752 + + +## Public Pages + +In Admin → Access, there is an option for giving a list of [globs][glob] +to name URLs which get treated as if the visitor had [the default cap +set](#defcap). For example, you could take the [**Read**][o] capability +away from the “nobody” user category, who has it by default, to prevent +users without logins from pulling down your repository contents one +artifact at a time, yet give those users the ability to read the project +documentation by setting the glob to match your [embedded +documentation][edoc]’s URL root. + + +## Default User Capability Set + +In Admin → Access, you can define a default user capability set, which +is used as: + +1. the default caps for users newly created by an Admin or Setup user +2. the default caps for self-registered users, an option in that same UI +3. the effective caps for URIs considered [public pages](#pubpg) + +This defaults to [**Reader**][u]. + + + +
+ +[ref]: ./ref.html + +[a]: ./ref.html#a +[b]: ./ref.html#b +[c]: ./ref.html#c +[d]: ./ref.html#d +[e]: ./ref.html#e +[f]: ./ref.html#f +[g]: ./ref.html#g +[h]: ./ref.html#h +[i]: ./ref.html#i +[j]: ./ref.html#j +[k]: ./ref.html#k +[l]: ./ref.html#l +[m]: ./ref.html#m +[n]: ./ref.html#n +[o]: ./ref.html#o +[p]: ./ref.html#p +[q]: ./ref.html#q +[r]: ./ref.html#r +[s]: ./ref.html#s +[t]: ./ref.html#t +[u]: ./ref.html#u +[v]: ./ref.html#v +[w]: ./ref.html#w +[x]: ./ref.html#x +[y]: ./ref.html#y +[z]: ./ref.html#z + +[2]: ./ref.html#2 +[3]: ./ref.html#3 +[4]: ./ref.html#4 +[5]: ./ref.html#5 +[6]: ./ref.html#6 +[7]: ./ref.html#7 + +[glob]: https://en.wikipedia.org/wiki/Glob_(programming) +[japi]: https://docs.google.com/document/d/1fXViveNhDbiXgCuE7QDXQOKeFzf2qNUkBEgiUvoqFN4/view#heading=h.6k0k5plm18p1 +[sp]: ../sync.wiki +[sync]: /help?name=sync +[wp]: /help#webpages diff --git a/www/caps/login-groups.md b/www/caps/login-groups.md new file mode 100644 index 0000000..7cca2d7 --- /dev/null +++ b/www/caps/login-groups.md @@ -0,0 +1,128 @@ +# Login Groups + +The Admin → Login-Groups UI feature and its corresponding [`login-group` +command][lg] solve a common problem with Fossil: you’ve created multiple +repositories that some subset of users need access to, and you +don’t want to redundantly administer the user credentials for each +repository. + + +## Restrictions + +This feature ties changes to the “`user`” table in one repo to that in +one or more other repos, keyed by the user’s name. The interactions are +non-obvious because although the goal is for the end result to “just +work,” there are practical security and administration matters that +complicate things: + +1. Login group handling only works between joined repositories for the + subset of users with the same name. + +1. If you’re logged in on one repo in a group, any other repo in that + group that has a matching user record will accept your valid login. + +1. When you set up a login group between two repos, the user tables + aren’t merged, so even though you may have users that appear in + both, they will retain their initial passwords, credentials, and so + forth. + +1. The same is true after the login group is created: changes you make + to the user table in one repo in the group only propagate to the + other repos in the group when you check the “Apply changes to all + repositories“ box in the “Scope” section of the user edit screen. + Otherwise, user changes remain local to the repo you made them on. + +1. Login groups only affect [the HTTP interfaces][wo]. Contrast things + like `ssh://` clones, where unless you [go out of your way][sh] to + force them to run over one of the HTTP interfaces that pays + attention to Fossil’s RBAC system, login groups aren’t consulted. + + +## Interactions + +These restrictions combine in subtle and interesting ways. Examples: + +* **#1 and #2**: If you are logged into repo C as “charlie” and then + try to visit joined repo A where “charlie” doesn’t exist, your valid + login on C won’t get you into A. + +* **#2 and #3**: If “alice” exists in both of these same repos, + logging in on A gets her into C, but if she has different user + capabilities on each from the time before the two repos joined the + login group, her caps on A don’t apply to C, nor vice versa. + + Let us say F is a forum-only repo, and W is a wiki-only repo, and + that Alice has forum-posting rights on F and wiki-editing rights on + W. If both repos are joined by a login group, Alice can log in on F + and then access W without logging in on it separately, but she + cannot then post a forum message on W even though she could on F. + +* **#3 and #4**: If you change the caps for user “alice” on one repo + in a group and tell Fossil to apply the changes to all repos in the + group, the new caps will *overwrite* those on the other repos, not + merge with them. + + To extend the practical example from the prior point, let us say you + wish to grant Alice the “write unversioned” capability on both F and + W. If you check that single user cap box on F plus the “apply to + all” option, then “Apply Changes,” she will end up with forum + + unversioned caps on repo W, losing her wiki-editing caps in the + process. + + If you want user caps to differ on each repo, you must administer + them separately even if there is a common subset of caps between all + repos in the group for that user. Remember: selecting the “apply to + all” box calls for an overwrite operation, not a merge. + +* **#4 and #1**: If you make a change to an existing user “bob” in + repo B and select the “apply to all” option, it will only affect + other repos in the group that have a user “bob” configured. + + But, if you are instead creating user “bob” for the first time and + select that option, that user *will* be created in all repos. The + same is true of user deletion: that destructive action will + propagate through the group if you request it. + +* **#5 and #1**: If you have a user “daisy” on both repos A and B in a + login group, logging in over the web to A doesn’t let you push + changes into B over SSH. Without the workaround linked above, SSH + only pays attention to the operating system’s user authentication + system, not Fossil’s. + + Inversely, if Daisy successfully logs in over SSH to repo B, she + gains no access to any of the other repos in that group. She needs + at least one valid login over HTTP to one of the group’s repos. + + +## Discussion + +The end result of all of this is that you can have a subset of users +with credentials only on repo A, a different subset only on B, and a +third subset common to both. The only thing selecting which case applies +is restriction #1 above. + +Login groups have names. A repo can be in only one of these named login +groups at a time. + +Trust in login groups is transitive within a single server. Consider +this sequence: + + $ cd /path/to/A/checkout + $ fossil login-group join --name G ~/museum/B.fossil + $ cd /path/to/C/checkout + $ fossil login-group join ~/museum/B.fossil + +That creates login group G joining repo A to B, then joins C to B. +Although we didn’t explicitly tie C to A, a successful login on C gets +you into both A and B, within the restrictions set out above. + +Changes are transitive in the same way, provided you check that “apply +to all” box on the user edit screen. + +[lg]: /help/login-group +[sh]: ../server/any/http-over-ssh.md +[wo]: ./index.md#webonly + +----- + +*[Back to Administering User Capabilities](./)* diff --git a/www/caps/ref.html b/www/caps/ref.html new file mode 100644 index 0000000..9c218ff --- /dev/null +++ b/www/caps/ref.html @@ -0,0 +1,454 @@ +
+ + + +

Here we document each currently-defined user capability character in +more detail than the brief summary on the “key” page in the Fossil user editor. Each +row gives the capability letter used in the Fossil user editor followed +by the C code’s name for that cap within the FossilUserPerms +object, so you can use this reference both from the UI down and from the +C code up.

+ +

The mnemonics +given here vary from obviously-correct to post facto +rationalizations to the outright fanciful. To some extent, this is unavoidable.

+ + +

Reference

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
?NameDescription
aAdmin + Admin users have all of the capabilities below except for + setup, Private, and WrUnver. + See Admin vs. Setup for a more + nuanced discussion. Mnemonic: administrate. +
bAttach + Add attachments to wiki articles, technotes, or tickets. + Mnemonics: bind, button, bond, or bolt. + See also: B. +
cApndTkt + Append comments to existing tickets. Mnemonic: comment. +
dn/a + Legacy capability letter from Fossil's forebear CVSTrac, which has no useful + meaning in Fossil due to the nature of its durable Merkle tree design. + We recommend that you remove it in case we + ever reuse this letter for another purpose. See this + post for details. +
eRdAddr + View personal + identifying information (PII) about other users such as email + addresses. Mnemonics: show email addresses; or + Europe, home of GDPR. +
fNewWiki + Create new wiki articles. Mnemonic: fast, English + translation of the Hawaiian word wiki. +
gClone + Clone the repository. Note that this is distinct from check-out capability, o; and that upon cloning + not just files, but also tickets, wikis, technotes and forum posts + are tranferred. Mnemonic: get. +
hHyperlink + Get hyperlinks in generated HTML which link you to other parts of + the repository. This capability exists so we can deny it to the + “nobody” category, to prevent bots from + wandering around aimlessly in the site’s hyperlink web, chewing up server resources to little + good purpose. Mnemonic: hyperlink. +
iWrite + Check changes into the repository. Note that a lack of this + capability does not prevent you from checking changes into your + local clone, only from syncing those changes up to the parent + repo, and then only over HTTP. + Also note that not just files, but also tickets, wikis, technotes + and forum posts will be accepted from clones upon syncronization. + Granting this capability also grants o (Read) Mnemonics: + input, check in changes. +
jRdWiki + View wiki articles. Mnemonic: injest page content. (All + right, you critics, you do better, then.) +
kWrWiki + Edit wiki articles. Granting this capability also grants RdWiki and ApndWiki, + but it does not grant NewWiki! + Mnemonic: kontribute. +
lModWiki + Moderate wiki article appends. Appends do not get + saved permanently to the receiving repo’s block chain until Setup or someone with this cap approves it. + Mnemonic: allow. +
mApndWiki + Append content to existing wiki articles. Mnemonic: amend + wiki +
nNewTkt + File new tickets. Mnemonic: new ticket. +
oRead + Read content and history of files from a remote Fossil instance over + HTTP. See Reading vs. + Cloning. Mnemonic: check out remote repo contents. +
pPassword + Change one’s own password. Mnemonic: password. +
qModTkt + Moderate tickets: delete comments appended to tickets. Mnemonic: + quash noise commentary. +
rRdTkt + View existing tickets. Mnemonic: read tickets. +
sSetup + The all-powerful Setup user. + Mnemonics: setup or superuser. +
tTktFmt + Create new ticket report formats. Note that although this allows + the user to provide SQL code to be run in the server’s context, + and this capability is given to the untrusted “anonymous” user + category by default, this is a safe capability to give to users + because it is internally restricted to read-only queries on the + tickets table only. (This restriction is done with an SQLite + authorization hook, not by any method so weak as SQL text + filtering.) Mnemonic: new ticket report. +
un/a + Inherit all capabilities of the “reader” user category; does not + have a dedicated flag internally within Fossil. Mnemonic: + user +
vn/a + Inherit all capabilities of the “developer” user category; does + not have a dedicated flag internally within Fossil. Mnemonic: + developer. +
wWrTkt + Edit existing tickets. Granting this capability also grants RdTkt, ApndTkt, and + NewTkt. Mnemonic: write to ticket. +
xPrivate + Push or pull private branches. + Mnemonic: exclusivity; “x” connotes unknown material in + many Western languages due to its traditional + use in mathematics. +
yWrUnver + Push unversioned content. Mnemonic: + yield, sense + 4: “hand over.” +
zZip + Pull archives of particular repository versions via /zip, /tarball, and /sqlar URLs. This is an + expensive capability to grant, because creating such archives can + put a large load on a Fossil server which + you may then need to manage. + Mnemonic: zip file download. +
2RdForum + Read forum posts by other users. + Mnemonic: from thee 2 me. +
3WrForum + Create new forum threads, reply to threads created by others, and + edit one’s own posts. New posts are held for moderation and do + not appear in repo clones or syncs. Granting this capability also + grants RdForum. Mnemonic: post for + 3 audiences: me, the mods, and the Man. +
4WrTForum + Extends WrForum, bypassing the moderation + and sync restrictions. Mnemonic: post 4 immediate release. +
5ModForum + Moderate forum posts. + Granting this capability also grants WrTForum and RdForum, + so a user with this cap never has to moderate their own posts. + Mnemonic: “May I have 5 seconds of your time, honored + Gatekeeper?” +
6AdminForum + Users with this capability see a checkbox on unmoderated forum + posts labeled “Trust user X so that future posts by user X do not + require moderation.” Checking that box and then clicking the + moderator-only “Approve” button on that post grants WrTForum capability to that post’s author. + There is currently no UI for a user with this cap to + revoke trust from a user once it is granted; only Admin and Setup can + currently revoke granted caps. Granting this capability also + grants ModForum and those it in turn + grants. Mnemonic: “I’m 6 [sick] of hitting Approve on your + posts!” +
7EmailAlert + User can sign up for email alerts. + Mnemonic: Seven can + wait, I’ve got email to read now. +
AAnnounce + Send email announcements to users signed up to + receive them. Mnemonic: announce. +
BAttach to Forum + Add attachments to forum posts. See also: b. +
CChat + Allow access to the /chat room. +
DDebug + Enable debugging features. Mnemonic: debug. +
LIs-logged-in + This is not a real capability, but is used in certain capability + checks, e.g. via capexpr. It + resolves to true if the current user is logged in. + Mnemonic: Logged in. +
+ +
+ + diff --git a/www/cgi.wiki b/www/cgi.wiki new file mode 100644 index 0000000..6b9c375 --- /dev/null +++ b/www/cgi.wiki @@ -0,0 +1,210 @@ +CGI Script Configuration Options + +

Summary

+ +It is not necessary to have a central server in order to use Fossil. +But a central server can help a project run more smoothly by giving developers +a common point of rendezvous for syncing, and by providing a web-based +portal where developers and non-developers alike can learn about the +project and its current state. + +Setting up a server using Fossil is easy. +A [./server/|separate document] talks about all of the many different methods for +setting up a Fossil server, one of which is [./server/any/cgi.md | as a CGI +script]. CGI is the technique that the three +[./selfhost.wiki|self-hosting Fossil repositories] all use. + +Setting up a Fossil server using CGI is mostly about writing a short +script (usually just 2 lines) in the cgi-bin folder of an ordinary +web-server. But there are a lot of extra options that can be added +to this script, to customize the configuration. This article describes +those options. + +

CGI Script Options

+ +The CGI script used to launch a Fossil server will usually look something +like this: + + +#!/usr/bin/fossil +repository: /home/www/fossils/myproject.fossil + + +Of course, pathnames will likely be different. The first line +(the "[wikipedia:/wiki/Shebang_(Unix)|shebang]") +always gives the name of the Fossil executable. Subsequent lines are of +the form "property: argument ...". +The remainder of this document describes the available properties and +their arguments. + +
+ +

repository: PATH

+ +This property defines the Fossil repository that the server will use. +Every Fossil CGI requires either this property or the +[#directory|directory:] property (but not both). +Many Fossil CGI scripts have this one property and no other. + + +

directory: PATH

+ +The PATH is the name of a directory that contains one or more Fossil +repository files having the suffix ".fossil". If this property is +used instead of [#repository|repository:], then the Fossil +server is able to serve all of the repositories in the directory. The +specific repository used is selected by a prefix on the PATH_INFO. See +the notes for the [#repolist|repolist] option regarding name +collisions between subdirectories and repository files. + + +

notfound: URL

+ +If the [#directory|directory:] option is used and if the PATH_INFO +of the HTTP request does not correspond to any Fossil repository, then +the request redirects to URL. + + +

repolist

+ +This is a Boolean property. +If it is present, and if the [#directory:|directory:] option is used, +and if the PATH_INFO string is empty, then Fossil will show a list +of available Fossil repositories. + +The "skin" of the reply is determined by the first +repository in the list that has a non-zero +[/help/repolist-skin|repolist-skin] setting. + +If no repository has such a non-zero repolist-skin setting, then +the repository list is generic HTML without any decoration, with +the page title taken from the FOSSIL_REPOLIST_TITLE +environment variable. The variable can be defined in the CGI +control file using the [#setenv|setenv:] statement. + +The "Project Description" and "Login-Group" columns on the repolist page +are optional. They are hidden by default. Show them by +etting the FOSSIL_REPOLIST_SHOW environment variable to +a string that contains substrings "description" and/or "login-group". + +The repolist-generated page recurses into subdirectories and will list +all *.fossil files found, with the following exceptions: + + * Filenames starting with a period are treated as "hidden" and skipped. + + * Subdirectory names which match the base name of a fossil file in + the same directory are listed in the resulting page but are not + hyperlinked because the links would be ambiguous and the + repositories in the subdirectories would be unreachable to + clients. For example, any repositories under subdirectory + XYZ are unreachable if XYZ.fossil exists in + the same directory as XYZ, noting that this particular + name check is case-insensitive. The entries for such + repositories are clearly marked in the repolist page's output to + make the user aware of the problem. To make them accessible, + move them into a directory which does not share a base name with + a repository file. + + +

localauth

+ +This is a Boolean property. +If it is present, [./caps/ref.html#s | setup capability] +is granted to any HTTP request that +comes in over a loopback interface, such as 127.0.0.1. + + +

skin: NAME

+ +If NAME is the name of one of the built-in skins supported by Fossil, +then this option causes Fossil to display using that built-in skin, +and to ignore any custom skin that might be configured in the repository +itself. + +So, if you wanted to set up a server for a single Fossil project, but +also give users the option to use several of the different built-in +skins, you could create multiple CGI scripts, each with a different +"skin:" property, but all pointing to the same repository:. +Then users can select which skin to use by using the appropriate CGI. + + +

files: GLOBLIST

+ +The GLOBLIST argument is a comma-separate list of "globs" that specify +filenames. In [#directory|directory: mode], if the PATH_INFO +does not identify any Fossil repository, but it does refer some other +file in the directory, and that filename matches one of the glob patterns +in the GLOBLIST, then the file is returned as static content. + + +

setenv: NAME VALUE

+ +This parameter causes additional environment variable NAME to have VALUE. +This parameter can be repeated as many times as necessary. + + +

HOME: PATH

+ +This parameter is a short-hand for "setenv HOME PATH". + + +

cgi-debug: FILE

+ +Cause CGI-related debugging information to be appended in FILE. Use +this to help debug CGI problems. + + +

errorlog: FILENAME

+ +This setting causes the server to log any errors in FILENAME. +It is ok for multiple Fossil CGIs to share the same error log. + +Setting up an error log for Fossil servers is not required, but it +is recommended. + + +

timeout: N

+ +This property changes the timeout on each CGI request to N seconds. +If N is zero, then there is no timeout. If this property is omitted, +then the default timeout is 300 seconds (5 minutes). + + +

extroot: PATH

+ +This property defines the DOCUMENT_ROOT for the +[./serverext.wiki|CGI Server Extensions]. If this property +is present, then CGI Server Extensions are enabled. When this +property is omitted, CGI Server Extensions are disabled. + +A cascade of CGI invocations can occur here. Fossil itself is +started as CGI, then Fossil can turn around and invoke a sub-CGI +extension. The sub-CGI extension outputs reply text, when Fossil +then (optionally) augments with its own header and footer and returns +to the original requestor. The property controls the DOCUMENT_ROOT +of the sub-CGI. + + +

redirect: REPO URL

+ +Extract the "name" query parameter and search REPO for a check-in or +ticket that matches the value of "name", then redirect to URL. There +can be multiple "redirect:" lines that are processed in order. If the +repo name is "*", then an unconditional redirect to URL is taken. + + +

jsmode: VALUE

+ +Specifies the delivery mode for JavaScript files. See "[/help/http | +http --jsmode]" for the allowed values and their meanings. + + + + +This parameter causes the contents of the given file to override the +site's mainmenu configuration setting, in much the same way +that the skin setting overrides the skin. This can be used to +apply a common main menu to a number of sites, and centrally maintain +it, without having to copy its contents into each site. Note, however, +that the contents of this setting are not stored in the repository and +will not be cloned along with the repository. diff --git a/www/changes.wiki b/www/changes.wiki new file mode 100644 index 0000000..41cb3e3 --- /dev/null +++ b/www/changes.wiki @@ -0,0 +1,2297 @@ +Change Log + +

Changes for version 2.29 (pending)

    +
  1. Enhance the [/help/www/ci_edit|/ci_edit] web page to show more + context of the check-in being edited. +
  2. Honor the NO_COLOR environment variable in the + "[/help/system|fossil sys ls]" command. +
  3. On the [/help/www/info|/info webpage] (and similar) put a + "copy" button before the date. +
  4. Add the ".m4a" and ".otf" mimetypes. +
  5. When doing a "fossil update", if a file under management needs to + overwrite an unmanaged file, display the name of the backup that + is made of the unmanaged file, and use file_delete() to delete + the unmanaged file, even if that unmanaged file is read-only. +
  6. Improve the default prompts used by the + "[/help/sqlite3|fossil sql]" command. +
  7. The captcha now uses light-gray boxes as the background, instead of + spaces, to work around width inconsistencies in some fonts. +
  8. Forum post editing and replying was overhauled with a new UI. Clients + with JavaScript disabled will still see the older forms. +
  9. Forum posts may now have attachments if their poster has the new "B" + capability. +
  10. Add the "[/help/attachment-size-limit|attachment-size-limit]" setting + to limit the size of file attachments to wiki pages, tech notes, + tickets, and forum posts. +
  11. Add the "[/help/path-to-tag|path-to-tag]" setting, which if set to + something like "release" provides a link named "path-to-release" on + /info pages for checkins. Clicking that link shows the path to the + first release containing that check-in. +
  12. Addded several HTML5 tags deemed harmless to the MD/wiki allowlists. +
  13. Fix various pages on deployments where a custom skin emits + its own BODY tag without the expected CSS classes. +
  14. When behaving as an HTTP client, Fossil now always uses HTTP/1.1 and + it understands the "chunked" transfer encoding. +
  15. The "diff" command (in all its variations) when invoked with --tk + now accepts an extra argument of the form "HOST:DIRECTORY" or + "USER@HOST:DIRECTORY" to specify a directory on a remote host in which + the diff is to be taken. The diff is run using SSH, the results are + captured, then displayed locally using the Tcl/Tk diff GUI. +
+ +

Changes for version 2.28 (2026-03-11)

    +
  1. Improvements to [./antibot.wiki|anti-robot defenses]:
      +
    1. The default configuration now allows robots to download any tarball + or similar, to better support of automated build systems. +
    2. New special tag "zipX" for the [/help/robot-restrict|robot-restrict] + setting blocks robot access to tarballs, but with exceptions to support + automated build systems. +
    3. Tags of the form "ext/PATH" in the robot-restrict setting block access + by robots to specific [./serverext.wiki|CGI extension] at PATH. +
    4. Enhancements to the default value for the + [/help/robot-restrict|robot-restrict setting]. +
    +
  2. A drop-down menu of recent branches is now possible for the submenu, and + is used in the [/dir?ci=trunk|code browser]. +
  3. Simplified access to tarballs and ZIPs:
      +
    1. When in the [/dir?ci=trunk|code browser] at the top-level, + a new "Download" submenu option is available to take the + user to a page where he can download a tarball or ZIP archive. +
    2. New [/help/www/download|/download page] is available. When + configured using the new + [/help/suggested-downloads|suggested-downloads setting], a + link to [/download] named "Tarballs and ZIPs" appears in the + [/sitemap] and thus on the hamburger menu. +
    3. The robot-resistant [/help/www/rchvdwnld|/rchvdwnld page] can be + used to provide human access to on-demand generated ZIP archives + and tarballs. This page is used to help implement the two previous + subitems. +
    4. The filenames for tarballs and ZIPs are now standardized to + include a timestamp and a hash prefix. +
    5. New "[/help/get|fossil get]" command downloads and unpacks a specific + check-in without having to clone the repository. +
    +
  4. Web-based [/help/www/timeline|timeline] enhancements:
      +
    1. A new "Simple" view is available. This is compromise between "Verbose" + and "Compact" that shows only the check-in hash rather than the full + detail section. There is an ellipsis that one can click on to see the + full detail text. +
    2. The artifact hash in the detail section of each timeline entry is now + emphasized (controlled by CSS) to make it easier to locate amid all + the other text and links. +
    3. When clicking on the ellipsis in "Compact" or "Simple" views, the + ellipsis is replaced by a left arrow ("←") which can be clicked to + hide the extra detail again. +
    4. New setting [/help/timeline-mark-leaves|timeline-mark-leaves] controls + whether or not leaf check-ins are marked in the timeline. +
    5. "No-graph" timelines (using the "ng" query parameter) now show + branch colors and bare check-in circles on the left. The check-in + circles appear, but no lines connecting them. + ([/timeline?ng|example]). +
    +
  5. Labels in Markdown now have IDs generated using the GitHub "slugify" + algorithm. +
  6. The [/help/timeline|timeline command] is enhanced with the new options + "-u|--for-user" to filter by user, and "-r" to display + entries in chronological order. +
  7. The [/help/open|open command]'s new "--reopen REPOFILE" flag + can be used to fix a checkout after moving its repository file. +
  8. Update internal Unicode character tables, used in regular expression + handling, from version 13 to 17. +
  9. Add a zoom-message option to [/help/www/chat|/chat] to better support + pikchr diagrams. +
  10. The new [/help/system|fossil system] (often abbreviated as just "fossil sys") + provides a few common unix-like commands that are missing on some + platforms. This is a convenience for Unix-based developers who occasionally + need to work on ill-provisioned alternative platforms. The set of commands + currently offered is small, but is expected to grow in future releases. +
  11. The [/help/www/help|/help webpage] now accepts queries of the form + "/help/CMD" to show help for the command or setting CMD and + "/help/www/PAGE" to show help for webpage PAGE. +
  12. Add the -t and -T options to the [/help/praise|fossil praise] command. +
  13. The [/help/clone|fossil clone] command adds the --ipv6 option. +
  14. Add "-s" and "--stop" as aliases for the "--stop-on-error" option in + the [/help/all|fossil all] command. +
  15. Add the "-h|--hash" option to the [/help/whatis|fossil whatis] command. +
+ +

Changes for version 2.27 (2025-09-30)

    +
  1. Close a potential Denial-of-Service attack against any public-facing Fossil + server involving exponential behavior in Fossil's regexp implementation. +
  2. Fix a SQL injection on the [/help/www/file|/file page]. Thanks to + additional defenses built into Fossil, as well as good luck, this injection + is not exploitable for either data exfiltration or privilege escalation. The + only possible result of invoking the injection is a harmless SQL syntax error. +
  3. Strengthen robot defenses to help prevent public-facing servers from being + overwhelmed by the latest generation of AI spiders. +
      +
    1. New javascript captcha used to restrict access by user "nobody" to pages + listed in the [/help/robot-restrict|robot-restrict setting]. +
    2. The [/help/robot-exception|robot-exception setting] is available to allow + access to pages that match a regular expression. Use this, for example, to + allow curl scripts and similar to download release tarballs. +
    3. Require at least an anonymous login to access the /blame page and similar. +
    +
  4. [/help/www/timeline|Timeline] enhancements: +
      +
    1. The chng= query parameter on the [/help/www/timeline|timeline page] + so that it works with other query parameters like p=, d=, from=, and to=. +
    2. Always include nodes identify by sel1= and sel2= in the /timeline display. +
    3. Improved title when p= and d= are different. +
    +
  5. Enable the --editor option on the [/help/amend|fossil amend] command. +
  6. When walking the filesystem looking for Fossil repositories, avoid descending + into directories named "/proc". +
  7. Reduce memory requirements for sending authenticated sync protocol + messages. +
  8. Show numstat-style change statistics in the /info and /ckout pages. +
  9. Add the [/help/stash | stash rename] subcommand. +
  10. Add the "-h" option to the "[/help/ls|ls]" command to display + file hashes for a specific check-in when in verbose mode. +
+ +

Changes for version 2.26 (2025-04-30)

    +
  1. Enhancements to [/help/diff|fossil diff] and similar: +
      +
    1. The argument to the --from option can be a directory name, causing + Fossil to use files under that directory as the baseline for the diff. +
    2. For "gdiff", if no [/help/gdiff-command|gdiff-command setting] + is defined, Fossil tries to do a --tk diff if "tclsh" and "wish" + are available, or a --by diff if not. +
    3. The "Reload" button is added to --tk diffs, to bring the displayed + diff up to date with the latest changes on disk. +
    4. Add the "Hide diffs/Show diffs" toggle to web-UI diff pages that show + diffs of multiple files. +
    +
  2. Added the [/help/www/ckout|/ckout web page] to provide information + about pending changes in a working check-out +
  3. Enhancements to the [/help/ui|fossil ui] command: +
      +
    1. Defaults to using the new [/help/www/ckout|/ckout page] as its + start page. Or, if the new "--from PATH" option is present, the + default start page becomes "/ckout?exbase=PATH". +
    2. The new "--extpage FILENAME" option opens the named file as if it + where in a [./serverext.wiki|CGI extension]. Example usage: the + person editing this change log has + "fossil ui --extpage www/changes.wiki" running and hence can + press "Reload" on the web browser to view edits. +
    3. Accept both IPv4 and IPv6 connections on all platforms, including + Windows and OpenBSD. This also applies to the "fossil server" + command. +
    +
  4. Enhancements to [/help/merge|fossil merge]: +
      +
    1. Added the [/help/merge-info|fossil merge-info] command and + especially the --tk option to that command, to provide analysis + of the most recent merge or update operation. +
    2. When a merge conflict occurs, a new section is added to the conflict + text that shows Fossil's suggested resolution to the conflict. +
    +
  5. Enhancements to [/help/commit|fossil commit]: +
      +
    1. If Fossil sees potential formatting mistakes (ex: bad hyperlinks) + in the check-in comment, it will alert the developer and give + him or her the opportunity to edit the comment before continuing. + This feature is controllable by the + [/help/verify-comments|verify-comments setting]. +
    2. The new "--if-changes" option causes the commit to become + a quiet no-op if there are no pending changes. +
    3. Added the ability to sign check-ins with SSH keys. Artifacts signed + this way are ignored by all previous fossil versions, as if they + were plain-text file content instead of Fossil artifacts. +
    4. Issue a warning if a user tries to commit on a check-in where the + branch has been changed. +
    5. The interactive checkin comment prompt shows the formatting rules + set for that repository. +
    6. Add the "--editor" option. +
    +
  6. Deprecate the --comfmtflags and --comment-format global options and + no longer list them in the built-in help, but keep them working for + backwards compatibility. + Alternative TTY comment formatting can still be specified using the + [/help/comment-format|comment-format setting], if desired. The + default comment format is now called "canonical", not "legacy". +
  7. Enhancements to the [/help/www/timeline|/timeline page]: +
      +
    1. Added the "ml=" ("Merge-in List") query parameter that works + like "rl=" ("Related List") but adds "mionly" style related + check-ins instead of the full "rel" style. +
    2. For "tl=", "rl=", and "ml=", the order of the branches in the + graph now tries to match the order of the branches named in + the list. +
    3. The "ms=" ("Match Style") query parameter is honored for + "tl=", "rl=", and "ml=". +
    4. New query parameter "sl=BRANCHLIST" ("Sort List") strives to + put branches in the specified order in the graph. This + overrides any "tl=" or similar ordering. +
    5. In the various "from=","to=" query formats, if the one of the + end points is an ancestor of the other, then the "rel" modifier + omits check-ins that are not ancestors of the newer endpoint. +
    6. For "tl=" and similar query parameters, if the pattern contains + GLOB characters, then the matching style ("ms=") is set to GLOB + automatically and the "ms=" query parameter can be omitted. +
    7. Enhance the "ymd" query parameter so that when used like + "ymd=YYYYMMDD-YYYYMMDD" it shows all events in the range of + dates specified. +
    8. Accept the "Z" (Zulu-time) suffix on date arguments for the + "ymd" and "yw" query parameters. +
    9. The new "min" query parameter, when added to a from=,to= query, + collapses long runs of check-ins on the same branch into just + end-points. +
    10. The p= and d= parameters can now reference different check-ins, + in which case the timeline shows those check-ins that are both + ancestors of p= and descendants of d=. +
    11. The saturation and intensity of user-specified checkin and branch + background colors are automatically adjusted to keep the colors + compatible with the current skin, unless the + [/help/raw-bgcolor|raw-bgcolor setting] is turned on. +
    +
  8. The [/help/www/docfile|/docfile webpage] was added. It works like + /doc but keeps the title of markdown documents with the document rather + that moving it up to the page title. +
  9. Added the [/help/www/clusterlist|/clusterlist page] for analysis + and debugging +
  10. Added the "artifact_to_json(NAME)" SQL function that returns a JSON + decoding of the artifact described by NAME. +
  11. Improvements to the [/help/patch|fossil patch] command: +
      +
    1. Fix a bug in "fossil patch create" that causes + [/help/revert|fossil revert] operations that happened + on individualfiles after a [/help/merge|fossil merge] + to be omitted from the patch. +
    2. Added the [/help/patch|patch alias] command for managing + aliases for remote checkout names. +
    +
  12. Enhancements to on-line help and the [/help/help|fossil help] command: +
      +
    1. Add the ability to search the help text, either in the UI + (on the [/help/www/search|/search page]) or from the command-line + (using the "[/help/search|fossil search -h PATTERN]" command.) +
    2. Accepts an optional SUBCOMMAND argument following the + COMMAND argument and only shows results for the specified + subcommand, not the entire command. +
    3. The -u (--usage) option shows only the command-line syntax +
    4. The -o (--options) option shows only the command-line options +
    +
  13. Enhancements to the [./tickets.wiki|ticket system]: +
      +
    1. Added the ability to attach wiki pages to a ticket for extended + descriptions. +
    2. Added submenu to the 'View Ticket' page, to use it as + template for a new ticket. +
    3. Added button 'Submit and New' to create multiple tickets + in a row. +
    4. Link the version field in ticket view to a matching checkin or tag. +
    5. Show creation time in report and ticket view. +
    6. Show previous comments in edit ticket as reference. +
    +
  14. Added the "hash" query parameter to the + [/help/www/whatis|/whatis webpage]. +
  15. Add a "user permissions changes" [/doc/trunk/www/alerts.md|subscription] + which alerts subscribers when an admin creates a new user or + when a user's permissions change. +
  16. If the FOSSIL_REPOLIST_SHOW environment variable exists and contains + the substring "description", then the project description for each repository + is shown on the repository list page. The login-group for each project is + now only shown if the FOSSIL_REPOLIST_SHOW environment variable exists and + contains the substring "login-group". ([./cgi.wiki#repolist|More information]) +
  17. The [/doc/trunk/www/th1.md|TH1 script language] is enhanced for improved + security: +
      +
    1. TH1 now makes a distinction between + [/doc/trunk/www/th1.md#taint|tainted and untainted string values]. + This makes it more difficult to write custom TH1 scripts that + contain XSS or SQL-injection bugs. The + [/help/vuln-report|vuln-report] setting was added to control + what Fossil does when it encounters a potential TH1 + security problem. +
    2. The "--th" option was removed from the [/help/pikchr|fossil pikchr] + command. +
    3. The "enable_htmlify" TH1 command was removed. +
    +
  18. Make [/help/www/chat|/chat] better-behaved during server outages, reducing + the frequency of reconnection attempts over time and providing feedback + to the user when the connection is down. +
  19. The [/help/www/sqlar|/sqlar] page does not work for users who are not logged + in, nor are links to that page displayed to users who are not logged in. Being + logged in as "anonymous" is sufficient to overcome this restriction, assuming + that "anonymous" can download tarballs and ZIP archives. +
  20. Many other minor fixes and additions. +
+ +

Changes for version 2.25 (2024-11-06)

+ + * The "[/help/ui|fossil ui /]" command now works even for repositories + that have non-ASCII filenames + * Add the [/help/tree|fossil tree] command. + * On case-insensitive filesystems, store files using the filesystem's + preferred case rather than the case typed in by the user. + * Change the name "fossil cherry-pick" command to "fossil cherrypick", + which is more familiar to Git users. Retain the legacy name for + compatibility. + * Add new query parameters to the [/help/www/timeline|/timeline page]: + d2=, p2=, and dp2=. + * Add options to the [/help/tag|fossil tag] command that will list tag values. + * Add the -b|--brief option to the [/help/status|fossil status] command. + * Add ability to upload unversioned files via the [/help/www/uvlist|/uvlist page]. + * Add history search to the [/help/www/chat|/chat page]. + * Add Unix socket support to the [/help/server|server command]. + * On Windows, use the root certificates managed by the operating system + (requires OpenSSL 3.2.0 or greater). + * Take into account zero-width and double-width unicode characters when + formatting the command-line timeline. + * Update the built-in SQLite to version 3.47.0. Precompiled binaries are + linked against OpenSSL 3.4.0. + * Numerous minor fixes and additions. + + +

Changes for version 2.24 (2024-04-23)

+ + * Apache change work-around → As part of a security fix, the Apache webserver + mod_cgi module has stopped relaying the Content-Length field of the HTTP + reply header from the CGI programs back to the client in cases where the + connection is to be closed and the client is able to read until end-of-file. + The HTTP and CGI specs allow for this, though it does seem rude. + Older versions of Fossil were depending on the Content-Length header field + being set. To work around the change to Apache, Fossil has + been enhanced to cope with a missing Content-Length in the reply header. See + [forum:/forumpost/12ac403fd29cfc89|forum thread 12ac403fd29cfc89]. + * [./customskin.md|Skin] enhancements: +
    +
  • Reworked the default skin to make everything more readable: larger + fonts, more whitespace, deeper indents to show hierarchy and to + offset command examples, etc. Adjusted colors slightly to bring + things into better accord with the WCAG accessibility guidelines. + This constitutes a breaking change for those with + custom skins; see [./customskin.md#version-2.24 | this section of + the docs] for migration advice. +
  • Add a new link added to the [/login] page that allows the user to + [/skins|select their preferred skin]. This preference is stored in + the [/fdscookie|fossil display_settings cookie]. +
  • The /setup_skin_admin page is simplified to let administrators easily + select one of the built-in skins as a default, or to specify a + custom skin. +
+ * If an "ssh:" sync fails in a way that suggests that the fossil executable + could not be found on the remote host, then retry after adding a PATH= + prefix to the command. This helps "ssh:" to "just work" when the server + is a Mac. + * Enhancements to the [/help/www/timeline|/timeline page]: +
    +
  • Add the x= query paramater +
  • Add the shortcut tl= and rl= query parameters +
  • Add support for from=,ft= and from=,bt= query parameter combinations +
  • Automatically highlight the endpoints for from=,to= queries. +
  • Add the to2=Z query parameter to augment from=X,to=Y so that the + path from X to Z is shown if Y cannot be found. +
+ * Moved the /museum/repo.fossil file referenced from the Dockerfile from + the ENTRYPOINT to the CMD part to allow use of --repolist mode. + * The [/uvlist] page now shows the hash algorithm used so that + viewers don't have to guess. The hash is shown in a fixed-width + font for a more visually pleasing display. + * If the [/help/autosync|autosync setting] contains keyword "all", + the automatic sync occurs against all defined remote repositories, not + just the default. + * Markdown formatter: improved handling of indented fenced code blocks + that contain blank lines. + * When doing a "[/help/add|fossil add]" on a system with case-insensitive + but case-preserving filenames (Mac and Windows) try to use the filename + case as it is known to the filesystem, not the case entered by the + user on the command-line. See + [forum:/forumpost/30d9c0d131610f53|forum thread 30d9c0d131610f53]. + * Fix problems with one-click unsubscribe on email notifications. + * Import the latest [/doc/trunk/www/pikchr.md|Pikchr] containing support + for "diamond" objects. + * Add ability to render committed Pikchr files to SVG via + /doc/…/foo.pikchr?popup URLs. + * Update Fossil's internal robot detection logic so that it correctly + identifies the new GoogleOther crawler as a robot. + +

Changes for version 2.23 (2023-11-01)

+ + * Add ability to "close" forum threads, such that unprivileged users + may no longer respond to them. Only administrators can close + threads or respond to them by default, and the + [/help/forum-close-policy|forum-close-policy setting] can be + used to add that capability to moderators. + * Add the [/help/all|fossil all whatis] command. + * The [/help/status|fossil status] command and relevant UI pages now + correctly report files which were both renamed and edited as such. + * Show default value of settings that have a default in + [/help/help|fossil help SETTING] output. + * On timeline graphs, show closed check-ins using an X in the middle of the + node circle or box. + * New options for email notification: Get email only for the first + post in each new thread, and/or posts that are in reply to my posts. + * Fix a regression bug introduced in version 2.22 that caused FTS5 searches + to fail for terms containing non-ASCII characters. + * Improved defense-in-depth against malicious attack: +
    +
  • When an attempted SQL injection attack is detected, return + HTTP result code 418, which can signal the web server to sanction + the attacking IP address. +
  • Better defense against cross-site request forgery (CSRF) + attacks. +
  • Improvements to static analysis of source code (the codecheck1.c + file in the source tree). +
+ * Enhance the [/help/www/dir|treeview file listings] + ([/dir?type=tree&ci=trunk|example]) by displaying file sizes + and adding the option to sort by file size. + * The [/help/fts-config|fossil fts-config] command now shows how much + repository space is used by the full-text index. + * Changing a setting to an empty string is now the same as deleting the + setting, in most cases. There are a few exceptions, indicated by the + keep-empty flag on the setting definition. + * The [/help/branch|fossil branch list] command can now filter branches + that have/have not been merged into the current branch. + * Improvements to interactions with remote repositories over SSH: +
    +
  • Print the text of the SSH command that is run to do remote interaction, + for full disclosure to the operator. +
  • Add a PATH= argument to the [/help/ui|fossil ui remote:/] and + [/help/patch|fossil patch push/pull remote:...] commands so that + they work when the "remote" machine is a Mac and the "fossil" + executable is in the $HOME/bin directory. +
+ * Update built-in libraries SQLite, ZLib, Pikchr to their latest versions. + * Documentation enhancements and typo fixes. + + +

Changes for version 2.22 (2023-05-31)

+ * Enhancements to the [/help/www/timeline|/timeline webpage]:
    +
  1. Add the ft=TAG query parameter which in combination with d=Y + shows all descendants of Y up to TAG +
  2. Enhance the s=PATTERN (search) query parameter so that forum post + text is also searched when the "vfx" query parameter is used +
  3. Fix the u= (user) query parameter so that it works with a= and b= +
  4. Add the oldestfirst query parameter to show the events in reverse order. + Useful in combination with y=f and vfs and perhaps also u= to show all + forum events in chronological order +
  5. For the p=X and bt=Y query parameter combination, if Y is a tag that + identifies multiple check-ins, search backwards in time for Y beginning + at X +
+ * Administrators can select to skip sending notifications about new forum + posts. + * If the value N is negative in "--context N" or "-c N" to the various diff + commands, then treat it as infinite and show the complete file content. + * The stock OCI container no longer includes BusyBox, thus no longer + needs to start as root to chroot that power away. That in turn + frees us from needing to build and install the container as root, + since it no longer has to create a private /dev tree + inside the jail for Fossil's use. + * Add support for the trigram tokenizer for FTS5 search to enable + searching in Chinese. + * Comment lines (starting with a '#') are now supported inside + [./settings.wiki#versionable|versioned settings]. + * Default permissions for anonymous users in new repositories are + changed to "hz". + * The [/help/status|fossil status] command now detects when a + file used to be a symlink and has been replaced by a regular file. + (It previously checked for the inverse case only.) + * The [/help/empty-dirs|empty-dirs setting] now reuses the same + parser as the *-glob settings instead of its prior idiosyncratic + parser, allowing quoted whitespace in patterns. + * Enhancements to the [/help/www/reports|/reports webpage]: +
    +
  1. The by-week, by-month, and by-year options now show an estimated + size of the current week, month, or year as a dashed box. +
  2. New sub-categories "Merge Check-ins" and "Non-Merge Check-ins". +
+ +

Changes for version 2.21 (2023-02-25)

+ * Users can request a password reset. This feature is disabled by default. + Use the new [/help/self-pw-reset|self-pw-reset property] to enable it. + New web pages [/help/www/resetpw|/resetpw] and + [/help/www/reqpwreset|/reqpwreset] added. + * Add the [/help/repack|fossil repack] command (together with + [/help/all|fossil all repack]) as a convenient way to optimize the + size of one or all of the repositories on a system. + * Add the ability to put text descriptions on ticket report formats. + * Upgrade the test-find-pivot command to the [/help/merge-base|merge-base command]. + * The [/help/www/chat|/chat page] can now embed fossil-rendered + views of wiki/markdown/pikchr file attachments with the caveat that such + embedding happens in an iframe and thus does not inherit styles and such + from the containing browser window. + * The [/help/all|fossil all remote] subcommand added to "fossil all". + * Passwords for remembered remote repositories are now stored as irreversible + hashes rather than obscured clear-text, for improved security. + * Add the "nossl" and "nocompress" options to CGI. + * Update search infrastructure from FTS4 to FTS5. + * Add the [/help/www/deltachain|/deltachain] page for debugging purposes. + * Writes to the database are disabled by default if the HTTP request + does not come from the same origin. This enhancement is a defense in depth + measure only; it does not address any known vulnerabilities. + * Improvements to automatic detection and mitigation of attacks from + malicious robots. + +

Changes for version 2.20 (2022-11-16)

+ * Added the [/help/chat-timeline-user|chat-timeline-user setting]. If + it is not an empty string, then any changes that would appear on the timeline + are announced in [./chat.md|the chat room]. + * The /unsubscribe page now requests confirmation. [./alerts.md|Email notifications] + now contain only an "Unsubscribe" link, and not a link to subscription management. + * Added the "[/help/branch|fossil branch lsh]" subcommand to list the + most recently modified branches. + * More elements of the /info page are now inside of an accordion. + * Replace the --dryrun flag with --dry-run in all + commands which still used the former name, for consistency. + * Rebuilt [/file/Dockerfile | the stock Dockerfile] to create a "from scratch" + Busybox based container image via an Alpine Linux intermediary + * Added [/doc/trunk/www/containers.md | a new document] describing how to + customize, use, and run that container. + * Added "by hour of day" report to [/reports?view=byhour|the /reports page]. + * Improved correctness, usability, and efficiency for the case + [/timeline?r=deltify-tkt-blobs|when values in a TICKET's column + tend to be long and volatile]. + * Fixed a bug [/info/ea5afad31f478396 | introduced in 2.17] that + prevented clone --unversioned from completing the + retrieval of UV files from the remote repo. While fixing that, enabled + UV tracing output with clone --unversioned --verbose, making it + consonant with uv sync --verbose. + +

Changes for version 2.19 (2022-07-21)

+ * On file listing pages, sort filenames using the "uintnocase" collating + sequence, so that filenames that contains embedded integers sort in + numeric order even if they contain a different number of digits. + (Example: "fossil_80_..." comes before "fossil_100.png" in the + [/dir?ci=92fd091703a28c07&name=skins/blitz|/skins/blitz] directory listing.) + * Enhancements to the graph layout algorithm design to improve readability + and promote better situational awareness. + * Performance enhancement for the + [./checkin_names.wiki#root|"root:BRANCHNAME" style of tag], + accomplished using a Common Table Expression in the underlying SQL. + * Sort tag listings (command line and webpage) by taking numbers into + consideration so as to cater for tags that follow semantic versioning. + * On the wiki listings, omit by default wiki pages that are associated with + check-ins and branches. + * Add the new "[/help/describe|fossil describe]" command. + * Markdown subsystem extended with [../src/markdown.md#ftnts|footnotes support]. + See corresponding [../test/markdown-test3.md|test cases], + [/wiki?name=branch/markdown-footnotes#il|known limitations] and + [forum:/forumthread/ee1f1597e46ec07a|discussion]. + * Add the new special name "start:BRANCH" to refer to the first check-in of + the branch. + * Support [/wiki?name=branch/generated-tkt-mimetype&p|generated "mimetype"] + columns in the TICKET and TICKETCHNG tables. + * Fix [/timeline?r=fix_remote_url_overwrite_with_proxy|remote-url-overwrite] + bug where remote-url is overwritten by the proxy setting during sync + operation. Also require explicit "system" proxy setting to use + "http_proxy" environment variable. + * Reimplemented the [/pikchrshow] app to use a WebAssembly build of + pikchr so that it can render pikchrs on the client instead of requiring + a server round-trip. + * Add the [/help/email-listid|email-listid setting]. If set, it is + used as the List-ID header for all outbound notification emails. + * Add the "--branch" option to the "[/help/timeline|timeline]" command + to restrict the displayed items to a specific branch. + * Add the "--versions" option to "[/help/diff|fossil diff]" + to display details about the compared versions into the patch header. + * Numerous other minor enhancements. + +

Changes for version 2.18 (2022-02-23)

+ * Added support for [./ssl-server.md|SSL/TLS server mode] for commands + like "[/help/server|fossil server]" and "[/help/http|fossil http]" + * The new [/help/cherry-pick|cherry-pick command] is an alias for + [/help/merge|merge --cherrypick]. + * Add new setting "[/help/large-file-size|large-file-size]". If the size + of any file in a commit exceeds this size, a warning is issued. + * Query parameter "year=YYYY" is now accepted by [/help/www/timeline|/timeline]. + * The [/help/tar|tar] and [/help/zip|zip commands] no longer + sterilize the manifest file. + * Further improvement to diff alignment in cases that involve both + edits and indentation changes. + * [/doc/trunk/www/chat.md|Chat] improvements:
    +
  • [/help/www/chat|The /chat page] input options have been reworked + again for better cross-browser portability. +
  • When sending a [/help/www/chat|/chat] message fails, it is no longer + immediately lost and sending may optionally be retried. +
  • [/help/www/chat|/chat] can now optionally embed attachments of certain + types directly into message bodies via an iframe. +
  • Add the "--as FILENAME" option to the "[/help/chat|fossil chat send]" + command. +
  • Added the "[/help/chat|fossil chat pull]" command, available to + administrators only, for backing up the chat conversation. +
+ * Promote the test-detach command into the [/help/detach|detach command]. + * For "[/help/pull|fossil pull]" with the --from-parent-project option, + if no URL is specified then use the last URL from the most recent prior + "fossil pull --from-parent-project". + * Add options --project-name and --project-desc to the + "[/help/init|fossil init]" command. + * The [/help/www/ext|/ext page] generates the SERVER_SOFTWARE environment + variable for clients. + * Fix the REQUEST_URI [/doc/trunk/www/aboutcgi.wiki#cgivar|CGI variable] such + that it includes the query string. This is how most other systems understand + REQUEST_URI. + * Added the --transport-command option to [/help/sync|fossil sync] + and similar. + +

Changes for version 2.17 (2021-10-09)

+ + * Major improvements to the "diff" subsystem, including:
    +
  • Added new [/help/diff|formatting options]: --by, -b, --webpage, --json, --tcl. +
  • Partial-line matching for unified diffs +
  • Better partial-line matching for side-by-side diffs +
  • Buttons on web-based diffs to show more context +
  • Performance improvements +
+ * The --branchcolor option on [/help/commit|fossil commit] and + [/help/amend|fossil amend] can now take the value "auto" to + force Fossil to use its built-in automatic color choosing algorithm. + * Fossil now [./concepts.wiki#workflow|autosyncs] prior to running + [/help/open|fossil open]. + * Add the [/help/ticket-default-report|ticket-default-report setting], + which if set to the title of a ticket report causes that ticket report + to be displayed below the search box in the /ticket page. + * The "nc" query parameter to the [/help/www/timeline|/timeline] page + causes all graph coloring to be omitted. + * Improvements and bug fixes to the new "[/help/ui|fossil ui REMOTE]" + feature so that it works better on a wider variety of platforms. + * In [/help/www/wikiedit|/wikiedit], show the list of attachments for + the current page and list URLs suitable for pasting them into the page. + * Add the --no-http-compression option to [/help/sync|fossil sync] + and similar. + * Print total payload bytes on a [/help/sync|fossil sync] when using + the --verbose option. + * Add the close, reopen, hide, and + unhide subcommands to [/help/branch|the branch command]. + * The "-p" option to [/help/branch|fossil branch list] shows only + private branches. + * The [/md_rules|Markdown formatter] now interprets the content of + block HTML markup (such as <table>) in most cases. Only content + of <pre> and <script> is passed through verbatim. + * The [/help/wiki|wiki list command] no longer lists "deleted" + pages by default. Use the new --all option to include deleted + pages in the output. + * The [/help/all|fossil all git status] command only shows reports for + the subset of repositories that have a configured Git export. + * The [/help/www/chat|/chat] configuration was reimplemented and + provides new options, including the ability for a repository + administrator to + [./chat.md#notifications|extend the selection of notification sounds] + using unversioned files. + * Chat now uses fossil's full complement of markdown features, + instead of the prior small subset of markup it previously supported. + This retroactively applies to all chat messages, as they are + markdown-processed when they are sent instead of when they + are saved. + * Added a chat message preview mode so messages can be previewed + before being sent. Similarly, added a per-message ability to view + the raw un-parsed message text. + * The hotkey to activate preview mode in [/help/www/wikiedit|/wikiedit], + [/help/www/fileedit|/fileedit], and [/help/www/pikchrshow|/pikchrshow] + was changed from ctrl-enter to shift-enter in order to align with + [/help/www/chat|/chat]'s new preview feature and related future + changes. + +

Changes for Version 2.16 (2021-07-02)

+ * Security: Fix the client-side TLS so that it verifies that the + server hostname matches its certificate. + * The default "ssh" command on Windows is changed to "ssh" instead of the + legacy "plink", as ssh is now generally available on Windows systems. + Installations that still need to use the legacy "plink" can make that + happen by running: 'fossil set ssh-command "plink -ssh" --global'. + * Added the [./patchcmd.md|fossil patch] command. + * The [/help/ui|fossil ui] command is enhanced in multiple ways:
    +
  1. The REPOSITORY argument can be the name of a check-out directory. +
  2. If the REPOSITORY argument is prefixed by "HOST:" or "USER@HOST:" + then the ui is run on the remote machine and tunnelled back to the local + machine using ssh. (The latest version of fossil must be installed on + both the local and the remote for this to work correctly.) +
  3. The new --nobrowser and --fossilcmd options is provided. +
+ * The [/brlist|/brlist web page] allows the user to + select multiple branches to be displayed together in a single + timeline. + * The [./forum.wiki|Forum] provides a hyperlink on the author of each + post that goes to a timeline of recent posts by that same author. + * Added the "[/help/bisect|fossil bisect run]" command for improved + automation of bisects. + * The [/help/merge|fossil merge] command now does a better job merging + branches where files have been renamed between the current branch and the + branch being merged. + * The [/help/open|fossil open] command allows the repository file + to be inside the working directory without requiring the --force flag. + * The [/help/www/wikiedit|/wikiedit] and [/help/www/wikinew|/wikinew] + pages now default to markdown format. + * The [/help/www/login|/login] page now links to a user's forum post + timeline if the repository has forum posts. + * Tags may now be propagated for forum posts, wiki pages, and technotes. + The [/help/tag|tag command] can now manipulate and list such tags. + * [./caps/login-groups.md|Login-Groups] are now shown on the repository + list of the "[/help/all|fossil all ui]" command. + * Administrators can configure [./alerts.md|email alerts] to expire + a specific number of days (ex: 365) after the last user contact with + the Fossil server. This prevents alert emails being sent to + abandoned email accounts forever. + * SQL that defines [/tktsetup_tab|database objects for tickets] now + [/timeline?c=c717f1ef9a1a4c91|can DROP] a VIEW or an INDEX provided + that its name starts with 'ticket' or 'fx_'. + * Update the built-in SQLite to version 3.36.0. + * Numerous other minor enhancements. + +

Changes for Version 2.15 (2021-03-26) and Patch 2.15.1 on (2021-04-07) + and 2.15.2 on (2021-06-15)

+ * Patch 2.15.2: Fix the client-side TLS so that it verifies that the + server hostname matches its certificate. Upgrading to + the patch is recommended. + * Patch 2.15.1: Fix a data exfiltration bug in the server. Upgrading to + the patch is recommended. + * The [./defcsp.md|default CSP] has been relaxed slightly to allow + images to be loaded from any URL. All other resources are still + locked down by default. + * The built-in skins all use the "[/help/mainmenu|mainmenu]" + setting to determine the content of the main menu. + The ability to edit the + "mainmenu" setting is added on the /Admin/Configuration page. + * The hamburger menu is now available on most of the built-in skins. + * Any built-in skin named "X" can be used instead of the standard + repository skin by adding the URL parameter skin=X to the + request. The selection is persisted using the display + preferences cookie unless the "once" query parameter is also + included. The [/skins] page may be used to select a skin. + * The [/cookies] page now gives the user an opportunity to delete + individual cookies. And the /cookies page is linked from the + /sitemap, so that it appears in hamburger menus. + * The [/sitemap] extensions are now specified by a single new + "[/help/sitemap-extra|sitemap-extra setting]", + rather than a cluster of various + "sitemap-*" settings. The older settings are no longer used. + This change might require minor server configuration + adjustments on servers that use /sitemap extensions. + The /Admin/Configuration page provides the ability to edit + the new "sitemap-extra" setting. + * Added the "--ckout-alias NAME" option to + [/help/ui|fossil ui], [/help/server|fossil server], and + [/help/http|fossil http]. This option causes Fossil to + understand URIs of the form "/doc/NAME/..." as if they were + "[/help/www/doc|/doc/ckout/...]", to facilitate testing of + [./embeddeddoc.wiki|embedded documentation] changes prior to + check-in. + * For diff web pages, if the diff type (unified versus side-by-side) + is not specified by a query parameter, and if the + "[/help/preferred-diff-type|preferred-diff-type]" + setting is omitted or less than 1, then select the diff type based + on a guess of whether or not the request is coming from a mobile + device. Mobile gets unified and desktop gets side-by-side. + * The various pages which show diffs now have toggles to show/hide + individual diffs. + * Add the "[/help/preferred-diff-type|preferred-diff-type]" + setting to allow an admin to force a default diff type. + * The "pikchr-background" setting is now available in + "detail.txt" skin files, for better control of Pikchr + colors in inverted color schemes. + * Add the --list option to the + [/help/tarball|tarball], + [/help/zip|zip], and [/help/sqlar|sqlar] + commands. + * The javascript used to implement the hamburger menu on the + default built-in skin has been made generic so that it is usable + by a variety of skins, and promoted to an ordinary built-in + javascript file. + * New TH1 commands: + "[/doc/trunk/www/th1.md#bireqjs|builtin_request_js]", + "[/doc/trunk/www/th1.md#capexpr|capexpr]", + "foreach", "lappend", and "string match" + * The [/help/leaves|leaves command] now shows the branch point + of each leaf. + * The [/help/add|fossil add] command refuses to add files whose + names are reserved by Windows (ex: "aux") unless the --allow-reserved + option is included. This helps prevent Unix users from accidentally + creating check-ins that are unreadable by Windows users. + * Add the "re=" query parameter to the [/help/www/dir|/dir] webpage, + for symmetry with the [/help/www/tree|/tree] page. + * Update the built-in SQLite to version 3.35.0. + * The ./configure script now has the --print-minimum-sqlite-version option + that prints the minimum SQLite version required by the current version + of Fossil. This might be used by integrators who insist on building + Fossil to link against the system SQLite library rather than the + built-in copy of SQLite, to verify that their system SQLite library + is recent enough. + * Webpage that shows [/help/www/whistory|history of a wiki page] + gained client-side UI to help with comparison between two arbitrary + versions of a wiki (by the means of anchoring a "baseline" version) + and the ability to squeeze several sequential edits made by the same + user into a single "recycled" row (the latest edit in that sequence). + +

Changes for Version 2.14 (2021-01-20) and Patch 2.14.1 on (2021-04-07) + and 2.14.2 on (2021-06-15)

+ * Patch 2.14.2: Fix the client-side TLS so that it verifies that the + server hostname matches its certificate. Upgrading to + the patch is recommended. + * Patch 2.14.1: Fix a data exfiltration bug in the server. + Upgrading to the patch is recommended. + * Schema Update Notice #1: + This release drops a trigger from the database schema (replacing + it with a TEMP trigger that is created as needed). This + change happens automatically the first time you + add content to a repository using Fossil 2.14 or later. No + action is needed on your part. However, if you upgrade to + version 2.14 and then later downgrade or otherwise use an earlier + version of Fossil, the email notification mechanism may fail + to send out notifications for some events, due to the missing + trigger. If you want to + permanently downgrade an installation, then you should run + "[/help/rebuild|fossil rebuild]" after the downgrade + to get email notifications working again. If you are not using + email notification, then the schema change will not affect you in + any way. + * Schema Update Notice #2: + This release changes how the descriptions of wiki edits are stored + in the EVENT table, for improved display on timelines. You must + run "[/help/rebuild|fossil rebuild]" to take advantage of + this enhancement. Everything will still work without + "fossil rebuild", except you will get goofy descriptions of + wiki updates in the timeline. + * Add support for [./chat.md|Fossil chat]. + * The "[/help/clone|fossil clone]" command is enhanced so that + if the repository filename is omitted, an appropriate name is derived + from the remote URL and the newly cloned repo is opened. This makes + the clone command work more like Git, thus making it easier for + people transitioning from Git. + * Added the --mainbranch option to the [/help/git|fossil git export] + command. + * Added the --format option to the + "[/help/timeline|fossil timeline]" command. + * Enhance the --numstat option on the + "[/help/diff|fossil diff]" command so that it shows a total + number of lines added and deleted and total number of files + modified. + * Add the "contact" sub-command to [/help/user|fossil user]. + * Added commands "[/help/all|fossil all git export]" and + "[/help/all|fossil all git status]". + * Added the "df=CHECKIN" query parameter to the + [/help/www/timeline|/timeline page]. + * Improvements to the "[/sitemap]" page. Add subpages + [/sitemap-timeline] and [/sitemap-test]. + * Better text position in cylinder objects of Pikchr diagrams. + * New "details.txt" settings available to custom skins to better control + the rendering of Pikchr diagrams: +
    +
  • pikchr-foreground +
  • pikchr-scale +
  • pikchr-fontscale +
+ * Allow the use of SQL functions inside the ticket table definition + for custom ticket configurations. + * The built-in SQLite is updated to version 3.35.0 alpha containing + performance optimizations, especially performance associated with + startup, and minor improvements to the CLI. + * Performance optimizations to Fossil itself. + * Countless improvements and enhancements to the documentation + +

Changes for Version 2.13 (2020-11-01)

+ + * Added support for [./interwiki.md|interwiki links]. + * Enable <del> and <ins> markup in wiki. + * Improvements to the Forum threading display. + * Added support for embedding [./pikchr.md|pikchr] + markup in markdown and fossil-wiki content. + * The new "[/help/pikchr|pikchr]" command can render + pikchr scripts, optionally pre-processed with + [/doc/trunk/www/th1.md|TH1] blocks and variables exactly like + site skins are. + * The new [/help/www/pikchrshow|pikchrshow] page provides an + editor and previewer for pikchr markup. + * In [/help/www/wikiedit|/wikiedit] and + [/help/www/fileedit|/fileedit], Ctrl-Enter can now be used + initiate a preview and to toggle between the editor and preview + tabs. + * The /artifact and /file views, when in + line-number mode, now support interactive selection of a range + of lines to hyperlink to. + * Enhance the [/help/www/finfo|/finfo] webpage so that when query + parameters identify both a filename and a checkin, the resulting + graph tracks the identified file across renames. + * The built-in SQLite is updated to an alpha of version 3.34.0, and + the minimum SQLite version is increased to 3.34.0 because the + /finfo change in the previous bullet depends on enhancements to + recursive common table expressions that are only available in + SQLite 3.34.0 and later. + * Countless other minor refinements and documentation improvements. + +

Changes for Version 2.12.1 (2020-08-20)

+ + * (2.12.1): Fix client-side vulnerabilities discovered by Max Justicz. + * Security fix in the "[/help/git|fossil git export]" command. + The same fix is also backported to version 2.10.1 and 2.11.1. + New "safety-net" features were added to prevent similar problems + in the future. + * Enhancements to the graph display for cases when there are + many cherry-pick merges into a single check-in. + [/timeline?f=2d75e87b760c0a9|Example] + * Enhance the [/help/open|fossil open] command with the new + --workdir option and the ability to accept a URL as the repository + name, causing the remote repository to be cloned automatically. + Do not allow "fossil open" to open in a non-empty working directory + unless the --keep option or the new --force option is used. + * Enhance the markdown formatter to more closely follow the + [https://spec.commonmark.org/0.29/#emphasis-and-strong-emphasis|CommonMark specification] + with regard to text highlighting. + Underscores in the middle of identifiers (ex: fossil_printf()) + no longer need to be escaped. + * The markdown-to-html translator can prevent unsafe HTML + (for example: <script>) on user-contributed pages like forum and + tickets and wiki. The admin can adjust this behavior using + the [/help/safe-html|safe-html setting] on the Admin/Wiki page. + The default is to disallow unsafe HTML everywhere. + [https://fossil-scm.org/forum/forumpost/3714e6568f|Example]. + * Added the "collapse" and "expand" capability for long forum posts. + [https://fossil-scm.org/forum/forumpost/9297029862|Example] + * The "[/help/remote-url|fossil remote]" command now has options for + specifying multiple persistent remotes with symbolic names. Currently + only one remote can be used at a time, but that might change in the + future. + * Add the "Remember me?" checkbox on the login page. Use a session + cookie for the login if it is not checked. + * Added the experimental "[/help/hook|fossil hook]" command for + managing "hook scripts" that run before checkin or after a push. + * Enhance the [/help/revert|fossil revert] command so that it + is able to revert all files beneath a directory. + * Add the [/help/bisect|fossil bisect skip] command. + * Add the [/help/backup|fossil backup] command. + * Enhance [/help/bisect|fossil bisect ui] so that it shows all unchecked + check-ins in between the innermost "good" and "bad" check-ins. + * Added the --reset flag to the "[/help/add|fossil add]", + "[/help/rm|fossil rm]", and + "[/help/addremove|fossil addremove]" commands. + * Added the "--min N" and "--logfile FILENAME" + flags to the [/help/backoffice|backoffice] command, as well as other + enhancements to make the backoffice command a viable replacement for + automatic backoffice. Other incremental backoffice improvements. + * Added the [/help/www/fileedit|/fileedit page], which allows + editing of text files online. Requires explicit activation by + a setup user. + * Translate built-in help text into HTML for display on web pages. + [/help/help|Example]. + * On the [/help/www/timeline|/timeline] webpage, the combination + of query parameters "p=CHECKIN" and "bt=ANCESTOR" draws all + ancestors of CHECKIN going back to ANCESTOR. For example, + [/timeline?p=202006271506&bt=version-2.11] shows all ancestors + of the checkin that occurred on 2020-06-27 15:06 going back to + the 2.11 release. + * Update the built-in SQLite so that the + "[/help/sql|fossil sql]" command supports new output + modes ".mode box" and ".mode json". + * Add the "obscure()" SQL function to the + "[/help/sql|fossil sql]" command. + * Added virtual tables "helptext" and "builtin" to + the "[/help/sql|fossil sql]" command, providing access to the + dispatch table including all help text, and the builtin data files, + respectively. + * [./delta_format.wiki|Delta compression] is now applied to forum edits. + * The [/help/www/wikiedit|wiki editor] has been modernized and is + now Ajax-based. The WYSIWYG editing option for Fossil-format wiki + pages was removed. (Please let us know, via the site's Forum menu, + if that removal unduly impacts you.) This also changes the semantics + of the wiki "Sandbox": that pseudo-page may be freely edited but + no longer saved via the UI (the [/help/wiki|wiki CLI command] + can, though). + * The [/help/allow-symlinks|allow-symlinks setting] no longer + syncs. It must be activated individually on any clones which require + symlinks. + * Countless documentation enhancements. + +

Changes for Version 2.11 (2020-05-25)

+ + * (2.11.2): Backport security fixes from 2.12.1 + * (2.11.1): Backport security fix for the "fossil git export" command. + * Support [/md_rules|Markdown] in the default ticket configuration. + * Timestamp strings in [./checkin_names.wiki|object names] + can now omit punctation. So, for example, "202004181942" and + "2020-04-18 19:42" mean the same thing. + * Enhance backlink processing so that it works with Markdown-formatted + tickets and so that it works for wiki pages. + Ticket [a3572c6a5b47cd5a]. +
  • "[/help/rebuild|fossil rebuild]" is needed to + take full advantage of this fix. Fossil will continue + to work without the rebuild, but the new backlinks will be missing.
+ * The algorithm for finding the + [./tech_overview.wiki#configloc|location of the configuration database] + is enhanced to be XDG-compliant. + * Add a hide/show feature to + [./wikitheory.wiki#assocwiki|associated wiki] display on + check-in and branch information pages. + * Enhance the "[/help/info|fossil info]" command so that it + works with no arguments even if not within an open check-out. + * Many improvements to the forum and especially email notification + of forum posts, in response to community feedback after switching + SQLite support from a mailing list over to the forum. + * Minimum length of a self-registered user ID increased from 3 to 6 + characters. + * When the "vfx" query parameter is used on the + "[/help/www/timeline|/timeline]" page, it causes the complete + text of forum posts to be displayed. + * Rework the "[/help/grep|fossil grep]" command to be more useful. + * Expose the [/help/redirect-to-https|redirect-to-https] + setting to the [/help/settings|settings] command. + * Improve support for CGI on IIS web servers. + * The [./serverext.wiki|/ext page] can now render index files, + in the same way as the embedded docs. + * Most commands now support the Unix-conventional "--" + flag to treat all following arguments as filenames + instead of flags. + * Added the [/help/mimetypes|mimetypes config setting] + (versionable) to enable mimetype overrides and custom definitions. + * Add an option on the /Admin/Timeline setup page to set a default + timeline style other than "Modern". + * In [./embeddeddoc.wiki|embedded documentation], hyperlink URLs + of the form "/doc/$CURRENT/..." the "$CURRENT" text is translated + into the check-in hash for the document currently being viewed. + * Added the [/help/www/phantoms|/phantoms] webpage that shows all + phantom artifacts. + * Enhancements to phantom processing to try to reduce + bandwidth-using chatter about phantoms on the sync protocol. + * Security: Fossil now assumes that the schema of every + database it opens has been tampered with by an adversary and takes + extra precautions to ensure that such tampering is harmless. + * Security: Fossil now puts the Content-Security-Policy in the + HTTP reply header, in addition to also leaving it in the + HTML <head> section, so that it is always available, even + if a custom skin overrides the HTML <head> and omits + the CSP in the process. + * Output of the [/help/diff|fossil diff -y] command automatically + adjusts according to the terminal width. + * The Content-Security-Policy is now set using the + [/help/default-csp|default-csp setting]. + * Merge conflicts caused via the [/help/merge|merge] and + [/help/update|update] commands no longer leave temporary + files behind unless the new --keep-merge-files flag + is used. + * The [/help/www/artifact_stats|/artifact_stats page] is now accessible + to all users if the new "artifact_stats_enable" setting is turned + on. There is a new checkbox under the /Admin/Access menu to turn + that capability on and off. + * Add the [/help/tls-config|fossil tls-config] command for viewing + the TLS configuration and the list of SSL Cert exceptions. + * Captchas all include a button to read the captcha using an audio + file, so that they can be completed by the visually impaired. + * Stop using the IP address as part of the login cookie. + * Bug fix: fix the SSL cert validation logic so that if an exception + is allowed for particular site, the exception expires as soon as the + cert changes values. + * Bug fix: the FTS search into for forum posts is now kept up-to-date + correctly. + * Bug fix: the "fossil git export" command is now working on Windows + * Bug fix: display Technote items on the timeline correctly + * Bug fix: fix the capability summary matrix of the Security Audit + page so that it does not add "anonymous" capabilities to the + "nobody" user. + * Update internal Unicode character tables, used in regular expression + handling, from version 12.1 to 13. + * Many documentation enhancements. + * Many minor enhancements to existing features. + +

Changes for Version 2.10 (2019-10-04)

+ + * (2.10.2): backport security fixes from 2.12.1 + * (2.10.1): backport security fix for the "fossil git export" command. + * Added support for [./serverext.wiki|CGI-based Server Extensions]. + * Added the [/help/repolist-skin|repolist-skin] setting used to + add style to repository list pages. + * Enhance the hierarchical display of Forum threads to do less + indentation and to provide links back to the previous message + in the thread. Provide sequential numbers for all messages in + a forum thread. + * Add support for fenced code blocks and improved hyperlink + processing to the [/md_rules|markdown formatter]. + * Add support for hyperlinks to wiki pages in the + [/md_rules|markdown formatter]. + * Enhance the [/help/www/stat|/stat] page so that it gives the + option to show a breakdown of forum posts. + * The special check-in name "merge-in:BRANCH" means the source of + the most recent merge-in from the parent branch of BRANCH. + * Add hyperlinks to branch-diffs on the /info page and from + timelines of a branch. + * Add graphical context on the [/help/www/vdiff|/vdiff] page. + * Uppercase query parameters, POST parameters, and cookie names are + converted to all lowercase and entered into the parameter set, + instead of being discarded. + * Change the default [./hashpolicy.wiki|hash policy] to SHA3. + * Timeout [./server/any/cgi.md|CGI requests] after 300 seconds, or + some other value set by the + [./cgi.wiki#timeout|"timeout:" property] in the CGI script. + * The check-in lock interval is reduced from 24 hours to 60 seconds, + though the interval is now configurable using a setting. + An additional check for conflicts is added after interactive + check-in comment entry, to compensate for the reduced lock interval. + * Performance optimizations. + * Many documentation improvements. + +

Changes for Version 2.9 (2019-07-13)

+ + * Added the [/help/git|fossil git export] command and instructions + for [./mirrortogithub.md|creating a GitHub mirror of a Fossil project]. + * Improved handling of relative hyperlinks on the + [/help/www/artifact|/artifact] pages for wiki. For example, + hyperlinks and the lizard <img> now work correctly + for both [/artifact/2ff24ab0887cf522] and + [/doc/0d7ac90d575004c2415/www/index.wiki]. + * Enhancements to the timeline graph layout, to show more information + with less clutter. + * Added tool-tips to the /timeline graph. On by default but can be + disabled by setting the "Tooltip dwell time" to 0 in the timeline + configuration. + * Copy buttons added to various check-in hash and branch name links. + * Double-clicking on a /timeline graph node now jumps to the /info page + for the check-in. So, repurpose the timestamp hyperlink to show all + activity around that check-in in time. + * Added the [/help/touch|fossil touch] command, and the --setmtime + option on the [/help/open|fossil open] and + [/help/update|fossil update] commands. + * Many documentation enhancements. + * For the "[/help/update|fossil update]" and + "[/help/checkout|fossil checkout]" commands, if a + managed file is removed because it is no longer part of the target + check-in and the directory containing the file is empty after the + file is removed and the directory is not the current working + directory and is not on the [/help/empty-dirs|empty-dirs] + list, then also remove the directory. + * Update internal Unicode character tables, used in regular expression + handling, from version 11.0 to 12.1. + * In "[/help/regexp|fossil regexp]", "[/help/grep|fossil grep]" + and the TH1 "regexp" command, the -nocase option now removes multiple + diacritics from the same character (derived from SQLite's + remove_diacritics=2) + * Added the [/help/www/secureraw|/secureraw] page that requires the + complete SHA1 or SHA3 hash, not just a prefix, before it will deliver + content. + * Accept purely numeric ISO8601 date/time strings as long as they + do not conflict with a hash. Example: "20190510134217" instead of + "2019-05-10 13:42:17". This helps keep URLs shorter and less + complicated + * Support both "1)" and "1." for numbered lists in markdown, as + commonmark does. + * The sync and clone HTTP requests omit the extra /xfer path element + from the end of the request URI. All servers since 2010 know that + the HTTP request is for a sync or clone from the mimetype so the + extra path element is not needed. + * If an automatic sync gets a permanent redirect request, then update + the saved remote URL to the new address. + * Temporary filenames (for example used for external "diff" commands) + try to preserve the suffix of the original file. + * Added the [/help/www/thisdayinhistory|/thisdayinhistory] web page. + * Enhanced parsing of [/help/www/timeline|/timeline] query parameters + "ymd=", "ym=", and "yw=". All arguments are option (in which case they + default to the current time) and all accept ISO8601 date/times without + punctuation. + * Automatically disapprove pending moderation requests for a user when + that user is deleted. This helps in dealing with spam-bots. + * Improvements to the "Capability Summary" section in the + [/help/www/secaudit0|Security Audit] web-page. + * Use new "ci-lock" and "ci-lock-failed" pragmas in the + [./sync.wiki|sync protocol] to try to prevent accident forks + caused by concurrent commits when operating in auto-sync mode. + * Fix a bug ([https://fossil-scm.org/forum/forumpost/c51b9a1169|details]) + that can cause repository databases to be overwritten with debugging + output, thus corrupting the repository. This is only a factor when + CGI debugging is enabled, and even then is a rare occurrence, but it is + obviously an important fix. + +

Changes for Version 2.8 (2019-02-20)

+ + * Show cherry-pick merges as dotted lines on the timeline graph. + → The "fossil rebuild" command must be run to create and + populate the new "cherrypick" table in the repository in order + for this feature to operate. + * Add the ability to associate branches, check-ins, and tags with + specially-named Wiki pages. This gives the ability to better + document branches and tags, and provide more documentation on + check-ins beyond the check-in comment. The associated Wiki is + automatically displayed on /info pages for check-ins, and on + /timeline?r=BRANCH and /timeline?t=TAG pages for branches and + tags. This feature is on by default, but can be disabled in on + the Admin/Wiki page. + * Enhance the repository list page (shown for example by + "fossil all ui") so that it shows the name and last check-in + time for each project. The implementation of the repository + list page is now broken out into a separate source file (repolist.c). + * Allow users with Forum Supervisor permission ('6') to add Forum + Write Trusted permission ('4') to users as they are approving a + forum post by that user. + * When running a bisect, report the number of check-ins still in + the search range and the estimated number of bisect steps remaining. + Do this at each step of the bisect. + * Provide a permanent link to a bisect timeline using the bid= query + parameter. + * Make the chronological forum display feature available to all users, + and make it the default format on mobile devices. + * Break out Wiki setup into a separate /setup_wiki page, accessible + on the standard menus through Admin/Wiki. + * Add "Next" and "Previous" buttons on the /wdiff page, allowing + the user to step through the versions of a wiki page. + * Improve the display of the /whistory page. + * Omit the "HH:MM" timestamps on timeline graphs on narrow-screen + devices, to improve horizontal space uses. This helps make Fossil + more mobile-friendly. + * Enhance /wcontent to show a sortable list of Wiki pages together + with the number of revisions and the most recent change time for + each page. + * Hyperlinks to Wiki pages on the /timeline go to the specific + version of the Wiki page named in the timeline, not to the latest + version. + * Enhancements to the "amend", "tag", and "reparent" commands, including + adding options --override-date, --override-user, and --dry-run. + * Add the global --comment-format command-line option and the + comment-format setting to control the display of the command-line + timeline. + * Change the "fossil reparent" command so that it only works from + within an active checkout. + * On the /setup_ucap_list, show administrators how many users have + each capability. The counts are a hyperlink to the /setup_ulist + page showing the subset of users that have that capability. + * Provide the ability to redirect all HTTP pages to HTTPS. Formerly + one could cause this to occur for the /login page only. That option + still exists, but the redirect can now also be done for all pages. + * "Compress" the built-in javascript by omitting comments and + leading and trailing whitespace. + * Detect when the repository used by a checkout is swapped out for + a clone that uses different RID values, and make appropriate adjustments + to the checkout database to avoid any problems. + * Add the backoffice-disable setting to completely disable the + backoffice feature. + * Update the built-in SQLite to version 3.27.1. + * Various other small enhancements to webpages and documentation. + + +

Changes for Version 2.7 (2018-09-22)

+ + * Add the [./alerts.md|email alerts] feature for commits, ticket + changes, wiki changes, forum posts, and announcements. This is + still a work in progress. It is functional, but it is not as easy to + setup and use as it ought to be. + * Add the [./forum.wiki|discussion forum] feature. + * Add new user capabilities letters needed to support alerts and forum. + Formerly, user capabilities were letters from [a-z], but with the + enhancements, the supply of lower case letters was exhausted. + User capabilities are now letters in [a-zA-Z0-9]. + * The built-in skins are now responsive, providing better layout on + small screens, including mobile devices. + * The default skin now includes a hamburger menu that is generated + by the [/sitemap] page. + * All of the built-in skins now use a + [https://en.wikipedia.org/wiki/Content_Security_Policy|Content Security Policy (CSP)] + to help prevent cross-site injection and forgery attacks. There are no known + vulnerabilities in Fossil. The added CSP does not fix anything; it merely adds + another layer of defense. + * The [/sitemap] and other list pages show as multiple columns if + the viewing window is wide enough. + * There is an optional "js" file for each skin that can be used to + hold javascript. This file can be loaded by reference or can be + included in the header or footer. + * Add the [./backoffice.md|backoffice]. + * Update internal Unicode character tables, used in regular expression + handling, from version 10.0 to 11.0. + * Improvements to the "Security Audit" administration page + * Add the [/help/branch|fossil branch current] command. + * Add the [./grep.md|grep] command. + * Update the built-in SQLite to version 3.25.1. + * Some code and interfaces are in place to support sending and + receiving email directly via SMTP, but this feature is not yet + complete or ready for production use. + * The `mv-rm-files` setting is now compiled into Fossil in the + default Fossil configuration; no longer must you say + ./configure --with-legacy-mv-rm to make it available. The + setting remains disabled by default, however, so you must still say + fossil set mv-rm-files 1 to enable it on each repository + where you want hard mv/rm behavior. + +

Changes for Version 2.6 (2018-05-04)

+ + * Fix a bug that was causing crashes while trying to clone the TCL + repository. This fix is the main reason for the current release. + * Added the new "Classic" timeline viewing mode. "Classic" is the + same as "Verbose" in the previous release. The "Verbose" mode is + now like "Compact" except the extra check-in details are shown by + default. + * Add support for ETags:, Last-Modified:, and If-Modified-Since: + cache control mechanisms. + * Enhance the [/help/www/tarball|/tarball], + [/help/www/zip|/zip], and + [/help/www/sqlar|/sqlar] pages so that the checkin + name to be downloaded can be expressed as part of the URI, + and without the need for query parameters. + * On the [/help/www/timeline|/timeline] webpage, add the days=N + query parameter and enhance the ymd=DATE and yw=DATE query parameters + to accept 'now' as an argument to show the latest day or week. + * In the web page that comes up in response to the + [/help/all|fossil all ui] command, show the last modification + time for each repository, and allow click-to-sort on the modification + time column. + * In the tarball cache replacement algorithm, give extra weight to + tarballs that have been accessed more than once. + * Additional defenses against web-based attacks. There have not been + any known vulnerabilities. We are just being paranoid. + * Update the built-in SQLite to an alpha version of 3.24.0. + +

Changes for Version 2.5 (2018-02-07)

+ + * Numerous enhancements to the look and feel of the web interface. + Especially: Added separate "Modern", "Compact", "Verbose", and + "Columnar" view options on timelines. + * Common display settings (such as the "view" option and the number + of rows in a timeline) are held in a cookie and thus persist + across multiple pages. + * Rework the skin editing process so that changes are implemented + on one of nine /draft pages, evaluated, then merged back to the + default. + * Added the [/timeline?skin=ardoise&once|Ardoise] skin. + * Fix the "fossil server" command on Unix to be much more responsive + to multiple simultaneous web requests. + * Use the IPv6 stack for the "fossil ui" and "fossil server" + commands on Windows. + * Support for [https://sqlite.org/sqlar|SQL Archives] as a download + option. + * Fossil now automatically generates the + <html><head>...</head><body> + at the beginning of each web page if the configurable header + lacks a <body> tag. + * Added the /artifact_stats page, currently accessible only by + the administrator. + * Upgrade to the latest versions of SQLite and OpenSSL. + * Improved key bindings on the Tk diff screen generated by + "fossil diff --tk". + * Begin factoring out in-line javascript into separately loaded + script files. This is a step along the + road toward supporting a strict Content Security Policy. More work + is to be done. + * Initial infrastructure is in place to make use of the pledge() + system call in OpenBSD. More work is to be done. + +

Changes for Version 2.4 (2017-11-03)

+ + * New feature: URL Aliases. URL Aliases allow an administrator + to define their own URLs on the web interface that are rewritten to + built-in URLs with specific parameters. Create and configure URL Aliases + using the /Setup/URL_Aliases menu option in the web interface. + * Add tech-note search capability. + * Add the -r|--revision and -o|--origin options to the + [/help/annotate|annotate] command. + * Add the origin= query parameter to the [/help/www/annotate|/annotate] + webpage. + * The [/help/annotate|fossil annotate] command and the + [/help/www/annotate|/annotate] web page go backwards in time as far + as can be computed in 30 milliseconds by default, rather than stopping + after 20 steps. The new limit= query parameter or the --limit command-line + option can be used to alter this timeout. + * Provide separate [/help#settings|on-line help screens for each setting]. + * Back out support for the --no-dir-symlinks option + * Remove support from the legacy configuration sync protocol. The only + way now to do a configuration push or pull is to use the new protocol that + was added in 2011. + * Add the from= and to= query parameters to [/help/www/fdiff|/fdiff] + in order to get a diff of two files in the same check-in. + * Fix the "ssh://" protocol to prevent an attack whereby the attacker convinces + a victim to run a "clone" with a dodgy URL and thereby gains access to their + system. + * Provide a checkbox that will temporarily disable all ad-units. + * Improvements to the [/help/www/stat|/stat] page + * Various new hyperlinks to the [/help/www/bloblist|/bloblist] + and [/help/www/bigbloblist|/bigbloblist] pages. + * Correct the [/help/www/doc|/doc] page to support read-only repositories. + * Correct [/help/www/zip|/zip], [/help/www/tarball|/tarball], + [/help/zip|zip], and [/help/tarball|tarball] pages and commands to + honor the versioned manifest setting when outside of an open checkout + directory. + * The admin-log and access-log settings are now on by default for + new repositories. + * Update the built-in SQLite to version 3.21.0. + +

Changes for Version 2.3 (2017-07-21)

+ + * Update the built-in SQLite to version 3.20.0 (beta). + * Update internal Unicode character tables, used in regular expression + handling, from version 9.0 to 10.0. + * Show the last-sync-URL on the [/help/www/urllist|/urllist] page. + * Added the "Event Summary" activity report. + [/reports?type=ci&view=lastchng|example] + * Added the "Security Audit" page, available to administrators only + * Added the Last Login time to the user list page, for administrators only + * Added the --numstat option to the [/help/diff|fossil diff] command + * Limit the size of the heap and stack on unix systems, as a proactive + defense against the + [https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt|Stack Clash] + attack. + * Fix "database locked" warnings caused by "PRAGMA optimize". + * Fix a potential XSS vulnerability on the + [/help/www/help|/help] webpage. + * Documentation updates + +

Changes for Version 2.2 (2017-04-11)

+ + * GIT comment tags are now handled by Fossil during import/export. + * Show the content of README files on directory listings. + ([/file/skins|example]) + * Support for Basic Authentication if enabled (default off). + * Show the hash algorithms used on the + [/help/www/rcvfromlist|/rcvfromlist] page. + * The [/help/www/tarball|/tarball] and [/help/www/zip|/zip] pages + now use the the r= query parameter + to select which check-in to deliver. The uuid= query parameter + is still accepted for backwards compatibility. + * Update the built-in SQLite to version 3.18.0. + * Run "[https://www.sqlite.org/pragma.html#pragma_optimize|PRAGMA optimize]" + on the database connection as it is closing. + +

Changes for Version 2.1 (2017-03-10)

+ + * Add support for [./hashpolicy.wiki|hash policies] that control which + of the Hardened-SHA1 or SHA3-256 algorithms is used to name new + artifacts. + * Add the "gshow" and "gcat" subcommands to [/help/stash|fossil stash]. + * Add the [/help/www/juvlist|/juvlist] web page and use it to construct + the [/uv/download.html|Download Page] of the Fossil self-hosting website + using Ajax. + +

Changes for Version 2.0 (2017-03-03)

+ + * Use the + [https://github.com/cr-marcstevens/sha1collisiondetection|hardened SHA1] + implementation by Marc Stevens and Dan Shumow. + * Add the ability to read and understand + [./fileformat.wiki#names|artifact names] that are based on SHA3-256 + rather than SHA1, but do not actually generate any such names. + * Added the [/help/sha3sum|sha3sum] command. + * Update the built-in SQLite to version 3.17.0. + +

Changes for Version 1.37 (2017-01-16)

+ + * Add checkbox widgets to various web pages. See [/technote/8d18bf27e9| + this technote] for more information. To get the checkboxes to look as + intended, you must update the CSS in your repository and all clones. + * Add the [/help/all|fossil all ui] command + * Add the [/help/www/file|/file] webpage + * Enhance the [/help/www/brlist|/brlist] webpage to make use of branch colors. + * Add support for the ms=EXACT|LIKE|GLOB|REGEXP query parameter on the + [/help/www/timeline|/timeline] webpage, with associated form widgets. + * Enhance the [/help/changes|changes] and [/help/status|status] commands + with many new filter options so that specific kinds of changes can be + found without having to pipe through grep or sed. + * Enhanced the [/help/sqlite3|fossil sql] command so that it opens the + [./tech_overview.wiki#localdb|checkout database] and the + [./tech_overview.wiki#configdb|configuration database] in addition to the + repository database. + * TH1 enhancements: +
  • Add [unversioned content] command.
    • +
  • Add [unversioned list] command.
    • +
  • Add project_description variable.
    • +
+ * Rename crnl-glob [/help/settings|setting] to crlf-glob, but keep + crnl-glob as a compatibility alias. + * Added the --command option to the [/help/diff|diff] command. + * Fix a C99-ism that prevents the 1.36 release from building with MSVC. + * Fix [/help/ticket|ticket set] when using the "+" prefix with fields + from the "ticketchng" table. + * Remove the "fusefs" command from builds that do not have the underlying + support enabled. + * Fixes for incremental git import/export. + * Minor security enhancements to + [./encryptedrepos.wiki|encrypted repositories]. + * Update the built-in SQLite to version 3.16.2. + * Update the built-in Zlib to version 1.2.11. + + +

Changes for Version 1.36 (2016-10-24)

+ + * Add support for [./unvers.wiki|unversioned content], + the [/help/unversioned|fossil unversioned] command and the + [/help/www/uv|/uv] and [/uvlist] web pages. + * The [/uv/download.html|download page] is moved into + [./unvers.wiki|unversioned content] so that the self-hosting Fossil + websites no longer uses any external content. + * Added the "Search" button to the graphical diff generated by + the --tk option on the [/help/diff|diff] command. + * Added the "--checkin VERSION" option to the + [/help/diff|diff] command. + * Various performance enhancements to the [/help/diff|diff] command. + * Update internal Unicode character tables, used in regular expression + handling, from version 8.0 to 9.0. + * Update the built-in SQLite to version 3.15. Fossil now requires + the SQLITE_DBCONFIG_MAINDBNAME interface of SQLite which is only available + in SQLite version 3.15 and later and so Fossil will not work with + earlier SQLite versions. + * Fix [https://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg23618.html|multi-line timeline bug] + * Enhance the [/help/purge|fossil purge] command. + * New command [/help/shell|fossil shell]. + * SQL parameters whose names are all lower-case in Ticket Report SQL + queries are filled in using HTTP query parameter values. + * Added support for [./childprojects.wiki|child projects] that are + able to pull from their parent but not push. + * Added the -nocomplain option to the TH1 "query" command. + * Added support for the chng=GLOBLIST query parameter on the + [/help/www/timeline|/timeline] webpage. + +

Changes for Version 1.35 (2016-06-14)

+ + * Enable symlinks by default on all non-Windows platforms. + * Enhance the [/md_rules|Markdown formatting] so that hyperlinks that begin + with "/" are relative to the root of the Fossil repository. + * Rework the [/help/www/setup_ulist|/setup_list page] (the User List page) + to display all users in a click-to-sort table. + * Fix backslash-octal escape on filenames while importing from git + * When markdown documents begin with <h1> HTML elements, use that + header at the document title. + * Added the [/help/www/bigbloblist|/bigbloblist page]. + * Enhance the [/help/www/finfo|/finfo page] so that when it is showing + the ancestors of a particular file version, it only shows direct + ancestors and omits changes on branches, thus making it show the same set + of ancestors that are used for [/help/www/blame|/blame]. + * Added the --page option to the [/help/ui|fossil ui] command + * Added the [/help/bisect|fossil bisect ui] command + * Enhanced the [/help/diff|fossil diff] command so that it accepts + directory names as arguments and computes diffs on all files contained + within those directories. + * Fix the [/help/add|fossil add] command so that it shows "SKIP" for + files added that were already under management. + * TH1 enhancements: +
  • Add [array exists] command.
    • +
  • Add minimal [array names] command.
    • +
  • Add tcl_platform(engine) and tcl_platform(platform) array + elements.
    • +
+ * Get autosetup working with MinGW. + * Fix autosetup detection of zlib in the source tree. + * Added autosetup detection of OpenSSL when it may be present under the + "compat" subdirectory of the source tree. + * Added the [/help/reparent|fossil reparent] command + * Added --include and --exclude options to [/help/tarball|fossil tarball] + and [/help/zip|fossil zip] and the in= and ex= query parameters to the + [/help/www/tarball|/tarball] and [/help/www/zip|/zip] web pages. + * Add support for [./encryptedrepos.wiki|encrypted Fossil repositories]. + * If the FOSSIL_PWREADER environment variable is set, then use the program it + names in place of getpass() to read passwords and passphrases + * Option --baseurl now works on Windows. + * Numerous documentation improvements. + * Update the built-in SQLite to version 3.13.0. + +

Changes for Version 1.34 (2015-11-02)

+ + * Make the [/help/clean|fossil clean] command undoable for files less + than 10MiB. + * Update internal Unicode character tables, used in regular expression + handling, from version 7.0 to 8.0. + * Add the new [/help/amend|amend] command which is used to modify + tags of a "check-in". + * Fix bug in [/help/import|import] command, handling version 3 of + the svndump format for subversion. + * Add the [/help/all|all cache] command. + * TH1 enhancements: +
  • Add minimal [lsearch] command. Only exact + case-sensitive matching is supported.
    • +
  • Add the [glob_match], [markdown], + [dir], and [encode64] commands.
    • +
  • Add the [tclIsSafe] and [tclMakeSafe] commands to + the Tcl integration subsystem.
    • +
  • Add 'double', 'integer', and 'list' classes to the + [string is] command.
    • +
+ * Add the --undo option to the [/help/diff|diff] command. + * Build-in Antirez's "linenoise" command-line editing library for use with + the [/help/sqlite3|fossil sql] command on Unix platforms. + * Add [/help/stash|stash cat] as an alias for the + [/help/stash|stash show] command. + * Automatically pull before [/help/merge|fossil merge] when auto-sync + is enabled. + * Fix --hard option to [/help/mv|fossil mv] and [/help/rm|fossil rm] + to enable them to work properly with certain relative paths. + * Change the mimetype for ".n" and ".man" files to text/plain. + * Display improvements in the [/help/bisect|fossil bisect chart] command. + * Updated the built-in SQLite to version 3.9.1 and activated JSON1 and FTS5 + support (both currently unused within Fossil). + +

Changes for Version 1.33 (2015-05-23)

+ * Improved fork detection on [/help/update|fossil update], + [/help/status|fossil status] and related commands. + * Change the default skin to what used to be called "San Francisco Modern". + * Add the [/repo-tabsize] web page + * Add [/help/import|fossil import --svn], for importing a subversion + repository into fossil which was exported using "svnadmin dump". + * Add the "--compress-only" option to [/help/rebuild|fossil rebuild]. + * Use a pie chart on the [/reports?view=byuser] page. + * Enhanced [/help/clean|fossil clean --verily] so that it ignores + keep-glob and ignore-glob settings. Added the -x alias for --verily. + * Add the --soft and --hard options to [/help/rm|fossil rm] and + [/help/mv|fossil mv]. The default is still --soft, but that is + now configurable at compile-time or by the mv-rm-files setting. + * Improved ability to [./customgraph.md|customize the timeline graph]. + * Improvements to the [/sitemap] page. + * Automatically adjust the [/help/timeline|CLI timeline] to the terminal + width on Linux. + * Added [info commands] and [info vars] commands to TH1. + These commands perform the same function as their Tcl counterparts, + except they do not accept a pattern argument. + * Fix some obscure issues with TH1 expression processing. + * Fix titles in search results for documents that are not wiki, markdown, + or HTML. + * Formally translate TH1 to Tcl return codes and vice-versa, where + necessary, in the Tcl integration subsystem. + * Add [/help/leaves|fossil leaves -multiple], for finding multiple + leaves on the same branch. + * Added the "Blitz" skin option. + * Removed the ".fossil-settings/keep-glob" file. It should not have been + checked into the repository. + * Update the built-in SQLite to version 3.8.10.2. + * Make [/help/open|fossil open] honor ".fossil-settings/allow-symlinks". + * Allow [/help/add|fossil add] to be used on symlinks to nonexistent or + unreadable files in the same way as [/help/addremove|fossil addremove]. + * Added fork warning to be issued if sync produced a fork + * Update the [/help/www/info|info] page to report when a file becomes a + symlink. Additionally show the UUID for files whose types have changed + without changing contents or symlink target. + * Have [/help/changes|fossil changes] and + [/help/status|fossil status] report when executable or symlink status + changes on otherwise unmodified files. + * Permit filtering weekday and file [/help/www/reports|reports] by user. + Also ensure the user parameter is preserved when changing types. Add a + field for direct entry of the user name to each applicable report. + * Create parent directories of [/help/settings|empty-dirs] if they don't + already exist. + * Inhibit timeline links to wiki pages that have been deleted. + +

Changes for Version 1.32 (2015-03-14)

+ * When creating a new repository using [/help/init|fossil init], ensure + that the new repository is fully compatible with historical versions of + Fossil by having a valid manifest as RID 1. + * Anti-aliased rendering of arrowheads on timeline graphs. + * Added vi/less-style key bindings to the --tk diff GUI. + * Documentation updates to fix spellings and changes all "checkins" to + "check-ins". + * Add the --repolist option to server commands such as + [/help/server|fossil server] or [/help/http|fossil http]. + * Added the "Xekri" skin. + * Enhance the "ln=" query parameter on artifact displays to accept multiple + ranges, separate by spaces (or "+" when URL-encoded). + * Added [/help/forget|fossil forget] as an alias for + [/help/rm|fossil rm]. + +

Changes For Version 1.31 (2015-02-23)

+ * Change the auxiliary schema by adding columns MLINK.ISAUX and MLINK.PMID + columns to the schema, to support better drawing of file change graphs. + A [/help/rebuild|fossil rebuild] is recommended but is not required. + so that the new graph drawing logic can work effectively. + * Added [/search|search] over Check-in comments, Documents, Tickets and + Wiki. Disabled by default. The search can be either a full-scan or it + can use an index that is kept up-to-date automatically. The new + /srchsetup web-page and the [/help/fts-config|fts-config] command + were added to help configure the search capability. Expect further + enhancements to the search capabilities in subsequent releases. + * Added form elements to some submenus (in particular the /timeline) + for easier operation. + * Added the --ifneeded option to [/help/rebuild|fossil rebuild]. + * Added "override skins" using the "skin:" line of the CGI script or + using the --skin LABEL option on the [/help/server|server], + [/help/ui|ui], or [/help/http|http] commands. + * Embedded html documents that begin with + <doc class="fossil-doc"> are displayed with standard + headers and footers added. + * Allow <div style='...'> markup in [/wiki_rules|wiki]. + * Renamed "Events" to "Technical Notes", while updating the technote + display and control pages. Add support for technotes as plain text + or as Markdown. + * Added the [/md_rules] pages containing summary instructions on the + Markdown format. + * Added the --repolist and --nojail options to the various server commands + (ex: [/help/server|fossil server]). + * Added the [/help/all|fossil all add] subcommand to "fossil all". + * Improvements to the /login page. Some hyperlinks to pages that require + "anonymous" privileges are displayed even if the current user is "nobody" + but automatically redirect to /login. + * The [/help/www/doc|/doc] web-page will now try to deliver the file + "404.md" from the top-level directory (if such a file exists) in + place of its built-in 404 text. + * Download of Tarballs and ZIP Archives by user "nobody" is now enabled + by default in new repositories. + * Enhancements to the table sorting controls. More display tables + are now sortable. + * Add IPv6 support to [/help/sync|fossil sync] and + [/help/clone|fossil clone] + * Add more skins such as "San Francisco Modern" and "Eagle". + * During shutdown, check to see if the check-out database (".fslckout") + contains a lot of free space, and if it does, VACUUM it. + * Added the [/mimetype_list] page. + * Added the [/hash-collisions] page. + * Allow the user of Common Table Expressions in the SQL that defaults + ticket reports. + * Break out the components (css, footer, and header) for the + various built-in skins into separate files in the source tree. + +

Changes For Version 1.30 (2015-01-19)

+ * Added the [/help/bundle|fossil bundle] command. + * Added the [/help/purge|fossil purge] command. + * Added the [/help/publish|fossil publish] command. + * Added the [/help/unpublished|fossil unpublished] command. + * Enhance the [/tree] webpage to show the age of each file with the option + to sort by age. + * Enhance the [/brlist] webpage to show additional information about each branch + and to be sortable by clicking on column headers. + * Add support for Docker. Just install docker and type + "sudo docker run -d -p 8080:8080 nijtmans/fossil" to get it running. + * Add the [/help/fusefs|fossil fusefs DIRECTORY] command that mounts a + Fuse Filesystem at the given DIRECTORY and populates it with read-only + copies of all historical check-ins. This only works on systems that + support FuseFS. + * Add the administrative log that records all configuration. + * Added the [/sitemap] webpage. + * Added the [/bloblist] web page. + * Let [/help/new|fossil new] no longer create an initial empty commit + by default. The first commit after checking out an empty repository will + become the initial commit. + * Added the [/help/all|fossil all dbstat] and + [/help/all|fossil all info] commands. + * Update SQLite to version 3.8.8. + * Added the --verily option to the [/help/clean|fossil clean] command. + * Add the "autosync-tries" setting to control the number of autosync attempts + before returning an error. + * Added a compile-time option (--with-miniz) to build using miniz instead + of zlib. Disabled by default. + * Support customization of commands and webpages, including the ability to + add new ones, via the "TH1 hooks" feature. Disabled by default. Enabled + via a compile-time option. + * Add the [checkout], [render], [styleHeader], [styleFooter], + [trace], [getParameter], [setParameter], [artifact], and + [globalState] commands to TH1, primarily for use by TH1 hooks. + * Automatically adjust the width of command-line timeline output according to the + detected width of the terminal. + * Prompt the user to optionally fix invalid UTF-8 at check-in. + * Added a line-number toggle option to the [/help/www/info|/info] + and [/help/www/artifact|/artifact] pages. + * Most commands now issue errors rather than silently ignoring unrecognized + command-line options. + * Use full 40-character SHA1 hashes (instead of abbreviations) in most + internal URLs. + * The "ssh:" sync method on Windows now uses "plink.exe" instead of "ssh" as + the secure-shell client program. + * Prevent a partial clone when the connection is lost. + * Make the distinction between 301 and 302 redirects. + * Allow commits against a closed check-in as long as the commit goes onto + a different branch. + * Improved cache control in the web interface reduces unnecessary requests + for common resources like the page logo and CSS. + * Fix a rare and long-standing sync protocol bug + that would silently prevent the sync from running to completion. Before + this bug-fix it was sometimes necessary to do "fossil sync --verily" to + get two repositories in sync. + * Add the [/finfo?name=src/foci.c|files_of_checkin] virtual table - useful + for ad hoc queries in the [/help/sqlite3|fossil sql] interface, + and also used internally. + * Added the "$secureurl" TH1 variable for use in headers and footers. + * (Internal:) Add the ability to include resources as separate files in the + source tree that are converted into constant byte arrays in the compiled + binary. Use this feature to store the Tk script that implements the --tk + diff option in a separate file for easier editing. + * (Internal:) Implement a system of compile-time checks to help ensure + the correctness of printf-style formatting strings. + * Fix CVE-2014-3566, also known as the POODLE SSL 3.0 vulnerability. + * Numerous documentation fixes and improvements. + * Other obscure and minor bug fixes - see the timeline for details. + +

Changes For Version 1.29 (2014-06-12)

+ * Add the ability to display content, diffs and annotations for UTF16 + text files in the web interface. + * Add the "SaveAs..." and "Invert" buttons + to the graphical diff display that results + from using the --tk option with the [/help/diff | fossil diff] command. + * The [/reports] page now requires Read ("o") permissions. The "byweek" + report now properly propagates the selected year through the event type + filter links. + * The [/help/info | info command] now shows leaf status of the checkout. + * Add support for tunneling https through a http proxy (Ticket [e854101c4f]). + * Add option --empty to the "[/help/open | fossil open]" command. + * Enhanced [/help/www/fileage|the fileage page] to support a glob parameter. + * Add -w|--ignore-all-space and -Z|--ignore-trailing-space options to + [/help/annotate|fossil annotate], [/help/blame|fossil blame], + [/help/diff|fossil (g)diff], [/help/stash|fossil stash diff]. + * Add --strip-trailing-cr option to [/help/diff|fossil (g)diff] and + [/help/stash|fossil stash diff]. + * Add button "Ignore Whitespace" to /annotate, /blame, /ci, /fdiff + and /vdiff UI pages. + * Enhance [/reports?view=byweekday|/reports] with a "byweekday" view. + * Enhance the [/help/cat|fossil cat] command so that it works outside + of a checkout when using the -R command-line option. + * Use full-length SHA1 hashes, not abbreviations, in most hyperlinks. + * Correctly render the <title> markup on wiki pages in the + [/help/www/artifact|/artifact] webpage. + * Enhance the [/help/whatis|fossil whatis] command to report on attachments + and cluster artifacts. Added the [/help/test-whatis-all] command for + testing purposes. + * Add support for HTTP Basic Authentication on [/help/clone|clone] and + [/help/sync|sync]. + * Fix the [/help/stash|stash] so that it remembers added files and re-adds + them when the stash is applied. + * Fix the server so that it avoids writing to the database (and thus avoids + database locking issues) on a + [/help/pull|pull] or [/help/clone|clone]. + * Add support for [./server.wiki#loadmgmt|server load management] using both + a cache of expensive pages (the [/help/cache|fossil cache] command) + and by rejecting expensive page requests when the server load average is too + high. + * Add the [/help/praise|fossil praise] command as an alias for + [/help/blame|fossil blame] for subversion compatibility. + * Enhance the [/help/test-diff|fossil test-diff] command with -y or --tk + options so that it shows both filenames above their respective columns in + the side-by-side diff output. + * Issue a warning if a [/help/add|fossil add] command tries to add a file + that matches the ignore-glob. + * Add option -W|--width to "[/help/stash|fossil stash ls]" + and "[/help/leaves|fossil leaves]" commands. + * Enhance support for running as the root user. Now works on Haiku. + * Added the -empty option to [/help/new|fossil new], which + causes it to not create an initial empty commit. The first commit after + checking out a repo created this way will become the initial commit. + * Enhance sync operations by committing each round-trip to minimize number + of retransmits when autosync fails. Include option for + [/help/update| fossil update] and [/help/merge| fossil merge] to + continue even if missing content. + * Minor portability fixes for platforms where the char type is unsigned + by default. + +

Changes For Version 1.28 (2014-01-27)

+ * Enhance [/help/www/reports | /reports] to support event type filtering. + * When cloning a repository, the user name passed via the URL (if any) + is now used as the default local admin user's name. + * Enhance the SSH transport mechanism so that it runs a single instance of + the "fossil" executable on the remote side, obviating the need for a shell + on the remote side. Some users may need to add the "?fossil=/path/to/fossil" + query parameter to "ssh:" URIs if their fossil binary is not in a standard + place. + * Add the "[/help/blame | fossil blame]" command that works just like + "fossil annotate" but uses a different output format that includes the + user who made each change and omits line numbers. + * Add the "Tarball and ZIP-archive Prefix" configuration parameter under + Admin/Configuration. + * Fix CGI processing so that it works on web servers that do not + supply REQUEST_URI. + * Add options --dirsonly, --emptydirs, and --allckouts to the + "[/help/clean | fossil clean]" command. + * Ten-fold performance improvement in large "fossil blame" or + "fossil annotate" commands. + * Add option -W|--width and --offset to "[/help/timeline | fossil timeline]" + and "[/help/finfo | fossil finfo]" commands. + * Option -n|--limit of "[/help/timeline | fossil timeline]" now + specifies the number of entries, just like all other commands which + have the -n|--limit option. The various timeline-related functions + now output "--- ?? limit (??) reached ---" at the end whenever + appropriate. Use "-n 0" if no limit is desired. + * Fix handling of password embedded in Fossil URL. + * New --once option to [/help/clone | fossil clone] command + which does not store the URL or password when cloning. + * Modify [/help/ui | fossil ui] to respect "default user" in an open + repository. + * Fossil now hides check-ins that have the "hidden" tag in timeline webpages. + * Enhance /ci_edit page to add the "hidden" tag to check-ins. + * Advanced possibilities for commit and ticket change notifications over + http using TH1 scripting. + * Add --sha1sum and --integrate options + to the "[/help/commit | fossil commit]" command. + * Add the "clean" and "extra" subcommands to the + "[/help/all | fossil all]" command + * Add the --whatif option to "[/help/clean|fossil clean]" that works the + same as "--dry-run", + so that the name does not collide with the --dry-run option of "fossil all". + * Provide a configuration option to show dates on the web timeline + as "YYMMMDD HH:MM" + * Add an option to the "stats" webpage that allows an administrator to see + the current repository schema. + * Enhancements to the "[/help/www/vdiff|/vdiff]" webpage for more difference + display options. + * Added the "[/tree?ci=trunk&expand | /tree]" webpage as an alternative + to "/dir" and make it the default way of showing file lists. + * Send gzipped HTTP responses to clients that support it. + +

Changes For Version 1.27 (2013-09-11)

+ * Enhance the [/help/changes | fossil changes], + [/help/clean | fossil clean], [/help/extras | fossil extras], + [/help/ls | fossil ls] and [/help/status | fossil status] commands + to restrict operation to files and directories named on the command-line. + * New --integrate option to [/help/merge | fossil merge], which + automatically closes the merged branch when committing. + * Renamed /stats_report page to [/reports]. Graph width is now + relative, not absolute. + * Added yw=YYYY-WW (year-week) filter to timeline to limit the results + to a specific year and calendar week number, e.g. [/timeline?yw=2013-01]. + * Updates to SQLite to prevent opening a repository file using file descriptors + 1 or 2 on Unix. This fixes a bug under which an assertion failure could + overwrite part of a repository database file, corrupting it. + * Added support for unlimited line lengths in side-by-side diffs. + * New --close option to [/help/commit | fossil commit], which + immediately closes the branch being committed. + * Added chart option to [/help/bisect | fossil bisect]. + * Improvements to the "human or bot?" determination. + * Reports errors about missing CGI-standard environment variables for HTTP + servers which do not support them. + * Minor improvements to sync support on Windows. + * Added --scgi option to [/help/server | fossil server]. + * Internal improvements to the sync process. + * The internals of the JSON API are now MIT-licensed, so downstream + users/packagers are no longer affected by the "do no evil" license + clause. + +

Changes For Version 1.26 (2013-06-18)

+ * The argument to the --port option for the [/help/ui | fossil ui] and + [/help/server | fossil server] commands can take an IP address in addition + to the port number, causing Fossil to bind to just that one IP address. + * After prompting for a password, also ask if that password should be + remembered. + * Performance improvements to the diff engine. + * Fix the side-by-side diff engine to work better with multi-byte Unicode text. + * Color-coding in the web-based annotation (blame) display. Fix the annotation + engine so that it is no longer confused by time-warps. + * The markdown formatter is now available by default and can be used for + tickets, wiki, and embedded documentation. + * Add subcommands "fossil bisect log" and "fossil bisect status" to the + [/help/bisect | fossil bisect] command, as well as other bisect enhancements. + * Enhanced defenses that prevent spiders from using excessive CPU and bandwidth. + * Consistent use of the -n or --dry-run command line options. + * Win32: Fossil now understands Cygwin paths containing one or more of + the characters "*:<>?|. Those are normally forbidden in + win32. This means that the win32 fossil.exe is better usable in a Cygwin + environment. See + [http://cygwin.com/cygwin-ug-net/using-specialnames.html#pathnames-specialchars]. + * Cygwin: Fossil now understands win32 absolute paths starting with a drive + letter everywhere. The default value of the "case-sensitive" setting is + now FALSE, except when case-sensitivity is enabled in the Windows kernel. + See + [http://cygwin.com/cygwin-ug-net/using-specialnames.html#pathnames-casesensitive] + * Enhancements to /timeline.rss, adding more flags for filtering + results, including the ability to subscribe to changes made + to individual tickets. For example: [/timeline.rss?y=t&tkt=12fceeec82]. + * Improved handling of the differences between case-sensitive and + case-insensitive filesystems. + * JSON API: added the 'status' command to report local checkout status. + * Fixes to the --args support and documented this feature in the help. + * Added [/stats_report] page. + * Added ym=YYYY-MM filter to the [/timeline?ym=2013-06]. + * Fixed: config reset now re-installs default ticket report format. + * ssh:// and file:// protocols now ignore proxy settings. + * Added [/hash-color-test] web page. + * Cherry-pick merges are recorded internally (though no yet displayed on the + timeline graph.) + * Bring in the latest versions of SQLite, zlib, and autosetup from upstream. + +

Changes For Version 1.25 (2013-02-16)

+ * Enhancements to ticket processing. There are now two tables: TICKET and + TICKETCHNG. There is one row in TICKETCHNG for each ticket artifact. + Fields from ticket artifacts go into either or both of TICKET and + TICKETCHNG, whichever contain matching column names. Default ticket + edit and viewing scripts are updated to use TICKETCHNG. The TH1 + scripting language is enhanced to support this, including the new + "query" command for doing SQL queries against the repository database. + All changes should be backwards compatible. + * Add the ability to moderate ticket and wiki changes. Unmoderated changes + do not sync and may be deleted by the moderator if found to contain spam + or other objectionable content. + * Add javascript so that clicking on a node of the timeline graph selects + that node. Then clicking on a second node shows a diff between the + two nodes. Clicking on the selected node unselects it. + * Warn of unresolved merge conflicts in "fossil status" and disallow + commits of unresolved conflicts unless the --allow-conflict option + is used. + * Add javascript so that clicking on column headers in a ticket report + sorts by the indicated column. + * Add the "fossil cat" command which is basically an alias for + "fossil finfo -p". + * Hyperlinks with the class "button" are rendered as submenu buttons + on embedded documentation. + * The check-in comment editor on Windows now defaults to NotePad.exe. + * Correctly deal with BOMs in check-in comments. Also attempt to convert + check-in comments to UTF8 from other encodings. + * Allow the deletion of multiple stash entries using multiple arguments + to the "fossil stash rm" command. + * Enhance the "fossil server DIRECTORY" command to serve static content + files contained in DIRECTORY. For security, only files with a + recognized suffix (such as *.html, *.jpg, *.txt, etc) will be delivered + as static content, and *.fossil files are not on the list of recognized + suffixes. There are additional restrictions on the names of the files. + * Allow the "fossil ui" command to specify a directory as long as the + the --notfound option is used. + * Add a configuration option that causes timeline messages to be rendered + as text/x-fossil-plain (which is the same as text/plain except that + hyperlinks inside of [...] are decorated.) + * Only decorate [...] in check-in comments and tickets + if the contented text really is a valid hyperlink target. + * Improvements to the side-by-side diff algorithm, for a more + human-friendly display in some complex cases. + * Added [utime] and [stime] commands to TH1. These + commands can be used for things such as displaying the page rendering + time in the footer. + * Add the ability to pass command-line options of "fossil rebuild" to + "fossil all rebuild". + * Add the --deanalyze option to "fossil rebuild" (and "fossil all rebuild") + * Do not run the graphical merging tool nor leave merge-droppings after a + dry-run merge. Display an improved merge-summary message at the end of + the merge. + * Add options to "fossil commit" to override the various sanity checks. + Options added: --allow-empty, --allow-fork, --allow-older, and + --allow-conflict. + * Optionally require a CAPTCHA (controlled by a setting on the + Admin/Access webpage) when a user who is not logged in tries to + edit wiki, or a ticket, or an attachment. + * Improvements to the "ssh://" sync protocol, to help it move past + noisy motd comments. + * Add the uf=FILE-SHA1-HASH query parameter to the timeline, causing the + timeline to show only check-ins that contain the specific file identified + by FILE-SHA1-HASH. ("uf" stands for "uses file".) + * Enhance the file change annotator so that it follows the file across + name changes. + * Fix the server-side of the sync protocol so that it will not generate + a delta loop when a file changes from its original state, through two + or more intermediate states, and back to the original state, all within + a single sync. + * Show much less output during a sync operation, unless the --verbose + option is used. + * Set the action= attribute of <form> elements using javascript, + as an addition defense against spam-bots. + * Disallow invalid UTF8 characters (such as characters in the surrogate + pair range) in filenames. + * Judge the UserAgent strings issued by the NetSurf webbrowser to be + coming from a human, not from a bot. + * Add the zlib sources to the Fossil source tree (under compat/zlib) and + use those sources when compiling on (Windows) systems that do not have + a zlib library installed by default. + * Prompt the user with the option to convert non-UTF8 files into UTF8 + when committing. + * Allow the characters *[]? in filenames. + * Allow the --context option on diff commands to have a value of 0. + * Added the "dbstat" command. + * Enhanced "fossil merge" so that if the VERSION argument is omitted, Fossil + tries to merge any forks of the current branch. + * Improved detection of forks in a commit race. + * Added the --analyze option to "fossil rebuild". + +

Changes For Version 1.24 (2012-10-22)

+ * Added support for WYSIWYG editing of wiki pages. WYSIWYG is turned off + by default and can be turned on by setting a configuration option. + * Allow style= attribute to occur in HTML markup on wiki pages. + * Added the --tk option to the "fossil diff" and "fossil stash diff" + commands, causing color-coded diff output to be displayed in a Tcl/Tk + GUI window. This option only works if Tcl/Tk is installed on the + host. + * On Windows, make the "gdiff" command default to use WinDiff.exe. + * Update the "fossil stash" command so that it always prompts for a + comment if the -m option is omitted. + * Enhance the timeline webpages so that a=, b=, c=, d=, p=, and dp= + query parameters (and others) can all accept any valid check-in name + (such as branch names or labels) instead of just SHA1 hashes. + * Added the "fossil stash show" command. + * Added the "fileage" webpage with links to this page from the check-in + information page and from the file browser. + * Added --age and -t options to the "fossil ls" command. + * Added the --setmtime option to "fossil update". When used, the mtime + of all managed files is set to the time when the most recent version of + the file was checked in. + * Changed the "vdiff" webpage to show the complete text of files that + were added or removed (the equivalent of using the -N or --newfile + options with the "fossil diff" command-line.) + * Added the --temp option to "fossil clean" and "fossil extra", causing + those commands to only look at temporary files generated by Fossil, + such as merge-conflict reports or aborted check-in messages. + * Enhance the raw page download so that it can guess the mimetype of + attachments based on the filename. + * Change the behavior of the from= and to= query parameters on the + timeline page so that by default the path between the two specified + check-ins avoids merges. + * Add the --baseurl option to "fossil server" and "fossil http" commands, + so that those commands can be used with reverse proxies. + * If unable to determine the command-line user, do not guess. Instead + issue an error message. This helps prevent check-ins from accidentally + occurring under the wrong username. + * Include branch information in the output of file change listings + (the "finfo" webpage). + * Make the simplified view of file history, rather than the full view, + the default. + * In the "fossil configuration" command, allow the "css" option for + synchronizing, importing, or exporting just the CSS file. This makes + it easier to share CSS files across repositories by exporting from + one and importing to another. + * Add the (unsupported) "fossil test-orphans" command. + * Add the --template option to the "fossil init" command, to facilitate + creating new repositories based on a template repository. + * Add the diff-binary setting, which if enabled causes binary files to + be passed to the "gdiff" command for it to deal with, rather than simply + printing a "cannot diff binary files" error. + * Add the --unified option to the "fossil diff" command to force a unified + diff even if the --tk option (which normally implies a side-by-side diff) + is used. + * Present a choice of nearby branches and versions to diff against on the + check-in information page. + * Add the --force option to the "fossil merge" command that will force the + merge to occur even if it would be a no-op. This is sometimes useful for + documentation purposes. + * Add another built-in skin: "Enhanced Default". Other minor tweaks to + the existing skins. + * Add the "urllist" webpage, showing a list of URLs by which a server + instance of Fossil has been accessed. Requires "Administrator" privileges. + A link is on the "Setup" main page. + * Enable dynamic loading of the Tcl runtime for installations that want + to use Tcl as part of their configuration. This reduces the size of + the Fossil binary and allows any version of Tcl 8.4 or later to be used. + * Merge the latest SQLite changes from upstream. + * Lots of minor bug fixes. + +

Changes For Version 1.23 (2012-08-08)

+ * The default checkout database name is now ".fslckout" instead of + "_FOSSIL_" on Unix. Both names continue to work. + * Added the "fossil all changes" command + * Added the --ckout option to the "fossil all list" command + * Added the "public-pages" glob pattern that can be configured to allow + anonymous users to see embedded documentation on sites where source + code should not be accessible to anonymous users. + * Allow multiple --tag options on the same "fossil commit" command. + * Change the meaning of the --bgcolor option for "fossil commit" to only + change the color for that one commit. The new --branchcolor option + is available to set a persistent background color. + * Add the branch= query parameter to the vdiff page and the --branch option + to the "fossil diff" command. + * Check-in names of the form "root:BRANCH" now refer to the origin of + the branch. Hence to see all changes in a branch, use + "fossil diff --from root:BRANCH --to BRANCH". The --branch option on + the diff command is an alias for the same. + * Add the ability to configure ad-units to be displayed between the menu + bar and the content. + * Add the ability to set a background image as part of server configuration. + * Allow partial commits of cherrypick merges. + * Updates against an uncommitted merge are now a warning, not a fatal error. + * Prompt the user to continue if a check-in comment is unedited. + * Fixes to case sensitivity settings with the /dir webpage. + * Repositories now try to remember the locations of all checkouts and + web-access URLs and display this information with the + "fossil info $REPO" command. + * Improved defense against spiders: The src= attribute of + <a> elements is set using javascript after the page loads. + * Enhanced formatting of the user list page. + * If a file named in "fossil add" is missing, that is now a warning instead + of a fatal error. + * Fix side-by-side diff so that it displays correctly with + multi-byte UTF8 characters. + * Performance improvements in the diff logic. + * Other performance tweaks and documentation updates. + +

Changes For Version 1.22 (2012-03-17)

+ * Greatly improved "diff" processing including the new --brief option, + partial line matching, colorized in-line diffs, and better performance. + * Promote "allow-symlinks" to a versionable setting + * Harden the CGI processing logic against DOS attacks + * Add the ability to run TH1 scripts after sync requests + * Store the repository name in _FOSSIL_ as it is type in the "open" command, + possibly as a relative pathname. + * Make ".fslckout" the alternative name for the "_FOSSIL_" file. + * Change the "ssh:" transfer method to allow all access regardless of + user permission. + * Improvements to the timeline messages associated with tag changes. + (Requires a "[/help/rebuild | fossil rebuild]" to take effect.) + * Various additions and fixes for the JSON API. + * Improved merge-with-rename handling. + * --cherrypick merges use their origin's commit message by default. + * Added support for multiple concurrent logins per user. + * Update to use SQLite version 3.7.11. + * Various minor bug fixes. + +

Changes For Version 1.21 (2011-12-13)

+ * Added side-by-side diffs in the command-line interface + * Automatically enable hyperlinks if the UserAgent string in the + HTTP header suggests that the requestor is a human and not a bot. + * Show only commonly used commands with "fossil help". Use + "fossil help --all" to see the complete list now. + * Improvements to the "stash" command: (1) Stash all files, not just + those below the working directory. (2) Add the --detail option to + "list". (3) Confirm before "drop --all". (4) Add the "help" + subcommand. + * Add an Admin/Access setting to change the number of octets of the + IP address that are saved in login cookies - allowing this setting + to be changed to zero + * Promote the "test-md5sum" command to "md5sum". + * Added the "whatis" command. + * Stop showing the server-code in status outputs - it is no longer used + for anything. + * Added a compile-time option (--with-tcl) to build in full Tcl scripting + support via integration with TH1. + * Merged the JSON branch into trunk. Disabled by default. Enabled + by a compile-time option. Probably it will be enabled by default + in some future release. + * Update to use SQLite version 3.7.9 plus the alignment fix for Sparc. + align + +

Changes For Version 1.20 (2011-10-21)

+ * Added side-by-side diffs in HTML interface. [0bde74ea1e] + * Added support for symlinks. (Controlled by "allow-symlinks" setting, + off by default). [e4f1c1fe95] + * Fixed CLI annotate to show the proper file version in case there + are multiple equal versions in history. [e161670939] + * Timeline now shows tag changes (requires rebuild).[87540ed6e6] + * Fixed annotate to show "more relevant" versions of lines in + some cases. [e161670939] + * New command: ticket history. [98a855c508] + * Disabled SSLv2 in HTTPS client.[ea1d369d23] + * Fixed constant prompting regarding previously-saved SSL + certificates. [636804745b] + * Other SSL improvements. + * Added -R REPOFILE support to several more CLI commands. [e080560378] + * Generated tarballs now have constant timestamps, so they are + always identical for any given check-in. [e080560378] + * A number of minor HTML-related tweaks and fixes. + * Added --args FILENAME global CLI argument to import arbitrary + CLI arguments from a file (e.g. long file lists). [e080560378] + * Fixed significant memory leak in annotation of files with long + histories.[9929bab702] + * Added warnings when a merge operation overwrites local copies + (UNDO is available, but previously this condition normally went + silently unnoticed). [39f979b08c] + * Improved performance when adding many files. [a369dc7721] + * Improve merges which contain many file renames. [0b93b0f958] + * Added protection against timing attacks. [d4a341b49d] + * Firefox now remembers filled fields when returning to forms. [3fac77d7b0] + * Added the --stats option to the rebuild command. [f25e5e53c4] + * RSS feed now passes validation. [ce354d0a9f] + * Show overridden user when entering commit comment. [ce354d0a9f] + * Made rebuilding from web interface silent. [ce354d0a9f] + * Now works on MSVC with repos >2GB. [6092935ff2] + * A number of code cleanups to resolve warnings from various compilers. + * Update the built-in SQLite to version 3.7.9 beta. + +

Changes For Version 1.19 (2011-09-02)

+ * Added a ./configure script based on autosetup. + * Added the "[/help/winsrv | fossil winsrv]" command + for creating a Fossil service on Windows systems. + * Added "versionable settings" where settings that affect + the local tree can be stored in versioned files in the + .fossil-settings directory. + * Background colors for branches are chosen automatically if no + color is specified by the user. + * The status, changes and extras commands now show + pathnames relative to the current working directory, + unless overridden by command line options or the + "relative-paths" setting.
WARNING: This + change will break scripts which rely on the current + output when the current working directory is not the + repository root. + * Added "empty-dirs" versionable setting. + * Added support for client-side SSL certificates with "ssl-identity" + setting and --ssl-identity option. + * Added "ssl-ca-location" setting to specify trusted root + SSL certificates. + * Added the --case-sensitive BOOLEAN command-line option to many commands. + Default to true for Unix and false for Windows. + * Added the "Color-Test" submenu button on the branch list web page. + * Compatibility improvements to the git-export feature. + * Performance improvements on SHA1 checksums + * Update to the latest SQLite version 3.7.8 alpha. + * Fix the tarball generator to work with very log pathnames + +

Changes For Version 1.18 (2011-07-14)

+ * Added this Change Log + * Added sequential version numbering + * Added an optional configure script - the Makefile still works for most + systems. + * Improvements to the "annotate" algorithm: only search primary + ancestors and ignore branches. + * Update the "scrub" command to remove traces of login-groups and + subrepositories. + * Added the --type option to the "fossil tag find" command. + * In contexts where only a check-in makes sense, resolve branch and + tag names to checkins only, never events or other artifacts. + * Improved display of file renames on a diff. A rebuild is required + to take full advantage of this change. + * Update the built-in SQLite to version 3.7.7. diff --git a/www/chat.md b/www/chat.md new file mode 100644 index 0000000..23a310d --- /dev/null +++ b/www/chat.md @@ -0,0 +1,262 @@ +# Fossil Chat + +## Introduction + +Fossil’s developer chatroom feature provides an +ephemeral discussion venue for insiders. Design goals include: + + * **Simple but functional** → + Fossil chat is designed to provide a convenient real-time + communication mechanism for geographically dispersed developers. + Fossil chat is *not* intended as a replacement or competitor for + IRC, Slack, Discord, Telegram, Google Hangouts, etc. + + * **Low administration** → + You can activate the chatroom in seconds without having to + mess with configuration files or install new software. + In an existing [server setup](./server/), + simply enable the [C capability](/setup_ucap_list) for users + whom you want to give access to the chatroom. + + * **Ephemeral** → + Chat messages do not sync to peer repositories, and they are + automatically deleted after a configurable delay (default: 7 days). + Individual messages or the entire conversation + can be deleted at any time without impacting any other part + of the system. + +Fossil chat is designed for use by insiders - people with check-in +privileges or higher. It is not intended as a general-purpose gathering +place for random passers-by on the internet. +Fossil chat seeks to provide a communication venue for discussion +that does *not* become part of the permanent record for the project. +For persistent and durable discussion, use the [Forum](./forum.wiki). +Because the conversation is intended to be ephemeral, the chat messages +are local to a single repository. Chat content does not sync. + + +## Setup + +A Fossil repository must be functioning as a [server](./server/) in order +for chat to work. +To activate chat, simply add the [C capability](/setup_ucap_list) +to every user who is authorized to participate. Anyone who can read chat +can also post to chat. + +Setup ("s") and Admin ("a") users always have access to chat, without needing +the "C" capability. A common configuration is to add the "C" capability +to "Developer" so that any individual user who has the "v" capability will +also have access to chat. + +There are also some settings under /Admin/Chat that control the +behavior of chat, though the default settings are reasonable so in most +cases those settings can be ignored. The settings control things like +the amount of time that chat messages are retained before being purged +from the repository database. + +## Usage + +For users with appropriate permissions, simply browse to the +[/chat](/help/www/chat) to start up a chat session. The default +skin includes a "Chat" entry on the menu bar on wide screens for +people with chat privilege. There is also a "Chat" option on +the [Sitemap page](/sitemap), which means that chat will appear +as an option under the hamburger menu for many [skins](./customskin.md). + +Chat messages are subject to [Fossil's +full range of Markdown processing](/md_rules). Because chat messages are +stored as-is when they arrive from a client, this change applies +retroactively to messages stored by previous fossil versions. + +Files may be sent via chat using the file selection element at the +bottom of the page. If the desktop environment system supports it, +files may be dragged and dropped onto that element. Files are not +automatically sent - selection of a file can be cancelled using the +Cancel button which appears only when a file is selected. When the +Send button is pressed, any pending text is submitted along with the +selected file. Image files sent this way will, by default, appear +inline in messages, but each user may toggle that via the settings +popup menu, such that images instead appear as downloadable links. +Non-image files always appear in messages as download links. + +### Deletion of Messages + +
Message deletion is itself a type of message, which +is why deletions count towards updates in the recent activity list. (It +is counted for the person who performed the deletion, not the author of +the deleted comment.) That can potentially lead to odd corner cases +where a user shows up in the list but has no messages which are +currently visible because they were deleted, or an admin user who has +not posted anything but deleted a message. That is a known minor +cosmetic-only bug with a resolution of "will not fix."
+ +Any user may *locally* delete a given message by clicking on the "tab" +at the top of the message and clicking the button which appears. Such +deletions are local-only, and the messages will reappear if the page +is reloaded. The user who posted a given message, or any Admin users, +may additionally choose to globally delete a message from the chat +record, which deletes it not only from their own browser but also +propagates the removal to all connected clients the next time they +poll for new messages. + +### Customizing New-message Notification Sounds + +By default, the list of new-message notification sounds is limited to +a few built in to the fossil binary. In addition, any +[unversioned files](./unvers.wiki) named `alert-sounds/*.{mp3,wav,ogg}` +will be included in that list. To switch sounds, tap the "settings" +button. + +### Who's Online? + +Because the chat app has to be able to work over transient CGI-based +connections, as opposed to a stable socket connection to the server, +real-time tracking of "who's online" is not feasible. +Chat offers an optional feature, toggleable in the settings, +which can list users who have posted messages in the client's current +list of loaded messages. This is not the same thing as tracking who's +online, but it gives an overview of which users have been active most +recently, noting that "lurkers" (people who post no messages) will not +show up in that list, nor does the chat infrastructure have a way to +track and present those. That list can be used to filter messages on a +specific user by tapping on that user's name, tapping a second time to +remove the filter. + +### The `fossil chat` Command + +Type [fossil chat](/help/chat) from within any open check-out +to bring up a chatroom for the project that is in that checkout. +The new chat window will attempt to connect to the default sync +target for that check-out (the server whose URL is shown by the +[fossil remote](/help/remote) command). + +### Chat Messages From Robots + +The [fossil chat send](/help/chat) can be used by project-specific +robots to send notifications to the chatroom. For example, on the +[SQLite project](https://sqlite.org/) (for which the Fossil chatroom +feature, and indeed all of Fossil, was invented) there are long-running +fuzz servers that sometimes run across obscure problems. Whenever this +happens, a message is sent to the SQLite developers chatroom alerting +them to the problem. + +The recommended way to allow robots to send chat messages is to create +a new user on the server for each robot. Give each such robot account +the "C" privilege only. That means that the robot user account will be +able to send chat messages, but not do anything else. Then, in the +program or script that runs the robot, when it wants to send a chat +message, have it run a command like this: + +~~~~ +fossil chat send --remote https://robot:PASSWORD@project.org/fossil \ + --message 'MESSAGE TEXT' --file file-to-attach.txt +~~~~ + +Substitute the appropriate project URL, robot account +name and password, message text and file attachment, of course. + +### Chat Messages For Timeline Events + +If the [chat-timeline-user setting](/help/chat-timeline-user) is not an +empty string, then any change to the repository that would normally result +in a new timeline entry is announced in the chatroom. The announcement +appears to come from a user whose name is given by the chat-timeline-user +setting. + +This mechanism is similar to [email notification](./alerts.md) except that +the notification is sent via chat instead of via email. + +## Quirks + + - There is no message-editing capability. This is by design and + desire of `/chat`'s developers. + + - When `/chat` has problems connecting to the message poller (see + the next section) it will provide a subtle visual indicator of the + connection problem - a dotted line along the top of the input + field. If the connection recovers within a short time, that + indicator will go away, otherwise it will pop up a loud message + signifying that the connection to the poller is down and how long + it will wait to retry (progressively longer, up to some + maximum). `/chat` will recover automatically when the server is + reachable. Trying to send messages while the poller connection is + down is permitted, and the poller will attempt to recover + immediately if sending of a message succeeds. That applies to any + operations which send traffic, e.g. if the per-message "toggle + text mode" button is activated or a message is globally deleted. + +## Implementation Details + +*You do not need to understand how Fossil chat works in order to use it. +But many developers prefer to know how their tools work. +This section is provided for the benefit of those curious developers.* + +The [/chat](/help/www/chat) webpage downloads a small amount of HTML +and a small amount of javascript to run the chat session. The +javascript uses XMLHttpRequest (XHR) to download chat content, post +new content, or delete historical messages. The following web +interfaces are used by the XHR: + + * [/chat-poll](/help?name=/chat-poll) → + Downloads chat content as JSON. + Chat messages are numbered sequentially. + The client tells the server the latest chat message ID it currently + holds, and the server sends back subsequent messages. If there + are no subsequent messages, the /chat-poll page blocks until new + messages are available. + + * [/chat-send](/help?name=/chat-send) → + Sends a new chat message to the server. + + * [/chat-delete](/help?name=/chat-delete) → + Deletes a chat message. + +Fossil chat uses the venerable "hanging GET" or +"[long polling](wikipedia:/wiki/Push_technology#Long_polling)" +technique to receive asynchronous notification of new messages. +This is done because long polling works well with CGI and SCGI, +which are the usual mechanisms for setting up a Fossil server. +More advanced notification techniques such as +[Server-sent events](wikipedia:/wiki/Server-sent_events) and especially +[WebSockets](wikipedia:/wiki/WebSocket) might seem more appropriate for +a chat system, but those technologies are not compatible with CGI. + +Downloading of posted files and images uses a separate, non-XHR interface: + + * [/chat-download](/help?name=/chat-download) → + Fetches the file content associated with a post (one file per + post, maximum). In the UI, this is accessed via links to uploaded + files and via inlined image tags. + +Chat messages are stored on the server-side in the CHAT table of +the repository. + +~~~ +CREATE TABLE repository.chat( + msgid INTEGER PRIMARY KEY AUTOINCREMENT, + mtime JULIANDAY, -- Time for this entry - Julianday Zulu + lmtime TEXT, -- Client YYYY-MM-DDZHH:MM:SS when message originally sent + xfrom TEXT, -- Login of the sender + xmsg TEXT, -- Raw, unformatted text of the message + fname TEXT, -- Filename of the uploaded file, or NULL + fmime TEXT, -- MIMEType of the upload file, or NULL + mdel INT, -- msgid of another message to delete + file BLOB -- Text of the uploaded file, or NULL +); +~~~ + +The CHAT table is not cross-linked with any other tables in the repository +schema. An administrator can "DROP TABLE chat;" at any time, without +harm (apart from deleting all chat history, of course). The CHAT table +is dropped when running [fossil scrub --verily](/help/scrub). + +On the server-side, message text is stored exactly as entered by the +users. The /chat-poll page queries the CHAT table and constructs a +JSON reply described in the [/chat-poll +documentation](/help/www/chat-poll). The message text is translated +into HTML before being converted to JSON so that the text can be +safely added to the display using assignment to `innerHTML`. Though +`innerHTML` assignment is generally considered unsafe, it is only so +with untrusted content from untrusted sources. The chat content goes +through sanitization steps which eliminate any potential security +vulnerabilities of assigning that content to `innerHTML`. diff --git a/www/checkin.wiki b/www/checkin.wiki new file mode 100644 index 0000000..88266b8 --- /dev/null +++ b/www/checkin.wiki @@ -0,0 +1,89 @@ +Check-in Checklist + +

Always run the following checklist prior to every +check-in or commit to the Fossil repository:

+ +Before every check-in: + + 0. fossil user default → your username is correct. + + 1. fossil diff → +
    +
  1. No stray changes +
  2. All changes comply with the license +
  3. All inputs are scrubbed before use +
  4. No injection attacks via %s formats +
+ + 2. fossil extra → no unmanaged files need to be added. + + 3. The check-in will go onto the desired branch. + → Check-ins to trunk normally require approval from + the lead programmer (drh). + + 4. auto-sync is on, or the system clock is verified + + 5. If sources files have been added or removed, ensure all makefiles + and configure scripts have been updated accordingly. + +Before every check-in to trunk: + + 6. No compiler warnings on the development machine. + + 7. The fossil executable that results from a build actually works. + + +
+

Commentary

+ +Before you go ahead and push content back to the servers, make sure +that the username you are using by default matches your username +within the project. Also remember to enable the localauth setting +if you intend to make changes via a locally served web UI. + +Item 1 is the most important step. Consider using gdiff +instead of diff if you have a graphical differ configured. Or +use the command-line option --tk. Also consider the -v +command-line option to show the complete text of newly added files. +The recommended command for completing checklist item 1 is: + + fossil diff --tk -v + +Look carefully at every changed line in item 1. +Make sure that you are not about to commit unrelated changes. +If there are two or more unrelated changes present, consider +breaking up the commit into two or more separate commits. +Always make 100% sure that all changes are compatible with the +BSD license, that you have the authority to commit the code in accordance +with the [/doc/trunk/www/copyright-release.html | CLA] that you have +signed and have on file, and that +no NDAs, copyrights, patents, or trademarks are infringed by the check-in. +Also check very carefully to make sure that +you are not introducing security vulnerabilities. Pay particular attention +to the possibility of SQL or HTML injection attacks. + +Item 2 verifies that you have not added source files but failed to +do the necessary "fossil add" to manage them. If you commit +without bringing the new file under source control, the check-in will +be broken. That, in turn, can cause complications far in the future +when we are bisecting for a bug. + +For item 3, Run "fossil status" or the equivalent to +make sure your changes are going into the branch you think they are. +Also double-check the branch name when entering change comments. +Never check into trunk unless you are expressly authorized to do so. + +For Item 4, if you are on-network, make sure auto-sync is enabled. This +will minimize the risk of an unintended fork. It will also give you a +warning if you system clock is set incorrectly. If you are off-network, +make sure that your system clock is correct or at least close to correct +so that your check-in does not appear out-of-sequence on timelines. +On-network commits are preferred, especially for trunk commits. + +Items 6 and 7 help to ensure that check-ins on the trunk always work. +Knowing that the trunk always works makes bisecting much easier. Items +6 and 7 are recommended for all check-ins, even those that are on a branch. +But they are especially important for trunk check-ins. We desire that +all trunk check-ins work at all times. Any experimental, unstable, or +questionable changes should go on a branch and be merged into trunk after +further testing. diff --git a/www/checkin_names.wiki b/www/checkin_names.wiki new file mode 100644 index 0000000..f3d8758 --- /dev/null +++ b/www/checkin_names.wiki @@ -0,0 +1,316 @@ +Check-in Names + +
+Quick Reference +
    +
  • Hash prefix +
  • Branch name +
  • Tag name +
  • Timestamp: YYYY-MM-DD HH:MM:SS +
  • tag-name : timestamp +
  • root : branchname +
  • start : branchname +
  • merge-in : branchname +
  • Special names: +
      +
    • tip +
    • current +
    • next +
    • previous or prev +
    • ckout (embedded docs only) +
    +
+
+ +Many Fossil [/help|commands] and [./webui.wiki | web interface] URLs accept +check-in names as an argument. For example, the "[/help/info|info]" command +accepts an optional check-in name to identify the specific check-in +about which information is desired: + +
+fossil info checkin-name
+
+ +You are perhaps reading this page from the following URL: + + +https://fossil-scm.org/home/doc/trunk/www/checkin_names.wiki + + +This is an example of an [./embeddeddoc.wiki | embedded documentation] +page URL. The "trunk" element of the pathname is a +[./glossary.md#check-in | check-in] name that +determines which version of the documentation to display. + +Fossil provides a variety of ways to specify a check-in. This +document describes the various methods. + +

Canonical Check-in Name

+ +The canonical name of a check-in is the hash of its +[./fileformat.wiki#manifest | manifest] expressed as a +[./hashes.md | long lowercase hexadecimal number]. For example: + +
+fossil info e5a734a19a9826973e1d073b49dc2a16aa2308f9
+
+ +The full 40 or 64 character hash is unwieldy to remember and type, though, +so Fossil also accepts a unique prefix of the hash, using any combination +of upper and lower case letters, as long as the prefix is at least 4 +characters long. Hence the following commands all +accomplish the same thing as the above: + +
+fossil info e5a734a19a9
+fossil info E5a734A
+fossil info e5a7
+
+ +Fossil uses this feature itself, identifying check-ins by 8 to 16-character +prefixes of the canonical name in places where it doesn't want to chew +up the screen real estate required to display the whole hash. + +

Tags And Branch Names

+ +Using a tag or branch name where a check-in name is expected causes +Fossil to choose the most recent check-in with that tag or branch name. +So for example, the most recent check-in that +is tagged with "release" as of this writing is [b98ce23d4fc]. +The command: + +
+fossil info release
+
+ +…results in the following output: + +
+hash:         b98ce23d4fc3b734cdc058ee8a67e6dad675ca13 2020-08-20 13:27:04 UTC
+parent:       40feec329163103293d98dfcc2d119d1a16b227a 2020-08-20 13:01:51 UTC
+tags:         release, branch-2.12, version-2.12.1
+comment:      Version 2.12.1 (user: drh)
+
+ +There are multiple check-ins that are tagged with "release" but +(as of this writing) the [b98ce23d4fc] +check-in is the most recent so it is the one that is selected. + +Note that unlike some other version control systems, a "branch" in Fossil +is not anything special: it is simply a sequence of check-ins that +share a common tag, so the same mechanism that resolves tag names +also resolves branch names. + + +Note also that there can — in theory, if rarely in practice — be an ambiguity +between tag names and canonical names. Suppose, for example, you had a +check-in with the canonical name deed28aa99… and you +also happened to have tagged a different check-in with "deed2". If +you use the "deed2" name, does it choose the canonical name or the tag +name? In such cases, you can prefix the tag name with "tag:". +For example: + +
+fossil info tag:deed2
+
+ +The "tag:deed2" name will refer to the most recent check-in +tagged with "deed2" rather than the +check-in whose canonical name begins with "deed2". + +

Whole Branches

+ +Usually when a branch name is specified, it means the latest check-in on +that branch, but for some commands (ex: [/help/purge|purge]) a branch name +on the argument means the earliest connected check-in on the branch. This +seems confusing when being explained here, but it works out to be intuitive +in practice. + +For example, the command "fossil purge XYZ" means to purge the check-in XYZ +and all of its descendants. But when XYZ is in the form of a branch name, one +generally wants to purge the entire branch, not just the last check-in on the +branch. And so for this reason, commands like purge will interpret a branch +name to be the first check-in of the branch rather than the last. If there +are two or more branches with the same name, then these commands will select +the first check-in of the branch that has the most recent check-in. What +happens is that Fossil searches for the most recent check-in with the given +tag, just as it always does. But if that tag is a branch name, it then walks +back down the branch looking for the first check-in of that branch. + +Again, this behavior only occurs on a few commands where it make sense. + +

Timestamps

+ +A timestamp in one of the formats shown below means the most recent +check-in that occurs no later than the timestamp given: + + 1. YYYY-MM-DD + 2. YYYY-MM-DD HH:MM + 3. YYYY-MM-DD HH:MM:SS + 4. YYYY-MM-DD HH:MM:SS.SSS + 5. YYYYMMDD + 6. YYYYMMDDHHMM + 7. YYYYMMDDHHMMSS + +In the second through the fourth forms, +the space between the day and the year can optionally be +replaced by an uppercase T, and the entire timestamp can +optionally be followed by "z" or "Z". In the fourth +form with fractional seconds, any number of digits may follow the +decimal point, though due to precision limits only the first three +digits will be significant. The final three pure-digit forms +without punctuation are only valid if the number they encode is +not also the prefix of an artifact hash. + +In its default configuration, Fossil interprets and displays all dates +in Universal Coordinated Time (UTC). This tends to work the best for +distributed projects where participants are scattered around the globe. +But there is an option on the Admin/Timeline page of the web interface to +switch to local time. The "Z" suffix on a timestamp check-in +name is meaningless if Fossil is in the default mode of using UTC for +everything, but if Fossil has been switched to local time mode, then the +"Z" suffix means to interpret that particular timestamp using +UTC instead of local time. + +You may prefix a timestamp with the string “date:”, in which case +processing stops immediately, whether the string is parsed correctly and +refers to anything within the repository or not. The prefix is therefore +useful when the date could be misinterpreted as a tag. For example, a +repo could have release tags like “2020-04-01”, the date the release was +cut, but you could force Fossil to interpret that string as a date +rather than as a tag by passing “date:2020-04-01”. + +For an example of how timestamps are useful, +consider the homepage for the Fossil website itself: + +
+https://fossil-scm.org/home/doc/trunk/www/index.wiki
+
+ +The bold component of that URL is a check-in name. To see the stored content +of the Fossil website repository as of January 1, 2009, one has merely to change +the URL to the following: + +
+https://fossil-scm.org/home/doc/2009-01-01/www/index.wiki
+
+ +(Note that this won't roll you back to the skin and other +cosmetic configurations as of that date. It also won't change screens +like the timeline, which has an independent date selector.) + +

Tag And Timestamp

+ +A check-in name can also take the form of a tag or branch name followed by +a colon and then a timestamp. The combination means to take the most +recent check-in with the given tag or branch which is not more recent than +the timestamp. So, for example: + +
+fossil update trunk:2010-07-01T14:30
+
+ +Would cause Fossil to update the working check-out to be the most recent +check-in on the trunk that is not more recent than 14:30 (UTC) on +July 1, 2010. + +

Root Of A Branch

+ +A branch name that begins with the "root:" prefix refers to the +last check-in on the parent branch prior to the beginning of the branch. +Such a label is useful, for example, in computing all diffs for a single +branch. The following example will show all changes in the hypothetical +branch "xyzzy": + +
+fossil diff --from root:xyzzy --to xyzzy
+
+ + +That doesn't do what you might expect after you merge the parent +branch's changes into the child branch: the above command will include +changes made on the parent branch as well. + +You can solve this by using the prefix "merge-in:" instead of +"root:" to tell Fossil to find +the most recent merge-in point for that branch. +The resulting diff will then show only the changes in +the branch itself, omitting +any changes that have already been merged in from the parent branch. + + +The prefix "start:" gives the first check-in of the named branch. + +The prefixes "root:", "start:", and "merge-in:" +can be chained: one can say for example + +
+fossil info merge-in:xyzzy:2022-03-01
+
+ +to get information about the most recent merge-in point on the branch +"xyzzy" that happened on or before March 1, 2022. + +

Special Tags

+ +The tag "tip" means the most recent check-in. The "tip" tag is practically +equivalent to the timestamp "9999-12-31". + +This special name works anywhere you can pass a "NAME", such as with +/info URLs: + +
+http://localhost:8080/info/tip
+
+ +There are several other special names, but they only work from within a +check-out directory because they are relative to the current checked-out +version: + + * "current": the current checked-out version + * "next": the youngest child of the current checked-out version + * "previous" or "prev": the primary (non-merge) parent of "current" + +Therefore, you can use these names in a fossil info command, +but not in an /info URL, for example. + +For embedded documentation URLs only, there is one more special name, +"ckout". See [./embeddeddoc.wiki#ckout | its coverage elsewhere] for +more details. You cannot currently use "ckout" anywhere other than in +/doc URLs. + + +

Additional Examples

+ +To view the changes in the most recent check-in prior to the version currently +checked out: + +
+fossil diff --from previous --to current
+
+ +Suppose you are of the habit of tagging each release with a "release" tag. +Then to see everything that has changed on the trunk since the last release: + +
+fossil diff --from release --to trunk
+
+ + +

Resolution Order

+ +Fossil currently resolves name strings to artifact hashes in the +following order: + + # Exact matches on [#special | the special names] + # [#timestamps | Timestamps], with preference to ISO8601 forms + # [#tagpfx | tag:TAGNAME] + # [#root | root:BRANCH] + # [#start | start:BRANCH] + # [#merge-in | merge-in:BRANCH] + # [#tag-ts | TAGNAME:timestamp] + # Full artifact hash or hash prefix. + # Any other type of symbolic name that Fossil extracts from + artifacts. + +
diff --git a/www/childprojects.wiki b/www/childprojects.wiki new file mode 100644 index 0000000..ad38bec --- /dev/null +++ b/www/childprojects.wiki @@ -0,0 +1,57 @@ +Child Projects + +

Background

+ +The default behavior of Fossil is to share everything (all check-ins, +tickets, wiki, etc) between all clients and all servers. Such a policy +helps to promote a cohesive design for a cathedral-style project run +by a small cliche of developers - the sort of project for which Fossil +was designed. + +But sometimes it is desirable to branch off a side project that does not +sync back to the master but does continue to track changes in the master. +For example, the master project might be an open-source project like +[https://www.sqlite.org/|SQLite] and a team might want to do a proprietary +closed-source enhancement to that master project in a separate repository. +All changes in the master project should flow forward into the derived +project, but care must be taken to prevent proprietary content from the +derived project from leaking back into the master. + +

Child Projects

+ +A scenario such as the above can be accomplished in Fossil by creating +a child project. The child project is able to freely pull from the parent, +but the parent cannot push or pull from the child nor is the child able to +push to the parent. Content flows from parent to child only, and then only +at the request of the child. + +

Creating a Child Project

+ +To create a new child project, first clone the parent. Then make manual +SQL changes to the child repository as follows: + + +UPDATE config SET name='parent-project-code' WHERE name='project-code'; +UPDATE config SET name='parent-project-name' WHERE name='project-name'; +INSERT INTO config(name,value) + VALUES('project-code',lower(hex(randomblob(20)))); +INSERT INTO config(name,value) + VALUES('project-name','CHILD-PROJECT-NAME'); + + +Modify the CHILD-PROJECT-NAME in the last statement to be the name of +the child project, of course. + +The repository is now a separate project, independent from its parent. +Clone the new project to the developers as needed. + +The child project and the parent project will not normally be able to sync +with one another, since they are now separate projects with distinct +project codes. However, if the +"--from-parent-project" command-line option is provided to the +"[/help/pull|fossil pull]" command in the child, and the URL of +parent repository is also provided on the command-line, then updates to +the parent project that occurred after the child was created will be added +to the child repository. Thus, by periodically doing a +pull --from-parent-project, the child project is able to stay up to date +with all the latest changes in the parent. diff --git a/www/chroot.md b/www/chroot.md new file mode 100644 index 0000000..882a176 --- /dev/null +++ b/www/chroot.md @@ -0,0 +1,45 @@ +# The Server Chroot Jail + +If you run Fossil as root in any mode that [serves data on the +network][srv], and you're running it on Unix or a compatible OS, Fossil +will drop itself into a [`chroot(2)` jail][cj] shortly after starting +up. The usual reason for launching Fossil +as root is to allow it to bind to TCP port 80 for HTTP +service, since normal users are restricted to ports 1024 and higher. + +Fossil uses the owner of the Fossil repository file as its new user +ID when it drops root privileges. + +When Fossil enters a chroot jail, it needs to have all of its dependencies +inside the chroot jail in order to continue work. There are several +resources that need to be inside the chroot jail with Fossil in order for +Fossil to work correctly: + +* the repository file(s) + +* `/dev/null` — create it with `mknod(8)` inside the jail directory + ([Linux example][mnl], [OpenBSD example][obsd]) + +* `/dev/urandom` — ditto + +* `/proc` — you might need to mount this virtual filesystem inside the + jail on Linux systems that make use of [Fossil’s server load + shedding feature][fls] + +* any shared libraries your `fossil` binary is linked to, unless you + [configured Fossil with `--static`][bld] to avoid it + +Fossil does all of this as one of many layers of defense against +hacks and exploits. You can prevent Fossil from entering the chroot +jail using the --nojail option to the +[fossil server command](/help/server) +but you cannot make Fossil hold onto root privileges. Fossil always drops +root privilege before accepting inputs, for security. + + +[bld]: https://fossil-scm.org/home/doc/trunk/www/build.wiki +[cj]: https://en.wikipedia.org/wiki/Chroot +[fls]: ./loadmgmt.md +[mnl]: https://fossil-scm.org/forum/forumpost/90caff30cb +[srv]: ./server/ +[obsd]: ./server/openbsd/fastcgi.md#chroot diff --git a/www/ckout-workflows.md b/www/ckout-workflows.md new file mode 100644 index 0000000..6c36183 --- /dev/null +++ b/www/ckout-workflows.md @@ -0,0 +1,142 @@ +# Check-Out Workflows + +Because Fossil separates the concept of “check-out directory” from +“repository DB file,” it gives you the freedom to choose from several +working styles. Contrast Git, where the two concepts are normally +intermingled in a single working directory, which strongly encourages +the “update in place” working style. + + +## Multiple-Checkout Workflow + +With Fossil, it is routine to have multiple check-outs from the same +repository: + + fossil clone https://example.com/repo /path/to/repo.fossil + + mkdir -p ~/src/my-project/trunk + cd ~/src/my-project/trunk + fossil open /path/to/repo.fossil # implicitly opens “trunk” + + mkdir ../release + cd ../release + fossil open /path/to/repo.fossil release + + mkdir ../my-other-branch + cd ../my-other-branch + fossil open /path/to/repo.fossil my-other-branch + + mkdir ../scratch + cd ../scratch + fossil open /path/to/repo.fossil abcd1234 + + mkdir ../test + cd ../test + fossil open /path/to/repo.fossil 2019-04-01 + +Now you have five separate check-out directories: one each for: + +* trunk +* the latest tagged public release +* an alternate branch you’re working on +* a “scratch” directory for experiments you don’t want to do in the + other check-out directories; and +* a “test” directory where you’re currently running a long-running + test to evaluate a user bug report against the version as of last + April Fool’s Day. + +Each check-out operates independently of the others. + +This multiple-checkouts working style is especially useful when Fossil stores source code in programming languages +where there is a “build” step that transforms source files into files +you actually run or distribute. Contrast a switch-in-place workflow, +where you have to rebuild all outputs from the source files +that differ between those versions whenever you switch versions. In the above model, +you switch versions with a “`cd`” command instead, so that you only have +to rebuild outputs from files you yourself change. + +This style is also useful when a check-out directory may be tied up with +some long-running process, as with the “test” example above, where you +might need to run an hours-long brute-force replication script to tickle +a [Heisenbug][hb], forcing it to show itself. While that runs, you can +open a new terminal tab, “`cd ../trunk`”, and get back +to work. + +[hb]: https://en.wikipedia.org/wiki/Heisenbug + + + +## Single-Checkout Workflows + +Nevertheless, it is possible to work in a more typical Git sort of +style, switching between versions in a single check-out directory. + +#### The Idiomatic Fossil Way + +The most idiomatic way is as follows: + + fossil clone https://example.com/repo /path/to/repo.fossil + mkdir work-dir + cd work-dir + fossil open /path/to/repo.fossil + ...work on trunk... + + fossil update my-other-branch + ...work on your other branch in the same directory... + +Basically, you replace the `cd` commands in the multiple checkouts +workflow above with `fossil up` commands. + + +#### Opening a Repository by URI + +You can instead open the repo’s URI directly: + + mkdir work-dir + cd work-dir + fossil open https://example.com/repo + +Now you have “trunk” open in `work-dir`, with the repo file stored as +`repo.fossil` in that same directory. + +Users of Git may be surprised that it doesn’t create a directory for you +and that you `cd` into it *before* the clone-and-open step, not after. +This is because we’re overloading the “open” command, which already had +the behavior of opening into the current working directory. Changing it +to behave like `git clone` would therefore make the behavior surprising +to Fossil users. (See [our discussions][caod] if you want the full +details.) + + +#### Git-Like Clone-and-Open + +Fossil also supports a more Git-like alternative: + + fossil clone https://fossil-scm.org/fossil + cd fossil + +This results in a `fossil.fossil` repo DB file and a `fossil/` working +directory. + +Note that our `clone URI` behavior does not commingle the repo and +check-out, solving our major problem with the Git design. + +If you want the repo to be named something else, adjust the URL: + + fossil clone https://fossil-scm.org/fossil/fsl + +That gets you `fsl.fossil` checked out into `fsl/`. + +For sites where the repo isn’t served from a subdirectory like this, you +might need another form of the URL. For example, you might have your +repo served from `dev.example.com` and want it cloned as `my-project`: + + fossil clone https://dev.example.com/repo/my-project + +The `/repo` addition is the key: whatever comes after is used as the +repository name. [See the docs][clone] for more details. + +[caod]: https://fossil-scm.org/forum/forumpost/3f143cec74 +[clone]: /help/clone + +
diff --git a/www/co-vs-up.md b/www/co-vs-up.md new file mode 100644 index 0000000..b0e74ca --- /dev/null +++ b/www/co-vs-up.md @@ -0,0 +1,31 @@ +# Checkout vs Update + +Fossil has two commands that look like they do the same thing on initial +examination, [`fossil update`][up] and [`fossil checkout`][co], but +there are several key differences: + +1. `fossil checkout` aborts if there are changed files in the local + directory unless you give the `--force` option, whereas + `fossil update` merges upstream changes with your local changes. + Since Fossil tends to follow the CVS command design, and CVS + popularized the [merge on update][cvsmu] workflow, we expect that + Fossil’s update behavior is more likely to be what you want. + +2. Update triggers an autosync attempt; checkout does not. + +3. Several features in `fossil update` do not exist in + `fossil checkout`, so developing a habit to type `fossil up` + means you’re more likely to have the features you want at hand. + +4. Inversely, the `fossil checkout --keep` feature doesn’t exist in + `fossil update`, but it’s a rarely-needed operation, so it doesn’t + provide a good reason to develop a habit of using `fossil checkout` + instead. + +In summary, these are two separate commands; neither is an alias for the +other. They overlap enough that they can be used interchangeably for +some use cases, but `update` is more powerful and more broadly useful. + +[co]: /help/checkout +[cvsmu]: http://web.mit.edu/gnu/doc/html/cvs_7.html#SEC37 +[up]: /help/update diff --git a/www/collisions.ipynb b/www/collisions.ipynb new file mode 100755 index 0000000..644786e --- /dev/null +++ b/www/collisions.ipynb @@ -0,0 +1,139 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "*Pull in the packages we need:*" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "require(stats)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Parameters\n", + "\n", + "* **cpd** - Checkins per day. Defaults to 20.\n", + "\n", + "* **days** - Number of days to simulate. Defaults to 10000, or roughly\n", + " 40 working years.\n", + "\n", + "* **winSz** - Size of the commit window as a fraction of `workSec`. Defaults\n", + " to 0.01%, which is roughly 3 seconds for the default 8-hour work day, a\n", + " a long-ish commit time for Fossil.\n", + "\n", + "* **workSec** - Seconds in working day, defaulting to 8 hours. This value\n", + " only affects the reporting output, not the results of the underlying\n", + " simulation. It's a scaling parameter to humanize the results.\n", + " \n", + "* **spread** - Adjustment factor in computing the standard deviation for our \n", + " normally-distributed random numbers. The default gives \"nice\" distributions,\n", + " spread over the working day with a low chance of generating values outside\n", + " the working day.\n", + " \n", + " The smaller this value gets (<4), the more spread out the checkins, and\n", + " so the lower the chance of collisions. You might want to decrease it a bit\n", + " to simulate early and late workers.\n", + " \n", + " As you increase this value (>4) you're simulating a work environment\n", + " where people tend to check their work in closer and closer to mid-day,\n", + " which increases the chance of collision." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "collisions <- function(\n", + " cpd = 20,\n", + " days = 10000,\n", + " winSz = 0.01 / 100,\n", + " workSec = 8 * 60 * 60,\n", + " spread = 4)\n", + "{\n", + " cat(\"Running simulation...\\n\")\n", + "\n", + " day = 0\n", + " collisions = 0\n", + " winSec = workSec * winSz\n", + " mean = workSec / 2\n", + " sd = workSec / spread\n", + "\n", + " while (day < days) {\n", + " # Create the commit time vector as random values in a normal\n", + " # distribution.\n", + " times = sort(rnorm(cpd, mean, sd))\n", + "\n", + " # Are there any pairs in the time vector that are within the\n", + " # passed window size? If so, that's a Fossil checkin collision.\n", + " i = 1\n", + " while (i < cpd) {\n", + " if (abs(times[i] - times[i + 1]) < winSec) {\n", + " collisions = collisions + 1\n", + " }\n", + " i = i + 1\n", + " }\n", + " \n", + " day = day + 1\n", + " }\n", + " \n", + " cat(\"Found\", collisions, \"collisions in\", days, (workSec / 3600),\n", + " \"hour working days with\", winSec, \"second windows.\\n\")\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "*Run the following cell, possibly changing parameters documented above:*" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Running simulation...\n", + "Found 422 collisions in 10000 8 hour working days with 2.88 second windows.\n" + ] + } + ], + "source": [ + "collisions()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "R", + "language": "R", + "name": "ir" + }, + "language_info": { + "codemirror_mode": "r", + "file_extension": ".r", + "mimetype": "text/x-r-source", + "name": "R", + "pygments_lexer": "r", + "version": "3.3.2" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/www/colordiff.md b/www/colordiff.md new file mode 100644 index 0000000..5d321e5 --- /dev/null +++ b/www/colordiff.md @@ -0,0 +1,104 @@ +# Colorized Diffs + +The oldest and most widely compatible method to get colorized diffs in +Fossil is to use its web UI: + + fossil ui --page '/vdiff?from=2024-04-01&to=trunk' + +That syntax is admittedly awkward, and it doesn’t work where “from” is +the current checkout. Fortunately, there are many other methods to get +colorized `diff` output from Fossil. + + + +## `fossil diff -b` + +This produces a graphical diff in HTML format and sends it to the +user’s default web browser for viewing. + + + + +## `fossil diff -tk` + +You may be surprised to learn that the prior feature doesn’t use any of +the skinning or chrome from Fossil UI. This is because it is meant as a +functional replacement for an older method of getting colorized diffs, +“`fossil diff -tk`”. The feature was added after Apple stopped shipping +Tcl/Tk in macOS, and the third-party replacements often failed to work +correctly. It’s useful on other platforms as well. + + + +## Delegate to Git + +It may be considered sacrilege by some, but the most direct method for +those who want Git-like diff behavior may be to delegate diff behavior +to Git: + + fossil set --global diff-command 'git diff --no-index' + +The flag permits it to diff files that aren’t inside a Git repository. + + + +## GNU Diffutils + +If your system is from 2016 or later, it may include [GNU Diffutils][gd] +3.4 or newer, which lets you say: + + fossil set --global diff-command 'diff -dwu --color=always' + +You might think you could give `--color=auto`, but that fails with +commands like “`fossil diff | less`” since the pipe turns the output +non-interactive from the perspective of the underlying `diff` instance. + +This use of unconditional colorization means you will then have to +remember to add the `-i` option to `fossil diff` commands when producing +`patch(1)` files or piping diff output to another command that doesn’t +understand ANSI escape sequences, such as [`diffstat`][ds]. + +[ds]: https://invisible-island.net/diffstat/ +[gd]: https://www.gnu.org/software/diffutils/ + + + +## Bat, the Cat with Wings + +We can work around the `--color=auto` problem by switching from GNU less +as our pager to [`bat`][bat], as it can detect GNU diff output and +colorize it for you: + + fossil set --global diff-command 'diff -dwu --color=auto' + fossil diff | bat + +In this author’s experience, that works a lot more reliably than GNU +less’s ANSI color escape code handling, even when you set `LESS=-R` in +your environment. + +The reason we don’t leave the `diff-command` unset in this case is that +Fossil produces additional lines at the start which confuse the diff +format detection in `bat`. Forcing output through an external diff +command solves that. It also means that if you forget to pipe the output +through `bat`, you still get colorized output from GNU diff. + +[bat]: https://github.com/sharkdp/bat + + + +## Colordiff + +A method that works on systems predating GNU diffutils 3.4 or the +widespread availability of `bat` is to install [`colordiff`][cdurl], as +it is included in [many package systems][cdpkg], including ones for +outdated OSes. That then lets you say: + + fossil set --global diff-command 'colordiff -dwu' + +The main reason we list this alternative last is that it has the same +limitation of unconditional color as [above](#diffutils). + +[cdurl]: https://www.colordiff.org/ +[cdpkg]: https://repology.org/project/colordiff/versions + +
diff --git a/www/concepts.wiki b/www/concepts.wiki new file mode 100644 index 0000000..56bd522 --- /dev/null +++ b/www/concepts.wiki @@ -0,0 +1,472 @@ +Fossil Concepts + +

1.0 Introduction

+ +[./index.wiki | Fossil] is a +[http://en.wikipedia.org/wiki/Software_configuration_management | software configuration management] system. +Fossil is software that is designed to control and track the +development of a software project and to record the history +of the project. +There are many such systems in use today. Fossil strives to +distinguish itself from the others by being extremely simple +to setup and operate. + +This document is intended as a quick introduction to the concepts +behind Fossil. + +See also: + + * [./glossary.md | Glossary] + * [./quickstart.wiki | Quick start guide] + +

2.0 Composition Of A Project

+ + +R1: cylinder "Remote" "Repository" fill 0xadd8e6 rad 70% +R2: cylinder same "Remote" "Repository" at 2.5*R1.wid right of R1 + spline <-> from R1.e to 0.6 then to 0.4 then to R2.w + text "HTTPS" at .5 +R3: cylinder same "Local" "Repository" fill 0x90ee90 \ + at dist(R1.e,R2.w) below .5 + spline <-> from .5 to 0.6 to 0.5 to .5 \ + "HTTPS" above behind R1 + spline <-> from R2.sw to .6 to .5 to R3.ne "HTTPS" ljust +T1: line from 1.0cm heading 200 from R3.sw go 2.2cm heading 150 then 2.2cm west close \ + fill 0xffff00 "Local" below "Source Tree" below +T2: line from 1.0cm heading 160 from R3.se same "Local" below "Source Tree" below + line <-> from R3.sw to T1.start + line <-> from R3.se to T2.start + + +A software project normally consists of a "source tree". +A source tree is a hierarchy of files that are used to generate +the end product. The source tree changes over time as the +software grows and expands and as features are added and bugs +are fixed. A snapshot of the source tree at any point in time +is called a "version" or "revision" or a "baseline" of the product. +In Fossil, we use the name "check-in". + +A "repository" is a database that contains copies of all historical +check-ins for a project. Check-ins are normally stored in the +repository in a highly space-efficient compressed format (delta encoding). +But that is an implementation detail that you the user need not worry over. +Think of the repository as a safe place where all your old check-ins are +securely stored away and available for retrieval whenever you need +them. + +A repository in Fossil is a single file on your disk. This file +might be rather large (dozens or hundreds of megabytes for a large +or long running project) but it is nevertheless just a file. You +can move it around, rename it, write it out to a memory stick, or +do anything else you normally do with files. + +Each source tree that is controlled by Fossil is associated with +a single repository on the local disk drive. You can tie two or more +source trees to a single repository if you want (though one +tree per repository is the most common configuration.) So a +single repository can be associated with many source trees, but +each source tree is associated with only one repository. + +Fossil source trees may not overlap. A Fossil source tree is identified +by a file named "_FOSSIL_" (or ".fslckout", but this article will always +use the name "_FOSSIL_") in the root directory of the source tree. Every +file that is a sibling of _FOSSIL_ and every file in every subfolder is +considered potentially a part of the source tree. The _FOSSIL_ file +contains (among other things) the pathname of the repository with which +the source tree is associated. On the other hand, the repository has +no record of its source trees. So you are free to delete a source tree +or move it around without consequence. But if you move or rename or +delete a repository, then any source trees associated with that repository +will no longer be able to locate their repository and will stop working. + +When multiple developers are working on the same project, each +developer typically has his or her own local repository and an associated +source tree in which to work. Developers share their work by +"syncing" the content of their local repositories either directly +or through a central server. Changes can "push" from the local +repository into a remote repository. Or changes can "pull" from a +remote repository into a local repository. Or one can do a "sync" +which is a shortcut for doing both a push and a pull at the same time. +Fossil also has the concept of "cloning". A "clone" is like a "pull", +except that instead of beginning with an existing local repository, +a clone begins with nothing and creates a new local repository that +is a duplicate of a remote repository. + +Communication between repositories is normally via HTTPS. (SSH is also +supported, as is unencrypted HTTP.) Remote +repositories are identified by URL. You can also point a web browser +at a repository and get human-readable status, history, and tracking +information about the project. + +

2.1 Identification Of Artifacts

+ +A particular version of a particular file is called an "artifact". Each +artifact has a universally unique name which is the SHA1 or SHA3-256 hash of the +content of that file expressed as either 40 or 64 characters of +lower-case hexadecimal. (See the [./hashpolicy.wiki|hash policy +document] for information on which algorithm is used, when.) Such a hash +is referred to as the Artifact ID. These hash algorithms were created +with Fossil's purpose in mind: to provide a highly forgery-resistant +identifier for a blob of data, such as a file. Given any file, it is +simple to find the artifact ID for that file. But given an artifact ID, +it is computationally intractable to generate a file that will have that +same artifact ID. + +Artifact IDs look something like this: + +
+6089f0b563a9db0a6d90682fe47fd7161ff867c8
+59712614a1b3ccfd84078a37fa5b606e28434326
+19dbf73078be9779edd6a0156195e610f81c94f9
+b4104959a67175f02d6b415480be22a239f1f077
+997c9d6ae03ad114b2b57f04e9eeef17dcb82788
+
+ +When referring to an artifact using Fossil, you can use a unique +prefix of the artifact ID that is four characters or longer. This saves +a lot of typing. When displaying artifact IDs, Fossil will usually only +show the first 10 digits since that is normally enough to uniquely +identify a file. + +Changing (or adding or removing) a single byte in a file results +in a completely different artifact ID. And since the artifact ID is the name of +the artifact, making any change to a file results in a new artifact. +In this way, artifacts are immutable. + +A repository is really just an unordered collection of +artifacts. New artifacts can be added to the repository, but +existing artifacts can never be removed. (Well, almost never. There +is a [./shunning.wiki | "shunning"] mechanism that allows spam or other +inappropriate content to be removed if absolutely necessary, but such +removal is discouraged.) +Fossil is designed in +such a way that it can be handed a set of artifacts in any +order and it can figure out the relationship between those +artifacts and reconstruct the complete development history of +a software project. + +

2.2 Manifests

+ +Associated with every check-in is a special file called the +[./fileformat.wiki#manifest| "manifest"]. The manifest is a +listing of all other files in +that source tree. The manifest contains the (complete) artifact ID +of the file and the name of the file as it appears on disk, +and thus serves as a mapping from artifact ID to disk name. The artifact ID +of the manifest is the identifier for the entire check-in. When +you look at a "timeline" of changes in Fossil, the ID associated +with each check-in or commit is really just the artifact ID of the +manifest for that check-in. + +The manifest file is not normally a real file on disk. Instead, +the manifest is computed in memory by Fossil whenever it needs it. +However, the "fossil setting manifest on" command will cause the +manifest file to be materialized to disk, if desired. Both Fossil +itself, and SQLite cause the manifest file to be materialized to disk +so that the makefiles for these projects can read the manifest and +embed version information in generated binaries. + +Fossil automatically generates a manifest whenever you "commit" +a new check-in. So this is not something that you, the developer, +need to worry with. The format of a manifest is intentionally +designed to be simple to parse, so that if +you want to read and interpret a manifest, either by hand or +with a script, that is easy to do. But you will probably never +need to do so. + +In addition to identifying all files in the check-in, a +manifest also contains a check-in comment, the date and time +when the check-in was established, who created the check-in, +and links to other check-ins from which the current check-in +is derived. There are also a couple of checksums used to verify +the integrity of the check-in. And the whole manifest might +be PGP clearsigned. + +

2.3 Key concepts

+ +
    +
  • A check-in is a set of files arranged + in a hierarchy.
    • +
  • A repository keeps a record of historical check-ins.
    • +
  • Repositories share their changes using push, pull, + sync, and clone.
    • +
  • A particular version of a particular file is an artifact + that is identified by an artifact ID.
    • +
  • Artifacts tracked by Fossil are inherently immutable.
    • +
  • Fossil automatically generates a manifest file that identifies + every artifact in a check-in.
    • +
  • The artifact ID of the manifest is the identifier of the check-in.
    • +
+ +

3.0 Fossil - The Program

+ +Fossil is software. The implementation of Fossil is in the form +of a single executable named "fossil" (or "fossil.exe" on Windows). +To install Fossil on your system, +all you have to do is obtain a copy of this one executable file (either +by downloading a +pre-compiled version +or [./build.wiki | compiling it yourself]) and then +putting that file somewhere on your PATH. + +Fossil is completely self-contained. It is not necessary to +install any other software in order to use Fossil. You do not need +CVS, gzip, diff, rsync, Python, Perl, Tcl, Java, Apache, PostgreSQL, MySQL, +SQLite, patch, or any similar software on your system in order to use +Fossil effectively. You will want to have some kind of text editor +for entering check-in comments. Fossil will use whatever text editor +is identified by your VISUAL environment variable. Fossil will also +use GPG to clearsign your manifests if you happen to have it installed, +but Fossil will skip that step if GPG is missing from your system. +You can optionally set up Fossil to use external "diff" programs, +though Fossil has an excellent built-in "diff" algorithm that works +fine for most people. If you happen to have Tcl/Tk installed on your +system, Fossil will use it to generate a graphical "diff" display when +you use the --tk option to the "diff" command, but this too is entirely +optional. + + +To uninstall Fossil, simply delete the executable. + +To upgrade an older version of Fossil to a newer version, just +replace the old executable with the new one. You might need to +run "fossil all rebuild" to restructure your repositories after +an upgrade. Running "all rebuild" never hurts, so when upgrading it +is a good policy to run it even if it is not strictly necessary. + +To use Fossil, simply type the name of the executable in your +shell, followed by one of the various built-in commands and +arguments appropriate for that command. For example: + +
fossil help
+ +In the next section, when we say things like "use the help +command" we mean to use the command name "help" as the first +token after the name of the Fossil executable, as shown above. + +

4.0 Workflow

+ + + down +R1: cylinder "Remote" "Repository" fill 0xadd8e6 rad 70% + move 150% +R2: cylinder same "Local" "Repository" fill 0x90ee90 + move 120% +T1: line go 2.2cm heading 150 then 2.2cm west close \ + fill 0xffff00 "Local" below "Source Tree" below + arrow from R2.n+(-0.25cm,+0.25cm) to R1.s+(-0.25cm,-0.25cm) \ + "push " rjust + arrow from R1.s+(+0.25cm,-0.25cm) to R2.n+(+0.25cm,+0.25cm) \ + " pull" ljust " clone" ljust + arrow from T1.start+(-0.25cm,+0cm) to R2.s+(-0.25cm,-0.25cm) \ + "commit " rjust + arrow from R2.s+(+0.25cm,-0.25cm) to T1.start+(+0.25cm,+0cm) \ + " open" ljust " update" ljust " merge" ljust + + + +Fossil has two modes of operation: "autosync" and +"manual-merge" +Autosync mode is reminiscent of CVS or SVN in that it automatically +keeps your changes in synchronization with your co-workers through +the use of a central server. The manual-merge mode is the standard workflow +for GIT or Mercurial in that your local repository develops +independently of your coworkers and you share and merge your changes manually. +An interesting feature of Fossil is that it supports both autosync +and manual-merge work flows. + +The default setting for Fossil is to be in autosync mode. You +can change the autosync setting or check the current autosync +setting using commands like: + +
+fossil setting autosync on
+fossil setting autosync off
+fossil settings
+
+ +By default, Fossil runs with autosync mode turned on. The +authors find that projects run more smoothly in autosync mode since +autosync helps to prevent pointless forking and merging and helps keeps +all collaborators working on exactly the same code rather than on their +own personal forks of the code. In the author's view, manual-merge mode +should be reserved for disconnected operation. + +

4.1 Autosync Workflow

+ +
    +
  1. +Establish a local repository using either the new command +to start a new project, or the clone command to make a clone +of a repository for an existing project. +
    2. + +
  2. +Establish one or more source trees using +the open command with the name of the repository file as its +argument. +
    3. + +
  3. +The open command in the previous step populates your local source +tree with a copy of the latest check-in. Usually this is what you want. +In the rare cases where it is not, use the update command to +switch to a different check-in. +Use the timeline or leaves commands +to identify alternative check-ins to switch to. +
    4. + +
  4. +Edit the code. Add new files to the source tree using the add +command. Omit files from future check-ins using the rm command. +(Even when you remove files from future check-ins, those files continue +to exist in historical check-ins.) Test your changes. +
    5. + +
  5. +Create a new check-in using the commit command. You will be prompted +for a check-in comment and also for your GPG key if you have GPG installed. +The commit copies the edits you have made in your local source +tree into your local repository. After your commit completes, Fossil will +automatically push your changes back to the server +you cloned from or whatever server you most recently synced with. +
    6. + +
  6. +When your coworkers make their own changes, you can merge those changes +into your local local source tree using the update command. +In autosync mode, update will first go back to the server you +cloned from or with which you most recently synced, and pull down all +recent changes into your local repository. Then it will merge recent +changes into your local source tree. If you do an update and +find that it messes something up in your source tree (perhaps a co-worker +checked in incompatible changes) you can use the undo command +to back out the changes. +
    7. + +
  7. +Repeat all of the above until you have generated great software. +
    8. +
+ +

4.2 Manual-Merge Workflow

+ +When autosync is disabled, the commit command is decoupled from +push and the update command is decoupled from pull. +That means you have to do a few extra steps in order to accomplish the +push and pull tasks manually. + +
    +
  1. +Establish a local repository using either the new command +to start a new project, or the clone command to make a clone +of a repository for an existing project. The default setting for +a new repository is with autosync on, so you will need to turn it off +using the setting autosync off command with a -R option +to specify the repository. +
    2. + +
  2. +Establish one or more source trees by changing your working directory +to where you want the root of the source tree to be, then issuing +the open command with the name of the repository file as its +argument. +
    3. + +
  3. +The open command in the previous step populates your local source +tree with a copy of the latest check-in. Usually this is what you want. +In the rare cases where it is not, use the update command to +switch to a different check-in. +Use the timeline or leaves commands +to identify alternative check-ins to switch to. +
    4. + +
  4. +Edit the code. Add new files to the source tree using the add +command. Omit files from future check-ins using the rm command. +(Even when you remove files from future check-ins, those files continue +to exist in historical check-ins.) Test your changes. +
    5. + +
  5. +Create a new check-in using the commit command. You will be prompted +for a check-in comment and also for your GPG key if you have GPG installed. +The commit copies the edits you have made in your local source +tree into your local repository. +
    6. + +
  6. +Use the push command to push your changes out to a server +where your co-workers can access them. +
    7. + +
  7. +When co-workers make their own changes, use the pull command +to pull those changes into your local repository. Note that pull +does not move the changes into your local source tree, only into your +local repository. +
    8. + +
  8. +Once changes are in your local repository, use +the update command to merge them to your local source tree. +If you merge in some changes and find that the changes do not work out +or are not to your liking, you can back out the changes using the +undo command. +
    9. + +
  9. +If two or more people ran "commit" against the same check-in, this will +result in a [./branching.wiki | fork] which you may want to resolve by +running merge followed by another commit. +
    10. + +
  10. +Repeat all of the above until you have generated great software. +
    11. +
+ +

5.0 Setting Up A Fossil Server

+ +With other configuration management software, setting up a server is +a lot of work and normally takes time, patience, and a lot of system +knowledge. Fossil is designed to avoid this frustration. Setting up +a server with Fossil is ridiculously easy. You have four options: + + # Stand-alone server. + Simply run the [/help/server|fossil server] or + [/help/ui|fossil ui] command from the command-line. +

+ # CGI. + Install a 2-line CGI script on a CGI-enabled web-server like Apache. +

+ # SCGI. + Start an SCGI server using the + [/help/server| fossil server --scgi] command for handling + SCGI requests from web-servers like Nginx. +

+ # Inetd or Stunnel. + Configure programs like inetd, xinetd, or stunnel to hand off HTTP requests + directly to the [/help/http|fossil http] command. + +See the [./server/ | How To Configure A Fossil Server] document +for details. + +

6.0 Review Of Key Concepts

+ +
    +
  • The fossil program is a self-contained stand-alone executable. + Just put it somewhere on your PATH to install it.
    • +
  • Use the clone or new commands to create a new repository.
    • +
  • Use the open command to create a new source tree.
    • +
  • Use the add and rm or delete commands to add and + remove files from the local source tree.
    • +
  • Use the commit command to create a new check-in.
    • +
  • Use the update command to merge in changes from others.
    • +
  • The push and pull commands can be used to share changes + manually, but these things happen automatically in the default + autosync mode.
    • +
diff --git a/www/contact.md b/www/contact.md new file mode 100644 index 0000000..75e62ed --- /dev/null +++ b/www/contact.md @@ -0,0 +1,14 @@ +# Contact Information + +## Questions, Suggestions, and Bug Reports + +The developers for Fossil monitor the [Fossil Forum][1]. Post there +with questions, improvement suggestions, and/or bug reports. + +[1]: https://fossil-scm.org/forum/forum + +## Security Problems and Vulnerabilities + +If you think you have discovered a security vulnerability in Fossil +and want to report the problem privately, send +email to "drh at sqlite dot org". diff --git a/www/containers.md b/www/containers.md new file mode 100644 index 0000000..ac5f02f --- /dev/null +++ b/www/containers.md @@ -0,0 +1,996 @@ +# OCI Containers + +This document shows how to build Fossil into [OCI] compatible containers +and how to use those containers in interesting ways. We start off using +the original and still most popular container development and runtime +platform, [Docker], but since you have more options than that, we will +show some of these options later on. + +[Docker]: https://www.docker.com/ +[OCI]: https://opencontainers.org/ + + +## 1. Quick Start + +Fossil ships a `Dockerfile` at the top of its source tree, +[here][DF], which you can build like so: + + $ docker build -t fossil . + +If the image built successfully, you can create a container from it and +test that it runs: + + $ docker run --name fossil -p 9999:8080/tcp fossil + +This shows us remapping the internal TCP listening port as 9999 on the +host. This feature of OCI runtimes means there’s little point to using +the “`fossil server --port`” feature inside the container. We can let +Fossil default to 8080 internally, then remap it to wherever we want it +on the host instead. + +Our stock `Dockerfile` configures Fossil with the default feature set, +so you may wish to modify the `Dockerfile` to add configuration options, +add APK packages to support those options, and so forth. + +The Fossil `Makefile` provides two convenience targets, +“`make container-image`” and “`make container-run`”. The first creates a +versioned container image, and the second does that and then launches a +fresh container based on that image. You can pass extra arguments to the +first command via the Makefile’s `DBFLAGS` variable and to the second +with the `DCFLAGS` variable. (DB is short for “`docker build`”, and DC +is short for “`docker create`”, a sub-step of the “run” target.) +To get the custom port setting as in +second command above, say: + + $ make container-run DCFLAGS='-p 9999:8080/tcp' + +Contrast the raw “`docker`” commands above, which create an +_unversioned_ image called `fossil:latest` and from that a container +simply called `fossil`. The unversioned names are more convenient for +interactive use, while the versioned ones are good for CI/CD type +applications since they avoid a conflict with past versions; it lets you +keep old containers around for quick roll-backs while replacing them +with fresh ones. + +[DF]: /file/Dockerfile + + +## 2. Repository Storage Options + +If you want the container to serve an existing repository, there are at +least two right ways to do it. + +The wrong way is to use the `Dockerfile COPY` command, because by baking +the repo into the image at build time, it will become one of the image’s +base layers. The end result is that each time you build a container from +that image, the repo will be reset to its build-time state. Worse, +restarting the container will do the same thing, since the base image +layers are immutable. This is almost certainly not what you +want. + +The correct ways put the repo into the _container_ created from the +_image_, not in the image itself. + + +### 2.1 Storing the Repo Inside the Container + +The simplest method is to stop the container if it was running, then +say: + + $ docker cp /path/to/my-project.fossil fossil:/museum/repo.fossil + $ docker start fossil + $ docker exec fossil chown -R 499 /museum + +That copies the local Fossil repo into the container where the server +expects to find it, so that the “start” command causes it to serve from +that copied-in file instead. Since it lives atop the immutable base +layers, it persists as part of the container proper, surviving restarts. + +Notice that the copy command changes the name of the repository +database. The container configuration expects it to be called +`repo.fossil`, which it almost certainly was not out on the host system. +This is because there is only one repository inside this container, so +we don’t have to name it after the project it contains, as is +traditional. A generic name lets us hard-code the server start command. + +If you skip the “chown” command above and put “`http://localhost:9999/`” +into your browser, expecting to see the copied-in repo’s home page, you +will get an opaque “Not Found” error. This is because the user and group +ID of the file will be that of your local user on the container’s host +machine, which is unlikely to map to anything in the container’s +`/etc/passwd` and `/etc/group` files, effectively preventing the server +from reading the copied-in repository file. 499 is the default “`fossil`” +user ID inside the container, causing Fossil to run with that user’s +privileges after it enters the chroot. (See [below](#args) for how to +change this default.) You don’t have to restart the server after fixing +this with `chmod`: simply reload the browser, and Fossil will try again. + + +### 2.2 Storing the Repo Outside the Container + +The simple storage method above has a problem: containers are +designed to be killed off at the slightest cause, rebuilt, and +redeployed. If you do that with the repo inside the container, it gets +destroyed, too. The solution is to replace the “run” command above with +the following: + + $ docker run \ + --publish 9999:8080 \ + --name fossil-bind-mount \ + --volume ~/museum:/museum \ + fossil + +Because this bind mount maps a host-side directory (`~/museum`) into the +container, you don’t need to `docker cp` the repo into the container at +all. It still expects to find the repository as `repo.fossil` under that +directory, but now both the host and the container can see that repo DB. + +Instead of a bind mount, you could instead set up a separate +[volume](https://docs.docker.com/storage/volumes/), at which point you +_would_ need to `docker cp` the repo file into the container. + +Either way, files in these mounted directories have a lifetime +independent of the container(s) they’re mounted into. When you need to +rebuild the container or its underlying image — such as to upgrade to a +newer version of Fossil — the external directory remains behind and gets +remapped into the new container when you recreate it with `--volume/-v`. + + +#### 2.2.1 WAL Mode Interactions + +You might be aware that OCI containers allow mapping a single file into +the repository rather than a whole directory. Since Fossil repositories +are specially-formatted SQLite databases, you might be wondering why we +don’t say things like: + + --volume ~/museum/my-project.fossil:/museum/repo.fossil + +That lets us have a convenient file name for the project outside the +container while letting the configuration inside the container refer to +the generic “`/museum/repo.fossil`” name. Why should we have to name +the repo generically on the outside merely to placate the container? + +The reason is, you might be serving that repo with [WAL mode][wal] +enabled. If you map the repo DB alone into the container, the Fossil +instance inside the container will write the `-journal` and `-wal` files +alongside the mapped-in repository inside the container. That’s fine as +far as it goes, but if you then try using the same repo DB from outside +the container while there’s an active WAL, the Fossil instance outside +won’t know about it. It will think it needs to write *its own* +`-journal` and `-wal` files *outside* the container, creating a high +risk of [database corruption][dbcorr]. + +If we map a whole directory, both sides see the same set of WAL files. +[Testing](https://tangentsoft.com/sqlite/dir/walbanger?ci=trunk) +gives us a reasonable level of confidence that using WAL across a +container boundary is safe when used in this manner. + +[dbcorr]: https://www.sqlite.org/howtocorrupt.html#_deleting_a_hot_journal +[wal]: https://www.sqlite.org/wal.html + + +## 3. Security + +### 3.1 Why Not Chroot? + +Prior to 2023.03.26, the stock Fossil container relied on [the chroot +jail feature](./chroot.md) to wall away the shell and other tools +provided by [BusyBox]. It included that as a bare-bones operating system +inside the container on the off chance that someone might need it for +debugging, but the thing is, Fossil is self-contained, needing none of +that power in the main-line use cases. + +Our weak “you might need it” justification collapsed when we realized +you could restore this basic shell environment with a one-line change to +the `Dockerfile`, as shown [below](#run). + +[BusyBox]: https://www.busybox.net/BusyBox.html + + +### 3.2 Dropping Unnecessary Capabilities + +The example commands above create the container with [a default set of +Linux kernel capabilities][defcap]. Although Docker strips away almost +all of the traditional root capabilities by default, and Fossil doesn’t +need any of those it does take away, Docker does leave some enabled that +Fossil doesn’t actually need. You can tighten the scope of capabilities +by adding “`--cap-drop`” options to your container creation commands. + +Specifically: + +* **`AUDIT_WRITE`**: Fossil doesn’t write to the kernel’s auditing + log, and we can’t see any reason you’d want to be able to do that as + an administrator shelled into the container, either. Auditing is + something done on the host, not from inside each individual + container. + +* **`CHOWN`**: The Fossil server never even calls `chown(2)`, and our + image build process sets up all file ownership properly, to the + extent that this is possible under the limitations of our + automation. + + Curiously, stripping this capability doesn’t affect your ability to + run commands like “`chown -R fossil:fossil /museum`” when + you’re using bind mounts or external volumes — as we recommend + [above](#bind-mount) — because it’s the host OS’s kernel + capabilities that affect the underlying `chown(2)` call in that + case, not those of the container. + + If for some reason you did have to change file ownership of + in-container files, it’s best to do that by changing the + `Dockerfile` to suit, then rebuilding the container, since that + bakes the need for the change into your reproducible build process. + If you had to do it without rebuilding the container, [there’s a + workaround][capchg] for the fact that capabilities are a create-time + change, baked semi-indelibly into the container configuration. + +* **`FSETID`**: Fossil doesn’t use the SUID and SGID bits itself, and + our build process doesn’t set those flags on any of the files. + Although the second fact means we can’t see any harm from leaving + this enabled, we also can’t see any good reason to allow it, so we + strip it. + +* **`KILL`**: The only place Fossil calls `kill(2)` is in the + [backoffice], and then only for processes it created on earlier + runs; it doesn’t need the ability to kill processes created by other + users. You might wish for this ability as an administrator shelled + into the container, but you can pass the “`docker exec --user`” + option to run commands within your container as the legitimate owner + of the process, removing the need for this capability. + +* **`MKNOD`**: As of 2023.03.26, the stock container uses the + runtime’s default `/dev` node tree. Prior to this, we had to create + `/dev/null` and `/dev/urandom` inside [the chroot jail](#chroot), + but even then, these device nodes were created at build time and + were never changed at run time, so we didn’t need this run-time + capability even then. + +* **`NET_BIND_SERVICE`**: With containerized deployment, Fossil never + needs the ability to bind the server to low-numbered TCP ports, not + even if you’re running the server in production with TLS enabled and + want the service bound to port 443. It’s perfectly fine to let the + Fossil instance inside the container bind to its default port (8080) + because you can rebind it on the host with the + “`docker create --publish 443:8080`” option. It’s the container’s + _host_ that needs this ability, not the container itself. + + (Even the container runtime might not need that capability if you’re + [terminating TLS with a front-end proxy](./ssl.wiki#server). You’re + more likely to say something like “`-p localhost:12345:8080`” and then + configure the reverse proxy to translate external HTTPS calls into + HTTP directed at this internal port 12345.) + +* **`NET_RAW`**: Fossil itself doesn’t use raw sockets, and while + you could [swap out the run layer](#run) for something more + functional that *does* make use of raw sockets, there’s little call + for it. The best reason I can come up with is to be able to run + utilities like `ping` and `traceroute`, but since we aren’t doing + anything clever with the networking configuration, there’s no + particularly compelling reason to run these from inside the + container. If you need to ping something, do it on the host. + + If we did not take this hard-line stance, an attacker that broke + into the container and gained root privileges might use raw sockets + to do a wide array of bad things to any network the container is + bound to. + +* **`SETFCAP, SETPCAP`**: There isn’t much call for file permission + granularity beyond the classic Unix ones inside the container, so we + drop root’s ability to change them. + +All together, we recommend adding the following options to your +“`docker run`” commands, as well as to any “`docker create`” command +that will be followed by “`docker start`”: + + --cap-drop AUDIT_WRITE \ + --cap-drop CHOWN \ + --cap-drop FSETID \ + --cap-drop KILL \ + --cap-drop MKNOD \ + --cap-drop NET_BIND_SERVICE \ + --cap-drop NET_RAW \ + --cap-drop SETFCAP \ + --cap-drop SETPCAP + +In the next section, we’ll show a case where you create a container +without ever running it, making these options pointless. + +[backoffice]: ./backoffice.md +[defcap]: https://docs.docker.com/engine/security/#linux-kernel-capabilities +[capchg]: https://stackoverflow.com/a/45752205/142454 + + + +## 4. Extracting a Static Binary + +Our 2-stage build process uses Alpine Linux only as a build host. Once +we’ve got everything reduced to a single static Fossil binary, +we throw all the rest of it away. + +A secondary benefit falls out of this process for free: it’s arguably +the easiest way to build a purely static Fossil binary for Linux. Most +modern Linux distros make this [surprisingly difficult][lsl], but Alpine’s +back-to-basics nature makes static builds work the way they used to, +back in the day. If that’s all you’re after, you can do so as easily as +this: + + $ docker build -t fossil . + $ docker create --name fossil-static-tmp fossil + $ docker cp fossil-static-tmp:/bin/fossil . + $ docker container rm fossil-static-tmp + +The result is six or seven megs, depending on the CPU architecture you +build for. It’s built stripped. + +[lsl]: https://stackoverflow.com/questions/3430400/linux-static-linking-is-dead + + +## 5. Customization Points + +### 5.1 Fossil Version + +The default version of Fossil fetched in the build is the version in the +checkout directory at the time you run it. You could override it to get +a release build like so: + + $ docker build -t fossil --build-arg FSLVER=version-2.20 . + +Or equivalently, using Fossil’s `Makefile` convenience target: + + $ make container-image DBFLAGS='--build-arg FSLVER=version-2.20' + +While you could instead use the generic +“`release`” tag here, it’s better to use a specific version number +since container builders cache downloaded files, hoping to +reuse them across builds. If you ask for “`release`” before a new +version is tagged and then immediately after, you might expect to get +two different tarballs, but because the underlying source tarball URL +remains the same when you do that, you’ll end up reusing the +old tarball from cache. This will occur +even if you pass the “`docker build --no-cache`” option. + +This is why we default to pulling the Fossil tarball by checkin ID +rather than let it default to the generic “`trunk`” tag: so the URL will +change each time you update your Fossil source tree, forcing the builder to +pull a fresh tarball. + + +### 5.2 User & Group IDs + +The “`fossil`” user and group IDs inside the container default to 499. +Why? Regular user IDs start at 500 or 1000 on most Unix type systems, +leaving those below it for system users like this Fossil daemon owner. +Since it’s typical for these to start at 0 and go upward, we started at +500 and went *down* one instead to reduce the chance of a conflict to as +close to zero as we can manage. + +To change it to something else, say: + + $ make container-image DBFLAGS='--build-arg UID=501' + +This is particularly useful if you’re putting your repository on a +separate volume since the IDs “leak” out into the host environment via +file permissions. You may therefore wish them to mean something on both +sides of the container barrier rather than have “499” appear on the host +in “`ls -l`” output. + + +### 5.3 Container Engine + +Although the Fossil container build system defaults to Docker, we allow +for use of any OCI container system that implements the same interfaces. +We go into more details about this [below](#light), but +for now, it suffices to point out that you can switch to Podman while +using our `Makefile` convenience targets unchanged by saying: + + $ make CENGINE=podman container-run + + +### 5.4 Fossil Configuration Options + +You can use this same mechanism to enable non-default Fossil +configuration options in your build. For instance, to turn on +the JSON API and the TH1 docs extension: + + $ make container-image \ + DBFLAGS='--build-arg FSLCFG="--json --with-th1-docs"' + +If you also wanted [the Tcl evaluation extension](./th1.md#tclEval), +that brings us to [the next point](#run). + + +### 5.5 Elaborating the Run Layer + +If you want a basic shell environment for temporary debugging of the +running container, that’s easily added. Simply change this line in the +`Dockerfile`… + + FROM scratch AS run + +…to this: + + FROM busybox AS run + +Rebuild and redeploy to give your Fossil container a [BusyBox]-based +shell environment that you can get into via: + + $ docker exec -it -u fossil $(make container-version) sh + +That command assumes you built it via “`make container`” and are +therefore using its versioning scheme. + +You will likely want to remove the `PATH` override in the “RUN” stage +when doing this since it’s written for the case where everything is in +`/bin`, and that will no longer be the case with a more full-featured +“`run`” layer. As long as the parent layer’s `PATH` value contains +`/bin`, delegating to it is more likely the correct thing. + +Another useful case to consider is that you’ve installed a [server +extension](./serverext.wiki) and you need an interpreter for that +script. The first option above won’t work except in the unlikely case that +it’s written for one of the bare-bones script interpreters that BusyBox +ships.(^[BusyBox]’s `/bin/sh` is based on the old 4.4BSD Lite Almquist +shell, implementing little more than what POSIX specified in 1989, plus +equally stripped-down versions of `awk` and `sed`.) + +Let’s say the extension is written in Python. Because this is one of the +most popular programming languages in the world, we have many options +for achieving this. For instance, there is a whole class of +“[distroless]” images that will do this efficiently by changing +“`STAGE 2`” in the `Dockefile` to this: + + ## --------------------------------------------------------------------- + ## STAGE 2: Pare that back to the bare essentials, plus Python. + ## --------------------------------------------------------------------- + FROM cgr.dev/chainguard/python:latest + USER root + ARG UID=499 + ENV PATH "/sbin:/usr/sbin:/bin:/usr/bin" + COPY --from=builder /tmp/fossil /bin/ + COPY --from=builder /bin/busybox.static /bin/busybox + RUN [ "/bin/busybox", "--install", "/bin" ] + RUN set -x \ + && echo "fossil:x:${UID}:${UID}:User:/museum:/false" >> /etc/passwd \ + && echo "fossil:x:${UID}:fossil" >> /etc/group \ + && install -d -m 700 -o fossil -g fossil log museum + +You will also have to add `busybox-static` to the APK package list in +STAGE 1 for the `RUN` script at the end of that stage to work, since the +[Chainguard Python image][cgimgs] lacks a shell, on purpose. The need to +install root-level binaries is why we change `USER` temporarily here. + +Build it and test that it works like so: + + $ make container-run && + docker exec -i $(make container-version) python --version + 3.11.2 + +The compensation for the hassle of using Chainguard over something more +general purpose like changing the `run` layer to Alpine and then adding +a “`apk add python`” command to the `Dockerfile` +is huge: we no longer leave a package manager sitting around inside the +container, waiting for some malefactor to figure out how to abuse it. + +Beware that there’s a limit to this über-jail’s ability to save you when +you go and provide a more capable runtime layer like this. The container +layer should stop an attacker from accessing any files out on the host +that you haven’t explicitly mounted into the container’s namespace, but +it can’t stop them from making outbound network connections or modifying +the repo DB inside the container. + +[cgimgs]: https://github.com/chainguard-images/images/tree/main/images +[distroless]: https://www.chainguard.dev/unchained/minimal-container-images-towards-a-more-secure-future +[MTA]: https://en.wikipedia.org/wiki/Message_transfer_agent + + +### 5.6 Email Alerts + +The nature of our single static binary container precludes two of the +options for [sending email alerts](./alerts.md) from Fossil: + +* pipe to a command +* SMTP relay host + +There is no `/usr/sbin/sendmail` inside the container, and the container +cannot connect out to a TCP service on the host by default. + +While it is possible to get around the first lack by [elaborating the +run layer](#run), to inject a full-blown Sendmail setup into the +container would go against the whole idea of containerization. +Forwarding an SMTP relay port into the container isn’t nearly as bad, +but it’s still bending the intent behind containers out of shape. + +A far better option in this case is the “store emails in database” +method since the containerized Fossil binary knows perfectly well how to +write SQLite DB files without relying on any external code. Using the +paths in the configuration recommended above, the database path should +be set to something like `/museum/mail.db`. This, along with the use of +[bind mounts](#bind-mount) means you can have a process running outside +the container that passes the emails along to the host-side MTA. + +The included [`email-sender.tcl`](/file/tools/email-sender.tcl) script +works reasonably well for this, though in my own usage, I had to make +two changes to it: + +1. The shebang line at the top has to be `#!/usr/bin/tclsh` on my server. +2. I parameterized the `DBFILE` variable at the top thus: + + set DBFILE [lindex $argv 0] + +I then wanted a way to start this Tcl script on startup and keep it +running, which made me reach for systemd. My server is set to allow user +services to run at boot(^”Desktop” class Linuxes tend to disable that by +default under the theory that you don’t want those services to run until +you’ve logged into the GUI as that user. If you find yourself running +into this, [enable linger +mode](https://www.freedesktop.org/software/systemd/man/loginctl.html).) +so I was able to create a unit file called +`~/.local/share/systemd/user/alert-sender@.service` with these contents: + + [Unit] + Description=Fossil email alert sender for %I + + [Service] + WorkingDirectory=/home/fossil/museum + ExecStart=/home/fossil/bin/alert-sender %I/mail.db + Restart=always + RestartSec=3 + + [Install] + WantedBy=default.target + +I was then able to enable email alert forwarding for select repositories +after configuring them per [the docs](./alerts.md) by saying: + + $ systemctl --user daemon-reload + $ systemctl --user enable --now alert-sender@myproject + +Because this is a parameterized script and we’ve set our repository +paths predictably, you can do this for as many repositories as you need +to by passing their names after the “`@`” sign in the commands above. + + +## 6. Lightweight Alternatives to Docker + +Those afflicted with sticker shock at seeing the size of a [Docker +Desktop][DD] installation — 1.65 GB here — might’ve immediately +“noped” out of the whole concept of containers. The first thing to +realize is that when it comes to actually serving simple containers like +the ones shown above is that [Docker Engine][DE] suffices, at about a +quarter of the size. + +Yet on a small server — say, a $4/month ten gig Digital Ocean droplet — +that’s still a big chunk of your storage budget. It takes ~60:1 overhead +merely to run a Fossil server container? Once again, I wouldn’t +blame you if you noped right on out of here, but if you will be patient, +you will find that there are ways to run Fossil inside a container even +on entry-level cloud VPSes. These are well-suited to running Fossil; you +don’t have to resort to [raw Fossil service][srv] to succeed, +leaving the benefits of containerization to those with bigger budgets. + +For the sake of simple examples in this section, we’ll assume you’re +integrating Fossil into a larger web site, such as with our [Debian + +nginx + TLS][DNT] plan. This is why all of the examples below create +the container with this option: + + --publish 127.0.0.1:9999:8080 + +The assumption is that there’s a reverse proxy running somewhere that +redirects public web hits to localhost port 9999, which in turn goes to +port 8080 inside the container. This use of port +publishing effectively replaces the use of the +“`fossil server --localhost`” option. + +For the nginx case, you need to add `--scgi` to these commands, and you +might also need to specify `--baseurl`. + +Containers are a fine addition to such a scheme as they isolate the +Fossil sections of the site from the rest of the back-end resources, +thus greatly reducing the chance that they’ll ever be used to break into +the host as a whole. + +(If you wanted to be double-safe, you could put the web server into +another container, restricting it to reading from the static web +site directory and connecting across localhost to back-end dynamic +content servers such as Fossil. That’s way outside the scope of this +document, but you can find ready advice for that elsewhere. Seeing how +we do this with Fossil should help you bridge the gap in extending +this idea to the rest of your site.) + +[DD]: https://www.docker.com/products/docker-desktop/ +[DE]: https://docs.docker.com/engine/ +[DNT]: ./server/debian/nginx.md +[srv]: ./server/ + + +### 6.1 Stripping Docker Engine Down + +The core of Docker Engine is its [`containerd`][ctrd] daemon and the +[`runc`][runc] container runtime. Add to this the out-of-core CLI program +[`nerdctl`][nerdctl] and you have enough of the engine to run Fossil +containers. The big things you’re missing are: + +* **BuildKit**: The container build engine, which doesn’t matter if + you’re building elsewhere and shipping the images to the target. + A good example is using a container registry as an + intermediary between the build and deployment hosts. + +* **SwarmKit**: A powerful yet simple orchestrator for Docker that you + probably aren’t using with Fossil anyway. + +In exchange, you get a runtime that’s about half the size of Docker +Engine. The commands are essentially the same as above, but you say +“`nerdctl`” instead of “`docker`”. You might alias one to the other, +because you’re still going to be using Docker to build and ship your +container images. + +[ctrd]: https://containerd.io/ +[nerdctl]: https://github.com/containerd/nerdctl +[runc]: https://github.com/opencontainers/runc + + +### 6.2 Podman + +A lighter-weight [rootless][rl] [drop-in replacement][whatis] that +doesn’t give up the image builder is [Podman]. Initially created by +Red Hat and thus popular on that family of OSes, it will run on +any flavor of Linux. It can even be made to run [on macOS via Homebrew][pmmac] +or [on Windows via WSL2][pmwin]. + +On Ubuntu 22.04, the installation size is about 38 MiB, roughly a +tenth the size of Docker Engine. + +For our purposes here, the only thing that changes relative to the +examples at the top of this document are the initial command: + + $ podman build -t fossil . + $ podman run --name fossil -p 9999:8080/tcp fossil + +Your Linux package repo may have a `podman-docker` package which +provides a “`docker`” script that calls “`podman`” for you, eliminating +even the command name difference. With that installed, the `make` +commands above will work with Podman as-is. + +The only difference that matters here is that Podman doesn’t have the +same [default Linux kernel capability set](#caps) as Docker, which +affects the `--cap-drop` flags recommended above to: + + $ podman create \ + --name fossil \ + --cap-drop CHOWN \ + --cap-drop FSETID \ + --cap-drop KILL \ + --cap-drop NET_BIND_SERVICE \ + --cap-drop SETFCAP \ + --cap-drop SETPCAP \ + --publish 127.0.0.1:9999:8080 \ + localhost/fossil + $ podman start fossil + +[pmmac]: https://podman.io/getting-started/installation.html#macos +[pmwin]: https://github.com/containers/podman/blob/main/docs/tutorials/podman-for-windows.md +[Podman]: https://podman.io/ +[rl]: https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md +[whatis]: https://docs.podman.io/en/latest/index.html + + +### 6.3 `systemd-container` + +If even the Podman stack is too big for you, the next-best option I’m +aware of is the `systemd-container` infrastructure on modern Linuxes, +available since version 239 or so. Its runtime tooling requires only +about 1.4 MiB of disk space: + + $ sudo apt install systemd-container btrfs-tools + +That command assumes the primary test environment for +this guide, Ubuntu 22.04 LTS with `systemd` 249. For best +results, `/var/lib/machines` should be a btrfs volume, because +[`$REASONS`][mcfad]. For CentOS Stream 9 and other Red Hattish +systems, you will have to make several adjustments, which we’ve +collected [below](#nspawn-centos) to keep these examples clear. + +We’ll assume your Fossil repository stores something called +“`myproject`” within `~/museum/myproject/repo.fossil`, named according +to the reasons given [above](#repo-inside). We’ll make consistent use of +this naming scheme in the examples below so that you will be able to +replace the “`myproject`” element of the various file and path names. + +If you use [the stock `Dockerfile`][DF] to generate your +base image, `nspawn` won’t recognize it as containing an OS unless you +change the “`FROM scratch AS os`” line at the top of the second stage +to something like this: + + FROM gcr.io/distroless/static-debian11 AS os + +Using that as a base image provides all the files `nspawn` checks for to +determine whether the container is sufficiently close to a Linux VM for +the following step to proceed: + + $ make container + $ docker container export $(make container-version) | + machinectl import-tar - myproject + +Next, create `/etc/systemd/nspawn/myproject.nspawn`: + +---- + + [Exec] + WorkingDirectory=/ + Parameters=bin/fossil server \ + --baseurl https://example.com/myproject \ + --create \ + --jsmode bundled \ + --localhost \ + --port 9000 \ + --scgi \ + --user admin \ + museum/repo.fossil + DropCapability= \ + CAP_AUDIT_WRITE \ + CAP_CHOWN \ + CAP_FSETID \ + CAP_KILL \ + CAP_MKNOD \ + CAP_NET_BIND_SERVICE \ + CAP_NET_RAW \ + CAP_SETFCAP \ + CAP_SETPCAP + ProcessTwo=yes + LinkJournal=no + Timezone=no + + [Files] + Bind=/home/fossil/museum/myproject:/museum + + [Network] + VirtualEthernet=no + +---- + +If you recognize most of that from the `Dockerfile` discussion above, +congratulations, you’ve been paying attention. The rest should also +be clear from context. + +Some of this is expected to vary: + +* The references to `example.com` and `myproject` are stand-ins for + your actual web site and repository name. + +* The command given in the `Parameters` directive assumes you’re + setting up [SCGI proxying via nginx][DNT], but with adjustment, + it’ll work with the other repository service methods we’ve + [documented][srv]. + +* The path in the host-side part of the `Bind` value must point at the + directory containing the `repo.fossil` file referenced in said + command so that `/museum/repo.fossil` refers to your repo out + on the host for the reasons given [above](#bind-mount). + +That being done, we also need a generic `systemd` unit file called +`/etc/systemd/system/fossil@.service`, containing: + +---- + + [Unit] + Description=Fossil %i Repo Service + Wants=modprobe@tun.service modprobe@loop.service + After=network.target systemd-resolved.service modprobe@tun.service modprobe@loop.service + + [Service] + ExecStart=systemd-nspawn --settings=override --read-only --machine=%i bin/fossil + + [Install] + WantedBy=multi-user.target + +---- + +You shouldn’t have to change any of this because we’ve given the +`--setting=override` flag, meaning any setting in the nspawn file +overrides the setting passed to `systemd-nspawn`. This arrangement +not only keeps the unit file simple, it allows multiple services to +share the base configuration, varying on a per-repo level through +adjustments to their individual `*.nspawn` files. + +You may then start the service in the normal way: + + $ sudo systemctl enable fossil@myproject + $ sudo systemctl start fossil@myproject + +You should then find it running on localhost port 9000 per the nspawn +configuration file above, suitable for proxying Fossil out to the +public using nginx via SCGI. If you aren’t using a front-end proxy +and want Fossil exposed to the world via HTTPS, you might say this instead in +the `*.nspawn` file: + + Parameters=bin/fossil server \ + --cert /path/to/cert.pem \ + --create \ + --jsmode bundled \ + --port 443 \ + --user admin \ + museum/repo.fossil + +You would also need to un-drop the `CAP_NET_BIND_SERVICE` capability +to allow Fossil to bind to this low-numbered port. + +We use the `systemd` template file feature to allow multiple Fossil +servers running on a single machine, each on a different TCP port, +as when proxying them out as subdirectories of a larger site. +To add another project, you must first clone the base “machine” layer: + + $ sudo machinectl clone myproject otherthing + +That will not only create a clone of `/var/lib/machines/myproject` +as `../otherthing`, it will create a matching `otherthing.nspawn` file for you +as a copy of the first one. Adjust its contents to suit, then enable +and start it as above. + +[mcfad]: https://www.freedesktop.org/software/systemd/man/machinectl.html#Files%20and%20Directories + + +### 6.3.1 Getting It Working on a RHEL Clone + +The biggest difference between doing this on OSes like CentOS versus +Ubuntu is that RHEL (thus also its clones) doesn’t ship btrfs in +its kernel, thus ships with no package repositories containing `mkfs.btrfs`, which +[`machinectl`][mctl] depends on for achieving its various purposes. + +Fortunately, there are workarounds. + +First, the `apt install` command above becomes: + + $ sudo dnf install systemd-container + +Second, you have to hack around the lack of `machinectl import-tar`: + + $ rootfs=/var/lib/machines/fossil + $ sudo mkdir -p $rootfs + $ docker container export fossil | sudo tar -xf -C $rootfs - + +The parent directory path in the `rootfs` variable is important, +because although we aren’t able to use `machinectl` on such systems, the +`systemd-nspawn` developers assume you’re using them together; when you give +`--machine`, it assumes the `machinectl` directory scheme. You could +instead use `--directory`, allowing you to store the rootfs wherever +you like, but why make things difficult? It’s a perfectly sensible +default, consistent with the [LHS] rules. + +The final element — the machine name — can be anything +you like so long as it matches the nspawn file’s base name. + +Finally, since you can’t use `machinectl clone`, you have to make +a wasteful copy of `/var/lib/machines/myproject` when standing up +multiple Fossil repo services on a single machine. (This is one +of the reasons `machinectl` depends on `btrfs`: cheap copy-on-write +subvolumes.) Because we give the `--read-only` flag, you can simply +`cp -r` one machine to a new name rather than go through the +export-and-import dance you used to create the first one. + +[LHS]: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html +[mctl]: https://www.freedesktop.org/software/systemd/man/machinectl.html + + +### 6.3.2 What Am I Missing Out On? + +For all the runtime size savings in this method, you may be wondering +what you’re missing out on relative to Podman, which takes up +roughly 27× more disk space. Short answer: lots. Long answer: + +1. **Build system.** You’ll have to build and test your containers + some other way. This method is only suitable for running them + once they’re built. + +2. **Orchestration.** All of the higher-level things like + “compose” files, Docker Swarm mode, and Kubernetes are + unavailable to you at this level. You can run multiple + instances of Fossil, but on a single machine only and with a + static configuration. + +3. **Image layer sharing.** When you update an image using one of the + above methods, Docker and Podman are smart enough to copy only + changed layers. Furthermore, when you base multiple containers + on a single image, they don’t make copies of the base layers; + they can share them, because base layers are immutable, thus + cannot cross-contaminate. + + Because we use `systemd-nspawn --read-only`, we get *some* + of this benefit, particularly when using `machinectl` with + `/var/lib/machines` as a btrfs volume. Even so, the disk space + and network I/O optimizations go deeper in the Docker and Podman + worlds. + +4. **Tooling.** Hand-creating and modifying those `systemd` + files sucks compared to “`podman container create ...`” This + is but one of many affordances you will find in the runtimes + aimed at daily-use devops warriors. + +5. **Network virtualization.** In the scheme above, we turn off the + `systemd` private networking support because in its default mode, it + wants to hide containerized services entirely. While there are + [ways][ndcmp] to expose Fossil’s single network service port under + that scheme, it adds a lot of administration complexity. In the + big-boy container runtimes, `docker create --publish` fixes all this + up in a single option, whereas `systemd-nspawn --port` does + approximately *none* of that despite the command’s superficial + similarity. + + From a purely functional point of view, this isn’t a huge problem if + you consider the inbound service direction only, being external + connections to the Fossil service we’re providing. Since we do want + this Fossil service to be exposed — else why are we running it? — we + get all the control we need via `fossil server --localhost` and + similar options. + + The complexity of the `systemd` networking infrastructure’s + interactions with containers make more sense when you consider the + outbound path. Consider what happens if you enable Fossil’s + optional TH1 docs feature plus its Tcl evaluation feature. That + would enable anyone with the rights to commit to your repository the + ability to make arbitrary network connections on the Fossil host. + Then, let us say you have a client-server DBMS server on that same + host, bound to localhost for private use by other services on the + machine. Now that DBMS is open to access by a rogue Fossil committer + because the host’s loopback interface is mapped directly into the + container’s network namespace. + + Proper network virtualization would protect you in this instance. + +This author expects that the set of considerations is broader than +presented here, but that it suffices to make our case as it is: if you +can afford the space of Podman or Docker, we strongly recommend using +either of them over the much lower-level `systemd-container` +infrastructure. You’re getting a considerable amount of value for the +higher runtime cost; it isn’t pointless overhead. + +(Incidentally, these are essentially the same reasons why we no longer +talk about the `crun` tool underpinning Podman in this document. It’s +even more limited than `nspawn`, making it even more difficult to administer while +providing no runtime size advantage. The `runc` tool underpinning +Docker is even worse on this score, being scarcely easier to use than +`crun` while having a much larger footprint.) + +[ndcmp]: https://wiki.archlinux.org/title/systemd-networkd#Usage_with_containers + + +### 6.3.3 Violated Assumptions + +The `systemd-container` infrastructure has a bunch of hard-coded +assumptions baked into it. We papered over these problems above, +but if you’re using these tools for other purposes on the machine +you’re serving Fossil from, you may need to know which assumptions +our container violates and the resulting consequences. + +Some of it we discussed above already, but there’s one big class of +problems we haven’t covered yet. It stems from the fact that our stock +container starts a single static executable inside a bare-bones container +rather than “boot” an OS image. That causes a bunch of commands to fail: + +* **`machinectl poweroff`** will fail because the container + isn’t running dbus. + +* **`machinectl start`** will try to find an `/sbin/init` + program in the rootfs, which we haven’t got. We could + rename `/bin/fossil` to `/sbin/init` and then hack + the chroot scheme to match, but ick. (This, incidentally, + is why we set `ProcessTwo=yes` above even though Fossil is + perfectly capable of running as PID 1, a fact we depend on + in the other methods above.) + +* **`machinectl shell`** will fail because there is no login + daemon running, which we purposefully avoided adding by + creating a “`FROM scratch`” container. (If you need a + shell, say: `sudo systemd-nspawn --machine=myproject /bin/sh`) + +* **`machinectl status`** won’t give you the container logs + because we disabled the shared journal, which was in turn + necessary because we don’t run `systemd` *inside* the + container, just outside. + +If these are problems for you, you may wish to build a +fatter container using `debootstrap` or similar. ([External +tutorial][medtut].) + +[medtut]: https://medium.com/@huljar/setting-up-containers-with-systemd-nspawn-b719cff0fb8d + +
diff --git a/www/contribute.wiki b/www/contribute.wiki new file mode 100644 index 0000000..9fe477b --- /dev/null +++ b/www/contribute.wiki @@ -0,0 +1,128 @@ +Contributing To Fossil + +Fossil users are encouraged to contribute enhancements back to the +project. This note outlines some of the procedures for making +useful contributions. + +

1.0 Contributor Agreement

+ +In order to accept non-trivial contributions, we must have a +[./copyright-release.pdf | Contributor Agreement (PDF)] +(or [./copyright-release.html | as HTML]) on file for you. We require +this in order to maintain clear title to the Fossil code and prevent +the introduction of code with incompatible licenses or other entanglements +that might cause legal problems for Fossil users. Many +lawyer-rich organizations require this as a precondition to using +Fossil. + +If you do not wish to submit a Contributor Agreement, we would still +welcome your suggestions and example code, but we will not use your code +directly: we will be forced to re-implement your changes from scratch, which +might take longer. + +We've made exceptions for "trivial" changes in the past, but the +definition of that term is up to the project leader. + +

2.0 Submitting Patches

+ +Suggested changes or bug fixes can be submitted by creating a patch +against the current source tree: + +
fossil diff -i > my-change.patch
+ +Alternatively, you can create a binary patch: + +
fossil patch create my-change.db
+ +Post patches to +[https://fossil-scm.org/forum | the forum] or email them to +drh@sqlite.org. Be sure to +describe in detail what the patch does and which version of Fossil +it is written against. It's best to make patches against tip-of-trunk +rather than against past releases. + +If your change is more complicated than a patch can properly encode, you +may submit [/help/bundle | a Fossil bundle] instead. Unlike patches, +bundles can contain multiple commits, check-in comments, file renames, +file deletions, branching decisions, and more which patch(1) +files cannot. It's best to make a bundle of a new branch so the change +can be integrated, tested, enhanced, and merged down to trunk in a +controlled fashion. + +A contributor agreement is not strictly necessary to submit a patch or bundle, +but without a contributor agreement on file, your contribution will be +used for reference only: it will not be applied to the code. This +may delay acceptance of your contribution. + +Your contribution might not be accepted even if you do have +a contributor agreement on file. Please do not take this personally +or as an affront to your coding ability. Sometimes contributions are rejected +because they seem to be taking the project in a direction that the +architect does not want to go. In other cases, there might be an alternative +implementation of the same feature being prepared separately. + +

3.0 Check-in Privileges

+ +Check-in privileges are granted on a case-by-case basis. Your chances +of getting check-in privileges are much improved if you have a history +of submitting quality patches and/or making thoughtful posts on +[https://fossil-scm.org/forum | the forum]. +A signed contributor agreement is, of course, a prerequisite for check-in +privileges.

+ +Contributors are asked to make all non-trivial changes on a branch. The +Fossil Architect (Richard Hipp) will merge changes onto the trunk.

+ +Contributors are required to follow the +[./checkin.wiki | pre-checkin checklist] prior to every check-in to +the Fossil self-hosting repository. This checklist is short and succinct +and should only require a few seconds to follow. Contributors +should print out a copy of the pre-checkin checklist and keep +it on a note card beside their workstations for quick reference. + +Contributors should review the +[./style.wiki | Coding Style Guidelines] and mimic the coding style +used through the rest of the Fossil source code. Your code should +blend in. A third-party reader should be unable to distinguish your +code from any other code in the source corpus. + +

4.0 Testing

+ +Fossil's [../test/release-checklist.wiki | release checklist] is of +primary benefit to the project leader, followed by him at release time, +but contributors are encouraged to run through its steps when making +major changes, since if the change doesn't pass this checklist, it won't +be included in the next release. + +

5.0 UI and Documentation Language

+ +The Fossil project uses American English in its web interface and +documentation. Until there is some provision for translating the UI and +docs into other languages and dialects, we ask that you do not commit +changes that conflict with this. + +We aren't opposed to such a project, but it would be a huge amount of +work, which no one's stepped up to do yet. Not only is each individual +translation a large ongoing job its own right, there is no +infrastructure for it yet, so the first few translations will be harder +than any future translation built on that infrastructure. + +More immediately, we're likely to reject, revert, or rework commits that +use other English dialects. One example that comes up occasionally is +"artefact" versus "artifact." The UI and docs use the American English +spelling pervasively, so you have poor options if you insist on +"artefact:" + + * attempt to slip one-off changes by your peers + * attempt to change all American English usages to Commonwealth English + * make the Fossil UI and docs translatable, then contribute a + Commonwealth English translation + +Only the latter is likely to succeed. + +

6.0 See Also

+ + * [./build.wiki | How To Compile And Install Fossil] + * [./makefile.wiki | The Fossil Build Process] + * [./tech_overview.wiki | A Technical Overview of Fossil] + * [./adding_code.wiki | Adding Features To Fossil] diff --git a/www/copyright-release.html b/www/copyright-release.html new file mode 100644 index 0000000..222d3f5 --- /dev/null +++ b/www/copyright-release.html @@ -0,0 +1,106 @@ +

+Fossil SCM Contributor Agreement +

+ +

+This agreement applies to your contribution of material to the +Fossil Software Configuration Management System ("Fossil") that is +managed by Hipp, Wyrick & Company, Inc. ("Hwaci") and +sets out the intellectual property rights you grant to Hwaci in the +contributed material. +The terms "contribution" and "contributed material" mean any source code, +object code, patch, tool, sample, graphic, specification, manual, +documentation, or any other material posted, submitted, or uploaded by +you to the Fossil project. + The term "you" means the person identified +and signing at the bottom of this document. If your contribution +is on behalf of a company, the term "you" also means the company +identified in the signature area below. + +

    +

  1. +With respect to any worldwide copyrights, or copyright applications and +registrations, in your contribution: +

      +
    • You hereby assign to Hwaci joint ownership, and to the extent that + such assignment is or becomes invalid, ineffective or unenforceable, + you hereby grant to Hwaci a perpetual, irrevocable, non-exclusive, + worldwide, no-charge, royalty-free, unrestricted license to exercise + all rights under those copyrights, including the right to sublicense. +
    • You agree that both you and Hwaci can do all things in relation to your + contribution as if each of us were the sole owners, and if one of us + makes a derivative work of your contribution, the one who makes + (or has made) the derivative work will be the sole owner of that + derivative work. +
    • You agree that you will not assert any moral rights in your + contribution against Hwaci, Hwaci's licensees or transferees, or + any other user or consumer of your contribution. +
    • You agree that Hwaci may register a copyright in your contribution and + exercise all ownership rights associated with it. +
    • You agree that neither you nor Hwaci has any duty to consult with, + obtain the consent of, or pay or render an accounting to the other + for any use or distribution of your contribution. +
    + +

  2. +With respect to any patents you own, or that you can license without payment +to any third party, and which are relevant to your contribution, you hereby +grant to Hwaci a perpetual, irrevocable, non-exclusive, worldwide, no-charge, +royalty-free license to +make, have made, use, sell, offer to sell, import, and otherwise +transfer your contribution in whole or in part, alone or in +combination with or included in any product, work or materials arising +out of the Fossil project, and to sublicense these same rights. + +

  3. +Except as set out above, you keep all right, title, and interest in your +contribution. The rights that you grant to Hwaci under this agreement +are effective on the date that you first submitted your contribution to the +Fossil project, even if your submission took place before the date that +you sign this agreement. + +

  4. +You represent and warrant the following: +

      +
    • Your contribution is an original work and that you can legally + grant the rights set out in this agreement. +
    • Your contribution does not, to the best of your knowledge and belief, + violate any third party's copyrights, trademarks, patents, + or other intellectual property rights. +
    • You are authorized to sign this agreement on behalf of your + company (if applicable). +
    +
+ +

By filling in the following information and signing your name, +you agree to be bound by all of the terms +set forth in this agreement. Please print clearly.

+ +

+ + + + + +
Your name & email: + +  

  + +

Company name:
(if applicable)		 + +  

  + + 

Postal address: + + +

 

 

 

+ +
Signature: 

 

Date: 

 

+ +

Send completed forms to: +

+Hipp, Wyrick & Company, Inc.
+6200 Maple Cove Lane
+Charlotte, NC 28269-1086
+USA +

diff --git a/www/copyright-release.pdf b/www/copyright-release.pdf new file mode 100644 index 0000000..1a73987 Binary files /dev/null and b/www/copyright-release.pdf differ diff --git a/www/css-tricks.md b/www/css-tricks.md new file mode 100644 index 0000000..c09013c --- /dev/null +++ b/www/css-tricks.md @@ -0,0 +1,74 @@ +# Fossil CSS Tips and Tricks + +Many aspects of Fossil's appearance can be customized by +[customizing the site skin](customskin.md). This document +details certain specific CSS tweaks which users have asked +about via the forums. + +This is a "living document" - please feel free to suggest +additions via [the Fossil forum](https://fossil-scm.org/forum/). + +This document is *not* an introduction to CSS - the web is +full of tutorials on that topic. It covers only the specifics +of customizing certain CSS-based behaviors in a Fossil UI. That said... + +## Is it Really `!important`? + +By and large, CSS's `!important` qualifier is not needed when +customizing Fossil's CSS. On occasion, however, particular styles may +be set directly on DOM elements when Fossil generates its HTML, and +such cases require the use of `!important` to override them. + + + +# Main UI CSS + +## Number of Columns in `/dir` View + +The width of columns on the [`/dir` page](/dir) is calculated +dynamically as the page is generated, to attempt to fit the widest +name in a given directory. The number of columns is determined +automatically by CSS. To modify the number of columns and/or the entry width: + +```css +div.columns { + columns: WIDTH COLUMN_COUNT !important; + /* Examples: + columns: 20ex 3 !important + columns: auto auto !important + */ +} +/* The default rule uses div.columns, but it can also be selected using: */ +div.columns.files { ... } +``` + +The `!important` qualifier is required here because the style values are dynamically +calculated and applied when the HTML is emitted. + +The file list itself can be further customized via: + +```css +div.columns > ul { + ... +} +ul.browser { + ... +} +``` + + + +# Forum-specific CSS + +## Limiting Display Length of Long Posts + +Excessively long posts can make scrolling through threads problematic, +especially on mobile devices. The amount of a post which is visible can +be configured using: + +```css +div.forumPostBody { + max-height: 25em; /* change to the preferred maximum effective height */ + overflow: auto; /* tells the browser to add scrollbars as needed */ +} +``` diff --git a/www/css/diff.md b/www/css/diff.md new file mode 100644 index 0000000..9a9e5e0 --- /dev/null +++ b/www/css/diff.md @@ -0,0 +1,112 @@ +Notes On Diff Formatting +======================== + +There are two main kinds of diff display for the web interface: +unified and side-by-side. Both displays are implemented using +a <table>. The unified diff is a 4-column table, and the +side-by-side diff is a 5-column table. In a page like /info that +might show multiple file diffs, each file diff is in a separate +<table>. For side-by-side diffs, a small amount of Javascript +code is used to resize the text columns so that they fill the screen +width and to keep horizontal scrollbars in sync. + +For the unified diff, the basic structure +is like this: + +> ~~~~ + + + + + + + +
+     Line numbers for the left-hand file
+  
+     Line numbers for the right-hand file
+  
+     Change marks.  "+" or "=" or nothing
+  
+     The text
+
+~~~~ + +The structure for a side-by-side diff follows the +same basic pattern, though with 5 columns instead of +4, and slightly different class names: + +> ~~~~ + + + + + + + + +
+     Line numbers for the left-hand file
+  
+     The text for the left side
+  
+     Change marks.  "+" or "=" or nothing
+  
+     Line numbers for the right-hand file
+  
+     The text on the right-hand side
+
+~~~~ + +The outer <table> always has class "diff". The "diffu" class +is added for unified diffs and the "splitdiff" class is added for +side-by-side diffs. + +All line-number columns have the "diffln" class. They also always +have one of "difflnl" or "difflnr" depending on whether they hold +line numbers for the left or right files, respectively. + +Text is always kept in a separate column so that it can be scraped +and copied by the user. All text columns have the "difftxt" class. +One additional class "difftxtu", "difftxtl", or "difftxtr" is added +depending on if the text is for a unified diff, the left column of +a side-by-side diff, or the right column of a side-by-side diff. + +The content of all columns is a single <pre> that contains the +appropriate diff-text for that column. Scrolling is done on the +<pre> element. + +Within text columns, highlighting is done with <del> and +<ins> markup. All text on a line that contains an isert or +delete is surrounded by <ins>...</ins> or +<del>..</del>. Within that line, specific characters +of text that specifically inserted deleted have an additional +layer of <ins> or <del> markup. Thus CSS like the +following is appropriate: + +> ~~~~ +td.difftxt ins { + background-color: #dafbe1; /* Light green for the whole line */ + text-decoration: none; +} +td.difftxt ins > ins { + background-color: #a0e4b2; /* Dark green for specific characters that change */ + text-decoration: none; +} +~~~~ + +In a side-by-side diff, if an interior <ins> or <del> that mark +specific characters that change correspond to a delete/insert on the other +side, they they have the "edit" class tag. (ex: <ins class='edit'> +or <del class='edit'>). Some skins choose to paint these "modified" +regions blue: + +> ~~~~ +td.difftxt ins > ins.edit { + background-color: #c0c0ff; /* Blue for "modified" text region */ + text-decoration: none; +} +~~~~ + +Line number text also has <ins> and <del> tags for lines which +are pure insert or pure delete. But the tags do not nest for line numbers. diff --git a/www/css/index.md b/www/css/index.md new file mode 100644 index 0000000..d3835df --- /dev/null +++ b/www/css/index.md @@ -0,0 +1,12 @@ +Cascading Style Sheet Notes +=========================== + +This is a collection of technical notes that document the design of +the Document Object Model (DOM) and corresponding Cascading Style Sheet (CSS) +attributes used for customing the look-and-feel of Fossil. These notes +are of interest to people who want to customize the Fossil skin or +enhance the internal display logic. + +This is a collection of documents that we hope will grow over time. + + * [Diff styling](./diff.md) diff --git a/www/custom_ticket.wiki b/www/custom_ticket.wiki new file mode 100644 index 0000000..eac0532 --- /dev/null +++ b/www/custom_ticket.wiki @@ -0,0 +1,153 @@ +Customizing The Ticket System + +

Introduction

+ +This guide will explain how to add the "assigned_to" and "submitted_by" fields +to the ticket system in Fossil, as well as making the system more useful. You +must have "admin" access to the repository to implement these instructions. + +

First modify the TICKET table

+ +Click on the "Admin" menu, then "Tickets", then "Table". After the other fields +and before the final ")", insert: +
+  ,
+  assigned_to TEXT,
+  opened_by TEXT
+
+ +And "Apply Changes". You have just added two more fields to the ticket +database! NOTE: I won't tell you to "Apply Changes" after each step from here +on out. Now, how do you use these fields? + +

Next add assignees

+ +Back to the "Tickets" admin page, and click "Common". Add something like this: +
+set assigned_choices {
+  unassigned
+  tom
+  dick
+  harriet
+}
+
+ +Obviously, choose names corresponding to the logins on your system. The +'unassigned' entry is important, as it prevents you from having a NULL in that +field (which causes problems later when editing). + +

Now modify the 'new ticket' page

+ +Back to the "Tickets" admin page, and click "New Ticket Page". This is a little +more tricky. Edit the top part: + + +if {[info exists submit]} { + set status Open + set opened_by $login + set assigned_to "unassigned" + submit_ticket +} + + +Note the "set opened_by" bit -- that will automatically set the "opened_by" +field to the login name of the bug reporter. Now, skip to the part with "EMail" +and modify it like so: + + +enable_output expr { "$login" eq "anonymous"} + + + EMail: + + + + Not publicly visible. Used by developers to contact you with questions. + + +enable_output 1 + + +This bit of code will get rid of the "email" field entry for logged-in users. +Since we know the user's information, we don't have to ask for it. NOTE: it +might be good to automatically scoop up the user's email and put it here. + +You might also want to enable people to actually assign the ticket to a specific +person during creation. For this to work, you need to add the code +for "assigned_to" as shown below under the heading "Modify the 'edit ticket' page". +This will give you an additional combobox where you can choose a person during +ticket creation. + +

Modify the 'view ticket' page

+ +Look for the text "Contact:" (about halfway through). Then insert these lines +after the closing tr tag and before the "enable_output" line: + + + + Assigned to: + $ + + Opened by: + $ + + + + +This will add a row which displays these two fields, in the event the user has +ticket "edit" capability. + +

Modify the 'edit ticket' page

+ +Before the "Severity:" line, add this: + + + + Assigned to: + + combobox assigned_to $assigned_choices 1 + + + + +That will give you a drop-down list of assignees. The first argument to the TH1 +command 'combobox' is the database field which the combobox is associated to. +The next argument is the list of choices you want to show in the combobox (and +that you specified in the second step above.) The last argument should be 1 for +a true combobox (see the TH1 documentation for +details). + +Now, similar to the previous +section, look for "Contact:" and add this: + + + + Reported by: + + + + + + +

What next?

+ +Now you can add custom reports which select based on the person to whom the +ticket is assigned. For example, an "Assigned to me" report could be: + + +SELECT + CASE WHEN status IN ('Open','Verified') THEN '#f2dcdc' + WHEN status='Review' THEN '#e8e8e8' + WHEN status='Fixed' THEN '#cfe8bd' + WHEN status='Tested' THEN '#bde5d6' + WHEN status='Deferred' THEN '#cacae5' + ELSE '#c8c8c8' END AS 'bgcolor', + substr(tkt_uuid,1,10) AS '#', + datetime(tkt_mtime) AS 'mtime', + type, + status, + subsystem, + title +FROM ticket +WHERE assigned_to=user() + diff --git a/www/customgraph.md b/www/customgraph.md new file mode 100644 index 0000000..1490f8b --- /dev/null +++ b/www/customgraph.md @@ -0,0 +1,228 @@ +# Customizing the Timeline Graph + +Beginning with version 1.33, Fossil gives users and skin authors significantly +more control over the look and feel of the timeline graph. + +## Basic Style Options + +Fossil includes several options for changing the graph's style without having +to delve into CSS. These can be found in the details.txt file of your skin or +under Admin/Skins/Details in the web UI. + +* **`timeline-arrowheads`** + + Set this to `0` to hide arrowheads on primary child lines. + +* **`timeline-circle-nodes`** + + Set this to `1` to make check-in nodes circular instead of square. + +* **`timeline-color-graph-lines`** + + Set this to `1` to colorize primary child lines. + +* **`white-foreground`** + + Set this to `1` if your skin uses white (or any light color) text. + This tells Fossil to generate darker background colors for branches. + + +## Advanced Styling + +If the above options aren't enough for you, it's time to get your hands dirty +with CSS. To get started, I recommend first copying all the [graph-related CSS +rules](#default-css) to your stylesheet. Then it's simply a matter of making +the necessary changes to achieve the look you want. So, next, let's look at the +various graph elements and what purpose they serve. + +Each element used to construct the timeline graph falls into one of two +categories: visible elements and positioning elements. We'll start with the +latter, less obvious type. + +## Positioning Elements + +These elements aren't intended to be seen. They're only used to help position +the graph and its visible elements. + +* **`.tl-canvas`** + + Set the left and right margins on this class to give the desired amount + of space between the graph and its adjacent columns in the timeline. + + **Additional Classes** + + * `.sel`: See [`.tl-node`](#tl-node) for more information. + +* **`.tl-rail`** + + Think of rails as invisible vertical lines on which check-in nodes are + placed. The more simultaneous branches in a graph, the more rails required + to draw it. Setting the `width` property on this class determines the + maximum spacing between rails. This spacing is automatically reduced as + the number of rails increases. If you change the `width` of `.tl-node` + elements, you'll probably need to change this value, too. + +* **`.tl-mergeoffset`** + + A merge line often runs vertically right beside a primary child line. This + class's `width` property specifies the maximum spacing between the two. + Setting this value to `0` will eliminate the vertical merge lines. + Instead, the merge arrow will extend directly off the primary child line. + As with rail spacing, this is also adjusted automatically as needed. + +* **`.tl-nodemark`** + + In the timeline table, the second cell in each check-in row contains an + invisible div with this class. These divs are used to determine the + vertical position of the nodes. By setting the `margin-top` property, + you can adjust this position. + +## Visible Elements + +These are the elements you can actually see on the timeline graph: the nodes, +arrows, and lines. Each of these elements may also have additional classes +attached to them, depending on their context. + +* **`.tl-node`** + + A node exists for each check-in in the timeline. + + **Additional Classes** + + * `.leaf`: Specifies that the check-in is a leaf (i.e. that it has no + children in the same branch). + + * `.merge`: Specifies that the check-in contains a merge. + + * `.sel`: When the user clicks a node to designate it as the beginning + of a diff, this class is added to both the node itself and the + [`.tl-canvas`](#tl-canvas) element. The class is removed from both + elements when the node is clicked again. + +* **`.tl-arrow`** + + Arrows point from parent nodes to their children. Technically, this + class is just for the arrowhead. The rest of the arrow is composed + of [`.tl-line`](#tl-line) elements. + + There are six additional classes that are used to distinguish the different + types of arrows. However, only these combinations are valid: + + * `.u`: Up arrow that points to a child from its primary parent. + + * `.u.sm`: Smaller up arrow, used when there is limited space between + parent and child nodes. + + * `.merge.l` or `.merge.r`: Merge arrow pointing either to the left or + right. + + * `.warp`: A timewarped arrow (always points to the right), used when a + misconfigured clock makes a check-in appear to have occurred before its + parent ([example](https://www.sqlite.org/src/timeline?c=2010-09-29&nd)). + +* **`.tl-line`** + + Along with arrows, lines connect parent and child nodes. Line thickness is + determined by the `width` property, regardless of whether the line is + horizontal or vertical. You can also use borders to create special line + styles. Here's a CSS snippet for making dotted merge lines: + + .tl-line.merge { + width: 0; + background: transparent; + border: 0 dotted #000; + } + .tl-line.merge.h { + border-top-width: 1px; + } + .tl-line.merge.v { + border-left-width: 1px; + } + + **Additional Classes** + + * `.merge`: A merge line. + + * `.h` or `.v`: Horizontal or vertical. + + * `.warp`: A timewarped line. + + +## Default Timeline Graph CSS + + .tl-canvas { + margin: 0 6px 0 10px; + } + .tl-rail { + width: 18px; + } + .tl-mergeoffset { + width: 2px; + } + .tl-nodemark { + margin-top: 5px; + } + .tl-node { + width: 10px; + height: 10px; + border: 1px solid #000; + background: #fff; + cursor: pointer; + } + .tl-node.leaf:after { + content: ''; + position: absolute; + top: 3px; + left: 3px; + width: 4px; + height: 4px; + background: #000; + } + .tl-node.sel:after { + content: ''; + position: absolute; + top: 2px; + left: 2px; + width: 6px; + height: 6px; + background: red; + } + .tl-arrow { + width: 0; + height: 0; + transform: scale(.999); + border: 0 solid transparent; + } + .tl-arrow.u { + margin-top: -1px; + border-width: 0 3px; + border-bottom: 7px solid #000; + } + .tl-arrow.u.sm { + border-bottom: 5px solid #000; + } + .tl-line { + background: #000; + width: 2px; + } + .tl-arrow.merge { + height: 1px; + border-width: 2px 0; + } + .tl-arrow.merge.l { + border-right: 3px solid #000; + } + .tl-arrow.merge.r { + border-left: 3px solid #000; + } + .tl-line.merge { + width: 1px; + } + .tl-arrow.warp { + margin-left: 1px; + border-width: 3px 0; + border-left: 7px solid #600000; + } + .tl-line.warp { + background: #600000; + } diff --git a/www/customskin.md b/www/customskin.md new file mode 100644 index 0000000..2df2b13 --- /dev/null +++ b/www/customskin.md @@ -0,0 +1,598 @@ +# Skinning the Fossil Web Interface + +The Fossil web interface comes with a pre-configured look and feel. The default +look and feel works fine in many situations. However, you may want to change +the look and feel (the "skin") of Fossil to better suite your own individual tastes. +This document provides background information to aid you in that task. + +## Built-in Skins + +Fossil comes with [multiple built-in skins](/skins). If the default skin does not +suite your tastes, perhaps one of the other built-in skins will work better. +If nothing else, the built-in skins can serve as examples or templates that +you can use to develop your own custom skin. + +The sources to these built-ins can +be found in the Fossil source tree under the skins/ folder. The +[skins/](/dir?ci=trunk&name=skins) +folder contains a separate subfolder for each built-in skin, with each +subfolders holding at least these five files: + + * css.txt + * details.txt + * footer.txt + * header.txt + * js.txt + +Try out the built-in skins by using the --skin option on the +[fossil ui](/help/ui) or [fossil server](/help/server) commands. + +## Sharing Skins + +The skin of a repository is not part of the versioned state and does not +"push" or "pull" like checked-in files. The skin is local to the +repository. However, skins can be shared between repositories using +the [fossil config](/help/configuration) command. +The "fossil config push skin" command will send the local skin to a remote +repository and the "fossil config pull skin" command will import a skin +from a remote repository. The "fossil config export skin FILENAME" +will export the skin for a repository into a file FILENAME. This file +can then be imported into a different repository using the +"fossil config import FILENAME" command. Unlike "push" and "pull", +the "export" and "import" commands are able to move skins between +repositories for different projects. So, for example, if you have a +group of related repositories, you can develop a skin for one of them, +then get a consistent look across all the repositories by exporting +the skin from the first repository and importing into all the others. + +The file generated by "fossil config export" could be checked into +one of your repositories and versioned, if desired. This will not +automatically change the skin when looking backwards in time, but it +will provide an historical record of what the skin used to be and +allow the historical look of the repositories to be recreated if +necessary. + +When cloning a repository, the skin of the new repository is initialized to +the skin of the repository from which it was cloned. + +# Structure Of A Fossil Web Page + +Every HTML page generated by Fossil has the same basic structure: + +| Fossil-Generated HTML Header | +| Skin Header | +| Fossil-Generated Content | +| Skin Footer | +| Fossil-Generated HTML Footer | + +By default, Fossil starts every generated HTML page with this: + + + + + + + .... + + + + +Fossil used to require a static version of this in every skin’s Header +area, but over time, we have found good cause to generate multiple +elements at runtime. + +One such is the `FEATURE` element, being either the top-level HTTP +request routing element (e.g. `doc`) or an aggregate feature class that +groups multiple routes under a single name. A prime example is `forum`, +which groups the `/forummain`, `/forumpost`, and `/forume2` routes, +allowing per-feature CSS. For instance, to style `
` tags +specially for forum posts written in Markdown, leaving all other block +quotes alone, you could say: + + body.forum div.markdown blockquote { + margin-left: 10px; + } + +You can [override this generated HTML header](#override) by including a +“``” tag somewhere in the Header area of the skin, but it is +almost always best to limit a custom skin’s Header section to something +like this: + +