diff options
Diffstat (limited to 'www')
207 files changed, 44145 insertions, 0 deletions
diff --git a/www/aboutcgi.wiki b/www/aboutcgi.wiki new file mode 100644 index 0000000..362f536 --- /dev/null +++ b/www/aboutcgi.wiki @@ -0,0 +1,301 @@ +<title>How CGI Works In Fossil</title> + +<h2>Introduction</h2> + +CGI or "Common Gateway Interface" is a venerable yet reliable technique for +generating dynamic web content. This article gives a quick background on how +CGI works and describes how Fossil can act as a CGI service. + +This is a "how it works" guide. This document provides background +information on the CGI protocol so that you can better understand what +is going on behind the scenes. If you just want to set up Fossil +as a CGI server, see the [./server/ | Fossil Server Setup] page. Or +if you want to develop CGI-based extensions to Fossil, see +the [./serverext.wiki|CGI Server Extensions] page. + +<h2>A Quick Review Of CGI</h2> + +An HTTP request is a block of text that is sent by a client application +(usually a web browser) and arrives at the web server over a network +connection. The HTTP request contains a URL that describes the information +being requested. The URL in the HTTP request is typically the same URL +that appears in the URL bar at the top of the web browser that is making +the request. The URL might contain a "?" character followed by +query parameters. The HTTP will usually also contain other information +such as the name of the application that made the request, whether or +not the requesting application can accept a compressed reply, POST +parameters from forms, and so forth. + +The job of the web server is to interpret the HTTP request and formulate +an appropriate reply. +The web server is free to interpret the HTTP request in any way it wants, +but most web servers follow a similar pattern, described below. +(Note: details may vary from one web server to another.) + +Suppose the filename component of the URL in the HTTP request looks like this: + +<pre>/one/two/timeline/four</pre> + +Most web servers will search their content area for files that match +some prefix of the URL. The search starts with <b>/one</b>, then goes to +<b>/one/two</b>, then <b>/one/two/timeline</b>, and finally +<b>/one/two/timeline/four</b> is checked. The search stops at the first +match. + +Suppose the first match is <b>/one/two</b>. If <b>/one/two</b> is an +ordinary file in the content area, then that file is returned as static +content. The "<b>/timeline/four</b>" suffix is silently ignored. + +If <b>/one/two</b> is a CGI script (or program), then the web server +executes the <b>/one/two</b> script. The output generated by +the script is collected and repackaged as the HTTP reply. + +Before executing the CGI script, the web server will set up various +environment variables with information useful to the CGI script: +<table> +<tr><th>Variable<th>Meaning +<tr><td>GATEWAY_INTERFACE<td>Always set to "CGI/1.0" +<tr><td>REQUEST_URI + <td>The input URL from the HTTP request. +<tr><td>SCRIPT_NAME + <td>The prefix of the input URL that matches the CGI script name. + In this example: "/one/two". +<tr><td>PATH_INFO + <td>The suffix of the URL beyond the name of the CGI script. + In this example: "timeline/four". +<tr><td>QUERY_STRING + <td>The query string that follows the "?" in the URL, if there is one. +</table> + +There are other CGI environment variables beyond those listed above. +Many Fossil servers implement the +[https://fossil-scm.org/home/test-env/two/three?abc=xyz|test-env] +webpage that shows some of the CGI environment +variables that Fossil pays attention to. + +In addition to setting various CGI environment variables, if the HTTP +request contains POST content, then the web server relays the POST content +to standard input of the CGI script. + +In summary, the task of the +CGI script is to read the various CGI environment variables and +the POST content on standard input (if any), figure out an appropriate +reply, then write that reply on standard output. +The web server will read the output from the CGI script, reformat it +into an appropriate HTTP reply, and relay the result back to the +requesting application. +The CGI script exits as soon as it generates a single reply. +The web server will (usually) persist and handle multiple HTTP requests, +but a CGI script handles just one HTTP request and then exits. + +The above is a rough outline of how CGI works. +There are many details omitted from this brief discussion. +See other on-line CGI tutorials for further information. + +<h2>How Fossil Acts As A CGI Program</h2> + +An appropriate CGI script for running Fossil will look something +like the following: + +<pre> +#!/usr/bin/fossil +repository: /home/www/repos/project.fossil +</pre> + +The first line of the script is a +"[https://en.wikipedia.org/wiki/Shebang_%28Unix%29|shebang]" +that tells the operating system what program to use as the interpreter +for this script. On unix, when you execute a script that starts with +a shebang, the operating system runs the program identified by the +shebang with a single argument that is the full pathname of the script +itself. +In our example, the interpreter is Fossil, and the argument might +be something like "/var/www/cgi-bin/one/two" (depending on how your +particular web server is configured). + +The Fossil program that is run as the script interpreter +is the same Fossil that runs when +you type ordinary Fossil commands like "fossil sync" or "fossil commit". +But in this case, as soon as it launches, the Fossil program +recognizes that the GATEWAY_INTERFACE environment variable is +set to "CGI/1.0" and it therefore knows that it is being used as +CGI rather than as an ordinary command-line tool, and behaves accordingly. + +When Fossil recognizes that it is being run as CGI, it opens and reads +the file identified by its sole argument (the file named by +<code>argv[1]</code>). In our example, the second line of that file +tells Fossil the location of the repository it will be serving. +Fossil then starts looking at the CGI environment variables to figure +out what web page is being requested, generates that one web page, +then exits. + +Usually, the webpage being requested is the first term of the +PATH_INFO environment variable. (Exceptions to this rule are noted +in the sequel.) For our example, the first term of PATH_INFO +is "timeline", which means that Fossil will generate +the [/help/www/timeline|/timeline] webpage. + +With Fossil, terms of PATH_INFO beyond the webpage name are converted into +the "name" query parameter. Hence, the following two URLs mean +exactly the same thing to Fossil: +<ol type='A'> +<li> [https://fossil-scm.org/home/info/c14ecc43] +<li> [https://fossil-scm.org/home/info?name=c14ecc43] +</ol> + +In both cases, the CGI script is called "/fossil". For case (A), +the PATH_INFO variable will be "info/c14ecc43" and so the +"[/help/www/info|/info]" webpage will be generated and the suffix of +PATH_INFO will be converted into the "name" query parameter, which +identifies the artifact about which information is requested. +In case (B), the PATH_INFO is just "info", but the same "name" +query parameter is set explicitly by the URL itself. + +<h2>Serving Multiple Fossil Repositories From One CGI Script</h2> + +The previous example showed how to serve a single Fossil repository +using a single CGI script. +On a website that wants to serve multiple repositories, one could +simply create multiple CGI scripts, one script for each repository. +But it is also possible to serve multiple Fossil repositories from +a single CGI script. + +If the CGI script for Fossil contains a "directory:" line instead of +a "repository:" line, then the argument to "directory:" is the name +of a directory that contains multiple repository files, each ending +with ".fossil". For example: + +<pre> +#!/usr/bin/fossil +directory: /home/www/repos +</pre> + +Suppose the /home/www/repos directory contains files named +<b>one.fossil</b>, <b>two.fossil</b>, and <b>subdir/three.fossil</b>. +Further suppose that the name of the CGI script (relative to the root +of the webserver document area) is "cgis/example2". Then to +see the timeline for the "three.fossil" repository, the URL would be: + +<pre> +http://example.com/cgis/example2/subdir/three/timeline +</pre> + +Here is what happens: +<ol> +<li> The input URI on the HTTP request is + <b>/cgis/example2/subdir/three/timeline</b> +<li> The web server searches prefixes of the input URI until it finds + the "cgis/example2" script. The web server then sets + PATH_INFO to the "subdir/three/timeline" suffix and invokes the + "cgis/example2" script. +<li> Fossil runs and sees the "directory:" line pointing to + "/home/www/repos". Fossil then starts pulling terms off the + front of the PATH_INFO looking for a repository. It first looks + at "/home/www/resps/subdir.fossil" but there is no such repository. + So then it looks at "/home/www/repos/subdir/three.fossil" and finds + a repository. The PATH_INFO is shortened by removing + "subdir/three/" leaving it at just "timeline". +<li> Fossil looks at the rest of PATH_INFO to see that the webpage + requested is "timeline". +</ol> +<a id="cgivar"></a> + +The web server sets many environment variables in step 2 in addition +to just PATH_INFO. The following diagram shows a few of these variables +and their relationship to the request URL: + +<verbatim type="pikchr"> +charwid = 0.075 +thickness = 0 + +SCHEME: box "https://" mono fit +DOMAIN: box "example.com" mono fit +SCRIPT: box "/cgis/example2" mono fit +PATH: box "/subdir/three/timeline" mono fit +QUERY: box "?c=55d7e1" mono fit + +thickness = 0.01 + +DB: box at 0.3 below DOMAIN "HTTP_HOST" mono fit invis +SB: box at 0.3 below SCRIPT "SCRIPT_NAME" mono fit invis +PB: box at 0.3 below PATH "PATH_INFO" mono fit invis +QB: box at 0.3 below QUERY "QUERY_STRING" mono fit invis +RB: box at 0.5 above PATH "REQUEST_URI" mono fit invis + +color = lightgray + +box at SCHEME width SCHEME.width height SCHEME.height +line fill 0x7799CC behind QUERY \ + from SCRIPT.nw \ + to RB.sw \ + to RB.se \ + to QUERY.ne \ + close +line fill 0x99CCFF behind DOMAIN \ + from DOMAIN.nw \ + to DOMAIN.sw \ + to DB.n \ + to DOMAIN.se \ + to DOMAIN.ne \ + close +line fill 0xCCEEFF behind SCRIPT \ + from SCRIPT.nw \ + to SCRIPT.sw \ + to SB.n \ + to SCRIPT.se \ + to SCRIPT.ne \ + close +line fill 0x99CCFF behind PATH \ + from PATH.nw \ + to PATH.sw \ + to PB.n \ + to PATH.se \ + to PATH.ne \ + close +line fill 0xCCEEFF behind QUERY \ + from QUERY.nw \ + to QUERY.sw \ + to QB.n \ + to QUERY.se \ + to QUERY.ne \ + close +</verbatim> + +<h2>Additional CGI Script Options</h2> + +The CGI script can have additional options used to fine-tune +Fossil's behavior. See the [./cgi.wiki|CGI script documentation] +for details. + +<h2>Additional Observations</h2> +<ol type="I"> +<li><p> +Fossil does not distinguish between the various HTTP methods (GET, PUT, +DELETE, etc). Fossil figures out what it needs to do purely from the +webpage term of the URI.</p></li> +<li><p> +Fossil does not distinguish between query parameters that are part of the +URI, application/x-www-form-urlencoded or multipart/form-data encoded +parameter that are part of the POST content, and cookies. Each information +source is seen as a space of key/value pairs which are loaded into an +internal property hash table. The code that runs to generate the reply +can then reference various properties values. +Fossil does not care where the value of each property comes from (POST +content, cookies, or query parameters) only that the property exists +and has a value.</p></li> +<li><p> +The "[/help/ui|fossil ui]" and "[/help/server|fossil server]" commands +are implemented using a simple built-in web server that accepts incoming HTTP +requests, translates each request into a CGI invocation, then creates a +separate child Fossil process to handle each request. In other words, CGI +is used internally to implement "fossil ui/server". +<br><br> +SCGI is processed using the same built-in web server, just modified +to parse SCGI requests instead of HTTP requests. Each SCGI request is +converted into CGI, then Fossil creates a separate child Fossil +process to handle each CGI request.</p></li> +<li><p> +Fossil is itself often launched using CGI. But Fossil can also then +turn around and launch [./serverext.wiki|sub-CGI scripts to implement +extensions].</p></li> +</ol> diff --git a/www/aboutdownload.wiki b/www/aboutdownload.wiki new file mode 100644 index 0000000..4eb09af --- /dev/null +++ b/www/aboutdownload.wiki @@ -0,0 +1,101 @@ +<title>How The Fossil Download Page Works</title> + +<h2>1.0 Overview</h2> + +The [/uv/download.html|Download] page for the Fossil self-hosting +repository is implemented using [./unvers.wiki|unversioned files]. +The "download.html" screen itself, and the various build products +are all stored as unversioned content. The download.html page +uses XMLHttpRequest() to retrieve the [/help/www/juvlist|/juvlist] webpage +for a list of all unversioned files. Javascript in the +[/uv/download.js?mimetype=text/plain|download.js] file (which is +sourced by "download.html") then figures out which unversioned files are +build products and paints appropriate icons on the displayed +download page. + +Except, the "Source Tarball" download products are not stored as +unversioned files. They are computed on-demand by the +[/help/www/tarball|/tarball web page]. + +When a new version is generated, the developers use the +[/help/uv|fossil uv edit] command to make minor changes +to the "[/uv/download.js?mimetype=text/plain|download.js]" +file so that it knows about the +new version number. Then the developers run +the [/help/uv|fossil uv add] command for each +build product. Finally, the +[/help/uv|fossil uv sync] command is run to push all +the content up to servers. All +[./selfhost.wiki|three self-hosting repositories] for Fossil +are updated automatically. + +<h2>2.0 Details</h2> + +The current text of the "download.html" and "download.js" files can +be seen at: + + * [/uv/download.html?mimetype=text/plain] + * [/uv/download.js?mimetype=text/plain] + +Notice how the hyperlinks above use the "mimetype=text/plain" +query parameter in order to display the file as plain text +instead of the usual HTML or Javascript. + +The default mimetype for "download.html" is +text/html. But because the entire page is enclosed within + +<verbatim><div class='fossil-doc' data-title='Download Page'>...</div></verbatim> + +Fossil knows to add its standard header and footer information to the +document, making it look just like any other page. See +"[./embeddeddoc.wiki|embedded documentation]" for further details on +how this <div class='fossil-doc'> markup works. + +With each new release, the "releases" variable in the javascript on +the [/uv/download.js?mimetype=text/plain|download.js] page is +edited (using "[/help/uv|fossil uv edit download.js]") to add +details of the release. + +When the JavaScript in the "download.js" file runs, it requests +a listing of all unversioned content using the /juvlist URL. +([/juvlist|sample /juvlist output]). The content of the download page is +constructed by matching unversioned files against regular expressions +in the "releases" variable. + +Build products need to be constructed on different machines. The precompiled +binary for Linux is compiled on Linux, the precompiled binary for Windows +is compiled on Windows11, and so forth. After a new release is tagged, +the release manager goes around to each of the target platforms, checks +out the release and compiles it, then runs +[/help/uv|fossil uv add] for the build product followed by +[/help/uv|fossil uv sync] to push the new build product to the +[./selfhost.wiki|various servers]. This process is repeated for +each build product. + +When older builds are retired from the download page, the +[/uv/download.js?mimetype=text/plain|download.js] page is again +edited to remove the corresponding entry from the "release" variable +and the edit is synced using +[/help/uv|fossil uv sync]. This causes the build products to +disappear from the download page immediately. But those build products +are still taking up space in the unversioned content table of the +server repository. To purge the obsolete build products, one or +more [/help/uv|fossil uv rm] commands are run, followed by +another [/help/uv|fossil uv sync]. It is important to purge +obsolete build products since they take up a lot of space. +At [/repo-tabsize] you can see that the unversioned table takes up +a substantial fraction of the repository. + +<h2>3.0 Security</h2> + +Only users with the [/setup_ulist_notes|"y" permission] are allowed +to push unversioned content up to the servers. Having the ability +to push check-ins (the [/setup_ulist_notes|"i" permission]) is not +sufficient. + +On the Fossil project there are (as of 2023-07-31) 71 people who have +check-in privileges. But only the project lead can push unversioned +content and thus change the build products on the download page. +Minimizing the number of people who can change the build products +helps to ensure that rogue binaries do not slip onto the download page +unnoticed. diff --git a/www/adding_code.wiki b/www/adding_code.wiki new file mode 100644 index 0000000..1192907 --- /dev/null +++ b/www/adding_code.wiki @@ -0,0 +1,228 @@ +<title>Adding Features To Fossil</title> + +<h2>1.0 Introduction</h2> + +This article provides a brief overview of how to write new C-code code that +extends or enhances the core Fossil binary. + +New features can be added to a Fossil server using +[./serverext.wiki|external CGI programs], +but that is not what this article is about. +This article focuses on how to make changes +to Fossil itself. + +<h2>2.0 Programming Language</h2> + +Fossil is written in C-89. There are specific [./style.wiki | style guidelines] +that are required for any new code that will be accepted into the Fossil core. +But, of course, if you are writing an extension just for yourself, you can +use any programming style you want. + +The source code for Fossil is not sent directly into the C compiler. +There are three separate code [./makefile.wiki#preprocessing|preprocessors] +that run over the code first. + + 1. The <b>mkindex</b> preprocessor scans all regular source files looking + for special comments that contain "help" text and which identify routines + that implement specific commands or which generate particular web pages. + + 2. The <b>makeheaders</b> preprocessor generates all the ".h" files + automatically. Fossil programmers write ".c" files only and let the + makeheaders preprocessor create the ".h" files. + + 3. The <b>translate</b> preprocessor converts source code lines that + begin with "@" into string literals, or into print statements that + generate web page output, depending on context. + +The [./makefile.wiki|Makefile] for Fossil takes care of running these +preprocessors with all the right arguments and in the right order. So it is +not necessary to understand the details of how these preprocessors work. +(Though, the sources for all three preprocessors are included in the source +tree and are well commented, if you want to dig deeper.) It is only necessary +to know that these preprocessors exist and hence will affect the way you +write code. + +<h2>3.0 Adding New Source Code Files</h2> + +New source code files are added in the "src/" subdirectory of the Fossil +source tree. Suppose one wants to add a new source code file named +"xyzzy.c". The first step is to add this file to the various makefiles. +Do so by editing the file tools/makemake.tcl and adding "xyzzy" (without +the final ".c") to the list of source modules at the top of that script. +Save the result and then run the makemake.tcl script using a TCL +interpreter. The command to run the makemake.tcl script is: + +<verbatim> +tclsh makemake.tcl +</verbatim> + +The working directory must be src/ when the command above is run. +Note that TCL is not normally required to build Fossil, but +it is required for this step. If you do not have a TCL interpreter on +your system already, they are easy to install. A popular choice is the +[http://www.activestate.com/activetcl|Active Tcl] installation from +ActiveState. + +After the makefiles have been updated, create the xyzzy.c source file +from the following template: + +<verbatim> +/* +** Copyright boilerplate goes here. +***************************************************** +** High-level description of what this module goes +** here. +*/ +#include "config.h" +#include "xyzzy.h" + +#if INTERFACE +/* Exported object (structure) definitions or #defines +** go here */ +#endif /* INTERFACE */ + +/* New code goes here */ +</verbatim> + +Note in particular the <b>#include "xyzzy.h"</b> line near the top. +The "xyzzy.h" file is automatically generated by makeheaders. Every +normal Fossil source file must have a #include at the top that imports +its private header file. (Some source files, such as "sqlite3.c" are +exceptions to this rule. Don't worry about those exceptions. The +files you write will require this #include line.) + +The "#if INTERFACE ... #endif" section is optional and is only needed +if there are structure definitions or typedefs or macros that need to +be used by other source code files. The makeheaders preprocessor +uses definitions in the INTERFACE section to help it generate header +files. See [../tools/makeheaders.html | makeheaders.html] for additional +information. + +After creating a template file such as shown above, and after updating +the makefiles, you should be able to recompile Fossil and have it include +your new source file, even before your source file contains any code. +It is recommended that you try this. + +Be sure to [/help/add|fossil add] your new source file to the self-hosting +Fossil repository and then [/help/commit|commit] your changes! + +<h2 id="newcmd">4.0 Creating A New Command</h2> + +By "commands" we mean the keywords that follow "fossil" when invoking +Fossil from the command-line. So, for example, in + +<verbatim> +fossil diff xyzzy.c +</verbatim> + +The "command" is "diff". Commands may optionally be followed by +arguments and/or options. To create new commands in Fossil, add code +(either to an existing source file, or to a new source file created as +described above) according to the following template: + +<verbatim> +/* +** COMMAND: xyzzy +** +** Help text goes here. Backslashes must be escaped. +*/ +void xyzzy_cmd(void){ + /* Implement the command here */ + fossil_print("Hello, World!\n"); +} +</verbatim> + +The example above creates a new command named "xyzzy" that prints the +message "Hello, World!" on the console. This command is a normal command +that will show up in the list of command from [/help/help|fossil help]. +If you add an asterisk to the end of the command name, like this: + +<verbatim> +** COMMAND: xyzzy* +</verbatim> + +Then the command will only show up if you add the "--all" option to +[/help/help|fossil help]. Or, if the command name starts with +"test" then the command will be considered experimental and will only +show up when the --test option is used with [/help/help|fossil help]. + +The example above is a fully functioning Fossil command. You can add +the text shown to an existing Fossil source file, recompiling then test +it out by typing: + +<verbatim> +./fossil xyzzy +./fossil help xyzzy +./fossil xyzzy --help +</verbatim> + +The name of the C function that implements the command can be anything +you like (as long as it does not collide with some other symbol in the +Fossil code) but it is traditional to name the function +"<i>commandname</i><b>_cmd</b>", as is done in the example. + +You could also use "printf()" instead of "fossil_print()" to generate +the output text, if desired. But "fossil_print()" is recommended as +it has extra logic to insert \r characters at the right times on +Windows systems. + +Once you have the command running, you can then start adding code to +make it do useful things. There are lots of utility functions in +Fossil for parsing command-line options and for +opening and accessing and manipulating the repository and +the working check-out. Study implementations of existing commands +to get an idea of how things are done. You can easily find the implementations +of existing commands by searching for "COMMAND: <i>name</i>" in the +files of the "src/" directory. + +<h2 id="newpage">5.0 Creating A New Web Page</h2> + +As with commands, new webpages can be added simply by inserting a function +that generates the webpage together with a special header comment. A +template follows: + +<verbatim> +/* +** WEBPAGE: helloworld +*/ +void helloworld_page(void){ + style_header("Hello World!"); + @ <p>Hello, World!</p> + style_footer(); +} +</verbatim> + +Add the code above to a new or existing Fossil source code file, then +recompile fossil and run [/help/ui|fossil ui] then enter +"http://localhost:8080/helloworld" in your web browser and the routine +above will generate a web page that says "Hello World." +It really is that simple. + +The special "WEBPAGE:" comment is picked up by the "mkindex" preprocessor +and used to generate a table that maps the "helloworld" webpage name +into a pointer to the "helloworld_page()" function. The function that +implements a webpage can be named anything you like (as long as it does +not collide with another name) but the traditional name is +"<i>pagename</i><b>_page</b>". + +HTML pages begin with a call to style_header() and end with the call to +style_footer(). Content is generated by the "@" lines that are translated +(by the "translate" preprocessor) into printf-like code that generates the +content of the webpage. Different techniques are used to generate +non-HTML content. In the unlikely event that you need to generate +non-HTML content, look at existing webpage implementations +(ex: "logo" or "style.css") to see how that is done. + +There are lots of other things that a real web-page implementation will +need to do, such as verifying user credentials, parsing query parameters, +and interacting with the repository. But now that you have the general +idea of how webpages are implemented, you can look at the many other +webpage implementations already built into Fossil to see how all that +works. + +<h2>6.0 See Also</h2> + + * [./makefile.wiki|The Fossil Build Process] + * [./tech_overview.wiki|A Technical Overview Of Fossil] + * [./contribute.wiki|Contributing To The Fossil Project] + * [./serverext.wiki|Adding CGI Extensions To A Fossil Server] diff --git a/www/alerts.md b/www/alerts.md new file mode 100644 index 0000000..6a516e9 --- /dev/null +++ b/www/alerts.md @@ -0,0 +1,727 @@ +# Email Alerts + +## Overview + +Beginning with version 2.7, Fossil can send email messages to +subscribers to alert them to changes in the repository: + + * New [checkins](/help/ci) + * [Ticket](./tickets.wiki) changes + * [Wiki](./wikitheory.wiki) page changes + * New and edited [forum](./forum.wiki) posts + * Users receiving [new permissions](./caps/index.md) (admins only) + * Announcements + +Subscribers can elect to receive emails as soon as these events happen, +or they can receive a daily digest of the events instead. + +Email alerts are sent by a [Fossil server](./server/), which must be +[set up](#quick) by the Fossil administrator to send email. + +Email alerts do not currently work if you are only using Fossil from the +command line. + +A bit of terminology: Fossil uses the terms "email alerts" and +"notifications" interchangeably. We stick to the former term in this +document except when referring to parts of the Fossil UI still using the +latter term. + + +## Setup Prerequisites + +Much of this document describes how to set up Fossil's email alert +system. To follow this guide, you will need a Fossil UI browser window +open to the [Admin → Notification](/setup_notification) Fossil UI screen +on the Fossil server that will be sending these email alerts, logged +in as a user with [**Admin** capability](./caps/ref.html#a). It is not possible to work on a +clone of the server's repository and push the configuration changes up +to that repo as an Admin user, [on purpose](#backup). + +<a id="cd"></a> +You will also need a CLI window open with its working directory changed +to a checkout directory of the Fossil repository you are setting up to +send email. If you don't `cd` to such a checkout directory first, +you'll need to add `-R /path/to/repo.fossil` to each `fossil` command +below to tell Fossil which repository you mean it to apply the command +to. + +There are other prerequisites for email service, but since they vary +depending on the configuration you choose, we'll cover these inline +below. + + +<a id="quick"></a> +## Quick Email Service Setup + +If you've already got a working OpenSMTPD, Postfix, Exim, Sendmail, +or similar server on the machine running your Fossil instance(s), +and you aren't using Fossil's [chroot jail feature](./chroot.md) +to wall Fossil off from the rest of the machine, it's +fairly simple to set up email alerts. + +(Otherwise, skip [ahead](#advanced) to the sections on advanced email +service setup.) + +This is our "quick setup" option even though setting up an SMTP mail +server is not trivial, because there are many other reasons to have such +a server set up already: internal project email service, `cron` +notifications, server status monitoring notifications... + +With that out of the way, the Fossil-specific steps are easy: + +1. Go to [Admin → Notification](/setup_notification) and fill out all + of the **Required** fields: + + * **Canonical server URL** — Use the suggested URL + + * **"From" email address** — `forum-bounces@example.com` is + traditional, but suit yourself + + * **Repository nickname** — See the suggested examples on the web page. + +2. Set "Email Send Method" to "Pipe to a command" + +3. Set the "Administrator email address" to a suitable valid email + address on that machine. It could be the same value you used for + the "From" address above, or it could be a different value like + `admin@example.com`. + +Save your changes. + +At the command line, say + + $ fossil set email-send-command + +If that gives a blank value instead of `sendmail -ti`, say + + $ fossil set email-send-command "sendmail -ti" + +to force the setting. That works around a [known +bug](https://fossil-scm.org/forum/forumpost/840b676410) which may be +squished by the time you read this. + +If you're running Postfix or Exim, you might think that command is +wrong, since you aren't running Sendmail. These mail servers provide a +`sendmail` command for compatibility with software like Fossil that has +no good reason to care exactly which SMTP server implementation is +running at a given site. There may be other SMTP servers that also +provide a compatible `sendmail` command, in which case they may work +with Fossil using the same steps as above. + +<a id="status"></a> +If you reload the Admin → Notification page, the Status section at the +top should show: + + Outgoing Email: Piped to command "sendmail -ti" + Pending Alerts: 0 normal, 0 digest + Subscribers: 0 active, 0 total + +Before you move on to the next section, you might like to read up on +[some subtleties](#pipe) with the "pipe to a command" method that we did +not cover above. + + +<a id="usage"></a> +## Usage and Testing + +Now that email service from Fossil is set up, you can test it and begin +using it. + + +<a id="sub" name="subscribe"></a> +### Subscribing to Alerts + +In the Status output above, we saw that there are no subscribers, so the +next step is to add the first one. + +Go to the `/subscribe` page on your Fossil instance to sign up for email +alerts. At the very least, you will need to sign up for "Forum Posts" +and "Announcements" to complete the testing steps below. + +If you're logged in with a Fossil repository user account and put the +same user name and email address into this forum as you used for your +user information under Admin → Users, Fossil will simply tie your alert +preferences to your login record, and the email address in your user's +Contact Info field will be considered already-verified. Otherwise, +Fossil will create an alert-only record, and you will have to verify the +email address before Fossil will send alerts to it. + +This shows a key aspect of the way Fossil's email alerts system works, +by the way: a user can be signed up for email alerts without having a +full-fledged Fossil user account. Only when both user names are the same +are the two records tied together under the hood. For more on this, see +[Users vs Subscribers below](#uvs). + +If you are seeing the following complaint from Fossil: + +> Use a different login with greater privilege than FOO to access /subscribe + +...then the repository's administrator forgot to give the +[**EmailAlert** capability][cap7] +to that user or to a user category that the user is a member of. + +After a subscriber signs up for alerts for the first time, a single +verification email is sent to that subscriber's given email address. +The new subscriber must click a link in that email in order to activate +the subscription. + +Subscription verification emails are only sent once. This is a defense +against malicious robots that try to harass innocent Internet users by +having subscription pages send multiple verification emails. If the +initial subscription verification does not go through correctly, an +administrator must [intervene](#admin) to reset the subscription. + +Every subscriber-only email address has a [long random hexadecimal +security code](#scode) that serves in place of a password. All email +alerts contain a link in their footer back to the Fossil server, +incorporating this security code, which allows the subscriber to adjust +their subscription options. If a user doesn't have any of those emails, +they can request a link via email by visiting the `/alerts` or +`/unsubscribe` page on the repository. + +Those with Fossil repository logins can adjust their email alert +settings by visiting the `/alerts` page on the repository. With the +default skin, you can get there by clicking the "Logout" link in the +upper right corner of any Fossil UI page then clicking the "Email +Alerts" link. That link is also available via the Sitemap (`/sitemap`) +and via the default skin's hamburger menu (☰). + +[cap7]: ./caps/ref.html#7 + + +<a id="unsub" name="unsubscribe"></a> +### Unsubscribing + +To unsubscribe from alerts, visit the `/alerts` page on the repository, +click the "Unsubscribe" button, then check the "Unsubscribe" checkbox to +verify your action and press the "Unsubscribe" button a second time. + +This interlock is intended to prevent accidental unsubscription. + + +<a id="test"></a> +### Test Email Service + +The easiest way to test email sending from Fossil is via the "[Send +Announcement](/announce)" link at the top of the "Email Notification +Setup" page. Put your email address in the "To:" line and a test +message below, then press "Send Message" to verify that outgoing email +is working. + +Another method is from the command line: + + $ fossil alerts test-message you@example.com --body README.md --subject Test + +That should send you an email with "Test" in the subject line and the +contents of your project's `README.md` file in the body. + +That command assumes that your project contains a "readme" file, but of +course it does, because you have followed the [Programming Style Guide +Checklist][cl], right? Right. + +[cl]: https://sendgrid.com/blog/programming-style-guide-checklist/ + + +<a id="cap7" name="ucap"></a> +### User Capabilities + +Once email alerts are working, you may need to [adjust the default user +capabilities](./caps/) to give "[Email Alerts][cap7]" capability to any +[user category](./caps/#ucat) or [individual user](./caps/#ucap) that +needs to use the subscription setup pages, `/subscribe` and `/alerts`. +[**Admin**][capa] and [**Setup**][caps] users always have this +capability. + +To allow any passer-by on the Internet to subscribe, give the "Email +Alerts" capability to the "nobody" user category. To require that a +person solve a simple CAPTCHA first, give that capability to the +"anonymous" user category instead. + +[capa]: ./caps/ref.html#a +[caps]: ./caps/ref.html#s + + +<a id="first" name="frist"></a> +### First Post + +I suggest taking the time to compose a suitable introductory message +especially for your project's forum, one which a new user would find +helpful. + +Wait a few seconds, and you should receive an email alert with the +post's subject and body text in the email. + + +<a id="trouble"></a> +### Troubleshooting + +If email alerts aren't working, there are several useful commands you +can give to figure out why. + +(Be sure to [`cd` into a repo checkout directory](#cd) first!) + + $ fossil alerts status + +This should give much the same information as you saw [above](#status). +One difference is that, since you've created a forum post, the +`pending-alerts` value should only be zero if you did in fact get the +requested email alert. If it's zero, check your mailer's spam folder. If +it's nonzero, continue with these troubleshooting steps. + + $ fossil backoffice + +That forces Fossil to run its ["back office" process](./backoffice.md). +Its only purpose at the time of this writing is to push out alert +emails, but it might do other things later. Sometimes it can get stuck +and needs to be kicked. For that reason, you might want to set up a +crontab entry to make sure it runs occasionally. + + $ fossil alerts send + +This should also kick off the backoffice processing, if there are any +pending alerts to send out. + + $ fossil alert pending + +Show any pending alerts. The number of lines output here should equal +the [status output above](#status). + + $ fossil test-add-alerts f5900 + $ fossil alert send + +Manually create an email alert and push it out immediately. + +The `f` in the first command's final parameter means you're scheduling a +"forum" alert. The integer is the ID of a forum post, which you can find +by visiting `/timeline?showid` on your Fossil instance. + +The second command above is necessary because the `test-add-alerts` +command doesn't kick off a backoffice run. + + $ fossil ale send + +This only does the same thing as the final command above, rather than +send you an ale, as you might be hoping. Sorry. + + +<a id="advanced"></a> +## Advanced Email Setups + +Fossil offers several methods of sending email: + + 1. Pipe the email message text into a command. + 2. Store email messages as entries in an SQLite database. + 3. Store email messages as individual files in a directory. + 4. Send emails to an SMTP relay. + 5. Send emails directly to the recipients via SMTP. + +This wide range of options allows Fossil to talk to pretty much any +SMTP setup. + +The first four options let Fossil delegate email handling to an existing +[MTA] so that Fossil does not need to implement the [roughly two +dozen][mprotos] separate [RFCs][rfcs] required in order to properly +support SMTP email in this complex world we've built. As well, this +design choice means you do not need to do duplicate configuration, such +as to point Fossil at your server's TLS certificate in order to support +users behind mail servers that require STARTTLS encryption. + +[mprotos]: http://sqlite.1065341.n5.nabble.com/Many-ML-emails-going-to-GMail-s-SPAM-tp98685p98722.html +[rfcs]: https://en.wikipedia.org/wiki/Request_for_Comments + + +<a id="pipe"></a> +### Method 1: Pipe to a Command + +This is our ["quick setup" option](#quick) above, but there are some +details we ignored which we'll cover now. + +Fossil pipes the email message in [RFC 822 format][rfc822] to the +standard input of the command you gave as the "Email Send Method", +defaulting to `sendmail -ti`. This constitutes a protocol between Fossil +and the SMTP [message transfer agent (MTA)][MTA]. Any other MTA which +speaks the same protocol can be used in place of the most common +options: Sendmail, Exim, and Postfix. + +The `-t` option tells the command to expect the list of email recipients +in a `To` header in the RFC 822 message presented on its standard input. +Without this option, the `sendmail` command expects to receive the +recipient list on the command line, but that's not possible with the +current design of this email sending method. Therefore, if you're +attempting to use a less common MTA which cannot parse the recipient +list from the `To` header in the email message, you might need to look +for a different MTA. + +The `-i` option is only needed for MTAs that take a dot/period at the +beginning of a line of standard input text as "end of message." Fossil +doesn't attempt to escape such dots, so if the line wrapping happens to +occur such that a dot or period in an alert message is at the beginning +of a line, you'll get a truncated email message without this option. +Statistically, this will happen about once every 70 or so messages, so +it is important to give this option if your MTA treats leading dots on a +line this way. + +<a id="msmtp"></a> +The [`msmtp`][msmtp] SMTP client is compatible with this +protocol if you give it the `-t` option. It’s a useful option on a server +hosting a Fossil repository which doesn't otherwise require a separate +SMTP server for other purposes, such as because you’ve got a separate +provider for your email and merely need a way to let Fossil feed +messages into it. + +It is probably also possible to configure [`procmail`][pmdoc] to work +with this protocol. If you know how to do it, a patch to this document +or a how-to on [the Fossil forum][ff] would be appreciated. + +[ff]: https://fossil-scm.org/forum/ +[msmtp]: https://marlam.de/msmtp/ +[MTA]: https://en.wikipedia.org/wiki/Message_transfer_agent +[pmdoc]: http://pm-doc.sourceforge.net/doc/ +[rfc822]: https://www.w3.org/Protocols/rfc822/ + + +<a id="db"></a> +### Method 2: Store in a Database + +The self-hosting Fossil repository at <https://fossil-scm.org/> +currently uses this method rather than [the pipe method](#pipe) because +it is running inside of a restrictive [chroot jail][cj] which is unable +to hand off messages to the local MTA directly. + +When you configure a Fossil server this way, it adds outgoing email +messages to an SQLite database file. A separate daemon process can then +extract those messages for further disposition. + +Fossil uses a short TCL script (seen at [](/file/tools/email-sender.tcl)) +that continuously monitors this database for new messages and hands any that +it finds off to a local MTA using the same [pipe to MTA protocol](#pipe) +as above. + +In this way, outbound email alerts escape the chroot jail without +requiring that we insert a separate MTA configuration inside that jail. +We only need to arrange that the same SQLite DB file be visible both +inside and outside the chroot jail, which we do by naming the database +file in the "Store Emails In This Database" setting under Admin → +Notification. The Tcl script has this path hard-coded as +`/home/www/fossil/emailqueue.db`, but you will probably need to adjust +that for your local purposes. + +This method may work with other similar technologies besides `chroot`: +Docker containers, LXC containers, BSD jails, Solaris zones, etc. + +With suitable file share mappings, this method may even work with +virtual machine or distributed computing setups where the MTA and Fossil +servers are not on the same machine, though beware the [risk of DB +corruption][rdbc] if used with a file sharing technology that doesn't +use proper file locking. + +You can start this Tcl script as a daemon automatically on most Unix and +Unix-like systems by adding the following line to the `/etc/rc.local` +file of the server that hosts the repository sending email alerts: + + /usr/bin/tclsh /home/www/fossil/email-sender.tcl & + +[cj]: https://en.wikipedia.org/wiki/Chroot +[rdbc]: https://www.sqlite.org/howtocorrupt.html#_filesystems_with_broken_or_missing_lock_implementations + + +<a id="dir"></a> +### Method 3: Store in a Directory + +This method is functionally very similar to [the DB method](#db), +differing only in that messages are written to a directory in the +filesystem. You should therefore read that section and make the minor +adjustments required by the storage method. + +This method may work over a file sharing mechanism that doesn't do file +locking properly, as long as the reading process is somehow restricted +from reading a message file as it's being written. + +It might be useful in testing and debugging to temporarily switch to +this method, since you can easily read the generated email messages +without needing to involve an [MTA]. + + +<a id="relay"></a> +### Method 4: SMTP Relay + +In this configuration, the Fossil server contacts an open SMTP relay and +sends the messages to it. This method is only appropriate when: + +1. You have a local MTA that doesn't accept [the pipe + protocol](#pipe). + +2. The MTA is willing to accept anonymous submissions, since Fossil + currently has no way to authenticate itself to the MTA. This is [an + unsafe configuration][omr] in most cases, but some SMTP servers make + an exception for connections coming from a `localhost` or LAN + address, choosing to accept such submissions as inherently safe. + +If you have a local MTA meeting criterion #1 but not #2, we'd suggest +using a more powerful SMTP client such as [msmtp](#msmtp) along with one +of the other methods above. + +[omr]: https://en.wikipedia.org/wiki/Open_mail_relay + + +<a id="direct"></a> +### Method 5: Direct SMTP Send + +As of Fossil 2.7, the code to support this method is incomplete, so you +cannot currently select it as an option in Admin → Notification. + + +<a id="uvs"></a> +## Users vs Subscribers + +Fossil makes a distinction between "users" and "subscribers". A user is +someone with a username and password: that is, someone who can log into +the Fossil repository. A subscriber is someone who receives email +alerts. Users can also be subscribers and subscribers can be users, but +that does not have to be the case. It is possible to be a user without +being a subscriber and to be a subscriber without being a user. + +In the repository database file, users are tracked with the `user` table +and subscribers are tracked via the `subscriber` table. + + +<a id="admin"></a> +## Administrator Activities + +The "[List Subscribers](/subscribers)" button at the top of the Admin → +Notification screen gives a list of subscribers, which gives a Fossil +server administrator a lot of power over those subscriptions. + +Clicking an email address in this subscriber list opens the same +`/alerts` page that the user can see for their own subscription, but +with more information and functionality than normal users get: + +* Subscription creation and modification timestamps. + +* The IP address the user had when they last made a change via either + `/subscribe` or `/alert`. + +* The user's login name, if they are not [a mere subscriber](#uvs). A + Fossil Admin user is allowed to modify this, either to tie a + subscription-only record to an existing Fossil user account or to + break that tie. + +* The "Do not call" checkbox allows a Fossil Admin user to mark a given + email address so that Fossil never sends email to that address. This + is distinct from unsubscribing that email address because it prevents + Fossil from accepting a new subscription for that address. + +* The Verified checkbox is initially unchecked for subscriber-only + email addresses until the user clicks the link in the verification + email. This checkbox lets the Fossil Admin user manually verify the + user, such as in the case where the verification email message got + lost. Unchecking this box does not cause another verification email + to be sent. + +* Admin users (only) may activate the "user elevation" subscription, + which sends a notification when a user is created or is explicitly + assigned permission they did not formerly have. + +This screen also allows a Fossil Admin user to perform other activities +on behalf of a subscriber which they could do themselves, such as to +[unsubscribe](#unsub) them. + + +<a id="backup"></a> +## Cloning, Syncing, and Backups + +That’s [covered elsewhere](./backup.md#alerts). + + +<a id="pages" name="commands"></a> +## Controlling the Email Alert System + +This section collects the list of Fossil UI pages and CLI commands that +control the email alert system, some of which have not been mentioned so +far: + +Commands: + + * The [`alerts`](/help/alerts) command + * The [`test-alert`](/help/test-alert) command + * The [`test-add-alerts`](/help/test-add-alerts) command + +Web pages available to users and subscribers: + + * The [`/subscribe`](/help/www/subscribe) page + * The [`/alerts`](/help/www/alerts) page + * The [`/unsubscribe`](/help/www/unsubscribe) page + * The [`/renew`](/help/www/renew) page + * The [`/contact_admin`](/help/www/contact_admin) page + +Administrator-only web pages: + + * The [`/setup_notification`](/help/www/setup_notification) page + * The [`/subscribers`](/help/www/subscribers) page + + +<a id="design"></a> +## Design of Email Alerts + +This section describes the low-level design of the email alert system in +Fossil. This expands on the high-level administration focused material +above with minimal repetition. + +This section assumes expert-level systems knowledge. If the material +above sufficed for your purposes, feel free to skip this section, which +runs to the end of this document. + + +<a id="datades"></a> +### Data Design + +There are two new tables in the repository database, starting with +Fossil 2.7. These tables are not created in new repositories by +default. The tables only come into existence as needed when email +alerts are configured and used. + + * <b>SUBSCRIBER</b> → + The subscriber table records the email address for people who + want to receive email notifications. Each subscriber has a + `subscriberCode` which is a random 32-byte blob that uniquely + identifies the subscriber. There are also fields to indicate + what kinds of notifications the subscriber wishes to receive, + whether or not the email address of the subscriber has been + verified, etc. + + * <b>PENDING\_ALERT</b> → + The PENDING\_ALERT table contains records that define events + about which alert emails might need to be sent. + A pending\_alert always refers to an entry in the + EVENT table. The EVENT table is part of the standard schema + and records timeline entries. In other words, there is one + row in the EVENT table for each possible timeline entry. The + PENDING\_ALERT table refers to EVENT table entries for which + we might need to send alert emails. + +As pointed out above, ["subscribers" are distinct from "users"](#uvs). +The SUBSCRIBER.SUNAME field is the optional linkage between users and +subscribers. + + +<a id="stdout"></a> +### The "stdout" Method + +The [list of mail sending methods](#advanced) above left out an +internal-only method called "stdout" which simply writes the text of the +email message on standard output. The "stdout" method is used for +testing and debugging. If you need something similar and can't modify +your local Fossil instance to use this method, you might temporarily +switch to [the "dir" method](#dir) instead. + + +<a id="msgfmt"></a> +### Message Format + +The email messages generated by Fossil have a [well-formed +header][rfc822]. The downstream processing is expected to extract the +"To:", "From:", "Subject:" and whatever other attributes it needs from +the email header text. + +These emails use the `text/plain` MIME type with the UTF-8 character +set. We currently use a transfer encoding of `quoted-printable`, but +there is commented-out code in Fossil to switch to `base64` encoding, +which Fossil used in the early days leading up to the 2.7 release. + +If you switch Fossil back to `base64` mode, you may want to build a +utility program that ships in the Fossil source tree named +["tools/decode-email.c"](/file/tools/decode-email.c) which can decode +these messages into a human-readable format. + + +<a id="inbound" name="bounces"></a> +### Dealing with Inbound Email + +Inbound email messages — for example, bounces from failed alert emails — +should be relayed to the `fossil email inbound` command. That command +is currently a no-op place-holder. At some point, we will need to +design and write a bounce-message processing system for Fossil. + + +<a id="password" name="scode" name="verification"></a> +### Passwords vs Subscriber Codes + +When anonymous passers-by on the Internet sign up for email alerts, +their email address must first be verified. An email message is sent to +the address supplied inviting the user to click on a link. The link +includes a pseudorandom 128-bit blob encoded as 32 hexadecimal digits, +which serves in place of a password for that email address. (This is +stored in the database as `subscriber.subscriberCode`.) If anyone visits +the link, the email address is verified. + +Knowledge of the `subscriberCode` is sufficient to control a +subscription. + +Because this code is included in plain text in email alert messages, it +is not as secure as a separate password, but it has several virtues: + +* It is easier for the average subscriber to deal with in that they + don't have to come up with yet another password and store it safely. + +* If the `subscriberCode` is stolen, the worst that can happen is that + the thief can change that email address's subscription settings. + Contrast a password which may be shared with other services, which + then compromises those other services. + +* No PII other than the subscriber's email address is available to an + attacker with the `subscriberCode`. Nor can knowledge of the + `subscriberCode` lead to an email flood or other annoyance attack, as + far as I can see. + +If the `subscriberCodes` for a Fossil repository are ever compromised, +new ones can be generated as follows: + + UPDATE subscriber SET subscriberCode=randomblob(32); + +Since this then affects all new email alerts going out from Fossil, your +end users may never even realize that they're getting new codes, as long +as they don't click on the URLs in the footer of old alert messages. + +With that in mind, a Fossil server administrator could choose to +randomize the `subscriberCodes` periodically, such as just before the +daily digest emails are sent out each day. + +**Important:** All of the above is distinct from the passwords for users +with a Fossil repository login. Such users also have subscriber codes, +but those codes can only be used to modify the user's email alert +settings. That code cannot allow a user to log into the user's Fossil +repository account. + + +<a id="processing"></a> +### Internal Processing Flow + +Almost all of the email alert code is found in the +[`src/alerts.c`](/file/src/alerts.c) source file. + +When email alerts are enabled, a trigger is created in the schema +(`email_trigger1`) that adds a new entry to the `PENDING_ALERT` table +every time a row is added to the `EVENT` table. During a +`fossil rebuild`, the `EVENT` table is rebuilt from scratch; since we do not +want users to get alerts for every historical check-in, the trigger is +disabled during `rebuild`. + +Email alerts are sent out by the `alert_send_alerts()` function, which +is normally called automatically due to the `email-autoexec` setting, +which defaults to enabled. If that setting is disabled or if the user +simply wants to force email alerts to be sent immediately, they can give +a `fossil alert send` command, such as via a `cron` script. Each time +this function is called, the alert messages are moved further down the +chain, so you cannot cause duplicate alerts by calling it too often. + +Digests are handled by recording the time of the last digest in the +`email-last-digest` setting, and only sending a new digest if the +current time is one day or later after the last digest. + +Individual emails are sent to each subscriber. I (drh) ran tests and +found that I could send about 1200 emails/second, which is fast enough +that I do not need to resort to trying to notify multiple subscribers +with a single email. Because each subscriber gets a separate email, the +system can include information in the email that is unique to the +subscriber, such as a link to the page to edit their subscription. That +link includes the `subscriberCode`. diff --git a/www/antibot.wiki b/www/antibot.wiki new file mode 100644 index 0000000..c38cfec --- /dev/null +++ b/www/antibot.wiki @@ -0,0 +1,276 @@ +<title>Defense Against Robots</title> + +A typical Fossil website can have billions and billions of pages, +and many of those pages (for example diffs and annotations and tarballs) +can be expensive to compute. +If a robot walks a Fossil-generated website, +it can present a crippling bandwidth and CPU load. +A "robots.txt" file can help, but in practice, most robots these +days ignore the robots.txt file, so it won't help much. + +A Fossil website is intended to be used +interactively by humans, not walked by robots. This article +describes the techniques used by Fossil to try to welcome human +users while keeping out robots. + +<h2>Defenses Are Enabled By Default</h2> + +In the latest implementations of Fossil, most robot defenses are +enabled by default. You can probably get by with standing up a +public-facing Fossil instance in the default configuration. But +you can also customize the defenses to serve your particular needs. + +<h2>Customizing Anti-Robot Defenses</h2> + +Admin users can configure robot defenses on the +"Robot Defense Settings" page (/setup_robot). +That page is accessible (to Admin users) from the default menu bar +by click on the "Admin" menu choice, then selecting the +"Robot-Defense" link from the list. + +<h2>The Hyperlink User Capability</h2> + +Every Fossil web session has a "user". For random passers-by on the internet +(and for robots) that user is "nobody". The "anonymous" user is also +available for humans who do not wish to identify themselves. The difference +is that "anonymous" requires a login (using a password supplied via +a CAPTCHA) whereas "nobody" does not require a login. +The site administrator can also create logins with +passwords for specific individuals. + +Users without the <b>[./caps/ref.html#h | Hyperlink]</b> capability +do not see most Fossil-generated hyperlinks. This is +a simple defense against robots, since [./caps/#ucat | the "nobody" +user category] does not have this capability by default. +Users must log in (perhaps as +"anonymous") before they can see any of the hyperlinks. A robot +that cannot log into your Fossil repository will be unable to walk +its historical check-ins, create diffs between versions, pull zip +archives, etc. by visiting links, because there are no links. + +A text message appears at the top of each page in this situation to +invite humans to log in as anonymous in order to activate hyperlinks. + +But requiring a login, even an anonymous login, can be annoying. +Fossil provides other techniques for blocking robots which +are less cumbersome to humans. + +<h2>Automatic Hyperlinks Based on UserAgent and Javascript</h2> + +Fossil has the ability to selectively enable hyperlinks for users +that lack the <b>Hyperlink</b> capability based on their UserAgent string in the +HTTP request header and on the browsers ability to run Javascript. + +The UserAgent string is a text identifier that is included in the header +of most HTTP requests that identifies the specific maker and version of +the browser (or robot) that generated the request. Typical UserAgent +strings look like this: + +<ul> +<li> Mozilla/5.0 (Windows NT 6.1; rv:19.0) Gecko/20100101 Firefox/19.0 +<li> Mozilla/4.0 (compatible; MSIE 8.0; Windows_NT 5.1; Trident/4.0) +<li> Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) +<li> Wget/1.12 (openbsd4.9) +</ul> + +The first two UserAgent strings above identify Firefox 19 and +Internet Explorer 8.0, both running on Windows NT. The third +example is the robot used by Google to index the internet. +The fourth example is the "wget" utility running on OpenBSD. +Thus the first two UserAgent strings above identify the requester +as human whereas the second two identify the requester as a robot. +Note that the UserAgent string is completely under the control +of the requester and so a malicious robot can forge a UserAgent +string that makes it look like a human. But most robots want +to "play nicely" on the internet and are quite open +about the fact that they are a robot. And so the UserAgent string +provides a good first-guess about whether or not a request originates +from a human or a robot. + +The [/help/auto-hyperlink|auto-hyperlink] setting, shown as +"<b>Enable hyperlinks based on User-Agent and/or Javascript</b>" on +the Robot Defense Settings page, +can be set to "HTTP Header Only" or "HTTP Header and Javascript" or "off". +If the UserAgent string looks like a human and not a robot, then +Fossil will enable hyperlinks even if the <b>Hyperlink</b> capability +is omitted from the user permissions. This setting gives humans easy +access to the hyperlinks while preventing robots +from walking the billions of pages on a typical Fossil site. + +If the setting is "HTTP Header Only" (2), then the hyperlinks are simply +enabled and that is all. But if the setting is "HTTP Header and Javascript" (1), +then the hyperlinks are not enabled directly. +Instead, the HTML code that is generated contains anchor tags ("<a>") +with "href=" attributes that point to [/honeypot] rather than the correct +link. JavaScript code is added to the end of the page that goes back and +fills in the correct "href=" attributes of +the anchor tags with the true hyperlink targets, thus enabling the hyperlinks. +This extra step of using JavaScript to enable the hyperlink targets +is a security measure against robots that forge a human-looking +UserAgent string. Most robots do not bother to run JavaScript and +so to the robot the empty anchor tag will be useless. But all modern +web browsers implement JavaScript, so hyperlinks will show up +normally for human users. +(UPDATE 2026-08-06: The previous two sentences +were true when this document was first written in 2013, but they are +not true any more. Most robots these days have normal-looking UserAgent +strings and do run Javascript.) + +If the [/help/auto-hyperlink|"auto-hyperlink"] setting is (2) +"<b>Enable hyperlinks using HTTP-Header and/or Javascript</b>", +then there are now two additional sub-settings that control when +hyperlinks are enabled. + +The first new sub-setting is a delay (in milliseconds) before setting +the "href=" attributes on anchor tags. The default value for this +delay is 10 milliseconds. The idea here is that a robots will try to +interpret the links on the page immediately, and will not wait for delayed +scripts to be run, and thus will never enable the true links. + +The second sub-setting waits to run the +JavaScript that sets the "href=" attributes on anchor tags until after +at least one "mousedown" or "mousemove" event has been detected on the +<body> element of the page. The thinking here is that robots will not be +simulating mouse motion and so no mouse events will ever occur and +hence the hyperlinks will never become enabled for robots. + +See also [./loadmgmt.md|Managing Server Load] for a description +of how expensive pages can be disabled when the server is under heavy +load. + +<h2>Do Not Allow Robot Access To Certain Pages</h2> + +The [/help/robot-restrict|robot-restrict setting] is a comma-separated +list of GLOB patterns for pages for which robot access is prohibited. +The default value is: + +<blockquote><pre> +timelineX,diff,annotate,fileage,file,finfo,reports,tree,hexdump,download +</pre></blockquote> + +Each entry corresponds to the first path element on the URI for a +Fossil-generated page. If Fossil does not know for certain that the +HTTP request is coming from a human, then any attempt to access one of +these pages brings up a javascript-powered captcha. The user has to +click the accept button the captcha once, and that sets a cookie allowing +the user to continue surfing without interruption for 15 minutes or so +before being presented with another captcha. + +To disable this feature (not recommended for public-facing websites) +change this setting to "off". If robots become particularly troublesome, +change this setting to "*" (the GLOB wildcard) which restricts robots +from accessing any page of the Fossil webpage other than "/login". +Robots have gotten so bad lately (as of 2026) that the use of "*" is +advisable for any public-facing website that is running on a low-power +platform such as a RaspberryPI or a $6/month VPS, or any platform with +limited bandwidth. + +Some path elements have special meanings: + + * <b>timelineX →</b> + This means a subset of /timeline/ pages that are considered + "expensive". The exact definition of which timeline pages are + expensive and which are not is still the subject of active + experimentation and is likely to change by the time you read this + text. The idea is that anybody (including robots) can see a timeline + of the most recent changes, but timelines of long-ago change or that + contain lists of file changes or other harder-to-compute values are + prohibited. + + * <b>zip →</b> + The special "zip" keyword also matches "/tarball/" and "/sqlar/". + + * <b>zipX →</b> + This is like "zip" in that it restricts access to "/zip/", "/tarball"/ + and "/sqlar/" but with exceptions:<ol type="a"> + <li><p> If the [/help/robot-zip-leaf|robot-zip-leaf] setting is + true, then tarballs of leaf check-ins are allowed. This permits + URLs that attempt to download the latest check-in on trunk or + from a named branch, for example. + <li><p> If a check-in has a tag that matches the GLOB list in + [/help/robot-zip-tag|robot-zip-tag], then tarballs of that + check-in are allowed. This allow check-ins tagged with + "release" or "allow-robots" (for example) to be downloaded + without restriction. + </ol> + The "zipX" restriction is not in the default robot-restrict setting. + This is something you might want to add, depending on your needs. + + * <b>diff →</b> + This matches /vdiff/ and /fdiff/ and /vpatch/ and any other page that + is primarily about showing the difference between two check-ins or two + file versioons. + + * <b>annotate →</b> + This also matches /blame/ and /praise/. + +Other special keywords may be added in the future. + +The default [/help/robot-restrict|robot-restrict] +setting has been shown (circa 2026) to do a reasonable job of keeping +robots from consuming all available CPU and bandwidth while will +still allowing humans access to the full power of the site without +having to be logged in. + +One possible enhancement is to add "zipX" to the +[/help/robot-restrict|robot-restrict] setting, +and enable [help?cmd=robot-zip-leaf|robot-zip-leaf] +and configure [help?cmd=robot-zip-tag|robot-zip-tag]. +Do this if you find that robots downloading lots of +obscure tarballs is causing load issues on your site. + +<h2>Anti-robot Exception RegExps</h2> + +The [/help/robot-exception|robot-exception setting] under the name +of <b>Exceptions to anti-robot restrictions</b> is a list of +[/re_rules|regular expressions], one per line, that match +URIs that will bypass the captcha and allow robots full access. The +intent of this setting is to allow automated build scripts +to download specific tarballs of project snapshots. + +The recommended value for this setting allows robots to use URIs of the +following form: + +<blockquote> +<b>https://</b><i>DOMAIN</i><b>/tarball/release/</b><i>HASH</i><b>/</b><i>NAME</i><b>.tar.gz</b> +</blockquote> + +The <i>HASH</i> part of this URL can be any valid +[./checkin_names.wiki|check-in name]. The link works as long as that +check-in is tagged with the "release" symbolic tag. In this way, +robots are permitted to download tarballs (and ZIP archives) of official +releases, but not every intermediate check-in between releases. Humans +who are willing to click the captcha can still download whatever they +want, but robots are blocked by the captcha. This prevents aggressive +robots from downloading tarballs of every historical check-in of your +project, once per day, which many robots these days seem eager to do. + +For example, on the Fossil project itself, this URL will work, even for +robots: + +<blockquote> +https://fossil-scm.org/home/tarball/release/version-2.27/fossil-scm.tar.gz +</blockquote> + +But the next URL will not work for robots because check-in 3bbd18a284c8bd6a +is not tagged as a "release": + +<blockquote> +https://fossil-scm.org/home/tarball/release/3bbd18a284c8bd6a/fossil-scm.tar.gz +</blockquote> + +The second URL will work for humans, just not robots. + +<h2>The Ongoing Struggle</h2> + +Fossil currently does a good job of providing easy access to humans +while keeping out troublesome robots. However, robots +continue to grow more sophisticated, requiring ever more advanced +defenses. This "arms race" is unlikely to end. The developers of +Fossil will continue to try improve the robot defenses of Fossil so +check back from time to time for the latest releases and updates. + +Readers of this page who have suggestions on how to improve the robot +defenses in Fossil are invited to submit your ideas to the Fossil Users +forum: +[https://fossil-scm.org/forum]. diff --git a/www/apple-touch-icon.png b/www/apple-touch-icon.png Binary files differnew file mode 100644 index 0000000..5b9178b --- /dev/null +++ b/www/apple-touch-icon.png diff --git a/www/background.jpg b/www/background.jpg Binary files differnew file mode 100644 index 0000000..f788555 --- /dev/null +++ b/www/background.jpg diff --git a/www/backoffice.md b/www/backoffice.md new file mode 100644 index 0000000..b49b7f9 --- /dev/null +++ b/www/backoffice.md @@ -0,0 +1,225 @@ +Backoffice +========== + +This is technical documentation about the internal workings of Fossil. +Ordinary Fossil users do not need to know about anything covered by this +document. The information here is intended for people who want to enhance +or extend the Fossil code, or who just want a deeper understanding of +the internal workings of Fossil. + +What Is The Backoffice +---------------------- + +The backoffice is a mechanism used by a +[Fossil server](./server/) to do low-priority +background work that is not directly related to the user interface. Here +are some examples of the kinds of work that backoffice performs: + + 1. Sending email alerts and notifications + 2. Sending out daily digests of email notifications + 3. Other background email handling chores + 4. Automatic syncing of peer repositories + 5. Repository maintenance and optimization + +(As of 2018-08-07, only items 1 and 2 have actually been implemented.) +The idea is that the backoffice handles behind-the-scenes work that does +not have tight latency requirements. + +When Backoffice Runs +-------------------- + +A backoffice process is usually launched automatically by a webpage +request. After each webpage is generated, Fossil checks to see if any +backoffice work needs to be done. If there is work to do, and no other +process is already assigned to do the work, then a new backoffice process +is started to do the work. + +This happens for every webpage, regardless of how that webpage is launched, +and regardless of the purpose of the webpage. This also happens on the +server for "[fossil sync](/help/sync)" and +"[fossil clone](/help/clone)" commands which are implemented as +web requests - albeit requests that the human user never sees. +Web requests can arrive at the Fossil server via direct TCP/IP (for example +when Fossil is started using commands like "[fossil server](/help/server)") +or via [CGI](./server/any/cgi.md) or +[SCGI](./server/any/scgi.md) or via SSH. +A backoffice process might be started regardless of the origin of the +request. + +The backoffice is not a daemon. Each backoffice process runs for a short +while and then exits. This helps keep Fossil easy to manage, since there +are no daemons to start and stop. To upgrade Fossil to a new version, +you simply replace the older "fossil" executable with the newer one, and +the backoffice processes will (within a minute or so) start using the new +one. (Upgrading the executable on Windows is more complicated, since on +Windows it is not possible to replace an executable file that is in active +use. But Windows users probably already know this.) + +The backoffice is serialized and rate limited. No more than a single +backoffice process will be running at once, and backoffice runs will not +occur more frequently than once every 60 seconds. (The 60-second spacing +is controlled by the BKOFCE_LEASE_TIME macro in the +[backoffice.c](/file/src/backoffice.c) source file.) + +If a Fossil server is idle, then no backoffice processes will be running. +That means there are no extra processes sitting around taking up memory +and process table slots for seldom accessed repositories. +The backoffice is an on-demand system. +A busy repository will usually have a backoffice +running at all times. But an infrequently accessed repository will only have +backoffice processes running for a minute or two following the most recent +access. + +Manually Running The Backoffice +------------------------------- + +The automatic backoffice runs are sufficient for most installations. +However, the daily digest of email notifications is handled by the +backoffice. If a Fossil server can sometimes go more than a day without +being accessed, then the automatic backoffice will never run, and the +daily digest might not go out until somebody does visit a webpage. +If this is a problem, an administrator can set up a cron job to +periodically run: + + fossil backoffice _REPOSITORY_ + +That command will cause backoffice processing to occur immediately. +Note that this is almost never necessary for an internet-facing +Fossil repository, since most repositories will get multiple accesses +per day from random robots, which will be sufficient to kick off the +daily digest emails. And even for a private server, if there is very +little traffic, then the daily digests are probably a no-op anyhow +and won't be missed. + +Automatic Backoffice Does Not Work On Some Systems +-------------------------------------------------- + +We have observed that the automatic backoffice does not work on +some systems - OpenBSD in particular. We still do not understand why +this is. (If you have insights, please share them on the +[Fossil Forum](https://fossil-scm.org/forum) so that we can perhaps +fix the problem.) For now, the backoffice must be run manually +on OpenBSD systems. + +To set up a fully-manual backoffice, first disable the automatic backoffice +using the "[backoffice-disable](/help/backoffice-disable)" setting. + + fossil setting backoffice-disable on + +Then arrange to invoke the backoffice separately using a command +like this: + + fossil backoffice --poll 30 _REPOSITORY-LIST_ + +Multiple repositories can be named. This one command will handle +launching the backoffice for all of them. There are additional useful +command-line options. See the "[fossil backoffice](/help/backoffice)" +documentation for details. + +The backoffice processes that are run manually using the "fossil backoffice" +command do not normally use a lease. That means that if you run the +"fossil backoffice" command with --poll and you forget to disable +automatic backoffice by setting the "backoffice-disable" flag, then +you might have one backoffice running due to a command and another due +to a webpage access, both at the same time. This is harmless. The +only downside is that it uses extra CPU time. + +How Backoffice Is Implemented +----------------------------- + +The backoffice is implemented by the +"[backoffice.c](/file/src/backoffice.c)" source file. + +Serialization and rate limiting is handled by a single entry in the +repository database CONFIG table named "backoffice". This entry is +called "the lease". The value of the lease +is a text string representing four integers, which +are respectively: + + 1. The process id of the "current" backoffice process + 2. The lease expiration time of the current backoffice process + 3. The process id of the "next" backoffice process + 4. The lease expiration time for the next backoffice process + +Times are expressed in seconds since 1970. A process id of zero means +"no process". Sometimes the process id will be non-zero even if there +is no corresponding process. Fossil knows how to figure out whether or +not a process still exists. + +You can print out a decoded copy of the current backoffice lease using +this command: + + fossil test-backoffice-lease -R _REPOSITORY_ + +If a system has been idle for a long time, then there will be no +backoffice processes. (Either the process id entries in the lease +will be zero, or there will exist no process associated with the +process id.) When a new web request comes in, the system +sees that no backoffice process is active and so it kicks off a separate +process to run backoffice. + +The new backoffice process becomes the "current" process. It sets a +lease expiration time for itself to be 60 seconds in the future. +Then it does the backoffice processing and exits. Note that usually +the backoffice process will exit long before its lease expires. That +is ok. The lease is there to limit the rate at which backoffice processes +run. + +If a new backoffice process starts up and sees that the "current" lease has +yet to expire, the new process makes itself the "next" backoffice process +and sets its expiration time to be 60 seconds past the expiration time of +the "current" backoffice process. The "next" process then puts itself to +sleep until the "current" lease expires. After the "current" +lease expires and the "current" process has itself exited, then +the "next" process promotes itself to the new "current" process. It +sets the current lease expiration to be 60 seconds in the future, runs +whatever backoffice work is needed, then exits. + +If a new backoffice process starts up and finds that there is already +a "current" lease and a "next" process, it exits without doing anything. +This should happen only rarely, since the lease information is checked +prior to spawning the backoffice process, so a conflict will only happen +in a race. + +Because the "backoffice" entry of the CONFIG table is in the repository +database, access to the lease is serialized. The lease prevents more +than one backoffice process from running at a time. It prevents +backoffice processes from running more frequently than once every 60 seconds. +And, it guarantees (assuming processes are not killed out-of-band) that +every web request will be followed within 60 seconds by a backoffice +run. + +Debugging The Backoffice +------------------------ + +The backoffice should "just work". It should not require administrator +attention. However, if you suspect that something is not working right, +there are some debugging aids. + +We have already mentioned the command that shows the backoffice lease +for a repository: + + fossil test-backoffice-lease -R _REPOSITORY_ + +Running that command every few seconds should show what is going on with +backoffice processing in a particular repository. + +There are also settings that control backoffice behavior. The +"backoffice-nodelay" setting prevents the "next" process from taking a +lease and sleeping. If "backoffice-nodelay" is set, that causes all +backoffice processes to exit either immediately or after doing whatever +backoffice works needs to be done. If something is going wrong and +backoffice leases are causing delays in webpage processing, then setting +"backoffice-nodelay" to true can work around the problem until the bug +can be fixed. The "backoffice-logfile" setting is the name of a log +file onto which is appended a short message every time a backoffice +process actually starts to do the backoffice work. This log file can +be used to verify that backoffice really is running, if there is any +doubt. The "backoffice-disable" setting prevents automatic backoffice +processing, if true. Use this to completely disable backoffice processing +that occurs automatically after each HTTP request. The "backoffice-disable" +setting does not affect the operation of the manual +"fossil backoffice" command. +Most installations should leave "backoffice-nodelay" and "backoffice-disable" +set to their default values of off and +leave "backoffice-logfile" unset or set to an empty string. diff --git a/www/backup.md b/www/backup.md new file mode 100644 index 0000000..198be34 --- /dev/null +++ b/www/backup.md @@ -0,0 +1,304 @@ +# Backing Up a Remote Fossil Repository + +One of the great benefits of Fossil and other [distributed version control systems][dvcs] +is that cloning a repository makes a backup. If you are running a project with multiple +developers who share their work using a [central server][server] and the server hardware +catches fire, the clones of the repository on each developer +workstation *may* serve as a suitable backup. + +[dvcs]: wikipedia:/wiki/Distributed_version_control +[server]: ./server/whyuseaserver.wiki + +We say “may” because +it turns out not everything in a Fossil repository is copied when cloning. You +don’t even always get copies of all historical file artifacts. More than +that, a Fossil repository typically contains +other useful information that is not always shared as part of a clone, which might need +to be backed up separately. To wit: + + +## <a id="pii"></a> Sensitive Information + +Fossil purposefully does not clone certain sensitive information unless +you’re logged in as a user with [Setup] capability. As an example, a local clone +may have a different `user` table than the remote, because only a +Setup user is allowed to see the full version for privacy and security +reasons. + + +## <a id="config"></a> Configuration Drift + +Fossil allows the local configuration to differ in several areas from +that of the remote. You get a copy +of *some* of these configuration areas on initial clone — not all! — but after that, +remote configuration changes mostly do not sync down automatically. + + +#### <a id="skin"></a> Skin + +Changes to the remote’s skin don’t sync down, on purpose, since you may +want to have a different skin on the local clone than on the remote. You +can ask for updates with [`fossil config pull skin`][cfg], but that does +not happen automatically during the course of normal development. + + +#### <a id="alerts"></a> Email Alerts + +The Admin → Notification settings do not get copied on clone or sync, +and it is not possible to push such settings from one repository to +another. We did this on purpose because you may have a network of peer +repositories, and you only want one repository sending email alerts. If +Fossil were to automatically replicate the email alert settings to a +separate repository, subscribers would get multiple alerts for each +event, which would be *bad.* + +The only element of the email alert configuration that can be pulled +over the sync protocol on demand is the subscriber list, via +[`fossil config pull subscriber`][cfg]. + + +#### <a id="project"></a> Project Configuration + +This is normally generated once during `fossil init` and never changed, +so Fossil doesn’t pull this information without being forced, on +purpose. You could accidentally merge two separate Fossil repos by +pushing one repo’s project config up to another, for example. + + +#### <a id="other-cfg"></a> Others + +A repo’s URL aliases, [interwiki configuration](./interwiki.md), and +[ticket customizations](./custom_tcket.wiki) also do not normally sync. + +[cfg]: /help/configuration + + + +## <a id="private"></a> Private Branches + +The very nature of Fossil’s [private branch feature][pbr] ensures that +remote clones don’t get a copy of those branches. Normally this is +exactly what you want, but in the case of making backups, you probably +want to back up these branches as well. One of the two backup methods below +provides this. + + +## <a id="shun"></a> Shunned Artifacts + +Fossil purposefully doesn’t sync [shunned artifacts][shun]. If you want +your local clone to be a precise match to the remote, it needs to track +changes to the shun table as well. + + +## <a id="uv"></a> Unversioned Artifacts + +Data in Fossil’s [unversioned artifacts table][uv] doesn’t sync down by +default unless you specifically ask for it. Like local configuration +data, it doesn’t get pulled as part of a normal `fossil sync`, but +*unlike* the config data, you don’t get unversioned files as part of the +initial clone unless you ask for it by passing the `--unversioned/-u` +flag. + + +## <a id="ait"></a>Autosync Is Intransitive + +If you’re using Fossil in a truly distributed mode, rather than the +simple central-and-clones model that is more common, there may be no +single source of truth in the network because Fossil’s autosync feature +isn’t transitive. + +That is, if you cloned from server A, and then you stand that up on a +server B, then if I clone from your server as my repository C, your changes to B +autosync up to A, but not down to me on C until I do something locally +that triggers autosync. The inverse is also true: if I commit something +on C, it will autosync up to B, but A won’t get a copy until someone on +B does something to trigger a sync there. + +An easy way to run into this problem is to set up failover servers +`svr1` thru `svr3.example.com`, then set `svr2` and `svr3` up to sync +with the first. If all of the users normally clone from `svr1`, their +commits don’t get to `svr2` and `svr3` until something on one of the +servers pushes or pulls the changes down to the next server in the sync +chain. + +Likewise, if `svr1` falls over and all of the users re-point their local +clones at `svr2`, then `svr1` later reappears, `svr1` is likely to +remain a stale copy of the old version of the repository until someone +causes it to sync with `svr2` or `svr3` to catch up again. And then if +you originally designed the sync scheme to treat `svr1` as the primary +source of truth, those users still syncing with `svr2` won’t have their +commits pushed up to `svr1` unless you’ve set up bidirectional sync, +rather than have the two backup servers do `pull` only. + + +# <a id="sync-solution"></a> Solution 1: Explicit Pulls + +The following script solves most of the above problems for the use case +where you want a *nearly-complete* clone of the remote repository using nothing +but the normal Fossil sync protocol. It only does so if you are logged into +the remote as a user with Setup capability, however. + +``` shell +#!/bin/sh +fossil sync --unversioned +fossil configuration pull all +fossil rebuild +``` + +The last step is needed to ensure that shunned artifacts on the remote +are removed from the local clone. The second step includes +`fossil conf pull shun`, but until those artifacts are actually rebuilt +out of existence, your backup will be “more than complete” in the sense +that it will continue to have information that the remote says should +not exist any more. That would be not so much a “backup” as an +“archive,” which might not be what you want. + + +# <a id="sql-solution"></a> Solution 2: SQL-Level Backup + +The first method doesn’t get you a copy of the remote’s +[private branches][pbr], on purpose. It may also miss other info on the +remote, such as SQL-level customizations that the sync protocol can’t +see. (Some [ticket system customization][tkt] schemes rely on this ability, for example.) You can +solve such problems if you have access to the remote server, which +allows you to get a SQL-level backup by delegating handling of locking +and transaction isolation to +[the `backup` command][bu], allowing the user to safely back up an in-use +repository. + +If you have SSH access to the remote server, something like this will work: + +``` shell +#!/bin/bash +bf=repo-$(date +%Y-%m-%d).fossil +ssh example.com "cd museum ; fossil backup -R repo.fossil backups/$bf" && + scp example.com:museum/backups/$bf ~/museum/backups +``` + +Beware that this method does not solve [the intransitive sync +problem](#ait), in and of itself: if you do a SQL-level backup of a +stale repo DB, you have a *stale backup!* You should therefore run this +on every node that may need to serve as a backup so that at least *one* +of the backups is also up-to-date. + + +# <a id="enc"></a> Encrypted Off-Site Backups + +A useful refinement that you can apply to both methods above is +encrypted off-site backups. You may wish to store backups of your +repositories off-site on a service such as Dropbox, Google Drive, iCloud, +or Microsoft OneDrive, where you don’t fully trust the service not to +leak your information. This addition to the prior scripts will encrypt +the resulting backup in such a way that the cloud copy is a useless blob +of noise to anyone without the key: + +```shell +iter=152830 +pass="h8TixP6Mt6edJ3d6COaexiiFlvAM54auF2AjT7ZYYn" +gd="$HOME/Google Drive/Fossil Backups/$bf.xz.enc" +fossil sql -R ~/museum/backups/"$bf" .dump | xz -9 | + openssl enc -e -aes-256-cbc -pbkdf2 -iter $iter -pass pass:"$pass" -out "$gd" +``` + +If you’re adding this to the first script above, remove the +“`-R repo-name`” bit so you get a dump of the repository backing the +current working directory. + +Change the `pass` value to some other long random string, and change the +`iter` value to something in the hundreds of thousands range. A good source for +the first is [here][grcp], and for the second, [here][rint]. + +You may find posts online written by people recommending millions of +iterations for PBKDF2, but they’re generally talking about this in the +context of memorizable passwords, where adding even one more character +to the password is a significant burden. Given our script’s purely +random maximum-length passphrase, there isn’t much more that increasing +the key derivation iteration count can do for us. + +Conversely, if you were to reduce the passphrase to 41 characters, that +would drop the key strength by roughly 2⁶, being the entropy value per +character for using most of printable ASCII in our passphrase. To make +that lost strength up on the PBKDF2 end, you’d have to multiply your +iterations by 2⁶ = 64 times. It’s easier to use a max-length passphrase +in this situation than get crazy with key derivation iteration counts. + +(This, by the way, is why the example passphrase above is 42 characters: +with 6 bits of entropy per character, that gives you a key size of 252, +as close as we can get to our chosen encryption algorithm’s 256-bit key +size without going over. If it pleases you to give it 43 random +characters for a passphrase in order to pick up those last four bits of +security, you’re welcome to do so.) + +Compressing the data before encrypting it removes redundancies that can +make decryption easier, and it results in a smaller backup than you get +with the previous script alone, at the expense of a lot of CPU time +during the backup. You may wish to switch to a less space-efficient +compression algorithm that takes less CPU power, such as [`lz4`][lz4]. +Changing up the compression algorithm also provides some +security-thru-obscurity, which is useless on its own, but it *is* a +useful adjunct to strong encryption. + +This requires OpenSSL 1.1 or higher. If you’re on 1.0 or older, you +won’t have the `-pbkdf2` and `-iter` options, and you may have to choose +a different cipher algorithm; both changes are likely to weaken the +encryption significantly, so you should install a newer version rather +than work around the lack of these features. + +Beware that macOS ships a fork of OpenSSL called [LibreSSL][lssl] that +lacked this capability until Ventura (13.0). If you’re on Monterey (12) +or older, we recommend use of the [Homebrew][hb] OpenSSL package rather +than give up on the security afforded by use of configurable-iteration +PBKDF2. To avoid a conflict with the platform’s `openssl` binary, +Homebrew’s installation is [unlinked][hbul] by default, so you have to +give an explicit path to it, one of: + + /usr/local/opt/openssl/bin/openssl ... # Intel x86 Macs + /opt/homebrew/opt/openssl/bin/openssl ... # ARM Macs (“Apple silicon”) + +[lssl]: https://www.libressl.org/ + + +## <a id="rest"></a> Restoring From An Encrypted Backup + +The “restore” script for the above fragment is basically an inverse of +it, but it’s worth showing it because there are some subtleties to take +care of. If all variables defined in earlier scripts are available, then +restoration is: + +``` +openssl enc -d -aes-256-cbc -pbkdf2 -iter $iter -pass pass:"$pass" -in "$gd" | + xz -d | fossil sql --no-repository ~/museum/restored-repo.fossil +``` + +We changed the `-e` to `-d` on the `openssl` command to get decryption, +and we changed the `-out` to `-in` so it reads from the encrypted backup +file and writes the result to stdout. + +The decompression step is trivial. + +The last change is tricky: we used `fossil sql` above to ensure that +we’re using the same version of SQLite to write the encrypted backup DB +as was used to maintain the repository. We must also do that on +restoration: +Fossil serves as a dogfooding project for SQLite, +often making use of the latest features, so it is quite likely that a given +random `sqlite3` binary in your `PATH` will be unable to understand the +file created by “`fossil sql .dump`”! The tricky bit is, you can’t just +pipe the decrypted SQL dump into `fossil sql`, because on startup, Fossil +normally goes looking for tables created by `fossil init`, and it won’t +find them in a newly-created repo DB. We get around this by passing +the `--no-repository` flag, which suppresses this behavior. Doing it +this way saves you from needing to go and build a matching version of +`sqlite3` just to restore the backup. + +[bu]: /help/backup +[grcp]: https://www.grc.com/passwords.htm +[hb]: https://brew.sh +[hbul]: https://docs.brew.sh/FAQ#what-does-keg-only-mean +[lz4]: https://lz4.github.io/lz4/ +[pbr]: ./private.wiki +[rint]: https://www.random.org/integers/?num=1&min=100000&max=1000000&col=5&base=10&format=html&rnd=new +[Setup]: ./caps/admin-v-setup.md#apsu +[shun]: ./shunning.wiki +[tkt]: ./tickets.wiki +[uv]: ./unvers.wiki diff --git a/www/blame.wiki b/www/blame.wiki new file mode 100644 index 0000000..f3ffb3e --- /dev/null +++ b/www/blame.wiki @@ -0,0 +1,62 @@ +<title>The Annotate Algorithm</title> + +<h2>1.0 Introduction</h2> + +The [/help/annotate|fossil annotate], +[/help/blame|fossil blame], and +[/help/praise|fossil praise] commands, and the +[/help/www/annotate|/annotate], +[/help/www/blame|/blame], and +[/help/www/praise|/praise] web pages are all used to show the most +recent check-in that modified each line of a particular file. +This article overviews the algorithm used to compute the annotation +for a file in Fossil. + +<h2>2.0 Algorithm</h2> + +<ol type='1'> +<li>Locate the check-in that contains the file that is to be + annotated. Call this check-in C0. +<li>Find all direct ancestors of C0. A direct ancestor is the closure + of the primary parent of C0. Merged in branches are not part of + the direct ancestors of C0. +<li>Prune the list of ancestors of C0 so that it contains only + check-ins in which the file to be annotated was modified. +<li>Load the complete text of the file to be annotated from check-in C0. + Call this version of the file F0. +<li>Parse F0 into lines. Mark each line as "unchanged". +<li>For each ancestor of C0 on the pruned list (call the ancestor CX), + beginning with the most + recent ancestor and moving toward the oldest ancestor, do the + following steps: +<ol type='a'> +<li>Load the text for the file to be annotated as it existed in check-in CX. + Call this text FX. +<li>Compute a diff going from FX to F0. +<li>For each line of F0 that is changed in the diff and which was previously + marked "unchanged", update the mark to indicated that line + was modified by CX. +</ol> +<li>Show each line of F0 together with its change mark, appropriately + formatted. +</ol> + +<h2>3.0 Discussion and Notes</h2> + +The time-consuming part of this algorithm is step 6b - computing the +diff from all historical versions of the file to the version of the file +under analysis. For a large file that has many historical changes, this +can take several seconds. For this reason, the default +[/help/www/annotate|/annotate] webpage only shows those lines that were +changed by the 20 most recent modifications to the file. This allows +the loop on step 6 to terminate after only 19 diffs instead of the hundreds +or thousands of diffs that might be required for a frequently modified file. + +As currently implemented (as of 2015-12-12) the annotate algorithm does not +follow files across name changes. File name change information is +available in the database, and so the algorithm could be enhanced to follow +files across name changes by modifications to step 3. + +Step 2 is interesting in that it is +[/artifact/6cb824a0417?ln=196-201 | implemented] using a +[https://www.sqlite.org/lang_with.html#recursivecte|recursive common table expression]. diff --git a/www/blockchain.md b/www/blockchain.md new file mode 100644 index 0000000..af8c613 --- /dev/null +++ b/www/blockchain.md @@ -0,0 +1,478 @@ +# Is Fossil A Blockchain? + +The Fossil version control system shares a lot of similarities with +other blockchain based technologies, but it also differs from the more common +sorts of blockchains. This document will discuss the term’s +applicability, so you can decide whether applying the term to Fossil +makes sense to you. + + +## The Dictionary Argument + +The [Wikipedia definition of "blockchain"][bcwp] begins: + +> + "A blockchain…is a growing list of records, called blocks, which are linked using + cryptography… Each block contains a cryptographic hash of the previous + block, a timestamp, and transaction data (generally represented as a Merkle tree)." + +Point-for-point, Fossil follows this partial definition. +The blocks +are Fossil’s ["manifest" artifacts](./fileformat.wiki#manifest). Each +manifest has a cryptographically-strong [SHA-1] or [SHA-3] hash linking it to +one or more “parent” blocks. The manifest also contains a timestamp and +the transactional data needed to express a commit to the repository. +To traverse the Fossil repository from the tips of its [DAG] to the +root by following the parent hashes in each manifest is to traverse +a Merkle tree. +Every change in Fossil starts by adding one or more manifests to +the repository, extending this Merkle tree. + +[bcwp]: https://en.wikipedia.org/wiki/Blockchain +[DAG]: https://en.wikipedia.org/wiki/Directed_acyclic_graph +[SHA-1]: https://en.wikipedia.org/wiki/SHA-1 +[SHA-3]: https://en.wikipedia.org/wiki/SHA-3 + + + +<a id="currency"></a> +## Cryptocurrency + +Because blockchain technology was first popularized as Bitcoin, many +people associate the term with cryptocurrency. Fossil has nothing to do +with cryptocurrency, so a claim that “Fossil is a blockchain” may fail +to communicate the speaker’s concepts clearly due to conflation with +cryptocurrency. + +Cryptocurrency has several features and requirements that Fossil doesn’t +provide, either because it doesn’t need them or because we haven’t +gotten around to creating the feature. Whether these are essential to +the definition of “blockchain” and thus disqualify Fossil as a blockchain +is for you to decide. + +Cryptocurrencies must prevent three separate types of fraud to be useful: + +* **Type 1** is modification of existing currency. To draw an analogy + to paper money, we wish to prevent someone from using green and + black markers to draw extra zeroes on a US $10 bill so that it + claims to be a $100 bill. + +* **Type 2** is creation of new fraudulent currency that will pass + in commerce. To extend our analogy, it is the creation of new + US $10 bills. There are two sub-types to this fraud. In terms of + our analogy, they are: + + * **Type 2a**: copying an existing legitimate $10 bill<br><br> + + * **Type 2b**: printing a new $10 bill that is unlike an existing + legitimate one, yet which will still pass in commerce + +* **Type 3** is double-spending existing legitimate cryptocurrency. + There is no analogy in paper money due to its physical form; it is a + problem unique to digital currency due to its infinitely-copyable + nature. + +How does all of this compare to Fossil? + +1. <a id="signatures"></a>**Signatures.** Cryptocurrencies use a chain + of [digital signatures][dsig] to prevent Type 1 and Type 3 frauds. This + chain forms an additional link between the blocks, separate from the + hash chain that applies an ordering and lookup scheme to the blocks. + [_Blockchain: Simple Explanation_][bse] explains this “hash chain” + vs. “block chain” distinction in more detail. + + These signatures prevent modification of the face value of each + transaction (Type 1 fraud) by ensuring that only the one signing a + new block has the private signing key that could change an issued + block after the fact. + + The fact that these signatures are also *chained* prevents Type + 3 frauds by making the *prior* owner of a block sign it over to + the new owner. To avoid an O(n²) auditing problem as a result, + cryptocurrencies add a separate chain of hashes to make checking + for double-spending quick and easy. + + Fossil has [a disabled-by-default feature][cs] to call out to an + external copy of [PGP] or [GPG] to sign commit manifests before + inserting them into the repository. You can couple that with + a server-side [after-receive hook][arh] to reject unsigned commits. + + Although there are several distinctions you can draw between the way + Fossil’s commit signing scheme works and the way block signing works + in cryptocurrencies, only one is of material interest for our + purposes here: Fossil commit signatures apply only to a single + commit. Fossil does not sign one commit over to the next “owner” of + that commit in the way that a blockchain-based cryptocurrency must + when transferring currency from one user to another, beacuse there + is no useful analog to the double-spending problem in Fossil. The + closest you can come to this is double-insert of commits into the + blockchain, which we’ll address shortly. + + What Fossil commit signatures actually do is provide in-tree forgery + prevention, both Type 1 and Type 2. You cannot modify existing + commits (Type 1 forgery) because you do not have the original + committer’s private signing key, and you cannot forge new commits + attesting to come from some other trusted committer (Type 2) because + you don’t have any of their private signing keys, either. + Cryptocurrencies use the work problem to prevent Type 2 + forgeries, but the application of that to Fossil is a matter we get + to [later](#work). + + Although you have complete control over the contents of your local + Fossil repository clone, you cannot perform Type 1 forgery on its + contents short of executing a [preimage attack][prei] on the hash + algorithm. ([SHA3-256][SHA-3] by default in the current version of + Fossil.) Even if you could, Fossil’s sync protocol will prevent the + modification from being pushed into another repository: the remote + Fossil instance says, “I’ve already got that one, thanks,” and + ignores the push. Thus, short of breaking into the remote server + and modifying the repository in place, you couldn’t make use of + a preimage attack even if you had that power. Further, that would be an attack on the + server itself, not on Fossil’s data structures, so while it is + useful to think through this problem, it is not helpful in answering + our questions here. + + The Fossil sync protocol’s duplication detection also prevents the closest analog to Type 3 + frauds in Fossil: copying a commit manifest in your local repo clone + won’t result in a double-commit on sync. + + In the absence of digital signatures, Fossil’s [RBAC system][caps] + restricts Type 2 forgery to trusted committers. Thus once again + we’re reduced to an infosec problem, not a data structure design + question. + + (Inversely, enabling commit clearsigning is a good idea + if you have committers on your repo whom you don’t trust not to + commit Type 2 frauds. But let us be clear: your choice of setting + does not answer the question of whether Fossil is a blockchain.) + + If Fossil signatures prevent Type 1 and Type 2 frauds, you + may wonder why they are not enabled by default. It is because + they are defense-in-depth measures, not the minimum sufficient + measures needed to prevent repository fraud, unlike the equivalent + protections in a cryptocurrency blockchain. Fossil provides its + primary protections through other means, so it doesn’t need to + mandate signatures. + + Also, Fossil is not itself a [PKI], and there is no way for regular + users of Fossil to link it to a PKI, since doing so would likely + result in an unwanted [PII] disclosure. There is no email address + in a Fossil commit manifest that you could use to query one of the + public PGP keyservers, for example. It therefore becomes a local + policy matter as to whether you even *want* to have signatures, + because they’re not without their downsides. + +2. <a id="work"></a>**Work Contests.** Cryptocurrencies prevent Type 2b forgeries + by setting up some sort of contest that ensures that new coins can come + into existence only by doing some difficult work task. This “mining” + activity results in a coin that took considerable work to create, + which thus has economic value by being a) difficult to re-create, + and b) resistant to [debasement][dboc]. + + Fossil repositories are most often used to store the work product of + individuals, rather than cryptocoin mining machines. There is + generally no contest in trying to produce the most commits. There + may be an implicit contest to produce the “best” commits, but that + is a matter of project management, not something that can be + automatically mediated through objective measures. + + Incentives to commit to the repository come from outside of Fossil; + they are not inherent to its nature, as with cryptocurrencies. + Moreover, there is no useful sense in which we could say that one + commit “re-creates” another. Commits are generally products of + individual human intellect, thus necessarily unique in all but + trivial cases. This is foundational to copyright law. + +3. <a id="lcr"></a>**Longest Chain Rule.** Cryptocurrencies generally + need some way to distinguish which blocks are legitimate and which + not. They do this in part by identifying the linear chain with the + greatest cumulative [work time](#work) as the legitimate chain. All + blocks not on that linear chain are considered “orphans” and are + ignored by the cryptocurrency software. + + Its inverse is sometimes called the “51% attack” because a single + actor would have to do slightly more work than the entire rest of + the community using a given cryptocurrency in order for their fork + of the currency to be considered the legitimate fork. This argument + soothes concerns that a single bad actor could take over the + network. + + The closest we can come to that notion in Fossil is the default + “trunk” branch, but there’s nothing in Fossil that delegitimizes + other branches just because they’re shorter, nor is there any way in + Fossil to score the amount of work that went into a commit. Indeed, + [forks and branches][fb] are *valuable and desirable* things in + Fossil. + +This much is certain: Fossil is definitely not a cryptocurrency. Whether +this makes it “not a blockchain” is a subjective matter. + +[arh]: ./hooks.md +[bse]: https://www.researchgate.net/publication/311572122_What_is_Blockchain_a_Gentle_Introduction +[caps]: ./caps/ +[cs]: /help/clearsign +[dboc]: https://en.wikipedia.org/wiki/Debasement +[dsig]: https://en.wikipedia.org/wiki/Digital_signature +[fb]: ./branching.wiki +[GPG]: https://gnupg.org/ +[PGP]: https://www.openpgp.org/ +[PII]: https://en.wikipedia.org/wiki/Personal_data +[PKI]: https://en.wikipedia.org/wiki/Public_key_infrastructure +[pow]: https://en.wikipedia.org/wiki/Proof_of_work +[prei]: https://en.wikipedia.org/wiki/Preimage_attack + + + +<a id="dlt"></a> +## Distributed Ledgers + +Cryptocurrencies are an instance of [distributed ledger technology][dlt]. If +we can convince ourselves that Fossil is also a distributed +ledger, then we might think of Fossil as a peer technology, +having at least some qualifications toward being considered a blockchain. + +A key tenet of DLT is that records be unmodifiable after they’re +committed to the ledger, which matches quite well with Fossil’s design +and everyday use cases. Fossil puts up multiple barriers to prevent +modification of existing records and injection of incorrect records. + +Yet, Fossil also has [purge] and [shunning][shun]. Doesn’t that mean +Fossil cannot be a distributed ledger? + +These features only remove existing commits from the repository. If you want a +currency analogy, they are ways to burn a paper bill or to melt a [fiat +coin][fc] down to slag. In a cryptocurrency, you can erase your “wallet” +file, effectively destroying money in a similar way. These features +do not permit forgery of either type described above: you can’t use them +to change the value of existing commits (Type 1) or add new commits to +the repository (Type 2). + +What if we removed those features from Fossil, creating an append-only +Fossil variant? Is it a DLT then? Arguably still not, because [today’s Fossil +is an AP-mode system][ctap], which means +there can be no guaranteed consensus on the content of the ledger at any +given time. An AP-mode accounts receivable system would allow +different bottom-line totals at different sites, because you’ve +cast away “C” to get AP-mode operation. (See the prior link or +[Wikipedia’s article on the CAP theorem][cap] if you aren’t following +this terminology.) + +By the same token, you cannot guarantee that the command +“`fossil info tip`” gives the same result everywhere. You would need to +recast Fossil as a CA or CP-mode system to solve that. +(Everyone not +partitioned away from the majority of the network at any rate, in the CP +case.) + +What are the prospects for CA-mode or CP-mode Fossil? [We don’t want +CA-mode Fossil][ctca], but [CP-mode could be useful][ctcp]. Until the latter +exists, this author believes Fossil is not a distributed ledger in a +technologically defensible sense. + +The most common technologies answering to the label “blockchain” are all +DLTs, so if Fossil is not a DLT, then it is not a blockchain in that +sense. + +[ctap]: ./cap-theorem.md#ap +[ctca]: ./cap-theorem.md#ca +[ctcp]: ./cap-theorem.md#cp +[cap]: https://en.wikipedia.org/wiki/CAP_theorem +[dlt]: https://en.wikipedia.org/wiki/Distributed_ledger +[DVCS]: https://en.wikipedia.org/wiki/Distributed_version_control +[fc]: https://en.wikipedia.org/wiki/Fiat_money +[purge]: /help/purge +[shun]: ./shunning.wiki + + +<a id="dpc"></a> +## Distributed Partial Consensus + +If we can’t get DLT, can we at least get some kind of distributed +consensus at the level of individual Fossil’s commits? + +Many blockchain based technologies have this property: given some +element of the blockchain, you can make certain proofs that it either is +a legitimate part of the whole blockchain, or it is not. + +Unfortunately, this author doesn’t see a way to do that with Fossil. +Given only one “block” in Fossil’s putative “blockchain” — a commit, in +Fossil terminology — all you can prove is whether it is internally +consistent, that it is not corrupt. That then points you at the parent(s) of that +commit, which you can repeat the exercise on, back to the root of the +DAG. This is what the enabled-by-default [`repo-cksum` setting][rcks] +does. + +If cryptocurrencies worked this way, you wouldn’t be able to prove that +a given cryptocoin was legitimate without repeating the proof-of-work +calculations for the entire cryptocurrency scheme! Instead, you only +need to check a certain number of signatures and proofs-of-work in order +to be reasonably certain that you are looking at a legitimate section of +the whole blockchain. + +What would it even mean to prove that a given Fossil commit “*belongs*” +to the repository you’ve extracted it from? For a software project, +isn’t that tantamount to automatic code review, where the server would +be able to reliably accept or reject a commit based solely on its +content? That sounds nice, but this author believes we’ll need to invent +[AGI] first. + +A better method to provide distributed consensus for Fossil would be to +rely on the *natural* intelligence of its users: that is, distributed +commit signing, so that a commit is accepted into the blockchain only +once some number of users countersign it. This amounts to a code review +feature, which Fossil doesn’t currently have. + +Solving that problem basically requires solving the [PKI] problem first, +since you can’t verify the proofs of these signatures if you can’t first +prove that the provided signatures belong to people you trust. This is a +notoriously hard problem in its own right. + +A future version of Fossil could instead provide [consensus in the CAP +sense][ctcp]. For instance, you could say that if a quorum of servers +all have a given commit, it “belongs.” Fossil’s strong hashing tech +would mean that querying whether a given commit is part of the +“blockchain” would be as simple as going down the list of servers and +sending each an HTTP GET `/info` query for the artifact ID, concluding +that the commit is legitimate once you get enough HTTP 200 status codes back. All of this is +hypothetical, because Fossil doesn’t do this today. + +[AGI]: https://en.wikipedia.org/wiki/Artificial_general_intelligence +[rcks]: /help/repo-cksum + + + +<a id="anon"></a> +## Anonymity + +Many blockchain based technologies go to extraordinary lengths to +allow anonymous use of their service. + +As typically configured, Fossil does not: commits synced between servers +always at least have a user name associated with them, which the remote +system must accept through its [RBAC system][caps]. That system can run +without having the user’s email address, but it’s needed if [email +alerts][alert] are enabled on the server. The remote server logs the IP +address of the commit for security reasons. That coupled with the +timestamp on the commit could sufficiently deanonymize users in many +common situations. + +It is possible to configure Fossil so it doesn’t do this: + +* You can give [Write capability][capi] to user category “nobody,” so + that anyone that can reach your server can push commits into its + repository. + +* You could give that capability to user category “anonymous” instead, + which requires that the user log in with a CAPTCHA, but which doesn’t + require that the user otherwise identify themselves. + +* You could enable [the `self-register` setting][sreg] and choose not to + enable [commit clear-signing][cs] so that anonymous users could push + commits into your repository under any name they want. + +On the server side, you can also [scrub] the logging that remembers +where each commit came from. + +Commit source info isn’t transmitted from the remote server on clone or pull: +the size of the `rcvfrom` table after initial clone is 1, containing +only the remote server’s IP address. On each pull containing new +artifacts, your local `fossil` instance adds another entry to this +table, likely with the same IP address unless the server has moved or +you’re using [multiple remotes][mrep]. This table is far more +interesting on the server side, containing the IP addresses of all +contentful pushes; thus [the `scrub` command][scrub]. + +Because Fossil doesn’t +remember IP addresses in commit manifests or require commit signing, it +allows at least *pseudonymous* commits. When someone clones a remote +repository, they don’t learn the email address, IP address, or any other +sort of [PII] of prior committers, on purpose. + +Some people say that private, permissioned blockchains (as you may +imagine Fossil to be) are inherently problematic by the very reason that +they don’t bake anonymous contribution into their core. The very +existence of an RBAC is a moving piece that can break. Isn’t it better, +the argument goes, to have a system that works even in the face of +anonymous contribution, so that you don’t need an RBAC? Cryptocurrencies +do this, for example: anyone can “mine” a new coin and push it into the +blockchain, and there is no central authority restricting the transfer +of cryptocurrency from one user to another. + +We can draw an analogy to encryption, where an algorithm is +considered inherently insecure if it depends on keeping any information +from an attacker other than the key. Encryption schemes that do +otherwise are derided as “security through obscurity.” + +You may be wondering what any of this has to do with whether Fossil is a +blockchain, but that is exactly the point: all of this is outside +Fossil’s core hash-chained repository data structure. If you take the +position that you don’t have a “blockchain” unless it allows anonymous +contribution, with any needed restrictions provided only by the very +structure of the managed data, then Fossil does not qualify. + +Why do some people care about this distinction? Consider Bitcoin, +wherein an anonymous user cannot spam the blockchain with bogus coins +because its [proof-of-work][pow] protocol allows such coins to be +rejected immediately. There is no equivalent in Fossil: it has no +technology that allows the receiving server to look at the content of a +commit and automatically judge it to be “good.” Fossil relies on its +RBAC system to provide such distinctions: if you have a commit bit, your +commits are *ipso facto* judged “good,” insofar as any human work +product can be so judged by a blob of compiled C code. This takes us +back to the [digital ledger question](#dlt), where we can talk about +what it means to later correct a bad commit that got through the RBAC +check. + +We may be willing to accept pseudonymity, rather than full anonymity. +If we configure Fossil as above, either bypassing the RBAC or abandoning +human control over it, scrubbing IP addresses, etc., is it then a public +permissionless blockchain in that sense? + +We think not, because there is no [longest chain rule](#lcr) or anything +like it in Fossil. + +For a fair model of how a Fossil repository might behave under such +conditions, consider GitHub: here one user can fork another’s repository +and make an arbitrary number of commits to their public fork. Imagine +this happens 10 times. How does someone come along later and +*automatically* evaluate which of the 11 forks of the code (counting the +original repository among their number) is the “best” one? For a +computer software project, the best we could do to approximate this +devolves to a [software project cost estimation problem][scost]. These +methods are rather questionable in their own right, being mathematical +judgement values on human work products, but even if we accept their +usefulness, then we still cannot say which fork is better based solely +on their scores under these metrics. We may well prefer to use the fork +of a software program that took *less* effort, being smaller, more +self-contained, and with a smaller attack surface. + + +[alert]: ./alerts.md +[capi]: ./caps/ref.html#i +[mrep]: /help/remote +[scost]: https://en.wikipedia.org/wiki/Software_development_effort_estimation +[scrub]: /help/scrub +[sreg]: /help/self-register + + +# Conclusion + +This author believes it is technologically indefensible to call Fossil a +“blockchain” in any sense likely to be understood by a majority of those +you’re communicating with. Using a term in a nonstandard way just because you can +defend it means you’ve failed any goal that requires clear communication. +The people you’re communicating your ideas to must have the +same concept of the terms you use. + +What term should you use instead? Fossil stores a DAG of hash-chained +commits, so an indisputably correct term is a [Merkle tree][mt], named +after [its inventor][drrm]. You could also use the more generic term +“hash tree.” + +Fossil is a technological peer to many common sorts of blockchain +technology. There is a lot of overlap in concepts and implementation +details, but when speaking of what most people understand as +“blockchain,” Fossil is not that. + +[drrm]: https://en.wikipedia.org/wiki/Ralph_Merkle +[mt]: https://en.wikipedia.org/wiki/Merkle_tree diff --git a/www/branching.wiki b/www/branching.wiki new file mode 100644 index 0000000..90eb265 --- /dev/null +++ b/www/branching.wiki @@ -0,0 +1,767 @@ +<title>Branching, Forking, Merging, and Tagging</title> +<h2>Background</h2> + +In a simple and perfect world, the development of a project would proceed +linearly, as shown in Figure 1. + +<verbatim type="pikchr center toggle"> +ALL: [circle rad 0.1in thickness 1.5px "1" +arrow right 40% +circle same "2" +arrow same +circle same "3" +arrow same +circle same "4"] +box invis "Figure 1" big fit with .n at .3cm below ALL.s +</verbatim> + +Each circle represents a check-in. For the sake of clarity, the check-ins +are given small consecutive numbers. In a real system, of course, the +check-in numbers would be long hexadecimal hashes since it is not possible +to allocate collision-free sequential numbers in a distributed system. +But as sequential numbers are easier to read, we will substitute them for +the long hashes in this document. + +The arrows in Figure 1 show the evolution of a project. The initial +check-in is 1. Check-in 2 is derived from 1. In other words, check-in 2 +was created by making edits to check-in 1 and then committing those edits. +We say that 2 is a <i>child</i> of 1 +and that 1 is a <i>parent</i> of 2. +Check-in 3 is derived from check-in 2, making +3 a child of 2. We say that 3 is a <i>descendant</i> of both 1 and 2 and that 1 +and 2 are both <i>ancestors</i> of 3. + +<h2 id="dag">DAGs</h2> + +The graph of check-ins is a +[http://en.wikipedia.org/wiki/Directed_acyclic_graph | directed acyclic graph], +commonly shortened to <i>DAG</i>. Check-in 1 is the <i>root</i> of the DAG +since it has no ancestors. Check-in 4 is a <i>leaf</i> of the DAG since +it has no descendants. (We will give a more precise definition later of +"leaf.") + +Alas, reality often interferes with the simple linear development of a +project. Suppose two programmers make independent modifications to check-in 2. +After both changes are committed, the check-in graph looks like Figure 2: + +<verbatim type="pikchr center toggle"> +ALL: [circle rad 0.1in thickness 1.5px "1" +arrow right 40% +circle same "2" +circle same "3" at 2nd circle+(.4,.3) +arrow from 2nd circle to 3rd circle chop +circle same "4" at 2nd circle+(.4,-.3) +arrow from 2nd circle to 4th circle chop] +box invis "Figure 2" big fit with .n at .3cm below ALL.s +</verbatim> + +The graph in Figure 2 has two leaves: check-ins 3 and 4. Check-in 2 has +two children, check-ins 3 and 4. We call this state a <i>fork</i>. + +Fossil tries to prevent forks, primarily through its +"[./concepts.wiki#workflow | autosync]" mechanism. + +Suppose two programmers named Alice and +Bob are each editing check-in 2 separately. Alice finishes her edits +and commits her changes first, resulting in check-in 3. When Bob later +attempts to commit his changes, Fossil verifies that check-in 2 is still +a leaf. Fossil sees that check-in 3 has occurred and aborts Bob's commit +attempt with a message "would fork." This allows Bob to do a "fossil +update" to pull in Alice's changes, merging them into his own +changes. After merging, Bob commits check-in 4 as a child of check-in 3. +The result is a linear graph as shown in Figure 1. This is how CVS +works. This is also how Fossil works in autosync mode. + +But perhaps Bob is off-network when he does his commit, so he has no way +of knowing that Alice has already committed her changes. Or, it could +be that Bob has turned off "autosync" mode in Fossil. Or, maybe Bob +just doesn't want to merge in Alice's changes before he has saved his +own, so he forces the commit to occur using the "--allow-fork" option to +the <b>[/help/commit | fossil commit]</b> command. For any of these +reasons, two commits against check-in 2 have occurred, so the DAG now +has two leaves. + +In such a condition, a person working with this repository has a +dilemma: which version of the project is the "latest" in the sense of +having the most features and the most bug fixes? When there is more +than one leaf in the graph, you don't really know, which is why we +would ideally prefer to have linear check-in graphs. + +Fossil resolves such problems using the check-in time on the leaves to +decide which leaf to use as the parent of new leaves. When a branch is +forked as in Figure 2, Fossil will choose check-in 4 as the parent for a +later check-in 5, but <i>only</i> if it has sync'd that check-in down +into the local repository. If autosync is disabled or the user is +off-network when that fifth check-in occurs so that check-in 3 is the +latest on that branch at the time within that clone of the repository, +Fossil will make check-in 3 the parent of check-in 5! We show practical +consequences of this [#bad-fork | later in this article]. + +Fossil also uses a forked branch's leaf check-in timestamps when +checking out that branch: it gives you the fork with the latest +check-in, which in turn selects which parent your next check-in will be +a child of. This situation means development on that branch can fork +into two independent lines of development, based solely on which branch +tip is newer at the time the next user starts his work on it. + +Because of these potential problems, we strongly recommend that you do +not intentionally create forks on long-lived shared working branches +with "--allow-fork". (Prime example: trunk.) The inverse case — +intentional forks on short-lived single-developer branches — is far +easier to justify, since presumably the lone developer is never confused +about why there are two or more leaves on that branch. Further +justifications for intentional forking are [#forking | given below]. + +Let us return to Figure 2. To resolve such situations before they can +become a real problem, Alice can use the <b>[/help/merge | fossil +merge]</b> command to merge Bob's changes into her local copy of +check-in 3. Without arguments, that command merges all leaves on the +current branch. Alice can then verify that the merge is sensible and if +so, commit the results as check-in 5. This results in a DAG as shown in +Figure 3. + +<verbatim type="pikchr center toggle"> +ALL: [circle rad 0.1in thickness 1.5px "1" +arrow right 40% +circle same "2" +circle same "3" at 2nd circle+(.4,.3) +arrow from 2nd circle to 3rd circle chop +circle same "4" at 2nd circle+(.4,-.3) +arrow from 2nd circle to 4th circle chop +circle same "5" at 3rd circle+(.4,-.3) +arrow from 3rd circle to 5th circle chop +arrow dashed .03 from 4th circle to 5th circle chop] +box invis "Figure 3" big fit with .n at .2cm below ALL.s +</verbatim> + +Check-in 5 is a child of check-in 3 because it was created by editing +check-in 3, but since check-in 5 also inherits the changes from check-in 4 by +virtue of the merge, we say that check-in 5 is a <i>merge child</i> +of check-in 4 and that it is a <i>direct child</i> of check-in 3. +The graph is now back to a single leaf, check-in 5. + +We have already seen that if Fossil is in autosync mode then Bob would +have been warned about the potential fork the first time he tried to +commit check-in 4. If Bob had updated his local check-out to merge in +Alice's check-in 3 changes, then committed, the fork would have +never occurred. The resulting graph would have been linear, as shown +in Figure 1. + +Realize that the graph of Figure 1 is a subset of Figure 3. If you hold your +hand over the ④ in Figure 3, it looks +exactly like Figure 1 except that the leaf has a different check-in +number. That is just a notational difference: the two check-ins +have exactly the same content. + +Inversely, Figure 3 is a +superset of Figure 1. The check-in 4 of Figure 3 captures additional +state which is omitted from Figure 1. Check-in 4 of Figure 3 holds a +copy of Bob's local checkout before he merged in Alice's changes. That +snapshot of Bob's changes, which is independent of Alice's changes, is +omitted from Figure 1. + +Some people say that the development approach taken in +Figure 3 is better because it preserves this extra intermediate state. +Others say that the approach taken in Figure 1 is better because it is +much easier to visualize linear development and because the +merging happens automatically instead of as a separate manual step. We +will not take sides in that debate. We will simply point out that +Fossil enables you to do it either way. + +<h2 id="branching">The Alternative to Forking: Branching</h2> + +Having more than one leaf in the check-in DAG is called a "fork." This +is usually undesirable and either avoided entirely, +as in Figure 1, or else quickly resolved as shown in Figure 3. +But sometimes, one does want to have multiple leaves. For example, a project +might have one leaf that is the latest version of the project under +development and another leaf that is the latest version that has been +tested. +When multiple leaves are desirable, we call this <i>branching</i> +instead of <i>forking</i>: + +Figure 4 shows an example of a project where there are two branches, one +for development work and another for testing. + +<verbatim type="pikchr center toggle"> +ALL: [circle rad 0.1in thickness 1.5px fill white "1" +arrow 40% +C2: circle same "2" +arrow same +circle same "3" +arrow same +C5: circle same "5" +arrow same +C7: circle same "7" +arrow same +C8: circle same "8" +arrow same +C10: circle same "10" +C4: circle same at 3rd circle-(0,.35) "4" +C6: circle same at (1/2 way between C5 and C7,C4) "6" +C9: circle same at (1/2 way between C8 and C10,C4) "9" +arrow from C2 to C4 chop +arrow from C4 to C6 chop +arrow from C6 to C9 chop +arrow dashed 0.03 from C6 to C7 chop +arrow same from C9 to C10 +layer = 0 +box fill 0x9bcdfc color 0x9bcdfc wid (C10.e.x - C2.w.x) ht C6.height*1.5 at C6.c +box invis "test" fit with .sw at last box.sw] +box invis "Figure 4" big with .n at 0 below ALL.s +</verbatim> + +Figure 4 diagrams the following scenario: the project starts and +progresses to a point where (at check-in 2) +it is ready to enter testing for its first release. +In a real project, of course, there might be hundreds or thousands of +check-ins before a project reaches this point, but for simplicity of +presentation we will say that the project is ready after check-in 2. +The project then splits into two branches that are used by separate +teams. The testing team, using the blue branch, finds and fixes a few +bugs with check-ins 6 and 9. Meanwhile, the development +team, working on the top uncolored branch, +is busy adding features for the second +release. Of course, the development team would like to take advantage of +the bug fixes implemented by the testing team, so periodically the +changes in the test branch are merged into the dev branch. This is +shown by the dashed merge arrows between check-ins 6 and 7 and between +check-ins 9 and 10. + +In both Figures 2 and 4, check-in 2 has two children. In Figure 2, +we call this a "fork." In diagram 4, we call it a "branch." What is +the difference? As far as the internal Fossil data structures are +concerned, there is no difference. The distinction is in the intent. +In Figure 2, the fact that check-in 2 has multiple children is an +accident that stems from concurrent development. In Figure 4, giving +check-in 2 multiple children is a deliberate act. To a good +approximation, we define forking to be by accident and branching to +be by intent. Apart from that, they are the same. + +When the fork is intentional, it helps humans to understand what is +going on if we <i>name</i> the forks. This is not essential to Fossil's +internal data model, but humans have trouble working with long-lived +branches identified only by the commit ID currently at its tip, being a +long string of hex digits. Therefore, Fossil conflates two concepts: +branching as intentional forking and the naming of forks as branches. +They are in fact separate concepts, but since Fossil is intended to be +used primarily by humans, we combine them in Fossil's human user +interfaces. + +<p class="blockquote"> +<b>Key Distinction:</b> A branch is a <i>named, intentional</i> fork. +</p> + +Unnamed forks <i>may</i> be intentional, but most of the time, they're +accidental and left unnamed. + +Fossil offers two primary ways to create named, intentional forks, +a.k.a. branches. First: + +<pre> +$ fossil commit --branch my-new-branch-name +</pre> + +This is the method we recommend for most cases: it creates a branch as +part of a check-in using the version in the current checkout directory +as its basis. (This is normally the tip of the current branch, though +it doesn't have to be. You can create a branch from an ancestor check-in +on a branch as well.) After making this branch-creating +check-in, your local working directory is switched to that branch, so +that further check-ins occur on that branch as well, as children of the +tip check-in on that branch. + +The second, more complicated option is: + +<pre> +$ fossil branch new my-new-branch-name trunk +$ fossil update my-new-branch-name +$ fossil commit +</pre> + +Not only is this three commands instead of one, the first of which is +longer than the entire simpler command above, you must give the second command +before creating any check-ins, because until you do, your local working +directory remains on the same branch it was on at the time you issued +the command, so that the commit would otherwise put the new material on +the original branch instead of the new one. + +In addition to those problems, the second method is a violation of the +[https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it|YAGNI +Principle]. We recommend that you wait until you actually need the +branch before you create it using the first command above. + +The "trunk" is just another named branch in Fossil. It is simply +the default branch name for the first check-in and every check-in made as +one of its direct descendants. It is special only in that it is Fossil's +default when it has no better idea of which branch you mean. + + +<h2 id="forking">Justifications For Forking</h2> + +The primary cases where forking is justified over branching are all when +it is done purely in software in order to avoid losing information: + +<ol> + <li><p id="offline">By Fossil itself when two users check in children to the same + leaf of a branch, as in Figure 2. + <br><br> + If the fork occurs because + autosync is disabled on one or both of the repositories or because + the user doing the check-in has no network connection at the moment + of the commit, Fossil has no way of knowing that it is creating a + fork until the two repositories are later synchronized.</p></li> + + <li><p id="dist-clone">By Fossil when the cloning hierarchy is more + than 2 levels deep. + <br><br> + [./sync.wiki|Fossil's synchronization protocol] is a two-party + negotiation; syncs don't automatically propagate up the clone tree + beyond that. Because of that, if you have a master repository and + Alice clones it, then Bobby clones from Alice's repository, a + check-in by Bobby that autosyncs with Alice's repo will <i>not</i> + also autosync with the master repo. The master doesn't get a copy of + Bobby's check-in until Alice <i>separately</i> syncs with the master. + If Carol cloned from the master repo and checks something in that + creates a fork relative to Bobby's check-in, the master repo won't + know about that fork until Alice syncs her repo with the master. + Even then, realize that Carol still won't know about the fork until + she subsequently syncs with the master repo. + <br><br> + One way to deal with this is to just accept it as a fact of using a + [https://en.wikipedia.org/wiki/Distributed_version_control|Distributed + Version Control System] like Fossil. + <br><br> + Another option, which we recommend you consider carefully, is to + make it a local policy that check-ins be made only directly against the master + repo or one of its immediate child clones so that the autosync + algorithm can do its job most effectively. Any clones deeper than + that should be treated as read-only and thus get a copy of the new + state of the world only once these central repos have negotiated + that new state. This policy avoids a class of inadvertent fork you + might not need to tolerate. Since [#bad-fork|forks on long-lived + shared working branches can end up dividing a team's development + effort], a team may easily justify this restriction on distributed + cloning.</p></li> + + <li><p id="automation">You've automated Fossil, so you use + <b>fossil commit --allow-fork</b> commands to prevent Fossil from + refusing the check-in simply because it would create a fork. + <br><br> + If you are writing such a tool — e.g. a shell script to make + multiple manipulations on a Fossil repo — it's better to make it + smart enough to detect this condition and cope with it, such as + by making a call to <b>[/help/update | fossil update]</b> + and checking for a merge conflict. That said, if the alternative is + losing information, you may feel justified in creating forks that an + interactive user must later manually clean up with <b>fossil merge</b> + commands.</p></li> +</ol> + +That leaves only one case where we can recommend use of "--allow-fork" +by interactive users: when you're working on a personal branch so that +creating a dual-tipped branch isn't going to cause any other user an +inconvenience or risk [#bad-fork | inadvertently forking the development +effort]. In such a case, the lone developer working on that branch is +not confused, since the fork in development is intentional. Sometimes it +simply makes no sense to bother creating a name, cluttering the global +branch namespace, simply to convert an intentional fork into a "branch." +This is especially the case when the fork is short-lived. + +There's a common generalization of that case: you're a solo developer, +so that the problems with branching vs forking simply don't matter. In +that case, feel free to use "--allow-fork" as much as you like. + + +<h2 id="fix">Fixing Forks</h2> + +If your local checkout is on a forked branch, you can usually fix a fork +automatically with: + +<pre> +$ fossil merge +</pre> + +Normally you need to pass arguments to <b>fossil merge</b> to tell it +what you want to merge into the current basis view of the repository, +but without arguments, the command seeks out and fixes forks. + + +<h2 id="tags">Tags And Properties</h2> + +Tags and properties are used in Fossil to help express the intent, and +thus to distinguish between forks and branches. Figure 5 shows the +same scenario as Figure 4 but with tags and properties added: + +<verbatim type="pikchr center toggle"> +ALL: [arrowht = 0.07 +C1: circle rad 0.1in thickness 1.5px fill white "1" +arrow 40% +C2: circle same "2" +arrow same +circle same "3" +arrow same +C5: circle same "5" +arrow same +C7: circle same "7" +arrow same +C8: circle same "8" +arrow same +C10: circle same "10" +C4: circle same at 3rd circle-(0,.35) "4" +C6: circle same at (1/2 way between C5 and C7,C4) "6" +C9: circle same at (1/2 way between C8 and C10,C4) "9" +arrow from C2 to C4 chop +arrow from C4 to C6 chop +arrow from C6 to C9 chop +arrow dashed 0.03 from C6 to C7 chop +arrow same from C9 to C10 +layer = 0 +box fill 0x9bcdfc color 0x9bcdfc wid (C10.e.x - C2.w.x) ht C6.height*1.5 at C6.c +text " test" above ljust at last box.sw +box fill lightgray "branch=trunk" "sym-trunk" fit with .ne at C1-(0.05,0.3); +line color gray from last box.ne to C1 chop +box same "branch=test" "sym-test" "bgcolor=blue" "cancel=sym-trunk" fit \ + with .n at C4-(0,0.3) +line color gray from last box.n to C4 chop +box same "sym-release-1.0" "closed" fit with .n at C9-(0,0.3) +line color gray from last box.n to C9 chop] +box invis "Figure 5" big fit with .n at 0.2cm below ALL.s +</verbatim> + +A <i>tag</i> is a name that is attached to a check-in. A +<i>property</i> is a name/value pair. Internally, Fossil implements +tags as properties with a NULL value. So, tags and properties really +are much the same thing, and henceforth we will use the word "tag" +to mean either a tag or a property. + +A tag can be a one-time tag, a propagating tag or a cancellation tag. +A one-time tag only applies to the check-in to which it is attached. A +propagating tag applies to the check-in to which it is attached and also +to all direct descendants of that check-in. A <i>direct descendant</i> +is a descendant through direct children. Tag propagation does not +cross merges. Tag propagation also stops as soon +as it encounters another check-in with the same tag. A cancellation tag +is attached to a single check-in in order to either override a one-time +tag that was previously placed on that same check-in, or to block +tag propagation from an ancestor. + +The initial check-in of every repository has two propagating tags. In +Figure 5, that initial check-in is check-in 1. The <b>branch</b> tag +tells (by its value) what branch the check-in is a member of. +The default branch is called "trunk." All tags that begin with "<b>sym-</b>" +are symbolic name tags. When a symbolic name tag is attached to a +check-in, that allows you to refer to that check-in by its symbolic +name rather than by its hexadecimal hash name. When a symbolic name +tag propagates (as does the <b>sym-trunk</b> tag) then referring to that +name is the same as referring to the most recent check-in with that name. +Thus the two tags on check-in 1 cause all descendants to be in the +"trunk" branch and to have the symbolic name "trunk." + +Check-in 4 has a <b>branch</b> tag which changes the name of the branch +to "test." The branch tag on check-in 4 propagates to check-ins 6 and 9. +But because tag propagation does not follow merge links, the <b>branch=test</b> +tag does not propagate to check-ins 7, 8, or 10. Note also that the +<b>branch</b> tag on check-in 4 blocks the propagation of <b>branch=trunk</b> +so that it cannot reach check-ins 6 or 9. This causes check-ins 4, 6, and +9 to be in the "test" branch and all others to be in the "trunk" branch. + +Check-in 4 also has a <b>sym-test</b> tag, which gives the symbolic name +"test" to check-ins 4, 6, and 9. Because tags do not propagate across +merges, check-ins 7, 8, and 10 do not inherit the <b>sym-test</b> tag and +are hence not known by the name "test." +To prevent the <b>sym-trunk</b> tag from propagating from check-in 1 +into check-ins 4, 6, and 9, there is a cancellation tag for +<b>sym-trunk</b> on check-in 4. The net effect is that +check-ins on the trunk go by the symbolic name of "trunk" and check-ins +on the test branch go by the symbolic name "test." + +The <b>bgcolor=blue</b> tag on check-in 4 causes the background color +of timelines to be blue for check-in 4 and its direct descendants. + +Figure 5 also shows two one-time tags on check-in 9. (The diagram does +not make a graphical distinction between one-time and propagating tags.) +The <b>sym-release-1.0</b> tag means that check-in 9 can be referred to +using the more meaningful name "release-1.0." The <b>closed</b> tag means +that check-in 9 is a "closed leaf." A closed leaf is a leaf that should +never have direct children. + +<h2 id="bad-fork">How Can Forks Divide Development Effort?</h2> + +[#dist-clone|Above], we stated that forks carry a risk that development +effort on a branch can be divided among the forks. It might not be +immediately obvious why this is so. To see it, consider this swim lane +diagram: + +<verbatim type="pikchr center toggle"> + $laneh = 0.75 + +ALL: [ + # Draw the lanes + down + box width 3.5in height $laneh fill 0xacc9e3 + box same fill 0xc5d8ef + box same as first box + box same as 2nd box + line from 1st box.sw+(0.2,0) up until even with 1st box.n \ + "Alan" above aligned + line from 2nd box.sw+(0.2,0) up until even with 2nd box.n \ + "Betty" above aligned + line from 3rd box.sw+(0.2,0) up until even with 3rd box.n \ + "Charlie" above aligned + line from 4th box.sw+(0.2,0) up until even with 4th box.n \ + "Darlene" above aligned + + # fill in content for the Alice lane + right +A1: circle rad 0.1in at end of first line + (0.2,-0.2) \ + fill white thickness 1.5px "1" + arrow right 50% + circle same "2" + arrow right until even with first box.e - (0.65,0.0) + ellipse "future" fit fill white height 0.2 width 0.5 thickness 1.5px +A3: circle same at A1+(0.8,-0.3) "3" fill 0xc0c0c0 + arrow from A1 to last circle chop "fork!" below aligned + + # content for the Betty lane +B1: circle same as A1 at A1-(0,$laneh) "1" + arrow right 50% + circle same "2" + arrow right until even with first ellipse.w + ellipse same "future" +B3: circle same at A3-(0,$laneh) "3" + arrow right 50% + circle same as A3 "4" + arrow from B1 to 2nd last circle chop + + # content for the Charlie lane +C1: circle same as A1 at B1-(0,$laneh) "1" + arrow 50% + circle same "2" + arrow right 0.8in "goes" "offline" +C5: circle same as A3 "5" + arrow right until even with first ellipse.w \ + "back online" above "pushes 5" below "pulls 3 & 4" below + ellipse same "future" + + # content for the Darlene lane +D1: circle same as A1 at C1-(0,$laneh) "1" + arrow 50% + circle same "2" + arrow right until even with C5.w + circle same "5" + arrow 50% + circle same as A3 "6" + arrow right until even with first ellipse.w + ellipse same "future" +D3: circle same as B3 at B3-(0,2*$laneh) "3" + arrow 50% + circle same "4" + arrow from D1 to D3 chop +] +box invis "Figure 6" big fit with .n at 0.2cm below ALL.s +</verbatim> + +This is a happy, cooperating team. That is an important restriction on +our example, because you must understand that this sort of problem can +arise without any malice, selfishness, or willful ignorance in sight. +All users on this diagram start out with the same view of the +repository, cloned from the same master repo, and all of them are +working toward their shared vision of a unified future. + +All users, except possibly Alan, start out with the same two initial +check-ins in their local working clones, 1 & 2. It might be that Alan +starts out with only check-in 1 in his local clone, but we'll deal with +that detail later. + +It doesn't matter which branch this happy team is working on, only that +our example makes the most sense if you think of it as a long-lived shared +working branch like trunk. Each user makes +only one check-in, shaded light gray in the diagram. + +<h3 id="bf-alan">Step 1: Alan</h3> + +Alan sets the stage for this problem by creating a +fork from check-in 1 as check-in 3. How and why Alan did this doesn't +affect what happens next, though we will walk through the possible cases +and attempt to assign blame [#post-mortem|in the <i>post mortem</i>]. +For now, you can assume that Alan did this out of unavoidable ignorance. + +<h3 id="bf-betty">Step 2: Betty</h3> + +Because Betty's local clone is autosyncing with +the same upstream repository as Alan's clone, there are a number of ways +she can end up seeing Alan's check-in 3 as the latest on that branch: + +<ol> + <li><p>The working check-out directory she's using at the moment was + on a different branch at the time Alan made check-in 3, so Fossil + sees that as the tip at the time she switches her working directory + to that branch with a <b>fossil update $BRANCH</b> command. (There is an + implicit autosync in that command, if the option was enabled at the + time of the update.)</p></li> + + <li><p>The same thing, only in a fresh checkout directory with a + <b>[/help/open | fossil open $REPO $BRANCH]</b> command.</p></li> + + <li><p>Alan makes his check-in 3 while Betty has check-in 1 or 2 as + the tip in her local clone, but because she's working with an + autosync'd connection to the same upstream repository as Alan, on + attempting what will become check-in 4, she gets the "would fork" + message from <b>fossil commit</b>, so she dutifully updates her clone + and tries again, moving her work to be a child of the new tip, + check-in 3. (If she doesn't update, she creates a <i>second</i> + fork, which simply complicates matters beyond what we need here for + our illustration.)</p></li> +</ol> + +For our purposes here, it doesn't really matter which one happened. All +that matters is that Alan's check-in 3 becomes the parent of Betty's +check-in 4 because it was the newest tip of the working branch at the +time Betty does her check-in. + +<h3 id="bf-charlie">Step 3: Charlie</h3> + +Meanwhile, Charlie went offline after syncing +his repo with check-in 2 as the latest on that branch. When he checks +his changes in, it is as a child of 2, not of 4, because Charlie doesn't +know about check-ins 3 & 4 yet. He does this at an absolute wall clock +time <i>after</i> Alan and Betty made their check-ins, so when Charlie +comes back online and pushes his check-in 5 to the master repository and +learns about check-ins 3 and 4 during Fossil sync, Charlie inadvertently +revives the other side of the fork. + +<h3 id="bf-darlene">Step 4: Darlene</h3> + +Darlene sees all of this, because she joins in +on the work on this branch after Alan, Betty, and Charlie made their +check-ins and pushed them to the master repository. She's taking one of +the same three steps as we [#bf-betty|outlined for Betty above]. +Regardless of her path to this view, it happens after Charlie pushed his +check-in 5 to the master repo, so Darlene sees that as the latest on the +branch, causing her work to be saved as a child of check-in 5, not of +check-in 4, as it would if Charlie didn't come back online and sync +before Darlene started work on that branch. + +<h3 id="post-mortem">Post Mortem</h3> + +The end result of all of this is that even though everyone makes only one check-in +and no one disables autosync without genuine need, +half of the check-ins end up on one side of the fork and half on +the other. + +A future user — his mother calls him Edward, but please call him Eddie — +can then join in on the work on this branch and end up on <i>either</i> side of +the fork. If Eddie joins in with the state of the repository as drawn +above, he'll end up on the top side of the fork, because check-in 6 is +the latest, but if Alan or Betty makes a seventh check-in to that branch +first, it will be as a child of check-in 4 since that's the version in +their local check-out directories. Since that check-in 7 will then be the latest, +Eddie will end up on the bottom side of the fork instead. + +In all of this, realize that neither side of the fork is obviously +"correct." Every participant was doing the right thing by their own +lights at the time they made their lone check-in. + +Who, then, is to blame? + +We can only blame the consequences of creating the fork on Alan if he +did so on purpose, as by passing "--allow-fork" when creating a check-in +on a shared working branch. Alan might have created it inadvertently by +going offline while check-in 1 was the tip of the branch in his local +clone, so that by the time he made his check-in 3, check-in 2 had +arrived at the shared parent repository from someone else. (Francine?) +When Alan rejoins the network and does an autosync, he learns about +check-in 2. Since his #3 is already checked into his local clone because +autosync was off or blocked, the sync creates an unavoidable fork. We +can't blame either Alan or Francine here: they were both doing the right +thing given their imperfect view of the state of the global situation. + +The same is true of Betty, Charlie, and Darlene. None of them tried to +create a fork, and none of them chose a side in this fork to participate +in. They just took Fossil's default and assumed it was correct. + +The only blame I can assign here is on any of these users who believed +forks couldn't happen before this did occur, and I blame them only for +their avoidable ignorance. (You, dear reader, have been ejected from +that category by reading this very document.) Any time someone can work +without getting full coordination from every other clone of the repo, +forks are possible. Given enough time, they're all but inevitable. This +is a general property of DVCSes, not just of Fossil. + +This sort of consequence is why forks on shared working branches are +bad, which is why [./concepts.wiki#workflow|Fossil tries so hard to avoid them], why it warns you +about it when they do occur, and why it makes it relatively [#fix|quick and +painless to fix them] when they do occur. + + +<h2>Review Of Terminology</h2> + +<dl> +<dt><b>Branch</b></dt> +<dd><p>A branch is a set of check-ins with the same value for their +"branch" property.</p></dd> +<dt><b>Leaf</b></dt> +<dd><p>A leaf is a check-in with no children in the same branch.</p></dd> +<dt><b>Closed Leaf</b></dt> +<dd><p>A closed leaf is any leaf with the <b>closed</b> tag. These leaves +are intended to never be extended with descendants and hence are omitted +from lists of leaves in the command-line and web interface.</p></dd> +<dt><b>Open Leaf</b></dt> +<dd><p>A open leaf is a leaf that is not closed.</p></dd> +<dt><b>Fork</b></dt> +<dd><p>A fork is when a check-in has two or more direct (non-merge) +children in the same branch.</p></dd> +<dt><b>Branch Point</b></dt> +<dd><p>A branch point occurs when a check-in has two or more direct (non-merge) +children in different branches. A branch point is similar to a fork, +except that the children are in different branches.</p></dd> +</dl> + +Check-in 4 of Figure 3 is not a leaf because it has a child (check-in 5) +in the same branch. Check-in 9 of Figure 5 also has a child (check-in 10) +but that child is in a different branch, so check-in 9 is a leaf. Because +of the <b>closed</b> tag on check-in 9, it is a closed leaf. + +Check-in 2 of Figure 3 is considered a "fork" +because it has two children in the same branch. Check-in 2 of Figure 5 +also has two children, but each child is in a different branch, hence in +Figure 5, check-in 2 is considered a "branch point." + +<h2>Differences With Other DVCSes</h2> + +<h3 id="single">Single DAG</h3> + +Fossil keeps all check-ins on a single DAG. Branches are identified with +tags. This means that check-ins can be freely moved between branches +simply by altering their tags. + +Most other DVCSes maintain a separate DAG for each branch. + +<h3 id="unique">Branch Names Need Not Be Unique</h3> + +Fossil does not require that branch names be unique, as in some VCSes, +most notably Git. Just as with unnamed branches (which we call forks) +Fossil resolves such ambiguities using the timestamps on the latest +check-in in each branch. If you have two branches named "foo" and you say +<b>fossil update foo</b>, you get the tip of the "foo" branch with the most +recent check-in. + +This fact is helpful because it means you can reuse branch names, which +is especially useful with utility branches. There are several of these +in the SQLite and Fossil repositories: "broken-build," "declined," +"mistake," etc. As you might guess from these names, such branch names +are used in renaming the tip of one branch to shunt it off away from the +mainline of that branch due to some human error. (See +<b>[/help/amend | fossil +amend]</b> and the Fossil UI check-in amendment features.) This is a +workaround for Fossil's [./shunning.wiki|normal inability to forget +history]: we usually don't want to actually <i>remove</i> history, but +would like to sometimes set some of it aside under a new label. + +Because some VCSes can't cope with duplicate branch names, Fossil +collapses such names down on export using the same time stamp based +arbitration logic, so that only the branch with the newest check-in gets +the branch name in the export. + +All of the above is true of tags in general, not just branches. diff --git a/www/bugtheory.wiki b/www/bugtheory.wiki new file mode 100644 index 0000000..56c2edd --- /dev/null +++ b/www/bugtheory.wiki @@ -0,0 +1,124 @@ +<title>Bug-Tracking In Fossil</title> +<h2>Introduction</h2> + +A bug-report in fossil is called a "ticket". Tickets are tracked +separately from code check-ins. + +Some other distributed bug-tracking systems store tickets as files within +the source tree and thereby leverage the syncing and merging capabilities +of the versioning system to sync and merge tickets. This approach is +rejected in fossil for three reasons: + + 1. Check-ins in fossil are immutable. So if + tickets were part of the check-in, then there would be no way to add + new tickets to a check-in as new bugs are discovered. + + 2. Any project of reasonable size and complexity will generate thousands + and thousands of tickets, and we do not want all those ticket files + cluttering the source tree. + + 3. We want tickets to be managed from the web interface and to have a + permission system that is distinct from check-in permissions. + In other words, we do not want to restrict the creation and editing + of tickets to developers with check-in privileges and an installed + copy of the fossil executable. Casual passers-by on the internet should + be permitted to create tickets. + +Recall that a fossil repository consists of an +unordered collection of <i>artifacts</i>. (See the +<a href="fileformat.wiki">file format document</a> for details.) +Some artifacts have a special format, and among those are +<a href="fileformat.wiki#tktchng">Ticket Change Artifacts</a>. +One or more ticket change artifacts are associated with each +ticket. A ticket is created by a ticket change artifact. +Each subsequent modification of the ticket is a separate artifact. + +The "push", "pull", and "sync" algorithms share ticket change artifacts +between repositories in the same way as every other artifact. In fact, +the sync algorithm has no knowledge of the meaning of the artifacts it +is syncing. As far as the sync algorithm is concerned, all artifacts are +alike. After the sync has occurs, the individual repositories must +make sense of the meaning of the various artifacts for themselves. + +<h2>Interpretation Of Ticket Change Artifacts</h2> + +<i>Note: The following is implementation detail which can be safely ignored by +casual users of fossil.</i> + +Every ticket change artifact contains (among other things) + + * a timestamp, + * a ticket ID, and + * one or more name/value pairs. + +The current state of a ticket is found by replaying all ticket +change artifacts with the same ticket ID in timestamp order. +For a given ticket, all values are initially NULL. As each +ticket change artifact is encountered, values are either replaced +or appended, according to a flag on the name/value pair. The current +values for the fields of a ticket are the values that remain at the +end of the replay process. + +To create a new ticket, one inserts a ticket change +artifact with a new ID. The ticket ID +is a random 40-character lower-case hexadecimal number. The "tktnew" +page in the fossil web interface creates new ticket IDs +using a good source of randomness to insure uniqueness. +The name/value pairs on the initial ticket change +artifact are the initial values for the fields in the ticket. + +Amending a ticket means simply creating a new artifact with the +same ticket ID and with name/value pairs for those fields which +are changing. Fields of the ticket which are not being modified +should not appear as name/value pairs in the new artifact. + +This approach to storing ticket state means that independently entered +changes are automatically merged together when artifacts are shared +between repositories. Tickets do not branch. This approach also makes +it trivial to track the historic progression of changes to a ticket. + +In order for this scheme to work, the system clocks on machines that +add new ticket changes artifacts have to be set close to reality. +It is OK for a ticket change artifact timestamp to be off by a few +minutes or even a few hours. But if a timestamp on a ticket change +artifact is off by months or years, it can seriously confuse the +replay algorithm for determining the current ticket state. There +are techniques available to cause rogue artifacts to be ignored +by fossil. So if a ticket change artifact with a bad timestamp +does get into a repository, it can be removed by an administrator. +But the best approach is to take steps to insure that timestamps +are approximately correct in the first place. + +<h2>Local Configuration</h2> + +The ticket change artifacts are part of the global state for a project. +The global state is that which is shared between repositories during +a sync. Each repository also has local state which is not (normally) +shared. The name/value pairs for a ticket are part of global state, +but the interpretation and display of that information is local state. +Hence, each repository is free to set up its own ticket display and +input formats and reporting rules according to its own particular needs. + +Each repository defines its own TICKET table in its database. There is +one row in the TICKET table for each unique ticket ID. The +names of columns in the TICKET table correspond to the names in +the name/value pairs of ticket change artifacts. When running the replay +algorithm, if a name/value pair is seen which has no corresponding column +in the TICKET table, then that name/value pair is ignored. Columns can +be added or removed from the TICKET table at any time. Whenever the +TICKET table is modified, the replay algorithm automatically reruns +to repopulate the table using the new column names. Note that the TICKET +table schema and content is part of the local state of a repository +and is not shared with other repositories during a sync, push, or pull. + +Each repository also defines scripts used to generate web pages for +creating new tickets, viewing existing tickets, and modifying an +existing ticket. These scripts consist of HTML with an embedded +scripts written in a Tcl-like language called "[./th1.md|TH1]". Every +new fossil repository is created with default scripts. Administrators +wishing to customize their ticket entry, viewing, and editing screens +should modify the default scripts to suit their needs. These screen +generator scripts are part of the local state of a repository and are +not shared with other repositories during a sync, push, or pull. + +<i>To be continued...</i> diff --git a/www/build-icons/linux.gif b/www/build-icons/linux.gif Binary files differnew file mode 100644 index 0000000..eaa5cb7 --- /dev/null +++ b/www/build-icons/linux.gif diff --git a/www/build-icons/linux64.gif b/www/build-icons/linux64.gif Binary files differnew file mode 100644 index 0000000..b254a5b --- /dev/null +++ b/www/build-icons/linux64.gif diff --git a/www/build-icons/mac.gif b/www/build-icons/mac.gif Binary files differnew file mode 100644 index 0000000..30f9538 --- /dev/null +++ b/www/build-icons/mac.gif diff --git a/www/build-icons/openbsd.gif b/www/build-icons/openbsd.gif Binary files differnew file mode 100644 index 0000000..88b76e3 --- /dev/null +++ b/www/build-icons/openbsd.gif diff --git a/www/build-icons/src.gif b/www/build-icons/src.gif Binary files differnew file mode 100644 index 0000000..810d39b --- /dev/null +++ b/www/build-icons/src.gif diff --git a/www/build-icons/win32.gif b/www/build-icons/win32.gif Binary files differnew file mode 100644 index 0000000..9098e86 --- /dev/null +++ b/www/build-icons/win32.gif diff --git a/www/build.wiki b/www/build.wiki new file mode 100644 index 0000000..7787461 --- /dev/null +++ b/www/build.wiki @@ -0,0 +1,543 @@ +<title>Compiling and Installing Fossil</title> + +<h2>0.0 Using A Pre-compiled Binary</h2> + +[/uv/download.html|Pre-compiled binaries] are available for recent +releases. Just download +the appropriate executable for your platform +and put it on your $PATH. +To uninstall, simply delete the executable. +To upgrade from an older release, just overwrite the older binary with +the newer one. + +For details about how those binaries are built, see +[/wiki?name=Release+Build+How-To | the Release Build How-To wiki page]. + + +<h2>0.1 Executive Summary</h2> + +Building and installing is very simple. Three steps: + +<ol> +<li> Download and unpack a source tarball or ZIP. +<li> <b>./configure; make</b> +<li> Move the resulting "fossil" or "fossil.exe" executable to someplace on +your $PATH. +</ol> + +<hr> + +<h2>1.0 Obtaining The Source Code</h2> + +Fossil is self-hosting, so you can obtain a ZIP archive or tarball +containing a snapshot of the <em>latest</em> version directly from +Fossil's own fossil repository. Additionally, source archives of +<em>released</em> versions of +fossil are available from the [/uv/download.html|downloads page]. +To obtain a development version of fossil, follow these steps: + +<ol> +<li>Point your web browser to [https://fossil-scm.org/]</li> + +<li>Click on the [/timeline|Timeline] +link at the top of the page.</li> + +<li>Select a version of Fossil you want to download. The latest +version on the trunk branch is usually a good choice. Click on its +link.</li> + +<li>Finally, click on one of the +"Zip Archive" or "Tarball" links, according to your preference. +These links will build a ZIP archive or a gzip-compressed tarball of the +complete source code and download it to your computer.</li> +</ol> + +<h2>Aside: Is it really safe to use an unreleased development version of +the Fossil source code?</h2> + +Yes! Any check-in on the +[/timeline?t=trunk | trunk branch] of the Fossil +[https://fossil-scm.org/ | self-hosting repository] +will work fine. (Dodgy code is always on a branch.) In the unlikely +event that you pick a version with a serious bug, it still won't +clobber your files. Fossil uses several +[./selfcheck.wiki | self-checks] prior to committing any +repository change that prevent loss-of-work due to bugs. + +The Fossil [./selfhost.wiki | self-hosting repositories], especially +the one at [https://fossil-scm.org/home], usually run a version +of trunk that is less than a week or two old. Look at the bottom +left-hand corner of this screen (to the right of "This page was +generated in...") to see exactly which version of Fossil is +rendering this page. It is always safe to use whatever version +of the Fossil code you find running on the main Fossil website. + +<h2>2.0 Compiling</h2> + +<ol> +<li value="5"> +Unpack the ZIP or tarball you downloaded then +<b>cd</b> into the directory created.</li> + +<li><i>(Optional, Debian-compatible Linux only)</i> +Make sure you have all the necessary tools and libraries at hand by running: +<b>sudo apt install tcl-dev tk libssl-dev zlib1g-dev</b>. + +<li><i>(Optional, Unix only)</i> +Run <b>./configure</b> to construct a makefile. + +<ol type="a"> +<li> +The build system for Fossil on Unix-like systems assumes that the +OpenSSL development and runtime files are available on your system, +because unprotected repositories are trivial to attack otherwise. +Indeed, some public Fossil repositories — including Fossil's own — today +run in an HTTPS-only mode, so that you can't even do an anonymous clone +from them without using the TLS features added to Fossil by OpenSSL. To +weaken that stance could allow a +[https://en.wikipedia.org/wiki/Man-in-the-middle_attack|man in the +middle attack], such as one that substitutes malicious code into your +Fossil repository clone. + +You can force the Fossil build system to avoid searching for, building +against, and linking to the OpenSSL library by passing +<b>--with-openssl=none</b> to the <tt>configure</tt> script. + +If you do not have the OpenSSL development libraries on your system, +we recommend that you install them, typically via your OS's package +manager. The Fossil build system goes to a lot of effort to seek these +out wherever they may be found, so that is typically all you need to +do. + +For more advanced use cases, see the [./ssl.wiki#openssl-bin|OpenSSL +discussion in the "TLS and Fossil" document]. +</li> + +<li> +To build a statically linked binary, you can <i>try</i> adding +the <b>--static</b> option, but +[https://stackoverflow.com/questions/3430400/linux-static-linking-is-dead +| it may well not work]. If your platform of choice is affected by this, +the simplest workaround we're aware of is to build a Fossil container, +then [./containers.md#static | extract the static executable from it]. +</li> + +<li> +To enable the native [./th1.md#tclEval | Tcl integration feature] feature, +add the <b>--with-tcl=1</b> and <b>--with-tcl-private-stubs=1</b> options. +</li> + +<li> +Other configuration options can be seen by running +<b>./configure --help</b> +</li> +</ol> + +<li>Run "<b>make</b>" to build the "fossil" or "fossil.exe" executable. +The details depend on your platform and compiler.</li> + +<ol type="a"> +<li><i>Unix</i> → the configure-generated Makefile should work on +all Unix and Unix-like systems. Simply type "<b>make</b>".</li> + +<li><i>Unix without running "configure"</i> → if you prefer to avoid +running configure, you can also use: <b>make -f Makefile.classic</b>. You may +want to make minor edits to Makefile.classic to configure the build for your +system.</li> + +<li><i>MinGW / MinGW-w64</i> → The best-supported path is to build +via the MinGW specific Makefile under a POSIX build of GNU make: +"<b>make -f win/Makefile.mingw</b>".</li> + +To enable the native [./th1.md#tclEval | Tcl integration feature], use a +command line like the following (all on one line): + +<pre>make -f win/Makefile.mingw FOSSIL_ENABLE_TCL=1 FOSSIL_ENABLE_TCL_STUBS=1 FOSSIL_ENABLE_TCL_PRIVATE_STUBS=1</pre> + +<li><i>MSYS2 / Cygwin</i> → This is something of a hybrid between +options "a" and "c" above: it configures and builds +<code>fossil.exe</code> much as on Linux, but you get a native Windows +executable out at the end. The primary downside is that this type of +executable can become confused when attempting to interoperate with +fully native Windows EXEs by making assumptions that only hold true when +all elements are running under the Cygwin/MSYS environment. The MSVC and +MinGW options do not have this limitation. + +Even so, there is value to Linux/Unix natives in having this hybrid +while off in Windows-land. The simpler of the two paths is MSYS2, since +it lets you install the necessary prerequisites in a single command +after installing the base environment: + +<pre>pacman -sS gcc make openssl-devel zlib-devel</pre> + +The equivalent in Cygwin's <code>setup.exe</code> requires stepping +through the GUI package chooser, and then if you miss one of the +prereqs, going all the way back through it again until you get it right. + +<li><i>MSVC</i> → Use the MSVC makefile.</li> + +<em>NB:</em> Run the following <code>nmake</code> commands from a "x64 Native +Tools Command Prompt"; <code>buildmsvc.bat</code> is able to automatically load +the build tools (x64 by default, pass "x86" as the first argument to use the +x86 tools), so it can be called from a normal command prompt. + +First, change to the "win/" subdirectory ("<b>cd win</b>"), then run +"<b>nmake /f Makefile.msc</b>".<br><br>Alternatively, the batch +file "<b>win\buildmsvc.bat</b>" may be used and it will attempt to +detect and use the latest installed version of MSVC.<br><br>To enable +the optional <a href="https://www.openssl.org/">OpenSSL</a> support, +first <a href="https://www.openssl.org/source/">download the official +source code for OpenSSL</a> and extract it to an appropriately named +"<b>openssl</b>" subdirectory within the local +[/tree?ci=trunk&name=compat | compat] directory then make sure that some recent +<a href="http://www.perl.org/">Perl</a> binaries are installed locally, +and finally run one of the following commands: +<pre> +nmake /f Makefile.msc FOSSIL_ENABLE_SSL=1 FOSSIL_BUILD_SSL=1 PERLDIR=C:\full\path\to\Perl\bin +</pre> +<pre> +buildmsvc.bat FOSSIL_ENABLE_SSL=1 FOSSIL_BUILD_SSL=1 PERLDIR=C:\full\path\to\Perl\bin +</pre> +To enable the optional native [./th1.md#tclEval | Tcl integration feature], +run one of the following commands or add the "FOSSIL_ENABLE_TCL=1" +argument to one of the other NMAKE command lines: +<pre> +nmake /f Makefile.msc FOSSIL_ENABLE_TCL=1 +</pre> +<pre> +buildmsvc.bat FOSSIL_ENABLE_TCL=1 +</pre> + +<li><i>Cygwin</i> → The same as other Unix-like systems. It is +recommended to configure using: "<b>configure --disable-internal-sqlite</b>", +making sure you have the "libsqlite3-devel" , "zlib-devel" and +"openssl-devel" packages installed first.</li> +</ol> +</ol> + +<h2>3.0 Installing</h2> + +<ol> +<li value="9"> +The finished binary is named "fossil" (or "fossil.exe" on Windows). +Put this binary in a +directory that is somewhere on your PATH environment variable. +It does not matter where. +</li> + +<li> +<b>(Optional:)</b> +To uninstall, just delete the binary. +</li> +</ol> + +<h2>4.0 Additional Considerations</h2> + +<ul> +<li> + If the makefiles that come with Fossil do not work for + you, or for some other reason you want to know how to build + Fossil manually, then refer to the + [./makefile.wiki | Fossil Build Process] document which describes + in detail what the makefiles do behind the scenes. +</li> + +<li> + The fossil executable is self-contained and stand-alone and usually + requires no special libraries or other software to be installed. However, + the "--tk" option to the [/help/diff|diff command] requires that Tcl/Tk + be installed on the local machine. You can get Tcl/Tk from + [http://www.activestate.com/activetcl|ActiveState]. +</li> + +<li> + To build on older Macs (circa 2002, MacOS 10.2) edit the Makefile + generated by configure to add the following lines: + <pre> + TCC += -DSQLITE_WITHOUT_ZONEMALLOC + TCC += -D_BSD_SOURCE + TCC += -DWITHOUT_ICONV + TCC += -Dsocketlen_t=int + TCC += -DSQLITE_MAX_MMAP_SIZE=0 + </pre> +</li> +</ul> + + +<h2 id="docker" name="oci">5.0 Building a Docker Container</h2> + +The information on building Fossil inside an +[https://opencontainers.org/ | OCI container] is now in +[./containers.md | a separate document]. + +This includes the instructions on using the OCI container as an +expedient intermediary for building a statically-linked Fossil binary on +modern Linux platforms, which otherwise make this difficult. + + +<h2>6.0 Building on/for Android</h2> + +<h3>6.1 Cross-compiling from Linux</h3> + +The following instructions for building Fossil for Android via Linux, +without requiring a rooted OS, are adapted from +[forum:/forumpost/e0e9de4a7e | a forum post]. + +On the development machine, from the fossil source tree: + +<pre><code>export CC=$NDK_PATH/toolchains/llvm/prebuilt/linux-x86_64/bin/armv7a-linux-androideabi21-clang +./configure --with-openssl=none +make +</code></pre> + + +On the Android device, enable the <em>USB debugging</em> option from +Developer menu in Device Options. Connect the device to the development +system with USB. If it's configured and connected properly, +the device should show up in the output of <code>adb devices</code>: + +<pre><code>sudo adb devices +</code></pre> + +Copy the resulting fossil binary onto the device... + +<pre><code>sudo adb push fossil /data/local/tmp +</code></pre> + +And run it from an <code>adb</code> shell: + +<pre><code>sudo adb shell +> cd /data/local/tmp +# Fossil requires a HOME directory to work with: +> export HOME=$PWD +> export PATH=$PWD:$PATH +> fossil version +This is fossil version 2.11 [e5653a4ceb] 2020-03-26 18:54:02 UTC +</code></pre> + +The output might, or might not, include warnings such as: + +<pre><code>WARNING: linker: ./fossil: unused DT entry: type 0x6ffffef5 arg 0x1464 +WARNING: linker: ./fossil: unused DT entry: type 0x6ffffffe arg 0x1ba8 +WARNING: linker: ./fossil: unused DT entry: type 0x6fffffff arg 0x2 +</code></pre> + +The source of such warnings is not 100% certain. +Some information about these (reportedly harmless) warnings can +be found +[https://stackoverflow.com/a/41900551 | on this StackOverflow post]. + + +<a id='fuzzer'></a> +<h2>7.0 Building for Fuzz Testing</h2> + +This feature is primarily intended for fossil's developers and may +change at any time. It is only known to work on Linux systems and has +been seen to work on x86/64 and ARM. + +Fossil has builtin support for processing specific features using +<tt>libfuzzer</tt>. The features which can be tested this way are +found in the help text for the [/help/test-fuzz|test-fuzz +command]. + +Fuzzing requires: + + * The clang C compiler. + * libfuzzer. On Ubuntu-derived systems, it can be installed with + <tt>apt install libfuzzer-XYZ</tt>, where XYZ is a version number + (several versions may be available on any given system) + + +Compile as follows: + +<pre><code>make clean +make TCCFLAGS='-DFOSSIL_FUZZ -fsanitize=fuzzer,address,undefined -O0 -g' CC=clang +</code></pre> + +The resulting <tt>fossil</tt> binary differs from the standard +one primarily in that it runs the <tt>test-fuzz</tt> command by +default. It needs to be told what to fuzz and needs to be given a +directory of input files to seed the fuzzer with: + + +<pre><code>$ mkdir cases + # Copy input files into ./cases. e.g. when fuzzing the markdown + # processor, copy any to-be-tested .md files into that directory. + # Then start the fuzzer: +$ ./fossil-fuzz --fuzztype markdown cases +</code></pre> + +As it works, it writes its mutated test files into the "cases" +directory, each one named in the form of a hash. When it finds a +problem it will produce a stack trace for the offending code, will +output the name of the file which triggered the crash (named +<tt>cases/SOME_HASH</tt>) and may, depending on the nature of the +problem, produce a file named <tt>crash-SOMETHING</tt>. In theory the +crash file can be fed directly back into the fuzzer to reproduce the +problem: + +<pre></code>$ ./fossil-fuzz --fuzztype markdown crash-SOMETHING +</code></pre> + +But whether or not it will genuinely crash may depend on static +app-level state which might not trigger the crash when running an +individual test. + +For a detailed information about the fuzzer's flags and features, see: + + * [https://llvm.org/docs/LibFuzzer.html] + * [https://github.com/google/fuzzing/blob/master/tutorial/libFuzzerTutorial.md] + +Flags for the fuzzer can be passed directly to fossil, +e.g. <tt>-jobs=4</tt> to start four fuzzer jobs in parallel, but doing +so may cause the fuzzer to <em>strip the --fuzztype flag</em>, leading +to it testing the wrong thing. When passing on fuzzer-specific flags +along with <tt>--fuzztype</tt>, be sure to check your system's process +list to ensure that your <tt>--fuzztype</tt> flag is there. + + +<a id='wasm'></a> +<h2>8.0 Building WebAssembly Components</h2> + +Fossil uses one component built as +[https://developer.mozilla.org/en-US/docs/WebAssembly | WebAssembly] +a.k.a. WASM. Because compiling WASM code requires non-trivial +client-side tooling, the repository includes compiled copies of these +pieces. Most Fossil hackers should never need to concern themselves +with the WASM parts, but this section describes how to for those who +want or need to do so. + +<strong>The bits described in this section are necessary when updating +<tt>extsrc/pikchr.c</tt></strong> from the upstream source, or the +fossil binary will use a different version of pikchr than +[/pikchrshow] does (as the latter runs the WASM build of pikchr). + +These instructions have only ever been tested on Linux systems. They +"should" work on any Unix-like system supported by Emscripten. The +fossil makefiles for Windows builds <em>do not</em> include any of the +WASM-related components (patches to add that would be welcomed, of +course). + +The first step is to configure the tree with support for +[https://emscripten.org/|Emscripten]. This requires that the system +has the Emscripten SDK (a.k.a. emsdk) installed, as documented at: + +[https://emscripten.org/docs/getting_started/downloads.html] + +For instructions on keeping the SDK up to date, see: + +[https://emscripten.org/docs/tools_reference/emsdk.html] + +<div class="sidebar">Getting Emscripten up and running is trivial and +painless, at least on Linux systems, but the installer downloads +many hundreds of megabytes of tools and dependencies, all of which +will be installed under the single SDK directory (as opposed to +being installed at the system level). It does, however, require +that python3 be installed at the system level and it can +optionally make use of a system-level cmake for certain tasks +unrelated to how fossil uses the SDK.</div> + +After installing the SDK, configure the fossil tree with emsdk +support: + +<pre><code>$ ./configure --with-emsdk=/path/to/emsdk \ + --and-other-options... +</code></pre> + +If the <tt>--with-emsdk</tt> flag is not provided, the configure +script will check for the environment variable <tt>EMSDK</tt>, which +is one of the standard variables the SDK environment uses. If that +variable is found, its value will implicitly be used in place of the +missing <tt>--with-emsdk</tt> flag. Thus, if the <tt>emsdk_env.sh</tt> +script is sourced into the shell before running the configure script, +the SDK will be detected even without the config flag. + +The configure script installs some makefile variables which tell the +build where to find the SDK and it generates a script named +<tt>tools/emcc.sh</tt> (from the template file +<tt>[/file/tools/emcc.sh.in|/tools/emcc.sh.in]</tt>), which is a +wrapper around the Emscripten C compiler (<tt>emcc</tt>). The wrapper +script uses the configure-time state to attempt to set up the various +environment variables which are required by <tt>emcc</tt> and will +fail if it cannot do so. Once it's set up the environment, it passes +on all of its arguments to <tt>emcc</tt>. + +The WASM-related build parts are set up such that none of them should +ever trigger implicitly (e.g. via dependencies resolution) in a normal +build cycle. They are instead explicitly built as described below. + +From the top of the source tree, all WASM-related components can be +built with: + +<pre><code>$ make wasm</code></pre> + +<div class="sidebar">The file +<tt>[/file/extsrc/pikcher-worker.js|extsrc/pikcher-worker.js]</tt> +is hand-coded and intended to be loaded as a "Worker" in +JavaScript. That file loads the main module and provides an +interface via which a main JavaScript thread can communicate with +pikchr running in a Worker thread. The file +<tt>[/file/src/fossil.page.pikchrshowasm.js|src/fossil.page.pikchrshowasm.js]</tt> +implements the [/pikchrshow] app and demonstrates how +<tt>pikchr-worker.js</tt> is used.</div> + +As of this writing, those parts include: + + * <tt>extsrc/pikchr.wasm</tt> is a WASM-compiled form of + <tt>extsrc/pikchr.c</tt>. + * <tt>extsrc/pikchr.js</tt> is JS/WASM glue code generated by Emscripten + to give JS code access to the API exported by the WASM file. + +When a new version of <tt>extsrc/pikchr.c</tt> is installed, the +files <tt>pikchr.{js,wasm}</tt> will need to be recompiled to account +for that. Running <tt>make wasm</tt> will, if the build is set up for +the emsdk, recompile those: + +<pre><code>$ rm extsrc/pikchr.{js,wasm} +# ^^^^ that rm has proven necessary in order to ensure rebuilds +$ make wasm +./tools/emcc.sh -o extsrc/pikchr.js ... +$ ls -la extsrc/pikchr.{js,wasm} +-rw-rw-r-- 1 stephan stephan 17263 Jun 8 03:59 extsrc/pikchr.js +-rw-rw-r-- 1 stephan stephan 97578 Jun 8 03:59 extsrc/pikchr.wasm +</code></pre> + +<div class="sidebar">If that fails with a message along the lines of +“<code>setting `EXPORTED_RUNTIME_METHODS` expects `<class 'list'>` but +got `<class 'str'>`</code>” then the emcc being invoked is too old: emcc +changed the format of list-type arguments at some point. The required +minimum version is unknown, but any SDK version from May 2022 or later +"should" (as of this writing) suffice. Any older version may or may not +work.</div> + +After that succeeds, we need to run the normal build so that those +generated files can be compiled in to the fossil binary, accessible +via the [/help/www/builtin|/builtin page]: + +<pre><code>$ make</code></pre> + +Before checking in those newly-built files, they need to be tested by +running the [/pikchrshow] page. If that page loads, the compilation +process fundamentally worked (a load failure will be made obvious to +the viewer). If it fails to load then the browser's dev tools console +likely provides at least a small hint (and <em>sometimes</em> a useful +hint) about the nature of the problem. Don't check those files in +until [/pikchrshow] runs, though! + +Should pikchr's C interface ever change, <tt>pikchr-worker.js</tt> +will need to be updated to accommodate it, but such modification is +typically trivial. + +<h3>8.1 Solutions other than Emscripten?</h3> + +Emscripten is not the only option for building C as WASM, but it +provides a complete toolchain which eliminates many other steps which +must otherwise be accounted for on a per-project basis. Despite its +convenience, it behooves us to explore other build options for the +sake of portability and avoiding what amounts to vendor lock-in. + +For later refererence, here are articles specifically covering +building WASM projects without using Emscripten: + + * [https://surma.dev/things/c-to-webassembly/] + * [https://schellcode.github.io/webassembly-without-emscripten] diff --git a/www/cap-theorem.md b/www/cap-theorem.md new file mode 100644 index 0000000..d65a89f --- /dev/null +++ b/www/cap-theorem.md @@ -0,0 +1,173 @@ +# Fossil and the CAP Theorem + +[The CAP theorem][cap] is a fundamental mathematical proof about +distributed systems. A software system can no more get around it than a +physical system can get past *c*, the [speed of light][sol] constant. + +Fossil is a distributed system, so it can be useful to think about it in +terms of the CAP theorem. We won’t discuss the theorem itself or how you +reason using its results here. For that, we recommend [this article][tut]. + +[cap]: https://en.wikipedia.org/wiki/CAP_theorem +[sol]: https://en.wikipedia.org/wiki/Speed_of_light +[tut]: https://www.ibm.com/cloud/learn/cap-theorem + + +<a id="ap"></a> +## Fossil Is an AP-Mode System + +As with all common [DVCSes][dvcs], Fossil is an AP-mode system, meaning +that your local clone isn’t necessarily consistent with all other clones +(C), but the system is always available for use (A) and +partition-tolerant (P). This is what allows you to turn off Fossil’s +autosync mode, go off-network, and continue working with Fossil, even +though only a single node (your local repo clone) is accessible at the +time. + +You may consider that going back online restores “C”, because upon sync, +you’re now consistent with the repo you cloned from. But, if another +user has gone offline in the meantime, and they’ve made commits to their +disconnected repo, *you* aren’t consistent with *them.* Besides which, +if another user commits to the central repo, that doesn’t push the +change down to you automatically: even if all users of a Fossil system +are online at the same instant, and they’re all using autosync, Fossil +doesn’t guarantee consistency across the network. + +There’s no getting around the CAP theorem! + +[dvcs]: https://en.wikipedia.org/wiki/Distributed_version_control + + +<a id="ca"></a> +## CA-Mode Fossil + +What would it mean to redesign Fossil to be CA-mode? + +It means we get a system that is always consistent (C) and available (A) +as long as there are no partitions (P). + +That’s basically [CVS] and [Subversion][svn]: you can only continue +working with the repository itself as long as your connection to the central repo server functions. + +It’s rather trivial to talk about single-point-of-failure systems like +CVS or Subversion as +CA-mode. Another common example used this way is a classical RDBMS, but +aren’t we here to talk about distributed systems? What’s a good example +of a *distributed* CA-mode system? + +A better example is [Kafka], which in its default configuration assumes +it being run on a corporate LAN in a single data center, so network +partitions are exceedingly rare. It therefore sacrifices partition +tolerance to get the advantages of CA-mode operation. In its particular application of +this mode, a +message isn’t “committed” until all running brokers have a copy of it, +at which point the message becomes visible to the client(s). In that +way, all clients always see the same message store as long as all of the +Kafka servers are up and communicating. + +How would that work in Fossil terms? + +If there is only one central server and I clone it on my local laptop, +then CA mode means I can only commit if the remote Fossil is available, +so in that sense, it devolves to the old CVS model. + +What if there are three clones? Perhaps there is a central server *A*, +the clone *B* on my laptop, and the clone *C* on your laptop. Doesn’t CA +mode now mean that my commit on *B* doesn’t exist after I commit it to +the central repo *A* until you, my coworker, *also* pull down the copy +of that commit to your laptop *C*, validating the commit through the +network? + +That’s one way to design the system, but another way would be to scope +the system to only talk about proper *servers*, not about the clients. +In that model, a CA-mode Fossil alternative might require 2+ servers to +be running for proper replication. When I make a commit, if all of the +configured servers aren’t online, I can’t commit. This is basically CVS +with replication, but without any useful amount of failover. + +[CVS]: https://en.wikipedia.org/wiki/Concurrent_Versions_System +[Kafka]: https://engineering.linkedin.com/kafka/intra-cluster-replication-apache-kafka +[svn]: https://en.wikipedia.org/wiki/Apache_Subversion + + +<a id="cp"></a> +## CP-Mode Fossil + +What if we modify our CA-mode system above with “warm spares”? We can +say that commits must go to all of the spares as well as the active +servers, but a loss of one active server requires that one warm spare +come into active state, and all of the clients learn that the spare is +now considered “active.” At this point, you have a CP-mode system, not a +CA-mode system, because it’s now partition-tolerant (P) but it becomes +unavailable when there aren’t enough active servers or warm +spares to promote to active status. + +CP is your classical [BFT] style distributed consensus system, where the +system is available only if the client can contact a *majority* of the +servers. This is a formalization of the warm spare concept above: with +*N* server nodes, you need at least ⌊*N* / 2⌋ + 1 of them to be online +for a commit to succeed. + +Many distributed database systems run in CP mode because consistency (C) and +partition-tolerance (P) is a useful combination. What you lose is +always-available (A) operation: with a suitably bad partition, the +system goes down for users on the small side of that partition. + +An optional CP mode for Fossil would be attractive in some ways since in +some sense Fossil is a distributed DBMS, but in practical terms, it +means Fossil would then not be a [DVCS] in the most useful sense, being +that you could work while your client is disconnected from the remote +Fossil it cloned from. + +A fraught question is whether the non-server Fossil clones count as +“nodes” in this sense. + +If they do count, then if there are only two systems, the central server +and the clone on my laptop, then it stands to reason from the formula +above that I can only commit if the central server is available. In that +scheme, a CP-mode Fossil is basically like CVS. + +But what happens if my company hires a coworker to help me with the +project, and this person makes their own clone of the central repo? The +equation says I still need 2 nodes to be available for a commit, so if +my new coworker goes off-network, that doesn’t affect whether I can make +commits. Likewise, if I go off-network, my coworker can make commits to +the central server. + +But what happens if the central server goes down? The equation says we +still have 2 nodes, so we should be able to commit, right? Sure, but +only if my laptop and communicate directly to my coworker’s laptop! If +it can’t, that’s also a network partition, so *N=1* on both sides in +that case. The implication is that for a true CP-mode Fossil, we’d need +some kind of peer-to-peer networking layer so that our laptops can +accept commits from the other, so that when the central server comes +online, one of us can send the results up to it to get it caught up. + +But doesn’t that then mean there is no security? How does [Fossil’s RBAC +system][caps] work if peer-to-peer commits are allowed? + +You can instead reconceptualize the system as “node” meaning only server +nodes, so that client-only systems don’t count. This allows you to have +an RBAC system again. + +With just one central server, ⌊1/2⌋+1=1, so you get CVS-like behavior: +if the server’s up, you can commit. + +If you set up 2 servers for redundancy, both must be up for commits to +be allowed, since otherwise you could end up with half the commits going +to the server on one side of a network partition, half going to the +other, and no way to arbitrate among the two once the partition is +lifted. + +(Today’s AP-mode Fossil has this capability, but the necessary cost is +“C”, consistency! Once again, you can’t get around the CAP theorem.) + +3 servers is more sensible: any client that can see at least 2 of them +can commit. + +Will there ever be a CP-mode Fossil? This author doubts it, but as I’ve +shown, it would be useful in contexts where you’d rather have a +guarantee of consistency than availability. + +[BFT]: https://en.wikipedia.org/wiki/Byzantine_fault +[caps]: ./caps/ diff --git a/www/caps/admin-v-setup.md b/www/caps/admin-v-setup.md new file mode 100644 index 0000000..db1b412 --- /dev/null +++ b/www/caps/admin-v-setup.md @@ -0,0 +1,461 @@ +# Differences Between Setup and Admin Users + +This document explains the distinction between [Setup users][caps] and +[Admin users][capa]. For other information about use types, see: + +* [Administering User Capabilities](./) +* [How Moderation Works](../forum.wiki#moderation) +* [Users vs Subscribers](../alerts.md#uvs) +* [Defense Against Spiders](../antibot.wiki) + + +## <a id="philosophy"></a>Philosophical Core + +The Setup user "owns" the Fossil repository and may delegate a subset of +that power to one or more Admin users. + +The Setup user can grant Admin capability and take it away, but Admin +users cannot grant themselves Setup capability, either directly via the +Admin → Users UI page or via any indirect means. (If you discover +indirect means to elevate Admin privilege to Setup, it's a bug, so +please [report it][forum]!) + +It is common for the Setup user to have administrative control over the +host system running the Fossil repository, whereas it makes no sense for +Admin users to have that ability. If an Admin-only user had `root` +access on a Linux box running the Fossil instance they are an Admin on, +they could elevate their capability to Setup in several ways. (The +`fossil user` command, the `fossil sql` command, editing the repository +DB file directly, etc.) Therefore, if you wish to grant someone +Setup-like capability on a Fossil repository but you're unwilling to +give them a login on the host system, you probably want to grant +them Admin capability instead. + +Admin power is delegated from Setup. When a Setup user grants Admin +capability, it is an expression of trust in that user's judgement. + +Admin-only users must not fight against the policies of the Setup user. +Such a rift would be just cause for the Setup user to strip the Admin +user's capabilities. This may then create a fork in the project’s +development effort as the ex-Admin takes their clone and stands it up +elsewhere, so they may become that fork’s Setup user. + +A useful rule of thumb here is that Admin users should only change +things that the Setup user has not changed from the stock configuration. +In this way, an Admin-only user can avoid overriding the Setup user's +choices. + +You can also look at the role of Admin from the other direction, up +through the [user power hierarchy][ucap] rather than down from Setup. An +Admin user is usually a “super-developer” role, given full control over +the repository’s managed content: versioned artifacts in [the hash tree][bc], +[unversioned content][uv], forum posts, wiki articles, +tickets, etc. + +We’ll explore these distinctions in the rest of this document. + +[bc]: ../blockchain.md +[ucap]: ./index.md#ucap +[uv]: ../unvers.wiki + + +## <a id="binary"></a>No Granularity + +Fossil doesn’t make any distinction between these two user types beyond +this binary choice: Setup or Admin. + +A few features of Fossil are broken down so that only part of the +feature is accessible to Admin, with the rest left only to Setup users, +but for the most part each feature affected by this distinction is +either Admin + Setup or Setup-only. + +We could add more capability letters to break down individual +sub-features, but we’d run out of ASCII alphanumerics pretty quickly, +and we might even run out of ASCII punctuation and symbols. Then would +we need to shift to Unicode? + +Consider the Admin → Settings page, which is currently restricted to +Setup users only: you might imagine breaking this up into several +subsets so that some settings can be changed by Admin users. Is that a +good idea? Maybe, but it should be done only after due consideration. It +would definitely be wrong to assign a user capability bit to *each* +setting on that page. + +Now consider the opposite sort of case, Admin → Skins. Fossil grants +Admin users full access to this page so that the Admins can maintain and +extend the skin as the repository evolves, not so Admins can switch the +entire skin to another without consulting with the Setup user first. How +would Fossil decide, using user capabilities only, which skin changes +the Admin user is allowed to do, and which must be left to Setup? Do we +assign a separate capability letter to each step in `/setup_skin`? Do we +assign one more each to the five sections of a skin? (Header, Footer, +CSS, JavaScript, and Details.) It quickly becomes unmanageable. + + + +## <a id="capgroups"></a>Capability Groups + +We can break up the set of powers the Admin user capability grants into +several groups, then defend each group as a coherent whole. + + +### <a id="security"></a>Security + +While establishing the Fossil repository's security policy is a task for +the Setup user, *maintaining* that policy is something that Fossil +allows a Setup user to delegate to trustworthy users via the Admin user +capability: + +* **Manage users**: The only thing an Admin-only user cannot do on the + Admin → Users page is grant Setup capability, either to themselves + or to other users. The intent is that Admin users be able to take + some of the load of routine user management tasks off the shoulders + of the Setup user: delete accounts created by spammers, fix email + alert subscriptions, reset passwords, etc. + +* **Security audit**: The Admin → Security-Audit page runs several + tests on the Fossil repository's configuration, then reports + potential problems it found and offers canned solutions. Those + canned solutions do not do anything that an Admin-user could not do + via other means, so this page offers the Admin-only user no more + power than they otherwise had. For example, this page's "Take it + Private" feature can also be done manually via Admin → Users. This + page is a convenience, not a grant of new power to the Admin-only + user. + +* **Logging**:<a id="log"></a> Admin-only users get to see the various + Fossil logs in case they need to use them to understand a problem + they're empowered to solve. An obvious example is a spam attack: the + Admin might want to find the user's last-used IP, see if they cloned + the repository, see if they attempted to brute-force an existing + login before self-registering, etc. + +Some security-conscious people might be bothered by the fact that +Admin-only users have these abilities. Think of a large IT organization: +if the CIO hires a [tiger team][tt] to test the company's internal IT +defenses, the line grunts fix the reported problems, not the CIO. + + +### <a id="administrivia"></a>Administrivia + +It is perfectly fine for a Fossil repository to only have Setup users, +no Admin users. The smaller the repository, the more likely the +repository has no Admin-only users. If the Setup user neither needs nor +wants to grant Admin power to others, there is no requirement in Fossil +to do so. [Setup capability is a pure superset of Admin capability.][sia] + +As the number of users on a Fossil repository grows, the value in +delegating administrivia also grows, because the Setup user typically +has other time sinks they consider more important. + +Admin users can take over the following routine tasks on behalf of the +Setup user: + +* **Shunning**: After user management, this is one of the greatest + powers of an Admin-only user. Fossil grants access to the Admin → + Shunned page to Admin users rather than reserve it to Setup users + because one of the primary purposes of [the Fossil shunning + system][shun] is to clean up after a spammer, and that's + exactly the sort of administrivia we wish to delegate to Admin users. + + Coupled with the Rebuild button on the same page, an Admin user has + the power to delete the repository's entire + [hash tree][bc]! This makes this feature a pretty good + razor in deciding whether to grant someone Admin capability: do you + trust that user to shun Fossil artifacts responsibly? + + Realize that shunning is cooperative in Fossil. As long as there are + surviving repository clones, an Admin-only user who deletes the + whole hash tree has merely caused a nuisance. An Admin-only user + cannot permanently destroy the repository unless the Setup user has + been so silly as to have no up-to-date clones. + +* **Moderation**: According to [the user power hierarchy][ucap], + Admins are greater than Moderators, so control over + what Moderators can do clearly belongs to both Admins and to the + Setup user(s). + +* **Status**: Although the Fossil `/stat` page is visible to every + user with Read capability, there are several additional things this + page gives access to when a user also has the Admin capability: + + * <p>[Email alerts][ale] and [backoffice](../backoffice.md) + status. Admin-only users cannot modify the email alerts setup, + but they can see some details about its configuration and + current status.</p> + + * <p>The `/urllist` page, which is a read-only page showing the + ways the repository can be accessed and how it has been accessed + in the past. Logically, this is an extension to logging, + [covered above](#log).</p> + + * <p>The Fossil repository SQL schema. This is not particularly + sensitive information, since you get more or less the same + information when you clone the repository. It's restricted to + Admin because it's primarily useful in debugging SQL errors, + which happen most often when Fossil itself is in flux and the + schema isn't being automatically updated correctly. That puts + this squarely into the "administrivia" category.</p> + + * <p>Web cache status, environment, and logging: more + administrivia meant to help the Admin debug problems.</p> + +* **Configure search** + +[ale]: ../alerts.md +[shun]: ../shunning.wiki + + +### <a id="cosmetics"></a>Cosmetics + +While the Setup user is responsible for setting up the initial "look" of +a Fossil repository, the Setup user entrusts Admin users with +*maintaining* that look. An Admin-only user therefore has the following +special abilities: + +* Modify the repository skin + +* Create and modify URL aliases + +* Manage the "ad units" feature, if enabled. + +* Adjust the `/timeline` display preferences. + +* Change the "logo" element displayed by some skins. + +These capabilities allow an Admin-only user to affect the branding and +possibly even the back-end finances of a project. This is why we began +this document with a philosophical discussion: if you cannot entrust a +user with these powers, you should not grant that user Admin capability. + + +## <a id="clones"></a>Clones and Backups + +Fossil is a *distributed* version control system, which has direct +effects on the “Setup user” concept in the face of clones. When you +clone a repository, your local user becomes a Setup user on the local +clone even if you are not one on the remote repository. This may be +surprising to you, but it should also be sensible once you realize that +your operating system will generally give you full control over the +local repository file. What use trying to apply remote restrictions on +the local file, then? + +The distinctions above therefore are intransitive: they apply only +within a single repository instance. + +Fossil behaves differently when you do a clone as a user with Setup +capability on the remote repository, which primarily has effects on the +fidelity of clone-as-backup, which we cover [elsewhere](../backup.md). +We strongly encourage you to read that document if you expect to use a +clone as a complete replacement for the remote repository. + + +## <a id="apsu"></a>The All-Powerful Setup User + +Setup users get [every user capability](./ref.html) of Fossil except for +[two exceptionally dangerous capabilities](#dcap), which they can later +grant to themselves or to others. + +In addition, Setup users can use every feature of the Fossil UI. If Fossil can do a +thing, a Setup user on that repo can make Fossil do it. + +Setup users can do many things that Admin users cannot. They may not +only use all of the Admin UI features, they may also: + +* See record IDs (RIDs) on screens that show them +* See the MIME type of attachments on [`/ainfo` pages](/help/www/ainfo) +* See a remote repo’s HTTP [cache status](/help/www/cachestat) + and [pull cache entries](/help/www/cacheget) +* Edit a Setup user’s account! + +The “Admin” feature of Fossil UI is so-named because Admin users can use +about half of its functions, but only Setup can use these pages: + +* **Access**: This page falls under the [Security](#security) + category above, but like Configuration, it's generally something set + up once and never touched, so only Setup users should change it. + +* **Configuration**: This page nominally falls + under [Cosmetics](#cosmetics) above, but it's such a core part of the Fossil + configuration — something every Setup user is expected to fully + specify on initial repository setup — that we have trouble + justifying any case where an Admin-only user would have good cause + to modify any of it. This page is generally set up once and then + never touched again. + +* **Email-Server**: This is an experimental SMTP server feature which + is currently unused in Fossil. Should we get it working, it will + likely remain Setup-only, since it will likely be used as a + replacement for the platform’s default SMTP server, a powerful + position for a piece of software to take. + +* **Login-Group**: [Login groups][lg] allow one Fossil repository to + delegate user access to another. Since an Admin-only user on one + repo might not have such access to another repo on the same host + system, this must be a Setup-only task. + +* **Notification**: This is the main UI for setting up integration + with a platform’s SMTP service, for use in sending out [email + notifications][ale]. Because this screen can set commands to execute + on the host, and because finishing the configuration requires a + login on the Fossil host system, it is not appropriate to give Admin + users access to it. + +* **Settings**: The [repository settings][rs] available via Admin → + Settings have too wide a range of power to allow modification by + Admin-only users: + + * <p><b>Harmless</b>: Admin-only users on a repository may well + have checkin rights on the repository, so the fact that + versionable settings like `crlf-glob` can also be set at the + repository level seems like a thing we might want to allow + Admin-only users the ability to change. Since Fossil currently + has no way to allow only some settings to be changed by + Admin-only users and some not, we can't just show these harmless + settings to Admin-only users.</p> + + * <p><b>Low-Risk</b>: The <tt>admin-log</tt> setting controls + whether the Fossil admin log is generated. Since we've <a + href="#log">already decided</a> that Admin-only users can see + this log, it seems fine that the Admin users can choose whether + this log gets generated in the first place.</p> + + <p>There's a small risk that a rogue Admin user could disable + the log before doing something evil that the log would capture, + so ideally, we'd want to restrict changing this setting from 1 + to 0 to Setup only while allowing Admin-only users to change it + from 0 to 1. Fossil doesn't currently allow that.</p> + + * <p><b>Risky</b>: The <tt>https-login</tt> setting falls under + the "Security" section above, but it should probably never be + adjusted by Admin-only users. Sites that want it on will never + want it to be disabled without a very good reason.</p> + + <p>There is also an inverse risk: if the site has a front-end + HTTPS proxy that uses HTTP to communicate over localhost to + Fossil, enabling this setting will create an infinite redirect + loop! (Ask me how I know.)</p> + + * <p><b>Dangerous</b>: The <tt>email-send-command</tt> setting + could allow a rogue Admin to run arbitrary commands on the host + system, unless it's prevented via some kind of host-specific + restriction. (chroot, jails, SELinux, VMs, etc.) Since it makes + no sense to trust Admin-only users with <tt>root</tt> level + access on the host system, we almost certainly don't want to + allow them to change such settings.</p> + +* **SQL**: The Admin → SQL feature allows the Setup user to enter raw + SQL queries against the Fossil repository via Fossil UI. This not + only allows arbitrary ability to modify the repository hash tree + and its backing data tables, it can probably also be used to damage + the host such as via `PRAGMA temp_store = FILE`. + +* **Tickets**: This section allows input of arbitrary TH1 code that + runs on the server, affecting the way the Fossil ticketing system + works. The justification in the **TH1** section below therefore + applies. + +* **TH1**: The [TH1 language][th1] is quite restricted relative to the + Tcl language it descends from, so this author does not believe there + is a way to damage the Fossil repository or its host via the Admin → + TH1 feature, which allows execution of arbitrary TH1 code within the + repository's execution context. Nevertheless, interpreters are a + well-known source of security problems, so it seems best to restrict + this feature to Setup-only users as long as we lack a good reason + for Admin-only users to have access to it. + +* **Transfers**: This is for setting up TH1 hooks on various actions, + so the justification in the **TH1** section above applies. + +* **Wiki**: These are mainly cosmetic and usability settings. We might + open this up to Admin users in the future. + +Just remember, [user caps affect Fossil’s web interfaces only][webo]. A +user is a Setup user by default on their local clone of a repo, and +Fossil’s ability to protect itself against malicious (or even simply +incorrect) pushes is limited. Someone with clone and push capability on +your repo could clone it, modify their local repo, and then push the +changes back to your repo. Be careful who you give that combination of +capabilities to! + +When you run [`fossil ui`][fui], you are the Setup user on that repo +through that UI instance, regardless of the capability set defined in +the repo’s user table. This is true even if you cloned a remote repo +where you do not have Setup caps. This is why `ui` always binds to +`localhost` without needing the `--localhost` flag: in this mode, anyone +who can connect to that repo’s web UI has full power over that repo. + + +## <a id="dcap"></a>Dangerous Capabilities Initially Denied to Everyone + +There are two capabilities that Fossil doesn’t grant by default to Setup +or Admin users automatically. They are exceptionally dangerous, so +Fossil makes these users grant themselves (or others) these capabilities +deliberately, hopefully after careful consideration. + + +### <a id="y"></a>Write Unversioned + +Fossil currently doesn’t distinguish the sub-operations of +[`fossil uv`](/help/uv); they’re all covered by [**WrUnver**][capy] +(“y”) capability. Since some of these operations are unconditionally +destructive due to the nature of unversioned content, and since this +goes against Fossil’s philosophy of immutable history, nobody gets cap +“y” on a Fossil repo by default, not even the Setup or Admin users. A +Setup or Admin user must grant cap “y” to someone — not necessarily +themselves! — before modifications to remote +unversioned content are possible. + +Operations on unversioned content made without this capability affect +your local clone only. In this way, your local unversioned file table +can have different content from that in its parent repo. This state of +affairs will continue until your user either gets cap “y” and syncs that +content with its parent or you say `fossil uv revert` to make your local +unversioned content table match that of its parent repo. + + +### <a id="x"></a>Private Branch Push + +For private branches to remain private, they must never be accidentally +pushed to a public repository. It can be [difficult to impossible][shun] +to recover from such a mistake, so nobody gets [**Private**][capx] (“x”) +capability on a Fossil repo by default, not even Admin or Setup users. + +There are two common uses for private branches. + +One use is part of a local social contract allowing individual +developers to work on some things in private until they’re ready to push +them up to the parent repository. This goes against [a core tenet][fdp] +of Fossil’s design philosophy, but Fossil allows it, so some development +organizations do this. If yours is one of these, you might give cap “x” +to the “developer” category. + +The other use is in development organizations that follow the Fossil +philosophy, where you do not work in private unless you absolutely must. +You may have a public-facing project — let’s call it “SQLite” for the +sake of argument — but then someone comes along and commissions a custom +modification to your project which they wish to keep proprietary. You +do your work on a private branch, which you absolutely must never push +to the public repo, because that would be illegal. (Breach of contract, +copyright violation on a work-for-hire agreement, etc.) If you are using +Fossil in this way, we recommend that you give “x” capability to a +special developer account only, if at all, to minimize the chance of an +accidental push. + + +[capa]: ./ref.html#a +[caps]: ./ref.html#s +[capx]: ./ref.html#x +[capy]: ./ref.html#y + +[fcp]: https://fossil-scm.org/home/help/configuration +[fdp]: ../fossil-v-git.wiki#devorg +[forum]: https://fossil-scm.org/forum/ +[fui]: /help/ui +[lg]: ./login-groups.md +[rs]: https://fossil-scm.org/home/doc/trunk/www/settings.wiki +[sia]: https://fossil-scm.org/home/artifact?ln=1259-1260&name=0fda31b6683c206a +[snoy]: https://fossil-scm.org/forum/forumpost/00e1c4ecff +[th1]: ../th1.md +[tt]: https://en.wikipedia.org/wiki/Tiger_team#Security +[webo]: ./#webonly diff --git a/www/caps/impl.md b/www/caps/impl.md new file mode 100644 index 0000000..1f9b5a4 --- /dev/null +++ b/www/caps/impl.md @@ -0,0 +1,111 @@ +# Implementation Details of User Capabilities + +## <a id="choices"></a>Capability Letter Choices + +We [assigned][ref] user capability characters using only lowercase ASCII +letters at first, so those are the most important within Fossil: they +control the functions most core to Fossil’s operation. Once we used up +most of the lowercase letters, we started using uppercase, and then +during the development of the [forum feature][for] we assigned most of +the decimal numerals. All of the lowercase ASCII letters are now +assigned. Eventually, we might have to start using ASCII +punctuation and symbols. We expect to run out of reasons to define new caps before +we’re forced to switch to Unicode, though the possibilities for [mnemonic][mn] +assignments with emoji are intriguing. <span style="vertical-align: +bottom">😉</span> + +The existing caps are usually mnemonic, especially among the +earliest and therefore most central assignments, made when we still had +lots of letters to choose from. There is still hope for good future +mnemonic assignments among the uppercase letters, which are mostly still +unused. + + +## <a id="bitfield"></a>Why Not Bitfields? + +Some may question the use of ASCII character strings for [capability +sets][ucap] instead of bitfields, which are more efficient, both in +terms of storage and processing time. + +Fossil handles these character strings in one of two ways. For most HTTP +hits, Fossil [expands][sexp] the string into a [`struct` full of +flags][sff] so that later code can just do simple Boolean tests. In a +minority of cases, where Fossil only needs to check for the presence of +a single flag, it just does a [`strchr()` call][sc] on the string +instead. + +Both methods are slower than bit testing in a bitfield, but keep the +execution context in mind: at the front end of an HTTP request handler, +where the nanosecond differences in such implementation details are +completely swamped by the millisecond scale ping time of that repo’s +network connection, followed by the required I/O to satisfy the request. +Either method is plenty fast in that context. + +In exchange for this immeasurable cost per hit, we get human-readable +capability sets. + + +## <a id="filter"></a>Why Doesn’t Fossil Filter “Bad” Artifacts on Sync? + +Fossil is more trusting about the content it receives from a remote +clone during sync than you might expect. Common manifestations of this +design choice are: + +1. A user may be able to impersonate other users. This can be + [accidental](./index.md#defuser) as well as purposeful. + +2. If your local system clock is out-of-sync with absolute time, + artifacts committed to that repo will appear with the “wrong” time + when sync’d. If the time sync error is big enough, it can make + check-ins appear to go back in time and other bad effects. + +3. You can purposely overwrite good timestamps with bad ones and push + those changes up to the remote with no interference, even though + Fossil tries to make that a Setup-only operation. + +All of this falls out of two of Fossil’s design choices: sync is +all-or-nothing, and [the Fossil hash tree][bc] is immutable. Fossil +would have to violate one or both of these principles to filter such +problems out of incoming syncs. + +We have considered auto-[shunning][shun] “bad” content on sync, but this +is [difficult][asd] due to [the design of the sync protocol][dsp]. This +is not an impossible set of circumstances, but implementing a robust +filter on this input path would be roughly as difficult as writing a +basic [inter-frame video codec][ifvc]: do-able, but still a lot of +work. Patches to do this will be thoughtfully considered. + +We can’t simply change content as it arrives. Such manipulations would +change the artifact manifests, which would change the hashes, which +would require rewriting all parts of the block chain from that point out +to the tips of those branches. The local Fossil repo must then go +through the same process as the remote one on subsequent syncs in order +to build up a sync sequence that the remote can understand. Even if +you’re willing to accept all of that, this would break all references to +the old artifact IDs in forum posts, wiki articles, check-in comments, +tickets, etc. + +The bottom line here is that [**Clone**](./ref.html#g) and +[**Write**](./ref.html#i) are a potent combination of user capabilities. +Be careful who you give that pair to! + + +----- + +*[Back to Administering User Capabilities](./)* + +<!-- add padding so anchor links always scroll ref’d section to top --> +<div style="height: 75em"></div> + +[asd]: https://fossil-scm.org/forum/forumpost/ce4a3b5f3e +[bc]: ../blockchain.md +[dsp]: https://fossil-scm.org/fossil/doc/trunk/www/sync.wiki +[for]: ./forum.wiki +[ifvc]: https://en.wikipedia.org/wiki/Inter_frame +[mn]: https://en.wikipedia.org/wiki/Mnemonic +[ref]: ./ref.html +[sexp]: /artifact?ln=1223-1298&name=889d6724 +[sff]: /artifact?ln=80-117&name=52d2860f +[sc]: https://en.cppreference.com/w/c/string/byte/strchr +[shun]: ../shunning.wiki +[ucap]: ./index.md#ucap diff --git a/www/caps/index.md b/www/caps/index.md new file mode 100644 index 0000000..3492e5e --- /dev/null +++ b/www/caps/index.md @@ -0,0 +1,405 @@ +# Administering User Capabilities (a.k.a. Permissions) + +Fossil includes a powerful [role-based access control system][rbac] +which affects which users have which capabilities(^Some parts of the +Fossil code call these “permissions” instead, but since there is [a +clear and present risk of confusion](#webonly) with operating system +level file permissions in this context, we avoid using that term for +Fossil’s RBAC capability flags in these pages.) within a given +[served][svr] Fossil repository. We call this the “caps” system for +short. + +Fossil stores a user’s caps as an unordered string of ASCII characters, +one capability per, [currently](./impl.md#choices) limited to +[alphanumerics][an]. Caps are case-sensitive: “**A**” and “**a**” are +different user capabilities. + +This is a complex topic, so some sub-topics have their own documents: + +1. [Login Groups][lg] +2. [Implementation Details](./impl.md) +3. [User Capability Reference](./ref.html) + +[an]: https://en.wikipedia.org/wiki/Alphanumeric +[avs]: ./admin-v-setup.md +[lg]: ./login-groups.md +[rbac]: https://en.wikipedia.org/wiki/Role-based_access_control + + +## <a id="ucat"></a>User Categories + +Before we explain individual user capabilities and their proper +administration, we want to talk about an oft-overlooked and +misunderstood feature of Fossil: user categories. + +Fossil defines four user categories. Two of these apply based on the +user’s login status: **nobody** and **anonymous**. The other two act +like Unix or LDAP user groups: **reader** and **developer**. Because we +use the word “group” for [another purpose][lg] in Fossil, we will +avoid using it that way again in this document. The correct term in +Fossil is “category.” + +Fossil user categories give you a way to define capability sets for four +hard-coded situations within the Fossil C source code. Logically +speaking: + +> *(developer* ∨ *reader)* ≥ *anonymous* ≥ *nobody* + +When a user visits a [served Fossil repository][svr] via its web UI, +they initially get the capabilities of the “nobody” user category. This +category would be better named “everybody” because it applies whether +you’re logged in or not. + +When a user logs in as “anonymous” via [`/login`](/help?name=/login) they +get all of the “nobody” category’s caps plus those assigned to the +“anonymous” user category. It would be better named “user” because it +affects all logged-in users, not just those logged in via Fossil’s +anonymous user feature. + +When a user with either the “reader” ([**u**][u]) or “developer” +([**v**][v]) capability letter logs in, they get their [individual user +caps](#ucap) plus those assigned to this special user category. They +also get those assigned to the “anonymous” and “nobody” categories. + +Because “developer” users do not automatically inherit “reader” caps, +it is standard practice to give both letters to your “developer” users: +**uv**. You could instead just assign cap **u** to the “developer” +category. + +Fossil shows how these capabilities apply hierarchically in the user +editing screen (Admin → Users → name) with the `[N]` `[A]` `[D]` `[R]` +tags next to each capability check box. If a user gets a capability from +one of the user categories already assigned to it, there is no value in +redundantly assigning that same cap to the user explicitly. For example, +with the default **ei** cap set for the “developer” category, the cap +set **ve** is redundant because **v** grants **ei**, which includes +**e**. + +We suggest that you lean heavily on these fixed user categories when +setting up new users. Ideally, your users will group neatly into one of +the predefined categories, but if not, you might be able to shoehorn +them into our fixed scheme. For example, the administrator of a +wiki-only Fossil repo for non-developers could treat the “developer” +user category as if it were called “author,” and a forum-only repo could +treat the same category as if it were called “member.” + +There is currently no way to define custom user categories. + +[svr]: ../server/ + + +## <a id="ucap"></a>Individual User Capabilities + +When one or more users need to be different from the basic capabilities +defined in user categories, you can assign caps to individual users. You +may want to have the [cap reference][ref] open when doing such work. + +It is useful at this time to expand on the logical +expression [above](#cat), which covered only the four fixed user categories. +When we bring the individual user capabilities into it, the complete +expression of the way Fossil implements user power becomes: + +> *setup* ≥ *admin* ≥ *moderator* ≥ *(developer* ∨ *reader)* ≥ *[subscriber]* ≥ *anonymous* ≥ *nobody* + +The two additions at the top are clear: [setup is all-powerful][apsu], +and since admin users have [all capabilities][ref] except for Setup +capability, they are [subordinate only to the setup user(s)][avsp]. + +The moderator insertion could go anywhere from where it’s shown now down +to above the “anonymous” level, depending on what other caps you give to +your moderators. Also, there is not just one type of moderator: Fossil +has [wiki][l], [ticket][q], and [forum][5] moderators, each +independent of the others. Usually your moderators are fairly +high-status users, with developer capabilities or higher, but Fossil +does allow the creation of low-status moderators. + +The placement of “subscriber” in that hierarchy is for the +sort of subscriber who has registered an account on the repository +purely to [receive email alerts and announcements][7]. Users with +additional caps can also be subscribers, but not all users *are* in fact +subscribers, which is why we show it in square brackets. (See [Users vs +Subscribers](../alerts.md#uvs).) + +[apsu]: ./admin-v-setup.md#apsu +[avsp]: ./admin-v-setup.md#philosophy + + +## <a id="new"></a>New Repository Defaults + +Fossil creates one user account in new repos, which is named after your +OS user name [by default](#defuser). + +Fossil gives the initial repository user the [all-powerful Setup +capability][apsu]. + +Users who visit a [served repository][svr] without logging in get the +“nobody” user category’s caps which default to +**[g][g][j][j][o][o][r][r][z][z]**: clone the repo, read the wiki, +check-out files via the web UI, view tickets, and pull version archives. +This default is suited to random passers-by on a typical FOSS project’s +public web site and its code repository. + +Users who [prove they are not a bot][bot] by logging in — even if only +as “anonymous” — get the “nobody” capability set plus +**[h][h][m][m][n][n][c][c]**: see internal hyperlinks, append to +existing wiki articles, file new tickets, and comment on existing +tickets. We chose these additional capabilities as those we don’t want +bots to have, but which a typical small FOSS project would be happy to +give anonymous humans visiting the project site. + +The “reader” user category is typically assigned to users who want to be +identified within the repository but who primarily have a passive role +in the project. The default capability set on a Fossil repo adds +**[k][k][p][p][t][t][w][w]** caps to those granted by “nobody” and +“anonymous”. This category is not well-named, because the default caps +are all about modifying repository content: edit existing wiki pages, +change one’s own password, create new ticket report formats, and modify +existing tickets. This category would be better named “participant”. + +Those in the “developer” category get the “nobody” and “anonymous” cap +sets plus **[e][e][i][i]**: view +sensitive user material and check in changes. + +[bot]: ../antibot.wiki + + +## <a id="pvt"></a>Consequences of Taking a Repository Private + +When you click Admin → Security-Audit → “Take it private,” one of the +things it does is set the user capabilities for the “nobody” and +“anonymous” user categories to blank, so that users who haven’t logged +in can’t even see your project’s home page, and the option to log in as +“anonymous” isn’t even offered. Until you log in with a user name, all +you see is the repository’s skin and those few UI elements that work +without any user capability checks at all, such as the “Login” link. + +Beware: Fossil does not reassign the capabilities these users had to +other users or to the “reader” or “developer” user category! All users +except those with Setup capability will lose all capabilities they +inherited from “nobody” and “anonymous” categories. Setup is the [lone +exception][apsu]. + +If you will have non-Setup users in your private repo, you should parcel +out some subset of the capability set the “nobody” and “anonymous” +categories had to other categories or to individual users first. + + +## <a id="read-v-clone"></a>Reading vs. Cloning + +Fossil has two capabilities that are often confused: +[**Read**](./ref.html#o) and [**Clone**](./ref.html#g). + +The **Read** capability has nothing to do with reading data from a local +repository, because [caps affect Fossil’s web interfaces +only](#webonly). Once you’ve cloned a remote repository to your local +machine, you can do any reading you want on that repository irrespective +of whether your local user within that repo has <b>Read</b> capability. +The repo clone is completely under your user’s power at that point, +affected only by OS file permissions and such. If you need to prevent +that, you want to deny **Clone** capability instead. + +Withholding the **Read** capability has a different effect: it +prevents a web client from viewing [embedded documentation][edoc], +using [the file browser](/help?name=/dir), +exploring the [history](/help?name=/timeline) of check-ins, +and pulling file content via the [`/artifact`](/help?name=/artifact), +[`/file`](/help?name=/file), and [`/raw`](/help?name=/raw) URLs. +It is common to withhold **Read** capability from low-status visitors +on private or semi-private repos to prevent them from pulling individual +elements of the repo over the web one at a time, as someone may do when +denied the bulk **Clone** capability. + +[edoc]: ../embeddeddoc.wiki + + +## <a id="defuser"></a>Default User Name + +By default, Fossil assumes your OS user account name is the same as the +one you use in any Fossil repository. It is the [default for a new +repository](#new), though you can override this with [the `--admin-user` +option][auo]. Fossil has other ways of overriding this in other contexts +such as the `name@` syntax in clone URLs. + +It’s simplest to stick with the default; a mismatch can cause problems. +For example, if you clone someone else’s repo anonymously, turn off +autosync, and make check-ins to that repository, they will be assigned +to your OS user name by default. If you later get a login on the remote +repository under a different name and sync your repo with it, your +earlier “private” check-ins will get synced to the remote under your OS +user name! + +When such problems occur, you can amend the check-in to hide the +incorrect name from Fossil reports, but the original values remain in +the repository [forever][shun]. It is [difficult enough][fos] to fix +such problems automatically during sync that we are unlikely to ever do +so. + +[auo]: /help?name=new +[fos]: ./impl.md#filter +[shun]: ../shunning.wiki + + + +## <a id="utclone"></a>Cloning the User Table + +When cloning over HTTP, the initial user table in the local clone is set +to its “[new state:](#new)” only one user with Setup capability, named +after either your OS user account, per the default above, or after the +user given in the clone URL. + +There is one exception: if you clone as a named Setup user, you get a +complete copy of the user information. This restriction keeps the user +table private except for the only user allowed to make absolutely +complete clones of a remote repo, such as for failover or backup +purposes. Every other user’s clone is missing this and a few other +items, either for information security or PII privacy reasons. + +When cloning with file system paths, `file://` URLs, or over SSH, you +get a complete clone, including the parent repo’s complete user table. + +All of the above applies to [login groups][lg] as well. + + +## <a id="webonly"></a>Caps Affect Web Interfaces Only + +Fossil’s user capability system only affects accesses over `http[s]://` +URLs. This includes clone, sync/push/pull, the [UI pages][wp], and [the +JSON API][japi]. For everything else, the user caps aren’t consulted at +all. + +The only checks made when working directly with a local repository are +the operating system’s file system permissions. This should strike you +as sensible, since if you have read access to the repository file, you +can do anything you want to that repo DB including giving your user’s +record the [**Setup**][s] capability, after which Fossil’s user +capability system is effectively bypassed. (Or, create another Setup +user, with the same end effect.) If you’re objecting that you need +*write* access to the DB file to achieve this, realize that you can copy +a read-only file to another location, giving yourself write access to +it. + +This is why the `fossil ui` command +gives you Setup permissions within Fossil UI: it can’t usefully prevent +you from doing anything through the UI since only the local file system +permissions actually matter, and you can’t start `fossil ui` without +having at least read access to that file. + +What may be more surprising to you is that this is also true when +working on a *clone* done over a local file path, except that there are +then two sets of file system permission checks: once to modify the +working check-out’s repo clone DB file, then again on [sync][sync] with +the parent DB file. The Fossil capability checks are effectively +defeated because your user has [**Setup**][s] capability on both sides +of the sync. Be aware that those file checks do still matter, however: +Fossil requires write access to a repo DB while cloning from it, so you +can’t clone from a read-only repo DB file over a local file path. + +Even more surprising to you may be the fact that user caps do not affect +cloning and syncing over SSH! (Not unless you go [out of your way][sshfc] +to patch around it, at any rate.) When you make a change to such a +repository, the stock Fossil behavior is that the change first goes to the +local repo clone where file system +permissions are all that matter, but then upon sync, the situation is +effectively the same as when the parent repo is on the local file +system. The reason behind this is that if you can log into the remote +system over SSH and that user has the necessary file system permissions +on that remote repo DB file to allow clone and sync operations, then +we’re back in the same situation as with local files: there’s no point +trying to enforce the Fossil user capabilities when you can just modify +the remote DB directly, so the operation proceeds unimpeded by any user +capability settings on the remote repo. + +Where this gets confusing is that *all* Fossil syncs are done over the +HTTP protocol, including those done over `file://` and `ssh://` URLs, +not just those done over `http[s]://` URLs: + +* For `ssh://` URLs, Fossil pipes the HTTP conversation through a + local SSH client to a remote instance of Fossil running the + [`test-http`](/help?name=test-http) command to receive the tunneled + HTTP connection. [This interface is intentionally permissionless][sxycap]. + +* For `file://` URLs — as opposed to plain local file paths — + the “sending” Fossil instance writes its side of + the HTTP conversation out to a temporary file in the same directory + as the local repo clone and then calls itself on the “receiving” + repository to read that same HTTP transcript file back in to apply + those changes to that repository. Presumably Fossil does this + instead of using a pipe to ease portability to Windows. + +Despite use of HTTP for these URL types, the fact remains that +checks for capabilities like [**Read**][o] and [**Write**][i] within the +HTTP conversation between two Fossil instances only have a useful effect +when done over an `http[s]://` URL. + +[sshfc]: ../server/any/http-over-ssh.md +[sxycap]: /file?ci=ec5efceb8aac6cb4&name=src/main.c&ln=2748-2752 + + +## <a id="pubpg"></a>Public Pages + +In Admin → Access, there is an option for giving a list of [globs][glob] +to name URLs which get treated as if the visitor had [the default cap +set](#defcap). For example, you could take the [**Read**][o] capability +away from the “nobody” user category, who has it by default, to prevent +users without logins from pulling down your repository contents one +artifact at a time, yet give those users the ability to read the project +documentation by setting the glob to match your [embedded +documentation][edoc]’s URL root. + + +## <a id="defcap"></a>Default User Capability Set + +In Admin → Access, you can define a default user capability set, which +is used as: + +1. the default caps for users newly created by an Admin or Setup user +2. the default caps for self-registered users, an option in that same UI +3. the effective caps for URIs considered [public pages](#pubpg) + +This defaults to [**Reader**][u]. + + +<!-- add padding so anchor links always scroll ref’d section to top --> +<div style="height: 75em"></div> + +[ref]: ./ref.html + +[a]: ./ref.html#a +[b]: ./ref.html#b +[c]: ./ref.html#c +[d]: ./ref.html#d +[e]: ./ref.html#e +[f]: ./ref.html#f +[g]: ./ref.html#g +[h]: ./ref.html#h +[i]: ./ref.html#i +[j]: ./ref.html#j +[k]: ./ref.html#k +[l]: ./ref.html#l +[m]: ./ref.html#m +[n]: ./ref.html#n +[o]: ./ref.html#o +[p]: ./ref.html#p +[q]: ./ref.html#q +[r]: ./ref.html#r +[s]: ./ref.html#s +[t]: ./ref.html#t +[u]: ./ref.html#u +[v]: ./ref.html#v +[w]: ./ref.html#w +[x]: ./ref.html#x +[y]: ./ref.html#y +[z]: ./ref.html#z + +[2]: ./ref.html#2 +[3]: ./ref.html#3 +[4]: ./ref.html#4 +[5]: ./ref.html#5 +[6]: ./ref.html#6 +[7]: ./ref.html#7 + +[glob]: https://en.wikipedia.org/wiki/Glob_(programming) +[japi]: https://docs.google.com/document/d/1fXViveNhDbiXgCuE7QDXQOKeFzf2qNUkBEgiUvoqFN4/view#heading=h.6k0k5plm18p1 +[sp]: ../sync.wiki +[sync]: /help?name=sync +[wp]: /help#webpages diff --git a/www/caps/login-groups.md b/www/caps/login-groups.md new file mode 100644 index 0000000..7cca2d7 --- /dev/null +++ b/www/caps/login-groups.md @@ -0,0 +1,128 @@ +# Login Groups + +The Admin → Login-Groups UI feature and its corresponding [`login-group` +command][lg] solve a common problem with Fossil: you’ve created multiple +repositories that some subset of users need access to, and you +don’t want to redundantly administer the user credentials for each +repository. + + +## Restrictions + +This feature ties changes to the “`user`” table in one repo to that in +one or more other repos, keyed by the user’s name. The interactions are +non-obvious because although the goal is for the end result to “just +work,” there are practical security and administration matters that +complicate things: + +1. Login group handling only works between joined repositories for the + subset of users with the same name. + +1. If you’re logged in on one repo in a group, any other repo in that + group that has a matching user record will accept your valid login. + +1. When you set up a login group between two repos, the user tables + aren’t merged, so even though you may have users that appear in + both, they will retain their initial passwords, credentials, and so + forth. + +1. The same is true after the login group is created: changes you make + to the user table in one repo in the group only propagate to the + other repos in the group when you check the “Apply changes to all + repositories“ box in the “Scope” section of the user edit screen. + Otherwise, user changes remain local to the repo you made them on. + +1. Login groups only affect [the HTTP interfaces][wo]. Contrast things + like `ssh://` clones, where unless you [go out of your way][sh] to + force them to run over one of the HTTP interfaces that pays + attention to Fossil’s RBAC system, login groups aren’t consulted. + + +## Interactions + +These restrictions combine in subtle and interesting ways. Examples: + +* **#1 and #2**: If you are logged into repo C as “charlie” and then + try to visit joined repo A where “charlie” doesn’t exist, your valid + login on C won’t get you into A. + +* **#2 and #3**: If “alice” exists in both of these same repos, + logging in on A gets her into C, but if she has different user + capabilities on each from the time before the two repos joined the + login group, her caps on A don’t apply to C, nor vice versa. + + Let us say F is a forum-only repo, and W is a wiki-only repo, and + that Alice has forum-posting rights on F and wiki-editing rights on + W. If both repos are joined by a login group, Alice can log in on F + and then access W without logging in on it separately, but she + cannot then post a forum message on W even though she could on F. + +* **#3 and #4**: If you change the caps for user “alice” on one repo + in a group and tell Fossil to apply the changes to all repos in the + group, the new caps will *overwrite* those on the other repos, not + merge with them. + + To extend the practical example from the prior point, let us say you + wish to grant Alice the “write unversioned” capability on both F and + W. If you check that single user cap box on F plus the “apply to + all” option, then “Apply Changes,” she will end up with forum + + unversioned caps on repo W, losing her wiki-editing caps in the + process. + + If you want user caps to differ on each repo, you must administer + them separately even if there is a common subset of caps between all + repos in the group for that user. Remember: selecting the “apply to + all” box calls for an overwrite operation, not a merge. + +* **#4 and #1**: If you make a change to an existing user “bob” in + repo B and select the “apply to all” option, it will only affect + other repos in the group that have a user “bob” configured. + + But, if you are instead creating user “bob” for the first time and + select that option, that user *will* be created in all repos. The + same is true of user deletion: that destructive action will + propagate through the group if you request it. + +* **#5 and #1**: If you have a user “daisy” on both repos A and B in a + login group, logging in over the web to A doesn’t let you push + changes into B over SSH. Without the workaround linked above, SSH + only pays attention to the operating system’s user authentication + system, not Fossil’s. + + Inversely, if Daisy successfully logs in over SSH to repo B, she + gains no access to any of the other repos in that group. She needs + at least one valid login over HTTP to one of the group’s repos. + + +## Discussion + +The end result of all of this is that you can have a subset of users +with credentials only on repo A, a different subset only on B, and a +third subset common to both. The only thing selecting which case applies +is restriction #1 above. + +Login groups have names. A repo can be in only one of these named login +groups at a time. + +Trust in login groups is transitive within a single server. Consider +this sequence: + + $ cd /path/to/A/checkout + $ fossil login-group join --name G ~/museum/B.fossil + $ cd /path/to/C/checkout + $ fossil login-group join ~/museum/B.fossil + +That creates login group G joining repo A to B, then joins C to B. +Although we didn’t explicitly tie C to A, a successful login on C gets +you into both A and B, within the restrictions set out above. + +Changes are transitive in the same way, provided you check that “apply +to all” box on the user edit screen. + +[lg]: /help/login-group +[sh]: ../server/any/http-over-ssh.md +[wo]: ./index.md#webonly + +----- + +*[Back to Administering User Capabilities](./)* diff --git a/www/caps/ref.html b/www/caps/ref.html new file mode 100644 index 0000000..9c218ff --- /dev/null +++ b/www/caps/ref.html @@ -0,0 +1,454 @@ +<div class='fossil-doc' data-title="User Capability Reference"> + +<style type="text/css"> + p#backlink { + /* Make empty space below the table so hyperlinks to named anchors + near the bottom of the table still scroll that row to the top of + the user's browser, even on tall screens. */ + margin-bottom: 75em; + } + + tr > th { + background-color: #e8e8e8; + vertical-align: top; + } + body.fossil-dark-style tr > th { + color: #000; + opacity: 0.85; + } + + tr.cols th { + white-space: nowrap; + } + + td, th { + padding: 0.4em; + } +</style> + +<p>Here we document each currently-defined user capability character in +more detail than the brief summary on the <a +href="/setup_ucap_list">“key” page</a> in the Fossil user editor. Each +row gives the capability letter used in the Fossil user editor followed +by the C code’s name for that cap within the <tt>FossilUserPerms</tt> +object, so you can use this reference both from the UI down and from the +C code up.</p> + +<p>The <a href="https://en.wikipedia.org/wiki/Mnemonic">mnemonics</a> +given here vary from obviously-correct to <i>post facto</i> +rationalizations to the outright fanciful. To <a +href="./impl.md#choices">some extent</a>, this is unavoidable.</p> + + +<h2>Reference</h2> + +<table> + <tr class="cols"> + <th>?</th> + <th>Name</th> + <th style="text-align: left">Description</th> + </tr> + + <tr id="a"> + <th>a</th> + <th>Admin</th> + <td> + Admin users have <em>all</em> of the capabilities below except for + <a href="#s">setup</a>, <a herf="#x">Private</a>, and <a href="#y">WrUnver</a>. + See <a href="admin-v-setup.md">Admin vs. Setup</a> for a more + nuanced discussion. Mnemonic: <b>a</b>dministrate. + </td> + </tr> + + <tr id="b"> + <th>b</th> + <th>Attach</th> + <td> + Add attachments to wiki articles, technotes, or tickets. + Mnemonics: <b>b</b>ind, <b>b</b>utton, <b>b</b>ond, or <b>b</b>olt. + See also: <a href='#B'>B</a>. + </td> + </tr> + + <tr id="c"> + <th>c</th> + <th>ApndTkt</th> + <td> + Append comments to existing tickets. Mnemonic: <b>c</b>omment. + </td> + </tr> + + <tr id="d"> + <th>d</th> + <th>n/a</th> + <td> + Legacy capability letter from Fossil's forebear <a + href="http://cvstrac.org/">CVSTrac</a>, which has no useful + meaning in Fossil due to the nature of its durable Merkle tree design. + We recommend that you remove it in case we + ever reuse this letter for another purpose. See <a + href="https://fossil-scm.org/forum/forumpost/43c78f4bef">this + post</a> for details. + </td> + </tr> + + <tr id="e"> + <th>e</th> + <th>RdAddr</th> + <td> + View <a + href="https://en.wikipedia.org/wiki/Personal_data">personal + identifying information</a> (PII) about other users such as email + addresses. Mnemonics: show <b>e</b>mail addresses; or + <b>E</b>urope, home of <a + href="https://en.wikipedia.org/wiki/General_Data_Protection_Regulation">GDPR</a>. + </td> + </tr> + + <tr id="f"> + <th>f</th> + <th>NewWiki</th> + <td> + Create new wiki articles. Mnemonic: <b>f</b>ast, English + translation of the Hawaiian word <a + href="https://en.wikipedia.org/wiki/History_of_wikis#WikiWikiWeb,_the_first_wiki"><i>wiki</i></a>. + </td> + </tr> + + <tr id="g"> + <th>g</th> + <th>Clone</th> + <td> + Clone the repository. Note that this is distinct from <a + href="#o">check-out capability, <b>o</b></a>; and that upon cloning + not just files, but also tickets, wikis, technotes and forum posts + are tranferred. Mnemonic: <b>g</b>et. + </td> + </tr> + + <tr id="h"> + <th>h</th> + <th>Hyperlink</th> + <td> + Get hyperlinks in generated HTML which link you to other parts of + the repository. This capability exists so we can deny it to the + “nobody” category, to <a href="../antibot.wiki">prevent bots from + wandering around aimlessly</a> in the site’s hyperlink web, <a + href="../loadmgmt.md">chewing up server resources</a> to little + good purpose. Mnemonic: <b>h</b>yperlink. + </td> + </tr> + + <tr id="i"> + <th>i</th> + <th>Write</th> + <td> + Check changes into the repository. Note that a lack of this + capability does not prevent you from checking changes into your + local clone, only from syncing those changes up to the parent + repo, and then <a href="./basics.md#webonly">only over HTTP</a>. + Also note that not just files, but also tickets, wikis, technotes + and forum posts will be accepted from clones upon syncronization. + Granting this capability also grants <b>o (Read)</b> Mnemonics: + <b>i</b>nput, check <b>i</b>n changes. + </td> + </tr> + + <tr id="j"> + <th>j</th> + <th>RdWiki</th> + <td> + View wiki articles. Mnemonic: in<b>j</b>est page content. (All + right, you critics, you do better, then.) + </td> + </tr> + + <tr id="k"> + <th>k</th> + <th>WrWiki</th> + <td> + Edit wiki articles. Granting this capability also grants <a + href="#j"><b>RdWiki</b></a> and <a href="#m"><b>ApndWiki</b></a>, + but it does <em>not</em> grant <a href="#f"><b>NewWiki</b></a>! + Mnemonic: <b>k</b>ontribute. + </td> + </tr> + + <tr id="l"> + <th>l</th> + <th>ModWiki</th> + <td> + Moderate <a href="#m">wiki article appends</a>. Appends do not get + saved permanently to the receiving repo’s block chain until <a + href="#s">Setup</a> or someone with this cap approves it. + Mnemonic: a<b>l</b>low. + </td> + </tr> + + <tr id="m"> + <th>m</th> + <th>ApndWiki</th> + <td> + Append content to existing wiki articles. Mnemonic: a<b>m</b>end + wiki + </td> + </tr> + + <tr id="n"> + <th>n</th> + <th>NewTkt</th> + <td> + File new tickets. Mnemonic: <b>n</b>ew ticket. + </td> + </tr> + + <tr id="o"> + <th>o</th> + <th>Read</th> + <td> + Read content and history of files from a remote Fossil instance over + HTTP. See <a href="index.md#read-v-clone">Reading vs. + Cloning</a>. Mnemonic: check <b>o</b>ut remote repo contents. + </td> + </tr> + + <tr id="p"> + <th>p</th> + <th>Password</th> + <td> + Change one’s own password. Mnemonic: <b>p</b>assword. + </td> + </tr> + + <tr id="q"> + <th>q</th> + <th>ModTkt</th> + <td> + Moderate tickets: delete comments appended to tickets. Mnemonic: + <b>q</b>uash noise commentary. + </td> + </tr> + + <tr id="r"> + <th>r</th> + <th>RdTkt</th> + <td> + View existing tickets. Mnemonic: <b>r</b>ead tickets. + </td> + </tr> + + <tr id="s"> + <th>s</th> + <th>Setup</th> + <td> + The <a href="./admin-v-setup.md#apsu">all-powerful Setup user</a>. + Mnemonics: <b>s</b>etup or <b>s</b>uperuser. + </td> + </tr> + + <tr id="t"> + <th>t</th> + <th>TktFmt</th> + <td> + Create new ticket report formats. Note that although this allows + the user to provide SQL code to be run in the server’s context, + and this capability is given to the untrusted “anonymous” user + category by default, this is a safe capability to give to users + because it is internally restricted to read-only queries on the + tickets table only. (This restriction is done with an SQLite + authorization hook, not by any method so weak as SQL text + filtering.) Mnemonic: new <b>t</b>icket report. + </td> + </tr> + + <tr id="u"> + <th>u</th> + <th>n/a</th> + <td> + Inherit all capabilities of the “reader” user category; does not + have a dedicated flag internally within Fossil. Mnemonic: + <a href="./index.md#ucat"><b>u</b>ser</a> + </td> + </tr> + + <tr id="v"> + <th>v</th> + <th>n/a</th> + <td> + Inherit all capabilities of the “developer” user category; does + not have a dedicated flag internally within Fossil. Mnemonic: + de<b>v</b>eloper. + </td> + </tr> + + <tr id="w"> + <th>w</th> + <th>WrTkt</th> + <td> + Edit existing tickets. Granting this capability also grants <a + href="#r"><b>RdTkt</b></a>, <a href="#c"><b>ApndTkt</b></a>, and + <a href="#n"><b>NewTkt</b></a>. Mnemonic: <b>w</b>rite to ticket. + </td> + </tr> + + <tr id="x"> + <th>x</th> + <th>Private</th> + <td> + Push or pull <a href="../private.wiki">private branches</a>. + Mnemonic: e<b>x</b>clusivity; “x” connotes unknown material in + many Western languages due to its <a + href="https://en.wikipedia.org/wiki/La_Géométrie#The_text">traditional + use in mathematics</a>. + </td> + </tr> + + <tr id="y"> + <th>y</th> + <th>WrUnver</th> + <td> + Push <a href="../unvers.wiki">unversioned content</a>. Mnemonic: + <b>y</b>ield, <a href="https://en.wiktionary.org/wiki/yield">sense + 4</a>: “hand over.” + </td> + </tr> + + <tr id="z"> + <th>z</th> + <th>Zip</th> + <td> + Pull archives of particular repository versions via <a + href="/help/www/zip"><tt>/zip</tt></a>, <a + href="/help/www/tarball"><tt>/tarball</tt></a>, and <a + href="/help/www/sqlar"><tt>/sqlar</tt></a> URLs. This is an + expensive capability to grant, because creating such archives can + put a large load on <a href="../server/">a Fossil server</a> which + you may then need to <a href="../loadmgmt.md">manage</a>. + Mnemonic: <b>z</b>ip file download. + </td> + </tr> + + <tr id="2"> + <th>2</th> + <th>RdForum</th> + <td> + Read <a href="../forum.wiki">forum posts</a> by other users. + Mnemonic: from thee <b>2</b> me. + </td> + </tr> + + <tr id="3"> + <th>3</th> + <th>WrForum</th> + <td> + Create new forum threads, reply to threads created by others, and + edit one’s own posts. New posts are <a + href="../forum.wiki#moderation">held for moderation</a> and do + not appear in repo clones or syncs. Granting this capability also + grants <a href="#2"><b>RdForum</b></a>. Mnemonic: post for + <b>3</b> audiences: me, <a href="#5">the mods</a>, and <a + href="https://en.wikipedia.org/wiki/The_Man">the Man</a>. + </td> + </tr> + + <tr id="4"> + <th>4</th> + <th>WrTForum</th> + <td> + Extends <a href="#3"><b>WrForum</b></a>, bypassing the moderation + and sync restrictions. Mnemonic: post <b>4</b> immediate release. + </td> + </tr> + + <tr id="5"> + <th>5</th> + <th>ModForum</th> + <td> + <a href="../forum.wiki#moderation">Moderate forum posts</a>. + Granting this capability also grants <a + href="#4"><b>WrTForum</b></a> and <a href="#2"><b>RdForum</b></a>, + so a user with this cap never has to moderate their own posts. + Mnemonic: “May I have <b>5</b> seconds of your time, honored + Gatekeeper?” + </td> + </tr> + + <tr id="6"> + <th>6</th> + <th>AdminForum</th> + <td> + Users with this capability see a checkbox on unmoderated forum + posts labeled “Trust user X so that future posts by user X do not + require moderation.” Checking that box and then clicking the + moderator-only “Approve” button on that post grants <a + href="#4"><b>WrTForum</b></a> capability to that post’s author. + There is currently no UI for a user with this cap to + <em>revoke</em> trust from a user once it is granted; only <a + href="#a"><b>Admin</b></a> and <a href="#s"><b>Setup</b></a> can + currently revoke granted caps. Granting this capability also + grants <a href="#5"><b>ModForum</b></a> and those it in turn + grants. Mnemonic: “I’m <b>6</b> [sick] of hitting Approve on your + posts!” + </td> + </tr> + + <tr id="7"> + <th>7</th> + <th>EmailAlert</th> + <td> + User can sign up for <a href="../alerts.md">email alerts</a>. + Mnemonic: <a href="https://en.wikipedia.org/wiki/Heaven_Can_Wait">Seven can + wait</a>, I’ve got email to read now. + </td> + </tr> + + <tr id="A"> + <th>A</th> + <th>Announce</th> + <td> + Send email announcements to users <a href="#7">signed up to + receive them</a>. Mnemonic: <b>a</b>nnounce. + </td> + </tr> + + <tr id="B"> + <th>B</th> + <th>Attach to Forum</th> + <td> + Add attachments to forum posts. See also: <a href='#b'>b</a>. + </td> + </tr> + + <tr id="C"> + <th>C</th> + <th>Chat</th> + <td> + Allow access to the <tt>/chat</tt> room. + </td> + </tr> + + <tr id="D"> + <th>D</th> + <th>Debug</th> + <td> + Enable debugging features. Mnemonic: <b>d</b>ebug. + </td> + </tr> + + <tr id="L"> + <th>L</th> + <th>Is-logged-in</th> + <td> + This is not a real capability, but is used in certain capability + checks, e.g. via <a href="../th1.md#capexpr">capexpr</a>. It + resolves to true if the current user is logged in. + Mnemonic: <b>L</b>ogged in. + </td> + </tr> + +</table> + +<hr/> + +<p id="backlink"><a href="./"><em>Back to Administering User +Capabilities</em></a></p> diff --git a/www/cgi.wiki b/www/cgi.wiki new file mode 100644 index 0000000..6b9c375 --- /dev/null +++ b/www/cgi.wiki @@ -0,0 +1,210 @@ +<title>CGI Script Configuration Options</title> + +<h1>Summary</h1> + +It is not necessary to have a central server in order to use Fossil. +But a central server can help a project run more smoothly by giving developers +a common point of rendezvous for syncing, and by providing a web-based +portal where developers and non-developers alike can learn about the +project and its current state. + +Setting up a server using Fossil is easy. +A [./server/|separate document] talks about all of the many different methods for +setting up a Fossil server, one of which is [./server/any/cgi.md | as a CGI +script]. CGI is the technique that the three +[./selfhost.wiki|self-hosting Fossil repositories] all use. + +Setting up a Fossil server using CGI is mostly about writing a short +script (usually just 2 lines) in the cgi-bin folder of an ordinary +web-server. But there are a lot of extra options that can be added +to this script, to customize the configuration. This article describes +those options. + +<h1>CGI Script Options</h1> + +The CGI script used to launch a Fossil server will usually look something +like this: + +<verbatim> +#!/usr/bin/fossil +repository: /home/www/fossils/myproject.fossil +</verbatim> + +Of course, pathnames will likely be different. The first line +(the "[wikipedia:/wiki/Shebang_(Unix)|shebang]") +always gives the name of the Fossil executable. Subsequent lines are of +the form "<b>property: argument ...</b>". +The remainder of this document describes the available properties and +their arguments. + +<hr> + +<h2 id="repository">repository: <i>PATH</i></h2> + +This property defines the Fossil repository that the server will use. +Every Fossil CGI requires either this property or the +[#directory|<b>directory:</b>] property (but not both). +Many Fossil CGI scripts have this one property and no other. + + +<h2 id="directory">directory: <i>PATH</i></h2> + +The PATH is the name of a directory that contains one or more Fossil +repository files having the suffix ".fossil". If this property is +used instead of [#repository|<b>repository:</b>], then the Fossil +server is able to serve all of the repositories in the directory. The +specific repository used is selected by a prefix on the PATH_INFO. See +the notes for the [#repolist|<b>repolist</b>] option regarding name +collisions between subdirectories and repository files. + + +<h2 id="notfound">notfound: <i>URL</i></h2> + +If the [#directory|<b>directory:</b>] option is used and if the PATH_INFO +of the HTTP request does not correspond to any Fossil repository, then +the request redirects to URL. + + +<h2 id="repolist">repolist</h2> + +This is a Boolean property. +If it is present, and if the [#directory:|<b>directory:</b>] option is used, +and if the PATH_INFO string is empty, then Fossil will show a list +of available Fossil repositories. + +The "skin" of the reply is determined by the first +repository in the list that has a non-zero +[/help/repolist-skin|repolist-skin] setting. + +If no repository has such a non-zero repolist-skin setting, then +the repository list is generic HTML without any decoration, with +the page title taken from the <tt>FOSSIL_REPOLIST_TITLE</tt> +environment variable. The variable can be defined in the CGI +control file using the [#setenv|<tt>setenv:</tt>] statement. + +The "Project Description" and "Login-Group" columns on the repolist page +are optional. They are hidden by default. Show them by +etting the <tt>FOSSIL_REPOLIST_SHOW</tt> environment variable to +a string that contains substrings "description" and/or "login-group". + +The repolist-generated page recurses into subdirectories and will list +all <tt>*.fossil</tt> files found, with the following exceptions: + + * Filenames starting with a period are treated as "hidden" and skipped. + + * Subdirectory names which match the base name of a fossil file in + the same directory are listed in the resulting page but are not + hyperlinked because the links would be ambiguous and the + repositories in the subdirectories would be unreachable to + clients. For example, any repositories under subdirectory + <tt>XYZ</tt> are unreachable if <tt>XYZ.fossil</tt> exists in + the same directory as <tt>XYZ</tt>, noting that this particular + name check is case-insensitive. The entries for such + repositories are clearly marked in the repolist page's output to + make the user aware of the problem. To make them accessible, + move them into a directory which does not share a base name with + a repository file. + + +<h2 id="localauth">localauth</h2> + +This is a Boolean property. +If it is present, [./caps/ref.html#s | setup capability] +is granted to any HTTP request that +comes in over a loopback interface, such as 127.0.0.1. + + +<h2 id="skin">skin: <i>NAME</i></h2> + +If NAME is the name of one of the built-in skins supported by Fossil, +then this option causes Fossil to display using that built-in skin, +and to ignore any custom skin that might be configured in the repository +itself. + +So, if you wanted to set up a server for a single Fossil project, but +also give users the option to use several of the different built-in +skins, you could create multiple CGI scripts, each with a different +"<b>skin:</b>" property, but all pointing to the same <b>repository:</b>. +Then users can select which skin to use by using the appropriate CGI. + + +<h2 id="files">files: </i>GLOBLIST</i></h2> + +The GLOBLIST argument is a comma-separate list of "globs" that specify +filenames. In [#directory|<b>directory:</b> mode], if the PATH_INFO +does not identify any Fossil repository, but it does refer some other +file in the directory, and that filename matches one of the glob patterns +in the GLOBLIST, then the file is returned as static content. + + +<h2 id="setenv">setenv: <i>NAME VALUE</i></h2> + +This parameter causes additional environment variable NAME to have VALUE. +This parameter can be repeated as many times as necessary. + + +<h2 id="HOME">HOME: <i>PATH</i></h2> + +This parameter is a short-hand for "<b>setenv HOME <i>PATH</i></b>". + + +<h2 id="cgi-debug">cgi-debug: <i>FILE</i></h2> + +Cause CGI-related debugging information to be appended in <i>FILE</i>. Use +this to help debug CGI problems. + + +<h2 id="errorlog">errorlog: <i>FILENAME</i></h2> + +This setting causes the server to log any errors in FILENAME. +It is ok for multiple Fossil CGIs to share the same error log. + +Setting up an error log for Fossil servers is not required, but it +is recommended. + + +<h2 id="timeout">timeout: <i>N</i></h2> + +This property changes the timeout on each CGI request to N seconds. +If N is zero, then there is no timeout. If this property is omitted, +then the default timeout is 300 seconds (5 minutes). + + +<h2 id="extroot">extroot: <i>PATH</i></h2> + +This property defines the DOCUMENT_ROOT for the +[./serverext.wiki|CGI Server Extensions]. If this property +is present, then CGI Server Extensions are enabled. When this +property is omitted, CGI Server Extensions are disabled. + +A cascade of CGI invocations can occur here. Fossil itself is +started as CGI, then Fossil can turn around and invoke a sub-CGI +extension. The sub-CGI extension outputs reply text, when Fossil +then (optionally) augments with its own header and footer and returns +to the original requestor. The property controls the DOCUMENT_ROOT +of the sub-CGI. + + +<h2 id="redirect">redirect: <i>REPO URL</i></h2> + +Extract the "name" query parameter and search REPO for a check-in or +ticket that matches the value of "name", then redirect to URL. There +can be multiple "redirect:" lines that are processed in order. If the +repo name is "*", then an unconditional redirect to URL is taken. + + +<h2 id="jsmode">jsmode: <i>VALUE</i></h2> + +Specifies the delivery mode for JavaScript files. See "[/help/http | +http --jsmode]" for the allowed values and their meanings. + + +<h2 id="mainmenu">mainmenu: <i>FILE</i></h2> + +This parameter causes the contents of the given file to override the +site's <tt>mainmenu</tt> configuration setting, in much the same way +that the <tt>skin</tt> setting overrides the skin. This can be used to +apply a common main menu to a number of sites, and centrally maintain +it, without having to copy its contents into each site. Note, however, +that the contents of this setting are not stored in the repository and +will not be cloned along with the repository. diff --git a/www/changes.wiki b/www/changes.wiki new file mode 100644 index 0000000..41cb3e3 --- /dev/null +++ b/www/changes.wiki @@ -0,0 +1,2297 @@ +<title>Change Log</title> + +<h2 id='v2_29'>Changes for version 2.29 (pending)</h2><ol> + <li> Enhance the [/help/www/ci_edit|/ci_edit] web page to show more + context of the check-in being edited. + <li> Honor the NO_COLOR environment variable in the + "[/help/system|fossil sys ls]" command. + <li> On the [/help/www/info|/info webpage] (and similar) put a + "copy" button before the date. + <li> Add the ".m4a" and ".otf" mimetypes. + <li> When doing a "fossil update", if a file under management needs to + overwrite an unmanaged file, display the name of the backup that + is made of the unmanaged file, and use file_delete() to delete + the unmanaged file, even if that unmanaged file is read-only. + <li> Improve the default prompts used by the + "[/help/sqlite3|fossil sql]" command. + <li> The captcha now uses light-gray boxes as the background, instead of + spaces, to work around width inconsistencies in some fonts. + <li> Forum post editing and replying was overhauled with a new UI. Clients + with JavaScript disabled will still see the older forms. + <li> Forum posts may now have attachments if their poster has the new "B" + capability. + <li> Add the "[/help/attachment-size-limit|attachment-size-limit]" setting + to limit the size of file attachments to wiki pages, tech notes, + tickets, and forum posts. + <li> Add the "[/help/path-to-tag|path-to-tag]" setting, which if set to + something like "release" provides a link named "path-to-release" on + /info pages for checkins. Clicking that link shows the path to the + first release containing that check-in. + <li> Addded several HTML5 tags deemed harmless to the MD/wiki allowlists. + <li> Fix various pages on deployments where a custom skin emits + its own <code>BODY</code> tag without the expected CSS classes. + <li> When behaving as an HTTP client, Fossil now always uses HTTP/1.1 and + it understands the "chunked" transfer encoding. + <li> The "diff" command (in all its variations) when invoked with --tk + now accepts an extra argument of the form "HOST:DIRECTORY" or + "USER@HOST:DIRECTORY" to specify a directory on a remote host in which + the diff is to be taken. The diff is run using SSH, the results are + captured, then displayed locally using the Tcl/Tk diff GUI. +</ol> + +<h2 id='v2_28'>Changes for version 2.28 (2026-03-11)</h2><ol> + <li> Improvements to [./antibot.wiki|anti-robot defenses]:<ol type="a"> + <li> The default configuration now allows robots to download any tarball + or similar, to better support of automated build systems. + <li> New special tag "zipX" for the [/help/robot-restrict|robot-restrict] + setting blocks robot access to tarballs, but with exceptions to support + automated build systems. + <li> Tags of the form "ext/PATH" in the robot-restrict setting block access + by robots to specific [./serverext.wiki|CGI extension] at PATH. + <li> Enhancements to the default value for the + [/help/robot-restrict|robot-restrict setting]. + </ol> + <li> A drop-down menu of recent branches is now possible for the submenu, and + is used in the [/dir?ci=trunk|code browser]. + <li> Simplified access to tarballs and ZIPs:<ol type="a"> + <li> When in the [/dir?ci=trunk|code browser] at the top-level, + a new "Download" submenu option is available to take the + user to a page where he can download a tarball or ZIP archive. + <li> New [/help/www/download|/download page] is available. When + configured using the new + [/help/suggested-downloads|suggested-downloads setting], a + link to [/download] named "Tarballs and ZIPs" appears in the + [/sitemap] and thus on the hamburger menu. + <li> The robot-resistant [/help/www/rchvdwnld|/rchvdwnld page] can be + used to provide human access to on-demand generated ZIP archives + and tarballs. This page is used to help implement the two previous + subitems. + <li> The filenames for tarballs and ZIPs are now standardized to + include a timestamp and a hash prefix. + <li> New "[/help/get|fossil get]" command downloads and unpacks a specific + check-in without having to clone the repository. + </ol> + <li> Web-based [/help/www/timeline|timeline] enhancements:<ol type="a"> + <li> A new "Simple" view is available. This is compromise between "Verbose" + and "Compact" that shows only the check-in hash rather than the full + detail section. There is an ellipsis that one can click on to see the + full detail text. + <li> The artifact hash in the detail section of each timeline entry is now + emphasized (controlled by CSS) to make it easier to locate amid all + the other text and links. + <li> When clicking on the ellipsis in "Compact" or "Simple" views, the + ellipsis is replaced by a left arrow ("←") which can be clicked to + hide the extra detail again. + <li> New setting [/help/timeline-mark-leaves|timeline-mark-leaves] controls + whether or not leaf check-ins are marked in the timeline. + <li> "No-graph" timelines (using the "ng" query parameter) now show + branch colors and bare check-in circles on the left. The check-in + circles appear, but no lines connecting them. + ([/timeline?ng|example]). + </ol> + <li> Labels in Markdown now have IDs generated using the GitHub "slugify" + algorithm. + <li> The [/help/timeline|timeline command] is enhanced with the new options + "<tt>-u|--for-user</tt>" to filter by user, and "<tt>-r</tt>" to display + entries in chronological order. + <li> The [/help/open|open command]'s new "<tt>--reopen REPOFILE</tt>" flag + can be used to fix a checkout after moving its repository file. + <li> Update internal Unicode character tables, used in regular expression + handling, from version 13 to 17. + <li> Add a zoom-message option to [/help/www/chat|/chat] to better support + pikchr diagrams. + <li> The new [/help/system|fossil system] (often abbreviated as just "fossil sys") + provides a few common unix-like commands that are missing on some + platforms. This is a convenience for Unix-based developers who occasionally + need to work on ill-provisioned alternative platforms. The set of commands + currently offered is small, but is expected to grow in future releases. + <li> The [/help/www/help|/help webpage] now accepts queries of the form + "/help/CMD" to show help for the command or setting CMD and + "/help/www/PAGE" to show help for webpage PAGE. + <li> Add the -t and -T options to the [/help/praise|fossil praise] command. + <li> The [/help/clone|fossil clone] command adds the --ipv6 option. + <li> Add "-s" and "--stop" as aliases for the "--stop-on-error" option in + the [/help/all|fossil all] command. + <li> Add the "-h|--hash" option to the [/help/whatis|fossil whatis] command. +</ol> + +<h2 id='v2_27'>Changes for version 2.27 (2025-09-30)</h2><ol> + <li> Close a potential Denial-of-Service attack against any public-facing Fossil + server involving exponential behavior in Fossil's regexp implementation. + <li> Fix a SQL injection on the [/help/www/file|/file page]. Thanks to + additional defenses built into Fossil, as well as good luck, this injection + is not exploitable for either data exfiltration or privilege escalation. The + only possible result of invoking the injection is a harmless SQL syntax error. + <li> Strengthen robot defenses to help prevent public-facing servers from being + overwhelmed by the latest generation of AI spiders. + <ol type="a"> + <li> New javascript captcha used to restrict access by user "nobody" to pages + listed in the [/help/robot-restrict|robot-restrict setting]. + <li> The [/help/robot-exception|robot-exception setting] is available to allow + access to pages that match a regular expression. Use this, for example, to + allow curl scripts and similar to download release tarballs. + <li> Require at least an anonymous login to access the /blame page and similar. + </ol> + <li> [/help/www/timeline|Timeline] enhancements: + <ol type="a"> + <li> The chng= query parameter on the [/help/www/timeline|timeline page] + so that it works with other query parameters like p=, d=, from=, and to=. + <li> Always include nodes identify by sel1= and sel2= in the /timeline display. + <li> Improved title when p= and d= are different. + </ol> + <li> Enable the --editor option on the [/help/amend|fossil amend] command. + <li> When walking the filesystem looking for Fossil repositories, avoid descending + into directories named "/proc". + <li> Reduce memory requirements for sending authenticated sync protocol + messages. + <li> Show numstat-style change statistics in the /info and /ckout pages. + <li> Add the [/help/stash | stash rename] subcommand. + <li> Add the "-h" option to the "[/help/ls|ls]" command to display + file hashes for a specific check-in when in verbose mode. + </ol> + +<h2 id='v2_26'>Changes for version 2.26 (2025-04-30)</h2><ol> + <li>Enhancements to [/help/diff|fossil diff] and similar: + <ol type="a"> + <li> The argument to the --from option can be a directory name, causing + Fossil to use files under that directory as the baseline for the diff. + <li> For "gdiff", if no [/help/gdiff-command|gdiff-command setting] + is defined, Fossil tries to do a --tk diff if "tclsh" and "wish" + are available, or a --by diff if not. + <li> The "Reload" button is added to --tk diffs, to bring the displayed + diff up to date with the latest changes on disk. + <li> Add the "Hide diffs/Show diffs" toggle to web-UI diff pages that show + diffs of multiple files. + </ol> + <li>Added the [/help/www/ckout|/ckout web page] to provide information + about pending changes in a working check-out + <li>Enhancements to the [/help/ui|fossil ui] command: + <ol type="a"> + <li> Defaults to using the new [/help/www/ckout|/ckout page] as its + start page. Or, if the new "--from PATH" option is present, the + default start page becomes "/ckout?exbase=PATH". + <li> The new "--extpage FILENAME" option opens the named file as if it + where in a [./serverext.wiki|CGI extension]. Example usage: the + person editing this change log has + "fossil ui --extpage www/changes.wiki" running and hence can + press "Reload" on the web browser to view edits. + <li> Accept both IPv4 and IPv6 connections on all platforms, including + Windows and OpenBSD. This also applies to the "fossil server" + command. + </ol> + <li>Enhancements to [/help/merge|fossil merge]: + <ol type="a"> + <li> Added the [/help/merge-info|fossil merge-info] command and + especially the --tk option to that command, to provide analysis + of the most recent merge or update operation. + <li> When a merge conflict occurs, a new section is added to the conflict + text that shows Fossil's suggested resolution to the conflict. + </ol> + <li>Enhancements to [/help/commit|fossil commit]: + <ol type="a"> + <li> If Fossil sees potential formatting mistakes (ex: bad hyperlinks) + in the check-in comment, it will alert the developer and give + him or her the opportunity to edit the comment before continuing. + This feature is controllable by the + [/help/verify-comments|verify-comments setting]. + <li> The new "--if-changes" option causes the commit to become + a quiet no-op if there are no pending changes. + <li> Added the ability to sign check-ins with SSH keys. Artifacts signed + this way are ignored by all previous fossil versions, as if they + were plain-text file content instead of Fossil artifacts. + <li> Issue a warning if a user tries to commit on a check-in where the + branch has been changed. + <li> The interactive checkin comment prompt shows the formatting rules + set for that repository. + <li> Add the "--editor" option. + </ol> + <li>Deprecate the --comfmtflags and --comment-format global options and + no longer list them in the built-in help, but keep them working for + backwards compatibility. + Alternative TTY comment formatting can still be specified using the + [/help/comment-format|comment-format setting], if desired. The + default comment format is now called "canonical", not "legacy". + <li>Enhancements to the [/help/www/timeline|/timeline page]: + <ol type="a"> + <li> Added the "ml=" ("Merge-in List") query parameter that works + like "rl=" ("Related List") but adds "mionly" style related + check-ins instead of the full "rel" style. + <li> For "tl=", "rl=", and "ml=", the order of the branches in the + graph now tries to match the order of the branches named in + the list. + <li> The "ms=" ("Match Style") query parameter is honored for + "tl=", "rl=", and "ml=". + <li> New query parameter "sl=BRANCHLIST" ("Sort List") strives to + put branches in the specified order in the graph. This + overrides any "tl=" or similar ordering. + <li> In the various "from=","to=" query formats, if the one of the + end points is an ancestor of the other, then the "rel" modifier + omits check-ins that are not ancestors of the newer endpoint. + <li> For "tl=" and similar query parameters, if the pattern contains + GLOB characters, then the matching style ("ms=") is set to GLOB + automatically and the "ms=" query parameter can be omitted. + <li> Enhance the "ymd" query parameter so that when used like + "ymd=YYYYMMDD-YYYYMMDD" it shows all events in the range of + dates specified. + <li> Accept the "Z" (Zulu-time) suffix on date arguments for the + "ymd" and "yw" query parameters. + <li> The new "min" query parameter, when added to a from=,to= query, + collapses long runs of check-ins on the same branch into just + end-points. + <li> The p= and d= parameters can now reference different check-ins, + in which case the timeline shows those check-ins that are both + ancestors of p= and descendants of d=. + <li> The saturation and intensity of user-specified checkin and branch + background colors are automatically adjusted to keep the colors + compatible with the current skin, unless the + [/help/raw-bgcolor|raw-bgcolor setting] is turned on. + </ol> + <li>The [/help/www/docfile|/docfile webpage] was added. It works like + /doc but keeps the title of markdown documents with the document rather + that moving it up to the page title. + <li>Added the [/help/www/clusterlist|/clusterlist page] for analysis + and debugging + <li>Added the "artifact_to_json(NAME)" SQL function that returns a JSON + decoding of the artifact described by NAME. + <li>Improvements to the [/help/patch|fossil patch] command: + <ol type="a"> + <li> Fix a bug in "fossil patch create" that causes + [/help/revert|fossil revert] operations that happened + on individualfiles after a [/help/merge|fossil merge] + to be omitted from the patch. + <li> Added the [/help/patch|patch alias] command for managing + aliases for remote checkout names. + </ol> + <li>Enhancements to on-line help and the [/help/help|fossil help] command: + <ol type="a"> + <li> Add the ability to search the help text, either in the UI + (on the [/help/www/search|/search page]) or from the command-line + (using the "[/help/search|fossil search -h PATTERN]" command.) + <li> Accepts an optional SUBCOMMAND argument following the + COMMAND argument and only shows results for the specified + subcommand, not the entire command. + <li> The -u (--usage) option shows only the command-line syntax + <li> The -o (--options) option shows only the command-line options + </ol> + <li>Enhancements to the [./tickets.wiki|ticket system]: + <ol type="a"> + <li> Added the ability to attach wiki pages to a ticket for extended + descriptions. + <li> Added submenu to the 'View Ticket' page, to use it as + template for a new ticket. + <li> Added button 'Submit and New' to create multiple tickets + in a row. + <li> Link the version field in ticket view to a matching checkin or tag. + <li> Show creation time in report and ticket view. + <li> Show previous comments in edit ticket as reference. + </ol> + <li>Added the "hash" query parameter to the + [/help/www/whatis|/whatis webpage]. + <li>Add a "user permissions changes" [/doc/trunk/www/alerts.md|subscription] + which alerts subscribers when an admin creates a new user or + when a user's permissions change. + <li>If the FOSSIL_REPOLIST_SHOW environment variable exists and contains + the substring "description", then the project description for each repository + is shown on the repository list page. The login-group for each project is + now only shown if the FOSSIL_REPOLIST_SHOW environment variable exists and + contains the substring "login-group". ([./cgi.wiki#repolist|More information]) + <li>The [/doc/trunk/www/th1.md|TH1 script language] is enhanced for improved + security: + <ol type="a"> + <li> TH1 now makes a distinction between + [/doc/trunk/www/th1.md#taint|tainted and untainted string values]. + This makes it more difficult to write custom TH1 scripts that + contain XSS or SQL-injection bugs. The + [/help/vuln-report|vuln-report] setting was added to control + what Fossil does when it encounters a potential TH1 + security problem. + <li> The "--th" option was removed from the [/help/pikchr|fossil pikchr] + command. + <li> The "enable_htmlify" TH1 command was removed. + </ol> + <li>Make [/help/www/chat|/chat] better-behaved during server outages, reducing + the frequency of reconnection attempts over time and providing feedback + to the user when the connection is down. + <li>The [/help/www/sqlar|/sqlar] page does not work for users who are not logged + in, nor are links to that page displayed to users who are not logged in. Being + logged in as "anonymous" is sufficient to overcome this restriction, assuming + that "anonymous" can download tarballs and ZIP archives. + <li>Many other minor fixes and additions. +</ol> + +<h2 id='v2_25'>Changes for version 2.25 (2024-11-06)</h2> + + * The "[/help/ui|fossil ui /]" command now works even for repositories + that have non-ASCII filenames + * Add the [/help/tree|fossil tree] command. + * On case-insensitive filesystems, store files using the filesystem's + preferred case rather than the case typed in by the user. + * Change the name "fossil cherry-pick" command to "fossil cherrypick", + which is more familiar to Git users. Retain the legacy name for + compatibility. + * Add new query parameters to the [/help/www/timeline|/timeline page]: + d2=, p2=, and dp2=. + * Add options to the [/help/tag|fossil tag] command that will list tag values. + * Add the -b|--brief option to the [/help/status|fossil status] command. + * Add ability to upload unversioned files via the [/help/www/uvlist|/uvlist page]. + * Add history search to the [/help/www/chat|/chat page]. + * Add Unix socket support to the [/help/server|server command]. + * On Windows, use the root certificates managed by the operating system + (requires OpenSSL 3.2.0 or greater). + * Take into account zero-width and double-width unicode characters when + formatting the command-line timeline. + * Update the built-in SQLite to version 3.47.0. Precompiled binaries are + linked against OpenSSL 3.4.0. + * Numerous minor fixes and additions. + + +<h2 id='v2_24'>Changes for version 2.24 (2024-04-23)</h2> + + * Apache change work-around → As part of a security fix, the Apache webserver + mod_cgi module has stopped relaying the Content-Length field of the HTTP + reply header from the CGI programs back to the client in cases where the + connection is to be closed and the client is able to read until end-of-file. + The HTTP and CGI specs allow for this, though it does seem rude. + Older versions of Fossil were depending on the Content-Length header field + being set. To work around the change to Apache, Fossil has + been enhanced to cope with a missing Content-Length in the reply header. See + [forum:/forumpost/12ac403fd29cfc89|forum thread 12ac403fd29cfc89]. + * [./customskin.md|Skin] enhancements: + <ul> + <li> Reworked the default skin to make everything more readable: larger + fonts, more whitespace, deeper indents to show hierarchy and to + offset command examples, etc. Adjusted colors slightly to bring + things into better accord with the WCAG accessibility guidelines. + This constitutes a <strong>breaking change</strong> for those with + custom skins; see [./customskin.md#version-2.24 | this section of + the docs] for migration advice. + <li> Add a new link added to the [/login] page that allows the user to + [/skins|select their preferred skin]. This preference is stored in + the [/fdscookie|fossil display_settings cookie]. + <li> The /setup_skin_admin page is simplified to let administrators easily + select one of the built-in skins as a default, or to specify a + custom skin. + </ul> + * If an "ssh:" sync fails in a way that suggests that the fossil executable + could not be found on the remote host, then retry after adding a PATH= + prefix to the command. This helps "ssh:" to "just work" when the server + is a Mac. + * Enhancements to the [/help/www/timeline|/timeline page]: + <ul> + <li> Add the x= query paramater + <li> Add the shortcut tl= and rl= query parameters + <li> Add support for from=,ft= and from=,bt= query parameter combinations + <li> Automatically highlight the endpoints for from=,to= queries. + <li> Add the to2=Z query parameter to augment from=X,to=Y so that the + path from X to Z is shown if Y cannot be found. + </ul> + * Moved the /museum/repo.fossil file referenced from the Dockerfile from + the ENTRYPOINT to the CMD part to allow use of --repolist mode. + * The [/uvlist] page now shows the hash algorithm used so that + viewers don't have to guess. The hash is shown in a fixed-width + font for a more visually pleasing display. + * If the [/help/autosync|autosync setting] contains keyword "all", + the automatic sync occurs against all defined remote repositories, not + just the default. + * Markdown formatter: improved handling of indented fenced code blocks + that contain blank lines. + * When doing a "[/help/add|fossil add]" on a system with case-insensitive + but case-preserving filenames (Mac and Windows) try to use the filename + case as it is known to the filesystem, not the case entered by the + user on the command-line. See + [forum:/forumpost/30d9c0d131610f53|forum thread 30d9c0d131610f53]. + * Fix problems with one-click unsubscribe on email notifications. + * Import the latest [/doc/trunk/www/pikchr.md|Pikchr] containing support + for "diamond" objects. + * Add ability to render committed Pikchr files to SVG via + <samp>/doc/…/foo.pikchr?popup</samp> URLs. + * Update Fossil's internal robot detection logic so that it correctly + identifies the new GoogleOther crawler as a robot. + +<h2 id='v2_23'>Changes for version 2.23 (2023-11-01)</h2> + + * Add ability to "close" forum threads, such that unprivileged users + may no longer respond to them. Only administrators can close + threads or respond to them by default, and the + [/help/forum-close-policy|forum-close-policy setting] can be + used to add that capability to moderators. + * Add the [/help/all|fossil all whatis] command. + * The [/help/status|fossil status] command and relevant UI pages now + correctly report files which were both renamed <b>and</b> edited as such. + * Show default value of settings that have a default in + [/help/help|fossil help SETTING] output. + * On timeline graphs, show closed check-ins using an X in the middle of the + node circle or box. + * New options for email notification: Get email only for the first + post in each new thread, and/or posts that are in reply to my posts. + * Fix a regression bug introduced in version 2.22 that caused FTS5 searches + to fail for terms containing non-ASCII characters. + * Improved defense-in-depth against malicious attack: + <ul> + <li> When an attempted SQL injection attack is detected, return + HTTP result code 418, which can signal the web server to sanction + the attacking IP address. + <li> Better defense against cross-site request forgery (CSRF) + attacks. + <li> Improvements to static analysis of source code (the codecheck1.c + file in the source tree). + </ul> + * Enhance the [/help/www/dir|treeview file listings] + ([/dir?type=tree&ci=trunk|example]) by displaying file sizes + and adding the option to sort by file size. + * The [/help/fts-config|fossil fts-config] command now shows how much + repository space is used by the full-text index. + * Changing a setting to an empty string is now the same as deleting the + setting, in most cases. There are a few exceptions, indicated by the + keep-empty flag on the setting definition. + * The [/help/branch|fossil branch list] command can now filter branches + that have/have not been merged into the current branch. + * Improvements to interactions with remote repositories over SSH: + <ul> + <li> Print the text of the SSH command that is run to do remote interaction, + for full disclosure to the operator. + <li> Add a PATH= argument to the [/help/ui|fossil ui remote:/] and + [/help/patch|fossil patch push/pull remote:...] commands so that + they work when the "remote" machine is a Mac and the "fossil" + executable is in the $HOME/bin directory. + </ul> + * Update built-in libraries SQLite, ZLib, Pikchr to their latest versions. + * Documentation enhancements and typo fixes. + + +<h2 id='v2_22'>Changes for version 2.22 (2023-05-31)</h2> + * Enhancements to the [/help/www/timeline|/timeline webpage]: <ol type="a"> + <li> Add the ft=TAG query parameter which in combination with d=Y + shows all descendants of Y up to TAG + <li> Enhance the s=PATTERN (search) query parameter so that forum post + text is also searched when the "vfx" query parameter is used + <li> Fix the u= (user) query parameter so that it works with a= and b= + <li> Add the oldestfirst query parameter to show the events in reverse order. + Useful in combination with y=f and vfs and perhaps also u= to show all + forum events in chronological order + <li> For the p=X and bt=Y query parameter combination, if Y is a tag that + identifies multiple check-ins, search backwards in time for Y beginning + at X + </ol> + * Administrators can select to skip sending notifications about new forum + posts. + * If the value N is negative in "--context N" or "-c N" to the various diff + commands, then treat it as infinite and show the complete file content. + * The stock OCI container no longer includes BusyBox, thus no longer + needs to start as root to chroot that power away. That in turn + frees us from needing to build and install the container as root, + since it no longer has to create a private <tt>/dev</tt> tree + inside the jail for Fossil's use. + * Add support for the trigram tokenizer for FTS5 search to enable + searching in Chinese. + * Comment lines (starting with a '#') are now supported inside + [./settings.wiki#versionable|versioned settings]. + * Default permissions for anonymous users in new repositories are + changed to "hz". + * The [/help/status|fossil status] command now detects when a + file used to be a symlink and has been replaced by a regular file. + (It previously checked for the inverse case only.) + * The [/help/empty-dirs|empty-dirs setting] now reuses the same + parser as the *-glob settings instead of its prior idiosyncratic + parser, allowing quoted whitespace in patterns. + * Enhancements to the [/help/www/reports|/reports webpage]: + <ol type="a"> + <li> The by-week, by-month, and by-year options now show an estimated + size of the current week, month, or year as a dashed box. + <li> New sub-categories "Merge Check-ins" and "Non-Merge Check-ins". + </ol> + +<h2 id='v2_21'>Changes for version 2.21 (2023-02-25)</h2> + * Users can request a password reset. This feature is disabled by default. + Use the new [/help/self-pw-reset|self-pw-reset property] to enable it. + New web pages [/help/www/resetpw|/resetpw] and + [/help/www/reqpwreset|/reqpwreset] added. + * Add the [/help/repack|fossil repack] command (together with + [/help/all|fossil all repack]) as a convenient way to optimize the + size of one or all of the repositories on a system. + * Add the ability to put text descriptions on ticket report formats. + * Upgrade the test-find-pivot command to the [/help/merge-base|merge-base command]. + * The [/help/www/chat|/chat page] can now embed fossil-rendered + views of wiki/markdown/pikchr file attachments with the caveat that such + embedding happens in an iframe and thus does not inherit styles and such + from the containing browser window. + * The [/help/all|fossil all remote] subcommand added to "fossil all". + * Passwords for remembered remote repositories are now stored as irreversible + hashes rather than obscured clear-text, for improved security. + * Add the "nossl" and "nocompress" options to CGI. + * Update search infrastructure from FTS4 to FTS5. + * Add the [/help/www/deltachain|/deltachain] page for debugging purposes. + * Writes to the database are disabled by default if the HTTP request + does not come from the same origin. This enhancement is a defense in depth + measure only; it does not address any known vulnerabilities. + * Improvements to automatic detection and mitigation of attacks from + malicious robots. + +<h2 id='v2_20'>Changes for version 2.20 (2022-11-16)</h2> + * Added the [/help/chat-timeline-user|chat-timeline-user setting]. If + it is not an empty string, then any changes that would appear on the timeline + are announced in [./chat.md|the chat room]. + * The /unsubscribe page now requests confirmation. [./alerts.md|Email notifications] + now contain only an "Unsubscribe" link, and not a link to subscription management. + * Added the "[/help/branch|fossil branch lsh]" subcommand to list the + most recently modified branches. + * More elements of the /info page are now inside of an accordion. + * Replace the <tt>--dryrun</tt> flag with <tt>--dry-run</tt> in all + commands which still used the former name, for consistency. + * Rebuilt [/file/Dockerfile | the stock Dockerfile] to create a "from scratch" + Busybox based container image via an Alpine Linux intermediary + * Added [/doc/trunk/www/containers.md | a new document] describing how to + customize, use, and run that container. + * Added "by hour of day" report to [/reports?view=byhour|the /reports page]. + * Improved correctness, usability, and efficiency for the case + [/timeline?r=deltify-tkt-blobs|when values in a TICKET's column + tend to be long and volatile]. + * Fixed a bug [/info/ea5afad31f478396 | introduced in 2.17] that + prevented <tt>clone --unversioned</tt> from completing the + retrieval of UV files from the remote repo. While fixing that, enabled + UV tracing output with <tt>clone --unversioned --verbose</tt>, making it + consonant with <tt>uv sync --verbose</tt>. + +<h2 id='v2_19'>Changes for version 2.19 (2022-07-21)</h2> + * On file listing pages, sort filenames using the "uintnocase" collating + sequence, so that filenames that contains embedded integers sort in + numeric order even if they contain a different number of digits. + (Example: "fossil_80_..." comes before "fossil_100.png" in the + [/dir?ci=92fd091703a28c07&name=skins/blitz|/skins/blitz] directory listing.) + * Enhancements to the graph layout algorithm design to improve readability + and promote better situational awareness. + * Performance enhancement for the + [./checkin_names.wiki#root|"root:BRANCHNAME" style of tag], + accomplished using a Common Table Expression in the underlying SQL. + * Sort tag listings (command line and webpage) by taking numbers into + consideration so as to cater for tags that follow semantic versioning. + * On the wiki listings, omit by default wiki pages that are associated with + check-ins and branches. + * Add the new "[/help/describe|fossil describe]" command. + * Markdown subsystem extended with [../src/markdown.md#ftnts|footnotes support]. + See corresponding [../test/markdown-test3.md|test cases], + [/wiki?name=branch/markdown-footnotes#il|known limitations] and + [forum:/forumthread/ee1f1597e46ec07a|discussion]. + * Add the new special name "start:BRANCH" to refer to the first check-in of + the branch. + * Support [/wiki?name=branch/generated-tkt-mimetype&p|generated "mimetype"] + columns in the <var>TICKET</var> and <var>TICKETCHNG</var> tables. + * Fix [/timeline?r=fix_remote_url_overwrite_with_proxy|remote-url-overwrite] + bug where remote-url is overwritten by the proxy setting during sync + operation. Also require explicit "system" proxy setting to use + "http_proxy" environment variable. + * Reimplemented the [/pikchrshow] app to use a WebAssembly build of + pikchr so that it can render pikchrs on the client instead of requiring + a server round-trip. + * Add the [/help/email-listid|email-listid setting]. If set, it is + used as the List-ID header for all outbound notification emails. + * Add the "--branch" option to the "[/help/timeline|timeline]" command + to restrict the displayed items to a specific branch. + * Add the "--versions" option to "[/help/diff|fossil diff]" + to display details about the compared versions into the patch header. + * Numerous other minor enhancements. + +<h2 id='v2_18'>Changes for version 2.18 (2022-02-23)</h2> + * Added support for [./ssl-server.md|SSL/TLS server mode] for commands + like "[/help/server|fossil server]" and "[/help/http|fossil http]" + * The new [/help/cherry-pick|cherry-pick command] is an alias for + [/help/merge|merge --cherrypick]. + * Add new setting "[/help/large-file-size|large-file-size]". If the size + of any file in a commit exceeds this size, a warning is issued. + * Query parameter "year=YYYY" is now accepted by [/help/www/timeline|/timeline]. + * The [/help/tar|tar] and [/help/zip|zip commands] no longer + sterilize the manifest file. + * Further improvement to diff alignment in cases that involve both + edits and indentation changes. + * [/doc/trunk/www/chat.md|Chat] improvements:<ul> + <li> [/help/www/chat|The /chat page] input options have been reworked + again for better cross-browser portability. + <li> When sending a [/help/www/chat|/chat] message fails, it is no longer + immediately lost and sending may optionally be retried. + <li> [/help/www/chat|/chat] can now optionally embed attachments of certain + types directly into message bodies via an iframe. + <li> Add the "--as FILENAME" option to the "[/help/chat|fossil chat send]" + command. + <li> Added the "[/help/chat|fossil chat pull]" command, available to + administrators only, for backing up the chat conversation. + </ul> + * Promote the test-detach command into the [/help/detach|detach command]. + * For "[/help/pull|fossil pull]" with the --from-parent-project option, + if no URL is specified then use the last URL from the most recent prior + "fossil pull --from-parent-project". + * Add options --project-name and --project-desc to the + "[/help/init|fossil init]" command. + * The [/help/www/ext|/ext page] generates the SERVER_SOFTWARE environment + variable for clients. + * Fix the REQUEST_URI [/doc/trunk/www/aboutcgi.wiki#cgivar|CGI variable] such + that it includes the query string. This is how most other systems understand + REQUEST_URI. + * Added the --transport-command option to [/help/sync|fossil sync] + and similar. + +<h2 id='v2_17'>Changes for version 2.17 (2021-10-09)</h2> + + * Major improvements to the "diff" subsystem, including: <ul> + <li> Added new [/help/diff|formatting options]: --by, -b, --webpage, --json, --tcl. + <li> Partial-line matching for unified diffs + <li> Better partial-line matching for side-by-side diffs + <li> Buttons on web-based diffs to show more context + <li> Performance improvements + </ul> + * The --branchcolor option on [/help/commit|fossil commit] and + [/help/amend|fossil amend] can now take the value "auto" to + force Fossil to use its built-in automatic color choosing algorithm. + * Fossil now [./concepts.wiki#workflow|autosyncs] prior to running + [/help/open|fossil open]. + * Add the [/help/ticket-default-report|ticket-default-report setting], + which if set to the title of a ticket report causes that ticket report + to be displayed below the search box in the /ticket page. + * The "nc" query parameter to the [/help/www/timeline|/timeline] page + causes all graph coloring to be omitted. + * Improvements and bug fixes to the new "[/help/ui|fossil ui REMOTE]" + feature so that it works better on a wider variety of platforms. + * In [/help/www/wikiedit|/wikiedit], show the list of attachments for + the current page and list URLs suitable for pasting them into the page. + * Add the --no-http-compression option to [/help/sync|fossil sync] + and similar. + * Print total payload bytes on a [/help/sync|fossil sync] when using + the --verbose option. + * Add the <tt>close</tt>, <tt>reopen</tt>, <tt>hide</tt>, and + </tt>unhide</tt> subcommands to [/help/branch|the branch command]. + * The "-p" option to [/help/branch|fossil branch list] shows only + private branches. + * The [/md_rules|Markdown formatter] now interprets the content of + block HTML markup (such as <table>) in most cases. Only content + of <pre> and <script> is passed through verbatim. + * The [/help/wiki|wiki list command] no longer lists "deleted" + pages by default. Use the new <tt>--all</tt> option to include deleted + pages in the output. + * The [/help/all|fossil all git status] command only shows reports for + the subset of repositories that have a configured Git export. + * The [/help/www/chat|/chat] configuration was reimplemented and + provides new options, including the ability for a repository + administrator to + [./chat.md#notifications|extend the selection of notification sounds] + using unversioned files. + * Chat now uses fossil's full complement of markdown features, + instead of the prior small subset of markup it previously supported. + This retroactively applies to all chat messages, as they are + markdown-processed when they are sent instead of when they + are saved. + * Added a chat message preview mode so messages can be previewed + before being sent. Similarly, added a per-message ability to view + the raw un-parsed message text. + * The hotkey to activate preview mode in [/help/www/wikiedit|/wikiedit], + [/help/www/fileedit|/fileedit], and [/help/www/pikchrshow|/pikchrshow] + was changed from ctrl-enter to shift-enter in order to align with + [/help/www/chat|/chat]'s new preview feature and related future + changes. + +<h2 id='v2_16'>Changes for Version 2.16 (2021-07-02)</h2> + * <b>Security:</b> Fix the client-side TLS so that it verifies that the + server hostname matches its certificate. + * The default "ssh" command on Windows is changed to "ssh" instead of the + legacy "plink", as ssh is now generally available on Windows systems. + Installations that still need to use the legacy "plink" can make that + happen by running: '<tt>fossil set ssh-command "plink -ssh" --global</tt>'. + * Added the [./patchcmd.md|fossil patch] command. + * The [/help/ui|fossil ui] command is enhanced in multiple ways:<ol> + <li> The REPOSITORY argument can be the name of a check-out directory. + <li> If the REPOSITORY argument is prefixed by "HOST:" or "USER@HOST:" + then the ui is run on the remote machine and tunnelled back to the local + machine using ssh. (The latest version of fossil must be installed on + both the local and the remote for this to work correctly.) + <li> The new --nobrowser and --fossilcmd options is provided. + </ol> + * The [/brlist|/brlist web page] allows the user to + select multiple branches to be displayed together in a single + timeline. + * The [./forum.wiki|Forum] provides a hyperlink on the author of each + post that goes to a timeline of recent posts by that same author. + * Added the "[/help/bisect|fossil bisect run]" command for improved + automation of bisects. + * The [/help/merge|fossil merge] command now does a better job merging + branches where files have been renamed between the current branch and the + branch being merged. + * The [/help/open|fossil open] command allows the repository file + to be inside the working directory without requiring the --force flag. + * The [/help/www/wikiedit|/wikiedit] and [/help/www/wikinew|/wikinew] + pages now default to markdown format. + * The [/help/www/login|/login] page now links to a user's forum post + timeline if the repository has forum posts. + * Tags may now be propagated for forum posts, wiki pages, and technotes. + The [/help/tag|tag command] can now manipulate and list such tags. + * [./caps/login-groups.md|Login-Groups] are now shown on the repository + list of the "[/help/all|fossil all ui]" command. + * Administrators can configure [./alerts.md|email alerts] to expire + a specific number of days (ex: 365) after the last user contact with + the Fossil server. This prevents alert emails being sent to + abandoned email accounts forever. + * SQL that defines [/tktsetup_tab|database objects for tickets] now + [/timeline?c=c717f1ef9a1a4c91|can DROP] a VIEW or an INDEX provided + that its name starts with '<code>ticket</code>' or '<code>fx_</code>'. + * Update the built-in SQLite to version 3.36.0. + * Numerous other minor enhancements. + +<h2 id='v2_15'>Changes for Version 2.15 (2021-03-26) and Patch 2.15.1 on (2021-04-07) + and 2.15.2 on (2021-06-15)</h2> + * <b>Patch 2.15.2:</b> Fix the client-side TLS so that it verifies that the + server hostname matches its certificate. <b>Upgrading to + the patch is recommended.</b> + * <b>Patch 2.15.1:</b> Fix a data exfiltration bug in the server. <b>Upgrading to + the patch is recommended.</b> + * The [./defcsp.md|default CSP] has been relaxed slightly to allow + images to be loaded from any URL. All other resources are still + locked down by default. + * The built-in skins all use the "[/help/mainmenu|mainmenu]" + setting to determine the content of the main menu. + The ability to edit the + "mainmenu" setting is added on the /Admin/Configuration page. + * The hamburger menu is now available on most of the built-in skins. + * Any built-in skin named "X" can be used instead of the standard + repository skin by adding the URL parameter <tt>skin=X</tt> to the + request. The selection is persisted using the display + preferences cookie unless the "once" query parameter is also + included. The [/skins] page may be used to select a skin. + * The [/cookies] page now gives the user an opportunity to delete + individual cookies. And the /cookies page is linked from the + /sitemap, so that it appears in hamburger menus. + * The [/sitemap] extensions are now specified by a single new + "[/help/sitemap-extra|sitemap-extra setting]", + rather than a cluster of various + "sitemap-*" settings. The older settings are no longer used. + <b>This change might require minor server configuration + adjustments on servers that use /sitemap extensions.</b> + The /Admin/Configuration page provides the ability to edit + the new "sitemap-extra" setting. + * Added the "--ckout-alias NAME" option to + [/help/ui|fossil ui], [/help/server|fossil server], and + [/help/http|fossil http]. This option causes Fossil to + understand URIs of the form "/doc/NAME/..." as if they were + "[/help/www/doc|/doc/ckout/...]", to facilitate testing of + [./embeddeddoc.wiki|embedded documentation] changes prior to + check-in. + * For diff web pages, if the diff type (unified versus side-by-side) + is not specified by a query parameter, and if the + "[/help/preferred-diff-type|preferred-diff-type]" + setting is omitted or less than 1, then select the diff type based + on a guess of whether or not the request is coming from a mobile + device. Mobile gets unified and desktop gets side-by-side. + * The various pages which show diffs now have toggles to show/hide + individual diffs. + * Add the "[/help/preferred-diff-type|preferred-diff-type]" + setting to allow an admin to force a default diff type. + * The "pikchr-background" setting is now available in + "detail.txt" skin files, for better control of Pikchr + colors in inverted color schemes. + * Add the <tt>--list</tt> option to the + [/help/tarball|tarball], + [/help/zip|zip], and [/help/sqlar|sqlar] + commands. + * The javascript used to implement the hamburger menu on the + default built-in skin has been made generic so that it is usable + by a variety of skins, and promoted to an ordinary built-in + javascript file. + * New TH1 commands: + "[/doc/trunk/www/th1.md#bireqjs|builtin_request_js]", + "[/doc/trunk/www/th1.md#capexpr|capexpr]", + "foreach", "lappend", and "string match" + * The [/help/leaves|leaves command] now shows the branch point + of each leaf. + * The [/help/add|fossil add] command refuses to add files whose + names are reserved by Windows (ex: "aux") unless the --allow-reserved + option is included. This helps prevent Unix users from accidentally + creating check-ins that are unreadable by Windows users. + * Add the "re=" query parameter to the [/help/www/dir|/dir] webpage, + for symmetry with the [/help/www/tree|/tree] page. + * Update the built-in SQLite to version 3.35.0. + * The ./configure script now has the --print-minimum-sqlite-version option + that prints the minimum SQLite version required by the current version + of Fossil. This might be used by integrators who insist on building + Fossil to link against the system SQLite library rather than the + built-in copy of SQLite, to verify that their system SQLite library + is recent enough. + * Webpage that shows [/help/www/whistory|history of a wiki page] + gained client-side UI to help with comparison between two arbitrary + versions of a wiki (by the means of anchoring a "baseline" version) + and the ability to squeeze several sequential edits made by the same + user into a single "recycled" row (the latest edit in that sequence). + +<h2 id='v2_14'>Changes for Version 2.14 (2021-01-20) and Patch 2.14.1 on (2021-04-07) + and 2.14.2 on (2021-06-15)</h2> + * <b>Patch 2.14.2:</b> Fix the client-side TLS so that it verifies that the + server hostname matches its certificate. <b>Upgrading to + the patch is recommended.</b> + * <b>Patch 2.14.1:</b> Fix a data exfiltration bug in the server. + <b>Upgrading to the patch is recommended.</b> + * <b>Schema Update Notice #1:</b> + This release drops a trigger from the database schema (replacing + it with a TEMP trigger that is created as needed). This + change happens automatically the first time you + add content to a repository using Fossil 2.14 or later. No + action is needed on your part. However, if you upgrade to + version 2.14 and then later downgrade or otherwise use an earlier + version of Fossil, the email notification mechanism may fail + to send out notifications for some events, due to the missing + trigger. If you want to + permanently downgrade an installation, then you should run + "[/help/rebuild|fossil rebuild]" after the downgrade + to get email notifications working again. If you are not using + email notification, then the schema change will not affect you in + any way. + * <b>Schema Update Notice #2:</b> + This release changes how the descriptions of wiki edits are stored + in the EVENT table, for improved display on timelines. You must + run "[/help/rebuild|fossil rebuild]" to take advantage of + this enhancement. Everything will still work without + "fossil rebuild", except you will get goofy descriptions of + wiki updates in the timeline. + * Add support for [./chat.md|Fossil chat]. + * The "[/help/clone|fossil clone]" command is enhanced so that + if the repository filename is omitted, an appropriate name is derived + from the remote URL and the newly cloned repo is opened. This makes + the clone command work more like Git, thus making it easier for + people transitioning from Git. + * Added the --mainbranch option to the [/help/git|fossil git export] + command. + * Added the --format option to the + "[/help/timeline|fossil timeline]" command. + * Enhance the --numstat option on the + "[/help/diff|fossil diff]" command so that it shows a total + number of lines added and deleted and total number of files + modified. + * Add the "contact" sub-command to [/help/user|fossil user]. + * Added commands "[/help/all|fossil all git export]" and + "[/help/all|fossil all git status]". + * Added the "df=CHECKIN" query parameter to the + [/help/www/timeline|/timeline page]. + * Improvements to the "[/sitemap]" page. Add subpages + [/sitemap-timeline] and [/sitemap-test]. + * Better text position in cylinder objects of Pikchr diagrams. + * New "details.txt" settings available to custom skins to better control + the rendering of Pikchr diagrams: + <ul> + <li> pikchr-foreground + <li> pikchr-scale + <li> pikchr-fontscale + </ul> + * Allow the use of SQL functions inside the ticket table definition + for custom ticket configurations. + * The built-in SQLite is updated to version 3.35.0 alpha containing + performance optimizations, especially performance associated with + startup, and minor improvements to the CLI. + * Performance optimizations to Fossil itself. + * Countless improvements and enhancements to the documentation + +<h2 id='v2_13'>Changes for Version 2.13 (2020-11-01)</h2> + + * Added support for [./interwiki.md|interwiki links]. + * Enable <del> and <ins> markup in wiki. + * Improvements to the Forum threading display. + * Added support for embedding [./pikchr.md|pikchr] + markup in markdown and fossil-wiki content. + * The new "[/help/pikchr|pikchr]" command can render + pikchr scripts, optionally pre-processed with + [/doc/trunk/www/th1.md|TH1] blocks and variables exactly like + site skins are. + * The new [/help/www/pikchrshow|pikchrshow] page provides an + editor and previewer for pikchr markup. + * In [/help/www/wikiedit|/wikiedit] and + [/help/www/fileedit|/fileedit], Ctrl-Enter can now be used + initiate a preview and to toggle between the editor and preview + tabs. + * The <tt>/artifact</tt> and <tt>/file</tt> views, when in + line-number mode, now support interactive selection of a range + of lines to hyperlink to. + * Enhance the [/help/www/finfo|/finfo] webpage so that when query + parameters identify both a filename and a checkin, the resulting + graph tracks the identified file across renames. + * The built-in SQLite is updated to an alpha of version 3.34.0, and + the minimum SQLite version is increased to 3.34.0 because the + /finfo change in the previous bullet depends on enhancements to + recursive common table expressions that are only available in + SQLite 3.34.0 and later. + * Countless other minor refinements and documentation improvements. + +<h2 id='v2_12'>Changes for Version 2.12.1 (2020-08-20)</h2> + + * (2.12.1): Fix client-side vulnerabilities discovered by Max Justicz. + * Security fix in the "[/help/git|fossil git export]" command. + The same fix is also backported to version 2.10.1 and 2.11.1. + New "safety-net" features were added to prevent similar problems + in the future. + * Enhancements to the graph display for cases when there are + many cherry-pick merges into a single check-in. + [/timeline?f=2d75e87b760c0a9|Example] + * Enhance the [/help/open|fossil open] command with the new + --workdir option and the ability to accept a URL as the repository + name, causing the remote repository to be cloned automatically. + Do not allow "fossil open" to open in a non-empty working directory + unless the --keep option or the new --force option is used. + * Enhance the markdown formatter to more closely follow the + [https://spec.commonmark.org/0.29/#emphasis-and-strong-emphasis|CommonMark specification] + with regard to text highlighting. + Underscores in the middle of identifiers (ex: fossil_printf()) + no longer need to be escaped. + * The markdown-to-html translator can prevent unsafe HTML + (for example: <script>) on user-contributed pages like forum and + tickets and wiki. The admin can adjust this behavior using + the [/help/safe-html|safe-html setting] on the Admin/Wiki page. + The default is to disallow unsafe HTML everywhere. + [https://fossil-scm.org/forum/forumpost/3714e6568f|Example]. + * Added the "collapse" and "expand" capability for long forum posts. + [https://fossil-scm.org/forum/forumpost/9297029862|Example] + * The "[/help/remote-url|fossil remote]" command now has options for + specifying multiple persistent remotes with symbolic names. Currently + only one remote can be used at a time, but that might change in the + future. + * Add the "Remember me?" checkbox on the login page. Use a session + cookie for the login if it is not checked. + * Added the experimental "[/help/hook|fossil hook]" command for + managing "hook scripts" that run before checkin or after a push. + * Enhance the [/help/revert|fossil revert] command so that it + is able to revert all files beneath a directory. + * Add the [/help/bisect|fossil bisect skip] command. + * Add the [/help/backup|fossil backup] command. + * Enhance [/help/bisect|fossil bisect ui] so that it shows all unchecked + check-ins in between the innermost "good" and "bad" check-ins. + * Added the <tt>--reset</tt> flag to the "[/help/add|fossil add]", + "[/help/rm|fossil rm]", and + "[/help/addremove|fossil addremove]" commands. + * Added the "<tt>--min</tt> <i>N</i>" and "<tt>--logfile</tt> <i>FILENAME</i>" + flags to the [/help/backoffice|backoffice] command, as well as other + enhancements to make the backoffice command a viable replacement for + automatic backoffice. Other incremental backoffice improvements. + * Added the [/help/www/fileedit|/fileedit page], which allows + editing of text files online. Requires explicit activation by + a setup user. + * Translate built-in help text into HTML for display on web pages. + [/help/help|Example]. + * On the [/help/www/timeline|/timeline] webpage, the combination + of query parameters "p=CHECKIN" and "bt=ANCESTOR" draws all + ancestors of CHECKIN going back to ANCESTOR. For example, + [/timeline?p=202006271506&bt=version-2.11] shows all ancestors + of the checkin that occurred on 2020-06-27 15:06 going back to + the 2.11 release. + * Update the built-in SQLite so that the + "[/help/sql|fossil sql]" command supports new output + modes ".mode box" and ".mode json". + * Add the "<tt>obscure()</tt>" SQL function to the + "[/help/sql|fossil sql]" command. + * Added virtual tables "<tt>helptext</tt>" and "<tt>builtin</tt>" to + the "[/help/sql|fossil sql]" command, providing access to the + dispatch table including all help text, and the builtin data files, + respectively. + * [./delta_format.wiki|Delta compression] is now applied to forum edits. + * The [/help/www/wikiedit|wiki editor] has been modernized and is + now Ajax-based. The WYSIWYG editing option for Fossil-format wiki + pages was removed. (Please let us know, via the site's Forum menu, + if that removal unduly impacts you.) This also changes the semantics + of the wiki "Sandbox": that pseudo-page may be freely edited but + no longer saved via the UI (the [/help/wiki|wiki CLI command] + can, though). + * The [/help/allow-symlinks|allow-symlinks setting] no longer + syncs. It must be activated individually on any clones which require + symlinks. + * Countless documentation enhancements. + +<h2 id='v2_11'>Changes for Version 2.11 (2020-05-25)</h2> + + * (2.11.2): Backport security fixes from 2.12.1 + * (2.11.1): Backport security fix for the "fossil git export" command. + * Support [/md_rules|Markdown] in the default ticket configuration. + * Timestamp strings in [./checkin_names.wiki|object names] + can now omit punctation. So, for example, "202004181942" and + "2020-04-18 19:42" mean the same thing. + * Enhance backlink processing so that it works with Markdown-formatted + tickets and so that it works for wiki pages. + Ticket [a3572c6a5b47cd5a]. + <ul><li> "[/help/rebuild|fossil rebuild]" is needed to + take full advantage of this fix. Fossil will continue + to work without the rebuild, but the new backlinks will be missing.</ul> + * The algorithm for finding the + [./tech_overview.wiki#configloc|location of the configuration database] + is enhanced to be XDG-compliant. + * Add a hide/show feature to + [./wikitheory.wiki#assocwiki|associated wiki] display on + check-in and branch information pages. + * Enhance the "[/help/info|fossil info]" command so that it + works with no arguments even if not within an open check-out. + * Many improvements to the forum and especially email notification + of forum posts, in response to community feedback after switching + SQLite support from a mailing list over to the forum. + * Minimum length of a self-registered user ID increased from 3 to 6 + characters. + * When the "vfx" query parameter is used on the + "[/help/www/timeline|/timeline]" page, it causes the complete + text of forum posts to be displayed. + * Rework the "[/help/grep|fossil grep]" command to be more useful. + * Expose the [/help/redirect-to-https|redirect-to-https] + setting to the [/help/settings|settings] command. + * Improve support for CGI on IIS web servers. + * The [./serverext.wiki|/ext page] can now render index files, + in the same way as the embedded docs. + * Most commands now support the Unix-conventional "<tt>--</tt>" + flag to treat all following arguments as filenames + instead of flags. + * Added the [/help/mimetypes|mimetypes config setting] + (versionable) to enable mimetype overrides and custom definitions. + * Add an option on the /Admin/Timeline setup page to set a default + timeline style other than "Modern". + * In [./embeddeddoc.wiki|embedded documentation], hyperlink URLs + of the form "/doc/$CURRENT/..." the "$CURRENT" text is translated + into the check-in hash for the document currently being viewed. + * Added the [/help/www/phantoms|/phantoms] webpage that shows all + phantom artifacts. + * Enhancements to phantom processing to try to reduce + bandwidth-using chatter about phantoms on the sync protocol. + * Security: Fossil now assumes that the schema of every + database it opens has been tampered with by an adversary and takes + extra precautions to ensure that such tampering is harmless. + * Security: Fossil now puts the Content-Security-Policy in the + HTTP reply header, in addition to also leaving it in the + HTML <head> section, so that it is always available, even + if a custom skin overrides the HTML <head> and omits + the CSP in the process. + * Output of the [/help/diff|fossil diff -y] command automatically + adjusts according to the terminal width. + * The Content-Security-Policy is now set using the + [/help/default-csp|default-csp setting]. + * Merge conflicts caused via the [/help/merge|merge] and + [/help/update|update] commands no longer leave temporary + files behind unless the new <tt>--keep-merge-files</tt> flag + is used. + * The [/help/www/artifact_stats|/artifact_stats page] is now accessible + to all users if the new "artifact_stats_enable" setting is turned + on. There is a new checkbox under the /Admin/Access menu to turn + that capability on and off. + * Add the [/help/tls-config|fossil tls-config] command for viewing + the TLS configuration and the list of SSL Cert exceptions. + * Captchas all include a button to read the captcha using an audio + file, so that they can be completed by the visually impaired. + * Stop using the IP address as part of the login cookie. + * Bug fix: fix the SSL cert validation logic so that if an exception + is allowed for particular site, the exception expires as soon as the + cert changes values. + * Bug fix: the FTS search into for forum posts is now kept up-to-date + correctly. + * Bug fix: the "fossil git export" command is now working on Windows + * Bug fix: display Technote items on the timeline correctly + * Bug fix: fix the capability summary matrix of the Security Audit + page so that it does not add "anonymous" capabilities to the + "nobody" user. + * Update internal Unicode character tables, used in regular expression + handling, from version 12.1 to 13. + * Many documentation enhancements. + * Many minor enhancements to existing features. + +<h2 id='v2_10'>Changes for Version 2.10 (2019-10-04)</h2> + + * (2.10.2): backport security fixes from 2.12.1 + * (2.10.1): backport security fix for the "fossil git export" command. + * Added support for [./serverext.wiki|CGI-based Server Extensions]. + * Added the [/help/repolist-skin|repolist-skin] setting used to + add style to repository list pages. + * Enhance the hierarchical display of Forum threads to do less + indentation and to provide links back to the previous message + in the thread. Provide sequential numbers for all messages in + a forum thread. + * Add support for fenced code blocks and improved hyperlink + processing to the [/md_rules|markdown formatter]. + * Add support for hyperlinks to wiki pages in the + [/md_rules|markdown formatter]. + * Enhance the [/help/www/stat|/stat] page so that it gives the + option to show a breakdown of forum posts. + * The special check-in name "merge-in:BRANCH" means the source of + the most recent merge-in from the parent branch of BRANCH. + * Add hyperlinks to branch-diffs on the /info page and from + timelines of a branch. + * Add graphical context on the [/help/www/vdiff|/vdiff] page. + * Uppercase query parameters, POST parameters, and cookie names are + converted to all lowercase and entered into the parameter set, + instead of being discarded. + * Change the default [./hashpolicy.wiki|hash policy] to SHA3. + * Timeout [./server/any/cgi.md|CGI requests] after 300 seconds, or + some other value set by the + [./cgi.wiki#timeout|"timeout:" property] in the CGI script. + * The check-in lock interval is reduced from 24 hours to 60 seconds, + though the interval is now configurable using a setting. + An additional check for conflicts is added after interactive + check-in comment entry, to compensate for the reduced lock interval. + * Performance optimizations. + * Many documentation improvements. + +<h2 id='v2_9'>Changes for Version 2.9 (2019-07-13)</h2> + + * Added the [/help/git|fossil git export] command and instructions + for [./mirrortogithub.md|creating a GitHub mirror of a Fossil project]. + * Improved handling of relative hyperlinks on the + [/help/www/artifact|/artifact] pages for wiki. For example, + hyperlinks and the lizard <img> now work correctly + for both [/artifact/2ff24ab0887cf522] and + [/doc/0d7ac90d575004c2415/www/index.wiki]. + * Enhancements to the timeline graph layout, to show more information + with less clutter. + * Added tool-tips to the /timeline graph. On by default but can be + disabled by setting the "Tooltip dwell time" to 0 in the timeline + configuration. + * Copy buttons added to various check-in hash and branch name links. + * Double-clicking on a /timeline graph node now jumps to the /info page + for the check-in. So, repurpose the timestamp hyperlink to show all + activity around that check-in in time. + * Added the [/help/touch|fossil touch] command, and the --setmtime + option on the [/help/open|fossil open] and + [/help/update|fossil update] commands. + * Many documentation enhancements. + * For the "[/help/update|fossil update]" and + "[/help/checkout|fossil checkout]" commands, if a + managed file is removed because it is no longer part of the target + check-in and the directory containing the file is empty after the + file is removed and the directory is not the current working + directory and is not on the [/help/empty-dirs|empty-dirs] + list, then also remove the directory. + * Update internal Unicode character tables, used in regular expression + handling, from version 11.0 to 12.1. + * In "[/help/regexp|fossil regexp]", "[/help/grep|fossil grep]" + and the TH1 "regexp" command, the -nocase option now removes multiple + diacritics from the same character (derived from SQLite's + remove_diacritics=2) + * Added the [/help/www/secureraw|/secureraw] page that requires the + complete SHA1 or SHA3 hash, not just a prefix, before it will deliver + content. + * Accept purely numeric ISO8601 date/time strings as long as they + do not conflict with a hash. Example: "20190510134217" instead of + "2019-05-10 13:42:17". This helps keep URLs shorter and less + complicated + * Support both "1)" and "1." for numbered lists in markdown, as + commonmark does. + * The sync and clone HTTP requests omit the extra /xfer path element + from the end of the request URI. All servers since 2010 know that + the HTTP request is for a sync or clone from the mimetype so the + extra path element is not needed. + * If an automatic sync gets a permanent redirect request, then update + the saved remote URL to the new address. + * Temporary filenames (for example used for external "diff" commands) + try to preserve the suffix of the original file. + * Added the [/help/www/thisdayinhistory|/thisdayinhistory] web page. + * Enhanced parsing of [/help/www/timeline|/timeline] query parameters + "ymd=", "ym=", and "yw=". All arguments are option (in which case they + default to the current time) and all accept ISO8601 date/times without + punctuation. + * Automatically disapprove pending moderation requests for a user when + that user is deleted. This helps in dealing with spam-bots. + * Improvements to the "Capability Summary" section in the + [/help/www/secaudit0|Security Audit] web-page. + * Use new "ci-lock" and "ci-lock-failed" pragmas in the + [./sync.wiki|sync protocol] to try to prevent accident forks + caused by concurrent commits when operating in auto-sync mode. + * Fix a bug ([https://fossil-scm.org/forum/forumpost/c51b9a1169|details]) + that can cause repository databases to be overwritten with debugging + output, thus corrupting the repository. This is only a factor when + CGI debugging is enabled, and even then is a rare occurrence, but it is + obviously an important fix. + +<h2 id='v2_8'>Changes for Version 2.8 (2019-02-20)</h2> + + * Show cherry-pick merges as dotted lines on the timeline graph. + → The "fossil rebuild" command must be run to create and + populate the new "cherrypick" table in the repository in order + for this feature to operate. + * Add the ability to associate branches, check-ins, and tags with + specially-named Wiki pages. This gives the ability to better + document branches and tags, and provide more documentation on + check-ins beyond the check-in comment. The associated Wiki is + automatically displayed on /info pages for check-ins, and on + /timeline?r=BRANCH and /timeline?t=TAG pages for branches and + tags. This feature is on by default, but can be disabled in on + the Admin/Wiki page. + * Enhance the repository list page (shown for example by + "fossil all ui") so that it shows the name and last check-in + time for each project. The implementation of the repository + list page is now broken out into a separate source file (repolist.c). + * Allow users with Forum Supervisor permission ('6') to add Forum + Write Trusted permission ('4') to users as they are approving a + forum post by that user. + * When running a bisect, report the number of check-ins still in + the search range and the estimated number of bisect steps remaining. + Do this at each step of the bisect. + * Provide a permanent link to a bisect timeline using the bid= query + parameter. + * Make the chronological forum display feature available to all users, + and make it the default format on mobile devices. + * Break out Wiki setup into a separate /setup_wiki page, accessible + on the standard menus through Admin/Wiki. + * Add "Next" and "Previous" buttons on the /wdiff page, allowing + the user to step through the versions of a wiki page. + * Improve the display of the /whistory page. + * Omit the "HH:MM" timestamps on timeline graphs on narrow-screen + devices, to improve horizontal space uses. This helps make Fossil + more mobile-friendly. + * Enhance /wcontent to show a sortable list of Wiki pages together + with the number of revisions and the most recent change time for + each page. + * Hyperlinks to Wiki pages on the /timeline go to the specific + version of the Wiki page named in the timeline, not to the latest + version. + * Enhancements to the "amend", "tag", and "reparent" commands, including + adding options --override-date, --override-user, and --dry-run. + * Add the global --comment-format command-line option and the + comment-format setting to control the display of the command-line + timeline. + * Change the "fossil reparent" command so that it only works from + within an active checkout. + * On the /setup_ucap_list, show administrators how many users have + each capability. The counts are a hyperlink to the /setup_ulist + page showing the subset of users that have that capability. + * Provide the ability to redirect all HTTP pages to HTTPS. Formerly + one could cause this to occur for the /login page only. That option + still exists, but the redirect can now also be done for all pages. + * "Compress" the built-in javascript by omitting comments and + leading and trailing whitespace. + * Detect when the repository used by a checkout is swapped out for + a clone that uses different RID values, and make appropriate adjustments + to the checkout database to avoid any problems. + * Add the backoffice-disable setting to completely disable the + backoffice feature. + * Update the built-in SQLite to version 3.27.1. + * Various other small enhancements to webpages and documentation. + + +<h2 id='v2_7'>Changes for Version 2.7 (2018-09-22)</h2> + + * Add the [./alerts.md|email alerts] feature for commits, ticket + changes, wiki changes, forum posts, and announcements. This is + still a work in progress. It is functional, but it is not as easy to + setup and use as it ought to be. + * Add the [./forum.wiki|discussion forum] feature. + * Add new user capabilities letters needed to support alerts and forum. + Formerly, user capabilities were letters from [a-z], but with the + enhancements, the supply of lower case letters was exhausted. + User capabilities are now letters in [a-zA-Z0-9]. + * The built-in skins are now responsive, providing better layout on + small screens, including mobile devices. + * The default skin now includes a hamburger menu that is generated + by the [/sitemap] page. + * All of the built-in skins now use a + [https://en.wikipedia.org/wiki/Content_Security_Policy|Content Security Policy (CSP)] + to help prevent cross-site injection and forgery attacks. There are no known + vulnerabilities in Fossil. The added CSP does not fix anything; it merely adds + another layer of defense. + * The [/sitemap] and other list pages show as multiple columns if + the viewing window is wide enough. + * There is an optional "js" file for each skin that can be used to + hold javascript. This file can be loaded by reference or can be + included in the header or footer. + * Add the [./backoffice.md|backoffice]. + * Update internal Unicode character tables, used in regular expression + handling, from version 10.0 to 11.0. + * Improvements to the "Security Audit" administration page + * Add the [/help/branch|fossil branch current] command. + * Add the [./grep.md|grep] command. + * Update the built-in SQLite to version 3.25.1. + * Some code and interfaces are in place to support sending and + receiving email directly via SMTP, but this feature is not yet + complete or ready for production use. + * The `mv-rm-files` setting is now compiled into Fossil in the + default Fossil configuration; no longer must you say + <tt>./configure --with-legacy-mv-rm</tt> to make it available. The + setting remains disabled by default, however, so you must still say + <tt>fossil set mv-rm-files 1</tt> to enable it on each repository + where you want hard <tt>mv/rm</tt> behavior. + +<h2 id='v2_6'>Changes for Version 2.6 (2018-05-04)</h2> + + * Fix a bug that was causing crashes while trying to clone the TCL + repository. This fix is the main reason for the current release. + * Added the new "Classic" timeline viewing mode. "Classic" is the + same as "Verbose" in the previous release. The "Verbose" mode is + now like "Compact" except the extra check-in details are shown by + default. + * Add support for ETags:, Last-Modified:, and If-Modified-Since: + cache control mechanisms. + * Enhance the [/help/www/tarball|/tarball], + [/help/www/zip|/zip], and + [/help/www/sqlar|/sqlar] pages so that the checkin + name to be downloaded can be expressed as part of the URI, + and without the need for query parameters. + * On the [/help/www/timeline|/timeline] webpage, add the days=N + query parameter and enhance the ymd=DATE and yw=DATE query parameters + to accept 'now' as an argument to show the latest day or week. + * In the web page that comes up in response to the + [/help/all|fossil all ui] command, show the last modification + time for each repository, and allow click-to-sort on the modification + time column. + * In the tarball cache replacement algorithm, give extra weight to + tarballs that have been accessed more than once. + * Additional defenses against web-based attacks. There have not been + any known vulnerabilities. We are just being paranoid. + * Update the built-in SQLite to an alpha version of 3.24.0. + +<h2 id='v2_5'>Changes for Version 2.5 (2018-02-07)</h2> + + * Numerous enhancements to the look and feel of the web interface. + Especially: Added separate "Modern", "Compact", "Verbose", and + "Columnar" view options on timelines. + * Common display settings (such as the "view" option and the number + of rows in a timeline) are held in a cookie and thus persist + across multiple pages. + * Rework the skin editing process so that changes are implemented + on one of nine /draft pages, evaluated, then merged back to the + default. + * Added the [/timeline?skin=ardoise&once|Ardoise] skin. + * Fix the "fossil server" command on Unix to be much more responsive + to multiple simultaneous web requests. + * Use the IPv6 stack for the "fossil ui" and "fossil server" + commands on Windows. + * Support for [https://sqlite.org/sqlar|SQL Archives] as a download + option. + * Fossil now automatically generates the + <html><head>...</head><body> + at the beginning of each web page if the configurable header + lacks a <body> tag. + * Added the /artifact_stats page, currently accessible only by + the administrator. + * Upgrade to the latest versions of SQLite and OpenSSL. + * Improved key bindings on the Tk diff screen generated by + "fossil diff --tk". + * Begin factoring out in-line javascript into separately loaded + script files. This is a step along the + road toward supporting a strict Content Security Policy. More work + is to be done. + * Initial infrastructure is in place to make use of the pledge() + system call in OpenBSD. More work is to be done. + +<h2 id='v2_4'>Changes for Version 2.4 (2017-11-03)</h2> + + * New feature: URL Aliases. URL Aliases allow an administrator + to define their own URLs on the web interface that are rewritten to + built-in URLs with specific parameters. Create and configure URL Aliases + using the /Setup/URL_Aliases menu option in the web interface. + * Add tech-note search capability. + * Add the -r|--revision and -o|--origin options to the + [/help/annotate|annotate] command. + * Add the origin= query parameter to the [/help/www/annotate|/annotate] + webpage. + * The [/help/annotate|fossil annotate] command and the + [/help/www/annotate|/annotate] web page go backwards in time as far + as can be computed in 30 milliseconds by default, rather than stopping + after 20 steps. The new limit= query parameter or the --limit command-line + option can be used to alter this timeout. + * Provide separate [/help#settings|on-line help screens for each setting]. + * Back out support for the --no-dir-symlinks option + * Remove support from the legacy configuration sync protocol. The only + way now to do a configuration push or pull is to use the new protocol that + was added in 2011. + * Add the from= and to= query parameters to [/help/www/fdiff|/fdiff] + in order to get a diff of two files in the same check-in. + * Fix the "ssh://" protocol to prevent an attack whereby the attacker convinces + a victim to run a "clone" with a dodgy URL and thereby gains access to their + system. + * Provide a checkbox that will temporarily disable all ad-units. + * Improvements to the [/help/www/stat|/stat] page + * Various new hyperlinks to the [/help/www/bloblist|/bloblist] + and [/help/www/bigbloblist|/bigbloblist] pages. + * Correct the [/help/www/doc|/doc] page to support read-only repositories. + * Correct [/help/www/zip|/zip], [/help/www/tarball|/tarball], + [/help/zip|zip], and [/help/tarball|tarball] pages and commands to + honor the versioned manifest setting when outside of an open checkout + directory. + * The admin-log and access-log settings are now on by default for + new repositories. + * Update the built-in SQLite to version 3.21.0. + +<h2 id='v2_3'>Changes for Version 2.3 (2017-07-21)</h2> + + * Update the built-in SQLite to version 3.20.0 (beta). + * Update internal Unicode character tables, used in regular expression + handling, from version 9.0 to 10.0. + * Show the last-sync-URL on the [/help/www/urllist|/urllist] page. + * Added the "Event Summary" activity report. + [/reports?type=ci&view=lastchng|example] + * Added the "Security Audit" page, available to administrators only + * Added the Last Login time to the user list page, for administrators only + * Added the --numstat option to the [/help/diff|fossil diff] command + * Limit the size of the heap and stack on unix systems, as a proactive + defense against the + [https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt|Stack Clash] + attack. + * Fix "database locked" warnings caused by "PRAGMA optimize". + * Fix a potential XSS vulnerability on the + [/help/www/help|/help] webpage. + * Documentation updates + +<h2 id='v2_2'>Changes for Version 2.2 (2017-04-11)</h2> + + * GIT comment tags are now handled by Fossil during import/export. + * Show the content of README files on directory listings. + ([/file/skins|example]) + * Support for Basic Authentication if enabled (default off). + * Show the hash algorithms used on the + [/help/www/rcvfromlist|/rcvfromlist] page. + * The [/help/www/tarball|/tarball] and [/help/www/zip|/zip] pages + now use the the r= query parameter + to select which check-in to deliver. The uuid= query parameter + is still accepted for backwards compatibility. + * Update the built-in SQLite to version 3.18.0. + * Run "[https://www.sqlite.org/pragma.html#pragma_optimize|PRAGMA optimize]" + on the database connection as it is closing. + +<h2 id='v2_1'>Changes for Version 2.1 (2017-03-10)</h2> + + * Add support for [./hashpolicy.wiki|hash policies] that control which + of the Hardened-SHA1 or SHA3-256 algorithms is used to name new + artifacts. + * Add the "gshow" and "gcat" subcommands to [/help/stash|fossil stash]. + * Add the [/help/www/juvlist|/juvlist] web page and use it to construct + the [/uv/download.html|Download Page] of the Fossil self-hosting website + using Ajax. + +<h2 id='v2_0'>Changes for Version 2.0 (2017-03-03)</h2> + + * Use the + [https://github.com/cr-marcstevens/sha1collisiondetection|hardened SHA1] + implementation by Marc Stevens and Dan Shumow. + * Add the ability to read and understand + [./fileformat.wiki#names|artifact names] that are based on SHA3-256 + rather than SHA1, but do not actually generate any such names. + * Added the [/help/sha3sum|sha3sum] command. + * Update the built-in SQLite to version 3.17.0. + +<h2 id='v1_37'>Changes for Version 1.37 (2017-01-16)</h2> + + * Add checkbox widgets to various web pages. See [/technote/8d18bf27e9| + this technote] for more information. To get the checkboxes to look as + intended, you must update the CSS in your repository and all clones. + * Add the [/help/all|fossil all ui] command + * Add the [/help/www/file|/file] webpage + * Enhance the [/help/www/brlist|/brlist] webpage to make use of branch colors. + * Add support for the ms=EXACT|LIKE|GLOB|REGEXP query parameter on the + [/help/www/timeline|/timeline] webpage, with associated form widgets. + * Enhance the [/help/changes|changes] and [/help/status|status] commands + with many new filter options so that specific kinds of changes can be + found without having to pipe through grep or sed. + * Enhanced the [/help/sqlite3|fossil sql] command so that it opens the + [./tech_overview.wiki#localdb|checkout database] and the + [./tech_overview.wiki#configdb|configuration database] in addition to the + repository database. + * TH1 enhancements: + <ul><li>Add <nowiki>[unversioned content]</nowiki> command.</li> + <li>Add <nowiki>[unversioned list]</nowiki> command.</li> + <li>Add project_description variable.</li> + </ul> + * Rename crnl-glob [/help/settings|setting] to crlf-glob, but keep + crnl-glob as a compatibility alias. + * Added the --command option to the [/help/diff|diff] command. + * Fix a C99-ism that prevents the 1.36 release from building with MSVC. + * Fix [/help/ticket|ticket set] when using the "+" prefix with fields + from the "ticketchng" table. + * Remove the "fusefs" command from builds that do not have the underlying + support enabled. + * Fixes for incremental git import/export. + * Minor security enhancements to + [./encryptedrepos.wiki|encrypted repositories]. + * Update the built-in SQLite to version 3.16.2. + * Update the built-in Zlib to version 1.2.11. + + +<h2 id='v1_36'>Changes for Version 1.36 (2016-10-24)</h2> + + * Add support for [./unvers.wiki|unversioned content], + the [/help/unversioned|fossil unversioned] command and the + [/help/www/uv|/uv] and [/uvlist] web pages. + * The [/uv/download.html|download page] is moved into + [./unvers.wiki|unversioned content] so that the self-hosting Fossil + websites no longer uses any external content. + * Added the "Search" button to the graphical diff generated by + the --tk option on the [/help/diff|diff] command. + * Added the "--checkin VERSION" option to the + [/help/diff|diff] command. + * Various performance enhancements to the [/help/diff|diff] command. + * Update internal Unicode character tables, used in regular expression + handling, from version 8.0 to 9.0. + * Update the built-in SQLite to version 3.15. Fossil now requires + the SQLITE_DBCONFIG_MAINDBNAME interface of SQLite which is only available + in SQLite version 3.15 and later and so Fossil will not work with + earlier SQLite versions. + * Fix [https://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg23618.html|multi-line timeline bug] + * Enhance the [/help/purge|fossil purge] command. + * New command [/help/shell|fossil shell]. + * SQL parameters whose names are all lower-case in Ticket Report SQL + queries are filled in using HTTP query parameter values. + * Added support for [./childprojects.wiki|child projects] that are + able to pull from their parent but not push. + * Added the -nocomplain option to the TH1 "query" command. + * Added support for the chng=GLOBLIST query parameter on the + [/help/www/timeline|/timeline] webpage. + +<h2 id='v1_35'>Changes for Version 1.35 (2016-06-14)</h2> + + * Enable symlinks by default on all non-Windows platforms. + * Enhance the [/md_rules|Markdown formatting] so that hyperlinks that begin + with "/" are relative to the root of the Fossil repository. + * Rework the [/help/www/setup_ulist|/setup_list page] (the User List page) + to display all users in a click-to-sort table. + * Fix backslash-octal escape on filenames while importing from git + * When markdown documents begin with <h1> HTML elements, use that + header at the document title. + * Added the [/help/www/bigbloblist|/bigbloblist page]. + * Enhance the [/help/www/finfo|/finfo page] so that when it is showing + the ancestors of a particular file version, it only shows direct + ancestors and omits changes on branches, thus making it show the same set + of ancestors that are used for [/help/www/blame|/blame]. + * Added the --page option to the [/help/ui|fossil ui] command + * Added the [/help/bisect|fossil bisect ui] command + * Enhanced the [/help/diff|fossil diff] command so that it accepts + directory names as arguments and computes diffs on all files contained + within those directories. + * Fix the [/help/add|fossil add] command so that it shows "SKIP" for + files added that were already under management. + * TH1 enhancements: + <ul><li>Add <nowiki>[array exists]</nowiki> command.</li> + <li>Add minimal <nowiki>[array names]</nowiki> command.</li> + <li>Add tcl_platform(engine) and tcl_platform(platform) array + elements.</li> + </ul> + * Get autosetup working with MinGW. + * Fix autosetup detection of zlib in the source tree. + * Added autosetup detection of OpenSSL when it may be present under the + "compat" subdirectory of the source tree. + * Added the [/help/reparent|fossil reparent] command + * Added --include and --exclude options to [/help/tarball|fossil tarball] + and [/help/zip|fossil zip] and the in= and ex= query parameters to the + [/help/www/tarball|/tarball] and [/help/www/zip|/zip] web pages. + * Add support for [./encryptedrepos.wiki|encrypted Fossil repositories]. + * If the FOSSIL_PWREADER environment variable is set, then use the program it + names in place of getpass() to read passwords and passphrases + * Option --baseurl now works on Windows. + * Numerous documentation improvements. + * Update the built-in SQLite to version 3.13.0. + +<h2 id='v1_34'>Changes for Version 1.34 (2015-11-02)</h2> + + * Make the [/help/clean|fossil clean] command undoable for files less + than 10MiB. + * Update internal Unicode character tables, used in regular expression + handling, from version 7.0 to 8.0. + * Add the new [/help/amend|amend] command which is used to modify + tags of a "check-in". + * Fix bug in [/help/import|import] command, handling version 3 of + the svndump format for subversion. + * Add the [/help/all|all cache] command. + * TH1 enhancements: + <ul><li>Add minimal <nowiki>[lsearch]</nowiki> command. Only exact + case-sensitive matching is supported.</li> + <li>Add the <nowiki>[glob_match]</nowiki>, <nowiki>[markdown]</nowiki>, + <nowiki>[dir]</nowiki>, and <nowiki>[encode64]</nowiki> commands.</li> + <li>Add the <nowiki>[tclIsSafe] and [tclMakeSafe]</nowiki> commands to + the Tcl integration subsystem.</li> + <li>Add 'double', 'integer', and 'list' classes to the + <nowiki>[string is]</nowiki> command.</li> + </ul> + * Add the --undo option to the [/help/diff|diff] command. + * Build-in Antirez's "linenoise" command-line editing library for use with + the [/help/sqlite3|fossil sql] command on Unix platforms. + * Add [/help/stash|stash cat] as an alias for the + [/help/stash|stash show] command. + * Automatically pull before [/help/merge|fossil merge] when auto-sync + is enabled. + * Fix --hard option to [/help/mv|fossil mv] and [/help/rm|fossil rm] + to enable them to work properly with certain relative paths. + * Change the mimetype for ".n" and ".man" files to text/plain. + * Display improvements in the [/help/bisect|fossil bisect chart] command. + * Updated the built-in SQLite to version 3.9.1 and activated JSON1 and FTS5 + support (both currently unused within Fossil). + +<h2 id='v1_33'>Changes for Version 1.33 (2015-05-23)</h2> + * Improved fork detection on [/help/update|fossil update], + [/help/status|fossil status] and related commands. + * Change the default skin to what used to be called "San Francisco Modern". + * Add the [/repo-tabsize] web page + * Add [/help/import|fossil import --svn], for importing a subversion + repository into fossil which was exported using "svnadmin dump". + * Add the "--compress-only" option to [/help/rebuild|fossil rebuild]. + * Use a pie chart on the [/reports?view=byuser] page. + * Enhanced [/help/clean|fossil clean --verily] so that it ignores + keep-glob and ignore-glob settings. Added the -x alias for --verily. + * Add the --soft and --hard options to [/help/rm|fossil rm] and + [/help/mv|fossil mv]. The default is still --soft, but that is + now configurable at compile-time or by the mv-rm-files setting. + * Improved ability to [./customgraph.md|customize the timeline graph]. + * Improvements to the [/sitemap] page. + * Automatically adjust the [/help/timeline|CLI timeline] to the terminal + width on Linux. + * Added <nowiki>[info commands] and [info vars]</nowiki> commands to TH1. + These commands perform the same function as their Tcl counterparts, + except they do not accept a pattern argument. + * Fix some obscure issues with TH1 expression processing. + * Fix titles in search results for documents that are not wiki, markdown, + or HTML. + * Formally translate TH1 to Tcl return codes and vice-versa, where + necessary, in the Tcl integration subsystem. + * Add [/help/leaves|fossil leaves -multiple], for finding multiple + leaves on the same branch. + * Added the "Blitz" skin option. + * Removed the ".fossil-settings/keep-glob" file. It should not have been + checked into the repository. + * Update the built-in SQLite to version 3.8.10.2. + * Make [/help/open|fossil open] honor ".fossil-settings/allow-symlinks". + * Allow [/help/add|fossil add] to be used on symlinks to nonexistent or + unreadable files in the same way as [/help/addremove|fossil addremove]. + * Added fork warning to be issued if sync produced a fork + * Update the [/help/www/info|info] page to report when a file becomes a + symlink. Additionally show the UUID for files whose types have changed + without changing contents or symlink target. + * Have [/help/changes|fossil changes] and + [/help/status|fossil status] report when executable or symlink status + changes on otherwise unmodified files. + * Permit filtering weekday and file [/help/www/reports|reports] by user. + Also ensure the user parameter is preserved when changing types. Add a + field for direct entry of the user name to each applicable report. + * Create parent directories of [/help/settings|empty-dirs] if they don't + already exist. + * Inhibit timeline links to wiki pages that have been deleted. + +<h2 id='v1_33'>Changes for Version 1.32 (2015-03-14)</h2> + * When creating a new repository using [/help/init|fossil init], ensure + that the new repository is fully compatible with historical versions of + Fossil by having a valid manifest as RID 1. + * Anti-aliased rendering of arrowheads on timeline graphs. + * Added vi/less-style key bindings to the --tk diff GUI. + * Documentation updates to fix spellings and changes all "checkins" to + "check-ins". + * Add the --repolist option to server commands such as + [/help/server|fossil server] or [/help/http|fossil http]. + * Added the "Xekri" skin. + * Enhance the "ln=" query parameter on artifact displays to accept multiple + ranges, separate by spaces (or "+" when URL-encoded). + * Added [/help/forget|fossil forget] as an alias for + [/help/rm|fossil rm]. + +<h2 id='v1_31'>Changes For Version 1.31 (2015-02-23)</h2> + * Change the auxiliary schema by adding columns MLINK.ISAUX and MLINK.PMID + columns to the schema, to support better drawing of file change graphs. + A [/help/rebuild|fossil rebuild] is recommended but is not required. + so that the new graph drawing logic can work effectively. + * Added [/search|search] over Check-in comments, Documents, Tickets and + Wiki. Disabled by default. The search can be either a full-scan or it + can use an index that is kept up-to-date automatically. The new + /srchsetup web-page and the [/help/fts-config|fts-config] command + were added to help configure the search capability. Expect further + enhancements to the search capabilities in subsequent releases. + * Added form elements to some submenus (in particular the /timeline) + for easier operation. + * Added the --ifneeded option to [/help/rebuild|fossil rebuild]. + * Added "override skins" using the "skin:" line of the CGI script or + using the --skin LABEL option on the [/help/server|server], + [/help/ui|ui], or [/help/http|http] commands. + * Embedded html documents that begin with + <doc class="fossil-doc"> are displayed with standard + headers and footers added. + * Allow <div style='...'> markup in [/wiki_rules|wiki]. + * Renamed "Events" to "Technical Notes", while updating the technote + display and control pages. Add support for technotes as plain text + or as Markdown. + * Added the [/md_rules] pages containing summary instructions on the + Markdown format. + * Added the --repolist and --nojail options to the various server commands + (ex: [/help/server|fossil server]). + * Added the [/help/all|fossil all add] subcommand to "fossil all". + * Improvements to the /login page. Some hyperlinks to pages that require + "anonymous" privileges are displayed even if the current user is "nobody" + but automatically redirect to /login. + * The [/help/www/doc|/doc] web-page will now try to deliver the file + "404.md" from the top-level directory (if such a file exists) in + place of its built-in 404 text. + * Download of Tarballs and ZIP Archives by user "nobody" is now enabled + by default in new repositories. + * Enhancements to the table sorting controls. More display tables + are now sortable. + * Add IPv6 support to [/help/sync|fossil sync] and + [/help/clone|fossil clone] + * Add more skins such as "San Francisco Modern" and "Eagle". + * During shutdown, check to see if the check-out database (".fslckout") + contains a lot of free space, and if it does, VACUUM it. + * Added the [/mimetype_list] page. + * Added the [/hash-collisions] page. + * Allow the user of Common Table Expressions in the SQL that defaults + ticket reports. + * Break out the components (css, footer, and header) for the + various built-in skins into separate files in the source tree. + +<h2 id='v1_30'>Changes For Version 1.30 (2015-01-19)</h2> + * Added the [/help/bundle|fossil bundle] command. + * Added the [/help/purge|fossil purge] command. + * Added the [/help/publish|fossil publish] command. + * Added the [/help/unpublished|fossil unpublished] command. + * Enhance the [/tree] webpage to show the age of each file with the option + to sort by age. + * Enhance the [/brlist] webpage to show additional information about each branch + and to be sortable by clicking on column headers. + * Add support for Docker. Just install docker and type + "sudo docker run -d -p 8080:8080 nijtmans/fossil" to get it running. + * Add the [/help/fusefs|fossil fusefs DIRECTORY] command that mounts a + Fuse Filesystem at the given DIRECTORY and populates it with read-only + copies of all historical check-ins. This only works on systems that + support FuseFS. + * Add the administrative log that records all configuration. + * Added the [/sitemap] webpage. + * Added the [/bloblist] web page. + * Let [/help/new|fossil new] no longer create an initial empty commit + by default. The first commit after checking out an empty repository will + become the initial commit. + * Added the [/help/all|fossil all dbstat] and + [/help/all|fossil all info] commands. + * Update SQLite to version 3.8.8. + * Added the --verily option to the [/help/clean|fossil clean] command. + * Add the "autosync-tries" setting to control the number of autosync attempts + before returning an error. + * Added a compile-time option (--with-miniz) to build using miniz instead + of zlib. Disabled by default. + * Support customization of commands and webpages, including the ability to + add new ones, via the "TH1 hooks" feature. Disabled by default. Enabled + via a compile-time option. + * Add the <nowiki>[checkout], [render], [styleHeader], [styleFooter], + [trace], [getParameter], [setParameter], [artifact], and + [globalState]</nowiki> commands to TH1, primarily for use by TH1 hooks. + * Automatically adjust the width of command-line timeline output according to the + detected width of the terminal. + * Prompt the user to optionally fix invalid UTF-8 at check-in. + * Added a line-number toggle option to the [/help/www/info|/info] + and [/help/www/artifact|/artifact] pages. + * Most commands now issue errors rather than silently ignoring unrecognized + command-line options. + * Use full 40-character SHA1 hashes (instead of abbreviations) in most + internal URLs. + * The "ssh:" sync method on Windows now uses "plink.exe" instead of "ssh" as + the secure-shell client program. + * Prevent a partial clone when the connection is lost. + * Make the distinction between 301 and 302 redirects. + * Allow commits against a closed check-in as long as the commit goes onto + a different branch. + * Improved cache control in the web interface reduces unnecessary requests + for common resources like the page logo and CSS. + * Fix a rare and long-standing sync protocol bug + that would silently prevent the sync from running to completion. Before + this bug-fix it was sometimes necessary to do "fossil sync --verily" to + get two repositories in sync. + * Add the [/finfo?name=src/foci.c|files_of_checkin] virtual table - useful + for ad hoc queries in the [/help/sqlite3|fossil sql] interface, + and also used internally. + * Added the "$secureurl" TH1 variable for use in headers and footers. + * (Internal:) Add the ability to include resources as separate files in the + source tree that are converted into constant byte arrays in the compiled + binary. Use this feature to store the Tk script that implements the --tk + diff option in a separate file for easier editing. + * (Internal:) Implement a system of compile-time checks to help ensure + the correctness of printf-style formatting strings. + * Fix CVE-2014-3566, also known as the POODLE SSL 3.0 vulnerability. + * Numerous documentation fixes and improvements. + * Other obscure and minor bug fixes - see the timeline for details. + +<h2 id='v1_29'>Changes For Version 1.29 (2014-06-12)</h2> + * Add the ability to display content, diffs and annotations for UTF16 + text files in the web interface. + * Add the "SaveAs..." and "Invert" buttons + to the graphical diff display that results + from using the --tk option with the [/help/diff | fossil diff] command. + * The [/reports] page now requires Read ("o") permissions. The "byweek" + report now properly propagates the selected year through the event type + filter links. + * The [/help/info | info command] now shows leaf status of the checkout. + * Add support for tunneling https through a http proxy (Ticket [e854101c4f]). + * Add option --empty to the "[/help/open | fossil open]" command. + * Enhanced [/help/www/fileage|the fileage page] to support a glob parameter. + * Add -w|--ignore-all-space and -Z|--ignore-trailing-space options to + [/help/annotate|fossil annotate], [/help/blame|fossil blame], + [/help/diff|fossil (g)diff], [/help/stash|fossil stash diff]. + * Add --strip-trailing-cr option to [/help/diff|fossil (g)diff] and + [/help/stash|fossil stash diff]. + * Add button "Ignore Whitespace" to /annotate, /blame, /ci, /fdiff + and /vdiff UI pages. + * Enhance [/reports?view=byweekday|/reports] with a "byweekday" view. + * Enhance the [/help/cat|fossil cat] command so that it works outside + of a checkout when using the -R command-line option. + * Use full-length SHA1 hashes, not abbreviations, in most hyperlinks. + * Correctly render the <title> markup on wiki pages in the + [/help/www/artifact|/artifact] webpage. + * Enhance the [/help/whatis|fossil whatis] command to report on attachments + and cluster artifacts. Added the [/help/test-whatis-all] command for + testing purposes. + * Add support for HTTP Basic Authentication on [/help/clone|clone] and + [/help/sync|sync]. + * Fix the [/help/stash|stash] so that it remembers added files and re-adds + them when the stash is applied. + * Fix the server so that it avoids writing to the database (and thus avoids + database locking issues) on a + [/help/pull|pull] or [/help/clone|clone]. + * Add support for [./server.wiki#loadmgmt|server load management] using both + a cache of expensive pages (the [/help/cache|fossil cache] command) + and by rejecting expensive page requests when the server load average is too + high. + * Add the [/help/praise|fossil praise] command as an alias for + [/help/blame|fossil blame] for subversion compatibility. + * Enhance the [/help/test-diff|fossil test-diff] command with -y or --tk + options so that it shows both filenames above their respective columns in + the side-by-side diff output. + * Issue a warning if a [/help/add|fossil add] command tries to add a file + that matches the ignore-glob. + * Add option -W|--width to "[/help/stash|fossil stash ls]" + and "[/help/leaves|fossil leaves]" commands. + * Enhance support for running as the root user. Now works on Haiku. + * Added the <tt>-empty</tt> option to [/help/new|fossil new], which + causes it to not create an initial empty commit. The first commit after + checking out a repo created this way will become the initial commit. + * Enhance sync operations by committing each round-trip to minimize number + of retransmits when autosync fails. Include option for + [/help/update| fossil update] and [/help/merge| fossil merge] to + continue even if missing content. + * Minor portability fixes for platforms where the char type is unsigned + by default. + +<h2>Changes For Version 1.28 (2014-01-27)</h2> + * Enhance [/help/www/reports | /reports] to support event type filtering. + * When cloning a repository, the user name passed via the URL (if any) + is now used as the default local admin user's name. + * Enhance the SSH transport mechanism so that it runs a single instance of + the "fossil" executable on the remote side, obviating the need for a shell + on the remote side. Some users may need to add the "?fossil=/path/to/fossil" + query parameter to "ssh:" URIs if their fossil binary is not in a standard + place. + * Add the "[/help/blame | fossil blame]" command that works just like + "fossil annotate" but uses a different output format that includes the + user who made each change and omits line numbers. + * Add the "Tarball and ZIP-archive Prefix" configuration parameter under + Admin/Configuration. + * Fix CGI processing so that it works on web servers that do not + supply REQUEST_URI. + * Add options --dirsonly, --emptydirs, and --allckouts to the + "[/help/clean | fossil clean]" command. + * Ten-fold performance improvement in large "fossil blame" or + "fossil annotate" commands. + * Add option -W|--width and --offset to "[/help/timeline | fossil timeline]" + and "[/help/finfo | fossil finfo]" commands. + * Option -n|--limit of "[/help/timeline | fossil timeline]" now + specifies the number of entries, just like all other commands which + have the -n|--limit option. The various timeline-related functions + now output "--- ?? limit (??) reached ---" at the end whenever + appropriate. Use "-n 0" if no limit is desired. + * Fix handling of password embedded in Fossil URL. + * New <tt>--once</tt> option to [/help/clone | fossil clone] command + which does not store the URL or password when cloning. + * Modify [/help/ui | fossil ui] to respect "default user" in an open + repository. + * Fossil now hides check-ins that have the "hidden" tag in timeline webpages. + * Enhance <tt>/ci_edit</tt> page to add the "hidden" tag to check-ins. + * Advanced possibilities for commit and ticket change notifications over + http using TH1 scripting. + * Add --sha1sum and --integrate options + to the "[/help/commit | fossil commit]" command. + * Add the "clean" and "extra" subcommands to the + "[/help/all | fossil all]" command + * Add the --whatif option to "[/help/clean|fossil clean]" that works the + same as "--dry-run", + so that the name does not collide with the --dry-run option of "fossil all". + * Provide a configuration option to show dates on the web timeline + as "YYMMMDD HH:MM" + * Add an option to the "stats" webpage that allows an administrator to see + the current repository schema. + * Enhancements to the "[/help/www/vdiff|/vdiff]" webpage for more difference + display options. + * Added the "[/tree?ci=trunk&expand | /tree]" webpage as an alternative + to "/dir" and make it the default way of showing file lists. + * Send gzipped HTTP responses to clients that support it. + +<h2>Changes For Version 1.27 (2013-09-11)</h2> + * Enhance the [/help/changes | fossil changes], + [/help/clean | fossil clean], [/help/extras | fossil extras], + [/help/ls | fossil ls] and [/help/status | fossil status] commands + to restrict operation to files and directories named on the command-line. + * New --integrate option to [/help/merge | fossil merge], which + automatically closes the merged branch when committing. + * Renamed <tt>/stats_report</tt> page to [/reports]. Graph width is now + relative, not absolute. + * Added <tt>yw=YYYY-WW</tt> (year-week) filter to timeline to limit the results + to a specific year and calendar week number, e.g. [/timeline?yw=2013-01]. + * Updates to SQLite to prevent opening a repository file using file descriptors + 1 or 2 on Unix. This fixes a bug under which an assertion failure could + overwrite part of a repository database file, corrupting it. + * Added support for unlimited line lengths in side-by-side diffs. + * New --close option to [/help/commit | fossil commit], which + immediately closes the branch being committed. + * Added <tt>chart</tt> option to [/help/bisect | fossil bisect]. + * Improvements to the "human or bot?" determination. + * Reports errors about missing CGI-standard environment variables for HTTP + servers which do not support them. + * Minor improvements to sync support on Windows. + * Added <tt>--scgi</tt> option to [/help/server | fossil server]. + * Internal improvements to the sync process. + * The internals of the JSON API are now MIT-licensed, so downstream + users/packagers are no longer affected by the "do no evil" license + clause. + +<h2>Changes For Version 1.26 (2013-06-18)</h2> + * The argument to the --port option for the [/help/ui | fossil ui] and + [/help/server | fossil server] commands can take an IP address in addition + to the port number, causing Fossil to bind to just that one IP address. + * After prompting for a password, also ask if that password should be + remembered. + * Performance improvements to the diff engine. + * Fix the side-by-side diff engine to work better with multi-byte Unicode text. + * Color-coding in the web-based annotation (blame) display. Fix the annotation + engine so that it is no longer confused by time-warps. + * The markdown formatter is now available by default and can be used for + tickets, wiki, and embedded documentation. + * Add subcommands "fossil bisect log" and "fossil bisect status" to the + [/help/bisect | fossil bisect] command, as well as other bisect enhancements. + * Enhanced defenses that prevent spiders from using excessive CPU and bandwidth. + * Consistent use of the -n or --dry-run command line options. + * Win32: Fossil now understands Cygwin paths containing one or more of + the characters <nowiki>"*:<>?|</nowiki>. Those are normally forbidden in + win32. This means that the win32 fossil.exe is better usable in a Cygwin + environment. See + [http://cygwin.com/cygwin-ug-net/using-specialnames.html#pathnames-specialchars]. + * Cygwin: Fossil now understands win32 absolute paths starting with a drive + letter everywhere. The default value of the "case-sensitive" setting is + now FALSE, except when case-sensitivity is enabled in the Windows kernel. + See + [http://cygwin.com/cygwin-ug-net/using-specialnames.html#pathnames-casesensitive] + * Enhancements to /timeline.rss, adding more flags for filtering + results, including the ability to subscribe to changes made + to individual tickets. For example: [/timeline.rss?y=t&tkt=12fceeec82]. + * Improved handling of the differences between case-sensitive and + case-insensitive filesystems. + * JSON API: added the 'status' command to report local checkout status. + * Fixes to the <tt>--args</tt> support and documented this feature in the help. + * Added [/stats_report] page. + * Added <tt>ym=YYYY-MM</tt> filter to the [/timeline?ym=2013-06]. + * Fixed: <tt>config reset</tt> now re-installs default ticket report format. + * <tt>ssh://</tt> and <tt>file://</tt> protocols now ignore proxy settings. + * Added [/hash-color-test] web page. + * Cherry-pick merges are recorded internally (though no yet displayed on the + timeline graph.) + * Bring in the latest versions of SQLite, zlib, and autosetup from upstream. + +<h2>Changes For Version 1.25 (2013-02-16)</h2> + * Enhancements to ticket processing. There are now two tables: TICKET and + TICKETCHNG. There is one row in TICKETCHNG for each ticket artifact. + Fields from ticket artifacts go into either or both of TICKET and + TICKETCHNG, whichever contain matching column names. Default ticket + edit and viewing scripts are updated to use TICKETCHNG. The TH1 + scripting language is enhanced to support this, including the new + "query" command for doing SQL queries against the repository database. + All changes should be backwards compatible. + * Add the ability to moderate ticket and wiki changes. Unmoderated changes + do not sync and may be deleted by the moderator if found to contain spam + or other objectionable content. + * Add javascript so that clicking on a node of the timeline graph selects + that node. Then clicking on a second node shows a diff between the + two nodes. Clicking on the selected node unselects it. + * Warn of unresolved merge conflicts in "fossil status" and disallow + commits of unresolved conflicts unless the --allow-conflict option + is used. + * Add javascript so that clicking on column headers in a ticket report + sorts by the indicated column. + * Add the "fossil cat" command which is basically an alias for + "fossil finfo -p". + * Hyperlinks with the class "button" are rendered as submenu buttons + on embedded documentation. + * The check-in comment editor on Windows now defaults to NotePad.exe. + * Correctly deal with BOMs in check-in comments. Also attempt to convert + check-in comments to UTF8 from other encodings. + * Allow the deletion of multiple stash entries using multiple arguments + to the "fossil stash rm" command. + * Enhance the "fossil server DIRECTORY" command to serve static content + files contained in DIRECTORY. For security, only files with a + recognized suffix (such as *.html, *.jpg, *.txt, etc) will be delivered + as static content, and *.fossil files are not on the list of recognized + suffixes. There are additional restrictions on the names of the files. + * Allow the "fossil ui" command to specify a directory as long as the + the --notfound option is used. + * Add a configuration option that causes timeline messages to be rendered + as text/x-fossil-plain (which is the same as text/plain except that + hyperlinks inside of <nowiki>[...]</nowiki> are decorated.) + * Only decorate <nowiki>[...]</nowiki> in check-in comments and tickets + if the contented text really is a valid hyperlink target. + * Improvements to the side-by-side diff algorithm, for a more + human-friendly display in some complex cases. + * Added <nowiki>[utime] and [stime]</nowiki> commands to TH1. These + commands can be used for things such as displaying the page rendering + time in the footer. + * Add the ability to pass command-line options of "fossil rebuild" to + "fossil all rebuild". + * Add the --deanalyze option to "fossil rebuild" (and "fossil all rebuild") + * Do not run the graphical merging tool nor leave merge-droppings after a + dry-run merge. Display an improved merge-summary message at the end of + the merge. + * Add options to "fossil commit" to override the various sanity checks. + Options added: --allow-empty, --allow-fork, --allow-older, and + --allow-conflict. + * Optionally require a CAPTCHA (controlled by a setting on the + Admin/Access webpage) when a user who is not logged in tries to + edit wiki, or a ticket, or an attachment. + * Improvements to the "ssh://" sync protocol, to help it move past + noisy motd comments. + * Add the uf=FILE-SHA1-HASH query parameter to the timeline, causing the + timeline to show only check-ins that contain the specific file identified + by FILE-SHA1-HASH. ("uf" stands for "uses file".) + * Enhance the file change annotator so that it follows the file across + name changes. + * Fix the server-side of the sync protocol so that it will not generate + a delta loop when a file changes from its original state, through two + or more intermediate states, and back to the original state, all within + a single sync. + * Show much less output during a sync operation, unless the --verbose + option is used. + * Set the action= attribute of <form> elements using javascript, + as an addition defense against spam-bots. + * Disallow invalid UTF8 characters (such as characters in the surrogate + pair range) in filenames. + * Judge the UserAgent strings issued by the NetSurf webbrowser to be + coming from a human, not from a bot. + * Add the zlib sources to the Fossil source tree (under compat/zlib) and + use those sources when compiling on (Windows) systems that do not have + a zlib library installed by default. + * Prompt the user with the option to convert non-UTF8 files into UTF8 + when committing. + * Allow the characters <nowiki>*[]?</nowiki> in filenames. + * Allow the --context option on diff commands to have a value of 0. + * Added the "dbstat" command. + * Enhanced "fossil merge" so that if the VERSION argument is omitted, Fossil + tries to merge any forks of the current branch. + * Improved detection of forks in a commit race. + * Added the --analyze option to "fossil rebuild". + +<h2>Changes For Version 1.24 (2012-10-22)</h2> + * Added support for WYSIWYG editing of wiki pages. WYSIWYG is turned off + by default and can be turned on by setting a configuration option. + * Allow style= attribute to occur in HTML markup on wiki pages. + * Added the --tk option to the "fossil diff" and "fossil stash diff" + commands, causing color-coded diff output to be displayed in a Tcl/Tk + GUI window. This option only works if Tcl/Tk is installed on the + host. + * On Windows, make the "gdiff" command default to use WinDiff.exe. + * Update the "fossil stash" command so that it always prompts for a + comment if the -m option is omitted. + * Enhance the timeline webpages so that a=, b=, c=, d=, p=, and dp= + query parameters (and others) can all accept any valid check-in name + (such as branch names or labels) instead of just SHA1 hashes. + * Added the "fossil stash show" command. + * Added the "fileage" webpage with links to this page from the check-in + information page and from the file browser. + * Added --age and -t options to the "fossil ls" command. + * Added the --setmtime option to "fossil update". When used, the mtime + of all managed files is set to the time when the most recent version of + the file was checked in. + * Changed the "vdiff" webpage to show the complete text of files that + were added or removed (the equivalent of using the -N or --newfile + options with the "fossil diff" command-line.) + * Added the --temp option to "fossil clean" and "fossil extra", causing + those commands to only look at temporary files generated by Fossil, + such as merge-conflict reports or aborted check-in messages. + * Enhance the raw page download so that it can guess the mimetype of + attachments based on the filename. + * Change the behavior of the from= and to= query parameters on the + timeline page so that by default the path between the two specified + check-ins avoids merges. + * Add the --baseurl option to "fossil server" and "fossil http" commands, + so that those commands can be used with reverse proxies. + * If unable to determine the command-line user, do not guess. Instead + issue an error message. This helps prevent check-ins from accidentally + occurring under the wrong username. + * Include branch information in the output of file change listings + (the "finfo" webpage). + * Make the simplified view of file history, rather than the full view, + the default. + * In the "fossil configuration" command, allow the "css" option for + synchronizing, importing, or exporting just the CSS file. This makes + it easier to share CSS files across repositories by exporting from + one and importing to another. + * Add the (unsupported) "fossil test-orphans" command. + * Add the --template option to the "fossil init" command, to facilitate + creating new repositories based on a template repository. + * Add the diff-binary setting, which if enabled causes binary files to + be passed to the "gdiff" command for it to deal with, rather than simply + printing a "cannot diff binary files" error. + * Add the --unified option to the "fossil diff" command to force a unified + diff even if the --tk option (which normally implies a side-by-side diff) + is used. + * Present a choice of nearby branches and versions to diff against on the + check-in information page. + * Add the --force option to the "fossil merge" command that will force the + merge to occur even if it would be a no-op. This is sometimes useful for + documentation purposes. + * Add another built-in skin: "Enhanced Default". Other minor tweaks to + the existing skins. + * Add the "urllist" webpage, showing a list of URLs by which a server + instance of Fossil has been accessed. Requires "Administrator" privileges. + A link is on the "Setup" main page. + * Enable dynamic loading of the Tcl runtime for installations that want + to use Tcl as part of their configuration. This reduces the size of + the Fossil binary and allows any version of Tcl 8.4 or later to be used. + * Merge the latest SQLite changes from upstream. + * Lots of minor bug fixes. + +<h2>Changes For Version 1.23 (2012-08-08)</h2> + * The default checkout database name is now ".fslckout" instead of + "_FOSSIL_" on Unix. Both names continue to work. + * Added the "fossil all changes" command + * Added the --ckout option to the "fossil all list" command + * Added the "public-pages" glob pattern that can be configured to allow + anonymous users to see embedded documentation on sites where source + code should not be accessible to anonymous users. + * Allow multiple --tag options on the same "fossil commit" command. + * Change the meaning of the --bgcolor option for "fossil commit" to only + change the color for that one commit. The new --branchcolor option + is available to set a persistent background color. + * Add the branch= query parameter to the vdiff page and the --branch option + to the "fossil diff" command. + * Check-in names of the form "root:BRANCH" now refer to the origin of + the branch. Hence to see all changes in a branch, use + "fossil diff --from root:BRANCH --to BRANCH". The --branch option on + the diff command is an alias for the same. + * Add the ability to configure ad-units to be displayed between the menu + bar and the content. + * Add the ability to set a background image as part of server configuration. + * Allow partial commits of cherrypick merges. + * Updates against an uncommitted merge are now a warning, not a fatal error. + * Prompt the user to continue if a check-in comment is unedited. + * Fixes to case sensitivity settings with the /dir webpage. + * Repositories now try to remember the locations of all checkouts and + web-access URLs and display this information with the + "fossil info $REPO" command. + * Improved defense against spiders: The src= attribute of + <a> elements is set using javascript after the page loads. + * Enhanced formatting of the user list page. + * If a file named in "fossil add" is missing, that is now a warning instead + of a fatal error. + * Fix side-by-side diff so that it displays correctly with + multi-byte UTF8 characters. + * Performance improvements in the diff logic. + * Other performance tweaks and documentation updates. + +<h2>Changes For Version 1.22 (2012-03-17)</h2> + * Greatly improved "diff" processing including the new --brief option, + partial line matching, colorized in-line diffs, and better performance. + * Promote "allow-symlinks" to a versionable setting + * Harden the CGI processing logic against DOS attacks + * Add the ability to run TH1 scripts after sync requests + * Store the repository name in _FOSSIL_ as it is type in the "open" command, + possibly as a relative pathname. + * Make ".fslckout" the alternative name for the "_FOSSIL_" file. + * Change the "ssh:" transfer method to allow all access regardless of + user permission. + * Improvements to the timeline messages associated with tag changes. + (Requires a "[/help/rebuild | fossil rebuild]" to take effect.) + * Various additions and fixes for the JSON API. + * Improved merge-with-rename handling. + * --cherrypick merges use their origin's commit message by default. + * Added support for multiple concurrent logins per user. + * Update to use SQLite version 3.7.11. + * Various minor bug fixes. + +<h2>Changes For Version 1.21 (2011-12-13)</h2> + * Added side-by-side diffs in the command-line interface + * Automatically enable hyperlinks if the UserAgent string in the + HTTP header suggests that the requestor is a human and not a bot. + * Show only commonly used commands with "fossil help". Use + "fossil help --all" to see the complete list now. + * Improvements to the "stash" command: (1) Stash all files, not just + those below the working directory. (2) Add the --detail option to + "list". (3) Confirm before "drop --all". (4) Add the "help" + subcommand. + * Add an Admin/Access setting to change the number of octets of the + IP address that are saved in login cookies - allowing this setting + to be changed to zero + * Promote the "test-md5sum" command to "md5sum". + * Added the "whatis" command. + * Stop showing the server-code in status outputs - it is no longer used + for anything. + * Added a compile-time option (--with-tcl) to build in full Tcl scripting + support via integration with TH1. + * Merged the JSON branch into trunk. Disabled by default. Enabled + by a compile-time option. Probably it will be enabled by default + in some future release. + * Update to use SQLite version 3.7.9 plus the alignment fix for Sparc. + align + +<h2>Changes For Version 1.20 (2011-10-21)</h2> + * Added side-by-side diffs in HTML interface. [0bde74ea1e] + * Added support for symlinks. (Controlled by "allow-symlinks" setting, + off by default). [e4f1c1fe95] + * Fixed CLI annotate to show the proper file version in case there + are multiple equal versions in history. [e161670939] + * Timeline now shows tag changes (requires rebuild).[87540ed6e6] + * Fixed annotate to show "more relevant" versions of lines in + some cases. [e161670939] + * New command: ticket history. [98a855c508] + * Disabled SSLv2 in HTTPS client.[ea1d369d23] + * Fixed constant prompting regarding previously-saved SSL + certificates. [636804745b] + * Other SSL improvements. + * Added -R REPOFILE support to several more CLI commands. [e080560378] + * Generated tarballs now have constant timestamps, so they are + always identical for any given check-in. [e080560378] + * A number of minor HTML-related tweaks and fixes. + * Added --args FILENAME global CLI argument to import arbitrary + CLI arguments from a file (e.g. long file lists). [e080560378] + * Fixed significant memory leak in annotation of files with long + histories.[9929bab702] + * Added warnings when a merge operation overwrites local copies + (UNDO is available, but previously this condition normally went + silently unnoticed). [39f979b08c] + * Improved performance when adding many files. [a369dc7721] + * Improve merges which contain many file renames. [0b93b0f958] + * Added protection against timing attacks. [d4a341b49d] + * Firefox now remembers filled fields when returning to forms. [3fac77d7b0] + * Added the --stats option to the rebuild command. [f25e5e53c4] + * RSS feed now passes validation. [ce354d0a9f] + * Show overridden user when entering commit comment. [ce354d0a9f] + * Made rebuilding from web interface silent. [ce354d0a9f] + * Now works on MSVC with repos >2GB. [6092935ff2] + * A number of code cleanups to resolve warnings from various compilers. + * Update the built-in SQLite to version 3.7.9 beta. + +<h2>Changes For Version 1.19 (2011-09-02)</h2> + * Added a ./configure script based on autosetup. + * Added the "[/help/winsrv | fossil winsrv]" command + for creating a Fossil service on Windows systems. + * Added "versionable settings" where settings that affect + the local tree can be stored in versioned files in the + .fossil-settings directory. + * Background colors for branches are chosen automatically if no + color is specified by the user. + * The status, changes and extras commands now show + pathnames relative to the current working directory, + unless overridden by command line options or the + "relative-paths" setting.<br><b>WARNING:</b> This + change will break scripts which rely on the current + output when the current working directory is not the + repository root. + * Added "empty-dirs" versionable setting. + * Added support for client-side SSL certificates with "ssl-identity" + setting and --ssl-identity option. + * Added "ssl-ca-location" setting to specify trusted root + SSL certificates. + * Added the --case-sensitive BOOLEAN command-line option to many commands. + Default to true for Unix and false for Windows. + * Added the "Color-Test" submenu button on the branch list web page. + * Compatibility improvements to the git-export feature. + * Performance improvements on SHA1 checksums + * Update to the latest SQLite version 3.7.8 alpha. + * Fix the tarball generator to work with very log pathnames + +<h2>Changes For Version 1.18 (2011-07-14)</h2> + * Added this Change Log + * Added sequential version numbering + * Added an optional configure script - the Makefile still works for most + systems. + * Improvements to the "annotate" algorithm: only search primary + ancestors and ignore branches. + * Update the "scrub" command to remove traces of login-groups and + subrepositories. + * Added the --type option to the "fossil tag find" command. + * In contexts where only a check-in makes sense, resolve branch and + tag names to checkins only, never events or other artifacts. + * Improved display of file renames on a diff. A rebuild is required + to take full advantage of this change. + * Update the built-in SQLite to version 3.7.7. diff --git a/www/chat.md b/www/chat.md new file mode 100644 index 0000000..23a310d --- /dev/null +++ b/www/chat.md @@ -0,0 +1,262 @@ +# Fossil Chat + +## Introduction + +Fossil’s developer chatroom feature provides an +ephemeral discussion venue for insiders. Design goals include: + + * **Simple but functional** → + Fossil chat is designed to provide a convenient real-time + communication mechanism for geographically dispersed developers. + Fossil chat is *not* intended as a replacement or competitor for + IRC, Slack, Discord, Telegram, Google Hangouts, etc. + + * **Low administration** → + You can activate the chatroom in seconds without having to + mess with configuration files or install new software. + In an existing [server setup](./server/), + simply enable the [C capability](/setup_ucap_list) for users + whom you want to give access to the chatroom. + + * **Ephemeral** → + Chat messages do not sync to peer repositories, and they are + automatically deleted after a configurable delay (default: 7 days). + Individual messages or the entire conversation + can be deleted at any time without impacting any other part + of the system. + +Fossil chat is designed for use by insiders - people with check-in +privileges or higher. It is not intended as a general-purpose gathering +place for random passers-by on the internet. +Fossil chat seeks to provide a communication venue for discussion +that does *not* become part of the permanent record for the project. +For persistent and durable discussion, use the [Forum](./forum.wiki). +Because the conversation is intended to be ephemeral, the chat messages +are local to a single repository. Chat content does not sync. + + +## Setup + +A Fossil repository must be functioning as a [server](./server/) in order +for chat to work. +To activate chat, simply add the [C capability](/setup_ucap_list) +to every user who is authorized to participate. Anyone who can read chat +can also post to chat. + +Setup ("s") and Admin ("a") users always have access to chat, without needing +the "C" capability. A common configuration is to add the "C" capability +to "Developer" so that any individual user who has the "v" capability will +also have access to chat. + +There are also some settings under /Admin/Chat that control the +behavior of chat, though the default settings are reasonable so in most +cases those settings can be ignored. The settings control things like +the amount of time that chat messages are retained before being purged +from the repository database. + +## <a id="usage"></a>Usage + +For users with appropriate permissions, simply browse to the +[/chat](/help/www/chat) to start up a chat session. The default +skin includes a "Chat" entry on the menu bar on wide screens for +people with chat privilege. There is also a "Chat" option on +the [Sitemap page](/sitemap), which means that chat will appear +as an option under the hamburger menu for many [skins](./customskin.md). + +Chat messages are subject to [Fossil's +full range of Markdown processing](/md_rules). Because chat messages are +stored as-is when they arrive from a client, this change applies +retroactively to messages stored by previous fossil versions. + +Files may be sent via chat using the file selection element at the +bottom of the page. If the desktop environment system supports it, +files may be dragged and dropped onto that element. Files are not +automatically sent - selection of a file can be cancelled using the +Cancel button which appears only when a file is selected. When the +Send button is pressed, any pending text is submitted along with the +selected file. Image files sent this way will, by default, appear +inline in messages, but each user may toggle that via the settings +popup menu, such that images instead appear as downloadable links. +Non-image files always appear in messages as download links. + +### Deletion of Messages + +<div class="sidebar">Message deletion is itself a type of message, which +is why deletions count towards updates in the recent activity list. (It +is counted for the person who performed the deletion, not the author of +the deleted comment.) That can potentially lead to odd corner cases +where a user shows up in the list but has no messages which are +currently visible because they were deleted, or an admin user who has +not posted anything but deleted a message. That is a known minor +cosmetic-only bug with a resolution of "will not fix."</div> + +Any user may *locally* delete a given message by clicking on the "tab" +at the top of the message and clicking the button which appears. Such +deletions are local-only, and the messages will reappear if the page +is reloaded. The user who posted a given message, or any Admin users, +may additionally choose to globally delete a message from the chat +record, which deletes it not only from their own browser but also +propagates the removal to all connected clients the next time they +poll for new messages. + +### <a id='notifications'></a>Customizing New-message Notification Sounds + +By default, the list of new-message notification sounds is limited to +a few built in to the fossil binary. In addition, any +[unversioned files](./unvers.wiki) named `alert-sounds/*.{mp3,wav,ogg}` +will be included in that list. To switch sounds, tap the "settings" +button. + +### <a id='connection'></a> Who's Online? + +Because the chat app has to be able to work over transient CGI-based +connections, as opposed to a stable socket connection to the server, +real-time tracking of "who's online" is not feasible. +Chat offers an optional feature, toggleable in the settings, +which can list users who have posted messages in the client's current +list of loaded messages. This is not the same thing as tracking who's +online, but it gives an overview of which users have been active most +recently, noting that "lurkers" (people who post no messages) will not +show up in that list, nor does the chat infrastructure have a way to +track and present those. That list can be used to filter messages on a +specific user by tapping on that user's name, tapping a second time to +remove the filter. + +### <a id="cli"></a> The `fossil chat` Command + +Type [fossil chat](/help/chat) from within any open check-out +to bring up a chatroom for the project that is in that checkout. +The new chat window will attempt to connect to the default sync +target for that check-out (the server whose URL is shown by the +[fossil remote](/help/remote) command). + +### <a id="robots"></a> Chat Messages From Robots + +The [fossil chat send](/help/chat) can be used by project-specific +robots to send notifications to the chatroom. For example, on the +[SQLite project](https://sqlite.org/) (for which the Fossil chatroom +feature, and indeed all of Fossil, was invented) there are long-running +fuzz servers that sometimes run across obscure problems. Whenever this +happens, a message is sent to the SQLite developers chatroom alerting +them to the problem. + +The recommended way to allow robots to send chat messages is to create +a new user on the server for each robot. Give each such robot account +the "C" privilege only. That means that the robot user account will be +able to send chat messages, but not do anything else. Then, in the +program or script that runs the robot, when it wants to send a chat +message, have it run a command like this: + +~~~~ +fossil chat send --remote https://robot:PASSWORD@project.org/fossil \ + --message 'MESSAGE TEXT' --file file-to-attach.txt +~~~~ + +Substitute the appropriate project URL, robot account +name and password, message text and file attachment, of course. + +### <a id="chat-robot"></a> Chat Messages For Timeline Events + +If the [chat-timeline-user setting](/help/chat-timeline-user) is not an +empty string, then any change to the repository that would normally result +in a new timeline entry is announced in the chatroom. The announcement +appears to come from a user whose name is given by the chat-timeline-user +setting. + +This mechanism is similar to [email notification](./alerts.md) except that +the notification is sent via chat instead of via email. + +## Quirks + + - There is no message-editing capability. This is by design and + desire of `/chat`'s developers. + + - When `/chat` has problems connecting to the message poller (see + the next section) it will provide a subtle visual indicator of the + connection problem - a dotted line along the top of the input + field. If the connection recovers within a short time, that + indicator will go away, otherwise it will pop up a loud message + signifying that the connection to the poller is down and how long + it will wait to retry (progressively longer, up to some + maximum). `/chat` will recover automatically when the server is + reachable. Trying to send messages while the poller connection is + down is permitted, and the poller will attempt to recover + immediately if sending of a message succeeds. That applies to any + operations which send traffic, e.g. if the per-message "toggle + text mode" button is activated or a message is globally deleted. + +## Implementation Details + +*You do not need to understand how Fossil chat works in order to use it. +But many developers prefer to know how their tools work. +This section is provided for the benefit of those curious developers.* + +The [/chat](/help/www/chat) webpage downloads a small amount of HTML +and a small amount of javascript to run the chat session. The +javascript uses XMLHttpRequest (XHR) to download chat content, post +new content, or delete historical messages. The following web +interfaces are used by the XHR: + + * [/chat-poll](/help?name=/chat-poll) → + Downloads chat content as JSON. + Chat messages are numbered sequentially. + The client tells the server the latest chat message ID it currently + holds, and the server sends back subsequent messages. If there + are no subsequent messages, the /chat-poll page blocks until new + messages are available. + + * [/chat-send](/help?name=/chat-send) → + Sends a new chat message to the server. + + * [/chat-delete](/help?name=/chat-delete) → + Deletes a chat message. + +Fossil chat uses the venerable "hanging GET" or +"[long polling](wikipedia:/wiki/Push_technology#Long_polling)" +technique to receive asynchronous notification of new messages. +This is done because long polling works well with CGI and SCGI, +which are the usual mechanisms for setting up a Fossil server. +More advanced notification techniques such as +[Server-sent events](wikipedia:/wiki/Server-sent_events) and especially +[WebSockets](wikipedia:/wiki/WebSocket) might seem more appropriate for +a chat system, but those technologies are not compatible with CGI. + +Downloading of posted files and images uses a separate, non-XHR interface: + + * [/chat-download](/help?name=/chat-download) → + Fetches the file content associated with a post (one file per + post, maximum). In the UI, this is accessed via links to uploaded + files and via inlined image tags. + +Chat messages are stored on the server-side in the CHAT table of +the repository. + +~~~ +CREATE TABLE repository.chat( + msgid INTEGER PRIMARY KEY AUTOINCREMENT, + mtime JULIANDAY, -- Time for this entry - Julianday Zulu + lmtime TEXT, -- Client YYYY-MM-DDZHH:MM:SS when message originally sent + xfrom TEXT, -- Login of the sender + xmsg TEXT, -- Raw, unformatted text of the message + fname TEXT, -- Filename of the uploaded file, or NULL + fmime TEXT, -- MIMEType of the upload file, or NULL + mdel INT, -- msgid of another message to delete + file BLOB -- Text of the uploaded file, or NULL +); +~~~ + +The CHAT table is not cross-linked with any other tables in the repository +schema. An administrator can "DROP TABLE chat;" at any time, without +harm (apart from deleting all chat history, of course). The CHAT table +is dropped when running [fossil scrub --verily](/help/scrub). + +On the server-side, message text is stored exactly as entered by the +users. The /chat-poll page queries the CHAT table and constructs a +JSON reply described in the [/chat-poll +documentation](/help/www/chat-poll). The message text is translated +into HTML before being converted to JSON so that the text can be +safely added to the display using assignment to `innerHTML`. Though +`innerHTML` assignment is generally considered unsafe, it is only so +with untrusted content from untrusted sources. The chat content goes +through sanitization steps which eliminate any potential security +vulnerabilities of assigning that content to `innerHTML`. diff --git a/www/checkin.wiki b/www/checkin.wiki new file mode 100644 index 0000000..88266b8 --- /dev/null +++ b/www/checkin.wiki @@ -0,0 +1,89 @@ +<title>Check-in Checklist</title> + +<h2><u>Always</u> run the following checklist prior to <u>every</u> +check-in or commit to the Fossil repository:</h2> + +Before every check-in: + + 0. <b>fossil user default</b> → your username is correct. + + 1. <b>fossil diff</b> → + <ol type="a"> + <li> No stray changes + <li> All changes comply with the license + <li> All inputs are scrubbed before use + <li> No injection attacks via %s formats + </ol> + + 2. <b>fossil extra</b> → no unmanaged files need to be added. + + 3. The check-in will go onto the desired branch. + → Check-ins to trunk normally require approval from + the lead programmer (drh). + + 4. auto-sync is on, or the system clock is verified + + 5. If sources files have been added or removed, ensure all makefiles + and configure scripts have been updated accordingly. + +Before every check-in to <b>trunk</b>: + + 6. No compiler warnings on the development machine. + + 7. The fossil executable that results from a build actually works. + + +<hr> +<h2>Commentary</h2> + +Before you go ahead and push content back to the servers, make sure +that the username you are using by default matches your username +within the project. Also remember to enable the localauth setting +if you intend to make changes via a locally served web UI. + +Item 1 is the most important step. Consider using <b>gdiff</b> +instead of <b>diff</b> if you have a graphical differ configured. Or +use the command-line option <b>--tk</b>. Also consider the <b>-v</b> +command-line option to show the complete text of newly added files. +The recommended command for completing checklist item 1 is: + + <b>fossil diff --tk -v</b> + +Look carefully at every changed line in item 1. +Make sure that you are not about to commit unrelated changes. +If there are two or more unrelated changes present, consider +breaking up the commit into two or more separate commits. +Always make 100% sure that all changes are compatible with the +BSD license, that you have the authority to commit the code in accordance +with the [/doc/trunk/www/copyright-release.html | CLA] that you have +signed and have on file, and that +no NDAs, copyrights, patents, or trademarks are infringed by the check-in. +Also check very carefully to make sure that +you are not introducing security vulnerabilities. Pay particular attention +to the possibility of SQL or HTML injection attacks. + +Item 2 verifies that you have not added source files but failed to +do the necessary "<b>fossil add</b>" to manage them. If you commit +without bringing the new file under source control, the check-in will +be broken. That, in turn, can cause complications far in the future +when we are bisecting for a bug. + +For item 3, Run "<b>fossil status</b>" or the equivalent to +make sure your changes are going into the branch you think they are. +Also double-check the branch name when entering change comments. +Never check into trunk unless you are expressly authorized to do so. + +For Item 4, if you are on-network, make sure auto-sync is enabled. This +will minimize the risk of an unintended fork. It will also give you a +warning if you system clock is set incorrectly. If you are off-network, +make sure that your system clock is correct or at least close to correct +so that your check-in does not appear out-of-sequence on timelines. +On-network commits are preferred, especially for trunk commits. + +Items 6 and 7 help to ensure that check-ins on the trunk always work. +Knowing that the trunk always works makes bisecting much easier. Items +6 and 7 are recommended for all check-ins, even those that are on a branch. +But they are especially important for trunk check-ins. We desire that +all trunk check-ins work at all times. Any experimental, unstable, or +questionable changes should go on a branch and be merged into trunk after +further testing. diff --git a/www/checkin_names.wiki b/www/checkin_names.wiki new file mode 100644 index 0000000..f3d8758 --- /dev/null +++ b/www/checkin_names.wiki @@ -0,0 +1,316 @@ +<title>Check-in Names</title> + +<div class="sidebar no-label"> +<b>Quick Reference</b> +<ul> +<li> Hash prefix +<li> Branch name +<li> Tag name +<li> Timestamp: <i>YYYY-MM-DD HH:MM:SS</i> +<li> <i>tag-name</i> <big><b>:</b></big> <i>timestamp</i> +<li> <b>root <big>:</big></b> <i>branchname</i> +<li> <b>start <big>:</big></b> <i>branchname</i> +<li> <b>merge-in <big>:</big></b> <i>branchname</i> +<li> Special names: +<ul> +<li> <b>tip</b> +<li> <b>current</b> +<li> <b>next</b> +<li> <b>previous</b> or <b>prev</b> +<li> <b>ckout</b> (<a href='./embeddeddoc.wiki'>embedded docs</a> only) +</ul> +</ul> +</div> + +Many Fossil [/help|commands] and [./webui.wiki | web interface] URLs accept +check-in names as an argument. For example, the "[/help/info|info]" command +accepts an optional check-in name to identify the specific check-in +about which information is desired: + +<pre style="white-space: pre-wrap"> +fossil info <i>checkin-name</i> +</pre> + +You are perhaps reading this page from the following URL: + +<verbatim> +https://fossil-scm.org/home/doc/trunk/www/checkin_names.wiki +</verbatim> + +This is an example of an [./embeddeddoc.wiki | embedded documentation] +page URL. The "trunk" element of the pathname is a +[./glossary.md#check-in | check-in] name that +determines which version of the documentation to display. + +Fossil provides a variety of ways to specify a check-in. This +document describes the various methods. + +<h2 id="canonical">Canonical Check-in Name</h2> + +The canonical name of a check-in is the hash of its +[./fileformat.wiki#manifest | manifest] expressed as a +[./hashes.md | long lowercase hexadecimal number]. For example: + +<pre> +fossil info e5a734a19a9826973e1d073b49dc2a16aa2308f9 +</pre> + +The full 40 or 64 character hash is unwieldy to remember and type, though, +so Fossil also accepts a unique prefix of the hash, using any combination +of upper and lower case letters, as long as the prefix is at least 4 +characters long. Hence the following commands all +accomplish the same thing as the above: + +<pre> +fossil info e5a734a19a9 +fossil info E5a734A +fossil info e5a7 +</pre> + +Fossil uses this feature itself, identifying check-ins by 8 to 16-character +prefixes of the canonical name in places where it doesn't want to chew +up the screen real estate required to display the whole hash. + +<h2 id="tags">Tags And Branch Names</h2> + +Using a tag or branch name where a check-in name is expected causes +Fossil to choose the most recent check-in with that tag or branch name. +So for example, the most recent check-in that +is tagged with "release" as of this writing is [b98ce23d4fc]. +The command: + +<pre> +fossil info release +</pre> + +…results in the following output: + +<pre> +hash: b98ce23d4fc3b734cdc058ee8a67e6dad675ca13 2020-08-20 13:27:04 UTC +parent: 40feec329163103293d98dfcc2d119d1a16b227a 2020-08-20 13:01:51 UTC +tags: release, branch-2.12, version-2.12.1 +comment: Version 2.12.1 (user: drh) +</pre> + +There are multiple check-ins that are tagged with "release" but +(as of this writing) the [b98ce23d4fc] +check-in is the most recent so it is the one that is selected. + +Note that unlike some other version control systems, a "branch" in Fossil +is not anything special: it is simply a sequence of check-ins that +share a common tag, so the same mechanism that resolves tag names +also resolves branch names. + +<a id="tagpfx"></a> +Note also that there can — in theory, if rarely in practice — be an ambiguity +between tag names and canonical names. Suppose, for example, you had a +check-in with the canonical name deed28aa99… and you +also happened to have tagged a different check-in with "deed2". If +you use the "deed2" name, does it choose the canonical name or the tag +name? In such cases, you can prefix the tag name with "tag:". +For example: + +<pre> +fossil info tag:deed2 +</pre> + +The "tag:deed2" name will refer to the most recent check-in +tagged with "deed2" rather than the +check-in whose canonical name begins with "deed2". + +<h2 id="whole-branches">Whole Branches</h2> + +Usually when a branch name is specified, it means the latest check-in on +that branch, but for some commands (ex: [/help/purge|purge]) a branch name +on the argument means the earliest connected check-in on the branch. This +seems confusing when being explained here, but it works out to be intuitive +in practice. + +For example, the command "fossil purge XYZ" means to purge the check-in XYZ +and all of its descendants. But when XYZ is in the form of a branch name, one +generally wants to purge the entire branch, not just the last check-in on the +branch. And so for this reason, commands like purge will interpret a branch +name to be the first check-in of the branch rather than the last. If there +are two or more branches with the same name, then these commands will select +the first check-in of the branch that has the most recent check-in. What +happens is that Fossil searches for the most recent check-in with the given +tag, just as it always does. But if that tag is a branch name, it then walks +back down the branch looking for the first check-in of that branch. + +Again, this behavior only occurs on a few commands where it make sense. + +<h2 id="timestamps">Timestamps</h2> + +A timestamp in one of the formats shown below means the most recent +check-in that occurs no later than the timestamp given: + + 1. <i>YYYY-MM-DD</i> + 2. <i>YYYY-MM-DD HH:MM</i> + 3. <i>YYYY-MM-DD HH:MM:SS</i> + 4. <i>YYYY-MM-DD HH:MM:SS.SSS</i> + 5. <i>YYYYMMDD</i> + 6. <i>YYYYMMDDHHMM</i> + 7. <i>YYYYMMDDHHMMSS</i> + +In the second through the fourth forms, +the space between the day and the year can optionally be +replaced by an uppercase <b>T</b>, and the entire timestamp can +optionally be followed by "<b>z</b>" or "<b>Z</b>". In the fourth +form with fractional seconds, any number of digits may follow the +decimal point, though due to precision limits only the first three +digits will be significant. The final three pure-digit forms +without punctuation are only valid if the number they encode is +not also the prefix of an artifact hash. + +In its default configuration, Fossil interprets and displays all dates +in Universal Coordinated Time (UTC). This tends to work the best for +distributed projects where participants are scattered around the globe. +But there is an option on the Admin/Timeline page of the web interface to +switch to local time. The "<b>Z</b>" suffix on a timestamp check-in +name is meaningless if Fossil is in the default mode of using UTC for +everything, but if Fossil has been switched to local time mode, then the +"<b>Z</b>" suffix means to interpret that particular timestamp using +UTC instead of local time. + +You may prefix a timestamp with the string “date:”, in which case +processing stops immediately, whether the string is parsed correctly and +refers to anything within the repository or not. The prefix is therefore +useful when the date could be misinterpreted as a tag. For example, a +repo could have release tags like “2020-04-01”, the date the release was +cut, but you could force Fossil to interpret that string as a date +rather than as a tag by passing “date:2020-04-01”. + +For an example of how timestamps are useful, +consider the homepage for the Fossil website itself: + +<pre> +https://fossil-scm.org/home/doc/<b>trunk</b>/www/index.wiki +</pre> + +The bold component of that URL is a check-in name. To see the stored content +of the Fossil website repository as of January 1, 2009, one has merely to change +the URL to the following: + +<pre> +https://fossil-scm.org/home/doc/<b>2009-01-01</b>/www/index.wiki +</pre> + +(Note that this won't roll you back to the <i>skin</i> and other +cosmetic configurations as of that date. It also won't change screens +like the timeline, which has an independent date selector.) + +<h2 id="tag-ts">Tag And Timestamp</h2> + +A check-in name can also take the form of a tag or branch name followed by +a colon and then a timestamp. The combination means to take the most +recent check-in with the given tag or branch which is not more recent than +the timestamp. So, for example: + +<pre> +fossil update trunk:2010-07-01T14:30 +</pre> + +Would cause Fossil to update the working check-out to be the most recent +check-in on the trunk that is not more recent than 14:30 (UTC) on +July 1, 2010. + +<h2 id="root">Root Of A Branch</h2> + +A branch name that begins with the "<tt>root:</tt>" prefix refers to the +last check-in on the parent branch prior to the beginning of the branch. +Such a label is useful, for example, in computing all diffs for a single +branch. The following example will show all changes in the hypothetical +branch "xyzzy": + +<pre> +fossil diff --from root:xyzzy --to xyzzy +</pre> + +<a id="merge-in"></a> +That doesn't do what you might expect after you merge the parent +branch's changes into the child branch: the above command will include +changes made on the parent branch as well. + +You can solve this by using the prefix "<tt>merge-in:</tt>" instead of +"<tt>root:</tt>" to tell Fossil to find +the most recent merge-in point for that branch. +The resulting diff will then show only the changes in +the branch itself, omitting +any changes that have already been merged in from the parent branch. + +<a id="start"></a> +The prefix "<tt>start:</tt>" gives the first check-in of the named branch. + +The prefixes "<tt>root:</tt>", "<tt>start:</tt>", and "<tt>merge-in:</tt>" +can be chained: one can say for example + +<pre> +fossil info merge-in:xyzzy:2022-03-01 +</pre> + +to get information about the most recent merge-in point on the branch +"xyzzy" that happened on or before March 1, 2022. + +<h2 id="special">Special Tags</h2> + +The tag "tip" means the most recent check-in. The "tip" tag is practically +equivalent to the timestamp "9999-12-31". + +This special name works anywhere you can pass a "NAME", such as with +<tt>/info</tt> URLs: + +<pre> +http://localhost:8080/info/tip +</pre> + +There are several other special names, but they only work from within a +check-out directory because they are relative to the current checked-out +version: + + * "current": the current checked-out version + * "next": the youngest child of the current checked-out version + * "previous" or "prev": the primary (non-merge) parent of "current" + +Therefore, you can use these names in a <tt>fossil info</tt> command, +but not in an <tt>/info</tt> URL, for example. + +For embedded documentation URLs only, there is one more special name, +"ckout". See [./embeddeddoc.wiki#ckout | its coverage elsewhere] for +more details. You cannot currently use "ckout" anywhere other than in +<tt>/doc</tt> URLs. + + +<h2 id="examples">Additional Examples</h2> + +To view the changes in the most recent check-in prior to the version currently +checked out: + +<pre> +fossil diff --from previous --to current +</pre> + +Suppose you are of the habit of tagging each release with a "release" tag. +Then to see everything that has changed on the trunk since the last release: + +<pre> +fossil diff --from release --to trunk +</pre> + + +<h2 id="order">Resolution Order</h2> + +Fossil currently resolves name strings to artifact hashes in the +following order: + + # Exact matches on [#special | the special names] + # [#timestamps | Timestamps], with preference to ISO8601 forms + # [#tagpfx | tag:TAGNAME] + # [#root | root:BRANCH] + # [#start | start:BRANCH] + # [#merge-in | merge-in:BRANCH] + # [#tag-ts | TAGNAME:timestamp] + # Full artifact hash or hash prefix. + # Any other type of symbolic name that Fossil extracts from + artifacts. + +<div style="height:50em" id="this-space-intentionally-left-blank"></div> diff --git a/www/childprojects.wiki b/www/childprojects.wiki new file mode 100644 index 0000000..ad38bec --- /dev/null +++ b/www/childprojects.wiki @@ -0,0 +1,57 @@ +<title>Child Projects</title> + +<h2>Background</h2> + +The default behavior of Fossil is to share everything (all check-ins, +tickets, wiki, etc) between all clients and all servers. Such a policy +helps to promote a cohesive design for a cathedral-style project run +by a small cliche of developers - the sort of project for which Fossil +was designed. + +But sometimes it is desirable to branch off a side project that does not +sync back to the master but does continue to track changes in the master. +For example, the master project might be an open-source project like +[https://www.sqlite.org/|SQLite] and a team might want to do a proprietary +closed-source enhancement to that master project in a separate repository. +All changes in the master project should flow forward into the derived +project, but care must be taken to prevent proprietary content from the +derived project from leaking back into the master. + +<h2>Child Projects</h2> + +A scenario such as the above can be accomplished in Fossil by creating +a child project. The child project is able to freely pull from the parent, +but the parent cannot push or pull from the child nor is the child able to +push to the parent. Content flows from parent to child only, and then only +at the request of the child. + +<h2>Creating a Child Project</h2> + +To create a new child project, first clone the parent. Then make manual +SQL changes to the child repository as follows: + +<verbatim> +UPDATE config SET name='parent-project-code' WHERE name='project-code'; +UPDATE config SET name='parent-project-name' WHERE name='project-name'; +INSERT INTO config(name,value) + VALUES('project-code',lower(hex(randomblob(20)))); +INSERT INTO config(name,value) + VALUES('project-name','CHILD-PROJECT-NAME'); +</verbatim> + +Modify the CHILD-PROJECT-NAME in the last statement to be the name of +the child project, of course. + +The repository is now a separate project, independent from its parent. +Clone the new project to the developers as needed. + +The child project and the parent project will not normally be able to sync +with one another, since they are now separate projects with distinct +project codes. However, if the +"--from-parent-project" command-line option is provided to the +"[/help/pull|fossil pull]" command in the child, and the URL of +parent repository is also provided on the command-line, then updates to +the parent project that occurred after the child was created will be added +to the child repository. Thus, by periodically doing a +pull --from-parent-project, the child project is able to stay up to date +with all the latest changes in the parent. diff --git a/www/chroot.md b/www/chroot.md new file mode 100644 index 0000000..882a176 --- /dev/null +++ b/www/chroot.md @@ -0,0 +1,45 @@ +# The Server Chroot Jail + +If you run Fossil as root in any mode that [serves data on the +network][srv], and you're running it on Unix or a compatible OS, Fossil +will drop itself into a [`chroot(2)` jail][cj] shortly after starting +up. The usual reason for launching Fossil +as root is to allow it to bind to TCP port 80 for HTTP +service, since normal users are restricted to ports 1024 and higher. + +Fossil uses the owner of the Fossil repository file as its new user +ID when it drops root privileges. + +When Fossil enters a chroot jail, it needs to have all of its dependencies +inside the chroot jail in order to continue work. There are several +resources that need to be inside the chroot jail with Fossil in order for +Fossil to work correctly: + +* the repository file(s) + +* `/dev/null` — create it with `mknod(8)` inside the jail directory + ([Linux example][mnl], [OpenBSD example][obsd]) + +* `/dev/urandom` — ditto + +* `/proc` — you might need to mount this virtual filesystem inside the + jail on Linux systems that make use of [Fossil’s server load + shedding feature][fls] + +* any shared libraries your `fossil` binary is linked to, unless you + [configured Fossil with `--static`][bld] to avoid it + +Fossil does all of this as one of many layers of defense against +hacks and exploits. You can prevent Fossil from entering the chroot +jail using the <tt>--nojail</tt> option to the +[fossil server command](/help/server) +but you cannot make Fossil hold onto root privileges. Fossil always drops +root privilege before accepting inputs, for security. + + +[bld]: https://fossil-scm.org/home/doc/trunk/www/build.wiki +[cj]: https://en.wikipedia.org/wiki/Chroot +[fls]: ./loadmgmt.md +[mnl]: https://fossil-scm.org/forum/forumpost/90caff30cb +[srv]: ./server/ +[obsd]: ./server/openbsd/fastcgi.md#chroot diff --git a/www/ckout-workflows.md b/www/ckout-workflows.md new file mode 100644 index 0000000..6c36183 --- /dev/null +++ b/www/ckout-workflows.md @@ -0,0 +1,142 @@ +# Check-Out Workflows + +Because Fossil separates the concept of “check-out directory” from +“repository DB file,” it gives you the freedom to choose from several +working styles. Contrast Git, where the two concepts are normally +intermingled in a single working directory, which strongly encourages +the “update in place” working style. + + +## <a id="mcw"></a> Multiple-Checkout Workflow + +With Fossil, it is routine to have multiple check-outs from the same +repository: + + fossil clone https://example.com/repo /path/to/repo.fossil + + mkdir -p ~/src/my-project/trunk + cd ~/src/my-project/trunk + fossil open /path/to/repo.fossil # implicitly opens “trunk” + + mkdir ../release + cd ../release + fossil open /path/to/repo.fossil release + + mkdir ../my-other-branch + cd ../my-other-branch + fossil open /path/to/repo.fossil my-other-branch + + mkdir ../scratch + cd ../scratch + fossil open /path/to/repo.fossil abcd1234 + + mkdir ../test + cd ../test + fossil open /path/to/repo.fossil 2019-04-01 + +Now you have five separate check-out directories: one each for: + +* trunk +* the latest tagged public release +* an alternate branch you’re working on +* a “scratch” directory for experiments you don’t want to do in the + other check-out directories; and +* a “test” directory where you’re currently running a long-running + test to evaluate a user bug report against the version as of last + April Fool’s Day. + +Each check-out operates independently of the others. + +This multiple-checkouts working style is especially useful when Fossil stores source code in programming languages +where there is a “build” step that transforms source files into files +you actually run or distribute. Contrast a switch-in-place workflow, +where you have to rebuild all outputs from the source files +that differ between those versions whenever you switch versions. In the above model, +you switch versions with a “`cd`” command instead, so that you only have +to rebuild outputs from files you yourself change. + +This style is also useful when a check-out directory may be tied up with +some long-running process, as with the “test” example above, where you +might need to run an hours-long brute-force replication script to tickle +a [Heisenbug][hb], forcing it to show itself. While that runs, you can +open a new terminal tab, “`cd ../trunk`”, and get back +to work. + +[hb]: https://en.wikipedia.org/wiki/Heisenbug + + + +## <a id="scw"></a> Single-Checkout Workflows + +Nevertheless, it is possible to work in a more typical Git sort of +style, switching between versions in a single check-out directory. + +#### <a id="idiomatic"></a> The Idiomatic Fossil Way + +The most idiomatic way is as follows: + + fossil clone https://example.com/repo /path/to/repo.fossil + mkdir work-dir + cd work-dir + fossil open /path/to/repo.fossil + ...work on trunk... + + fossil update my-other-branch + ...work on your other branch in the same directory... + +Basically, you replace the `cd` commands in the multiple checkouts +workflow above with `fossil up` commands. + + +#### <a id="open"></a> Opening a Repository by URI + +You can instead open the repo’s URI directly: + + mkdir work-dir + cd work-dir + fossil open https://example.com/repo + +Now you have “trunk” open in `work-dir`, with the repo file stored as +`repo.fossil` in that same directory. + +Users of Git may be surprised that it doesn’t create a directory for you +and that you `cd` into it *before* the clone-and-open step, not after. +This is because we’re overloading the “open” command, which already had +the behavior of opening into the current working directory. Changing it +to behave like `git clone` would therefore make the behavior surprising +to Fossil users. (See [our discussions][caod] if you want the full +details.) + + +#### <a id="clone"></a> Git-Like Clone-and-Open + +Fossil also supports a more Git-like alternative: + + fossil clone https://fossil-scm.org/fossil + cd fossil + +This results in a `fossil.fossil` repo DB file and a `fossil/` working +directory. + +Note that our `clone URI` behavior does not commingle the repo and +check-out, solving our major problem with the Git design. + +If you want the repo to be named something else, adjust the URL: + + fossil clone https://fossil-scm.org/fossil/fsl + +That gets you `fsl.fossil` checked out into `fsl/`. + +For sites where the repo isn’t served from a subdirectory like this, you +might need another form of the URL. For example, you might have your +repo served from `dev.example.com` and want it cloned as `my-project`: + + fossil clone https://dev.example.com/repo/my-project + +The `/repo` addition is the key: whatever comes after is used as the +repository name. [See the docs][clone] for more details. + +[caod]: https://fossil-scm.org/forum/forumpost/3f143cec74 +[clone]: /help/clone + +<div style="height:50em" id="this-space-intentionally-left-blank"></div> diff --git a/www/co-vs-up.md b/www/co-vs-up.md new file mode 100644 index 0000000..b0e74ca --- /dev/null +++ b/www/co-vs-up.md @@ -0,0 +1,31 @@ +# Checkout vs Update + +Fossil has two commands that look like they do the same thing on initial +examination, [`fossil update`][up] and [`fossil checkout`][co], but +there are several key differences: + +1. `fossil checkout` aborts if there are changed files in the local + directory unless you give the `--force` option, whereas + `fossil update` merges upstream changes with your local changes. + Since Fossil tends to follow the CVS command design, and CVS + popularized the [merge on update][cvsmu] workflow, we expect that + Fossil’s update behavior is more likely to be what you want. + +2. Update triggers an autosync attempt; checkout does not. + +3. Several features in `fossil update` do not exist in + `fossil checkout`, so developing a habit to type `fossil up` + means you’re more likely to have the features you want at hand. + +4. Inversely, the `fossil checkout --keep` feature doesn’t exist in + `fossil update`, but it’s a rarely-needed operation, so it doesn’t + provide a good reason to develop a habit of using `fossil checkout` + instead. + +In summary, these are two separate commands; neither is an alias for the +other. They overlap enough that they can be used interchangeably for +some use cases, but `update` is more powerful and more broadly useful. + +[co]: /help/checkout +[cvsmu]: http://web.mit.edu/gnu/doc/html/cvs_7.html#SEC37 +[up]: /help/update diff --git a/www/collisions.ipynb b/www/collisions.ipynb new file mode 100755 index 0000000..644786e --- /dev/null +++ b/www/collisions.ipynb @@ -0,0 +1,139 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "*Pull in the packages we need:*" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "require(stats)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Parameters\n", + "\n", + "* **cpd** - Checkins per day. Defaults to 20.\n", + "\n", + "* **days** - Number of days to simulate. Defaults to 10000, or roughly\n", + " 40 working years.\n", + "\n", + "* **winSz** - Size of the commit window as a fraction of `workSec`. Defaults\n", + " to 0.01%, which is roughly 3 seconds for the default 8-hour work day, a\n", + " a long-ish commit time for Fossil.\n", + "\n", + "* **workSec** - Seconds in working day, defaulting to 8 hours. This value\n", + " only affects the reporting output, not the results of the underlying\n", + " simulation. It's a scaling parameter to humanize the results.\n", + " \n", + "* **spread** - Adjustment factor in computing the standard deviation for our \n", + " normally-distributed random numbers. The default gives \"nice\" distributions,\n", + " spread over the working day with a low chance of generating values outside\n", + " the working day.\n", + " \n", + " The smaller this value gets (<4), the more spread out the checkins, and\n", + " so the lower the chance of collisions. You might want to decrease it a bit\n", + " to simulate early and late workers.\n", + " \n", + " As you increase this value (>4) you're simulating a work environment\n", + " where people tend to check their work in closer and closer to mid-day,\n", + " which increases the chance of collision." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "collisions <- function(\n", + " cpd = 20,\n", + " days = 10000,\n", + " winSz = 0.01 / 100,\n", + " workSec = 8 * 60 * 60,\n", + " spread = 4)\n", + "{\n", + " cat(\"Running simulation...\\n\")\n", + "\n", + " day = 0\n", + " collisions = 0\n", + " winSec = workSec * winSz\n", + " mean = workSec / 2\n", + " sd = workSec / spread\n", + "\n", + " while (day < days) {\n", + " # Create the commit time vector as random values in a normal\n", + " # distribution.\n", + " times = sort(rnorm(cpd, mean, sd))\n", + "\n", + " # Are there any pairs in the time vector that are within the\n", + " # passed window size? If so, that's a Fossil checkin collision.\n", + " i = 1\n", + " while (i < cpd) {\n", + " if (abs(times[i] - times[i + 1]) < winSec) {\n", + " collisions = collisions + 1\n", + " }\n", + " i = i + 1\n", + " }\n", + " \n", + " day = day + 1\n", + " }\n", + " \n", + " cat(\"Found\", collisions, \"collisions in\", days, (workSec / 3600),\n", + " \"hour working days with\", winSec, \"second windows.\\n\")\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "*Run the following cell, possibly changing parameters documented above:*" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Running simulation...\n", + "Found 422 collisions in 10000 8 hour working days with 2.88 second windows.\n" + ] + } + ], + "source": [ + "collisions()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "R", + "language": "R", + "name": "ir" + }, + "language_info": { + "codemirror_mode": "r", + "file_extension": ".r", + "mimetype": "text/x-r-source", + "name": "R", + "pygments_lexer": "r", + "version": "3.3.2" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/www/colordiff.md b/www/colordiff.md new file mode 100644 index 0000000..5d321e5 --- /dev/null +++ b/www/colordiff.md @@ -0,0 +1,104 @@ +# Colorized Diffs + +The oldest and most widely compatible method to get colorized diffs in +Fossil is to use its web UI: + + fossil ui --page '/vdiff?from=2024-04-01&to=trunk' + +That syntax is admittedly awkward, and it doesn’t work where “from” is +the current checkout. Fortunately, there are many other methods to get +colorized `diff` output from Fossil. + + +<a id="ui"></a> +## `fossil diff -b` + +This produces a graphical diff in HTML format and sends it to the +user’s default web browser for viewing. + + + +<a id="ui"></a> +## `fossil diff -tk` + +You may be surprised to learn that the prior feature doesn’t use any of +the skinning or chrome from Fossil UI. This is because it is meant as a +functional replacement for an older method of getting colorized diffs, +“`fossil diff -tk`”. The feature was added after Apple stopped shipping +Tcl/Tk in macOS, and the third-party replacements often failed to work +correctly. It’s useful on other platforms as well. + + +<a id="git"></a> +## Delegate to Git + +It may be considered sacrilege by some, but the most direct method for +those who want Git-like diff behavior may be to delegate diff behavior +to Git: + + fossil set --global diff-command 'git diff --no-index' + +The flag permits it to diff files that aren’t inside a Git repository. + + +<a id="diffutils"></a> +## GNU Diffutils + +If your system is from 2016 or later, it may include [GNU Diffutils][gd] +3.4 or newer, which lets you say: + + fossil set --global diff-command 'diff -dwu --color=always' + +You might think you could give `--color=auto`, but that fails with +commands like “`fossil diff | less`” since the pipe turns the output +non-interactive from the perspective of the underlying `diff` instance. + +This use of unconditional colorization means you will then have to +remember to add the `-i` option to `fossil diff` commands when producing +`patch(1)` files or piping diff output to another command that doesn’t +understand ANSI escape sequences, such as [`diffstat`][ds]. + +[ds]: https://invisible-island.net/diffstat/ +[gd]: https://www.gnu.org/software/diffutils/ + + +<a id="bat"></a> +## Bat, the Cat with Wings + +We can work around the `--color=auto` problem by switching from GNU less +as our pager to [`bat`][bat], as it can detect GNU diff output and +colorize it for you: + + fossil set --global diff-command 'diff -dwu --color=auto' + fossil diff | bat + +In this author’s experience, that works a lot more reliably than GNU +less’s ANSI color escape code handling, even when you set `LESS=-R` in +your environment. + +The reason we don’t leave the `diff-command` unset in this case is that +Fossil produces additional lines at the start which confuse the diff +format detection in `bat`. Forcing output through an external diff +command solves that. It also means that if you forget to pipe the output +through `bat`, you still get colorized output from GNU diff. + +[bat]: https://github.com/sharkdp/bat + + +<a id="colordiff"></a> +## Colordiff + +A method that works on systems predating GNU diffutils 3.4 or the +widespread availability of `bat` is to install [`colordiff`][cdurl], as +it is included in [many package systems][cdpkg], including ones for +outdated OSes. That then lets you say: + + fossil set --global diff-command 'colordiff -dwu' + +The main reason we list this alternative last is that it has the same +limitation of unconditional color as [above](#diffutils). + +[cdurl]: https://www.colordiff.org/ +[cdpkg]: https://repology.org/project/colordiff/versions + +<div style="height:50em" id="this-space-intentionally-left-blank"></div> diff --git a/www/concepts.wiki b/www/concepts.wiki new file mode 100644 index 0000000..56bd522 --- /dev/null +++ b/www/concepts.wiki @@ -0,0 +1,472 @@ +<title>Fossil Concepts</title> + +<h2>1.0 Introduction</h2> + +[./index.wiki | Fossil] is a +[http://en.wikipedia.org/wiki/Software_configuration_management | software configuration management] system. +Fossil is software that is designed to control and track the +development of a software project and to record the history +of the project. +There are many such systems in use today. Fossil strives to +distinguish itself from the others by being extremely simple +to setup and operate. + +This document is intended as a quick introduction to the concepts +behind Fossil. + +See also: + + * [./glossary.md | Glossary] + * [./quickstart.wiki | Quick start guide] + +<h2>2.0 Composition Of A Project</h2> + +<verbatim type="pikchr float-right"> +R1: cylinder "Remote" "Repository" fill 0xadd8e6 rad 70% +R2: cylinder same "Remote" "Repository" at 2.5*R1.wid right of R1 + spline <-> from R1.e to 0.6<R1.se,R2.sw> then to 0.4<R1.ne,R2.nw> then to R2.w + text "HTTPS" at .5<R1.ne,R2.nw> +R3: cylinder same "Local" "Repository" fill 0x90ee90 \ + at dist(R1.e,R2.w) below .5<R1,R2> + spline <-> from .5<R1.s,R1.se> to 0.6<R1.s,R3.w> to 0.5<R1.se,R3.n> to .5<R3.nw,R3.n> \ + "HTTPS" above behind R1 + spline <-> from R2.sw to .6<R2.sw,R3.n> to .5<R2.s,R3.e> to R3.ne "HTTPS" ljust +T1: line from 1.0cm heading 200 from R3.sw go 2.2cm heading 150 then 2.2cm west close \ + fill 0xffff00 "Local" below "Source Tree" below +T2: line from 1.0cm heading 160 from R3.se same "Local" below "Source Tree" below + line <-> from R3.sw to T1.start + line <-> from R3.se to T2.start +</verbatim> + +A software project normally consists of a "source tree". +A source tree is a hierarchy of files that are used to generate +the end product. The source tree changes over time as the +software grows and expands and as features are added and bugs +are fixed. A snapshot of the source tree at any point in time +is called a "version" or "revision" or a "baseline" of the product. +In Fossil, we use the name "check-in". + +A "repository" is a database that contains copies of all historical +check-ins for a project. Check-ins are normally stored in the +repository in a highly space-efficient compressed format (delta encoding). +But that is an implementation detail that you the user need not worry over. +Think of the repository as a safe place where all your old check-ins are +securely stored away and available for retrieval whenever you need +them. + +A repository in Fossil is a single file on your disk. This file +might be rather large (dozens or hundreds of megabytes for a large +or long running project) but it is nevertheless just a file. You +can move it around, rename it, write it out to a memory stick, or +do anything else you normally do with files. + +Each source tree that is controlled by Fossil is associated with +a single repository on the local disk drive. You can tie two or more +source trees to a single repository if you want (though one +tree per repository is the most common configuration.) So a +single repository can be associated with many source trees, but +each source tree is associated with only one repository. + +Fossil source trees may not overlap. A Fossil source tree is identified +by a file named "_FOSSIL_" (or ".fslckout", but this article will always +use the name "_FOSSIL_") in the root directory of the source tree. Every +file that is a sibling of _FOSSIL_ and every file in every subfolder is +considered potentially a part of the source tree. The _FOSSIL_ file +contains (among other things) the pathname of the repository with which +the source tree is associated. On the other hand, the repository has +no record of its source trees. So you are free to delete a source tree +or move it around without consequence. But if you move or rename or +delete a repository, then any source trees associated with that repository +will no longer be able to locate their repository and will stop working. + +When multiple developers are working on the same project, each +developer typically has his or her own local repository and an associated +source tree in which to work. Developers share their work by +"syncing" the content of their local repositories either directly +or through a central server. Changes can "push" from the local +repository into a remote repository. Or changes can "pull" from a +remote repository into a local repository. Or one can do a "sync" +which is a shortcut for doing both a push and a pull at the same time. +Fossil also has the concept of "cloning". A "clone" is like a "pull", +except that instead of beginning with an existing local repository, +a clone begins with nothing and creates a new local repository that +is a duplicate of a remote repository. + +Communication between repositories is normally via HTTPS. (SSH is also +supported, as is unencrypted HTTP.) Remote +repositories are identified by URL. You can also point a web browser +at a repository and get human-readable status, history, and tracking +information about the project. + +<h3 id="artifacts">2.1 Identification Of Artifacts</h3> + +A particular version of a particular file is called an "artifact". Each +artifact has a universally unique name which is the <a +href="http://en.wikipedia.org/wiki/SHA1">SHA1</a> or <a +href="http://en.wikipedia.org/wiki/SHA3">SHA3-256</a> hash of the +content of that file expressed as either 40 or 64 characters of +lower-case hexadecimal. (See the [./hashpolicy.wiki|hash policy +document] for information on which algorithm is used, when.) Such a hash +is referred to as the Artifact ID. These hash algorithms were created +with Fossil's purpose in mind: to provide a highly forgery-resistant +identifier for a blob of data, such as a file. Given any file, it is +simple to find the artifact ID for that file. But given an artifact ID, +it is computationally intractable to generate a file that will have that +same artifact ID. + +Artifact IDs look something like this: + +<pre> +6089f0b563a9db0a6d90682fe47fd7161ff867c8 +59712614a1b3ccfd84078a37fa5b606e28434326 +19dbf73078be9779edd6a0156195e610f81c94f9 +b4104959a67175f02d6b415480be22a239f1f077 +997c9d6ae03ad114b2b57f04e9eeef17dcb82788 +</pre> + +When referring to an artifact using Fossil, you can use a unique +prefix of the artifact ID that is four characters or longer. This saves +a lot of typing. When displaying artifact IDs, Fossil will usually only +show the first 10 digits since that is normally enough to uniquely +identify a file. + +Changing (or adding or removing) a single byte in a file results +in a completely different artifact ID. And since the artifact ID is the name of +the artifact, making any change to a file results in a new artifact. +In this way, artifacts are immutable. + +A repository is really just an unordered collection of +artifacts. New artifacts can be added to the repository, but +existing artifacts can never be removed. (Well, almost never. There +is a [./shunning.wiki | "shunning"] mechanism that allows spam or other +inappropriate content to be removed if absolutely necessary, but such +removal is discouraged.) +Fossil is designed in +such a way that it can be handed a set of artifacts in any +order and it can figure out the relationship between those +artifacts and reconstruct the complete development history of +a software project. + +<h3>2.2 Manifests</h3> + +Associated with every check-in is a special file called the +[./fileformat.wiki#manifest| "manifest"]. The manifest is a +listing of all other files in +that source tree. The manifest contains the (complete) artifact ID +of the file and the name of the file as it appears on disk, +and thus serves as a mapping from artifact ID to disk name. The artifact ID +of the manifest is the identifier for the entire check-in. When +you look at a "timeline" of changes in Fossil, the ID associated +with each check-in or commit is really just the artifact ID of the +manifest for that check-in. + +The manifest file is not normally a real file on disk. Instead, +the manifest is computed in memory by Fossil whenever it needs it. +However, the "fossil setting manifest on" command will cause the +manifest file to be materialized to disk, if desired. Both Fossil +itself, and SQLite cause the manifest file to be materialized to disk +so that the makefiles for these projects can read the manifest and +embed version information in generated binaries. + +Fossil automatically generates a manifest whenever you "commit" +a new check-in. So this is not something that you, the developer, +need to worry with. The format of a manifest is intentionally +designed to be simple to parse, so that if +you want to read and interpret a manifest, either by hand or +with a script, that is easy to do. But you will probably never +need to do so. + +In addition to identifying all files in the check-in, a +manifest also contains a check-in comment, the date and time +when the check-in was established, who created the check-in, +and links to other check-ins from which the current check-in +is derived. There are also a couple of checksums used to verify +the integrity of the check-in. And the whole manifest might +be PGP clearsigned. + +<h3 id="keyconc">2.3 Key concepts</h3> + +<ul> +<li>A <b>check-in</b> is a set of files arranged + in a hierarchy.</li> +<li>A <b>repository</b> keeps a record of historical check-ins.</li> +<li>Repositories share their changes using <b>push</b>, <b>pull</b>, + <b>sync</b>, and <b>clone</b>.</li> +<li>A particular <u>version</u> of a particular file is an <b>artifact</b> + that is identified by an <b>artifact ID</b>.</li> +<li>Artifacts tracked by Fossil are inherently immutable.</li> +<li>Fossil automatically generates a <b>manifest</b> file that identifies + every artifact in a check-in.</li> +<li>The artifact ID of the manifest is the identifier of the check-in.</li> +</ul> + +<h2>3.0 Fossil - The Program</h2> + +Fossil is software. The implementation of Fossil is in the form +of a single executable named "fossil" (or "fossil.exe" on Windows). +To install Fossil on your system, +all you have to do is obtain a copy of this one executable file (either +by downloading a +<a href="https://fossil-scm.org/home/uv/download.html">pre-compiled version</a> +or [./build.wiki | compiling it yourself]) and then +putting that file somewhere on your PATH. + +Fossil is completely self-contained. It is not necessary to +install any other software in order to use Fossil. You do <u>not</u> need +CVS, gzip, diff, rsync, Python, Perl, Tcl, Java, Apache, PostgreSQL, MySQL, +SQLite, patch, or any similar software on your system in order to use +Fossil effectively. You will want to have some kind of text editor +for entering check-in comments. Fossil will use whatever text editor +is identified by your VISUAL environment variable. Fossil will also +use GPG to clearsign your manifests if you happen to have it installed, +but Fossil will skip that step if GPG is missing from your system. +You can optionally set up Fossil to use external "diff" programs, +though Fossil has an excellent built-in "diff" algorithm that works +fine for most people. If you happen to have Tcl/Tk installed on your +system, Fossil will use it to generate a graphical "diff" display when +you use the --tk option to the "diff" command, but this too is entirely +optional. + + +To uninstall Fossil, simply delete the executable. + +To upgrade an older version of Fossil to a newer version, just +replace the old executable with the new one. You might need to +run "<b>fossil all rebuild</b>" to restructure your repositories after +an upgrade. Running "all rebuild" never hurts, so when upgrading it +is a good policy to run it even if it is not strictly necessary. + +To use Fossil, simply type the name of the executable in your +shell, followed by one of the various built-in commands and +arguments appropriate for that command. For example: + +<pre>fossil help</pre> + +In the next section, when we say things like "use the <b>help</b> +command" we mean to use the command name "help" as the first +token after the name of the Fossil executable, as shown above. + +<h2 id="workflow">4.0 Workflow</h2> + +<verbatim type="pikchr float-right"> + down +R1: cylinder "Remote" "Repository" fill 0xadd8e6 rad 70% + move 150% +R2: cylinder same "Local" "Repository" fill 0x90ee90 + move 120% +T1: line go 2.2cm heading 150 then 2.2cm west close \ + fill 0xffff00 "Local" below "Source Tree" below + arrow from R2.n+(-0.25cm,+0.25cm) to R1.s+(-0.25cm,-0.25cm) \ + "push " rjust + arrow from R1.s+(+0.25cm,-0.25cm) to R2.n+(+0.25cm,+0.25cm) \ + " pull" ljust " clone" ljust + arrow from T1.start+(-0.25cm,+0cm) to R2.s+(-0.25cm,-0.25cm) \ + "commit " rjust + arrow from R2.s+(+0.25cm,-0.25cm) to T1.start+(+0.25cm,+0cm) \ + " open" ljust " update" ljust " merge" ljust +</verbatim> + + +Fossil has two modes of operation: <i>"autosync"</i> and +<i>"manual-merge"</i> +Autosync mode is reminiscent of CVS or SVN in that it automatically +keeps your changes in synchronization with your co-workers through +the use of a central server. The manual-merge mode is the standard workflow +for GIT or Mercurial in that your local repository develops +independently of your coworkers and you share and merge your changes manually. +An interesting feature of Fossil is that it supports both autosync +and manual-merge work flows. + +The default setting for Fossil is to be in autosync mode. You +can change the autosync setting or check the current autosync +setting using commands like: + +<pre> +fossil setting autosync on +fossil setting autosync off +fossil settings +</pre> + +By default, Fossil runs with autosync mode turned on. The +authors find that projects run more smoothly in autosync mode since +autosync helps to prevent pointless forking and merging and helps keeps +all collaborators working on exactly the same code rather than on their +own personal forks of the code. In the author's view, manual-merge mode +should be reserved for disconnected operation. + +<h3>4.1 Autosync Workflow</h3> + +<ol> +<li> +Establish a local repository using either the <b>new</b> command +to start a new project, or the <b>clone</b> command to make a clone +of a repository for an existing project. +</li> + +<li> +Establish one or more source trees using +the <b>open</b> command with the name of the repository file as its +argument. +</li> + +<li> +The <b>open</b> command in the previous step populates your local source +tree with a copy of the latest check-in. Usually this is what you want. +In the rare cases where it is not, use the <b>update</b> command to +switch to a different check-in. +Use the <b>timeline</b> or <b>leaves</b> commands +to identify alternative check-ins to switch to. +</li> + +<li> +Edit the code. Add new files to the source tree using the <b>add</b> +command. Omit files from future check-ins using the <b>rm</b> command. +(Even when you remove files from future check-ins, those files continue +to exist in historical check-ins.) Test your changes. +</li> + +<li> +Create a new check-in using the <b>commit</b> command. You will be prompted +for a check-in comment and also for your GPG key if you have GPG installed. +The commit copies the edits you have made in your local source +tree into your local repository. After your commit completes, Fossil will +automatically <b>push</b> your changes back to the server +you cloned from or whatever server you most recently synced with. +</li> + +<li> +When your coworkers make their own changes, you can merge those changes +into your local local source tree using the <b>update</b> command. +In autosync mode, <b>update</b> will first go back to the server you +cloned from or with which you most recently synced, and pull down all +recent changes into your local repository. Then it will merge recent +changes into your local source tree. If you do an <b>update</b> and +find that it messes something up in your source tree (perhaps a co-worker +checked in incompatible changes) you can use the <b>undo</b> command +to back out the changes. +</li> + +<li> +Repeat all of the above until you have generated great software. +</li> +</ol> + +<h3>4.2 Manual-Merge Workflow</h3> + +When autosync is disabled, the <b>commit</b> command is decoupled from +<b>push</b> and the <b>update</b> command is decoupled from <b>pull</b>. +That means you have to do a few extra steps in order to accomplish the +<b>push</b> and <b>pull</b> tasks manually. + +<ol> +<li> +Establish a local repository using either the <b>new</b> command +to start a new project, or the <b>clone</b> command to make a clone +of a repository for an existing project. The default setting for +a new repository is with autosync on, so you will need to turn it off +using the <b>setting autosync off</b> command with a <b>-R</b> option +to specify the repository. +</li> + +<li> +Establish one or more source trees by changing your working directory +to where you want the root of the source tree to be, then issuing +the <b>open</b> command with the name of the repository file as its +argument. +</li> + +<li> +The <b>open</b> command in the previous step populates your local source +tree with a copy of the latest check-in. Usually this is what you want. +In the rare cases where it is not, use the <b>update</b> command to +switch to a different check-in. +Use the <b>timeline</b> or <b>leaves</b> commands +to identify alternative check-ins to switch to. +</li> + +<li> +Edit the code. Add new files to the source tree using the <b>add</b> +command. Omit files from future check-ins using the <b>rm</b> command. +(Even when you remove files from future check-ins, those files continue +to exist in historical check-ins.) Test your changes. +</li> + +<li> +Create a new check-in using the <b>commit</b> command. You will be prompted +for a check-in comment and also for your GPG key if you have GPG installed. +The commit copies the edits you have made in your local source +tree into your local repository. +</li> + +<li> +Use the <b>push</b> command to push your changes out to a server +where your co-workers can access them. +</li> + +<li> +When co-workers make their own changes, use the <b>pull</b> command +to pull those changes into your local repository. Note that <b>pull</b> +does not move the changes into your local source tree, only into your +local repository. +</li> + +<li> +Once changes are in your local repository, use +the <b>update</b> command to merge them to your local source tree. +If you merge in some changes and find that the changes do not work out +or are not to your liking, you can back out the changes using the +<b>undo</b> command. +</li> + +<li> +If two or more people ran "commit" against the same check-in, this will +result in a [./branching.wiki | fork] which you may want to resolve by +running <b>merge</b> followed by another <b>commit</b>. +</li> + +<li> +Repeat all of the above until you have generated great software. +</li> +</ol> + +<h2>5.0 Setting Up A Fossil Server</h2> + +With other configuration management software, setting up a server is +a lot of work and normally takes time, patience, and a lot of system +knowledge. Fossil is designed to avoid this frustration. Setting up +a server with Fossil is ridiculously easy. You have four options: + + # <b>Stand-alone server.</b> + Simply run the [/help/server|fossil server] or + [/help/ui|fossil ui] command from the command-line. + <br><br> + # <b>CGI.</b> + Install a 2-line CGI script on a CGI-enabled web-server like Apache. + <br><br> + # <b>SCGI.</b> + Start an SCGI server using the + [/help/server| fossil server --scgi] command for handling + SCGI requests from web-servers like Nginx. + <br><br> + # <b>Inetd or Stunnel.</b> + Configure programs like inetd, xinetd, or stunnel to hand off HTTP requests + directly to the [/help/http|fossil http] command. + +See the [./server/ | How To Configure A Fossil Server] document +for details. + +<h2>6.0 Review Of Key Concepts</h2> + +<ul> +<li>The <b>fossil</b> program is a self-contained stand-alone executable. + Just put it somewhere on your PATH to install it.</li> +<li>Use the <b>clone</b> or <b>new</b> commands to create a new repository.</li> +<li>Use the <b>open</b> command to create a new source tree.</li> +<li>Use the <b>add</b> and <b>rm</b> or <b>delete</b> commands to add and + remove files from the local source tree.</li> +<li>Use the <b>commit</b> command to create a new check-in.</li> +<li>Use the <b>update</b> command to merge in changes from others.</li> +<li>The <b>push</b> and <b>pull</b> commands can be used to share changes + manually, but these things happen automatically in the default + autosync mode.</li> +</ul> diff --git a/www/contact.md b/www/contact.md new file mode 100644 index 0000000..75e62ed --- /dev/null +++ b/www/contact.md @@ -0,0 +1,14 @@ +# Contact Information + +## Questions, Suggestions, and Bug Reports + +The developers for Fossil monitor the [Fossil Forum][1]. Post there +with questions, improvement suggestions, and/or bug reports. + +[1]: https://fossil-scm.org/forum/forum + +## Security Problems and Vulnerabilities + +If you think you have discovered a security vulnerability in Fossil +and want to report the problem privately, send +email to "drh at sqlite dot org". diff --git a/www/containers.md b/www/containers.md new file mode 100644 index 0000000..ac5f02f --- /dev/null +++ b/www/containers.md @@ -0,0 +1,996 @@ +# OCI Containers + +This document shows how to build Fossil into [OCI] compatible containers +and how to use those containers in interesting ways. We start off using +the original and still most popular container development and runtime +platform, [Docker], but since you have more options than that, we will +show some of these options later on. + +[Docker]: https://www.docker.com/ +[OCI]: https://opencontainers.org/ + + +## 1. Quick Start + +Fossil ships a `Dockerfile` at the top of its source tree, +[here][DF], which you can build like so: + + $ docker build -t fossil . + +If the image built successfully, you can create a container from it and +test that it runs: + + $ docker run --name fossil -p 9999:8080/tcp fossil + +This shows us remapping the internal TCP listening port as 9999 on the +host. This feature of OCI runtimes means there’s little point to using +the “`fossil server --port`” feature inside the container. We can let +Fossil default to 8080 internally, then remap it to wherever we want it +on the host instead. + +Our stock `Dockerfile` configures Fossil with the default feature set, +so you may wish to modify the `Dockerfile` to add configuration options, +add APK packages to support those options, and so forth. + +The Fossil `Makefile` provides two convenience targets, +“`make container-image`” and “`make container-run`”. The first creates a +versioned container image, and the second does that and then launches a +fresh container based on that image. You can pass extra arguments to the +first command via the Makefile’s `DBFLAGS` variable and to the second +with the `DCFLAGS` variable. (DB is short for “`docker build`”, and DC +is short for “`docker create`”, a sub-step of the “run” target.) +To get the custom port setting as in +second command above, say: + + $ make container-run DCFLAGS='-p 9999:8080/tcp' + +Contrast the raw “`docker`” commands above, which create an +_unversioned_ image called `fossil:latest` and from that a container +simply called `fossil`. The unversioned names are more convenient for +interactive use, while the versioned ones are good for CI/CD type +applications since they avoid a conflict with past versions; it lets you +keep old containers around for quick roll-backs while replacing them +with fresh ones. + +[DF]: /file/Dockerfile + + +## 2. <a id="storage"></a>Repository Storage Options + +If you want the container to serve an existing repository, there are at +least two right ways to do it. + +The wrong way is to use the `Dockerfile COPY` command, because by baking +the repo into the image at build time, it will become one of the image’s +base layers. The end result is that each time you build a container from +that image, the repo will be reset to its build-time state. Worse, +restarting the container will do the same thing, since the base image +layers are immutable. This is almost certainly not what you +want. + +The correct ways put the repo into the _container_ created from the +_image_, not in the image itself. + + +### <a id="repo-inside"></a> 2.1 Storing the Repo Inside the Container + +The simplest method is to stop the container if it was running, then +say: + + $ docker cp /path/to/my-project.fossil fossil:/museum/repo.fossil + $ docker start fossil + $ docker exec fossil chown -R 499 /museum + +That copies the local Fossil repo into the container where the server +expects to find it, so that the “start” command causes it to serve from +that copied-in file instead. Since it lives atop the immutable base +layers, it persists as part of the container proper, surviving restarts. + +Notice that the copy command changes the name of the repository +database. The container configuration expects it to be called +`repo.fossil`, which it almost certainly was not out on the host system. +This is because there is only one repository inside this container, so +we don’t have to name it after the project it contains, as is +traditional. A generic name lets us hard-code the server start command. + +If you skip the “chown” command above and put “`http://localhost:9999/`” +into your browser, expecting to see the copied-in repo’s home page, you +will get an opaque “Not Found” error. This is because the user and group +ID of the file will be that of your local user on the container’s host +machine, which is unlikely to map to anything in the container’s +`/etc/passwd` and `/etc/group` files, effectively preventing the server +from reading the copied-in repository file. 499 is the default “`fossil`” +user ID inside the container, causing Fossil to run with that user’s +privileges after it enters the chroot. (See [below](#args) for how to +change this default.) You don’t have to restart the server after fixing +this with `chmod`: simply reload the browser, and Fossil will try again. + + +### 2.2 <a id="bind-mount"></a>Storing the Repo Outside the Container + +The simple storage method above has a problem: containers are +designed to be killed off at the slightest cause, rebuilt, and +redeployed. If you do that with the repo inside the container, it gets +destroyed, too. The solution is to replace the “run” command above with +the following: + + $ docker run \ + --publish 9999:8080 \ + --name fossil-bind-mount \ + --volume ~/museum:/museum \ + fossil + +Because this bind mount maps a host-side directory (`~/museum`) into the +container, you don’t need to `docker cp` the repo into the container at +all. It still expects to find the repository as `repo.fossil` under that +directory, but now both the host and the container can see that repo DB. + +Instead of a bind mount, you could instead set up a separate +[volume](https://docs.docker.com/storage/volumes/), at which point you +_would_ need to `docker cp` the repo file into the container. + +Either way, files in these mounted directories have a lifetime +independent of the container(s) they’re mounted into. When you need to +rebuild the container or its underlying image — such as to upgrade to a +newer version of Fossil — the external directory remains behind and gets +remapped into the new container when you recreate it with `--volume/-v`. + + +#### 2.2.1 <a id="wal-mode"></a>WAL Mode Interactions + +You might be aware that OCI containers allow mapping a single file into +the repository rather than a whole directory. Since Fossil repositories +are specially-formatted SQLite databases, you might be wondering why we +don’t say things like: + + --volume ~/museum/my-project.fossil:/museum/repo.fossil + +That lets us have a convenient file name for the project outside the +container while letting the configuration inside the container refer to +the generic “`/museum/repo.fossil`” name. Why should we have to name +the repo generically on the outside merely to placate the container? + +The reason is, you might be serving that repo with [WAL mode][wal] +enabled. If you map the repo DB alone into the container, the Fossil +instance inside the container will write the `-journal` and `-wal` files +alongside the mapped-in repository inside the container. That’s fine as +far as it goes, but if you then try using the same repo DB from outside +the container while there’s an active WAL, the Fossil instance outside +won’t know about it. It will think it needs to write *its own* +`-journal` and `-wal` files *outside* the container, creating a high +risk of [database corruption][dbcorr]. + +If we map a whole directory, both sides see the same set of WAL files. +[Testing](https://tangentsoft.com/sqlite/dir/walbanger?ci=trunk) +gives us a reasonable level of confidence that using WAL across a +container boundary is safe when used in this manner. + +[dbcorr]: https://www.sqlite.org/howtocorrupt.html#_deleting_a_hot_journal +[wal]: https://www.sqlite.org/wal.html + + +## 3. <a id="security"></a>Security + +### 3.1 <a id="chroot"></a>Why Not Chroot? + +Prior to 2023.03.26, the stock Fossil container relied on [the chroot +jail feature](./chroot.md) to wall away the shell and other tools +provided by [BusyBox]. It included that as a bare-bones operating system +inside the container on the off chance that someone might need it for +debugging, but the thing is, Fossil is self-contained, needing none of +that power in the main-line use cases. + +Our weak “you might need it” justification collapsed when we realized +you could restore this basic shell environment with a one-line change to +the `Dockerfile`, as shown [below](#run). + +[BusyBox]: https://www.busybox.net/BusyBox.html + + +### 3.2 <a id="caps"></a>Dropping Unnecessary Capabilities + +The example commands above create the container with [a default set of +Linux kernel capabilities][defcap]. Although Docker strips away almost +all of the traditional root capabilities by default, and Fossil doesn’t +need any of those it does take away, Docker does leave some enabled that +Fossil doesn’t actually need. You can tighten the scope of capabilities +by adding “`--cap-drop`” options to your container creation commands. + +Specifically: + +* **`AUDIT_WRITE`**: Fossil doesn’t write to the kernel’s auditing + log, and we can’t see any reason you’d want to be able to do that as + an administrator shelled into the container, either. Auditing is + something done on the host, not from inside each individual + container. + +* **`CHOWN`**: The Fossil server never even calls `chown(2)`, and our + image build process sets up all file ownership properly, to the + extent that this is possible under the limitations of our + automation. + + Curiously, stripping this capability doesn’t affect your ability to + run commands like “`chown -R fossil:fossil /museum`” when + you’re using bind mounts or external volumes — as we recommend + [above](#bind-mount) — because it’s the host OS’s kernel + capabilities that affect the underlying `chown(2)` call in that + case, not those of the container. + + If for some reason you did have to change file ownership of + in-container files, it’s best to do that by changing the + `Dockerfile` to suit, then rebuilding the container, since that + bakes the need for the change into your reproducible build process. + If you had to do it without rebuilding the container, [there’s a + workaround][capchg] for the fact that capabilities are a create-time + change, baked semi-indelibly into the container configuration. + +* **`FSETID`**: Fossil doesn’t use the SUID and SGID bits itself, and + our build process doesn’t set those flags on any of the files. + Although the second fact means we can’t see any harm from leaving + this enabled, we also can’t see any good reason to allow it, so we + strip it. + +* **`KILL`**: The only place Fossil calls `kill(2)` is in the + [backoffice], and then only for processes it created on earlier + runs; it doesn’t need the ability to kill processes created by other + users. You might wish for this ability as an administrator shelled + into the container, but you can pass the “`docker exec --user`” + option to run commands within your container as the legitimate owner + of the process, removing the need for this capability. + +* **`MKNOD`**: As of 2023.03.26, the stock container uses the + runtime’s default `/dev` node tree. Prior to this, we had to create + `/dev/null` and `/dev/urandom` inside [the chroot jail](#chroot), + but even then, these device nodes were created at build time and + were never changed at run time, so we didn’t need this run-time + capability even then. + +* **`NET_BIND_SERVICE`**: With containerized deployment, Fossil never + needs the ability to bind the server to low-numbered TCP ports, not + even if you’re running the server in production with TLS enabled and + want the service bound to port 443. It’s perfectly fine to let the + Fossil instance inside the container bind to its default port (8080) + because you can rebind it on the host with the + “`docker create --publish 443:8080`” option. It’s the container’s + _host_ that needs this ability, not the container itself. + + (Even the container runtime might not need that capability if you’re + [terminating TLS with a front-end proxy](./ssl.wiki#server). You’re + more likely to say something like “`-p localhost:12345:8080`” and then + configure the reverse proxy to translate external HTTPS calls into + HTTP directed at this internal port 12345.) + +* **`NET_RAW`**: Fossil itself doesn’t use raw sockets, and while + you could [swap out the run layer](#run) for something more + functional that *does* make use of raw sockets, there’s little call + for it. The best reason I can come up with is to be able to run + utilities like `ping` and `traceroute`, but since we aren’t doing + anything clever with the networking configuration, there’s no + particularly compelling reason to run these from inside the + container. If you need to ping something, do it on the host. + + If we did not take this hard-line stance, an attacker that broke + into the container and gained root privileges might use raw sockets + to do a wide array of bad things to any network the container is + bound to. + +* **`SETFCAP, SETPCAP`**: There isn’t much call for file permission + granularity beyond the classic Unix ones inside the container, so we + drop root’s ability to change them. + +All together, we recommend adding the following options to your +“`docker run`” commands, as well as to any “`docker create`” command +that will be followed by “`docker start`”: + + --cap-drop AUDIT_WRITE \ + --cap-drop CHOWN \ + --cap-drop FSETID \ + --cap-drop KILL \ + --cap-drop MKNOD \ + --cap-drop NET_BIND_SERVICE \ + --cap-drop NET_RAW \ + --cap-drop SETFCAP \ + --cap-drop SETPCAP + +In the next section, we’ll show a case where you create a container +without ever running it, making these options pointless. + +[backoffice]: ./backoffice.md +[defcap]: https://docs.docker.com/engine/security/#linux-kernel-capabilities +[capchg]: https://stackoverflow.com/a/45752205/142454 + + + +## 4. <a id="static"></a>Extracting a Static Binary + +Our 2-stage build process uses Alpine Linux only as a build host. Once +we’ve got everything reduced to a single static Fossil binary, +we throw all the rest of it away. + +A secondary benefit falls out of this process for free: it’s arguably +the easiest way to build a purely static Fossil binary for Linux. Most +modern Linux distros make this [surprisingly difficult][lsl], but Alpine’s +back-to-basics nature makes static builds work the way they used to, +back in the day. If that’s all you’re after, you can do so as easily as +this: + + $ docker build -t fossil . + $ docker create --name fossil-static-tmp fossil + $ docker cp fossil-static-tmp:/bin/fossil . + $ docker container rm fossil-static-tmp + +The result is six or seven megs, depending on the CPU architecture you +build for. It’s built stripped. + +[lsl]: https://stackoverflow.com/questions/3430400/linux-static-linking-is-dead + + +## 5. <a id="custom" name="args"></a>Customization Points + +### <a id="pkg-vers"></a> 5.1 Fossil Version + +The default version of Fossil fetched in the build is the version in the +checkout directory at the time you run it. You could override it to get +a release build like so: + + $ docker build -t fossil --build-arg FSLVER=version-2.20 . + +Or equivalently, using Fossil’s `Makefile` convenience target: + + $ make container-image DBFLAGS='--build-arg FSLVER=version-2.20' + +While you could instead use the generic +“`release`” tag here, it’s better to use a specific version number +since container builders cache downloaded files, hoping to +reuse them across builds. If you ask for “`release`” before a new +version is tagged and then immediately after, you might expect to get +two different tarballs, but because the underlying source tarball URL +remains the same when you do that, you’ll end up reusing the +old tarball from cache. This will occur +even if you pass the “`docker build --no-cache`” option. + +This is why we default to pulling the Fossil tarball by checkin ID +rather than let it default to the generic “`trunk`” tag: so the URL will +change each time you update your Fossil source tree, forcing the builder to +pull a fresh tarball. + + +### 5.2 <a id="uids"></a>User & Group IDs + +The “`fossil`” user and group IDs inside the container default to 499. +Why? Regular user IDs start at 500 or 1000 on most Unix type systems, +leaving those below it for system users like this Fossil daemon owner. +Since it’s typical for these to start at 0 and go upward, we started at +500 and went *down* one instead to reduce the chance of a conflict to as +close to zero as we can manage. + +To change it to something else, say: + + $ make container-image DBFLAGS='--build-arg UID=501' + +This is particularly useful if you’re putting your repository on a +separate volume since the IDs “leak” out into the host environment via +file permissions. You may therefore wish them to mean something on both +sides of the container barrier rather than have “499” appear on the host +in “`ls -l`” output. + + +### 5.3 <a id="cengine"></a>Container Engine + +Although the Fossil container build system defaults to Docker, we allow +for use of any OCI container system that implements the same interfaces. +We go into more details about this [below](#light), but +for now, it suffices to point out that you can switch to Podman while +using our `Makefile` convenience targets unchanged by saying: + + $ make CENGINE=podman container-run + + +### 5.4 <a id="config"></a>Fossil Configuration Options + +You can use this same mechanism to enable non-default Fossil +configuration options in your build. For instance, to turn on +the JSON API and the TH1 docs extension: + + $ make container-image \ + DBFLAGS='--build-arg FSLCFG="--json --with-th1-docs"' + +If you also wanted [the Tcl evaluation extension](./th1.md#tclEval), +that brings us to [the next point](#run). + + +### 5.5 <a id="run"></a>Elaborating the Run Layer + +If you want a basic shell environment for temporary debugging of the +running container, that’s easily added. Simply change this line in the +`Dockerfile`… + + FROM scratch AS run + +…to this: + + FROM busybox AS run + +Rebuild and redeploy to give your Fossil container a [BusyBox]-based +shell environment that you can get into via: + + $ docker exec -it -u fossil $(make container-version) sh + +That command assumes you built it via “`make container`” and are +therefore using its versioning scheme. + +You will likely want to remove the `PATH` override in the “RUN” stage +when doing this since it’s written for the case where everything is in +`/bin`, and that will no longer be the case with a more full-featured +“`run`” layer. As long as the parent layer’s `PATH` value contains +`/bin`, delegating to it is more likely the correct thing. + +Another useful case to consider is that you’ve installed a [server +extension](./serverext.wiki) and you need an interpreter for that +script. The first option above won’t work except in the unlikely case that +it’s written for one of the bare-bones script interpreters that BusyBox +ships.(^[BusyBox]’s `/bin/sh` is based on the old 4.4BSD Lite Almquist +shell, implementing little more than what POSIX specified in 1989, plus +equally stripped-down versions of `awk` and `sed`.) + +Let’s say the extension is written in Python. Because this is one of the +most popular programming languages in the world, we have many options +for achieving this. For instance, there is a whole class of +“[distroless]” images that will do this efficiently by changing +“`STAGE 2`” in the `Dockefile` to this: + + ## --------------------------------------------------------------------- + ## STAGE 2: Pare that back to the bare essentials, plus Python. + ## --------------------------------------------------------------------- + FROM cgr.dev/chainguard/python:latest + USER root + ARG UID=499 + ENV PATH "/sbin:/usr/sbin:/bin:/usr/bin" + COPY --from=builder /tmp/fossil /bin/ + COPY --from=builder /bin/busybox.static /bin/busybox + RUN [ "/bin/busybox", "--install", "/bin" ] + RUN set -x \ + && echo "fossil:x:${UID}:${UID}:User:/museum:/false" >> /etc/passwd \ + && echo "fossil:x:${UID}:fossil" >> /etc/group \ + && install -d -m 700 -o fossil -g fossil log museum + +You will also have to add `busybox-static` to the APK package list in +STAGE 1 for the `RUN` script at the end of that stage to work, since the +[Chainguard Python image][cgimgs] lacks a shell, on purpose. The need to +install root-level binaries is why we change `USER` temporarily here. + +Build it and test that it works like so: + + $ make container-run && + docker exec -i $(make container-version) python --version + 3.11.2 + +The compensation for the hassle of using Chainguard over something more +general purpose like changing the `run` layer to Alpine and then adding +a “`apk add python`” command to the `Dockerfile` +is huge: we no longer leave a package manager sitting around inside the +container, waiting for some malefactor to figure out how to abuse it. + +Beware that there’s a limit to this über-jail’s ability to save you when +you go and provide a more capable runtime layer like this. The container +layer should stop an attacker from accessing any files out on the host +that you haven’t explicitly mounted into the container’s namespace, but +it can’t stop them from making outbound network connections or modifying +the repo DB inside the container. + +[cgimgs]: https://github.com/chainguard-images/images/tree/main/images +[distroless]: https://www.chainguard.dev/unchained/minimal-container-images-towards-a-more-secure-future +[MTA]: https://en.wikipedia.org/wiki/Message_transfer_agent + + +### 5.6 <a id="alerts"></a>Email Alerts + +The nature of our single static binary container precludes two of the +options for [sending email alerts](./alerts.md) from Fossil: + +* pipe to a command +* SMTP relay host + +There is no `/usr/sbin/sendmail` inside the container, and the container +cannot connect out to a TCP service on the host by default. + +While it is possible to get around the first lack by [elaborating the +run layer](#run), to inject a full-blown Sendmail setup into the +container would go against the whole idea of containerization. +Forwarding an SMTP relay port into the container isn’t nearly as bad, +but it’s still bending the intent behind containers out of shape. + +A far better option in this case is the “store emails in database” +method since the containerized Fossil binary knows perfectly well how to +write SQLite DB files without relying on any external code. Using the +paths in the configuration recommended above, the database path should +be set to something like `/museum/mail.db`. This, along with the use of +[bind mounts](#bind-mount) means you can have a process running outside +the container that passes the emails along to the host-side MTA. + +The included [`email-sender.tcl`](/file/tools/email-sender.tcl) script +works reasonably well for this, though in my own usage, I had to make +two changes to it: + +1. The shebang line at the top has to be `#!/usr/bin/tclsh` on my server. +2. I parameterized the `DBFILE` variable at the top thus: + + set DBFILE [lindex $argv 0] + +I then wanted a way to start this Tcl script on startup and keep it +running, which made me reach for systemd. My server is set to allow user +services to run at boot(^”Desktop” class Linuxes tend to disable that by +default under the theory that you don’t want those services to run until +you’ve logged into the GUI as that user. If you find yourself running +into this, [enable linger +mode](https://www.freedesktop.org/software/systemd/man/loginctl.html).) +so I was able to create a unit file called +`~/.local/share/systemd/user/alert-sender@.service` with these contents: + + [Unit] + Description=Fossil email alert sender for %I + + [Service] + WorkingDirectory=/home/fossil/museum + ExecStart=/home/fossil/bin/alert-sender %I/mail.db + Restart=always + RestartSec=3 + + [Install] + WantedBy=default.target + +I was then able to enable email alert forwarding for select repositories +after configuring them per [the docs](./alerts.md) by saying: + + $ systemctl --user daemon-reload + $ systemctl --user enable --now alert-sender@myproject + +Because this is a parameterized script and we’ve set our repository +paths predictably, you can do this for as many repositories as you need +to by passing their names after the “`@`” sign in the commands above. + + +## 6. <a id="light"></a>Lightweight Alternatives to Docker + +Those afflicted with sticker shock at seeing the size of a [Docker +Desktop][DD] installation — 1.65 GB here — might’ve immediately +“noped” out of the whole concept of containers. The first thing to +realize is that when it comes to actually serving simple containers like +the ones shown above is that [Docker Engine][DE] suffices, at about a +quarter of the size. + +Yet on a small server — say, a $4/month ten gig Digital Ocean droplet — +that’s still a big chunk of your storage budget. It takes ~60:1 overhead +merely to run a Fossil server container? Once again, I wouldn’t +blame you if you noped right on out of here, but if you will be patient, +you will find that there are ways to run Fossil inside a container even +on entry-level cloud VPSes. These are well-suited to running Fossil; you +don’t have to resort to [raw Fossil service][srv] to succeed, +leaving the benefits of containerization to those with bigger budgets. + +For the sake of simple examples in this section, we’ll assume you’re +integrating Fossil into a larger web site, such as with our [Debian + +nginx + TLS][DNT] plan. This is why all of the examples below create +the container with this option: + + --publish 127.0.0.1:9999:8080 + +The assumption is that there’s a reverse proxy running somewhere that +redirects public web hits to localhost port 9999, which in turn goes to +port 8080 inside the container. This use of port +publishing effectively replaces the use of the +“`fossil server --localhost`” option. + +For the nginx case, you need to add `--scgi` to these commands, and you +might also need to specify `--baseurl`. + +Containers are a fine addition to such a scheme as they isolate the +Fossil sections of the site from the rest of the back-end resources, +thus greatly reducing the chance that they’ll ever be used to break into +the host as a whole. + +(If you wanted to be double-safe, you could put the web server into +another container, restricting it to reading from the static web +site directory and connecting across localhost to back-end dynamic +content servers such as Fossil. That’s way outside the scope of this +document, but you can find ready advice for that elsewhere. Seeing how +we do this with Fossil should help you bridge the gap in extending +this idea to the rest of your site.) + +[DD]: https://www.docker.com/products/docker-desktop/ +[DE]: https://docs.docker.com/engine/ +[DNT]: ./server/debian/nginx.md +[srv]: ./server/ + + +### 6.1 <a id="nerdctl" name="containerd"></a>Stripping Docker Engine Down + +The core of Docker Engine is its [`containerd`][ctrd] daemon and the +[`runc`][runc] container runtime. Add to this the out-of-core CLI program +[`nerdctl`][nerdctl] and you have enough of the engine to run Fossil +containers. The big things you’re missing are: + +* **BuildKit**: The container build engine, which doesn’t matter if + you’re building elsewhere and shipping the images to the target. + A good example is using a container registry as an + intermediary between the build and deployment hosts. + +* **SwarmKit**: A powerful yet simple orchestrator for Docker that you + probably aren’t using with Fossil anyway. + +In exchange, you get a runtime that’s about half the size of Docker +Engine. The commands are essentially the same as above, but you say +“`nerdctl`” instead of “`docker`”. You might alias one to the other, +because you’re still going to be using Docker to build and ship your +container images. + +[ctrd]: https://containerd.io/ +[nerdctl]: https://github.com/containerd/nerdctl +[runc]: https://github.com/opencontainers/runc + + +### 6.2 <a id="podman"></a>Podman + +A lighter-weight [rootless][rl] [drop-in replacement][whatis] that +doesn’t give up the image builder is [Podman]. Initially created by +Red Hat and thus popular on that family of OSes, it will run on +any flavor of Linux. It can even be made to run [on macOS via Homebrew][pmmac] +or [on Windows via WSL2][pmwin]. + +On Ubuntu 22.04, the installation size is about 38 MiB, roughly a +tenth the size of Docker Engine. + +For our purposes here, the only thing that changes relative to the +examples at the top of this document are the initial command: + + $ podman build -t fossil . + $ podman run --name fossil -p 9999:8080/tcp fossil + +Your Linux package repo may have a `podman-docker` package which +provides a “`docker`” script that calls “`podman`” for you, eliminating +even the command name difference. With that installed, the `make` +commands above will work with Podman as-is. + +The only difference that matters here is that Podman doesn’t have the +same [default Linux kernel capability set](#caps) as Docker, which +affects the `--cap-drop` flags recommended above to: + + $ podman create \ + --name fossil \ + --cap-drop CHOWN \ + --cap-drop FSETID \ + --cap-drop KILL \ + --cap-drop NET_BIND_SERVICE \ + --cap-drop SETFCAP \ + --cap-drop SETPCAP \ + --publish 127.0.0.1:9999:8080 \ + localhost/fossil + $ podman start fossil + +[pmmac]: https://podman.io/getting-started/installation.html#macos +[pmwin]: https://github.com/containers/podman/blob/main/docs/tutorials/podman-for-windows.md +[Podman]: https://podman.io/ +[rl]: https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md +[whatis]: https://docs.podman.io/en/latest/index.html + + +### 6.3 <a id="nspawn"></a>`systemd-container` + +If even the Podman stack is too big for you, the next-best option I’m +aware of is the `systemd-container` infrastructure on modern Linuxes, +available since version 239 or so. Its runtime tooling requires only +about 1.4 MiB of disk space: + + $ sudo apt install systemd-container btrfs-tools + +That command assumes the primary test environment for +this guide, Ubuntu 22.04 LTS with `systemd` 249. For best +results, `/var/lib/machines` should be a btrfs volume, because +[`$REASONS`][mcfad]. For CentOS Stream 9 and other Red Hattish +systems, you will have to make several adjustments, which we’ve +collected [below](#nspawn-centos) to keep these examples clear. + +We’ll assume your Fossil repository stores something called +“`myproject`” within `~/museum/myproject/repo.fossil`, named according +to the reasons given [above](#repo-inside). We’ll make consistent use of +this naming scheme in the examples below so that you will be able to +replace the “`myproject`” element of the various file and path names. + +If you use [the stock `Dockerfile`][DF] to generate your +base image, `nspawn` won’t recognize it as containing an OS unless you +change the “`FROM scratch AS os`” line at the top of the second stage +to something like this: + + FROM gcr.io/distroless/static-debian11 AS os + +Using that as a base image provides all the files `nspawn` checks for to +determine whether the container is sufficiently close to a Linux VM for +the following step to proceed: + + $ make container + $ docker container export $(make container-version) | + machinectl import-tar - myproject + +Next, create `/etc/systemd/nspawn/myproject.nspawn`: + +---- + + [Exec] + WorkingDirectory=/ + Parameters=bin/fossil server \ + --baseurl https://example.com/myproject \ + --create \ + --jsmode bundled \ + --localhost \ + --port 9000 \ + --scgi \ + --user admin \ + museum/repo.fossil + DropCapability= \ + CAP_AUDIT_WRITE \ + CAP_CHOWN \ + CAP_FSETID \ + CAP_KILL \ + CAP_MKNOD \ + CAP_NET_BIND_SERVICE \ + CAP_NET_RAW \ + CAP_SETFCAP \ + CAP_SETPCAP + ProcessTwo=yes + LinkJournal=no + Timezone=no + + [Files] + Bind=/home/fossil/museum/myproject:/museum + + [Network] + VirtualEthernet=no + +---- + +If you recognize most of that from the `Dockerfile` discussion above, +congratulations, you’ve been paying attention. The rest should also +be clear from context. + +Some of this is expected to vary: + +* The references to `example.com` and `myproject` are stand-ins for + your actual web site and repository name. + +* The command given in the `Parameters` directive assumes you’re + setting up [SCGI proxying via nginx][DNT], but with adjustment, + it’ll work with the other repository service methods we’ve + [documented][srv]. + +* The path in the host-side part of the `Bind` value must point at the + directory containing the `repo.fossil` file referenced in said + command so that `/museum/repo.fossil` refers to your repo out + on the host for the reasons given [above](#bind-mount). + +That being done, we also need a generic `systemd` unit file called +`/etc/systemd/system/fossil@.service`, containing: + +---- + + [Unit] + Description=Fossil %i Repo Service + Wants=modprobe@tun.service modprobe@loop.service + After=network.target systemd-resolved.service modprobe@tun.service modprobe@loop.service + + [Service] + ExecStart=systemd-nspawn --settings=override --read-only --machine=%i bin/fossil + + [Install] + WantedBy=multi-user.target + +---- + +You shouldn’t have to change any of this because we’ve given the +`--setting=override` flag, meaning any setting in the nspawn file +overrides the setting passed to `systemd-nspawn`. This arrangement +not only keeps the unit file simple, it allows multiple services to +share the base configuration, varying on a per-repo level through +adjustments to their individual `*.nspawn` files. + +You may then start the service in the normal way: + + $ sudo systemctl enable fossil@myproject + $ sudo systemctl start fossil@myproject + +You should then find it running on localhost port 9000 per the nspawn +configuration file above, suitable for proxying Fossil out to the +public using nginx via SCGI. If you aren’t using a front-end proxy +and want Fossil exposed to the world via HTTPS, you might say this instead in +the `*.nspawn` file: + + Parameters=bin/fossil server \ + --cert /path/to/cert.pem \ + --create \ + --jsmode bundled \ + --port 443 \ + --user admin \ + museum/repo.fossil + +You would also need to un-drop the `CAP_NET_BIND_SERVICE` capability +to allow Fossil to bind to this low-numbered port. + +We use the `systemd` template file feature to allow multiple Fossil +servers running on a single machine, each on a different TCP port, +as when proxying them out as subdirectories of a larger site. +To add another project, you must first clone the base “machine” layer: + + $ sudo machinectl clone myproject otherthing + +That will not only create a clone of `/var/lib/machines/myproject` +as `../otherthing`, it will create a matching `otherthing.nspawn` file for you +as a copy of the first one. Adjust its contents to suit, then enable +and start it as above. + +[mcfad]: https://www.freedesktop.org/software/systemd/man/machinectl.html#Files%20and%20Directories + + +### 6.3.1 <a id="nspawn-rhel"></a>Getting It Working on a RHEL Clone + +The biggest difference between doing this on OSes like CentOS versus +Ubuntu is that RHEL (thus also its clones) doesn’t ship btrfs in +its kernel, thus ships with no package repositories containing `mkfs.btrfs`, which +[`machinectl`][mctl] depends on for achieving its various purposes. + +Fortunately, there are workarounds. + +First, the `apt install` command above becomes: + + $ sudo dnf install systemd-container + +Second, you have to hack around the lack of `machinectl import-tar`: + + $ rootfs=/var/lib/machines/fossil + $ sudo mkdir -p $rootfs + $ docker container export fossil | sudo tar -xf -C $rootfs - + +The parent directory path in the `rootfs` variable is important, +because although we aren’t able to use `machinectl` on such systems, the +`systemd-nspawn` developers assume you’re using them together; when you give +`--machine`, it assumes the `machinectl` directory scheme. You could +instead use `--directory`, allowing you to store the rootfs wherever +you like, but why make things difficult? It’s a perfectly sensible +default, consistent with the [LHS] rules. + +The final element — the machine name — can be anything +you like so long as it matches the nspawn file’s base name. + +Finally, since you can’t use `machinectl clone`, you have to make +a wasteful copy of `/var/lib/machines/myproject` when standing up +multiple Fossil repo services on a single machine. (This is one +of the reasons `machinectl` depends on `btrfs`: cheap copy-on-write +subvolumes.) Because we give the `--read-only` flag, you can simply +`cp -r` one machine to a new name rather than go through the +export-and-import dance you used to create the first one. + +[LHS]: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html +[mctl]: https://www.freedesktop.org/software/systemd/man/machinectl.html + + +### 6.3.2 <a id="nspawn-weaknesses"></a>What Am I Missing Out On? + +For all the runtime size savings in this method, you may be wondering +what you’re missing out on relative to Podman, which takes up +roughly 27× more disk space. Short answer: lots. Long answer: + +1. **Build system.** You’ll have to build and test your containers + some other way. This method is only suitable for running them + once they’re built. + +2. **Orchestration.** All of the higher-level things like + “compose” files, Docker Swarm mode, and Kubernetes are + unavailable to you at this level. You can run multiple + instances of Fossil, but on a single machine only and with a + static configuration. + +3. **Image layer sharing.** When you update an image using one of the + above methods, Docker and Podman are smart enough to copy only + changed layers. Furthermore, when you base multiple containers + on a single image, they don’t make copies of the base layers; + they can share them, because base layers are immutable, thus + cannot cross-contaminate. + + Because we use `systemd-nspawn --read-only`, we get *some* + of this benefit, particularly when using `machinectl` with + `/var/lib/machines` as a btrfs volume. Even so, the disk space + and network I/O optimizations go deeper in the Docker and Podman + worlds. + +4. **Tooling.** Hand-creating and modifying those `systemd` + files sucks compared to “`podman container create ...`” This + is but one of many affordances you will find in the runtimes + aimed at daily-use devops warriors. + +5. **Network virtualization.** In the scheme above, we turn off the + `systemd` private networking support because in its default mode, it + wants to hide containerized services entirely. While there are + [ways][ndcmp] to expose Fossil’s single network service port under + that scheme, it adds a lot of administration complexity. In the + big-boy container runtimes, `docker create --publish` fixes all this + up in a single option, whereas `systemd-nspawn --port` does + approximately *none* of that despite the command’s superficial + similarity. + + From a purely functional point of view, this isn’t a huge problem if + you consider the inbound service direction only, being external + connections to the Fossil service we’re providing. Since we do want + this Fossil service to be exposed — else why are we running it? — we + get all the control we need via `fossil server --localhost` and + similar options. + + The complexity of the `systemd` networking infrastructure’s + interactions with containers make more sense when you consider the + outbound path. Consider what happens if you enable Fossil’s + optional TH1 docs feature plus its Tcl evaluation feature. That + would enable anyone with the rights to commit to your repository the + ability to make arbitrary network connections on the Fossil host. + Then, let us say you have a client-server DBMS server on that same + host, bound to localhost for private use by other services on the + machine. Now that DBMS is open to access by a rogue Fossil committer + because the host’s loopback interface is mapped directly into the + container’s network namespace. + + Proper network virtualization would protect you in this instance. + +This author expects that the set of considerations is broader than +presented here, but that it suffices to make our case as it is: if you +can afford the space of Podman or Docker, we strongly recommend using +either of them over the much lower-level `systemd-container` +infrastructure. You’re getting a considerable amount of value for the +higher runtime cost; it isn’t pointless overhead. + +(Incidentally, these are essentially the same reasons why we no longer +talk about the `crun` tool underpinning Podman in this document. It’s +even more limited than `nspawn`, making it even more difficult to administer while +providing no runtime size advantage. The `runc` tool underpinning +Docker is even worse on this score, being scarcely easier to use than +`crun` while having a much larger footprint.) + +[ndcmp]: https://wiki.archlinux.org/title/systemd-networkd#Usage_with_containers + + +### 6.3.3 <a id="nspawn-assumptions"></a>Violated Assumptions + +The `systemd-container` infrastructure has a bunch of hard-coded +assumptions baked into it. We papered over these problems above, +but if you’re using these tools for other purposes on the machine +you’re serving Fossil from, you may need to know which assumptions +our container violates and the resulting consequences. + +Some of it we discussed above already, but there’s one big class of +problems we haven’t covered yet. It stems from the fact that our stock +container starts a single static executable inside a bare-bones container +rather than “boot” an OS image. That causes a bunch of commands to fail: + +* **`machinectl poweroff`** will fail because the container + isn’t running dbus. + +* **`machinectl start`** will try to find an `/sbin/init` + program in the rootfs, which we haven’t got. We could + rename `/bin/fossil` to `/sbin/init` and then hack + the chroot scheme to match, but ick. (This, incidentally, + is why we set `ProcessTwo=yes` above even though Fossil is + perfectly capable of running as PID 1, a fact we depend on + in the other methods above.) + +* **`machinectl shell`** will fail because there is no login + daemon running, which we purposefully avoided adding by + creating a “`FROM scratch`” container. (If you need a + shell, say: `sudo systemd-nspawn --machine=myproject /bin/sh`) + +* **`machinectl status`** won’t give you the container logs + because we disabled the shared journal, which was in turn + necessary because we don’t run `systemd` *inside* the + container, just outside. + +If these are problems for you, you may wish to build a +fatter container using `debootstrap` or similar. ([External +tutorial][medtut].) + +[medtut]: https://medium.com/@huljar/setting-up-containers-with-systemd-nspawn-b719cff0fb8d + +<div style="height:50em" id="this-space-intentionally-left-blank"></div> diff --git a/www/contribute.wiki b/www/contribute.wiki new file mode 100644 index 0000000..9fe477b --- /dev/null +++ b/www/contribute.wiki @@ -0,0 +1,128 @@ +<title>Contributing To Fossil</title> + +Fossil users are encouraged to contribute enhancements back to the +project. This note outlines some of the procedures for making +useful contributions. + +<h2>1.0 Contributor Agreement</h2> + +In order to accept non-trivial contributions, we <u>must</u> have a +[./copyright-release.pdf | Contributor Agreement (PDF)] +(or [./copyright-release.html | as HTML]) on file for you. We require +this in order to maintain clear title to the Fossil code and prevent +the introduction of code with incompatible licenses or other entanglements +that might cause legal problems for Fossil users. Many +lawyer-rich organizations require this as a precondition to using +Fossil. + +If you do not wish to submit a Contributor Agreement, we would still +welcome your suggestions and example code, but we will not use your code +directly: we will be forced to re-implement your changes from scratch, which +might take longer. + +We've made exceptions for "trivial" changes in the past, but the +definition of that term is up to the project leader. + +<h2>2.0 Submitting Patches</h2> + +Suggested changes or bug fixes can be submitted by creating a patch +against the current source tree: + +<pre>fossil diff -i > my-change.patch</pre> + +Alternatively, you can create a binary patch: + +<pre>fossil patch create my-change.db</pre> + +Post patches to +[https://fossil-scm.org/forum | the forum] or email them to +<a href="mailto:drh@sqlite.org">drh@sqlite.org</a>. Be sure to +describe in detail what the patch does and which version of Fossil +it is written against. It's best to make patches against tip-of-trunk +rather than against past releases. + +If your change is more complicated than a patch can properly encode, you +may submit [/help/bundle | a Fossil bundle] instead. Unlike patches, +bundles can contain multiple commits, check-in comments, file renames, +file deletions, branching decisions, and more which <tt>patch(1)</tt> +files cannot. It's best to make a bundle of a new branch so the change +can be integrated, tested, enhanced, and merged down to trunk in a +controlled fashion. + +A contributor agreement is not strictly necessary to submit a patch or bundle, +but without a contributor agreement on file, your contribution will be +used for reference only: it will not be applied to the code. This +may delay acceptance of your contribution. + +Your contribution might not be accepted even if you do have +a contributor agreement on file. Please do not take this personally +or as an affront to your coding ability. Sometimes contributions are rejected +because they seem to be taking the project in a direction that the +architect does not want to go. In other cases, there might be an alternative +implementation of the same feature being prepared separately. + +<h2>3.0 Check-in Privileges</h2> + +Check-in privileges are granted on a case-by-case basis. Your chances +of getting check-in privileges are much improved if you have a history +of submitting quality patches and/or making thoughtful posts on +[https://fossil-scm.org/forum | the forum]. +A signed contributor agreement is, of course, a prerequisite for check-in +privileges.</p> + +Contributors are asked to make all non-trivial changes on a branch. The +Fossil Architect (Richard Hipp) will merge changes onto the trunk.</p> + +Contributors are required to follow the +[./checkin.wiki | pre-checkin checklist] prior to every check-in to +the Fossil self-hosting repository. This checklist is short and succinct +and should only require a few seconds to follow. Contributors +should print out a copy of the pre-checkin checklist and keep +it on a note card beside their workstations for quick reference. + +Contributors should review the +[./style.wiki | Coding Style Guidelines] and mimic the coding style +used through the rest of the Fossil source code. Your code should +blend in. A third-party reader should be unable to distinguish your +code from any other code in the source corpus. + +<h2>4.0 Testing</h2> + +Fossil's [../test/release-checklist.wiki | release checklist] is of +primary benefit to the project leader, followed by him at release time, +but contributors are encouraged to run through its steps when making +major changes, since if the change doesn't pass this checklist, it won't +be included in the next release. + +<h2>5.0 UI and Documentation Language</h2> + +The Fossil project uses American English in its web interface and +documentation. Until there is some provision for translating the UI and +docs into other languages and dialects, we ask that you do not commit +changes that conflict with this. + +We aren't opposed to such a project, but it would be a huge amount of +work, which no one's stepped up to do yet. Not only is each individual +translation a large ongoing job its own right, there is no +infrastructure for it yet, so the first few translations will be harder +than any future translation built on that infrastructure. + +More immediately, we're likely to reject, revert, or rework commits that +use other English dialects. One example that comes up occasionally is +"artefact" versus "artifact." The UI and docs use the American English +spelling pervasively, so you have poor options if you insist on +"artefact:" + + * attempt to slip one-off changes by your peers + * attempt to change all American English usages to Commonwealth English + * make the Fossil UI and docs translatable, then contribute a + Commonwealth English translation + +Only the latter is likely to succeed. + +<h2>6.0 See Also</h2> + + * [./build.wiki | How To Compile And Install Fossil] + * [./makefile.wiki | The Fossil Build Process] + * [./tech_overview.wiki | A Technical Overview of Fossil] + * [./adding_code.wiki | Adding Features To Fossil] diff --git a/www/copyright-release.html b/www/copyright-release.html new file mode 100644 index 0000000..222d3f5 --- /dev/null +++ b/www/copyright-release.html @@ -0,0 +1,106 @@ +<h1 align="center"> +Fossil SCM Contributor Agreement +</h1> + +<p> +This agreement applies to your contribution of material to the +Fossil Software Configuration Management System ("Fossil") that is +managed by Hipp, Wyrick & Company, Inc. ("Hwaci") and +sets out the intellectual property rights you grant to Hwaci in the +contributed material. +The terms "contribution" and "contributed material" mean any source code, +object code, patch, tool, sample, graphic, specification, manual, +documentation, or any other material posted, submitted, or uploaded by +you to the Fossil project. + The term "you" means the person identified +and signing at the bottom of this document. If your contribution +is on behalf of a company, the term "you" also means the company +identified in the signature area below. + +<ol> +<li><p> +With respect to any worldwide copyrights, or copyright applications and +registrations, in your contribution: +<ul> +<li> You hereby assign to Hwaci joint ownership, and to the extent that + such assignment is or becomes invalid, ineffective or unenforceable, + you hereby grant to Hwaci a perpetual, irrevocable, non-exclusive, + worldwide, no-charge, royalty-free, unrestricted license to exercise + all rights under those copyrights, including the right to sublicense. +<li> You agree that both you and Hwaci can do all things in relation to your + contribution as if each of us were the sole owners, and if one of us + makes a derivative work of your contribution, the one who makes + (or has made) the derivative work will be the sole owner of that + derivative work. +<li> You agree that you will not assert any moral rights in your + contribution against Hwaci, Hwaci's licensees or transferees, or + any other user or consumer of your contribution. +<li> You agree that Hwaci may register a copyright in your contribution and + exercise all ownership rights associated with it. +<li> You agree that neither you nor Hwaci has any duty to consult with, + obtain the consent of, or pay or render an accounting to the other + for any use or distribution of your contribution. +</ul> + +<li><p> +With respect to any patents you own, or that you can license without payment +to any third party, and which are relevant to your contribution, you hereby +grant to Hwaci a perpetual, irrevocable, non-exclusive, worldwide, no-charge, +royalty-free license to +make, have made, use, sell, offer to sell, import, and otherwise +transfer your contribution in whole or in part, alone or in +combination with or included in any product, work or materials arising +out of the Fossil project, and to sublicense these same rights. + +<li><p> +Except as set out above, you keep all right, title, and interest in your +contribution. The rights that you grant to Hwaci under this agreement +are effective on the date that you first submitted your contribution to the +Fossil project, even if your submission took place before the date that +you sign this agreement. + +<li><p> +You represent and warrant the following: +<ul> +<li> Your contribution is an original work and that you can legally + grant the rights set out in this agreement. +<li> Your contribution does not, to the best of your knowledge and belief, + violate any third party's copyrights, trademarks, patents, + or other intellectual property rights. +<li> You are authorized to sign this agreement on behalf of your + company (if applicable). +</ul> +</ol> + +<p>By filling in the following information and signing your name, +you agree to be bound by all of the terms +set forth in this agreement. Please print clearly.</p> + +<p><table width="80%" border="1" cellpadding="0" cellspacing="0" align="center"> +<tr><td width="20%" valign="top">Your name & email:</td><td width="80%"> + + <!-- Replace this line with your name and email --> <p> + +</td></tr> +<tr><td valign="top">Company name:<br>(if applicable)</td><td> + + <!-- Replace this line with your company name --> <p> + + </td></tr> +<tr><td valign="top">Postal address:</td><td> + + <!-- Replace this line and the next line with your postal address --> + <p> </p><p> </p><p> </p> + +</td></tr> +<tr><td valign="top">Signature:</td><td> <p> </td></tr> +<tr><td valign="top">Date:</td><td> <p> </td></tr> +</table> + +<p>Send completed forms to: +<blockquote> +Hipp, Wyrick & Company, Inc.<br> +6200 Maple Cove Lane<br> +Charlotte, NC 28269-1086<br> +USA +</p> diff --git a/www/copyright-release.pdf b/www/copyright-release.pdf Binary files differnew file mode 100644 index 0000000..1a73987 --- /dev/null +++ b/www/copyright-release.pdf diff --git a/www/css-tricks.md b/www/css-tricks.md new file mode 100644 index 0000000..c09013c --- /dev/null +++ b/www/css-tricks.md @@ -0,0 +1,74 @@ +# Fossil CSS Tips and Tricks + +Many aspects of Fossil's appearance can be customized by +[customizing the site skin](customskin.md). This document +details certain specific CSS tweaks which users have asked +about via the forums. + +This is a "living document" - please feel free to suggest +additions via [the Fossil forum](https://fossil-scm.org/forum/). + +This document is *not* an introduction to CSS - the web is +full of tutorials on that topic. It covers only the specifics +of customizing certain CSS-based behaviors in a Fossil UI. That said... + +## Is it Really `!important`? + +By and large, CSS's `!important` qualifier is not needed when +customizing Fossil's CSS. On occasion, however, particular styles may +be set directly on DOM elements when Fossil generates its HTML, and +such cases require the use of `!important` to override them. + + +<!-- ============================================================ --> +# Main UI CSS + +## Number of Columns in `/dir` View + +The width of columns on the [`/dir` page](/dir) is calculated +dynamically as the page is generated, to attempt to fit the widest +name in a given directory. The number of columns is determined +automatically by CSS. To modify the number of columns and/or the entry width: + +```css +div.columns { + columns: WIDTH COLUMN_COUNT !important; + /* Examples: + columns: 20ex 3 !important + columns: auto auto !important + */ +} +/* The default rule uses div.columns, but it can also be selected using: */ +div.columns.files { ... } +``` + +The `!important` qualifier is required here because the style values are dynamically +calculated and applied when the HTML is emitted. + +The file list itself can be further customized via: + +```css +div.columns > ul { + ... +} +ul.browser { + ... +} +``` + + +<!-- ============================================================ --> +# Forum-specific CSS + +## Limiting Display Length of Long Posts + +Excessively long posts can make scrolling through threads problematic, +especially on mobile devices. The amount of a post which is visible can +be configured using: + +```css +div.forumPostBody { + max-height: 25em; /* change to the preferred maximum effective height */ + overflow: auto; /* tells the browser to add scrollbars as needed */ +} +``` diff --git a/www/css/diff.md b/www/css/diff.md new file mode 100644 index 0000000..9a9e5e0 --- /dev/null +++ b/www/css/diff.md @@ -0,0 +1,112 @@ +Notes On Diff Formatting +======================== + +There are two main kinds of diff display for the web interface: +unified and side-by-side. Both displays are implemented using +a <table>. The unified diff is a 4-column table, and the +side-by-side diff is a 5-column table. In a page like /info that +might show multiple file diffs, each file diff is in a separate +<table>. For side-by-side diffs, a small amount of Javascript +code is used to resize the text columns so that they fill the screen +width and to keep horizontal scrollbars in sync. + +For the unified diff, the basic structure +is like this: + +> ~~~~ +<table class='diff udiff'> +<tr> + <td class='diffln difflnl'><pre> + Line numbers for the left-hand file + </pre></td> + <td class='diffln difflnr'><pre> + Line numbers for the right-hand file + </pre></td> + <td class='diffsep'><pre> + Change marks. "+" or "=" or nothing + </pre></td> + <td class='difftxt difftxtu'><pre> + The text + </pre></td> +</tr> +</table> +~~~~ + +The structure for a side-by-side diff follows the +same basic pattern, though with 5 columns instead of +4, and slightly different class names: + +> ~~~~ +<table class='diff splitdiff'> +<tr> + <td class='diffln difflnl'><pre> + Line numbers for the left-hand file + </pre></td> + <td class='difftxt difftxtl'><pre> + The text for the left side + </pre></td> + <td class='diffsep'><pre> + Change marks. "+" or "=" or nothing + </pre></td> + <td class='diffln difflnr'><pre> + Line numbers for the right-hand file + </pre></td> + <td class='difftxt difftxtr'><pre> + The text on the right-hand side + </pre></td> +</tr> +</table> +~~~~ + +The outer <table> always has class "diff". The "diffu" class +is added for unified diffs and the "splitdiff" class is added for +side-by-side diffs. + +All line-number columns have the "diffln" class. They also always +have one of "difflnl" or "difflnr" depending on whether they hold +line numbers for the left or right files, respectively. + +Text is always kept in a separate column so that it can be scraped +and copied by the user. All text columns have the "difftxt" class. +One additional class "difftxtu", "difftxtl", or "difftxtr" is added +depending on if the text is for a unified diff, the left column of +a side-by-side diff, or the right column of a side-by-side diff. + +The content of all columns is a single <pre> that contains the +appropriate diff-text for that column. Scrolling is done on the +<pre> element. + +Within text columns, highlighting is done with <del> and +<ins> markup. All text on a line that contains an isert or +delete is surrounded by <ins>...</ins> or +<del>..</del>. Within that line, specific characters +of text that specifically inserted deleted have an additional +layer of <ins> or <del> markup. Thus CSS like the +following is appropriate: + +> ~~~~ +td.difftxt ins { + background-color: #dafbe1; /* Light green for the whole line */ + text-decoration: none; +} +td.difftxt ins > ins { + background-color: #a0e4b2; /* Dark green for specific characters that change */ + text-decoration: none; +} +~~~~ + +In a side-by-side diff, if an interior <ins> or <del> that mark +specific characters that change correspond to a delete/insert on the other +side, they they have the "edit" class tag. (ex: <ins class='edit'> +or <del class='edit'>). Some skins choose to paint these "modified" +regions blue: + +> ~~~~ +td.difftxt ins > ins.edit { + background-color: #c0c0ff; /* Blue for "modified" text region */ + text-decoration: none; +} +~~~~ + +Line number text also has <ins> and <del> tags for lines which +are pure insert or pure delete. But the tags do not nest for line numbers. diff --git a/www/css/index.md b/www/css/index.md new file mode 100644 index 0000000..d3835df --- /dev/null +++ b/www/css/index.md @@ -0,0 +1,12 @@ +Cascading Style Sheet Notes +=========================== + +This is a collection of technical notes that document the design of +the Document Object Model (DOM) and corresponding Cascading Style Sheet (CSS) +attributes used for customing the look-and-feel of Fossil. These notes +are of interest to people who want to customize the Fossil skin or +enhance the internal display logic. + +This is a collection of documents that we hope will grow over time. + + * [Diff styling](./diff.md) diff --git a/www/custom_ticket.wiki b/www/custom_ticket.wiki new file mode 100644 index 0000000..eac0532 --- /dev/null +++ b/www/custom_ticket.wiki @@ -0,0 +1,153 @@ +<title>Customizing The Ticket System</title> + +<h2>Introduction</h2> + +This guide will explain how to add the "assigned_to" and "submitted_by" fields +to the ticket system in Fossil, as well as making the system more useful. You +must have "admin" access to the repository to implement these instructions. + +<h2>First modify the TICKET table</h2> + +Click on the "Admin" menu, then "Tickets", then "Table". After the other fields +and before the final ")", insert: +<pre> + , + assigned_to TEXT, + opened_by TEXT +</pre> + +And "Apply Changes". You have just added two more fields to the ticket +database! NOTE: I won't tell you to "Apply Changes" after each step from here +on out. Now, how do you use these fields? + +<h2>Next add assignees</h2> + +Back to the "Tickets" admin page, and click "Common". Add something like this: +<pre> +set assigned_choices { + unassigned + tom + dick + harriet +} +</pre> + +Obviously, choose names corresponding to the logins on your system. The +'unassigned' entry is important, as it prevents you from having a NULL in that +field (which causes problems later when editing). + +<h2>Now modify the 'new ticket' page</h2> + +Back to the "Tickets" admin page, and click "New Ticket Page". This is a little +more tricky. Edit the top part: + +<verbatim> +if {[info exists submit]} { + set status Open + set opened_by $login + set assigned_to "unassigned" + submit_ticket +} +</verbatim> + +Note the "set opened_by" bit -- that will automatically set the "opened_by" +field to the login name of the bug reporter. Now, skip to the part with "EMail" +and modify it like so: + +<verbatim> +<th1>enable_output expr { "$login" eq "anonymous"}</th1> +<tr> +<td align="right"> + EMail: + <input type="text" name="private_contact" value="$<private_contact>" size="30"> +</td> +<td> + <u>Not publicly visible</u>. Used by developers to contact you with questions. +</td> +</tr> +<th1>enable_output 1</th1> +</verbatim> + +This bit of code will get rid of the "email" field entry for logged-in users. +Since we know the user's information, we don't have to ask for it. NOTE: it +might be good to automatically scoop up the user's email and put it here. + +You might also want to enable people to actually assign the ticket to a specific +person during creation. For this to work, you need to add the code +for "assigned_to" as shown below under the heading "Modify the 'edit ticket' page". +This will give you an additional combobox where you can choose a person during +ticket creation. + +<h2>Modify the 'view ticket' page</h2> + +Look for the text "Contact:" (about halfway through). Then insert these lines +after the closing tr tag and before the "enable_output" line: + +<verbatim> +<tr> + <td align="right">Assigned to:</td><td bgcolor="#d0d0d0"> + $<assigned_to> + </td> + <td align="right">Opened by:</td><td bgcolor="#d0d0d0"> + $<opened_by> + </td> +</tr> +</verbatim> + +This will add a row which displays these two fields, in the event the user has +<a href="./caps/ref.html#w">ticket "edit" capability</a>. + +<h2>Modify the 'edit ticket' page</h2> + +Before the "Severity:" line, add this: + +<verbatim> +<tr> + <td align="right">Assigned to:</td> + <td> + <th1>combobox assigned_to $assigned_choices 1</th1> + </td> +</tr> +</verbatim> + +That will give you a drop-down list of assignees. The first argument to the TH1 +command 'combobox' is the database field which the combobox is associated to. +The next argument is the list of choices you want to show in the combobox (and +that you specified in the second step above.) The last argument should be 1 for +a true combobox (see the <a href="th1.md#combobox">TH1 documentation</a> for +details). + +Now, similar to the previous +section, look for "Contact:" and add this: + +<verbatim> +<tr> + <td align="right">Reported by:</td> + <td> + <input type="text" name="opened_by" size="40" value="$<opened_by>"> + </td> +</tr> +</verbatim> + +<h2>What next?</h2> + +Now you can add custom reports which select based on the person to whom the +ticket is assigned. For example, an "Assigned to me" report could be: + +<verbatim> +SELECT + CASE WHEN status IN ('Open','Verified') THEN '#f2dcdc' + WHEN status='Review' THEN '#e8e8e8' + WHEN status='Fixed' THEN '#cfe8bd' + WHEN status='Tested' THEN '#bde5d6' + WHEN status='Deferred' THEN '#cacae5' + ELSE '#c8c8c8' END AS 'bgcolor', + substr(tkt_uuid,1,10) AS '#', + datetime(tkt_mtime) AS 'mtime', + type, + status, + subsystem, + title +FROM ticket +WHERE assigned_to=user() +</verbatim> diff --git a/www/customgraph.md b/www/customgraph.md new file mode 100644 index 0000000..1490f8b --- /dev/null +++ b/www/customgraph.md @@ -0,0 +1,228 @@ +# Customizing the Timeline Graph + +Beginning with version 1.33, Fossil gives users and skin authors significantly +more control over the look and feel of the timeline graph. + +## <a id="basic-style"></a>Basic Style Options + +Fossil includes several options for changing the graph's style without having +to delve into CSS. These can be found in the details.txt file of your skin or +under Admin/Skins/Details in the web UI. + +* **`timeline-arrowheads`** + + Set this to `0` to hide arrowheads on primary child lines. + +* **`timeline-circle-nodes`** + + Set this to `1` to make check-in nodes circular instead of square. + +* **`timeline-color-graph-lines`** + + Set this to `1` to colorize primary child lines. + +* **`white-foreground`** + + Set this to `1` if your skin uses white (or any light color) text. + This tells Fossil to generate darker background colors for branches. + + +## <a id="adv-style"></a>Advanced Styling + +If the above options aren't enough for you, it's time to get your hands dirty +with CSS. To get started, I recommend first copying all the [graph-related CSS +rules](#default-css) to your stylesheet. Then it's simply a matter of making +the necessary changes to achieve the look you want. So, next, let's look at the +various graph elements and what purpose they serve. + +Each element used to construct the timeline graph falls into one of two +categories: visible elements and positioning elements. We'll start with the +latter, less obvious type. + +## <a id="pos-elems"></a>Positioning Elements + +These elements aren't intended to be seen. They're only used to help position +the graph and its visible elements. + +* <a id="tl-canvas"></a>**`.tl-canvas`** + + Set the left and right margins on this class to give the desired amount + of space between the graph and its adjacent columns in the timeline. + + **Additional Classes** + + * `.sel`: See [`.tl-node`](#tl-node) for more information. + +* <a id="tl-rail"></a>**`.tl-rail`** + + Think of rails as invisible vertical lines on which check-in nodes are + placed. The more simultaneous branches in a graph, the more rails required + to draw it. Setting the `width` property on this class determines the + maximum spacing between rails. This spacing is automatically reduced as + the number of rails increases. If you change the `width` of `.tl-node` + elements, you'll probably need to change this value, too. + +* <a id="tl-mergeoffset"></a>**`.tl-mergeoffset`** + + A merge line often runs vertically right beside a primary child line. This + class's `width` property specifies the maximum spacing between the two. + Setting this value to `0` will eliminate the vertical merge lines. + Instead, the merge arrow will extend directly off the primary child line. + As with rail spacing, this is also adjusted automatically as needed. + +* <a id="tl-nodemark"></a>**`.tl-nodemark`** + + In the timeline table, the second cell in each check-in row contains an + invisible div with this class. These divs are used to determine the + vertical position of the nodes. By setting the `margin-top` property, + you can adjust this position. + +## <a id="vis-elems"></a>Visible Elements + +These are the elements you can actually see on the timeline graph: the nodes, +arrows, and lines. Each of these elements may also have additional classes +attached to them, depending on their context. + +* <a id="tl-node"></a>**`.tl-node`** + + A node exists for each check-in in the timeline. + + **Additional Classes** + + * `.leaf`: Specifies that the check-in is a leaf (i.e. that it has no + children in the same branch). + + * `.merge`: Specifies that the check-in contains a merge. + + * `.sel`: When the user clicks a node to designate it as the beginning + of a diff, this class is added to both the node itself and the + [`.tl-canvas`](#tl-canvas) element. The class is removed from both + elements when the node is clicked again. + +* <a id="tl-arrow"></a>**`.tl-arrow`** + + Arrows point from parent nodes to their children. Technically, this + class is just for the arrowhead. The rest of the arrow is composed + of [`.tl-line`](#tl-line) elements. + + There are six additional classes that are used to distinguish the different + types of arrows. However, only these combinations are valid: + + * `.u`: Up arrow that points to a child from its primary parent. + + * `.u.sm`: Smaller up arrow, used when there is limited space between + parent and child nodes. + + * `.merge.l` or `.merge.r`: Merge arrow pointing either to the left or + right. + + * `.warp`: A timewarped arrow (always points to the right), used when a + misconfigured clock makes a check-in appear to have occurred before its + parent ([example](https://www.sqlite.org/src/timeline?c=2010-09-29&nd)). + +* <a id="tl-line"></a>**`.tl-line`** + + Along with arrows, lines connect parent and child nodes. Line thickness is + determined by the `width` property, regardless of whether the line is + horizontal or vertical. You can also use borders to create special line + styles. Here's a CSS snippet for making dotted merge lines: + + .tl-line.merge { + width: 0; + background: transparent; + border: 0 dotted #000; + } + .tl-line.merge.h { + border-top-width: 1px; + } + .tl-line.merge.v { + border-left-width: 1px; + } + + **Additional Classes** + + * `.merge`: A merge line. + + * `.h` or `.v`: Horizontal or vertical. + + * `.warp`: A timewarped line. + + +## <a id="default-css"></a>Default Timeline Graph CSS + + .tl-canvas { + margin: 0 6px 0 10px; + } + .tl-rail { + width: 18px; + } + .tl-mergeoffset { + width: 2px; + } + .tl-nodemark { + margin-top: 5px; + } + .tl-node { + width: 10px; + height: 10px; + border: 1px solid #000; + background: #fff; + cursor: pointer; + } + .tl-node.leaf:after { + content: ''; + position: absolute; + top: 3px; + left: 3px; + width: 4px; + height: 4px; + background: #000; + } + .tl-node.sel:after { + content: ''; + position: absolute; + top: 2px; + left: 2px; + width: 6px; + height: 6px; + background: red; + } + .tl-arrow { + width: 0; + height: 0; + transform: scale(.999); + border: 0 solid transparent; + } + .tl-arrow.u { + margin-top: -1px; + border-width: 0 3px; + border-bottom: 7px solid #000; + } + .tl-arrow.u.sm { + border-bottom: 5px solid #000; + } + .tl-line { + background: #000; + width: 2px; + } + .tl-arrow.merge { + height: 1px; + border-width: 2px 0; + } + .tl-arrow.merge.l { + border-right: 3px solid #000; + } + .tl-arrow.merge.r { + border-left: 3px solid #000; + } + .tl-line.merge { + width: 1px; + } + .tl-arrow.warp { + margin-left: 1px; + border-width: 3px 0; + border-left: 7px solid #600000; + } + .tl-line.warp { + background: #600000; + } diff --git a/www/customskin.md b/www/customskin.md new file mode 100644 index 0000000..2df2b13 --- /dev/null +++ b/www/customskin.md @@ -0,0 +1,598 @@ +# Skinning the Fossil Web Interface + +The Fossil web interface comes with a pre-configured look and feel. The default +look and feel works fine in many situations. However, you may want to change +the look and feel (the "skin") of Fossil to better suite your own individual tastes. +This document provides background information to aid you in that task. + +## <a id="builtin"></a>Built-in Skins + +Fossil comes with [multiple built-in skins](/skins). If the default skin does not +suite your tastes, perhaps one of the other built-in skins will work better. +If nothing else, the built-in skins can serve as examples or templates that +you can use to develop your own custom skin. + +The sources to these built-ins can +be found in the Fossil source tree under the skins/ folder. The +[skins/](/dir?ci=trunk&name=skins) +folder contains a separate subfolder for each built-in skin, with each +subfolders holding at least these five files: + + * css.txt + * details.txt + * footer.txt + * header.txt + * js.txt + +Try out the built-in skins by using the --skin option on the +[fossil ui](/help/ui) or [fossil server](/help/server) commands. + +## <a id="sharing"></a>Sharing Skins + +The skin of a repository is not part of the versioned state and does not +"push" or "pull" like checked-in files. The skin is local to the +repository. However, skins can be shared between repositories using +the [fossil config](/help/configuration) command. +The "fossil config push skin" command will send the local skin to a remote +repository and the "fossil config pull skin" command will import a skin +from a remote repository. The "fossil config export skin FILENAME" +will export the skin for a repository into a file FILENAME. This file +can then be imported into a different repository using the +"fossil config import FILENAME" command. Unlike "push" and "pull", +the "export" and "import" commands are able to move skins between +repositories for different projects. So, for example, if you have a +group of related repositories, you can develop a skin for one of them, +then get a consistent look across all the repositories by exporting +the skin from the first repository and importing into all the others. + +The file generated by "fossil config export" could be checked into +one of your repositories and versioned, if desired. This will not +automatically change the skin when looking backwards in time, but it +will provide an historical record of what the skin used to be and +allow the historical look of the repositories to be recreated if +necessary. + +When cloning a repository, the skin of the new repository is initialized to +the skin of the repository from which it was cloned. + +# Structure Of A Fossil Web Page + +Every HTML page generated by Fossil has the same basic structure: + +| Fossil-Generated HTML Header | +| Skin Header | +| Fossil-Generated Content | +| Skin Footer | +| Fossil-Generated HTML Footer | + +By default, Fossil starts every generated HTML page with this: + + <html> + <head> + <base href="..."> + <meta http-equiv="Content-Security-Policy" content="...."> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <title>....</title> + <link rel="stylesheet" href="..." type="text/css"> + </head> + <body class="FEATURE"> + +Fossil used to require a static version of this in every skin’s Header +area, but over time, we have found good cause to generate multiple +elements at runtime. + +One such is the `FEATURE` element, being either the top-level HTTP +request routing element (e.g. `doc`) or an aggregate feature class that +groups multiple routes under a single name. A prime example is `forum`, +which groups the `/forummain`, `/forumpost`, and `/forume2` routes, +allowing per-feature CSS. For instance, to style `<blockquote>` tags +specially for forum posts written in Markdown, leaving all other block +quotes alone, you could say: + + body.forum div.markdown blockquote { + margin-left: 10px; + } + +You can [override this generated HTML header](#override) by including a +“`<body>`” tag somewhere in the Header area of the skin, but it is +almost always best to limit a custom skin’s Header section to something +like this: + +<div class="sidebar" id="version-2.24">Prior to Fossil 2.24, we used +generic `<div>` elements to mark up these sections of the header, but we +switched to these semantic tag names to give browser accessibility +features more freedom to do intelligent things with the page content. +Those who made custom skins based on the old way of doing things will +need to track this change when upgrading, else the corresponding CSS +will mistarget the page header elements. Also, if you’re using Fossil’s +chat feature, failing to track this change will cause it to miscalculate +the message area size, resulting in double scrollbars. Simply diffing +your custom header in the skin editor against the stock version should +be sufficient to show what you need to change.</div> + + + <header> + ... + </header> + <nav class="mainmenu" title="Main Menu"> + ... + </nav> + <nav id="hbdrop" class="hbdrop" title="sitemap"></nav> + +See the stock skins’ headers for ideas of what to put in place of the +ellipses. + +The Fossil-generated Content section immediately follows this Header. +It will look like this: + + <div class="content"> + ... Fossil-generated content here ... + </div> + +After the Content is the custom Skin Footer section which should +follow this template: + + <footer> + ... skin-specific stuff here ... + </footer> + +As with the `<header>` change called out above, this, too, is a breaking +change in Fossil 2.24. + +Finally, Fossil always adds its own footer (unless overridden) +to close out the generated HTML: + + </body> + </html> + +## <a id="mainmenu"></a>Changing the Main Menu Contents + +The actual text content of the skin’s main menu is not +part of the skin proper if you’re using one of the stock skins. +If you look at the Header section of the skin, you’ll find a +`<div class="mainmenu">` element whose contents are set by a short +[TH1](./th1.md) script from the contents of the **Main Menu** section of +the Setup → Configuration screen. + +This feature allows the main menu contents to stay the same across +different skins, so you no longer have to reapply menu customizations +when trying different skins. + +See the [`capexpr`](./th1.md#capexpr) section of the TH1 docs for help +on interpreting the default contents of this block. + + +## <a id="override"></a>Overriding the HTML Header and Footer + +Notice that the `<html>`, `<head>`, and opening `<body>` +elements at the beginning of the document, +and the closing `</body>` and `</html>` elements at the end are automatically +generated by Fossil. This is recommended. + +However, for maximum design flexibility, Fossil allows those elements to be +supplied as part of the configurable Skin Header and Skin Footer. +If the Skin Header contains the text "`<body`", then Fossil assumes that +the Skin Header and Skin Footer will handle all of the `<html>`, +`<head>`, and `<body>` text itself, and the Fossil-generated header and +footer will be blank. + +When overriding the HTML Header in this way, you will probably want to use some +of the [TH1 variables documented below](#vars) such as `$stylesheet_url` +to avoid hand-writing code that Fossil can generate for you. + +# Designing, Debugging, and Installing A Custom Skin + +It is possible to develop a new skin from scratch. But a better and easier +approach is to use one of the existing built-in skins as a baseline and +make incremental modifications, testing after each step, to obtain the +desired result. + +The skin is controlled by five files: + +<dl> +<dt><b>css.txt</b></dt> + +<dd>The css.txt file is the text of the CSS for Fossil. +Fossil might add additional CSS elements after +the css.txt file, if it sees that the css.txt omits some +CSS components that Fossil needs. But for the most part, +the content of the css.txt is the CSS for the page.</dd> + +<dt><b>details.txt</b><dt> + +<dd>The details.txt file is short list of settings that control +the look and feel, mostly of the timeline. The default +details.txt file looks like this: + +<pre> +pikchr-background: "" +pikchr-fontscale: "" +pikchr-foreground: "" +pikchr-scale: "" +timeline-arrowheads: 1 +timeline-circle-nodes: 1 +timeline-color-graph-lines: 1 +white-foreground: 0 +</pre> + +The three "timeline-" settings in details.txt control the appearance +of certain aspects of the timeline graph. The number on the +right is a boolean - "1" to activate the feature and "0" to +disable it. The "white-foreground:" setting should be set to +"1" if the page color has light-color text on a darker background, +and "0" if the page has dark text on a light-colored background. + +If the "pikchr-foreground" setting +is defined and is not an empty string then it specifies a +foreground color to use for [pikchr diagrams](./pikchr.md). The +default pikchr foreground color is black, or white if the +"white-foreground" boolean is set. The "pikchr-background" +settings does the same for the pikchr diagram background color. +If the "pikchr-fontscale" and "pikchr-scale" values are not +empty strings, then they should be floating point values (close +to 1.0) that specify relative scaling of the fonts in pikchr +diagrams and other elements of the diagrams, respectively. +</dd> + +<dt><b>footer.txt</b> and <b>header.txt</b></dt> + +<dd>The footer.txt and header.txt files contain the Skin Footer +and Skin Header respectively. Of these, the Skin Header is +the most important, as it contains the markup used to generate +the banner and menu bar for each page. + +Both the footer.txt and header.txt file are +[processed using TH1](#headfoot) prior to being output as +part of the overall web page.</dd> + +<dt><b>js.txt</b></dt> + +<dd>The js.txt file is optional. It is intended to be javascript. +The complete text of this javascript might be inserted into +the Skin Footer, after being processed using TH1, using +code like the following in the "footer.txt" file: + +<pre> +<script nonce="$nonce"> + <th1>styleScript</th1> +</script> +</pre> + +The js.txt file was originally used to insert javascript +that controls the hamburger menu in the default skin. More +recently, the javascript for the hamburger menu was moved into +a separate built-in file. Skins that use the hamburger menu +typically cause the javascript to be loaded by including the +following TH1 code in the "header.txt" file: + +<pre> +<th1>builtin_request_js hbmenu.js</th1> +</pre> + +The difference between `styleScript` and `builtin_request_js` +is that the `styleScript` command interprets the file +using TH1 and injects the content directly into the output +stream, whereas the `builtin_request_js` command inserts the +Javascript verbatim and does so at some unspecified future time +down inside the Fossil-generated footer. +You can use either +approach in custom skins that you create. + +Note that the "js.txt" file is *not* automatically inserted into +the generate HTML for a page. You, the skin designer, must +cause the javascript to be inserted by issuing appropriate +TH1 commands in the "header.txt" or "footer.txt" files.</dd> +</dl> + +Developing a new skin is simply a matter of creating appropriate +versions of these five control files. + +### Skin Development Using The Web Interface + +Users with admin privileges can use the Admin/Skin configuration page +on the web interface to develop a new skin. The development of a new +skin occurs without disrupting the existing skin. So you can work on +a new skin for a Fossil instance while the existing skin is still in +active use. + +The new skin is a "draft" skin. You initialize one of 9 draft skins +to either the current skin or to one of the built-in skins. Then +use forms to edit the 5 control files described above. The new +skin can be tested after each edit. Finally, once the new skin is +working as desired, the draft skin is "published" and becomes the +new live skin that most users see. + +### Skin Development Using A Local Text Editor + +An alternative approach is to copy the five control files for your +baseline skin into a temporary working directory (here called +"./newskin") and then launch the [fossil ui](/help/ui) command +with the "--skin ./newskin" option. If the argument to the --skin +option contains a "/" character, then the five control files are +read out of the directory named. You can then edit the control +files in the ./newskin folder using you favorite text editor, and +press "Reload" on your browser to see the effects. + + +### Disabling The Web Browser Cache During Development + +Fossil is aggressive about asking the web browser to cache +resources. While developing a new skin, it is often helpful to +put your web browser into developer mode and disable the cache. +If you fail to do this, then you might make some change to your skin +under development and press "Reload" only to find that the display +did not change. After you have finished work your skin, the +caches should synchronize with your new design and you can reactivate +your web browser's cache and take it out of developer mode. + +## <a id="headfoot"></a>Header and Footer Processing + +The `header.txt` and `footer.txt` control files of a skin are the HTML text +of the Skin Header and Skin Footer, except that before being inserted +into the output stream, the text is run through a +[TH1 interpreter](./th1.md) that might adjust the text as follows: + + * All text within <th1>...</th1> is omitted from the + output and is instead run as a TH1 script. That TH1 + script has the opportunity to insert new text in place of itself, + or to inhibit or enable the output of subsequent text. + + * Text of the form "$NAME" or "$<NAME>" is replaced with + the value of the TH1 variable NAME. See the [TH1 Variables](#vars) + section for more information on the two possible variable formats. + +For example, first few lines of a typical Skin Header will look +like this: + + <header> + <div class="title"><h1>$<project_name></h1>$<title>/div> + +After variables are substituted by TH1, that will look more like this: + + <header> + <div class="title"><h1>Project Name</h1>Page Title</div> + +As you can see, two TH1 variable substitutions were done. + +The same TH1 interpreter is used for both the header and the footer +and for all scripts contained within them both. Hence, any global +TH1 variables that are set by the header are available to the footer. + +## <a id="menu"></a>Customizing the ≡ Hamburger Menu + +The menu bar of the default skin has an entry to open a drop-down menu with +additional navigation links, represented by the ≡ button (hence the name +"hamburger menu"). The Javascript logic to open and close the hamburger menu +when the button is clicked is usually handled by a script named +"hbmenu.js" that is one of the [built-in resource files](/test-builtin-files) +that are part of Fossil. + +The ≡ button for the hamburger menu is added to the menu bar by the following +TH1 commands in the `header.txt` file, right before the menu bar links: + + html "<a id='hbbtn' href='$home/sitemap'>☰</a>" + builtin_request_js hbmenu.js + +The hamburger button can be repositioned between the other menu links (but the +drop-down menu is always left-aligned with the menu bar), or it can be removed +by deleting the above statements. The "html" statement inserts the appropriate +`<a>` for the hamburger menu button (some skins require something slightly +different - for example the ardoise skins wants "`<li><a>`"). The +"builtin_request_js hbmenu.js" asks Fossil to include the "hbmenu.js" +resource files in the Fossil-generated footer. + +The hbmenu.js script requires +the following `<div>` element somewhere in your header, in which to build +the hamburger menu. + + <div id='hbdrop'></div> + +Out of the box, the contents of the panel is populated with the [Site +Map](/sitemap), but only if the panel does not already contain any HTML +elements (that is, not just comments, plain text or non-presentational white +space). So the hamburger menu can be customized by replacing the empty `<div +id='hbdrop'></div>` element with a menu structure knitted according to the +following template: + + <div id="hbdrop" data-anim-ms="400"> + <ul class="columns" style="column-width: 20em; column-count: auto"> + <!-- NEW GROUP WITH HEADING LINK --> + <li> + <a href="$home$index_page">Link: Home</a> + <ul> + <li><a href="$home/timeline">Link: Timeline</a></li> + <li><a href="$home/dir?ci=tip">Link: File List</a></li> + </ul> + </li> + <!-- NEW GROUP WITH HEADING TEXT --> + <li> + Heading Text + <ul> + <li><a href="$home/doc/trunk/www/customskin.md">Link: Theming</a></li> + <li><a href="$home/doc/trunk/www/th1.md">Link: TH1 Scripts</a></li> + </ul> + </li> + <!-- NEXT GROUP GOES HERE --> + </ul> + </div> + +The custom `data-anim-ms` attribute can be added to the panel element to direct +the Javascript logic to override the default menu animation duration of 400 ms. +A faster animation duration of 80-200 ms may be preferred for smaller menus. The +animation is disabled by setting the attribute to `"0"`. + + +## <a id="vars"></a>TH1 Variables + +Before expanding the TH1 within the header and footer, Fossil first +initializes a number of TH1 variables to values that depend on +repository settings and the specific page being generated. + +Variables holding text that is loaded from "external, potentially untrusted" +sources (including the repository settings) are treated as [tainted strings] +(./th1.md#taint) and must be noted in the `$<NAME>` form, instead of `$NAME`, +or they may trigger an error (see the linked document for details). The +`$<NAME>` form corresponds to the TH1 statement `puts [ htmlize "$NAME" ]`, +where the [htmlize](./th1.md#htmlize) function escapes the tainted string, +making it safe for output in HTML code. + + + * **`project_name`** - The project_name variable is filled with the + name of the project as configured under the Admin/Configuration + menu. This is a [tainted string](./th1.md#taint) variable and must + be used as `$<project_name>`. + + * **`project_description`** - The project_description variable is + filled with the description of the project as configured under + the Admin/Configuration menu. This is a [tainted string] + (./th1.md#taint) variable and must be used as `$<project_description>`. + + * **`mainmenu`** - The mainmenu variable contains a TCL list with the main + menu entries. See the [mainmenu](/help/mainmenu) setting for details. + + * **`title`** - The title variable holds the title of the page being + generated. + + The title variable is special in that it is deleted after + the header script runs and before the footer script. This is + necessary to avoid a conflict with a variable by the same name used + in my ticket-screen scripts. + + * **`baseurl`** - The root of the URL namespace for this server. + + * **`secureurl`** - The same as $baseurl except that if the scheme is + "http:" it is changed to "https:" + + * **`home`** - The $baseurl without the scheme and hostname. For example, + if the $baseurl is "http://projectX.com/cgi-bin/fossil" then the + $home will be just "/cgi-bin/fossil". + + * **`index_page`** - The landing page URI as + specified by the Admin/Configuration setup page. + + * **`current_page`** - The name of the page currently being processed, + without the leading "/" and without query parameters. + Examples: "timeline", "doc/trunk/README.txt", "wiki". + + * **`current_checkin`** - The [check-in name](./checkin_names.wiki) to + which the current + [/ci](/help/www/ci), + [/dir](/help/www/dir), + [/tree](/help/www/tree), + [/timeline](/help/www/timeline) or + [/vinfo](/help/www/vinfo) + page refers, or undefined for all other pages. This variable is + derived from query parameters and is therefore a [tainted string] + (./th1.md#taint). + + * **`csrf_token`** - A token used to prevent cross-site request forgery. + + * **`default_csp`** - [Fossil’s default CSP](./defcsp.md) unless + [overridden by custom TH1 code](./defcsp.md#th1). Useful within + the skin for inserting the CSP into a `<meta>` tag within [a + custom `<head>` element](#headfoot). + + * **`nonce`** - The value of the cryptographic nonce for the request + being processed. + + * **`release_version`** - The release version of Fossil. Ex: "1.31" + + * **`manifest_version`** - A prefix on the check-in hash of the + specific version of fossil that is running. Ex: "\[47bb6432a1\]" + + * **`manifest_date`** - The date of the source-code check-in for the + version of fossil that is running. + + * **`compiler_name`** - The name and version of the compiler used to + build the fossil executable. + + * **`login`** - This variable only exists if the user has logged in. + The value is the username of the user. + + * **`stylesheet_url`** - A URL for the internal style-sheet maintained + by Fossil. + + * **`logo_image_url`** - A URL for the logo image for this project, as + configured on the Admin/Logo page. + + * **`background_image_url`** - A URL for a background image for this + project, as configured on the Admin/Logo page. + +All of the above are variables in the sense that either the header or the +footer is free to change or erase them. But they should probably be treated +as constants. New predefined values are likely to be added in future +releases of Fossil. + + +## <a id="procedure"></a>Suggested Skin Customization Procedure + +Developers are free, of course, to develop new skins using any method they +want, but the following is a technique that has worked well in the past and +can serve as a starting point for future work: + + 1. Select a built-in skin that is closest to the desired look. Make + copies of the css, footer, and header into files name "css.txt", + "details.txt", + "footer.txt", and "header.txt" in some temporary directory. + + If the Fossil source code is available, then these three files can + be copied directly out of one of the subdirectories under skins. If + sources are not easily at hand, then a copy/paste out of the + CSS, footer, and header editing screens under the Admin menu will + work just as well. The important point is that the three files + be named exactly "css.txt", "footer.txt", and "header.txt" and that + they all be in the same directory. + + 2. Run the [fossil ui](/help/ui) command with an extra + option "--skin SKINDIR" where SKINDIR is the name of the directory + in which the three txt files were stored in step 1. This will bring + up the Fossil website using the tree files in SKINDIR. + + 3. Edit the *.txt files in SKINDIR. After making each small change, + press Reload on the web browser to see the effect of that change. + Iterate until the desired look is achieved. + + 4. Copy/paste the resulting css.txt, details.txt, + header.txt, and footer.txt files + into the CSS, details, header, and footer configuration screens + under the Admin/Skins menu. Alternately, import them using the + process described below. + +An alternative to step 4 is to convert the skin files into a form +which can be imported into a repository using `fossil config import`. +It requires compiling [a small tool from the fossil source +tree](/file/tools/skintxt2config.c): + +> +``` +$ cc -o s2c /path/to/fossil/checkout/tools/skintxt2config.c +``` + +With that in place, the custom skin files can be converted with: + +> +``` +$ ./s2c yourskin/*.txt > skin.config +``` + +It can be imported into an arbitrary fossil repository with: + +> +``` +$ fossil config import skin.config +``` + +And it can be pushed to a remote repository with: + +> +``` +$ fossil config push skin +``` + +That approach has proven to be an effective way to locally develop +skin changes then push them to a "live" site. + + +## See Also + +* [Customizing the Timeline Graph](customgraph.md) diff --git a/www/defcsp.md b/www/defcsp.md new file mode 100644 index 0000000..0e61fda --- /dev/null +++ b/www/defcsp.md @@ -0,0 +1,490 @@ +# The Default Content Security Policy (CSP) + +When Fossil’s web interface generates an HTML page, it normally includes +a [Content Security Policy][csp] (CSP) in the `<head>`. The CSP specifies +allowed sources for external resources such as images, +CSS, javascript, and so forth. +The purpose of CSP is to provide an extra layer of protection against +[cross-site scripting][xss] (XSS) and code injection +attacks. Compatible web browsers will not use external resources unless +they are specifically allowed by the CSP, which dramatically reduces +the attack surface of the application. + +Fossil does not rely on CSP for security. +A Fossil server should be secure from attack even without CSP. +Fossil includes built-in server-side content filtering logic. +For example, Fossil purposely breaks `<script>` tags when it finds +them in Markdown and Fossil Wiki documents. And the Fossil build +process scans the source code for potential injection vulnerabilities +and refuses to compile if any problems are found. +However, CSP provides an additional layer of defense against undetected +bugs that might lead to a vulnerability. + +## The Default Restrictions + +The default CSP used by Fossil is as follows: + +<pre> +default-src 'self' data:; +script-src 'self' 'nonce-$nonce'; +style-src 'self' 'unsafe-inline'; +img-src * data:; +</pre> + +The default is recommended for most installations. However, +the site administrators can overwrite this default CSP using the +[default-csp setting](/help/default-csp). For example, +CSP restrictions can be completely disabled by setting the default-csp to: + + default-src *; + +The following sections detail the maining of the default CSP setting. + +### <a id="base"></a> default-src 'self' data: + +This policy means mixed-origin content isn’t allowed, so you can’t refer +to resources on other web domains. Browsers will ignore a link like the +one in the following Markdown under our default CSP: + +  + +If you look in the browser’s developer console, you should see a CSP +error when attempting to render such a page. + +The default policy does allow inline `data:` URIs, which means you could +[data-encode][de] your image content and put it inline within the +document: + +  + +That method is best used for fairly small resources. Large `data:` URIs +are hard to read and edit. There are secondary problems as well: if you +put a large image into a Fossil forum post this way, anyone subscribed +to email alerts will get a copy of the raw URI text, which can amount to +pages and pages of [ugly Base64-encoded text][b64]. + +For inline images within [embedded documentation][ed], it suffices to +store the referred-to files in the repo and then refer to them using +repo-relative URLs: + +  + +This avoids bloating the doc text with `data:` URI blobs: + +There are many other cases, [covered below](#serving). + +[b64]: https://en.wikipedia.org/wiki/Base64 +[svr]: ./server/ + + +### <a id="img"></a> img-src * data: + +It was not always thus, but after careful consideration, we’ve chosen to +leave the source of inline images unrestricted by default in Fossil. +This allows you to pull them in from remote systems, to pull them from +within the Fossil repository itself, or to use `data:` URIs. + +If you are certain all images come from only within the repository, you +can close off certain risks — tracking pixels, broken image format +decoders, system dialog box spoofing, etc. — by changing this to +“`img-src 'self'`” possibly followed by “`data:`” if you will also use +`data:` URIs. + + +### <a id="style"></a> style-src 'self' 'unsafe-inline' + +This policy allows CSS information to come from separate files hosted +under the Fossil repo server’s Internet domain. It also allows inline CSS +`<style>` tags within the document text. + +The `'unsafe-inline'` declaration allows CSS within individual HTML +elements: + + <p style="margin-left: 4em">Indented text.</p> + +As the "`unsafe-`" prefix on the name implies, the `'unsafe-inline'` +feature is suboptimal for security. However, there are +a few places in the Fossil-generated HTML that benefit from this +flexibility and the work-arounds are verbose and difficult to maintain. +Furthermore, the harm that can be done with style injections is far +less than the harm possible with injected javascript. And so the +`'unsafe-inline'` compromise is accepted for now, though it might +go away in some future release of Fossil. + + +### <a id="script"></a> script-src 'self' 'nonce-%s' + +This policy disables in-line JavaScript and only allows `<script>` +elements if the `<script>` includes a `nonce` attribute that matches the +one declared by the CSP. That nonce is a large random number, unique for +each HTTP page generated by Fossil, so an attacker cannot guess the +value, so the browser will ignore an attacker’s injected JavaScript. + +That nonce can only come from one of three sources, all of which should +be protected at the system administration level on the Fossil server: + +* **Fossil server C code:** All code paths in Fossil that emit + `<script>` elements include the `nonce` attribute. There are several + cases, such as the “JavaScript” section of a [custom skin][cs]. + That text is currently inserted into each HTML page generated by + Fossil,¹ which means it needs to include a `nonce` attribute to + allow it to run under this default CSP. We consider JavaScript + emitted via these paths to be safe because it’s audited by the + Fossil developers. We assume that you got your Fossil server’s code + from a trustworthy source and that an attacker cannot replace your + Fossil server binary. + +* **TH1 code:** The Fossil TH1 interpreter pre-defines the + [`$nonce` variable](./th1.md#nonce) for use in [custom skins][cs]. For + example, some of the stock skins that ship with Fossil include a + wall clock feature up in the corner that updates once a minute. + These paths are safe in the default Fossil configuration because + only the [all-powerful Setup user][su] can write TH1 code that + executes in the server’s running context. + + There is, however, [a default-disabled path](#xss) to beware of, + covered in the next section. + +* **[CGI server extensions][ext]:** Fossil exports the nonce to the + CGI in the `FOSSIL_NONCE` environment variable, which it can then + use in `<script>` elements it generates. Because these extensions + can only be installed by the Fossil server’s system administrator, + this path is also considered safe. + +[ext]: ./serverext.wiki +[su]: ./caps/admin-v-setup.md#apsu + + +#### <a id="xss"></a>Cross-Site Scripting via Ordinary User Capabilities + +We’re so restrictive about how we treat JavaScript because it can lead +to difficult-to-avoid scripting attacks. If we used the same CSP for +`<script>` tags [as for `<style>` tags](#style), anyone with check-in +rights on your repository could add a JavaScript file to your repository +and then refer to it from other content added to the site. Since +JavaScript code can access any data from any URI served under its same +Internet domain, and many Fossil users host multiple Fossil repositories +under a single Internet domain, such a CSP would only be safe if all of +those repositories are trusted equally. + +Consider [the Chisel hosting service](http://chiselapp.com/), which +offers free Fossil repository hosting to anyone on the Internet, all +served under the same `http://chiselapp.com/user/$NAME/$REPO` URL +scheme. Any one of those hundreds of repositories could trick you into +visiting their repository home page, set to [an HTML-formatted embedded +doc page][hfed] via Admin → Configuration → Index Page, with this +content: + + <script src="/doc/trunk/bad.js"></script> + +That script can then do anything allowed in JavaScript to *any other* +Chisel repository your browser can access. The possibilities for mischief +are *vast*. For just one example, if you have login cookies on four +different Chisel repositories, your attacker could harvest the login +cookies for all of them through this path if we allowed Fossil to serve +JavaScript files under the same CSP policy as we do for CSS files. + +This is why the default configuration of Fossil has no way for [embedded +docs][ed], [wiki articles][wiki], [tickets][tkt], [forum posts][fp], or +[tech notes][tn] to automatically insert a nonce into the page content. +This is all user-provided content, which could link to user-provided +JavaScript via check-in rights, effectively giving all such users a +capability that is usually reserved to the repository’s administrator. + +The default-disabled [TH1 documents feature][edtf] is the only known +path around this restriction. If you are serving a Fossil repository +that has any user you do not implicitly trust to a level that you would +willingly run any JavaScript code they’ve provided, blind, you **must +not** give the `--with-th1-docs` option when configuring Fossil, because +that allows substitution of the [pre-defined `$nonce` TH1 +variable](./th1.md#nonce) into [HTML-formatted embedded docs][hfed]: + + <script src="/doc/trunk/bad.js" nonce="$nonce"></script> + +Even with this feature enabled, you cannot put `<script>` tags into +Fossil Wiki or Markdown-formatted content, because our HTML generators +for those formats purposely strip or disable such tags in the output. +Therefore, if you trust those users with check-in rights to provide +JavaScript but not those allowed to file tickets, append to wiki +articles, etc., you might justify enabling TH1 docs on your repository, +since the only way to create or modify HTML-formatted embedded docs is +through check-ins. + +[ed]: ./embeddeddoc.wiki +[edtf]: ./embeddeddoc.wiki#th1 +[hfed]: ./embeddeddoc.wiki#html + + +## <a id="serving"></a>Serving Files Within the Limits + +There are several ways to serve files within the above restrictions, +avoiding the need to [override the default CSP](#override). In +decreasing order of simplicity and preference: + +1. Within [embedded documentation][ed] (only!) you can refer to files + stored in the repo using document-relative file URLs: + +  + +2. Relative file URLs don’t work from [wiki articles][wiki], + [tickets][tkt], [forum posts][fp], or [tech notes][tn], but you can + still refer to them inside the repo with [`/doc`][du] or + [`/raw`][ru] URLs: + +  + <img src="/raw/logo.png" style="float: right; margin-left: 2em"> + +3. Store the files as [unversioned content][uv], referred to using + [`/uv`][uu] URLs instead: + +  + +4. Use the [optional CGI server extensions feature](./serverext.wiki) + to serve such content via `/ext` URLs. + +5. Put Fossil behind a [front-end proxy server][svr] as a virtual + subdirectory within the site, so that our default CSP’s “self” rules + match static file routes on that same site. For instance, your repo + might be at `https://example.com/code`, allowing documents in that + repo to refer to: + + * images as `/image/foo.png` + * JavaScript files as `/js/bar.js` + * CSS style sheets as `/style/qux.css` + + Although those files are all outside the Fossil repo at `/code`, + keep in mind that it is the browser’s notion of “self” that matters + here, not Fossil’s. All resources come from the same Internet + domain, so the browser cannot distinguish Fossil-provided content + from static content served directly by the proxy server. + + This method opens up many other potential benefits, such as + [TLS encryption][tls], high-performance tuning via custom HTTP + headers, integration with other web technologies like PHP, etc. + +You might wonder why we rank in-repo content as most preferred above. It +is because the first two options are the only ones that cause such +resources to be included in an initial clone or in subsequent repo +syncs. The methods further down the list have a number of undesirable +properties: + +1. Relative links to out-of-repo files break in `fossil ui` when run on + a clone. + +2. Absolute links back to the public repo instance solve that: + +  + + ...but using them breaks some types of failover and load-balancing + schemes, because it creates a [single point of failure][spof]. + +3. Absolute links fail when one’s purpose in using a clone is to + recover from the loss of a project web site by standing that clone + up [as a server][svr] elsewhere. You probably forgot to copy such + external resources in the backup copies, so that when the main repo + site disappears, so do those files. + +Unversioned content is in the middle of the first list above — between +fully-external content and fully in-repo content — because it isn’t +included in a clone unless you give the `--unversioned` flag. If you +then want updates to the unversioned content to be included in syncs, +you have to give the same flag to [a `sync` command](/help/sync). +There is no equivalent with other commands such as `up` and `pull`, so +you must then remember to give `fossil uv` commands when necessary to +pull new unversioned content down. + +Thus our recommendation that you refer to in-repo resources exclusively. + +[du]: /help/www/doc +[fp]: ./forum.wiki +[ru]: /help/www/raw +[spof]: https://en.wikipedia.org/wiki/Single_point_of_failure +[tkt]: ./tickets.wiki +[tn]: ./event.wiki +[tls]: ./server/debian/nginx.md +[uu]: /help/www/uv +[uv]: ./unvers.wiki +[wiki]: ./wikitheory.wiki + + +## <a id="override"></a>Overriding the Default CSP + +If you wish to relax the default CSP’s restrictions or to tighten them +further, there are multiple ways to accomplish that. + +The following methods are listed in top-down order to give the simplest +and most straightforward method first. Further methods dig down deeper +into the stack, which is helpful to understand even if you end up using +a higher-level method. + + +### <a id="cspsetting"></a>The `default-csp` Setting + +If the [`default-csp` setting](/help/default-csp) is defined and is +not an empty string, its value is injected into the page using +[TH1](./th1.md) via one or more of the methods below, depending on the +skin you’re using and local configuration. + +Changing this setting is the easiest way to set a nonstandard CSP on +your site. + +Because a blank setting tells Fossil to use its hard-coded default CSP, +you have to say something like the following to get a repository without +content security policy restrictions: + + $ fossil set -R /path/to/served/repo.fossil default-csp 'default-src *' + +We recommend that instead of using the command line to change this +setting that you do it via the repository’s web interface, in +Admin → Settings. Write your CSP rules in the edit box marked +"`default-csp`". Do not add hard newlines in that box: the setting needs +to be on a single long line. Beware that changes take effect +immediately, so be careful with your edits: you could end up locking +yourself out of the repository with certain CSP changes! + +There are a few reasons why changing this setting via the command line +is inadvisable, except for very short settings like the example above: + +1. You have to be sure to set it on the repository where you want the + CSP to apply. Changing this setting on your local clone doesn’t + affect the remote repo you cloned from, which is most likely where + you want the CSP restrictions. + +2. For more complicated CSPs, the quoting rules for your shell and the + CSP syntax may interact, making it difficult or impossible to set + your desired CSP via the command line. Setting it via the web UI + doesn’t have this problem. + + + +### <a id="th1"></a>TH1 Setup Hook + +Fossil sets [the TH1 variable `$default_csp`][thvar] from the +`default-csp` setting and uses *that* to inject the value into generated +HTML pages in its stock configuration. + +This means that another way you can override this value is to use +the [`th1-setup` hook script](./th1-hooks.md), which runs before TH1 +processing happens during skin processing: + + $ fossil set th1-setup "set default_csp {default-src 'self'}" + +After [the above](#admin-ui), this is the cleanest method. + +[thvar]: ./customskin.md#vars + + + +### <a id="csrc"></a>Fossil C Source Code + +When you do neither of the above things, Fossil uses +[a hard-coded default](/info?ln=527-530&name=65a555d0d4fb846b). + +We tell you about this not to suggest that you hack the Fossil C source +code to change the CSP but simply to document the next step before we +move down-stack. + + + +### <a id="header"></a>Skin Header + +[In the normal case](./customskin.md#override), Fossil injects the CSP +retrieved by one of the above methods into the header of all HTML +documents it generates: + +```HTML +<head>... + <meta http-equiv="Content-Security-Policy" content="..."> + ... +``` + +Fossil skips this when you’re using a custom skin *and* its +[Header section](./customskin.md#headfoot) includes a `<body>` tag. This +is because prior to Fossil 2.5, the Header for a custom skin normally +contained everything from the opening `<html>` tag through the leading +`<body>` tag. From that version onward, Fossil now generates that header +when possible, so that the skin’s Header normally provides only the +opening tags of the document body, rather than the HTML header. + +When we added CSP support in Fossil 2.7, we made use of that mechanism +to inject the CSP into the generated HTML document header. + +For backwards compatibility, Fossil skips this when the skin’s Header +includes a `<body>` tag. Fossil takes that as a hint that it’s dealing +with a skin made in the pre-Fossil-2.5 days and doesn’t try to blindly +override it. + +The problem then is that you may be a Fossil user from the days before +Fossil 2.5, and you may be using a custom skin. This includes users who +selected one of the stock skins, since for the purposes of this section, +there is no difference between the cases. If you go into Admin → Skins → +Header and find a `<body>` tag, none of the above will apply to your +repo since Fossil will not be injecting its CSP into your pages. + +If you selected one of the stock skins (e.g. Khaki) prior to upgrading +to Fossil 2.5+ and didn’t make any changes to it since that time, you +can take the simplest option, which is to simply revert to the stock +version of the skin, so your pages will have the CSP injected, at which +point this document will begin describing what Fossil does with that +repo. + +If you’re using a customized version of one of the stock skins, the +skinning mechanism has a diff feature to make it easier to fold your +local changes into the stock version. + +If you’re using a fully customized skin, we recommend replicating the +method that [the Bootstrap skin uses][dcinj].² Alone among the stock +Fossil skins, Bootstrap still does old-style Header processing, +providing the entire HTML header and the start of the document body. + +We do *not* recommend injecting an explicit `Content-Security-Policy` +meta tag into a header to override Fossil’s default CSP. That means you +have to edit the skin every time you want to change the CSP. Use the TH1 +`$default_csp` variable like the Bootstrap skin does so you can use one +of the methods above with your custom skin, so the CSP can vary +independently of the skin. + +[dcinj]: /info?ln=7&name=bef080a6929a3e6f + + +### <a id="fep"></a>Front-End Proxy + +If your Fossil repo is behind some sort of HTTP [front-end proxy][svr], +the [preferred method][pmcsp] for setting the CSP is via a custom HTTP +header, which most HTTP reverse proxy programs allow. + +Beware that if you have a CSP set via both the HTTP and HTML headers +that the two CSPs [merge](https://stackoverflow.com/a/51153816/142454), +taking the most restrictive elements of each CSP. If you wish the proxy +layer’s setting to completely override Fossil’s setting, you will need +to combine that with one of the methods above to either remove the +Fossil-provided CSP or to make Fossil provide a no-restrictions CSP +which the front-end proxy can then tighten down. + +[pmcsp]: https://developers.google.com/web/fundamentals/security/csp/#the_meta_tag + + + +------------ + + +**Asides and Digressions:** + +1. Fossil might someday switch to serving the “JavaScript” section of a + custom skin as a virtual text file, allowing it to be cached by the + browser, reducing page load times. + +2. The stock Bootstrap skin *did* provide redundant CSP text from + Fossil 2.7 through Fossil 2.9, so setting the CSP via the higher + level methods did not work with that skin. We fixed this in Fossil + 2.10, but if you selected the Bootstrap skin prior to that, you’re + now running on a *copy* of it stored in your repo settings table, so + the change to the stock version of the skin won’t affect that repo + automatically. You will have to either merge the diffs in with your + local changes or revert to the stock version of the skin. + + +[cs]: ./customskin.md +[csp]: https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP +[de]: https://dopiaza.org/tools/datauri/index.php +[xss]: https://en.wikipedia.org/wiki/Cross-site_scripting diff --git a/www/delta-manifests.md b/www/delta-manifests.md new file mode 100644 index 0000000..6899cbe --- /dev/null +++ b/www/delta-manifests.md @@ -0,0 +1,246 @@ +# Delta Manifests + +<div class="sidebar">Do not confuse these with the core [Fossil delta +format](./delta_format.wiki). This document describes an optional +feature not enabled by default.</div> + +This article describes "delta manifests," a special-case form of +checkin manifest which is intended to take up far less space than +a normal checkin manifest, in particular for repositories with +many files. We'll see, however, that the space savings, if indeed +there are any, come with some caveats. + +This article assumes that the reader is at least moderately familiar +with Fossil's [artifact file format](./fileformat.wiki), in particular +the structure of checkin manifests, and it won't make much sense to +readers unfamiliar with that topic. + +# Background and Motivation of Delta Manifests + +A checkin manifest includes a list of every file in that checkin. A +moderately-sized project can easily have a thousand files, and every +checkin manifest will include those thousand files. As of this writing +Fossil's own checkins contain 989 files and the manifests are 80kb +each. Thus a checkin which changes only 2 bytes of source code +ostensibly costs another 80kb of storage for the manifest for that +change. + +Delta manifests were conceived as a mechanism to help combat that +storage overhead. + +# Makeup of a Delta Manifest + +A delta manifest is structured like a normal manifest (called a +"baseline" manifest) except that it has *two types of parents*: the +P-card which is part of (nearly) every manifest and a so-called +baseline (denoted by a B-card). The P-card tells us which artifact(s) +is/are the parents for purposes of the SCM version DAG. The B-card +tells us which manifest to use as a basis for this delta. The B-card +need not be, and often is not, the same as the P-card. Here's an +example: + +``` +B c04ce8aaf1170966c6f8abcce8b57e72a0fa2b81 +C Minor\sdoc\supdates... +D 2021-03-11T18:56:24.686 +F bindings/s2/shell_extend.c 6d8354c693120a48cfe4798812cd24499be174b2 +<15 F-cards snipped for brevity> +F src/repo.c 2f224cb0e59ccd90ba89c597e40b8e8d87506638 +P 61d3e64e6fb1a93d4a7b0182e4c6b94d178d66d9 +R a84ec2e8e1eb37ff0d94cac262795e23 +U stephan +Z 536e6d26dd8dbe2779d9e5f52a15518e +``` + +The B-card names another manifest, by its unique ID, the same way that +a P-card does. A manifest may have multiple P-card parents (the second +and subsequent ones denoting merge parents) but B-cards always refer +to exactly one parent. + +What unambiguously distinguishes this as a delta is the existence of +the B-card. All deltas have a B-card and no other type of artifact has +one. What also, but not unambiguously, distinguishes it as a delta is +that it has only 17 F-cards, whereas a baseline manifest in that same +repository has (as of this writing) 291 F-cards. In this particular +case, the delta manifest is 1363 bytes, compared to 20627 bytes for +the next checkin - a baseline manifest. That's a significant saving in +F-cards, especially if a repository contains thousands of files. That +savings, however, comes with caveats which we'll address below. + +Trivia regarding the B-card: + +- The B-card always refers to a baseline manifest, not another delta. +- Deltas may not chain with another delta, but any number of deltas + may have the same B-card. It is quite common for a series of delta + manifest checkins, each of which derives (in the P-card sense) from + the one before it, to have the same B-card. + +A delta manifest is functionally identical to a normal manifest except +that it has a B-card and how it records F-cards. Namely, it only +records F-cards which have changed at some point between this delta +and the version represented by the delta's B-card. This recording of +F-card *differences* also means that delta manifests, unlike normal +manifests, have to explicitly record deleted F-cards. Baseline +manifests do not record deletions. Instead, they include a list of +every file which is part of that checkin. Deltas, however, record the +differences between their own version and a baseline version, and thus +have to record deletions. They do this by including F-cards which have +only a file name and no hash. + +Iterating over F-cards in a manifest is something several important +internal parts of Fossil have to do. Iterating over a baseline +manifest, e.g. when performing a checkout, is straightforward: simply +walk through the list in the order the cards are listed. A delta, +however, introduces a significant wrinkle to that process. In short, +when iterating over a delta's F-cards, code has to compare the delta's +list to the baseline's list. If the delta has an entry the parent does +not have, or which is a newer entry for the same file, the delta's +entry is used. If the delta is missing an entry which the baseline +has, the baseline's entry is used. When a deletion F-card is +discovered in the delta (recall that baselines do not record +deletions), iteration over that card is skipped - the internal +algorithms which iterate over F-cards never report deletions to the +code iterating over those cards. The reason for that is consistency: +only deltas record file deletions, but the fact that it's a delta is +an internal detail, not something which higher-level code should +concern itself with. If higher-level iteration code were shown file +deletions, they would effectively be dealing with a leaky abstraction +and special-case handling which only applies to delta manifests. The +F-card iteration API hides such details from its users (other +Fossil-internal APIs). + + +# When does Fossil Create Deltas? + +By default, Fossil never creates delta manifests. It can be told to do +so using the `--delta` flag to the [`commit` +command](/help/commit). (Before doing so in your own repositories, +please read the section below about the caveats!) When a given +repository gets a delta manifest for the first time, Fossil records +that fact in the repository's `config` table with an entry named +`seen-delta-manifest`. If, in later sessions, Fossil sees that that +setting has a true value, it will *consider* creating delta manifests +by default. + +Conversely, the [`forbid-delta-manifests` repository config +setting](/help/forbid-delta-manifests) may be used to force Fossil to +*never* create deltas. That setting will propagate to other repository +clones via the sync process, to try to ensure that no clone introduces +a delta manifests. We'll cover reasons why one might want to use that +setting later on. + +After creating a delta manifest during the commit process, Fossil +examines the size of the delta. If, in Fossil's opinion, the space +savings are not significant enough to warrant the delta's own +overhead, it will discard the delta and create a new baseline manifest +instead. (The heuristic it uses for that purpose is tucked away in +Fossil's checkin algorithm.) + + +# Caveats + +Delta manifests may appear, on the surface, to be a great way to save +a few bytes of repository space. There are, however, caveats... + +## Space Savings? + +Though deltas were conceived as a way to save storage space, that +benefit is *not truly achieved* because... + +When a manifest is created, Fossil stores its parent version as a +[fossil delta](./delta_format.wiki) (as opposed to a delta manifest) +which succinctly descibes the differences between the parent and its +new child. This form of compression is extremely space-efficient and +can reduce the real storage space requirements of a manifest from tens +or hundreds of kilobytes down to a kilobyte or less for checkins which +modify only a few files. As an example, as of this writing, Fossil's +[tip checkin baseline manifest](/artifact/decd537016bf) is 80252 bytes +(uncompressed), and the delta-compressed baseline manifest of the +[previous checkin](/artifact/2f7c93f49c0e) is stored as a mere 726 +bytes of Fossil-delta'd data (not counting the z-lib compression which +gets applied on top of that). In this case, the tip version modified 7 +files compared to its parent version. + +Thus delta manifests do not *actually* save much storage space. They +save *some*, in particular in the tip checkin version: Fossil +delta-compresses *older* versions of checkins against the child +versions, as opposed to delta-compressing the children against the +parents. The reason is to speed up access for the most common case - +the latest version. Thus tip-version delta manifests are more +storage-space efficient than tip-version baseline manifests. Once the +next version is committed, though, and Fossil deltification is applied +to those manifests, that difference in space efficiency shrinks +tremendously, often to the point of insignificance. + +We can observe the Fossil-delta compression savings using a bit of +3rd-party code which can extract Fossil-format blobs both with and +without applying their deltas: + +``` +$ f-acat tip > A # tip version's manifest +$ f-acat prev --raw > B # previous manifest in its raw fossil-deltified form +$ f-acat prev > C # previous manifest fossil-undelta'd +$ ls -la A B C +-rw-rw-r-- 1 user user 80252 Mar 12 07:09 A # tip +-rw-rw-r-- 1 user user 726 Mar 12 07:09 B # previous: delta'd +-rw-rw-r-- 1 user user 80256 Mar 12 07:09 C # previous: undelta'd +``` + +For comparison's sake, when looking at a separate repository which +uses delta manifests, a delta-compressed delta manifest takes up +approximately the same space as a delta-compressed baseline manifest +(to within 10 bytes for the test samples). + +i.e. delta manifests may not save any storage space except for the tip +version! (*Surprise!*) + +In terms of RAM costs, deltas usually cost more memory than baseline +manifests. The reason is because traversing a delta requires having +not only that delta in memory, but also its baseline version. Delta +manifests are seldom used in ways which do not require also loading +their baselines. Thus Fossil internally requires two manifest objects +for most operations with a delta manifest, whereas a baseline has but +one. The difference in RAM cost is directly proportional to the size +of the delta manifest. + +## Manifests as Proof of Code Integrity + +Delta manifests have at least one more notable caveat, this one +arguably more significant than an apparent lack of space savings: +they're useless for purposes of publishing a manifest which downstream +clients can use to verify the integrity of their copy of the software. + +Consider this use case: [the SQLite project](https://sqlite.org) +publishes source code to many thousands of downstream consumers, many +of whom would like to be able to verify that the copy they have +downloaded is actually the copy published by the project. This is +easily achieved by providing a copy of the downloaded version's +manifest, as it contains a hash of every single file the project +published and the manifest itself has a well-known hash and is +cryptographically tamper-proof. It's mathematically extremely improbable for a +malicious party to modify such a manifest and re-publish it as an +"official" one, as the various hashes (F-cards, R-card, Z-card, *and* +the hash of the manifest itself) would not line up. A collision-based +attack would have to defeat *all four of those hashes*, which is +practically impossible to do. Thus a Fossil checkin manifest can be used +to provide strong assurances that a given copy of the software has not +been tampered with since being exported by Fossil. + +*However*, that use case is *only possible with baseline manifests*. +A delta manifest is *essentially useless* for that purpose. The +algorithm for traversing F-cards of a delta manifest is not trivial +for arbitrary clients to reproduce, e.g. using a shell script. While +it *could* be done in any higher-level programming language (or some +truly unsightly shell code), it would be an onerous burden on +downstream consumers and would not be without risks of having bugs +which invalidate the strong guarantees provided by the manifest. + +It's worth noting that the core Fossil project repository does not use +delta manifests, at least in part for the same reason the SQLite +project does not: the ability to provide a manifest which clients can +easily use to verify the integrity of the code they've downloaded. The +[`forbid-delta-manifests` config +setting](/help/forbid-delta-manifests) is used to ensure that none are +introduced into the repository beyond the few which were introduced +solely for testing purposes. + diff --git a/www/delta_encoder_algorithm.wiki b/www/delta_encoder_algorithm.wiki new file mode 100644 index 0000000..87f8216 --- /dev/null +++ b/www/delta_encoder_algorithm.wiki @@ -0,0 +1,248 @@ +<title>Fossil Delta Encoding Algorithm</title> + +<h2>Abstract</h2> + +<p>A key component for the efficient storage of multiple revisions of +a file in fossil repositories is the use of delta-compression, i.e. to +store only the changes between revisions instead of the whole +file.</p> + +<p>This document describes the encoding algorithm used by Fossil to +generate deltas. It is targeted at developers working on either +<a href="index.wiki">fossil</a> itself, or on tools compatible with +it. The exact format of the generated byte-sequences, while in general +not necessary to understand encoder operation, can be found in the +companion specification titled "<a href="delta_format.wiki">Fossil +Delta Format</a>". +</p> + +<p>The algorithm is inspired +by <a href="http://samba.anu.edu.au/rsync/">rsync</a>.</p> + +<h2 id="argresparam">1.0 Arguments, Results, and Parameters</h2> + +<p>The encoder takes two byte-sequences as input, the "original", and +the "target", and returns a single byte-sequence containing the +"delta" which transforms the original into the target upon its +application.</p> + +<p>Note that the data of a "byte-sequence" includes its length, +i.e. the number of bytes contained in the sequence.</p> + +<p>The algorithm has one parameter named "NHASH", the size of the +"sliding window" for the "rolling hash", in bytes. These two terms are +explained in the next section. The value of this parameter has to be a +power of two for the algorithm to work. For Fossil the value of this +parameter is set to "16".</p> + +<h2 id="operation">2.0 Operation</h2> + +<p>The algorithm is split into three phases which generate +the <a href="delta_format.wiki#header">header</a>, +<a href="delta_format.wiki#slist">segment list</a>, +and <a href="delta_format.wiki#trailer">trailer</a> of the delta, per +its general <a href="delta_format.wiki#structure">structure</a>.</p> + +<p>The two phases generating header and trailer are not covered here +as their implementation trivially follows directly from the +specification of the <a href="delta_format.wiki">delta format</a>.</p> + +<p>This leaves the segment-list. Its generation is done in two phases, +a pre-processing step operating on the "original" byte-sequence, +followed by the processing of the "target" byte-sequence using the +information gathered by the first step.</p> + +<h3 id="preprocessing">2.1 Preprocessing the original</h3> + +<p>A major part of the processing of the "target" is to find a range +in the "original" which contains the same content as found at the +current location in the "target".</p> + +<p>A naive approach to this would be to search the whole "original" +for such content. This however is very inefficient as it would search +the same parts of the "original" over and over. What is done instead +is to sample the "original" at regular intervals, compute signatures +for the sampled locations and store them in a hash table keyed by +these signatures.</p> + +<p>That is what happens in this step. The following processing step +can then the compute signature for its current location and then has +to search only a narrow set of locations in the "original" for +possible matches, namely those which have the same signature.</p> + +<p>In detail:</p> + +<ol> +<li>The "original" is split into chunks of NHASH bytes. Note that a +partial chunk of less than NHASH bytes at the end of "original" is +ignored. +</li> +<li>The <a href="#rollhash">rolling hash</a> of each chunk is +computed. +</li> +<li>A hash table is filled, mapping from the hashes of the chunks to +the list of chunk locations having this hash. +</li> +</ol> + +<h3 id="processing">2.2 Processing the target</h3> + +<p>This, the main phase of the encoder, processes the target in a loop +from beginning to end. The state of the encoder is captured by two +locations, the "base" and the "slide". "base" points to the first byte +of the target for which no delta output has been generated yet, and +"slide" is the location of the window used to look in the "origin" for +commonalities. This window is NHASH bytes long.</p> + +<p>Initially both "base" and "slide" point to the beginning of the +"target". In each iteration of the loop the encoder decides whether to +<ul> +<li>emit a single instruction +to <a href="delta_format.wiki#copyrange">copy a range</a>, or +</li> +<li>emit two instructions, first +to <a href="delta_format.wiki#insertlit">insert a literal</a>, then +to <a href="delta_format.wiki#copyrange">copy a range</a>, or +</li> +<li>move the window forward one byte. +</li> +</ul> +</p> + +<verbatim type="pikchr float-right"> +TARGET: [ + scale = 0.8 + down + "Target" bold + box fill palegreen width 150% height 200% "Processed" + GI: box same as first box fill yellow height 25% "Gap → Insert" + CC: box same fill orange height 200% "Common → Copy" + W: box same as GI fill lightgray width 125% height 200% "Window" bold + box same as CC height 125% "" + box same fill white "" +] + +[ "Base" bold ; right ; arrow 33% ] with .e at TARGET.GI.nw +[ "Slide" bold ; right ; arrow 33% ] with .e at TARGET.W.nw + +ORIGIN: [ + down + "Origin" bold + B1: box fill white + B2: box fill orange height 200% + B3: box fill white height 200% +] with .nw at 0.75 right of TARGET.ne + +arrow from TARGET.W.e to ORIGIN.B2.w "Signature" aligned above +</verbatim> + +<p>To make this decision the encoder first computes the hash value for +the NHASH bytes in the window and then looks at all the locations in +the "origin" which have the same signature. This part uses the hash +table created by the pre-processing step to efficiently find these +locations.</p> + +<p>For each of the possible candidates the encoder finds the maximal +range of bytes common to both "origin" and "target", going forward and +backward from "slide" in the "target", and the candidate location in +the "origin". This search is constrained on the side of the "target" +by the "base" (backward search), and the end of the "target" (forward +search), and on the side of the "origin" by the beginning and end of +the "origin", respectively.</p> + +<p>There are input files for which the hash chains generated by the +pre-processing step can become very long, leading to long search times +and affecting the performance of the delta generator. To limit the +effect such long chains can have the actual search for candidates is +bounded, looking at most N candidates. Currently N is set to 250.</p> + +<p>From the ranges for all the candidates the best (= largest) common +range is taken and it is determined how many bytes are needed to +encode the bytes between the "base" and the end of that range. If the +range extended back to the "base" then this can be done in a single +copy instruction. Otherwise, i.e if there is a gap between the "base" +and the beginning of the range then two instructions are needed, one +to insert the bytes in the gap as a literal, and a copy instruction +for the range itself. The general situation at this point can be seen +in the picture to the right.</p> + +<p>If the number of bytes needed to encode both gap (if present), and +range is less than the number of bytes we are encoding the encoder +will emit the necessary instructions as described above, set "base" +and "slide" to the end of the encoded range and start the next +iteration at that point.</p> + +<p>If, on the other hand, the encoder either did not find candidate +locations in the origin, or the best range coming out of the search +needed more bytes to encode the range than there were bytes in the +range, then no instructions are emitted and the window is moved one +byte forward. The "base" is left unchanged in that case.</p> + +<p>The processing loop stops at one of two conditions: +<ol> +<li>The encoder decided to move the window forward, but the end of the +window reached the end of the "target". +</li> +<li>After the emission of instructions the new "base" location is +within NHASH bytes of end of the "target", i.e. there are no more than +at most NHASH bytes left. +</li> +</ol> +</p> + +<p>If the processing loop left bytes unencoded, i.e. "base" not +exactly at the end of the "target", as is possible for both end +conditions, then one last insert instruction is emitted to put these +bytes into the delta.<p> + +<h2 id="exceptions">3.0 Exceptions</h2> + +<p>If the "original" is at most NHASH bytes long no compression of +changes is possible, and the segment-list of the delta consists of a +single literal which contains the entire "target".</p> + +<p>This is actually equivalent to the second end condition of the +processing loop described in the previous section, just checked before +actually entering the loop.</p> + +<h2 id="rollhash">4.0 The rolling hash</h2> + +<p>The rolling hash described below and used to compute content +signatures was chosen not only for good hashing properties, but also +to enable the easy (incremental) recalculation of its value for a +sliding window, i.e. where the oldest byte is removed from the window +and a new byte is shifted in.<p> + +<h3 id="rhdef">4.1 Definition</h3> + +<p>Assuming an array Z of NHASH bytes (indexing starting at 0) the +hash V is computed via</p> + +<div align="center"><img src="encode1.gif"></div> +<div align="center"><img src="encode2.gif"></div> +<div align="center"><img src="encode3.gif"></div> + +where A and B are unsigned 16-bit integers (hence the <u>mod</u>), and +V is a 32-bit unsigned integer with B as MSB, A as LSB. + +<h3 id="rhincr">4.2 Incremental recalculation</h3> + +<p>Assuming an array Z of NHASH bytes (indexing starting at 0) with +hash V (and components A and B), the dropped +byte <img src="encode4.gif" align="center">, and the new byte +<img src="encode5.gif" align="center"> , the new hash can +be computed incrementally via: </p> + +<div align="center"><img src="encode6.gif"></div> +<div align="center"><img src="encode7.gif"></div> +<div align="center"><img src="encode8.gif"></div> + +<p>For A, the regular sum, it can be seen easily that this the correct +way recomputing that component.</p> + +<p>For B, the weighted sum, note first that <img src="encode4.gif" +align="center"> has the weight NHASH in the sum, so that is what has +to be removed. Then adding in <img src="encode9.gif" align="center"> +adds one weight factor to all the other values of Z, and at last adds +in <img src="encode5.gif" align="center"> with weight 1, also +generating the correct new sum</p> diff --git a/www/delta_format.wiki b/www/delta_format.wiki new file mode 100644 index 0000000..ecc2a1e --- /dev/null +++ b/www/delta_format.wiki @@ -0,0 +1,316 @@ +<title>Fossil Delta Format</title> +<h1>1.0 Overview</h1> + +Fossil achieves efficient storage and low-bandwidth synchronization +through the use of delta-compression. Instead of storing +or transmitting the complete content of an artifact, Fossil stores or +transmits only the changes relative to a related artifact. + +This document describes the delta-encoding format used by Fossil. +The intended audience is developers working on either +<a href="index.wiki">Fossil</a> itself, or on tools compatible with +Fossil. Understanding of this document is <em>not</em> required for +ordinary users of Fossil. This document is an implementation detail. + +This document only describes the delta file format. A +[./delta_encoder_algorithm.wiki|separate document] describes one possible +algorithm for generating deltas in this format. + +<h2>1.1 Sample Software And Analysis Tools</h2> + +The core routines used to generate and apply deltas in this format +are contained in the [../src/delta.c|delta.c] source file. Interface +logic, including "test-*" commands to directly manipulate deltas are +contained in the [../src/deltacmd.c|deltacmd.c] source file. SQL functions +to create, apply, and analyze deltas are implemented by code in the +[../src/deltafunc.c|deltafunc.c] source file. + +The following command-line tools are available to create and apply +deltas and to test the delta logic: + + * [/help/test-delta|fossil test-delta] → Run self-tests of + the delta logic + + * [/help/test-delta-create|fossil test-delta-create X Y] → compute + a delta that converts file X into file Y. Output that delta. + + * [/help/test-delta-apply|fossil test-delta-apply X D] → apply + delta D to input file X and output the result. + + * [/help/test-delta-analyze|fossil test-delta-analyze X Y] → compute + and delta that converts file X into file Y but instead of writing the + delta to output, write performance information about the delta. + +When running the [/help/sqlite3|fossil sql] command to get an +interactive SQL session connected to the repository, the following +additional SQL functions are provided: + + * <b>delta_create(</b><i>X</i><b>,</b><i>Y</i><b>)</b> → + Compute a data that carries blob X into blob Y and return that delta + as a blob. + + * <b>delta_apply(</b><i>X</i><b>,</b><i>D</i><b>)</b> → + Apply delta D to input blob X return a new blob which is the result. + + + * <b>delta_output_size(</b><i>D</i>)</b> → + Return the size of the output that would result from applying delta D. + + * <b>delta_parse(</b><i>D</i>)</b> → This is a table-valued function + that returns one row for the header, for the trailer, and for each segment + in delta D. + + +<h1 id="structure">2.0 Structure</h1> +<verbatim type="pikchr"> + leftmargin = 0.1 + box height 50% "Header" + box same "Segments" + box same "Trailer" +</verbatim> + +A delta consists of three parts, a "header", a "trailer", and a +"segment-list" between them. + +Both header and trailer provide information about the target +helping the decoder, and the segment-list describes how the target can +be constructed from the original. + +<h2 id="header">2.1 Header</h2> +<verbatim type="pikchr"> + leftmargin = 0.1 + box height 50% "Size" + box same "\"\\n\"" +</verbatim> + +The header consists of a single number followed by a newline +character (ASCII 0x0a). The number is the length of the target in +bytes. + +This means that, given a delta, the decoder can compute the size of +the target (and allocate any necessary memory based on that) by simply +reading the first line of the delta and decoding the number found +there. In other words, before it has to decode everything else. + +<h2 id="trailer">2.2 Trailer</h2> +<verbatim type="pikchr"> + leftmargin = 0.1 + box height 50% "Checksum" + box same "\";\"" +</verbatim> + +The trailer consists of a single number followed by a semicolon (ASCII +0x3b). This number is a checksum of the target and can be used by a +decoder to verify that the delta applied correctly, reconstructing the +target from the original. + +The checksum is computed by treating the target as a series of +32-bit integer numbers (MSB first), and summing these up, modulo +2^32-1. A target whose length is not a multiple of 4 is padded with +0-bytes (ASCII 0x00) at the end. + +By putting this information at the end of the delta a decoder has +it available immediately after the target has been reconstructed +fully. + +<h2 id="slist">2.3 Segment-List</h2> +<verbatim type="pikchr"> + leftmargin = 0.1 + PART1: [ + B1: box height 50% width 15% "" + B2: box same "" + B3: box same "" + "***" + box height 50% width 15% "" + I1: line down 50% from B2 .s + arrow right until even with B3.e + box "Insert Literal" height 50% + line down 75% from I1 .s + arrow right until even with B3.e + box "Copy Range" height 50% + ] + down + PART2: [ + "" + box "Length" height 50% + right + box "\":\"" same + box "Bytes" same + ] with .nw at previous.sw +</verbatim> + +The segment-list of a delta describes how to create the target from +the original by a combination of inserting literal byte-sequences and +copying ranges of bytes from the original. This is where the +compression takes place, by encoding the large common parts of +original and target in small copy instructions. + +The target is constructed from beginning to end, with the data +generated by each instruction appended after the data of all previous +instructions, with no gaps. + +<h3 id="insertlit">2.3.1 Insert Literal</h3> + +A literal is specified by two elements, the size of the literal in +bytes, and the bytes of the literal itself. + +<verbatim type="pikchr"> + leftmargin = 0.1 + box "Length" height 50% + box "\":\"" same + box "Bytes" same +</verbatim> + +The length is written first, followed by a colon character (ASCII +0x3a), followed by the bytes of the literal. + +<h3 id="copyrange">2.3.2 Copy Range</h3> + +A range to copy is specified by two numbers, the offset of the +first byte in the original to copy, and the size of the range, in +bytes. The size zero is special, its usage indicates that the range +extends to the end of the original. + +<verbatim type="pikchr"> + leftmargin = 0.1 + box "Length" height 50% + box "\"@\"" same + box "Offset" same + box "\",\"" same +</verbatim> + + +The length is written first, followed by an "at" character (ASCII +0x40), then the offset, followed by a comma (ASCII 0x2c). + +<h1 id="intcoding">3.0 Encoding of integers</h1> + +The format currently handles only 32 bit integer numbers. They are +written base-64 encoded, MSB first, and without leading +"0"-characters, except if they are significant (i.e. 0 => "0"). + +The base-64 encoding uses one character for each 6 bits of +the integer to be encoded. The encoding characters are: + +<pre> +0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~ +</pre> + +The least significant 6 bits of the integer are encoded by +the first character, followed by +the next 6 bits, and so on until all non-zero bits of the integer +are encoded. The minimum number of encoding characters is used. +Note that for integers less than 10, the base-64 coding is a +ASCII decimal rendering of the number itself. + +<h1 id="examples">4.0 Examples</h1> + +<h2 id="examplesint">4.1 Integer encoding</h2> + +<table> +<tr> +<th>Value</th> +<th>Encoding</th> +</tr> +<tr> +<td>0</td> +<td>0</td> +</tr> +<tr> +<td>6246</td> +<td>1Xb</td> +</tr> +<tr> +<td>-1101438770</td> +<td>2zMM3E</td> +</tr> +</table> + +<h2 id="examplesdelta">4.2 Delta encoding</h2> + +An example of a delta using the specified encoding is: + +<pre> +1Xb +4E@0,2:thFN@4C,6:scenda1B@Jd,6:scenda5x@Kt,6:pieces79@Qt,F: Example: eskil~E@Y0,2zMM3E;</pre> +</pre> + +This can be taken apart into the following parts: + +<table> +<tr><th>What </th> <th>Encoding </th><th>Meaning </th><th>Details</th></tr> +<tr><td>Header</td> <td>1Xb </td><td>Size </td><td> 6246 </td></tr> +<tr><td>S-List</td> <td>4E@0, </td><td>Copy </td><td> 270 @ 0 </td></tr> +<tr><td> </td> <td>2:th </td><td>Literal </td><td> 2 'th' </td></tr> +<tr><td> </td> <td>FN@4C, </td><td>Copy </td><td> 983 @ 268 </td></tr> +<tr><td> </td> <td>6:scenda </td><td>Literal </td><td> 6 'scenda' </td></tr> +<tr><td> </td> <td>1B@Jd, </td><td>Copy </td><td> 75 @ 1256 </td></tr> +<tr><td> </td> <td>6:scenda </td><td>Literal </td><td> 6 'scenda' </td></tr> +<tr><td> </td> <td>5x@Kt, </td><td>Copy </td><td> 380 @ 1336 </td></tr> +<tr><td> </td> <td>6:pieces </td><td>Literal </td><td> 6 'pieces' </td></tr> +<tr><td> </td> <td>79@Qt, </td><td>Copy </td><td> 457 @ 1720 </td></tr> +<tr><td> </td> <td>F: Example: eskil</td><td>Literal </td><td> 15 ' Example: eskil'</td></tr> +<tr><td> </td> <td>~E@Y0, </td><td>Copy </td><td> 4046 @ 2176 </td></tr> +<tr><td>Trailer</td><td>2zMM3E </td><td>Checksum</td><td> -1101438770 </td></tr> +</table> + +The unified diff behind the above delta is + +<verbatim> +bluepeak:(761) ~/Projects/Tcl/Fossil/Devel/devel > diff -u ../DELTA/old ../DELTA/new +--- ../DELTA/old 2007-08-23 21:14:40.000000000 -0700 ++++ ../DELTA/new 2007-08-23 21:14:33.000000000 -0700 +@@ -5,7 +5,7 @@ + + * If the server does not have write permission on the database + file, or on the directory containing the database file (and +- it is thus unable to update database because it cannot create ++ it is thus unable to update the database because it cannot create + a rollback journal) then it currently fails silently on a push. + It needs to return a helpful error. + +@@ -27,8 +27,8 @@ + * Additional information displayed for the "vinfo" page: + + + All leaves of this version that are not included in the +- descendant list. With date, user, comment, and hyperlink. +- Leaves in the descendant table should be marked as such. ++ descendant list. With date, user, comment, and hyperlink. ++ Leaves in the descendant table should be marked as such. + See the compute_leaves() function to see how to find all + leaves. + + Add file diff links to the file change list. +@@ -37,7 +37,7 @@ + + * The /xfer handler (for push, pull, and clone) does not do + delta compression. This results in excess bandwidth usage. +- There are some code in xfer.c that are sketches of ideas on ++ There are some pieces in xfer.c that are sketches of ideas on + how to do delta compression, but nothing has been implemented. + + * Enhancements to the diff and tkdiff commands in the cli. +@@ -45,7 +45,7 @@ + single file. Allow diffs against any two arbitrary versions, + not just diffs against the current check-out. Allow + configuration options to replace tkdiff with some other +- visual differ of the users choice. ++ visual differ of the users choice. Example: eskil. + + * Ticketing interface (expand this bullet) +</verbatim> + + + +<h1 id="notes">Notes</h1> + +<ul> +<li>Pure text files generate a pure text delta. +</li> +<li>Binary files generate a delta that may contain some binary data. +</li> +<li>The delta encoding does not attempt to compress the content. +It was considered to be much +more sensible to do compression using a separate general-purpose +compression library, like <a href="http://www.zlib.net">zlib</a>. +</li> +</ul> diff --git a/www/embeddeddoc.wiki b/www/embeddeddoc.wiki new file mode 100644 index 0000000..6a6643d --- /dev/null +++ b/www/embeddeddoc.wiki @@ -0,0 +1,268 @@ +<title>Project Documentation</title> + +Fossil provides a built-in <a href="wikitheory.wiki">wiki</a> +that can be used to store the +documentation for a project. This is sufficient for many projects. +If your project is well-served by wiki documentation, then you +need read no further. + +But fossil also supports embedding project documentation as +files in the source tree. There are several potential advantages +to this approach: + + 1. The documentation files are versioned together with the + source code files so it is always clear what version of + the documentation goes with a particular release. + + 2. The documentation files can be edited using your favorite + text editor instead of having to use the web-based wiki + editor. + + 3. Only people with check-in privileges can modify the documentation. + (This might be either an advantage or disadvantage, depending + on the nature of your project.) + +We will call documentation that is included as files in the source tree +"embedded documentation". + +<h1>1.0 Fossil Support For Embedded Documentation</h1> + +The fossil web interface supports embedded documentation using +the "/doc" page. To access embedded documentation, one points +a web browser to a fossil URL of the following form: + +<pre> +<i><baseurl></i><big><b>/doc/</b></big><i><version></i><big><b>/</b></big><i><filename></i> +</pre> + +The <i><baseurl></i> is the main URL used to access the fossil web server. +For example, the <i><baseurl></i> for the fossil project itself is +[https://fossil-scm.org/home]. +If you launch the web server using the "[/help/ui|fossil ui]" command line, +then the <i><baseurl></i> is usually +<b>http://localhost:8080/</b>. + +The <i><version></i> is the +[./checkin_names.wiki|name of a check-in] +that contains the embedded document. This might be a hash prefix for +the check-in, or it might be the name of a branch or tag, or it might +be a timestamp. See the prior link +for more possibilities and examples. + +The <i id="ckout"><version></i> can +also be the special identifier "<b>ckout</b>". +The "<b>ckout</b>" keywords means to +pull the documentation file from the local source tree on disk, not +from the any check-in. The "<b>ckout</b>" keyword +only works when you start your server using the +"[/help/server|fossil server]" or "[/help?cmd=ui|fossil ui]" +commands. The "/doc/ckout" URL is intended to show a preview of +the documentation you are currently editing but have not yet checked in. + +The original designed purpose of the "ckout" feature is to allow the +user to preview local changes to documentation before committing the +change. This is an important facility, since unlike other document +languages like HTML, there is still a lot of variation among rendering +engines for Fossil's markup languages, Markdown and Fossil Wiki. Your +changes may look fine in your whizzy GUI Markdown editor, but what +actually matters is how <i>Fossil</i> will interpret your Markdown text. +Therefore, you should always preview your edits before committing them. + +To help make "ckout" easier to use, the "[/help?cmd=ui|fossil ui]" +command has the "--ckout-alias NAME" option that makes NAME an +alias for "ckout". If you are editing a collection of documents +that have hardcoded links to one another in the form of +"/doc/trunk/...", for example, you can test your changes by +running "fossil ui --ckout-alias trunk" and all of the links will +point to your uncommitted edits rather than to the latest trunk +check-in. + +Finally, the <i><filename></i> element of the URL is the +pathname of the documentation file relative to the root of the source +tree. + +The mimetype (and thus the rendering) of documentation files is +determined by the file suffix. Fossil currently understands +[/mimetype_list|many different file suffixes], +including all the popular ones such as ".css", ".gif", ".htm", +".html", ".jpg", ".jpeg", ".png", and ".txt". + +Documentation files whose names end in ".wiki" use the +[/wiki_rules | fossil wiki markup] - +a safe subset of HTML together with some wiki rules for paragraph +breaks, lists, and hyperlinks. +Documentation files ending in ".md" or ".markdown" use the +[/md_rules | Markdown markup language]. +Documentation files ending in ".txt" are plain text. +Wiki, markdown, and plain text documentation files +are rendered with the standard fossil header and footer added. +Most other mimetypes are delivered directly to the requesting +web browser without interpretation, additions, or changes. + +<h2 id="html">1.1 HTML Rendering With Fossil Headers And Footers</h2> + +Files with the mimetype "text/html" (the .html or .htm suffix) are +usually rendered directly to the browser without interpretation. +However, if the file begins with a <div> element like this: + + <b><div class='fossil-doc' data-title='<i>Title Text</i>'></b> + +Then the standard Fossil header and footer are added to the document +prior to being displayed. The "class='fossil-doc'" attribute is +required for this to occur. The "data-title='...'" attribute is +optional, but if it is present the text will become the title displayed +in the Fossil header. An example of this can be seen in Fossil +Documentation Index www/permutedindex.html: + + * [/file/www/permutedindex.html?txt|source text for <b>www/permutedindex.html</b>] + * [/doc/trunk/www/permutedindex.html|<b>www/permutedindex.html</b> rendered as HTML] + +Beware that such HTML files render in the same browser security context +as all other embedded documentation served from Fossil; they are not +fully-independent web pages. One practical consequence of this is that +embedded <tt><script></tt> tags will cause a +[https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP | Content +Security Policy] error in your browser with the default CSP as served by +Fossil. See the documentation on [./customskin.md#headfoot | Header and +Footer Processing] and [./defcsp.md | The Default CSP]. + + +<h1>2.0 Server-Side Text Substitution</h1> + +Fossil can do a few types of substitution of server-side information +into the embedded document. + +<h2>2.1 "$ROOT" In HTML and Markdown Hyperlinks</h2> + +Hyperlinks in Markdown and HTML embedded documents can reference +the root of the Fossil repository using the special text "$ROOT" +at the beginning of a URL. For example, a Markdown hyperlink to +the Markdown formatting rules might be +written in the embedded document like this: + +<verbatim> +[Markdown formatting rules]($ROOT/wiki_rules) +</verbatim> + +Depending on how the how the Fossil server is configured, that hyperlink +might be renderer like one of the following: + +<verbatim> +<a href="/wiki_rules">Wiki formatting rule</a> +<a href="/cgi-bin/fossil/wiki_rules">Wiki formatting rules</a> +</verbatim> + +So, in other words, the "$ROOT" text is converted into whatever +the "<baseurl>" is for the document. + +This substitution works for HTML and Markdown documents. +It does not work for Wiki embedded documents, since with +Wiki you can just begin a URL with "/" and it automatically knows +to prepend the $ROOT. + +<h2>2.2 "$CURRENT" In "/doc/" Hyperlinks</h2> + +Similarly, URLs of the form "/doc/$CURRENT/..." have the check-in +hash of the check-in currently being viewed substituted in place of +the "$CURRENT" text. This feature, in combination with the "$ROOT" +substitution above, allows an absolute path to be used for hyperlinks. + +For example, if an embedded document documented wanted to reference +some other document in a separate file named "www/otherdoc.md", +it could use a URL like this: + +<verbatim> +[Other Document]($ROOT/doc/$CURRENT/www/otherdoc.md) +</verbatim> + +As with "$ROOT", this substitution only works for Markdown and HTML +documents. For Wiki documents, you would need to use a relative URL. + +<h2 id="th1">2.3 TH1 Documents</h2> + +Enabling TH1 document support requires the following: + + * Configure the build with the <code>--with-th1-docs</code> flag. + * Enable the <code>th1-docs</code> setting, which is only available + after building with <code>--with-th1-docs</code>. + * Affected files must have a <code>.th1</code> file extension. + * The code to run must be embedded in blocks of + <code><th1>...</th1></code>. + +Fossil will substitute the value of [./th1.md | TH1 expressions] +within the <code><th1>...</th1></code> blocks into +the output HTML. + +Since TH1 is a full scripting language, this feature essential grants +the ability to execute code on the server to anyone with check-in +privilege for the project. +This is a security risk that needs to be carefully managed. +The feature is off by default. +Administrators should understand and carefully assess the risks +before enabling the use of TH1 within embedded documentation. + + +<h1>3.0 Examples</h1> + +This file that you are currently reading is an example of +embedded documentation. The name of this file in the fossil +source tree is "<b>www/embeddeddoc.wiki</b>". +You are perhaps looking at this +file using the URL: + +<pre>[https://fossil-scm.org/home/doc/trunk/www/embeddeddoc.wiki]</pre> + +The first part of this path, the "[https://fossil-scm.org/home]", +is the base URL. You might have originally typed: +[https://fossil-scm.org/]. The web server at the fossil-scm.org +site automatically redirects such links by appending "home". The +"home" file on fossil-scm.org is really a [./server/any/cgi.md|CGI script] +which runs the fossil web service in CGI mode. +The "home" CGI script looks like this: + +<pre> +#!/usr/bin/fossil +repository: /fossil/fossil.fossil +</pre> + +This is one of the many ways to set up a +<a href="./server/">Fossil server</a>. + +The "<b>/trunk/</b>" part of the URL tells fossil to use +the documentation files from the most recent trunk check-in. +If you wanted to see an historical version of this document, +you could substitute the name of a check-in for "<b>/trunk/</b>". +For example, to see the version of this document associated with +check-in [9be1b00392], simply replace the "<b>/trunk/</b>" with +"<b>/9be1b00392/</b>". You can also substitute the symbolic name +for a particular version or branch. For example, you might +replace "<b>/trunk/</b>" with "<b>/experimental/</b>" to get the latest +version of this document in the "experimental" branch. The symbolic name +can also be a date and time string in any of the following formats:</p> + +<ul> +<li> <i>YYYY-MM-DD</i> +<li> <i>YYYY-MM-DD<b>T</b>HH:MM</i> +<li> <i>YYYY-MM-DD<b>T</b>HH:MM:SS</i> +</ul> + +When the symbolic name is a date and time, fossil shows the version +of the document that was most recently checked in as of the date +and time specified. So, for example, to see what the fossil website +looked like at the beginning of 2010, enter: + +<pre><a href="https://fossil-scm.org/home/doc/2010-01-01/www/index.wiki">https://fossil-scm.org/home/doc/<b>2010-01-01</b>/www/index.wiki +</a></pre> + +The file that encodes this document is stored in the fossil source tree under +the name "<b>www/embeddeddoc.wiki</b>" and so that name forms the +last part of the URL for this document. + +As I sit writing this documentation file, I am testing my work by +running the "<b>fossil ui</b>" command line and viewing +<b>http://localhost:8080/doc/ckout/www/embeddeddoc.wiki</b> in +Firefox. I am doing this even though I have not yet checked in +the "<b>www/embeddeddoc.wiki</b>" file for the first time. Using +the special "<b>ckout</b>" version identifier on the "<b>/doc</b>" page +it is easy to make multiple changes to multiple files and see how they all +look together before committing anything to the repository. diff --git a/www/encode1.gif b/www/encode1.gif Binary files differnew file mode 100644 index 0000000..dab94c4 --- /dev/null +++ b/www/encode1.gif diff --git a/www/encode2.gif b/www/encode2.gif Binary files differnew file mode 100644 index 0000000..5526450 --- /dev/null +++ b/www/encode2.gif diff --git a/www/encode3.gif b/www/encode3.gif Binary files differnew file mode 100644 index 0000000..431080f --- /dev/null +++ b/www/encode3.gif diff --git a/www/encode4.gif b/www/encode4.gif Binary files differnew file mode 100644 index 0000000..daa85e5 --- /dev/null +++ b/www/encode4.gif diff --git a/www/encode5.gif b/www/encode5.gif Binary files differnew file mode 100644 index 0000000..fb4e4fe --- /dev/null +++ b/www/encode5.gif diff --git a/www/encode6.gif b/www/encode6.gif Binary files differnew file mode 100644 index 0000000..a1cc86b --- /dev/null +++ b/www/encode6.gif diff --git a/www/encode7.gif b/www/encode7.gif Binary files differnew file mode 100644 index 0000000..c3ce40d --- /dev/null +++ b/www/encode7.gif diff --git a/www/encode8.gif b/www/encode8.gif Binary files differnew file mode 100644 index 0000000..4abf30d --- /dev/null +++ b/www/encode8.gif diff --git a/www/encode9.gif b/www/encode9.gif Binary files differnew file mode 100644 index 0000000..2f98096 --- /dev/null +++ b/www/encode9.gif diff --git a/www/encryptedrepos.wiki b/www/encryptedrepos.wiki new file mode 100644 index 0000000..c9eb923 --- /dev/null +++ b/www/encryptedrepos.wiki @@ -0,0 +1,82 @@ +<title>How To Use Encrypted Repositories</title> + +<h2>Introduction</h2> + +Fossil can be compiled so that it works with encrypted repositories using +the [https://www.sqlite.org/see/doc/trunk/www/readme.wiki|SQLite Encryption Extension]. +This technical note explains the process. + +<h2>Building An Encryption-Enabled Fossil</h2> + +The SQLite Encryption Extension (SEE) is proprietary software and requires +[https://sqlite.org/purchase/see|purchasing a license]. + +Assuming you have an SEE license, the first step of compiling Fossil to +use SEE is to create an SEE-enabled version of the SQLite database source code. +This alternative SQLite database source file should be called "sqlite3-see.c" +and should be placed in the extsrc/ subfolder of the Fossil sources, right beside +the public-domain "sqlite3.c" source file. Also make a copy of the SEE-enabled +"shell.c" file, renamed as "shell-see.c", and place it in the extsrc/ subfolder +beside the original "shell.c". + +Add the --with-see command-line option to the configuration script to enable +the use of SEE on unix-like systems. + +<pre> +./configure --with-see; make +</pre> + +To build for Windows using MSVC, add +the "USE_SEE=1" argument to the "nmake" command line. + +<pre> +nmake -f makefile.msc USE_SEE=1 +</pre> + +<h2>Using Encrypted Repositories</h2> + +Any Fossil repositories whose filename ends with ".efossil" is taken to be +an encrypted repository. Fossil will prompt for the encryption password and +attempt to open the repository database using that password. + +Every invocation of fossil on an encrypted repository requires retyping the +encryption password. +To avoid excess password typing, consider using the "fossil shell" +command which prompts for the password just once, then reuses it for each +subsequent Fossil command entered at the prompt. + +On Windows, the "fossil server", "fossil ui", and "fossil shell" commands do not +(currently) work on an encrypted repository. + +<h2>Additional Security</h2> + +Use the FOSSIL_SECURITY_LEVEL environment for additional protection. + +<pre> +export FOSSIL_SECURITY_LEVEL=1 +</pre> + +A setting of 1 or greater +prevents fossil from trying to remember the previous sync password. + +<pre> +export FOSSIL_SECURITY_LEVEL=2 +</pre> + +A setting of 2 or greater +causes all password prompts to be preceded by a random translation matrix similar +to the following: + +<pre> +abcde fghij klmno pqrst uvwyz +qresw gjymu dpcoa fhkzv inlbt +</pre> + +When entering the password, the user must substitute the letter on the second +line that corresponds to the letter on the first line. Uppercase substitutes +for uppercase inputs, and lowercase substitutes for lowercase inputs. Letters +that are not in the translation matrix (digits, punctuation, and "x") are not +modified. For example, given the +translation matrix above, if the password is "pilot-9crazy-xube", then the user +must type "fmpav-9ekqtb-xirw". This simple substitution cypher helps prevent +password capture by keyloggers. diff --git a/www/env-opts.md b/www/env-opts.md new file mode 100644 index 0000000..ec42074 --- /dev/null +++ b/www/env-opts.md @@ -0,0 +1,509 @@ +Environment Variables and Global Options +======================================== + +Fossil uses a number of environment variables and supports a number +of global options. Most of these seem to be primarily documented in +the source code, with the primary exception of `--args` which is +described in the usage text printed by running fossil with no +arguments at all. + + + +Global Options +-------------- + +The following options are understood by the fossil command itself, and +are collected before any subcommand begins processing. + +`--args FILENAME`: Read the file `FILENAME` and replace these two +arguments with its content. Each line of the file is assumed to be an +argument unless it starts with '-' and contains a space, in which case +it is assumed to be another flag and is treated as such. `--args +FILENAME` may be used in conjunction with any other flags. + +`--case-sensitive BOOL`: Override the `case-sensitive` setting, which +can override the native preferences of the platform for case sensitive +file names: insensitive on Windows, sensitive on Unix. There are +probably odd interactions possible if you mix case sensitive and case +insensitive file systems on any single platform. This option or the +global setting should be used to force the case sensitivity to the +most sensible condition. + +`--cgitrace`: Active CGI tracing. + +`--chdir DIRECTORY`: Change to the named directory before processing +any commands. + + +`--comfmtflags NUMBER`: Specify flags that control how check-in comments +and certain other text outputs are formatted for display. The flags are +individual bits in `NUMBER`, which must be specified in base 10: + + * _0_ — Uses the revised algorithm with no special handling. + + * _1_ — Uses the canonical algorithm, other flags are ignored. + + * _2_ — Trims leading and trailing carriage-returns and line-feeds + where they do not materially impact pre-existing formatting + (i.e. at the start of the comment string _and_ right before + line indentation). + + * _4_ — Trims leading and trailing spaces where they do not materially + impact the pre-existing formatting (i.e. at the start of the + comment string _and_ right before line indentation). + + * _8_ — Attempts to break lines on word boundaries while honoring the + logical line length. + + * _16_ — Looks for the original comment text within the text being + printed. Upon matching, a new line will be emitted, thus + preserving more of the pre-existing formatting. + + +`--comment-format NUMBER`: Alias for `--comfmtflags NUMBER`. + + +> NOTE: As of Fossil version 2.26, use of the `--comfmtflags` and +> `--comment-format` options is no longer recommended and they are +> no longer documented, but retained for backwards compatibility. + + +`--errorlog ERRLOG`: Name a file to which fossil will log panics, +errors, and warnings. + + +`--help`: If `--help` is found anywhere on the command line, translate +the command to `fossil help cmdname` where `cmdname` is the first +argument that does not begin with a `-` character. If all arguments +start with `-`, translate to `fossil help argv[1] argv[2]...`. + +`--httptrace`: (Sets `g.fHttpTrace`.) Trace outbound HTTP requests. + +`--localtime`: Override the `timeline-utc` option to explicitly use +local time. + +`--nocgi`: Prevent fossil from acting as a CGI by default even if the +`GATEWAY_INTERFACE` environment variable is set. + +`--no-th-hook`: (Sets `g.fNoThHook`.) Override the `th1-hooks` setting +and prevent any TH1 hooks from being executed. + +`--quiet`: (Sets `g.fQuiet`.) Cause fossil to suppress various messages and progress +indicators that would otherwise be printed. + +`--sqltrace`: (Sets `g.SqlTrace`.) Implies `--sqlstats`. Trace certain +SQLite database activity, especially showing every SQL query +processed. + +`--sqlstats`: (Sets `g.fSqlStats`.) Print a number of performance +statistics about each SQLite database used when it is closed. + +`--sshtrace`: (Sets `g.fSshTrace`.) + +`--ssl-identity`: The fully qualified name of the file containing the client +certificate and private key to use, in PEM format. It can be created by +concatenating the client certificate and private key files. This identity will +be presented to SSL servers to authenticate the client, in addition to the +normal password authentication. + +`--systemtrace`: (Sets `g.fSystemTrace`.) Trace all commands launched +as sub processes. + +`--user LOGIN`: (Sets `g.zLogin`) Also `-U LOGIN`. Set the user name +used with the repository. + +`--utc`: Override the `timeline-utc` option to explicitly use +UTC time. + +`--vfs VFSNAME`: Load the named VFS into SQLite. + + +Environment Variables +--------------------- + +The location of the user's account-wide [configuration database][configdb] +depends on the operating system and on the existence of various +environment variables and/or files. See the discussion of the +[configuration database location algorithm][configloc] for details. + +`EDITOR`: Name the editor to use for check-in and stash comments. +Overridden by the local or global `editor` setting or the `VISUAL` +environment variable. + +`FOSSIL_BREAK`: If set, an opportunity will be created to attach a +debugger to the Fossil process prior to any significant work being +performed. + +`FOSSIL_FORCE_TICKET_MODERATION`: If set, *ALL* changes for tickets +will be required to go through moderation (even those performed by the +local interactive user via the command line). This can be useful for +local (or remote) testing of the moderation subsystem and its impact +on the contents and status of tickets. + +`FOSSIL_FORCE_WIKI_MODERATION`: If set, *ALL* changes for wiki pages +will be required to go through moderation (even those performed by the +local interactive user via the command line). This can be useful for +local (or remote) testing of the moderation subsystem and its impact +on the contents and status of wiki pages. + + +`FOSSIL_HOME`: Location of [configuration database][configdb]. +See the [configuration database location][configloc] description +for additional information. + +`FOSSIL_REPOLIST_TITLE`: The page title of the "Repository List" page +loaded by the `fossil all ui` or `fossil ui /` commands. Only used if +none of the listed repositories has the `repolist_skin` property set. +Can be set from the [CGI control file][cgictlfile]. + +`FOSSIL_REPOLIST_SHOW`: If this variable exists and has a text value +that contains the substring "description", then the "Project Description" +column appears on the repolist page. If it contains the substring +"login-group", then the Login-Group column appears on the repolist page. +Can be set from the [CGI control file][cgictlfile]. + +`FOSSIL_USE_SEE_TEXTKEY`: If set, treat the encryption key string for +SEE as text to be hashed into the actual encryption key. This has no +effect if Fossil was not compiled with SEE support enabled. + + +`FOSSIL_USER`: Name of the default user account if the checkout, local +or global `default-user` setting is not present. The first environment +variable found in the environment from the list `FOSSIL_USER`, `USER`, +`LOGNAME`, and `USERNAME` is the user name. If none of those are set, +then the default user name is "root". See the discussion of Fossil +Username below for a lot more detail. + + +`FOSSIL_SECURITY_LEVEL`: If set to any of the values listed below, +additional measures for password security will be enabled (also see +[How To Use Encrypted Repositories][encryptedrepos.wiki]): + +[encryptedrepos.wiki]: /doc/trunk/www/encryptedrepos.wiki + + * _≥1_ — Do not remember passwords. + + * _≥2_ — Use a scrambled matrix for password input. + + +`FOSSIL_TCL_PATH`: When Tcl stubs support is configured, point to a +specific file or folder containing the version of Tcl to load at run +time. + +`FOSSIL_TEST_DANGEROUS_IGNORE_OPEN_CHECKOUT`: When set to the literal +value `YES_DO_IT`, the test suite will relax the constraint that some +tests may not run within an open checkout. This is subject to removal +in the future. + +`FOSSIL_VFS`: Name a VFS to load into SQLite. + +`GATEWAY_INTERFACE`: If present and the `--nocgi` option is not, assume +fossil is invoked from a web server as a CGI command, and act +accordingly. + +`HOME`: Potential location of the [configuration database][configdb]. +See the [configuration database location][configloc] description for details. + +`HOMEDRIVE`, `HOMEPATH`: (Windows) Location of the `~/.fossil` file. +The first environment variable found in the environment from the list +`FOSSIL_HOME`, `LOCALAPPDATA` (Windows), `APPDATA` (Windows), +`HOMEDRIVE` and `HOMEPATH` (Windows, used together), and `HOME` is +used as the location of the `~/.fossil` file. + +`HTTP_HOST`: If defined, included in error log messages. + +`http_proxy`: If the global or local settings `proxy` is not set, this +is used as the default value for the `proxy` setting. + + +`HTTP_USER_AGENT`: If defined, included in error log messages. + + +`LOCALAPPDATA`: (Windows) Location of the `~/.fossil` file. The first +environment variable found in the environment from the list +`FOSSIL_HOME`, `LOCALAPPDATA` (Windows), `APPDATA` (Windows), +`HOMEDRIVE` and `HOMEPATH` (Windows, used together), and `HOME` is +used as the location of the `~/.fossil` file. + +`LOGNAME`: Name of the logged in user on many Unix-like platforms. +Used as the fossil user name if `FOSSIL_USER` is not specified. See +the discussion of Fossil Username below for a lot more detail. + +`NO_COLOR`: If defined and not set to a `false` value (i.e. "off", "no", +"false", "0"), the `fossil search` command skips colorization of console +output using ANSI escape codes (VT100). + +`PATH`: Used by most platforms to locate programs invoked without a +fully qualified name. Explicitly used by `fossil ui` on certain platforms +to choose the browser to launch. + +`PATH_INFO`: If defined, included in error log messages. + +`QUERY_STRING`: If defined, included in error log messages. + +`REMOTE_ADDR`: If defined, included in error log messages. + +`REMOTE_HOST`: Used by `fossil http` run from `stunnel` to identify +the remote host. + +`REQUEST_METHOD`: If defined, included in error log messages. + +`REQUEST_URI`: If defined, included in error log messages. + +`SCRIPT_NAME`: If defined, included in error log messages. + +`SSH_CONNECTION`: Informs CGI processing if the remote client is SSH. + +`SSL_CERT_FILE`, `SSL_CERT_DIR`: Override the [`ssl-ca-location`] +(/help/ssl-ca-location) setting. + +`SQLITE_FORCE_PROXY_LOCKING`: From `sqlite3.c`, 1 means force always +use proxy, 0 means never use proxy, and undefined means use proxy for +non-local files only. + +`SQLITE_TMPDIR`: Names the temporary file location for SQLite. When +set, this will be used instead of `TMPDIR`. + + +`SYSTEMROOT`: (Windows) Used to locate `notepad.exe` as a +fall back comment editor. + +`TERM`: If the linenoise library is used (almost certainly not on +Windows), it will check `TERM` to verify that the interactive terminal +is not named on a short list on terminals known to not work with +linenoise. Linenoise is a library that provides command history and +command line editing to interactive programs, and can be used in the +`fossil sqlite3` command. + +`TH1_DELETE_INTERP`: Set this variable to ask fossil to explicitly +delete the TH1 interpreter, if it is loaded, then check that it +released all of its allocated memory, when exiting fossil. This is not +strictly necessary, but makes debugging memory leaks easier. See +[main.c near line 386](/artifact/e75796be5338a81c?ln=386,391) for the +code. + +`TH1_ENABLE_DOCS`: Override the local or global setting `tcl-docs` +to enable TH1 documents in fossil. + +`TH1_ENABLE_HOOKS`: Override the local or global setting `tcl-hooks` +to enable TH1 hooks in fossil. + +`TH1_ENABLE_TCL`: Override the local or global setting `tcl` to enable +Tcl in fossil. + +`TH1_TEST_ANON_CAPS`: Override the default anonymous permissions used +when processing the `--set-anon-caps` option for the `test-th-eval`, +`test-th-render`, and `test-th-source` test commands. + +`TH1_TEST_USER_CAPS`: Override the default user permissions used when +processing the `--set-user-caps` option for the `test-th-eval`, +`test-th-render`, and `test-th-source` test commands. + +`TMPDIR`: Names the temporary file location for SQLite. + + +`USER`: Name of the logged in user on many Unix-like platforms. +Used as the fossil user name if `FOSSIL_USER` is not specified. See +the discussion of Fossil Username below for a lot more detail. + +`USERNAME`: Name of the logged in user on Windows platforms. +Used as the fossil user name if `FOSSIL_USER` is not specified. See +the discussion of Fossil Username below for a lot more detail. + +`VISUAL`: Name the editor to use for check-in and stash comments. +Overrides the `EDITOR` environment variable. Overridden by the local +or global `editor` setting. + + + +Notes on Related Values +----------------------- + +### CGI and JSON Parameters + + +The JSON API implementation looks up many values in the first of +several places searched. This unifies the parameter handling logic, +allows the caller to choose whether to prefer URL parameters, request +headers, or the POST payload, and allows the `fossil json` command to +share most of the same logic as the `/json` API path. The search order +is a POST payload, GET/COOKIE/non-JSON POST, JSON POST, the system +environment. + +See the comment above the implementation of [`json_getenv`][json.c] +for some further discussion. + +[json.c]: /artifact/6df1d80dece8968b?ln=277,290 + +### Comment Editor + +The editor used to edit a check-in or stash comment is named by the +local or global setting `editor`. If neither is set, then the environment +variables `VISUAL`, and `EDITOR` are checked in that order. + +On Windows, if no editor is named, then Notepad is used. Note that the +operation will be aborted if `notepad.exe` is not found in the Windows +folder. + +On Unix-like platforms, if no editor is named, then a message is +displayed on stdout, and stdin is read until a single line containing +only a dot is seen. + + +### Error logging + +If logging errors to a file, fossil will include the values of the +following environment variables in the error log entry if they are +defined: `HTTP_HOST`, `HTTP_USER_AGENT`, `PATH_INFO`, `QUERY_STRING`, +`REMOTE_ADDR`, `REQUEST_METHOD`, `REQUEST_URI`, and `SCRIPT_NAME`. + + + +### Fossil Username + +In absence of any explicit setting, fossil will use the same name you +logged in to your platform with, as the user name when interacting +with local and remote repositories. Note that only the name is shared, +fossil makes no attempt to share or leverage any platform's +authentication mechanisms or passwords. + +When logging in to a repository, it tries a series of sources for the +user name, and the first non-blank name that succeeds is the logged in +user. The order is: + +1. The --user and -U command-line options. +2. If running within an open checkout (the local database is open), + check in its table of values stored per open checkout for the + value stored by `fossil user default USERNAME`. +3. The default user in the repository (setting `default-user`) +4. The `FOSSIL_USER` environment variable. +5. The `USER` environment variable. +6. The `LOGNAME` environment variable. +7. The `USERNAME` environment variable. +8. Check if the user can be extracted from the remote URL, if + there is a remote URL. + +Items 2 and 3 are both set by `fossil user default USERNAME`, the +first within an open checkout, the second outside and using the `-R +REPOSITORY` option to identify the repository. Both cases require that +the named user be present in the repository when the default user is +assigned. Although the default user is internally stored as if it were +a setting named `default-user`, it is not accessible through +the `fossil set` command. + +Items 5, 6, and 7 cover most of the names of an environment variable +set automatically by the platform with the name of the platform's +logged in user for use by programs. Historically, `USER` comes from +Unix System-V, `LOGNAME` from BSD, and `USERNAME` from Windows, but +many Linux distributions will set both `USER` and `LOGNAME` for broad +compatibility. + +When creating a new repository, fossil needs a user name for the admin +user granted the "s" permission. But since fossil generally expects +that `fossil new` or `fossil clone` are used outside of any checkout +(especially when run for the first time without any checkouts at all +or the users's global settings database), it looks in a shorter list +of places for a non-blank name. In the special case of a clone, +`default-user` can be copied from the original, and so it can be set +in the clone even before any users have been created, and in that case +it will be the new admin user. If `default-user` is not set, then the +first found environment variable from the list `FOSSIL_USER`, `USER`, +`LOGNAME`, and `USERNAME`, is the user name. As a final fallback, if +none of those are set, then the default user name is "root". + + +### Configuration Database Location + +Fossil keeps some information pertinent to each user in the user's +[configuration database file][configdb]. +The configuration database file includes the global settings +and the list of repositories and checkouts used by `fossil all`. + +The location of the configuration database file depends on the +operating system and on the existence of various environment +variables and/or files. In brief, the configuration database is +usually: + + * Traditional unix → "`$HOME/.fossil`" + * Windows → "`%LOCALAPPDATA%/_fossil`" + * [XDG-unix][xdg] → "`$HOME/.config/fossil.db`" + +[xdg]: https://www.freedesktop.org/wiki/ + +See the [configuration database location +algorithm][configloc] discussion for full information. + +### SQLite VFS to use + +See [the SQLite documentation](http://www.sqlite.org/vfs.html) for an +explanation of what a VFS actually is and what it does. + +If the default VFS underneath SQLite is not suitable, an alternative +can be selected with either the `--vfs VFSNAME` option or the +`FOSSIL_VFS` environment variable. The `--vfs` option takes +precedence. + + +### <a id="temp"></a>Temporary File Location + +Fossil places some temporary files in the checkout directory. Most notably, +supporting files related to merge conflicts are placed in the same +folder as the merge result. + +Other temporary files need a different home. The rules for choosing one are +complicated. + +Fossil-specific code uses `FOSSIL_TEMP`, `TEMP`, and `TMP`, in that +order. Fossil’s own test suite prepends `FOSSIL_TEST_TEMP` to that list. + +The underlying SQLite code uses several different path sets for its temp +files, depending on the platform type. + +On Unix-like platforms, excepting Cygwin, SQLite first checks the +environment variables `SQLITE_TMPDIR` and `TMPDIR`, in that order. If +neither is defined, it falls back to a hard-coded list of paths: +`/var/tmp`, `/usr/tmp`, and `/tmp`. If all of that fails, it uses the +current working directory. + +For Cygwin builds, SQLite instead uses the first defined variable in +this list: `SQLITE_TMPDIR`, `TMPDIR`, `TMP`, `TEMP`, and `USERPROFILE`. + +For native Windows builds, SQLite simply calls the OS’s [`GetTempPath()` +API][gtp]. See that reference page for details. + +[gtp]: https://msdn.microsoft.com/en-us/library/windows/desktop/aa364992%28v=vs.85%29.aspx + + + +That said, it is not unusual for utilities on all platforms to assume +that `TEMP` or `TMP` point somewhere safe for temporary files. + +If the identified temporary folder is not writable, then weird things +will happen on all platforms. + + +### Web browser + +Occasionally, fossil wants to launch a web browser for the user, most +obviously as part of the `fossil ui` command. In that specific case, +the browser is launched pointing at the web server started by +`fossil ui` listening on a private TCP port. + +On all platforms, if the local or global settings `web-browser` is +set, that is the command used to open a URL. + +Otherwise, the specific actions vary by platform. + +On Unix-like platforms other than Apple's, it looks for the first +program from the list `xdg-open`, `gnome-open`, `firefox`, and +`google-chrome` that it can find on the `PATH`. + +On Apple platforms, it assumes that `open` is the command to open a +URL in the user's configured default browser. + +On Windows platforms, it assumes that `start` is the command to open +a URL in the user's configured default browser. + +[configdb]: ./tech_overview.wiki#configdb +[configloc]: ./tech_overview.wiki#configloc +[cgictlfile]: ./cgi.wiki diff --git a/www/event.wiki b/www/event.wiki new file mode 100644 index 0000000..0b70ee6 --- /dev/null +++ b/www/event.wiki @@ -0,0 +1,120 @@ +<title>Technical Notes</title> + +<h2>What Is A "Technote"?</h2> + +In Fossil, a "technical note" or "technote" (formerly called an "event") +is a special kind of [./wikitheory.wiki | wiki page] +that is associated with a point in time rather than having a page name. +Each technote causes a single entry to appear on the +[/timeline?y=e | Timeline Page]. +Clicking on the timeline link will display the text of the technote. +The wiki content, the timeline entry text, the +time of the technote, and the timeline background color can all be edited. + +As with check-ins, wiki, and tickets, all technotes automatically synchronize +to other repositories. Hence, technotes can be viewed, created, and edited +off-line. And the complete edit history for technotes is maintained +for auditing purposes. + +Possible uses for technotes include: + + * <b>Milestones</b>. Project milestones, such as releases or beta-test + cycles, can be recorded as technotes. The timeline entry for the technote + can be something simple like "Version 1.2.3" perhaps with a bright + color background to draw attention to the entry and the wiki content + can contain release notes, for example. + + * <b>Blog Entries</b>. Blog entries from developers describing the current + state of a project, or rationale for various design decisions, or + roadmaps for future development, can be entered as technotes. + + * <b>Process Checkpoints</b>. For projects that have a formal process, + technotes can be used to record the completion or the initiation of + various process steps. For example, a technote can be used to record + the successful completion of a long-running test, perhaps with + performance results and details of where the test was run and who + ran it recorded in the wiki content. The technote can be added + from a script. + + * <b>News Articles</b>. Significant occurrences in the life cycle of + a project can be recorded as news articles using technotes. Perhaps the + domain name of the canonical website for a project changes, or new + server hardware is obtained. Such happenings are appropriate for + reporting as news. + + * <b>Announcements</b>. Changes to the composition of the development + team or acquisition of new project sponsors can be communicated as + announcements which can be implemented as technotes. + + * <b>Signed Checksums</b>. Technotes containing cryptographically signed + checksums can be linked to repository artifacts, thereby creating a + traceable, auditable chain so that users can readily verify the integrity + and authenticity of project deliverables. And the command line interface + to technotes enables embedding such processes in scripts. + +No project is required to use technotes. But technotes can help many projects +stay better organized and provide a better historical record of the +development progress. + +<h2>Viewing Technotes</h2> + +Because technotes are considered a special kind of wiki, +users must have permission to read wiki in order read technotes. +Enable the "j" permission under the /Setup/Users menu in order +to give specific users or user classes the ability to view wiki +and technotes. + +Technotes show up on the timeline. Click on the hyperlink beside the +technote title to see the complete text. + +<h2>Creating, Editing and Viewing Technotes</h2> + +There is a hyperlink under the /wikihelp menu that can be used to create +new technotes. And there is a submenu hyperlink on technote displays for +editing existing technotes. + +Technotes can also be created using the <b>wiki create</b> command: + +<verbatim> +fossil wiki create TestTechnote -t now --technote-bgcolor lightgreen technote.md +Created new tech note 2021-03-15 13:05:56 +</verbatim> + +This command inserts a light green technote in the timeline at 2021-03-15 13:05:56, with +the contents of file <b>technote.md</b> and comment "TestTechnote". Specifying a different time using +<b>-t DATETIME</b> will insert the technote at the specified timestamp location in the timeline. +Different technotes can have the same timestamp. + +The first argument to create, <b>TECHNOTE-COMMENT</b>, is the title text for the technote +that appears in the timeline. + +To view all technotes, use the <b>wiki ls</b> command: + +<verbatim> +fossil wiki ls --technote --show-technote-ids +z739263a134bf0da1d28e939f4c4367f51ef4c51 2020-12-19 13:20:19 +e15a918a8bed71c2ac091d74dc397b8d3340d5e1 2018-09-22 17:40:10 +</verbatim> + +A technote ID is the UUID of the technote. + +To view an individual technote, use the <b>wiki export</b> command: + +<verbatim> +fossil wiki export --technote version-2.16 +Release Notes 2021-07-02 + +This note describes changes in the Fossil snapshot for ... +</verbatim> + +The <b>-t|--technote</b> option to the <b>export</b> subcommand takes one of +three identifiers: <b>DATETIME</b>; <b>TECHNOTE-ID</b>; and <b>TAG</b>. +See the [/help/wiki | wiki help] for specifics. + +Users must have check-in privileges (permission "i") in order to +create or edit technotes. In addition, users must have create-wiki +privilege (permission "f") to create new technotes and edit-wiki +privilege (permission "k") in order to edit existing technotes. + +Technote content may be formatted as [/wiki_rules | Fossil wiki], +[/md_rules | Markdown], or plain text. diff --git a/www/faq.tcl b/www/faq.tcl new file mode 100755 index 0000000..3a17bb8 --- /dev/null +++ b/www/faq.tcl @@ -0,0 +1,181 @@ +#!/usr/bin/tclsh +# +# Run this to generate the FAQ +# +set cnt 1 +proc faq {question answer} { + set ::faq($::cnt) [list [string trim $question] [string trim $answer]] + incr ::cnt +} + +faq { + What GUIs are available for fossil? +} { + The fossil executable comes with a [./webui.wiki | web-based GUI] built in. + Just run: + + <pre> + <b>fossil [/help/ui|ui]</b> <i>REPOSITORY-FILENAME</i> + </pre> + + And your default web browser should pop up and automatically point to + the fossil interface. (Hint: You can omit the <i>REPOSITORY-FILENAME</i> + if you are within an open check-out.) +} + +faq { + What is the difference between a "branch" and a "fork"? +} { + This is a big question - too big to answer in a FAQ. Please + read the <a href="branching.wiki">Branching, Forking, Merging, + and Tagging</a> document. +} + + +faq { + How do I create a new branch? +} { + There are lots of ways: + + When you are checking in a new change using the <b>[/help/commit|commit]</b> + command, you can add the option "--branch <i>BRANCH-NAME</i>" to + make the new check-in be the first check-in for a new branch. + + If you want to create a new branch whose initial content is the + same as an existing check-in, use this command: + + <pre> + <b>fossil [/help/branch|branch] new</b> <i>BRANCH-NAME BASIS</i> + </pre> + + The <i>BRANCH-NAME</i> argument is the name of the new branch and the + <i>BASIS</i> argument is the name of the check-in that the branch splits + off from. + + If you already have a fork in your check-in tree and you want to convert + that fork to a branch, you can do this from the web interface. + First locate the check-in that you want to be + the initial check-in of your branch on the timeline and click on its + link so that you are on the <b>ci</b> page. Then find the "<b>edit</b>" + link (near the "Commands:" label) and click on that. On the + "Edit Check-in" page, check the box beside "Branching:" and fill in + the name of your new branch to the right and press the "Apply Changes" + button. +} + +faq { + How do I tag a check-in? +} { + There are several ways: + + When you are checking in a new change using the <b>[/help/commit|commit]</b> + command, you can add a tag to that check-in using the + "--tag <i>TAGNAME</i>" command-line option. You can repeat the --tag + option to give a check-in multiple tags. Tags need not be unique. So, + for example, it is common to give every released version a "release" tag. + + If you want to add a tag to an existing check-in, you can use the + <b>[/help/tag|tag]</b> command. For example: + + <pre> + <b>fossil [/help/branch|tag] add</b> <i>TAGNAME</i> <i>CHECK-IN</i> + </pre> + + The CHECK-IN in the previous line can be any + [./checkin_names.wiki | valid check-in name format]. + + You can also add (and remove) tags from a check-in using the + [./webui.wiki | web interface]. First locate the check-in that you + want to tag on the timeline, then click on the link to go the detailed + information page for that check-in. Then find the "<b>edit</b>" + link (near the "Commands:" label) and click on that. There are + controls on the edit page that allow new tags to be added and existing + tags to be removed. +} + +faq { + How do I create a private branch that won't get pushed back to the + main repository. +} { + Use the <b>--private</b> command-line option on the + <b>commit</b> command. The result will be a check-in which exists in + your local repository only and is never pushed to other repositories. + All descendants of a private check-in are also private. + + Unless you specify something different using the <b>--branch</b> and/or + <b>--bgcolor</b> options, the new private check-in will be put on a branch + named "private" with an orange background color. + + You can merge from the trunk into your private branch in order to keep + your private branch in sync with the latest changes on the trunk. Once + you have everything in your private branch the way you want it, you can + then merge your private branch back into the trunk and push. Only the + final merge operation will appear in other repositories. It will seem + as if all the changes that occurred on your private branch occurred in + a single check-in. + Of course, you can also keep your branch private forever simply + by not merging the changes in the private branch back into the trunk. + + [./private.wiki | Additional information] +} + +faq { + How can I delete inappropriate content from my fossil repository? +} { + See the article on [./shunning.wiki | "shunning"] for details. +} + +faq { + How do I make a clone of the fossil self-hosting repository? +} { + Any of the following commands should work: + + <pre> + fossil [/help/clone|clone] https://fossil-scm.org/ fossil.fossil + fossil [/help/clone|clone] https://www2.fossil-scm.org/ fossil.fossil + fossil [/help/clone|clone] https://www3.fossil-scm.org/site.cgi fossil.fossil + </pre> + + Once you have the repository cloned, you can open a local check-out + as follows: + + <pre> + mkdir src; cd src; fossil [/help/open|open] ../fossil.fossil + </pre> + + Thereafter you should be able to keep your local check-out up to date + with the latest code in the public repository by typing: + + <pre> + fossil [/help/update|update] + </pre> +} + +faq { + How do I import or export content from and to other version control systems? +} { + Please see [./inout.wiki | Import And Export] +} + + + +############################################################################# +# Code to actually generate the FAQ +# +puts "<title>Fossil FAQ</title>\n" +puts "Note: See also <a href=\"qandc.wiki\">Questions and Criticisms</a>.\n" + +puts {<ol>} +for {set i 1} {$i<$cnt} {incr i} { + puts "<li><a href=\"#q$i\">[lindex $faq($i) 0]</a></li>" +} +puts {</ol>} +puts {<hr>} + +for {set i 1} {$i<$cnt} {incr i} { + puts "<p id=\"q$i\"><b>($i) [lindex $faq($i) 0]</b></p>\n" + set body [lindex $faq($i) 1] + regsub -all "\n *" [string trim $body] "\n" body + puts "$body</li>\n" +} +puts {</ol>} diff --git a/www/faq.wiki b/www/faq.wiki new file mode 100644 index 0000000..0c8a9c3 --- /dev/null +++ b/www/faq.wiki @@ -0,0 +1,149 @@ +<title>Fossil FAQ</title> + +Note: See also <a href="qandc.wiki">Questions and Criticisms</a>. + +<ol> +<li><a href="#q1">What GUIs are available for fossil?</a></li> +<li><a href="#q2">What is the difference between a "branch" and a "fork"?</a></li> +<li><a href="#q3">How do I create a new branch?</a></li> +<li><a href="#q4">How do I tag a check-in?</a></li> +<li><a href="#q5">How do I create a private branch that won't get pushed back to the + main repository.</a></li> +<li><a href="#q6">How can I delete inappropriate content from my fossil repository?</a></li> +<li><a href="#q7">How do I make a clone of the fossil self-hosting repository?</a></li> +<li><a href="#q8">How do I import or export content from and to other version control systems?</a></li> +</ol> +<hr> +<p id="q1"><b>(1) What GUIs are available for fossil?</b></p> + +The fossil executable comes with a [./webui.wiki | web-based GUI] built in. +Just run: + +<pre> +<b>fossil [/help/ui|ui]</b> <i>REPOSITORY-FILENAME</i> +</pre> + +And your default web browser should pop up and automatically point to +the fossil interface. (Hint: You can omit the <i>REPOSITORY-FILENAME</i> +if you are within an open check-out.)</li> + +<p id="q2"><b>(2) What is the difference between a "branch" and a "fork"?</b></p> + +This is a big question - too big to answer in a FAQ. Please +read the <a href="branching.wiki">Branching, Forking, Merging, +and Tagging</a> document.</li> + +<p id="q3"><b>(3) How do I create a new branch?</b></p> + +There are lots of ways: + +When you are checking in a new change using the <b>[/help/commit|commit]</b> +command, you can add the option "--branch <i>BRANCH-NAME</i>" to +make the new check-in be the first check-in for a new branch. + +If you want to create a new branch whose initial content is the +same as an existing check-in, use this command: + +<pre> +<b>fossil [/help/branch|branch] new</b> <i>BRANCH-NAME BASIS</i> +</pre> + +The <i>BRANCH-NAME</i> argument is the name of the new branch and the +<i>BASIS</i> argument is the name of the check-in that the branch splits +off from. + +If you already have a fork in your check-in tree and you want to convert +that fork to a branch, you can do this from the web interface. +First locate the check-in that you want to be +the initial check-in of your branch on the timeline and click on its +link so that you are on the <b>ci</b> page. Then find the "<b>edit</b>" +link (near the "Commands:" label) and click on that. On the +"Edit Check-in" page, check the box beside "Branching:" and fill in +the name of your new branch to the right and press the "Apply Changes" +button.</li> + +<p id="q4"><b>(4) How do I tag a check-in?</b></p> + +There are several ways: + +When you are checking in a new change using the <b>[/help/commit|commit]</b> +command, you can add a tag to that check-in using the +"--tag <i>TAGNAME</i>" command-line option. You can repeat the --tag +option to give a check-in multiple tags. Tags need not be unique. So, +for example, it is common to give every released version a "release" tag. + +If you want to add a tag to an existing check-in, you can use the +<b>[/help/tag|tag]</b> command. For example: + +<pre> +<b>fossil [/help/branch|tag] add</b> <i>TAGNAME</i> <i>CHECK-IN</i> +</pre> + +The CHECK-IN in the previous line can be any +[./checkin_names.wiki | valid check-in name format]. + +You can also add (and remove) tags from a check-in using the +[./webui.wiki | web interface]. First locate the check-in that you +want to tag on the timeline, then click on the link to go the detailed +information page for that check-in. Then find the "<b>edit</b>" +link (near the "Commands:" label) and click on that. There are +controls on the edit page that allow new tags to be added and existing +tags to be removed.</li> + +<p id="q5"><b>(5) How do I create a private branch that won't get pushed back to the + main repository.</b></p> + +Use the <b>--private</b> command-line option on the +<b>commit</b> command. The result will be a check-in which exists in +your local repository only and is never pushed to other repositories. +All descendants of a private check-in are also private. + +Unless you specify something different using the <b>--branch</b> and/or +<b>--bgcolor</b> options, the new private check-in will be put on a branch +named "private" with an orange background color. + +You can merge from the trunk into your private branch in order to keep +your private branch in sync with the latest changes on the trunk. Once +you have everything in your private branch the way you want it, you can +then merge your private branch back into the trunk and push. Only the +final merge operation will appear in other repositories. It will seem +as if all the changes that occurred on your private branch occurred in +a single check-in. +Of course, you can also keep your branch private forever simply +by not merging the changes in the private branch back into the trunk. + +[./private.wiki | Additional information]</li> + +<p id="q6"><b>(6) How can I delete inappropriate content from my fossil repository?</b></p> + +See the article on [./shunning.wiki | "shunning"] for details.</li> + +<p id="q7"><b>(7) How do I make a clone of the fossil self-hosting repository?</b></p> + +Any of the following commands should work: + +<pre> +fossil [/help/clone|clone] https://fossil-scm.org/ fossil.fossil +fossil [/help/clone|clone] https://www2.fossil-scm.org/ fossil.fossil +fossil [/help/clone|clone] https://www3.fossil-scm.org/site.cgi fossil.fossil +</pre> + +Once you have the repository cloned, you can open a local check-out +as follows: + +<pre> +mkdir src; cd src; fossil [/help/open|open] ../fossil.fossil +</pre> + +Thereafter you should be able to keep your local check-out up to date +with the latest code in the public repository by typing: + +<pre> +fossil [/help/update|update] +</pre></li> + +<p id="q8"><b>(8) How do I import or export content from and to other version control systems?</b></p> + +Please see [./inout.wiki | Import And Export]</li> + +</ol> diff --git a/www/fileedit-page.md b/www/fileedit-page.md new file mode 100644 index 0000000..c8f7eeb --- /dev/null +++ b/www/fileedit-page.md @@ -0,0 +1,317 @@ +# The fileedit Page + +This document describes the limitations of, caveats for, and +disclaimers for the [](/fileedit) page, which provides users with + basic editing features for files via the web interface when they + have [checkin privileges](./caps/index.md). + +# Important Caveats and Disclaimers + +Predictably, the ability to edit files in a repository from a web +browser halfway around the world comes with several obligatory caveats +and disclaimers... + +## <a id="cap"></a> `/fileedit` Does *Nothing* by Default. + +In order to "activate" it, a user with [the "setup" +permission](./caps/index.md) must set the +[fileedit-glob](/help/fileedit-glob) repository setting to a +comma- or newline-delimited list of globs representing a whitelist of +files which may be edited online. Any user with commit access may then +edit files matching one of those globs. Certain pages within the UI +get an "edit" link added to them when the current user's permissions +and the whitelist both permit editing of that file. + +## <a id="csrf"></a> CSRF & HTTP Referrer Headers + +In order to protect against [Cross-site Request Forgery (CSRF)][csrf] +attacks, Fossil UI features which write to the database require that +the browser send the so-called [HTTP `Referer` header][referer] +(noting that the misspelling of "referrer" is a historical accident +which has long-since been standardized!). Modern browsers, by default, +include such information automatically for *interactive* actions which +lead to a request, e.g. clicking on a link back to the same +server. However, `/fileedit` uses asynchronous ["XHR"][xhr] +connections, which browsers *may* treat differently than strictly +interactive elements. + +- **Firefox**: configuration option `network.http.sendRefererHeader` + specifies whether the `Referer` header is sent. It must have a value + of 2 (which is the default) for XHR requests to get the `Referer` + header. Purely interactive Fossil features, in which users directly + activate links or forms, work with a level of 1 or higher. +- **Chrome**: apparently requires an add-on in order to change this + policy, so Chrome without such an add-on will not suppress this + header. +- **Safari**: ??? +- **Other browsers**: ??? + +If `/fileedit` shows an error message saying "CSRF violation," the +problem is that the browser is not sending a `Referer` header to XHR +connections. Fossil does not offer a way to disable its CSRF +protections. + +[referer]: https://en.wikipedia.org/wiki/HTTP_referer +[csrf]: https://en.wikipedia.org/wiki/Cross-site_request_forgery +[xhr]: https://en.wikipedia.org/wiki/XMLHttpRequest + +## <a id="commit"></a> `/fileedit` **Works by Creating Commits** + +Thus any edits made via that page become a normal part of the +repository. + +## <a id="intent"></a> `/fileedit` is *Intended* for use with Embedded Docs + +... and similar text files, and is most certainly +**not intended for editing code**. + +Editing files with unusual syntax requirements, e.g. hard tabs in +makefiles, may break them. *You Have Been Warned.* + +Similarly, though every effort is made to retain the end-of-line +style used by being-edited files, the round-trip through an HTML +textarea element may change the EOLs. The Commit section of the page +offers three different options for how to treat newlines when saving +changes. **Files with mixed EOL styles** *will be normalized to a single +EOL style* when modified using `/fileedit`. When "inheriting" the EOL +style from a previous version which has mixed styles, the first EOL +style detected in the previous version of the file is used. + +## <a id="checkout"></a> `/fileedit` **is Not a Replacement for a Checkout** + +A full-featured checkout allows far more possibilities than this basic +online editor permits, and the feature scope of `/fileedit` is +intentionally kept small, implementing only the bare necessities +needed for performing basic edits online. It *is not, and will never +be, a replacement for a checkout.* + +It is to be expected that users will want to do "more" with this +page, and we generally encourage feature requests, but be aware that +certain types of ostensibly sensible feature requests *will be +rejected* for `/fileedit`. These include, but are not limited to: + +- Features which are already provided by other pages, e.g. +the ability to create a new named branch or add tags. +- Features which would require re-implementing significant +capabilities provided only within a checkout (e.g. merging files). +- The ability to edit/manipulate files which are in a local +checkout. (If you have a checkout, use your local editor, not +`/fileedit`.) +- Editing of non-text files, e.g. images. Use a checkout and your +preferred graphics editor. +- Support for syncing/pulling/pushing of a repository before and/or +after edits. Those features cannot be *reliably* provided via a web +interface for several reasons. + +Similarly, some *potential* features have significant downsides, +abuses, and/or implementation hurdles which make the decision of +whether or not to implement them subject to notable contributor +debate. e.g. the ability to add new files or remove/rename older +files. + + +## <a id="storage"></a> `/fileedit` **Stores Only Limited Local Edits While Working** + +When changes are made to a given checkin/file combination, +`/fileedit` will, if possible, store them in [`window.localStorage` +or `window.sessionStorage`][html5storage], if available, but... + +- Which storage is used is unspecified and may differ across + environments. +- If neither of those is available, the storage is transient and + will not survive a page reload. In this case, the UI issues a clear + warning in the editor tab. +- It stores only the most recent checkin/file combinations which have + been modified (exactly how many may differ - the number will be + noted somewhere in the UI). Note that changing the "executable bit" + is counted as a modification, but the checkin *comment* is *not* + and is reset after a commit. +- If its internal limit on the number of modified files is exceeded, + it silently discards the oldest edits to keep the list at its limit. + +Edits are saved whenever the editor component fires its "change" +event, which essentially means as soon as it loses input focus. Thus +to force the browser to save any pending changes, simply click +somwhere on the page outside of the editor. + +Exactly how long `localStorage` will survive, and how much it or +`sessionStorage` can hold, is environment-dependent. `sessionStorage` +will survive until the current browser tab is closed, but it survives +across reloads of the same tab. + +If `/fileedit` determines that no persistent storage is available a +warning is displayed on the editor page. + +[html5storage]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API + +## <a id="power"></a> The Power is Yours, but... + +> "With great power comes great responsibility." + +**Use this feature judiciously, *if at all*.** + +Now, with those warnings and caveats out of the way... + +----- + +# <a id="tips"></a> Tips and Tricks + +## <a id="global-js"></a> `fossil` Global-scope JS Object + +`/fileedit` is largely implemented in JavaScript, and makes heavy use +of the global-scope `fossil` object, which provides +infrastructure-level features intended for use by Fossil UI pages. +(That said, that infrastructure was introduced with `/fileedit`, and +most pages do not use it.) + +The `fossil.page` object represents the UI's current page (on pages +which make use of this API - most do not). That object supports +listening to page-specific events so that JS code installed via +[client-side edits to the site skin's footer](customskin.md) may react +to those changes somehow. The next section describes one such use for +such events... + +## <a id="syn-hl"></a> Integrating Syntax Highlighting + +Assuming a repository has integrated a 3rd-party syntax highlighting +solution, it can probably (depending on its API) be told how to +highlight `/fileedit`'s wiki/markdown-format previews. Here are +instructions for doing so with [highlightjs](https://highlightjs.org/): + +At the very bottom of the [site skin's footer](customskin.md), add a +script tag similar to the following: + +```javascript +<script nonce="$<nonce>"> +if(window.fossil && fossil.page && fossil.page.name==='fileedit'){ + fossil.page.addEventListener( + 'fileedit-preview-updated', + (ev)=>{ + if(ev.detail.previewMode==='wiki'){ + ev.detail.element.querySelectorAll( + 'code[class^=language-]' + ).forEach((e)=>hljs.highlightBlock(e)); + } + } + ); +} +</script> +``` + +Note that the `nonce="$<nonce>"` part is intended to be entered +literally as shown above. It will be expanded to contain the current +request's nonce value when the page is rendered. + +The first line of the script just ensures that the expected JS-level +infrastructure is loaded. It's only loaded in the `/fileedit` page and +possibly pages added or "upgraded" since `/fileedit`'s introduction. + +The part in the `if` block adds an event listener to the `/fileedit` +app which gets called when the preview is refreshed. That event +contains 3 properties: + +- `previewMode`: a string describing the current preview mode: `wiki` + (which includes Fossil-native wiki and markdown), `text`, + `htmlInline`, `htmlIframe`. We should "probably" only highlight wiki + text, and thus the example above limits its work to that type of + preview. It won't work with `htmlIframe`, as that represents an + iframe element which contains a complete HTML document. +- `element`: the DOM element in which the preview is rendered. +- `mimetype`: the mimetype of the being-previewed content, as determined + by Fossil (by its file extension). + +The event listener callback shown above doesn't use the `mimetype`, +but makes use of the other two. It fishes all `code` blocks out of +the preview which explicitly have a CSS class named +`language-`something, and then asks highlightjs to highlight them. + +## <a id="editor"></a> Integrating a Custom Editor Widget + +(These instructions also work for the `/wikiedit` page by replacing +"fileedit" with "wikiedit" in any strings or symbol names!) + +It is possible to replace `/fileedit`'s basic text-editing widget (a +`textarea` element) with a fancy 3rd-party editor widget by following +these instructions... + +All JavaScript code which follows is assumed to be in a script tag +similar to the one shown in the previous section: + +```javascript +<script nonce="$<nonce>"> +if(window.fossil && fossil.page && fossil.page.name==='fileedit'){ + // code specific to the fileedit page goes here +} +</script> +``` + +First, install proxy functions so that `fossil.page.fileContent()` +can get and set your content: + +``` +fossil.page.setContentMethods( + function(){ return text-form content of your widget }, + function(content){ set text-form content of your widget } +}; +``` + +Secondly, we need to alert the editor app when there are changes so +that it can do things like store edits locally so that they are not +lost on a page reload. How that is done is completely dependent on the +3rd-party editor widget, but it generically looks something like: + +``` +myCustomWidget.on('eventName', ()=>fossil.page.notifyOfChange()); +``` + +Lastly, if the 3rd-party editor does *not* hide or remove the native +editor widget, and does not inject itself into the DOM on the caller's +behalf, we can replace the native widget with the 3rd-party one with: + +```javascript +fossil.page.replaceEditorWidget(yourNewWidgetElement); +``` + +That method must be passed a DOM element and may only be called once: +it *removes itself* the first time it is called. + +That should be all there is to it. When `fossil.page` needs to get the +being-edited content, it will call the installed content-getter +function with no arguments, and when it sets the content (immediately +after (re)loading a file or grabbing local edits), it will pass that +content to the installed content-setter method. Those, in turn will +trigger the installed proxies and fire any relevant events. + +Below is an example of Fossil skin footer content which plugs in the +TinyMCE HTML editor into the `/wikiedit` page, but the process is +identical for `/fileedit` (noting that `/fileedit` may need to be able +to edit multiple types of files for which a special-purpose editor +like TinyMCE may not be suitable). Note that any paths to CSS and JS +resources of course need to be modified to suit one's own +installation. + +``` +<!-- TinyMCE CSS and JS: --> +<link href="$<home>/doc/ckout/skin.min.css" rel="stylesheet" type="text/css"> +<link href="$<home>/doc/ckout/content.min.css" rel="stylesheet" type="text/css"> +<script src='$<home>/doc/ckout/tinymce.min.js'></script> +<script src='$<home>/doc/ckout/theme.min.js'></script> +<script src='$<home>/doc/ckout/icons.min.js'></script> +<!-- Integrate TinyMCE into /wikiedit: --> +<script nonce="$<nonce>"> +if(window.fossil && window.fossil.page.name==='wikiedit'){ + window.fossil.onPageLoad( function(){ + const elemId = 'wikiedit-content-editor'; + tinymce.init({selector: 'textarea#'+elemId}); + const widget = tinymce.get(elemId); + fossil.page.setContentMethods( + function(){return widget.getContent()}, + function(content){widget.setContent(content)} + ); + widget.on('change', function(){ + if(widget.isDirty()) fossil.page.notifyOfChange(); + }); + }); +} +</script> +``` diff --git a/www/fileformat.wiki b/www/fileformat.wiki new file mode 100644 index 0000000..3eaa3cc --- /dev/null +++ b/www/fileformat.wiki @@ -0,0 +1,995 @@ +<title>Fossil File Formats</title> + +The global state of a fossil repository is kept simple so that it can +endure in useful form for decades or centuries. +A fossil repository is intended to be readable, +searchable, and extensible by people not yet born. + +The global state of a fossil repository is an unordered +set of <i>artifacts</i>. +An artifact might be a source code file, the text of a wiki page, +part of a trouble ticket, a description of a check-in including all +the files in that check-in with the check-in comment and so forth. +Artifacts are broadly grouped into two types: content artifacts and +structural artifacts. Content artifacts are the raw project source-code +files that are checked into the repository. Structural artifacts have +special formatting rules and are used to show the relationships between +other artifacts in the repository. It is possible for an artifact to +be both a structure artifact and a content artifact, though this is +rare. Artifacts can be text or binary. + +In addition to the global state, +each fossil repository also contains local state. +The local state consists of web-page formatting +preferences, authorized users, ticket display and reporting formats, +and so forth. The global state is shared in common among all +repositories for the same project, whereas the local state is often +different in separate repositories. +The local state is not versioned and is not synchronized +with the global state. +The local state is not composed of artifacts and is not intended to be enduring. +This document is concerned with global state only. Local state is only +mentioned here in order to distinguish it from global state. + +<h2 id="names">1.0 Artifact Names</h2> + +Each artifact in the repository is named by a hash of its content. +No prefixes, suffixes, or other information is added to an artifact before +the hash is computed. The artifact name is just the (lower-case +hexadecimal) hash of the raw artifact. + +Fossil currently computes artifact names using either SHA1 or SHA3-256. It +is relatively easy to add new algorithms in the future, but there are no +plans to do so at this time. + +When referring to artifacts in using tty commands or webpage URLs, it is +sufficient to specify a unique prefix for the artifact name. If the input +prefix is not unique, Fossil will show an error. Within a structural +artifact, however, all references to other artifacts must be the complete +hash. + +Prior to Fossil version 2.0, all names were formed from the SHA1 hash of +the artifact. The key innovation in Fossil 2.0 was adding support for +alternative hash algorithms. + +<h2 id="structural">2.0 Structural Artifacts</h2> + +A structural artifact is an artifact with a particular format +that is used to define the relationships between other artifacts in the +repository. +Fossil recognizes the following kinds of structural +artifacts: + +<ul> +<li> [#manifest | Manifests] </li> +<li> [#cluster | Clusters] </li> +<li> [#ctrl | Control (a.k.a. Tag) Artifacts] </li> +<li> [#wikichng | Wiki Pages] </li> +<li> [#tktchng | Ticket Changes] </li> +<li> [#attachment | Attachments] </li> +<li> [#event | TechNotes] </li> +<li> [#forum | Forum Posts] </li> +</ul> + +These eight structural artifact types are described in subsections below. + +Structural artifacts are UTF-8 text. The artifact may be PGP clearsigned. +After removal of the PGP clearsign header and suffix (if any) a structural +artifact consists of one or more "cards" separated by a single newline +(ASCII: 0x0a) character. Each card begins with a single +character "card type". Zero or more arguments may follow +the card type. All arguments are separated from each other +and from the card-type character by a single space +character. There is no surplus white space between arguments +and no leading or trailing whitespace except for the newline +character that acts as the card separator. All cards must be in strict +lexicographical order (except for an +[./fileformat.wiki#outofordercards|historical bug compatibility]). +There may not be any duplicate cards. + +In the current implementation (as of 2017-02-27) the artifacts that +make up a fossil repository are stored as delta- and zlib-compressed +blobs in an <a href="http://www.sqlite.org/">SQLite</a> database. This +is an implementation detail and might change in a future release. For +the purpose of this article "file format" means the format of the artifacts, +not how the artifacts are stored on disk. It is the artifact format that +is intended to be enduring. The specifics of how artifacts are stored on +disk, though stable, is not intended to live as long as the +artifact format. + +<h3 id="manifest">2.1 The Manifest</h3> + +A manifest defines a check-in. +A manifest contains a list of artifacts for +each file in the project and the corresponding filenames, as +well as information such as parent check-ins, the username of the +programmer who created the check-in, the date and time when +the check-in was created, and any check-in comments associated +with the check-in. + +Allowed cards in the manifest are as follows: + +<div class="indent"> +<b>B</b> <i>baseline-manifest</i><br> +<b>C</b> <i>checkin-comment</i><br> +<b>D</b> <i>time-and-date-stamp</i><br> +<b>F</b> <i>filename</i> ?<i>hash</i>? ?<i>permissions</i>? ?<i>old-name</i>?<br> +<b>N</b> <i>mimetype</i><br> +<b>P</b> <i>artifact-hash</i>+<br> +<b>Q</b> (<b>+</b>|<b>-</b>)<i>artifact-hash</i> ?<i>artifact-hash</i>?<br> +<b>R</b> <i>repository-checksum</i><br> +<b>T</b> (<b>+</b>|<b>-</b>|<b>*</b>)<i>tag-name</i> <b>*</b> ?<i>value</i>?<br> +<b>U</b> <i>user-login</i><br> +<b>Z</b> <i>manifest-checksum</i> +</div> + +A manifest may optionally have a single <b>B</b> card. The <b>B</b> card specifies +another manifest that serves as the "baseline" for this manifest. A +manifest that has a <b>B</b> card is called a delta-manifest and a manifest +that omits the <b>B</b> card is a baseline-manifest. The other manifest +identified by the argument of the <b>B</b> card must be a baseline-manifest. +A baseline-manifest records the complete contents of a check-in. +A delta-manifest records only changes from its baseline. + +A manifest must have exactly one <b>C</b> card. The sole argument to +the <b>C</b> card is a check-in comment that describes the check-in that +the manifest defines. The check-in comment is text. The following +escape sequences are applied to the text: +A space (ASCII 0x20) is represented as "\s" (ASCII 0x5C, 0x73). A +newline (ASCII 0x0a) is "\n" (ASCII 0x5C, x6E). A backslash +(ASCII 0x5C) is represented as two backslashes "\\". Apart from +space and newline, no other whitespace characters are allowed in +the check-in comment. Nor are any unprintable characters allowed +in the comment. + +A manifest must have exactly one <b>D</b> card. The sole argument to +the <b>D</b> card is a date-time stamp in the ISO8601 format. The +date and time should be in coordinated universal time (UTC). +The format one of: + +<pre class="indent"><i>YYYY-MM-DD<b>T</b>HH:MM:SS +YYYY-MM-DD<b>T</b>HH:MM:SS.SSS</i></pre> + +A manifest has zero or more <b>F</b> cards. Each <b>F</b> card identifies a file +that is part of the check-in. There are one, two, three, or four +arguments. The first argument is the pathname of the file in the +check-in relative to the root of the project file hierarchy. No ".." +or "." directories are allowed within the filename. Space characters +are escaped as in <b>C</b> card comment text. Backslash characters and +newlines are not allowed within filenames. The directory separator +character is a forward slash (ASCII 0x2F). The second argument to the +<b>F</b> card is the lower-case hexadecimal artifact hash of +the content artifact. The second argument is required for baseline +manifests but is optional for delta manifests. When the second +argument to the <b>F</b> card is omitted, it means that the file has been +deleted relative to the baseline (files removed in baseline manifests +versions are <em>not</em> added as <b>F</b> cards). The optional 3rd argument +defines any special access permissions associated with the file. This +can be defined as "x" to mean that the file is executable or "l" +(small letter ell) to mean a symlink. All files are always readable +and writable. This can be expressed by "w" permission if desired but +is optional. The file format might be extended with new permission +letters in the future. The optional 4th argument is the name of the +same file as it existed in the parent check-in. If the name of the +file is unchanged from its parent, then the 4th argument is omitted. + +Sidebar: due to a minor bug in Fossil's early history (specifically +2008-08-11 to 2009-01-20), manifest parsers must be able to tolerate a +single space character after the UUID value even if no permissions +value is set. Normally the space would indicate that another token +is expected, but this one case requires leniency to allow +[/artifact/32b480faa3465591|a single known historical artifact] to parse. +For full details see +[https://claude.ai/code/artifact/c121be7a-5c6a-4a18-ab7f-63d086ac8071|this article]. + +A manifest has zero or one <b>N</b> cards. The <b>N</b> card specifies the mimetype for the +text in the comment of the <b>C</b> card. If the <b>N</b> card is omitted, a default mimetype +is used. Note that the <b>N</b> card has never actually been used by +any Fossil implementation. The implementation has always interpreted +check-in comments according to the [/wiki_rules|Fossil Wiki formatting rules]. +There are no current plans to ever change that. + +A manifest has zero or one <b>P</b> cards. Most manifests have one <b>P</b> card. +The <b>P</b> card has a varying number of arguments that +define other manifests from which the current manifest +is derived. Each argument is a lowercase +hexadecimal artifact hash of a predecessor manifest. All arguments +to the <b>P</b> card must be unique within that card. +The first argument is the artifact hash of the direct ancestor of the manifest. +Other arguments define manifests with which the first was +merged to yield the current manifest. Most manifests have +a <b>P</b> card with a single argument. The first manifest in the +project has no ancestors and thus has no <b>P</b> card or (depending +on the Fossil version) an empty <b>P</b> card (no arguments). + +A manifest has zero or more <b>Q</b> cards. A <b>Q</b> card is similar to a <b>P</b> card +in that it defines a predecessor to the current check-in. But +whereas a <b>P</b> card defines the immediate ancestor or a merge +ancestor, the <b>Q</b> card is used to identify a single check-in or a small +range of check-ins which were cherry-picked for inclusion in or +exclusion from the current manifest. The first argument of +the <b>Q</b> card is the artifact ID of another manifest (the "target") +which has had its changes included or excluded in the current manifest. +The target is preceded by "+" or "-" to show inclusion or +exclusion, respectively. The optional second argument to the +<b>Q</b> card is another manifest artifact ID which is the "baseline" +for the cherry-pick. If omitted, the baseline is the primary +parent of the target. The +changes included or excluded consist of all changes moving from +the baseline to the target. + +The <b>Q</b> card was added to the interface specification on 2011-02-26. +Older versions of Fossil will reject manifests that contain <b>Q</b> cards. + +A manifest may optionally have a single <b>R</b> card. The <b>R</b> card has +a single argument which is the MD5 checksum of all files in +the check-in except the manifest itself. The checksum is expressed +as 32 characters of lowercase hexadecimal. The checksum is +computed as follows: For each file in the check-in (except for +the manifest itself) in strict sorted lexicographical order, +take the pathname of the file relative to the root of the +repository, append a single space (ASCII 0x20), the +size of the file in ASCII decimal, a single newline +character (ASCII 0x0A), and the complete text of the file. +Compute the MD5 checksum of the result. + +A manifest might contain one or more <b>T</b> cards used to set +[./branching.wiki#tags | tags or properties] +on the check-in. The format of the <b>T</b> card is the same as +described in <i>Control Artifacts</i> section below, except that the +second argument is the single character "<b>*</b>" instead of an +artifact ID. The <b>*</b> in place of the artifact ID indicates that +the tag or property applies to the current artifact. It is not +possible to encode the current artifact ID as part of an artifact, +since the act of inserting the artifact ID would change the artifact ID, +hence a <b>*</b> is used to represent "self". <b>T</b> cards are typically +added to manifests in order to set the <b>branch</b> property and a +symbolic name when the check-in is intended to start a new branch. + +Each manifest has a single <b>U</b> card. The argument to the <b>U</b> card is +the login of the user who created the manifest. The login name +is encoded using the same character escapes as is used for the +check-in comment argument to the <b>C</b> card. + +A manifest must have a single <b>Z</b> card as its last line. The argument +to the <b>Z</b> card is a 32-character lowercase hexadecimal MD5 hash +of all prior lines of the manifest up to and including the newline +character that immediately precedes the "Z", excluding any PGP +clear-signing prefix. The <b>Z</b> card is +a sanity check to prove that the manifest is well-formed and +consistent. + +A sample manifest from Fossil itself can be seen +[/artifact/28987096ac | here]. + +<h3 id="cluster">2.2 Clusters</h3> + +A cluster is an artifact that declares the existence of other artifacts. +Clusters are used during repository synchronization to help +reduce network traffic. As such, clusters are an optimization and +may be removed from a repository without loss or damage to the +underlying project code. + +Allowed cards in the cluster are as follows: + +<div class="indent"> +<b>M</b> <i>artifact-id</i><br /> +<b>Z</b> <i>checksum</i> +</div> + +A cluster contains one or more <b>M</b> cards followed by a single <b>Z</b> card. +Each <b>M</b> card has a single argument which is the artifact ID of +another artifact in the repository. The <b>Z</b> card works exactly like +the <b>Z</b> card of a manifest. The argument to the <b>Z</b> card is the +lower-case hexadecimal representation of the MD5 checksum of all +prior cards in the cluster. The <b>Z</b> card is required. + +An example cluster from Fossil can be seen +[/artifact/d03dbdd73a2a8 | here]. + +<h3 id="ctrl">2.3 Control (a.k.a. Tag) Artifacts</h3> + +Control artifacts are used to assign properties to other artifacts +within the repository. Allowed cards in a control artifact are as +follows: + +<div class="indent"> +<b>D</b> <i>time-and-date-stamp</i><br /> +<b>T</b> (<b>+</b>|<b>-</b>|<b>*</b>)<i>tag-name</i> <i>artifact-id</i> ?<i>value</i>?<br /> +<b>U</b> <i>user-name</i><br /> +<b>Z</b> <i>checksum</i><br /> +</div> + +Control articles are also referred to as Tag artifacts, but tags can +also be applied via other artifact types, as described in +[#summary|the Card Summary table]. + +A control artifact must have one <b>D</b> card, one <b>U</b> card, one <b>Z</b> card and +one or more <b>T</b> cards. No other cards or other text is +allowed in a control artifact. Control artifacts might be PGP +clearsigned. + +The <b>D</b> card and the <b>Z</b> card of a control artifact are the same +as in a manifest. + +The <b>T</b> card represents a [./branching.wiki#tags | tag or property] +that is applied to +some other artifact. The <b>T</b> card has two or three values. The +second argument is the lowercase artifact ID of the artifact +to which the tag is to be applied. The +first value is the tag name. The first character of the tag +is either "+", "-", or "*". The "+" means the tag should be added +to the artifact. The "-" means the tag should be removed. +The "*" character means the tag should be added to the artifact +and all direct descendants (but not descendants through a merge) down +to but not including the first descendant that contains a +more recent "-", "*", or "+" tag with the same name. +The optional third argument is the value of the tag. A tag +without a value is a Boolean. + +When two or more tags with the same name are applied to the +same artifact, the tag with the latest (most recent) date is +used. + +Some tags have special meaning. The "comment" tag when applied +to a check-in will override the check-in comment of that check-in +for display purposes. The "user" tag overrides the name of the +check-in user. The "date" tag overrides the check-in date. +The "branch" tag sets the name of the branch that at check-in +belongs to. Symbolic tags begin with the "sym-" prefix. + +The <b>U</b> card is the name of the user that created the control +artifact. The <b>Z</b> card is the usual required artifact checksum. + +An example control artifact can be seen [/info/9d302ccda8 | here]. + + +<h3 id="wikichng">2.4 Wiki Pages</h3> + +A wiki artifact defines a single version of a +single wiki page. +Wiki artifacts accept +the following card types: + +<div class="indent"> +<b>C</b> <i>change-comment</i><br> +<b>D</b> <i>time-and-date-stamp</i><br /> +<b>L</b> <i>wiki-title</i><br /> +<b>N</b> <i>mimetype</i><br /> +<b>P</b> <i>parent-artifact-id</i>+<br /> +<b>U</b> <i>user-name</i><br /> +<b>W</b> <i>size</i> <b>\n</b> <i>text</i> <b>\n</b><br /> +<b>Z</b> <i>checksum</i> +</div> + +The <b>D</b> card is the date and time when the wiki page was edited. +The <b>P</b> card specifies the parent wiki pages, if any. The <b>L</b> card +gives the name of the wiki page. The optional <b>N</b> card specifies +the mimetype of the wiki text. If the <b>N</b> card is omitted, the +mimetype is assumed to be text/x-fossil-wiki. +The <b>U</b> card specifies the login +of the user who made this edit to the wiki page. The <b>Z</b> card is +the usual checksum over the entire artifact and is required. + +The <b>W</b> card is used to specify the text of the wiki page. The +argument to the <b>W</b> card is an integer which is the number of bytes +of text in the wiki page. That text follows the newline character +that terminates the <b>W</b> card. The wiki text is always followed by one +extra newline. + +The <b>C</b> card on a wiki page is optional. The argument is a comment +that explains why the changes were made. The ability to have a <b>C</b> +card on a wiki page artifact was added on 2019-12-02 at the suggestion +of user George Krivov and is not currently used or generated by the +implementation. Older versions of Fossil will reject a wiki-page +artifact that includes a <b>C</b> card. + +An example wiki artifact can be seen +[/artifact?name=7b2f5fd0e0&txt=1 | here]. + +Historical note: though <b>U</b> cards are technically required for +wiki pages by this specification, Fossil has historically elided them +if the submitter is "nobody". Very few repositories enable "nobody" to +write to wiki pages, so relatively few wiki artifacts without U cards +exist, but they do exist. Implementations are recommended to accept a +missing U card as an alias for a U card with the value +<tt>nobody</tt>. + +<h3 id="tktchng">2.5 Ticket Changes</h3> + +A ticket-change artifact represents a change to a trouble ticket. +The following cards are allowed on a ticket change artifact: + +<div class="indent"> +<b>D</b> <i>time-and-date-stamp</i><br /> +<b>J</b> ?<b>+</b>?<i>name</i> ?<i>value</i>?<br /> +<b>K</b> <i>ticket-id</i><br /> +<b>U</b> <i>user-name</i><br /> +<b>Z</b> <i>checksum</i> +</div> + +The <b>D</b> card is the usual date and time stamp and represents the point +in time when the change was entered. The <b>U</b> card is the login of the +programmer who entered this change. The <b>Z</b> card is the required checksum over +the entire artifact. + +Every ticket has a distinct ticket-id: +40-character lower-case hexadecimal number. +The ticket-id is given in the <b>K</b> card. A ticket exists if it contains one or +more changes. The first "change" to a ticket is what brings the +ticket into existence. + +<b>J</b> cards specify changes to the "value" of "fields" in the ticket. +If the <i>value</i> parameter of the <b>J</b> card is omitted, then the +field is set to an empty string. +Each fossil server has a ticket configuration which specifies the fields it +understands. The ticket configuration is part of the local state for +the repository and thus can vary from one repository to another. +Hence a <b>J</b> card might specify a <i>field</i> that does not exist in the +local ticket configuration. If a <b>J</b> card specifies a <i>field</i> that +is not in the local configuration, then that <b>J</b> card +is simply ignored. + +The first argument of the <b>J</b> card is the field name. The second +value is the field value. If the field name begins with "+" then +the value is appended to the prior value. Otherwise, the value +on the <b>J</b> card replaces any previous value of the field. +The field name and value are both encoded using the character +escapes defined for the <b>C</b> card of a manifest. + +An example ticket-change artifact can be seen +[/artifact/91f1ec6af053 | here]. + +<h3 id="attachment">2.6 Attachments</h3> + +An attachment artifact associates some other artifact that is the +attachment (the source artifact) with a ticket, wiki page, technical +note, or forum post to which the attachment is connected (the target +artifact). The following cards are allowed on an attachment artifact: + +<div class="indent"> +<b>A</b> <i>filename target</i> ?<i>source</i>?<br /> +<b>C</b> <i>comment</i><br /> +<b>D</b> <i>time-and-date-stamp</i><br /> +<b>N</b> <i>mimetype</i><br /> +<b>U</b> <i>user-name</i><br /> +<b>Z</b> <i>checksum</i> +</div> + +The <b>A</b> card specifies a filename for the attachment in its first +argument. The second argument to the <b>A</b> card is the name of the +wiki page, ticket, technical note, or full hash of a forum post to +which the attachment is connected. The third argument is either +missing or else it is the lower-case artifact ID of the attachment +itself. A missing third argument means that the attachment should be +deleted. If <i>target</i> is a forum post, the hash provided should +generally be that of the first version of the post for reasons +[#forumpost-tag|explained elsewhere]. + +The <b>C</b> card is an optional comment describing what the attachment is about. +The <b>C</b> card is optional, but there can only be one. + +A single <b>D</b> card is required to give the date and time when the attachment +was applied. + +There may be zero or one <b>N</b> cards. The <b>N</b> card specifies the mimetype of the +comment text provided in the <b>C</b> card. If the <b>N</b> card is omitted, the <b>C</b> card +mimetype is taken to be text/plain. + +A single <b>U</b> card gives the name of the user who added the attachment. +If an attachment is added anonymously, then the <b>U</b> card may be omitted. + +The <b>Z</b> card is the usual checksum over the rest of the attachment artifact. +The <b>Z</b> card is required. + + +<h3 id="event">2.7 Technical Notes</h3> + +A technical note or "technote" artifact (formerly known as an "event" artifact) +associates a timeline comment and a page of text +(similar to a wiki page) with a point in time. Technotes can be used +to record project milestones, release notes, blog entries, process +checkpoints, or news articles. +The following cards are allowed on a technote artifact: + +<div class="indent"> +<b>C</b> <i>comment</i><br> +<b>D</b> <i>time-and-date-stamp</i><br /> +<b>E</b> <i>technote-time</i> <i>technote-id</i><br /> +<b>N</b> <i>mimetype</i><br /> +<b>P</b> <i>parent-artifact-id</i>+<br /> +<b>T</b> <b>+</b><i>tag-name</i> <b>*</b> ?<i>value</i>?<br /> +<b>U</b> <i>user-name</i><br /> +<b>W</b> <i>size</i> <b>\n</b> <i>text</i> <b>\n</b><br /> +<b>Z</b> <i>checksum</i> +</div> + +The <b>C</b> card contains text that is displayed on the timeline for the +technote. The <b>C</b> card is optional, but there can only be one. + +A single <b>D</b> card is required to give the date and time when the +technote artifact was created. This is different from the time at which +the technote appears on the timeline. + +A single <b>E</b> card gives the time of the technote (the point on the timeline +where the technote is displayed) and a unique identifier for the technote. +When there are multiple artifacts with the same technote-id, the one with +the most recent <b>D</b> card is the only one used. The technote-id must be a +40-character lower-case hexadecimal string. + +The optional <b>N</b> card specifies the mimetype of the text of the technote +that is contained in the <b>W</b> card. If the <b>N</b> card is omitted, then the +<b>W</b> card text mimetype is assumed to be text/x-fossil-wiki, which is the +Fossil wiki format. + +The optional <b>P</b> card specifies a prior technote with the same technote-id +from which the current technote is an edit. The <b>P</b> card is a hint to the +system that it might be space efficient to store one technote as a delta of +the other. + +A technote might contain one or more <b>T</b> cards used to set +[./branching.wiki#tags | tags or properties] +on the technote. The format of the <b>T</b> card is the same as +described in [#ctrl | Control Artifacts] section above, except that the +second argument is the single character "<b>*</b>" instead of an +artifact ID and the name is always prefaced by "<b>+</b>". +The <b>*</b> in place of the artifact ID indicates that +the tag or property applies to the current artifact. It is not +possible to encode the current artifact ID as part of an artifact, +since the act of inserting the artifact ID would change the artifact ID, +hence a <b>*</b> is used to represent "self". The "<b>+</b>" on the +name means that tags can only be "add" and they can only be non-propagating +tags. In a technote, <b>T</b> cards are normally used to set the background +display color for timelines. + +The optional <b>U</b> card gives the name of the user who entered the technote. + +A single <b>W</b> card provides wiki text for the document associated with the +technote. The format of the <b>W</b> card is exactly the same as for a +[#wikichng | wiki artifact]. + +The <b>Z</b> card is the required checksum over the rest of the artifact. + +<h3 id="forum">2.8 Forum Posts</h3> + +Forum posts are intended as a mechanism for users and developers to +discuss a project. Forum posts are like messages on a mailing list. + +The following cards are allowed on an forum post artifact: + +<div class="indent"> +<b>D</b> <i>time-and-date-stamp</i><br /> +<b>G</b> <i>thread-root</i><br /> +<b>H</b> <i>thread-title</i><br /> +<b>I</b> <i>in-reply-to</i><br /> +<b>N</b> <i>mimetype</i><br /> +<b>P</b> <i>parent-artifact-id</i><br /> +<b>U</b> <i>user-name</i><br /> +<b>W</b> <i>size</i> <b>\n</b> <i>text</i> <b>\n</b><br /> +<b>Z</b> <i>checksum</i> +</div> + +Every forum post must have either one <b>I</b> card and one <b>G</b> card +or one <b>H</b> card. +Forum posts are organized into topic threads. The initial +post for a thread (the root post) has an <b>H</b> card giving the title or +subject for that thread. The argument to the <b>H</b> card is a string +in the same format as a comment string in a <b>C</b> card. +All follow-up posts have an <b>I</b> card that +indicates which prior post in the same thread the current forum +post is replying to, and a <b>G</b> card specifying the root post for +the entire thread. The argument to <b>G</b> and <b>I</b> cards is the +artifact hash for the prior forum post to which the card refers. + +In theory, it is sufficient for follow-up posts to have only an +<b>I</b> card, since the <b>G</b> card value could be computed by following a +chain of <b>I</b> cards. However, the <b>G</b> card is required in order to +associate the artifact with a forum thread in the case where an +intermediate artifact in the <b>I</b> card chain is shunned or otherwise +becomes unreadable. + +A single <b>D</b> card is required to give the date and time when the +forum post was created. + +The optional <b>N</b> card specifies the mimetype of the text of the technote +that is contained in the <b>W</b> card. If the <b>N</b> card is omitted, then the +<b>W</b> card text mimetype is assumed to be text/x-fossil-wiki, which is the +Fossil wiki format. + +The optional <b>P</b> card specifies a prior forum post for which this +forum post is an edit. For display purposes, only the child post +is shown, though the historical post is retained as a record. +If <b>P</b> cards are used and there exist multiple versions of the same +forum post, then <b>I</b> cards for other artifacts refer to whichever +version of the post was current at the time the reply was made, +but <b>G</b> cards refer to the initial, unedited root post for the thread. +Thus, following the chain of <b>I</b> cards back to the root of the thread +may land on a different post than the one given in the <b>G</b> card. +However, following the chain of <b>I</b> cards back to the thread root, +then following <b>P</b> cards back to the initial version of the thread +root must give the same artifact as is provided by the <b>G</b> card, +otherwise the artifact containing the <b>G</b> card is considered invalid +and should be ignored. + +In general, <b>P</b> cards may contain multiple arguments, indicating a +merge. But since forum posts cannot be merged, the +<b>P</b> card of a forum post may only contain a single argument. + +The <b>U</b> card gives name of the user who entered the forum post. + +A single <b>W</b> card provides wiki text for the forum post. +The format of the <b>W</b> card is exactly the same as for a +[#wikichng | wiki artifact]. + +The <b>Z</b> card is the required checksum over the rest of the artifact. + +<a name="forumpost-tag"></a> +<h4>2.8.1 Tags and Attachments on Forum Posts</h4> + +When adding [#ctrl|tags] or [#attachment|attachments] to [#forum|forum +posts] it is generally up to the application to tag, or attach to, +only the first version of any given post. For example, if post X has +two edits then a tag applied, or attachment added, by the user to post +X+2 should generally be applied to version X instead. Though this +complicates the app logic for applying tags, it simplifies the app's +location of tags for purposes of applying tag-/attachment-dependent +logic. As of this writing (May 2026), no current Fossil use cases +would be improved by tagging specific subsequent versions of posts, +e.g. the hypothetical X+1. + +Forum posts, because they have a P-card, support propagating tags. + + +<h2 id="summary">3.0 Card Summary</h2> + +The following table summarizes the various kinds of cards that appear +on Fossil artifacts. A blank entry means that combination of card and +artifact is not legal. A number or range of numbers indicates the number +of times a card may (or must) appear in the corresponding artifact type. +e.g. a value of 1 indicates a required unique card and 1+ indicates that one +or more such cards are required. + +<table> +<tr> +<th>⇩ Card Format / Used By ⇨</th> +<th>Manifest</th> +<th>Cluster</th> +<th>Control</th> +<th>Wiki</th> +<th>Ticket</th> +<th>Attachment</th> +<th>Technote</th> +<th>Forum</th> +</tr> +<tr> +<td><b>A</b> <i>filename</i> <i>target</i> ?<i>source</i>?</td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td align=center><b>1</b></td> +<td> </td> +<td> </td> +</tr> +<tr> +<td><b>B</b> <i>baseline</i></td> +<td align=center><b>0-1</b></td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +</tr> +<tr> +<td><b>C</b> <i>comment-text</i></td> +<td align=center><b>1</b></td> +<td> </td> +<td> </td> +<td align=center><b>0-1</b></td> +<td> </td> +<td align=center><b>0-1</b></td> +<td align=center><b>0-1</b></td> +<td> </td> +</tr> +<tr> +<td><b>D</b> <i>date-time-stamp</i></td> +<td align=center><b>1</b></td> +<td> </td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +</tr> +<tr> +<td><b>E</b> <i>technote-time technote-id</i></td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td align=center><b>1</b></td> +<td> </td> +</tr> +<tr> +<td><b>F</b> <i>filename</i> ?<i>uuid</i>? ?<i>permissions</i>? ?<i>oldname</i>?</td> +<td align=center><b>0+</b></td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +</tr> +<tr> +<td><b>G</b> <i>thread-root</i></td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td align=center><b>0-1</b></td> +</tr> +<tr> +<td><b>H</b> <i>thread-title</i></td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td align=center><b>0-1</b><sup><nowiki>[4]</nowiki></sup></td> +</tr> +<tr> +<td><b>I</b> <i>in-reply-to</i></td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td align=center><b>0-1</b><sup><nowiki>[4]</nowiki></sup></td> +</tr> +<tr> +<td><b>J</b> <i>name</i> ?<i>value</i>?</td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td align=center><b>1+</b></td> +<td> </td> +<td> </td> +<td> </td> +</tr> +<tr> +<td><b>K</b> <i>ticket-uuid</i></td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td align=center><b>1</b></td> +<td> </td> +<td> </td> +<td> </td> +</tr> +<tr> +<td><b>L</b> <i>wiki-title</i></td> +<td> </td> +<td> </td> +<td> </td> +<td align=center><b>1</b></td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +</tr> +<tr> +<td><b>M</b> <i>uuid</i></td> +<td> </td> +<td align=center><b>1+</b></td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +</tr> +<tr> +<td><b>N</b> <i>mimetype</i></td> +<td align=center><b>0-1</b></td> +<td> </td> +<td> </td> +<td align=center><b>0-1</b></td> +<td> </td> +<td align=center><b>0-1</b></td> +<td align=center><b>0-1</b></td> +<td align=center><b>0-1</b></td> +</tr> +<tr> +<td><b>P</b> <i>uuid ...</i></td> +<td align=center><b>0-1</b></td> +<td> </td> +<td> </td> +<td align=center><b>0-1</b></td> +<td> </td> +<td> </td> +<td align=center><b>0-1</b></td> +<td align=center><b>0-1</b><sup><nowiki>[5]</nowiki></sup></td> +</tr> +<tr> +<td><b>Q</b> (<b>+</b>|<b>-</b>)<i>uuid</i> ?<i>uuid</i>?</td> +<td align=center><b>0+</b></td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +</tr> +<tr> +<td><b>R</b> <i>md5sum</i></td> +<td align=center><b>0-1</b></td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<td> </td> +<tr> +<td><b>T</b> (<b>+</b>|<b>*</b>|<b>-</b>)<i>tagname</i> <i>uuid</i> ?<i>value</i>?<sup><nowiki>[1]</nowiki></sup></td> +<td align=center><b>0+</b></td> +<td> </td> +<td align=center><b>1+</b><sup><nowiki>[2]</nowiki></sup></td> +<td> </td> +<td> </td> +<td> </td> +<td align=center><b>0+</b><sup><nowiki>[3]</nowiki></sup></td> +<td> </td> +</tr> +<tr> +<td><b>U</b> <i>username</i></td> +<td align=center><b>1</b></td> +<td> </td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>0-1</b></td> +<td align=center><b>0-1</b></td> +<td align=center><b>1</b></td> +</tr> +<tr> +<td><b>W</b> <i>size</i> <b>\n</b> <i>text</i> <b>\n</b></td> +<td> </td> +<td> </td> +<td> </td> +<td align=center><b>1</b></td> +<td> </td> +<td> </td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +</tr> +<tr> +<td><b>Z</b> <i>md5sum</i></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +<td align=center><b>1</b></td> +</tr> +</table> + +Footnotes: + +1) T-card names may not be made up of only hexadecimal characters, as + they would be indistinguishable from a hash prefix. + +2) Tags in [#ctrl | Control Artifacts] may not be + self-referential. i.e. their target hash may not be <tt>*</tt>. + +3) Tags in [#event | Technotes] must be self-referential. i.e. their + target hash must be <tt>*</tt>. Similarly, technote tags may only + be non-propagating "add" tags. i.e. their name prefix must be + <tt>+</tt>. + +4) [#forum | Forum Posts] must have either one H-card or one + I-card, not both. + +5) [#forum | Forum Post] P-cards may have only a single parent + hash. i.e. they may not have merge parents. + +<h2 id="addenda">4.0 Addenda</h2> + +This section contains additional information which may be useful when +implementing algorithms described above. + +<h3 id="outofordercards">4.1 Relaxed Card Ordering Due To An Historical Bug</h3> + +All cards of a structural artifact should be in lexicographical order. +The Fossil implementation verifies this and rejects any structural +artifact which has out-of-order cards. Futhermore, when Fossil is +generating new structural artifacts, it runs the generated artifact +through the parser to confirm that all cards really are in the correct +order before committing the transaction. In this way, Fossil prevents +bugs in the code from accidentally inserting misformatted artifacts. +The test parse of newly created artifacts is part of the +[./selfcheck.wiki|self-check strategy] of Fossil. It takes a +few more CPU cycles to double check each artifact before inserting it. +The developers consider those CPU cycles well-spent. + +However, the card-order safety check was accidentally disabled due to +[15d04de574383d61|a bug]. +And while that bug was lurking undetected in the code, +[5e67a7f4041a36ad|another bug] caused the N cards of Technical Notes +to occur after the P card rather than before. +Thus for a span of several years, Technical Note artifacts were being +inserted into Fossil repositories that had their N and P cards in the +wrong order. + +Both bugs have now been fixed. However, to prevent historical +Technical Note artifacts that were inserted by users in good faith +from being rejected by newer Fossil builds, the card ordering +requirement is relaxed slightly. The actual implementation is this: + +<p class=blockquote> +"All cards must be in strict lexicographic order, except that the +N and P cards of a Technical Note artifact are allowed to be +interchanged." +</p> + +Future versions of Fossil might strengthen this slightly to only allow +the out of order N and P cards for Technical Notes entered before +a certain date. + +<h3>4.2 R-Card Hash Calculation</h3> + +Given a manifest file named <tt>MF</tt>, the following Bash shell code +demonstrates how to compute the value of the <b>R</b> card in that manifest. +This example uses manifest [28987096ac]. Lines starting with <tt>#</tt> are +shell input and other lines are output. This demonstration assumes that the +file versions represented by the input manifest are checked out +under the current directory. + +<nowiki><pre> +# head MF +-----BEGIN PGP SIGNED MESSAGE----- +Hash: SHA1 + +C Make\sthe\s"clearsign"\sPGP\ssigning\sdefault\sto\soff. +D 2010-02-23T15:33:14 +F BUILD.txt 4f7988767e4e48b29f7eddd0e2cdea4555b9161c +F COPYRIGHT-GPL2.txt 06877624ea5c77efe3b7e39b0f909eda6e25a4ec +... + +# grep '^R ' MF +R c0788982781981c96a0d81465fec7192 + +# for i in $(awk '/^F /{print $2}' MF); do \ + echo $i $(stat -c '%s' $i); \ + cat $i; \ +done | md5sum +c0788982781981c96a0d81465fec7192 - +</pre></nowiki> + +Minor caveats: the above demonstration will work only when none of the +filenames in the manifest are "fossilized" (encoded) because they contain +spaces. In that case the shell-generated hash would differ because the +<tt>stat</tt> calls will fail to find such files (which are output in encoded +form here). That approach also won't work for delta manifests. Calculating +the <b>R</b> card for delta manifests requires traversing both the delta and its baseline in +lexical order of the files, preferring the delta's copy if both contain +a given file. diff --git a/www/fiveminutes.wiki b/www/fiveminutes.wiki new file mode 100644 index 0000000..4ab51e4 --- /dev/null +++ b/www/fiveminutes.wiki @@ -0,0 +1,94 @@ +<title>Up and running in 5 minutes as a single user</title> + +<p align="center"><b><i> +The following document was contributed by Gilles Ganault on 2013-01-08. +</i></b> +</p><hr> + +<h1>Up and running in 5 minutes as a single user</h1> + +This short document explains the main basic Fossil commands for a single +user, i.e. with no additional users, with no need to synchronize with some remote +repository, and no need for branching/forking. + +<h2>Create a new repository</h2> + +<tt>fossil new c:\test.repo</tt> + +This will create the new SQLite binary file that holds the repository, i.e. +files, tickets, wiki, etc. It can be located anywhere, although it's considered +best practice to keep it outside the work directory where you will work on files +after they've been checked out of the repository. + +<h2>Open the repository</h2> + +<tt>cd c:\temp\test.fossil +<br> +fossil open c:\test.repo</tt> + +This will check out the last revision of all the files in the repository, +if any, into the current work directory. In addition, it will create a binary +file _FOSSIL_ to keep track of changes (on non-Windows systems it is called +<tt>.fslckout</tt>). + +<h2>Add new files</h2> + +<tt>fossil add .</tt> + +To tell Fossil to add new files to the repository. The files aren't actually +added until you run "<tt>fossil commit</tt>. When using ".", it tells Fossil +to add all the files in the current directory recursively, i.e. including all +the files in all the subdirectories. + +Note: To tell Fossil to ignore some extensions: + +<tt>fossil settings ignore-glob "*.o,*.obj,*.exe" --global</tt> + +<h2>Remove files that haven't been committed yet</h2> + +<tt>fossil delete myfile.c</tt> + +This will simply remove the item from the list of files that were previously +added through "<tt>fossil add</tt>". + +<h2>Check current status</h2> + +<tt>fossil changes</tt> + +This shows the list of changes that have been done and will be committed the +next time you run "<tt>fossil commit</tt>". It's a useful command to run before +running "<tt>fossil commit</tt>" just to check that things are OK before proceeding. + +<h2>Commit changes</h2> + +To actually apply the pending changes to the repository, e.g. new files marked +for addition, checked-out files that have been edited and must be checked-in, +etc. + +<tt>fossil commit -m "Added stuff"</tt> + +If no file names are provided on the command-line then all changes will be checked in, +otherwise just the listed file(s) will be checked in. + +<h2>Compare two revisions of a file</h2> + +If you wish to compare the last revision of a file and its checked out version +in your work directory: + +<tt>fossil gdiff myfile.c</tt> + +If you wish to compare two different revisions of a file in the repository: + +<tt>fossil finfo myfile</tt> + +Note the first hash, which is the hash of the commit +when the file was committed. + +<tt>fossil gdiff --from HASH#1 --to HASH#2 myfile.c</tt> + +<h2>Cancel changes and go back to previous revision</h2> + +<tt>fossil revert myfile.c</tt> + +Fossil does not prompt when reverting a file. It simply reminds the user about the +"undo" command, just in case the revert was a mistake. diff --git a/www/forum.wiki b/www/forum.wiki new file mode 100644 index 0000000..2489fac --- /dev/null +++ b/www/forum.wiki @@ -0,0 +1,614 @@ +<title>Fossil Forums</title> + +<h2>Introduction</h2> + +Fossil includes a built-in discussion forum, designed to substitute +for a mailing list. Email notification is available to receive posts, +but the web-based UI must be used to enter new posts. Advantages of +the forum include: + + * <b>Easy to Administer:</b> If you have already set up a + [./server/|Fossil server] with [./alerts.md|email alerts] + then turning on the forum feature + is just a matter of flipping some permission bits. There is + no new software to install and configure, and the same logins + and passwords work. + + * <b>Consistent Display:</b> Forum posts can be in [/md_rules|Markdown], + [/wiki_rules|Fossil Wiki], or plain text. Whichever format is used, the result is + displayed consistently across all platforms and operating systems and + between mobile devices and desktops. + + * <b>Editable:</b> Forum posts can be amended after they are sent, + to fix typos or provide updates. The original posts are preserved + as part of the historical record, but only the amended posts are + displayed by default. + + * <b>Built-in Full-Text Search:</b> Forum posts can be included in + the index used by the built-in Fossil search logic. + + * <b>Off-Line Access:</b> Because forum posts are synced along with + all other artifacts, you can search the forum, or add new posts, or + edit existing posts, all while off-line. + + * <b>Automatically Cross-Referenced To Other Fossil Artifacts:</b> Because forum + posts are normal Fossil artifacts, you can link from them to + other Fossil artifacts (check-ins, wiki, tickets) and from those other + artifacts to forum posts. The reverse links are recognized and + displayed automatically on the receiver. + + * <b>Malefactor Resistant:</b> Because Fossil accepts forum posts + only via the web UI, it is more resistent to spam. Passers-by + can post to the forum anonymously (subject to moderation), without + the hassle of a sign-up process. + + * <b>Distributed and Tamper-Proof:</b> Posts are stored in the Fossil + repository using the same [./fileformat.wiki | DAG/Merkle-tree design] + that Fossil uses to store your check-ins, wiki documents, etc. + Forum posts sync to cloned repositories. + +<h2>Example Installations</h2> + +Both the [forum:/forum|Fossil project itself] and the +[https://sqlite.org/forum/forum|SQLite project] use the Fossil forum in place +of mailing lists. The forum has worked well on both projects. The ability +to post anonymously provides a low-resistance path for people to report +problems, resulting in more problems being reported and fixed. +The ability to moderate and amend forum posts means that the +forums contain better information. And backups and archives +are as easy as running "clone". + +Both Fossil and SQLite keep their forums as separate repositories. +But there is no requirement to do this. A forum can be coresident in +the same repository with the source code. + +<h2 id="setup">Setting up a Fossil Forum</h2> + +<h3 id="caps">Capabilities</h3> + +By default, no Fossil user has permission to use the forums except for +users with Setup and Admin capabilities, which get these as part of the +large package of other capabilities they get. + +For public Fossil repositories that wish to accept new users without +involving a human, go into Admin → Access and enable the "Allow +users to register themselves" setting. You may also wish to give users +in [./caps/#ucat | the <tt>anonymous</tt> user category] the +<b>[./caps/ref.html#2 | RdForum]</b> and +<b>[./caps/ref.html#3 | WrForum]</b> +capabilities: this allows people to post without creating an account +simply by solving [./antibot.wiki | a simple CAPTCHA]. + +For a private repository, you probably won't want to give the +<tt>anonymous</tt> user any forum access, but you may wish to give the +<b>RdForum</b> capability to users in the <tt>reader</tt> category. + +For either type of repository, you are likely to want to give at least +the <b>[./caps/ref.html#4 | WrTForum]</b> capability to users in the <tt>developer</tt> +category. If you did not give the <b>RdForum</b> capability to +<tt>anonymous</tt> above, you should give <tt>developer</tt> that +capability here if you choose to give it <b>WrForum</b> or +<b>WrTForum</b> capability. + +If you want to use the email alert feature, by default only those +users in the Setup and Admin user categories can make use of it. Grant +the <b>[./caps/ref.html#7 | EmailAlert]</b> capability to give others access to this feature. +Alternately, you can handle alert signups outside of Fossil, with +a Setup or Admin users manually signing users up via Admin → +Notification. You'll want to grant this capability to the +<tt>nobody</tt> user category if you want anyone to sign up without any +restrictions. Give it to <tt>anonymous</tt> instead if you want the +user to solve a simple CAPTCHA before signing up. Or, give it to +<tt>reader</tt> or <tt>developer</tt> if you want only users with Fossil +logins to have this ability. (That's assuming you give one or both of +these capabilities to every user on your Fossil repository.) + +By following this advice, you should not need to tediously add +capabilities to individual accounts except in atypical cases, such as +to grant the <b>[./caps/ref.html#5 | ModForum]</b> capability to an uncommonly +highly-trusted user. + + +<h3 id="skin">Skin Setup</h3> + +If you create a new Fossil repository with version 2.7 or newer, its +default skin is already set up correctly for typical forum +configurations. + +If you have an existing repository, you have two choices if you want its +skin to be upgraded to support forums: + +<ol> + <li>Go into Admin → Skins and switch from your current skin to + one of the stock skins. If you were on a stock skin, just switch away + from your current one to the actual stock skin, since they will be + different after the upgrade.</li> + + <li>If you have local customization that you do not want to throw + away, you can use the diff feature of Fossil's skin editor to show how + the skins differ.</li> +</ol> + +The remainder of this section summarizes the differences you're expected +to see when taking option #2. + +The first thing is that you'll need to add something like the following +to the Header part of the skin to create the navbar link: + +<verbatim> +if {[anycap 23456] || [anoncap 2] || [anoncap 3]} { + menulink /forum Forum +} +</verbatim> + +These rules say that any logged-in user with any [./caps/ref.html#2 | +forum-related capability] or an anonymous user <b>RdForum</b> or +<b>WrForum</b> capability will see the "Forum" navbar +link, which just takes you to <tt>/forum</tt>. + +The exact code you need here varies depending on which skin you're +using. Follow the style you see for the other navbar links. + +The new forum feature also brings many new CSS styles to the table. If +you're using the stock skin or something sufficiently close, the changes +may work with your existing skin as-is. Otherwise, you might need to +adjust some things, such as the background color used for the selected +forum post: + +<verbatim> +div.forumSel { + background-color: rgba(0, 0, 0, 0.05); +} +</verbatim> + +That overrides the default — a hard-coded light cyan — with a 95% +transparent black overlay instead, which simply darkens your skin's +normal background color underneath the selected post. That should work +with almost any background color except for very dark background colors. +For dark skins, an inverse of the above trick will work better: + +<verbatim> +div.forumSel { + background-color: rgba(255, 255, 255, 0.05); +} +</verbatim> + +That overlays the background with 5% white to lighten it slightly. + +Another new forum-related CSS style you might want to reflect into your +existing skin is: + +<verbatim> +div.forumPosts a:visited { + color: #6A7F94; +} +</verbatim> + +This changes the clicked-hyperlink color for the forum post links on the +main <tt>/forum</tt> page only, which allows your browser's history +mechanism to show which threads a user has read and which not. The link +color will change back to the normal link color — indicating "unread" — +when a reply is added to an existing thread because that changes where +the link from the <tt>/forum</tt> page points, taking you to the newest +post in the thread. + +The color given above is suitable for the stock skin. + +Beware that when changing this example, there are some +[https://hacks.mozilla.org/2010/03/privacy-related-changes-coming-to-css-vistited/ +| stringent restrictions] in modern browsers to prevent snoopy web sites +from brute-forcing your browsing history. (See the link for the method, +which explains the restrictions.) + + +<h3 id="search">Enable Forum Search</h3> + +One of the underlying assumptions of the forum feature is that you will +want to be able to search the forum archives, so the <tt>/forum</tt> +page always includes a search box. Since that depends on search being +enabled on the Fossil repository, Fossil warns that search is disabled +until you go into Admin → Search and enable the "Search Forum" +setting. + +You may want to enable some of the other Fossil search features while +you're in there. All of this does come at some CPU and I/O cost, which +is why it's disabled by default. + + +<h3 id="sso">Single Sign-On</h3> + +If you choose to host your discussion forums within the same repository +as your project's other Fossil-managed content, you inherently have a +single sign-on system. Contrast third-party mailing list and forum +software where you either end up with two separate user tables and +permission sets, or you must go to significant effort to integrate the +two login systems. + +You may instead choose to host your forums in a Fossil repository +separate from your project's main Fossil repository. A good reason to do +this is that you have a public project where very few of those +participating in the forum have special capability bits for project +assets managed by Fossil, so you wish to segregate the two user sets. + +Yet, what of the users who will have logins on both repositories? Some +users will be trusted with access to the project's main Fossil +repository, and these users will probably also participate in the +project's Fossil-hosted forum. Fossil has a feature to solve this +problem: [./caps/login-groups.md | login groups]. + + +<h3 id="alerts">Email Alerts (a.k.a. Notifications)</h3> + +Internet email service has become rather complicated since its initial +simple and insecure implementation decades ago. Fossil's role in all of +this is rather small at the moment, but the details of the integration +are complex enough to justify [./alerts.md | a separate document]. + +(The other reason that document is separate is that Fossil's email +alerts system also gets used by features of Fossil other than the +forum.) + + +<h2 id="access">Accessing the Forum</h2> + +There are many paths to a repository's Fossil forum: + +<ul> + <li> + If you're using the default Fossil skin as shipped with Fossil + 2.7+ or one [#skin | updated] to support it, there + is a Forum button in the navbar which appears for users able to + access the forum. With the default skin, that button will only + appear if the user's browser window is at least + 1200 pixels wide. The + Fossil admin can adjust this limit in the skin's CSS section, down + near the bottom in the definition of the `wideonly` style. + </li> + + <li>The other stock skins have this button in them as of 2.7 as well, + without the screen width restriction, since the navbar in those skins + wraps on narrow screens more gracefully than the default skin + does.</li> + + <li>Users who set up their Fossil repository under prior versions and + who now have local skin changes they don't want to overwrite by + reverting to the stock 2.7 version of the skin they chose to start + with can easily [#skin | edit their skin] to include these links.</li> + + <li>A "Forum" link appears in the drop-down panel when you click the + default skin's hamburger menu (☰) while logged in as any user + with one or more of the [#caps | user capabilities listed above].</li> + + <li>That same link also appears on the repository's <tt>/sitemap</tt> + page, since it provides the content for the hamburger menu's + panel.</li> +</ul> + +The forum uses JavaScript to allow for a more comfortable editing +experience than is possible with static HTML. If one takes issue with, +or has issues with, the JavaScript-based forum editing and response +features, there are several ways around them: (1) Clients with JS +disabled should see the legacy (pre-mid-2026) input forms and +buttons. (2) Adding <code>?nojs</code> to any <code>/forumpost</code> +URL will disable plugging-in of the JS UI elements. (3) When forum +posts are visited via <code>/info</code> instead of +<code>/forumpost</code>, the JS-based UI elements do not plug +themselves in, leaving the legacy UI intact. + + +<h2 id="moderation">How Moderation Works</h2> + +In this section, we're going to call all of the following a "forum +update:" + + * create a new post + * reply to an existing post + * edit a post or reply + +When a person with the normal <b>WrForum</b> capability +updates the forum, Fossil saves the update in its block chain, but this +update is impermanent because of two other table updates made at the +same time: + +<ol> + <li>Fossil saves the update artifact's ID in its <tt>private</tt> + table, preventing Fossil from sending such artifacts to any of the + repository's clones. (This is the same mechanism behind + [./private.wiki | private branches].)</li> + + <li>Fossil also adds a reference to that artifact in the + <tt>modreq</tt> table, which backs the moderation feature. This is + what causes Fossil to leave out the Reply button when rendering that + post's HTML in the forum's web interface.</li> +</ol> + +When a moderator approves an update, Fossil deletes these table entries, +making the update [./shunning.wiki | semi-permanent]. This changes how Fossil renders the +HTML for that update. It also means the artifact will now sync to +users with <b>[./caps/ref.html#g | Clone]</b> capability. + +When a forum user edits a moderator-approved artifact, what actually +happens under the hood is that Fossil writes another artifact to the +repository which refers to the original version as its parent, causing +Fossil UI to present the new version instead of the original. The +original version remains in the repository, just as with historical +checkins. The parent must remain in the repository for referential +integrity purposes. + +When you "Delete" a moderator-approved post or reply through Fossil UI, +it's actually an edit with blank replacement content. The only way to +truly delete such artifacts is through [./shunning.wiki | shunning]. + +When a user with <b>WrTForum</b> capability +updates the forum, it happens in the same way except that Fossil skips +the <tt>private</tt> and <tt>modreq</tt> table insertions. + +When a moderator rejects an update, that artifact is unceremoniously +removed from the tip of the block chain. This is safe because Fossil +prevents replies to a reply or post awaiting moderator approval, so +referential integrity cannot be harmed. Rejecting an edit is even +safer, since the original post remains behind, so that replies continue +to refer to that original post. + + +<h2 id="mod-user">Using the Moderation Feature</h2> + +Having described all of the work that Fossil performs under the hood on +behalf of its users, we can now give the short list of work left for the +repository's administrators and moderators: + +<ol> + <li>Add the <b>[./caps/ref.html#5 | ModForum]</b> capability to any of + your users who should have this ability. You don't need to do this + for any user with <b>[./caps/ref.html#s | Setup]</b> or + <b>[./caps/ref.html#a | Admin]</b> capability; it's + [/artifact/b16221ffb736caa2?ln=1246-1257 | already included].</li> + + <li>When someone updates the forum, an entry will appear in the + timeline if the type filter is set to "Forum" or "Any Type". If that + user has only the <b>WrForum</b> capability, any + other user with the <b>ModForum</b> capability + will see a conditional link appear at the top of the main forum + page: "Moderation Requests". Clicking this takes the moderator to + the <tt>/modreq</tt> page. A moderator may wish to keep the main + forum page open in a browser tab, reloading it occasionally to see + when the "Moderation Requests" link reappears.</li> + + <li>A moderator viewing an update pending moderation sees two + buttons at the bottom, "Approve" and "Reject" in place of the + "Delete" button that the post's creator sees. Beware that both + actions are durable and have no undo. Be careful!</li> +</ol> + +<h2 id="close-post">Closing Forum Posts</h2> + +As of version 2.23 the forum interface supports the notion of +"closing" posts. By default, only users with the [./caps/index.md|'s' +and 'a' capabilities] may close or re-open posts, or reply to closed +posts. If the [/help/forum-close-policy|forum-close-policy +configuration option] is enabled then users with +[./caps/index.md|forum-moderator permissions] may also perform those +actions. + +Closing a post has the following implications: + + * Only authorized users may edit or respond to such posts, recursively + through all responses of that post. + * Only authorized users may re-open a closed post. + +A forum thread may be closed at any given point in the conversation, +not just the starting point of the thread, and affects that specific +post and all existing responses to it. + +Note that closing a post is effectively an "advisory lock" and may be +bypassed. Any user, admin or otherwise, who can push changes to a +repository may bypass closure of a post by setting the appropriate +artifact tags on a local copy and pushing those changes to a remote +copy of the forum. + +The option to close a post, permissions permitting, appears as a +"Close" button on the currently-selected post. If the selected post is +alread closed, a "Re-open" button will be shown instead. In order to +re-open a post, <em>the closed post itself</em> must be +selected. Selecting a response to that post, which is implicitly +closed by the closure of its parent, will <em>not</em> offer a re-open +option. + +Though forum users are permitted to delete their own posts, they are +not permitted, without appropriate permissions, to close their own +posts. This is intentional, as closing one's own post can be used to +antagonize other forum users. For example, by posting something +trollish or highly controversial in nature and closing the post to +further responses. + +<h2 id="status">Setting Post Statuses</h2> + +As of version 2.29 forum posts may be tagged with a status. The +setting <tt>forum-statuses</tt> controls whether this feature is is +enabled. If it is not set, is not valid JSON5, or has only a single +entry, then status are not shown in the forum. + +<tt>forum-statuses</tt> is a JSON5-format array of objects in the +following format: + +<verbatim> +[ + {label:"Open", value:"open"}, + {label:"Resolved", value:"resolved"} + ... +]</verbatim> + +The list defines the legal statuses for forum posts and syncs with +other project-level settings. All "label" and "value" members must be +unique within the scope of that list. + +The "value" values must be legal for use as HTML "dataset" members so +that they can be used for with CSS selectors to apply per-status +styling. + +If the list is valid JSON5 and has two or more entries then the forum +behavior changes in the following ways: + + * Each TR element of the main <tt>/forum</tt> view table gets a + <tt>data-status="X"</tt> member, where "X" is the value part from + the associated status. This can be used to style the statuses + in the site skin using CSS selectors such as<br> + <tt>body.cpage-forum div.forumPosts tr[data-status="open"]</tt><br> + Because there is no default status list, no predefined styles + are applied. + * The forum list shows the "label" value of the current status and + a selection list of available filters. It elides the status column + when filtering on a specific status. + * The root of each thread, when selected, gets a new UI control for + displaying or editing the status, depending on permissions. + * A drop-down selection list is added to the <tt>/forum</tt> page + for filtering based on the status. + +A list with only a single entry is treated as empty, the justification +being that if there is only one option then the UI does not need to be +cluttered with it. + +This setting is applied via a <tt>status</tt> tag on the first version +of a legal post. That tag may be set to any value from the "values" +members of the <tt>forum-statuses</tt> list. + +Status tag rules: + + * <tt>status</tt> is only observed on the root of each thread. The UI + does nothing with that tag on responses. + * The first list entry is considered to be the default status. Any post + which has no <tt>status</tt> tag, or has a <tt>status</tt> tag value + which is not in that list, is assumed for most purposes to have the + value of the first entry in the list. + * The tag is applied only to the first version of any given post. The + UI ensures that the tag is applied to the proper post, even when + rendering a newer version. + +Caveat: a "closed" status is not recommended because it's easy to confuse with +the <a href='#close-post'>"closed" tag feature</a>, which behaves considerably +differently and predates that "status" tag support by about three years. The +"closed" semantics cannot be trivially consolidated with those of "status" +but we reserve the right to do so at some point. + +<h2 id="attachments">Attachments</h2> + +As of version 2.29 users with the [./caps/index.md|'B' capability] may +attach files to forum posts. Files may not be attached until a forum +post is saved for the first time, after which an "Attach" button will +appear in the post when it is selected. Attached files undergo +moderation exactly like forum posts do. When a moderator accepts a +pending-moderation posts, all files attached to it which are also +pending approval are also approved. Similarly, when a moderator +rejects a pending-moderation post, all files attached to it which are +also pending approval are rejected. + +Developer notes regarding the save-before-attach limitation: + + * We cannot add the attachment form to the current post editor + because (A) the Preview option would upload the files each time + (B) we cannot pre-populate file selection elements so could not + restore them when rendering the preview's response. + * To work around that we need to reimplement the forum editor + using AJAX-friendly APIs. That would allow us to render a preview + without reloading the page, to attach multiple files at once, + to support pasting a screenshot as an attachment, and to submit both + a new post and its attachments at the same time. + * We have AJAX-capable preview code in /wiki and /fileedit. + * We have paste-screenshot code in /chat but integrating it here will + require adding a "paste" button to the attachment UI. + +Developer notes regarding automatic acceptance/rejection of +attachments: + + * There is a race condition here where an admin may approve + attachments they have not yet seen: user creates posts, admin gets + notified, admin reads post, user attaches a file at that time, + admin approves post. + +TBD is whether to accept that case or to remove automatic approval of +attached files. + +<h2 id="drafts">Draft Edits</h2> + +The forum uses, if available, JavaScript's <code>localStorage</code> +or <code>sessionStorage</code> to store local drafts of new +posts, replies, and edits to existing posts. Each time an editor +input field loses focus, the draft is saved. When viewing a forum +thread, the Edit and Reply buttons get clearly marked if they +have an associated local draft. + +Local drafts for replies and responses (but not new posts) are +automatically purged at semi-random intervals. Every Nth visit to a +forum page has a small chance to run the cleanup, and drafts older +than 10 days old are purged. + +If neither <code>localStorage</code> nor <code>sessionStorage</code> +are available, it uses a transient storage object which is recreated +on each page-load, so it cannot store drafts outside of a single +thread at a time. + +Beware: Each draft is keyed to the post it is editing or replied to, +which means that editing or replying to the same post from different +tabs will cause collisions. The editor to most recently lose the focus +will save the draft, overwriting the other. + +In browsers which support Web Locks, attempts to edit resp. reply to a +post which is already being edited resp. replied to in another tab +will show an error for the second and subsequent tabs. The first tab +to use the edit/reply buttons will hold a separate Web Lock for each +and forbid edit/reply access to the post in other tabs until the +locking tab is closed. + + +<h2 id="padding">Curious Extra Page Padding</h2> + +Users may notice that the forum pages get a bunch of padding added to +the bottom of the page when an editor widget is opened. This is not a +bug, but a workaround to help avoid the UI jumping around when +rendering a preview wildly resizes the widget. The effect is +especially helpful when working on posts while the bottom footer of +the page is in the viewport, as the footer is a hard boundary against +scrolling. With the extra padding, the editor widget's top edge shifts +around less when the preview is shown and hidden. + +<h2 id="fork">Forked Edits</h2> + +It is possible to "fork" forum messages, in the same way that any +given check-in may cause a fork. This can happen when a message is +edited concurrently from two or more browser tabs or via two or more +fossil instances which subsequently sync with each other. + +When a forum post is forked, the UI currently behaves as it does for +concurrent wiki page edits, showing only the most recent +edit. Currently (2026-06) the UI is incapable of showing the older +forked copies but (A) the data are still part of the SCM history and +(B) there are plans to improve the UI to show such forks. + +For trivia's sake, here's a query which finds posts which have +forks: + +<pre><verbatim> +SELECT bp.rid AS parentRid, bp.uuid parentUuid, count(f.fprev) forkCount +FROM blob b, blob bp, forumpost f +WHERE b.rid=f.fpid +AND f.fprev=bp.rid +GROUP BY f.fprev +HAVING forkCount>1; +</verbatim></pre> + +And here's one which lists forked posts: + +<pre><verbatim> +WITH multiparent(mpid) AS ( + SELECT fprev FROM forumpost f + GROUP BY fprev + HAVING count(fprev)>1 +) +SELECT datetime(fmtime), forumpost.* +FROM forumpost, multiparent +WHERE fprev=mpid +ORDER BY mpid, fpid, fmtime; +</verbatim></pre> + +The telling part is the <code>fprev</code> column: those with matching +values forked from the post referred to by <code>fprev</code>. diff --git a/www/foss-cklist.wiki b/www/foss-cklist.wiki new file mode 100644 index 0000000..cdf90f3 --- /dev/null +++ b/www/foss-cklist.wiki @@ -0,0 +1,112 @@ +<title>Checklist For Successful Open-Source Projects</title> +<nowiki> + +This checklist is loosely derived from Tom "Spot" Callaway's Fail Score +blog post <a href="http://spot.livejournal.com/308370.html"> +http://spot.livejournal.com/308370.html</a> (see also +<a href="http://www.theopensourceway.org/book/The_Open_Source_Way-How_to_tell_if_a_FLOSS_project_is_doomed_to_FAIL.html">[1]</a>). +Tom's original post assigned point scores to the various elements and +by adding together the individual points, the reader is supposed to be able +to judge the likelihood that the project will fail. +The point scores, and the items on the list, clearly reflect Tom's +biases and are not necessarily those of the larger open-source community. +Nevertheless, the policy of the Fossil shall be to strive for a perfect +score. + +This checklist is an inversion of Tom's original post in that it strives to +say what the project should do in order to succeed rather than what it +should not do to avoid failure. The point values are omitted. + +See also: +<ul> +<li><a href="http://offog.org/articles/packaging/"> +http://offog.org/articles/packaging/</a> +<li> +<a href="http://www.gnu.org/prep/standards/standards.html#Managing-Releases"> +http://www.gnu.org/prep/standards/standards.html#Managing-Releases</a> +</ul> + +<hr> + +<ol> +<li>The source code size is less than 100 MB, uncompressed. + +<li>The project uses a Version Control System (VCS). +<ol type="a"> +<li>The VCS has a working web interface. +<li>There is documentation on how to use the VCS. +<li>The VCS is general-purpose, not something hacked together for this + one project. +</ol> + +<li>The project comes with documentation on how to build from source +and that documentation is lucid, correct, and up-to-date. + +<li>The project is configurable using an autoconf-generated configure +script, or the equivalent, and does not require: +<ol type="a"> +<li>Manually editing flat files +<li>Editing code header files +</ol> + +<li>The project should be buildable using commonly available and + standard tools like "make". + +<li>The project does not depend on 3rd-party proprietary build tools. + +<li>The project is able to dynamically link against standard libraries + such as zlib and libjpeg. +<ol type="a"> +<li>The project does not ship with the sources to standard libraries. + <i>(On the Fossil project, we will allow the SQLite library sources + to be included in the source tree as long as a system-installed + SQLite library can be used in its stead.)</i> +<li>The project does not use slightly modified versions of standard + libraries. Any required bug fixes in standard libraries are pushed + back upstream. +</ol> + + +<li>"make install" works and can be configured to target any + directory of the installer's choosing. +<ol type="a"> +<li>The project contains no unconfigurable hard-coded pathnames like + "/opt" or "/usr/local". +<li>After installation, the source tree can be moved or deleted and + the application will continue working. +</ol> + + +<li>The source code uses only \n for line endings, not \r\n. + +<li>The code does not depend on any special compiler features or bugs. + +<li>The project has a mailing list and announces releases on + the mailing list. + +<li>The project has a bug tracker. + +<li>The project has a website. + +<li>Release version numbers are in the traditional X.Y or X.Y.Z format. + +<li>Releases can be downloaded as tarball using + gzip or bzip2 compression. + +<li>Releases unpack into a versioned top-level directory. + (ex: "projectname-1.2.3/"). + +<li>A statement of license appears at the top of every source code file + and the complete text of the license is included in the source code + tarball. + +<li>There are no incompatible licenses in the code. + +<li>The project has not been blithely proclaimed "public domain" without +having gone through the tedious and exacting legal steps to actually put it +in the public domain. + +<li>There is an accurate change log in the code and on the website. + +<li>There is documentation in the code and on the website. +</ol> diff --git a/www/fossil-from-msvc.wiki b/www/fossil-from-msvc.wiki new file mode 100644 index 0000000..351b38f --- /dev/null +++ b/www/fossil-from-msvc.wiki @@ -0,0 +1,51 @@ +<h1>Integrating Fossil in the Microsoft Express 2010 IDE</h1> + +<i>Contributed by Gilles Ganault on 2013-05-24.</i> + +The Express version of Visual Studio doesn't support add-in's and plug-in's, +but it's not an issue since it's still possible to use Fossil through the +External Tools menu and Fossil is a CLI application anyway: + +<ol type="1"> + <li>Tools > Settings > Expert Settings</li> + <li>Tools > External Tools, where the items in this list map + to "External Tool X" that we'll add to our own Fossil + menu later: </li> + <ol type="1"> + <li>Rename the default "[New Tool 1]" to eg. + "Commit" 2. + </li> + <li>Change Command to where Fossil is located eg. + "c:\fossil.exe"</li> + <li>Change Arguments to the required command, eg. + "commit -m". + The user will be prompted to type the comment that Commit expects</li> + <li>Set "Initial Directory" to point it to the work directory + where the source files are currently checked out + by Fossil (eg. c:\Workspace). It's also possible to use system + variables such as "$(ProjectDir)" instead of hard-coding the path</li> + <li>Check "Prompt for arguments", since Commit + requires typing a comment. Useless for commands like Changes + that don't require arguments</li> + <li>Uncheck "Close on Exit", so we can see what Fossil says + before closing the DOS box. Note that "Use Output Window" + will display the output in a child window within the IDE instead of + opening a DOS box</li> + <li>Click on OK</li> + </ol> + <li>Tools > Customize > Commands</li> + <ol type="1"> + <li>With "Menu bar = Menu Bar" selected, click on "Add + New Menu". A new "Fossil" menu is displayed in the + IDE's menu bar</li> + <li>Click on "Modify Selection" to rename it + "Fossil", and...</li> + <li>Use the "Move Down" button to move it lower in + the list</li> + </ol> + <li>Still in Customize dialog: In the "Menu bar" combo, select + the new Fossil menu you just created, and Click on "Add Command...": + From Categories, select Tools, and select "External Command 1". + Click on Close. It's unfortunate that the IDE doesn't say which command + maps to "External Command X".</li> +</ol> diff --git a/www/fossil-is-not-relational.md b/www/fossil-is-not-relational.md new file mode 100644 index 0000000..ba2671a --- /dev/null +++ b/www/fossil-is-not-relational.md @@ -0,0 +1,407 @@ +# Fossil is not Relational + +***An Introduction to the Fossil Data Model*** + +Upon hearing that Fossil is based on sqlite, it's natural for people +unfamiliar with its internals to assume that Fossil stores its +SCM-relevant data in a database-friendly way and that the SCM history +can be modified via SQL. The truth, however, is *far stranger than +that.* + +This document introduces, at a relatively high level: + +1) The underlying enduring and immutable data format, which is + independent of any specific storage engine. + +2) The `blob` table: Fossil's single point of SCM-relevant data + storage. + +3) The transformation of (1) from its immutable raw form to a + *transient* database-friendly form. + +4) Some of the consequences of this model. + + +# Part 1: Artifacts + +```pikchr center +AllObjects: [ +A: file "Artifacts" fill lightskyblue; +down; move to A.s; move 50%; +F: file "Client" "files"; +right; move 1; up; move 50%; +B: cylinder "blob table" +right; +arrow from A.e to B.w; +arrow from F.e to B.w; +arrow dashed from B.e; +C: box rad 0.1 "Crosslink" "process"; +arrow +AUX: cylinder "Auxiliary" "tables" +arc -> cw dotted from AUX.s to B.s; +] # end of AllObjects +``` + + +The centerpiece of Fossil's architecture is a data format which +describes what we call "artifacts." Each artifact represents the state +of one atomic unit of SCM-relevant data, such as a single checkin, a +single wiki page edit, a single modification to a ticket, creation or +cancellation of tags, and similar SCM constructs. In the cases of +checkins and ticket updates, an artifact may record changes to +multiple files resp. ticket fields, but the change as a whole +is atomic. Though we often refer to both fossil-specific SCM data +and client-side content as artifacts, this document uses the term +artifact solely for the former purpose. + +From [the data format's main documentation][dataformat]: + +> The global state of a fossil repository is kept simple so that it +> can endure in useful form for decades or centuries. A fossil +> repository is intended to be readable, searchable, and extensible by +> people not yet born. + +[dataformat]: ./fileformat.wiki + +This format has the following major properties: + +- It is <u>**syntactically simple**</u>, easily and efficiently + parsable in any programming language. It is also entirely + human-readable. + +- It is <u>**immutable**</u>. An artifact is identified by its unique + hash value. Any modification to an artifact changes that hash, + thereby changing its identity. + +- It is <u>**not generic**</u>. It is custom-made for its purpose and + makes no attempt at providing a generic format. It contains *only* + what it *needs* to function, with zero bloat. + +- It <u>**holds all SCM-relevant data except for client-level file + content**</u>, the latter instead being referenced by their unique + hash values. Storage of the client-side content is an implementation + detail delegated to higher-level applications. + +- <u>**Auditability**</u>. By following the hash references in + artifacts it is possible to unambiguously trace the origin of any + modification to the SCM state. Combined with higher-level tools + (specifically, Fossil's database), this audit trail can easily be + traced both backwards and forwards in time, using any given version + in the SCM history as a starting point. + +Notably, the artifact file format <u>does not</u>... + +- Specify any specific storage mechanism for the SCM's raw bytes, + which includes both artifacts themselves and client-side file + content. The file format refers to all such content solely by its + unique hash value. + +- Specify any optimizations such as storing file-level changes as + deltas between two versions of that content. + +Such aspects are all considered to be implementation details of +higher-level applications (be they in the main fossil binary or a +hypothetical 3rd-party application), and have no effect on the +underlying artifact data model. That said, in Fossil: + +- All raw byte content (artifacts and client files) is stored in + the `blob` database table. + +- Fossil uses delta and zlib compression to keep the storage size of + changes from one version of a piece of content to the next to a + minimum. + + +## Sidebar: SCM-relevant vs Non-SCM-relevant State + +Certain data in Fossil are "SCM-relevant" and certain data are not. In +short, SCM-relevant data are managed in a way consistent with +controlled versioning of that data. Conversely, non-SCM-relevant data +are essentially any state neither specified by nor unambiguously +refererenced by the artifact file format and are therefore not +versioned. + +SCM-relevant state includes: + +- Any and all data stored in the bodies of artifacts. This includes, + but is not limited to: wiki/ticket/forum content, tags, file names + and Fossil-side permissions, the name of each user who introduces + any given artifact into the data store, the timestamp of each such + change, the inheritance tree of checkins, and many other pieces of + metadata. + +- Raw file content of versioned files. These data are external to + artifacts, which refer to them by their hashes. How they are stored + is not the concern of the data model, but (spoiler alert!) Fossil + stores them in an SQLite database, one record per distinct hash, in + its `blob` table (which we will cover more very soon). + +Non-SCM-relevant state includes: + +- Fossil's list of users and their metadata (permissions, email + address, etc.). Artifacts themselves reference users only by their + user names. Artifacts neither care whether, nor guarantee that, user + "drh" in one artifact is in fact the same "drh" referenced in + another artifact. + +- All Fossil UI configuration, e.g. the site's skin, config settings, + and project name. + +- In short, any tables in a Fossil repository file except for the + `blob` table. Most, but not all, of these tables are transient + caches for the data specified by the artifact files (which are + stored in the `blob` table), and can safely be destroyed and rebuilt + from the collection of artifacts with no loss of state to the + repository. *All* of them, except for `blob` and `delta`, can be + destroyed with no loss of *SCM-relevant* data. + +## Terminology Hair-splitting: Manifest vs. Artifact + +We sometimes refer to artifacts as "manifests," which is technically a +term for artifacts which record checkins. The various other artifact +types are arguably not "manifests," but are sometimes referred to as +such because the internal APIs use that term. + + +## A Very Basic Example + +The following artifact, truncated for brevity, represents a typical +checkin artifact (a.k.a. a manifest): + +``` +C Bug\sfix\sin\sthe\slocal\sdatabase\sfinder. +D 2007-07-30T13:01:08 +F src/VERSION 24bbb3aad63325ff33c56d777007d7cd63dc19ea +F src/add.c 1a5dfcdbfd24c65fa04da865b2e21486d075e154 +F src/blob.c 8ec1e279a6cd0cfd5f1e3f8a39f2e9a1682e0113 +<SNIP> +F www/selfcheck.html 849df9860df602dc2c55163d658c6b138213122f +P 01e7596a984e2cd2bc12abc0a741415b902cbeea +R 74a0432d81b956bfc3ff5a1a2bb46eb5 +U drh +Z c9dcc06ecead312b1c310711cb360bc3 +``` + +Each line is a single data record called a "card." The first letter of +each line tells us the type of data stored on that line and the +following space-separated tokens contain the data for that +line. Tokens which themselves contain spaces (notably the checkin +comment) have those escaped as `\s`. The raw text of wiki +pages/comments, forum posts, and ticket bodies/comments is stored +directly in the corresponding artifact, but is stored in a way which +makes such escaping unnecessary. + +The hashes seen above are a critical component of the architecture: + +- The `F` (file) records refer to the content of those files by the +hash of that content. Where that content is stored is *not* specified +by the data model. + +- The `P` (parent) line is the hash code of the parent version (itself + an artifact). + +- The `Z` line is a hash of all of the content of *this artifact* + which precedes the `Z` line. Thus any change to the content of an + artifact changes both the artifact's identity (its hash) and its `Z` + value, making it impossible to inject modified artifacts into an + existing artifact tree. + +- The `R` line is yet another consistency-checking hash which we won't + go into here except to say that it's an internal consistency + check/line of defense against modification of file content + referenced by the artifact. + +# Part 2: The `blob` Table + +```pikchr center +AllObjects: [ +A: file "Artifacts"; +down; move to A.s; move 50%; +F: file "Client" "files" fill lightskyblue; +right; move 1; up; move 50%; +B: cylinder "blob table" fill lightskyblue; +right; +arrow from A.e to B.w; +arrow from F.e to B.w; +arrow dashed from B.e; +C: box rad 0.1 "Crosslink" "process"; +arrow +AUX: cylinder "Auxiliary" "tables" +arc -> cw dotted from AUX.s to B.s; +] # end of AllObjects +``` + + +The `blob` table is the core-most storage of a Fossil repository +database, storing all SCM-relevant data (and *only* SCM-relevant +data). Each row of this table holds a single artifact or the content +for a single version of a single client-side file. Slightly truncated +for clarity, its schema contains the following fields: + +- **`uuid`**: the hash code of the blob's contents. +- **`rid`**: a unique integer key for this record. This is how the + blob table is mapped to other (transient) tables, but the RIDs are + specific to one given copy of a repository and must not be used for + cross-repository referencing. The RID is a private/internal value of + no use to a user unless they're building SQL queries for use with + the Fossil db schema. +- **`size`**: the size, in bytes, of the blob's contents, or -1 for + "phantom" blobs (those which Fossil knows should exist because it's + seen them referenced somewhere, but for which it has not been given + any content). +- **`content`**: the blob's raw content bytes, with the caveat that + Fossil is free to store it in an "alternate representation." + Specifically, the `content` field often holds a zlib-compressed + delta from a previous version of the blob's content (a separate + entry in the `blob` table), and an auxiliary table named `delta` + maps such blobs to their previous versions, such that Fossil can + reconstruct the real content from them by applying the delta to its + previous version (and such deltas may be chained). Thus extraction + of the content from this field cannot be performed via vanilla SQL, + and requires a Fossil-specific function which knows how to convert + any internal representations of the content to its original form. + + +## Sidebar: How does `blob` Distinguish Between Artifacts and Client Content? + +Notice that the `blob` table has no flag saying "this record is an +artifact" or "this record is client data." Similarly, there is no +place in the database dedicated to keeping track of which `blob` +records are artifacts and which are file content. + +That said, (A) the type of a blob can be implied via certain table +relationships and (B) the `event` table (the `/timeline`'s main data +source) incidentally has a list of artifacts and their sub-types +(checkin, wiki, tag, etc.). However, given that all of those +relationships, including the timeline, are *transient*, how can Fossil +distinguish between the two types of data? + +Fossil's artifact format is extremely rigid and is *strictly* enforced +internally, with very near zero room provided for leniency (barring a +small handful of historical compatibility cases). Every artifact which +is internally created is re-parsed for validity before it is committed +to the database, making it impossible that Fossil can inject an +invalid artifact into the repository. Because of the strictness of the +artifact parser, the chances that any given piece of arbitrary client +data could be successfully parsed as an artifact, even if it is +syntactically 99% similar to an artifact, are *effectively zero*. + +Thus Fossil's rule of interpreting the contents of the blob table is: +if it can be parsed as an artifact, it *is* an artifact, else it is +opaque client-side data. + +That rule is most often relevant in operations like `rebuild` and +`reconstruct`, both of which necessarily have to sort out artifacts +and non-artifact blobs from arbitrary collections of blobs. + +It is, in fact, possible to store an artifact unrelated to the current +repository in that repository, and it *will be parsed and processed as +an artifact* (see below), but it likely refers to other artifacts or +blobs which are not part of the current repository, thereby possibly +introducing "strange" data into the UI. If this happens, it's +potentially slightly confusing but is functionally harmless. + + +# Part 3: Crosslinking + +```pikchr center +AllObjects: [ +A: file "Artifacts"; +down; move to A.s; move 50%; +F: file "Client" "files"; +right; move 1; up; move 50%; +B: cylinder "blob table" +right; +arrow from A.e to B.w; +arrow from F.e to B.w; +arrow dashed from B.e; +C: box rad 0.1 "Crosslink" "process" fill lightskyblue; +arrow +AUX: cylinder "Auxiliary" "tables" fill lightskyblue; +arc -> cw dotted from AUX.s to B.s; +] # end of AllObjects +``` + +Once an artifact is stored in the `blob` table, how does one perform +SQL queries against its plain-text format? In short: *One Does Not +Simply Query the Artifacts*. + +Crosslinking, as its colloquially known, is a one-way processing step +which transforms an immutable artifact's state into something +database-friendly. Crosslinking happens automatically every time +Fossil generates, or is given, a new artifact. Crosslinking of any +given artifact may update many different auxiliary tables, *all* of +which are transient in the sense that they may be destroyed and then +recreated by crosslinking all artifacts from the `blob` table (which +is exactly what the `rebuild` command does). The overwhelming majority +of individual database records in any Fossil repository are found in +these transient auxiliary tables, though the `blob` table tends to +account for the overwhelming majority of a repository's disk space. + +This approach to mapping data from artifacts to the db gives Fossil +the freedom to change its database model, effectively at will, with +minimal client-side disruption (at most, a call to `rebuild`). This +allows, for example, Fossil to take advantage of new improvements in +sqlite without affecting compatibility with older repositories. + +Auxiliary tables hold data mappings such as: + +- Child/parent relationships of checkins. (The `plink` table.) +- Records of file names and changes to files. (The `mlink` and `filename` tables.) +- Timeline entries. (The `event` table.) + +And numerous other bits and pieces. + +The many auxiliary tables maintained by the app-level code reference +the `blob` table via its RID field, as that's far more efficient than +using hashes (`blob.uuid`) as foreign keys. The contexts of those +auxiliary data unambiguously tell us whether the referenced blobs are +artifacts or file content, so there is no efficiency penalty there for +hosting both opaque blobs and artifacts in the `blob` table. + +The complete SQL schemas for the core-most auxiliary tables can be found +at: + +[](/finfo/src/schema.c?ci=trunk) + +Noting, however, that all database tables are effectively internal +APIs, with no API stability guarantees and subject to change at any +time. Thus their structures generally should not be relied upon in +client-side scripts. + + +# Part 4: Implications and Consequences of the Model + +*Some* of the implications and consequences of Fossil's data model +combined with the higher-level access via SQL include: + +- **Provable immutability of history.** Fossil offers only one option + for modifying history: "shunning" is the forceful removal of an + artifact from the `blob` table and the creation of a db record + stating that the shunned hash may no longer be synced into this + repository. Shunning effectively leaves a hole in the SCM history, + and is only intended to be used for removal of illegal, dangerous, + or private information which should never have been added to the + repository. + +- **Complete separation of SCM-relevant data and app-level data + structures**. This allows the application to update its structures + at will without significant backwards-compatibility concerns. In + Fossil's case, "data structures" primarily refers to the SQL + schema. Bringing a given repository schema up to date vis a vis a + given fossil binary version simply means rebuilding the repository + with that fossil binary. There are exceptionally rare cases, namely + the switch from SHA1 to SHA3-256 ushered in with Fossil 2.0, which + can lead to true incompatibility. e.g. a Fossil 1.x client cannot + use a repository database which contains SHA3 hashes, regardless of + a rebuild. + +- **Two-way compatibility with other hypothetical clients** which also + implement the same underlying data model. So far there are none, but + it's conceivably possible. + +- **Provides a solid basis for reporting.** Fossil's real-time metrics + and reporting options are arguably the most powerful and flexible + yet seen in an SCM. + +- Very probably several more things. diff --git a/www/fossil-v-git.wiki b/www/fossil-v-git.wiki new file mode 100644 index 0000000..7c64c7f --- /dev/null +++ b/www/fossil-v-git.wiki @@ -0,0 +1,984 @@ +<title>Fossil Versus Git</title> + +<h2>1.0 Don't Stress!</h2> + +The feature sets of Fossil and [http://git-scm.com | Git] overlap in +many ways. Both are +[https://en.wikipedia.org/wiki/Distributed_version_control | distributed +version control systems] which store a tree of check-in objects to a +local repository clone. In both systems, the local clone starts out as a +full copy of the remote parent. New content gets added to the local +clone and then later optionally pushed up to the remote, and changes to +the remote can be pulled down to the local clone at will. Both systems +offer diffing, patching, branching, merging, cherry-picking, bisecting, +private branches, a stash, etc. + +Fossil has inbound and outbound Git conversion features, so if you start +out using one DVCS and later decide you like the other better, you can +easily [./inout.wiki | move your version-controlled file content].¹ + +In this document, we set all of that similarity and interoperability +aside and focus on the important differences between the two, especially +those that impact the user experience. + +Keep in mind that you are reading this on a Fossil website, and though +we try to be fair, the information here will inevitably +be biased in favor of Fossil purely because we spend most of our +time using Fossil, not Git. Ask around for second opinions from +people who have used <em>both</em> Fossil and Git. + +If you want a more practical, less philosophical guide to moving from +Git to Fossil, see our [./gitusers.md | Git to Fossil Translation Guide]. + + +<h2>2.0 Differences Between Fossil And Git</h2> + +Differences between Fossil and Git are summarized by the following table, +with further description in the text that follows. + +<table style="width: fit-content"> +<tr><th>GIT</th><th>FOSSIL</th><th>more</th></tr> +<tr> + <td>File versioning only</td> + <td> + VCS, tickets, wiki, docs, notes, forum, chat, UI, + [https://en.wikipedia.org/wiki/Role-based_access_control|RBAC] + </td> + <td><a href="#features">2.1 ↓</a></td> +</tr> +<tr> + <td>A federation of many small programs</td> + <td>One self-contained, stand-alone executable</td> + <td><a href="#selfcontained">2.2 ↓</a></td> +</tr> +<tr> + <td>Custom key/value data store</td> + <td>[https://sqlite.org/mostdeployed.html|The most used SQL database in the world]</td> + <td><a href="#durable">2.3 ↓</a></td> +</tr> +<tr> + <td>Runs natively on POSIX systems</td> + <td>Runs natively on both POSIX and Windows</td> + <td><a href="#portable">2.4 ↓</a></td> +</tr> +<tr> + <td>Bazaar-style development</td> + <td>Cathedral-style development</td> + <td><a href="#devorg">2.5.1 ↓</a></td> +</tr> +<tr> + <td>Designed for Linux kernel development</td> + <td>Designed for SQLite development</td> + <td><a href="#scale">2.5.2 ↓</a></td> +</tr> +<tr> + <td>Focus on individual branches</td> + <td>Focus on the entire tree of changes</td> + <td><a href="#branches">2.5.3 ↓</a></td> +</tr> +<tr> + <td>One check-out per repository</td> + <td>Many check-outs per repository</td> + <td><a href="#checkouts">2.6 ↓</a></td> +</tr> +<tr> + <td>Remembers what you should have done</td> + <td>Remembers what you actually did</td> + <td><a href="#history">2.7 ↓</a></td> +</tr> +<tr> + <td>Commit first</td> + <td>Test first</td> + <td><a href="#testing">2.8 ↓</a></td> +</tr> +<tr> + <td>SHA-1 or SHA-2</td> + <td>SHA-1 and/or SHA-3, in the same repository</td> + <td><a href="#hash">2.9 ↓</a></td> +</tr> +</table> + +<h3 id="features">2.1 Featureful</h3> + +Git provides file versioning services only, whereas Fossil adds +an integrated [./wikitheory.wiki | wiki], +[./bugtheory.wiki | ticketing & bug tracking], +[./embeddeddoc.wiki | embedded documentation], +[./event.wiki | technical notes], a [./forum.wiki | web forum], +and a [./chat.md | chat service], +all within a single nicely-designed [./customskin.md|skinnable] web +[/help/ui|UI], +protected by [./caps/ | a fine-grained role-based +access control system]. +These additional capabilities are available for Git as 3rd-party +add-ons, but with Fossil they are integrated into +the design, to the point that it approximates +"[https://github.com/ | GitHub]-in-a-box." + +Even if you only want straight version control, Fossil has affordances +not available in Git. + +For instance, Fossil can do operations over all local repo clones and +check-out directories with a single command. You can say "<tt>fossil +all sync</tt>" on a laptop prior to taking it off the network hosting +those repos, as before going on a trip. It doesn't matter if those +repos are private and restricted to your company network or public +Internet-hosted repos, you get synced up with everything you need while +off-network. + +You get the same capability with several other Fossil +sub-commands as well, such as "<tt>fossil all changes</tt>" to get a list of files +that you forgot to commit prior to the end of your working day, across +all repos. + +Whenever Fossil is told to modify the local checkout in some destructive +way ([/help/rm|fossil rm], [/help/update|fossil update], +[/help/revert|fossil revert], etc.) Fossil remembers the prior state +and is able to return the check-out directory to that state with a +<tt>fossil undo</tt> command. While you cannot undo a commit in Fossil +— [#history | on purpose!] — as long as the change remains confined to +the local check-out directory only, Fossil makes undo +[https://git-scm.com/book/en/v2/Git-Basics-Undoing-Things|easier than in +Git]. + +For developers who choose to self-host projects rather than rely on a +3rd-party service such as GitHub, Fossil is much easier to set up: +the stand-alone Fossil executable together with a [./server/any/cgi.md|2-line CGI script] +suffice to instantiate a full-featured developer website. To accomplish +the same using Git requires locating, installing, configuring, integrating, +and managing a wide assortment of separate tools. Standing up a developer +website using Fossil can be done in minutes, whereas doing the same using +Git requires hours or days. + +Fossil is small, complete, and self-contained. If you clone +[https://github.com/git/git|Git's self-hosting repository], you get just +Git's source code. If you clone Fossil's self-hosting repository, you +get the entire Fossil website — source code, documentation, ticket +history, and so forth.² That means you get a copy of this very article +and all of its historical versions, plus the same for all of the other +public content on this site. + + +<h3 id="selfcontained" name="selfcontained">2.2 Self Contained</h3> + +Git is actually a collection of many small tools, each doing one small +part of the job, which can be recombined (by experts) to perform +powerful operations. Git has a lot of complexity and many dependencies, +so that most people end up installing it via some kind of package +manager, simply because the creation of complicated binary packages is +best delegated to people skilled in their creation. Normal Git users are +not expected to build Git from source and install it themselves. + +Fossil is a single self-contained stand-alone executable which +depends only on common platform libraries in its default configuration. +To install one of [https://fossil-scm.org/home/uv/download.html | our +precompiled binaries], unpack the executable from the archive and put it +somewhere in your <tt>PATH</tt>. To uninstall it, delete the executable. + +This policy is particularly useful when running Fossil inside a +restrictive container, anything from [./chroot.md | classic chroot +jails] to modern [https://en.wikipedia.org/wiki/OS-level_virtualization +| OS-level virtualization mechanisms] such as +[https://en.wikipedia.org/wiki/Docker_(software) | Docker]. +Our [./containers.md | stock container image] is under 11 MB when +uncompressed and running because it contains nothing but a single +statically-linked binary. + +If you build a dynamically linked binary instead, a Linux +x86_64 build of Fossil drops to under 5 MB on-disk, stripped. +It will depend only on widespread +platform libraries with stable ABIs such as glibc, zlib, and openssl. + +Much the same is true on Windows, where our precompiled static binaries +are distributed inside a ZIP archive containing "<tt>fossil.exe</tt>" +and not a thing else, with a size on-par that of the Linux container +build. There is no "<tt>setup.exe</tt>" to run; just copy it into your +<tt>%PATH%</tt>. + +Fossil is easy to build from sources. Just run +"<tt>./configure && make</tt>" on POSIX systems and +"<tt>nmake /f Makefile.msc</tt>" on Windows. + +Git, by contrast, takes 25 MiB on Ubuntu 26.04 across 940 files. +That doesn't count platform facilities like the POSIX shell and script +interpreters its full feature set requires. A fairer comparison to +Fossil's single static binary container is the Docker Hardened Image for +[https://hub.docker.com/hardened-images/catalog/dhi/git |Git 2.x on +Alpine], which presently weighs in at 36.3 megs unpacked and 607 files, +if you count all the <tt>busybox</tt> symlinks which would otherwise be +separate binaries on a more traditional Linux system. Worst of all are +the "Git for Windows" packages where they end up needing to ship a large +yet nerfed Linux-like userland to support Git's many loosely coupled +pieces, approaching a hundred megs and thousands of files. + +Some say that Git more closely adheres to the Unix philosophy, +summarized as "many small tools, loosely joined," but we have many +examples of other successful Unix software that violates that principle +to good effect, from Apache to Python to ZFS. We can infer from that +that this is not an absolute principle of good software design. +Sometimes "many features, tightly-coupled" works better. What actually +matters is effectiveness and efficiency. We believe Fossil achieves +this. + +The above size comparisons aren't apples-to-apples anyway. We've +compared the size of Fossil with all of its [#features | many built-in +features] to a relatively bare-bones Git installation. You must add a +lot of third-party software to Git to give it a Fossil-equivalent +feature set. + +Consider [https://about.gitlab.com/|GitLab], which wraps Git in enough +features to bring it to a rough (very rough) parity with Fossil. While +it is certainly ahead in some areas — automation comes to mind — it must +be noted that GitLab's own forums are on Discourse, not self-hosted as +with Fossil. You may then argue that Discourse is superior to the Fossil +forum feature, but this merely brings us back to another of Fossil's +advantages: when the posts are part of the repo, you can migrate your +entire project to another host merely by standing that repo back up on +other hardware. Choosing to assemble your project hosting from multiple +pieces requires each to have their own backups, their own management +processes, and their own migration strategies. + +Even then, these disparate services do not cooperate at the same level +as in Fossil, where the ability to have a forum post linking to a wiki +article linking to a trouble ticket linking to a commit falls out of the +model nearly for free. These are all internal links, mind, potentially +using nothing but repository artifact hashes, all backed by SQLite's +referential integrity. With something like GitLab, its "internal +references" are tacked on after the fact, not Git repo hashes at all, +and when it comes to external services like Discourse, history tells us +you're storing up tech debt which will come due at some future point in +the form of piles of broken links when one piece or the other needs +changing out. + +Furthermore, GitLab is far more resource hungry even in its +[https://docs.gitlab.com/omnibus/settings/memory_constrained_envs/ | +minimal configuration], hence more costly to run than the equivalent +Fossil setup. [https://hub.docker.com/r/gitlab/gitlab-ce/ | The +official GitLab Community Edition container] currently clocks in at +3.44 GiB, independent of add-ons like Discourse. You can expect it to +cost around 8× as much to host it on a cloud service. Even pared down +to the minimum, there remain the consequences from the difference in +basic technology choices: Ruby and PostgreSQL vs C and SQLite. + +The Fossil project itself is [./selfhost.wiki|hosted on a small and +inexpensive VPS]. A bare-bones $5/month VPS or a +spare Raspberry Pi is sufficient to run a full-up project +site, complete with tickets, wiki, chat, and forum, in addition to +being a code repository. + +<h3 id="durable" name="database">2.3 Query Language</h3> + +The baseline data structures for Fossil and Git are the same, modulo +formatting details. Both systems manage a +[https://en.wikipedia.org/wiki/Directed_acyclic_graph | directed acyclic +graph] (DAG) of [https://en.wikipedia.org/wiki/Merkle_tree | Merkle +tree] structured check-in objects. +Check-ins are identified by a cryptographic hash of the check-in +contents, and each check-in refers to its parent via the parent's hash. + +The difference is that Git stores its objects as individual files in the +<tt>.git</tt> folder or compressed into bespoke key/value +[https://git-scm.com/book/en/v2/Git-Internals-Packfiles|pack-files], +whereas Fossil stores its objects in a [https://www.sqlite.org/|SQLite] +database file which provides ACID transactions and a high-level query +language. + +This difference is more than an implementation detail; it has important +practical consequences. + +One notable example is that it is difficult to find the descendants +of check-ins in Git. +One can easily locate the ancestors of a particular Git check-in +by following the pointers embedded in the check-in object, but +going the other direction is difficult enough +that neither native Git nor the big "forge" facilities +like GitHub and GitLab provide this capability short of crawling the +[https://www.git-scm.com/docs/git-log|commit log]. In Fossil, +we can find descendants using a simple SQL query, which then allows +us to see [/timeline?df=release&y=ci|all check-ins since the last release], +as but one example. +Git lets you see "what came before," but Fossil makes it just as +easy to also see "what came after." + +Leaf check-ins in Git that lack a "ref" become "detached," making them +difficult to locate and subject to garbage collection. This +[https://stackoverflow.com/q/3965676 | detached head +state] problem has caused grief for +[https://www.google.com/search?q=git+detached+head+state | +untold millions of Git users]. With +Fossil, detached heads are simply impossible because we can always find +our way back into the Merkle tree using one or more of the relations +in the SQL database. + +The SQL query capabilities of Fossil make it easier to track the +changes for one particular file within a project. For example, +you can easily find +[/finfo/www/fossil-v-git.wiki|the complete edit history of this one document], +or even +[/finfo/www/fossil-v-git.wiki?ubg|the same history color-coded by committer], +Both come down to simple SQL queries in Fossil, with procedural code +only being used to format the result for display. +The same result could be obtained from Git, but because the data is +in a key/value store, much more procedural code has to be written to +walk the data and compute the result. + +The ease of querying Fossil data using SQL means that status or +history information about the project under management is easier +to obtain, hence more likely to happen, giving its developers better +situational awareness. +Fossil reports tend to be more detailed and useful. +Compare [/timeline?c=6df7a853ec16865b|this Fossil timeline] +to +[https://github.com/drhsqlite/fossil-mirror/commits/master?after=f720c106d297ca1f61bccb30c5c191b88a626d01+34 | +its closest equivalent in the GitHub mirror]. Judge for yourself: which of those +reports is more useful to a developer trying to understand what happened? + +<h3 id="portable">2.4 Portable</h3> + +Fossil is largely written in ISO C, almost purely conforming to the +original 1989 standard. We make very little use of +[https://en.wikipedia.org/wiki/C99|C99], and we do not knowingly make +any use of +[https://en.wikipedia.org/wiki/C11_(C_standard_revision)|C11]. Fossil +does call POSIX and Windows APIs where necessary, but it's about +as portable as you can ask given that ISO C doesn't define all of the +facilities Fossil needs to do its thing. (Network sockets, file locking, +etc.) There are certainly well-known platforms Fossil hasn't been ported +to yet, but that's most likely due to lack of interest rather than +inherent difficulties in doing the port. We believe the most stringent +limit on its portability is that it assumes at least a 32-bit CPU and +several megs of flat-addressed memory.³ Fossil isn't quite as +[https://www.sqlite.org/custombuild.html|portable as SQLite], but it's +close. + +Over half of the C code in Fossil is actually an embedded copy of the +current version of SQLite. Much of what is Fossil-specific after you set +SQLite itself aside is SQL code calling into SQLite. The number of lines +of SQL code in Fossil isn't large by percentage, but since SQL is such +an expressive, declarative language, it has an outsized contribution to +Fossil's user-visible functionality. + +Fossil isn't entirely C and SQL code. Its web UI [./javascript.md | +uses JavaScript where +necessary]. The server-side +UI scripting uses a custom minimal +[https://en.wikipedia.org/wiki/Tcl|Tcl] dialect called +[./th1.md|TH1], which is +embedded into Fossil itself. Fossil's build system and test suite are +largely based on Tcl.⁴ All of this is quite portable. + +About half of Git's code is POSIX C, and about a third is POSIX shell +code. This is largely why the so-called "Git for Windows" distributions +(both [https://git-scm.com/download/win|first-party] and +[https://gitforwindows.org/|third-party]) are actually an +[https://www.msys2.org/wiki/Home/|MSYS POSIX portability environment] bundled +with all of the Git stuff, because it would be too painful to port Git +natively to Windows. Git is a foreign citizen on Windows, speaking to it +only through a translator.⁵ + +While Fossil does lean toward POSIX norms when given a choice — LF-only +line endings are treated as first-class citizens over CR+LF, for example +— the Windows build of Fossil is truly native. + +The third-party extensions to Git tend to follow this same pattern. +[https://docs.gitlab.com/ee/install/install_methods.html#microsoft-windows | +GitLab isn't portable to Windows at all], +for example. For that matter, GitLab isn't even officially supported on +macOS, the BSDs, or uncommon Linuxes! We have many users who regularly +build and run Fossil on all of these systems. + + +<h3 id="vs-linux">2.5 Linux vs. SQLite</h3> + +Fossil and Git promote different development styles because each one was +specifically designed to support the creator's main software +development project: [https://en.wikipedia.org/wiki/Linus_Torvalds|Linus +Torvalds] designed Git to support development of +[https://www.kernel.org/|the Linux kernel], and +[https://en.wikipedia.org/wiki/D._Richard_Hipp|D. Richard Hipp] designed +Fossil to support the development of [https://sqlite.org/|SQLite]. +Both projects must rank high on any objective list of "most +important FOSS projects," yet these two projects are almost entirely unlike +one another, so it is natural that the DVCSes created to support these +projects also differ in many ways. + +In the following sections, we will explain how four key differences +between the Linux and SQLite software development projects dictated the +design of each DVCS's low-friction usage path. + +When deciding between these two DVCSes, you should ask yourself, "Is my +project more like Linux or more like SQLite?" + + +<h4 id="devorg">2.5.1 Development Organization</h4> + +Eric S. Raymond's seminal essay-turned-book +"[https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar|The +Cathedral and the Bazaar]" details the two major development +organization styles found in +[https://en.wikipedia.org/wiki/Free_and_open-source_software|FOSS] +projects. As it happens, Linux and SQLite fall on opposite sides of this +dichotomy. Differing development organization styles dictate a different +design and low-friction usage path in the tools created to support each +project. + +Git promotes the Linux kernel's bazaar development style, in which a +loosely-associated mass of developers contribute their work through +[https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows#_dictator_and_lieutenants_workflow|a +hierarchy of lieutenants] who manage and clean up these contributions +for consideration by Linus Torvalds, who has the power to cherry-pick +individual contributions into his version of the Linux kernel. Git +allows an anonymous developer to rebase and push specific locally-named +private branches, so that a Git repo clone often isn't really a clone at +all: it may have an arbitrary number of differences relative to the +repository it originally cloned from. Git encourages siloed development. +Select work in a developer's local repository may remain private +indefinitely. + +All of this is exactly what one wants when doing bazaar-style +development. + +Fossil's normal mode of operation differs on every one of these points, +with the specific designed-in goal of promoting SQLite's cathedral +development model: + + * <b>Personal engagement:</b> SQLite's developers know each + other by name and work together daily on the project. + + * <b>Trust over hierarchy:</b> SQLite's developers check + changes into their local repository, and these are immediately and + automatically synchronized up to the central repository; there is no + "[https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows#_dictator_and_lieutenants_workflow|dictator + and lieutenants]" hierarchy as with Linux kernel contributions. D. + Richard Hipp rarely overrides decisions made by those he has trusted + with commit access on his repositories. Fossil allows you to give + [./caps/admin-v-setup.md|some users] more power over what + they can do with the repository, but Fossil [./caps/index.md#ucap | + only loosely supports] the enforcement of a development organization's + social and power hierarchies. Fossil is a great fit for + [https://en.wikipedia.org/wiki/Flat_organization|flat + organizations]. + + * <b>No easy drive-by contributions:</b> Git + [https://www.git-scm.com/docs/git-request-pull|pull requests] offer + a low-friction path to accepting + [https://www.jonobacon.com/2012/07/25/building-strong-community-structural-integrity/|drive-by + contributions]. Fossil's closest equivalents are its unique + [/help/bundle|bundle] and [/help/patch|patch] features, which require higher engagement + than firing off a PR.⁶ This difference comes directly from the + initial designed purpose for each tool: the SQLite project doesn't + accept outside contributions from previously-unknown developers, but + the Linux kernel does. + + * <b>No rebasing:</b> When your local repo clone syncs changes + up to its parent, those changes are sent exactly as they were + committed locally. [#history|There is no rebasing mechanism in + Fossil, on purpose.] + + * <b>Sync over push:</b> Explicit pushes are uncommon in + Fossil-based projects: the default is to rely on + [/help/autosync|autosync mode] instead, in which each commit + syncs immediately to its parent repository. This is a mode so you + can turn it off temporarily when needed, such as when working + offline. Fossil is still a truly distributed version control system; + it's just that its starting default is to assume you're rarely out + of communication with the parent repo. + <br><br> + This is not merely a reflection of modern always-connected computing + environments. It is a conscious decision in direct support of + SQLite's cathedral development model: we don't want developers going + dark, then showing up weeks later with a massive bolus of changes + for us to integrate all at once. + [https://en.wikipedia.org/wiki/Jim_McCarthy_(author)|Jim McCarthy] + put it well in his book on software project management, + <i>[https://www.amazon.com/dp/0735623198/|Dynamics of Software + Development]</i>: "[https://www.youtube.com/watch?v=oY6BCHqEbyc|Beware + of a guy in a room]." + + * <b>Branch names sync:</b> Unlike in Git, branch names in + Fossil are not purely local labels. They sync along with everything + else, so everyone sees the same set of branch names. Fossil's design + choice here is a direct reflection of the Linux vs. SQLite project + outlook: SQLite's developers collaborate closely on a single + coherent project, whereas Linux's developers go off on tangents and + occasionally send selected change sets to each other. + + * <b>Private branches are rare:</b> + [/doc/trunk/www/private.wiki|Private branches exist in Fossil], but + they're normally used to handle rare exception cases, whereas in + many Git projects, they're part of the straight-line development + process. + + * <b>Identical clones:</b> Fossil's autosync system tries to + keep each local clone identical to the repository it cloned + from. + +Where Git encourages siloed development, Fossil fights against it. +Fossil places a lot of emphasis on synchronizing everyone's work and on +reporting on the state of the project and the work of its developers, so +that everyone — especially the project leader — can maintain a better +mental picture of what is happening, leading to better situational +awareness. + +By contrast, "…[https://docs.github.com/en/get-started/quickstart/contributing-to-projects|forking is +at the core of social coding at GitHub]". As of June 2026, +[https://github.com/search?q=is:public|Github hosts 324 million distinct +software projects], most of which were created by forking a +previously-existing project. Since this is +[https://www.griddynamics.com/blog/number-software-developers-world | +~11× the number of developers in the world], it beggars belief that +most of these forks are still under active development. The vast bulk +of these must be abandoned one-off efforts. This is part of the nature +of bazaar style development. + +You can think about this difference in terms of +[https://en.wikipedia.org/wiki/Feedback | feedback loop size], which we +know from the mathematics of +[https://en.wikipedia.org/wiki/Control_theory | control theory] to +directly affect the speed at which any system can safely make changes. +The larger the feedback loop, the slower the whole system must run in +order to avoid loss of control. The same concept shows up in other +contexts, such as in the [https://en.wikipedia.org/wiki/OODA_loop | OODA +loop] concept. +Committing your changes to private branches in order to delay a public +push to the parent repo increases the size of your collaborators' +control loops, either causing them to slow their work in order to safely +react to your work, or to over-correct in response to each change. + +Each DVCS can be used in the opposite style, but doing so works against +their low-friction paths. + + +<h4 id="scale">2.5.2 Scale</h4> + +The Linux kernel has a far bigger developer community than that of +SQLite: there are thousands and thousands of contributors to Linux, most +of whom do not know each other's names. These thousands are responsible +for producing roughly 73× more code than is in SQLite. (32.0 +[https://en.wikipedia.org/wiki/Source_lines_of_code|MLOC] vs. 0.44 MLOC +according to [https://github.com/boyter/scc | scc].) The Linux +kernel and its development process were already uncommonly large back in +2005 when Git was designed, specifically to support the consequences of +having such a large set of developers working on such a large code base. + +95% of the code in SQLite comes from just six programmers, and 62% of +it is from the lead developer alone. The SQLite developers know each +other well and interact daily. Fossil was designed for this development +model. + +When choosing your DVCS, we think you should ask yourself whether the +scale of your software configuration management problems is closer to +those Linus Torvalds designed Git to cope with or whether your work's +scale is closer to that of SQLite, for which D. Richard Hipp designed +Fossil. An [https://en.wikipedia.org/wiki/Impact_wrench|automotive air +impact wrench] running at 8000 RPM driving an M8 socket-cap bolt at 16 +cm/s is not the best way to hang a picture on the living room wall. + +Fossil works well for projects several times the size of SQLite, +[https://core.tcl-lang.org/tcl/ | such as Tcl], with a repository over +twice the size and with many more core committers. + + +<h4 id="branches">2.5.3 Individual Branches vs. The Entire Change History</h4> + +Both Fossil and Git store history as a directed acyclic graph (DAG) +of changes, but Git tends to focus more on individual branches of +the DAG, whereas Fossil puts more emphasis on the entire DAG. + +While a common usage pattern in Git is to only synchronize +a single branch — <tt>git pull upstream feature/branch</tt> — instead +of all refs, Fossil does not give you a choice; it +syncs the entire DAG or nothing. Git commands, +GitHub, and GitLab tend to show only a single branch at +a time, whereas Fossil usually shows all parallel branches at +once. Git has commands like "rebase" that help keep all relevant +changes on a single branch, whereas Fossil encourages a style of +many concurrent branches constantly springing into existence, +undergoing active development in parallel for a few days or weeks, then +merging back into the main line and disappearing. + +This difference in emphasis arises from the different purposes of +the two systems. Git's focus on individual branches +is exactly what you want for a highly-distributed bazaar-style project +such as Linux. Linus Torvalds does not want to see every check-in +by every contributor to Linux: such extreme visibility does not scale +well. Contrast Fossil, which was written for the cathedral-style SQLite project +and its handful of active committers. Seeing all +changes on all branches all at once helps keep the whole team +up-to-date with what everybody else is doing, resulting in a more +tightly focused and cohesive implementation. + + +<h3 id="checkouts">2.6 One vs. Many Check-outs per Repository</h3> + +Because Git commingles the repository data with the initial checkout of +that repository, the default mode of operation in Git is to stick to that +single work/repo tree, even when that's a shortsighted way of working. + +Fossil doesn't work that way. A Fossil repository is an SQLite database +file which is normally stored outside the working checkout directory. You can +[/help/open | open] a Fossil repository any number of times into +any number of working directories. A common usage pattern is to have one +working directory per active working branch, so that switching branches +is done with a <tt>cd</tt> command rather than by checking out the +branches successively in a single working directory. + +Fossil does allow you to switch branches within a working checkout +directory, and this is also often done. It is simply that there is no +inherent penalty to either choice in Fossil as there is in Git. The +standard advice is to use a switch-in-place workflow in Fossil when +the disturbance from switching branches is small, and to use multiple +checkouts when you have long-lived working branches that are different +enough that switching in place is disruptive. + +While you can [./gitusers.md#worktree | use Git in the Fossil style] via +its worktree feature, tutorials continue to teach the style of having +one working directory only. Git can even fight you on this, as when +working with a forked repository; it is best to have independent clones +of the upstream and your fork, to allow separate "remote" lists and +such. This can result in two working directories but each having a +captive repo clone each, which isn't in the spirit of <tt>git +worktree</tt> at all. Yet, it beats the alternative, which then +highlights a gap in the model of diverging and re-converging forks. +Ideally, Git would let you create one of these forks as a worktree while +maintaining a strong separation between your fork and the upstream repo, +but it ends up being too much hassle to bother with. + +There are +[https://duckduckgo.com/?q=git+worktree+problem | several known +problems] with the single worktree style, ones which don't happen in +Fossil because of the clear [./ckout-workflows.md | separation] between +a Fossil repository and each working directory. + +This distinction matters because switching branches inside a single working directory loses local context +on each switch. + +For instance, in any software project where the runnable program must be +built from source files, you invalidate build objects on each switch, +artificially increasing the time required to switch versions. Most obviously, this +affects software written in statically-compiled programming languages +such as C, Java, and Haskell, but it can even affect programs written in +dynamic languages like JavaScript. A typical +[https://en.wikipedia.org/wiki/Single-page_application | SPA] build +process involves several passes: [http://browserify.org/ | Browserify] to convert +[https://nodejs.org/ | Node] packages so they'll run in a web browser, +[https://sass-lang.com | SASS] to CSS translation, +transpilation of [https://www.typescriptlang.org | Typescript] to JavaScript, +[https://github.com/mishoo/UglifyJS | uglification], etc. +Once all that processing work is done for a given input +file in a given working directory, why re-do that work just to switch +versions? If most of the files that differ between versions don't change +very often, you can save substantial time by switching branches with +<tt>cd</tt> rather than swapping versions in-place within a working +checkout directory. + +For another example, you might have an active long-running test grinding +away in a working directory, then get a call from a customer requiring +that you switch to a stable branch to answer questions in terms of the +version that customer is running. You don't want to stop the test in +order to switch your lone working directory to the stable branch. + +Disk space is cheap. Having several working directories — each with its +own local state — makes switching versions cheap and fast. + +Plus, +<tt>cd</tt> is faster to type than <tt>git checkout</tt> or <tt>fossil +update</tt>. + +<h3 id="history">2.7 What you should have done vs. What you actually did</h3> + +Git puts a lot of emphasis on maintaining +a "clean" check-in history. Extraneous and experimental branches by +individual developers often never make it into the main repository. +Branches may be rebased before being pushed to make +it appear as if development had been linear, or "squashed" to make it +appear that multiple commits were made as a single commit. +There are [https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History | +other history rewriting mechanisms in Git] as well. Git strives to record what +the development of a project should have looked like had there been no +mistakes. + +Fossil, in contrast, puts more emphasis on recording exactly what happened, +including all of the messy errors, dead-ends, experimental branches, and +so forth. One might argue that this +makes the history of a Fossil project "messy," but another point of view +is that this makes the history "accurate." In actual practice, the +superior reporting tools available in Fossil mean that this incidental mess +is not a factor. + +Like Git, Fossil has an [/help/amend|amend command] for modifying +prior commits, but unlike in Git, this works not by replacing data in +the repository, but by adding a correction record to the repository that +affects how later Fossil operations present the corrected data. The old +information is still there in the repository, it is just overridden from +the amendment point forward. + +Fossil lacks almost every other history rewriting mechanism listed on +the Git documentation page linked above. [./rebaseharm.md | There is no +rebase] in Fossil, on purpose, thus no way to reorder or copy commits +around in the commit hash tree. There is no commit squashing, dropping, +or interactive patch-based cherry-picking of commit elements in Fossil. +There is nothing like Git's <tt>filter-branch</tt> in Fossil. + +The lone exception is deleting commits. Fossil has two methods for doing +that, both of which have stringent limitations, on purpose. + +The first is [/doc/trunk/www/shunning.wiki | shunning]. See that +document for details, but briefly, you only get mandatory compliance +for shun requests within a single repository. Shun requests do not +propagate automatically between repository clones. A Fossil repository +administrator can <i>cooperatively</i> pull another repo's shun requests +across a sync boundary, so that two admins can get together and agree to +shun certain committed artifacts, but a person cannot force their local +shun requests into another repo without having admin-level control over +the receiving repo as well. Fossil's shun feature isn't for fixing up +everyday bad commits, it's for dealing with extreme situations: public +commits of secret material, ticket/wiki/forum spam, law enforcement +takedown demands, etc. + +There is also the experimental [/help/purge | <tt>purge</tt> +command], which differs from shunning in ways that aren't especially +important in the context of this document. At a 30000 foot level, you +can think of purging as useful only when you've turned off Fossil's +autosync feature and want to pluck artifacts out of its hash tree before +they get pushed. In that sense, it's approximately the same as +<tt>git rebase -i, drop</tt>. However, given that Fossil defaults to +having autosync enabled [#devorg | for good reason], the purge command +isn't very useful in practice: once a commit has been pushed into +another repo, shunning is more useful if you need to delete it from +history. + +If these accommodations strike you as incoherent with respect to +Fossil's philosophy of durable, unchanging commits, realize that if +shunning and purging were removed from Fossil, you could still remove +artifacts from the repository with SQL <tt>DELETE</tt> statements; the +repository database file is, after all, directly modifiable, being +writable by your user. Where the Fossil philosophy really takes hold is +in making it difficult to violate the integrity of the hash tree. +It's somewhat tangential, but the document [./blockchain.md | "Is Fossil +a Blockchain?"] touches on this and related topics. + +One commentator characterized Git as recording history according to +the victors, whereas Fossil records history as it actually happened. + + +<h3 id="testing">2.8 Test Before Commit</h3> + +One of the things that falls out of Git's default separation of commit +from push is that there are several Git sub-commands that jump straight +to the commit step before a change could possibly be tested. Fossil, by +contrast, makes the equivalent change to the local working check-out +only, requiring a separate check-in step to commit the change. This +design difference falls naturally out of Fossil's default-enabled +autosync feature and its philosophy of [#history | not offering history +rewriting features]. + +The prime example in Git is rebasing: the change happens to the local +repository immediately if successful, even though you haven't tested the +change yet. It's possible to argue for such a design in a tool like Git +since it lacks an autosync feature, because you can still test the +change before pushing local changes to the parent repo, but in the +meantime you've made a durable change to your local Git repository. You +must do something drastic like <tt>git reset --hard</tt> to revert that +rebase or rewrite history before pushing it if the rebase causes a +problem. If you push your rebased local repo up to the parent without +testing first, you cannot fix it without violating +[https://www.atlassian.com/git/tutorials/merging-vs-rebasing#the-golden-rule-of-rebasing +| the golden rule of rebasing]. + +Lesser examples are the Git <tt>merge</tt>, <tt>cherry-pick</tt>, and +<tt>revert</tt> commands, all of which apply work from one branch onto +another, and all of which commit their change to the local repository +immediately without giving you +an opportunity to test the change first unless you give the +<tt>--no-commit</tt> option. Otherwise, you're back in the same boat: +reset the local repository or rewrite history to fix things, then maybe +retry. + +Fossil cannot sensibly work that way because of its default-enabled +autosync feature and its purposeful paucity of commands for modifying +commits, as discussed in [#history | the prior section]. + +Instead of jumping straight to the commit step, Fossil +applies the proposed merge to the local working directory only, +requiring a separate check-in step before the change is committed to the +repository. This gives you a chance to test the change first, +either manually or by running your software's automatic tests. (Ideally, +both!) Thus, Fossil doesn't need rebase, squashing, +<tt>reset --hard</tt>, or other Git commit mutating mechanisms. + +Because Fossil requires an explicit commit for a merge, it has the nice +side benefit that it makes you give an explicit commit <i>message</i> +for each merge, whereas Git writes that commit message itself by default +unless you give the optional <tt>--edit</tt> flag to override it. + +We don't look at this difference as a workaround in Fossil for autosync, +but instead as a test-first philosophical difference: +<tt>fossil commit</tt> is a <i>commitment</i>. When every commit is +pushed to the parent repo by default, it encourages a working style in +which every commit is tested first. It encourages thinking before +acting. We believe this is an inherently good thing. + +Incidentally, this is a good example of Git's messy command design. +These three commands: + +<pre> +$ git merge HASH +$ git cherry-pick HASH +$ git revert HASH +</pre> + +...are all the same command in Fossil: + +<pre> +$ fossil merge HASH +$ fossil merge --cherrypick HASH +$ fossil merge --backout HASH +</pre> + +If you think about it, they're all the same function: apply work done on +one branch to another. All that changes between these commands is how +much work gets applied — just one check-in or a whole branch — and the +merge direction. This is the sort of thing we mean when we point out +that Fossil's command interface is simpler than Git's: there are fewer +concepts to keep track of in your mental model of Fossil's internal +operation. + +Fossil's implementation of the feature is also simpler to describe. The +brief online help for <tt>[/help/merge | fossil merge]</tt> is +currently 50 lines long, to which you want to add the ~800 lines of +[./branching.wiki | the branching document]. The equivalent +documentation in Git is the aggregation of the man pages for the above +three commands, which is approaching 1400 lines as of this writing, much of it mutually redundant. +(e.g. Git's <tt>--edit</tt> and <tt>--no-commit</tt> options get +described three times, each time differently.) Fossil's +documentation is not only more concise, it gives a nice split of brief +online help and full online documentation. + + +<h3 id="hash">2.9 Hash Algorithm: SHA-3 vs SHA-2 vs SHA-1</h3> + +Fossil started out using 160-bit SHA-1 hashes to identify check-ins, +just as in Git. That changed in early 2017 when news of the +[https://shattered.io/|SHAttered attack] broke, demonstrating that SHA-1 +collisions were now practical to create. Two weeks later, the creator of +Fossil delivered a new release allowing a clean migration to +[https://en.wikipedia.org/wiki/SHA-3|256-bit SHA-3] with +[./hashpolicy.wiki|full backwards compatibility] to old SHA-1 based +repositories. + +In October 2019, after the last of the major binary +package repos offering Fossil upgraded to Fossil 2.<i>x</i>, +we switched the default hash mode so that +the conversion to SHA-3 is fully automatic. +This not +only solves the SHAttered problem, it should prevent a reoccurrence of +similar problems for the foreseeable future. + +Meanwhile, the Git community took until August 2018 to publish +[https://git-scm.com/docs/hash-function-transition/ | their plan] for +solving the same problem by moving to SHA-256, a variant of the +[https://en.wikipedia.org/wiki/SHA-2 | older SHA-2 algorithm]. That is +now technically implemented in the sense that <tt>git init +--object-format=sha256</tt> exists, but note well: this is not only an +optional setting, Git forge support is mixed, most notably +[https://github.com/GitoxideLabs/gitoxide/issues/281 | lacking in +GitHub], plus also BitBucket and others. This is doubtless because of +this warning in the latest ([https://git-scm.com/docs/git-init/2.54.0 | +as of this writing]) <tt>git init</tt> docs: + +<blockquote>Note: At present, there is no interoperability between +SHA-256 repositories and SHA-1 repositories.</blockquote> + +Although we are now in the <i>tenth year</i> of this situation, there +remains hope that Git will manage to make the transition without taking +the full decade: the +[https://www.deployhq.com/blog/git-3-0-on-the-horizon-what-git-users-need-to-know-about-the-next-major-release +| latest plan] is that Git 3.0 will finally <i>(finally!)</i> switch to +SHA256 by default, forcing the issue. Given the track record, we are +taking a "show me" stance on this claim. + +Always remember, attacks only get better, never worse. +Almost three years after Fossil solved this problem, the +[https://sha-mbles.github.io/ | SHAmbles attack] was published, further +weakening the case for continuing to use SHA-1. + +The practical impact of attacks like SHAttered and SHAmbles on the +Git and Fossil Merkle trees isn't clear, but you want to have your repositories +moved over to a stronger hash algorithm <i>before</i> someone figures out how +to make use of the weaknesses in the old one. Fossil's solution is long +since [https://repology.org/project/fossil/versions | universally deployed]. + +<hr/> + +<h3>Asides and Digressions</h3> + +<i><small><ol> + <li><p>[./mirrorlimitations.md|Many + things are lost] in making a Git mirror of a Fossil repo due to + limitations of Git relative to Fossil. GitHub adds some of these + missing features to stock + Git, but because they're not part of Git proper, + [./mirrortogithub.md|exporting a Fossil repository to GitHub] will + still not include them; Fossil tickets do not become GitHub issues, + for example. + + <li><p>The <tt>fossil-scm.org</tt> web site is actually hosted in + several parts, so that it is not strictly true that "everything" on + it is in the self-hosting Fossil project repo. The web forum is + hosted as [https://fossil-scm.org/forum/|a separate Fossil repo] + from the [https://fossil-scm.org/home/|main Fossil self-hosting + repo] for administration reasons, and the Download page content + isn't normally synchronized with a "<tt>fossil clone</tt>" command unless + you add the "-u" option. (See "[./aboutdownload.wiki|How the + Download Page Works]" for details.) + Chat history is deliberately not synced as + chat messages are intended to be ephemeral. + There may also be some purely + static elements of the web site served via D. Richard Hipp's own + lightweight web server, + <tt>[https://sqlite.org/althttpd/|althttpd]</tt>, + which is configured as a front end to Fossil running in CGI mode on + these sites. + + <li><p>This means you can give up waiting for Fossil to be ported to + the PDP-11, but we remain hopeful that someone may eventually port + it to [https://en.wikipedia.org/wiki/Z/OS|z/OS]. + + <li><p>"Why is there all this Tcl in and around Fossil?" you may + ask. It is because D. Richard Hipp is a long-time Tcl user and + contributor. SQLite started out as an embedded database for Tcl + specifically. ([https://sqlite.org/tclsqlite.html | [Reference]]) + When he then created Fossil to manage the development of SQLite, it + was natural for him to use Tcl-based tools for its scripting, build + system, test system, etc. It came full circle in 2011 when + [https://www.reddit.com/r/programming/comments/fwrx5/tcl_and_tk_move_away_from_cvs_to_fossil/ + | the Tcl and Tk projects moved from CVS to Fossil]. + + <li><p>A minority of the pieces of the Git core software suite are + written in other languages, primarily Perl, Python, and Tcl. (e.g. + <tt>git-send-mail</tt>, <tt>git-p4</tt>, and <tt>gitk</tt>, + respectively.) Although these interpreters are quite portable, they + aren't installed by default everywhere, and on some platforms you + can't count on them at all. (Not just Windows, but also the BSDs and + many other non-Linux platforms.) This expands the dependency + footprint of Git considerably. It is why the current Git for Windows + distribution is 44.7 MiB but the current <tt>fossil.exe</tt> + zip file for Windows is 2.24 MiB. Fossil is much smaller + despite using a roughly similar amount of high-level scripting code + because its interpreters are compact and built into Fossil itself. + + <li><p>Both Fossil and Git support + [https://en.wikipedia.org/wiki/Patch_(Unix)|<tt>patch(1)</tt> + files] — unified diff formatted output — for accepting drive-by contributions, but it's a + lossy contribution path for both systems. Unlike Git PRs and Fossil + bundles, patch files collapse multiple checkins together, they don't + include check-in comments, and they cannot encode changes made above + the individual file content layer: you lose branching decisions, + tag changes, file renames, and more when using patch files. The + [./patchcmd.md | <tt>fossil patch</tt> command] + also solves these problems, but it is because it works like a Fossil + bundle, only for uncommitted changes; it doesn't use Larry Wall's + <tt>patch</tt> tool to apply unified diff output to the receiving + Fossil checkout.</p></li> +</ol></i></small> diff --git a/www/fossil.eps b/www/fossil.eps new file mode 100644 index 0000000..2e0cb84 --- /dev/null +++ b/www/fossil.eps @@ -0,0 +1,129 @@ +%!PS-Adobe-3.0 EPSF-3.0 +%%BoundingBox: 0 0 249 284 +%%Pages: 0 +%%Creator: Sun Microsystems, Inc. +%%Title: none +%%CreationDate: none +%%LanguageLevel: 2 +%%EndComments +%%BeginProlog +%%BeginResource: procset SDRes-Prolog 1.0 0 +/b4_inc_state save def +/dict_count countdictstack def +/op_count count 1 sub def +userdict begin +0 setgray 0 setlinecap 1 setlinewidth 0 setlinejoin 10 setmiterlimit[] 0 setdash newpath +/languagelevel where {pop languagelevel 1 ne {false setstrokeadjust false setoverprint} if} if +/bdef {bind def} bind def +/c {setgray} bdef +/l {neg lineto} bdef +/rl {neg rlineto} bdef +/lc {setlinecap} bdef +/lj {setlinejoin} bdef +/lw {setlinewidth} bdef +/ml {setmiterlimit} bdef +/ld {setdash} bdef +/m {neg moveto} bdef +/ct {6 2 roll neg 6 2 roll neg 6 2 roll neg curveto} bdef +/r {rotate} bdef +/t {neg translate} bdef +/s {scale} bdef +/sw {show} bdef +/gs {gsave} bdef +/gr {grestore} bdef +/f {findfont dup length dict begin +{1 index /FID ne {def} {pop pop} ifelse} forall /Encoding ISOLatin1Encoding def +currentdict end /NFont exch definefont pop /NFont findfont} bdef +/p {closepath} bdef +/sf {scalefont setfont} bdef +/ef {eofill}bdef +/pc {closepath stroke}bdef +/ps {stroke}bdef +/pum {matrix currentmatrix}bdef +/pom {setmatrix}bdef +/bs {/aString exch def /nXOfs exch def /nWidth exch def currentpoint nXOfs 0 rmoveto pum nWidth aString stringwidth pop div 1 scale aString show pom moveto} bdef +%%EndResource +%%EndProlog +%%BeginSetup +%%EndSetup +%%Page: 1 1 +%%BeginPageSetup +%%EndPageSetup +pum +0.02835 0.02837 s +0 -10008 t +/tm matrix currentmatrix def +tm setmatrix +-4875 -2599 t +1 1 s +0.500 c 7207 8311 m 7036 8281 6990 7727 7105 7073 ct 7221 6420 7453 5915 7625 5945 ct +7796 5975 7842 6529 7727 7183 ct 7612 7836 7379 8342 7207 8311 ct p ef +7607 10301 m 7445 10335 7221 9926 7107 9387 ct 6992 8847 7030 8382 7192 8348 ct +7354 8313 7578 8723 7693 9262 ct 7807 9802 7769 10267 7607 10301 ct p ef +8749 11736 m 8626 11826 8289 11574 7995 11173 ct 7701 10773 7563 10375 7686 10285 ct +7809 10195 8146 10446 8440 10847 ct 8734 11248 8872 11645 8749 11736 ct p ef +10691 12070 m 10668 12234 10205 12303 9658 12223 ct 9111 12143 8686 11946 8710 11782 ct +8734 11618 9197 11550 9744 11629 ct 10291 11709 10715 11906 10691 12070 ct p ef +10761 12053 m 10721 11951 11064 11722 11526 11541 ct 11989 11360 12396 11296 12436 11398 ct +12476 11500 12134 11729 11671 11910 ct 11208 12091 10801 12155 10761 12053 ct +p ef +12410 11353 m 12365 11314 12499 11087 12709 10847 ct 12920 10607 13126 10444 13171 10483 ct +13216 10522 13082 10749 12872 10989 ct 12662 11230 12455 11392 12410 11353 ct +p ef +7901 12525 m 7790 12525 7700 12212 7700 11826 ct 7700 11440 7790 11127 7901 11127 ct +8012 11127 8102 11440 8102 11826 ct 8102 12212 8012 12525 7901 12525 ct p ef +7826 12575 m 7765 12668 7415 12547 7044 12306 ct 6672 12064 6420 11793 6480 11700 ct +6541 11607 6891 11728 7262 11970 ct 7634 12211 7886 12482 7826 12575 ct p ef +6460 11694 m 6435 11723 6333 11673 6234 11584 ct 6134 11494 6073 11399 6099 11371 ct +6124 11343 6225 11392 6325 11482 ct 6425 11571 6485 11666 6460 11694 ct p ef +13183 10437 m 13129 10414 13185 10157 13309 9863 ct 13433 9569 13579 9350 13633 9373 ct +13688 9396 13632 9653 13508 9947 ct 13384 10241 13238 10460 13183 10437 ct p ef +9900 11524 m 9790 11524 9701 11211 9701 10825 ct 9701 10439 9790 10126 9900 10126 ct +10009 10126 10098 10439 10098 10825 ct 10098 11211 10009 11524 9900 11524 ct +p ef +9828 11574 m 9767 11667 9417 11546 9046 11305 ct 8674 11063 8422 10792 8482 10699 ct +8543 10606 8893 10727 9264 10969 ct 9636 11210 9888 11481 9828 11574 ct p ef +6085 9230 m 5976 9131 6097 8819 6357 8533 ct 6616 8247 6915 8095 7024 8194 ct +7133 8293 7012 8605 6752 8892 ct 6493 9178 6194 9329 6085 9230 ct p ef +5910 9182 m 5805 9210 5629 8884 5516 8456 ct 5403 8027 5396 7657 5501 7629 ct +5605 7601 5782 7927 5895 8356 ct 6008 8784 6014 9154 5910 9182 ct p ef +8630 9345 m 8548 9270 8691 8978 8951 8692 ct 9210 8406 9487 8234 9569 8309 ct +9651 8383 9507 8675 9248 8961 ct 8989 9247 8712 9419 8630 9345 ct p ef +8566 9556 m 8478 9624 8188 9394 7918 9043 ct 7647 8692 7499 8352 7587 8285 ct +7675 8217 7965 8447 8235 8798 ct 8506 9149 8654 9489 8566 9556 ct p ef +6579 11626 m 6549 11638 6480 11543 6425 11413 ct 6371 11283 6352 11167 6382 11154 ct +6412 11141 6481 11237 6535 11367 ct 6590 11497 6609 11613 6579 11626 ct p ef +5952 11674 m 5959 11642 6081 11642 6223 11674 ct 6366 11707 6475 11759 6468 11791 ct +6461 11823 6339 11823 6197 11790 ct 6054 11758 5945 11706 5952 11674 ct p ef +5384 7615 m 5359 7644 5257 7594 5158 7505 ct 5058 7415 4997 7320 5023 7292 ct +5048 7264 5149 7313 5249 7403 ct 5349 7492 5409 7587 5384 7615 ct p ef +5503 7547 m 5473 7559 5404 7464 5349 7334 ct 5295 7204 5276 7088 5306 7075 ct +5336 7062 5405 7158 5459 7288 ct 5514 7418 5533 7534 5503 7547 ct p ef +4875 7595 m 4882 7562 5004 7563 5147 7595 ct 5289 7628 5399 7680 5392 7712 ct +5385 7744 5263 7744 5120 7711 ct 4977 7679 4868 7627 4875 7595 ct p ef +9764 8248 m 9739 8220 9798 8125 9898 8036 ct 9997 7946 10097 7897 10123 7925 ct +10148 7952 10088 8047 9989 8137 ct 9890 8226 9789 8276 9764 8248 ct p ef +9641 8179 m 9611 8167 9631 8051 9685 7920 ct 9740 7790 9809 7694 9839 7707 ct +9869 7720 9850 7836 9795 7966 ct 9741 8097 9672 8192 9641 8179 ct p ef +10269 8227 m 10276 8259 10167 8311 10024 8343 ct 9882 8375 9760 8375 9753 8343 ct +9746 8311 9856 8259 9998 8227 ct 10141 8195 10262 8195 10269 8227 ct p ef +9749 10085 m 9724 10113 9623 10064 9523 9974 ct 9424 9885 9363 9790 9389 9762 ct +9414 9734 9515 9783 9615 9872 ct 9714 9961 9775 10057 9749 10085 ct p ef +9868 10017 m 9838 10029 9770 9934 9715 9804 ct 9661 9674 9641 9558 9671 9545 ct +9702 9533 9770 9628 9824 9758 ct 9879 9888 9898 10004 9868 10017 ct p ef +9242 10064 m 9249 10032 9371 10032 9513 10064 ct 9656 10097 9765 10149 9758 10181 ct +9751 10213 9629 10213 9487 10180 ct 9344 10148 9235 10096 9242 10064 ct p ef +6841 4401 m 6726 4212 7272 3670 8061 3191 ct 8849 2711 9581 2475 9696 2664 ct +9811 2853 9265 3395 8477 3875 ct 7688 4355 6956 4590 6841 4401 ct p ef +9097 6042 m 8862 6353 8161 6221 7532 5745 ct 6902 5270 6583 4632 6818 4321 ct +7053 4009 7754 4142 8384 4617 ct 9013 5092 9333 5730 9097 6042 ct p ef +7880 5222 m 7747 5124 7957 4611 8349 4075 ct 8742 3540 9168 3185 9302 3283 ct +9435 3381 9225 3894 8833 4430 ct 8440 4965 8014 5320 7880 5222 ct p ef +9003 6100 m 8784 6059 8750 5255 8927 4303 ct 9103 3351 9423 2612 9642 2652 ct +9861 2693 9895 3498 9719 4450 ct 9543 5401 9222 6140 9003 6100 ct p ef +0 10008 t +pom +count op_count sub {pop} repeat countdictstack dict_count sub {end} repeat b4_inc_state restore +%%PageTrailer +%%Trailer +%%EOF diff --git a/www/fossil.gif b/www/fossil.gif Binary files differnew file mode 100644 index 0000000..55a5ac5 --- /dev/null +++ b/www/fossil.gif diff --git a/www/fossil2.eps b/www/fossil2.eps new file mode 100644 index 0000000..4fde4f0 --- /dev/null +++ b/www/fossil2.eps @@ -0,0 +1,215 @@ +%!PS-Adobe-3.0 EPSF-3.0 +%%BoundingBox: 0 0 555 735 +%%Pages: 0 +%%Creator: Sun Microsystems, Inc. +%%Title: none +%%CreationDate: none +%%LanguageLevel: 2 +%%EndComments +%%BeginProlog +%%BeginResource: SDRes +/b4_inc_state save def +/dict_count countdictstack def +/op_count count 1 sub def +userdict begin +0 setgray 0 setlinecap 1 setlinewidth 0 setlinejoin 10 setmiterlimit[] 0 setdash newpath +/languagelevel where {pop languagelevel 1 ne {false setstrokeadjust false setoverprint} if} if +/bdef {bind def} bind def +/c {setgray} bdef +/l {neg lineto} bdef +/rl {neg rlineto} bdef +/lc {setlinecap} bdef +/lj {setlinejoin} bdef +/lw {setlinewidth} bdef +/ml {setmiterlimit} bdef +/ld {setdash} bdef +/m {neg moveto} bdef +/ct {6 2 roll neg 6 2 roll neg 6 2 roll neg curveto} bdef +/r {rotate} bdef +/t {neg translate} bdef +/s {scale} bdef +/sw {show} bdef +/gs {gsave} bdef +/gr {grestore} bdef +/f {findfont dup length dict begin +{1 index /FID ne {def} {pop pop} ifelse} forall /Encoding ISOLatin1Encoding def +currentdict end /NFont exch definefont pop /NFont findfont} bdef +/p {closepath} bdef +/sf {scalefont setfont} bdef +/ef {eofill}bdef +/pc {closepath stroke}bdef +/ps {stroke}bdef +/pum {matrix currentmatrix}bdef +/pom {setmatrix}bdef +/bs {/aString exch def /nXOfs exch def /nWidth exch def currentpoint nXOfs 0 rmoveto pum nWidth aString stringwidth pop div 1 scale aString show pom moveto} bdef +%%EndResource +%%EndProlog +%%BeginSetup +%%EndSetup +%%Page: 1 1 +%%BeginPageSetup +%%EndPageSetup +pum +0.02833 0.02833 s +0 -25940 t +/tm matrix currentmatrix def +gs +tm setmatrix +-1000 -1000 t +1 1 s +1000 1000 m 20589 1000 l 20589 26939 l 1000 26939 l 1000 1000 l eoclip newpath +gs +1000 1000 m 20589 1000 l 20589 26939 l 1000 26939 l 1000 1000 l eoclip newpath +1000 1000 m 20590 1000 l 20590 26940 l 1000 26940 l 1000 1000 l eoclip newpath +0.500 c 7207 8311 m 7036 8281 6989 7726 7104 7073 ct 7220 6420 7453 5913 7624 5944 ct +7796 5974 7842 6529 7727 7182 ct 7612 7835 7378 8341 7207 8311 ct p ef +7607 10301 m 7446 10335 7220 9925 7106 9386 ct 6991 8847 7030 8381 7191 8347 ct +7353 8312 7578 8723 7693 9262 ct 7807 9800 7768 10267 7607 10301 ct p ef +8749 11736 m 8627 11825 8288 11574 7995 11173 ct 7701 10772 7562 10374 7685 10284 ct +7808 10194 8146 10446 8440 10847 ct 8734 11247 8872 11646 8749 11736 ct p ef +10683 12127 m 10664 12260 10204 12303 9657 12224 ct 9111 12145 8683 11972 8702 11840 ct +8721 11707 9181 11664 9727 11743 ct 10274 11822 10702 11995 10683 12127 ct p ef +10761 12053 m 10721 11951 11064 11721 11526 11540 ct 11989 11359 12397 11296 12437 11397 ct +12477 11499 12134 11729 11671 11910 ct 11209 12091 10801 12154 10761 12053 ct +p ef +12410 11353 m 12366 11314 12500 11087 12710 10847 ct 12920 10607 13127 10444 13171 10483 ct +13216 10522 13082 10749 12872 10989 ct 12662 11229 12455 11392 12410 11353 ct +p ef +7901 12525 m 7790 12525 7700 12212 7700 11826 ct 7700 11440 7790 11127 7901 11127 ct +8012 11127 8102 11440 8102 11826 ct 8102 12212 8012 12525 7901 12525 ct p ef +7825 12576 m 7765 12669 7414 12548 7043 12306 ct 6671 12065 6419 11793 6479 11700 ct +6540 11607 6891 11728 7262 11969 ct 7633 12211 7886 12483 7825 12576 ct p ef +6460 11695 m 6434 11723 6333 11674 6233 11584 ct 6133 11495 6072 11399 6098 11371 ct +6123 11342 6225 11392 6325 11481 ct 6425 11571 6485 11666 6460 11695 ct p ef +13184 10437 m 13129 10414 13185 10156 13309 9862 ct 13434 9569 13580 9349 13634 9372 ct +13688 9395 13632 9653 13508 9947 ct 13384 10240 13238 10460 13184 10437 ct p ef +9899 11524 m 9790 11524 9700 11211 9700 10825 ct 9700 10439 9790 10126 9899 10126 ct +10008 10126 10098 10439 10098 10825 ct 10098 11211 10008 11524 9899 11524 ct +p ef +9827 11575 m 9767 11668 9416 11547 9045 11305 ct 8673 11064 8421 10792 8481 10699 ct +8542 10606 8893 10727 9264 10968 ct 9635 11210 9888 11482 9827 11575 ct p ef +6085 9230 m 5976 9132 6098 8819 6357 8533 ct 6616 8247 6915 8096 7024 8194 ct +7133 8293 7012 8605 6753 8892 ct 6493 9178 6194 9329 6085 9230 ct p ef +5910 9183 m 5806 9210 5629 8885 5516 8456 ct 5403 8028 5396 7658 5501 7630 ct +5605 7602 5782 7928 5895 8357 ct 6008 8785 6014 9155 5910 9183 ct p ef +8630 9344 m 8547 9270 8691 8977 8950 8691 ct 9209 8405 9486 8234 9568 8308 ct +9650 8383 9507 8675 9248 8961 ct 8989 9247 8712 9419 8630 9344 ct p ef +8566 9557 m 8478 9625 8187 9395 7917 9044 ct 7646 8693 7498 8353 7586 8285 ct +7674 8217 7965 8448 8235 8799 ct 8505 9150 8654 9490 8566 9557 ct p ef +6578 11626 m 6548 11638 6479 11543 6424 11413 ct 6370 11283 6351 11166 6381 11153 ct +6412 11141 6481 11236 6535 11366 ct 6589 11496 6609 11613 6578 11626 ct p ef +5952 11673 m 5959 11641 6081 11641 6224 11674 ct 6366 11706 6476 11759 6469 11791 ct +6462 11823 6340 11823 6197 11791 ct 6055 11758 5945 11706 5952 11673 ct p ef +5384 7616 m 5358 7644 5257 7595 5157 7505 ct 5057 7416 4996 7320 5022 7292 ct +5047 7263 5149 7313 5249 7402 ct 5349 7492 5409 7587 5384 7616 ct p ef +5502 7547 m 5472 7559 5403 7464 5348 7334 ct 5294 7204 5275 7087 5305 7074 ct +5336 7062 5405 7157 5459 7287 ct 5513 7417 5533 7534 5502 7547 ct p ef +4875 7594 m 4882 7562 5004 7562 5147 7594 ct 5289 7627 5399 7679 5392 7712 ct +5385 7744 5263 7744 5120 7711 ct 4978 7679 4868 7626 4875 7594 ct p ef +9763 8248 m 9738 8220 9798 8124 9897 8035 ct 9996 7946 10097 7896 10122 7924 ct +10147 7951 10087 8047 9988 8136 ct 9889 8225 9787 8275 9763 8248 ct p ef +9639 8179 m 9608 8166 9628 8050 9682 7920 ct 9737 7790 9806 7694 9836 7707 ct +9867 7719 9847 7836 9793 7966 ct 9739 8096 9669 8192 9639 8179 ct p ef +10269 8228 m 10277 8260 10166 8312 10024 8344 ct 9881 8376 9759 8376 9752 8344 ct +9745 8311 9855 8259 9998 8227 ct 10140 8195 10262 8196 10269 8228 ct p ef +9749 10085 m 9724 10113 9622 10064 9523 9975 ct 9424 9886 9363 9791 9388 9762 ct +9414 9734 9516 9784 9615 9872 ct 9714 9961 9774 10057 9749 10085 ct p ef +9868 10017 m 9839 10029 9770 9933 9715 9803 ct 9661 9673 9642 9557 9671 9544 ct +9701 9532 9770 9628 9824 9758 ct 9879 9888 9898 10004 9868 10017 ct p ef +9242 10063 m 9249 10031 9371 10031 9514 10064 ct 9656 10096 9766 10149 9759 10181 ct +9752 10213 9630 10213 9487 10181 ct 9345 10148 9235 10096 9242 10063 ct p ef +6841 4401 m 6726 4212 7272 3669 8060 3190 ct 8848 2710 9581 2475 9696 2664 ct +9811 2853 9265 3396 8477 3875 ct 7689 4354 6956 4590 6841 4401 ct p ef +9098 6041 m 8863 6352 8161 6220 7532 5745 ct 6903 5271 6583 4632 6818 4321 ct +7053 4009 7755 4142 8384 4617 ct 9013 5091 9333 5730 9098 6041 ct p ef +7879 5222 m 7746 5124 7956 4610 8348 4075 ct 8740 3540 9167 3185 9300 3283 ct +9433 3380 9224 3895 8832 4430 ct 8440 4964 8012 5319 7879 5222 ct p ef +9004 6100 m 8786 6059 8751 5255 8927 4303 ct 9103 3351 9424 2612 9642 2652 ct +9861 2693 9896 3498 9719 4449 ct 9543 5401 9222 6140 9004 6100 ct p ef +106 lw 1 lj 9435 13668 m 9429 13652 l 9421 13637 l 9412 13622 l 9401 13607 l +9390 13593 l 9377 13579 l 9363 13565 l 9347 13552 l 9331 13539 l 9313 13526 l +9295 13515 l 9275 13503 l 9254 13493 l 9233 13482 l 9211 13473 l 9188 13464 l +9164 13456 l 9139 13448 l 9114 13441 l 9088 13435 l 9062 13430 l 9035 13425 l +9008 13422 l 8981 13419 l 8953 13416 l 8925 13415 l 8898 13414 l 8870 13414 l +8842 13415 l 8814 13417 l 8787 13419 l 8759 13422 l 8732 13426 l 8706 13431 l +8680 13437 l 8654 13443 l 8629 13450 l 8605 13458 l 8581 13466 l 8558 13475 l +8536 13485 l 8514 13495 l 8494 13506 l 8475 13517 l 8456 13529 l 8439 13542 l +8423 13555 l 8408 13568 l 8394 13582 l 8381 13596 l 8370 13611 l 8360 13626 l +8351 13641 l 8344 13656 l 8338 13672 l 8333 13687 l 8330 13703 l 8328 13719 l +8327 13735 l 8328 13751 l 8330 13767 l 8334 13783 l 8338 13798 l 8345 13814 l +8352 13829 l 8361 13844 l 8372 13859 l 8383 13874 l 8396 13888 l 8410 13902 l +8425 13915 l 8441 13928 l 8459 13940 l 8477 13952 l 8497 13964 l 8517 13974 l +8539 13985 l 8561 13994 l 8584 14003 l 8608 14011 l 8632 14019 l 8657 14026 l +8683 14032 l 8709 14037 l 8736 14042 l 8763 14046 l 8790 14049 l 8818 14052 l +8846 14053 l 8873 14054 l 8901 14054 l 8929 14053 l 8957 14051 l ps +8347 14443 m 8354 14459 l 8362 14474 l 8371 14489 l 8382 14504 l 8394 14518 l +8407 14533 l 8421 14546 l 8436 14560 l 8453 14573 l 8471 14585 l 8490 14597 l +8510 14608 l 8530 14619 l 8552 14629 l 8575 14639 l 8598 14647 l 8622 14656 l +8647 14663 l 8673 14670 l 8699 14676 l 8725 14681 l 8752 14685 l 8779 14689 l +8807 14692 l 8835 14694 l 8863 14695 l 8891 14696 l 8919 14696 l 8947 14695 l +8975 14693 l 9002 14690 l 9030 14687 l 9057 14682 l 9083 14677 l 9109 14672 l +9135 14665 l 9160 14658 l 9184 14650 l 9208 14641 l 9231 14632 l 9253 14622 l +9274 14612 l 9294 14600 l 9313 14589 l 9332 14576 l 9349 14564 l 9364 14550 l +9379 14537 l 9393 14523 l 9405 14508 l 9416 14494 l 9425 14479 l 9434 14463 l +9441 14448 l 9446 14432 l 9451 14416 l 9453 14400 l 9455 14384 l 9455 14368 l +9453 14352 l 9451 14336 l 9446 14320 l 9441 14305 l 9434 14289 l 9426 14274 l +9416 14259 l 9405 14244 l 9393 14230 l 9380 14216 l 9365 14202 l 9349 14189 l +9332 14176 l 9314 14164 l 9295 14152 l 9275 14141 l 9254 14130 l 9232 14120 l +9209 14111 l 9185 14102 l 9161 14094 l 9136 14087 l 9110 14081 l 9084 14075 l +9057 14070 l 9030 14066 l 9003 14062 l 8975 14059 l 8948 14057 l ps +10974 13668 m 10968 13652 l 10960 13637 l 10951 13622 l 10940 13607 l +10929 13593 l 10916 13579 l 10902 13565 l 10886 13552 l 10870 13539 l +10853 13526 l 10834 13515 l 10814 13503 l 10794 13493 l 10772 13482 l +10750 13473 l 10727 13464 l 10703 13456 l 10679 13448 l 10653 13441 l +10628 13435 l 10602 13430 l 10575 13425 l 10548 13422 l 10521 13419 l +10493 13416 l 10465 13415 l 10438 13414 l 10410 13414 l 10382 13415 l +10354 13417 l 10327 13419 l 10300 13422 l 10273 13426 l 10246 13431 l +10220 13437 l 10194 13443 l 10169 13450 l 10145 13458 l 10121 13466 l +10098 13475 l 10076 13485 l 10055 13495 l 10035 13506 l 10015 13517 l +9997 13529 l 9980 13542 l 9964 13555 l 9949 13568 l 9935 13582 l 9922 13596 l +9911 13611 l 9901 13626 l 9892 13641 l 9885 13656 l 9879 13672 l 9874 13687 l +9871 13703 l 9869 13719 l 9868 13735 l 9869 13751 l 9871 13767 l 9875 13783 l +9879 13798 l 9886 13814 l 9893 13829 l 9902 13844 l 9912 13859 l 9924 13874 l +9937 13888 l 9951 13902 l 9966 13915 l 9982 13928 l 10000 13940 l +10018 13952 l 10037 13964 l 10058 13974 l 10079 13985 l 10101 13994 l +10124 14003 l 10148 14011 l 10173 14019 l 10198 14026 l 10223 14032 l +10250 14037 l 10276 14042 l 10303 14046 l 10330 14049 l 10358 14052 l +10386 14053 l 10413 14054 l 10441 14054 l 10469 14053 l 10497 14051 l +ps +9888 14443 m 9895 14459 l 9903 14474 l 9912 14489 l 9923 14504 l 9934 14518 l +9948 14533 l 9962 14546 l 9977 14560 l 9994 14573 l 10012 14585 l +10031 14597 l 10050 14608 l 10071 14619 l 10093 14629 l 10115 14639 l +10139 14647 l 10163 14656 l 10188 14663 l 10213 14670 l 10239 14676 l +10265 14681 l 10292 14685 l 10320 14689 l 10347 14692 l 10375 14694 l +10403 14695 l 10431 14696 l 10459 14696 l 10487 14695 l 10514 14693 l +10542 14690 l 10569 14687 l 10596 14682 l 10623 14677 l 10649 14672 l +10675 14665 l 10700 14658 l 10724 14650 l 10748 14641 l 10770 14632 l +10792 14622 l 10813 14612 l 10834 14600 l 10853 14589 l 10871 14576 l +10888 14564 l 10904 14550 l 10918 14537 l 10932 14523 l 10944 14508 l +10955 14494 l 10965 14479 l 10973 14463 l 10980 14448 l 10985 14432 l +10990 14416 l 10992 14400 l 10994 14384 l 10994 14368 l 10992 14352 l +10990 14336 l 10986 14320 l 10980 14305 l 10973 14289 l 10965 14274 l +10955 14259 l 10944 14244 l 10932 14230 l 10919 14216 l 10904 14202 l +10888 14189 l 10871 14176 l 10853 14164 l 10834 14152 l 10814 14141 l +10793 14130 l 10771 14120 l 10748 14111 l 10725 14102 l 10700 14094 l +10675 14087 l 10650 14081 l 10624 14075 l 10597 14070 l 10570 14066 l +10543 14062 l 10515 14059 l 10488 14057 l ps +7205 14684 m 6849 14684 6559 14412 6559 14078 ct 6559 13744 6849 13472 7205 13472 ct +7561 13472 7851 13744 7851 14078 ct 7851 14412 7561 14684 7205 14684 ct pc +5427 13589 m 5427 14657 l ps +5374 13560 m 6242 13560 l ps +5374 13914 m 6090 13914 l ps +11776 13397 m 11776 14697 l ps +12625 13414 m 12625 14680 l ps +12577 14640 m 13493 14640 l ps +gr +gs +1000 1000 m 20589 1000 l 20589 26939 l 1000 26939 l 1000 1000 l eoclip newpath +gr +gr +0 25940 t +pom +count op_count sub {pop} repeat countdictstack dict_count sub {end} repeat b4_inc_state restore +%%PageTrailer +%%Trailer +%%EOF diff --git a/www/fossil2.gif b/www/fossil2.gif Binary files differnew file mode 100644 index 0000000..af6fe11 --- /dev/null +++ b/www/fossil2.gif diff --git a/www/fossil3.gif b/www/fossil3.gif Binary files differnew file mode 100644 index 0000000..bc372d9 --- /dev/null +++ b/www/fossil3.gif diff --git a/www/fossil_logo_small.gif b/www/fossil_logo_small.gif Binary files differnew file mode 100644 index 0000000..88cf7bf --- /dev/null +++ b/www/fossil_logo_small.gif diff --git a/www/fossil_logo_small2.gif b/www/fossil_logo_small2.gif Binary files differnew file mode 100644 index 0000000..c1d747f --- /dev/null +++ b/www/fossil_logo_small2.gif diff --git a/www/fossil_logo_small3.gif b/www/fossil_logo_small3.gif Binary files differnew file mode 100644 index 0000000..06da386 --- /dev/null +++ b/www/fossil_logo_small3.gif diff --git a/www/fossil_prompt.sh b/www/fossil_prompt.sh new file mode 100644 index 0000000..f99dabd --- /dev/null +++ b/www/fossil_prompt.sh @@ -0,0 +1,14 @@ +function set_prompt() { + case `fossil status -b` in + clean) + PS1="\[\e[1;32m\]\u@\h\[\e[0m\]:\[\e[1;36m\]\w\$\[\e[0m\] " + ;; + dirty) + PS1="\[\e[1;32m\]\u@\h\[\e[0m\]:\[\e[38;5;202m\]\w\$\[\e[0m\] " + ;; + *) + PS1="\[\e[1;32m\]\u@\h\[\e[0m\]:\w\$ " + ;; + esac +} +PROMPT_COMMAND=set_prompt diff --git a/www/fossil_prompt.wiki b/www/fossil_prompt.wiki new file mode 100644 index 0000000..5c60aef --- /dev/null +++ b/www/fossil_prompt.wiki @@ -0,0 +1,22 @@ +<title>Fossilized Bash Prompt</title> + +Dan Kennedy has contributed a +[./fossil_prompt.sh?mimetype=text/plain | bash script] +that manipulates the bash prompt to show the status of +the Fossil repository that the user is currently visiting. +The prompt shows the branch, version, and time stamp for the +current checkout, and the prompt changes colors from blue to +red when there are uncommitted changes. + +To try out this script, simply download it from the link above, then +type: + +<pre> +. fossil_prompt.sh +</pre> + +For a permanent installation, you can graft the code into your +<tt>.bashrc</tt> file in your home directory. + +The code is very simple (only 32 non-comment lines, as of this writing) +and hence easy to customize. diff --git a/www/gitusers.md b/www/gitusers.md new file mode 100644 index 0000000..94197b4 --- /dev/null +++ b/www/gitusers.md @@ -0,0 +1,1335 @@ +# Git to Fossil Translation Guide + +## Introduction + +Fossil shares many similarities with Git. In many cases, the +sub-commands are identical: [`fossil bisect`][fbis] does essentially the +same thing as [`git bisect`][gbis], for example. + +Yet, Fossil is not merely Git with a bunch of commands misspelled. If +that were the case, we could give you a two-column translation table +which would tell you [how to say things like “`git reset --hard HEAD`”](#reset) in +this funny ol’ Fossil dialect of Git and be done. The purpose of this +document is to cover all the cases where there is no simple 1:1 mapping, +usually because of intentional design differences in Fossil that prevent +it from working exactly like Git. We choose to explain these differences +since to understand the conversion, you need to know why each difference +exists. + +We focus on practical command examples here, leaving discussions of the +philosophical underpinnings that drive these command differences to [another +document][fvg]. The [case studies](#cs1) do get a bit philosophical, but +it is with the aim of illustrating how these Fossil design differences +cause Fossil to behave materially differently from Git in everyday +operation. + +We present this from the perspective of Git users moving to Fossil, but +it is also possible to read this document as a Fossil user who speaks +only pidgin Git, who may often have questions of the form, “Now how do I +do X in Git again?” + +This document’s authors are intimately familiar with Fossil, so it is +difficult for us to anticipate the perspective of people who are +intimately familiar with Git. If you have a lot of prior Git +experience, we welcome your contributions and questions on the [Fossil +Forum][ffor]. + +While we do try to explain Fossil-specific terminology inline here +as-needed, you may find it helpful to skim [the Fossil glossary][gloss]. +It will give you another take on our definitions here, and it may help +you to understand some of the other Fossil docs better. + +[fbis]: /help/bisect +[gbis]: https://git-scm.com/docs/git-bisect +[ffor]: https://fossil-scm.org/forum +[fvg]: ./fossil-v-git.wiki + + +<a id="mwd"></a> +## Repositories and Checkouts Are Distinct + +A repository and a check-out are distinct in Fossil, allowing them to be +stored in separate directory trees, whereas the two are commingled by +default with Git, with the repository stored in a `.git` subdirectory +underneath your working directory. This difference shows up in several +separate places when it comes to moving from Git to Fossil. + + + +#### <a id="cwork" name="scw"></a> Checkout Workflows + +A Fossil repository is an SQLite database storing the entire history of a +project. It is not normally stored inside the working tree. +A Fossil working tree — [also called a check-out](./glossary.md#check-out) — is a directory +that contains a snapshot of your project that you are currently working +on, extracted for you from the repository database file by the `fossil` +program. + +There are ways to +[emulate the Fossil working style in Git](#worktree), but because they’re not +designed into the core concept of the tool, Git tutorials usually +advocate a switch-in-place working mode instead, so that is how most +users end up working with Git. Contrast [Fossil’s check-out workflow +document][ckwf] to see the practical differences. + +There is one Git-specific detail we wish to add beyond what that +document already covers. This command: + + git checkout some-branch + +…is best given as: + + fossil update some-branch + +…in Fossil. There is a [`fossil checkout`][co] command, but it has +[several differences](./co-vs-up.md) that make it less broadly useful +than [`fossil update`][up] in everyday operation, so we recommend that +Git users moving to Fossil develop a habit of typing `fossil up` rather +than `fossil checkout`. That said, one of those differences does match +up with Git users’ expectations: `fossil checkout` doesn’t pull changes +from the remote repository into the local clone as `fossil update` does. +We think this is less broadly useful, but that’s the subject of the next +section. + +[ckwf]: ./ckout-workflows.md +[co]: /help/checkout + + +#### <a id="pullup"></a> Update vs Pull + +The closest equivalent to [`git pull`][gpull] is not +[`fossil pull`][fpull], but in fact [`fossil up`][up]. + +This is because +Fossil tends to follow the CVS command design: `cvs up` pulls +changes from the central CVS repository and merges them into the local +working directory, so that’s what `fossil up` does, too. (This design +choice also tends to make Fossil feel comfortable to Subversion +expatriates.) + +The `fossil pull` command is simply the reverse of +`fossil push`, so that `fossil sync` [is functionally equivalent +to](./sync.wiki#sync): + + fossil push ; fossil pull + +There is no implicit “and update the local working directory” step in Fossil’s +push, pull, or sync commands, as there is with `git pull`. + +Someone coming from the Git perspective may perceive that `fossil up` +has two purposes: + +* Without the optional `VERSION` argument, it updates the working + check-out to the tip of the current branch, as `git pull` does. + +* Given a `VERSION` argument, it updates to the named version. If that’s the + name of a branch, it updates to the *tip* of that branch, as + `git checkout BRANCH` does. + +In fact, these are the same operation, so they’re the same command in +Fossil. The first form simply allows the `VERSION` to be implicit: the +tip of the current branch. + +We think this is a more sensible command design than `git pull` vs +`git checkout`. ([…vs `git checkout` vs `git checkout`!][gcokoan]) + +[fpull]: /help/pull +[gpull]: https://git-scm.com/docs/git-pull +[gcokoan]: https://stevelosh.com/blog/2013/04/git-koans/#s2-one-thing-well + + +#### <a id="close" name="dotfile"></a> Closing a Check-Out + +The [`fossil close`][close] command dissociates a check-out directory from the +Fossil repository database, _nondestructively_ inverting [`fossil open`][open]. +(Contrast Git’s [closest alternative](#worktree), `git worktree remove`, which *is* +destructive!) This Fossil command does not +remove the managed files, and unless you give the `--force` +option, it won’t let you close the check-out with uncommitted changes to +those managed files. + +The `close` command also refuses to run without `--force` when you have +certain other precious per-checkout data that Fossil stores in the +`.fslckout` file at the root of a check-out directory. This is an SQLite +database that keeps track of local state such as what version you have +checked out, the contents of the [stash] for that working directory, the +[undo] buffers, per-checkout [settings][set], and so forth. The stash +and undo buffers are considered precious uncommitted changes, +so you have to force Fossil to discard these as part of closing the +check-out. + +Thus, `.fslckout` is not the same thing as `.git`! + +In native Windows builds of Fossil — that is, excluding Cygwin and WSL +builds, which follow POSIX conventions — this file is called `_FOSSIL_` +instead to get around the historical 3-character extension limit with +certain legacy filesystems. + +Closing a check-out directory is a rare operation. One use case +is that you’re about to delete the directory, so you want Fossil to forget about it +for the purposes of commands like [`fossil all`][all]. Even that isn’t +necessary, because Fossil will detect that this has happened and forget +the working directory for you. + +[all]: /help/all + + +#### <a id="worktree"></a> Git Worktrees + +There are at least three different ways to get [Fossil-style multiple +check-out directories][mcw] with Git. + +The old way is to simply symlink the `.git` directory between working +trees: + + mkdir ../foo-branch + ln -s ../actual-clone-dir/.git . + git checkout foo-branch + +The symlink trick has a number of problems, the largest being that +symlinks weren’t available on Windows until Vista, and until the Windows +10 Creators Update was released in spring of 2017, you had to be an +Administrator to use the feature besides. ([Source][wsyml]) Git 2.5 solved +this problem back when Windows XP was Microsoft’s current offering +by adding the `git-worktree` command: + + git worktree add ../foo-branch foo-branch + cd ../foo-branch + +That is approximately equivalent to this in Fossil: + + mkdir ../foo-branch + cd ../foo-branch + fossil open /path/to/repo.fossil foo-branch + +The Fossil alternative is wordier, but since this tends to be one-time setup, +not something you do everyday, the overhead is insignificant. This author keeps a “scratch” check-out +for cases where it’s inappropriate to reuse the “trunk” check-out, +isolating all of my expedient switch-in-place actions to that one +working directory. Since the other peer check-outs track long-lived +branches, and that set rarely changes once a development machine is set +up, I rarely pay the cost of these wordier commands. + +That then leads us to the closest equivalent in Git to [closing a Fossil +check-out](#close): + + git worktree remove . + +Note, however, that unlike `fossil close`, once the Git command +determines that there are no uncommitted changes, it blows away all of +the checked-out files! Fossil’s alternative is shorter, easier to +remember, and safer. + +There’s another way to get Fossil-like separate worktrees in Git: + + git clone --separate-git-dir repo.git https://example.com/repo + +This allows you to have your Git repository directory entirely separate +from your working tree, with `.git` in the check-out directory being a +file that points to `../repo.git`, in this example. + +[mcw]: ./ckout-workflows.md#mcw +[wsyml]: https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/ + + +#### <a id="iip"></a> Init in Place + +To illustrate the differences that Fossil’s separation of repository +from working directory creates in practice, consider this common Git “init in place” +method for creating a new repository from an existing tree of files, +perhaps because you are placing that project under version control for +the first time: + + cd long-established-project + git init + git add * + git commit -m "Initial commit of project." + +The closest equivalent in Fossil is: + + cd long-established-project + fossil init .fsl + fossil open --force .fsl + fossil add * + fossil ci -m "Initial commit of project." + +Note that unlike in Git, you can abbreviate the “`commit`” command in +Fossil as “`ci`” for compatibility with CVS, Subversion, etc. + +This creates a `.fsl` repo DB at the root of the project check-out to +emulate the `.git` repo dir. We have to use the `--force` flag on +opening the new repo because Fossil expects you to open a repo into an +empty directory in order to avoid spamming the contents of a repo over +an existing directory full of files. Here, we know the directory +contains files that will soon belong in the repository, though, so we +override this check. From then on, Fossil works like Git, for the +purposes of this example. + +We’ve drawn this example to create a tight parallel between Fossil and +Git, not to commend this `.fsl`-at-project-root trick to you. A better +choice would be `~/museum/home/long-established-project.fossil`, if +you’re following [the directory scheme exemplified in the glossary](./glossary.md#repository). That said, it +does emphasize an earlier point: Fossil doesn’t care where you put the +repo DB file or what you name it. + + +[clone]: /help/clone +[close]: /help/close +[gloss]: ./glossary.md +[open]: /help/open +[set]: /help/setting +[server]: /help/server +[stash]: /help/stash +[undo]: /help/undo + + +## <a id="log"></a> Fossil’s Timeline Is the “Log” + +Git users often need to use the `git log` command to dig linearly through +commit histories due to its [weak data model][wdm], giving [O(n) +performance][ocomp]. + +Fossil parses a huge amount of information out of commits that allow it +to produce its [timeline CLI][tlc] and [its `/timeline` web view][tlw] +using indexed SQL lookups, which generally have the info you would have +to manually extract from `git log`, produced much more quickly than Git +can, all else being equal: operations over [SQLite’s B-tree data structures][btree] +generally run in O(log n) time, faster than O(n) for equal *n* when the +constants are equal. + +Yet the constants are *not* equal because Fossil +reads from a single disk file rather than visit potentially many +files in sequence as Git must, so the OS’s buffer cache can result in +[still better performance][35pct]. + +Unlike Git’s log, Fossil’s timeline shows info across all branches by +default, a feature for maintaining better situational awareness. +It is possible to restrict the timeline to a single branch using `fossil timeline -b`. +Similarly, to restrict the timeline using the web UI equivalent, +click the name of a branch on the `/timeline` or `/brlist` page. (Or +manually, by adding the `r=` query parameter.) Note that even in this +case, the Fossil timeline still shows other branches where they interact +with the one you’ve referenced in this way; again, better situational +awareness. + + +#### <a id="emu-log"></a> Emulating `git log` + +If you truly need a backwards-in-time-only view of history in Fossil to +emulate `git log`, this is as close as you can currently come: + + fossil timeline parents current + +Again, though, this isn’t restricted to a single branch, as `git log` +is. + +Another useful rough equivalent is: + + git log --raw + fossil time -v + +This shows what changed in each version, though Fossil’s view is more a +summary than a list of raw changes. To dig deeper into single commits, +you can use Fossil’s [`info` command][infoc] or its [`/info` view][infow]. + +Inversely, you may more exactly emulate the default `fossil timeline` +output with `git log --name-status`. + + +#### <a id="whatchanged"></a> What Changed? + +A related — though deprecated — command is `git whatchanged`, which gives results similar to +`git log --raw`, so we cover it here. + +Though there is no `fossil whatchanged` command, the same sort of +information is available. For example, to pull the current changes from +the remote repository and then inspect them before updating the local +working directory, you might say this in Git: + + git fetch + git whatchanged ..@{u} + +…which you can approximate in Fossil as: + + fossil pull + fossil up -n + fossil diff --from tip + +To invert the `diff` to show a more natural patch, the command needs to +be a bit more complicated, since you can’t currently give `--to` +without `--from`. + + fossil diff --from current --to tip + +Rather than use the “dry run” form of [the `update` command][up], you can +say: + + fossil timeline after current + +…or if you want to restrict the output to the current branch: + + fossil timeline descendants current + + +#### <a id="ckin-names"></a> Symbolic Check-In Names + +Note the use of [human-readable symbolic version names][scin] in Fossil +rather than [Git’s cryptic notations][gcn]. + +For a more dramatic example of this, let us ask Git, “What changed since the +beginning of last month?” being October 2020 as I write this: + + git log master@{2020-10-01}..HEAD + +That’s rather obscure! Fossil answers the same question with a simpler +command: + + fossil timeline after 2020-10-01 + +You may need to add `-n 0` to bypass the default output limit of +`fossil timeline`, 20 entries. Without that, this command reads +almost like English. + +Some Git users like to write commands like the above so: + + git log @{2020-10-01}..@ + +Is that better? “@” now means two different things: an at-time reference +and a shortcut for `HEAD`! + +If you are one of those that like short commands, Fossil’s method is +less cryptic: it lets you shorten words in most cases up to the point +that they become ambiguous. For example, you may abbreviate the last +`fossil` command in the prior section: + + fossil tim d c + +…beyond which the `timeline` command becomes ambiguous with `ticket`. + +Some Fossil users employ shell aliases, symlinks, or scripts to shorten +the command still further: + + alias f=fossil + f tim d c + +Granted, that’s rather obscure, but you you can also choose something +intermediate like “`f time desc curr`”, which is reasonably clear. + +[35pct]: https://www.sqlite.org/fasterthanfs.html +[btree]: https://sqlite.org/btreemodule.html +[gcn]: https://git-scm.com/docs/gitrevisions +[infoc]: /help/info +[infow]: /help/www/info +[ocomp]: https://www.bigocheatsheet.com/ +[tlc]: /help/timeline +[tlw]: /help/www/timeline +[up]: /help/update +[wdm]: ./fossil-v-git.wiki#durable + + +## <a id="dhead"></a> Detached HEAD State + +The SQL indexes in Fossil which we brought up above have a useful +side benefit: you cannot have a [detached HEAD state][gdh] in Fossil, +the source of untold pain and data loss in Git. It simply cannot be done +in Fossil, because the indexes always let us find our way back into the +hash tree. + + +## <a id="slcom"></a> Summary Line Convention in Commit Comments + +The Git convention of a [length-limited summary line][lsl] at the start +of commit comments is not enforced or obeyed by default in Fossil. +However, there is a setting under Admin → Timeline → “Truncate comment +at first blank line (Git-style)” to change this for `/timeline` +displays. Alternately, you could enable the “Allow block-markup in +timeline” setting under Admin → Timeline, then apply [local skin +customizations][cskin] to put that first comment in bold or whatever +suits. + +Because this isn’t a typical Fossil convention, you’re likely to find +other odd differences between it and Git-based infrastructure. For +instance, Vim doesn’t ship with syntax support for Fossil commit +messages if you set `EDITOR=vim` in your shell environment, so you won’t +get over-limit highlighting for first-line text beyond the 50th +character and such, because it doesn’t recognize Fossil commit messages +and apply similar rules as to Git commit messages. + +[cskin]: ./customskin.md +[lsl]: https://chris.beams.io/posts/git-commit/#limit-50 + + +<a id="autocommit"></a> +## Fossil Never Auto-Commits + +There are several features in Git besides its `commit` command that +produce a new commit to the repository, and by default, they do it +without prompting or even asking for a commit message. These include +Git’s [`rebase`](#rebase), `merge`, and [`cherrypick`](#cpickrev) +commands, plus the [commit splitting](#comsplit) sub-feature +“`git commit -p`”. + +Fossil never does this, on firm philosophical grounds: we wish to be +able to test that each potentially repository-changing command does not +break anything _before_ freezing it immutably into the [Merkle +tree](./blockchain.md). Where Fossil has equivalent commands, they +modify the checkout tree alone, requiring a separate `commit` command +afterward, withheld until the user has satisfied themselves that the +command’s result is correct. + +We believe this is the main reason Git lacks an [autosync](#autosync) +feature: making push a manual step gives the user a chance to rewrite +history after triggering one of these autocommits locally, should the +automatic commit fail to work out as expected. Fossil chooses the +inverse path under the philosophy that commits are *commitments,* not +something you’re allowed to go back and rewrite later. + +This is also why there is no automatic commit message writing feature in +Fossil, as in these autocommit-triggering Git commands. The user is +meant to write the commit message by hand after they are sure it’s +correct, in clear-headed retrospective fashion. Having the tool do it +prospectively before one can test the result is simply backwards. + + +<a id="staging"></a> +## There Is No Staging Area + +Fossil omits the "Git index" or "staging area" concept. When you +type "`fossil commit`" _all_ changes in your check-out are committed, +by default. There is no need for the "-a" option as with Git. + +If you only want to commit _some_ of the changes, list the names +of the files or directories you want to commit as arguments, like this: + + fossil commit src/feature.c doc/feature.md examples/feature + +Note that the last element is a directory name, meaning “any changed +file under the `examples/feature` directory.” + + +<a id="comsplit"></a> +## Commit Splitting + +[Git’s commit splitting features][gcspl] rely on +other features of Git that Fossil purposefully lacks, as covered in the +prior two sections: [autocommit](#autocommit) and [the staging +area](#staging). + +While there is no direct Fossil equivalent for +`git add -p`, `git commit -p`, or `git rebase -i`, you can get the same +effect by converting an uncommitted change set to a patch and then +running it through [Patchouli]. + +Rather than use `fossil diff -i` to produce such a patch, a safer and +more idiomatic method would be: + + fossil stash save -m 'my big ball-o-hackage' + fossil stash diff > my-changes.patch + +That stores your changes in the stash, then lets you operate on a copy +of that patch. Each time you re-run the second command, it will take the +current state of the working directory into account to produce a +potentially different patch, likely smaller because it leaves out patch +hunks already applied. + +In this way, the combination of working tree and stash replaces the need +for Git’s index feature. + +We believe we know how to do commit splitting in a way compatible with +the Fossil philosophy, without following Git’s ill-considered design +leads. It amounts to automating the above process through an interactive +variant of [`fossil stash apply`][stash], as currently prototyped in the +third-party tool [`fnc`][fnc] and [its interactive `stash` +command][fncsta]. We merely await someone’s [contribution][ctrb] of this +feature into Fossil proper. + +[ctrb]: https://fossil-scm.org/fossil/doc/trunk/www/contribute.wiki +[fnc]: https://fnc.bsdbox.org/ +[fncsta]: https://fnc.bsdbox.org/uv/doc/fnc.1.html#stash +[gcspl]: https://git-scm.com/docs/git-rebase#_splitting_commits +[Patchouli]: https://pypi.org/project/patchouli/ + + +<a id="bneed"></a> +## Create Branches at Point of Need, Rather Than Ahead of Need + +Fossil prefers that you create new branches as part of the first commit +on that branch: + + fossil commit --branch my-branch + +If that commit is successful, your local check-out directory is then +switched to the tip of that branch, so subsequent commits don’t need the +“`--branch`” option. You simply say `fossil commit` again to continue +adding commits to the tip of that branch. + +To switch back to the parent branch, say something like: + + fossil update trunk + +(This is approximately equivalent to `git checkout master`.) + +Fossil does also support the Git style, creating the branch ahead of +need: + + fossil branch new my-branch + fossil up my-branch + ...work on first commit... + fossil commit + +This is more verbose, giving the same overall effect though the initial +actions are inverted: create a new branch for the first commit, switch +the check-out directory to that branch, and make that first commit. As +above, subsequent commits are descendants of that initial branch commit. +We think you’ll agree that creating a branch as part of the initial +commit is simpler. + +Fossil also allows you to move a check-in to a different branch +*after* you commit it, using the "`fossil amend`" command. +For example: + + fossil amend current --branch my-branch + +This works by inserting a tag into the repository that causes the web UI +to relabel commits from that point forward with the new name. Like Git, +Fossil’s fundamental data structure is the interlinked DAG of commit +hashes; branch names are supplemental data for making it easier for the +humans to understand this DAG, so this command does not change the core +history of the project, only annotate it for better display to the +humans. + +(The version string “current” is one of the [special check-in names][scin] in Fossil. See +that document for the many other names you can give to “`amend`”, or +indeed to any other Fossil command documented to accept a `VERSION` or +`NAME` string.) + +[scin]: ./checkin_names.wiki + + +<a id="syncall"></a> +## Sync Is All-or-Nothing + +Fossil does not support the concept of syncing, pushing, or pulling +individual branches. When you sync/push/pull in Fossil, it +processes all artifacts in its hash tree: +branches, tags, wiki articles, tickets, forum posts, technotes… +This is [not quite “everything,” full stop][bu], but it’s close. +[Fossil is an AP-mode system][capt], which in this case means it works +*very hard* to ensure that all repos are as close to identical as it can +make them under this eventually-consistent design philosophy. + +Branch *names* sync automatically in Fossil, not just the +content of those branches. That means this common Git command: + + git push origin master + +…is simply this in Fossil: + + fossil push + +Fossil doesn’t need to be told what to push or where to push it: it just +keeps using the same remote server URL you gave it last +until you [tell it to do something different][rem]. It pushes all +branches, not just one named local branch. + +[capt]: ./cap-theorem.md +[rem]: /help/remote + + +<a id="autosync"></a> +## Autosync + +Fossil’s [autosync][wflow] feature, normally enabled, has no +equivalent in Git. If you want Fossil to behave like Git, you can turn +it off: + + fossil set autosync 0 + +Let’s say that you have a typical server-and-workstations model with two +working clones on different machines, that you have disabled autosync, +and that this common sequence then occurs: + +1. Alice commits to her local clone and *separately* pushes the change + up to Condor — their central server — in typical Git fashion. +2. Bob does the same. +3. Alice brings Bob’s changes down from Condor with “`fossil pull`,” sees + what he did to their shared working branch, and becomes most wrathful. + (Tsk, tsk.) + +We’ll get to what you do about this situation [below](#reset), but for +now let us focus on the fact that disabling autosync makes it easier for +[forks] to occur in the development history. If all three machines had +been online and syncing at the time the sequence above began, Bob would +have been warned in step 2 that committing to the central repo would +create a fork and would be invited to fix it before committing. +Likewise, it would allow Fossil to warn Alice about the new +tip-of-branch commit the next time she triggers an implicit autosync at +step 3, giving her a chance to bring Bob’s changes down in a +non-conflicting manner, allowing work to proceed with minimal fuss. + +Fossil, being a distributed version control system, cannot guarantee +that sequence of events. Because it allows Alice’s work to proceed +asynchronously, it gives her the chance to create *another* inadvertent +fork before she can trigger an autosync. This is not a serious problem; +Fossil resolves it the same way as with Bob, by inviting her to fix this +second fork in the same manner as it did with Bob. It gets both parties +back onto a single track as expeditiously as possible by moving the +synchronization point out of the expensive human-time workflow and into +the software system, where it’s cheapest to resolve. + +Autosync provides Fossil with most of the advantages of a centralized +version control system while retaining the advantages of distributed +version control: + +1. Your work stays synced up with your coworkers’ efforts as long as your + machine can connect to the remote repository. At need, you can go + off-network and continue work atop the last version you synced with + the remote. + +2. You get implicit off-machine backup of your commits. Unlike + centralized version control, though, you can still work while + disconnected; your changes will sync up with the remote once you get + back online. + +3. Because there is [little distinction][bu] between the clones in the Fossil + model — unlike in Git, where clones often quickly diverge from each + other, quite possibly on purpose — the backup advantage applies in inverse + as well: if the remote server falls over dead, one of those with a + clone of that repository can stand it back up, and everyone can get + back to work simply by re-pointing their local repo at the new + remote. If the failed remote comes back later, it can sync with the + new central version, then perhaps take over as the primary source of + truth once again. + +[bu]: ./backup.md +[forks]: ./branching.wiki +[setup]: ./caps/admin-v-setup.md#apsu +[wflow]: ./concepts.wiki#workflow + + +<a id="reset"></a> +## Resetting the Repository + +Extending from [the prior item](#syncall), you may correctly infer that +“[delete the project and download a fresh copy][x1597]” has no part in +the Fossil Way. Ideally, you should never find yourself forced into +desperate measures like this:(^Parsing the output of `fossil status` is +usually a mistake since it relies on a potentially unstable interface. +We make no guarantee that there will always be a line beginning with +“`repo`” and that it will be separated from the repository’s file name +by a colon. The simplified example above is also liable to become +confused by whitespace in file names.) + + +``` +$ repo=$(fossil status | grep ^repo | cut -f2 -d:) +$ url=$(fossil remote) +$ fossil close # Stop here and think if it warns you! +$ mv $repo ${repo}.old +$ fossil clone $url $repo +$ fossil open --force $repo +``` + +What, then, should you as a Git transplant do instead when you find +yourself reaching for “`git reset`”? + +Since the correct answer to that depends on why you think it’s a good +solution to your immediate problem, we’ll take our motivating scenario +from the problem setup above, where we discussed Fossil’s [autosync] +feature. Let us further say Alice’s pique results from a belief that +Bob’s commit is objectively wrong-headed and should be expunged +henceforth. Since Fossil goes out of its way to ensure that [commits are +durable][wdm], it should be no further surprise that there is no easier +method to reset Bob’s clone in favor of Alice’s than the above sequence +in Fossil’s command set. Except in extreme situations, we believe that +sort of thing is unnecessary. + +Instead, Bob can say something like this: + +``` +fossil amend --branch MISTAKE --hide --close -m "mea culpa" tip +fossil up trunk +fossil push +``` + +Unlike in Git, the “`amend`” command doesn’t modify prior committed +artifacts. Bob’s first command doesn’t delete anything, merely tells +Fossil to hide his mistake from timeline views by inserting a few new +records into the local repository to change how the client interprets +the data it finds there henceforth.(^One to change the tag marking this +commit’s branch name to “`MISTAKE`,” one to mark that branch as hidden, +and one to close it to further commits.) + +Bob’s second command switches his working directory back to the prior +commit on that branch. We’re presuming it was “`trunk`” for the sake of +the example, but it works for any parent branch name. The command works +because the name “`trunk`” now means something different to Fossil by +virtue of the first command. + +Bob’s third command pushes the changes up to the central machine to +inform everyone else of his amendment.(^Amendments don’t autosync in +Fossil because they don’t change any previous commits, allowing the +other clones to continue working safely with their existing commit +hashes.) + +In this scheme, Alice then needs to say “`fossil update trunk`” in order +to return her check-out’s parent commit to the previous version lest her +next attempted commit land atop this mistake branch. The fact that Bob +marked the branch as closed will prevent that from going through, cluing +Alice into what she needs to do to remedy the situation, but that merely +shows why it’s a better workflow if Alice makes the amendment herself: + +``` +fossil amend --branch MISTAKE --hide --close \ + -m "shunt Bob’s erroneous commit off" tip +fossil up trunk +fossil push +``` + +Then she can fire off an email listing Bob’s assorted failings and go +about her work. This asynchronous workflow solves the problem without +requiring explicit coordination with Bob. When he gets his email, he can +then say “`fossil up trunk`” himself, which by default will trigger an +autosync, pulling down Alice’s amendments and getting him back onto her +development track. + +Remember that [branch names need not be unique](#btnames) in Fossil. You +are free to reuse this “`MISTAKE`” branch name as often as you need to. + +[autosync]: #autosync +[x1597]: https://xkcd.com/1597/ + + +<a id="trunk"></a> +## The Main Branch Is Called "`trunk`" + +In Fossil, the default name for the main branch +is "`trunk`". The "`trunk`" branch in Fossil corresponds to the +"`master`" branch in stock Git or to [the “`main`” branch in GitHub][mbgh]. + +Because the `fossil git export` command has to work with both stock Git +and with GitHub, Fossil uses Git’s traditional default rather than +GitHub’s new default: your Fossil repo’s “trunk” branch becomes “master” +when [mirroring to GitHub][mirgh] unless you give the `--mainbranch` +option. + +We do not know what happens on subsequent exports if you later rename +this branch on the GitHub side. + +[mbgh]: https://github.com/github/renaming +[mirgh]: ./mirrortogithub.md + + +<a id="unmanaged"></a> +## Status Does Not Show Unmanaged Files + +The "`fossil status`" command shows you what files in your check-out have +been edited and scheduled for adding or removing at the next commit. +But unlike "`git status`", the "`fossil status`" command does not warn +you about unmanaged files in your local check-out. There is a separate +"`fossil extras`" command for that. + + +<a id="rebase"></a> +## There Is No Rebase + +Fossil does not support rebase, [on purpose][3]. + +This is a deliberate design decision that the Fossil community has +thought about carefully and discussed many times, resulting in the +linked document. If you are fond of rebase, you should read it carefully +before expressing your views: it not only answers many of the common +arguments in favor of rebase known at the time the document’s first +draft was written, it’s been revised multiple times to address less +common objections as well. Chances are not good that you are going to +come up with a new objection that we haven’t already considered and +addressed there. + +There is only one sub-feature of `git rebase` that is philosophically +compatible with Fossil yet which currently has no functional equivalent. +We [covered this and the workaround for its lack](#comsplit) above. + +[3]: ./rebaseharm.md + + +<a id="cdiff"></a> +## Colorized Diffs + +When you run `git diff` on an ANSI X3.64 capable terminal, it uses color +to distinguish insertions, deletions, and replacements, but as of this +writing, `fossil diff` produces traditional uncolored [unified diff +format][udiff] output, suitable for producing a [patch file][pfile]. + +There are [many methods](./colordiff.md) for solving this. +Viewed this way, Fossil doesn’t lack colorized diffs, it simply has +*one* method where they *aren’t* colorized. + +[pfile]: https://en.wikipedia.org/wiki/Patch_(Unix) +[udiff]: https://en.wikipedia.org/wiki/Diff#Unified_format + + +## <a id="show"></a> Showing Information About Commits + +While there is no direct equivalent to Git’s “`show`” command, similar +functionality is present in Fossil under other commands: + + +#### <a id="patch"></a> Show a Patch for a Commit + + git show -p COMMIT_ID + +…gives much the same output as + + fossil diff --checkin COMMIT_ID + +…only without the patch email header. Git comes out of the [LKML] world, +where emailing a patch is a normal thing to do. Fossil is [designed for +cohesive teams][devorg] where drive-by patches are rarer. + +You can use any of [Fossil’s special check-in names][scin] in place of +the `COMMIT_ID` in this and later examples. Fossil docs usually say +“`VERSION`” or “`NAME`” where this is allowed, since the version string +or name might not refer to a commit ID, but instead to a forum post, a +wiki document, etc. For instance, the following command answers the question “What did +I just commit?” + + fossil diff --checkin tip + +…or equivalently using a different symbolic commit name: + + fossil diff --from prev + +[devorg]: ./fossil-v-git.wiki#devorg +[LKML]: https://lkml.org/ + + +#### <a id="cmsg"></a> Show a Specific Commit Message + + git show -s COMMIT_ID + + +…is + + fossil time -n 1 COMMIT_ID + +…or with a shorter, more obvious command, though with more verbose output: + + fossil info COMMIT_ID + +The `fossil info` command isn’t otherwise a good equivalent to +`git show`; it just overlaps its functionality in some areas. Much of +what’s missing is present in the corresponding [`/info` web +view][infow], though. + + +#### <a id="dstat"></a> Diff Statistics + +Fossil’s closest internal equivalent to commands like +`git show --stat` is: + + fossil diff -i --from 2020-04-01 --numstat + +The `--numstat` output is a bit cryptic, so we recommend delegating +this task to [the widely-available `diffstat` tool][dst], which gives +a histogram in its default output mode rather than bare integers: + + fossil diff -i -v --from 2020-04-01 | diffstat + +We gave the `-i` flag in both cases to force Fossil to use its internal +diff implementation, bypassing [your local `diff-command` setting][dcset] +since the `--numstat` option has no effect when you have an external diff +command set. + +If you leave off the `-v` flag in the second example, the `diffstat` +output won’t include info about any newly-added files. + +[dcset]: https://fossil-scm.org/home/help/diff-command +[dst]: https://invisible-island.net/diffstat/diffstat.html + + +<a id="btnames"></a> +## Branch and Tag Names + +Fossil has no special restrictions on the names of tags and branches, +though you might want to keep [Git's tag and branch name restrictions][gcrf] +in mind if you plan on [mirroring your Fossil repository to GitHub][mirgh]. + +Fossil does not require tag and branch names to be unique. It is +common, for example, to put a “`release`” tag on every release for a +Fossil-hosted project. This does not create a conflict in Fossil, since +Fossil resolves the ambiguity in a predictable way: the newest match +wins. Therefore, “`fossil up release`” always gets you the current +release in a project that uses this tagging convention. + +[The `fossil git export` command][fge] squashes repeated tags down to a +single instance to avoid confusing Git, exporting only the newest tag, +emulating Fossil’s own ambiguity resolution rule as best it can within +Git’s limitations. + +[fge]: /help/git +[gcrf]: https://git-scm.com/docs/git-check-ref-format + + + + +<a id="cpickrev"></a> +## Cherry-Picking and Reverting Commits + +Git’s separate "`git cherry-pick`" and “`git revert`” commands are +options to the [`fossil merge` command][merge]: `--cherrypick` and +`--backout`, respectively. We view this as sensible, since these are +both merge operations, and the two actions differ only in direction. + +Unlike in Git, the Fossil file format remembers cherrypicks and backouts +and can later show them as dashed lines on the graphical timeline. + +[merge]: /help/merge + + + +<a id="mvrm"></a> +## File Moves and Renames Are Soft by Default + +The "[`fossil mv`][mv]" and "[`fossil rm`][rm]" commands work like they +do in CVS in that they schedule the changes for the next commit by +default: they do not actually rename or delete the files in your +check-out. + +If you don’t like that default, you can change it globally: + + fossil setting --global mv-rm-files 1 + +Now these commands behave like in Git in any Fossil repository where +this setting hasn’t been overridden locally. + +If you want to keep Fossil’s soft `mv/rm` behavior most of the time, you +can cast it away on a per-command basis: + + fossil mv --hard old-name new-name + +[mv]: /help/mv +[rm]: /help/rm + + +---- + + +## <a id="cvdate" name="cs1"></a> Case Study 1: Checking Out a Version by Date + +Let’s get into something a bit more complicated: a case study showing +how the concepts lined out above cause Fossil to materially differ in +day-to-day operation from Git. + +Why would you want to check out a version of a project by date? Perhaps +your customer gave you a vague bug report referencing only a +date rather than a version. Or, you may be poking semi-randomly through +history to find a “good” version to anchor the start point of a +[`fossil bisect`][fbis] operation. + +My search engine’s first result for “git checkout by date” is [this +highly-upvoted accepted Stack Overflow answer][gcod]. The first command +it gives is based on Git’s [`rev-parse` feature][grp]: + + git checkout master@{2020-03-17} + +There are a number of weaknesses in this command. From least to most +critical: + +1. It’s a bit cryptic. Leave off the refname or punctuation, and it + means something else. You cannot simplify the cryptic incantation in + the typical use case. + +2. A date string in Git without a time will be interpreted as + “[at the local wall clock time on the given date][gapxd],” so the + command means something different from one second to the next! This + can be a serious problem if there are multiple commits on that date, because + the command will give different results depending on the time of + day you run it. + +3. It gives misleading output if there is no close match for the date + in the local [reflog]. It starts out empty after a fresh clone, and + while it does build up as you use that clone, Git [automatically + prunes][gle] the reflog to 90 days of history by default. This means + the command above can give different results from one machine to the + next, or even from one day to the next on the same clone. + + The command won’t fail outright if the reflog can’t resolve the + given date: it simply gives the closest commit it can come up with, + even if it’s months or years out from your target! Sometimes it + gives a warning about the reflog not going back far enough to give a + useful result, and sometimes it doesn’t. If you’re on a fresh clone, + you are likely to get the “tip” commit’s revision ID no matter what + date value you give. + + Git tries its best, but because it’s working from a purgeable and + possibly-stale local cache, you cannot trust its results. + +Consequently, we cannot recommend this command at all. It’s unreliable even in the +best case. + +That same Stack Overflow answer therefore goes on to recommend an +entirely different command: + + git checkout $(git rev-list -n 1 --first-parent --before="2020-03-17" master) + +We believe you get such answers to Git help requests in part +because of its lack of an always-up-to-date [index into its log](#log) and in +part because of its “small tools loosely joined” design philosophy. This +sort of command is therefore composed piece by piece: + +<p style="text-align:center">◆ ◆ ◆</p> + +“Oh, I know, I’ll search the rev-list, which outputs commit IDs by +parsing the log backwards from `HEAD`! Easy!” + + git rev-list --before=2020-03-17 + +“Blast! Forgot the commit ID!” + + git rev-list --before=2020-03-17 master + +“Double blast! It just spammed my terminal with revision IDs! I need to +limit it to the single closest match: + + git rev-list -n 1 --before=2020-03-17 master + +“Okay, it gives me a single revision ID now, but is it what I’m after? +Let’s take a look…” + + git show $(git rev-list -n 1 --before=2020-03-17 master) + +“Oops, that’s giving me a merge commit, not what I want. +Off to search the web… Okay, it says I need to give either the +`--first-parent` or `--no-merges` flag to show only regular commits, +not merge-commits. Let’s try the first one:” + + git show $(git rev-list -n 1 --first-parent --before=2020-03-17 master) + +“Better. Let’s check it out:” + + git checkout $(git rev-list -n 1 --first-parent --before=2020-03-17 master) + +“Success, I guess?” + +<p style="text-align:center">◆ ◆ ◆</p> + +This vignette is meant to explain some of Git’s popularity: it rewards +the sort of people who enjoy puzzles, many of whom are software +developers and thus need a tool like Git. Too bad if you’re just a +normal user. + +And too bad if you’re a Windows user who doesn’t want to use [Git +Bash][gbash], since neither of the stock OS command shells have a +command interpolation feature needed to run that horrid command. + +This alternative command still has weakness #2 above: if you run the +second `git show` command above on [Git’s own repository][gitgh], your +results may vary because there were four non-merge commits to Git on the +17th of March, 2020. + +You may be asking with an exasperated huff, “What is your *point*, man?” +The point is that the equivalent in Fossil is simply: + + fossil up 2020-03-17 + +…which will *always* give the commit closest to midnight UTC on the 17th +of March, 2020, no matter whether you do it on a fresh clone or a stale +one. The answer won’t shift about from one clone to the next or from +one local time of day to the next. We owe this reliability and stability +to three Fossil design choices: + +* Parse timestamps from all commits on clone into a local commit index, + then maintain that index through subsequent commits and syncs. + +* Use an indexed SQL `ORDER BY` query to match timestamps to commit + IDs for a fast and consistent result. + +* Round incomplete timestamp strings up using [rules][frud] consistent across + computers and local time of day. + +[frud]: https://fossil-scm.org/home/file/src/name.c?ci=d2a59b03727bc3&ln=122-141 +[gbash]: https://appuals.com/what-is-git-bash/ +[gapxd]: https://github.com/git/git/blob/7f7ebe054a/date.c#L1298-L1300 +[gcod]: https://stackoverflow.com/a/6990682/142454 +[gdh]: https://www.git-tower.com/learn/git/faq/detached-head-when-checkout-commit/ +[gitgh]: https://github.com/git/git/ +[gle]: https://git-scm.com/docs/git-reflog#_options_for_expire +[gmc]: https://github.com/git/git/commit/67b0a24910fbb23c8f5e7a2c61c339818bc68296 +[grp]: https://git-scm.com/docs/git-rev-parse +[reflog]: https://git-scm.com/docs/git-reflog + +---- + +## <a id="morigin" name="cs2"></a> Case Study 2: Multiple "origin" Servers + +Now let us consider a common use case at the time of this writing — during the +COVID-19 pandemic — where you’re working from home a lot, going into the +office one part-day a week only to do things that have to be done +on-site at the office. Let us also say you have no remote +access back into the work LAN, such as because your site IT is paranoid +about security. You may still want off-machine backups of your commits +while working from home, +so you need the ability to quickly switch between the “home” and +“work” remote repositories, with your laptop acting as a kind of +[sneakernet][sn] link between the big development server at the office +and your family’s home NAS. + +#### Git Method + +We first need to clone the work repo down to our laptop, so we can work on it +at home: + + git clone https://dev-server.example.com/repo + cd repo + git remote rename origin work + +The last command is optional, strictly speaking. We could continue to +use Git’s default name for the work repo’s origin — sensibly enough +called “`origin`” — but it makes later commands harder to understand, so +we rename it here. This will also make the parallel with Fossil easier +to draw. + +The first time we go home after this, we have to reverse-clone the work +repo up to the NAS: + + ssh my-nas.local 'git init --bare /SHARES/dayjob/repo.git' + git push --all ssh://my-nas.local//SHARES/dayjob/repo.git + +Realize that this is carefully optimized down to these two long +commands. In practice, we’d expect a user typing these commands by hand from memory +to need to give four or more commands here instead. +Packing the “`git init`” call into the “`ssh`” call is something more +often done in scripts and documentation examples than done interactively, +which then necessitates a third command before the push, “`exit`”. +There’s also a good chance that you’ll forget the need for the `--bare` +option here to avoid a fatal complaint from Git that the laptop can’t +push into a non-empty repo. If you fall into this trap, among the many +that Git lays for newbies, you have to nuke the incorrectly initted +repo, search the web or Git man pages to find out about `--bare`, and try again. + +Having navigated that little minefield, +we can tell Git that there is a second origin, a “home” repo in +addition to the named “work” repo we set up earlier: + + git remote add home ssh://my-nas.local//SHARES/dayjob/repo.git + git config master.remote home + +We don’t have to push or pull because the remote repo is a complete +clone of the repo on the laptop at this point, so we can just get to +work now, committing along the way to get our work safely off-machine +and onto our home NAS, like so: + + git add + git commit + git push + +We didn’t need to give a remote name on the push because we told it the +new upstream is the home NAS earlier. + +Now Friday comes along, and one of your office-mates needs a feature +you’re working on. You agree to come into the office later that +afternoon to sync up via the dev server: + + git push work master # send your changes from home up + git pull work master # get your coworkers’ changes + +Alternately, we could add “`--set-upstream/-u work`” to the first +command if we were coming into work long enough to do several Git-based things, not just pop in and sync. +That would allow the second to be just “`git pull`”, but the cost is +that when returning home, you’d have to manually reset the upstream +again. + +This example also shows a consequence of that fact that +[Git doesn’t sync branch names](#syncall): you have to keep repeating +yourself like an obsequious supplicant: “Master, master.” Didn’t we +invent computers to serve humans, rather than the other way around? + + +#### Fossil Method + +Now we’re going to do the same thing using Fossil, with +the commands arranged in blocks corresponding to those above for comparison. + +We start the same way, cloning the work repo down to the laptop: + + fossil clone https://dev-server.example.com/repo + cd repo + fossil remote add work https://dev-server.example.com/repo + +We’ve chosen the new “`fossil clone URI`” syntax rather than separate +`clone` and `open` commands to make the parallel with Git clearer. [See +above](#mwd) for more on that topic. + +Our [`remote` command][rem] is longer than the Git equivalent because +Fossil currently has no short command +to rename an existing remote. Worse, unlike with Git, we can’t just keep +using the default remote name because Fossil uses that slot in its +configuration database to store the *current* remote name, so on +switching from work to home, the home URL will overwrite the work URL if +we don’t give it an explicit name first. + +Although the Fossil commands are longer, so far, keep it in perspective: +they’re one-time setup costs, +easily amortized to insignificance by the shorter day-to-day commands +below. + +On first beginning to work from home, we reverse-clone the Fossil repo +up to the NAS: + + rsync repo.fossil my-nas.local:/SHARES/dayjob/ + +Now we’re beginning to see the advantage of Fossil’s simpler model, +relative to the tricky “`git init && git push`” sequence above. +Fossil’s alternative is almost impossible to get +wrong: copy this to that. *Done.* + +We’re relying on the `rsync` feature that creates up to one level of +missing directory (here, `dayjob/`) on the remote. If you know in +advance that the remote directory already exists, you could use a +slightly shorter `scp` command instead. Even with the extra 2 characters +in the `rsync` form, it’s much shorter because a Fossil repository is a +single SQLite database file, not a tree containing a pile of assorted +files. Because of this, it works reliably without any of [the caveats +inherent in using `rsync` to clone a Git repo][grsync]. + +Now we set up the second remote, which is again simpler in the Fossil +case: + + fossil remote add home ssh://my-nas.local//SHARES/dayjob/repo.fossil + fossil remote home + +The first command is nearly identical to the Git version, but the second +is considerably simpler. And to be fair, you won’t find the +“`git config`” command above in all Git tutorials. The more common +alternative we found with web searches is even longer: +“`git push --set-upstream home master`”. + +Where Fossil really wins is in the next step, making the initial commit +from home: + + fossil ci + +It’s one short command for Fossil instead of three for Git — or two if +you abbreviate it as “`git commit -a && git push`” — because of Fossil’s +[autosync] feature and deliberate omission of a +[staging feature](#staging). + +The “Friday afternoon sync-up” case is simpler, too: + + fossil remote work + fossil sync + +Back at home, it’s simpler still: we may be able to do away with the second command, +saying just “`fossil remote home`” because the sync will happen as part +of the next commit, thanks once again to Fossil’s autosync feature. If +the working branch now has commits from other developers after syncing +with the central repository, though, you’ll want to say “`fossil up`” to +avoid creating an inadvertent fork in the branch. + +(Which is easy to fix if it occurs: `fossil merge` without arguments +means “merge open tips on the current branch.”) + +[grsync]: https://stackoverflow.com/q/1398018/142454 +[qs]: ./quickstart.wiki +[shwmd]: ./fossil-v-git.wiki#checkouts +[sn]: https://en.wikipedia.org/wiki/Sneakernet diff --git a/www/globs.md b/www/globs.md new file mode 100644 index 0000000..7f1a6a5 --- /dev/null +++ b/www/globs.md @@ -0,0 +1,618 @@ +# File Name Glob Patterns + +A [glob pattern][glob] is a text expression that matches one or more +file names using wildcards familiar to most users of a command line. +For example, `*` is a glob that matches any name at all, and +`Readme.txt` is a glob that matches exactly one file. For purposes of +Fossil's globs, a complete path name is just a string, +and the globs do not apply any special meaning to the directory part +of the name. Thus, the glob `*` matches any name, including any +directory prefix, and `*/*` matches a name with _one or more_ +directory components. + +A glob should not be confused with a [regular expression][regexp] (RE) +even though they use some of the same special characters for similar +purposes. [They are not fully compatible][greinc] pattern +matching languages. Fossil uses globs when matching file names with the +settings described in this document, not REs. + +[glob]: https://en.wikipedia.org/wiki/Glob_(programming) +[greinc]: https://unix.stackexchange.com/a/57958/138 +[regexp]: https://en.wikipedia.org/wiki/Regular_expression + +[Fossil’s `*-glob` settings](#settings) hold one or more patterns to cause Fossil to +give matching named files special treatment. Glob patterns are also +accepted in options to certain commands and as query parameters to +certain Fossil UI web pages. For consistency, settings such as +`empty-dirs` are parsed as a glob even though they aren’t then *applied* +as a glob since it allows [the same syntax rules](#syntax) to apply. + +Where Fossil also accepts globs in commands, this handling may interact +with your OS’s command shell or its C runtime system, because they may +have their own glob pattern handling. We will detail such interactions +below. + + +## <a id="syntax"></a>Syntax + +Where Fossil accepts glob patterns, it will usually accept a *list* of +individual patterns separated from the others by whitespace or commas. + +The parser allows whitespace and commas in a pattern by quoting _the +entire pattern_ with either single or double quotation marks. Internal +quotation marks are treated literally. Moreover, a pattern that begins +with a quote mark ends when the first instance of the same mark occurs, +_not_ at a whitespace or comma. Thus, this: + + "foo bar"qux + +…constitutes _two_ patterns rather than one with an embedded space, in +contravention of normal shell quoting rules. + +A list matches a file when any pattern in that list matches. + +A pattern must consume and +match the *entire* file name to succeed. Partial matches are failed matches. + +Most characters in a glob pattern consume a single character of the file +name and must match it exactly. For instance, “a” in a glob simply +matches the letter “a” in the file name unless it is inside a special +character sequence. + +Other characters have special meaning, and they may include otherwise +normal characters to give them special meaning: + +:Pattern |:Effect +--------------------------------------------------------------------- +`*` | Matches any sequence of zero or more characters +`?` | Matches exactly one character +`[...]` | Matches one character from the enclosed list of characters +`[^...]` | Matches one character *not* in the enclosed list + +Note that unlike [POSIX globs][pg], these special characters and +sequences are allowed to match `/` directory separators as well as the +initial `.` in the name of a hidden file or directory. This is because +Fossil file names are stored as complete path names. The distinction +between file name and directory name is “underneath” Fossil in this sense. + +[pg]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_13 + +The bracket expressions above require some additional explanation: + + * A range of characters may be specified with `-`, so `[a-f]` matches + exactly the same characters as `[abcdef]`. Ranges reflect Unicode + code points without any locale-specific collation sequence. + Therefore, this particular sequence never matches the Unicode + pre-composed character `é`, for example. (U+00E9) + + * This dependence on character/code point ordering may have other + effects to surprise you. For example, the glob `[A-z]` not only + matches upper and lowercase ASCII letters, it also matches several + punctuation characters placed between `Z` and `a` in both ASCII and + Unicode: `[`, `\`, `]`, `^`, `_`, and <tt>\`</tt>. + + * You may include a literal `-` in a list by placing it last, just + before the `]`. + + * You may include a literal `]` in a list by making the first + character after the `[` or `[^`. At any other place, `]` ends the list. + + * You may include a literal `^` in a list by placing it anywhere + except after the opening `[`. + + * Beware that a range must be specified from low value to high + value: `[z-a]` does not match any character at all, preventing the + entire glob from matching. + +Some examples of character lists: + +:Pattern |:Effect +--------------------------------------------------------------------- +`[a-d]` | Matches any one of `a`, `b`, `c`, or `d` but not `ä` +`[^a-d]` | Matches exactly one character other than `a`, `b`, `c`, or `d` +`[0-9a-fA-F]` | Matches exactly one hexadecimal digit +`[a-]` | Matches either `a` or `-` +`[][]` | Matches either `]` or `[` +`[^]]` | Matches exactly one character other than `]` +`[]^]` | Matches either `]` or `^` +`[^-]` | Matches exactly one character other than `-` + +White space means the specific ASCII characters TAB, LF, VT, FF, CR, +and SPACE. Note that this does not include any of the many additional +spacing characters available in Unicode such as +U+00A0, NO-BREAK SPACE. + +Because both LF and CR are white space and leading and trailing spaces +are stripped from each glob in a list, a list of globs may be broken +into lines between globs when the list is stored in a file, as for a +versioned setting. + +Note that 'single quotes' and "double quotes" are the ASCII straight +quote characters, not any of the other quotation marks provided in +Unicode and specifically not the "curly" quotes preferred by +typesetters and word processors. + + +## File Names to Match + +Before it is compared to a glob pattern, each file name is transformed +to a canonical form: + + * all directory separators are changed to `/` + * redundant slashes are removed + * all `.` path components are removed + * all `..` path components are resolved + +(There are additional details we are ignoring here, but they cover rare +edge cases and follow the principle of least surprise.) + +The glob must match the *entire* canonical file name to be considered a +match. + +The goal is to have a name that is the simplest possible for each +particular file, and that will be the same regardless of the platform +you run Fossil on. This is important when you have a repository cloned +from multiple platforms and have globs in versioned settings: you want +those settings to be interpreted the same way everywhere. + +Beware, however, that all glob matching in Fossil is case sensitive +regardless of host platform and file system. This will not be a surprise +on POSIX platforms where file names are usually treated case +sensitively. However, most Windows file systems are case preserving but +case insensitive. That is, on Windows, the names `ReadMe` and `README` +are usually names of the same file. The same is true in other cases, +such as by default on macOS file systems and in the file system drivers +for Windows file systems running on non-Windows systems. (e.g. exfat on +Linux.) Therefore, write your Fossil glob patterns to match the name of +the file as checked into the repository. + +Some example cases: + +:Pattern |:Effect +-------------------------------------------------------------------------------- +`README` | Matches only a file named `README` in the root of the tree. It does not match a file named `src/README` because it does not include any characters that consume (and match) the `src/` part. +`*/README` | Matches `src/README`. Unlike Unix file globs, it also matches `src/library/README`. However it does not match the file `README` in the root of the tree. +`*README` | Matches `src/README` as well as the file `README` in the root of the tree as well as `foo/bar/README` or any other file named `README` in the tree. However, it also matches `A-DIFFERENT-README` and `src/DO-NOT-README`, or any other file whose name ends with `README`. +`src/README` | Matches `src\README` on Windows because all directory separators are rewritten as `/` in the canonical name before the glob is matched. This makes it much easier to write globs that work on both Unix and Windows. +`*.[ch]` | Matches every C source or header file in the tree at the root or at any depth. Again, this is (deliberately) different from Unix file globs and Windows wild cards. + + +## Where Globs are Used + +### <a id="settings"></a>Settings that are Globs + +These settings are all lists of glob patterns: + +:Setting |:Description +-------------------------------------------------------------------------------- +`binary-glob` | Files that should be treated as binary files for committing and merging purposes +`clean-glob` | Files that the [`clean`][] command will delete without prompting or allowing undo +`crlf-glob` | Files in which it is okay to have `CR`, `CR`+`LF` or mixed line endings. Set to "`*`" to disable CR+LF checking +`crnl-glob` | Alias for the `crlf-glob` setting +`encoding-glob` | Files that the [`commit`][] command will ignore when issuing warnings about text files that may use another encoding than ASCII or UTF-8. Set to "`*`" to disable encoding checking +`ignore-glob` | Files that the [`add`][], [`addremove`][], [`clean`][], and [`extras`][] commands will ignore +`keep-glob` | Files that the [`clean`][] command will keep + +All may be [versioned, local, or global](settings.wiki). Use `fossil +settings` to manage local and global settings, or a file in the +repository's `.fossil-settings/` folder at the root of the tree named +for each for versioned setting. + +Using versioned settings for these not only has the advantage that +they are tracked in the repository just like the rest of your project, +but you can more easily keep longer lists of more complicated glob +patterns than would be practical in either local or global settings. + +The `ignore-glob` is an example of one setting that frequently grows +to be an elaborate list of files that should be ignored by most +commands. This is especially true when one (or more) IDEs are used in +a project because each IDE has its own ideas of how and where to cache +information that speeds up its browsing and building tasks but which +need not be preserved in your project's history. + +Although the `empty-dirs` setting is not a list of glob patterns as +such, it is *parsed* that way for consistency among the settings, +allowing [the list parsing rules above](#syntax) to apply. + + +### <a id="commands"></a>Commands that Refer to Globs + +Many of the commands that respect the settings containing globs have +options to override some or all of the settings. These options are +usually named to correspond to the setting they override, such as +`--ignore` to override the `ignore-glob` setting. These commands are: + + * [`add`][] + * [`addremove`][] + * [`changes`][] + * [`clean`][] + * [`commit`][] + * [`extras`][] + * [`merge`][] + * [`settings`][] + * [`status`][] + * [`touch`][] + * [`unset`][] + +The commands [`tarball`][] and [`zip`][] produce compressed archives of a +specific checkin. They may be further restricted by options that +specify glob patterns that name files to include or exclude rather +than archiving the entire checkin. + +The commands [`http`][], [`cgi`][], [`server`][], and [`ui`][] that +implement or support with web servers provide a mechanism to name some +files to serve with static content where a list of glob patterns +specifies what content may be served. + +[`add`]: /help/add +[`addremove`]: /help/addremove +[`changes`]: /help/changes +[`clean`]: /help/clean +[`commit`]: /help/commit +[`extras`]: /help/extras +[`merge`]: /help/merge +[`settings`]: /help/settings +[`status`]: /help/status +[`touch`]: /help/touch +[`unset`]: /help/unset + +[`tarball`]: /help/tarball +[`zip`]: /help/zip + +[`http`]: /help/http +[`cgi`]: /help/cgi +[`server`]: /help/server +[`ui`]: /help/ui + + +### Web Pages that Refer to Globs + +The [`/timeline`][] page supports the query parameter `chng=GLOBLIST` that +names a list of glob patterns defining which files to focus the +timeline on. It also has the query parameters `t=TAG` and `r=TAG` that +names a tag to focus on, which can be configured with `ms=STYLE` to +use a glob pattern to match tag names instead of the default exact +match or a couple of other comparison styles. + +The pages [`/tarball`][] and [`/zip`][] generate compressed archives +of a specific checkin. They may be further restricted by query +parameters that specify glob patterns that name files to include or +exclude rather than taking the entire checkin. + +[`/timeline`]: /help/www/timeline +[`/tarball`]: /help/www/tarball +[`/zip`]: /help/www/zip + + +## Platform Quirks + +Fossil glob patterns are based on the glob pattern feature of POSIX +shells. Fossil glob patterns also have a quoting mechanism, discussed +[above](#syntax). Because other parts of your operating system may interpret glob +patterns and quotes separately from Fossil, it is often difficult to +give glob patterns correctly to Fossil on the command line. Quotes and +special characters in glob patterns are likely to be interpreted when +given as part of a `fossil` command, causing unexpected behavior. + +These problems do not affect [versioned settings files](settings.wiki) +or Admin → Settings in Fossil UI. Consequently, it is better to +set long-term `*-glob` settings via these methods than to use `fossil +settings` commands. + +That advice does not help you when you are giving one-off glob patterns +in `fossil` commands. The remainder of this section gives remedies and +workarounds for these problems. + + +### <a id="posix"></a>POSIX Systems + +If you are using Fossil on a system with a POSIX-compatible shell +— Linux, macOS, the BSDs, Unix, Cygwin, WSL etc. — the shell +may expand the glob patterns before passing the result to the `fossil` +executable. + +Sometimes this is exactly what you want. Consider this command for +example: + + $ fossil add RE* + +If you give that command in a directory containing `README.txt` and +`RELEASE-NOTES.txt`, the shell will expand the command to: + + $ fossil add README.txt RELEASE-NOTES.txt + +…which is compatible with the `fossil add` command's argument list, +which allows multiple files. + +Now consider what happens instead if you say: + + $ fossil add --ignore RE* src/*.c + +This *does not* do what you want because the shell will expand both `RE*` +and `src/*.c`, causing one of the two files matching the `RE*` glob +pattern to be ignored and the other to be added to the repository. You +need to say this in that case: + + $ fossil add --ignore 'RE*' src/*.c + +The single quotes force a POSIX shell to pass the `RE*` glob pattern +through to Fossil untouched, which will do its own glob pattern +matching. There are other methods of quoting a glob pattern or escaping +its special characters; see your shell's manual. + +Beware that Fossil's `--ignore` option does not override explicit file +mentions: + + $ fossil add --ignore 'REALLY SECRET STUFF.txt' RE* + +You might think that would add everything beginning with `RE` *except* +for `REALLY SECRET STUFF.txt`, but when a file is both given +explicitly to Fossil and also matches an ignore rule, Fossil asks what +you want to do with it in the default case; and it does not even ask +if you gave the `-f` or `--force` option along with `--ignore`. + +The spaces in the ignored file name above bring us to another point: +such file names must be quoted in Fossil glob patterns, lest Fossil +interpret it as multiple glob patterns, but the shell interprets +quotation marks itself. + +One way to fix both this and the previous problem is: + + $ fossil add --ignore "'REALLY SECRET STUFF.txt'" READ* + +The nested quotation marks cause the inner set to be passed through to +Fossil, and the more specific glob pattern at the end — that is, +`READ*` vs `RE*` — avoids a conflict between explicitly-listed +files and `--ignore` rules in the `fossil add` command. + +Another solution would be to use shell escaping instead of nested +quoting: + + $ fossil add --ignore "\"REALLY SECRET STUFF.txt\"" READ* + +It bears repeating that the two glob patterns here are not interpreted +the same way when running this command from a *subdirectory* of the top +checkout directory as when running it at the top of the checkout tree. +If these files were in a subdirectory of the checkout tree called `doc` +and that was your current working directory, the command would instead +have to be: + + $ fossil add --ignore "'doc/REALLY SECRET STUFF.txt'" READ* + +The Fossil glob pattern still needs the `doc/` prefix because +Fossil always interprets glob patterns from the base of the checkout +directory, not from the current working directory as POSIX shells do. + +When in doubt, use `fossil status` after running commands like the +above to make sure the right set of files were scheduled for insertion +into the repository before checking the changes in. You never want to +accidentally check something like a password, an API key, or the +private half of a public cryptographic key into Fossil repository that +can be read by people who should not have such secrets. + + +### <a id="windows"></a>Windows + +Before we get into Windows-specific details here, beware that this +section does not apply to the several Microsoft Windows extensions that +provide POSIX semantics to Windows, for which you want to use the advice +in [the POSIX section above](#posix) instead: + + * the ancient and rarely-used [Microsoft POSIX subsystem][mps]; + * its now-discontinued replacement feature, [Services for Unix][sfu]; or + * their modern replacement, the [Windows Subsystem for Linux][wsl] + +[mps]: https://en.wikipedia.org/wiki/Microsoft_POSIX_subsystem +[sfu]: https://en.wikipedia.org/wiki/Windows_Services_for_UNIX +[wsl]: https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux + +(The latter is sometimes incorrectly called "Bash on Windows" or "Ubuntu +on Windows," but the feature provides much more than just Bash or Ubuntu +for Windows.) + +Neither standard Windows command shell — `cmd.exe` or PowerShell +— expands glob patterns the way POSIX shells do. Windows command +shells rely on the command itself to do the glob pattern expansion. The +way this works depends on several factors: + + * the version of Windows you are using + * which OS upgrades have been applied to it + * the compiler that built your Fossil executable + * whether you are running the command interactively + * whether the command is built against a runtime system that does this + at all + * whether the Fossil command is being run from a file named `*.BAT` vs + being named `*.CMD` + +Usually (but not always!) the C runtime library that your `fossil.exe` +executable is built against does this glob expansion on Windows so the +program proper does not have to. This may then interact with the way the +Windows command shell you’re using handles argument quoting. Because of +these differences, it is common to find perfectly valid Fossil command +examples that were written and tested on a POSIX system which then fail +when tried on Windows. + +The most common problem is figuring out how to get a glob pattern passed +on the command line into `fossil.exe` without it being expanded by the C +runtime library that your particular Fossil executable is linked to, +which tries to act like [the POSIX systems described above](#posix). Windows is +not strongly governed by POSIX, so it has not historically hewed closely +to its strictures. + +For example, consider how you would set `crlf-glob` to `*` in order to +get normal Windows text files with CR+LF line endings past Fossil's +"looks like a binary file" check. The naïve approach will not work: + + C:\...> fossil setting crlf-glob * + +The C runtime library will expand that to the list of all files in the +current directory, which will probably cause a Fossil error because +Fossil expects either nothing or option flags after the setting's new +value, not a list of file names. (To be fair, the same thing will happen +on POSIX systems, only at the shell level, before `.../bin/fossil` even +gets run by the shell.) + +Let's try again: + + C:\...> fossil setting crlf-glob '*' + +Quoting the argument like that will work reliably on POSIX, but it may +or may not work on Windows. If your Windows command shell interprets the +quotes, it means `fossil.exe` will see only the bare `*` so the C +runtime library it is linked to will likely expand the list of files in +the current directory before the `setting` command gets a chance to +parse the command line arguments, causing the same failure as above. +This alternative only works if you’re using a Windows command shell that +passes the quotes through to the executable *and* you have linked Fossil +to a C runtime library that interprets the quotes properly itself, +resulting in a bare `*` getting clear down to Fossil’s `setting` command +parser. + +An approach that *will* work reliably is: + + C:\...> echo * | fossil setting crlf-glob --args - + +This works because the built-in Windows command `echo` does not expand its +arguments, and the `--args -` option makes Fossil read further command +arguments from its standard input, which is connected to the output +of `echo` by the pipe. (`-` is a common Unix convention meaning +"standard input," which Fossil obeys.) A [batch script][fng.cmd] to automate this trick was +posted on the now-inactive Fossil Mailing List. + +[fng.cmd]: https://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg25099.html + +(Ironically, this method will *not* work on POSIX systems because it is +not up to the command to expand globs. The shell will expand the `*` in +the `echo` command, so the list of file names will be passed to the +`fossil` standard input, just as with the first example above!) + +Another (usually) correct approach which will work on both Windows and +POSIX systems: + + C:\...> fossil setting crlf-glob *, + +This works because the trailing comma prevents the glob pattern from +matching any files, unless you happen to have files named with a +trailing comma in the current directory. If the pattern matches no +files, it is passed into Fossil's `main()` function as-is by the C +runtime system. Since Fossil uses commas to separate multiple glob +patterns, this means "all files from the root of the Fossil checkout +directory downward and nothing else," which is of course equivalent to +"all managed files in this repository," our original goal. + + +## Experimenting + +To preview the effects of command line glob pattern expansion for +various glob patterns (unquoted, quoted, comma-terminated), for any +combination of command shell, OS, C run time, and Fossil version, +precede the command you want to test with [`test-echo`][] like so: + + $ fossil test-echo setting crlf-glob "*" + C:\> echo * | fossil test-echo setting crlf-glob --args - + +The [`test-glob`][] command is also handy to test if a string +matches a glob pattern. + +[`test-echo`]: /help/test-echo +[`test-glob`]: /help/test-glob + + +## Converting `.gitignore` to `ignore-glob` + +Many other version control systems handle the specific case of +ignoring certain files differently from Fossil: they have you create +individual "ignore" files in each folder, which specify things ignored +in that folder and below. Usually some form of glob patterns are used +in those files, but the details differ from Fossil. + +In many simple cases, you can just store a top level "ignore" file in +`.fossil-settings/ignore-glob`. But as usual, there will be lots of +edge cases. + +[Git has a rich collection of ignore files][gitignore] which +accumulate rules that affect the current command. There are global +files, per-user files, per workspace unmanaged files, and fully +version controlled files. Some of the files used have no set name, but +are called out in configuration files. + +[gitignore]: https://git-scm.com/docs/gitignore + +In contrast, Fossil has a global setting and a local setting, but the local setting +overrides the global rather than extending it. Similarly, a Fossil +command's `--ignore` option replaces the `ignore-glob` setting rather +than extending it. + +With that in mind, translating a `.gitignore` file into +`.fossil-settings/ignore-glob` may be possible in many cases. Here are +some of features of `.gitignore` and comments on how they relate to +Fossil: + + * "A blank line matches no files...": same in Fossil. + * "A line starting with # serves as a comment...": same in Fossil, including + the possibility of escaping an initial `#` with a backslash to allow globs + beginning with a hash. + * "Trailing spaces are ignored unless they are quoted..." is similar + in Fossil. All whitespace before and after a glob is trimmed in + Fossil unless quoted with single or double quotes. Git uses + backslash quoting instead, which Fossil does not. + * "An optional prefix "!" which negates the pattern...": not in + Fossil. + * Git's globs are relative to the location of the `.gitignore` file: + Fossil's globs are relative to the root of the workspace. + * Git's globs and Fossil's globs treat directory separators + differently. Git includes a notation for zero or more directories + that is not needed in Fossil. + +### Example + +In a project with source and documentation: + + work + +-- doc + +-- src + +The file `doc/.gitignore` might contain: + + # Finished documents by pandoc via LaTeX + *.pdf + # Intermediate files + *.tex + *.toc + *.log + *.out + *.tmp + +Entries in `.fossil-settings/ignore-glob` with similar effect, also +limited to the `doc` folder: + + doc/*.pdf + doc/*.tex, doc/*.toc, doc/*.log, doc/*.out, doc/*.tmp + + + + + +## Implementation and References + +The implementation of the Fossil-specific glob pattern handling is here: + +:File |:Description +-------------------------------------------------------------------------------- +[`src/glob.c`][] | pattern list loading, parsing, and generic matching code +[`src/file.c`][] | application of glob patterns to file names + +[`src/glob.c`]: https://fossil-scm.org/home/file/src/glob.c +[`src/file.c`]: https://fossil-scm.org/home/file/src/file.c + +See the [Adding Features to Fossil][aff] document for broader details +about finding and working with such code. + +The actual pattern matching leverages the `GLOB` operator in SQLite, so +you may find [its documentation][gdoc], [source code][gsrc] and [test +harness][gtst] helpful. + +[aff]: ./adding_code.wiki +[gdoc]: https://sqlite.org/lang_expr.html#like +[gsrc]: https://www.sqlite.org/src/artifact?name=9d52522cc8ae7f5c&ln=570-768 +[gtst]: https://www.sqlite.org/src/artifact?name=66a2c9ac34f74f03&ln=586-673 diff --git a/www/glossary.md b/www/glossary.md new file mode 100644 index 0000000..8fff002 --- /dev/null +++ b/www/glossary.md @@ -0,0 +1,466 @@ +# Glossary + +There are several terms-of-art in Fossil that have specific meanings +which are either not immediately obvious to an outsider or which have +technical associations that can lead someone to either use the terms +incorrectly or to get the wrong idea from someone using those terms +correctly. We hope to teach users how to properly “speak Fossil” with +this glossary. + +The bullet-point lists following each definition are meant to be +clarifying and illustrative. They are not part of the definitions +themselves. + + +## <a id="project"></a>Project + +A collection of one or more computer files that serve some conceptually +unified purpose, which purpose changes and evolves over time, with the +history of that project being a valuable record. + +* We qualify the Fossil definition of this common word like this to + set aside cases where a zip file or tarball would suffice. If you + can pack your project up into such an archive once and be done with + it, Fossil is overkill. + + And yet that is often just the beginning, since there is often a + need for something to be changed, so now you have “version 2” of the + archive file. If you can foresee yourself creating versioned archive + files for your project, then you probably should be using Fossil for + it instead, then using Fossil’s [zip] or [tarball] command to + automatically produce archives of the latest version rather than + manually produce and track versions of the archive. The web version + of these commands ([`/zip`][zw] and [`/tarball`][tw]) are + particularly useful for public distribution of the latest version of + a project’s files. + +* Fossil was designed to host the SQLite software project, which is + comprised of source code, makefiles, scripts, documentation files, + and so forth. Fossil is also useful for many other purposes, such as + a fiction book project where each chapter is held in a separate file + and assembled into a finished whole deliverable. + +* We speak of projects being more than one file because even though + Fossil can be made to track the history of a single file, it is far + more often the case that when you get to something of a scale + sufficient to be called a “project,” there is more than one + version-tracked file involved, if not at the start, then certainly + by the end of the project. + + To take the example of a fiction book above, instead of putting each + chapter in a separate file, you could use a single AsciiDoc file for + the entire book project rather than make use of its [include + facility][AIF] to assemble it from chapter files, since that does at + least solve the [key problems][IFRS] inherent in version-tracking + something like Word’s DOCX format with Fossil instead. + + While Fossil will happily track the single file containing the prose + of your book project for you, you’re still likely to want separate + files for the cover artwork, a style sheet for use in converting the + source document to HTML, scripts to convert that intermediate output + to PDF and ePub in a reliably repeatable fashion, a `README` file + containing instructions to the printing house, and so forth. + +* Fossil requires that all the files for a project be collected into a + single directory hierarchy, owned by a single user with full rights + to modify those files. Fossil is not a good choice for managing a + project that has files scattered hither and yon all over the file + system, nor of collections of files with complicated ownership and + access rights. + + A project made of an operating system + installation’s configuration file set is not a good use of Fossil, + because you’ll have all of your OS’s *other* files intermixed. + Worse, Fossil doesn’t track OS permissions, so even if you were to + try to use Fossil as a system deployment tool by archiving versions + of the OS configuration files and then unpacking them on a new + system, the extracted project files would have read/write access by + the user who did the extraction, which probably isn’t what you were + wanting. + + Even with these problems aside, do you really want a `.fslckout` + SQLite database at the root of your filesystem? Are you prepared for + the consequences of saying `fossil clean --verily` on such a system? + You can constrain that with [the `ignore-glob` setting][IGS], but + are you prepared to write and maintain all the rules needed to keep + Fossil from blowing away the untracked portions of the file system? + We believe Fossil is a poor choice for a whole-system configuration + backup utility. + + As a counterexample, a project tracking your [Vim] configuration + history is a much better use of Fossil, because it’s all held within + `~/.vim`, and your user has full rights to that subdirectory. + +[AIF]: https://docs.asciidoctor.org/asciidoc/latest/directives/include/ +[IGS]: /help/ignore-glob +[IFRS]: ./image-format-vs-repo-size.md +[tarball]: /help/tarball +[tw]: /help/www/tarball +[Vim]: https://www.vim.org/ +[zip]: /help/zip +[zw]: /help/www/zip + + +## Repository <a id="repository" name="repo"></a> + +A single file that contains all historical versions of all files in a +project, which can be [cloned] to other machines and +[synchronized][sync] with them. Jargon: repo. + +* A Fossil repo is similar to an archive file in that it is a single + file that stores compressed versions of one or more files. Files can be + extracted from the repo, and new files can be added to the repo, + but a Fossil repo has other capabilities + above and beyond what simple archive formats can do. + +* Fossil does not care what you name your repository files, though + we do suggest “`.fossil`” as a standard extension. There is + only one place in Fossil where that convention is required, being the + [`fossil server DIRECTORY`][svrcmd] command, since it serves up + `*.fossil` files from `DIRECTORY`. If you don’t use that feature, + you can name your repo files anything you like. + +* Cloned and synced repos redundantly store all available information + about that project, so if any one repo is lost, all of the cloned + historical content of the project as of the last sync is preserved + in each surviving repo. + + Each user generally clones the project repository down to each + computer they use to participate in that project, and there is + usually at least one central host for the project as well, so there + is usually plenty of redundancy in any given Fossil-based project. + + That said, a Fossil repository clone is a backup only in a limited + sense, because some information can’t be cloned, some doesn’t sync + by default, and some data is neither clonable nor syncable. We cover + these limitations and the workarounds for them in a separate + document, [Backing Up a Remote Fossil Repository][backup]. + +* Rather than be a backup, a Fossil repository clone is a + communication method for coordinating work on the project among + multiple machines and users: local work done against one repository + is communicated up to its parent repository on the next sync, and + work done on other repositories that were previously synced up to + that parent get pulled down into the local repo clone. + + This bidirectional sync is automatic and on by default in Fossil. + You can [disable it][asdis], or you can [push] or [pull] + unidirectionally, if you wish. + + The Fossil philosophy is that all project information resides in + each clone of the repository. In the ideal world, this would occur + instantly and automatically, but in actual use, Fossil falls + somewhat short of that mark. Some limitations are simply + technological: a given clone may be temporarily out of communication + with its parent repository, network delays exist, and so forth. + Fossil is [an AP mode system][CAP]. (This is sometimes called + “eventual consistency.”) Other cases come down to administrative + necessity, as covered in [the backup doc][backup]. + +* Fossil doesn’t require that you have redundant clones. Whether you + do or not is a local decision based on usage needs, communication + requirements, desire for backups, and so forth. + +* Fossil doesn’t care where the repositories are stored, but we + recommend keeping them all in a single subdirectory such as + "`~/fossils`" or "`%USERPROFILE%\Fossils`". A flat set of files + suffices for simple purposes, but you may have use for something + more complicated. This author uses a scheme like the following on + mobile machines that shuttle between home and the office: + +``` pikchr toggle indent +box "~/museum/" fit +move right 0.1 +line right dotted +move right 0.05 +box invis "where one stores valuable fossils" ljust + +arrow down 50% from first box.s then right 50% +box "work/" fit +move right 0.1 +line dotted +move right 0.05 +box invis "projects from $DAYJOB" ljust + +arrow down 50% from 2nd vertex of previous arrow then right 50% +box "home/" fit +move right 0.1 +line dotted right until even with previous line.end +move right 0.05 +box invis "personal at-home projects" ljust + +arrow down 50% from 2nd vertex of previous arrow then right 50% +box "other/" fit +move right 0.1 +line dotted right until even with previous line.end +move right 0.05 +box invis "clones of Fossil itself, SQLite, etc." ljust +``` + +[asdis]: /help/autosync +[backup]: ./backup.md +[CAP]: ./cap-theorem.md +[cloned]: /help/clone +[pull]: /help/pull +[push]: /help/push +[svrcmd]: /help/server +[sync]: /help/sync + +[repository]: #repo +[repositories]: #repo + + +## Version / Revision / Hash / UUID <a id="version" name="hash"></a> + +These terms all mean the same thing: a long hexadecimal +[SHA hash value](./hashes.md) that uniquely identifies a particular +[check-in](#ci). + +We’ve listed the alternatives in decreasing preference order: + +* **Version** and **revision** are near-synonyms in common usage. + Fossil’s code and documentation use both interchangeably because + Fossil was created to manage the development of the SQLite project, + which formerly used [CVS], the Concurrent Versions System. CVS in + turn started out as a front-end to [RCS], the Revision Control + System, but even though CVS uses “version” in its name, it numbers + check-ins using a system derived from RCS’s scheme, which it calls + “Revisions” in user-facing output. Fossil inherits this confusion + honestly. + +* **Hash** refers to the [SHA1 or SHA3-256 hash](./hashes.md) of the + content of the checked-in data, uniquely identifying that version of + the managed files. It is a strictly correct synonym, used more often + in low-level contexts than the term “version.” + +* **UUID** is a deprecated term still found in many parts of the + Fossil internals and (decreasingly) its documentation. The problem + with using this as a synonym for a Fossil-managed version of the + managed files is that there are [standards][UUID] defining the + format of a “UUID,” none of which Fossil follows, not even the + [version 4][ruuid] (random) format, the type of UUID closest in + meaning and usage to a Fossil hash.(^A pre-Fossil 2.0 style SHA1 + hash is 160 bits, not the 128 bits many people expect for a proper + UUID, and even if you truncate it to 128 bits to create a “good + enough” version prefix, the 6 bits reserved in the UUID format for + the variant code cannot make a correct declaration except by a + random 1:64 chance. The SHA3-256 option allowed in Fossil 2.0 and + higher doesn’t help with this confusion, making a Fossil version + hash twice as large as a proper UUID. Alas, the term will never be + fully obliterated from use since there are columns in the Fossil + repository format that use the obsolete term; we cannot change this + without breaking backwards compatibility.) + +You will find all of these synonyms used in the Fossil documentation. +Some day we may settle on a single term, but it doesn’t seem likely. + +[CVS]: https://en.wikipedia.org/wiki/Concurrent_Versions_System +[hash]: #version +[RCS]: https://en.wikipedia.org/wiki/Revision_Control_System +[ruuid]: https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random) +[snfs]: https://en.wikipedia.org/wiki/Snapshot_(computer_storage)#File_systems +[UUID]: https://en.wikipedia.org/wiki/Universally_unique_identifier#Version_4_(random) +[version]: #version + + +## Check-in <a id="check-in" name="ci"></a> + +A [version] of the project’s files that have been committed to the +[repository]; as such, it is sometimes called a “commit” instead. A +check-in is a snapshot of the project at an instant in time, as seen from +a single [check-out’s](#co) perspective. It is sometimes styled +“`CHECKIN`”, especially in command documentation where any +[valid check-in name][ciname] can be used. + +* There is a harmless conflation of terms here: any of the various + synonyms for [version] may be used where “check-in” is more accurate, + and vice versa, because there is a 1:1 relation between them. A + check-in *has* a version, but a version suffices to uniquely look up + a particular commit.[^snapshot] + +* Combining both sets of synonyms results in a list of terms that is + confusing to new Fossil users, but it’s easy + enough to internalize the concepts. [Committing][commit] creates a + *commit.* It may also be said to create a checked-in *version* of a + particular *revision* of the project’s files, thus creating an + immutable *snapshot* of the project’s state at the time of the + commit. Fossil users find each of these different words for the + same concept useful for expressive purposes among ourselves, but to + Fossil proper, they all mean the same thing. + +* Check-ins are immutable. + +* Check-ins exist only inside the repository. Contrast a + [check-out](#co). + +* Check-ins may have [one or more names][ciname], but only the + [hash] is globally unique, across all time; we call it the check-in’s + canonical name. The other names are either imprecise, contextual, or + change their meaning over time and across [repositories]. + +[^snapshot]: You may sometimes see the term “snapshot” used as a synonym + for a check-in or the version number identifying said check-in. We + must warn against this usage because there is a potential confusion + here: [the `stash` command][stash] uses the term “snapshot,” as does + [the `undo` system][undo] to make a distinction with check-ins. + Nevertheless, there is a conceptual overlap here between Fossil and + systems that do use the term “snapshot,” the primary distinction being + that Fossil will capture only changes to files you’ve [added][add] to + the [repository], not to everything in [the check-out directory](#co) + at the time of the snapshot. (Thus [the `extras` command][extras].) + Contrast a snapshot taken by a virtual machine system or a + [snapshotting file system][snfs], which captures changes to everything + on the managed storage volume. + +[add]: /help/add +[ciname]: ./checkin_names.wiki +[extras]: /help/extras +[stash]: /help/stash +[undo]: /help/undo + + + +## Check-out <a id="check-out" name="co"></a> + +A set of files extracted from a [repository] that represent a +particular [check-in](#ci) of the [project](#project). + +* Unlike a check-in, a check-out is mutable. It may start out as a + version of a particular check-in extracted from the repository, but + the user is then free to make modifications to the checked-out + files. Once those changes are formally [committed][commit], they + become a new immutable check-in, derived from its parent check-in. + +* You can switch from one check-in to another within a check-out + directory by passing those names to [the `fossil update` + command][update]. + +* Check-outs relate to repositories in a one-to-many fashion: it is + common to have a single repo clone on a machine but to have it + [open] in [multiple working directories][mwd]. Check-out directories + are associated with the repos they were created from by settings + stored in the check-out directory. This is in the `.fslckout` file on + POSIX type systems, but for historical compatibility reasons, it’s + called `_FOSSIL_` by native Windows builds of Fossil. + + (Contrast the Cygwin and WSL Fossil binaries, which use POSIX file + naming rules.) + +* In the same way that one cannot extract files from a zip archive + without having a copy of that zip file, one cannot make check-outs + without access to the repository file or a clone thereof. + +* Because a Fossil repository is an SQLite database file, the same + rules for avoiding data corruption apply to it. In particular, it is + [nearly a hard requirement][h2cflp] that the repository clone be on + the same machine as the one where you make check-outs and the + subsequent check-ins. + + That said, the relative locations of the repo and the check-out + within the local file system are arbitrary. The repository may be + located inside the folder holding the check-out, but it certainly + does not have to be, and it usually is not. As an example, [the + Fossil plugin for Visual Studio Code][fpvsc] defaults to storing the + repo clone within the project directory as a file called `.fsl`, but + this is because VSCode’s version control features assume it’s being + used with Git, where the repository is the `.git` subdirectory + contents. With Fossil, [different check-out workflows][cwork] are + preferred. + +[commit]: /help/commit +[cwork]: ./ckout-workflows.md +[h2cflp]: https://www.sqlite.org/howtocorrupt.html#_file_locking_problems +[fpvsc]: https://marketplace.visualstudio.com/items?itemName=koog1000.fossil +[open]: /help/open +[mwd]: ./ckout-workflows.md#mcw +[update]: /help/update + + +## <a id="docs"></a>Embedded Documentation + +Serving as an alternative to Fossil’s built-in [wiki], the [embedded +documentation feature][edoc] stores the same type of marked-up text +files, but under Fossil’s powerful version control features. + +* The simple rule for determining whether to use the wiki or embedded + docs for any given document is whether the content is considered + “evergreen,” as with a Wikipedia article. + + While Fossil’s wiki feature does store the history of each + document’s changes, Fossil always presents the current version of + the document unless you manually go out of your way to dig back into + the history. Then, having done so, links from that historical + version of the wiki document take you to the current versions of the + target documents, not to the version contemporaneous with the source + document. + + The consequence is that if you say something like… + + $ fossil up 2020-04-01 + $ fossil ui --page wcontent + + …you will **not** see the list of wiki articles as of April Fool’s Day in 2020, but + instead the list of *current* wiki article versions, the same as if you ran it + from a check-out of the tip-of-trunk. + + Contrast embedded docs, which are not only version-controlled as + normal files are in Fossil, they participate in all the tagging, + branching, and other versioning features. There are several + consequences of this, such as that Fossil’s [special check-in + names][ciname] work with embedded doc URLs: + + * <p>If you visit an embedded doc as `/doc/release/file.md` and + then click on a relative link from that document, you will remain on + the release branch. This lets you see not only the release + version of a software project but also the documentation as of + that release.</p> + + * <p>If you visit `/doc/2020-04-01/file.md`, you will not only + pull up the version of `file.md` as of that date, relative links + will take you to contemporaneous versions of those embedded docs + as well.</p> + + * <p>If you say `fossil up 2020-04-01 && fossil ui` and then visit + `/doc/ckout/file.md`, you’ll not only see the checked-out + version of the file as of that date, relative links will show + you other files within that checkout.</p> + +* Fossil’s wiki presents a flat list of articles, while embedded docs + are stored in the repository’s file hierarchy, a powerful + organizational tool well-suited to complicated documentation. + +* Your repository’s Home page is a good candidate for the wiki, as is + documentation meant for use only with the current version of the + repository’s contents. + +* If you are at all uncertain whether to use the wiki or the embedded + documentation feature, prefer the latter, since it is inherently + more powerful, and when you use the [`/fileedit` feature][fef], the + workflow is scarcely different from using the wiki. + + (This very file is embedded documentation: clone + [Fossil’s self-hosting repository][fshr] and you will find it as + `www/glossary.md`.) + +[edoc]: ./embeddeddoc.wiki +[fef]: ./fileedit-page.md +[fshr]: ./selfhost.wiki +[wiki]: ./wikitheory.wiki + + +## <a id="cap"></a>Capability + +Fossil includes a powerful [role-based access control system][rbac] +which affects which users have permission to do certain things within a given +[repository]. You can read more about this complex topic +[here](./caps/). + +Some people — and indeed certain parts of Fossil’s own code — use the +term “permissions” instead, but since [operating system file permissions +also play into this](./caps/#webonly), we prefer the term “capabilities” +(or “caps” for short) when talking about Fossil’s RBAC system to avoid a +confusion here. + +[rbac]: https://en.wikipedia.org/wiki/Role-based_access_control + +<div style="height:50em" id="this-space-intentionally-left-blank"></div> diff --git a/www/grep.md b/www/grep.md new file mode 100644 index 0000000..7b222c5 --- /dev/null +++ b/www/grep.md @@ -0,0 +1,160 @@ +# Fossil grep vs POSIX grep + +As of Fossil 2.7, there is a `grep` command which acts roughly like +POSIX's `grep -E` over all historical versions of one or more managed files. +This document explains the commonalities and divergences between [POSIX +`grep`](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html) +and Fossil `grep`. + + +## Options + +Fossil `grep` implements about half of the options specified for +POSIX `grep`: + +| Option | Meaning +|--------|------------------------------------------------------------- +| `-c` | report the count of matches rather than the matched text +| `-i` | ignore case in matches +| `-l` | list a checkin ID prefix for matching historical versions of the file +| `-q` | no output; return only a status code indicating the success of the match +| `-s` | suppress error output about missing files +| `-v` | invert the sense of the match + +That leaves several divergences at the option level from POSIX `grep`: + +* You cannot give more than one pattern, as with `grep -e` or + `grep -f`. + +* There is no equivalent of `grep -F` to do literal fixed-string + matches only. + +* There is no `-x` option for doing a whole-line match. + +* `fossil grep -l` does not do precisely the same thing as POSIX + `grep -l`: it lists checkin ID prefixes, not file names. + +* Fossil always gives the line number in its output, which is to say + that it acts like `grep -n`. There is no way to disable the line + number in `fossil grep` output. + +Patches to remove those limitations will be thoughtfully considered. + +Fossil `grep` doesn’t support any of the GNU and BSD `grep` extensions. +For instance, it doesn’t support the common `-R` extension to POSIX, +which would presumably search a subtree of managed files. If Fossil does +one day get this feature, it would have a different option letter, since +`-R` in Fossil has a different meaning, by convention. Until then, you +can get the same effect on systems with a POSIX shell like so: + + $ fossil grep COMMAND: $(fossil ls src) + +If you run that in a check-out of the [Fossil self-hosting source +repository][fshsr], that returns the first line of the built-in +documentation for each Fossil command, across all historical versions. + +Fossil `grep` has extensions relative to these other `grep` standards, +such as `--verbose` to print each checkin ID considered, regardless of +whether it matches. This one is noteworthy here because the behavior +used to be under `-v` before we reassigned it to give POSIX-like `grep +-v` behavior. + +[fshsr]: ./selfhost.wiki + + +## Regular Expression Dialect + +Fossil contains a built-in regular expression engine implementing a +subset of the [POSIX extended regular expression][ere] dialect: + +[ere]: https://en.wikipedia.org/wiki/Regular_expression#POSIX_extended + +| Atom | Meaning +|---------|------------------------------------------------------------- +| `X*` | zero or more occurrences of X +| `X+` | one or more occurrences of X +| `X?` | zero or one occurrences of X +| `X{p,q}`| between p and q occurrences of X, inclusive +| `(X)` | match X +| <tt>X\|Y</tt>| X or Y +| `^X` | X occurring at the beginning of a line +| `X$` | X occurring at the end of a line +| `.` | Match any single character +| `\c` | Character `c` where `c` is one of <tt>{}()[]\|\*+?.\\</tt> +| `\c` | C-language escapes for `c` in `afnrtv`. ex: `\t` or `\n` +| `\uXXXX`| Where XXXX is exactly 4 hex digits, Unicode value XXXX +| `\xXX` | Where XX is exactly 2 hex digits, Unicode value XX +| `[abc]` | Any single character from the set `abc` +| `[^abc]`| Any single character not in the set `abc` +| `[a-z]` | Any single character in the range `a-z` +| `[^a-z]`| Any single character not in the range `a-z` +| `\b` | Word boundary +| `\w` | Word character: `[A-Za-z0-9_]` +| `\W` | Non-word character: `[^A-Za-z0-9_]` +| `\d` | Digit: `[0-9]` +| `\D` | Non-digit: `[^0-9]` +| `\s` | Whitespace character: `[ \t\r\n\v\f]` +| `\S` | Non-whitespace character: `[^ \t\r\n\v\f]` + +There are several restrictions in Fossil `grep` relative to a fully +POSIX compatible regular expression engine. Among them are: + +* There is currently no support for POSIX character classes such as + `[:lower:]`. + +* The values of `p` and `q` in the "`{p,q}`" syntax can be no greater + than 999. This is because the NFA that is used for regular expression + matching is proportional in size to the largest p or q value, and hence + allowing arbitrarily large values could result in a DoS attack. + +* Fossil `grep` does not currently attempt to take your operating + system's locale settings into account when doing this match. Since + Fossil has no way to mark a given file as having a particular + encoding, Fossil `grep` assumes the input files are in UTF-8 format. + + This means Fossil `grep` will not work correctly if the files in + your repository are in an encoding that is not backwards-compatible + with ASCII, such as UTF-16. Partially compatible encodings such as + ISO 8859 should work with Fossil `grep` as long as you stick to + their ASCII-compatible subset. + +The Fossil `grep` language is not a strict subset of POSIX extended +regular expressions. Some of the features documented above are +well-understood extensions to it, such as the "word" features `\b`, `\w` +and `\W`. + +Fossil `grep` is based on the Unicode engine from [SQLite's FTS5 +feature][fts5]. This means it does do things like Unicode-aware case +folding. Therefore, it is usually a user error to attempt to substitute +`[a-z]` for a lack of the POSIX character class `[:lower:]` if you are +grepping over pretty much any human written language other than English. +Use `fossil grep -i` instead, which does Unicode case folding. + +[fts5]: https://www.sqlite.org/fts5.html + + +## Algorithm Details + +Fossil `grep` uses a [nondeterministic finite automaton][nfa] for +matching, so the performance is bounded by ***O(nm)*** where ***n*** is +the size of the regular expression and ***m*** is the size of the input +string. + +[nfa]: https://en.wikipedia.org/wiki/Nondeterministic_finite_automaton + +In order to avoid [ReDoS attacks][redos], the Fossil regular expression +matcher was purposely written to avoid [implementation choices][rei] +which have the potential to require exponential evaluation time. This +constrains the possible feature set we can support in the Fossil `grep` +dialect. For instance, we are unlikely to ever add support for +[backtracking][bt]. + +[redos]: https://en.wikipedia.org/wiki/ReDoS +[rei]: https://en.wikipedia.org/wiki/Regular_expression#Implementations_and_running_times +[bt]: https://en.wikipedia.org/wiki/Backtracking + +The `X{p,q}` operator expands to `p` copies of `X` followed by `q-p` +copies of `X?` before RE evaluation. The ***O(nm)*** performance bound +above remains true for this case, but realize that it applies to the RE +*after* this expansion, not to the form as given by the user. In other +words, as `q-p` increases, so does the RE evaluation time. diff --git a/www/gsoc-ideas.md b/www/gsoc-ideas.md new file mode 100644 index 0000000..5d14f1a --- /dev/null +++ b/www/gsoc-ideas.md @@ -0,0 +1,177 @@ +# List of Projects and Tasks + +This list was made for the Fossil project's application for [Google Summer of Code](https://summerofcode.withgoogle.com/) in 2021. That application was +unsuccessful, but still this list is a starting point for anyone looking +for a place to start. We welcome newcomers, and invite developers to follow the simple +[procedures for contributing to Fossil](./contribute.wiki). The +[hacker how-to](./hacker-howto.wiki) is recommended reading. + +There are two implementations of the Fossil data model: + +* [the classic Fossil project](https://fossil-scm.org) , which is where this file is maintained and + which is as of 2021 how everyone interacts with Fossil objects +* [libfossil](https://fossil.wanderinghorse.net/r/libfossil), which is an independent project to manipulate Fossil objects from a library, or using commandline tools which are thin wrappers to the library + +As of 2021 the two implementations have an identical implementation of the +Fossil data model, are 100% compatible in terms of data access since they use +the same SQL, and are 100% binary compatible in terms of on-disk storage. + +The projects listed here are grouped by functionality - User Interface, Integration, Email, +etc. If you are looking for something easy to start with, then depending where +your interests lie, there are some small libfossil tasks and small +features to work on in the UI. + +# UI, Look and Feel + +Tasks for those interested in graphic/web design: + +* Add a quote button to the Forum, such as [discussed in this thread](https://fossil-scm.org/forum/forumpost/7ad03cd73d) +* Improve the documentation history-browsing page to enable selection of 2 arbitrary versions to diff, similar to the [Mediawiki history feature enabled on Wikipedia](https://en.wikipedia.org/w/index.php?title=Fossil_\(software\)&action=history) +* Allow diffing of Forum posts +* General touch-ups in the existing skins. This may, depending on how deep one + cares to dig, require digging into C code to find, and potentially modify, how + the HTML is generated. +* Creation of one or more new skins. This does not specifically require any C + know-how. +* Complete per-feature CSS facilities in [the Inskinerator](https://tangentsoft.com/inskinerator/dir) and add features to the Inskinerator + +# Projects Relating to Fossil Integration + +* Fossil hooks for pipelines with CI/CD such as static analysis, Buildbot, Gerrit, Travis and Jenkins are not well-documented and may need some further development. Make this work better, with configuration examples +* Create a [Pandoc](https://pandoc.org) filter that handles Fossil-style Markdown +* Create a [Pandoc filter that handles Pikchr](https://groups.google.com/g/pandoc-discuss/c/zZSspnHHsg0?pli=1) (Pikchr can be used with many kinds of layout, not just Markdown) +* Editor integration: [improve the Fossil VSCode plugin](https://marketplace.visualstudio.com/items?itemName=koog1000.fossil) or [create a Fossil plugin for Eclipse](https://marketplace.eclipse.org/taxonomy/term/26%2C31) +* Develop a test suite for the draft JSON API in libfossil. This JSON API is a way of integrating many kinds of systems with Fossil +* Re-implement the draft JSON API in libfossil to use the JSON capability in SQLite, now that SQLite has JSON. This is a large project and would start with feasibility analysis + +# Adding Inbound (Receiving) Email to Fossil + +This task involves designing a new feature and working with Fossil developers to +see how it can be feasible in practice. + +Fossil can [send email alerts](./alerts.md), but cannot receive email at all. +That is a good thing, because a complete [SMTP +MTA](https://en.wikipedia.org/wiki/MTA) is complicated and requires constant +maintenance, so Fossil should not try to be an MTA or ever listen to mail ports +on the Internet. + +There is one specific type of email reception that make sense for Fossil to +handle. When there is inbound mail related to a message that Fossil has +previously generated with a unique hash, Fossil already knows the context of +that message. An unknown sender cannot guess a valid hash although a malicious +sender could of course find a way to receive a valid hash and then use that to +gain access. The risk of automatic and non-specific spam is very low. + +A proposal to handle that would be to implement a Fossil command like this: + +``` +fossil email -R repo receive -t TYPE-OF-EMAIL -h HASH +``` + +Where the type of email would be one of a list something like this: + +* mail_bounce +* ticket_reply +* forum_reply + +This command is a non-network-aware [Mail Delivery +Agent](https://en.wikipedia.org/wiki/Mail_delivery_agent), and would be called +by an SMTP MTA such as Postfix, Courier or Exim. The MTA would need to be +configured to recognise that this is an email intended for Fossil, and what +type of email, and to extract its hash. People who configure MTAs are used to +doing this sort of thing, but no doubt Fossil would include a sample +[Postfix mail filter](http://www.postfix.org/FILTER_README.html#simple_filter) and +an equivalent driver for Exim. + +The Fossil command would reject anything that doesn't look like a bounce it is expecting. + +It is not certain that this design is the best one to address the inbound mail +problem. That is why the first part of this task is to find a workable design. + +# Work relating to the ticketing system in Fossil + +The Fossil SCM project uses tickets in a [somewhat unusual manner](https://fossil-scm.org/home/reportlist) +because the social programming +model has evolved to often use the Forum instead of ticketing. Other Fossil-using projects +use tickets in a more traditional report-a-bug manner. So this means that the +Fossil ticketing system user interface is underdeveloped. + +On the other hand, pretty much every software developer uses a ticketing system +at some point in their workflow, and Fossil is intended to be usable by most +developers. That means the ticketing system really needs to be further +developed. The underlying technology for the Fossil ticketing system is +guaranteed, so to improve it requires only user interface changes. + +Projects relating to the ticketing system include: + +* Improving the [Fossil cli for tickets](https://fossil-scm.org/forum/forumpost/d8e8a1cf92) which is confusing, as pointed out in that ticket. This is still classified as a "user interface" even though it isn't graphical. +* Alternatively, instead of improving Fossil's cli, implement a comprehensive ticket commandline with [libfossil's primitives](https://fossil.wanderinghorse.net/r/libfossil/wiki/home), look under the f-apps/ directory. +* Improving the Fossil web UI for ticketing, which is clunky to say the least. Fossil tries not be a heavy user of Javascript and Javascript libraries, but the wikiedit, chat and Forum code are all more advanced than ticketing, +and have UI features that would improve ticketing +* If there is an inbound email system as per the previous section "Adding Inbound (Receiving) Email to Fossil", then implement this system for ticketing + +# Tasks Requiring Fossil Data Model Knowledge + +The Fossil data model concepts are simple, but the implications are quite subtle and impressive. The data model +is designed to [endure for centuries](./fileformat.wiki), +be [easily accessible](./fossil-v-git.wiki#durable), and is [non-relational](./fossil-is-not-relational.md). +You will need to understand the data model to work on the following tasks: + +* Add the ability to tag non-checkin artifacts, something supported by + the data model but not the current CLI and UIs. This would open the + door to numerous new features, such as "sticky" forum posts and + per-file extended attributes. This could also relate to the RBAC + system. +* Implement "merge" and "stash" in libfossil +* Analyse the different kinds of [split/export/shallow clone](https://fossil-scm.org/forum/forumpost/1aa4f8ea8c6f96) use cases for Fossil including [complete bifurcation](https://fossil-scm.org/forum/forumpost/6434a06871). There are many proposals, relating to many different use cases, and a good analysis would help us to work out what should be implemented, and what should be implemented in Fossil and what is instead a libfossil wrapper + +# Fossil is cool + +There are many reasons why Fossil is just plain cool: + +* Fossil is symbiotically connected with [SQL and SQLite](5631123d66d96) +* Fossil is highly portable accross different operating systems +* Fossil is the [only credible alternative to Git](./fossil-v-git.wiki) +* Fossil is both ultra-long-term stable and has a high rate of development and new features +* Fossil has thought deeply about Comp Sci principles including [CAP Theorem](./cap-theorem.md) and [whether Fossil is a blockchain](./blockchain.md) +* Fossil has two independent implementations of the same data model: Fossil and libfossil + +and a lot, lot more, in the source, docs, forum and more. + + + + +``` pikchr center toggle +// Click to see the rendered diagram this describes, +// written in Fossil's built-in pikchr language, see https://pikchr.org +// +// based on pikchr script by Kees Nuyt, licensed +// https://creativecommons.org/licenses/by-nc-sa/4.0/ + +scale = 1.0 +eh = 0.5cm +ew = 0.2cm +ed = 2 * eh +er = 0.4cm +lws = 4.0cm +lwm = lws + er +lwl = lwm + er + +ellipse height eh width ew fill Bisque color CadetBlue +L1: line width lwl from last ellipse.n +line "click for" bold above width lwm from last ellipse.s +LV: line height eh down + +move right er down ed from last ellipse.n +ellipse height eh width ew fill Bisque color CadetBlue +L3: line "example of Fossil" bold width lws right from last ellipse.n to LV.end then down eh right ew +line width lwm right from last ellipse.s then to LV.start + +move right er down ed from last ellipse.n +ellipse height eh width ew fill Bisque color CadetBlue +line width lwl right from last ellipse.n then to L1.end +line "coolness" bold width lwl right from last ellipse.s then up eh + +``` + + diff --git a/www/hacker-howto.wiki b/www/hacker-howto.wiki new file mode 100644 index 0000000..72d0105 --- /dev/null +++ b/www/hacker-howto.wiki @@ -0,0 +1,17 @@ +<title>Fossil Developer How-To</title> + +The following links are of interest to programmers who want to modify +or enhance Fossil itself. Ordinary users can safely ignore this information. + + * [./build.wiki | How To Compile And Install Fossil] + * [./customskin.md | Theming Fossil] + * [./adding_code.wiki#newcmd | Adding New Commands To Fossil] + * [./makefile.wiki | The Fossil Build Process] + * [./tech_overview.wiki | A Technical Overview of Fossil] + * [./contribute.wiki|Contributing Code Or Enhancements To The Fossil Project] + * [./fileformat.wiki|Fossil Artifact File Format] + * [./sync.wiki|The Sync Protocol] + * [./style.wiki | Coding Style Guidelines] + * [./checkin.wiki | Pre-checkin Checklist] + * [../test/release-checklist.wiki | Release Checklist] + * [./backoffice.md | The "backoffice" subsystem] diff --git a/www/hashes.md b/www/hashes.md new file mode 100644 index 0000000..548e236 --- /dev/null +++ b/www/hashes.md @@ -0,0 +1,154 @@ +# Hashes: Fossil Artifact Identification + +All artifacts in Fossil are identified by a unique hash, currently using +[the SHA3 algorithm by default][hpol], but historically using the SHA1 +algorithm: + +| Algorithm | Raw Bits | Hexadecimal digits | +|-----------|----------|--------------------| +| SHA3-256 | 256 | 64 | +| SHA1 | 160 | 40 | + +There are many types of artifacts in Fossil: commits (a.k.a. check-ins), +tickets, ticket comments, wiki articles, forum postings, file data +belonging to check-ins, etc. ([More info...](./concepts.wiki#artifacts)). + +There is a loose hierarchy of terms used instead of “hash” in various +parts of the Fossil UI, which we cover in the sections below. + + +## Names + +Several Fossil interfaces accept [a wide variety of check-in +names][cin]: commit artifact hashes, ISO8601 date strings, branch names, +etc. Fossil interfaces that accept any of these options usually +document the parameter as “NAME”, so we will use that form to refer to +this specialized use. + +Artifact hashes are only one of many different types of NAME. We use +the broad term “NAME” to refer to the whole class of options. We use +more specific terms when we mean one particular type of NAME. + + +## Versions + +When an artifact hash refers to a specific commit, Fossil sometimes +calls it a “VERSION,” a “commit ID,” or a “check-in ID.” +We may eventually settle on one of these terms, but all three are +currently in common use within Fossil’s docs, UI, and programming +interfaces. + +A VERSION is a specific type of artifact hash, distinct +from, let us say, a wiki article artifact hash. + +A unique prefix of a VERSION hash is itself a VERSION. That is, if your +repository has exactly one commit artifact with a hash prefix of +“abc123”, then that is a valid version string as long as it remains +unambiguous. + + + +## <a id="uvh"></a>UUIDs + +Fossil uses the term “UUID” as a short alias for “artifact hash” in its +internals. There are a few places where this leaks out into external +interfaces, which we cover in the sections below. Going forward, we +prefer one of the terms above in public interfaces instead. + +Whether this short alias is correct is debateable. + +One argument is that since "UUID" is an acronym for “Universally Unique +Identifier,” and both SHA1 and SHA3-256 are larger and stronger than the +128-bit algorithms used by “proper” UUIDs, Fossil artifact hashes are +*more universally unique*. It is therefore quibbling to say that Fossil +UUIDs are not actually UUIDs. One wag suggested that Fossil artifact +hashes be called MUIDs: multiversally unique IDs. + +The common counterargument is that the acronym “UUID” was created for [a +particular type of universally-unique ID][uuid], with particular ASCII +and bitfield formats, and with particular meaning given to certain of +its bits. In that sense, no Fossil “UUID” can be used as a proper UUID. + +Be warned: attempting to advance the second position on the Fossil +discussion forum will get you nowhere at this late date. We’ve had the +debates, we’ve done the engineering, and we’ve made our evaluation. It’s +a settled matter: internally within Fossil, “UUID” is defined as in this +section’s leading paragraph. + +To those who remain unconvinced, “fixing” this would require touching +almost every source code file in Fossil in a total of about a thousand +separate locations. (Not exaggeration, actual data.) This would be a +massive undertaking simply to deal with a small matter of terminology, +with a high risk of creating bugs and downstream incompatibilities. +Therefore, we are highly unlikely to change this ourselves, and we are +also unlikely to accept a patch that attempts to fix it. + + +### Repository DB Schema + +The primary place where you find "UUID" in Fossil is in the `blob.uuid` +table column, in code dealing with that column, and in code manipulating +*other* data that *refers* to that column. This is a key lookup column +in the most important Fossil DB table, so it influences broad swaths of +the Fossil internals. + +For example, C code that refers to SQL result data on `blob.uuid` +usually calls the variable `zUuid`. That value may then be inserted into +a table like `ticket.tkt_uuid`, creating a reference back to +`blob.uuid`, and then be passed to a function like `uuid_to_rid()`. +There is no point renaming a single one of these in isolation: it would +create needless terminology conflicts, making the code hard to read and +understand, risking the creation of new bugs. + +You may have local SQL code that digs into the repository DB using these +column names. While you may rest easy, assured now that we are highly +unlikely to ever rename these columns, the Fossil repository DB schema +is not considered an external user interface, and internal interfaces +are subject to change at any time. We suggest switching to a more stable +API: [the JSON API][japi], [`timeline.rss`][trss], [TH1][th1], etc. + + +### TH1 Scripting Interfaces + +Some [TH1][th1] interfaces expose Fossil internals flowing from +`blob.uuid`, so “UUID” is a short alias for “artifact hash” in TH1. For +example, the `$tkt_uuid` variable — available when [customizing +the ticket system][ctkt] — is a ticket artifact hash, exposing the +`ticket.tkt_uuid` column, which has a SQL relation to `blob.uuid`. + +TH1 is a longstanding public programming interface. We cannot rename its +interfaces without breaking existing TH1 Fossil customizations. We are +also unlikely to provide a parallel set of variables with “better” +names, since that would create a mismatch with respect to the internals +they expose, creating a different sort of developer confusion in its +place. + + +### JSON API Parameters and Outputs + +[The JSON API][japi] frequently uses the term “UUID” in the same sort of way, +most commonly in [artifact][jart] and [timeline][jtim] APIs. As with +TH1, we can’t change this without breaking code that uses the JSON +API as originally designed, so we take the same stance. + + +### `manifest.uuid` + +If you have [the `manifest` setting][mset] enabled, Fossil writes a file +called `manifest.uuid` at the root of the check-out tree containing the +commit hash for the current checked-out version. Because this is a +public interface that existing code depends on, we are unwilling to +rename the file. + + +[cin]: ./checkin_names.wiki +[ctkt]: ./custom_ticket.wiki +[hpol]: ./hashpolicy.wiki +[japi]: ./json-api/ +[jart]: ./json-api/api-artifact.md +[jtim]: ./json-api/api-timeline.md +[mset]: /help/manifest +[th1]: ./th1.md +[trss]: /help/www/timeline.rss +[tvb]: ./branching.wiki +[uuid]: https://en.wikipedia.org/wiki/Universally_unique_identifier diff --git a/www/hashpolicy.wiki b/www/hashpolicy.wiki new file mode 100644 index 0000000..58a1396 --- /dev/null +++ b/www/hashpolicy.wiki @@ -0,0 +1,199 @@ +<title>Hash Policy</title> + +<h2>Executive Summary</h2> + +<b>Or: How To Avoid Reading This Article</b> + +There was much angst over the [http://www.shattered.io|SHAttered attack] +against SHA1 when it was announced in early 2017. If you are concerned +about this and its implications for Fossil, simply +[./quickstart.wiki#install|upgrade to Fossil 2.1 or later], and the +problem will go away. Everything will continue to work as before. + + * Legacy repositories will continue working just as + they always have, without any conversions or upgrades. + * Historical check-ins will keep their same historical + SHA1 names. + * New check-ins will get more secure SHA3-256 hash names. + * Everything will continue as if nothing happened. + * Your workflow will be unchanged. + +But if you are curious and want a deeper understanding of what is +going on, read on... + + +<h2>Introduction</h2> + +The first snapshot-based distributed version control system +was [http://www.monotone.ca|Monotone]. Many of the ideas behind the design +of Fossil were copied from Monotone, including the use of a SHA1 hash to +assign names to artifacts. Git and Mercurial did the same thing. + +The SHA1 hash algorithm is used only to create names for artifacts in Fossil +(and in Git, Mercurial, and Monotone). It is not used for security. +Nevertheless, when the [http://www.shattered.io|SHAttered attack] found +two different PDF files with the same SHA1 hash, many users learned that +"SHA1 is broken". They see that Fossil (and Git, Mercurial, and Monotone) +use SHA1 and they therefore conclude that "Fossil is broken". This is +not true, but it is a public relations problem. So the decision +was made to migrate Fossil away from SHA1. + +This article describes how that migration is occurring. + +<h2>Use Of Hardened SHA1</h2> + +In Fossil version 2.0 ([/timeline?c=version-2.0|2017-03-03]), +the internal SHA1 implementation was changed from a generic +FIPS PUB 180-4 SHA1 implementation to a "Hardened SHA1" +[[https://github.com/cr-marcstevens/sha1collisiondetection|1]] +[[https://marc-stevens.nl/research/papers/C13-S.pdf|2]]. + +The Hardened SHA1 algorithm automatically detects when the artifact +being hashed is specifically designed to exploit the known weaknesses +in the SHA1 algorithm, and when it detects such an attack it changes +the hash algorithm (by increasing the number of rounds in the compression +function) to make the algorithm secure again. If the attack detection +gets a false-positive, that means that Hardened SHA1 will get a different +answer than the standard FIPS PUB 180-4 SHA1, but the creators of +Hardened SHA1 (see the second paper +[[https://marc-stevens.nl/research/papers/C13-S.pdf|2]]) +report that the probability of a false-positive is vanishingly small - +less than 1 false positive out of 10<sup><font size=1>27</font></sup> +hashes. + +Hardened SHA1 is slower (and a lot bigger) but Fossil does not do that +much hashing, so performance is not really an issue. + +All versions of Fossil moving forward will use Hardened SHA1. So if +someone says "SHA1 is broken, and Fossil uses SHA1, therefore Fossil is +broken," you can rebut the argument by pointing out that Fossil uses +<em>Hardened SHA1</em>, not generic SHA1, and Hardened SHA1 is <em>not</em> +broken. + +<h2>Support For SHA3-256</h2> + +Prior to Fossil version 2.0 ([/timeline?c=version-2.0|2017-03-03]), +all artifacts in all Fossil repositories were named +by only a SHA1 hash. +Version 2.0 extended the [./fileformat.wiki|Fossil file format] +to allow artifacts to be named by either SHA1 or SHA3-256 hashes. +(SHA3-256 is the only variant of SHA3 that +Fossil uses for artifact naming, so for the remainder of this article +it will be called simply "SHA3". Similarly, "Hardened SHA1" will +be shortened to "SHA1" in the remaining text.) + +To be clear: Fossil (version 2.0 and later) +allows the SHA1 and SHA3 hashes to be mixed within +the same repository. Older check-ins, created years ago, +continue to be named using their legacy SHA1 hashes while +newer check-ins are named using modern SHA3 hashes. There +is no need to "convert" a repository from SHA1 over to SHA3. +You can see this in Fossil itself. The recent +[9d9ef82234f63758] check-in uses a SHA3 hash whereas the older +[1669115ab9d05c18] check-in uses a SHA1 hash. + +Other than permitting the use of SHA3 in addition to SHA1, there +were no file format changes in Fossil version 2.0 relative +to the previous version 1.37. Both Fossil 2.0 and Fossil 1.37 read +and write all the same repositories and sync with one another, as long +as none of the repositories contain artifacts named using SHA3. If +a repository does contain artifacts named using SHA3, Fossil 1.37 will +not know how to interpret those artifacts and will generate various warnings +and errors. + +<h2>How Fossil Decides Which Hash Algorithm To Use For New Artifacts</h2> + +If newer versions of Fossil are able to use either SHA1 or SHA3 to +name artifacts, which hash algorithm is actually used? That question +is answered by the "hash policy". These are the supported hash policies: + +<table cellpadding=10> +<tr> +<td valign='top'>sha1</td> +<td>Name all new artifacts using the (Hardened) SHA1 hash algorithm.</td> +</tr> +<tr> +<td valign='top'>auto</td> +<td>Name new artifacts using the SHA1 hash algorithm, but if any +artifacts are encountered which are already named using SHA3, then +automatically switch the hash policy to "sha3"</td> +</tr> +<tr> +<td valign='top'>sha3</td> +<td>Name new artifacts using the SHA3 hash algorithm if the artifact +does not already have a SHA1 name. If the artifact already has a SHA1 +name, then continue to use the older SHA1 name. Use SHA3 for new +artifacts that have never before been encountered.</td> +</tr> +<tr> +<td valign='top'>sha3-only</td> +<td>Name new artifacts using the SHA3 hash algorithm even if the artifact +already has a SHA1 name. In other words, force the use of SHA3. This can +cause some artifacts to be added to the repository twice, once under their +SHA1 name and again under their SHA3 name, but delta compression will +prevent that from causing repository size problems.</td> +</tr> +<tr> +<td valign='top'>shun-sha1</td> +<td>Like "sha3-only" but at this level do not accept a push of SHA1-named +artifacts. If another Fossil instance tries to push a SHA1-named artifact, +that artifact is discarded and ignored. +</tr> +</table> + +For Fossil 2.0, and obviously also for Fossil 1.37 and before, the +only hash policy supported was the one now called "sha1", meaning that +all new artifacts were named +using a SHA1 hash. +Even though Fossil 2.0 added the capability of understanding SHA3 hashes, it +never actually generates any SHA3 hashes. + +From Fossil 2.1 through 2.9, the default hash policy for legacy repositories +changed to "auto", meaning that +Fossil continued to generate only SHA1 hashes until it +encountered one artifact with a SHA3 hash. Once those older versions of +Fossil saw a single SHA3 hash, they +automatically switched to "sha3" mode and thereafter generated +only SHA3 hashes. + +When a new repository is created by cloning, the hash policy is copied +from the parent. + +For new repositories created using the +[/help/new|fossil new] command the default hash policy is "sha3". +That means new repositories +will normally hold nothing except SHA3 hashes. The hash policy for new +repositories can be overridden using the "--sha1" option to the +"fossil new" command. + +If you are still on Fossil 2.1 through 2.9 but you want Fossil to go +ahead and start using SHA3 hashes, change the hash policy to +"sha3" using a command like this: + +<verbatim> +fossil hash-policy sha3 +</verbatim> + +The next check-in will use a SHA3 hash, so that when that check-in is pushed +to colleagues, their clones will include the new SHA3-named artifact, so +their local Fossil instances will automatically convert their clones to +"sha3" mode as well. + +Of course, if some members of your team stubbornly refuse to upgrade past +Fossil 1.37, you should avoid changing the hash policy and creating +artifacts with SHA3 names, because once you do that your recalcitrant +coworkers will no longer be able to collaborate. + +<h2>A Pure SHA3 Future</h2> + +Fossil 2.10 changed the default hash policy to "sha3" mode even for +legacy repositories, so if you +upgrade to the latest version of Fossil, all of your new artifacts will +use a SHA3 hash. Legacy SHA1 artifacts continue to use their original +names, but new artifacts will use SHA3 names. You might not even notice +this automatic change over to stronger hashes. + +We decided to make the change to pure SHA3 since the last known distributor +of Fossil 1.x binaries — Debian 9 — was finally replaced in June 2019 +by Debian 10, which included Fossil 2.8. All other known sources of +Fossil 1.x binaries upgraded well before that point. diff --git a/www/hints.wiki b/www/hints.wiki new file mode 100644 index 0000000..63d6f0a --- /dev/null +++ b/www/hints.wiki @@ -0,0 +1,74 @@ +<title>Fossil Tips And Usage Hints</title> + +A collection of useful hints and tricks in no particular order: + + 1. Click on two nodes of any timeline graph in succession + to see a diff between the two versions. + + 2. Add the "--tk" option to "[/help/diff | fossil diff]" commands + to get a pop-up + window containing a complete side-by-side diff. (NB: The pop-up + window is run as a separate Tcl/Tk process, so you will need to + have Tcl/Tk installed on your machine for this to work. Visit + [http://www.activestate.com/activetcl] for a quick download of + Tcl/Tk if you do not already have it on your system.) + + 3. The "[/help/clean | fossil clean -x]" command is a great + alternative to "make clean". You can use "[/help/clean | fossil clean -f]" + as a slightly safer alternative if the "ignore-glob" setting is + not set. WARNING: make sure you did a "fossil add" for all source-files + you plan to commit, otherwise those files will be deleted without warning. + + 4. Use "[/help/all | fossil all changes]" to look for any uncommitted + edits in any of your Fossil projects. Use + "[/help/all | fossil all pull]" on your laptop + prior to going off network (for example, on a long plane ride) + to make sure you have all the latest content locally. Then run + "[/help/all|fossil all push]" when you get back online to upload + your changes. + + 5. To see an entire timeline, type "all" into the "Max:" entry box. + + 6. You can manually add a "c=CHECKIN" query parameter to the timeline + URL to get a snapshot of what was going on about the time of some + check-in. The "CHECKIN" can be + [./checkin_names.wiki | any valid check-in or version name], including + tags, branch names, and dates. For example, to see what was going + on in the Fossil repository on 2008-01-01, visit + [/timeline?c=2008-01-01]. + + 7. Further to the previous two hints, there are lots of query parameters + that you can add to timeline pages. The available query parameters + are tersely documented [/help/www/timeline | here]. + + 8. You can run "[/help/xdiff | fossil xdiff --tk $file1 $file2]" + to get a Tk pop-up window with side-by-side diffs of two files, even if + neither of the two files is part of any Fossil repository. Note that + this command is "xdiff", not "diff". Change <nobr>--tk</nobr> to + <nobr>--by</nobr> to see the diff in your web browser. + + 9. On web pages showing the content of a file (for example + [/artifact/c7dd1de9f]) you can manually + add a query parameter of the form "ln=FROM,TO" to the URL that + will cause the range of lines indicated to be highlighted. This + is useful in pointing out a few lines of code using a hyperlink + in an email or text message. Example: + [/artifact/c7dd1de9f?ln=28,30]. + Adding the "ln" query parameter without any argument simply turns + on line numbers. This feature only works right with files with + a mimetype of text/plain, of course. + + 10. When editing documentation to be checked in as managed files, you can + preview what the documentation will look like by using the special + "ckout" branch name in the "doc" URL while running "fossil ui". + See the [./embeddeddoc.wiki | embedded documentation] for details. + + 11. Use the "[/help/ui|fossil ui /]" command to bring up a menu of + all of your local Fossil repositories in your web browser. + + 12. If you have a bunch of Fossil repositories living on a remote machine + that you are able to access using ssh using a command like + "ssh login@remote", then you can bring up a user interface for all + those remote repositories using the command: + "[/help/ui|fossil ui login@remote:/]". This works by tunneling + all HTTP traffic through SSH to the remote machine. diff --git a/www/history.md b/www/history.md new file mode 100644 index 0000000..c3df911 --- /dev/null +++ b/www/history.md @@ -0,0 +1,95 @@ +# The History And Purpose Of Fossil + +Fossil is a [distributed version control system (DVCS)][100] written +beginning in [2007][105] by the [architect of SQLite][110] for the +purpose of managing the [SQLite project][115]. + +[100]: https://en.wikipedia.org/wiki/Distributed_version_control +[105]: /timeline?a=1970-01-01&n1=10 +[110]: https://sqlite.org/crew.html +[115]: https://sqlite.org/ + +Though Fossil was originally written specifically to support SQLite, +it is now also used by countless other projects. The SQLite architect (drh) +is still the top committer to Fossil, but there are also +[many other contributors][120]. + +[120]: /reports?type=ci&view=byuser + +## History + +The SQLite project started out using [CVS][300], as CVS was the most +commonly used version control system in that era (circa 2000). CVS +was an amazing version control system for its day in that it allowed +multiple developers to be editing the same file at the same time. + +[300]: https://en.wikipedia.org/wiki/Concurrent_Versions_System + +Though innovative and much loved in its time, CVS was not without problems. +Among those was a lack of visibility into the project history and the +lack of integrated bug tracking. To address these deficiencies, +the SQLite author developed the [CVSTrac][305] wrapper for CVS beginning +in [2002][310]. + +[305]: http://cvstrac.org/ +[310]: http://cvstrac.org/fossil/timeline?a=19700101&n1=10 + +CVSTrac greatly improved the usability of CVS and was adopted by +other projects. CVSTrac also [inspired the design][315] of [Trac][320], +which was a similar system that was (and is) far more widely used. + +[315]: https://trac.edgewall.org/wiki/TracHistory +[320]: https://trac.edgewall.org/ + +Historians can see the influence of CVSTrac on the development of +SQLite. [Early SQLite check-ins][325] that happened before CVSTrac +often had a check-in comment that was just a "smiley". +That was not an unreasonable check-in comment, as check-in comments +were scarcely seen and of questionable utility in raw CVS. CVSTrac +changed that, making check-in comments more visible and more useful. +The SQLite developers reacted by creating [better check-in comments][330]. + +[325]: https://sqlite.org/src/timeline?a=19700101&n1=10 +[330]: https://sqlite.org/src/timeline?c=20030101&n1=10&nd + +At about this same time, the [Monotone][335] system appeared. +Monotone was one of the first distributed version control systems. As far as +this author is aware, Monotone was the first VCS to make use of +SHA1 to identify artifacts. Monotone stored its content in an SQLite +database, which is what brought it to the attention of the SQLite architect. +These design choices were a major source of inspiration for Fossil. + +[335]: https://www.monotone.ca/ + +Beginning around 2005, the need for a better version control system +for SQLite began to become evident. The SQLite architect looked +around for a suitable replacement. Monotone, Git, and Mercurial were +all considered. But at that time, none of these supported sync +over ordinary HTTP, none could be run from an inexpensive shell +account on a leased server (this was before the widespread availability +of affordable virtual private servers), and none of them supported anything +resembling the wiki and ticket features of CVSTrac that had been +found to be so useful. And so, the SQLite architect began writing +his own DVCS. + +Early prototypes were done in [TCL][340]. As experiments proceeded, +however, it was found that the low-level byte manipulates needed for +things like delta compression and computing diffs +were better implemented in plain old C. +Experiments continued. Finally, a prototype capable of self-hosting +was devised on [2007-07-16][345]. + +[340]: https://www.tcl.tk/ +[345]: https://fossil-scm.org/fossil/timeline?c=200707211410&n1=10 + +The first project hosted by Fossil was Fossil itself. After a +few months of development work, the code was considered stable enough +to begin hosting the [SQLite documentation repository][350] which was +split off from the main SQLite CVS repository on [2007-11-12][355]. +After two years of development work on Fossil, the +SQLite source code itself was transferred to Fossil on +[2009-08-11][360]. + +[350]: https://www.sqlite.org/docsrc/doc/trunk/README.md +[355]: https://www.sqlite.org/docsrc/timeline?c=200711120345&n1=10 +[360]: https://sqlite.org/src/timeline?c=b0848925babde524&n1=12&y=ci diff --git a/www/hooks.md b/www/hooks.md new file mode 100644 index 0000000..1173a1c --- /dev/null +++ b/www/hooks.md @@ -0,0 +1,162 @@ +# Hooks + +Hooks are short scripts that Fossil runs at defined points of processing. +Administrators can use hooks to help enforce policy or connect Fossil to +a continuous integration (CI) system. + +## Interim Documentation. + + * This is a work-in-progress. The interface is in flux. + For the time being, the documentation is a list of + bullet points. We hope to transform this into a proper document + later, after things settle down. + + * Contributions and suggestions to the hook system and/or the + documentation are welcomed. + +## General Notes. + + * Each hooks has a "type", a "sequence", and a "command". The command + is a shell command that runs at the appropriate time. The type + is currently one of "after-receive", "before-commit", "commit-msg", + or "disabled". The sequence is an arbitrary integer. + + * There can be multiple hooks of the same type. When that is the + case, the hooks are run in order of ascending sequence. + + * Use the "fossil hook" command to create, edit, and delete hooks. + + * Use the "fossil hook test" command to test new hooks. + +## Hook Scripts + + * All scripts are expected to run relatively quickly. If a long-running + process is started by a hook, it should be run in the background so + that the original script can return. + + * The "%F" sequence inside the script is translated into the + name of the fossil executable. + + * The "%R" sequence in the script is translated into the name of + the repository. + + * The "%A" sequence becomes the name of an auxiliary input file, + the meaning of which depends on the hook type. The auxiliary filename + might be an empty string. Take care to use appropriate quoting! + +## Disabled Hooks + + * Hooks with type "disabled" never run. They are a place-holder for + scripts that might be converted to some other hook-type later. + For example, the command "fossil hook edit --type disabled ID" + can be used to temporarily disable the hook named ID, and then + "fossil hook edit --type after-receive ID" can be used to reenable + it later. + +## After-Receive Hooks + + * The "after-receive" hook is run by [the backoffice](./backoffice.md) + whenever new artifacts are received into the repository. The artifacts + have already been committed and so there is nothing that the + after-receive hook can do to block them. + + * The after-receive hooks are intended to be run on a server to start + up a background testing or CI process. But they can also be run + on the client side. The key point is that after-receive hooks are + invoked by backoffice, so backoffice must be running in order to + fire after-receive hooks. + + * The exit code from the after-receive script is ignored. + + * The standard input to the after-receive hook is a list of + new artifacts, one per line. The first token on each line is the + hash of the new artifact. After the hash is a human-readable text + description of what the artifact represents. + + * Sometimes the same artifact can represent two or more things. + For example, the same artifact might represent two or more files + in the check-out (assuming the files hold identical content). In + that case, the text description that is input to the after-receive + hook only shows one of the possible uses for the artifact. + + * If two or more pushes occur against a repository at about the same + time, then the set of artifacts added by both pushes might be + combined into a single after-receive callback. + + * Fossil holds a write transaction on the repository while the + after-receive hook is running. If the script needs to access the + database, then the database will need to be in WAL mode so that + readers can co-exist with the writer. Or the script might just + launch a background process that waits until the hook script finishes + and the transaction commits before it tries to access the repository + database. + + * A push might not deliver all of the artifacts for a checkin. If + Fossil knows that a /xfer HTTP request is incomplete, it will defer + running the after-receive push for 60 seconds, or until a complete + /xfer request is received. This helps to prevent after-receive hooks + from running when incomplete checkins exist in the repository, but + it does not provide hard guarantees, as there is no way to do that + in a distributed system. + + * The list of artifacts delivered to standard input of the + after-receive hook will not contain more than 24-hours worth + of artifacts. If the backoffice has been shut down for a while + such that after-receive hooks have not been running, and more + than 24-hours of changes have accumulated since the last run + of an after-receive hook, then only the most recent 24-hours + is included in the input. + +## Before-Commit Hooks + + * Before-commit hooks run during the "fossil commit" command before + the user is prompted for the check-in comment. Fossil holds + a write-transaction on the repository when the before-commit + hook is running, so the repository needs to be in WAL mode if the + script needs to access the repository. + + * The %A substitution is the name of a "commit description file" that + shows the details of the commit in progress. To see what a + "commit description file" looks like, set a before-commit hook + with a command of "cat %A" and then run a sample commit with + the --dry-run option. + + * If any before-commit hook returns a non-zero exit code, then + the commit is abandoned. All + before-commit hooks must exit(0) in order for the commit to + proceed. + + * The --no-validate flag to the "fossil commit" command prevents any + before-commit hooks from running. + + * The --trace flag to the "fossil commit" command shows each + before-commit hook as it is run. + + * If a before-commit hook fails, it should print an error message + on standard output or standard error. Otherwise, the user won't + know what went wrong, because Fossil won't tell them. + + * Nothing is written to standard input of the before-commit hook. + The information transmitted to the before-commit hook is contained + in the "%A" auxiliary file. The before-commit hook must open and + read that file if it wants access to the commit information. + +## Commit-Msg Hooks + + * Commit-msg hooks are not yet implemented. + + * The commit-msg hooks run during "fossil commit" after the check-in + message has been entered by the user. The "%A" argument to the + commit-msg hook is the text of the commit message. The intent + of the commit-msg hook is to validate the text of the commit + message to (for example) check for typos or ensure that it + conforms to standards. + + * If any commit-msg hook returns a non-zero exit code, then + the commit is abandoned. All + commit-msg hooks must exit(0) in order for the commit to + proceed. + + * Commit-msg hooks are advisory only. Each developer is in total + control of the local repository and can easily bypass the hooks + to cause a non-conforming checkin to be committed. diff --git a/www/hundredandone.md b/www/hundredandone.md new file mode 100644 index 0000000..330cf29 --- /dev/null +++ b/www/hundredandone.md @@ -0,0 +1,1093 @@ +# 101 Reasons Why Fossil Is Better Than Git + + 1. **Fossil comes as a single self-contained executable file**.<p> + Install Fossil by copying "fossil" (or "fossil.exe") to some place + on your $PATH (or %PATH%). Upgrade (or downgrade) by overwriting + that one file. Uninstall by deleting that one file.<p> + A Git installation requires hundreds of files. You want to use a + package manager to install, upgrade, or uninstall Git. If you + tinker with individual files of a Git installation, you run a high + risk of messing things up. + + 1. **Fossil comes with a built-in full-featured web interface**.<p> + The Fossil web interface meets or exceeds the capabilities + of GitHub, GitLab, Gitea, Forgejo, and similar. + Some people think of Fossil as "GitHub-in-a-box". + Git has "gitweb", but that CGI program + is so limited and difficult to set up and use that few people + even know it exists, so I do not count it + for the purposes of this point. + + 1. **You can run the Fossil web interface locally + using the "fossil ui" command.**<p> + Just type the command "fossil ui" from any Fossil checkout, + or add an argument that is the name of a Fossil repository file + or a directory that is the root of an open checkout + and Fossil automatically brings up a new window with the + Fossil web interface in your preferred web browser. + This works seamlessly on all platforms. There is nothing + extra to install. There is no configuration or setup. + It just works. + + 1. **The Fossil web interface shows a graphical timeline of changes.**<p> + See, for example <https://sqlite.org/src/timeline> or + <https://fossil-scm.org/home/timeline>. Similar timeline features + for Git are available from the command-line for a local clone + (for example using "gitk" or other third-party programs), but + nothing with anything close to the capabilities of the Fossil + timeline is available via a web interface, as far as I know.<p> + Using the web interface, one can easily check on the status + of a project when away from the office and without access to a clone + of the repository. + + 1. **The Fossil web interface view of a single check-in shows a + context graph of all other directly connected check-ins.**<p> + See the page for [check-in 59985724d71229bf](/info/59985724d71229bf) + for example. The context graph shows four other check-ins: + two direct descendants, one merge descendant, and one ancestor. + This context graph is useful in understanding how a particular + check-in fits into the history of the project. The context graph + can also be used to step forwards or backwards in time, by + clicking on the "check-in:" hash links for nearby check-ins. + + 1. **The graphical timeline dynamically adjusts its layout as you resize + your browser window.**<p> + The server sends down an HTML page that contains (among other things) + a JSON object that gives the basic structure of the + timeline graph, then Javascript renders the graph. The JS code is + small, does not use any third-party frameworks, and is (by default) + appended to the HTML page. Except for a separate CSS file, the HTML + is completely stand-alone. The local + web browser isn't required to go gather lots of separate resources. + + 1. **The graphical timeline works on a phone.**<p> + The graph layout automatically compresses on a small display, and + can seem a little cramped for a complex project. The timeline + does look better on a desktop. Even so, the timeline display is + functional on a phone and it is very convenient + to be able to see what is happening on a project + while away from the office and without access to a laptop. + + 1. **The graphical timeline is bandwidth efficient.**<p> + To display a timeline of recent activity on Fossil uses less + than 5% of the bandwidth as GitHub. In a typical example, + GitHub requires about 3.5MB of transfer to retrieve 56 different + resources compared to 150KB for just one HTML file and one CSS file + with Fossil. And for all that 3.5MB, GitHub only gives you a list + of recent check-ins without any indication of the branching + structure, whereas Fossil gives you an easy-to-read color-coded + graph. + + 1. **The Fossil web interface makes it easy to see a diff between + any two check-ins with just a couple of clicks.**<p> + On the timeline display, click on one node of the graph to + select it (a red dot will appear in the center of the node) and + then click on any other graph node, and Fossil will compute and + display a diff between those two check-ins. (Usage hint: + click the selected node a second time to deselect it.) + + 1. **The Fossil web-based diff page shows the context of the two + check-ins being diffed.**<p> + At the top of the web-based diff is a graph that shows + specifically the two check-ins being diffed and the context + around them. + [Example](/vdiff?from=052390edaf04a5f5&to=9b686daeeda7b4ca). + This helps to reduce any confusion about what you are looking at. + + 1. **In Fossil, a repository is distinct from a working checkout.**<p> + A Fossil repository can be colocated with the working checkout, as + they are required to be in Git. But most people keep the repository + separate. One common pattern is to put all Fossil repositories in + a single directory named $HOME/Fossils or $HOME/Museum and then open + working checkouts against each repository wherever they are needed. + + 1. **A single Fossil repository can support multiple working checkouts.** + <p> + Git has worktrees, but all worktrees checkouts must be on separate + branches. Fossil allows multiple working checkouts on the same branch + or even on the same check-in. One common pattern is to have one + checkout that is being edited, and another that is a pristine, unedited + version of the same check-in. This allows both to be compiled + simultaneously for performance comparison, or to step through both + binaries in two separate "gdb" sessions to hunt down a bug. + + 1. **A Fossil repository is a single disk file, not a directory + hierarchy.**<p> + You can "mv" a Fossil repository to a new place. You can "scp" + a Fossil repository to another machine. It is just a file, + specifically an SQLite database file. You can name a Fossil + repository anything you like. The usual convention is to give the + repository file a ".fossil" suffix, but that is not required. + + 1. **Fossil does not have a staging area**.</p> + A staging area does not add new capabilities, it only adds + complication. (Partial commits + are accomplished in Fossil simply by listing the subset of files to + be committed on the "fossil commit" command line.) + The staging area complicates the mental model + of the project that Git users need to keep up with, forcing the + developer to spend more effort thinking about the version control + system and hence less time thinking about the project they are + working on. + + 1. **Fossil remembers where all your repositories and working + checkouts are located.**<p> + This and other information (such as all your global settings) + is stored in a per-user database file + at $HOME/.config/fossil.db on unix or in %LOCALAPPDATA%/_fossil on + Windows. Fossil creates and manages that file automatically. + The user never has to know the file even exists. If you + move or rename repositories or checkouts, the database will get + temporarily out of sync with reality, but Fossil will automatically + fix the database the next time you do anything with the file or + checkout that was moved or renamed.<p> + You can get a list of repositories using the + "<tt>fossil all ls</tt>" command, or a list of open + checkouts using "<tt>fossil all ls --ckout</tt>". + + 1. **Fossil lets you bring up a web-based UI that shows all of + your repositories at once.**<p> + Run the command "<tt>fossil ui /</tt>" and your + default web browser will pop up a new tab that lists all of your + repositories together with the + associated project name and how recently that repository was modified. + Click on links to bring up repository-specific web-pages. Or click + on column headers to sort by that column.<p> + If you are like me and have hundreds of repositories on your desktop + system, this feature makes it easier to keep track of them all, or + just to remember what you called each one. + Are you on an infrequently used travel laptop and forgot where you + put a particular repository, this feature helps you find it. + + 1. **Fossil lets you quickly find uncommitted changes across all of + your open checkouts.**<p> + Simply run "<tt>fossil all changes</tt>" to get a quick + summary of every checkout on your local machine that needs a commit. + + 1. **Fossil lets you sync all of the changes in all of your local + repositories to their remotes, with a single command.**<p> + Run "<tt>fossil all sync</tt>" and all your local + repositories will sync up. This is useful, for example when + taking a laptop off-network. Before disconnecting, you sync + all of your repositories. Fossil itself keeps track of all + of your repositories, so you cannot accidentally forget one or two. + While off-network, you might commit changes to one or + more of those repositories. Once you reconnect, you simply + run "<tt>fossil all sync</tt>" again to push out your edits + back to the community. + + 1. **Fossil allows multiple check-ins to have the same tag.**<p> + For example, on the SQLite project, every release is tagged with + "release". + + 1. **The Fossil timeline can show all check-ins with a specific tag.**<p> + For example, to see all SQLite releases visit + <https://sqlite.org/src/timeline?t=release> or to see all + Fossil releases go to + <https://fossil-scm.org/home/timeline?t=release>. + + 1. **Fossil has a built-in wiki**.<p> + Wiki pages are stored in the same repository file as your code, so that + they push, pull, sync, and clone together with your code. + <p> + Git does not itself have a wiki. The various Git forges usually add + wikis, and store the wiki content in Git, though in a separate + Git repository. There is no option to store the wiki and source code + together such that they clone and sync as one, in the Git ecosystem. + + 1. **Fossil can associate a wiki page with a particular check-in.**<p> + Rather than including an oversized check-in comment on an + important check-in (such as the merge of a big new feature), Fossil + allows you to assign a wiki page to that check-in. That wiki page + is shown as part of the check-in information in the web interface. + See, for example, the "About" section of + [SQLite check-in 2019-11-21T18:28:44.463Z](https://sqlite.org/src/info/2019-11-21T18:28:44.463Z). + The wiki page associated with a check-in can be created and revised + after the check-in is committed and pushed. This feature can be used + to add notes or bug reports that occur long after the + check-in itself has been inserted into the DAG. + + 1. **Fossil can associate a wiki page with a particular branch.**<p> + You can create a wiki page for a branch that documents the purpose + of that branch. The wiki page can be displayed separately (like any + other wiki page) but is also displayed automatically at the top of + every timeline for that branch. See, for example, + <https://sqlite.org/src/timeline?r=autosetup>. + + 1. **Fossil keeps track of historical branch names.**<p> + Git does not track branches. Git only names each leaf + of the DAG and infers branches based on the names of leaves. + Fossil actually remembers the name of branches. + For example, suppose a customer asks + "Whatever became of that setlk-snapshot-fix branch you + started last year?" That branch still exists and you can + [see it on GitHub](https://github.com/sqlite/sqlite/commits/setlk-snapshot-fix) + and [in Fossil](https://sqlite.org/src/timeline?r=setlk-snapshot-fix). + Notice, though, that GitHub does not show how the branch was resolved. + (Aside: Also notice how much faster Fossil renders!) + Fossil clearly shows that the branch ended up being merged into trunk. + GitHub just shows us all ancestors of the leaf node labeled + "setlk-snapshot-fix", including ancestors that were in other branches + that got merged in, and ancestors that predate the founding of the + setlk-snapshot-fix branch. + <p> + GitHub is unable to show you the check-ins of branch setlk-snapshot-fix + only. There are other third-party tools that will show you that, I am + told, but they all require a local clone of the repository. Apparently + there is no way to see this information in a web browser using Git, + with or without third-party tools. + + 1. **Fossil has a built-in bug tracking system.**.<p> + Tickets are stored in the repository together with source code and + push/pull/sync the same as the code. The ticket system is configurable + and distributed. Repository administrators can determine what + fields appear in tickets and what users are allowed to create, + edit, or moderate tickets. + + 1. **Fossil allows multiple branches with the same name.**<p> + As an example, the SQLite project has multiple branches + named "experimental" + <https://sqlite.org/src/timeline?r=experimental> and many + others named "mistake" + <https://sqlite.org/src/timeline?r=mistake>. + Another example: + [the empty-table-optimizations branch(es)](https://sqlite.org/src/timeline?r=empty-table-optimizations) + which was initially merged to trunk and closed + on 2025-07-02, but then reopened and continued with more enhancements + until it was merged again on 2025-07-08. + + 1. **Fossil allows both legacy SHA1 hashes and newer SHA3-256 hashes + in the same repository.**<p> + Both Fossil and Git started out using only SHA1 hashes. But when the + [SHAttered attack](https://www.marc-stevens.nl/research/shattered.io/) + against SHA1 was published on 2017-02-23, the need to migrate to a + stronger hash algorithm was recognized. Fossil added the ability + to use SHA3-256 as an alternative on 2017-03-01 (six days after the + SHAttered attack was first published). + SHA3-256 is now the default for all new repositories and check-ins + in Fossil, though older check-ins that occurred + prior to SHAttered can still use their original SHA1 hash. Hence, + no repositories had to be rebuilt and no hyperlinks were broken.<p> + In contrast, after nine years, a Git repository can still only + support one hash algorithm at a time. + Newer Git repositories are able to use SHA2, though the default + is still SHA1. + + 1. **Fossil allows check-ins to be identified by timestamp**<p> + The canonical name for a check-in is its hash. Both Git and Fossil + allow a check-in to be identified by any unique prefix of its hash. + But only Fossil allows a check-in to also be identified by its + timestamp. The names "2026-02-02T16:03:24.852Z" and "fdebbedbd9a99165" + both refer to + [the same check-in](https://sqlite.org/src/info/fdebbedbd9a99165), + but the first one has the advantage of giving some time + context rather than just being a seemingly random sequence of hexadecimal + digits. It is possible that two or more check-ins can have the same + timestamp, in which case the timestamp would be ambiguous. And a + check-in timestamp can be changed after it is committed, by using + a special tag. So timestamp identifiers do not have the uniqueness + and stability guarantees as hash identifiers, but they are available + as an option and are often useful. + + 1. **Fossil has a built-in forum.**<p> + The forum content is replicated via push, pull, sync, and clone just like + source code. Forum posts can be enabled per user or for all users or + for all users and anonymous passers-by. Some users can be appointed + as moderators and posts from untrusted users can be held for + moderation. + <p> + The [Fossil Forum](https://fossil-scm.org/forum), the + [SQLite User Forum](https://sqlite.org/forum), and the + [SQLite Bugs Forum](https://sqlite.org/bugs) are all forums + set up for specific purposes and deliberately kept separate + from source code. But other projects such as + [Pikchr](https://pikchr.org/) comingle their forum and the source code + in the same repository. Thus when you clone the + Pikchr source repository, you also get all the Forum history. + + 1. **Fossil makes it easy to set up a project website using CGI.**<p> + If you have an internet-facing machine running a CGI-capable web + server, you can stand up a complete self-hosting project + website with a two-line CGI script. The canonical + [Fossil website](https://fossil-scm.org/home) is really just such + a CGI script. When you clone the Fossil source code, you don't + just get the code, you get the entire website. + <p> + The CGI script used to run the Fossil website looks + approximately like this: + <pre> + #!/usr/bin/fossil + repository: /Fossils/fossil.fossil</pre> + You can, of course, also create a self-hosting website using + GitLab or Forgejo or similar, but the setup and maintenance is + somewhat more involved. You'll also probably need a bigger machine + if you are using GitLab, whereas a Fossil website + works fine on a 2GiB Raspberry PI or a $6/month VPS. + I am told that Gitea and Forgejo also work well on a small machine. + No direct size and performance comparisons between Gitea/Forgejo and + Fossil have been made, as of this writing. + + 1. **Fossil makes it easy to set up a project website using SCGI.**<p> + CGI is easier, but some web servers (ex: Nginx) do not support CGI. + If you are using such a web server, you can also run a Fossil + server using SCGI. See the + [on-line Fossil SCGI documentation](/doc/trunk/www/scgi.wiki) + for details. + + 1. **Fossil makes it easy to set up a project website behind a + reverse proxy.**<p> + Simply run a [Fossil HTTP server](/doc/trunk/www/server/any/none.md) + and have your reverse proxy redirect requests to this new Fossil + server. You can also set up the Fossil server to work over named + pipes rather than a loopback, if your reverse proxy supports that. + + 1. **Fossil makes it easy to set up a project website without + using any web server at all.**<p> + The [fossil server command](/help/server) includes a --cert + option with which you can specify a TLS cert for encrypted + communication, and with the option, Fossil will accept ordinary + HTTPS requests from the open internet. There is no need to + install and configure a separate web server. Hence, the only + software you need to stand up a project website using Fossil is + the stand-alone "fossil" binary. + + 1. **A single Fossil server is able to host multiple projects.**<p> + By default, each Fossil server provides content for a single + repository. But you can launch a Fossil server that hosts + multiple repositories by putting all those repositories in a + directory and giving the directory name as the object to serve + instead of the repository name. For this mode of operation, + the repositories must be named with the "<tt>.fossil</tt>" file + suffix. Individual projects have URLs that begin with the + repository base name, omitting the "<tt>.fossil</tt>" suffix. + + 1. **Fossil automatically pushes after each commit, by default**.<p> + The [autosync setting](/help/autosync), which defaults to "on", causes + every commit to automatically push to the default remote. This helps + to keep the remote up-to-date and helps all the developers working on + the project keep up with what other developers are doing. + <p> + Long-time Git users might cringe at autosync, thinking that this + could cause problems if another developer commits ahead of you. + True enough, that would cause headaches for Git, but it does not + create problems for Fossil. + The worst that could happen is that the branch will fork. Fossil + will usually detect an impending fork and warn you. + But even if you don't get the warning (due to a race) or even if you + override the warning and force the commit anyhow, a fork on a branch + in Fossil is harmless. It shows cleanly in the timeline and + is easily resolved. So while autosync might cause issues with Git, + it is harmless when using Fossil. Since forks are harmless, + the benefits of autosync far outweigh the risks. + + 1. **Fossil supports embedded Pikchr in Wiki and in the Forum.**</p> + [Pikchr](https://pikchr.org/) is a + [PIC-like](https://en.wikipedia.org/wiki/Pic_language) markup language + for diagrams. Pikchr is designed for use with Markdown, but also works + with other markup languages. + The diagrams that appear in the Fossil documentation are + all drawn using Pikchr. + + 1. **Fossil has a Pikchr sandbox for experimenting with Pikchr scripts.** + <p> + The [pikchrshow page](/pikchrshow) allows users to experiment with and + refine their Pikchr diagrams prior to copy/pasting them into their + documents. + + 1. **The Fossil web interface supports "embedded documentation"**</p> + See the [Project Documentation](/doc/trunk/www/embeddeddoc.wiki) page + for details. Markdown, Wiki, plain-text, and HTML files in the + source tree can be rendered and used as documentation pages. This + is how all of the documentation files for Fossil itself are created. + + 1. **Fossil can easily host an entire project website, using only the + repository as the backing store.**<p> + The embedded documentation and wiki features allow you to write + web pages. The unversioned file feature gives you space to put + precompiled binaries or other transient and/or derived resources + without contaminating the source tree. The Forum and Ticket features + provide for community discussion and bug tracking. Fossil provides + everything you need to host a complete software project website. + Indeed, the + [canonical Fossil website](https://fossil-scm.org/home) is just + an instance of Fossil running on the self-hosting Fossil repository. + If you clone the Fossil self-hosting repository, you don't get just + code - you get the entire website. (Exception: + [Fossil Forum](https://sqlite.org/forum) is hosted separately using + a separate Fossil repository, so you'd actually need to clone that + one too, in order to get the whole website.) + + 1. **Using Fossil, backing up your project website is just a sync.**<p> + If you host your entire project website in a Fossil repository, + as Fossil itself does, then backing up that website is as simple + as creating a clone and keeping the clone synced. + + 1. **Fossil will render uncommitted changes to embedded documentation.** + <p> + Using the "<tt>fossil ui</tt>" and the + [/doc/VERSION/FILE](/help/www/doc) webpage, if the VERSION is the + special keyword "ckout", then the content is taken from the local + checkout rather than from the repository. This allows you to edit + embedded documentation files and then press Reload on your browser + to see how they will look and work on the + actual website without having to commit. + + 1. **Fossil has a wiki sandbox for experimenting with markup.**</p> + The [wiki sandbox](/wikiedit?name=Sandbox) allows users to experiment + with Markdown or other text markup languages supported by Fossil, + without making permanent changes to the repository. + + 1. **Fossil supports interwiki hyperlinks.**<p> + Core Git does not have any kind of markup language. Check-in comments + in Git are always displayed verbatim. But wrappers such as GitHub, + GitLab, Gitea, Forgejo, and similar generally support some variant + of Markdown. However, none of these system support links (apart from + full URLs) to other wiki systems. Fossil does support interwiki links + using the syntax "<tt>remote:path</tt>". So, for example, a check-in + in the SQLite source repository (<https://sqlite.org/src>) that wants + to reference a bug report (perhaps because it fixes the bug) can + include a link of the form "<tt>bugs:/info/</tt><i>HASH</i>" to + reference that bug. The source repository knows that "bugs:" refers + to the [SQLite Bug Forum](https://sqlite.org/bugs) and completes the + link accordingly. + <p> + Interwiki links are important because (1) they help keep hyperlinks + shorter, and (2) if the target wiki + ever moves to a new domain, all the links automatically adjust + by changing a single server setting, and (3) clones can have + different mappings for interwiki links, so that (for example) + a "bugs:" link in a clone of the source repository can map to + a clone of the Bugs Forum repository rather than the canonical + Bugs Forum. + + 1. **Fossil has a built-in chat server.**<p> + Users with appropriate permissions (usually just registered + developers, not anonymous passers-by) can bring up a web-based chat + window on any Fossil web-server instance. This feature allows + geographically distributed developers to collaborate interactively, + without having to involve a third-party chat provider such as Slack. + Nothing needs to be configured in order to activate Chat, other + than enabling the Chat privilege on the permission bits of the + users whom you want to have access to Chat. + + 1. **Fossil chat can be configured to send automatic notifications + when changes occur in the repository.**<p> + This helps developers keep up with what is happening in the repository. + The chat window beeps (or not, configurable individually by each user) + when new messages arrive, as an alert. + + 1. **The Fossil Chat system has hooks that allow external subsystems + to inject chat messages.**<p> + The SQLite developers use this to get notifications of testing + failures from our fuzz testing infrastructure. It could also + be leveraged to get chat notifications of CI/CD problems. + + 1. **Fossil chat is able to send attachments.**<p> + When the SQLite developers are working collaboratively on a problem + (while working, literally, on three different continents) we easily + send patches or diffs to one another over Chat. + + 1. **Fossil supports hyperlinks in check-in comments.**<p> + Check-in comments need not be just verbatim text (though they can + be, depending on repository settings). By default, check-in + comments can contain hyperlinks, including hyperlinks to + wiki pages, prior check-ins, forum posts, and interwiki hyperlinks. + + 1. **Fossil supports hyperlink back references**<p> + If the check-in comment for a newer commit contains a hyperlink + back to an older commit, then when the web interface shows the + details of the older commit, it also provides a forward + reference to the newer commit. + + 1. **Fossil supports a graphical timeline display of a bisect.**<p> + [For example](https://sqlite.org/src/timeline?bid=y2f0bde4bc8-ndfc790f998-ye2634e500c-yff205f2993-y6bb717acf7-nb48d951916-y8364d89c3b-n98a53fb276-y9d68971c58-y498ee8d514-n043ff54fb7-ye33da6d5dc). + This is not strictly necessary to make effective use of bisect, but + the graphical display does seem to help with situational awareness. + + 1. **Fossil can show you the first release in which a particular + check-in appears, with a single mouse click.**<p> + You have to configure the repository by giving it the name of the + tag that you use to mark releases, using the + [path-to-tag setting](/help/path-to-tag). Suppose you use the tag + name "release". Once you do that, then + when you are looking at the "info" page for a check-in, a link + named "path-to-release" appears in the overview section, and if + you click on that link, it brings up a new graph showing the + shortest path from that check-in to the next descendant check-in + tagged with "release". + <p> + Example: On the info page at + <https://sqlite.org/src/info/b1d7123bc619e3cb>, in the Overview + section at the top, to the right of the "Timelines:" label, you + will see the "path-to-release" link. Click that link to take you + to a page showing an abbreviated path from the original check-in + to the first "release" check-in that contains the change. To + see the full path, uncheck the "Brief" checkbox near the top of the page. + <p> + This feature is useful for when you bisect to find a bug, or a bug + fix, and you want to know the first release in which that bug or bug fix + appeared. + + 1. **Fossil allows you to revise a check-in comment without + rewriting history.**<p> + If you find a typo or other error in an historical check-in comment, + you can fix the problem in Fossil without having to rewrite all + subsequent history. The Fossil file format allows you to set + a special tag on the check-in that provides revised comment text. + The new tag causes both the command-line display + and the web interface to show the revised check-in comment rather + that the original. Note that the original check-in comment is + preserved, so there is still an immutable audit trail. But for + common use cases, only the newer revised comment is shown. + <p> + If you are using the web interface and if you have check-in privilege + on the repository, then on the /info page for the check-in, under + the "Overview" section, to the right of "Other Links:", there is + an "edit" link that will take you to a page that lets you change + the check-in comment from the web interface. This is the easiest + way to edit a check-in comment. + <p> + See [Fossil check-in b63d654041](/info/b63d65404) for an + example. The original comment is shown in the "Overview" + section of the check-in details, but the revised comment is shown + in the timeline. + + 1. **Fossil allows you to revise a check-in timestamp without + rewriting history.**<p> + When generating a new check-in, Fossil uses the current time on the + system where the commit is occurring. But if the system clock on + that system is incorrect, that can lead to a check-in with an + inaccurate timestamp. It can be the case that prior check-ins + have later timestamps or that subsequent check-ins can have + earlier timestamps, resulting in goofy-looking "time-warps" in the + timeline. This can be fixed by add a timestamp correction tag + to the faulty check-in to fix the timestamp. + <p> + If you are using the web interface and if you have check-in privilege + on the repository, then on the /info page for the check-in, under + the "Overview" section, to the right of "Other Links:", there is + an "edit" link that will take you to a page that lets you change + the timestamp for a check-in from the web interface. This is the + easiest way to make the change. + + 1. **Fossil allows you to move a check-in to a new branch without + rewriting history.**<p> + If you mistakenly commit to the wrong branch, you can move that + check-in to a new branch by attaching a special tag. + Note, however, that this will also move all subsequent check-ins + to that same new branch. + <p> + In the SQLite and Fossil projects, when developers mistakenly commit + on the wrong branch, the usual way we fix that is to move the + mistaken check-in to a branch named "mistake". Sometimes we also + set the "hidden" tag on that check-in as well, so that it does not + show up on ordinary timelines (though it is still part of the + immutable audit history and is visible with special options). + Then we just cherrypick the check-in onto the correct branch. + <p> + If you are using the web interface and if you have check-in privilege + on the repository, then on the /info page for the check-in, under + the "Overview" section, to the right of "Other Links:", there is + an "edit" link that will take you to a page that lets you move the + check-in to a new branch from the web interface. You can also do this + from the Fossil command-line, but the web interface is easier and less + error prone. + + 1. **Fossil supports unversioned files**.<p> + [Unversioned Files](/doc/trunk/www/unvers.wiki) are files held + in the repository but which are not versioned and which are not + synced by default. Unversioned files are used by Fossil itself + to store [Precompiled Binaries of Fossil](/uv/download.html). + <p> + Unversioned content is not synced *by default*. But unversioned + files will sync if you add the -u option to the + [fossil sync command](/help/sync). + There is also the [fossil uv sync](/help/uv) command. + + 1. **Fossil automatically selects check-in background colors according to + the branch that each check-in occurs on.**<p> + This helps to make the timeline easier to read at a glance, by + clearly showing which check-ins are on which branches. Developers + can assign specific colors to branches either when the branch is + first created, or after the branch has been running for a while. + But experience teaches us that it is better to just let the web + interface pick the branch colors automatically. The color is + derived from a hash of the branch name. + + 1. **The Fossil web interface contains a page that allows you to + preview what colors Fossil will choose for branch names.**<p> + On the [/hash-color-test page](/hash-color-test), one can enter + candidate branch names and see in advance what colors Fossil will + pick for that branch name. This seems like cheating, but I will + admit that I do this myself, sometimes... + + 1. **The Fossil web interface timeline can be asked to pick check-in + colors using the name of the committer, rather than the branch + name.**<p> + Simply add the "ubg" query parameter (mnemonic: User BackGround) and + the check-in colors will be determined by the committer login name + rather than the branch name or any preselected color name. This + results in a timeline that gives the reader a clearer view of who is + making changes. + [Example](/timeline?n=200&y=ci&ubg). + + 1. **Fossil tracks cherrypick merges.**<p> + Cherrypicks are recorded as part of the underlying + [Fossil file format](/doc/trunk/www/fileformat.wiki). + Cherrypicks appear on the timeline as thin dashed lines. + + 1. **Fossil draws arrows pointing forwards in time.**<p> + Forward-pointing arrows are more intuitive than arrows + that point backwards in time, like Git uses. Yes, I am aware + that the underlying implementation of Git has pointers going from child + to parent, and thus must necessarily go backwards in time. Fossil + has the same pointers. But just because the *implementation* + points backwards in time does not mean that the *user interface* + needs to do the same. Fossil flips those pointers around so that + they make more sense from the perspective of the human reader. + + 1. **Fossil implements some unix-like shell commands to use as + substitutes on systems that don't have them or that have inferior + implementations.**<p> + For example the "fossil system ls" command works like the standard + "ls" command on unix. It doesn't support all the options that a typical + "ls" implementation supports on Linux, but it is still way better than + having to run "dir". Other substitute commands include + "date", "pwd", "stty", "unzip", "which", and "zip". Probably + more will be added as needs arise. Having ready access to these + commands built into the standalone Fossil binary makes working on + non-Linux platforms more comfortable for unix geeks, and saves having + to hunt around and install system-specific alternatives. + + 1. **Fossil can copy all uncommitted changes from a checkout on a + remote system over to a checkout on the local machine.**<p> + The command is [fossil patch pull](/help/patch). It contacts the + remote system via SSH, updates its local checkout to the same baseline + as is found on the remote, then pulls over a minimal set of diffs and + applies them.<p> + This is very useful in pre-commit testing. For example, if you have + a big change on your desktop, and you want to test it before + committing, on multiple platforms, you can ssh over to those other + platforms and run "fossil patch pull ... && make test". When working + on SQLite, I will typically do that on a remote Mac, a remote + Win11 machine, and on a 32-core remote Linux machine that runs + faster than my desktop. + + 1. **Fossil can push uncommitted changes to a checkout on another + machine.**<p> + This is the same as the previous but in reverse. It is used, for + example, to push proposed changes up to a secure sandbox to be + reviewed by Claude/Codex/Copilot prior to commit. The sandbox is + not able to pull, for security reasons, but it can accept a push. + + 1. **Fossil lets you set up aliases for remote checkouts with which + you commonly push or pull.**<p> + My desktop is named "r21" and I normally do SQLite development + work in the directory ~/sqlite/sqlite. If I have uncommitted + changes that I want to test on Windows, I SSH over to the Win11 + machine then run commands like:<pre> + fossil patch pull r21:sqlite/sqlite -f + make clean test</pre> + But typing in "r21:sqlite/sqlite" can be tedious and error-prone, + maybe not so much in this particular example but definitely the + case for longer hostnames and subdirectory paths. Fortunately, + Fossil allows us to define patch alias. On my Win11 machine, + the alias named "@" is defined as "r21:sqlite/sqlite" and so I + can get by with typing just:<pre> + fossil patch pull @ -f</pre> + (Aside: the -f option tells the command to first "revert" any + uncommitted changes currently in the checkout prior to pulling + over the new ones from r21:sqlite/sqlite. Without that option, the + changes would be merged.) + + 1. **Fossil allows you to view uncommitted changes on a remote machine + in a web browser over SSH.**<p> + Suppose you have some edits on a remote, headless machine and you + would like to review those changes. Fossil lets you see those + changes using a command like:<pre> + fossil ui remote:path/to/checkout</pre> + Substitute the name of your remote and the path to the checkout + on the remote, of course.<p> + The way this works is that Fossil opens an SSH connection to the + remote machine that runs [fossil server](/help/server) on the + remote and that tunnels the HTTP content back through the SSH connection + to your desktop. At the same time, Fossil brings up your default + web browser and points it to the local end of your SSH tunnel. + That's a lot of network plumbing, but Fossil handles it all + automatically, so that you the developer don't need to think about it. + + 1. **Fossil lets you browse a repository on a remote, headless machine + over an SSH connection.**<p> + Just run "<tt>fossil ui remote:path/to/repository</tt>" (substituting + in the name of the remote machine and the path to the repository you + want to browse, of course) and Fossil will pop up a new page on your + default web browser (on your desktop!) that is connected over an + SSH tunnel to that remote repository. + + 1. **Fossil lets you see all repositories on a remote, headless machine + all at once, over SSH.**<p> + The command is "<tt>fossil ui remote:/</tt>". Substitute the name of + the remote machine, of course. The special pathname "/" indicates to + Fossil that you want to view all of the repositories on that machine. + It causes Fossil to consult the $HOME/.config/fossil.db file + (or %LOCALAPPDATA%/_fossil on Windows) to find the locations of all + repositories on the remote machine, then bring up a web page listing + all those repositories. Links on that initial page let you explore + deeper into the details of each repository. + + 1. **The Fossil web interface supports multiple timeline formats, to + accommodate personal tastes.**<p> + The different formats are called "Views". The current repertoire + includes "Modern", "Columnar", "Compact", "Simple", "Verbose", and + "Classic". Users can select whichever format they want and their + preference is remembered in a cookie. + + 1. **A Fossil web server admin can set the default timeline format.**<p> + Individual users have a lot of control over what their own timeline + displays look like, but the repository administrator can set the + default separately for each repository. + + 1. **Fossil is open and transparent about the cookies that it uses.**<p> + There are really only three: The login cookie if you are logged in, + the robot cookie indicating that you have previously passed a captcha + if you are not logged in, and the display preferences cookie. + If you visit the + [/cookies page](/cookies), Fossil will + show you all the cookies it uses and it will decode them for you to + show you exactly what they mean and what information they are holding, + and Fossil will give you an opportunity to delete each cookie + individually. + + 1. **The Fossil web interface comes with a variety of "skins" built in.**<p> + A skin determines the coloration and layout of Fossil web pages. + Visit the [/skins page](/skins) to see all the available skins and + which one is currently in use. + + 1. **Individual users get to choose their favorite Fossil skin.**<p> + The repository administrator sets the default skin, but if + individual users do not like that choice, they can select a different + skin and their choice is recorded in the display preferences cookie. + + 1. **Repository administrators can create new custom skins.**<p> + The current library of skins are mostly derived from custom skins that + users of Fossil have created over the years and generously donated + to the project. If none of the default skins work for you, you can + create your own, perhaps using one of the existing skins as a template. + + 1. **Most Fossil web-interface skins include a detailed hamburger + (☰) menu.**<p> + Clicking on the hamburger menu brings up a dropdown "site-map" + page that lets you quickly navigate to the information you want. + (Note: The presence and operation of the hamburger menu is a + skin-specific feature and might not be available on every skin, but + it is used on the more popular skins.) Git itself has nothing like + this. The various Git forges sometimes have a limited hamburger + menu, that it is not nearly as complete. I find it difficult to + locate the screens I want without consulting an AI. + + 1. **The Fossil web interface /sitemap page is responsive to + individual user permissions and capabilities.**<p> + Each user on the Fossil web interface, including the special + user "nobody" used if no login is attempted, has + "capabilities" assigned by the repository administrator. + Depending on capabilities, some pages will display differently or + will not display at all. Pages that a user does not have access + to are automatically omitted from the [/sitemap page](/sitemap). + + 1. **The Fossil web interface allows "anonymous" users.**<p> + The "anonymous" user is a human (we think, because he has solved + a captcha) but we do not know who. Users who do not want to + identify themselves but who also don't want to be mistaken for + a spider or robot can log in as anonymous. + <p> + The repository administrator has complete control over the capabilities + of anonymous. Anonymous can be completely banned, or maybe given + read-only capabilities, or given complete access, with lots of shade + in between, according to the needs of the project. + <p> + The "anonymous" user is distinct from user "nobody" in that we + believe anonymous is a real human, whereas user nobody is presumed + to be a robot. The repository administrator also has complete control + over the capabilities for user "nobody". + + 1. **The Fossil web interface has lots of built-in defenses against + abuse by spiders and robots.**<p> + Sadly, the internet is rapidly devolving such that most HTTP requests + now come from AI spiders trying to find training content, and/or robots + looking for website vulnerabilities. The flood of requests can + rapidly bog down an undefended server. Fossil includes a range of + defenses against aggressive bots that help keep the server load and + ISP costs down while still providing fast and detailed responses + to real humans. This is an on-going battle. But Fossil is, at least, + in the fight. Everything is easily configurable, via the web interface, + by repository administrators. + + 1. **The Fossil web interface includes a "security audit" page accessible + to repository administrators.**<p> + The security-audit pages give a succinct summary of how a repository + web interface is configured, with an eye toward operational security. + As with any full-featured web application, the Fossil web interface + has a large number of settings. A common worry among system + administrators is overlooking or omitting or misconfiguring some + security-sensitive setting. The security-audit page is designed to + reduce that worry. + <p> + The security-audit page shows at a glance how a repository web interface + is set up, and raises alerts about any settings that are questionable + or that might facilitate mischief. The page fits on a single screen + with minimal or no scrolling. After standing up a new Fossil server, + a quick glance at the security-audit page (accessible only to + administrators) gives peace of mind that all is well and that nothing + was overlooked. + + 1. **The Fossil web interface menu bar can be customized.**<p> + Repository administrators can customize the menu bar on the web + interface. Individual items can be added or omitted from the + menu bar based on user capabilities and/or whether or not the + client is a phone or other narrow-screen mobile device, a standard + desktop browser, or a wide-screen desktop browser. + + 1. **The Fossil web interface sitemap can be customized.**<p> + Repository administrators can add new entries to the + [/sitemap](/sitemap) that are shown or omitted + based on user capabilities. + + 1. **The Fossil web interface can be augmented with auxiliary content + and/or CGIs that exist outside of the repository.**<p> + The auxiliary content or CGI result uses the same theme and skin as + the default website, and blends right in. An example of this is + the + [SQLite Release Checklist](https://sqlite.org/src/ext/checklist/top/index). + The SQLite Release Checklist is a CGI that is completely separate from + Fossil, but appears to be integrated in the Fossil web interface. + It uses the same skin and interface settings. But the content is + created by a separate CGI program. Fossil passes down additional + CGI variables to tell the CGI what the Fossil user name is and what + capabilities that user has, among other things. In the case of the + SQLite Release Checklist, those additional settings mean only project + committers can make changes to the checklist (such as marking items + as "done") and that the checklist is read-only for the general public. + <p> + Additional information about this advanced feature of Fossil + can be seen at <https://fossil-scm.org/home/doc/trunk/www/serverext.wiki>. + + 1. **Fossil makes convenience commands available to run its + cryptographic hash algorithms (SHA1 and SHA3-256)**<p> + The "<tt>fossil sha1sum FILE ...</tt>" and + "<tt>fossil sha3sum FILE ...</tt>" commands will + compute SHA1 and SHA3-256 hashes on files. These commands are not + necessary to use Fossil, but they are still useful, and they are + not commonly installed on non-Linux platforms. Fossil ensures that + those hash functions are available wherever Fossil is available. + + 1. **Fossil exposes its 3-way-diff algorithm for external use.**<p> + The "<tt>fossil 3-way-merge ...</tt>" command works + like the classic unix "diff3" command in that it does a merge + of two variants of a file given a common ancestor. This is the + exact same algorithm that Fossil uses to compute merges internally, + simply exposed for external use. + + 1. **On a merge conflict, Fossil shows the conflicting inputs just + like other merge algorithms, but it also shows a suggested conflict + resolution.**<p> + The suggested conflict resolution is not always correct, but it is + sometimes, and its presence often makes resolving merge conflicts + simpler. + + 1. **Fossil supports single sign-in when serving multiple repositories + from the same host computer.**<p> + If you have a server that is hosting Fossil web interfaces for + multiple repositories, those repositories can be interconnected + into a common "login group" such that when a user logs into one + repository web interface, he is also automatically logged into + all other repositories in that login graph that hold the same + username. Furthermore, if the user + changes his password on one repository, it is automatically changed + on all the others within that login group. + + 1. **Experts can browse low-level details of a Fossil repository + using SQL.**<p> + A Fossil repository is just an SQLite database file. Low-level content + of that database file can be viewed and even changed using ordinary + SQL and the "<tt>fossil sql</tt>" command. The "fossil sql" + command brings up a standard SQLite command-line shell, already + connected to the repository database, and extended to include extra + functions (including table-valued functions) to help interpret the + low-level content of the repository. This feature is not needed nor + recommended for the average user. However, if you want to learn more + about the inner workings of Fossil, or if you want to generate some + custom reports about a repository, or if you are extending or + troubleshooting Fossil, the SQL interface is a great tool. + + 1. **The underlying artifacts of a Fossil repository are well-documented, + human-readable, and human-understandable.**<p> + A Fossil repository is an SQLite database file, but not every SQLite + database file is a Fossil repository. Fossil repositories store + "artifacts" in a very particular format. See + <https://fossil-scm.org/home/doc/trunk/www/fileformat.wiki> for + the details of that format. + <p> + This underlying format is text-only. It is designed to be easily + parsed and interpreted by programs written in any language. It is + designed to be easily understood by humans, even humans not yet born. + Many of the low-level artifact formats for Git, in contrast, are + binary and are only thinly documented. The only sure way to understand + the low-level Git format is, in my experience, to read the Git source + code. + + 1. **The Fossil web interface has a "This Day In History" page.**<p> + See that page for [Fossil](/thisdayinhistory) or + [SQLite](https://sqlite.org/src/thisdayinhistory). + The page shows multiple timeline snippets from various + days in the past: 1, 2, 5, 10, 15, 20 years ago. + <p> + This is something of a vanity page. It is difficult to describe a + real business need for this information. But the page does jog old + memories and helps developers keep perspective on how a project has + changed through the years. + <p> + The existence of this page illustrates how the + robust and modular design of the Fossil implementation + facilitates custom modifications that involve very little new code. + + 1. **Fossil allows you to update your current checkout even if it + contains uncommitted changes.**<p> + This is a very common idiom in Fossil: You are working on changes + and somebody commits ahead of you. You run + "<tt>fossil up</tt>" ("up" is short for "update") and the new + external changes are merged into your own uncommitted changes. You + continue working. + <p> + Doing this in Git appears to require multiple commands (or maybe + just one command with multiple verbose options) to interact with + the stash and to rebase your changes. + + 1. **Fossil lets you undo an update.**<p> + If you run "<tt>fossil up</tt>" on a checkout that contains + uncommitted changes, and the update does not go well (for example, + if there are a lot of merge conflicts) you can back out the update + by running "<tt>fossil undo</tt>". + <p> + The need for this does not arise often, because + "<tt>fossil up</tt>" normally just works. But the ability to + undo is a nice safety-net for the rare cases when the update goes awry. + + 1. **Fossil warns you if you try to commit and somebody else has + committed ahead of you.**<p> + You can then run "<tt>fossil up</tt>" and then retest and retry. + Or you can override the warning and force Fossil to commit anyhow, + thus forking the branch. Either way, + you enter the commit with more knowledge about what is happening, + and thus improved situational awareness. + + 1. **Every Fossil project has a unique identifier**.<p> + When a new repository is created, the unique identifier is created + and stored in the repository. That identifier is called the + "project code". The identifier is copied with every + clone. (Uniqueness is probabilistic. The identifier simply contains + enough of high-quality randomness to make it unlikely that there + will ever be a collision.) + <p> + It is not possible in Fossil to push to or pull from a repository + with the wrong project code. Thus you cannot contaminate one + project with code from another simply by specifying the wrong + remote and adding the --force flag, as is apparently possible in Git. + + 1. **Each Fossil repository keeps an audit trail.**<p> + For each new artifact received into a Fossil repository, by push or + pull, or by direct commit from the command line, Fossil records a + timestamp, username, and an IP address (where applicable) for + that artifact. If harmful or malicious content is added to + a repository, the repository administrator has the capability to + trace that content back to its source, so that appropriate + sanctions can be applied to the malefactor. + + 1. **Fossil stores content in a power-safe ACID database.**<p> + The repository content cannot be corrupted by a program crash, + system crash, or unexpected power loss. The repository moves + from one consistent state to another, atomically. This helps + ensure that the resources stored in Fossil are kept safe, + even if the Fossil implementation itself contains bugs. + <p> + The underlying database engine used by Fossil is SQLite, of course. + <p> + Git also claims to be transactional. However, because Git does + not use a separate database engine, the transactional integrity + of Git depends entirely upon the correctness of the Git code + itself. Git is thus far more sensitive to implementation errors. + + 1. **Fossil supports a built-in graphical diff tool.**<p> + Running "<tt>fossil gdiff</tt>" show the currently + uncommitted changes in a Tk-based graphical display. This + is built into Fossil and does not require any external tools + (though it does require Tcl/Tk). Git requires external tooling + in order to do the same. + + 1. **Fossil supports showing diffs in a web browser.**<p> + Adding the "<tt>-b</tt>" or "<tt>-by</tt>" option to any Fossil + diff command causes that diff to be rendered as a new page in + the users default web browser. Git does not have any such + capability, even with the aid of external programs, as far as + I am aware. + + 1. **The Fossil web interface provides a captcha-gated method to download + tarballs and ZIP archives for any check-in.**<p> + See, for example, <https://sqlite.org/src/rchvdwnld/20260704>. + That link provides access to the last check-in for the day 2026-07-04. + The last element of the patch can be any hash prefix, timestamp prefix, + or tag that references a check-in.<p> + Tarballs and ZIP archives are expensive to compute, not because Fossil + has trouble assembling the content, + but rather because the result must be run through zlib compression. + When computing a new archive, almost all of the CPU time is spent + inside of the zlib compression library. + For an SQLite-size tarball, the zlib compression alone can take as + much as 10 seconds of CPU time. If the tarball/ZIP-archive download + link is not protected by a captcha, multiple spiders will attempt to + download every possible tarball and ZIP archive, multiple times per + day, bringing your server to its knees. + + 1. **Repository administrators can create a cache of recently downloaded + tarballs and ZIP archives**<p> + That way, if there are common downloads (like the most recent release) + the archive does not get recomputed from scratch with each download. + A download of a cached archive file is fast. + The size of the cache is configurable by the repository administrator. + + 1. **The Fossil web interface has a "Repository Status" page that shows + interesting and useful facts about the repository being served.**<p> + See the [stat page for Fossil](/stat) for example. + + 1. **The Fossil web interface easily shows all of the contributors to + a project, and the number of changes they have committed.**<p> + See <https://fossil-scm.org/home/reports> for that report about + Fossil itself. + Many other reports can be generated from the same page by selecting + different options from the pull-down menus at the top. + <p>The page is captcha-gated since it is a magnet for spiders. + + 1. **The Fossil source code is easy to enhance with new web pages and + commands.**<p> + The design of the Fossil implementation makes it easy to extend with + new capabilities as needs arise. + <ul type="disk"> + <li> The code is simple C89. + <li> Each command or web page runs as a separate process which + exits when the action completes, so minor memory leaks are not + a concern. + <li> Preprocessors run over the Fossil source code before the + code reaches the C compiler to verify + that there are no SQL injections nor XSS vulnerabilities. + <li> New commands and webpages can be added simply by adding a new + procedure to do the necessary computation. The name, properties, + and documentation for the command or webpage are extracted from + specially formatted comments just prior to the procedure. + </ul><p> + The ease with which Fossil can be enhanced is part of the reason why + it has picked up so many useful features and has become so + powerful over its 19-year history. + +## Conclusion + +There are more reasons to prefer Fossil over Git, but +I think 101 are sufficient to prove my point. diff --git a/www/image-format-vs-repo-size.ipynb b/www/image-format-vs-repo-size.ipynb new file mode 100644 index 0000000..47aaa58 --- /dev/null +++ b/www/image-format-vs-repo-size.ipynb @@ -0,0 +1,1719 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Image Format vs Fossil Repository Size\n", + "\n", + "## Prerequisites\n", + "\n", + "This notebook was originally developed with standalone [JupyterLab] and Python 2 but was later moved to JupyterLab under [Anaconda] with Python 3. Backporting to Python 2 may require manual adjustment. Getting it running under stock JupyterLab or plain-old-Jupyter should be straightforward for one familiar with the tools. We will assume you're following in my footsteps, using Anaconda.\n", + "\n", + "One of the reasons we switched to Anaconda is that it comes with all but one of this notebook's prerequisites, that last remaining one of which you install so:\n", + "\n", + " $ pip install wand\n", + "\n", + "That should be done in a shell where \"`pip`\" is the version that came with Anaconda. Otherwise, the package will likely end up in some *other* Python package tree, which Anaconda's Python kernel may not be smart enough to find on its own.\n", + "\n", + "Note that you do *not* use `conda` for this: as of this writing, [Wand] is not available in a form that installs via `conda`.\n", + "\n", + "This notebook was written and tested on a macOS system where `/tmp` exists. Other platforms may require adjustments to the scripts below.\n", + "\n", + "[Anaconda]: https://www.anaconda.com/distribution/\n", + "[JupyterLab]: https://github.com/jupyterlab/\n", + "[Wand]: http://wand-py.org/\n", + "\n", + "\n", + "## Running\n", + "\n", + "The next cell generates the test repositories. This takes about 3 seconds to run on my machine. If you have to uncomment the \"`sleep`\" call in the inner loop, this will go up to about 45 seconds.\n", + "\n", + "The next cell produces the bar chart from the collected data, all but instantaneously.\n", + "\n", + "This split allows you to generate the expensive experimental data in a single pass, then play as many games as you like with the generated data.\n", + "\n", + "\n", + "## Discussion\n", + "\n", + "That is kept in [a separate document](image-format-vs-repo-size.md) so we can share that document with Fossil's Markdown renderer." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Created test directory /tmp/image-format-vs-repo-size\n", + "Created ../test-jpeg.fossil for format JPEG.\n", + "Created ../test-bmp.fossil for format BMP.\n", + "Created ../test-tiff.fossil for format TIFF.\n", + "Created ../test-png.fossil for format PNG.\n", + "Experiment completed in 3.0627901554107666 seconds.\n" + ] + } + ], + "source": [ + "import os\n", + "import random\n", + "import subprocess\n", + "import time\n", + "\n", + "from wand.color import Color\n", + "from wand.drawing import Drawing\n", + "from wand.image import Image\n", + "\n", + "import pandas as pd\n", + "\n", + "size = 256\n", + "iterations = 10\n", + "start = time.time()\n", + "repo_sizes = []\n", + "fossil = '/usr/local/bin/fossil'\n", + "\n", + "if not os.path.isfile(fossil): raise RuntimeError(\"No such executable \" + fossil)\n", + "if not os.access(fossil, os.X_OK): raise RuntimeError(\"Cannot execute \" + fossil)\n", + "\n", + "tdir = os.path.join('/tmp', 'image-format-vs-repo-size')\n", + "if not os.path.isdir(tdir): os.mkdir(tdir, 0o700)\n", + "print(\"Created test directory \" + tdir)\n", + " \n", + "formats = ['JPEG', 'BMP', 'TIFF', 'PNG']\n", + "for f in formats:\n", + " ext = f.lower()\n", + " wdir = os.path.join(tdir, 'work-' + ext)\n", + " if not os.path.isdir(wdir): os.mkdir(wdir, 0o700)\n", + " os.chdir(wdir)\n", + " repo = '../test-' + ext + '.fossil'\n", + " ifn = 'test.' + ext\n", + " ipath = os.path.join(wdir, ifn)\n", + " rs = []\n", + " \n", + " def add_repo_size():\n", + " rs.append(os.path.getsize(repo) / 1024.0 / 1024.0)\n", + " \n", + " def set_repo_page_size(n):\n", + " subprocess.run([\n", + " fossil,\n", + " 'rebuild',\n", + " '--compress',\n", + " '--pagesize',\n", + " str(n),\n", + " '--vacuum'\n", + " ])\n", + "\n", + " try:\n", + " # Create test repo\n", + " subprocess.run([fossil, 'init', repo])\n", + " subprocess.run([fossil, 'open', repo])\n", + " subprocess.run([fossil, 'set', 'binary-glob', \"*.{0}\".format(ext)])\n", + " set_repo_page_size(512) # minimum\n", + " add_repo_size()\n", + " set_repo_page_size(8192) # default\n", + " print(\"Created \" + repo + \" for format \" + f + \".\")\n", + "\n", + " # Create test image and add it to the repo\n", + " img = Image(width = size, height = size, depth = 8,\n", + " background = 'white')\n", + " img.alpha_channel = 'remove'\n", + " img.evaluate('gaussiannoise', 1.0)\n", + " img.save(filename = ipath)\n", + " subprocess.run([fossil, 'add', ifn])\n", + " subprocess.run([fossil, 'ci', '-m', 'initial'])\n", + " #print(\"Added initial \" + f + \" image.\")\n", + " add_repo_size()\n", + "\n", + " # Change a random pixel to a random RGB value and check it in\n", + " # $iterations times.\n", + " for i in range(iterations - 1):\n", + " with Drawing() as draw:\n", + " x = random.randint(0, size - 1)\n", + " y = random.randint(0, size - 1)\n", + "\n", + " r = random.randint(0, 255)\n", + " g = random.randint(0, 255)\n", + " b = random.randint(0, 255)\n", + " \n", + " draw.fill_color = Color('rgb({0},{1},{2})'.format(\n", + " r, g, b\n", + " ))\n", + " draw.color(x, y, 'point')\n", + " draw(img)\n", + " img.save(filename = ipath)\n", + " \n", + " # You might need to uncomment the next line if you find that\n", + " # the repo size doesn't change as expected. In some versions\n", + " # of Wand (or is it the ImageMagick underneath?) we have seen\n", + " # what appear to be asynchronous saves, with a zero-length file\n", + " # here if you don't wait for the save to complete.\n", + " #time.sleep(1.0)\n", + " \n", + " subprocess.run([fossil, 'ci', '-m', '\"change {0} step {1}'.format(\n", + " f, i\n", + " )])\n", + " add_repo_size()\n", + " \n", + " # Repo complete for this format\n", + " repo_sizes.append(pd.Series(rs, name=f))\n", + "\n", + " finally:\n", + " if os.path.exists(ipath): os.remove(ipath)\n", + " if os.path.exists(tdir):\n", + " if os.path.isfile(repo):\n", + " subprocess.run([fossil, 'close', '-f'])\n", + " os.unlink(repo)\n", + " os.chdir(tdir);\n", + " os.rmdir(wdir)\n", + " if os.path.exists(repo): os.remove(repo)\n", + " \n", + "print(\"Experiment completed in \" + str(time.time() - start) + \" seconds.\")" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "data": { + "image/svg+xml": [ + "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n", + "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n", + " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n", + "<!-- Created with matplotlib (https://matplotlib.org/) -->\n", + "<svg height=\"265.243125pt\" version=\"1.1\" viewBox=\"0 0 385.78125 265.243125\" width=\"385.78125pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n", + " <defs>\n", + " <style type=\"text/css\">\n", + "*{stroke-linecap:butt;stroke-linejoin:round;}\n", + " </style>\n", + " </defs>\n", + " <g id=\"figure_1\">\n", + " <g id=\"patch_1\">\n", + " <path d=\"M 0 265.243125 \n", + "L 385.78125 265.243125 \n", + "L 385.78125 0 \n", + "L 0 0 \n", + "z\n", + "\" style=\"fill:none;\"/>\n", + " </g>\n", + " <g id=\"axes_1\">\n", + " <g id=\"patch_2\">\n", + " <path d=\"M 43.78125 224.64 \n", + "L 378.58125 224.64 \n", + "L 378.58125 7.2 \n", + "L 43.78125 7.2 \n", + "z\n", + "\" style=\"fill:#ffffff;\"/>\n", + " </g>\n", + " <g id=\"patch_3\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 51.907464 224.64 \n", + "L 58.408434 224.64 \n", + "L 58.408434 138.354286 \n", + "L 51.907464 138.354286 \n", + "z\n", + "\" style=\"fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_4\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 84.412318 224.64 \n", + "L 90.913289 224.64 \n", + "L 90.913289 126.849524 \n", + "L 84.412318 126.849524 \n", + "z\n", + "\" style=\"fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_5\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 116.917172 224.64 \n", + "L 123.418143 224.64 \n", + "L 123.418143 118.220952 \n", + "L 116.917172 118.220952 \n", + "z\n", + "\" style=\"fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_6\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 149.422027 224.64 \n", + "L 155.922998 224.64 \n", + "L 155.922998 112.468571 \n", + "L 149.422027 112.468571 \n", + "z\n", + "\" style=\"fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_7\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 181.926881 224.64 \n", + "L 188.427852 224.64 \n", + "L 188.427852 109.592381 \n", + "L 181.926881 109.592381 \n", + "z\n", + "\" style=\"fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_8\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 214.431735 224.64 \n", + "L 220.932706 224.64 \n", + "L 220.932706 103.84 \n", + "L 214.431735 103.84 \n", + "z\n", + "\" style=\"fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_9\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 246.93659 224.64 \n", + "L 253.437561 224.64 \n", + "L 253.437561 98.087619 \n", + "L 246.93659 98.087619 \n", + "z\n", + "\" style=\"fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_10\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 279.441444 224.64 \n", + "L 285.942415 224.64 \n", + "L 285.942415 98.087619 \n", + "L 279.441444 98.087619 \n", + "z\n", + "\" style=\"fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_11\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 311.946299 224.64 \n", + "L 318.447269 224.64 \n", + "L 318.447269 82.268571 \n", + "L 311.946299 82.268571 \n", + "z\n", + "\" style=\"fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_12\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 344.451153 224.64 \n", + "L 350.952124 224.64 \n", + "L 350.952124 77.954286 \n", + "L 344.451153 77.954286 \n", + "z\n", + "\" style=\"fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_13\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 58.408434 224.64 \n", + "L 64.909405 224.64 \n", + "L 64.909405 125.411429 \n", + "L 58.408434 125.411429 \n", + "z\n", + "\" style=\"fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_14\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 90.913289 224.64 \n", + "L 97.41426 224.64 \n", + "L 97.41426 105.278095 \n", + "L 90.913289 105.278095 \n", + "z\n", + "\" style=\"fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_15\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 123.418143 224.64 \n", + "L 129.919114 224.64 \n", + "L 129.919114 105.278095 \n", + "L 123.418143 105.278095 \n", + "z\n", + "\" style=\"fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_16\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 155.922998 224.64 \n", + "L 162.423968 224.64 \n", + "L 162.423968 105.278095 \n", + "L 155.922998 105.278095 \n", + "z\n", + "\" style=\"fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_17\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 188.427852 224.64 \n", + "L 194.928823 224.64 \n", + "L 194.928823 105.278095 \n", + "L 188.427852 105.278095 \n", + "z\n", + "\" style=\"fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_18\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 220.932706 224.64 \n", + "L 227.433677 224.64 \n", + "L 227.433677 105.278095 \n", + "L 220.932706 105.278095 \n", + "z\n", + "\" style=\"fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_19\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 253.437561 224.64 \n", + "L 259.938532 224.64 \n", + "L 259.938532 105.278095 \n", + "L 253.437561 105.278095 \n", + "z\n", + "\" style=\"fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_20\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 285.942415 224.64 \n", + "L 292.443386 224.64 \n", + "L 292.443386 105.278095 \n", + "L 285.942415 105.278095 \n", + "z\n", + "\" style=\"fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_21\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 318.447269 224.64 \n", + "L 324.94824 224.64 \n", + "L 324.94824 105.278095 \n", + "L 318.447269 105.278095 \n", + "z\n", + "\" style=\"fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_22\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 350.952124 224.64 \n", + "L 357.453095 224.64 \n", + "L 357.453095 105.278095 \n", + "L 350.952124 105.278095 \n", + "z\n", + "\" style=\"fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_23\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 64.909405 224.64 \n", + "L 71.410376 224.64 \n", + "L 71.410376 128.287619 \n", + "L 64.909405 128.287619 \n", + "z\n", + "\" style=\"fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_24\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 97.41426 224.64 \n", + "L 103.915231 224.64 \n", + "L 103.915231 108.154286 \n", + "L 97.41426 108.154286 \n", + "z\n", + "\" style=\"fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_25\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 129.919114 224.64 \n", + "L 136.420085 224.64 \n", + "L 136.420085 108.154286 \n", + "L 129.919114 108.154286 \n", + "z\n", + "\" style=\"fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_26\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 162.423968 224.64 \n", + "L 168.924939 224.64 \n", + "L 168.924939 108.154286 \n", + "L 162.423968 108.154286 \n", + "z\n", + "\" style=\"fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_27\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 194.928823 224.64 \n", + "L 201.429794 224.64 \n", + "L 201.429794 108.154286 \n", + "L 194.928823 108.154286 \n", + "z\n", + "\" style=\"fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_28\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 227.433677 224.64 \n", + "L 233.934648 224.64 \n", + "L 233.934648 108.154286 \n", + "L 227.433677 108.154286 \n", + "z\n", + "\" style=\"fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_29\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 259.938532 224.64 \n", + "L 266.439502 224.64 \n", + "L 266.439502 108.154286 \n", + "L 259.938532 108.154286 \n", + "z\n", + "\" style=\"fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_30\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 292.443386 224.64 \n", + "L 298.944357 224.64 \n", + "L 298.944357 108.154286 \n", + "L 292.443386 108.154286 \n", + "z\n", + "\" style=\"fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_31\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 324.94824 224.64 \n", + "L 331.449211 224.64 \n", + "L 331.449211 108.154286 \n", + "L 324.94824 108.154286 \n", + "z\n", + "\" style=\"fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_32\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 357.453095 224.64 \n", + "L 363.954066 224.64 \n", + "L 363.954066 108.154286 \n", + "L 357.453095 108.154286 \n", + "z\n", + "\" style=\"fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_33\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 71.410376 224.64 \n", + "L 77.911347 224.64 \n", + "L 77.911347 129.725714 \n", + "L 71.410376 129.725714 \n", + "z\n", + "\" style=\"fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_34\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 103.915231 224.64 \n", + "L 110.416201 224.64 \n", + "L 110.416201 111.030476 \n", + "L 103.915231 111.030476 \n", + "z\n", + "\" style=\"fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_35\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 136.420085 224.64 \n", + "L 142.921056 224.64 \n", + "L 142.921056 80.830476 \n", + "L 136.420085 80.830476 \n", + "z\n", + "\" style=\"fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_36\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 168.924939 224.64 \n", + "L 175.42591 224.64 \n", + "L 175.42591 75.078095 \n", + "L 168.924939 75.078095 \n", + "z\n", + "\" style=\"fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_37\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 201.429794 224.64 \n", + "L 207.930765 224.64 \n", + "L 207.930765 70.76381 \n", + "L 201.429794 70.76381 \n", + "z\n", + "\" style=\"fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_38\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 233.934648 224.64 \n", + "L 240.435619 224.64 \n", + "L 240.435619 56.382857 \n", + "L 233.934648 56.382857 \n", + "z\n", + "\" style=\"fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_39\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 266.439502 224.64 \n", + "L 272.940473 224.64 \n", + "L 272.940473 40.56381 \n", + "L 266.439502 40.56381 \n", + "z\n", + "\" style=\"fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_40\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 298.944357 224.64 \n", + "L 305.445328 224.64 \n", + "L 305.445328 30.497143 \n", + "L 298.944357 30.497143 \n", + "z\n", + "\" style=\"fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_41\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 331.449211 224.64 \n", + "L 337.950182 224.64 \n", + "L 337.950182 26.182857 \n", + "L 331.449211 26.182857 \n", + "z\n", + "\" style=\"fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"patch_42\">\n", + " <path clip-path=\"url(#p70301a0bed)\" d=\"M 363.954066 224.64 \n", + "L 370.455036 224.64 \n", + "L 370.455036 17.554286 \n", + "L 363.954066 17.554286 \n", + "z\n", + "\" style=\"fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"matplotlib.axis_1\">\n", + " <g id=\"xtick_1\">\n", + " <g id=\"line2d_1\">\n", + " <defs>\n", + " <path d=\"M 0 0 \n", + "L 0 3.5 \n", + "\" id=\"m639e59548b\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n", + " </defs>\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"64.909405\" xlink:href=\"#m639e59548b\" y=\"224.64\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_1\">\n", + " <!-- 1 -->\n", + " <defs>\n", + " <path d=\"M 12.40625 8.296875 \n", + "L 28.515625 8.296875 \n", + "L 28.515625 63.921875 \n", + "L 10.984375 60.40625 \n", + "L 10.984375 69.390625 \n", + "L 28.421875 72.90625 \n", + "L 38.28125 72.90625 \n", + "L 38.28125 8.296875 \n", + "L 54.390625 8.296875 \n", + "L 54.390625 0 \n", + "L 12.40625 0 \n", + "z\n", + "\" id=\"DejaVuSans-49\"/>\n", + " </defs>\n", + " <g transform=\"translate(67.66878 238.0025)rotate(-90)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-49\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"xtick_2\">\n", + " <g id=\"line2d_2\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"97.41426\" xlink:href=\"#m639e59548b\" y=\"224.64\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_2\">\n", + " <!-- 2 -->\n", + " <defs>\n", + " <path d=\"M 19.1875 8.296875 \n", + "L 53.609375 8.296875 \n", + "L 53.609375 0 \n", + "L 7.328125 0 \n", + "L 7.328125 8.296875 \n", + "Q 12.9375 14.109375 22.625 23.890625 \n", + "Q 32.328125 33.6875 34.8125 36.53125 \n", + "Q 39.546875 41.84375 41.421875 45.53125 \n", + "Q 43.3125 49.21875 43.3125 52.78125 \n", + "Q 43.3125 58.59375 39.234375 62.25 \n", + "Q 35.15625 65.921875 28.609375 65.921875 \n", + "Q 23.96875 65.921875 18.8125 64.3125 \n", + "Q 13.671875 62.703125 7.8125 59.421875 \n", + "L 7.8125 69.390625 \n", + "Q 13.765625 71.78125 18.9375 73 \n", + "Q 24.125 74.21875 28.421875 74.21875 \n", + "Q 39.75 74.21875 46.484375 68.546875 \n", + "Q 53.21875 62.890625 53.21875 53.421875 \n", + "Q 53.21875 48.921875 51.53125 44.890625 \n", + "Q 49.859375 40.875 45.40625 35.40625 \n", + "Q 44.1875 33.984375 37.640625 27.21875 \n", + "Q 31.109375 20.453125 19.1875 8.296875 \n", + "z\n", + "\" id=\"DejaVuSans-50\"/>\n", + " </defs>\n", + " <g transform=\"translate(100.173635 238.0025)rotate(-90)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-50\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"xtick_3\">\n", + " <g id=\"line2d_3\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"129.919114\" xlink:href=\"#m639e59548b\" y=\"224.64\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_3\">\n", + " <!-- 3 -->\n", + " <defs>\n", + " <path d=\"M 40.578125 39.3125 \n", + "Q 47.65625 37.796875 51.625 33 \n", + "Q 55.609375 28.21875 55.609375 21.1875 \n", + "Q 55.609375 10.40625 48.1875 4.484375 \n", + "Q 40.765625 -1.421875 27.09375 -1.421875 \n", + "Q 22.515625 -1.421875 17.65625 -0.515625 \n", + "Q 12.796875 0.390625 7.625 2.203125 \n", + "L 7.625 11.71875 \n", + "Q 11.71875 9.328125 16.59375 8.109375 \n", + "Q 21.484375 6.890625 26.8125 6.890625 \n", + "Q 36.078125 6.890625 40.9375 10.546875 \n", + "Q 45.796875 14.203125 45.796875 21.1875 \n", + "Q 45.796875 27.640625 41.28125 31.265625 \n", + "Q 36.765625 34.90625 28.71875 34.90625 \n", + "L 20.21875 34.90625 \n", + "L 20.21875 43.015625 \n", + "L 29.109375 43.015625 \n", + "Q 36.375 43.015625 40.234375 45.921875 \n", + "Q 44.09375 48.828125 44.09375 54.296875 \n", + "Q 44.09375 59.90625 40.109375 62.90625 \n", + "Q 36.140625 65.921875 28.71875 65.921875 \n", + "Q 24.65625 65.921875 20.015625 65.03125 \n", + "Q 15.375 64.15625 9.8125 62.3125 \n", + "L 9.8125 71.09375 \n", + "Q 15.4375 72.65625 20.34375 73.4375 \n", + "Q 25.25 74.21875 29.59375 74.21875 \n", + "Q 40.828125 74.21875 47.359375 69.109375 \n", + "Q 53.90625 64.015625 53.90625 55.328125 \n", + "Q 53.90625 49.265625 50.4375 45.09375 \n", + "Q 46.96875 40.921875 40.578125 39.3125 \n", + "z\n", + "\" id=\"DejaVuSans-51\"/>\n", + " </defs>\n", + " <g transform=\"translate(132.678489 238.0025)rotate(-90)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-51\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"xtick_4\">\n", + " <g id=\"line2d_4\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"162.423968\" xlink:href=\"#m639e59548b\" y=\"224.64\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_4\">\n", + " <!-- 4 -->\n", + " <defs>\n", + " <path d=\"M 37.796875 64.3125 \n", + "L 12.890625 25.390625 \n", + "L 37.796875 25.390625 \n", + "z\n", + "M 35.203125 72.90625 \n", + "L 47.609375 72.90625 \n", + "L 47.609375 25.390625 \n", + "L 58.015625 25.390625 \n", + "L 58.015625 17.1875 \n", + "L 47.609375 17.1875 \n", + "L 47.609375 0 \n", + "L 37.796875 0 \n", + "L 37.796875 17.1875 \n", + "L 4.890625 17.1875 \n", + "L 4.890625 26.703125 \n", + "z\n", + "\" id=\"DejaVuSans-52\"/>\n", + " </defs>\n", + " <g transform=\"translate(165.183343 238.0025)rotate(-90)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-52\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"xtick_5\">\n", + " <g id=\"line2d_5\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"194.928823\" xlink:href=\"#m639e59548b\" y=\"224.64\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_5\">\n", + " <!-- 5 -->\n", + " <defs>\n", + " <path d=\"M 10.796875 72.90625 \n", + "L 49.515625 72.90625 \n", + "L 49.515625 64.59375 \n", + "L 19.828125 64.59375 \n", + "L 19.828125 46.734375 \n", + "Q 21.96875 47.46875 24.109375 47.828125 \n", + "Q 26.265625 48.1875 28.421875 48.1875 \n", + "Q 40.625 48.1875 47.75 41.5 \n", + "Q 54.890625 34.8125 54.890625 23.390625 \n", + "Q 54.890625 11.625 47.5625 5.09375 \n", + "Q 40.234375 -1.421875 26.90625 -1.421875 \n", + "Q 22.3125 -1.421875 17.546875 -0.640625 \n", + "Q 12.796875 0.140625 7.71875 1.703125 \n", + "L 7.71875 11.625 \n", + "Q 12.109375 9.234375 16.796875 8.0625 \n", + "Q 21.484375 6.890625 26.703125 6.890625 \n", + "Q 35.15625 6.890625 40.078125 11.328125 \n", + "Q 45.015625 15.765625 45.015625 23.390625 \n", + "Q 45.015625 31 40.078125 35.4375 \n", + "Q 35.15625 39.890625 26.703125 39.890625 \n", + "Q 22.75 39.890625 18.8125 39.015625 \n", + "Q 14.890625 38.140625 10.796875 36.28125 \n", + "z\n", + "\" id=\"DejaVuSans-53\"/>\n", + " </defs>\n", + " <g transform=\"translate(197.688198 238.0025)rotate(-90)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-53\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"xtick_6\">\n", + " <g id=\"line2d_6\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"227.433677\" xlink:href=\"#m639e59548b\" y=\"224.64\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_6\">\n", + " <!-- 6 -->\n", + " <defs>\n", + " <path d=\"M 33.015625 40.375 \n", + "Q 26.375 40.375 22.484375 35.828125 \n", + "Q 18.609375 31.296875 18.609375 23.390625 \n", + "Q 18.609375 15.53125 22.484375 10.953125 \n", + "Q 26.375 6.390625 33.015625 6.390625 \n", + "Q 39.65625 6.390625 43.53125 10.953125 \n", + "Q 47.40625 15.53125 47.40625 23.390625 \n", + "Q 47.40625 31.296875 43.53125 35.828125 \n", + "Q 39.65625 40.375 33.015625 40.375 \n", + "z\n", + "M 52.59375 71.296875 \n", + "L 52.59375 62.3125 \n", + "Q 48.875 64.0625 45.09375 64.984375 \n", + "Q 41.3125 65.921875 37.59375 65.921875 \n", + "Q 27.828125 65.921875 22.671875 59.328125 \n", + "Q 17.53125 52.734375 16.796875 39.40625 \n", + "Q 19.671875 43.65625 24.015625 45.921875 \n", + "Q 28.375 48.1875 33.59375 48.1875 \n", + "Q 44.578125 48.1875 50.953125 41.515625 \n", + "Q 57.328125 34.859375 57.328125 23.390625 \n", + "Q 57.328125 12.15625 50.6875 5.359375 \n", + "Q 44.046875 -1.421875 33.015625 -1.421875 \n", + "Q 20.359375 -1.421875 13.671875 8.265625 \n", + "Q 6.984375 17.96875 6.984375 36.375 \n", + "Q 6.984375 53.65625 15.1875 63.9375 \n", + "Q 23.390625 74.21875 37.203125 74.21875 \n", + "Q 40.921875 74.21875 44.703125 73.484375 \n", + "Q 48.484375 72.75 52.59375 71.296875 \n", + "z\n", + "\" id=\"DejaVuSans-54\"/>\n", + " </defs>\n", + " <g transform=\"translate(230.193052 238.0025)rotate(-90)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-54\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"xtick_7\">\n", + " <g id=\"line2d_7\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"259.938532\" xlink:href=\"#m639e59548b\" y=\"224.64\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_7\">\n", + " <!-- 7 -->\n", + " <defs>\n", + " <path d=\"M 8.203125 72.90625 \n", + "L 55.078125 72.90625 \n", + "L 55.078125 68.703125 \n", + "L 28.609375 0 \n", + "L 18.3125 0 \n", + "L 43.21875 64.59375 \n", + "L 8.203125 64.59375 \n", + "z\n", + "\" id=\"DejaVuSans-55\"/>\n", + " </defs>\n", + " <g transform=\"translate(262.697907 238.0025)rotate(-90)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-55\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"xtick_8\">\n", + " <g id=\"line2d_8\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"292.443386\" xlink:href=\"#m639e59548b\" y=\"224.64\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_8\">\n", + " <!-- 8 -->\n", + " <defs>\n", + " <path d=\"M 31.78125 34.625 \n", + "Q 24.75 34.625 20.71875 30.859375 \n", + "Q 16.703125 27.09375 16.703125 20.515625 \n", + "Q 16.703125 13.921875 20.71875 10.15625 \n", + "Q 24.75 6.390625 31.78125 6.390625 \n", + "Q 38.8125 6.390625 42.859375 10.171875 \n", + "Q 46.921875 13.96875 46.921875 20.515625 \n", + "Q 46.921875 27.09375 42.890625 30.859375 \n", + "Q 38.875 34.625 31.78125 34.625 \n", + "z\n", + "M 21.921875 38.8125 \n", + "Q 15.578125 40.375 12.03125 44.71875 \n", + "Q 8.5 49.078125 8.5 55.328125 \n", + "Q 8.5 64.0625 14.71875 69.140625 \n", + "Q 20.953125 74.21875 31.78125 74.21875 \n", + "Q 42.671875 74.21875 48.875 69.140625 \n", + "Q 55.078125 64.0625 55.078125 55.328125 \n", + "Q 55.078125 49.078125 51.53125 44.71875 \n", + "Q 48 40.375 41.703125 38.8125 \n", + "Q 48.828125 37.15625 52.796875 32.3125 \n", + "Q 56.78125 27.484375 56.78125 20.515625 \n", + "Q 56.78125 9.90625 50.3125 4.234375 \n", + "Q 43.84375 -1.421875 31.78125 -1.421875 \n", + "Q 19.734375 -1.421875 13.25 4.234375 \n", + "Q 6.78125 9.90625 6.78125 20.515625 \n", + "Q 6.78125 27.484375 10.78125 32.3125 \n", + "Q 14.796875 37.15625 21.921875 38.8125 \n", + "z\n", + "M 18.3125 54.390625 \n", + "Q 18.3125 48.734375 21.84375 45.5625 \n", + "Q 25.390625 42.390625 31.78125 42.390625 \n", + "Q 38.140625 42.390625 41.71875 45.5625 \n", + "Q 45.3125 48.734375 45.3125 54.390625 \n", + "Q 45.3125 60.0625 41.71875 63.234375 \n", + "Q 38.140625 66.40625 31.78125 66.40625 \n", + "Q 25.390625 66.40625 21.84375 63.234375 \n", + "Q 18.3125 60.0625 18.3125 54.390625 \n", + "z\n", + "\" id=\"DejaVuSans-56\"/>\n", + " </defs>\n", + " <g transform=\"translate(295.202761 238.0025)rotate(-90)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-56\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"xtick_9\">\n", + " <g id=\"line2d_9\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"324.94824\" xlink:href=\"#m639e59548b\" y=\"224.64\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_9\">\n", + " <!-- 9 -->\n", + " <defs>\n", + " <path d=\"M 10.984375 1.515625 \n", + "L 10.984375 10.5 \n", + "Q 14.703125 8.734375 18.5 7.8125 \n", + "Q 22.3125 6.890625 25.984375 6.890625 \n", + "Q 35.75 6.890625 40.890625 13.453125 \n", + "Q 46.046875 20.015625 46.78125 33.40625 \n", + "Q 43.953125 29.203125 39.59375 26.953125 \n", + "Q 35.25 24.703125 29.984375 24.703125 \n", + "Q 19.046875 24.703125 12.671875 31.3125 \n", + "Q 6.296875 37.9375 6.296875 49.421875 \n", + "Q 6.296875 60.640625 12.9375 67.421875 \n", + "Q 19.578125 74.21875 30.609375 74.21875 \n", + "Q 43.265625 74.21875 49.921875 64.515625 \n", + "Q 56.59375 54.828125 56.59375 36.375 \n", + "Q 56.59375 19.140625 48.40625 8.859375 \n", + "Q 40.234375 -1.421875 26.421875 -1.421875 \n", + "Q 22.703125 -1.421875 18.890625 -0.6875 \n", + "Q 15.09375 0.046875 10.984375 1.515625 \n", + "z\n", + "M 30.609375 32.421875 \n", + "Q 37.25 32.421875 41.125 36.953125 \n", + "Q 45.015625 41.5 45.015625 49.421875 \n", + "Q 45.015625 57.28125 41.125 61.84375 \n", + "Q 37.25 66.40625 30.609375 66.40625 \n", + "Q 23.96875 66.40625 20.09375 61.84375 \n", + "Q 16.21875 57.28125 16.21875 49.421875 \n", + "Q 16.21875 41.5 20.09375 36.953125 \n", + "Q 23.96875 32.421875 30.609375 32.421875 \n", + "z\n", + "\" id=\"DejaVuSans-57\"/>\n", + " </defs>\n", + " <g transform=\"translate(327.707615 238.0025)rotate(-90)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-57\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"xtick_10\">\n", + " <g id=\"line2d_10\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"357.453095\" xlink:href=\"#m639e59548b\" y=\"224.64\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_10\">\n", + " <!-- 10 -->\n", + " <defs>\n", + " <path d=\"M 31.78125 66.40625 \n", + "Q 24.171875 66.40625 20.328125 58.90625 \n", + "Q 16.5 51.421875 16.5 36.375 \n", + "Q 16.5 21.390625 20.328125 13.890625 \n", + "Q 24.171875 6.390625 31.78125 6.390625 \n", + "Q 39.453125 6.390625 43.28125 13.890625 \n", + "Q 47.125 21.390625 47.125 36.375 \n", + "Q 47.125 51.421875 43.28125 58.90625 \n", + "Q 39.453125 66.40625 31.78125 66.40625 \n", + "z\n", + "M 31.78125 74.21875 \n", + "Q 44.046875 74.21875 50.515625 64.515625 \n", + "Q 56.984375 54.828125 56.984375 36.375 \n", + "Q 56.984375 17.96875 50.515625 8.265625 \n", + "Q 44.046875 -1.421875 31.78125 -1.421875 \n", + "Q 19.53125 -1.421875 13.0625 8.265625 \n", + "Q 6.59375 17.96875 6.59375 36.375 \n", + "Q 6.59375 54.828125 13.0625 64.515625 \n", + "Q 19.53125 74.21875 31.78125 74.21875 \n", + "z\n", + "\" id=\"DejaVuSans-48\"/>\n", + " </defs>\n", + " <g transform=\"translate(360.21247 244.365)rotate(-90)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-49\"/>\n", + " <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_11\">\n", + " <!-- Checkin index -->\n", + " <defs>\n", + " <path d=\"M 64.40625 67.28125 \n", + "L 64.40625 56.890625 \n", + "Q 59.421875 61.53125 53.78125 63.8125 \n", + "Q 48.140625 66.109375 41.796875 66.109375 \n", + "Q 29.296875 66.109375 22.65625 58.46875 \n", + "Q 16.015625 50.828125 16.015625 36.375 \n", + "Q 16.015625 21.96875 22.65625 14.328125 \n", + "Q 29.296875 6.6875 41.796875 6.6875 \n", + "Q 48.140625 6.6875 53.78125 8.984375 \n", + "Q 59.421875 11.28125 64.40625 15.921875 \n", + "L 64.40625 5.609375 \n", + "Q 59.234375 2.09375 53.4375 0.328125 \n", + "Q 47.65625 -1.421875 41.21875 -1.421875 \n", + "Q 24.65625 -1.421875 15.125 8.703125 \n", + "Q 5.609375 18.84375 5.609375 36.375 \n", + "Q 5.609375 53.953125 15.125 64.078125 \n", + "Q 24.65625 74.21875 41.21875 74.21875 \n", + "Q 47.75 74.21875 53.53125 72.484375 \n", + "Q 59.328125 70.75 64.40625 67.28125 \n", + "z\n", + "\" id=\"DejaVuSans-67\"/>\n", + " <path d=\"M 54.890625 33.015625 \n", + "L 54.890625 0 \n", + "L 45.90625 0 \n", + "L 45.90625 32.71875 \n", + "Q 45.90625 40.484375 42.875 44.328125 \n", + "Q 39.84375 48.1875 33.796875 48.1875 \n", + "Q 26.515625 48.1875 22.3125 43.546875 \n", + "Q 18.109375 38.921875 18.109375 30.90625 \n", + "L 18.109375 0 \n", + "L 9.078125 0 \n", + "L 9.078125 75.984375 \n", + "L 18.109375 75.984375 \n", + "L 18.109375 46.1875 \n", + "Q 21.34375 51.125 25.703125 53.5625 \n", + "Q 30.078125 56 35.796875 56 \n", + "Q 45.21875 56 50.046875 50.171875 \n", + "Q 54.890625 44.34375 54.890625 33.015625 \n", + "z\n", + "\" id=\"DejaVuSans-104\"/>\n", + " <path d=\"M 56.203125 29.59375 \n", + "L 56.203125 25.203125 \n", + "L 14.890625 25.203125 \n", + "Q 15.484375 15.921875 20.484375 11.0625 \n", + "Q 25.484375 6.203125 34.421875 6.203125 \n", + "Q 39.59375 6.203125 44.453125 7.46875 \n", + "Q 49.3125 8.734375 54.109375 11.28125 \n", + "L 54.109375 2.78125 \n", + "Q 49.265625 0.734375 44.1875 -0.34375 \n", + "Q 39.109375 -1.421875 33.890625 -1.421875 \n", + "Q 20.796875 -1.421875 13.15625 6.1875 \n", + "Q 5.515625 13.8125 5.515625 26.8125 \n", + "Q 5.515625 40.234375 12.765625 48.109375 \n", + "Q 20.015625 56 32.328125 56 \n", + "Q 43.359375 56 49.78125 48.890625 \n", + "Q 56.203125 41.796875 56.203125 29.59375 \n", + "z\n", + "M 47.21875 32.234375 \n", + "Q 47.125 39.59375 43.09375 43.984375 \n", + "Q 39.0625 48.390625 32.421875 48.390625 \n", + "Q 24.90625 48.390625 20.390625 44.140625 \n", + "Q 15.875 39.890625 15.1875 32.171875 \n", + "z\n", + "\" id=\"DejaVuSans-101\"/>\n", + " <path d=\"M 48.78125 52.59375 \n", + "L 48.78125 44.1875 \n", + "Q 44.96875 46.296875 41.140625 47.34375 \n", + "Q 37.3125 48.390625 33.40625 48.390625 \n", + "Q 24.65625 48.390625 19.8125 42.84375 \n", + "Q 14.984375 37.3125 14.984375 27.296875 \n", + "Q 14.984375 17.28125 19.8125 11.734375 \n", + "Q 24.65625 6.203125 33.40625 6.203125 \n", + "Q 37.3125 6.203125 41.140625 7.25 \n", + "Q 44.96875 8.296875 48.78125 10.40625 \n", + "L 48.78125 2.09375 \n", + "Q 45.015625 0.34375 40.984375 -0.53125 \n", + "Q 36.96875 -1.421875 32.421875 -1.421875 \n", + "Q 20.0625 -1.421875 12.78125 6.34375 \n", + "Q 5.515625 14.109375 5.515625 27.296875 \n", + "Q 5.515625 40.671875 12.859375 48.328125 \n", + "Q 20.21875 56 33.015625 56 \n", + "Q 37.15625 56 41.109375 55.140625 \n", + "Q 45.0625 54.296875 48.78125 52.59375 \n", + "z\n", + "\" id=\"DejaVuSans-99\"/>\n", + " <path d=\"M 9.078125 75.984375 \n", + "L 18.109375 75.984375 \n", + "L 18.109375 31.109375 \n", + "L 44.921875 54.6875 \n", + "L 56.390625 54.6875 \n", + "L 27.390625 29.109375 \n", + "L 57.625 0 \n", + "L 45.90625 0 \n", + "L 18.109375 26.703125 \n", + "L 18.109375 0 \n", + "L 9.078125 0 \n", + "z\n", + "\" id=\"DejaVuSans-107\"/>\n", + " <path d=\"M 9.421875 54.6875 \n", + "L 18.40625 54.6875 \n", + "L 18.40625 0 \n", + "L 9.421875 0 \n", + "z\n", + "M 9.421875 75.984375 \n", + "L 18.40625 75.984375 \n", + "L 18.40625 64.59375 \n", + "L 9.421875 64.59375 \n", + "z\n", + "\" id=\"DejaVuSans-105\"/>\n", + " <path d=\"M 54.890625 33.015625 \n", + "L 54.890625 0 \n", + "L 45.90625 0 \n", + "L 45.90625 32.71875 \n", + "Q 45.90625 40.484375 42.875 44.328125 \n", + "Q 39.84375 48.1875 33.796875 48.1875 \n", + "Q 26.515625 48.1875 22.3125 43.546875 \n", + "Q 18.109375 38.921875 18.109375 30.90625 \n", + "L 18.109375 0 \n", + "L 9.078125 0 \n", + "L 9.078125 54.6875 \n", + "L 18.109375 54.6875 \n", + "L 18.109375 46.1875 \n", + "Q 21.34375 51.125 25.703125 53.5625 \n", + "Q 30.078125 56 35.796875 56 \n", + "Q 45.21875 56 50.046875 50.171875 \n", + "Q 54.890625 44.34375 54.890625 33.015625 \n", + "z\n", + "\" id=\"DejaVuSans-110\"/>\n", + " <path id=\"DejaVuSans-32\"/>\n", + " <path d=\"M 45.40625 46.390625 \n", + "L 45.40625 75.984375 \n", + "L 54.390625 75.984375 \n", + "L 54.390625 0 \n", + "L 45.40625 0 \n", + "L 45.40625 8.203125 \n", + "Q 42.578125 3.328125 38.25 0.953125 \n", + "Q 33.9375 -1.421875 27.875 -1.421875 \n", + "Q 17.96875 -1.421875 11.734375 6.484375 \n", + "Q 5.515625 14.40625 5.515625 27.296875 \n", + "Q 5.515625 40.1875 11.734375 48.09375 \n", + "Q 17.96875 56 27.875 56 \n", + "Q 33.9375 56 38.25 53.625 \n", + "Q 42.578125 51.265625 45.40625 46.390625 \n", + "z\n", + "M 14.796875 27.296875 \n", + "Q 14.796875 17.390625 18.875 11.75 \n", + "Q 22.953125 6.109375 30.078125 6.109375 \n", + "Q 37.203125 6.109375 41.296875 11.75 \n", + "Q 45.40625 17.390625 45.40625 27.296875 \n", + "Q 45.40625 37.203125 41.296875 42.84375 \n", + "Q 37.203125 48.484375 30.078125 48.484375 \n", + "Q 22.953125 48.484375 18.875 42.84375 \n", + "Q 14.796875 37.203125 14.796875 27.296875 \n", + "z\n", + "\" id=\"DejaVuSans-100\"/>\n", + " <path d=\"M 54.890625 54.6875 \n", + "L 35.109375 28.078125 \n", + "L 55.90625 0 \n", + "L 45.3125 0 \n", + "L 29.390625 21.484375 \n", + "L 13.484375 0 \n", + "L 2.875 0 \n", + "L 24.125 28.609375 \n", + "L 4.6875 54.6875 \n", + "L 15.28125 54.6875 \n", + "L 29.78125 35.203125 \n", + "L 44.28125 54.6875 \n", + "z\n", + "\" id=\"DejaVuSans-120\"/>\n", + " </defs>\n", + " <g transform=\"translate(175.885938 255.963437)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-67\"/>\n", + " <use x=\"69.824219\" xlink:href=\"#DejaVuSans-104\"/>\n", + " <use x=\"133.203125\" xlink:href=\"#DejaVuSans-101\"/>\n", + " <use x=\"194.726562\" xlink:href=\"#DejaVuSans-99\"/>\n", + " <use x=\"249.707031\" xlink:href=\"#DejaVuSans-107\"/>\n", + " <use x=\"307.617188\" xlink:href=\"#DejaVuSans-105\"/>\n", + " <use x=\"335.400391\" xlink:href=\"#DejaVuSans-110\"/>\n", + " <use x=\"398.779297\" xlink:href=\"#DejaVuSans-32\"/>\n", + " <use x=\"430.566406\" xlink:href=\"#DejaVuSans-105\"/>\n", + " <use x=\"458.349609\" xlink:href=\"#DejaVuSans-110\"/>\n", + " <use x=\"521.728516\" xlink:href=\"#DejaVuSans-100\"/>\n", + " <use x=\"585.205078\" xlink:href=\"#DejaVuSans-101\"/>\n", + " <use x=\"646.712891\" xlink:href=\"#DejaVuSans-120\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"matplotlib.axis_2\">\n", + " <g id=\"ytick_1\">\n", + " <g id=\"line2d_11\">\n", + " <defs>\n", + " <path d=\"M 0 0 \n", + "L -3.5 0 \n", + "\" id=\"m7e5aed6441\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n", + " </defs>\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7e5aed6441\" y=\"224.64\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_12\">\n", + " <!-- 0.0 -->\n", + " <defs>\n", + " <path d=\"M 10.6875 12.40625 \n", + "L 21 12.40625 \n", + "L 21 0 \n", + "L 10.6875 0 \n", + "z\n", + "\" id=\"DejaVuSans-46\"/>\n", + " </defs>\n", + " <g transform=\"translate(20.878125 228.439219)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-48\"/>\n", + " <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n", + " <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"ytick_2\">\n", + " <g id=\"line2d_12\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7e5aed6441\" y=\"187.824762\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_13\">\n", + " <!-- 0.2 -->\n", + " <g transform=\"translate(20.878125 191.623981)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-48\"/>\n", + " <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n", + " <use x=\"95.410156\" xlink:href=\"#DejaVuSans-50\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"ytick_3\">\n", + " <g id=\"line2d_13\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7e5aed6441\" y=\"151.009524\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_14\">\n", + " <!-- 0.4 -->\n", + " <g transform=\"translate(20.878125 154.808743)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-48\"/>\n", + " <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n", + " <use x=\"95.410156\" xlink:href=\"#DejaVuSans-52\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"ytick_4\">\n", + " <g id=\"line2d_14\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7e5aed6441\" y=\"114.194286\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_15\">\n", + " <!-- 0.6 -->\n", + " <g transform=\"translate(20.878125 117.993504)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-48\"/>\n", + " <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n", + " <use x=\"95.410156\" xlink:href=\"#DejaVuSans-54\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"ytick_5\">\n", + " <g id=\"line2d_15\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7e5aed6441\" y=\"77.379048\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_16\">\n", + " <!-- 0.8 -->\n", + " <g transform=\"translate(20.878125 81.178266)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-48\"/>\n", + " <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n", + " <use x=\"95.410156\" xlink:href=\"#DejaVuSans-56\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"ytick_6\">\n", + " <g id=\"line2d_16\">\n", + " <g>\n", + " <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"43.78125\" xlink:href=\"#m7e5aed6441\" y=\"40.56381\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_17\">\n", + " <!-- 1.0 -->\n", + " <g transform=\"translate(20.878125 44.363028)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-49\"/>\n", + " <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n", + " <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"text_18\">\n", + " <!-- Repo size (MiB) -->\n", + " <defs>\n", + " <path d=\"M 44.390625 34.1875 \n", + "Q 47.5625 33.109375 50.5625 29.59375 \n", + "Q 53.5625 26.078125 56.59375 19.921875 \n", + "L 66.609375 0 \n", + "L 56 0 \n", + "L 46.6875 18.703125 \n", + "Q 43.0625 26.03125 39.671875 28.421875 \n", + "Q 36.28125 30.8125 30.421875 30.8125 \n", + "L 19.671875 30.8125 \n", + "L 19.671875 0 \n", + "L 9.8125 0 \n", + "L 9.8125 72.90625 \n", + "L 32.078125 72.90625 \n", + "Q 44.578125 72.90625 50.734375 67.671875 \n", + "Q 56.890625 62.453125 56.890625 51.90625 \n", + "Q 56.890625 45.015625 53.6875 40.46875 \n", + "Q 50.484375 35.9375 44.390625 34.1875 \n", + "z\n", + "M 19.671875 64.796875 \n", + "L 19.671875 38.921875 \n", + "L 32.078125 38.921875 \n", + "Q 39.203125 38.921875 42.84375 42.21875 \n", + "Q 46.484375 45.515625 46.484375 51.90625 \n", + "Q 46.484375 58.296875 42.84375 61.546875 \n", + "Q 39.203125 64.796875 32.078125 64.796875 \n", + "z\n", + "\" id=\"DejaVuSans-82\"/>\n", + " <path d=\"M 18.109375 8.203125 \n", + "L 18.109375 -20.796875 \n", + "L 9.078125 -20.796875 \n", + "L 9.078125 54.6875 \n", + "L 18.109375 54.6875 \n", + "L 18.109375 46.390625 \n", + "Q 20.953125 51.265625 25.265625 53.625 \n", + "Q 29.59375 56 35.59375 56 \n", + "Q 45.5625 56 51.78125 48.09375 \n", + "Q 58.015625 40.1875 58.015625 27.296875 \n", + "Q 58.015625 14.40625 51.78125 6.484375 \n", + "Q 45.5625 -1.421875 35.59375 -1.421875 \n", + "Q 29.59375 -1.421875 25.265625 0.953125 \n", + "Q 20.953125 3.328125 18.109375 8.203125 \n", + "z\n", + "M 48.6875 27.296875 \n", + "Q 48.6875 37.203125 44.609375 42.84375 \n", + "Q 40.53125 48.484375 33.40625 48.484375 \n", + "Q 26.265625 48.484375 22.1875 42.84375 \n", + "Q 18.109375 37.203125 18.109375 27.296875 \n", + "Q 18.109375 17.390625 22.1875 11.75 \n", + "Q 26.265625 6.109375 33.40625 6.109375 \n", + "Q 40.53125 6.109375 44.609375 11.75 \n", + "Q 48.6875 17.390625 48.6875 27.296875 \n", + "z\n", + "\" id=\"DejaVuSans-112\"/>\n", + " <path d=\"M 30.609375 48.390625 \n", + "Q 23.390625 48.390625 19.1875 42.75 \n", + "Q 14.984375 37.109375 14.984375 27.296875 \n", + "Q 14.984375 17.484375 19.15625 11.84375 \n", + "Q 23.34375 6.203125 30.609375 6.203125 \n", + "Q 37.796875 6.203125 41.984375 11.859375 \n", + "Q 46.1875 17.53125 46.1875 27.296875 \n", + "Q 46.1875 37.015625 41.984375 42.703125 \n", + "Q 37.796875 48.390625 30.609375 48.390625 \n", + "z\n", + "M 30.609375 56 \n", + "Q 42.328125 56 49.015625 48.375 \n", + "Q 55.71875 40.765625 55.71875 27.296875 \n", + "Q 55.71875 13.875 49.015625 6.21875 \n", + "Q 42.328125 -1.421875 30.609375 -1.421875 \n", + "Q 18.84375 -1.421875 12.171875 6.21875 \n", + "Q 5.515625 13.875 5.515625 27.296875 \n", + "Q 5.515625 40.765625 12.171875 48.375 \n", + "Q 18.84375 56 30.609375 56 \n", + "z\n", + "\" id=\"DejaVuSans-111\"/>\n", + " <path d=\"M 44.28125 53.078125 \n", + "L 44.28125 44.578125 \n", + "Q 40.484375 46.53125 36.375 47.5 \n", + "Q 32.28125 48.484375 27.875 48.484375 \n", + "Q 21.1875 48.484375 17.84375 46.4375 \n", + "Q 14.5 44.390625 14.5 40.28125 \n", + "Q 14.5 37.15625 16.890625 35.375 \n", + "Q 19.28125 33.59375 26.515625 31.984375 \n", + "L 29.59375 31.296875 \n", + "Q 39.15625 29.25 43.1875 25.515625 \n", + "Q 47.21875 21.78125 47.21875 15.09375 \n", + "Q 47.21875 7.46875 41.1875 3.015625 \n", + "Q 35.15625 -1.421875 24.609375 -1.421875 \n", + "Q 20.21875 -1.421875 15.453125 -0.5625 \n", + "Q 10.6875 0.296875 5.421875 2 \n", + "L 5.421875 11.28125 \n", + "Q 10.40625 8.6875 15.234375 7.390625 \n", + "Q 20.0625 6.109375 24.8125 6.109375 \n", + "Q 31.15625 6.109375 34.5625 8.28125 \n", + "Q 37.984375 10.453125 37.984375 14.40625 \n", + "Q 37.984375 18.0625 35.515625 20.015625 \n", + "Q 33.0625 21.96875 24.703125 23.78125 \n", + "L 21.578125 24.515625 \n", + "Q 13.234375 26.265625 9.515625 29.90625 \n", + "Q 5.8125 33.546875 5.8125 39.890625 \n", + "Q 5.8125 47.609375 11.28125 51.796875 \n", + "Q 16.75 56 26.8125 56 \n", + "Q 31.78125 56 36.171875 55.265625 \n", + "Q 40.578125 54.546875 44.28125 53.078125 \n", + "z\n", + "\" id=\"DejaVuSans-115\"/>\n", + " <path d=\"M 5.515625 54.6875 \n", + "L 48.1875 54.6875 \n", + "L 48.1875 46.484375 \n", + "L 14.40625 7.171875 \n", + "L 48.1875 7.171875 \n", + "L 48.1875 0 \n", + "L 4.296875 0 \n", + "L 4.296875 8.203125 \n", + "L 38.09375 47.515625 \n", + "L 5.515625 47.515625 \n", + "z\n", + "\" id=\"DejaVuSans-122\"/>\n", + " <path d=\"M 31 75.875 \n", + "Q 24.46875 64.65625 21.28125 53.65625 \n", + "Q 18.109375 42.671875 18.109375 31.390625 \n", + "Q 18.109375 20.125 21.3125 9.0625 \n", + "Q 24.515625 -2 31 -13.1875 \n", + "L 23.1875 -13.1875 \n", + "Q 15.875 -1.703125 12.234375 9.375 \n", + "Q 8.59375 20.453125 8.59375 31.390625 \n", + "Q 8.59375 42.28125 12.203125 53.3125 \n", + "Q 15.828125 64.359375 23.1875 75.875 \n", + "z\n", + "\" id=\"DejaVuSans-40\"/>\n", + " <path d=\"M 9.8125 72.90625 \n", + "L 24.515625 72.90625 \n", + "L 43.109375 23.296875 \n", + "L 61.8125 72.90625 \n", + "L 76.515625 72.90625 \n", + "L 76.515625 0 \n", + "L 66.890625 0 \n", + "L 66.890625 64.015625 \n", + "L 48.09375 14.015625 \n", + "L 38.1875 14.015625 \n", + "L 19.390625 64.015625 \n", + "L 19.390625 0 \n", + "L 9.8125 0 \n", + "z\n", + "\" id=\"DejaVuSans-77\"/>\n", + " <path d=\"M 19.671875 34.8125 \n", + "L 19.671875 8.109375 \n", + "L 35.5 8.109375 \n", + "Q 43.453125 8.109375 47.28125 11.40625 \n", + "Q 51.125 14.703125 51.125 21.484375 \n", + "Q 51.125 28.328125 47.28125 31.5625 \n", + "Q 43.453125 34.8125 35.5 34.8125 \n", + "z\n", + "M 19.671875 64.796875 \n", + "L 19.671875 42.828125 \n", + "L 34.28125 42.828125 \n", + "Q 41.5 42.828125 45.03125 45.53125 \n", + "Q 48.578125 48.25 48.578125 53.8125 \n", + "Q 48.578125 59.328125 45.03125 62.0625 \n", + "Q 41.5 64.796875 34.28125 64.796875 \n", + "z\n", + "M 9.8125 72.90625 \n", + "L 35.015625 72.90625 \n", + "Q 46.296875 72.90625 52.390625 68.21875 \n", + "Q 58.5 63.53125 58.5 54.890625 \n", + "Q 58.5 48.1875 55.375 44.234375 \n", + "Q 52.25 40.28125 46.1875 39.3125 \n", + "Q 53.46875 37.75 57.5 32.78125 \n", + "Q 61.53125 27.828125 61.53125 20.40625 \n", + "Q 61.53125 10.640625 54.890625 5.3125 \n", + "Q 48.25 0 35.984375 0 \n", + "L 9.8125 0 \n", + "z\n", + "\" id=\"DejaVuSans-66\"/>\n", + " <path d=\"M 8.015625 75.875 \n", + "L 15.828125 75.875 \n", + "Q 23.140625 64.359375 26.78125 53.3125 \n", + "Q 30.421875 42.28125 30.421875 31.390625 \n", + "Q 30.421875 20.453125 26.78125 9.375 \n", + "Q 23.140625 -1.703125 15.828125 -13.1875 \n", + "L 8.015625 -13.1875 \n", + "Q 14.5 -2 17.703125 9.0625 \n", + "Q 20.90625 20.125 20.90625 31.390625 \n", + "Q 20.90625 42.671875 17.703125 53.65625 \n", + "Q 14.5 64.65625 8.015625 75.875 \n", + "z\n", + "\" id=\"DejaVuSans-41\"/>\n", + " </defs>\n", + " <g transform=\"translate(14.798438 154.609062)rotate(-90)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-82\"/>\n", + " <use x=\"69.419922\" xlink:href=\"#DejaVuSans-101\"/>\n", + " <use x=\"130.943359\" xlink:href=\"#DejaVuSans-112\"/>\n", + " <use x=\"194.419922\" xlink:href=\"#DejaVuSans-111\"/>\n", + " <use x=\"255.601562\" xlink:href=\"#DejaVuSans-32\"/>\n", + " <use x=\"287.388672\" xlink:href=\"#DejaVuSans-115\"/>\n", + " <use x=\"339.488281\" xlink:href=\"#DejaVuSans-105\"/>\n", + " <use x=\"367.271484\" xlink:href=\"#DejaVuSans-122\"/>\n", + " <use x=\"419.761719\" xlink:href=\"#DejaVuSans-101\"/>\n", + " <use x=\"481.285156\" xlink:href=\"#DejaVuSans-32\"/>\n", + " <use x=\"513.072266\" xlink:href=\"#DejaVuSans-40\"/>\n", + " <use x=\"552.085938\" xlink:href=\"#DejaVuSans-77\"/>\n", + " <use x=\"638.365234\" xlink:href=\"#DejaVuSans-105\"/>\n", + " <use x=\"666.148438\" xlink:href=\"#DejaVuSans-66\"/>\n", + " <use x=\"734.751953\" xlink:href=\"#DejaVuSans-41\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <g id=\"patch_43\">\n", + " <path d=\"M 43.78125 224.64 \n", + "L 43.78125 7.2 \n", + "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n", + " </g>\n", + " <g id=\"patch_44\">\n", + " <path d=\"M 378.58125 224.64 \n", + "L 378.58125 7.2 \n", + "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n", + " </g>\n", + " <g id=\"patch_45\">\n", + " <path d=\"M 43.78125 224.64 \n", + "L 378.58125 224.64 \n", + "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n", + " </g>\n", + " <g id=\"patch_46\">\n", + " <path d=\"M 43.78125 7.2 \n", + "L 378.58125 7.2 \n", + "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n", + " </g>\n", + " <g id=\"legend_1\">\n", + " <g id=\"patch_47\">\n", + " <path d=\"M 50.78125 73.9125 \n", + "L 105.828125 73.9125 \n", + "Q 107.828125 73.9125 107.828125 71.9125 \n", + "L 107.828125 14.2 \n", + "Q 107.828125 12.2 105.828125 12.2 \n", + "L 50.78125 12.2 \n", + "Q 48.78125 12.2 48.78125 14.2 \n", + "L 48.78125 71.9125 \n", + "Q 48.78125 73.9125 50.78125 73.9125 \n", + "z\n", + "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n", + " </g>\n", + " <g id=\"patch_48\">\n", + " <path d=\"M 52.78125 23.798437 \n", + "L 72.78125 23.798437 \n", + "L 72.78125 16.798437 \n", + "L 52.78125 16.798437 \n", + "z\n", + "\" style=\"fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"text_19\">\n", + " <!-- JPEG -->\n", + " <defs>\n", + " <path d=\"M 9.8125 72.90625 \n", + "L 19.671875 72.90625 \n", + "L 19.671875 5.078125 \n", + "Q 19.671875 -8.109375 14.671875 -14.0625 \n", + "Q 9.671875 -20.015625 -1.421875 -20.015625 \n", + "L -5.171875 -20.015625 \n", + "L -5.171875 -11.71875 \n", + "L -2.09375 -11.71875 \n", + "Q 4.4375 -11.71875 7.125 -8.046875 \n", + "Q 9.8125 -4.390625 9.8125 5.078125 \n", + "z\n", + "\" id=\"DejaVuSans-74\"/>\n", + " <path d=\"M 19.671875 64.796875 \n", + "L 19.671875 37.40625 \n", + "L 32.078125 37.40625 \n", + "Q 38.96875 37.40625 42.71875 40.96875 \n", + "Q 46.484375 44.53125 46.484375 51.125 \n", + "Q 46.484375 57.671875 42.71875 61.234375 \n", + "Q 38.96875 64.796875 32.078125 64.796875 \n", + "z\n", + "M 9.8125 72.90625 \n", + "L 32.078125 72.90625 \n", + "Q 44.34375 72.90625 50.609375 67.359375 \n", + "Q 56.890625 61.8125 56.890625 51.125 \n", + "Q 56.890625 40.328125 50.609375 34.8125 \n", + "Q 44.34375 29.296875 32.078125 29.296875 \n", + "L 19.671875 29.296875 \n", + "L 19.671875 0 \n", + "L 9.8125 0 \n", + "z\n", + "\" id=\"DejaVuSans-80\"/>\n", + " <path d=\"M 9.8125 72.90625 \n", + "L 55.90625 72.90625 \n", + "L 55.90625 64.59375 \n", + "L 19.671875 64.59375 \n", + "L 19.671875 43.015625 \n", + "L 54.390625 43.015625 \n", + "L 54.390625 34.71875 \n", + "L 19.671875 34.71875 \n", + "L 19.671875 8.296875 \n", + "L 56.78125 8.296875 \n", + "L 56.78125 0 \n", + "L 9.8125 0 \n", + "z\n", + "\" id=\"DejaVuSans-69\"/>\n", + " <path d=\"M 59.515625 10.40625 \n", + "L 59.515625 29.984375 \n", + "L 43.40625 29.984375 \n", + "L 43.40625 38.09375 \n", + "L 69.28125 38.09375 \n", + "L 69.28125 6.78125 \n", + "Q 63.578125 2.734375 56.6875 0.65625 \n", + "Q 49.8125 -1.421875 42 -1.421875 \n", + "Q 24.90625 -1.421875 15.25 8.5625 \n", + "Q 5.609375 18.5625 5.609375 36.375 \n", + "Q 5.609375 54.25 15.25 64.234375 \n", + "Q 24.90625 74.21875 42 74.21875 \n", + "Q 49.125 74.21875 55.546875 72.453125 \n", + "Q 61.96875 70.703125 67.390625 67.28125 \n", + "L 67.390625 56.78125 \n", + "Q 61.921875 61.421875 55.765625 63.765625 \n", + "Q 49.609375 66.109375 42.828125 66.109375 \n", + "Q 29.4375 66.109375 22.71875 58.640625 \n", + "Q 16.015625 51.171875 16.015625 36.375 \n", + "Q 16.015625 21.625 22.71875 14.15625 \n", + "Q 29.4375 6.6875 42.828125 6.6875 \n", + "Q 48.046875 6.6875 52.140625 7.59375 \n", + "Q 56.25 8.5 59.515625 10.40625 \n", + "z\n", + "\" id=\"DejaVuSans-71\"/>\n", + " </defs>\n", + " <g transform=\"translate(80.78125 23.798437)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-74\"/>\n", + " <use x=\"29.492188\" xlink:href=\"#DejaVuSans-80\"/>\n", + " <use x=\"89.794922\" xlink:href=\"#DejaVuSans-69\"/>\n", + " <use x=\"152.978516\" xlink:href=\"#DejaVuSans-71\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"patch_49\">\n", + " <path d=\"M 52.78125 38.476562 \n", + "L 72.78125 38.476562 \n", + "L 72.78125 31.476562 \n", + "L 52.78125 31.476562 \n", + "z\n", + "\" style=\"fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"text_20\">\n", + " <!-- BMP -->\n", + " <g transform=\"translate(80.78125 38.476562)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-66\"/>\n", + " <use x=\"68.603516\" xlink:href=\"#DejaVuSans-77\"/>\n", + " <use x=\"154.882812\" xlink:href=\"#DejaVuSans-80\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"patch_50\">\n", + " <path d=\"M 52.78125 53.154687 \n", + "L 72.78125 53.154687 \n", + "L 72.78125 46.154687 \n", + "L 52.78125 46.154687 \n", + "z\n", + "\" style=\"fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"text_21\">\n", + " <!-- TIFF -->\n", + " <defs>\n", + " <path d=\"M -0.296875 72.90625 \n", + "L 61.375 72.90625 \n", + "L 61.375 64.59375 \n", + "L 35.5 64.59375 \n", + "L 35.5 0 \n", + "L 25.59375 0 \n", + "L 25.59375 64.59375 \n", + "L -0.296875 64.59375 \n", + "z\n", + "\" id=\"DejaVuSans-84\"/>\n", + " <path d=\"M 9.8125 72.90625 \n", + "L 19.671875 72.90625 \n", + "L 19.671875 0 \n", + "L 9.8125 0 \n", + "z\n", + "\" id=\"DejaVuSans-73\"/>\n", + " <path d=\"M 9.8125 72.90625 \n", + "L 51.703125 72.90625 \n", + "L 51.703125 64.59375 \n", + "L 19.671875 64.59375 \n", + "L 19.671875 43.109375 \n", + "L 48.578125 43.109375 \n", + "L 48.578125 34.8125 \n", + "L 19.671875 34.8125 \n", + "L 19.671875 0 \n", + "L 9.8125 0 \n", + "z\n", + "\" id=\"DejaVuSans-70\"/>\n", + " </defs>\n", + " <g transform=\"translate(80.78125 53.154687)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-84\"/>\n", + " <use x=\"61.083984\" xlink:href=\"#DejaVuSans-73\"/>\n", + " <use x=\"90.576172\" xlink:href=\"#DejaVuSans-70\"/>\n", + " <use x=\"148.095703\" xlink:href=\"#DejaVuSans-70\"/>\n", + " </g>\n", + " </g>\n", + " <g id=\"patch_51\">\n", + " <path d=\"M 52.78125 67.832812 \n", + "L 72.78125 67.832812 \n", + "L 72.78125 60.832812 \n", + "L 52.78125 60.832812 \n", + "z\n", + "\" style=\"fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;\"/>\n", + " </g>\n", + " <g id=\"text_22\">\n", + " <!-- PNG -->\n", + " <defs>\n", + " <path d=\"M 9.8125 72.90625 \n", + "L 23.09375 72.90625 \n", + "L 55.421875 11.921875 \n", + "L 55.421875 72.90625 \n", + "L 64.984375 72.90625 \n", + "L 64.984375 0 \n", + "L 51.703125 0 \n", + "L 19.390625 60.984375 \n", + "L 19.390625 0 \n", + "L 9.8125 0 \n", + "z\n", + "\" id=\"DejaVuSans-78\"/>\n", + " </defs>\n", + " <g transform=\"translate(80.78125 67.832812)scale(0.1 -0.1)\">\n", + " <use xlink:href=\"#DejaVuSans-80\"/>\n", + " <use x=\"60.302734\" xlink:href=\"#DejaVuSans-78\"/>\n", + " <use x=\"135.107422\" xlink:href=\"#DejaVuSans-71\"/>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " </g>\n", + " <defs>\n", + " <clipPath id=\"p70301a0bed\">\n", + " <rect height=\"217.44\" width=\"334.8\" x=\"43.78125\" y=\"7.2\"/>\n", + " </clipPath>\n", + " </defs>\n", + "</svg>\n" + ], + "text/plain": [ + "<Figure size 432x288 with 1 Axes>" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "%config InlineBackend.figure_formats = ['svg']\n", + "\n", + "import matplotlib as mpl\n", + "import matplotlib.pyplot as plt\n", + "import pandas as pd\n", + "\n", + "os.chdir(tdir)\n", + "\n", + "# Merge per-format test data into a single DataFrame without the first\n", + "# first row, being the boring initial empty repo state.\n", + "data = pd.concat(repo_sizes, axis=1).drop(range(1))\n", + "\n", + "mpl.rcParams['figure.figsize'] = (6, 4)\n", + "ax = data.plot(kind = 'bar', colormap = 'coolwarm',\n", + " grid = False, width = 0.8,\n", + " edgecolor = 'white', linewidth = 2)\n", + "ax.axes.set_xlabel('Checkin index')\n", + "ax.axes.set_ylabel('Repo size (MiB)')\n", + "plt.savefig('image-format-vs-repo-size.svg', transparent=True)\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.6" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/www/image-format-vs-repo-size.md b/www/image-format-vs-repo-size.md new file mode 100644 index 0000000..9b8d4a7 --- /dev/null +++ b/www/image-format-vs-repo-size.md @@ -0,0 +1,302 @@ +# Image Format vs Fossil Repo Size + +## The Problem + +Fossil has a [delta compression][dc] feature which removes redundant +information from a file relative to its parent on check-in.[^delta-prgs] +That delta is then [zlib][zl]-compressed before being stored +in the Fossil repository database file. + +Storing pre-compressed data files in a Fossil repository defeats both of +these space-saving measures: + +1. Binary data compression algorithms turn the file data into + pseudorandom noise.[^prn] + + Typical data compression algorithms are not [hash functions][hf], + where the goal is that a change to each bit in the input has a + statistically even chance of changing every bit in the output, but + because they do approach that pathological condition, pre-compressed + data tends to defeat Fossil’s delta compression algorithm, there + being so little correlation between two different outputs from the + binary data compression algorithm. + +2. An ideal lossless binary data compression algorithm cannot be + applied more than once to make the data even smaller, since random + noise is incompressible. The consequence for our purposes here is + that pre-compressed data doesn’t benefit from Fossil’s zlib + compression. + +You might then ask, what does it matter if the space savings comes from +the application file format (e.g. JPEG, DOCX, Zip, etc.) or from Fossil +itself? It really doesn’t, as far as point 2 above goes, but point 1 +causes the Fossil repository to balloon out of proportion to the size of +the input data change on each checkin. This article will illustrate that +problem, quantify it, and give a solution to it. + +[dc]: ./delta_format.wiki +[hf]: https://en.wikipedia.org/wiki/Hash_function +[zl]: http://www.zlib.net/ + + +## <a id="formats"></a>Affected File Formats + +In this article’s core experiment, we use 2D image file formats, but +this article’s advice also applies to many other file types. For just a +few examples out of what must be thousands: + +* **Microsoft Office**: The [OOXML document format][oox] used from + Office 2003 onward (`.docx`, `.xlsx`, `.pptx`, etc.) are Zip files + containing an XML document file and several collateral files. + +* **Libre Office**: For the purposes of this article, its + [OpenDocument Format][odf] is designed the same basic way as OOXML. + +* **Java**: A Java [`.jar` file][jcl] is a Zip file containing JVM + `.class` files, manifest files, and more. + +* **Windows Installer:** An [`*.msi` file][wi] is a proprietary + database format that contains, among other things, [Microsoft + Cabinet][cab]-compressed files, which in turn may hold Windows + executables, which [may themselves be compressed][exc]. + +* **SVG, PDF, TIFF, etc.**: Many file formats are available in both + compressed and uncompressed forms. You should use the uncompressed + form with Fossil wherever practical, as we will show below. + + +[cab]: https://en.wikipedia.org/wiki/Cabinet_(file_format) +[exc]: https://en.wikipedia.org/wiki/Executable_compression +[jcl]: https://en.wikipedia.org/wiki/Java_(programming_language) +[odf]: https://en.wikipedia.org/wiki/OpenDocument +[oox]: https://en.wikipedia.org/wiki/Office_Open_XML +[wi]: https://en.wikipedia.org/wiki/Windows_Installer + + + +## <a id="demo"></a>Demonstration + +The companion `image-format-vs-repo-size.ipynb` file ([download][nbd], +[preview][nbp]) is a [JupyterLab][jl] notebook implementing the following +experiment: + +1. Create a new minimum-size Fossil repository. Save this initial size. + +2. Use [ImageMagick][im] via [Wand][wp] to generate a JPEG file of a + particular size — currently 256 px² — filled with Gaussian noise to + make data compression more difficult than with a solid-color image. + +3. Check that image into the new Fossil repo, and remember that size. + +4. Change a random pixel in the image to a random RGB value, save that + image, check it in, and remember the new Fossil repo size. + +5. Iterate on step 4 some number of times — currently 10 — and remember + the Fossil repo size at each step. + +6. Repeat the above steps for BMP, PNG, and TIFF.[^tiff-cmp] + +7. Create a bar chart showing how the Fossil repository size changes + with each checkin. + +We chose to use JupyterLab for this because it makes it easy for you to +modify the notebook to try different things. Want to see how the +results change with a different image size? Easy, change the `size` +value in the second cell of the notebook. Want to try more image +formats? You can put anything ImageMagick can recognize into the +`formats` list. Want to find the break-even point for images like those +in your own repository? Easily done with a small amount of code. + +[im]: https://www.imagemagick.org/ +[jl]: https://jupyter.org/ +[nbd]: ./image-format-vs-repo-size.ipynb +[nbp]: https://nbviewer.jupyter.org/urls/fossil-scm.org/fossil/doc/trunk/www/image-format-vs-repo-size.ipynb +[wp]: http://wand-py.org/ + + +## <a id="results"></a>Results + +Running the notebook gives a bar chart something like[^variance] this: + + + +There are a few key things we want to draw your attention to in that +chart: + +* BMP and uncompressed TIFF are nearly identical in size for all + checkins, and the repository growth rate is negligible past the + first commit.[^size-jump] We owe this economy to Fossil’s delta compression + feature: it is encoding each of those single-pixel changes in a very + small amount of repository space. + +* The JPEG and PNG bars increase by large amounts on most checkins + even though each checkin *also* encodes only a *single-pixel change*. + +* The size of the first checkin in the BMP and TIFF cases is roughly + the same as that for the PNG case, because both PNG and Fossil use + the zlib binary data compression algorithm. This shows that for + repos where the image files are committed only once, there is + virtually no penalty to using BMP or TIFF over PNG. The file sizes + likely differ only because of differences in zlib settings between + the cases. + +* Because JPEG’s lossy nature allows it to start smaller and have + smaller size increases than PNG, the crossover point with + BMP/TIFF isn’t until 7-9 checkins in typical runs of this [Monte + Carlo experiment][mce]. Given a choice among these four file + formats and a willingness to use lossy image compression, a rational + tradeoff is to choose JPEG for repositories where each image will + change fewer than that number of times. + + +[mce]: https://en.wikipedia.org/wiki/Monte_Carlo_method + + +## <a id="makefile"></a>Automated Recompression + +Since programs that produce and consume binary-compressed data files +often make it either difficult or impossible to work with the +uncompressed form, we want an automated method for producing the +uncompressed form to make Fossil happy while still having the compressed +form to keep our content creation applications happy. This `Makefile` +should[^makefile] do that for BMP, PNG, SVG, and XLSX files: + + .SUFFIXES: .bmp .png .svg .svgz + + .svgz.svg: + gzip -dc < $< > $@ + + .svg.svgz: + gzip -9c < $< > $@ + + .bmp.png: + convert -quality 95 $< $@ + + .png.bmp: + convert $< $@ + + SS_FILES := $(wildcard spreadsheet/*) + + + all: $(SS_FILES) illus.svg image.bmp doc-big.pdf + + reconstitute: illus.svgz image.png + ( cd spreadsheet ; zip -9 ../spreadsheet.xlsx) * ) + qpdf doc-big.pdf doc-small.pdf + + + $(SS_FILES): spreadsheet.xlsx + unzip $@ -d $< + + doc-big.pdf: doc-small.pdf + qpdf --stream-data=uncompress $@ $< + +This `Makefile` allows you to treat the compressed version as the +process input, but to actually check in only the changes against the +uncompressed version by typing “`make`” before “`fossil ci`”. This is +not actually an extra step in practice, since if you’ve got a +`Makefile`-based project, you should be building — and testing — it +before checking each change in anyway! + +Because this technique is based on dependency rules, only the necessary +files are generated on each `make` command. + +You only have to run “`make reconstitute`” *once* after opening a fresh +Fossil checkout to produce those compressed sources. After that, you +work with the compressed files in your content creation programs. Your +build system might include some kind of bootstrapping or +auto-configuration step that you could attach this to, so that it +doesn’t need to be run by hand. + +This `Makefile` illustrates two primary strategies: + + +### Input and Output File Formats Differ by Extension + +In the case of SVG and the bitmap image formats, the file name extension +differs between the cases, so we can use `make` suffix rules to get the +behavior we want. The top half of the `Makefile` just tells `make` how +to map from `*.svg` to `*.svgz` and vice versa, and the same for `*.bmp` +to/from `*.png`. + + +### Input and Output Use the Same Extension + +We don’t have that luxury for Excel and PDF files, each for a different +reason: + +* **Excel:** Excel has no way to work with the unpacked Zip file + contents at all, so we have to unpack it into a subdirectory, which + is what we check into Fossil. On making a fresh Fossil checkout, we + have to pack that subdirectory’s contents back up into an `*.xlsx` + file with “`make reconstitute`” so we can edit it with Excel again. + +* **PDF:** All PDF readers can display an uncompressed PDF file, but + many PDF-*producing* programs have no option for uncompressed + output. Since the file name extension is the same either way, we + treat the compressed PDF as the source to the process, yielding an + automatically-uncompressed PDF for the benefit of Fossil. Unlike + with the Excel case, there is no simple “file base name to directory + name” mapping, so we just created the `-big` to `-small` name scheme + here. + + +[^delta-prgs]: + This problem is not Fossil-specific. Several other programs also do + delta compression, so they’ll also be affected by this problem: + [rsync][rs], [Unison][us], [Git][git], etc. You should take this + article’s advice when using all such programs, not just Fossil. + When using file copying and synchronization programs *without* delta + compression, on the other hand, it’s best to use the most + highly-compressed file format you can tolerate, since they copy the + whole file any time any bit of it changes. + +[^prn]: + In fact, a good way to gauge the effectiveness of a given + compression scheme is to run its output through the same sort of + tests we use to gauge how “random” a given [PRNG][prng] is. Another + way to look at it is that if there is a discernible pattern in the + output of a compression scheme, that constitutes *information* (in + [the technical sense of that word][ith]) that could be further + compressed. + +[^tiff-cmp]: + We're using *uncompressed* TIFF here, not [LZW][lzw]- or + Zip-compressed TIFF, either of which would give similar results to + PNG, which is always zlib-compressed. + +[^variance]: + The raw data changes somewhat from one run to the next due to the + use of random noise in the image to make the zlib/PNG compression + more difficult, and the random pixel changes. Those test design + choices make this a [Monte Carlo experiment][mce]. We’ve found that + the overall character of the results doesn’t change from one run to + the next. + +[^size-jump]: + It’s not clear to me why there is a one-time jump in size for BMP + and TIFF past the first commit. I suspect it is due to the SQLite + indices being initialized for the first time. + Page size inflation might have something to do with it as well, + though we tried to control that by rebuilding the initial DB with a + minimal page size. If you re-run the program often enough, you will + sometimes see the BMP or TIFF bar jump higher than the other, again + likely due to one of the repos crossing a page boundary. + Another curious artifact in the data is that the BMP is slightly + larger than for the TIFF. This goes against expectation because a + low-tech format like BMP should have a small edge in this test + because TIFF metadata includes the option for multiple timestamps, + UUIDs, etc., which bloat the checkin size by creating many small + deltas. + +[^makefile]: + The `Makefile` above is not battle-tested. Please report bugs and + needed extensions [on the forum][for]. + +[for]: https://fossil-scm.org/forum/forumpost/15e677f2c8 +[git]: https://git-scm.com/ +[ith]: https://en.wikipedia.org/wiki/Information_theory +[lzw]: https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Welch +[prng]: https://en.wikipedia.org/wiki/Pseudorandom_number_generator +[rs]: https://rsync.samba.org/ +[us]: http://www.cis.upenn.edu/~bcpierce/unison/ diff --git a/www/image-format-vs-repo-size.svg b/www/image-format-vs-repo-size.svg new file mode 100644 index 0000000..3a3a282 --- /dev/null +++ b/www/image-format-vs-repo-size.svg @@ -0,0 +1,1472 @@ +<?xml version="1.0" encoding="utf-8" standalone="no"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" + "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<!-- Created with matplotlib (https://matplotlib.org/) --> +<svg height="288pt" version="1.1" viewBox="0 0 432 288" width="432pt" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> + <defs> + <style type="text/css"> +*{stroke-linecap:butt;stroke-linejoin:round;} + </style> + </defs> + <g id="figure_1"> + <g id="patch_1"> + <path d="M 0 288 +L 432 288 +L 432 0 +L 0 0 +z +" style="fill:none;"/> + </g> + <g id="axes_1"> + <g id="patch_2"> + <path d="M 54 252 +L 388.8 252 +L 388.8 34.56 +L 54 34.56 +z +" style="fill:none;"/> + </g> + <g id="patch_3"> + <path clip-path="url(#p9518871844)" d="M 62.126214 252 +L 68.627184 252 +L 68.627184 165.714286 +L 62.126214 165.714286 +z +" style="fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_4"> + <path clip-path="url(#p9518871844)" d="M 94.631068 252 +L 101.132039 252 +L 101.132039 154.209524 +L 94.631068 154.209524 +z +" style="fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_5"> + <path clip-path="url(#p9518871844)" d="M 127.135922 252 +L 133.636893 252 +L 133.636893 145.580952 +L 127.135922 145.580952 +z +" style="fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_6"> + <path clip-path="url(#p9518871844)" d="M 159.640777 252 +L 166.141748 252 +L 166.141748 139.828571 +L 159.640777 139.828571 +z +" style="fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_7"> + <path clip-path="url(#p9518871844)" d="M 192.145631 252 +L 198.646602 252 +L 198.646602 136.952381 +L 192.145631 136.952381 +z +" style="fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_8"> + <path clip-path="url(#p9518871844)" d="M 224.650485 252 +L 231.151456 252 +L 231.151456 131.2 +L 224.650485 131.2 +z +" style="fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_9"> + <path clip-path="url(#p9518871844)" d="M 257.15534 252 +L 263.656311 252 +L 263.656311 125.447619 +L 257.15534 125.447619 +z +" style="fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_10"> + <path clip-path="url(#p9518871844)" d="M 289.660194 252 +L 296.161165 252 +L 296.161165 125.447619 +L 289.660194 125.447619 +z +" style="fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_11"> + <path clip-path="url(#p9518871844)" d="M 322.165049 252 +L 328.666019 252 +L 328.666019 109.628571 +L 322.165049 109.628571 +z +" style="fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_12"> + <path clip-path="url(#p9518871844)" d="M 354.669903 252 +L 361.170874 252 +L 361.170874 105.314286 +L 354.669903 105.314286 +z +" style="fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_13"> + <path clip-path="url(#p9518871844)" d="M 68.627184 252 +L 75.128155 252 +L 75.128155 152.771429 +L 68.627184 152.771429 +z +" style="fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_14"> + <path clip-path="url(#p9518871844)" d="M 101.132039 252 +L 107.63301 252 +L 107.63301 132.638095 +L 101.132039 132.638095 +z +" style="fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_15"> + <path clip-path="url(#p9518871844)" d="M 133.636893 252 +L 140.137864 252 +L 140.137864 132.638095 +L 133.636893 132.638095 +z +" style="fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_16"> + <path clip-path="url(#p9518871844)" d="M 166.141748 252 +L 172.642718 252 +L 172.642718 132.638095 +L 166.141748 132.638095 +z +" style="fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_17"> + <path clip-path="url(#p9518871844)" d="M 198.646602 252 +L 205.147573 252 +L 205.147573 132.638095 +L 198.646602 132.638095 +z +" style="fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_18"> + <path clip-path="url(#p9518871844)" d="M 231.151456 252 +L 237.652427 252 +L 237.652427 132.638095 +L 231.151456 132.638095 +z +" style="fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_19"> + <path clip-path="url(#p9518871844)" d="M 263.656311 252 +L 270.157282 252 +L 270.157282 132.638095 +L 263.656311 132.638095 +z +" style="fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_20"> + <path clip-path="url(#p9518871844)" d="M 296.161165 252 +L 302.662136 252 +L 302.662136 132.638095 +L 296.161165 132.638095 +z +" style="fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_21"> + <path clip-path="url(#p9518871844)" d="M 328.666019 252 +L 335.16699 252 +L 335.16699 132.638095 +L 328.666019 132.638095 +z +" style="fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_22"> + <path clip-path="url(#p9518871844)" d="M 361.170874 252 +L 367.671845 252 +L 367.671845 132.638095 +L 361.170874 132.638095 +z +" style="fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_23"> + <path clip-path="url(#p9518871844)" d="M 75.128155 252 +L 81.629126 252 +L 81.629126 155.647619 +L 75.128155 155.647619 +z +" style="fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_24"> + <path clip-path="url(#p9518871844)" d="M 107.63301 252 +L 114.133981 252 +L 114.133981 135.514286 +L 107.63301 135.514286 +z +" style="fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_25"> + <path clip-path="url(#p9518871844)" d="M 140.137864 252 +L 146.638835 252 +L 146.638835 135.514286 +L 140.137864 135.514286 +z +" style="fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_26"> + <path clip-path="url(#p9518871844)" d="M 172.642718 252 +L 179.143689 252 +L 179.143689 135.514286 +L 172.642718 135.514286 +z +" style="fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_27"> + <path clip-path="url(#p9518871844)" d="M 205.147573 252 +L 211.648544 252 +L 211.648544 135.514286 +L 205.147573 135.514286 +z +" style="fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_28"> + <path clip-path="url(#p9518871844)" d="M 237.652427 252 +L 244.153398 252 +L 244.153398 135.514286 +L 237.652427 135.514286 +z +" style="fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_29"> + <path clip-path="url(#p9518871844)" d="M 270.157282 252 +L 276.658252 252 +L 276.658252 135.514286 +L 270.157282 135.514286 +z +" style="fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_30"> + <path clip-path="url(#p9518871844)" d="M 302.662136 252 +L 309.163107 252 +L 309.163107 135.514286 +L 302.662136 135.514286 +z +" style="fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_31"> + <path clip-path="url(#p9518871844)" d="M 335.16699 252 +L 341.667961 252 +L 341.667961 135.514286 +L 335.16699 135.514286 +z +" style="fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_32"> + <path clip-path="url(#p9518871844)" d="M 367.671845 252 +L 374.172816 252 +L 374.172816 135.514286 +L 367.671845 135.514286 +z +" style="fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_33"> + <path clip-path="url(#p9518871844)" d="M 81.629126 252 +L 88.130097 252 +L 88.130097 157.085714 +L 81.629126 157.085714 +z +" style="fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_34"> + <path clip-path="url(#p9518871844)" d="M 114.133981 252 +L 120.634951 252 +L 120.634951 138.390476 +L 114.133981 138.390476 +z +" style="fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_35"> + <path clip-path="url(#p9518871844)" d="M 146.638835 252 +L 153.139806 252 +L 153.139806 108.190476 +L 146.638835 108.190476 +z +" style="fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_36"> + <path clip-path="url(#p9518871844)" d="M 179.143689 252 +L 185.64466 252 +L 185.64466 102.438095 +L 179.143689 102.438095 +z +" style="fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_37"> + <path clip-path="url(#p9518871844)" d="M 211.648544 252 +L 218.149515 252 +L 218.149515 98.12381 +L 211.648544 98.12381 +z +" style="fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_38"> + <path clip-path="url(#p9518871844)" d="M 244.153398 252 +L 250.654369 252 +L 250.654369 83.742857 +L 244.153398 83.742857 +z +" style="fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_39"> + <path clip-path="url(#p9518871844)" d="M 276.658252 252 +L 283.159223 252 +L 283.159223 67.92381 +L 276.658252 67.92381 +z +" style="fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_40"> + <path clip-path="url(#p9518871844)" d="M 309.163107 252 +L 315.664078 252 +L 315.664078 57.857143 +L 309.163107 57.857143 +z +" style="fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_41"> + <path clip-path="url(#p9518871844)" d="M 341.667961 252 +L 348.168932 252 +L 348.168932 53.542857 +L 341.667961 53.542857 +z +" style="fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="patch_42"> + <path clip-path="url(#p9518871844)" d="M 374.172816 252 +L 380.673786 252 +L 380.673786 44.914286 +L 374.172816 44.914286 +z +" style="fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="matplotlib.axis_1"> + <g id="xtick_1"> + <g id="line2d_1"> + <defs> + <path d="M 0 0 +L 0 3.5 +" id="m769b2cfcfb" style="stroke:#000000;stroke-width:0.8;"/> + </defs> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="75.128155" xlink:href="#m769b2cfcfb" y="252"/> + </g> + </g> + <g id="text_1"> + <!-- 1 --> + <defs> + <path d="M 12.40625 8.296875 +L 28.515625 8.296875 +L 28.515625 63.921875 +L 10.984375 60.40625 +L 10.984375 69.390625 +L 28.421875 72.90625 +L 38.28125 72.90625 +L 38.28125 8.296875 +L 54.390625 8.296875 +L 54.390625 0 +L 12.40625 0 +z +" id="DejaVuSans-49"/> + </defs> + <g transform="translate(77.88753 265.3625)rotate(-90)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-49"/> + </g> + </g> + </g> + <g id="xtick_2"> + <g id="line2d_2"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="107.63301" xlink:href="#m769b2cfcfb" y="252"/> + </g> + </g> + <g id="text_2"> + <!-- 2 --> + <defs> + <path d="M 19.1875 8.296875 +L 53.609375 8.296875 +L 53.609375 0 +L 7.328125 0 +L 7.328125 8.296875 +Q 12.9375 14.109375 22.625 23.890625 +Q 32.328125 33.6875 34.8125 36.53125 +Q 39.546875 41.84375 41.421875 45.53125 +Q 43.3125 49.21875 43.3125 52.78125 +Q 43.3125 58.59375 39.234375 62.25 +Q 35.15625 65.921875 28.609375 65.921875 +Q 23.96875 65.921875 18.8125 64.3125 +Q 13.671875 62.703125 7.8125 59.421875 +L 7.8125 69.390625 +Q 13.765625 71.78125 18.9375 73 +Q 24.125 74.21875 28.421875 74.21875 +Q 39.75 74.21875 46.484375 68.546875 +Q 53.21875 62.890625 53.21875 53.421875 +Q 53.21875 48.921875 51.53125 44.890625 +Q 49.859375 40.875 45.40625 35.40625 +Q 44.1875 33.984375 37.640625 27.21875 +Q 31.109375 20.453125 19.1875 8.296875 +z +" id="DejaVuSans-50"/> + </defs> + <g transform="translate(110.392385 265.3625)rotate(-90)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-50"/> + </g> + </g> + </g> + <g id="xtick_3"> + <g id="line2d_3"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="140.137864" xlink:href="#m769b2cfcfb" y="252"/> + </g> + </g> + <g id="text_3"> + <!-- 3 --> + <defs> + <path d="M 40.578125 39.3125 +Q 47.65625 37.796875 51.625 33 +Q 55.609375 28.21875 55.609375 21.1875 +Q 55.609375 10.40625 48.1875 4.484375 +Q 40.765625 -1.421875 27.09375 -1.421875 +Q 22.515625 -1.421875 17.65625 -0.515625 +Q 12.796875 0.390625 7.625 2.203125 +L 7.625 11.71875 +Q 11.71875 9.328125 16.59375 8.109375 +Q 21.484375 6.890625 26.8125 6.890625 +Q 36.078125 6.890625 40.9375 10.546875 +Q 45.796875 14.203125 45.796875 21.1875 +Q 45.796875 27.640625 41.28125 31.265625 +Q 36.765625 34.90625 28.71875 34.90625 +L 20.21875 34.90625 +L 20.21875 43.015625 +L 29.109375 43.015625 +Q 36.375 43.015625 40.234375 45.921875 +Q 44.09375 48.828125 44.09375 54.296875 +Q 44.09375 59.90625 40.109375 62.90625 +Q 36.140625 65.921875 28.71875 65.921875 +Q 24.65625 65.921875 20.015625 65.03125 +Q 15.375 64.15625 9.8125 62.3125 +L 9.8125 71.09375 +Q 15.4375 72.65625 20.34375 73.4375 +Q 25.25 74.21875 29.59375 74.21875 +Q 40.828125 74.21875 47.359375 69.109375 +Q 53.90625 64.015625 53.90625 55.328125 +Q 53.90625 49.265625 50.4375 45.09375 +Q 46.96875 40.921875 40.578125 39.3125 +z +" id="DejaVuSans-51"/> + </defs> + <g transform="translate(142.897239 265.3625)rotate(-90)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-51"/> + </g> + </g> + </g> + <g id="xtick_4"> + <g id="line2d_4"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="172.642718" xlink:href="#m769b2cfcfb" y="252"/> + </g> + </g> + <g id="text_4"> + <!-- 4 --> + <defs> + <path d="M 37.796875 64.3125 +L 12.890625 25.390625 +L 37.796875 25.390625 +z +M 35.203125 72.90625 +L 47.609375 72.90625 +L 47.609375 25.390625 +L 58.015625 25.390625 +L 58.015625 17.1875 +L 47.609375 17.1875 +L 47.609375 0 +L 37.796875 0 +L 37.796875 17.1875 +L 4.890625 17.1875 +L 4.890625 26.703125 +z +" id="DejaVuSans-52"/> + </defs> + <g transform="translate(175.402093 265.3625)rotate(-90)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-52"/> + </g> + </g> + </g> + <g id="xtick_5"> + <g id="line2d_5"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="205.147573" xlink:href="#m769b2cfcfb" y="252"/> + </g> + </g> + <g id="text_5"> + <!-- 5 --> + <defs> + <path d="M 10.796875 72.90625 +L 49.515625 72.90625 +L 49.515625 64.59375 +L 19.828125 64.59375 +L 19.828125 46.734375 +Q 21.96875 47.46875 24.109375 47.828125 +Q 26.265625 48.1875 28.421875 48.1875 +Q 40.625 48.1875 47.75 41.5 +Q 54.890625 34.8125 54.890625 23.390625 +Q 54.890625 11.625 47.5625 5.09375 +Q 40.234375 -1.421875 26.90625 -1.421875 +Q 22.3125 -1.421875 17.546875 -0.640625 +Q 12.796875 0.140625 7.71875 1.703125 +L 7.71875 11.625 +Q 12.109375 9.234375 16.796875 8.0625 +Q 21.484375 6.890625 26.703125 6.890625 +Q 35.15625 6.890625 40.078125 11.328125 +Q 45.015625 15.765625 45.015625 23.390625 +Q 45.015625 31 40.078125 35.4375 +Q 35.15625 39.890625 26.703125 39.890625 +Q 22.75 39.890625 18.8125 39.015625 +Q 14.890625 38.140625 10.796875 36.28125 +z +" id="DejaVuSans-53"/> + </defs> + <g transform="translate(207.906948 265.3625)rotate(-90)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-53"/> + </g> + </g> + </g> + <g id="xtick_6"> + <g id="line2d_6"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="237.652427" xlink:href="#m769b2cfcfb" y="252"/> + </g> + </g> + <g id="text_6"> + <!-- 6 --> + <defs> + <path d="M 33.015625 40.375 +Q 26.375 40.375 22.484375 35.828125 +Q 18.609375 31.296875 18.609375 23.390625 +Q 18.609375 15.53125 22.484375 10.953125 +Q 26.375 6.390625 33.015625 6.390625 +Q 39.65625 6.390625 43.53125 10.953125 +Q 47.40625 15.53125 47.40625 23.390625 +Q 47.40625 31.296875 43.53125 35.828125 +Q 39.65625 40.375 33.015625 40.375 +z +M 52.59375 71.296875 +L 52.59375 62.3125 +Q 48.875 64.0625 45.09375 64.984375 +Q 41.3125 65.921875 37.59375 65.921875 +Q 27.828125 65.921875 22.671875 59.328125 +Q 17.53125 52.734375 16.796875 39.40625 +Q 19.671875 43.65625 24.015625 45.921875 +Q 28.375 48.1875 33.59375 48.1875 +Q 44.578125 48.1875 50.953125 41.515625 +Q 57.328125 34.859375 57.328125 23.390625 +Q 57.328125 12.15625 50.6875 5.359375 +Q 44.046875 -1.421875 33.015625 -1.421875 +Q 20.359375 -1.421875 13.671875 8.265625 +Q 6.984375 17.96875 6.984375 36.375 +Q 6.984375 53.65625 15.1875 63.9375 +Q 23.390625 74.21875 37.203125 74.21875 +Q 40.921875 74.21875 44.703125 73.484375 +Q 48.484375 72.75 52.59375 71.296875 +z +" id="DejaVuSans-54"/> + </defs> + <g transform="translate(240.411802 265.3625)rotate(-90)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-54"/> + </g> + </g> + </g> + <g id="xtick_7"> + <g id="line2d_7"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="270.157282" xlink:href="#m769b2cfcfb" y="252"/> + </g> + </g> + <g id="text_7"> + <!-- 7 --> + <defs> + <path d="M 8.203125 72.90625 +L 55.078125 72.90625 +L 55.078125 68.703125 +L 28.609375 0 +L 18.3125 0 +L 43.21875 64.59375 +L 8.203125 64.59375 +z +" id="DejaVuSans-55"/> + </defs> + <g transform="translate(272.916657 265.3625)rotate(-90)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-55"/> + </g> + </g> + </g> + <g id="xtick_8"> + <g id="line2d_8"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="302.662136" xlink:href="#m769b2cfcfb" y="252"/> + </g> + </g> + <g id="text_8"> + <!-- 8 --> + <defs> + <path d="M 31.78125 34.625 +Q 24.75 34.625 20.71875 30.859375 +Q 16.703125 27.09375 16.703125 20.515625 +Q 16.703125 13.921875 20.71875 10.15625 +Q 24.75 6.390625 31.78125 6.390625 +Q 38.8125 6.390625 42.859375 10.171875 +Q 46.921875 13.96875 46.921875 20.515625 +Q 46.921875 27.09375 42.890625 30.859375 +Q 38.875 34.625 31.78125 34.625 +z +M 21.921875 38.8125 +Q 15.578125 40.375 12.03125 44.71875 +Q 8.5 49.078125 8.5 55.328125 +Q 8.5 64.0625 14.71875 69.140625 +Q 20.953125 74.21875 31.78125 74.21875 +Q 42.671875 74.21875 48.875 69.140625 +Q 55.078125 64.0625 55.078125 55.328125 +Q 55.078125 49.078125 51.53125 44.71875 +Q 48 40.375 41.703125 38.8125 +Q 48.828125 37.15625 52.796875 32.3125 +Q 56.78125 27.484375 56.78125 20.515625 +Q 56.78125 9.90625 50.3125 4.234375 +Q 43.84375 -1.421875 31.78125 -1.421875 +Q 19.734375 -1.421875 13.25 4.234375 +Q 6.78125 9.90625 6.78125 20.515625 +Q 6.78125 27.484375 10.78125 32.3125 +Q 14.796875 37.15625 21.921875 38.8125 +z +M 18.3125 54.390625 +Q 18.3125 48.734375 21.84375 45.5625 +Q 25.390625 42.390625 31.78125 42.390625 +Q 38.140625 42.390625 41.71875 45.5625 +Q 45.3125 48.734375 45.3125 54.390625 +Q 45.3125 60.0625 41.71875 63.234375 +Q 38.140625 66.40625 31.78125 66.40625 +Q 25.390625 66.40625 21.84375 63.234375 +Q 18.3125 60.0625 18.3125 54.390625 +z +" id="DejaVuSans-56"/> + </defs> + <g transform="translate(305.421511 265.3625)rotate(-90)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-56"/> + </g> + </g> + </g> + <g id="xtick_9"> + <g id="line2d_9"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="335.16699" xlink:href="#m769b2cfcfb" y="252"/> + </g> + </g> + <g id="text_9"> + <!-- 9 --> + <defs> + <path d="M 10.984375 1.515625 +L 10.984375 10.5 +Q 14.703125 8.734375 18.5 7.8125 +Q 22.3125 6.890625 25.984375 6.890625 +Q 35.75 6.890625 40.890625 13.453125 +Q 46.046875 20.015625 46.78125 33.40625 +Q 43.953125 29.203125 39.59375 26.953125 +Q 35.25 24.703125 29.984375 24.703125 +Q 19.046875 24.703125 12.671875 31.3125 +Q 6.296875 37.9375 6.296875 49.421875 +Q 6.296875 60.640625 12.9375 67.421875 +Q 19.578125 74.21875 30.609375 74.21875 +Q 43.265625 74.21875 49.921875 64.515625 +Q 56.59375 54.828125 56.59375 36.375 +Q 56.59375 19.140625 48.40625 8.859375 +Q 40.234375 -1.421875 26.421875 -1.421875 +Q 22.703125 -1.421875 18.890625 -0.6875 +Q 15.09375 0.046875 10.984375 1.515625 +z +M 30.609375 32.421875 +Q 37.25 32.421875 41.125 36.953125 +Q 45.015625 41.5 45.015625 49.421875 +Q 45.015625 57.28125 41.125 61.84375 +Q 37.25 66.40625 30.609375 66.40625 +Q 23.96875 66.40625 20.09375 61.84375 +Q 16.21875 57.28125 16.21875 49.421875 +Q 16.21875 41.5 20.09375 36.953125 +Q 23.96875 32.421875 30.609375 32.421875 +z +" id="DejaVuSans-57"/> + </defs> + <g transform="translate(337.926365 265.3625)rotate(-90)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-57"/> + </g> + </g> + </g> + <g id="xtick_10"> + <g id="line2d_10"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="367.671845" xlink:href="#m769b2cfcfb" y="252"/> + </g> + </g> + <g id="text_10"> + <!-- 10 --> + <defs> + <path d="M 31.78125 66.40625 +Q 24.171875 66.40625 20.328125 58.90625 +Q 16.5 51.421875 16.5 36.375 +Q 16.5 21.390625 20.328125 13.890625 +Q 24.171875 6.390625 31.78125 6.390625 +Q 39.453125 6.390625 43.28125 13.890625 +Q 47.125 21.390625 47.125 36.375 +Q 47.125 51.421875 43.28125 58.90625 +Q 39.453125 66.40625 31.78125 66.40625 +z +M 31.78125 74.21875 +Q 44.046875 74.21875 50.515625 64.515625 +Q 56.984375 54.828125 56.984375 36.375 +Q 56.984375 17.96875 50.515625 8.265625 +Q 44.046875 -1.421875 31.78125 -1.421875 +Q 19.53125 -1.421875 13.0625 8.265625 +Q 6.59375 17.96875 6.59375 36.375 +Q 6.59375 54.828125 13.0625 64.515625 +Q 19.53125 74.21875 31.78125 74.21875 +z +" id="DejaVuSans-48"/> + </defs> + <g transform="translate(370.43122 271.725)rotate(-90)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-49"/> + <use x="63.623047" xlink:href="#DejaVuSans-48"/> + </g> + </g> + </g> + <g id="text_11"> + <!-- Checkin index --> + <defs> + <path d="M 64.40625 67.28125 +L 64.40625 56.890625 +Q 59.421875 61.53125 53.78125 63.8125 +Q 48.140625 66.109375 41.796875 66.109375 +Q 29.296875 66.109375 22.65625 58.46875 +Q 16.015625 50.828125 16.015625 36.375 +Q 16.015625 21.96875 22.65625 14.328125 +Q 29.296875 6.6875 41.796875 6.6875 +Q 48.140625 6.6875 53.78125 8.984375 +Q 59.421875 11.28125 64.40625 15.921875 +L 64.40625 5.609375 +Q 59.234375 2.09375 53.4375 0.328125 +Q 47.65625 -1.421875 41.21875 -1.421875 +Q 24.65625 -1.421875 15.125 8.703125 +Q 5.609375 18.84375 5.609375 36.375 +Q 5.609375 53.953125 15.125 64.078125 +Q 24.65625 74.21875 41.21875 74.21875 +Q 47.75 74.21875 53.53125 72.484375 +Q 59.328125 70.75 64.40625 67.28125 +z +" id="DejaVuSans-67"/> + <path d="M 54.890625 33.015625 +L 54.890625 0 +L 45.90625 0 +L 45.90625 32.71875 +Q 45.90625 40.484375 42.875 44.328125 +Q 39.84375 48.1875 33.796875 48.1875 +Q 26.515625 48.1875 22.3125 43.546875 +Q 18.109375 38.921875 18.109375 30.90625 +L 18.109375 0 +L 9.078125 0 +L 9.078125 75.984375 +L 18.109375 75.984375 +L 18.109375 46.1875 +Q 21.34375 51.125 25.703125 53.5625 +Q 30.078125 56 35.796875 56 +Q 45.21875 56 50.046875 50.171875 +Q 54.890625 44.34375 54.890625 33.015625 +z +" id="DejaVuSans-104"/> + <path d="M 56.203125 29.59375 +L 56.203125 25.203125 +L 14.890625 25.203125 +Q 15.484375 15.921875 20.484375 11.0625 +Q 25.484375 6.203125 34.421875 6.203125 +Q 39.59375 6.203125 44.453125 7.46875 +Q 49.3125 8.734375 54.109375 11.28125 +L 54.109375 2.78125 +Q 49.265625 0.734375 44.1875 -0.34375 +Q 39.109375 -1.421875 33.890625 -1.421875 +Q 20.796875 -1.421875 13.15625 6.1875 +Q 5.515625 13.8125 5.515625 26.8125 +Q 5.515625 40.234375 12.765625 48.109375 +Q 20.015625 56 32.328125 56 +Q 43.359375 56 49.78125 48.890625 +Q 56.203125 41.796875 56.203125 29.59375 +z +M 47.21875 32.234375 +Q 47.125 39.59375 43.09375 43.984375 +Q 39.0625 48.390625 32.421875 48.390625 +Q 24.90625 48.390625 20.390625 44.140625 +Q 15.875 39.890625 15.1875 32.171875 +z +" id="DejaVuSans-101"/> + <path d="M 48.78125 52.59375 +L 48.78125 44.1875 +Q 44.96875 46.296875 41.140625 47.34375 +Q 37.3125 48.390625 33.40625 48.390625 +Q 24.65625 48.390625 19.8125 42.84375 +Q 14.984375 37.3125 14.984375 27.296875 +Q 14.984375 17.28125 19.8125 11.734375 +Q 24.65625 6.203125 33.40625 6.203125 +Q 37.3125 6.203125 41.140625 7.25 +Q 44.96875 8.296875 48.78125 10.40625 +L 48.78125 2.09375 +Q 45.015625 0.34375 40.984375 -0.53125 +Q 36.96875 -1.421875 32.421875 -1.421875 +Q 20.0625 -1.421875 12.78125 6.34375 +Q 5.515625 14.109375 5.515625 27.296875 +Q 5.515625 40.671875 12.859375 48.328125 +Q 20.21875 56 33.015625 56 +Q 37.15625 56 41.109375 55.140625 +Q 45.0625 54.296875 48.78125 52.59375 +z +" id="DejaVuSans-99"/> + <path d="M 9.078125 75.984375 +L 18.109375 75.984375 +L 18.109375 31.109375 +L 44.921875 54.6875 +L 56.390625 54.6875 +L 27.390625 29.109375 +L 57.625 0 +L 45.90625 0 +L 18.109375 26.703125 +L 18.109375 0 +L 9.078125 0 +z +" id="DejaVuSans-107"/> + <path d="M 9.421875 54.6875 +L 18.40625 54.6875 +L 18.40625 0 +L 9.421875 0 +z +M 9.421875 75.984375 +L 18.40625 75.984375 +L 18.40625 64.59375 +L 9.421875 64.59375 +z +" id="DejaVuSans-105"/> + <path d="M 54.890625 33.015625 +L 54.890625 0 +L 45.90625 0 +L 45.90625 32.71875 +Q 45.90625 40.484375 42.875 44.328125 +Q 39.84375 48.1875 33.796875 48.1875 +Q 26.515625 48.1875 22.3125 43.546875 +Q 18.109375 38.921875 18.109375 30.90625 +L 18.109375 0 +L 9.078125 0 +L 9.078125 54.6875 +L 18.109375 54.6875 +L 18.109375 46.1875 +Q 21.34375 51.125 25.703125 53.5625 +Q 30.078125 56 35.796875 56 +Q 45.21875 56 50.046875 50.171875 +Q 54.890625 44.34375 54.890625 33.015625 +z +" id="DejaVuSans-110"/> + <path id="DejaVuSans-32"/> + <path d="M 45.40625 46.390625 +L 45.40625 75.984375 +L 54.390625 75.984375 +L 54.390625 0 +L 45.40625 0 +L 45.40625 8.203125 +Q 42.578125 3.328125 38.25 0.953125 +Q 33.9375 -1.421875 27.875 -1.421875 +Q 17.96875 -1.421875 11.734375 6.484375 +Q 5.515625 14.40625 5.515625 27.296875 +Q 5.515625 40.1875 11.734375 48.09375 +Q 17.96875 56 27.875 56 +Q 33.9375 56 38.25 53.625 +Q 42.578125 51.265625 45.40625 46.390625 +z +M 14.796875 27.296875 +Q 14.796875 17.390625 18.875 11.75 +Q 22.953125 6.109375 30.078125 6.109375 +Q 37.203125 6.109375 41.296875 11.75 +Q 45.40625 17.390625 45.40625 27.296875 +Q 45.40625 37.203125 41.296875 42.84375 +Q 37.203125 48.484375 30.078125 48.484375 +Q 22.953125 48.484375 18.875 42.84375 +Q 14.796875 37.203125 14.796875 27.296875 +z +" id="DejaVuSans-100"/> + <path d="M 54.890625 54.6875 +L 35.109375 28.078125 +L 55.90625 0 +L 45.3125 0 +L 29.390625 21.484375 +L 13.484375 0 +L 2.875 0 +L 24.125 28.609375 +L 4.6875 54.6875 +L 15.28125 54.6875 +L 29.78125 35.203125 +L 44.28125 54.6875 +z +" id="DejaVuSans-120"/> + </defs> + <g transform="translate(186.104688 283.323438)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-67"/> + <use x="69.824219" xlink:href="#DejaVuSans-104"/> + <use x="133.203125" xlink:href="#DejaVuSans-101"/> + <use x="194.726562" xlink:href="#DejaVuSans-99"/> + <use x="249.707031" xlink:href="#DejaVuSans-107"/> + <use x="307.617188" xlink:href="#DejaVuSans-105"/> + <use x="335.400391" xlink:href="#DejaVuSans-110"/> + <use x="398.779297" xlink:href="#DejaVuSans-32"/> + <use x="430.566406" xlink:href="#DejaVuSans-105"/> + <use x="458.349609" xlink:href="#DejaVuSans-110"/> + <use x="521.728516" xlink:href="#DejaVuSans-100"/> + <use x="585.205078" xlink:href="#DejaVuSans-101"/> + <use x="646.712891" xlink:href="#DejaVuSans-120"/> + </g> + </g> + </g> + <g id="matplotlib.axis_2"> + <g id="ytick_1"> + <g id="line2d_11"> + <defs> + <path d="M 0 0 +L -3.5 0 +" id="m80d71b6281" style="stroke:#000000;stroke-width:0.8;"/> + </defs> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#m80d71b6281" y="252"/> + </g> + </g> + <g id="text_12"> + <!-- 0.0 --> + <defs> + <path d="M 10.6875 12.40625 +L 21 12.40625 +L 21 0 +L 10.6875 0 +z +" id="DejaVuSans-46"/> + </defs> + <g transform="translate(31.096875 255.799219)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-48"/> + <use x="63.623047" xlink:href="#DejaVuSans-46"/> + <use x="95.410156" xlink:href="#DejaVuSans-48"/> + </g> + </g> + </g> + <g id="ytick_2"> + <g id="line2d_12"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#m80d71b6281" y="215.184762"/> + </g> + </g> + <g id="text_13"> + <!-- 0.2 --> + <g transform="translate(31.096875 218.983981)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-48"/> + <use x="63.623047" xlink:href="#DejaVuSans-46"/> + <use x="95.410156" xlink:href="#DejaVuSans-50"/> + </g> + </g> + </g> + <g id="ytick_3"> + <g id="line2d_13"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#m80d71b6281" y="178.369524"/> + </g> + </g> + <g id="text_14"> + <!-- 0.4 --> + <g transform="translate(31.096875 182.168743)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-48"/> + <use x="63.623047" xlink:href="#DejaVuSans-46"/> + <use x="95.410156" xlink:href="#DejaVuSans-52"/> + </g> + </g> + </g> + <g id="ytick_4"> + <g id="line2d_14"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#m80d71b6281" y="141.554286"/> + </g> + </g> + <g id="text_15"> + <!-- 0.6 --> + <g transform="translate(31.096875 145.353504)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-48"/> + <use x="63.623047" xlink:href="#DejaVuSans-46"/> + <use x="95.410156" xlink:href="#DejaVuSans-54"/> + </g> + </g> + </g> + <g id="ytick_5"> + <g id="line2d_15"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#m80d71b6281" y="104.739048"/> + </g> + </g> + <g id="text_16"> + <!-- 0.8 --> + <g transform="translate(31.096875 108.538266)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-48"/> + <use x="63.623047" xlink:href="#DejaVuSans-46"/> + <use x="95.410156" xlink:href="#DejaVuSans-56"/> + </g> + </g> + </g> + <g id="ytick_6"> + <g id="line2d_16"> + <g> + <use style="stroke:#000000;stroke-width:0.8;" x="54" xlink:href="#m80d71b6281" y="67.92381"/> + </g> + </g> + <g id="text_17"> + <!-- 1.0 --> + <g transform="translate(31.096875 71.723028)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-49"/> + <use x="63.623047" xlink:href="#DejaVuSans-46"/> + <use x="95.410156" xlink:href="#DejaVuSans-48"/> + </g> + </g> + </g> + <g id="text_18"> + <!-- Repo size (MiB) --> + <defs> + <path d="M 44.390625 34.1875 +Q 47.5625 33.109375 50.5625 29.59375 +Q 53.5625 26.078125 56.59375 19.921875 +L 66.609375 0 +L 56 0 +L 46.6875 18.703125 +Q 43.0625 26.03125 39.671875 28.421875 +Q 36.28125 30.8125 30.421875 30.8125 +L 19.671875 30.8125 +L 19.671875 0 +L 9.8125 0 +L 9.8125 72.90625 +L 32.078125 72.90625 +Q 44.578125 72.90625 50.734375 67.671875 +Q 56.890625 62.453125 56.890625 51.90625 +Q 56.890625 45.015625 53.6875 40.46875 +Q 50.484375 35.9375 44.390625 34.1875 +z +M 19.671875 64.796875 +L 19.671875 38.921875 +L 32.078125 38.921875 +Q 39.203125 38.921875 42.84375 42.21875 +Q 46.484375 45.515625 46.484375 51.90625 +Q 46.484375 58.296875 42.84375 61.546875 +Q 39.203125 64.796875 32.078125 64.796875 +z +" id="DejaVuSans-82"/> + <path d="M 18.109375 8.203125 +L 18.109375 -20.796875 +L 9.078125 -20.796875 +L 9.078125 54.6875 +L 18.109375 54.6875 +L 18.109375 46.390625 +Q 20.953125 51.265625 25.265625 53.625 +Q 29.59375 56 35.59375 56 +Q 45.5625 56 51.78125 48.09375 +Q 58.015625 40.1875 58.015625 27.296875 +Q 58.015625 14.40625 51.78125 6.484375 +Q 45.5625 -1.421875 35.59375 -1.421875 +Q 29.59375 -1.421875 25.265625 0.953125 +Q 20.953125 3.328125 18.109375 8.203125 +z +M 48.6875 27.296875 +Q 48.6875 37.203125 44.609375 42.84375 +Q 40.53125 48.484375 33.40625 48.484375 +Q 26.265625 48.484375 22.1875 42.84375 +Q 18.109375 37.203125 18.109375 27.296875 +Q 18.109375 17.390625 22.1875 11.75 +Q 26.265625 6.109375 33.40625 6.109375 +Q 40.53125 6.109375 44.609375 11.75 +Q 48.6875 17.390625 48.6875 27.296875 +z +" id="DejaVuSans-112"/> + <path d="M 30.609375 48.390625 +Q 23.390625 48.390625 19.1875 42.75 +Q 14.984375 37.109375 14.984375 27.296875 +Q 14.984375 17.484375 19.15625 11.84375 +Q 23.34375 6.203125 30.609375 6.203125 +Q 37.796875 6.203125 41.984375 11.859375 +Q 46.1875 17.53125 46.1875 27.296875 +Q 46.1875 37.015625 41.984375 42.703125 +Q 37.796875 48.390625 30.609375 48.390625 +z +M 30.609375 56 +Q 42.328125 56 49.015625 48.375 +Q 55.71875 40.765625 55.71875 27.296875 +Q 55.71875 13.875 49.015625 6.21875 +Q 42.328125 -1.421875 30.609375 -1.421875 +Q 18.84375 -1.421875 12.171875 6.21875 +Q 5.515625 13.875 5.515625 27.296875 +Q 5.515625 40.765625 12.171875 48.375 +Q 18.84375 56 30.609375 56 +z +" id="DejaVuSans-111"/> + <path d="M 44.28125 53.078125 +L 44.28125 44.578125 +Q 40.484375 46.53125 36.375 47.5 +Q 32.28125 48.484375 27.875 48.484375 +Q 21.1875 48.484375 17.84375 46.4375 +Q 14.5 44.390625 14.5 40.28125 +Q 14.5 37.15625 16.890625 35.375 +Q 19.28125 33.59375 26.515625 31.984375 +L 29.59375 31.296875 +Q 39.15625 29.25 43.1875 25.515625 +Q 47.21875 21.78125 47.21875 15.09375 +Q 47.21875 7.46875 41.1875 3.015625 +Q 35.15625 -1.421875 24.609375 -1.421875 +Q 20.21875 -1.421875 15.453125 -0.5625 +Q 10.6875 0.296875 5.421875 2 +L 5.421875 11.28125 +Q 10.40625 8.6875 15.234375 7.390625 +Q 20.0625 6.109375 24.8125 6.109375 +Q 31.15625 6.109375 34.5625 8.28125 +Q 37.984375 10.453125 37.984375 14.40625 +Q 37.984375 18.0625 35.515625 20.015625 +Q 33.0625 21.96875 24.703125 23.78125 +L 21.578125 24.515625 +Q 13.234375 26.265625 9.515625 29.90625 +Q 5.8125 33.546875 5.8125 39.890625 +Q 5.8125 47.609375 11.28125 51.796875 +Q 16.75 56 26.8125 56 +Q 31.78125 56 36.171875 55.265625 +Q 40.578125 54.546875 44.28125 53.078125 +z +" id="DejaVuSans-115"/> + <path d="M 5.515625 54.6875 +L 48.1875 54.6875 +L 48.1875 46.484375 +L 14.40625 7.171875 +L 48.1875 7.171875 +L 48.1875 0 +L 4.296875 0 +L 4.296875 8.203125 +L 38.09375 47.515625 +L 5.515625 47.515625 +z +" id="DejaVuSans-122"/> + <path d="M 31 75.875 +Q 24.46875 64.65625 21.28125 53.65625 +Q 18.109375 42.671875 18.109375 31.390625 +Q 18.109375 20.125 21.3125 9.0625 +Q 24.515625 -2 31 -13.1875 +L 23.1875 -13.1875 +Q 15.875 -1.703125 12.234375 9.375 +Q 8.59375 20.453125 8.59375 31.390625 +Q 8.59375 42.28125 12.203125 53.3125 +Q 15.828125 64.359375 23.1875 75.875 +z +" id="DejaVuSans-40"/> + <path d="M 9.8125 72.90625 +L 24.515625 72.90625 +L 43.109375 23.296875 +L 61.8125 72.90625 +L 76.515625 72.90625 +L 76.515625 0 +L 66.890625 0 +L 66.890625 64.015625 +L 48.09375 14.015625 +L 38.1875 14.015625 +L 19.390625 64.015625 +L 19.390625 0 +L 9.8125 0 +z +" id="DejaVuSans-77"/> + <path d="M 19.671875 34.8125 +L 19.671875 8.109375 +L 35.5 8.109375 +Q 43.453125 8.109375 47.28125 11.40625 +Q 51.125 14.703125 51.125 21.484375 +Q 51.125 28.328125 47.28125 31.5625 +Q 43.453125 34.8125 35.5 34.8125 +z +M 19.671875 64.796875 +L 19.671875 42.828125 +L 34.28125 42.828125 +Q 41.5 42.828125 45.03125 45.53125 +Q 48.578125 48.25 48.578125 53.8125 +Q 48.578125 59.328125 45.03125 62.0625 +Q 41.5 64.796875 34.28125 64.796875 +z +M 9.8125 72.90625 +L 35.015625 72.90625 +Q 46.296875 72.90625 52.390625 68.21875 +Q 58.5 63.53125 58.5 54.890625 +Q 58.5 48.1875 55.375 44.234375 +Q 52.25 40.28125 46.1875 39.3125 +Q 53.46875 37.75 57.5 32.78125 +Q 61.53125 27.828125 61.53125 20.40625 +Q 61.53125 10.640625 54.890625 5.3125 +Q 48.25 0 35.984375 0 +L 9.8125 0 +z +" id="DejaVuSans-66"/> + <path d="M 8.015625 75.875 +L 15.828125 75.875 +Q 23.140625 64.359375 26.78125 53.3125 +Q 30.421875 42.28125 30.421875 31.390625 +Q 30.421875 20.453125 26.78125 9.375 +Q 23.140625 -1.703125 15.828125 -13.1875 +L 8.015625 -13.1875 +Q 14.5 -2 17.703125 9.0625 +Q 20.90625 20.125 20.90625 31.390625 +Q 20.90625 42.671875 17.703125 53.65625 +Q 14.5 64.65625 8.015625 75.875 +z +" id="DejaVuSans-41"/> + </defs> + <g transform="translate(25.017187 181.969063)rotate(-90)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-82"/> + <use x="69.419922" xlink:href="#DejaVuSans-101"/> + <use x="130.943359" xlink:href="#DejaVuSans-112"/> + <use x="194.419922" xlink:href="#DejaVuSans-111"/> + <use x="255.601562" xlink:href="#DejaVuSans-32"/> + <use x="287.388672" xlink:href="#DejaVuSans-115"/> + <use x="339.488281" xlink:href="#DejaVuSans-105"/> + <use x="367.271484" xlink:href="#DejaVuSans-122"/> + <use x="419.761719" xlink:href="#DejaVuSans-101"/> + <use x="481.285156" xlink:href="#DejaVuSans-32"/> + <use x="513.072266" xlink:href="#DejaVuSans-40"/> + <use x="552.085938" xlink:href="#DejaVuSans-77"/> + <use x="638.365234" xlink:href="#DejaVuSans-105"/> + <use x="666.148438" xlink:href="#DejaVuSans-66"/> + <use x="734.751953" xlink:href="#DejaVuSans-41"/> + </g> + </g> + </g> + <g id="patch_43"> + <path d="M 54 252 +L 54 34.56 +" style="fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;"/> + </g> + <g id="patch_44"> + <path d="M 388.8 252 +L 388.8 34.56 +" style="fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;"/> + </g> + <g id="patch_45"> + <path d="M 54 252 +L 388.8 252 +" style="fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;"/> + </g> + <g id="patch_46"> + <path d="M 54 34.56 +L 388.8 34.56 +" style="fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;"/> + </g> + <g id="legend_1"> + <g id="patch_47"> + <path d="M 61 101.2725 +L 116.046875 101.2725 +Q 118.046875 101.2725 118.046875 99.2725 +L 118.046875 41.56 +Q 118.046875 39.56 116.046875 39.56 +L 61 39.56 +Q 59 39.56 59 41.56 +L 59 99.2725 +Q 59 101.2725 61 101.2725 +z +" style="fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;"/> + </g> + <g id="patch_48"> + <path d="M 63 51.158437 +L 83 51.158437 +L 83 44.158437 +L 63 44.158437 +z +" style="fill:#3b4cc0;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="text_19"> + <!-- JPEG --> + <defs> + <path d="M 9.8125 72.90625 +L 19.671875 72.90625 +L 19.671875 5.078125 +Q 19.671875 -8.109375 14.671875 -14.0625 +Q 9.671875 -20.015625 -1.421875 -20.015625 +L -5.171875 -20.015625 +L -5.171875 -11.71875 +L -2.09375 -11.71875 +Q 4.4375 -11.71875 7.125 -8.046875 +Q 9.8125 -4.390625 9.8125 5.078125 +z +" id="DejaVuSans-74"/> + <path d="M 19.671875 64.796875 +L 19.671875 37.40625 +L 32.078125 37.40625 +Q 38.96875 37.40625 42.71875 40.96875 +Q 46.484375 44.53125 46.484375 51.125 +Q 46.484375 57.671875 42.71875 61.234375 +Q 38.96875 64.796875 32.078125 64.796875 +z +M 9.8125 72.90625 +L 32.078125 72.90625 +Q 44.34375 72.90625 50.609375 67.359375 +Q 56.890625 61.8125 56.890625 51.125 +Q 56.890625 40.328125 50.609375 34.8125 +Q 44.34375 29.296875 32.078125 29.296875 +L 19.671875 29.296875 +L 19.671875 0 +L 9.8125 0 +z +" id="DejaVuSans-80"/> + <path d="M 9.8125 72.90625 +L 55.90625 72.90625 +L 55.90625 64.59375 +L 19.671875 64.59375 +L 19.671875 43.015625 +L 54.390625 43.015625 +L 54.390625 34.71875 +L 19.671875 34.71875 +L 19.671875 8.296875 +L 56.78125 8.296875 +L 56.78125 0 +L 9.8125 0 +z +" id="DejaVuSans-69"/> + <path d="M 59.515625 10.40625 +L 59.515625 29.984375 +L 43.40625 29.984375 +L 43.40625 38.09375 +L 69.28125 38.09375 +L 69.28125 6.78125 +Q 63.578125 2.734375 56.6875 0.65625 +Q 49.8125 -1.421875 42 -1.421875 +Q 24.90625 -1.421875 15.25 8.5625 +Q 5.609375 18.5625 5.609375 36.375 +Q 5.609375 54.25 15.25 64.234375 +Q 24.90625 74.21875 42 74.21875 +Q 49.125 74.21875 55.546875 72.453125 +Q 61.96875 70.703125 67.390625 67.28125 +L 67.390625 56.78125 +Q 61.921875 61.421875 55.765625 63.765625 +Q 49.609375 66.109375 42.828125 66.109375 +Q 29.4375 66.109375 22.71875 58.640625 +Q 16.015625 51.171875 16.015625 36.375 +Q 16.015625 21.625 22.71875 14.15625 +Q 29.4375 6.6875 42.828125 6.6875 +Q 48.046875 6.6875 52.140625 7.59375 +Q 56.25 8.5 59.515625 10.40625 +z +" id="DejaVuSans-71"/> + </defs> + <g transform="translate(91 51.158437)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-74"/> + <use x="29.492188" xlink:href="#DejaVuSans-80"/> + <use x="89.794922" xlink:href="#DejaVuSans-69"/> + <use x="152.978516" xlink:href="#DejaVuSans-71"/> + </g> + </g> + <g id="patch_49"> + <path d="M 63 65.836562 +L 83 65.836562 +L 83 58.836562 +L 63 58.836562 +z +" style="fill:#aac7fd;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="text_20"> + <!-- BMP --> + <g transform="translate(91 65.836562)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-66"/> + <use x="68.603516" xlink:href="#DejaVuSans-77"/> + <use x="154.882812" xlink:href="#DejaVuSans-80"/> + </g> + </g> + <g id="patch_50"> + <path d="M 63 80.514687 +L 83 80.514687 +L 83 73.514687 +L 63 73.514687 +z +" style="fill:#f7b89c;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="text_21"> + <!-- TIFF --> + <defs> + <path d="M -0.296875 72.90625 +L 61.375 72.90625 +L 61.375 64.59375 +L 35.5 64.59375 +L 35.5 0 +L 25.59375 0 +L 25.59375 64.59375 +L -0.296875 64.59375 +z +" id="DejaVuSans-84"/> + <path d="M 9.8125 72.90625 +L 19.671875 72.90625 +L 19.671875 0 +L 9.8125 0 +z +" id="DejaVuSans-73"/> + <path d="M 9.8125 72.90625 +L 51.703125 72.90625 +L 51.703125 64.59375 +L 19.671875 64.59375 +L 19.671875 43.109375 +L 48.578125 43.109375 +L 48.578125 34.8125 +L 19.671875 34.8125 +L 19.671875 0 +L 9.8125 0 +z +" id="DejaVuSans-70"/> + </defs> + <g transform="translate(91 80.514687)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-84"/> + <use x="61.083984" xlink:href="#DejaVuSans-73"/> + <use x="90.576172" xlink:href="#DejaVuSans-70"/> + <use x="148.095703" xlink:href="#DejaVuSans-70"/> + </g> + </g> + <g id="patch_51"> + <path d="M 63 95.192813 +L 83 95.192813 +L 83 88.192813 +L 63 88.192813 +z +" style="fill:#b40426;stroke:#ffffff;stroke-linejoin:miter;stroke-width:2;"/> + </g> + <g id="text_22"> + <!-- PNG --> + <defs> + <path d="M 9.8125 72.90625 +L 23.09375 72.90625 +L 55.421875 11.921875 +L 55.421875 72.90625 +L 64.984375 72.90625 +L 64.984375 0 +L 51.703125 0 +L 19.390625 60.984375 +L 19.390625 0 +L 9.8125 0 +z +" id="DejaVuSans-78"/> + </defs> + <g transform="translate(91 95.192813)scale(0.1 -0.1)"> + <use xlink:href="#DejaVuSans-80"/> + <use x="60.302734" xlink:href="#DejaVuSans-78"/> + <use x="135.107422" xlink:href="#DejaVuSans-71"/> + </g> + </g> + </g> + </g> + </g> + <defs> + <clipPath id="p9518871844"> + <rect height="217.44" width="334.8" x="54" y="34.56"/> + </clipPath> + </defs> +</svg> diff --git a/www/index.wiki b/www/index.wiki new file mode 100644 index 0000000..b8e8c9c --- /dev/null +++ b/www/index.wiki @@ -0,0 +1,108 @@ +<title>A Coherent Software Configuration Management System</title> + +<h3>What Is It?</h3> + +<div class="nomargins" style='float:right;border:2px solid #446979;padding:0 15px 10px 0;margin:0 50px 0 10px'> +<ul> +<li> [/uv/download.html | Download] +<li> [./quickstart.wiki | Quick Start] +<li> [./build.wiki | Install] +<li> [https://fossil-scm.org/forum | Support/Forum ] +<li> [./hints.wiki | Tips & Hints] +<li> [./changes.wiki | Change Log] +<li> [../COPYRIGHT-BSD2.txt | License] +<li> [./userlinks.wiki | User Links] +<li> [./hacker-howto.wiki | Hacker How-To] +<li> [./fossil-v-git.wiki | Fossil vs. Git] +<li> [./permutedindex.html | Doc Index] +</ul> +<p style="text-align:center"><img src="fossil3.gif" alt="Fossil logo"></p> +</div> + +Fossil is a simple, high-reliability, distributed +[https://en.wikipedia.org/wiki/Software_configuration_management | SCM] +system with these advanced features: + + 1. <b>Project Management</b> — + In addition to doing [./concepts.wiki | distributed version control] + like Git and Mercurial, + Fossil also supports [./bugtheory.wiki | bug tracking], + [./wikitheory.wiki | wiki], [./forum.wiki | forum], + [./alerts.md|email alerts], [./chat.md | chat], and + [./event.wiki | technotes]. + + 2. <b>Built-in Web Interface</b> — + Fossil has a built-in, [/skins | themeable], [./serverext.wiki | extensible], + and intuitive [./webui.wiki | web interface] + with a rich variety of information pages + ([./webpage-ex.md|examples]) promoting situational awareness. + <br><br> + This entire website is just a running instance of Fossil. + The pages you see here are all [./wikitheory.wiki | wiki] or + [./embeddeddoc.wiki | embedded documentation] or (in the case of + the [/uv/download.html|download] page) + [./unvers.wiki | unversioned files]. + When you clone Fossil from one of its + [./selfhost.wiki | self-hosting repositories], + you get more than just source code — you get this entire website. + + 3. <b>All-in-one</b> — + Fossil is a single self-contained, stand-alone executable. + To install, simply download a + [/uv/download.html | precompiled binary] + for Linux, Mac, or Windows and put it on your $PATH. + [./build.wiki | Easy-to-compile source code] is also available. + + + 4. <b>Self-host Friendly</b> — Stand up a project website + in minutes using [./server/ | a variety of techniques]. + Fossil is CPU and memory efficient. Most projects can be + hosted comfortably on a $5/month VPS or a Raspberry Pi. + You can also set up an automatic + [./mirrortogithub.md | GitHub mirror]. + + 5. <b>Simple Networking</b> — + Fossil uses ordinary HTTPS (or SSH if you prefer) + for network communications, so it works fine from behind + firewalls and [./quickstart.wiki#proxy|proxies]. + The protocol is + [./stats.wiki | bandwidth efficient] to the point that Fossil can be + used comfortably over dial-up, weak 3G, or airliner Wifi. + + 6. <b>Autosync</b> — + Fossil supports [./concepts.wiki#workflow | "autosync" mode] + which helps to keep projects moving + forward by reducing the amount of needless + [./branching.wiki | forking and merging] often + associated with distributed projects. + + 7. <b>Robust & Reliable</b> — + Fossil stores content using an [./fileformat.wiki | enduring file format] + in an SQLite database so that transactions are + atomic even if interrupted by a power loss or system crash. + Automatic [./selfcheck.wiki | self-checks] verify that all aspects of + the repository are consistent prior to each commit. + + 8. <b>Free and Open-Source</b> — [../COPYRIGHT-BSD2.txt|2-clause BSD license]. + +<hr> +<h3>Latest Release: 2.28 ([/timeline?c=version-2.28|2026-03-11])</h3> + + * [/uv/download.html|Download] + * [./changes.wiki#v2_28|Change Summary] + * [/timeline?p=version-2.28&d=version-2.27&y=ci|Check-ins in version 2.28] + * [/timeline?df=version-2.28&y=ci|Check-ins derived from the 2.28 release] + * [/timeline?t=release|Timeline of all past releases] + +<hr> +<h3>Quick Start</h3> + + 1. [/uv/download.html|Download] or install using a package manager or + [./build.wiki|compile from sources]. + 2. <tt>fossil init</tt> <i>REPOSITORY-DIR/new-repository</i> + 3. <tt>fossil open</tt> <i>REPOSITORY-DIR/new-repository</i> + 4. <tt>fossil add</tt> <i>files-or-directories</i> + 5. <tt>fossil commit -m</tt> "<i>commit message</i>" + 6. <tt>fossil ui</tt> + 7. Repeat steps 4, 5, and 6, in any order, as necessary. + See the [./quickstart.wiki|Quick Start Guide] for more detail. diff --git a/www/inout.wiki b/www/inout.wiki new file mode 100644 index 0000000..74b8e97 --- /dev/null +++ b/www/inout.wiki @@ -0,0 +1,141 @@ +<title>Import And Export</title> + +Fossil has the ability to import and export repositories from and to +[http://git-scm.com/ | Git]. And since most other version control +systems will also import/export from Git, that means that you can +import/export a Fossil repository to most version control systems using +Git as an intermediary. + +<h2>Git → Fossil</h2> + +To import a Git repository into Fossil, say something like: + +<pre> +cd git-repo +git fast-export --all | fossil import --git new-repo.fossil +</pre> + +The 3rd argument to the "fossil import" +command is the name of a new Fossil repository that is created to hold the Git +content. + +The --git option is not actually required. The git-fast-export file format +is currently the only VCS interchange format that Fossil understands. But +future versions of Fossil might be enhanced to understand other VCS +interchange formats, and so for compatibility, use of the +--git option is recommended. + +<a id="fx_git"></a> +Note that in new imports, Fossil defaults to using the email component of the +Git <em>committer</em> (or <em>author</em> if <code>--use-author</code> is +passed) to attribute check-ins in the imported repository. Alternatively, the +[/help/import | <code>--attribute</code>] option can be passed to have all +commits by a given committer attributed to a desired username. This will create +and populate the new <code>fx_git</code> table in the repository database to +maintain a record of correspondent usernames and email addresses that can be +used in subsequent exports or incremental imports. + +<h3>Converting Repositories on Windows</h3> + +The above commands work best on proper POSIX systems like Linux, macOS, +and the BSDs, where everything <tt>git</tt> sends is consumed by +<tt>fossil</tt> as soon as it can manage, with both programs working +concurrently. + +Historically, PowerShell indiscriminately sent objects — as opposed to raw +bytes — through its pipes, and buffered standard input for external processes. +This made it choke on the conversion when the in-flight repository size +exceeded available memory. Starting with version 7.4 (2023-11-16), PowerShell +supports byte stream piping between native commands and file redirection. + +If you are stuck with an older version, one workaround is to fall back to +<tt>cmd.exe</tt> — which doesn't seem to be affected by this problem. +Nevertheless, we instead recommend using +Microsoft's own [https://learn.microsoft.com/en-us/windows/wsl/ | Windows +Subsystem for Linux] or either of the two popular "Git for Windows" +distributions based on MSYS2. They handle pipes the POSIX way, avoiding +any dependency on the amount of data involved. + +<h2>Fossil → Git</h2> + +To convert a Fossil repository into a Git repository, run commands like +this: + +<pre> +git init new-repo +cd new-repo +fossil export --git ../repo.fossil | git fast-import +</pre> + +In other words, create a new Git repository, then pipe the output from the +"fossil export --git" command into the "git fast-import" command. + +Note that the "fossil export --git" command only exports the versioned files. +Tickets and wiki and events are not exported, since Git does not understand +those concepts. + +As with the "import" command, the --git option is not required +since the git-fast-export file format is currently the only VCS interchange +format that Fossil will generate. However, +future versions of Fossil might add the ability to generate other +VCS interchange formats, and so for compatibility, the use of the --git +option is recommended. + +<h2>Mirror A Fossil Repository In Git</h2> + +Fossil version 2.9 and later supports a simple mechanism for +doing a Git or +[./mirrortogithub.md|GitHub mirror of a Fossil repository]. +See that separate document for details. Fossil is self-hosting, +but a [https://github.com/drhsqlite/fossil-mirror|GitHub mirror of Fossil] +is available as a proof-of-concept. + +<h2>Bidirectional Synchronization</h2> +Fossil also has the ability to synchronize with a Git repository via repeated +imports and/or exports. To do this, it uses marks files to store a record of +artifacts which are known by both Git and Fossil to exist at a given point in +time. + +To illustrate, consider the example of a remote Fossil repository that a +user wants to import into a local Git repository. First, the user would clone +the remote repository and import it into a new Git repository: + +<pre> +fossil clone /path/to/remote/repo.fossil repo.fossil +mkdir repo +cd repo +fossil open ../repo.fossil +mkdir ../repo.git +cd ../repo.git +git init . +fossil export --git --export-marks ../repo/fossil.marks \ + ../repo.fossil | git fast-import \ + --export-marks=../repo/git.marks +</pre> + +Once the import has completed, the user would need to <tt>git checkout +trunk</tt>. At any point after this, new changes can be imported from the +remote Fossil repository: + +<pre> +cd ../repo +fossil pull +cd ../repo.git +fossil export --git --import-marks ../repo/fossil.marks \ + --export-marks ../repo/fossil.marks \ + ../repo.fossil | git fast-import \ + --import-marks=../repo/git.marks \ + --export-marks=../repo/git.marks +</pre> + +Changes in the Git repository can be exported to the Fossil repository and then +pushed to the remote: + +<pre> +git fast-export --import-marks=../repo/git.marks \ + --export-marks=../repo/git.marks --all | fossil import --git \ + --incremental --import-marks ../repo/fossil.marks \ + --export-marks ../repo/fossil.marks ../repo.fossil +cd ../repo +fossil push +</pre> diff --git a/www/interwiki.md b/www/interwiki.md new file mode 100644 index 0000000..fd9a33f --- /dev/null +++ b/www/interwiki.md @@ -0,0 +1,117 @@ +# Interwiki Links + +Interwiki links are a short-hand notation for links that target +external wikis or websites. For example, the following two +hyperlinks mean the same thing (assuming an appropriate [intermap](#intermap) +configuration): + + * [](wikipedia:MediaWiki#Interwiki_links) + * [](https://en.wikipedia.org/wiki/MediaWiki#Interwiki_links) + +Another example: The Fossil Forum is hosted in a separate repository +from the Fossil source code. This page is part of the +source code repository. Interwiki links can be used to more easily +refer to the forum repository: + + * [](forum:d5508c3bf44c6393df09c) + * [](https://fossil-scm.org/forum/info/d5508c3bf44c6393df09c) + +## Advantages Over Full URL Targets + + * Interwiki links are easier to write. There is less typing, + and fewer opportunities to make mistakes. + + * Interwiki links are easier to read. With well-chosen + intermap tags, the links are easier to understand. + + * Interwiki links continue to work after a domain change on the + target. If the target of a link moves to a different domain, + an interwiki link will continue to work, if the intermap is adjusted, + but a hard-coded link will be permanently broken. + + * Interwiki links allow clones to use a different target domain from the + original repository. + +## Details + +Fossil supports interwiki links in both the +[Fossil Wiki](/wiki_rules) and [Markdown](/md_rules) markup +styles. An interwiki link consists of a tag followed by a colon +and the link target: + +> <i>Tag</i><b>:</b><i>PageName</i> + +The Tag must consist of ASCII alphanumeric characters only - no +punctuation or whitespace or characters greater than U+007A. +The PageName is the link notation on the target wiki. +Three different classes of PageNames are recognized by Fossil: + + 1. <b>Path Links</b> → the PageName begins with the "/" character + or is an empty string. + + 2. <b>Hash Links</b> → the PageName is a hexadecimal number with + at least four digits. + + 3. <b>Wiki Links</b> → A PageName that is not a Path or Hash. + +The Intermap defines a base URL for each Tag. Path links are appended +directly to the URL contained in the Intermap. The Intermap can define +additional text to put in between the base URL and the PageName for +Hash and Wiki links, respectively. + +<a id="intermap"></a> +## Intermap + +The intermap defines a mapping from interwiki Tags to full URLs. The +Intermap can be viewed and managed using the [fossil interwiki][iwiki] +command or the [/intermap][imap] webpage. + +[iwiki]: /help/interwiki +[imap]: /intermap + +The current intermap for a server is seen on the [/intermap][imap] page +(which is read-only for non-Setup users) and at the bottom of the built-in +[Fossil Wiki rules](/wiki_rules) and [Markdown rules](/md_rules) +documentation pages. + +Each intermap entry stores, at a minimum, the base URL for the remote +wiki. The intermap entry might also store additional path text that +is used for Hash and Wiki links. If only the base URL is provided, +then the intermap will only allow Path style interwiki links. The +Hash and Wiki style interwiki links are only allowed if the necessary +extensions are provided in the intermap. + + +## Disadvantages and Limitations + + * Configuration is required. The intermap must be set up correctly + before interwiki links will work. This contrasts with ordinary + links that just work without any configuration. Cloning a repository + copies the intermap, but normal syncs do not keep the intermap in + sync. Use the "[fossil config pull interwiki][fcfg]" command to + synchronize the intermap. + + * The is no backlink tracking. For ordinary intrawiki links, Fossil keeps + track of both the source and target, and when displaying targets it + commonly shows links to that target. For example, if you mention a + check-in as part of a comment of another check-in, that new check-in + shows up in the "References" section of the target check-in. + ([example](31af805348690958). In other words, Fossil tracks not just + "_source→target_", but it also tracks "_target→source_". + But backtracking does not work for interwiki links, since the Fossil + running on the target has no way of scanning the source text and + hence has no way of knowing that it is a target of a link from the source. + +[fcfg]: /help/config + +## Intermap Storage Details + +The intermap is stored in the CONFIG table of the repository database, +in entries with names of the form "<tt>interwiki:</tt><i>Tag</i>". The +value for each such entry is a JSON string that defines the base URL +and extensions for Hash and Wiki links. + +## See Also + + 1. [](https://www.mediawiki.org/wiki/Manual:Interwiki) + 2. [](https://duckduckgo.com/?q=interwiki+links&ia=web) diff --git a/www/javascript.md b/www/javascript.md new file mode 100644 index 0000000..512837d --- /dev/null +++ b/www/javascript.md @@ -0,0 +1,642 @@ +# Use of JavaScript in Fossil + +## Philosophy & Policy + +The Fossil development project’s policy is to use JavaScript where it +helps make its web UI better, but to offer graceful fallbacks wherever +practical. The intent is that the UI be usable with JavaScript +entirely disabled. In almost all places where Fossil uses JavaScript, +it is an enhancement to provided functionality, and there is always +another way to accomplish a given end without using JavaScript. + +This is not to say that Fossil’s fall-backs for such cases are always as +elegant and functional as a no-JS purist might wish. That is simply +because [the vast majority of web users leave JavaScript unconditionally +enabled](#stats), and of the small minority of those that do not, a +large chunk use some kind of [conditional blocking](#block) instead, +rather than disable JavaScript entirely. +Fossil’s active developers do not deviate from that +norm enough that we have many no-JS purists among us, so the no-JS case +doesn’t get as much attention as some might want. We do [accept code +contributions][cg], and we are philosophically in favor of graceful +fall-backs, so you are welcome to appoint yourself the position of no-JS +czar for the Fossil project! + +Evil is in actions, not in objects: we do not believe JavaScript *can* +be evil. It is an active technology, but the actions that matter here +are those of writing the code and checking it into the Fossil project +repository. None of the JavaScript code in Fossil is evil, a fact we +enforce by being careful about who we give check-in rights on the +repository to and by policing what code does get contributed. The Fossil +project does not accept non-trivial outside contributions. + +We think it’s better to ask not whether Fossil requires JavaScript but +whether Fossil uses JavaScript *well*, so that [you can decide](#block) +to block or allow Fossil’s use of JavaScript. + +The Fossil developers want to see the project thrive, and we achieve +that best by making it usable and friendly to a wider audience than the +minority of static web app purists. Modern users generally expect a +smoother experience than was available with 1990s style HTTP +POST-and-response `<form>` based interaction. We also increase the set +of potential Fossil developers if we do not restrict them to such +antiquated methods. + +JavaScript is not perfect, but it's what we have, so we will use it +where we find it advantageous. + +[cg]: ./contribute.wiki + + +## <a id="debate"></a>Arguments Against JavaScript & Our Rebuttals + +There are many common arguments against the use of JavaScript. Rather than +rehash these same arguments on the [forum][ffor], we distill the common +ones we’ve heard before and give our stock answers to them here: + +1. “**It increases the size of the page download.**” + + The heaviest such pages served by Fossil only have about 15 kB of + compressed JavaScript. (You have to go out of your way to get Fossil + to serve uncompressed pages.) This is negligible, even over very + slow data connections. If you are still somehow on a 56 kbit/sec + analog telephone modem, this extra script code would download in + a few seconds. + + Most JavaScript-based Fossil pages use less code than that. + + Atop that, Fossil sends HTTP headers to the browser that allow it + to perform aggressive caching so that typical page loads will skip + re-loading this content on subsequent loads. These features are + currently optional: you must either set the + [`fossil server --jsmode bundle` option][fsrv] or the corresponding + `jsmode` control line + in your [`fossil cgi`][fcgi] script when setting up your + [Fossil server][fshome]. That done, Fossil’s JavaScript files will + load almost instantly from the browser’s cache after the initial + page load, rather than be re-transferred over the network. + + Between the improved caching and the fact that it’s quicker to + transfer a partial Ajax page load than reload the entire page, the + aggregate cost of such pages is typically *lower* than the older + methods based on HTTP POST with a full server round-trip. You can + expect to recover the cost of the initial page load in 1-2 + round-trips. If we were to double the amount of JavaScript code in + Fossil, the payoff time would increase to 2-4 round-trips. + +2. “**JavaScript is slow.**” + + It *was*, before September 2008. Google's introduction of [their V8 + JavaScript engine][v8] taught the world that JavaScript need not be + slow. This competitive pressure caused the other common JavaScript + interpreters to either improve or be replaced by one of the engines + that did improve to approach V8’s speed. + + Nowadays JavaScript is, as a rule, astoundingly fast. As the world + continues to move more and more to web-based applications and + services, JavaScript engine developers have ample motivation to keep + their engines fast and competitive. + + Ajax partial page updates are faster than + the no-JS alternative, a full HTTP POST round-trip to submit new + data to the remote server, retrieve an entire new HTML document, + and re-render the whole thing client-side. + +3. “<a id="3pjs"></a>**Third-party JavaScript cannot be trusted.**” + + Fossil does not use any third-party JavaScript libraries, not even + very common ones like jQuery. Every bit of JavaScript served by the + stock version of Fossil was written specifically for the Fossil + project and is stored [in its code repository][fsrc]. + + Therefore, if you want to hack on the JavaScript code served by + Fossil and mechanisms like [skin editing][cskin] don’t suffice for your + purposes, you can hack on the JavaScript in your local instance + directly, just as you can hack on its C, SQL, and Tcl code. Fossil + is free and open source software, under [a single license][2cbsd]. + +4. “<a id="snoop"></a>**JavaScript and cookies are used to snoop on web users.**” + + There is no tracking or other snooping technology in Fossil other than + that necessary for basic security, such as IP address logging on + check-ins. (This is in part why we have no [comprehensive user + statistics](#stats)!) + + Fossil attempts to set two cookies on all web clients: a login session + cookie and a display preferences cookie. These cookies are restricted to + the Fossil instance, so even this limited data cannot leak between + Fossil instances or into other web sites. + +5. “**JavaScript is fundamentally insecure.**” + + JavaScript is certainly sometimes used for nefarious ends, but if we + wish to have more features in Fossil, the alternative is to add more + code to the Fossil binary, [most likely in C][fslpl], a language + implicated in [over 4× more security vulnerabilities][whmsl]. + + Therefore, does it not make sense to place approximately four times + as much trust in Fossil’s JavaScript code as in its C code? + + The question is not whether JavaScript is itself evil, it is whether + its *authors* are evil. *Every byte* of JavaScript code used within + the Fossil UI is: + + * ...written by the Fossil developers, vetted by their peers. + + * ...[open source][flic] and [available][fsrc] to be inspected, + audited, and changed by its users. + + * ...compiled directly into the `fossil` binary in a + non-obfuscated form during the build process, so there are no + third-party servers delivering mysterious, obfuscated JavaScript + code blobs to the user. + + Local administrators can [modify the repository’s skin][cskin] to + inject additional JavaScript code into pages served by their Fossil + server. A typical case is to add a syntax highlighter like + [Prism.js][pjs] or [highlightjs][hljs] to the local repository. At + that point, your trust concern is not with Fossil’s use of + JavaScript, but with your trust in that repository’s administrator. + + Fossil's [default content security policy][dcsp] (CSP) + prohibits execution of JavaScript code which is delivered from + anywhere but the Fossil server which delivers the page. A local + administrator can change this CSP, but again this comes down to a + matter of trust with the administrator, not with Fossil itself. + +6. “**Cross-browser compatibility is poor.**” + + It most certainly was in the first decade or so of JavaScript’s + lifetime, resulting in the creation of powerful libraries like + jQuery to patch over the incompatibilities. Over time, the need for + such libraries has dropped as browser vendors have fixed the + incompatibilities. Cross-browser JavaScript compatibility issues + which affect web developers are, by and large, a thing of the past. + +7. “**Fossil UI works fine today without JavaScript. Why break it?**” + + While this is true today, and we have no philosophical objection to + it remaining true, we do not intend to limit ourselves to only those + features that can be created without JavaScript. The mere + availability of alternatives is not a good justification for holding + back on notable improvements when they're within easy reach. + + The no-JS case is a [minority position](#stats), so those that want + Fossil to have no-JS alternatives and graceful fallbacks will need + to get involved with the development if they want this state of + affairs to continue. + +8. “<a id="stats"></a>**A large number of users run without JavaScript enabled.**” + + That’s not what web audience measurements say: + + * [What percentage of browsers with javascript disabled?][s1] + * [How many people are missing out on JavaScript enhancement?][s2] + * [Just how many web users really disable cookies or JavaScript?][s3] + + Our sense of this data is that only about 0.2% of web users had + JavaScript disabled while participating in these studies. + + The Fossil user community is not typical of the wider web, but if we + were able to comprehensively survey our users, we’d expect to find + an interesting dichotomy. Because Fossil is targeted at software + developers, who in turn are more likely to be power-users, we’d + expect to find Fossil users to be more in favor of some amount of + JavaScript blocking than the average web user. Yet, we’d also expect + to find that our user base has a disproportionately high number who + run [powerful conditional blocking plugins](#block) in their + browsers, rather than block JavaScript entirely. We suspect that + between these two forces, the number of no-JS purists among Fossil’s + user base is still a tiny minority. + +9. “<a id="block"></a>**I block JavaScript entirely in my browser. That breaks Fossil.**” + + First, see our philosophy statements above. Briefly, we intend that + there always be some other way to get any given result without using + JavaScript, developer interest willing. + + But second, it doesn’t have to be all-or-nothing. We recommend that + those interested in blocking problematic uses of JavaScript use + tools like [NoScript][ns] or [uBlock Origin][ubo] to *selectively* + block JavaScript so the rest of the web can use the technology + productively, as it was intended. + + There are doubtless other useful tools of this sort. We recommend + these two only from our limited experience, not out of any wish to + exclude other tools. + + The primary difference between these two for our purposes is that + NoScript lets you select scripts to run on a page on a case-by-case + basis, whereas uBlock Origin delegates those choices to a group of + motivated volunteers who maintain allow/block lists to control all + of this; you can then override UBO’s stock rules as needed. + +10. “**My browser doesn’t even *have* a JavaScript interpreter.**” + + The Fossil open source project has no full-time developers, and only + a few of these part-timers are responsible for the bulk of the code + in Fossil. If you want Fossil to support such niche use cases, then + you will have to [get involved with its development][cg]: it’s + *your* uncommon itch. + +11. “<a id="compat"></a>**Fossil’s JavaScript code isn’t compatible with my browser.**” + + The Fossil project’s developers aim to remain compatible with + the largest portions of the client-side browser base. We use only + standards-defined JavaScript features which are known to work in the + overwhelmingly vast majority of browsers going back approximately 5 + years, at minimum, as documented by [Can I Use...?][ciu] We avoid use of + features added to the language more recently or those which are still in + flux in standards committees. + + We set this threshold based on the amount of time it typically takes for + new standards to propagate through the installed base. + + As of this writing, this means we are only using features defined in + [ECMAScript 2015][es2015], colloquially called “JavaScript 6.” That + is a sufficiently rich standard that it more than suffices for our + purposes, and it is [widely deployed][es6dep]. The biggest single + outlier remaining is MSIE 11, and [even Microsoft is moving their + own products off of it][ie11x]. + +[2cbsd]: https://fossil-scm.org/home/doc/trunk/COPYRIGHT-BSD2.txt +[ciu]: https://caniuse.com/ +[cskin]: ./customskin.md +[dcsp]: ./defcsp.md +[es2015]: https://ecma-international.org/ecma-262/6.0/ +[es6dep]: https://caniuse.com/#feat=es6 +[fcgi]: /help/cgi +[ffor]: https://fossil-scm.org/forum/ +[flic]: /doc/trunk/COPYRIGHT-BSD2.txt +[fshome]: /doc/trunk/www/server/ +[fslpl]: /doc/trunk/www/fossil-v-git.wiki#portable +[fsrc]: https://fossil-scm.org/home/file/src +[fsrv]: /help/server +[hljs]: https://fossil-scm.org/forum/forumpost/9150bc22ca +[ie11x]: https://techcommunity.microsoft.com/t5/microsoft-365-blog/microsoft-365-apps-say-farewell-to-internet-explorer-11-and/ba-p/1591666 +[ns]: https://noscript.net/ +[pjs]: https://fossil-scm.org/forum/forumpost/1198651c6d +[s1]: https://blockmetry.com/blog/javascript-disabled +[s2]: https://gds.blog.gov.uk/2013/10/21/how-many-people-are-missing-out-on-javascript-enhancement/ +[s3]: https://w3techs.com/technologies/overview/client_side_language/all +[ubo]: https://github.com/gorhill/uBlock/ +[v8]: https://en.wikipedia.org/wiki/V8_(JavaScript_engine) +[whmsl]: https://www.whitesourcesoftware.com/most-secure-programming-languages/ + + +---- + +## <a id="uses"></a>Places Where Fossil’s Web UI Uses JavaScript + +This section documents the areas where Fossil currently uses JavaScript +and what it does when these uses are blocked. It also gives common +workarounds where necessary. + + +### <a id="timeline"></a>Timeline Graph + +Fossil’s [web timeline][wt] uses JavaScript to render the graph +connecting the visible check-ins to each other, so you can visualize +parent/child relationships, merge actions, etc. We’re not sure it’s even +possible to render this in static HTML, even with the aid of SVG, due to +the vagaries of web layout among browser engines, screen sizes, etc. + +Fossil also uses JavaScript to handle clicks on the graph nodes to allow +diffs between versions, to display tooltips showing local context, etc. + +_Graceful Fallback:_ When JavaScript is disabled, this column of the +timeline simply collapses to zero width. All of the information you can +get from the timeline can be retrieved from Fossil in other ways not +using JavaScript: the “`fossil timeline`” command, the “`fossil info`” +command, by clicking around within the web UI, etc. + +_Potential Workaround:_ The timeline could be enhanced with `<noscript>` +tags that replace the graph with a column of checkboxes that control +what a series of form submit buttons do when clicked, replicating the +current JavaScript-based features of the graph using client-server round-trips. +For example, you could click two of those checkboxes and then a button +labeled “Diff Selected” to replicate the current “click two nodes to +diff them” feature. + +[wt]: https://fossil-scm.org/home/timeline + + +### <a id="wedit"></a>The New Wiki Editor + +The [new wiki editor][fwt] has many new features, a +few of which are impossible to get without use of JavaScript. + +First, it allows in-browser previews without losing client-side editor +state, such as where your cursor is. With the old editor, you had to +re-locate the place you were last editing on each preview, which would +reduce the incentive to use the preview function. In the new wiki +editor, you just click the Preview tab to see how Fossil interprets your +markup, then click back to the Editor tab to resume work with the prior +context undisturbed. + +Second, it continually saves your document state in client-side storage +in the background while you’re editing it so that if the browser closes +without saving the changes back to the Fossil repository, you can resume +editing from the stored copy without losing work. This feature is not so +much about saving you from crashes of various sorts, since computers are +so much more reliable these days. It is far more likely to save you from +the features of mobile OSes like Android and iOS which aggressively shut +down and restart apps to save on RAM. That OS design philosophy assumes +that there is a way for the app to restore its prior state from +persistent media when it’s restarted, giving the illusion that it was +never shut down in the first place. This feature of Fossil’s new wiki +editor provides that. + +With this change, we lost the old WYSIWYG wiki editor, available since +Fossil version 1.24. It hadn’t been maintained for years, it was +disabled by default, and no one stepped up to defend its existence when +this new editor was created, replacing it. If someone rescues that +feature, merging it in with the new editor, it will doubtless require +JavaScript in order to react to editor button clicks like the “**B**” +button, meaning “make \[selected\] text boldface.” There is no standard +WYSIWYG editor component in browsers, doubtless because it’s relatively +straightforward to create one using JavaScript. + +_Graceful Fallback:_ Fossil’s lack of +a script-free wiki editor mode is not from lack of +desire, but because the person who wrote the new wiki editor didn’t +want to maintain three different editors. (New Ajaxy editor, old +script-free HTML form based editor, and the old WYSIWYG JavaScript-based +editor.) If someone wants to implement a `<noscript>` alternative to the +new wiki editor, we will likely accept that [contribution][cg] as long +as it doesn’t interfere with the new editor. (The same goes for adding +a WYSIWYG mode to the new Ajaxy wiki editor.) + +_Workaround:_ You don’t have to use the browser-based wiki editor to +maintain your repository’s wiki at all. Fossil’s [`wiki` command][fwc] +lets you manipulate wiki documents from the command line. For example, +consider this Vi based workflow: + +```shell +$ vi 'My Article.wiki' # begin work on new article + ...write, write, write... +:w # save changes to disk copy +:!fossil wiki create 'My Article' '%' # current file (%) to new article + ...write, write, write some more... +:w # save again +:!fossil wiki commit 'My Article' '%' # update article from disk +:q # done writing for today + + ....days later... +$ vi # work sans named file today +:r !fossil wiki export 'My Article' - # pull article text into vi buffer + ...write, write, write yet more... +:w !fossil wiki commit - # vi buffer updates article +``` + +Extending this concept to other text editors is an exercise left to the +reader. + +[fwc]: /help/wiki +[fwt]: ./wikitheory.wiki + + +### <a id="fedit"></a>The File Editor + +Fossil’s [optional file editor feature][fedit] works +much like [the new wiki editor](#wedit), only on files committed to the +repository. + +The original designed purpose for this feature is to allow [embedded +documentation][edoc] to be interactively edited in the same way that +wiki articles can be. (Indeed, the associated `fileedit-glob` feature +allows you to restrict the editor to working *only* on files that can be +treated as embedded documentation.) This feature operates in much the +same way as the new wiki editor, so most of what we said above applies. + +_Workaround:_ This feature is an alternative to Fossil’s traditional +mode of file management: clone the repository, open it somewhere, edit a +file locally, and commit the changes. + +_Graceful Fallback:_ There is no technical reason why someone could not +write a `<noscript>` wrapped alternative to the current JavaScript based +`/fileedit` implementation. It would have all of the same downsides as +the old wiki editor: the users would lose their place on each save, they +would have no local backup if something crashes, etc. Still, we are +likely to accept such a [contribution][cg] as long as it doesn’t +interfere with the new editor. + +[edoc]: /doc/trunk/www/embeddeddoc.wiki +[fedit]: /doc/trunk/www/fileedit-page.md + + +### <a id="ln"></a>Line Numbering + +When viewing source files, Fossil offers to show line numbers in some +cases. ([Example][mainc].) Toggling them on and off is currently handled +in JavaScript, rather than forcing a page-reload via a button click. + +_Workaround:_ Manually edit the URL to give the “`ln`” query parameter +per [the `/file` docs](/help/www/file). + +_Potential Better Workaround:_ Someone sufficiently interested could +[provide a patch][cg] to add a `<noscript>` wrapped HTML button that +would reload the page with this parameter included/excluded to implement +the toggle via a server round-trip. + +A related feature is Fossil’s JavaScript-based interactive method +for selecting a range of lines by clicking the line numbers when they’re +visible. JavaScript lets us copy the resulting URL to the clipboard +to share your selection with others. + +_Workaround:_ These interactive features would be difficult and +expensive (in terms of network I/O) to implement without JavaScript. A +far simpler alternative is to manually edit the URL, per above. + +[mainc]: https://fossil-scm.org/home/artifact?ln&name=87d67e745 + + +### <a id="sxsdiff"></a>Side-by-Side Diff Mode + +The default “diff” view is a side-by-side mode. If either of the boxes +of output — the “from” and “to” versions of the repo contents for that +check-in — requires a horizontal scroll bar given the box content, font +size, browser window width, etc., both boxes will usually end up needing +to scroll since they should contain roughly similar content. Fossil +therefore scrolls both boxes when you drag the scroll bar on one because +if you want to examine part of a line scrolled out of the HTML element +in one box, you probably want to examine the same point on that line in +the other box. + +_Graceful Fallback:_ Manually scroll both boxes to sync their views. + +### <a id="diffcontext"></a>Diff Context Loading + +Fossil’s diff views can +dynamically load more lines of context around changed blocks. The UI +controls for this feature are injected using JavaScript when the page +initializes and make use of XHR requests to fetch data from the +fossil instance. + +_Graceful Fallback:_ The UI controls for this feature do not appear +when JS is unavailable, leaving the user with the "legacy" static diff +view. + + +### <a id="sort"></a>Table Sorting + +On pages showing a data table, the column headers may be clickable to do +a client-side sort of the data on that column. + +_Potential Workaround:_ This feature could be enhanced to do the sort on +the server side using a page re-load. + + +### <a id="tree"></a>File Browser Tree View + +The [file browser’s tree view mode][tv] uses JavaScript to handle clicks +on folders so they fold and unfold without needing to reload the entire +page. + +_Graceful Fallback:_ When JavaScript is disabled, clicks on folders +reload the page showing the folder contents instead. You then have to +use the browser’s Back button to return to the higher folder level. + +[tv]: https://fossil-scm.org/home/dir?type=tree + + +### <a id="hash"></a>Version Hashes + +In several places where the Fossil web UI shows a check-in hash or +similar, hovering over that check-in shows a tooltip with details about +the type of artifact the hash refers to and allows you to click to copy +the hash to the clipboard. + +_Graceful Fallback:_ When JavaScript is disabled, these tooltips simply +don’t appear, but you can still select and copy the hash using your +platform’s “copy selected text” feature. + + +### <a id="bots"></a>Anti-Bot Defenses + +Fossil has [anti-bot defenses][abd], and it has some JavaScript code +that, if run, can drop some of these defenses if it decides a given page +was loaded on behalf of a human, rather than a bot. + +_Graceful Fallback:_ You can use Fossil’s anonymous login feature to +convince the remote Fossil instance that you are not a bot. Coupled with +[the Fossil user capability system][caps], you can restore all +functionality that Fossil’s anti-bot defenses deny to random web clients +by default. + +[abd]: ./antibot.wiki +[caps]: ./caps/ + + +### <a id="hbm"></a>Hamburger Menu + +Several of the stock skins (including the default) include a “hamburger menu” (☰) which uses +JavaScript to show a simplified version of the Fossil UI site map using +an animated-in dropdown. + +_Graceful Fallback:_ Clicking the hamburger menu button with JavaScript +disabled will take you to the `/sitemap` page instead of showing a +simplified version of that page’s content in a drop-down. + +_Workaround:_ You can remove this button by [editing the skin][cskin] +header. + + +### <a id="clock"></a>Clock + +Some stock Fossil skins include JavaScript-based features such as the +current time of day. The Xekri skin includes this in its header, for +example. A clock feature requires JavaScript to get the time on initial +page load and then to update it once a minute. + +You may observe that the server could provide the current time when +generating the page, but the client and server may not be in the same +time zone, and there is no reliably-provided information from the client +that would let the server give the page load time in the client’s local +time zone. The server could only tell you *its* local time at page +request time, not the client’s time. That still wouldn’t be a “clock,” +since without client-side JavaScript code running, that part of the page +couldn’t update once a second. + +_Potential Graceful Fallback:_ You may consider showing the server’s +page generation time rather than the client’s wall clock time in the +local time zone to be a useful fallback for the current feature, so [a +patch to do this][cg] may well be accepted. Since this is not a +*necessary* Fossil feature, an interested user is unlikely to get the +core developers to do this work for them. + + +### <a id="chat"></a>Chat + +The [chat feature](./chat.md) is deeply dependent +on JavaScript. There is no obvious way to do this sort of thing without +active client-side code of some sort. + +_Potential Workaround:_ It would not be especially difficult for someone +sufficiently motivated to build a Fossil chat gateway, connecting to +IRC, Jabber, etc. The messages are stored in the repository’s `chat` +table with monotonically increasing IDs, so a poller that did something +like + + SELECT xfrom, xmsg FROM chat WHERE msgid > 1234; + +…would pull the messages submitted since the last poll. Making the +gateway bidirectional should be possible as well, as long as it properly +uses SQLite transactions. + +### <a id="brlist"></a>List of branches + +The [`/brlist`](/brlist) page uses JavaScript to enable +selection of several branches for further study via `/timeline`. +Client-side script interactively responds to checkboxes' events +and constructs a special hyperlink in the submenu. +Clicking this hyperlink loads a `/timeline` page that shows +only these selected branches (and the related check-ins). + +_Potential Workaround:_ A user can manually construct an appropriate +regular expession and put it into the "Tag Filter" entry of the +`/timeline` page (in its advanced mode). + +---- + +## <a id="future"></a>Future Plans for JavaScript in Fossil + +As of mid-2020, the informal provisional plan is to increase Fossil +UI's use of JavaScript considerably compared to its historically minimal +uses. To that end, a framework of Fossil-centric APIs is being developed +in conjunction with new features to consolidate Fossil's historical +hodgepodge of JavaScript snippets into a coherent code base. + +When deciding which features to port to JavaScript, the rules of thumb +for this ongoing effort are: + +- Pages which primarily display data (e.g. the timeline) will remain + largely static HTML with graceful fallbacks for all places they do + use JavaScript. Though JavaScript can be used effectively to power + all sorts of wonderful data presentation, Fossil currently doesn't + benefit greatly from doing so. We use JavaScript on these pages only + to improve their usability, not to define their primary operations. + +- Pages which act as editors of some sort (e.g. the `/info` page) are + prime candidates for getting the same treatment as the old wiki + editor: reimplemented from the ground up in JavaScript using Ajax + type techniques. Similarly, a JS-driven overhaul is planned for the + forum’s post editor. + +These are guidelines, not immutable requirements. Our development +direction is guided by our priorities: + +1) Features the developers themselves want to have and/or work on. + +2) Features end users request which catch the interest of one or more +developers, provided the developer(s) in question are in a position to +expend the effort. + +3) Features end users and co-contributors can convince a developer into +coding even when they really don't want to. + +In all of this, Fossil's project lead understandably has the final +say-so in whether any given feature indeed gets merged into the mainline +trunk. Development of any given feature, no matter how much effort was +involved, does not guarantee its eventual inclusion into the public +releases. diff --git a/www/json-api/_template.md b/www/json-api/_template.md new file mode 100644 index 0000000..92b9074 --- /dev/null +++ b/www/json-api/_template.md @@ -0,0 +1,22 @@ +# JSON API: X +([⬑JSON API Index](index.md)) + +Jump to: + +* [Foo](#foo) +* [Bar](#bar) +* [Baz](#baz) + +--- + +<a id="foo"></a> +# Foo + +<a id="bar"></a> +# Bar + +<a id="baz"></a> +# Baz + + +# Footnotes diff --git a/www/json-api/api-artifact.md b/www/json-api/api-artifact.md new file mode 100644 index 0000000..c0e9a54 --- /dev/null +++ b/www/json-api/api-artifact.md @@ -0,0 +1,155 @@ +# JSON API: /artifact +([⬑JSON API Index](index.md)) + +Jump to: + +* [Checkin Artifacts (Commits)](#checkin) +* [File Artifacts](#file) +* [Wiki Artifacts](#wiki) + +--- + +<a id="checkin"></a> +# Checkin Artifacts (Commits) + +Returns information about checkin artifacts (commits). + +**Status:** implemented 201110xx + +**Request:** `/json/artifact/COMMIT_HASH` + +**Required permissions:** "o" (was "h" prior to 20120408) + +**Response payload example: (CHANGED SIGNIFICANTLY ON 20120713)** + +```json +{ +"type":"checkin", +"name":"18dd383e5e7684e", // as given on CLI +"uuid":"18dd383e5e7684ecee327d3de7d3ff846069d1b2", +"isLeaf":false, +"user":"drh", +"comment":"Merge wideAnnotateUser and jsonWarnings into trunk.", +"timestamp":1330090810, +"parents":[ + // 1st entry is primary parent hash: + "3a44f95f40a193739aaafc2409f155df43e74a6f", + // Remaining entries are merged-in branch hashes: + "86f6e675eb3f8761d70d8b82b052ce2b297fffd2",\ + "dbf4ecf414881c9aad6f4f125dab9762589ef3d7"\ +], +"tags":["trunk"], +"files":[{ + "name":"src/diff.c", + // BLOB hashes, NOT commit hashes: + "uuid":"78c74c3b37e266f8f7e570d5cf476854b7af9d76", + "parent":"b1fa7e636cf4e7b6ed20bba2d2680397f80c096a", + "state":"modified", + "downloadPath":"/raw/src/diff.c?name=78c74c3b37e266f8f7e570d5cf476854b7af9d76" + }, + ...] +} +``` + +The "parents" property lists the parent hashes of the checkin. The +"parent" property of file entries refers to the parent hash of that +file. In the case of a merge there may be essentially an arbitrary +number. The first entry in the list is the "primary" parent. The primary +parent is the parent which was not pulled in via a merge operation. The +ordering of remaining entries is unspecified and may even change between +calls. For example: if, from branch C, we merge in A and B and then +commit, then in the artifact response for that commit the hash of branch +C will be in the first (primary) position, with the hashes for branches A +and B in the following entries (in an unspecified, and possibly +unstable, order). + +Note that the "uuid" and "parent" properties of the "files" entries +refer to raw blob hashes, not commit (a.k.a. check-in) hashes. See also +[the UUID vs. Hash discussion][uvh]. + +<a id="file"></a> +# File Artifacts + +Fetches information about file artifacts. + +**FIXME:** the content type guessing is currently very primitive, and +may (but I haven't seen this) mis-diagnose some non-binary files as +binary. Fossil doesn't yet have a mechanism for mime-type mappings. + +**Status:** implemented 20111020 + +**Required permissions:** "o" + +**Request:** `/json/artifact/FILE_HASH` + +**Request options:** + +- `format=(raw|html|none)` (default=none). If set, the contents of the + artifact are included if they are text, else they are not (JSON does + not do binary). The "html" flag runs it through the wiki parser. The + results of doing so are unspecified for non-embedded-doc files. The + "raw" format means to return the content as-is. "none" is the same + as not specifying this flag, and elides the content from the + response. +- DEPRECATED (use format instead): `includeContent=bool` (=false) (CLI: + `--content|-c`). If true, the full content of the artifact is returned + for text-only artifacts (but not for non-text artifacts). As of + 20120713 this option is only inspected if "format" is not specified. + +**Response payload example: (CHANGED SIGNIFICANTLY ON 20120713)** + +```json +{ +"type":"file", +"name":"same name specified as FILE_HASH argument", +"size": 12345, // in bytes, independent of format=... +"parent": "hash of parent file blob. Not set for first generation.", +"checkins":[{ + "name":"src/json_detail.h", + "timestamp":1319058803, + "comment":"...", + "user":"stephan", + "checkin":"d2c1ae23a90b24f6ca1d7637193a59d5ecf3e680", + "branch":"json", + "state":"added|modified|removed" + }, + ...], +/* The following "content" properties are only set if format=raw|html */ +"content": "file contents", +"contentSize": "size of content field, in bytes. Affected by the format option!", +"contentType": "text/plain", /* currently always text/plain */ +"contentFormat": "html|raw" +} +``` + +The "checkins" array lists all checkins which include this file, and a +file might have different names across different branches. The size and +hash, however, are the same across all checkins for a given blob. + +<a id="wiki"></a> +# Wiki Artifacts + +Returns information about wiki artifacts. + +**Status:** implemented 20111020, fixed to return the requested version +(instead of the latest) on 20120302. + +**Request:** `/json/artifact/WIKI_HASH` + +**Required permissions:** "j" + +**Options:** + +- DEPRECATED (use format instead): `bool includeContent` (=false). If + true then the raw content is returned with the wiki page, else no + content is returned.\ + CLI: `--includeContent|-c` +- The `--format` option works as for + [`/json/wiki/get`](api-wiki.md#get), and if set then it + implies the `includeContent` option. + +**Response payload example:** + +Currently the same as [`/json/wiki/get`](api-wiki.md#get). + +[uvh]: ../hashes.md#uvh diff --git a/www/json-api/api-auth.md b/www/json-api/api-auth.md new file mode 100644 index 0000000..96854da --- /dev/null +++ b/www/json-api/api-auth.md @@ -0,0 +1,271 @@ +# JSON API: X +([⬑JSON API Index](index.md)) + +Jump to: + +* [Introduction](#intro) +* [Capabilities (Access Rights)](#capabilities) +* [Login](#login) + * [Anonymous User Logins](#login-anonymous) +* [Logout](#logout) +* [Whoami](#whoami) + +--- + +<a id="intro"></a> +# Introduction + +**FIXME:** Ross found a bug: +[](/info/479aadb1d2645601) +(sending the authToken via `POST.envelope` isn't working. It should be.) + +The authentication-related operations are described in this section +(ordered alphabetically by operation name). + +The JSON API ties in to/piggybacks on fossil's existing cookie-based +login mechanism, but we must also keep in mind that not all JSON-using +clients support cookies. Cookie-capable clients can rely on cookies for +authentication, whereas non-cookie-aware clients will need to keep track +of the so-called "auth token" which is created via a login request, and +pass it in with each request (either as a GET parameter or a property of +the top-level POST request envelope) to prove to fossil that they are +authorized to access the requested resources. For most intents and +purposes, the "auth token" and the "login cookie" are the same thing (or +serve the same purpose), and the auth token is in fact just the value +part of the login cookie (which has a project-specific key). + +Note that fossil has two conventional user names which can show up in +various responses but do not refer to specific people: nobody and +anonymous. The nobody user is anyone who is not logged in. The anonymous +user is logged in but has no persistent user data (no associated user +name, email address, or similar). Normally the guest (nobody) user has +more access restrictions. The distinction between the two is largely +historical - it is a mechanism to keep bots from following the +multitudes of links generated by the HTML interface (especially the ZIP +files), while allowing interested users to do so by logging in as the +anonymous user (which bots have (or *had*, at the time) a harder time +doing because the password is randomly generated and protected from +brute-force attacks by hashing). In the JSON API, the distinction +between anonymous and nobody is not expected to be as prominent as it is +in the HTML interface because the reasons for the distinction don't +apply in *quite* the same ways to the JSON interface. It is possible, +however, that a given repo locks out guest access to, e.g. the wiki or +tickets, while still allowing anonymous (logged in) access. + +<a id="capabilities"></a> +# Capabilities (Access Rights) + +The "cap" request returns information about the so-called "capabilities" +(access rights) of the currently logged in user. This command is +basically the same as the "whoami" command, but returns capabilities in +a more exanded form (same data, different representation) and does not +include the authToken in the response. + +TODO: consider combining this and [whoami](#whoami) into a single +whoami response. We don't need both. We also don't really need the +permissionFlags member - the same info is already \[in a more cryptic form\] +in the capabilities string. + +**Status:** implemented 201109xx. + +**Required privileges:** none + +**Request:** `/json/cap` + +In CLI mode, permissions are not used/honored, and this command will +report that the caller has all permissions (which he effectively does). + +**Response payload example:** + +```json +{ +"userName":"json-demo", +"capabilities":"hgjorxz", /* raw fossil permissions list */ +"permissionFlags":{ /* Same info in a somewhat friendlier form */ + "setup": false, + "admin": false, + "delete": false, + "password": false, + "query": false, + "checkin": false, + "checkout": true, + "history": true, + "clone": false, + "readWiki": true, + "createWiki": false, + "appendWiki": false, + "editWiki": false, + "readTicket": true, + "createTicket": false, + "appendTicket": false, + "editTicket": false, + "attachFile": false, + "createTicketReport": false, + "readPrivate": false, + "zip": true, + "xferPrivate": false + } +} +``` + +**FIXME:** several new permissions have been added to fossil since +this API was implemented. + + +<a id="login"></a> +# Login + +**Status:** implemented 20110915 (anonymous login 20110918) + +**Required privileges:** none + +**Request:** (see below for Anonymous user special case) + +- `/json/login?name=...&password=...` +- `/json/login?n=...&p=...` (for symmetry with existing login mechanism) + +Or `POST.payload`: `{ "name": ..., "password": ...}` + +(POST is highly preferred because it keeps the password out of web +server logs!) + +**Response payload example:** (structure changed 20111001) + +```json +{ +"authToken":"...", +"name":"json-demo", +"capabilities":"hgjorxz", +"loginCookieName": "fossil-XXXXX" /*project-specific cookie name*/ +/* TODO: add authTokenExpiry timestamp (cookie expiry time) */ +} +``` + +We "should" be able to inherit fossil's `REMOTE_USER` handling without +any special support, but that is untested so far. (If you happen to test +this, please update this doc with (or otherwise report) your results!) + +The response *also* sets the conventional fossil login cookie (for +clients which can make use of cookies), using fossil's existing +mechanism for this. + +Further requests which require authentication must include the +`authToken` (from the returned payload value) in the request (or it must +be available via fossil's standard cookie) or access may (depending on +the request) be denied. The `authToken` may optionally be set in the +request envelope or as a GET parameter, and it *must* be given if the +request requires restricted access to a resource. e.g. if reading +tickets is disabled for the guest user then all non-guest users must +send authentication info in their requests in order to be able to fetch +ticket info. + +Cookie-aware clients should send the login-generated cookie with each +request, in which case they do not need to explicitly include the +`authToken` in the JSON envelope/GET arguments. If submitted, the +`authToken` is used, otherwise the cookie, if set, is used. Note that +fossil uses a project-dependent cookie name in order to help thwart +attacks, so there is no simple mapping of cookie *name* to auth +token. That said, the cookie's *value* is also the auth token's value. + +> Special case: when accessing fossil over a local server instance +which was started with the `--localauth` flag, the `authToken` is ignored +(neither validated nor used for any form of authentication). + + +<a id="login-anonymous"></a> +## Anonymous User Logins + +The Anonymous user requires special handling because he has a random +password. + +First fetch the password and the so-called "captcha seed" via this +request: + +`/json/anonymousPassword` + +It will return a payload in the form: + +```json +{ +"seed": a_32_bit_unsigned_integer, +"password": "1234abcd" /*hexadecimal STRING*/ +} +``` + +The "seed" and "password" values of the response payload must be set as +the "anonymousSeed" and "password" fields (respectively) of the +subsequent login request. The login request is identical to +non-anonymous login except that extra "anonymousSeed" property is +required. + +The password value *may* be time-limited, and *may* eventually become +invalidated due to old age. This is unspecified. + +***Potential***** (low-probability) bug regarding the seed value:** from +what I hear, some unusual JSON platforms don't support full 32-bit +precision. If absolutely necessary we could chop off a bit or two from +the seed value (*if* it ever becomes a problem and if DRH blesses it). +Or we could just make it a double. + + +<a id="logout"></a> +# Logout + +**Status:** implemented 20110916 + +**Required privileges:** none, but must be logged in + +**Request:** + + - `/json/logout?authToken=...token fetched by /login` + +Or: set the `authToken` property of the POST envelope (as opposed to the +`POST.payload`) + +Or: fossil's normal cookie mechanism is the fallback for the auth token. + +**Response payload:** The same as the "whoami" response, containing the +info for the "nobody" user. + +This request requires the authentication token, and subsequent logouts +without an intervening login will fail with the "auth token not +provided" error. In effect this request removes the login entry from the +user account, making the token invalid for future requests. In HTTP +mode, on success fossil's login cookie is unset by this call. + + +<a id="whoami"></a> +# Whoami + +This request fetches the current user's login name, capabilities, and +auth token. This can be used to check whether a login is active when the +client has not explicitly logged in (e.g. was logged in automatically +via a pre-existing cookie). + +**Status:** implemented 20110922 + +**Required privileges:** none + +**Request:** `/json/whoami` + +**Response payload example:** + +```json +{ +"name": "nobody", +"capabilities": "o", +"authToken": "fossil auth token (only for logged-in users)" +} +``` + +The reason authToken is included in the response is because it gives +client-side JavaScript code a way of fetching/checking for the auth +token at app startup. The token is normally sent as a cookie but parsing +the cookies in the browser is tedious, and fossil has a +project-dependent cookie name (which complicates parsing a bit). If +client code digs the cookie out of the browser, the app still wouldn't +know if the token is still valid, whereas whoami won't (or shouldn't!) +return an expired auth token. If the request does not include +authentication info (via the cookie, GET param, or request envelope) +then the response will not contain the authToken property and the user's +name will be "nobody". diff --git a/www/json-api/api-branch.md b/www/json-api/api-branch.md new file mode 100644 index 0000000..a2a002b --- /dev/null +++ b/www/json-api/api-branch.md @@ -0,0 +1,86 @@ +# JSON API: /branch +([⬑JSON API Index](index.md)) + +Jump to: + +* [Branch List](#list) +* [Create Branch](#create) + +--- + +<a id="list"></a> +# Branch List + +**Status:** implemented, at least in draft form, on 20110921. + +**Required privileges:** "o" + +**Request:** `/json/branch/list` + +**Response payload example:** + +```json +{ +"range":"closed", +"current":"json", /* only when there is a local opened checkout */ +"branches":[ + "artifact_description", + "bch", + "ben-changes-report", + "ben-safe-make", + "ben-security", + "ben-testing", + … +] +} +``` + +*Potential* TODO: add "tip" property which names the most recently +modified branch? (How to get this?) + +HTTP GET/`POST.payload` options: + +- `range`: a string in the set ("open", "closed", "all"), + case-sensitive, but only the first letter is actually evaluated. + Default="open". Only branches with this state are returned. + +CLI mode options (same semantics as HTTP equivalents), must come last on +the CLI: + +- `-r|--range all|closed|open` +- `-a` (equivalent to `-r all`) +- `-c` (equivalent to `-r closed`). Only one of `-a`/`-c` may be specified, + and if both are specified then which one takes precedence is + unspecified. + + +<a id="create"></a> +# Create Branch + +**Status:** implemented 20111002 + +**Required privileges:** "w" + +**Request:** `/json/branch/create` + +**Request options:** + +- `name=string` (REQUIRED) Name of new branch +- `basis=string` (default=trunk) Name of parent branch to branch from. +- `bgColor=string` (default=something ugly) In `#RRGGBB` form. (FIXME: + change the default to use fossil's random bgcolor technique.) +- `private=bool` (default=false) Determines whether the branch is + private or not. + +**Response payload example:** + +```json +{ +"name":"my-branch", +"basis":"my-other-branch", +"uuid":"de8115db4ce388ed8d0af666ae7d90e1410be4ca", +"isPrivate":true, +"bgColor":"#ff0000" +} +``` + diff --git a/www/json-api/api-checkout.md b/www/json-api/api-checkout.md new file mode 100644 index 0000000..0fd2437 --- /dev/null +++ b/www/json-api/api-checkout.md @@ -0,0 +1,54 @@ +# JSON API: /status +([⬑JSON API Index](index.md)) + +# Status of Local Checkout + +**Status:** implemented 20130223 + +**Required permissions:** n/a (local access only) + +**Request:** `/json/status` + +This command requires a local checkout and is the analog to the "fossil +status" command. + +**Request Options:** currently none. + +Payload example: + +```json +{ +"repository":"/home/stephan/cvs/fossil/fossil.fsl", +"localRoot":"/home/stephan/cvs/fossil/fossil/", +"checkout":{ + "uuid":"b38bb4f9bd27d0347b62ecfac63c4b8f57b5c93b", + "tags":["trunk"], + "datetime":"2013-02-22 17:34:19 UTC", + "timestamp":1361554459 + }, +/* "parent" info is currently missing. */ +"files":[ + {"name":"src/checkin.c", "status":"edited"} + ...], +"errorCount":0 /* see notes below */ +} +``` + +Notes: + +- The `checkout.tags` property follows the framework-wide convention + that the first tag in the list is the primary branch and any others + are secondary. +- `errorCount` is +1 for each missing file. Conflicts are not treated as + errors because the CLI 'status' command does not treat them as such. +- The `"status"` entry for each of the `"files"` entries will be either a + single string containing a single description of the status change, or + an array of strings if more than one change was made, e.g. `"edited"` + and `"renamed"`. The status strings include:\ + `deleted`, `new`, `notAFile`, `missing`, `updatedByMerge`, + `updatedByIntegrate`, `addedBymerge`, `addedByIntegrate`, + `conflict`, `edited`, `renamed` +- File objects with a `"renamed"` status will contain a `"priorName"` + property in addition to the `"name"` property reported in all cases. +- TODO: Info for the parent version is currently missing. +- TODO: "merged with" info for the checkout is currently missing. diff --git a/www/json-api/api-config.md b/www/json-api/api-config.md new file mode 100644 index 0000000..e89b26f --- /dev/null +++ b/www/json-api/api-config.md @@ -0,0 +1,36 @@ +# JSON API: /config +([⬑JSON API Index](index.md)) + +Jump to: + +* [Get Config](#get) +* [Set Config](#set) + +--- + +<a id="get"></a> +# Fetch Config + +**Status:** Implemented 20120217 + +**Required permissions:** "s" + +**Request:** `/json/config/get/Area[/Area2/...AreaN]` + +Where "Area" can be any combination of: *skin*, *ticket*, *project*, +*all*, or *skin-backup* (which is not included in "all" by default). + +**Response payload example:** + +```json +{ +"ignore-glob":"*~", +"project-description":"For testing Fossil's JSON API.", +"project-name":"fossil-json-tests" +} +``` + +<a id="set"></a> +# Set/Modify Config + +Not implemented. diff --git a/www/json-api/api-diff.md b/www/json-api/api-diff.md new file mode 100644 index 0000000..d001314 --- /dev/null +++ b/www/json-api/api-diff.md @@ -0,0 +1,44 @@ +# JSON API: /diff +([⬑JSON API Index](index.md)) + +# Diffs + +**Status:** implemented 20111007 + +**Required permissions:** "o" + +**Request:** `/json/diff[/version1[/version2]]` + +**Request options:** + +- `v1=string` Is the "from" version. It may optionally be the first + positional parameter/path element after the command name. +- `v2=string` Is the "to" version. It may optionally be the first + positional parameter/path element after the v1 part. +- `context=integer` (default=unspecified) Defines the number of context + lines to display in the diff.\ + CLI: `--context|-c` +- `sbs=bool` (default=false) Generates "side-by-side" diffs, but their + utility in JSON mode is questionable. +- `html=bool` (default=false) causes the output to be marked up with + HTML in the same manner as it is in the HTML interface. + +**Response payload example:** + +```json +{ +"from":"7a83a5cbd0424cefa2cdc001de60153aede530f5", +"to":"96920e7c04746c55ceed6e24fc82879886cb8197", +"diff":"@@ -1,7 +1,7 @@\\n-C factored\\\\sout..." +} +``` + +TODOs: + +- Unlike the standard diff command, which apparently requires a commit + hash, this one diffs individual file versions. If a commit hash is + provided, a diff of the manifests is returned. (That should be + considered a bug - we should return a combined diff in that case.) +- If hashes from two different types of artifacts are given, results + are unspecified. Garbage in, garbage out, and all that. +- For file diffs, add the file name(s) to the response payload. diff --git a/www/json-api/api-dir.md b/www/json-api/api-dir.md new file mode 100644 index 0000000..0e69394 --- /dev/null +++ b/www/json-api/api-dir.md @@ -0,0 +1,53 @@ +# JSON API: /dir +([⬑JSON API Index](index.md)) + +# Directory Listing + +**Status:** implemented 20120316 + +**Required privileges:** "o". Was "h" prior to 20120720, but the HTML +version of /dir changed permissions and this API was modified to match +it. + +**Request:** `/json/dir` + +Options: + +- `checkin=commit` (use "tip" for the latest). If checkin is not set + then all files from all versions of the tree are returned (but only + once per file - not with complete version info for each file in all + branches).\ + CLI: `--checkin|-ci CHECKIN` +- `name=subdirectory` name. To fetch the root directory, don't pass this + option, or use an empty value or "/".\ + CLI: use `--name|-n NAME` or pass it as the first argument after + the `dir` subcommand. + +**Response payload example:** + +```json +{ + "name":"/", /* dir name */ + "uuid":"ac6366218035ed62254c8d458f30801273e5d4fc", + "checkin":"tip", + "entries":[{ + "name": "ajax", /* file name/dir name fragment */ + "isDir": true, /* only set for directories */ + /* The following properties are ONLY set if + the 'checkin' parameter is provided. + */ + "uuid": "..." /*only for files, not dirs*/, + "size": number, + "timestamp": number + },...] +} +``` + +The checkin property is only set if the request includes it. The +entry-specific uuid and size properties (e.g. `entries[0].uuid`) are +only set if the checkin request property is set and they refer to the +latest version of that file for the given checkin. The `isDir` property is +only set on directory entries. + +This command does not recurse into subdirectories, though it "should be" +simple enough to add the option to do so. diff --git a/www/json-api/api-finfo.md b/www/json-api/api-finfo.md new file mode 100644 index 0000000..e6e757f --- /dev/null +++ b/www/json-api/api-finfo.md @@ -0,0 +1,53 @@ +# JSON API: /finfo +([⬑JSON API Index](index.md)) + +# File Information + +**Status:** implemented 2012-something, but output structure is likely +to change. + +**Required privileges:** "o" + +**Request:** `/json/finfo?name=path/to/file` + +Options: + +- `name=string`. Required. Use the absolute name of the file, including + any directory parts, and without a leading slash. e.g. + `"path/to/my.c"`.\ + CLI mode: `--name` or positional argument. +- `checkin=string`. Only return info related to this specific checkin, + as opposed to listing all checkins. If set, neither "before" nor + "after" have any effect.\ + CLI mode: `--checkin|-ci` +- `before=DATETIME` only lists checkins from on or before this time.\ + CLI mode: `--before|-b` +- `after=DATETIME` only lists checkins from on or after this time. + Using this option swaps the sort order, to provide reasonable + behaviour in conjunction with the limit option.\ + Only one of "before" and "after" may be specified, and if both are + specified then which one takes precedence is unspecified.\ + CLI mode: `--after|-a` +- `limit=integer` limits the output to (at most) the given number of + associated checkins.\ + CLI mode: `--limit|-n` + +**Response payload example (very likely to change!):** + +```json +{ +"name":"ajax/i-test/rhino-shell.js", +"checkins":[{ + "checkin":"6b7ddfefbfb871f793378d8f276fe829106ca49b", + "uuid":"2b735676d55e3d06d670ffbc643e5d3f748b95ea", + "timestamp":1329514170, + "user":"viriketo", + "comment":"<...snip...>", + "size":6293, + "state":"added|modified|removed" + },…] +} +``` + +**FIXME:** there is a semantic discrepancy between `/json/artifact`'s +`payload.checkins[N].uuid` and this command's. diff --git a/www/json-api/api-misc.md b/www/json-api/api-misc.md new file mode 100644 index 0000000..6bc788a --- /dev/null +++ b/www/json-api/api-misc.md @@ -0,0 +1,92 @@ +# JSON API: Misc. APIs +([⬑JSON API Index](index.md)) + +Some operations simply don't fit into a specific category (well, none +except "misc")... + +Jump to: + +* [Global State ("g")](#g) +* [Rebuild Repository](#rebuild) +* [Result Code Descriptions](#result-codes) + +--- + +<a id="g"></a> +# Global State ("g") + +Fossil's internal state is maintained in a global object called "g". And +thus this command is named "g"... + +**Status:** implemented 20111009 + +**Required permissions:** "a" or "s" + +**Request:** `/json/g` + +**Response payload example:** this is a debugging-only request, and has +no stable response payload structure. The result object contains all +kinds of details gleaned from the fossil environment. + +Easter egg: this output can be added to ANY response by passing the +`debugFossilG` boolean in the POST envelope or GET parameters, or as the +`--json-debug-g` flag in CLI mode. This requires admin or setup +privileges, though, and it is silently ignored for other users. + + +<a id="rebuild"></a> +# Rebuild Repository + +This request does the same as the "fossil rebuild" command, rebuilding +the repo-internal structure. This is often required after upgrading the +fossil binary on a system. There "are very probably" cases where calling +this over HTTP will not work (e.g. if the user table has changed enough +that the access rights cannot be validated without a rebuild, i.e. a +chicken/egg scenario). Another consideration is that this request can +take a long time to run - rebuilding the fossil repo on my laptop takes +about 21 seconds, which is likely longer than the timeout set on an XHR +request, meaning that the rebuild transaction will fail. It can safely +be run in CLI mode, where timeouts are not normally a concern. As a +preliminary benchmark: rebuilding the fossil repo (as of late 2011) +takes just over 21 seconds on my 32-bit 1.6GHz netbook. That said, most +repos are much smaller and rebuild in under a few seconds. + +**Status:** implemented 20110929 + +**Required privileges:** "a" + +**Request:** `/json/rebuild` + +Requires admin access rights. + +**Response payload:** none (response envelope `resultCode` reports failure). +Potential TODO: return the amount of time it took to rebuild. + + +<a id="result-codes"></a> +# Result Code Descriptions + +This request returns the full list of result codes documented for +Fossil's JSON API. Each result code is returned as an object containing +metadata about the result code. + +**Status:** implemented 20111006 + +**Required permissions:** none + +**Request:** `/json/resultCodes` + +**Response payload example:** + +```json +[{ + "resultCode":"FOSSIL-1000", + "cSymbol":"FSL_JSON_E_GENERIC", + "number":1000, + "description":"Generic error" + }, + … many more objects with the same structure … +] +``` + + diff --git a/www/json-api/api-query.md b/www/json-api/api-query.md new file mode 100644 index 0000000..e246625 --- /dev/null +++ b/www/json-api/api-query.md @@ -0,0 +1,85 @@ +# JSON API: /query +([⬑JSON API Index](index.md)) + +# SQL Query + +**Status:** implemented 20111008 + +**Required privileges:** "a" or "s" + +**Request:** `/json/query` + +Potential FIXME: restrict this to queries which return results, as opposed +to those which may modify data. + +Options: + +- `sql=string` The SQL code to run. It is expected that it be a SELECT + statement, but that is not enforced. This parameter may be set as a + POST.payload property, as the POST.payload itself, GET, or as a + positional parameter coming after the command name (CLI and HTTP + modes, though the escaping would be unsightly in HTTP mode). +- `format=string` (default="o"). "o" specifies that each result row + should be in the form of key/value pairs (o=object). "a" means each + row should be an array of values. + +**Example request:** + +POST to: `/json/query` + +```json +{ +"authToken": "...", +"payload": { + "sql": "SELECT * FROM reportfmt", + "format": "o" + } +} +``` + +**Response payload example:** (assuming the above example) + + +``` +{ +"columns":[ + "rn", + "owner", + "title", + "mtime", + "cols", + "sqlcode" + ], + "rows":[ + { + "rn":1, + "owner":"drh", + "title":"All Tickets", + "mtime":1303870798, + }, + … + ] +} +``` + +The column names are provided in a separate field because their order +is guaranteed to match the order of the query columns, whereas object +key/value pairs might get reordered (typically sorted by key) when +travelling through different JSON implementations. In this manner, +clients can e.g. be sure to render the columns in the proper +(query-specified) order. + +When in "array" mode the "rows" results will be an array of arrays. For +example, the above "rows" property would instead look like: + +`[ [1, "drh", "All Tickets", 1303870798, … ], … ]` + +Note the column *names* are never *guaranteed* to be exactly as they +appear in the SQL *unless* they are qualified with an AS, e.g. `SELECT +foo AS foo...`. When generating reports which need fixed column names, it +is highly recommended to use an AS qualifier for every column, even if +they use the same name as the column. This is the only way to guarantee +that the result column names will be stable. (FYI: that behaviour comes +from sqlite3, not the JSON bits, and this behaviour *has* been known to +change between sqlite3 versions (so this is not just an idle threat of +*potential* future incompatibility).) diff --git a/www/json-api/api-settings.md b/www/json-api/api-settings.md new file mode 100644 index 0000000..5acde97 --- /dev/null +++ b/www/json-api/api-settings.md @@ -0,0 +1,127 @@ +# JSON API: /settings +([⬑JSON API Index](index.md)) + +Jump to: + +* [Fetch Settings](#get) +* [Set Settings](#set) + +--- + +<a id="get"></a> +# Fetch Settings + +**Status:** Implemented 20230120 + +**Required permissions:** "o" + +**Request:** `/json/settings/get[?version=X]` + +**Response payload example:** + +```json +{ + "access-log":{ + "versionable":false, + "sensitive":false, + "defaultValue":"off", + "valueSource":null, + "value":null + }, +... + "binary-glob":{ + "versionable":true, + "sensitive":false, + "defaultValue":null, + "valueSource":"versioned", + "value":"*.gif\n*.ico\n*.jpg\n*.odp\n*.dia\n*.pdf\n*.png\n*.wav..." + }, +... + "web-browser":{ + "versionable":false, + "sensitive":true, + "defaultValue":null, + "valueSource":"repo", + "value":"firefox" + } +} +``` + +Each key in the payload is the name of a fossil-recognized setting, +modeled as an object. The keys of that are: + + +- `defaultValue`: the setting's default value, or `null` if no default + is defined. +- `value`: The current value of that setting. +- `valueSource`: one of (`"repo"`, `"checkout"`, `"versioned"`, or + `null`), specifying the data source where the setting was found. The + settings sources are checked in this order and the first one found + is the result: + - If `version=X` is provided, check-in `X` is searched for a + versionable-settings file. If found, its value is used and + `valueSource` will be `"versioned"`. If `X` is not a checkin, an + error response is produced with code `FOSSIL-3006`. + - If a versionable setting is found in the current checkout, its + value is used and `valueSource` will be `"versioned"` + - If the setting is found in checkout database's `vvar` table, its + value is used and `valueSource` will be `"checkout"`. + - If the setting is found in repository's `config` table, its + value is used and `valueSource` will be `"repo"`. + - If no value is found, `null` is used for both the `value` and + `valueSource` results. Note that _global settings are never + checked_ because they can leak information which have nothing + specifically to do with the given repository. +- `sensitive`: a value which fossil has flagged as sensitive can only + be fetched by a Setup user. For other users, they will always have + a `value` and `valueSource` of `null`. +- `versionable`: `true` if the setting is tagged as versionable, else + `false`. + +Note that settings are internally stored as strings, even if they're +semantically treated as numbers. The way settings are stored and +handled does not give us enough information to recognize their exact +data type here so they are passed on as-is. + + +<a id="set"></a> +# Set Settings + +**Status:** Implemented 20230120 + +**Required permissions:** "s" + +**Request:** `/json/settings/set` + +This call requires that the input payload be an object containing a +mapping of fossil-known configuration keys (case-sensitive) to +values. For example: + +```json +{ + "editor": "emacs", + "admin-log": true, + "auto-captcha": false +} +``` + +It iterates through each property, which must have a data type of +`null`, boolean, number, or string. A value of `null` _unsets_ +(deletes) the setting. Boolean values are stored as integer 0 +or 1. All other types are stored as-is. It only updates the +`repository.config` database and never updates a checkout or global +config database, nor is it capable of updating versioned settings +(^Updating versioned settings requires creating a full check-in.). + +It has no result payload but this may be changed in the future it +practice shows that it should return something specific. + +Error responses include: + +- `FOSSIL-2002`: called without "setup" permissions. +- `FOSSIL-3002`: called without a payload object. +- `FOSSIL-3001`: passed an unknown config option. +- `FOSSIL-3000`: a value has an unsupported data type. + +If an error is triggered, any settings made by this call up until that +point are discarded. diff --git a/www/json-api/api-stat.md b/www/json-api/api-stat.md new file mode 100644 index 0000000..e4cd902 --- /dev/null +++ b/www/json-api/api-stat.md @@ -0,0 +1,59 @@ +# JSON API: /stat +([⬑JSON API Index](index.md)) + +# Repository Stats + +**Status:** implemented + +**Required privileges:** "o" + +**Request:** `/json/stat` + +**Response payload example:** (fields marked with `*` are omitted in +"brief" mode) + +```json +{ +"projectName":"Fossil", +"projectDescription":"Fossil SCM", /* added 20120217 */ +"repositorySize":24464384, +* "blobCount":13612, +* "deltaCount":9348, +* "uncompressedArtifactSize":589205834, +* "averageArtifactSize":43292, +* "maxArtifactSize":4620758, +* "compressionRatio":"24:1", +* "checkinCount":3150, +* "fileCount":456, +* "wikiPageCount":23, +* "ticketCount":940, +"ageDays":1512, +"ageYears":4.139744, +"projectCode":"25d3a4b83202c0d616a5ed17334f180dac4434dc", +"compiler":"gcc-4.1.2 20080704 (Red Hat 4.1.2-50)", +"sqlite":{ + "version":"2011-09-04 01:27:00 [6b657ae750] (3.7.8)", + "pageCount":23891, + "pageSize":1024, + "freeList":2705, + "encoding":"UTF-8", + "journalMode":"delete" +} +} +``` + +**Options:** + +- "Full detail" mode:\ + **HTTP/payload parameter:** full=bool\ + **CLI MODE:** -f|--full with no value.\ + If true then all properties are included, else certain properties + are omitted from the payload (because they take a relatively long + time to calculate).\ + **TODO:** rename this to verbose, for consistency.\ + **Default=false**. *This is in contrast to the HTML interface*, + which defaults to full detail mode. Testing shows stat to have a + relatively high per-call cost/run time, so it defaults + to "brief" mode by default. Full-detail mode can, on slow hardware, + take half a minute to respond, whereas non-full mode takes well + under one second. diff --git a/www/json-api/api-tag.md b/www/json-api/api-tag.md new file mode 100644 index 0000000..a9faede --- /dev/null +++ b/www/json-api/api-tag.md @@ -0,0 +1,229 @@ +# JSON API: /tag +([⬑JSON API Index](index.md)) + +Jump to: + +* [Add](#add) +* [Cancel](#cancel) +* [Find](#find) +* [List](#list) + +--- + +<a id="add"></a> +# Add Tag + +**Status:** implemented 20111006 + +**Required permissions:** "i" + +**Request:** `/json/tag/add[/name[/checkin[/value]]]` + +**Request options:** + +- `name=string` The tag name. +- `checkin=string` The checkin to tag. May be a symbolic branch name. +- raw=bool (=false) If true, then the name is set as it is provided by + the client, else it gets "sym-" prefixed to it. Do not use this + unless you really know what you're doing. +- `value=string` (=null) An optional value. While tags *may* have values + in fossil, it is unusual for them to have a value. (This probably + has some interesting uses in custom UIs.) +- `propagate=bool` (=false) Sets the tag to propagate to all descendants + of the given checkin. + +In CLI modes, the name, checkin, and value parameters may optionally be +supplied as positional parameters (in that order, after the command +name). In HTTP mode they may optionally be the 4th-6th path elements or +specified via GET/`POST.envelope` parameters. + +**Response payload example:** + +```json +{ +"name":"my-tag", +"value":"abc", +"propagate":true, +"raw":false, +"appliedTo":"626ab2f3743543122cc11bc082a0603d2b5b2b1b" +} +``` + +The `appliedTo` property is the hash of the check-in to which the tag was +applied. This is the "resolved" version of the check-in name provided by +the client. + +<a id="cancel"></a> +# Cancel Tag + +**Status:** implemented 20111006 + +**Required permissions:** "i" + +**Request:** `/json/tag/cancel[/name[/checkin]]` + +**Request options:** + +- `name=string` The tag name. May optionally be provided as the 4th path + element. +- `checkin=string` The checkin to untag. May be a symbolic branch name. + May optionally be provided as the 5th path element. + +In CLI modes, the name and checkin parameters may optionally be supplied +as positional parameters (in that order, after the command name) or +using the `-name NAME` and `-checkin NAME` options. In HTTP mode they may +optionally be the 4th and 5th path elements. + +**Response payload:** none (resultCode indicates failure) + + +<a id="find"></a> +# Find Tag + +Fetches information about artifacts having a particular tag. + +Achtung: the output of this response is based on the HTML-mode +implementation, but it's not yet certain that it's exactly what we want +for JSON mode. The request options and response format may change. + +**Status:** implemented 20111006 + +**Required permissions:** "o" + +**Request:** `/json/tag/find[/tagName]` + +The response format differs somewhat depending on the options: + +- `name=string` The tag name to search for. Can optionally be the 3rd + path element. +- `limit=int` (defalt=0) Limits the number of results (0=no limit). + Since they are ordered from oldest to newest, the newest N results + will be returned. +- `type=string` (default=`*`) Searches only for the given type of + artifact (using fossil's conventional type naming: ci, e, t, w.) +- `raw=bool` (=false) If enabled, the response is an array of hashes + of the requested artifact type; otherwise, + it is an array of higher-level objects. If this is + true, the "name" property is interpreted as-is. If it is false, the + name is automatically prepended with "sym-" (meaning a branch). + (FIXME: the current semantics are confusing and hard to remember. + Re-do them.) + +**Response payload example, in RAW mode: (expect this format to change +at some point!)** + +```json +{ +"name":"sym-trunk" +"raw":true, +"type":"*", +"limit":2, +"artifacts":[ + "a28c83647dfa805f05f3204a7e146eb1f0d90505", + "dbda8d6ce9ddf01a16f6c81db2883f546298efb7" + ] +} +``` + +Very likely todo: return more information with that (at least the +artifact type and timestamp). Once the `/json/artifact` family of bits is +finished we could use that to return artifact-type-dependent values +here. + +**Response payload example, in non-raw mode:** + +``` +{ +"name":"trunk", +"raw":false, +"type":"*", +"limit":1, +"artifacts":[{ + "uuid":"4b0f813b8c59ac8f7fbbe33c0a369acc65407841", + "timestamp":1317833899, + "comment":"fixed [fc825dcf52]", + "user":"ron", + "eventType":"checkin" + }] +} +``` + +<a id="list"></a> +# List Tags + +**Status:** implemented 20111006 + +**Required permissions:** "o" + +**Request:** `/json/tag/list[/checkinName]` + +Potential fixme: we probably have too many different response formats +here. We should probably break this into multiple subcommands. + +The response format differs somewhat depending on the options: + +- `checkin=string` Lists the tags only for the given CHECKIN (can be a + branch name). If set, includeTickets is ignored (meaningless in this + combination). This option can be set as either a GET/`POST.payload` + option, as the last element of the request path, e.g. + `/json/tag/list/MYBRANCH` *or* with `POST.payload` set to a string + value. +- `raw=bool` (default=false) uses "raw" tag names +- `includeTickets=bool` (default=false) Determines whether `tkt-` tags + are included. There is one of these for each ticket, so there can be + many of them (over 900 in the main fossil repo as of this writing). + +**Response format when raw=false and no checkin is specified:** + +```json +{ +"raw":false, +"includeTickets":false, +"tags":[ + "bgcolor", + "branch", + "closed", + …all tag names... + ] +} +``` + +Enabling the `raw` option will leave the internal `sym-` prefix on tags +which have them but will not change the structure. If `includeTickets` is +true then `tkt-` entries (possibly very many!) will be included in the +output, else they are elided. + +**General notes:** + +The `tags` property will be null if there are no tags, every non-empty +repo has at least one tag (for the trunk branch). + +**Response format when raw=false and checkin is specified:** + +```json +{ +"raw":false, +"tags":{ + "json":null, + "json-multitag-test":null + } +} +``` + +The `null`s there are the tag values (most tags don't have values). + +If `raw=true` then the tags property changes slightly: + +```json +{ +"raw":true, +"tags":{ + "branch":"json", + "sym-json":null, + "sym-json-multitag-test":null + } +} +``` + +TODO?: change the tag values to objects in the form +`{value:..., tipUuid:string, propagating:bool}`. diff --git a/www/json-api/api-ticket.md b/www/json-api/api-ticket.md new file mode 100644 index 0000000..1d02153 --- /dev/null +++ b/www/json-api/api-ticket.md @@ -0,0 +1,136 @@ +# JSON API: Tickets +([⬑JSON API Index](index.md)) + +Jump to: + +* [Ticket Reports](#reports) + * [Fetch a Report](#report-get) + * [List Reports](#report-list) + * [Run a Report](#report-run) + + +--- + +# Tickets + +This API is incomplete. It is missing at least the following features: + +- Content for a given ticket ID +- History for a given ticket ID? +- An option to enable/disable the generation of hyperlinks, as links + won't be useful in most non-browser clients. + + +<a id="reports"></a> +# Ticket Reports + +<a id="report-get"></a> +## Fetch a Report + +**Status:** implemented 20111008 + +**Required privileges:** "t" (the thinking being that only those +permitted to create reports should be able to read their SQL code) + +**Request:** `/json/report/get[/REPORT_NUMBER]` + +**Request options:** + +- `report=number` The report number to fetch.\ + CLI: `-report|-r` \ + (Design note: `--number/-n` was not used because that parameter has a + different meaning (limit response count) in several commands.) May + optionally be defined via the 4th GET path element or CLI arg. + +**Response payload example:** + +```json +{ +"report":1, +"owner":"drh", +"title":"All Tickets", +"timestamp":"112443570187200", +"columns":"#ffffff Key:\r\n#f2dcdc Active\r\n...", +"sqlCode":"..." +} +``` + +<a id="report-list"></a> +## List Reports + +**Status:** implemented 20111008 + +**Required privileges:** "r" or "n" + +**Request:** `/json/report/list` + +**Response payload example:** + +```json +[ + { + "report":1, + "title":"All Tickets", + "owner":"drh" + }, + … +] +``` + +<a id="report-run"></a> +## Run a Report + +**Status:** implemented 20111008 + +**Required privileges:** "r" or "n" + +**Request:** `/json/report/run[/REPORT_NUMBER]` + +Request options: + +- `report|-r=int` Specifies which report to run. May optionally be + supplied as the 4th CLI arg or URL path element. +- `format|-f=string` (default="o") Specifies "array" or "object" output + format. + +**Response payload example:** + +```json +{ + "report":1, + "title":"All Tickets", + "sqlcode": "only set if requester has 't' privileges.", + "columnNames":[ … list of column names … ], + "tickets":[ + { + "bgcolor":"#cfe8bd", + "#":"fc825dcf52", + "timestamp":"112443570187200", + "type":"Code_Defect", + "status":"Fixed", + "subsystem":null, + "title":"\"config pull all\" asks to approve ssl cert" + }, + … + ] +} +``` + +Note that the column names of ticket reports are determined by the +reports themselves, and not C code. That means that we cannot return a +standard set of column names here. Fossil requires certain column naming +conventions for purposes of styling the HTML interface, e.g. the "\#" +column is always the uuid of the record and "bgcolor" (note the +different casing than bgColor used throughout the rest of this API!) has +a specific meaning to the HTML report browser. Fossil also allows the +tickets to be extended with client-specified fields, so we cannot +generically make these results fit into the API-wide naming scheme. Full +details are here: + +[](/doc/trunk/www/custom_ticket.wiki) + +and: + +[](/rptsql?rn=1) + +(That one may require non-default permission.) diff --git a/www/json-api/api-timeline.md b/www/json-api/api-timeline.md new file mode 100644 index 0000000..b0ff196 --- /dev/null +++ b/www/json-api/api-timeline.md @@ -0,0 +1,214 @@ +# JSON API: /timeline +([⬑JSON API Index](index.md)) + +Jump to: + +* [Introduction](#intro) +* [Branch Timeline](#branch) +* [Technote (formerly Event) Timeline](#technote) +* [Ticket Timeline](#ticket) +* [Wiki Timeline](#wiki) + +--- + +<a id="intro"></a> +# Introduction + +These requests return overview-level information about various types of +changes. The response payload differs for each artifact type, and the +current structures are almost certainly not "final" (e.g. we are still +undecided on how/whether to handle artifact links within commit messages +and whatnot). + +By default the entries are returned in chronological order from newest +to oldest, but some options might change that. + +FIXME (20120623): we have some inconsistent `type` vs. `eventType` in +the result sets. `type` is the current preferred choice (and it seems +unlikely that `eventType` is actually used in any client code). We +don't actually need either one (but a use for `type` is easily +envisioned), and we may get rid of both. + +**Common request options (via CLI, GET or POST.payload):** + +- `limit=integer` limits the number of entries in the response. Default + is unspecified (but is "quite possibly 20 or so"). A limit value of + 0 disables any limit, fetching all entries (which can take a really + long time and might overload clients which have very limited + memory).\ + CLI mode: `--limit|-n #` +- `after="YYYY-MM-DD[ HH:mm:ss]"` limits the search to items on or + after the given date string. Reverses the normal timeline sort + order. Alias: "a". Only one of "after" or "before" can be used, and + if both are specified then which one takes precedence is + unspecified.\ + CLI mode: `--after|-a "DATE[ TIME]"` +- `before="YYYY-MM-DD[ HH:mm:ss]"` limits the search to items on or + before the given date string.\ + CLI mode: `--before|-b "DATE[ TIME]" +- TODOs, still to be ported from the HTML-mode timeline: + - circa=DATETIME + - tag=string + - related=tag name + - string=search string + +<a id="branch"></a> +# Branch Timeline + +**Status:** partially implemented but undocumented because the utility +of the current impl is under question. It also doesn't understand most +of the common timeline options. + +<a id="checkin"></a> +# Checkin Timeline + +**Status:** implemented 201109xx + +**Required privileges:** "o" + +**Request:** `/json/timeline/checkin` + +**Response payload example:** + +```json +{ +"limit": number, /* if not set, all records are returned */ +"timeline":[{ + "uuid":"be700e84336941ef1bcd08d676310b75b9070f43", + "timestamp":1317094090, + "comment":"Added /json/timeline/ci showFiles to ajax test page.", + "user":"stephan", + "isLeaf":true, + "bgColor":null, /* not quite sure why this is null? */ + "type":"ci", + "parents": ["primary parent hash", "...other parent hashes"], + "tags":["json"], + "files":[{ + "name":"ajax/index.html", + "uuid":"9f00773a94cea6191dc3289aa24c0811b6d0d8fe", + "parent":"50e337c33c27529e08a7037a8679fb84b976ad0b", + "state":"modified" + }] + },...] +} +``` + +(Achtung: the `parents` property was called `prevUuid` prior to 20120316.) + +The `parents` property lists the checkins which were parents of this +commit. The first entry in the array is the "primary parent" - the one +which was not involved in a merge with the child. + +**Request options:** + +- `files=bool` toggles the addition of a "files" array property which + contains objects describing the files changed by the commit, + including their hash, previous hash, and state change type + (modified, added, or removed). ([“uuid” here means hash][uvh])\ + CLI mode: `--show-files|-f` +- `tag|branch=string` selects only entries with the given tag or "close + to" the given branch. Only one of these may be specified and if both + are specified, which one takes precedence is unspecified. If the + given tag/branch does not exist, an error response is generated. The + difference between the two is subtle - tag filters only on the given + tag (analog to the HTML interface's "r" option) whereas branch can + also return entries from other branches which were merged into the + requested branch (analog to the HTML interface's "b" option). If one + of these is specified, the response payload will contain a "tag" + *or* "branch" property with the tag/branch name given by the client. + +<a id="technote"></a> +# Technote (formerly Event) Timeline + +**Status:** implemented 20180803 + +**Required privileges:** "j" + +**Request:** + +- `/json/timeline/technote` +- DEPRECATED: `/json/timeline/event` (technotes were formerly called `events`) + +**Response payload example:** + +```json +{ +"limit": number, /* if not set, all records are returned */ +"timeline":[{ + "name":"8d18bf27e9f9ff8b9017edd55afc35701407d418", + "uuid":"b23962c88c123924a77fd663e91af094780d920a", + "timestamp":1478376113, + "comment":"Style update due to [8d880f0bb4]", + "user":"andygoth", + "eventType":"e" + },...] +} +``` + +The `uuid` of each entry can be passed to `/json/artifact` to fetch the raw +event content. + +<a id="ticket"></a> +# Ticket Timeline + +**Status:** implemented 201109xx + +**Required privileges:** "r" or "o" + +**Request:** `/json/timeline/ticket` + +**Response payload example:** + +```json +{ + "limit": number, /* if not set, all records are returned */ + "timeline":[{ + "uuid":"5065a5da060e181da49a618f8ae5dc245215e95b", + "timestamp":1316511322, + "user":"stephan", + "eventType":"t", + "comment":"Ticket [b64435dba9] <i>How to...</i>", + "briefComment":"Ticket [b64435dba9]: 2 changes", + "ticketUuid":"b64435dba9cceb709bd54fbc5883884d73f93491" + },...] +} +``` + +**Notice that there are two [hashes][uvh] for tickets** - `uuid` is the change +hash and `ticketUuid` is the actual ticket’s hash. This is an unfortunate +discrepancy vis-a-vis the other timeline entries, which only have one +hash. We may want to swap `uuid` to mean the ticket hash and change `uuid` +to `commitHash`. + +<a id="wiki"></a> +# Wiki Timeline + +**Status:** implemented 201109xx + +**Required privileges:** "j" or "o" + +**Requests:** + +- `/json/timeline/wiki` +- `/json/wiki/timeline` (alias) + +**Response payload example:** + +```json +{ +"limit": number, /* if not set, all records are returned */ +"timeline":[{ + "uuid":"4b2026f06eb48eaf187209fcb05ba5438c3b0ef0", + "timestamp":1331351121, + "comment":"Changes to wiki page [Page3]", + "user":"stephan", + "eventType":"w" + },...] +} +``` + +The `uuid` of each entry can be passed to `/json/artifact` or +`/json/wiki/get?uuid=...` to fetch the raw page and the hash of the +parent version. + +[uvh]: ../hashes.md#uvh diff --git a/www/json-api/api-user.md b/www/json-api/api-user.md new file mode 100644 index 0000000..c4f055f --- /dev/null +++ b/www/json-api/api-user.md @@ -0,0 +1,142 @@ +# JSON API: /user +([⬑JSON API Index](index.md)) + +Jump to: + +* [Get User Info](#get) +* [List Users](#list) +* [Save User](#save) + +--- + +<a id="get"></a> +# Get User Info + +**Status:** implemented 20110927. + +**Required privileges:** "a" or "s" + +**Request:** + +- POST `/json/user/get`\ + with `POST.payload.name=USERNAME` +- `/json/user/get?name=USERNAME` + +**Response payload example:** + +```json +{ + "uid":1, + "name":"stephan", + "capabilities":"abcdefhgijkmnopqrstuvwxz", + "info":"https://wanderinghorse.net/home/stephan/", + "timestamp":1316122562 +} +``` + +(What does that timestamp field represent, anyway?) + +<a id="list"></a> +# List Users + +**Status:** implemented 20110927. + +**Required privileges:** "a" or "s" + +**Request:** `/json/user/list` + +**Response payload example:** + +```json +[ + { + "uid":1, + "name":"stephan", + "capabilities":"abcdefhgijkmnoprstuvwxz", + "info":"", + "timestamp":1316122562 + }, + ... more users... +] +``` + + +<a id="save"></a> +# Save User + +Only admin/setup users may modify accounts other than their own. + +**Status:** implemented 20111021 *but* it is missing "login group" +support, so changes do not yet propagate to other repos within a group. + +**Required privileges:** 'p' or 'a' or 's', depending on the context. + +**Request:** `/json/user/save` + +All request options must come from the `POST.payload` and/or GET/CLI +parameters (exception: "name" must come from POST.payload or CLI). +GET/CLI parameters take precedence over those in `POST.payload`, the +intention being to use an input file as a template and overriding the +template's defaults via the CLI. The options include: + +- `name=string` Specifies the user name to change. When changing a + user's name, the current uid and the new name must be specified.\ + **Achtung:** due to fossil-internal ambiguity in the handling of the + "name" parameter, this parameter must come from the `POST.payload` + data or it will not be recognized. In CLI mode it may be specified + with the `--name` flag. +- `uid=int` Specifies the uid to change. At least one of uid or name are + required. A uid of -1 means to create a new user, in which case the + name must be provided. +- `password=string` Optionally changes the user's password. When + renaming existing or creating new users, be sure to always provide a + new password because any old password hash is invalidated by the + name change. +- `info=string` Optionally changes the user's info field. +- `capabilities=string` Optionally changes the user's capabilities + field. +- `forceLogout=bool` (=false, or true when renaming) Optionally clears + any current login info for the current user, which will invalidate + any active session. Requires 'a' or 's' privileges. Intended to be + used when disabling a user account, to ensure that any open session + is invalidated. When a user is renamed this option is implied (and + cannot be disabled) because renaming invalidates any currently + stored auth token (because the old name is part of the hash + equation). + +Fields which are not provided in the request will not be modified. +Non-admin/setup users cannot edit other users and may only change their +own data if they have the 'p' (password) privilege. + +As of 20120217, users who do not have the setup privilege may neither +change the setup privilege for any user nor edit another user who has +that privilege. That is, only users with setup access may propagate or +remove setup status and accounts with the setup privilege may only be +edited by themselves and other setup users. + +**Response payload:** Same as user/get, using the new/saved state of the +modified user. + +Example usage from the command line: + +```console +$ fossil json user save --name drh --password sqlite3 \ + --capabilities "as" --info "DRH" +$ fossil json user save --uid 1 --name richard \ + --password fossil \ + --info "Previously known as drh" +``` + +**Warnings:** + +- When creating a new user or renaming a user, if no (new) password is + specified in the save request then the user will not be able to log + in because the previous password (for existing users) is hashed + against the old name. +- Renaming a user invalidates any active login token because his old + name is a part of the hash. i.e. the user must log back in with the + new name after being renamed. + +**TODOs:** + +- Login group support. diff --git a/www/json-api/api-version.md b/www/json-api/api-version.md new file mode 100644 index 0000000..ab321b6 --- /dev/null +++ b/www/json-api/api-version.md @@ -0,0 +1,32 @@ +# JSON API: /version +([⬑JSON API Index](index.md)) + +# Version (a.k.a. HAI) + +**Status:** implemented + +**Required privileges:** none + +**Requests:** + +- `/json/version` +- `/json/HAI` (alias borrowed from LOLCATZ jargon) + +**Response payload example:** + +```json +{ +"manifestUuid":"20ff808f9809541d2eca6c49a17d5cbd16e1b93f", +"manifestVersion":"[20ff808f98]", +"manifestDate":"2011-09-09 16:49:23", +"manifestYear":"2011", +"releaseVersion":"1.19", +"releaseVersionNumber":119, +"jsonApiVersion": "YYYYMMDD" // added 20120409 +} +``` + +Those particular payload fields were chosen only because they're defined +in `VERSION.h`. We may want to add other information, but nothing comes to +mind at this time. + diff --git a/www/json-api/api-wiki.md b/www/json-api/api-wiki.md new file mode 100644 index 0000000..7e1899f --- /dev/null +++ b/www/json-api/api-wiki.md @@ -0,0 +1,277 @@ +# JSON API: /wiki +([⬑JSON API Index](index.md)) + +Jump to: + +* [Page List](#list) +* [Fetch a Page](#get) +* [Create or Save Page](#create-save) +* [Wiki Diffs](#diffs) +* [Preview](#preview) +* [Notes and TODOs](#todo) + +--- + +<a id="list"></a> +# Page List + +Returns a list of all pages, not including their content (which can be +arbitrarily large). + +**Status:** implemented 201109xx + +**Required privileges:** "j" or "o" + +**Request:** `/json/wiki/list` + +**Options:** + +- `bool verbose` (=false) Changes the results to return much more + information. Added 20120219. +- `glob=wildcard` string (default=`*`). If set, only page names + matching the given wildcard are returned. Added 20120325.\ + CLI: `--glob|-g STRING` +- `like=SQL LIKE string` (default=`%`). If set, only page names matching + the given SQL LIKE string are returned. Note that this match is + case-INsensitive. If both glob and like are given then only one will + work and which one takes precedence is unspecified. Added 20120325.\ + CLI: `--like|-l STRING` +- `invert=bool` (default=false). If set to a true value, the glob/like + filter has a reverse meaning (pages *not* matching the wildcard are + returned). Added 20120329.\ + CLI: `-i/--invert` + +**Response payload: format depends on "verbose" option** + +Non-verbose mode: + +```json +["PageName1",..."PageNameN"] +``` + +In verbose mode: + +```json +[{ +"name":"Apache On Windows XP", +"uuid":"a7e68df71b95d35321b9d9aeec3c8068f991926c", +"user":"jeffrimko", +"timestamp":1310227825, +"size":793 /* in bytes */ +},...] +``` + +The verbose-mode output is the same as the [`/json/wiki/get`](#get) output, but +without the content. The size property of each reflects the *byte* +length of the raw (non-HTMLized) page content. + +**Potential TODOs:** + +- Allow specifying (in the request) a list/array of names, as opposed + to listing all pages. The page count is rarely very high, though, so + an "overload" is very unlikely. (i have one wiki with about 47 pages + in it.) + +<a id="get"></a> +# Fetch a Page + +Fetches a single wiki page, including content and significant metadata. + +**Status:** implemented 20110922, but response format may change. + +**Required privileges:** "j" or "o" + +**Request:** + +- GET: `/json/wiki/get?name=PageName` +- GET: `/json/wiki/get/PageName` +- POST: `/json/wiki/get` with page name as `POST.payload` or + `POST.payload.name`. + +**Response payload example:** + +```json +{ +"name": "Fossil", +"uuid": "...hex string...", +"parent": "uuid of parent (not set for first version of page)", +"user": "anonymous", +"timestamp": 1286143975, +"size": 1906, /* In bytes, not UTF8 characters! + Affected by format option! */ +"content": "..." +} +``` + +**FIXME:** it's missing the mimetype (that support was added to fossil +after this was implemented). + +If given no page to load, or if asked to get a page which does not +exist, an error response is generated (a usage- or resource-not-found +error, respectively). + +**Options (via CLI/GET/`POST.payload`):** + +- `name=string`. The page to fetch. The latest version is fetched +unless the uuid paramter is supplied (in which case name is ignored). \ +CLI: `--name|-n string`\ +Optionally, the name may be the 4th positional argument/request path element. +- `uuid=string`. Fetches a specific version. The name parameter is +ignored when this is specified.\ +CLI: `--uid|-u string` +- `format=string ("raw"|"html"|"none")` (default="raw"). Specifies the +format of the "content" payload value.\ +CLI: `--format|-f string` \ +The "none" format means to return no content. In that case the size +refers to the same size as the "raw" format. + +**TODOs:** + +- Support passing an array of names in the request (and change + response to return an array). + +<a id="create-save"></a> +# Create or Save Page + +**Status:** implemented 20110922. + +**Required privileges:** "k" (save) or "f" (create) + +**Request:** + +- `/json/wiki/create` +- `/json/wiki/save` + +These work only in HTTP mode, not CLI mode. (FIXME: now that we can +simulate POST from a file, these could be used in CLI mode.) + +The semantic differences between save and create are: + +- Save will fail if the page doesn't already exist whereas create will + fail if it does. The createIfNotExists option (described below) can + be used to create new pages using the save operation. +- The content property is optional for the create request, whereas it + is required to be a string for save requests (but it *may* be an + empty string). This requirement for save is *almost* arbitrary, and + is intended to prevent accidental erasing of existing page content + via API misuse. + +**Response payload example:** + +The same as for [`/json/wiki/get`](#get) but the page content is not +included in the response (only the metadata). + +**Request options** (via GET or `POST.payload` object): + +- `name=string` specifies the page name. +- `content=string` is the body text.\ + Content is required for save (unless `createIfNotExists` is true *and* + the page does not exist), optional for create. It *may* be an empty + string. +- `mimetype=string` specifies the mimetype for the body, noting any any + unrecognized/unsupported mimetype is silently treated as + `text/x-fossil-wiki`. +- Save (not create) supports a `createIfNotExists` boolean option which + makes it a functional superset of the create/save features. i.e. it + will create if needed, else it will update. If createIfNotExists is + false (the default) then save will fail if given a page name which + does not refer to an existing page. +- **TODO:** add `commitMessage` string property. The fossil internals + don't have a way to do this at the moment (they can as of late 2019). + Since fossil wiki commits have always had the same default commit message, this is not a + high-priority addition. See:\ + [](/doc/trunk/www/fileformat.wiki#wikichng) +- **Potential TODO:** we *could* optionally also support + multi-page saving using an array of pages in the request payload:\ + `[… page objects … ]` + + + +<a id="diffs"></a> +# Wiki Diffs + +**Status:** implemented 20120304 + +**Required privileges:** "h" + +**Request:** + +- `/json/wiki/diff[/version1_UUID/version2_UUID]` + +**Response payload example:** + +```json +{ + "v1":"e32ccdcda59e930c77c1e01cebace5d71253f621", + "v2":"e15992f475760cdf3a9564d8f88cacb659ab4b07", + "diff":"@@ -1,4 +1,9 @@...<SNIP>..." +} +``` + +**Options:** + +- `v1=uuid` and `v2=uuid` specify the two versions to diff, and are + required parameters. They may optionally be specified as the two + URL/CLI parameters following the "diff" sub-command/path. + +This command does not verify that both UUIDs actually refer to the same +page name, but do verify that they refer to wiki content. + +Trivia: passing the same UUIDs to the `/json/diff` command will produce +very different results - that one diffs the manifests of the commits. + +**TODOs:** + +- Add options for changing the format of the diff, e.g. side-by-side + and enabling the HTML markup supported by the main fossil HTML GUI. +- Potentially do a name comparison to verify that the diff is against + the same page. That said, when "renaming" pages it might be useful + to diff two different pages. + +<a id="preview"></a> +# Preview + +**Status:** implemented 20120310 + +**Required privileges:** "k" (to limit its use to only those who can +edit wiki pages). This limitation is up for debate/reconsideration. + +**Request:** + +- `/json/wiki/preview` + +This command wiki-processes arbitrary text sent from the client. To help +curb potential abuse, its use is restricted to those with "k" access +rights. + +The `POST.payload` property must be either: + +1) A string containing Fossil wiki markup. + +2) An Object with a `body` property holding the text to render and a + `mimetype` property describing the wiki format: + `text/x-fossil-wiki` (the default), `text/x-markdown`, or + `text/plain`. Any unknown type is treated as `text/x-fossil-wiki`. + +The response payload is a string containing the rendered page. Whether +or not "all HTML" is allowed depends on site-level configuration +options, and that changes how the input is processed. + +Note that the links in the generated page are for the HTML interface, +and will not work as-is for arbitrary JSON clients. In order to +integrate the parsed content with JSON-based clients the HTML will +probably need to be post-processed, e.g. using jQuery to fish out the +links and re-map wiki page links to a JSON-capable page handler. + + +<a id="todo"></a> +# Notes and TODOs + +- When server-parsing the wiki content, the generated + intra-wiki/intra-site links will only be useful in the context of + the original fossil UI (or a work-alike), not arbitrary JSON + client apps. + +Potential TODOs: + +- `/wiki/history` analog to the [](/whistory) page. diff --git a/www/json-api/conventions.md b/www/json-api/conventions.md new file mode 100644 index 0000000..f3b470c --- /dev/null +++ b/www/json-api/conventions.md @@ -0,0 +1,649 @@ +# JSON API: General Conventions +([⬑JSON API Index](index.md)) + +Jump to: + +* [JSON Property Naming](#property-names) +* [HTTP GET Requests](#http-get) +* [HTTP POST Requests](#http-post) + * [POST Request Envelope](#request-envelope) +* [Request Parameter Data Types](#request-param-types) +* [Response Envelope](#response-envelope) +* [HTTP Response Headers](#http-response-header) +* [CLI vs. HTTP Mode](#cli-vs-http) +* [Simulating POSTed data](#simulating-post-data) +* [Indentation/Formatting of JSON Output](#json-indentation) +* [JSONP](#jsonp) +* [API Result Codes](#result-codes) + +--- + +<a id="property-names"></a> +# JSON Property Naming + +Since most JSON usage conventionally happens in JavaScript +environments, this API has (without an explicit decision ever having +been made) adopted the ubiquitous JavaScript convention of +`camelCaseWithStartingLower` for naming properties in JSON objects. + +<a id="http-get"></a> +# HTTP GET Requests + +Many (if not most) requests can be made via simple GET requests, e.g. we +*could* use any of the following patterns for a hypothetical JSON-format +timeline: + +- `https://..../timeline/json` +- `/timeline?format=json` +- `/timeline?json=1` +- `/timeline.json` +- `/json/timeline?...options...` + +The API settled on the `/json/...` convention, primarily because it +simplifies dispatching and argument-handling logic compared to the +`/[.../]foo.json` approach. Using `/json/...` allows us to unify that +logic for all JSON sub-commands, for both CLI and HTTP modes. + +<a id="http-post"></a> +# HTTP Post Requests + +Certain requests, mainly things like editing checkin messages and +committing new files entirely, require POST data. This is fundamentally +very simple to do - clients post plain/unencoded JSON using a common +wrapper envelope which contains the request-specific data to submit as +well as some request-independent information (like authentication data). + +<a id="request-envelope"></a> +## POST Request Envelope + +POST requests are sent to the same URL as their GET counterpart (if any, +else their own path), and are sent as plain-text/unencoded JSON wrapped +in a common request envelope with the following properties: + +- `requestId`: Optional arbitrary JSON value, not used by fossil, but + returned as-is in responses. +- `command`: Provides a secondary mechanism for specifying which JSON + command should be run. A request path of /json/foo/bar is equivalent + to a request with path=/json and command=foo/bar. Note that subpaths + do not work this way. e.g. path=/json/foo, command=bar will not + work, but path=/json, command=foo/bar will. This option is + particularly useful when generating JSON for piping in to CLI mode, + but it also has some response-dispatching uses on the client side. +- `authToken`: Authentication token. Created by a login + request. Determines what access rights the user has, and any given + request may require specific rights. In principle this is required + by any request which needs non-guest privileges, but cookie-aware + clients do not manually need to track this (it is managed as a + cookie by the agent/browser). + - Note that when accessing fossil over a local server instance + which was started with the `--localauth` flag, the `authToken` + will be ignored and need not be sent with any requests. The user + will automatically be given full privileges, as if they were + using CLI mode. +- `payload`: Command-specific parameters. Most can optionally come in + via GET parameters, but those taking complex structures expect them + to be placed here. +- `indent`: Optionally specifies indentation for the output. 0=no + indention. 1=a single TAB character for each level of + indentation. >1 means that many spaces per level. e.g. indent=7 + means to indent 7 spaces per object/array depth level. cson also + supports other flags for fine-tuning the output spacing, and adding + them to this interface might be interesting at some + point. e.g. whether or not to add a newline to the output. CLI mode + adds extra indentation by default, whereas CGI/server modes produce + unindented output by default. +- `jsonp`: Optional String (client function name). Requests which + include this will be returned with `Content-Type + application/javascript` and will be wrapped up in a function call + using the given name. e.g. if `jsonp=foo` then the result would look like:\ +`foo( {...the response envelope...} )` + +The API allows most of those (normally all but the payload) to come in +as either GET parameters or properties of the top-level POSTed request +JSON envelope, with GET taking priority over POST. (Reminder to self: we +could potentially also use values from cookies. Fossil currently only +uses 1 cookie (the login token), and I'd prefer to keep it that way.) + +POST requests without such an envelope will be rejected, generating a +Fossil/JSON error response (as opposed to an HTTP error response). GET +requests, by definition, never have an envelope. + +POSTed client requests *must* send a Content-Type header of either +`application/json` , `application/javascript`, or `text/plain`, or the +JSON-handling code will never see the POST data. The POST handler +optimistically assumes that type `text/plain` "might be JSON", since +`application/json` is a newer convention which many existing clients +do not use (as of the time these docs were written, back in 2011). + +## POST Envelope vs. `POST.payload` + + +When this document refers to POST data, it is referring to top-level +object in JSON-format POSTed input data. When we say `POST.payload` we +refer to the "payload" property of the POST data. While fossil's core +handles *form-urlencoded* POST data, if such data is sent in then +parsing it as JSON cannot succeed, which means that (at worst) the +JSON-mode bits will "not see" any POST data. Data POSTed to the JSON API +must be sent non-form-urlencoded (i.e. as plain text). + +Framework-level configuration options are always set via the top-level +POST envelope object or GET parameters. Request-specific options are set +either in `POST.payload` or GET parameters (though the former is required +in some cases). Here is an example which demonstrates the possibly +not-so-obvious difference between the two types of options (framework +vs. request-specific): + +```json +{ +"requestId":"my request", // standard envelope property (optional) +"command": "timeline/wiki", // also standard +"indent":2, // output indention is a framework-level option +"payload":{ // *anything* in the payload is request-specific + "limit":1 +} +} +``` + +When a given parameter is set in two places, e.g. GET and POST, or +POST-from-a-file and CLI parameters, which one takes precedence +depends on the concrete command handler (and may be unspecified). Most +will give precedence to CLI and GET parameters, but POSTed values are +technically preferred for non-string data because no additional "type +guessing" or string-to-whatever conversion has to be made (GET/CLI +parameters are *always* strings, even if they look like a number or +boolean). + + +<a id="request-param-types"></a> +# Request Parameter Data Types + +When parameters are sent in the form of GET or CLI arguments, they are +inherently strings. When they come in from JSON they keep their full +type (boolean, number, etc.). All parameters in this API specify what +*type* (or types) they must (or may) be. For strings, there is no +internal conversion/interpretation needed for GET- or CLI-provided +parameters, but for other types we sometimes have to convert strings to +other atomic types. This section describes how those string-to-whatever +conversions behave. + +No higher-level constructs, e.g. JSON **arrays** or **objects**, are +accepted in string form. Such parameters must be set in the POST +envelope or payload, as specified by the specific API. + +This API does not currently use any **floating-point** parameters, but +does return floating-point results in a couple of places. + +For **integer** parameters we use a conventional string-to-int algorithm +and assume base 10 (analog to atoi(3)). The API may err on the side of +usability when given technically invalid values. e.g. "123abc" will +likely be interpreted as the integer 123. No APIs currently rely on +integer parameters with more than 32 bits (signedness is call-dependent +but few, if any, use negative values). + +**Boolean** parameters are a bit schizophrenic... + +In **CLI mode**, boolean flags do not have a value, per se, and thus +require no string-to-bool conversion. e.g. +`fossil foo -aBoolOpt -non-bool-opt value`. + +Those which arrive as strings via **GET parameters** treat any of the +following as true: a string starting with a character in the set +`[1-9tT]`. All other string values are considered to be false for this +purpose. + +Those which are part of the **POST data** are normally (but not always - +it depends on the exact context) evaluated as the equivalent of +JavaScript booleans. e.g. if we have `POST.envelope.foo="f"`, and evaluate +it as a JSON boolean (as opposed to a string-to-bool conversion), the +result will be true because the underlying JSON API follows JavaScript +semantics for any-type-to-bool conversions. As long as clients always +send "proper" booleans in their POST data, the difference between +GET/CLI-provided booleans should never concern them. + +TODO: consider changing the GET-value-to-bool semantics to match the JS +semantics, for consistency (within the JSON API at least, but that might +cause inconsistencies vis-a-vis the HTML interface). + +<a id="response-envelope"></a> +# Response Envelope + +Every response comes in the form of an HTTP response or (in CLI mode) +JSON sent to stdout. The body of the response is a JSON object following +a common envelope format. The envelope has the following properties: + + +- `fossil`: Fossil server version string. This property is basically + "the official response envelope marker" - if it is set, clients can + "probably safely assume" that the object indeed came from one of the + Fossil/JSON APIs. This API never creates responses which do not + contain this property. +- `requestId`: Only set if the request contained it, and then it is + echoed back to the caller as-is. This can be used to determine + (client-side) which request a given response is coming in for + (assuming multiple asynchronous requests are pending). In practice + this generally isn’t needed because response handling tends to be + done by closures associated with the original request object (at + least in JavaScript code). In languages without closures it might + have some use. It may be any legal JSON value - it need not be + confined to a string or number. +- `resultCode`: Standardized result code string in the form + `FOSSIL-####`. Only error responses contain a `resultCode`. +- `resultText`: Possibly a descriptive string, possibly + empty. Supplements the resultCode, but can also be set on success + responses (but normally isn't). Clients must not rely on any + specific values being set here. +- `payload`: Request-specific response payload (data type/structure is + request-specific). The payload is never set for error responses, + only for success responses (and only those which actually have a + payload - not all do). +- `timestamp`: Response timestamp (GMT Unix Epoch). We use seconds + precision because I did not know at the time that Fossil actually + records millisecond precision. +- `payloadVersion`: Not initially needed, but reserved for future use + in maintaining version compatibility when the format of a given + response type's payload changes. If needed, the "first version" + value is assumed to be 0, for semantic [near-]compatibility with the + undefined value clients see when this property is not set. +- `command`: Normalized form of the command being run. It consists of + the "command" (non-argument) parts of the request path (or CLI + positional arguments), excluding the initial "/json/" part. e.g. the + "command" part of "/json/timeline/checkin?a=b" (CLI: json timeline + checkin...) is "timeline/checkin" (both in CLI and HTTP modes). +- `apiVersion`: Not yet used, but reserved for a numeric value which + represents the JSON API's version (which can be used to determine if + it has a given feature or not). This will not be implemented until + it's needed. +- `warnings`: Reserved for future use as a standard place to put + non-fatal warnings in responses. Will be an array but the warning + structure/type is not yet specified. Intended primarily as a + debugging tool, and will "probably not" become part of the public + client interface. +- `g`: Fossil administrators (those with the "a" or "s" permissions) + may set the `debugFossilG` boolean request parameter (CLI: + `--json-debug-g`) to enable this property for any given response. It + contains a good deal of the server-side internal state at the time + the response was generated, which is often useful in debuggering + problems. Trivia: it is called "g" because that's the name of + fossil's internal global state object. +- `procTimeMs`: For debugging only - generic clients must not rely on + this property. Contains the number of milliseconds the JSON command + processor needed to dispatch and process the command. TODO: move the + timer into the fossil core so that we can generically time its + responses and include the startup overhead in the time calculation. + + + +<a id="http-response-header"></a> +# HTTP Response Headers + +The Content-Type HTTP header of a response will be either +application/json, application/javascript, or text/plain, depending on +whether or not we are in JSONP mode or (failing that) the contents of +the "Accept" header sent in the request. The response type will be +text/plain if it cannot figure out what to do. The response's +Content-Type header *may* contain additional metadata, e.g. it might +look like: application/json; charset=utf-8 + +Apropos UTF-8: note that JSON is, by definition, Unicode and recommends +UTF-8 encoding (which is what we use). That means if your console cannot +handle UTF-8 then using this API in CLI mode might (depending on the +content) render garbage on your screen. + + +<a id="cli-vs-http"></a> +# CLI vs. HTTP Mode + +CLI (command-line interface) and HTTP modes (CGI and standalone server) +are consolidated in the same implementations and behave essentially +identically, with only minor exceptions. + +An HTTP path of `/json/foo` translates to the CLI command `fossil json +foo`. CLI mode takes options in the fossil-convention forms (e.g. `--foo 3` +or `-f 3`) whereas HTTP mode takes them via GET/POST data (e.g. `?foo=1`). +(Note that per long-standing fossil convention CLI parameters taking a +value do not use an equal sign before the value!) + +For example: + +- HTTP: `/json/timeline/wiki?after=2011-09-01&limit=3` +- CLI: `fossil json timeline wiki --after 2011-09-01 --limit 3` + +Some commands may only work in one mode or the other (for various +reasons). In CLI mode the user automatically has full setup/admin +access. + +In HTTP mode, request-specific options can also be specified in the +`POST.payload` data, and doing so actually has an advantage over +specifying them as URL parameters: posting JSON data retains the full +type information of the values, whereas GET-style parameters are always +strings and must be explicitly type-checked/converted (which may produce +unpredictable results when given invalid input). That said, oftentimes +it is more convenient to pass the options via URL parameters, rather +than generate the request envelope and payload required by POST +requests, and the JSON API makes some extra effort to treat GET-style +parameters type-equivalent to their POST counterparts. If a property +appears in both GET and `POST.payload`, GET-style parameters *typically* +take precedence over `POST.payload` by long-standing convention (=="PHP +does it this way by default"). + +(That is, however, subject to eventual reversal because of the +stronger type safety provided by POSTed JSON. Philosophically +speaking, though, GET *should* take precedence, in the same way that +CLI-provided options conventionally override app-configuration-level +options.) + +One notable functional difference between CLI and HTTP modes is that in +CLI mode error responses *might* be accompanied by a non-0 exit status +(they "should" always be, but there might be cases where that does not +yet happen) whereas in HTTP mode we always try to exit with code 0 to +avoid generating an HTTP 500 ("internal server error"), which could keep +the JSON response from being delivered. The JSON code only intentionally +allows an HTTP 500 when there is a serious internal error like +allocation or assertion failure. HTTP clients are expected to catch +errors by evaluating the response object, not the HTTP result code. + +<a id="simulating-post-data"></a> +# Simulating POSTed data + +We have a mechanism to feed request data to CLI mode via +files (simulating POSTed data), as demonstrated in this example: + +```console +$ cat in.json +{ "command": "timeline/wiki", "indent":2, "payload":{"limit":1}} +$ fossil json --json-input in.json # use filename - for stdin +``` + +The above is equivalent to: + +```console +$ echo '{"indent":2, "payload":{"limit":1}}' \ + | fossil json timeline wiki --json-input - +``` + +Note that the "command" JSON parameter is only checked when no json +subcommand is provided on the CLI or via the HTTP request path. Thus we +cannot pass the CLI args "json timeline" in conjunction with a "command" +string of "wiki" this way. + +***HOWEVER...*** + +Much of the existing JSON code was written before the `--json-input` +option was possible. Because of this, there might be some +"misinteractions" when providing request-specific options via *both* +CLI options and simulated POST data. Those cases will eventually be +ironed out (with CLI options taking precedence). Until then, when +"POSTing" data in CLI mode, for consistent/predictible results always +provide any options via the JSON request data, not CLI arguments. That +said, there "should not" be any blatant incompatibilities, but some +routines will prefer `POST.payload` over CLI/GET arguments, so there +are some minor inconsistencies across various commands with regards to +which source (POST/GET/CLI) takes precedence for a given option. The +precedence "should always be the same," but currently cannot be due to +core fossil implementation details (the internal consolidation of +GET/CLI/POST vars into a single set). + + +<a id="json-indentation"></a> +# Indentation/Formatting of JSON Output + +CLI mode accepts the `--indent|-I #` option to set the indention level +and HTTP mode accepts `indent=#` as a GET/POST parameter. The semantics +of the indention level are derived from the underlying JSON library and +have the following meanings: 0 (zero) or less disables all superfluous +indentation (this is the default in HTTP mode). A value of 1 uses 1 hard +TAB character (ASCII 0x09) per level of indention (the default in CLI +mode). Values greater than 1 use that many whitespaces (ASCII 32d) per +level of indention. e.g. a value of 7 uses 7 spaces per level of +indention. There is no way to specify one whitespace per level, but if +you *really* want one whitespace instead of one tab (same data size) you +can filter the output to globally replace ASCII 9dec (TAB) with ASCII +32dec (space). Because JSON string values *never* contain hard tabs +(they are represented by `\t`) there is no chance that such a global +replacement will corrupt JSON string contents - only the formatting will +be affected. + +Potential TODO: because extraneous indention "could potentially" be used +as a form of DoS, the option *might* be subject to later removal from HTTP +mode (in CLI it's fine). + +In HTTP mode no trailing newline is added to the output, whereas in CLI +mode one is normally appended (exception: in JSONP mode no newline is +appended, to (rather pedantically and arbitraily) allow the client to +add a semicolon at the end if he likes). There is currently no option to +control the newline behaviour, but the underlying JSON code supports +this option, so adding it to this API is just a matter of adding the +CLI/HTTP args for it. + +Pedantic note: internally the indention level is stored as a single +byte, so giving large indention values will cause harmless numeric +overflow (with only cosmetic effects), meaning, e.g., 257 will overflow +to the value 1. + +Potential TODO: consider changing cson's indention mechanism to use a +*signed* number, using negative values for tabs and positive for +whitespace count (or the other way around). This would require more +doc changes than code changes :/. + + +<a id="jsonp"></a> +# JSONP + +The API supports JSONP-style output. The caller specifies the callback +name and the JSON response will be wrapped in a function call to that +name. For HTTP mode pass the `jsonp=string` option (via GET or POST +envelope) and for CLI use `--jsonp string`. + +For example, if we pass the JSONP name `myCallback` then a response will +look like: + +```js +myCallback({...response...}) +``` + +Note that fossil does not evaluate the callback name itself, other than +to verify that it is-a string, so "garbage in, garbage out," and all +that. (Remember that CLI and GET parameters are *always* strings, even +if they *look* like numbers.) + + +<a id="result-codes"></a> +# API Result Codes + +Result codes are strings which tell the client whether or not a given +API call succeeded or failed, and if it failed *perhaps* some hint as to +why it failed. + +The result code is available via the resultCode property of every +*error* response envelope. Since having a result code value for success +responses is somewhat redundant, success responses contain no resultCode +property. In practice this simplifies error checking on the client side. + +The codes are strings in the form `FOSSIL-####`, where `####` is a +4-digit integral number, left-padded with zeros. The numbers follow +these conventions: + +- The number 0000 is reserved for the "not an error" (OK) case. Since + success responses do not contain a result code, clients won't see + this value (except in documentation). +- All numbers with a leading 0 are reserved for *potential* future use + in reporting non-fatal warnings. +- Despite *possibly* having leading zeros, the numbers are decimal, + not octal. Script code which uses eval() or similar to produce + integers from them may need to take that into account. +- The 1000ths and 100ths places of the number describe the general + category of the error, e.g. authentication- vs. database- vs. usage + errors. The 100ths place is more specific than the 1000ths place, + allowing two levels of sub-categorization (which "should be enough" + for this purpose). This separation allows the server administrator + to configure the level of granularity of error reporting. e.g. some + admins consider error messages to be security-relevant and like to + "dumb them down" on their way to the client, whereas developers + normally want to see very specific error codes when tracking down a + problem. We can offer a configuration option to "dumb down" error + codes to their generic category by simply doing a modulo 100 + (or 1000) against the native error code number. e.g. FOSSIL-1271 + could (via a simple modulo) be reduced to FOSSIL-1200 or + FOSSIL-1000, depending on the paranoia level of the sysadmin. I have + tried to order the result code numbers so that a dumb-down level of + 2 provides reasonably usable results without giving away too much + detail to malicious clients.\ + (**TODO:** `g.json.errorDetailParanoia` is used to set the + default dumb-down level, but it is currently set at compile-time. + It needs to be moved to a config option. We have a chicken/egg scenario + with error reporting and db access there (where the config is + stored).) +- Once a number is assigned to a given error condition (and actually + used somewhere), it may not be changed/redefined. JSON clients need + to be able to rely on stable result codes in order to provide + adequate error reporting to their clients, and possibly for some + error recovery logic as well (i.e. to decide whether to abort or + retry). + +The *tentative* list of result codes is shown in the following table. +These numbers/ranges are "nearly arbitrarily" chosen except for the +"special" value 0000. + +**Maintenance reminder:** these codes are defined in +[`src/json_detail.h`](/finfo/src/json_detail.h) (enum +`FossilJsonCodes`) and assigned default `resultText` values in +[`src/json.c:json_err_cstr()`](/finfo/src/json.c). Changes there need +to be reflected here (and vice versa). Also, we have assertions in +place to ensure that C-side codes are in the range 1000-9999, so do +not just go blindly change the numeric ranges used by the enum. + + +**`FOSSIL-0###`: Non-error Category** + +- `FOSSIL-0000`: Success/not an error. Succesful responses do not + contain a resultCode, so clients should never see this. +- `FOSSIL-0###`: Reserved for potential future use in reporting + non-fatal warnings. + + + +**`FOSSIL-1000`: Generic Errors Category** + +- `FOSSIL-1101`: Invalid request. Request envelope is invalid or + missing. +- `FOSSIL-1102`: Unknown JSON command. +- `FOSSIL-1103`: Unknown/unspecified error +- `FOSSIL-1104`: RE-USE +- `FOSSIL-1105`: A server-side timeout was reached. (i’m not sure we + can actually implement this one, though.) +- `FOSSIL-1106`: Assertion failed (or would have had we + continued). Note: if an `assert()` fails in CGI/server modes, the HTTP + response will be code 500 (Internal Server Error). We want to avoid + that and return a JSON response instead. All of that said - there seems + to be little reason to implement this, since assertions are "truly + serious" errors. +- `FOSSIL-1107`: Allocation/out of memory error. This cannot be reasonably + reported because fossil aborts if an allocation fails. +- `FOSSIL-1108`: Requested API is not yet implemented. +- `FOSSIL-1109`: Panic! Fossil's `fossil_panic()` or `cgi_panic()` was + called. In non-JSON HTML mode this produces an HTTP 500 + error. Clients "should" report this as a potential bug, as it + "possibly" indicates that the C code has incorrect argument- or + error handling somewhere. +- `FOSSIL-1110`: Reading of artifact manifest failed. Time to contact + your local fossil guru. +- `FOSSIL-1111`: Opening of file failed (e.g. POST data provided to + CLI mode). + + +**`FOSSIL-2000`: Authentication/Access Error Category** + +- `FOSSIL-2001`: Privileged request was missing authentication + token/cookie. +- `FOSSIL-2002`: Access to requested resource was denied. Oftentimes + the `resultText` property will contain a human-language description of + the access rights needed for the given command. +- `FOSSIL-2003`: Requested command is not available in the current + operating mode. Returned in CLI mode by commands which require HTTP + mode (e.g. login), and vice versa. FIXME: now that we can simulate + POST in CLI mode, we can get rid of this distinction for some of the + commands. +- `FOSSIL-2100`: Login Failed. +- `FOSSIL-2101`: Anonymous login attempt is missing the + "anonymousSeed" property (fetched via [the `/json/anonymousPassword` + request](api-auth.md#login-anonymous)). Note that this is more + specific form of `FOSSIL-3002`. + + +ONLY FOR TESTING purposes should the remaning 210X sub-codes be +enabled (they are potentially security-relevant, in that the client +knows which part of the request was valid/invalid): + +- `FOSSIL-2102`: Name not supplied in login request +- `FOSSIL-2103`: Password not supplied in login request +- `FOSSIL-2104`: No name/password match found + + +**`FOSSIL-3000`: Usage Error Category** + +- `FOSSIL-3001`: Invalid argument/parameter type(s) or value(s) in + request +- `FOSSIL-3002`: Required argument(s)/parameter(s) missing from + request +- `FOSSIL-3003`: Requested resource identifier is ambiguous (e.g. a + shortened hash that matches multiple artifacts, an abbreviated + date that matches multiple commits, etc.) +- `FOSSIL-3004`: Unresolved resource identifier. A branch/tag/uuid + provided by client code could not be resolved. This is a special + case of #3006. +- `FOSSIL-3005`: Resource already exists and overwriting/replacing is + not allowed. e.g. trying to create a wiki page or user which already + exists. FIXME? Consolidate this and resource-not-found into a + separate category for dumb-down purposes? +- `FOSSIL-3006`: Requested resource not found. e.g artifact ID, branch + name, etc. + + +**`FOSSIL-4000`: Database-related Error Category** + +- `FOSSIL-4001`: Statement preparation failed. +- `FOSSIL-4002`: Parameter binding failed. +- `FOSSIL-4003`: Statement execution failed. +- `FOSSIL-4004`: Database locked (this is not used anywhere, but + reserved for future use). + +Special-case DB-related errors... + +- `FOSSIL-4101`: Fossil Schema out of date (repo rebuild required). +- `FOSSIL-4102`: Fossil repo db could not be found. +- `FOSSIL-4103`: Repository db is not valid (possibly corrupt). +- `FOSSIL-4104`: Check-out not found. This is similar to FOSSIL-4102 + but indicates that a local checkout is required (but was not + found). Note that the 4102 gets triggered earlier than this one, and + so can appear in cases when a user might otherwise expect a 4104 + error. + + +Some of those error codes are of course "too detailed" for the client to +do anything with (e.g.. 4001-4004), but their intention is to make it +easier for Fossil developers to (A) track down problems and (B) support +clients who report problems. If a client reports, "I get a FOSSIL-4000, +how can I fix it?" then the developers/support personnel can't say much +unless they know if it's a 4001, 4002, 4003, 4004, or 4101 (in which +case they can probably zero in on the problem fairly quickly, since they +know which API call triggered it and they know (from the error code) the +general source of the problem). + +## Why Standard/Immutable Result Codes are Important + +- They are easily internationalized (i.e. associated with non-English + error text) +- Clients may be able to add automatic retry strategies for certain + problem types by examining the result code. e.g. if fossil returns a + locking or timeout error \[it currently does no special + timeout/locking handling, by the way\] the client could re-try, + whereas a usage error cannot be sensibly retried with the same + inputs. +- The "category" structure described above allows us some degree of + flexibility in how detailed the reported errors are reported. +- While the string prefix "FOSSIL-" on the error codes may seem + superfluous, it has one minor *potential* advantage on the client + side: when managing several unrelated data sources, these error + codes can be immediately identified (by higher-level code which may + be ignorant of the data source) as having come from the fossil API. + Think "ORA-111" vs. "111". diff --git a/www/json-api/hacking.md b/www/json-api/hacking.md new file mode 100644 index 0000000..1de360a --- /dev/null +++ b/www/json-api/hacking.md @@ -0,0 +1,353 @@ +# JSON API: Hacker's Guide +([⬑JSON API Index](index.md)) + +Jump to: + +* [Before Committing Changes](#before-committing) +* [JSON C API](#json-c-api) +* [Reporting Errors](#reporting-errors) +* [Getting Command Arguments](#command-args) +* [Creating JSON Data](#creating-json) + * [Creating JSON Values](#creating-json-values) + * [Converting SQL Query Results to JSON](#query-to-json) + +This section will only be of interest to those wanting to work on the +Fossil/JSON code. That said... + +If you happen to hack on the code and find something worth noting here +for others, please feel free to expand this section. It will only +improve via feedback from those working on the code. + +--- + +<a id="before-committing"></a> +# Before Committing Changes... + +Because this code lives in the trunk, there are certain +guidelines which must be followed before committing any changes: + +1. Read the [checkin preparation list](/doc/trunk/www/checkin.wiki). +2. Changes to the files `src/json_*.*`, and its related support code + (e.g. `ajax/*.*`), may be made freely without affecting mainline + users. Changes to other files, unless they are trivial or made for + purposes outside the JSON API (e.g. an unrelated bug fix), must be + reviewed carefully before committing. When in doubt, create a branch + and post a request for a review. +3. The Golden Rule is: *do not break the trunk build*. + + +<a id="json-c-api"></a> +# JSON C API + +libcson, the underlying JSON API, is a separate project, included in +fossil in "amalgamation" form: see `extsrc/cson_amalgamation.[ch]`. It has +thorough API docs and a good deal of information is in its wiki: + +[](https://fossil.wanderinghorse.net/wikis/cson/) + +In particular: + +[](https://fossil.wanderinghorse.net/wikis/cson/?page=CsonArchitecture) + +gives an overview of its architecture. Occasionally new versions of it +are pulled into the Fossil tree, but other developers generally need not +concern themselves with that. + +(Trivia: the cson wiki's back-end is fossil using this very JSON API, +living on top of a custom JavaScript+HTML5 application.) + +Only a small handful of low-level fossil routines actually input or +output JSON text (only for reading in POST data and sending the +response). In the C code we work with the higher-level JSON value +abstractions provided by cson (conceptually similar to an XML DOM). All +of the JSON-defined data types are supported, and we can construct JSON +output of near arbitrary complexity with the caveat that *cyclic data +structures are strictly forbidden*, and *will* cause memory corruption, +crashes, double free()'s, or other undefined behaviour. Because JSON +cannot, without client-specific semantic extensions to JSON, represent +cyclic structures, it is not anticipated that this will be a +problem/limitation when generating output for fossil. + + + +<a id="json-commands"></a> +# Architecture of JSON Commands + +In order to consolidate CLI/HTTP modes for JSON handling, this code +foregoes fossil's conventional command/path dispatching mechanism. Only +the top-most "json" command/path is dispatched directly by fossil's +core. The disadvantages of this are that we lose fossil's conventional +help text mechanism (which is based on code comments in the +command/path's dispatcher impl) and the ability to write abbreviated +command names in CLI mode ("json" itself may be abbreviated, but not the +subcommands). The advantages are that we can handle CLI/HTTP modes +almost identically (there are a couple of minor differences) by unifying +them under the same callback functions much more easily. + +The top-level "json" command/path uses its own dispatching mechanism +which uses either the path (in HTTP mode) or CLI positional arguments to +dispatch commands (stopping at the first "flag option" (e.g. -foo) in +CLI mode). The command handlers are simply callback functions which +return a cson\_value pointer (the C representation of an arbitrary JSON +value), representing the "payload" of the response (or NULL - not all +responses need a payload). On error these callbacks set the internal +JSON error state (detailed in a subsection below) and return NULL. The +top-level dispatcher then creates a response envelope and returns the +"payload" from the command (if any) to the caller. If a callback sets +the error state, the top-level dispatcher takes care to set the error +information in the response envelope. In summary: + +- The top-level dispatchers (`json_page_top()` and `json_cmd_top()`) + are called by fossil's core when the "json" command/path is called. + They initialize the JSON-mode global state, dispatch the requested + command, and handle the creation of the response envelope. They + prepare all the basic things which the individual subcommands need + in order to function. +- The command handlers (most are named `json_page_something()`) + implement the `fossil_json_f()` callback interface (see + [`src/json_detail.h`](/finfo/src/json_detail.h)). They are + responsible for permissions checking, setting any error state, and + passing back a payload (if needed - not all commands return a + payload). It is strictly forbidden for these callbacks to produce + any output on stdout/stderr, and doing so effectively corrupts the + out-bound JSON and HTTP headers. + +There is a wrench in all of that, however: the vast majority of fossil's +commands "fail fast" - they will `exit()` if they encounter an error. To +handle that, the fossil core error reporting routines have been +refactored a small bit to operate differently when we are running in +JSON mode. Instead of the conventional output, they generate a JSON +error response. In HTTP mode they exit with code 0 to avoid causing an +HTTP 500 error, whereas in CLI mode they will exit with a non-0 code. +Those routines still `exit()`, as in the conventional CLI/HTTP modes, but +they will exit differently. Because of this, it is perfectly fine for a +command handler to exit via one of fossil's conventional mechanisms +(e.g. `db_prepare()` can be fatal, and callbacks may call `fossil_panic()` +if they really want to). One exception is `fossil_exit()`, which does +_not_ generate any extra output and will `exit()` the app. In the JSON +API, as a rule of thumb, `fossil_exit()` is only used when we *want* a +failed request to cause an HTTP 500 error, and it is reserved for +allocation errors and similar truly catastrophic failures. That said... +libcson has been hacked to use `fossil_alloc()` and friends for memory +management, and those routines exit on error, so alloc error handling in +the JSON command handler code can afford to be a little lax (the +majority of *potential* errors clients get from the cson API have +allocation failure as their root cause). + +As a side-note: the vast majority (if not all) of the cson API calls are +"NULL-safe", meaning that will return an error code (or be a no-op) if +passed NULL arguments. e.g. the following chain of calls will not crash +if the value we're looking for does not exist, is-not-a String (see +`cson_value_get_string()` for important details), or if `myObj` is NULL: + +```c +const char * str = + cson_string_cstr( // get the C-string form of a cson_string + cson_value_get_string( // get its cson_string form + cson_object_get(myObj,"foo") // search for key in an Object + ) + ); +``` + +If `"foo"` is not found in `myObj` (or if `myObj` is NULL) then v will be +NULL, as opposed to stepping on a NULL pointer somewhere in that call +chain. + +Note that all cson JSON values except Arrays and Objects are *immutable* +- you cannot change a string's or number's value, for example. They also +use reference counting to manage ownership, as documented and +demonstrated on this page: + +[](https://fossil.wanderinghorse.net/wikis/cson/?page=TipsAndTricks) + +In short, after creating a new value you must eventually *either* add it +to a container (Object or Array) to transfer ownership *or* call +`cson_value_free()` to clean it up (exception: the Fossil/JSON command +callbacks *return* a value to transfer ownership to the dispatcher). +Sometimes it's more complex than that, but not normally. Any given value +may legally be stored in any number of containers (or multiple times +within one container), as long as *no cycles* are introduced (cycles +*will* cause undefined behaviour). Ownership is shared using reference +counting and the value will eventually be freed up when its last +remaining reference is freed (e.g. when the last container holding it is +cleaned up). For many examples of using cson in the context of fossil, +see the existing `json_page_xxx()` functions in `json_*.c`. + +<a id="reporting-errors"></a> +# Reporting Errors + +To report an error from a command callback, one abstractly needs to: + +- Set g.json.resultCode to one of the `FSL_JSON_E_xxx` values + (defined in [`src/json_detail.h`](/finfo/src/json_detail.h)). +- *Optionally* set `g.zErrMsg` to contain the (dynamically-allocated!) + error string to be sent to the client. If no error string is set + then a standard/generic string is used for the given error code. +- Clean up any resources created so far by the handler. +- Return NULL. If it returns non-NULL, the dispatcher will destroy the + value and not include it in the error response. + +That normally looks something like this: + +``` +if(!g.perm.Read){ + json_set_err(FSL_JSON_E_DENIED, "Requires 'o' permissions."); + return NULL; +} +``` + +`json_set_err()` is a variadic printf-like function, and can use the +printf extensions supported by mprintf() and friends (e.g. `%Q` and `%q`) +(but they are normally not needed in the context of JSON). If the error +string is NULL or empty then `json_err_cstr(errorCode)` is used to fetch +the standard/generic error string for the given code. + +When control returns to the top-level dispatching function it will check +`g.json.resultCode` and, if it is not 0, create an error response using +the `g.json.resultCode` and `g.zErrMsg` to construct the response's +`resultCode` and `resultText` properties. + +If a function wants to output an error and exit by itself, as opposed +to returning to the dispatcher, then it must behave slightly +differently. See the docs for `json_err()` (in +[`src/json.c`](/finfo/src/json.c)) for details, and search that file +for various examples of its usage. It is also used by fossil's core +error-reporting APIs, e.g. `fossil_panic()` (defined in [`src/main.c`](/finfo/src/main.c)). +That said, it would be "highly unusual" for a callback to need to do +this - it is *far* simpler (and more consistent/reliable) to set the +error state and return to the dispatcher. + +<a id="command-args"></a> +# Getting Command Arguments + +Positional parameters can be fetched usinig `json_command_arg(N)`, where +N is the argument position, with position 0 being the "json" +command/path. In CLI mode positional arguments have their obvious +meaning. In HTTP mode the request path (or the "command" request +property) is used to build up the "command path" instead. For example: + +CLI: `fossil json a b c` + +HTTP: `/json/a/b/c` + +HTTP POST or CLI with `--json-input`: /json with POSTed envelope +`{"command": "a/b/c" …}` + +Those will have identical "command paths," and `json_command_path(2)` +would return the "b" part. + +Caveat: a limitation of this support is that all CLI flags must come +*after* all *non-flag* positional arguments (e.g. file names or +subcommand names). Any argument starting with a dash ("-") is considered +by this code to be a potential "flag" argument, and all arguments after +it are ignored (because the generic handling cannot know if a flag +requires an argument, which changes how the rest of the arguments need +to be interpreted). + +To get named parameters, there are several approaches (plus some special +cases). Named parameters can normally come from any of the following +sources: + +- CLI arguments, e.g. `--foo bar` +- GET parameters: `/json/...?foo=bar` +- Properties of the POST envelope +- Properties of the `POST.payload` object (if any). + +To try to simplify the guessing process the API has a number of +functions which behave ever so slightly differently. A summary: + +- `json_getenv()` and `json_getenv_TYPE()` search the so-called "JSON + environment," which is a superset of the GET/POST/`POST.payload` (if + `POST.payload` is-a Object). +- `json_find_option_TYPE()`: searches the CLI args (only when in CLI + mode) and the JSON environment. +- The use of fossil's `P()` and `PD()` macros is discouraged in JSON + callbacks because they can only handle String data from the CLI or + GET parameters (not POST/`POST.payload`). (Note that `P()` and `PD()` + *normally* also handle POSTed keys, but they only "see" values + posted as form-urlencoded fields, and not JSON format.) +- `find_option()` (from `src/main.c`) "should" also be avoided in + JSON API handlers because it removes flag from the g.argv + arguments list. That said, the JSON API does use `find_option()` in + several of its option-finding convenience wrappers. + +For example code: the existing command callbacks demonstrate all kinds +of uses and the various styles of parameter/option inspection. Check out +any of the functions named `json_page_SOMETHING()`. + +<a href="creating-json"></a> +# Creating JSON Data + +<a href="creating-json-values"></a> +## Creating JSON Values + +cson has a fairly rich API for creating and manipulating the various +JSON-defined value types. For a detailed overview and demonstration I +recommend reading: + +[](https://fossil.wanderinghorse.net/wikis/cson/?page=HowTo) + +That said, the Fossil/JSON API has several convenience wrappers to save +a few bytes of typing: + +- `json_new_string("foo")` is easier to use than + `cson_value_new_string("foo", 3)`, and + `json_new_string_f("%s","foo")` is more flexible. +- `json_new_int()` is easier to type than `cson_value_new_integer()`. +- `cson_output_Blob()` and `cson_parse_Blob()` can write/read JSON + to/from fossil `Blob`-type objects. + +It also provides several lower-level JSON features which aren't of +general utility but provide necessary functionality for some of the +framework-level code (e.g. `cson_data_dest_cgi()`), which is only used +by the deepest of the JSON internals). + + +<a href="query-to-json"></a> +## Converting SQL Query Results to JSON + +The `cson_sqlite3_xxx()` family of functions convert `sqlite3_stmt` rows +to Arrays or Objects, or convert single columns to a JSON-compatible +form. See `json_stmt_to_array_of_obj()`, +`json_stmt_to_array_of_array()` (both in `src/json.c`), and +`cson_sqlite3_column_to_value()` and friends (in +`extsrc/cson_amalgamation.h`). They work in an intuitive way for numeric +types, but they optimistically/naively *assume* that any fields of type +TEXT or BLOB are actually UTF8 data, and treat them as such. cson's +string class only handles UTF8 data and it is semantically illegal to +feed them anything but UTF8. Violating this will likely result in +down-stream errors (e.g. when emitting the JSON string output). **The +moral of this story is:** *do not use these APIs to fetch binary data*. +JSON doesn't do binary and the `cson_string` class does not +protect itself against clients feeding it non-UTF8 data. + +Here's a basic example of using these features: + +```c +Stmt q = empty_Stmt; +cson_value * rows = NULL; +db_prepare(&q, "SELECT a AS a, b AS b, c AS c FROM foo"); +rows = json_stmt_to_array_of_obj( &sql, NULL ); +db_finalize(&q); +// side note: if db_prepare()/finalize() fail (==they exit()) +// then a JSON-format error reponse will be generated. +``` + +On success (and if there were results), `rows` is now an Array value, +each entry of which contains an Object containing the fields (key/value +pairs) of each row. `json_stmt_to_array_of_array()` returns each row +as an Array containing the column values (with no column name +information). + +**Note the seemingly superfluous use of the "AS" clause in the above +SQL.** Having them is actually significant! If a query does *not* use AS +clauses, the row names returned by the db driver *might* be different +than they appear in the query (this is documented behaviour of sqlite3). +Because the JSON API needs to return stable field names, we need to use +AS clauses to be guaranteed that the db driver will return the column +names we want. Note that the AS clause is often used to translate column +names into something more JSON-conventional or user-friendly, e.g. +"SELECT cap AS capabilities...". Alternately, we can convert the +individual `sqlite3_stmt` column values to JSON using +`cson_sqlite3_column_to_value()`, without referring directly to the +db-reported column name. diff --git a/www/json-api/index.md b/www/json-api/index.md new file mode 100644 index 0000000..757babe --- /dev/null +++ b/www/json-api/index.md @@ -0,0 +1,40 @@ +# JSON API Index + +This is the client-side documentation of Fossil's JSON API. The JSON +API aims to provide access to many of the primary fossil features via +AJAX-style interfaces. + +* [Introduction](intro.md) +* [General API Conventions](conventions.md) +* [Tips & Tricks](tips.md) +* [Hacking Guide](hacking.md) + +General warnings regarding the APIs linked to in the following list: + +- **NOTE** that request/response examples shown in the individual API +pages do not show [the standard request/response envelope](conventions.md) +(for brevity and sanity). +- **Achtung:** just because a given feature is described as being +implemented does not mean that the implementation is "final" - it may be +changed at any time until we find/implement useful APIs. + +The APIs, alphabetically by category: + +* [Artifact Info](api-artifact.md) +* [Authentication](api-auth.md) +* [Branches](api-branch.md) +* [Checkout Status](api-checkout.md) +* [Config](api-config.md) +* [Diffs](api-diff.md) +* [Directory Listing](api-dir.md) +* [File Info](api-finfo.md) +* [The Obligatory Misc. Category](api-misc.md) +* [Repository Stats](api-stat.md) +* [Settings](api-settings.md) +* [SQL Query](api-query.md) +* [Tags](api-tag.md) +* [Tickets](api-ticket.md) +* [Timeline](api-timeline.md) +* [User Management](api-user.md) +* [Version](api-version.md) +* [Wiki](api-wiki.md) diff --git a/www/json-api/intro.md b/www/json-api/intro.md new file mode 100644 index 0000000..d8a9f14 --- /dev/null +++ b/www/json-api/intro.md @@ -0,0 +1,172 @@ +# JSON API Introduction +([⬑JSON API Index](index.md)) + +Jump to: + +* [Why?](#why) +* [Building JSON Support](#builing) +* [Goals & Non-goals](#goals) +* [Potential Client-side Uses](#potential-uses) +* [Technical Problems and Considerations](#considerations) + +--- + +<a id="why"></a> +# Why? + +In September, 2011, Fossil contributor Stephan Beal had the great +pleasure of meeting D. Richard Hipp, Fossil's author, for lunch in +Munich, Germany. During the conversation Richard asked, "what does +Fossil need next?" Stephan's first answer was, "refactoring into a +library/client, as opposed to a monolithic app." We very quickly +agreed that the effort required would be "herculean," and second +choice was voiced, "a JSON API." They briefly discussed the idea and +Richard gave his blessing. That night work began. + +Why a JSON API? Because it is the next best thing to the +"librification" of Fossil, in that it makes Fossil's features +available to near-arbitrary applications using a simple, globally +available data format. + +<a id="building"></a> +# Building JSON Support + +In environments supported by fossil's `configure` script, +simply pass `--enable-json` to it: + +``` +$ ./configure --prefix=$HOME --enable-json ... +``` + +When built without that option, JSON support is disabled. **When +reconfiguring the source tree**, ***always be sure to do a "make +clean"*** (or equivalent for your platform) between builds (preferably +*before* reconfiguring), to ensure that everything is rebuilt properly. +If you fail to do that after enabling JSON on a tree which has already +been built, most of the sources will not be rebuilt properly. The reason +is that the JSON files are actually unconditionally compiled, but when +built without `--enable-json` they compile to empty object files. Thus +after a reconfigure the (empty) object files are still up-to-date +vis-a-vis the sources, and won't be rebuilt. + +To build Fossil with JSON support on Windows using the Microsoft C +compiler: + +``` +cd win +nmake -f Makefile.msc FOSSIL_ENABLE_JSON=1 +``` + +It has been seen to compile in VC versions 6 and higher. + +<a id="goals"></a> +# Goals & Non-goals + +The API described here is most certainly not +[*REST*](http://en.wikipedia.org/wiki/Representational_state_transfer)-conformant, +but is instead JSON over HTTP. The error reporting techniques of the +REST conventions (using HTTP error codes) "does not mesh" with my ideas +of separation of transport- vs. app-side errors. Additionally, REST +requires HTTP methods which are not specified by CGI (namely PUT and +DELETE), which means we can't possibly implement a REST-compatible +interface on top of fossil (which uses CGI mode even for its built-in +server). + +The **overall goals** of this effort include: + +- A JSON-based API off of which clients can build customized Fossil + UIs and special-purpose applications. e.g. a desktop notification + applet which polls for new timeline data. +- Arbitrary JSON-using clients should be able to use it. Though JSON + originates from JavaScript, it is truly a cross-platform data format + with a very high adoption rate. (There’s even a JSON implementation + for Oracle PL/SQL.) +- Fossil’s CGI and Server modes are the main targets and should be + supported equally. CLI JSON mode is of secondary concern (but is in + practice easier to test, so it’s generally implemented first). + +The ***non-goals*** include: + +- We won’t be able to implement *every* feature of Fossil via a JSON + interface, and we won’t try to. +- Binary data (e.g. commits of binary files or downloading ZIP files) + is not an initial goal, but "might be interesting" once the overall + infrastructure is in place and working well. See below for more + details about binary data in JSON. +- A "pure REST" interface is seemingly not possible due to REST + relying on HTTP methods not specified in the CGI standard (PUT and + DELETE). Additionally, REST-style error reporting cannot be used by + non-HTTP clients (which this code supports). + +Adding JSON support also gives us a framework off of which to +build/enhance other features. Some examples include: + +- **Internationalization**. Errors are reported via standard codes and + the raw artifact data is language-independent. +- The ability to author **special-case clients**, e.g. a ticket + poller. +- Use **arbitrary HTTP-capable languages** to implement such tools. + Programming languages which can execute programs and intercept their + stdout output can use the JSON API via a local fossil binary. +- **Automatable tests.** Many of fossil's test results currently have + to be "visually reviewed" for correctness after changes (e.g. + changes in the HTML interface). JSON structures can be + programmatically checked for correctness. Artifacts are immutable, + which allows us to be very specific in what data to expect as output + (for artifact-specific requests the payload data will often (but not + always) be the same across all requests and all time). + +<a id="potential-uses"></a> +# Potential Client-side Uses + +Some of the potential client-side uses of this API include... + +- Custom apps/applets to fetch timeline/ticket/etc. information from + arbitrary repositories. There are many possibilities here, including + "dashboard" sites which monitor several repositories. +- Custom post-commit triggers, by polling for changes and reacting to + them (e.g. sending mails). +- A custom wiki front-end which uses fossil as the back-end storage, + inheriting its versioning and user access support while providing a + completely custom wiki-centric UI. Such a wiki need not have, on the + surface, anything to do with fossil or source control, as fossil + would just become a glorified wiki back-end. This approach also + allows clients to serve wiki pages in a format of their choice - + since all rendering would be done client-side, they could use + whatever format they like. + + +<a id="considerations"></a> +# Technical Problems and Considerations + +A random list of considerations which need to be made and potential +problem areas... + +- **Binary data:** JSON is a text serialization method, and it takes + up the “payload” area of each HTTP request, so there is no + reasonable way to include binary data in the JSON message without + some sort of codec like Base64, for which there is no provision in + the current JSON API. You will therefore find no JSON API for + committing changes to a file in the repository, for example. Other + Fossil APIs such as [`/raw`](/help/www/raw) or + [`/fileedit`](../fileedit-page.md) may serve you better. +- **64-bit integers:** The JSON standard does not specify integer precision, + because it targets many different platforms, and not all of + them can support more than 32 bits. JavaScript (from which JSON + derives) supports 53 bits of integer precision, which may affect how + a given client-side JSON implementation sends large integers to Fossil’s JSON + API. Our JSON parser can cope with integers larger than 32 bits on input, and it + can emit them, but it requires platform support. If you’re running + Fossil on a 64-bit host, you should not run into problems in + this area, but if you’re on a legacy 32-bit only or a mixed 32/64-bit + system, it’s possible that some integers in the API could be + clipped. Realize however that this is a rare case: Fossil currently + cannot store files large enough to exceed a 32-bit `size_t` value, + and `time_t` won’t roll past 32-bit integers until 2038. We’re aware + of no other uses of integers in this API that could even in + principle exceed the range of a 32-bit integer. +- **Timestamps:** For portability, this API uses UTC Unix epoch + timestamps. (`time_t`) They are the most portable time representation out + there, easily usable in most programming environments. (In + hindsight, we might better have used a higher-precision time format, + but changing that now would break API compatibility.) diff --git a/www/json-api/tips.md b/www/json-api/tips.md new file mode 100644 index 0000000..885bdf8 --- /dev/null +++ b/www/json-api/tips.md @@ -0,0 +1,102 @@ +# JSON API: Tips and Tricks +([⬑JSON API Index](index.md)) + +Jump to: + +* [Beware of Content-Type and Encoding...](#content-type) + * [Using `curl` and `wget`](#curl-wget) +* [Example JavaScript](#javascript) +* [Demo Apps](#demo-apps) + +--- + +<a id="content-type"></a> +# Beware of Content-Type and Encoding... + +When posting data to fossil, make sure that the request sends: + +- **Content-Type** of `application/json`. Fossil also (currently) + accepts `application/javascript` and `text/plain` as JSON input, + but `application/json` is preferred. The client may optionally + send `;charset=utf-8` with the Content-Type, but any other + encoding produces undefined results. Behaviour without the charset + or with `;charset=utf-8` suffix is identical. +- **POST data must be an non-form-encoded JSON string** + (ASCII or UTF-8). jQuery, by default, form-urlencodes it, which the + fossil json bits cannot read. e.g. post the result of + `JSON.stringify(requestObject)`, without any additional encoding on + top of it. +- **When POSTing via jQuery**, set these AJAX options: + - `contentType:'application/json'` + - `dataType:'text'` + - `data:JSON.stringify(requestObject)` +- **When POSTing via XMLHttpRequest** (XHR), be sure to: + - `xhr.open( … )` + - `xhr.setRequestHeader("Content-Type", "application/json")` + - `xhr.send( JSON.stringify( requestObject ) )` + +The response will be (except in the case of an HTTP 500 error or +similar) a JSON or JSONP string, ready to be parsed by your favourite +`JSON.parse()` implementation or `eval()`'d directly. + +<a id="curl-wget"></a> +## Using `curl` and `wget` + +Both [curl](https://curl.haxx.se/) and +[wget](https://www.gnu.org/software/wget/) can be used to post data to +this API from the command line or scripts, but both require an extra +parameter to set the request encoding. + +Example: + +```console +$ cat x.json +{ +"payload": { + "sql": "SELECT * FROM reportfmt limit 1", + "format": "o" + } +} + +# Fossil has been started locally with: +# fossil server --localauth +# which allows the following requests to work without extra +# authenticaion: + +$ wget -q -O- \ + --post-file=x.json \ + --header="Content-Type: application/json" \ + 'http://localhost:8080/json/query' + +$ curl \ + --data-binary @x.json \ + --header 'Content-Type: application/json' \ + 'http://localhost:8080/json/query' +``` + +The relevant parts for encoding are the `--header` flag for `wget` and +`curl`, noting that they have different syntaxes for each +(`--header=X` vs `--header X`). + +<a id="javascript"></a> +# Example JavaScript (Browser and Shell) + +In the fossil source tree, [in the ajax directory](/dir/ajax), is test/demo code +implemented in HTML+JavaScript. While it is still quite experimental, it +demonstrates one approach to creating client-side wrapper APIs for +remote Fossil/JSON repositories. + +There is some additional JS test code, which uses the Rhino JS engine +(i.e. from the console, not the browser), under +[`ajax/i-test`](/dir/ajax/-itest). That adds a Rhino-based connection +back-end to the AJAJ API and uses it for running integration-style +tests against an arbitrary JSON-capable repository. + + +<a id="demo-apps"></a> +# Demo Apps + +Known in-the-wild apps using this API: + +- The wiki browsers/editors at [](https://fossil.wanderinghorse.net/wikis/) + diff --git a/www/loadmgmt.md b/www/loadmgmt.md new file mode 100644 index 0000000..c81fdfc --- /dev/null +++ b/www/loadmgmt.md @@ -0,0 +1,96 @@ +# Managing Server Load + +A Fossil server is very efficient and normally presents a very light +load on the server. The Fossil [self-hosting server][sh] is a 1/24th +slice VM at [Linode.com][lin] hosting 65 other repositories in addition +to Fossil, including some very high-traffic sites such as +<http://www.sqlite.org> and <http://system.data.sqlite.org>. This small +VM has a typical load of 0.05 to 0.1. A single HTTP request to Fossil +normally takes less than 10 milliseconds of CPU time to complete, so +requests can be arriving at a continuous rate of 20 or more per second, +and the CPU can still be mostly idle. + +However, there are some Fossil web pages that can consume large amounts +of CPU time, especially on repositories with a large number of files or +with long revision histories. High CPU usage pages include +[`/zip`](/help/zip), [`/tarball`](/help/tarball), +[`/annotate`](/help/annotate), and others. On very large repositories, +these commands can take 15 seconds or more of CPU time. If these kinds +of requests arrive too quickly, the load average on the server can grow +dramatically, making the server unresponsive. + +Fossil provides two capabilities to help avoid server overload problems +due to excessive requests to expensive pages: + +1. An optional cache is available that remembers the 10 most recently + requested `/zip` or `/tarball` pages and returns the precomputed + answer if the same page is requested again. + +2. Page requests can be configured to fail with a + “[503 Server Overload][503]” HTTP error if any request is + received while the host load average is too high. + +Both of these load-control mechanisms are turned off by default, but +they are recommended for high-traffic sites. Users with [admin +permissions](caps/index.md) are exempt from these restrictions, +provided they are logged in before the load gets too high (login is +disabled under high load). + +The webpage cache is activated using the [`fossil cache init`](/help/cache) +command-line on the server. Add a `-R` option to +specify the specific repository for which to enable caching. If running +this command as root, be sure to “`chown`” the cache database to give +the Fossil server write permission for the user ID of the web server; +this is a separate file in the same directory and with the same name as +the repository but with the “`.fossil`” suffix changed to “`.cache`”. + +To activate the server load control feature visit the Admin → Access +setup page in the administrative web interface; in the “**Server Load +Average Limit**” box enter the load average threshold above which “503 +Server Overload” replies will be issued for expensive requests. On the +self-hosting Fossil server, that value is set to 1.5, but you could +easily set it higher on a multi-core server. + +The maximum load average can also be set on the command line using +commands like this: + + fossil setting max-loadavg 1.5 + fossil all setting max-loadavg 1.5 + +The second form is especially useful for changing the maximum load +average simultaneously on a large number of repositories. + +Note that this load-average limiting feature is only available on +operating systems that support the [`getloadavg()`][gla] API. Most +modern Unix systems have this interface, but Windows does not, so the +feature will not work on Windows. + +Because Linux implements `getloadavg()` by accessing the `/proc/loadavg` +virtual file, you will need to make sure `/proc` is available to the +Fossil server. The most common reason for it to not be available is that +you are running a Fossil instance [inside a `chroot(2)` +jail](./chroot.md) and you have not mounted the `/proc` virtual file +system inside that jail. On the [self-hosting Fossil repositories][sh], +this was accomplished by adding a line to the `/etc/fstab` file: + + chroot_jail_proc /home/www/proc proc ro 0 0 + +The `/home/www/proc` pathname should be adjusted so that the `/proc` +component is at the root of the chroot jail, of course. + +To see if the load-average limiter is functional, visit the +[`/test-env`][hte] page of the server to view the current load average. +If the value for the load average is greater than zero, that means that +it is possible to activate the load-average limiter on that repository. +If the load average shows exactly "0.0", then that means that Fossil is +unable to find the load average. This can either be because it is in a +`chroot(2)` jail without `/proc` access, or because it is running on a +system that does not support `getloadavg()` and so the load-average +limiter will not function. + + +[503]: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.4 +[hte]: /help/www/test-env +[gla]: https://linux.die.net/man/3/getloadavg +[lin]: http://www.linode.com +[sh]: ./selfhost.wiki diff --git a/www/makefile.wiki b/www/makefile.wiki new file mode 100644 index 0000000..8cccfb0 --- /dev/null +++ b/www/makefile.wiki @@ -0,0 +1,327 @@ +<title>The Fossil Build Process</title> + +<h1>1.0 Introduction</h1> + +The build process for Fossil is tricky in that the source code +needs to be processed by three different preprocessor programs +before it is compiled. Most users will download a +[https://fossil-scm.org/home/uv/download.html | precompiled binary] +so this is of no consequence to them, and even those who +want to compile the code themselves can use one of the +[./build.wiki | existing makefiles]. +So most people do not need to be concerned with the +build complexities of Fossil. But hard-core developers who desire +a deep understanding of how Fossil is put together can benefit +from reviewing this article. + +<h1 id="srctour">2.0 Source Code Tour</h1> + +The source code for Fossil is found in the +[/dir?ci=trunk&name=src | src/] subdirectory of the +source tree. The src/ subdirectory contains all code, including +the code for the separate preprocessor programs. + +Each preprocessor program is a separate C program implemented in +a single file of C source code. The three preprocessor programs +are: + + 1. [/file/tools/mkindex.c | mkindex.c] + 2. [/file/tools/translate.c | translate.c] + 3. [/file/tools/makeheaders.c | makeheaders.c] + +Fossil uses [http://www.sqlite.org/ | SQLite] for on-disk +storage. The SQLite implementation is contained in three source +code files that do not participate in the preprocessing steps. +These three files that implement SQLite are: + + 4. sqlite3.c + 5. sqlite3.h + 6. shell.c + +All three SQLite source files are byte-for-byte copies of files by +the same name in the +standard [http://www.sqlite.org/amalgamation.html | amalgamation]. +The sqlite3.c file implements the database engine. The shell.c file +implements the command-line shell, which is accessed in fossil using +the [/help/sqlite3 | fossil sql] command. + +The shell.c command-line shell uses the [https://github.com/antirez/linenoise | +linenoise] library to implement line editing. linenoise comprises two +source files which were copied from the upstream repository with only +very minor portability edits: + + 7. linenoise.c + 8. linenoise.h + +The TH1 script engine is implemented using files: + + 9. th.c + 10. th.h + +The preprocessing steps are omitted for all of these imported +files. + +The VERSION.h header file is generated from other information sources +using a small program called: + + 11. [/file/tools/mkversion.c | mkversion.c] + +The builtin_data.h header file contains the definitions of C-language +byte-array constants that contain various resources such as scripts and +images. The builtin_data.h header file is generated from the original +resource files using a small program called: + + 12 [/file/tools/mkbuiltin.c | mkbuiltin.c] + +Examples of built-in resources include the [/file/src/diff.tcl | diff.tcl] +script used to implement the --tk option to [/help/diff| fossil diff], +the [/file/src/markdown.md | markdown documentation], and the various +CSS scripts, headers, and footers used to implement built-in skins. New +resources files are added to the "extra_files" variable in +[/file/tools/makemake.tcl | makemake.tcl]. + +The src/ subdirectory also contains documentation about the +makeheaders preprocessor program: + + 13. [../tools/makeheaders.html | makeheaders.html] + +Click on the link to read this documentation. In addition there is +a [http://www.tcl-lang.org/ | Tcl] script used to build the various makefiles: + + 14. makemake.tcl + +Running this Tcl script will automatically regenerate all makefiles. +In order to add a new source file to the Fossil implementation, simply +edit makemake.tcl to add the new filename, then rerun the script, and +all of the makefiles for all targets will be rebuilt. + +There is an optional code verification step implemented using + + 15. [/file/tools/codecheck1.c | codecheck1.c] + +This file implements a small utility program ("codecheck1") +that scans other Fossil source files looking for errors in printf-style +format strings. +The codecheck1 utility detects missing or surplus arguments on +printf-like functions and dangerous uses of "%s" that might +permit SQL injection or cross-site scripting attacks. This code +check step is run automatically on each build of Fossil, and can +also be run separately by typing "make codecheck". Note that the +built-in printf format checking of GCC does not function for Fossil +since Fossil implements its own printf (in the +[/file/src/printf.c | printf.c] source file) that includes special +features and formatting letters that are useful to Fossil. The +codecheck1 utility can be seen as an enhanced application-specific +replacement for the GCC printf format checker. + +Finally, there is one of the makefiles generated by makemake.tcl: + + 16. main.mk + +The main.mk makefile is invoked from the Makefile in the top-level +directory. The main.mk is generated by makemake.tcl and should not +be hand edited. Other makefiles generated by makemake.tcl are in +other subdirectories (currently all in the win/ subdirectory). + +All the other files in the src/ subdirectory (79 files at the time of +this writing) are C source code files that are subject to the +preprocessing steps described below. In the sequel, we will call these +other files "src.c" in order to have a convenient name. The reader +should understand that whenever "src.c" or "src.h" is used in the text +that follows, we really mean all (79) other source files other than +the exceptions described above. + +<h1>3.0 Automatically generated files</h1> + +The "VERSION.h" header file contains some C preprocessor macros that +identify the version of Fossil that is to be built. The VERSION.h file is +generated automatically from information extracted from the "manifest", +"manifest.uuid", and "VERSION" source files in the root directory of the +source tree. +(The "manifest" and "manifest.uuid" files are automatically generated and +updated by Fossil itself. See the [/help/setting | fossil set manifest] +command for additional information.) + +The VERSION.h header file is generated by +a C program: tools/mkversion.c. +To run the VERSION.h generator, first compile the tools/mkversion.c + source file into a command-line program (named "mkversion.exe") +then run: + +<pre> +mkversion.exe manifest.uuid manifest VERSION >VERSION.h +</pre> + +The pathnames in the above command might need to be adjusted to get the +directories right. The point is that the manifest.uuid, manifest, and +VERSION files +in the root of the source tree are the three arguments and +the generated VERSION.h file appears on standard output. + +The builtin_data.h header file is generated by a C program: tools/mkbuiltin.c. +The builtin_data.h file contains C-language byte-array definitions for +the content of resource files used by Fossil. To generate the +builtin_data.h file, first compile the mkbuiltin.c program, then run: + +<pre> +mkbuiltin.exe diff.tcl <i>OtherFiles...</i> >builtin_data.h +</pre> + +At the time of this writing, the "diff.tcl" script (a Tcl/Tk script used +to generate implement --tk option on the diff command) is the only resource +file processed using mkbuiltin.exe. However, new resources will likely be +added using this facility in future versions of Fossil. + +<h1 id="preprocessing">4.0 Preprocessing</h1> + +There are three preprocessors for the Fossil sources. The mkindex +and translate preprocessors can be run in any order. The makeheaders +preprocessor must be run after translate. + +<h2>4.1 The mkindex preprocessor</h2> + +The mkindex program scans the "src.c" source files looking for special +comments that identify routines that implement various Fossil commands, +web interface methods, and help text comments. The mkindex program +generates some C code that Fossil uses in order to dispatch commands and +HTTP requests and to show on-line help. Compile the mkindex program +from the mkindex.c source file. Then run: + +<pre> +./mkindex src.c >page_index.h +</pre> + +Note that "src.c" in the above is a stand-in for the (79) regular source +files of Fossil - all source files except for the exceptions described in +section 2.0 above. + +The output of the mkindex program is a header file that is #include-ed by +the main.c source file during the final compilation step. + +<h2>4.2 The translate preprocessor</h2> + +The translate preprocessor looks for lines of source code that begin +with "@" and converts those lines into string constants or (depending on +context) into special "printf" operations for generating the output of +an HTTP request. The translate preprocessor is a simple C program whose +sources are in the translate.c source file. The translate preprocess +is run on each of the other ordinary source files separately, like this: + +<pre> +./translate src.c >src_.c +</pre> + +In this case, the "src.c" file represents any single source file from the +set of ordinary source files as described in section 2.0 above. Note that +each source file is translated separately. By convention, the names of +the translated source files are the names of the input sources with a +single "_" character at the end. But a new makefile can use any naming +convention it wants - the "_" is not critical to the build process. + +After being translated, the output files (the "src_.c" files) should be +used for all subsequent preprocessing and compilation steps. + +<h2>4.3 The makeheaders preprocessor</h2> + +For each C source module "src.c", there is an automatically generated +header module "src.h" that contains all of the datatype and procedure +declarations needed by the source module. These header files are generated +automatically by the makeheaders program. The sources to makeheaders +are contained in a single file "makeheaders.c". Additional documentation +on makeheaders can be found in [../tools/makeheaders.html | tools/makeheaders.html]. + +The makeheaders program is run once. It scans all inputs source files and +generates header files for each one. Note that the sqlite3.c and shell.c +source files are not scanned by makeheaders. Makeheaders only runs over +"ordinary" source files, not the exceptional source files. However, +makeheaders also uses some extra header files as input. The general format +is like this: + +<pre> +makeheaders src_.c:src.h sqlite3.h th.h VERSION.h +</pre> + +In the example above the "src_.c" and "src.h" names represent all of the +(79) ordinary C source files, each as a separate argument. + +<h1>5.0 Compilation</h1> + +After all generated files have been created and all ordinary source files +have been preprocessed, the generated and preprocessed files can be +combined into a single executable using a C compiler. This can be done +all at once, or each preprocessed source file can be compiled into a +separate object code file and the resulting object code files linked +together in a final step. + +Some files require special C-preprocessor macro definitions. +When compiling sqlite.c, the following macros are recommended: + + * -DSQLITE_OMIT_LOAD_EXTENSION=1 + * -DSQLITE_ENABLE_DBSTAT_VTAB=1 + * -DSQLITE_ENABLE_FTS4=1 + * -DSQLITE_ENABLE_LOCKING_STYLE=0 + * -DSQLITE_LIKE_DOESNT_MATCH_BLOBS=1 + * -DSQLITE_THREADSAFE=0 + * -DSQLITE_DEFAULT_FILE_FORMAT=4 + * -DSQLITE_ENABLE_EXPLAIN_COMMENTS=1 + +The first three symbol definitions above are required; the others are merely +recommended. Extension loading is omitted as a security measure. The dbstat +virtual table is needed for the [/help/www/repo-tabsize|/repo-tabsize] page. +FTS4 is needed for the search feature. Fossil is single-threaded so mutexing +is disabled in SQLite as a performance enhancement. The +SQLITE_ENABLE_EXPLAIN_COMMENTS option makes the output of "EXPLAIN" queries +in the "[/help/sqlite3|fossil sql]" command much more readable. + +When compiling the shell.c source file, these macros are required: + + * -Dmain=sqlite3_main + * -DSQLITE_OMIT_LOAD_EXTENSION=1 + +The "main()" routine in the shell must be changed into sqlite3_main() +to prevent it from colliding with the real main() in Fossil, and to give +Fossil an entry point to jump to when the +[/help/sqlite3 | fossil sql] command is invoked. + +All the other source code files can be compiled without any special +options. + +<h1>6.0 Linkage</h1> + +Fossil needs to be linked against [http://www.zlib.net | zlib]. If +the HTTPS option is enabled, then it will also need to link against +the appropriate SSL implementation. And, of course, Fossil needs to +link against the standard C library. No other libraries or external +dependences are used. + +<h1>7.0 Debugging</h1> + +Debug mode is controlled via a FOSSIL_DEBUG preprocessor macro. This can be +set explicitly with the make command for the target platform. + +However, in practice it is instead recommended to add a respective configure +option for the target platform and then perform a clean build. This way the +Debug flags are consistently applied across the whole build process. For +example, use these Debug flags in addition to other flags passed to the +configure scripts: + +On Linux, *NIX and similar platforms: +<pre> +./configure --fossil-debug +</pre> + +On Windows: +<pre> +win\buildmsvc.bat FOSSIL_DEBUG=1 +</pre> + +The resulting fossil binary can then be loaded into a platform-specific +debugger. Source files displayed in the debugger correspond to the ones +generated from the translation stage of the build process, as that is what +was actually compiled into the respective object files that make up the +fossil binary. + +<h1>8.0 See Also</h1> + + * [./tech_overview.wiki | A Technical Overview Of Fossil] + * [./adding_code.wiki | How To Add Features To Fossil] diff --git a/www/mdtest/test1.md b/www/mdtest/test1.md new file mode 100644 index 0000000..598bc17 --- /dev/null +++ b/www/mdtest/test1.md @@ -0,0 +1,59 @@ +# Markdown Link-test + +This document exists solely as a test for some of the hyperlinking +capabilities of Markdown as implemented by Fossil. + +## Relative-Path Links + + * The index: [](../index.wiki) + + * Load management: [](../loadmgmt.md) + + * Site-map: [](../../../../sitemap) + + * Windows CGI: [](../server/windows/cgi.md) + +## The Magic $ROOT Path Prefix + +In text of the form `href="$ROOT/..."` in the HTML that markdown +generates, the $ROOT is replaced by the complete URI for the root +of the document tree. +Note that the $ROOT translation only occurs within the `<a href="...">` +element, not within the text of the hyperlink. So you should see the +$ROOT text on this page, but if you mouse-over the hyperlink the $ROOT +value should have been expanded to the actual document root. + + * Timeline: []($ROOT/timeline) + + * Site-map: []($ROOT/sitemap) + +The $ROOT prefix on markdown links is superfluous. The same link +works without the $ROOT prefix. (Though: the $ROOT prefix is required +for HTML documents.) + + * Timeline: [](/timeline) + + * Help: [](/help/help) + + * Site-map: [](/sitemap) + +## The Magic $CURRENT Document Version Translation + +In URI text of the form `.../doc/$CURRENT/...` the +$CURRENT value is converted to the version number of the document +currently being displayed. This conversion happens after translation +into HTML and only occurs on href='...' attributes so it does not occur +for plain text. + + * Document index: [](/doc/$CURRENT/www/index.wiki) + +Both the $ROOT and the $CURRENT conversions can occur on the same link. + + * Document index: []($ROOT/doc/$CURRENT/www/index.wiki) + +The translations must be contained within HTML markup in order to work. +They do not work for ordinary text that appears to be an href= attribute. + + * `x href='$ROOT/timeline'` + * `x action="$ROOT/whatever"` + * `x href="https://some-other-site.com/doc/$CURRENT/tail"` diff --git a/www/mirrorlimitations.md b/www/mirrorlimitations.md new file mode 100644 index 0000000..89a56ab --- /dev/null +++ b/www/mirrorlimitations.md @@ -0,0 +1,112 @@ +# Limitations On Git Mirrors + +The "<tt>[fossil git export](/help/git)</tt>" command can be used to +mirror a Fossil repository to Git. +([Setup instructions](./mirrortogithub.md) and an +[example](https://github.com/drhsqlite/fossil-mirror).) +But the export to Git is not perfect. Some information is lost during +export due to limitations in Git. This page describes what content of +Fossil is not included in an export to Git. + +## (1) Wiki, Tickets, Technotes, Forum + +Git only supports version control. The additional features of Fossil such +as Wiki, Tickets, Technotes, and the Forum are not supported in Git, +so those features are not included in an export. + +Third-party Git based tooling may add some of these features (e.g. +GitHub, GitLab) but because their data are not stored in the Git +repository, there is no single destination for Fossil to convert its +equivalent data *to*. For instance, Fossil tickets do not become GitHub +issues, because that is a proprietary feature of GitHub separate from +Git proper, stored outside the repository on the GitHub servers. + +You can also see the problem in its inverse case: you do not get a copy +of your GitHub issues when cloning the Git repository. You *do* get the +Fossil tickets, wiki, forum posts, etc. when cloning a remote Fossil +repo. + +## (2) Cherrypick Merges + +The Git client supports cherrypick merges but does not record the +cherrypick parent(s). + +Fossil tracks cherrypick merges in its repository and displays +cherrypicks in its timeline. (As an example, the dashed lines +[here](/timeline?c=0a9f12ce6655b7a5) are cherrypicks.) Because Git does +not have a way to represent this same information in its repository, the +history of Fossil cherrypicks cannot be exported to Git, only their +direct effects on the managed file data. + +## (3) Named Branches + +Git has only limited support for named branches. Git identifies the head +check-in of each branch. Depending on the check-in graph topology, this +is sufficient to infer the branch for many historical check-ins as well. +However, complex histories with lots of cross-merging +can lead to ambiguities. Fossil keeps +track of historical branch names unambiguously, +but the extra details about branch names that Fossil keeps +at hand cannot be exported to Git. + +An example of the kinds of ambiguities that arise when branch names +are not tracked is a "diamond-merge" history. In a diamond-merge, a +long-running development branch merges enhancements from trunk from +time to time and also periodically merges the development changes back +to trunk at moments when the branch is stable. +An example of diamond-merge in the Fossil source tree itself +can be seen at on the [bv-corrections01 branch](/timeline?r=bv-corrections01). +The distinction between checkins on the branch and checkins on trunk would +be lost in Git, which does not track branches for individual checkins, +and so you cannot (easily) tell which checkins are part of the branch and +which are part of trunk in a diamond-merge history on Git. For that +reason, diamond-merge histories are considered an anti-pattern in Git +and the usual recommendation for Git users is to employ +[rebase](./rebaseharm.md) to clean the history up. The point here is +that if your project has a diamond-merge history that shows up cleanly +in Fossil, it will export to Git and still be technically correct, but +the history display might be a jumbled mess that is difficult for humans +to comprehend. + +## (4) Non-unique Tags + +Git requires tags to be unique: each tag must refer to exactly one +check-in. Fossil does not have this restriction, and so it is common +in Fossil to tag multiple check-ins with the same name. For example, +it is common in Fossil to tag each check-in creating a release both +with a unique version tag *and* a common tag like "release" +so that all historical releases can be found at once. +([Example](/timeline?t=release).) + +Git does not allow this. The "release" tag must refer to just one +check-in. The work-around is that the non-unique tag in the Git export is +made to refer to only the most recent check-in with that tag. + +This is why the ["release" tag view][ghrtv] in the GitHub mirror of this +repository shows only the latest release version; contrast the prior +example. Both URLs are asking the repository the same question, but +because of Git's relatively impoverished data model, it cannot give the +same answer that Fossil does. + +[ghrtv]: https://github.com/drhsqlite/fossil-mirror/tree/release + +## (5) Amendments To Check-ins + +Check-ins are immutable in both Fossil and Git. +However, Fossil has a mechanism by which tags can be added to +its repository to provide after-the-fact corrections to prior check-ins. + +For example, tags can be added to check-ins that correct typos in the +check-in comment. The original check-in is immutable and so the +original comment is preserved in addition to the correction. But +software that displays the check-ins knows to look for the comment-change +tag and if present displays the corrected comment rather than the original. +([Example](/info/8ed91bbe44d0d383) changing the typo "os" into "so".) + +Git has no mechanism for providing corrections or clarifications to +historical check-ins. + +When exporting from Fossil to Git, the latest corrections to a Fossil check-in +are used to generate the corresponding Git check-in. But once the Git +check-in has been created, any subsequent corrections are omitted as there +is no way to transfer them to Git. diff --git a/www/mirrortogithub.md b/www/mirrortogithub.md new file mode 100644 index 0000000..9360ca6 --- /dev/null +++ b/www/mirrortogithub.md @@ -0,0 +1,166 @@ +# How To Mirror A Fossil Repository On GitHub + +Beginning with Fossil version 2.9, you can mirror a Fossil-based +project on GitHub (with [limitations](./mirrorlimitations.md)) +by following these steps: + +1. Create an account on GitHub if you do not have one already. Log + into that account. + +2. Create a new project. GitHub will ask you if you want to prepopulate + your project with various things like a README file. Answer "no" to + everything. You want a completely blank project. GitHub will then + supply you with a URL for your project that will look something + like this: + + https://github.com/username/project.git + +3. Back on your workstation, move to a checkout for your Fossil + project and type: + + <blockquote> + <pre> + $ fossil git export /path/to/git/repo --autopush \ + https://<font color="orange">username</font>:<font color="red">password</font>@github.com/username/project.git + </pre> + </blockquote> + + In place of the <code>/path/to...</code> argument above, put in + some directory name that is <i>outside</i> of your Fossil checkout. If + you keep multiple Fossil checkouts in a directory of their own, + consider using <code>../git-mirror</code> to place the Git export + mirror alongside them, for example. Fossil will create this + directory if necessary. This directory will become a Git + repository that holds a translation of your Fossil repository. + + The <code>--autopush</code> option tells Fossil that you want to + push the Git translation up to GitHub every time it is updated. + + The URL parameter is the same as the one GitHub gave you, but with + your GitHub <font color="orange">username</font> and <font + color="red">password</font> added. + + If your GitHub account uses two-factor authentication (2FA), you + will have to <a href="https://github.com/settings/tokens">generate + a personal access token</a> and use that in place of your actual + password in the URL. This token should have “repo” scope enabled, + only. + + You can also run the command above outside of any open checkout of + your project by supplying the “<code>-R repository</code>” + option. + +4. Get some coffee. Depending on the size of your project, the + initial "<code>fossil git export</code>" command in the previous + step might run for several minutes. + +5. And you are done! Assuming everything worked, your project is now + mirrored on GitHub. + +6. Whenever you update your project, simply run this command to update + the mirror: + + $ fossil git export + + Unlike with the first time you ran that command, you don’t need + the remaining arguments, because Fossil remembers those things. + Subsequent mirror updates should usually happen in a fraction of + a second. + +7. To see the status of your mirror, run: + + $ fossil git status + +## Notes: + + * Unless you specify --force, the mirroring only happens if the Fossil + repo has changed, with Fossil reporting "no changes", because Fossil + does not care about the success or failure of the mirror run. If a mirror + run failed (for example, due to an incorrect password, or a transient + error at github.com), Fossil will not retry until there has been a repo + change or --force is supplied. + + * The mirroring is one-way. If you check in changes on GitHub, those + changes will not be reabsorbed by Fossil. There are technical problems + that make a two-way mirror all but impossible. (This is not to be + confused with the ability to import a Fossil mirror from Github back + into a Fossil repository. That works, but it is not a mirror.) + + This also means that you cannot accept pull requests on GitHub. + + * The "`fossil git export`" command creates subprocesses that run "`git`" + commands, so you must have Git installed on your machine for any + of this to work. + + * The Git repository will have an extra unmanaged top-level directory named + "`.mirror_state`" that contains one or more files. Those files are + used to store the intermediate state of the translation so that + subsequent invocations of "`fossil git export`" will know where you + left off the last time and what new content needs to be moved over into + Git. Be careful not to mess with the `.mirror_state` directory or + any of its contents. Do not put those files under Git management. Do + not edit or delete them. + + * The name of the "trunk" branch is automatically translated into "master" + in the Git mirror unless you give the `--mainbranch` option. + + * Only check-ins and simple tags are translated to Git. Git does not + support wiki or tickets or unversioned content or any of the other + features of Fossil that make it so convenient to use, so those other + elements cannot be mirrored in Git. + + * In Git, all tags must be unique. If your Fossil repository has the + same tag on two or more check-ins, the tag will only be preserved on + the chronologically newest check-in. + + * There is a + [long list of restrictions](https://git-scm.com/docs/git-check-ref-format) + on tag and branch names in Git. If any of your Fossil tag or branch names + violate these rules, then the names are translated prior to being exported + to Git. The translation usually involves converting the offending characters + into underscores. + + * If your Fossil user contact info is not set and this repository was not + initially [imported from Git](./inout.wiki), `fossil git export` will + construct a generic `user@noemail.net` for the Git *committer* and *author* + email fields of each commit. However, Fossil will first attempt to parse an + email address from your user contact info, which can be set through a + Fossil [UI][ui] browser window or with the [`user contact`][usercmd] + subcommand on the command line. Alternatively, if this repository was + previously imported from Git using the [`--attribute`][attr] option, the + [`fx_git`][fxgit] table will be queried for correspondent email addresses. + Only if neither of these methods produce a user specified email will the + abovementioned generic address be used. + +[attr]: /help/import +[fxgit]: ./inout.wiki#fx_git +[ui]: /help/ui +[usercmd]: /help/user + + +## <a id='ex1'></a>Example GitHub Mirrors + +As of this writing (2019-03-16) Fossil’s own repository is mirrored +on GitHub at: + +> <https://github.com/drhsqlite/fossil-mirror> + +In addition, an official Git mirror of SQLite is available: + +> <https://github.com/sqlite/sqlite> + +The Fossil source repositories for these mirrors are at +<https://www2.fossil-scm.org/fossil> and <https://www2.sqlite.org/src>, +respectively. Both repositories are hosted on the same VM at +[Linode](https://www.linode.com). On that machine, there is a +[cron job](https://linux.die.net/man/8/cron) +that runs at 17 minutes after the hour, every hour that does: + + /usr/bin/fossil sync -u -R /home/www/fossil/fossil.fossil + /usr/bin/fossil sync -R /home/www/fossil/sqlite.fossil + /usr/bin/fossil git export -R /home/www/fossil/fossil.fossil + /usr/bin/fossil git export -R /home/www/fossil/sqlite.fossil + +The initial two "sync" commands pull in changes from the primary +Fossil repositories for Fossil and SQLite. The last two lines +export the changes to Git and push the results up to GitHub. diff --git a/www/mkindex.tcl b/www/mkindex.tcl new file mode 100755 index 0000000..a8d2bef --- /dev/null +++ b/www/mkindex.tcl @@ -0,0 +1,187 @@ +#!/usr/bin/env tclsh +# +# Run this TCL script to generate a WIKI page that contains a +# permuted index of the various documentation files. +# +# tclsh mkindex.tcl +# +# 2021-02-26: The permuted index feature has been removed because +# moderns don't understand such things, and seeing so many entries +# confuses them. +# + +set doclist { + aboutcgi.wiki {How CGI Works In Fossil} + aboutdownload.wiki {How The Download Page Works} + adding_code.wiki {Adding New Features To Fossil} + adding_code.wiki {Hacking Fossil} + alerts.md {Email Alerts And Notifications} + antibot.wiki {Defense against Spiders and Robots} + backoffice.md {The "Backoffice" mechanism of Fossil} + backup.md {Backing Up a Remote Fossil Repository} + blame.wiki {The Annotate/Blame Algorithm Of Fossil} + blockchain.md {Is Fossil A Blockchain?} + branching.wiki {Branching, Forking, Merging, and Tagging} + bugtheory.wiki {Bug Tracking In Fossil} + build.wiki {Compiling and Installing Fossil} + cap-theorem.md {Fossil and the CAP Theorem} + caps/ {Administering User Capabilities (a.k.a. Permissions)} + caps/admin-v-setup.md {Differences Between Setup and Admin Users} + caps/ref.html {User Capability Reference} + cgi.wiki {CGI Script Configuration Options} + changes.wiki {Fossil Changelog} + chat.md {Fossil Chat} + checkin_names.wiki {Check-in And Version Names} + checkin.wiki {Check-in Checklist} + childprojects.wiki {Child Projects} + chroot.md {Server Chroot Jail} + ckout-workflows.md {Check-Out Workflows} + co-vs-up.md {Checkout vs Update} + colordiff.md {Colorized Diffs} + copyright-release.html {Contributor License Agreement} + concepts.wiki {Fossil Core Concepts} + contact.md {Developer Contact Information} + containers.md {OCI Containers} + contribute.wiki {Contributing Code or Documentation To The Fossil Project} + css-tricks.md {Fossil CSS Tips and Tricks} + customgraph.md {Theming: Customizing the Timeline Graph} + customskin.md {Theming: Customizing The Appearance of Web Pages} + customskin.md {Custom Skins} + custom_ticket.wiki {Customizing The Ticket System} + defcsp.md {The Default Content Security Policy} + delta-manifests.md {Delta Manifests} + delta_encoder_algorithm.wiki {Fossil Delta Encoding Algorithm} + delta_format.wiki {Fossil Delta Format} + embeddeddoc.wiki {Embedded Project Documentation} + encryptedrepos.wiki {How To Use Encrypted Repositories} + env-opts.md {Environment Variables and Global Options} + event.wiki {Events} + faq.wiki {Frequently Asked Questions} + fileedit-page.md {The fileedit Page} + fileformat.wiki {Fossil File Format} + fiveminutes.wiki {Up and Running in 5 Minutes as a Single User} + forum.wiki {Fossil Forums} + foss-cklist.wiki {Checklist For Successful Open-Source Projects} + fossil-from-msvc.wiki {Integrating Fossil in the Microsoft Express 2010 IDE} + fossil-is-not-relational.md {Introduction to the (Non-relational) Fossil Data Model} + fossil_prompt.wiki {Fossilized Bash Prompt} + fossil-v-git.wiki {Fossil Versus Git} + gitusers.md {Git to Fossil Translation Guide} + globs.md {File Name Glob Patterns} + glossary.md {Glossary} + grep.md {Fossil grep vs POSIX grep} + hacker-howto.wiki {Hacker How-To} + hacker-howto.wiki {Fossil Developers Guide} + hashes.md {Hashes: Fossil Artifact Identification} + hashpolicy.wiki {Hash Policy: Choosing Between SHA1 and SHA3-256} + /help {Lists of Commands and Webpages} + hints.wiki {Fossil Tips And Usage Hints} + history.md {The Purpose And History Of Fossil} + hundredandone.md {101 Reasons Why Fossil Is Better Than Git} + index.wiki {Home Page} + inout.wiki {Import And Export To And From Git} + interwiki.md {Interwiki Links} + image-format-vs-repo-size.md {Image Format vs Fossil Repo Size} + javascript.md {Use of JavaScript in Fossil} + json-api/index.md {JSON API} + loadmgmt.md {Managing Server Load} + makefile.wiki {The Fossil Build Process} + mirrorlimitations.md {Limitations On Git Mirrors} + mirrortogithub.md {How To Mirror A Fossil Repository On GitHub} + /md_rules {Markdown Formatting Rules} + newrepo.wiki {How To Create A New Fossil Repository} + patchcmd.md {The "fossil patch" Command} + password.wiki {Password Management And Authentication} + pikchr.md {The Pikchr Diagram Language} + pop.wiki {Principles Of Operation} + private.wiki {Creating, Syncing, and Deleting Private Branches} + qandc.wiki {Questions And Criticisms} + quickstart.wiki {Fossil Quick Start Guide} + quotes.wiki + {Quotes: What People Are Saying About Fossil, Git, and DVCSes in General} + ../test/release-checklist.wiki {Pre-Release Testing Checklist} + rebaseharm.md {Rebase Considered Harmful} + reviews.wiki {Reviews} + selfcheck.wiki {Fossil Repository Integrity Self Checks} + selfhost.wiki {Fossil Self Hosting Repositories} + server/ {How To Configure A Fossil Server} + serverext.wiki {CGI Server Extensions} + serverext.wiki {Adding Extensions To A Fossil Server Using CGI Scripts} + settings.wiki {Fossil Settings} + /sitemap {Site Map} + shunning.wiki {Shunning: Deleting Content From Fossil} + stats.wiki {Performance Statistics} + style.wiki {Source Code Style Guidelines} + ssl.wiki {Using SSL with Fossil} + ssl-server.md {SSL/TLS Server Mode} + sync.wiki {The Fossil Sync Protocol} + tech_overview.wiki {A Technical Overview Of The Design And Implementation + Of Fossil} + tech_overview.wiki {SQLite Databases Used By Fossil} + th1.md {The TH1 Scripting Language} + theory1.wiki {Thoughts On The Design Of The Fossil DVCS} + tickets.wiki {The Fossil Ticket System} + unvers.wiki {Unversioned Files} + webpage-ex.md {Webpage Examples} + webui.wiki {The Fossil Web Interface} + whyallinone.md {Why Bundle Forum, Wiki, and other Web Software With Your DVCS?} + whyusefossil.wiki {Why You Should Use Fossil} + whyusefossil.wiki {Benefits Of Version Control} + wikitheory.wiki {Wiki In Fossil} + /wiki_rules {Wiki Formatting Rules} +} + +set permindex {} +set stopwords { + a about against and are as by for fossil from in of on or should the to use + used with +} +foreach {file title} $doclist { + set n [llength $title] + regsub -all {\s+} $title { } title + lappend permindex [list $title $file 1] + +# Disable the permutations. +# for {set i 0} {$i<$n-1} {incr i} { +# set prefix [lrange $title 0 $i] +# set suffix [lrange $title [expr {$i+1}] end] +# set firstword [string tolower [lindex $suffix 0]] +# if {[lsearch $stopwords $firstword]<0} { +# lappend permindex [list "$suffix — $prefix" $file 0] +# } +# } +} +set permindex [lsort -dict -index 0 $permindex] +set out [open permutedindex.html w] +fconfigure $out -encoding utf-8 -translation lf +puts $out \ +"<div class='fossil-doc' data-title='Index Of Fossil Documentation'>" +puts $out { +<form action='$ROOT/docsrch' method='GET' style="text-align:center"> +<input type="text" name="s" size="40" autofocus> +<input type="submit" value="Search Docs"> +</form> +<h2>Primary Documents:</h2> +<ul> +<li> <a href='quickstart.wiki'>Quick-start Guide</a> +<li> <a href='$ROOT/help'>Built-in help for commands and webpages</a> +<li> <a href='history.md'>Purpose and History of Fossil</a> +<li> <a href='build.wiki'>Compiling and installing Fossil</a> +<li> <a href='../COPYRIGHT-BSD2.txt'>License</a> +<li> <a href='userlinks.wiki'>Miscellaneous Docs for Fossil Users</a> +<li> <a href='hacker-howto.wiki'>Fossil Developer's Guide</a> +<li><a href='$ROOT/wiki?name=Release Build How-To'>Release Build How-To</a>, +a.k.a. how deliverables are built</li> +</li> +<li> <a href='$ROOT/wiki?name=To+Do+List'>To Do List (Wiki)</a> +<li> <a href='https://fossil-scm.org/fossil-book/'>Fossil book</a> +</ul> +<h2 id="pindex">Other Documents:</h2> +<ul>} +foreach entry $permindex { + foreach {title file bold} $entry break +# if {$bold} {set title <b>$title</b>} + if {[string match /* $file]} {set file ../../..$file} + puts $out "<li><a href=\"$file\">$title</a></li>" +} +puts $out "</ul></div>" diff --git a/www/newrepo.wiki b/www/newrepo.wiki new file mode 100644 index 0000000..fef3716 --- /dev/null +++ b/www/newrepo.wiki @@ -0,0 +1,163 @@ +<title>How To Create A New Fossil Repository</title> + +The [/doc/tip/www/quickstart.wiki|quickstart guide] explains how +to get up and running with fossil. But once you're running, what can +you do with it? This document will walk you through the process of +creating a fossil repository, populating it with files, and then +sharing it over the web. + +The first thing we need to do is create a fossil repository file: + +<verbatim> +$ fossil new demo.fossil +project-id: 9d8ccff5671796ee04e60af6932aa7788f0a990a +server-id: 145fe7d71e3b513ac37ac283979d73e12ca04bfe +admin-user: stephan (initial password is ******) +</verbatim> + +The numbers it spits out are unimportant (they are version +numbers). + +Now we have an empty repository file named <tt>demo.fossil</tt>. +There is nothing magical about the extension <tt>.fossil</tt> - it's +just a convention. You may name your files anything you like. + +The first thing we normally want to do is to run fossil as a local server so +that you can configure the access rights to the repo: + +<verbatim> +$ fossil ui demo.fossil +</verbatim> + +The <tt>ui</tt> command starts up a server (with an optional <tt>-port +NUMBER</tt> argument) and launches a web browser pointing at the +fossil server. From there it takes just a few moments to configure the +repo. Most importantly, go to the Admin menu, then the Users link, and +set your account name and password, and grant your account all access +privileges. (I also like to grant Clone access to the anonymous user, +but that's personal preference.) + +Once you are done, kill the fossil server (with Ctrl-C or equivalent) +and close the browser window. + +<div class="sidebar"> +It is not strictly required to configure a repository +this way, but if you are going to share a repo over the net then it +is highly recommended. If you are only going to work with the repo +locally, you can skip the configuration step and do it later if +you decide you want to share your repo. +</div> + +The next thing we need to do is <em>open</em> the repository. To do so +we create a working directory and then <tt>cd</tt> to it: + +<verbatim> +$ mkdir demo +$ cd demo +$ fossil open ../demo.fossil +</verbatim> + +That creates a file called <tt>_FOSSIL_</tt> in the current +directory, and this file contains all kinds of fossil-related +information about your local repository. Under Linux, the BSDs or +macOS, this will instead be called <tt>.fslckout</tt>. You can ignore it +for all purposes, but be sure not to accidentally remove it +or otherwise damage it - it belongs to fossil, not you. + +The next thing we need to do is add files to our repository. As it +happens, we have a few C source files lying around, which we'll +simply copy into our working directory. + +<verbatim> +$ cp ../csnip/*.{c,h} . +$ ls +clob.c clob.h clobz.c mkdep.c test-clob.c +tokenize_path.c tokenize_path.h vappendf.c vappendf.h +</verbatim> + +Fossil doesn't know about those files yet. Telling fossil about +a new file is a two-step process. First we <em>add</em> the file +to the repository, then we <em>commit</em> the file. This is a familiar +process for anyone who's worked with SCM systems before: + +<verbatim> +$ fossil add *.{c,h} +$ fossil commit -m "egg" +New_Version: d1296b4a08b9f8b943bb6c73698e51eed23f8f91 +</verbatim> + +We now have a working repository! The file <tt>demo.fossil</tt> +is the central storage, and we can share it amongst an arbitrary +number of trees. As a silly example: + +<verbatim> +$ cd ~/fossil +$ mkdir demo2 +$ cd demo2 +$ fossil open ../demo.fossil +ADD clob.c +ADD clob.h +ADD clobz.c +ADD mkdep.c +ADD test-clob.c +ADD tokenize_path.c +ADD tokenize_path.h +ADD vappendf.c +</verbatim> + +You may modify the repository (e.g. add, remove, or commit files) from +both working directories, and doing so might be useful when working on +a branch or experimental code. + +Making your repository available over the web is trivial to do. We +assume you have some web space where you can store your fossil file +and run a CGI script. If not, then this option is not for you. If +you do, then here's how... + +Copy the fossil repository file to your web server (it doesn't matter +where, really, but it "should" be unreachable by web browser traffic). + +In your <tt>cgi-bin</tt> (or equivalent) directory, create a file +which looks like this: + +<verbatim> +#!/path/to/fossil +repository: /path/to/my_repo.fossil +</verbatim> + +Make that script executable, and you're all ready to go: + +<verbatim> +$ chmod +x ~/www/cgi-bin/myrepo.cgi +</verbatim> + +Now simply point your browser to +<tt>https://my.domain/cgi-bin/myrepo.cgi</tt> and you should +be able to manage the repository from there. + +To check out a copy of your remote repository, use the +<em>clone</em> command: + +<verbatim> +$ fossil clone \ + https://MyAccountName:MyAccountPassword@my.domain/cgi-bin/myrepo.cgi +</verbatim> + +If you do not provide your password in the URL, fossil will +interactively prompt you for it. + +A clone is a local copy of a remote repository, and can be opened just +like a local one (as shown above). It is treated identically to your +local repository, with one very important difference. When you commit +changes to a cloned remote repository, they will be pushed back to the +remote repository. If you have <tt>autosync</tt> on then this sync +happens automatically, otherwise you will need to use the +<em>pull</em> command to get remote changes and the <em>push</em> +command to push your local commits to the remote repository. You must +of course have authorization to commit changes (access is configured +via the Admin/Users page mentioned above). + +You may always use the <em>server</em> or <em>ui</em> commands to +browse a cloned repository. You can even edit create or wiki entries, +etc., and they will be pushed to the remote side the next time you +push data to the remote. diff --git a/www/password.wiki b/www/password.wiki new file mode 100644 index 0000000..75258f4 --- /dev/null +++ b/www/password.wiki @@ -0,0 +1,143 @@ +<title>Fossil Password Management</title> + +Fossil handles user authentication using passwords. +Passwords are unique to each repository. Passwords are not part of the +persistent state of a project. Passwords are not versioned and +are not transmitted from one repository to another during a sync. +Passwords are local configuration information that can (and usually does) +vary from one repository to the next within the same project. + +Passwords are stored in the PW field of the USER table. +In older versions of Fossil (prior to +[/timeline?c=2010-01-10T20:56:30 | 2010-01-11]) the password +is stored as cleartext. In newer versions of Fossil, the password +can be either cleartext or an SHA1 hash (written as a 40-character +lower-case hexadecimal number). If the USER.PW field contains +a 40-character string, that string is assumed to be a SHA1 hash. +If the size of USER.PW is anything other than 40 characters, then +it is understood as a plain-text password. + +The SHA1 hash in the USER.PW field is a hash of a string composed of +the project-code, the user login, and the user cleartext password. +Suppose user "alice" with password "asdfg" had an account on the +Fossil self-hosting repository. Then the value of USER.PW +for alice would be the SHA1 hash of + +<pre> +CE59BB9F186226D80E49D1FA2DB29F935CCA0333/alice/asdfg +</pre> + +Note that by including the project-code and the login as part of the +hash, a different USER.PW value results even if two or more users on +the repository select the same "asdfg" password or if user alice +reuses the same password on multiple projects. + +Whenever a password is changed using the web interface or using the +"user" command-line method, the new password is stored using the SHA1 +encoding. Thus, cleartext passwords will gradually migrate to become +SHA1 passwords. All remaining cleartext passwords can be converted to +SHA1 passwords using the following command: + +<pre> +fossil test-hash-passwords <i>REPOSITORY-NAME</i> +</pre> + +Remember that converting from cleartext to SHA1 passwords is an +irreversible operation. + +The only way to insert a new cleartext password into the USER table +is to do so manually using SQL commands. For example: + +<pre> +UPDATE user SET pw='asdfg' WHERE login='alice'; +</pre> + +Note that an password that is an empty string or NULL will disable +all login for that user. Thus, to lock a user out of the system, +one has only to set their password to an empty string, using either +the web interface or direct SQL manipulation of the USER table. +Note also that the password field is +essentially ignored for the special users named "anonymous", "developer", +"reader", and "nobody". It is not possible to authenticate as users +"developer", "reader", or "nobody" and the authentication protocol +for "anonymous" uses one-time captchas not persistent passwords. + +<h2>Web Interface Authentication</h2> + +When a user logs into Fossil using the web interface, the login name +and password are sent in the clear to the server. For most modern fossil +server setups with [/help/redirect-to-https|redirect-to-https] enabled, +this will be protected by the +SSL connection over HTTPS so it cannot be easily viewed. The server then +hashes the password and compares it against the value stored in USER.PW. +If they match, the server sets a cookie on the client to record the +login. This cookie contains a large amount of high-quality randomness +and is thus intractable to guess. The value of the cookie and the IP +address of the client is stored in the USER.COOKIE and USER.IPADDR fields +of the USER table on the server. + +The USER.CEXPIRE field holds an expiration date +for the cookie, encoded as a Julian day number. On all subsequent +HTTP requests, the cookie value is matched against the USER table to +enable access to the repository. + +Note that in order to log into a Fossil server, it is necessary to +write information into the repository database. Hence, login is not +possible on a Fossil repository with a read-only database file. + +The user password is sent over the wire as cleartext on the initial +login attempt. The plan moving forward is to compute the SHA1 hash of +the password on the client using JavaScript and then send only the hash +over the wire, but that plan has not yet been set in code. + +<h2>Sync Protocol Authentication</h2> + +A different authentication mechanism is used when one repository wants +to sync (or push or pull or clone) another repository. When two +repositories are syncing, the one that initiates the transaction is +the client and the repository that responds is the server. The client +works by sending HTTP requests to the server with a method of "xfer" +and a content-type of "application/x-fossil". The content is Zlib-compressed +text consisting of "cards" of instructions. The first card of this content +is a "login" card responsible for authentication. The login card contains +the login name of the user and a "signature" where the signature is the +SHA1 hash of a nonce and the value of USER.PW. The nonce is the +SHA1 hash of the remainder of the request content after the newline +(ASCII 14) character that terminates the login card. + +Using this approach, the USER.PW value is treated as a shared secret +between the client and server. The USER.PW value is never sent over +the wire, but the protocol establishes that both client and server know +the value of USER.PW. Furthermore, the use of a SHA1 hash over the entire +message prevents an attacker from sniffing a valid login from a legitimate +users and then replaying the message modified content. + +If the USER.PW on the server holds a cleartext password, then the +server will also accept a login-card signature that is constructed +using either the cleartext password or the SHA1 hash of the password. +This means that when USER.PW holds a cleartext password, the login card +will work for both older and newer clients. If the USER.PW on the server +only holds the SHA1 hash of the password, then only newer clients will be +able to authenticate to the server. + +The client normally gets the login and password from the "remote URL". + +<pre> +http://<span style="color:blue">login:password</span>@servername.org/path +</pre> + +For older clients, the password is used for the shared secret as stated +in the URL and with no encoding. +For newer clients, the shared secret is derived from the password +by transforming the password using the SHA1 hash encoding +described above. However, if the first character of the password is +"*" (ASCII 0x2a) then the "*" is skipped and the rest of the password +is used directly as the share secret without the SHA1 encoding. + +<pre> +http://<span style="color:blue">login:*password</span>@servername.org/path +</pre> + +This *-before-the-password trick can be used by newer clients to +sync against a legacy server that does not understand the new SHA1 +password encoding. diff --git a/www/patchcmd.md b/www/patchcmd.md new file mode 100644 index 0000000..f98a4a4 --- /dev/null +++ b/www/patchcmd.md @@ -0,0 +1,112 @@ +# The "fossil patch" command + +The "[fossil patch](/help/patch)" command is designed to transfer +uncommitted changes from one check-out to another, including transfering +those changes to other machines. + +For example, if you are working on a Windows desktop and you want to +test your changes on a Linux server before you commit, you can use the +"fossil patch push" command to make a copy of all your changes on the +remote Linux server: + + fossil patch push linuxserver:/path/to/checkout + +In the previous line "linuxserver" is the name of the remote machine and +"/path/to/checkout" is an existing checkout directory for the same project +on the remote machine. + +The "fossil patch push" command works by first creating a patch file, +then transfering that patch file to the remote machine using "ssh", then +applying the patch. If you do not have ssh available, you can break these +steps apart as follows: + + 1. On the local machine: `fossil patch create mypatch.patch` + 2. Move "mypatch.patch" to the remote machine. + 3. On the remote machine: `fossil patch apply mypatch.patch` + +Step 2 can be accomplished by a variety of means including +posting the mypatch.patch file on [chat](./chat.md) or sending +it as an email attachment. + +## Setup + +The "fossil patch push" and "fossil patch pull" commands will only work if you have +"ssh" available on the local machine and if "fossil" is on the default +PATH on the remote machine. + +To check if Fossil is installed correctly on the remote, try a command +like this: + + ssh -T remote "fossil version" + +If the command above shows a recent version of Fossil, then you should be +set to go. If you get "fossil not found", or if the version shown is too +old, put a newer fossil executable on the default PATH. The default PATH +can be shown using: + + ssh -T remote 'echo $PATH' + +### Custom PATH Caveat + +On Unix-like systems, the init script for the user's login shell +(e.g. `~/.profile` or `~/.bash_profile`) may be configured to *not do +anything* when running under a non-interactive shell. Thus a fossil +binary installed to a custom directory might not be found. To allow +the patch command to use a fossil binary installed in a directory +which is normally added to the PATH via the interactive shell's init +script, it may be useful to disable that check. For example, +Ubuntu-derived systems sometimes start their default `.bashrc` with +something like: + +``` +# If not running interactively, don't do anything: +[ -z "$PS1" ] && return +# Or: +case $- in + *i*) ;; + *) return;; +esac +``` + +Commenting out that check will allow the patch command to run, for +example, `~/bin/fossil` if `~/bin` is added to the PATH via the init +script. To disable that check *only* when the shell is *not* running +over an SSH connection, something like the following should suffice: + +``` +if [ -z "$SSH_CONNECTION" ]; then + # ... the is-interactive check goes here ... +fi +``` + + +## Implementation Details + +The "fossil patch create" command records all of the local, uncommitted +changes in an SQLite database file. If the argument to "fossil patch create" +is a filename, then the patch-file database is written into that file. +If the argument is "-" then the database is written on standard output. + +The "fossil patch apply" command reads the patch-file database +and applies it to the local check-out. If a filename is given as an +argument, then the database is read from that file. If the argument is "-" +then the database is read from standard input. + +Hence the command: + + fossil patch push remote:projectA + +Is equivalent to: + + fossil patch create - | ssh -T remote 'cd projectA;fossil patch apply -' + +Likewise, a command like this: + + fossil patch pull remote:projB + +could be entered like this: + + ssh -T remote 'cd projB;fossil patch create -' | fossil patch apply - + +The "fossil patch view" command just opens the patch-file database and prints +a summary of its contents on standard output. diff --git a/www/permutedindex.html b/www/permutedindex.html new file mode 100644 index 0000000..27ba154 --- /dev/null +++ b/www/permutedindex.html @@ -0,0 +1,140 @@ +<div class='fossil-doc' data-title='Index Of Fossil Documentation'> + +<form action='$ROOT/docsrch' method='GET' style="text-align:center"> +<input type="text" name="s" size="40" autofocus> +<input type="submit" value="Search Docs"> +</form> +<h2>Primary Documents:</h2> +<ul> +<li> <a href='quickstart.wiki'>Quick-start Guide</a> +<li> <a href='$ROOT/help'>Built-in help for commands and webpages</a> +<li> <a href='history.md'>Purpose and History of Fossil</a> +<li> <a href='build.wiki'>Compiling and installing Fossil</a> +<li> <a href='../COPYRIGHT-BSD2.txt'>License</a> +<li> <a href='userlinks.wiki'>Miscellaneous Docs for Fossil Users</a> +<li> <a href='hacker-howto.wiki'>Fossil Developer's Guide</a> +<li><a href='$ROOT/wiki?name=Release Build How-To'>Release Build How-To</a>, +a.k.a. how deliverables are built</li> +</li> +<li> <a href='$ROOT/wiki?name=To+Do+List'>To Do List (Wiki)</a> +<li> <a href='https://fossil-scm.org/fossil-book/'>Fossil book</a> +</ul> +<h2 id="pindex">Other Documents:</h2> +<ul> +<li><a href="hundredandone.md">101 Reasons Why Fossil Is Better Than Git</a></li> +<li><a href="tech_overview.wiki">A Technical Overview Of The Design And Implementation Of Fossil</a></li> +<li><a href="serverext.wiki">Adding Extensions To A Fossil Server Using CGI Scripts</a></li> +<li><a href="adding_code.wiki">Adding New Features To Fossil</a></li> +<li><a href="caps/">Administering User Capabilities (a.k.a. Permissions)</a></li> +<li><a href="backup.md">Backing Up a Remote Fossil Repository</a></li> +<li><a href="whyusefossil.wiki">Benefits Of Version Control</a></li> +<li><a href="branching.wiki">Branching, Forking, Merging, and Tagging</a></li> +<li><a href="bugtheory.wiki">Bug Tracking In Fossil</a></li> +<li><a href="cgi.wiki">CGI Script Configuration Options</a></li> +<li><a href="serverext.wiki">CGI Server Extensions</a></li> +<li><a href="checkin_names.wiki">Check-in And Version Names</a></li> +<li><a href="checkin.wiki">Check-in Checklist</a></li> +<li><a href="ckout-workflows.md">Check-Out Workflows</a></li> +<li><a href="foss-cklist.wiki">Checklist For Successful Open-Source Projects</a></li> +<li><a href="co-vs-up.md">Checkout vs Update</a></li> +<li><a href="childprojects.wiki">Child Projects</a></li> +<li><a href="colordiff.md">Colorized Diffs</a></li> +<li><a href="build.wiki">Compiling and Installing Fossil</a></li> +<li><a href="contribute.wiki">Contributing Code or Documentation To The Fossil Project</a></li> +<li><a href="copyright-release.html">Contributor License Agreement</a></li> +<li><a href="private.wiki">Creating, Syncing, and Deleting Private Branches</a></li> +<li><a href="customskin.md">Custom Skins</a></li> +<li><a href="custom_ticket.wiki">Customizing The Ticket System</a></li> +<li><a href="antibot.wiki">Defense against Spiders and Robots</a></li> +<li><a href="delta-manifests.md">Delta Manifests</a></li> +<li><a href="contact.md">Developer Contact Information</a></li> +<li><a href="caps/admin-v-setup.md">Differences Between Setup and Admin Users</a></li> +<li><a href="alerts.md">Email Alerts And Notifications</a></li> +<li><a href="embeddeddoc.wiki">Embedded Project Documentation</a></li> +<li><a href="env-opts.md">Environment Variables and Global Options</a></li> +<li><a href="event.wiki">Events</a></li> +<li><a href="globs.md">File Name Glob Patterns</a></li> +<li><a href="cap-theorem.md">Fossil and the CAP Theorem</a></li> +<li><a href="changes.wiki">Fossil Changelog</a></li> +<li><a href="chat.md">Fossil Chat</a></li> +<li><a href="concepts.wiki">Fossil Core Concepts</a></li> +<li><a href="css-tricks.md">Fossil CSS Tips and Tricks</a></li> +<li><a href="delta_encoder_algorithm.wiki">Fossil Delta Encoding Algorithm</a></li> +<li><a href="delta_format.wiki">Fossil Delta Format</a></li> +<li><a href="hacker-howto.wiki">Fossil Developers Guide</a></li> +<li><a href="fileformat.wiki">Fossil File Format</a></li> +<li><a href="forum.wiki">Fossil Forums</a></li> +<li><a href="grep.md">Fossil grep vs POSIX grep</a></li> +<li><a href="quickstart.wiki">Fossil Quick Start Guide</a></li> +<li><a href="selfcheck.wiki">Fossil Repository Integrity Self Checks</a></li> +<li><a href="selfhost.wiki">Fossil Self Hosting Repositories</a></li> +<li><a href="settings.wiki">Fossil Settings</a></li> +<li><a href="hints.wiki">Fossil Tips And Usage Hints</a></li> +<li><a href="fossil-v-git.wiki">Fossil Versus Git</a></li> +<li><a href="fossil_prompt.wiki">Fossilized Bash Prompt</a></li> +<li><a href="faq.wiki">Frequently Asked Questions</a></li> +<li><a href="gitusers.md">Git to Fossil Translation Guide</a></li> +<li><a href="glossary.md">Glossary</a></li> +<li><a href="hacker-howto.wiki">Hacker How-To</a></li> +<li><a href="adding_code.wiki">Hacking Fossil</a></li> +<li><a href="hashpolicy.wiki">Hash Policy: Choosing Between SHA1 and SHA3-256</a></li> +<li><a href="hashes.md">Hashes: Fossil Artifact Identification</a></li> +<li><a href="index.wiki">Home Page</a></li> +<li><a href="aboutcgi.wiki">How CGI Works In Fossil</a></li> +<li><a href="aboutdownload.wiki">How The Download Page Works</a></li> +<li><a href="server/">How To Configure A Fossil Server</a></li> +<li><a href="newrepo.wiki">How To Create A New Fossil Repository</a></li> +<li><a href="mirrortogithub.md">How To Mirror A Fossil Repository On GitHub</a></li> +<li><a href="encryptedrepos.wiki">How To Use Encrypted Repositories</a></li> +<li><a href="image-format-vs-repo-size.md">Image Format vs Fossil Repo Size</a></li> +<li><a href="inout.wiki">Import And Export To And From Git</a></li> +<li><a href="fossil-from-msvc.wiki">Integrating Fossil in the Microsoft Express 2010 IDE</a></li> +<li><a href="interwiki.md">Interwiki Links</a></li> +<li><a href="fossil-is-not-relational.md">Introduction to the (Non-relational) Fossil Data Model</a></li> +<li><a href="blockchain.md">Is Fossil A Blockchain?</a></li> +<li><a href="json-api/index.md">JSON API</a></li> +<li><a href="mirrorlimitations.md">Limitations On Git Mirrors</a></li> +<li><a href="../../../help">Lists of Commands and Webpages</a></li> +<li><a href="loadmgmt.md">Managing Server Load</a></li> +<li><a href="../../../md_rules">Markdown Formatting Rules</a></li> +<li><a href="containers.md">OCI Containers</a></li> +<li><a href="password.wiki">Password Management And Authentication</a></li> +<li><a href="stats.wiki">Performance Statistics</a></li> +<li><a href="../test/release-checklist.wiki">Pre-Release Testing Checklist</a></li> +<li><a href="pop.wiki">Principles Of Operation</a></li> +<li><a href="qandc.wiki">Questions And Criticisms</a></li> +<li><a href="quotes.wiki">Quotes: What People Are Saying About Fossil, Git, and DVCSes in General</a></li> +<li><a href="rebaseharm.md">Rebase Considered Harmful</a></li> +<li><a href="reviews.wiki">Reviews</a></li> +<li><a href="chroot.md">Server Chroot Jail</a></li> +<li><a href="shunning.wiki">Shunning: Deleting Content From Fossil</a></li> +<li><a href="../../../sitemap">Site Map</a></li> +<li><a href="style.wiki">Source Code Style Guidelines</a></li> +<li><a href="tech_overview.wiki">SQLite Databases Used By Fossil</a></li> +<li><a href="ssl-server.md">SSL/TLS Server Mode</a></li> +<li><a href="backoffice.md">The "Backoffice" mechanism of Fossil</a></li> +<li><a href="patchcmd.md">The "fossil patch" Command</a></li> +<li><a href="blame.wiki">The Annotate/Blame Algorithm Of Fossil</a></li> +<li><a href="defcsp.md">The Default Content Security Policy</a></li> +<li><a href="fileedit-page.md">The fileedit Page</a></li> +<li><a href="makefile.wiki">The Fossil Build Process</a></li> +<li><a href="sync.wiki">The Fossil Sync Protocol</a></li> +<li><a href="tickets.wiki">The Fossil Ticket System</a></li> +<li><a href="webui.wiki">The Fossil Web Interface</a></li> +<li><a href="pikchr.md">The Pikchr Diagram Language</a></li> +<li><a href="history.md">The Purpose And History Of Fossil</a></li> +<li><a href="th1.md">The TH1 Scripting Language</a></li> +<li><a href="customskin.md">Theming: Customizing The Appearance of Web Pages</a></li> +<li><a href="customgraph.md">Theming: Customizing the Timeline Graph</a></li> +<li><a href="theory1.wiki">Thoughts On The Design Of The Fossil DVCS</a></li> +<li><a href="unvers.wiki">Unversioned Files</a></li> +<li><a href="fiveminutes.wiki">Up and Running in 5 Minutes as a Single User</a></li> +<li><a href="javascript.md">Use of JavaScript in Fossil</a></li> +<li><a href="caps/ref.html">User Capability Reference</a></li> +<li><a href="ssl.wiki">Using SSL with Fossil</a></li> +<li><a href="webpage-ex.md">Webpage Examples</a></li> +<li><a href="whyallinone.md">Why Bundle Forum, Wiki, and other Web Software With Your DVCS?</a></li> +<li><a href="whyusefossil.wiki">Why You Should Use Fossil</a></li> +<li><a href="../../../wiki_rules">Wiki Formatting Rules</a></li> +<li><a href="wikitheory.wiki">Wiki In Fossil</a></li> +</ul></div> diff --git a/www/pikchr.md b/www/pikchr.md new file mode 100644 index 0000000..09e20d6 --- /dev/null +++ b/www/pikchr.md @@ -0,0 +1,139 @@ +# The Pikchr Diagram Language + +Pikchr (pronounced "picture") is a [PIC][1]-like markup language for creating +diagrams in technical documentation. Source text for Pikchr diagrams +can be embedded directly in either [Markdown][2] or [Fossil Wiki][3]. +Fossil translates the Pikchr source text into SVG which is displayed as +part of the rendered wiki. + +[1]: wikipedia:/wiki/Pic_language +[2]: /md_rules +[3]: /wiki_rules + +For example, this document is written in Markdown. The following +is a sample Pikchr diagram: + +``` pikchr +arrow right 200% "Markdown" "Source" +box rad 10px "Markdown" "Formatter" "(markdown.c)" fit +arrow right 200% "HTML+SVG" "Output" +arrow <-> down 70% from last box.s +box same "Pikchr" "Formatter" "(pikchr.c)" fit +``` + +The diagram above was generated by the following lines of Markdown: + +~~~~~ +``` pikchr +arrow right 200% "Markdown" "Source" +box rad 10px "Markdown" "Formatter" "(markdown.c)" fit +arrow right 200% "HTML+SVG" "Output" +arrow <-> down 70% from last box.s +box same "Pikchr" "Formatter" "(pikchr.c)" fit +``` +~~~~~ + +See the [original Markdown source text of this document][4] for an +example of Pikchr in operation. + +[4]: ./pikchr.md?mimetype=text/plain + +Fossil allows Pikchr diagrams to appear anywhere that Markdown or +Fossil Wiki markup or used, including: + + * [Embedded documentation](./embeddeddoc.wiki) + * Stand-alone wiki pages + * [Wiki pages associated with particular branches or check-ins](./wikitheory.wiki#assocwiki) + * Check-in comments + * [Technical notes](./event.wiki) + * [Forum posts](./forum.wiki) + * [Bug reports and trouble tickets](./bugtheory.wiki) + +## Pikchr Is A Separate Project + +Even though the original author of Pikchr is the same as the original +creator of Fossil, the sources to the Pikchr formatter are maintained +as a [separate project named "pikchr.org"](https://pikchr.org). +Pikchr is a delivered as a single file of C code. The "pikchr.c" file +from the Pikchr project is periodically copied into the Fossil source +tree. Pikchr is maintained as a project distinct from Fossil so that it +can be used independently of Fossil. + +### Pikchr User Manual And Tutorials + +Complete documentation on the Pikchr language can be found on the +Pikchr project page: + + * <https://pikchr.org/> + +That website contains a user manual, tutorials, a language specification, +a summary of differences between Pikchr and legacy PIC, +and it hosts copies of historical PIC documentation. + +## How To Include Pikchr Diagrams In Fossil Documents + +To illustrate how to include Pikchr in Fossil markup, we will use the +following one-line Pikchr. Click to see the code: + +~~~ pikchr toggle +arrow; box "Hello" "World!" fit; arrow +~~~ + +For Markdown, the Pikchr code is put inside of a +[fenced code block][fcb]. A fenced code block is the text in between +``` ... ``` or between +~~~ ... ~~~ using three or +more ` or ~ characters. The fenced code block normally +displays its content verbatim, but if an "info string" of "pikchr" +follows the opening ``` or ~~~, then the +content is interpreted as Pikchr script and is replaced by the +equivalent SVG. +So either of these work: + +[fcb]: https://spec.commonmark.org/0.29/#fenced-code-blocks + +~~~~~~ +~~~ pikchr +arrow; box "Hello" "World!" fit; arrow +~~~ + +``` pikchr +arrow; box "Hello" "World!" fit; arrow +``` +~~~~~~ + +For Fossil Wiki, the Pikchr code goes within +`<verbatim type="pikchr"> ... </verbatim>`. Normally `<verbatim>` +content is displayed verbatim. The extra `type="pikchr"` attribute +causes the content to be interpreted as Pikchr and replaced by SVG. + +~~~~~~ +<verbatim type="pikchr"> +arrow; box "Hello" "World!" fit; arrow +</verbatim> +~~~~~~ + +## Extra Arguments In "Pikchr" Code Blocks + +Extra formatting arguments can be included in the fenced code block start +tag, or in the "`type=`" attribute of `<verbatim>`, to change the formatting +of the diagram. + + * **indent** → The diagram is indented from the left margin. + + * **center** → The diagram is centered + + * **toggle** → Clicking on the diagram toggles between showing + the rendered SVG and the original Pikchr source text. You can always + do this by holding down the Ctrl or Alt keys and clicking. The + "toggle" option just means you can toggle without holding down any + extra keys. + + * **float-left** → The diagram is shown at the left margin and + text fills in around the diagram. + + * **float-right** → The diagram is shown at the right margin and + text fills in around the diagram. + + * **source** → The display starts out showing the Pikchr source text. + The reader must click (or Alt-click or Ctrl-click) to show the diagram. diff --git a/www/pop.wiki b/www/pop.wiki new file mode 100644 index 0000000..5617dd9 --- /dev/null +++ b/www/pop.wiki @@ -0,0 +1,72 @@ +<title>Principles Of Operation</title> + +This page attempts to define the foundational principles upon +which Fossil is built. + + * A project consists of source files, wiki pages, and + trouble tickets, and control files (collectively "artifacts"). + All historical copies of all artifacts + are saved. The project maintains an audit + trail. + + * A project resides in one or more repositories. Each + repository is administered and operates independently + of the others. + + * Each repository has both global and local state. The + global state is common to all repositories (or at least + has the potential to be shared in common when the + repositories are fully synchronized). The local state + for each repository is private to that repository. + The global state represents the content of the project. + The local state identifies the authorized users and + access policies for a particular repository. + + * The global state of a repository is an unordered + collection of artifacts. Each artifact is named by a + cryptographic hash (SHA1 or SHA3-256) encoded in + lowercase hexadecimal. + In many contexts, the name can be + abbreviated to a unique prefix. A five- or six-character + prefix usually suffices to uniquely identify a file. + + * Because artifacts are named by a cryptographic hash, all artifacts + are immutable. Any change to the content of an artifact also + changes the hash that forms the artifacts name, thus + creating a new artifact. Both the old original version of the + artifact and the new change are preserved under different names. + + * It is theoretically possible for two artifacts with different + content to share the same hash. But finding two such + artifacts is so incredibly difficult and unlikely that we + consider it to be an impossibility. + + * The signature of an artifact is the cryptographic hash of the + artifact itself, exactly as it would appear in a disk file. No prefix + or meta-information about the artifact is added before computing + the hash. So you can + always find the signature of a file by using the + "sha1sum" or "sha3sum" or similar command-line utilities. + + * The artifacts that comprise the global state of a repository + are the complete global state of that repository. The SQLite + database that holds the repository contains additional information + about linkages between artifacts, but all of that added information + can be discarded and reconstructed by rescanning the content + artifacts. + + * Two repositories for the same project can synchronize + their global states simply by sharing artifacts. The local + state of repositories is not normally synchronized or + shared. + + * Every check-in has a special file at the top-level + named "manifest" which is an index of all other files in + that check-in. The manifest is automatically created and + maintained by the system. + + * The <a href="fileformat.wiki">file formats</a> + used by Fossil are all very simple so that with access + to the original content files, one can easily reconstruct + the content of a check-in without the need for any + special tools or software. diff --git a/www/private.wiki b/www/private.wiki new file mode 100644 index 0000000..5bf5b07 --- /dev/null +++ b/www/private.wiki @@ -0,0 +1,121 @@ +<title>Private Branches</title> + +By default, everything you check into a Fossil repository is shared +to all clones of that repository. In Fossil, you don't push and pull +individual branches; you push and pull everything all at once. + +But sometimes users want to keep some private work that is not +shared with others. This might be a preliminary or experimental change +that needs further refinement before it is shared and which might never +be shared at all. To do this in Fossil, simply commit the change with +the --private command-line option: + +<pre> +fossil commit --private +</pre> + +The --private option causes Fossil to put the check-in in a new branch +named "private". That branch will not participate in subsequent clone, +sync, push, or pull operations. The branch will remain on the one local +repository where it was created. Note that you only use the --private +option for the first check-in that creates the private branch. +Additional checkins into the private branch remain private automatically. + +<h2>Publishing Private Changes</h2> + +After additional work, one might desire to publish the changes associated +with a private branch. The usual way to do this is to merge those +changes into a public branch. For example: + +<pre> +fossil update trunk +fossil merge private +fossil commit +</pre> + +The private branch remains private and is not recorded as a parent +in the merge manifest's P-card, but all of the changes associated with +the private branch are now folded into the public branch and are hence +visible to other users of the project. + +A private branch created with Fossil version 1.30 or newer can also be +converted into a public branch using the <code>fossil publish</code> +command. However, there is no way to convert a private branch created with +older versions of Fossil into a public branch. + +<div class="sidebar"> +To avoid generating a missing artifact +reference on peer repositories without the private branch, the merge parent +is not recorded when merging the private branch into a public branch. As a +consequence, the web UI timeline does not draw a merge line from the private +merge parent to the public merge child. Moreover, repeat private-to-public +merge operations (without the [/help/merge | --force option]) with files +added on the private branch may only work once, but later abort with +"WARNING: no common ancestor for FILE", as the parent-child relationship is +not recorded. (See the [/doc/trunk/www/branching.wiki | Branching, Forking, +Merging, and Tagging] document for more information.) +</div> + +The <code>--integrate</code> option of <code>fossil merge</code> (to close +the merged branch when committing) is ignored for a private branch -- or the +check-in manifest of the resulting merge child would include a +<code>+close</code> tag referring to the leaf check-in on the private branch, +and generate a missing artifact reference on repository clones without that +private branch. It's still possible to close the leaf of the private branch +(after committing the merge child) with the <code>fossil amend --close</code> +command. + +<h2>Syncing Private Branches</h2> + +A private branch normally stays on the one repository where it was +originally created. But sometimes you want to share private branches +with another repository. For example, you might be building a cross-platform +application and have separate repositories on your Windows laptop, +your Linux desktop, and your iMac. You can transfer private branches +between these machines by using the --private option on the "sync", +"push", "pull", and "clone" commands. For example, if you are running +"fossil server" on your Linux box and you want to clone that repository +to your Mac, including all private branches, use: + +<verbatim> +fossil clone --private http://user@linux.localnetwork:8080/ mac-clone.fossil +</verbatim> + +You'll have to supply a username and password in order for this to work. +Fossil will not clone (or sync) private branches anonymously. + +By default, there are no users that can do private branch syncing. You +will have to give a user +the "Private" capability ("x") if you want them to be able to do this. +We deny such capability for normal users by default to add a barrier to +accidental syncing of a private branch to a public server. It is highly recommended that +you leave the "x" capability turned off on all repositories used for +collaboration (repositories to which many people push and pull) and +only enable "x" for local repositories when you need to share private +branches. + +Private branch sync only works if you use the --private command-line option. +Private branches are never synced via the auto-sync mechanism. Once +again, this restriction is designed to make it hard to accidentally +push private branches beyond their intended audience. + +<h2>Purging Private Branches</h2> + +You can remove all private branches from a repository using this command: + +<pre> +fossil scrub --private +</pre> + +Note that the above is a permanent and irreversible change. You will +be asked to confirm before continuing. Once the private branches are +removed, they cannot be retrieved (unless you have synced them to another +repository.) So be careful with the command. + +<h2>Additional Notes</h2> + +All of the features above apply to <u>all</u> private branches in a +single repository at once. There is no mechanism in Fossil (currently) +that allows you to push, pull, clone, sync, or scrub an individual +private branch within a repository that contains multiple private +branches. diff --git a/www/qandc.wiki b/www/qandc.wiki new file mode 100644 index 0000000..ace5108 --- /dev/null +++ b/www/qandc.wiki @@ -0,0 +1,154 @@ +<title>Questions And Criticisms</title> + +This page is a collection of real questions and criticisms that were +raised against Fossil early in its history (circa 2008). +This page is old and has not been kept up-to-date. See the +[/finfo?name=www/qandc.wiki|change history of this page]. + +<b>Fossil sounds like a lot of reinvention of the wheel. +Why create your own DVCS when you could have reused mercurial?</b> + +<div class="indent"> + I wrote fossil because none of the + other available DVCSes met my needs. If the other DVCSes do + meet your needs, then you might not need fossil. But they + don't meet mine, and so fossil is necessary for me. + + Features provided by fossil that one does not get with other + DVCSes include: + + <ol> + <li> Integrated <a href="wikitheory.wiki">wiki</a>. </li> + <li> Integrated <a href="bugtheory.wiki">bug tracking</a> </li> + <li> Immutable artifacts </li> + <li> Self-contained, stand-alone executable that can be run in + a <a href="http://en.wikipedia.org/wiki/Chroot">chroot jail</a> </li> + <li> Simple, well-defined, + <a href="fileformat.wiki">enduring file format</a> </li> + <li> Integrated <a href="webui.wiki">web interface</a> </li> + </ol> +</div> + +<b>Why should I use this rather than Trac?</b> + +<div class="indent"> + <ol> + <li> Fossil is distributed. You can view and/or edit tickets, wiki, and + code while off network, then sync your changes later. With Trac, you + can only view and edit tickets and wiki while you are connected to + the server. </li> + <li> Fossil is lightweight and fully self-contained. It is very easy + to setup on a low-resource machine. Fossil does not require an + administrator.</li> + <li> Fossil integrates code versioning into the same repository with + wiki and tickets. There is nothing extra to add or install. + Fossil is an all-in-one turnkey solution. </li> + </ol> +</div> + +<b>Love the concept here. Anyone using this for real work yet?</b> + +<div class="indent"> +Fossil is <a href="https://fossil-scm.org/">self-hosting</a>. +In fact, this page was probably delivered +to your web-browser via a working fossil instance. The same virtual +machine that hosts https://fossil-scm.org/ +(a <a href="http://www.linode.com/">Linode 720</a>) +also hosts 24 other fossil repositories for various small projects. +The documentation files for +<a href="http://www.sqlite.org/">SQLite</a> are hosted in a +fossil repository <a href="http://www.sqlite.org/docsrc/">here</a>, +for example. +Other projects are also adopting fossil. But fossil does not yet have +the massive user base of git or mercurial. +</div> + +<b>Fossil looks like the bug tracker that would be in your +Linksys Router's administration screen.</b> + +<div class="indent"> +I take a pragmatic approach to software: form follows function. +To me, it is more important to have a reliable, fast, efficient, +enduring, and simple DVCS than one that looks pretty. + +On the other hand, if you have patches that improve the appearance +of Fossil without seriously compromising its reliability, performance, +and/or maintainability, I will be happy to accept them. Fossil is +self-hosting. Send email to request a password that will let +you push to the main fossil repository. +</div> + +<b>It would be useful to have a separate application that +keeps the bug-tracking database in a versioned file. That file can +then be pushed and pulled along with the rest repository.</b> + +<div class="indent"> +Fossil already <u>does</u> push and pull bugs along with the files +in your repository. +But fossil does <u>not</u> track bugs as files in the source tree. +That approach to bug tracking was rejected for three reasons: + +<ol> +<li> Check-ins in fossil are immutable. So if + tickets were part of the check-in, then there would be no way to add + new tickets to a check-in as new bugs are discovered. + +<li> Any project of reasonable size and complexity will generate thousands + and thousands of tickets, and we do not want all those ticket files + cluttering the source tree. + +<li> We want tickets to be managed from the web interface and to have a + permission system that is distinct from check-in permissions. + In other words, we do not want to restrict the creation and editing + of tickets to developers with check-in privileges and an installed + copy of the fossil executable. Casual passers-by on the internet should + be permitted to create tickets. +</ol> + +These points are reiterated in the opening paragraphs of +the <a href="bugtheory.wiki">Bug-Tracking In Fossil</a> document. +</div> + +<b>Fossil is already the name of a plan9 versioned +append-only filesystem.</b> + +<div class="indent"> +I did not know that. Perhaps they selected the name for the same reason that +I did: because a repository with immutable artifacts preserves +an excellent fossil record of a long-running project. +</div> + +<b>The idea of storing a repository in a binary blob like an +SQLite database terrifies me.</b> + +<div class="indent"> +The use of SQLite to store the database is likely more stable and secure +than any other approach, due to the fact that SQLite is transactional. +Fossil also implements several internal +<a href="selfcheck.wiki">self-checks</a> to insure that no information +is ever lost. +</div> + + +<b>I am dubious of the benefits of including wikis and bug trackers +directly in the VCS - either they are under-featured compared to full +software like Trac, or the VCS is massively bloated compared to +Subversion or Bazaar.</b> + +<div class="indent"> +I have no doubt that Trac has many features that fossil lacks. But that +is not the point. Fossil has several key features that Trac lacks and that +I need: most notably the fact that +fossil supports disconnected operation. + +As for bloat: Fossil is a single self-contained executable. +You do not need any other packages +(diff, patch, merge, cvs, svn, rcs, git, python, perl, tcl, apache, +SQLite, and so forth) +in order to run fossil. Fossil runs just fine in a chroot jail all +by itself. And the self-contained fossil +executable is much less than 1MB in size. (Update 2015-01-12: Fossil has +grown in the years since the previous sentence was written but is still +much less than 2MB according to "size" when compiled using -Os on x64 Linux.) +Fossil is the very opposite of bloat. +</div> diff --git a/www/quickstart.wiki b/www/quickstart.wiki new file mode 100644 index 0000000..e75e939 --- /dev/null +++ b/www/quickstart.wiki @@ -0,0 +1,577 @@ +<title>Fossil Quick Start Guide</title> + +This is a guide to help you get started using the Fossil [https://en.wikipedia.org/wiki/Distributed_version_control|Distributed Version Control System] quickly +and painlessly. + +<h2 id="install">Installing</h2> + +Fossil is a single self-contained C program. You need to +either download a +[https://fossil-scm.org/home/uv/download.html|precompiled +binary] +or <a href="build.wiki">compile it yourself</a> from sources. +Install Fossil by putting the fossil binary +someplace on your $PATH. + +You can test that Fossil is present and working like this: + +<pre><b>fossil version +This is fossil version 2.25 [8f798279d5] 2024-11-06 12:59:09 UTC +</b></pre> + +<h2 id="workflow" name="fslclone">General Work Flow</h2> + +Fossil works with [./glossary.md#repository | repository files] +and [./glossary.md#check-out | check-out directories] using a +workflow like this: + +<ul> + <li>Create or clone a repository file. ([/help/init|fossil init] or + [/help/clone | fossil clone]) + <li>Check out a local tree. ([/help/open | fossil open]) + <li>Perform operations on the repository (including repository + configuration). +</ul> + +Fossil can be entirely driven from the command line. Many features +can also be conveniently accessed from the built-in web user interface. + +The following sections give a brief overview of these +operations. + +<h2 id="new">Starting A New Project</h2> + +To start a new project with Fossil, [/help/init | create a new empty repository]: + +<pre><b>fossil init</b> <i>repository-filename</i> +</pre> + +You can name the database anything you like, and you can place it anywhere in the filesystem. +The <tt>.fossil</tt> extension is traditional, but it is only required if you are going to use the +<tt>[/help/server | fossil server DIRECTORY]</tt> feature. + +Next, do something along the lines of: + +<pre> +<b>mkdir -p ~/src/project/trunk</b> +<b>cd ~/src/project/trunk</b> +<b>fossil open</b> <i>repository-filename</i> +<b>fossil add</b> foo.c bar.h qux.md +<b>fossil commit</b> +</pre> + +If your project directory already exists, obviating the <b>mkdir</b> +step, you will instead need to add the <tt>--force</tt> flag to the +<b>open</b> command to authorize Fossil to open the repo into a +non-empty checkout directory. (This is to avoid accidental opens into, +for example, your home directory.) + +The convention of naming your checkout directory after a long-lived +branch name like "trunk" is in support of Fossil's ability to have as +many open checkouts as you like. This author frequently has additional +checkout directories named <tt>../release</tt>, <tt>../scratch</tt>, +etc. The release directory is open to the branch of the same name, while +the scratch directory is used when disturbing one of the other +long-lived checkout directories is undesireable, as when performing a +[/help/bisect | bisect] operation. + + +<h2 id="clone">Cloning An Existing Repository</h2> + +Most fossil operations interact with a repository that is on the +local disk drive, not on a remote system. Hence, before accessing +a remote repository it is necessary to make a local copy of that +repository, a process called +"[/help/clone | cloning]". + +This is done as follows: + +<pre><b>fossil clone</b> <i>URL repository-filename</i> +</pre> + +The <i>URL</i> specifies the fossil repository +you want to clone. The <i>repository-filename</i> is the new local +filename into which the cloned repository will be written. For +example, to clone the source code of Fossil itself: + +<pre><b>fossil clone https://fossil-scm.org/ myclone.fossil</b></pre> + +If your logged-in username is 'exampleuser', you should see output something like this: + +<pre><b>Round-trips: 8 Artifacts sent: 0 received: 39421 +Clone done, sent: 2424 received: 42965725 ip: 10.10.10.0 +Rebuilding repository meta-data... +100% complete... +Extra delta compression... +Vacuuming the database... +project-id: 94259BB9F186226D80E49D1FA2DB29F935CCA0333 +server-id: 016595e9043054038a9ea9bc526d7f33f7ac0e42 +admin-user: exampleuser (intial remote-access password is "yoWgDR42iv")> +</b></pre> + +This <i>exampleuser</i> will be used by Fossil as the author of commits when +you checkin changes to the repository. It is also used by Fossil when you +make your repository available to others using the built-in server mode by +running <tt>[/help/server | fossil server]</tt> and will also be used when +running <tt>[/help/ui | fossil ui]</tt> to view the repository through +the Fossil UI. See the quick start topic for setting up a +<a href="#server">server</a> for more details. + +If the remote repository requires a login, include a +userid in the URL like this: + +<pre><b>fossil clone https://</b><i>remoteuserid</i><b>@www.example.org/ myclone.fossil</b></pre> + +You will be prompted separately for the password. +Use [https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters|"%HH"] escapes for special characters in the userid. +For example "/" would be replaced by "%2F" meaning that a userid of "Projects/Budget" would become "Projects%2FBudget") + +If you are behind a restrictive firewall, you might need +to <a href="#proxy">specify an HTTP proxy</a>. + +A Fossil repository is a single disk file. Instead of cloning, +you can just make a copy of the repository file (for example, using +"scp"). Note, however, that the repository file contains auxiliary +information above and beyond the versioned files, including some +sensitive information such as password hashes and email addresses. If you +want to share Fossil repositories directly by copying, consider running the +[/help/scrub|fossil scrub] command to remove sensitive information +before transmitting the file. + +<h2 id="import">Importing From Another Version Control System</h2> + +Rather than start a new project, or clone an existing Fossil project, +you might prefer to +<a href="./inout.wiki">import an existing Git project</a> +into Fossil using the [/help/import | fossil import] command. + +You can even decide to export your project back into git using the +[/help/git | fossil git] command, which is how the Fossil project maintains +[https://github.com/drhsqlite/fossil-mirror | its public GitHub mirror]. There +is no limit to the number of times a tree can be imported and exported between +Fossil and git. + +The [https://git-scm.com/docs/git-fast-export|Git fast-export format] has become +a popular way to move files between version management systems, including from +[https://www.mercurial-scm.org/|Mercurial]. +Fossil can also import [https://subversion.apache.org/|Subversion projects] directly. + +<h2 id="checkout">Checking Out A Local Tree</h2> + +To work on a project in fossil, you need to check out a local +copy of the source tree. Create the directory you want to be +the root of your tree, <tt>cd</tt> into that directory, and then: + +<pre><b>fossil open</b> <i>repository-filename</i></pre> + +For example: + +<pre><b>fossil open ../myclone.fossil + BUILD.txt + COPYRIGHT-BSD2.txt + README.md + ︙ +</tt></b></pre> + +This leaves you with the newest version of the tree +checked out. +From anywhere underneath the root of your local tree, you +can type commands like the following to find out the status of +your local tree: + +<pre> +<b>[/help/info | fossil info]</b> +<b>[/help/status | fossil status]</b> +<b>[/help/changes | fossil changes]</b> +<b>[/help/diff | fossil diff]</b> +<b>[/help/timeline | fossil timeline]</b> +<b>[/help/ls | fossil ls]</b> +<b>[/help/branch | fossil branch]</b> +</pre> + +If you created a new repository using "fossil init" some commands will not +produce much output. + +Note that Fossil allows you to make multiple check-outs in +separate directories from the same repository. This enables you, +for example, to do builds from multiple branches or versions at +the same time without having to generate extra clones. + +To switch a checkout between different versions and branches, +use: + +<pre> +<b>[/help/update | fossil update]</b> +<b>[/help/checkout | fossil checkout]</b> +</pre> + +[/help/update | update] honors the "autosync" option and +does a "soft" switch, merging any local changes into the target +version, whereas [/help/checkout | checkout] does not +automatically sync and does a "hard" switch, overwriting local +changes if told to do so. + +<h2 id="changes">Making and Committing Changes</h2> + +To add new files to your project or remove existing ones, use these +commands: + +<pre> +<b>[/help/add | fossil add]</b> <i>file...</i> +<b>[/help/rm | fossil rm]</b> <i>file...</i> +<b>[/help/addremove | fossil addremove]</b> <i>file...</i> +</pre> + +The command: + +<pre><b>[/help/changes | fossil changes]</b></pre> + +lists files that have changed since the last commit to the repository. For +example, if you edit the file "README.md": + +<pre><b>fossil changes +EDITED README.md +</b></pre> + +To see exactly what change was made you can use the command +<b>[/help/diff | fossil diff]</b>: + +<pre><b>fossil diff +Index: README.md +============================================================ +--- README.md ++++ README.md +@@ -1,5 +1,6 @@ ++Made some changes to the project +# Original text +</b></pre> + +"fossil diff" shows the difference between your tree on disk now and as +the tree was when you last committed changes. If you haven't committed +yet, then it shows the difference relative to the tip-of-trunk commit in +the repository, being what you get when you "fossil open" a repository +without specifying a version, populating the working directory. + +To see the most recent changes made to the repository by other users, use "fossil timeline" to +find out the most recent commit, and then "fossil diff" between that commit and the +current tree: + +<pre><b><verbatim>fossil timeline +=== 2021-03-28 === +03:18:54 [ad75dfa4a0] *CURRENT* Added details to frobnicate command (user: user-one tags: trunk) +=== 2021-03-27 === +23:58:05 [ab975c6632] Update README.md. (user: user-two tags: trunk) + ⋮ + +fossil diff --from current --to ab975c6632 +Index: frobnicate.c +============================================================ +--- frobnicate.c ++++ frobnicate.c +@@ -1,10 +1,11 @@ ++/* made a change to the source file */ +# Original text +</verbatim></b></pre> + +"current" is an alias for the checkout version, so the command +"fossil diff --from ad75dfa4a0 --to ab975c6632" gives identical results. + +To commit your changes to a local-only repository: + +<pre><b>fossil commit</b> <i>(... Fossil will start your editor, if defined)</i><b> +# Enter a commit message for this check-in. Lines beginning with # are ignored. +# +# user: exampleuser +# tags: trunk +# +# EDITED README.md +Edited file to add description of code changes +New_Version: 7b9a416ced4a69a60589dde1aedd1a30fde8eec3528d265dbeed5135530440ab +</b></pre> + +You will be prompted for check-in comments using whatever editor +is specified by your VISUAL or EDITOR environment variable. If none is +specified Fossil uses line-editing in the terminal. + +To commit your changes to a repository that was cloned from a remote +repository, you give the same command, but the results are different. +Fossil defaults to [./concepts.wiki#workflow|autosync] mode, a +single-stage commit that sends all changes committed to the local +repository immediately on to the remote parent repository. This only +works if you have write permission to the remote repository. + +<h2 id="naming">Naming of Files, Checkins, and Branches</h2> + +Fossil deals with information artifacts. This Quickstart document only deals +with files and collections of files, but be aware there are also tickets, wiki pages and more. +Every artifact in Fossil has a universally-unique hash id, and may also have a +human-readable name. + +The following are all equivalent ways of identifying a Fossil file, +checkin or branch artifact: + +<ul> +<li> the full unique SHA-256 hash, such as be836de35a821523beac2e53168e135d5ebd725d7af421e5f736a28e8034673a +<li> an abbreviated hash prefix, such as the first ten characters: be836de35a . This won't be universally unique, but it is usually unique within any one repository. As an example, the [https://fossil-scm.org/home/hash-collisions|Fossil project hash collisions] showed at the time of writing that there are no artifacts with identical first 8 characters +<li> a branch name, such as "special-features" or "juliet-testing". Each branch also has a unique SHA-256 hash +</ul> + +A special convenience branch is "trunk", which is Fossil's default branch name for +the first checkin, and the default for any time a branch name is needed but not +specified. + +This will get you started on identifying checkins. The +<a href="./checkin_names.wiki">Checkin Names document</a> is a complete reference, including +how timestamps can also be used. + +<h2 id="config">Accessing Your Local Repository's Web User Interface</h2> + +After you create a new repository, you usually want to do some local +configuration. This is most easily accomplished by firing up the Fossil +UI: + +<pre> +<b>fossil ui</b> <i>repository-filename</i> +</pre> + +You can shorten that to just [/help/ui | <b>fossil ui</b>] +if you are inside a checked-out local tree. + +This command starts an internal web server, after which Fossil +automatically launches your default browser, pointed at itself, +presenting a special view of the repository, its web user interface. + +You may override Fossil's logic for selecting the default browser so: + +<pre> +<b>fossil setting web-browser</b> <i>path-to-web-browser</i> +</pre> + +When launched this way, Fossil binds its internal web server to the IP +loopback address, 127.0.0.1, which it treats specially, bypassing all +user controls, effectively giving visitors the +[./caps/admin-v-setup.md#apsu | all-powerful Setup capabliity]. + +Why is that a good idea, you ask? Because it is a safe +presumption that only someone with direct file access to the repository +database file could be using the resulting web interface. Anyone who can +modify the repo DB directly could give themselves any and all access +with a SQL query, or even by direct file manipulation; no amount of +access control matters to such a user. + +(Contrast the [#server | many <i>other</i> ways] of setting Fossil up +as an HTTP server, where the repo DB is on the other side of the HTTP +server wall, inaccessible by all means other than Fossil's own +mediation. For this reason, the "localhost bypasses access control" +policy does <i>not</i> apply to these other interfaces. That is a very +good thing, since without this difference in policy, it would be unsafe +to bind a [/help/server | <b>fossil server</b>] instance to +localhost on a high-numbered port and then reverse-proxy it out to the +world via HTTPS, a practice this author does engage in, with confidence.) + +Once you are finished configuring Fossil, you may safely Control-C out +of the <b>fossil ui</b> command to shut down this privileged +built-in web server. Moreover, you may by grace of SQLite do this <i>at +any time</i>: all changes are either committed durably to the repo DB or +rolled back, in their totality. This includes configuration changes. + +<h2 id="sharing">Sharing Changes</h2> + +When [./concepts.wiki#workflow|autosync] is turned off, +the changes you [/help/commit | commit] are only +on your local repository. +To share those changes with other repositories, do: + +<pre> +<b>[/help/push | fossil push]</b> <i>URL</i> +</pre> + +Where <i>URL</i> is the http: URL of the server repository you +want to share your changes with. If you omit the <i>URL</i> argument, +fossil will use whatever server you most recently synced with. + +The [/help/push | push] command only sends your changes to others. To +Receive changes from others, use [/help/pull | pull]. Or go both ways at +once using [/help/sync | sync]: + +<pre> +<b>[/help/pull | fossil pull]</b> <i>URL</i> +<b>[/help/sync | fossil sync]</b> <i>URL</i> +</pre> + +When you pull in changes from others, they go into your repository, +not into your checked-out local tree. To get the changes into your +local tree, use [/help/update | update]: + +<pre> +<b>[/help/update | fossil update]</b> <i>VERSION</i> +</pre> + +The <i>VERSION</i> can be the name of a branch or tag or any +abbreviation to the 40-character +artifact identifier for a particular check-in, or it can be a +date/time stamp. ([./checkin_names.wiki | more info]) +If you omit +the <i>VERSION</i>, then fossil moves you to the +latest version of the branch you are currently on. + +The default behavior is for [./concepts.wiki#workflow|autosync] to +be turned on. That means that a [/help/pull|pull] automatically occurs +when you run [/help/update|update] and a [/help/push|push] happens +automatically after you [/help/commit|commit]. So in normal practice, +the push, pull, and sync commands are rarely used. But it is important +to know about them, all the same. + +<pre> +<b>[/help/checkout | fossil checkout]</b> <i>VERSION</i> +</pre> + +Is similar to update except that it does not honor the autosync +setting, nor does it merge in local changes - it prefers to overwrite +them and fails if local changes exist unless the <tt>--force</tt> +flag is used. + +<h2 id="branch" name="merge">Branching And Merging</h2> + +Use the --branch option to the [/help/commit | commit] command to start +a new branch at the point of need. ([./gitusers.md#bneed | Contrast git].) + +To merge two branches back together, first +[/help/update | update] to the branch you want to merge into. +Then do a [/help/merge|merge] of the other branch that you want to incorporate +the changes from. For example, to merge "featureX" changes into "trunk" +do this: + +<pre> +<b>fossil [/help/update|update] trunk</b> +<b>fossil [/help/merge|merge] featureX</b> +<i># make sure the merge didn't break anything...</i> +<b>fossil [/help/commit|commit] +</pre> + +The argument to the [/help/merge|merge] command can be any of the +version identifier forms that work for [/help/update|update]. +([./checkin_names.wiki|more info].) +The merge command has options to cherry-pick individual +changes, or to back out individual changes, if you don't want to +do a full merge. + +The merge command puts all changes in your working check-out. +No changes are made to the repository. +You must run [/help/commit|commit] separately +to add the merge changes into your repository to make them persistent +and so that your coworkers can see them. +But before you do that, you will normally want to run a few tests +to verify that the merge didn't cause logic breaks in your code. + +The same branch can be merged multiple times without trouble. Fossil +automatically keeps up with things and avoids conflicts when doing +multiple merges. So even if you have merged the featureX branch +into trunk previously, you can do so again and Fossil will automatically +know to pull in only those changes that have occurred since the previous +merge. + +If a merge or update doesn't work out (perhaps something breaks or +there are many merge conflicts) then you back up using: + +<pre> +<b>[/help/undo | fossil undo]</b> +</pre> + +This will back out the changes that the merge or update made to the +working checkout. There is also a [/help/redo|redo] command if you undo by +mistake. Undo and redo only work for changes that have +not yet been checked in using commit and there is only a single +level of undo/redo. + + +<h2 id="server">Setting Up A Server</h2> + +In addition to the inward-facing <b>fossil ui</b> mode covered [#config +| above], Fossil can also act as an outward-facing web server: + +<pre> +<b>[/help/server | fossil server]</b> <i>repository-filename</i> +</pre> + +Just as with <b>fossil ui</b>, you may omit the +<i>repository-filename</i> parameter when running this from within an open +check-out. + +<i>Unlike</i> <b>fossil ui</b> mode, Fossil binds to all network +interfaces by default in this mode, and it enforces the configured +[./caps/ | role-based access controls]. Further, because it is meant to +provide external web service, it doesn't try to launch a local web +browser pointing to a "Fossil UI" presentation; external visitors see +your repository's configured home page instead. + +To serve varying needs, there are additional ways to serve a Fossil repo +to external users: + +<ul> +<li>[./server/any/cgi.md|CGI], as used by Fossil's [./selfhost.wiki | + self-hosting repositories] +<li>[./server/any/scgi.md|SCGI] +<li>[./server/any/inetd.md|inetd] +<li>[./server/debian/service.md|systemd] +</ul> + +…along with [./server/#matrix | several other options]. + +We recommend that you read the [./server/whyuseaserver.wiki | Benefits +of a Fossil Server] article, because you might <i>need</i> to do this +and not yet know it. + +<h2 id="proxy">HTTP Proxies</h2> + +If you are behind a restrictive firewall that requires you to use +an HTTP proxy to reach the internet, then you can configure the proxy +in three different ways. You can tell fossil about your proxy using +a command-line option on commands that use the network, +<b>sync</b>, <b>clone</b>, <b>push</b>, and <b>pull</b>. + +<pre> +<b>fossil clone </b><i>URL</i> <b>--proxy</b> <i>Proxy-URL</i> +</pre> + +It is annoying to have to type in the proxy URL every time you +sync your project, though, so you can make the proxy configuration +persistent using the [/help/setting | setting] command: + +<pre> +<b>fossil setting proxy </b><i>Proxy-URL</i> +</pre> + +Or, you can set the "<b>http_proxy</b>" environment variable: + +<pre> +<b>export http_proxy=</b><i>Proxy-URL</i> +</pre> + +To stop using the proxy, do: + +<pre> +<b>fossil setting proxy off</b> +</pre> + +Or unset the environment variable. The fossil setting for the +HTTP proxy takes precedence over the environment variable and the +command-line option overrides both. If you have a persistent +proxy setting that you want to override for a one-time sync, that +is easily done on the command-line. For example, to sync with +a co-worker's repository on your LAN, you might type: + +<pre> +<b>fossil sync http://192.168.1.36:8080/ --proxy off</b> +</pre> + +<h2 id="links">Other Resources</h2> + +<ul> +<li> <a href="./gitusers.md">Hints For Users With Prior Git Experience</a> +<li> <a href="./whyusefossil.wiki">Why You Should Use Fossil</a> +<li> <a href="./history.md">The History and Purpose of Fossil</a> +<li> <a href="./branching.wiki">Branching, Forking, and Tagging</a> +<li> <a href="./hints.wiki">Fossil Tips and Usage Hints</a> +<li> <a href="./permutedindex.html">Comprehensive Fossil Doc Index</a> +</ul> diff --git a/www/quotes.wiki b/www/quotes.wiki new file mode 100644 index 0000000..54307be --- /dev/null +++ b/www/quotes.wiki @@ -0,0 +1,171 @@ +<title>What People Are Saying</title> + +The following are collected quotes from various forums and blogs about +Fossil, Git, and DVCSes in general. This collection is put together +by the creator of Fossil, so of course there is selection bias... + +<h2>On The Usability Of Git</h2> + +<ol> +<li>Git approaches the usability of iptables, which is to say, utterly +unusable unless you have the manpage tattooed on you arm. + +<p class="local-indent"> +<i>by mml at [http://news.ycombinator.com/item?id=1433387]</i> +</p> + +<li><nowiki>It's simplest to think of the state of your [git] repository +as a point in a high-dimensional "code-space", in which branches are +represented as n-dimensional membranes, mapping the spatial loci of +successive commits onto the projected manifold of each cloned +repository.</nowiki> + +<p class="local-indent"> +<i>by Jonathan Hartley at +[https://www.tartley.com/posts/a-guide-to-git-using-spatial-analogies]; +<br>Quoted here: [https://lwn.net/Articles/420152/].</i> +</p> + +<li>Git is not a Prius. Git is a Model T. +Its plumbing and wiring sticks out all over the place. +You have to be a mechanic to operate it successfully or you'll be +stuck on the side of the road when it breaks down. +And it <b>will</b> break down. + +<p class="local-indent"> +<i>Nick Farina at [http://nfarina.com/post/9868516270/git-is-simpler]</i> +</p> + +<li>Initial revision of "git", The information manager from hell + +<p class="local-indent"> +<i>Linus Torvalds - 2005-04-07 22:13:13<br> +Commit comment on the very first source-code check-in for git +</p> + +<li>I've been experimenting a lot with git at work. +Damn, it's complicated. +It has things to trip you up with that sane people just wouldn't ever both with +including the ability to allow you to commit stuff in such a way that you can't find +it again afterwards (!!!) +Demented workflow complexity on acid? +<p>* dkf really wishes he could use fossil instead</p> +<p class="local-indent"> +<i>by Donal K. Fellow (dkf) on the Tcl/Tk chatroom, 2013-04-09.</i> +</p> + +<li>[G]it is <i>designed</i> to forget things. + +<p class="local-indent"> +<i>[http://www.cs.cmu.edu/~davide/howto/git_lose.html] +</p> + +<li>[I]n nearly 31 years of using a computer i have, in total, lost more data to git +(while following the instructions!!!) than any other single piece of software. + +<p class="local-indent"> +<i>Stephan Beal on the [http://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg17181.html|Fossil mailing list] + 2014-09-01.</i> +</p> + +<li>If programmers _really_ wanted to help scientists, they'd build a version control +system that was more usable than Git. + +<p class="local-indent"> +<i>Tweet by Greg Wilson @gvwilson on 2015-02-22 17:47</i> +</p> + +<li><img src='xkcd-git.gif' align='top'> + +<p class="local-indent"><i>Randall Munroe. [http://xkcd.com/1597/]</i><p> + +</ol> + +<h2>On The Usability Of Fossil</h2> + +<ol> +<li value=11> +Fossil mesmerizes me with simplicity especially after I struggled to +get a bug-tracking system to work with mercurial. + +<p class="local-indent"> +<i>rawjeev at [https://stackoverflow.com/a/2100469/142454]</i> +</p> + +<li>Fossil is the best thing to happen +to my development workflow this year, as I am pretty sure that using +Git has resulted in the premature death of too many of my brain cells. +I'm glad to be able to replace Git in every place that I possibly can +with Fossil. + +<p class="local-indent"> +<i>Joe Prostko at [http://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg16716.html] +</p> + +<li>This is my favourite VCS. I can carry it on a USB. And it's a complete system, with it's own +server, ticketing system, Wiki pages, and a very, very helpful timeline visualization. And +the entire program in a single file! + +<p class="local-indent"> +<i>thunderbong commenting on hacker news: [https://news.ycombinator.com/item?id=9131619]</i> +</p> + + +</ol> + + +<h2>On Git Versus Fossil</h2> + +<ol> +<li value=14> +After prolonged exposure to fossil, i tend to get the jitters when I work with git... + +<p class="local-indent"> +<i>sriku - at [https://news.ycombinator.com/item?id=16104427]</i> +</p> + + +<li> +Just want to say thanks for fossil making my life easier.... +Also <nowiki>[for]</nowiki> not having a misanthropic command line interface. + +<p class="local-indent"> +<i>Joshua Paine at [http://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg02736.html]</i> +</p> + +<li>We use it at a large university to manage code that small teams write. +The runs everywhere, ease of installation and portability is something that +seems to be a good fit with the environment we have (highly ditrobuted, +sometimes very restrictive firewalls, OSX/Win/Linux). We are happy with it +and teaching a Msc/Phd student (read complete novice) fossil has just +been a smoother ride than Git was. + +<p class="local-indent"> +<i>viablepanic at [https://www.reddit.com/r/programming/comments/bxcto/why_not_fossil_scm/c0p30b4?utm_source=share&utm_medium=web2x&context=3]</i> +</p> + +<li>In the fossil community - and hence in fossil itself - development history +is pretty much sacrosanct. The very name "fossil" was to chosen to +reflect the unchanging nature of things in that history. +<br><br> +In git (or rather, the git community), the development history is part of +the published aspect of the project, so it provides tools for rearranging +that history so you can present what you "should" have done rather +than what you actually did. + +<p class="local-indent"> +<i>Mike Meyer on the Fossil mailing list, 2011-10-04</i> +</p> + +<li>github is such a pale shadow of what fossil does. + +<p class="local-indent"> +<i>dkf on the Tcl chatroom, 2013-12-06</i> +</p> + +<li>[With fossil] I actually enjoy keeping track of source files again. + +<p class="local-indent"> +<a href="https://wholesomedonut.prose.sh/using-fossil-not-git">https://wholesomedonut.prose.sh/using-fossil-not-git</a> +</p> +</ol> diff --git a/www/rebaseharm.md b/www/rebaseharm.md new file mode 100644 index 0000000..9b72d38 --- /dev/null +++ b/www/rebaseharm.md @@ -0,0 +1,522 @@ +# Rebase Considered Harmful + +Fossil deliberately omits a "rebase" command because the original +designer of Fossil (and [original author][vhist] of this article) considers rebase to be +an anti-pattern to be avoided. This article attempts to +explain that point of view. + +[vhist]: /finfo?name=www/rebaseharm.md&ubg + +## 1.0 Rebasing is dangerous + +Most people, even strident advocates of rebase, agree that rebase can +cause problems when misused. The Git rebase documentation talks about the +[golden rule of rebasing][golden]: never rebase on a public +branch. Horror stories of misused rebase abound, and the rebase +documentation devotes considerable space toward explaining how to +recover from rebase errors and/or misuse. + +## <a id="cap-loss"></a>2.0 Rebase provides no new capabilities + +Sometimes sharp and dangerous tools are justified, +because they accomplish things that cannot be +done otherwise, or at least cannot be done easily. +Rebase does not fall into that category, +because it provides no new capabilities. + +### <a id="orphaning"></a>2.1 A rebase is just a merge with historical references omitted + +A rebase is really nothing more than a merge (or a series of merges) +that deliberately forgets one of the parents of each merge step. +To help illustrate this fact, +consider the first rebase example from the +[Git documentation][gitrebase]. The merge looks like this: + +~~~ pikchr toggle center +scale = 0.8 +circle "C0" fit +arrow right 50% +circle same "C1" +arrow same +circle same "C2" +arrow same +circle same "C3" +arrow same +circle same "C5" +circle same "C4" at 1cm above C3 +arrow from C2 to C4 chop +arrow from C4 to C5 chop +~~~ + +And the rebase looks like this: + +~~~ pikchr toggle center +scale = 0.8 +circle "C0" fit +arrow right 50% +circle same "C1" +arrow same +circle same "C2" +arrow same +circle same "C3" +arrow same +circle same "C4'" +circle same "C4" at 1cm above C3 +arrow from C2 to C4 chop +~~~ + +As the [Git documentation][gitrebase] points out, check-ins C4\' and C5 +are identical. The only difference between C4\' and C5 is that C5 +records the fact that C4 is its merge parent but C4\' does not. + +Thus, a rebase is just a merge that forgets where it came from. + +The Git documentation acknowledges this fact (in so many words) and +justifies it by saying "rebasing makes for a cleaner history." I read +that sentence as a tacit admission that the Git history display +capabilities are weak and need active assistance from the user to +keep things manageable. +Surely a better approach is to record +the complete ancestry of every check-in but then fix the tool to show +a "clean" history in those instances where a simplified display is +desirable and edifying, but retain the option to show the real, +complete, messy history for cases where detail and accuracy are more +important. + +So, another way of thinking about rebase is that it is a kind of +merge that intentionally forgets some details in order to +not overwhelm the weak history display mechanisms available in Git. +Wouldn't it be better, less error-prone, and easier on users +to enhance the history display mechanisms in Git so that rebasing +for a clean, linear history became unnecessary? + +### <a id="clean-diffs"></a>2.2 Rebase does not actually provide better feature-branch diffs + +Another argument, often cited, is that rebasing a feature branch +allows one to see just the changes in the feature branch without +the concurrent changes in the main line of development. +Consider a hypothetical case: + +~~~ pikchr toggle center +scale = 0.8 +circle "C0" fit fill white +arrow right 50% +circle same "C1" +arrow same +circle same "C2" +arrow same +circle same "C4" +arrow same +circle same "C6" +circle same "C3" at last arrow.width + C0.rad*2 heading 30 from C2 +arrow right 50% +circle same "C5" +arrow from C2 to C3 chop +box ht C3.y-C2.y wid C6.e.x-C0.w.x+1.5*C1.rad at C2 behind C0 fill 0xc6e2ff color 0xaac5df +box ht previous.ht wid previous.wid*0.55 with .se at previous.ne \ + behind C0 fill 0x9accfc color 0xaac5df +text "feature" with .s at previous.n +text "main" with .n at first box.s +~~~ + +In the above, a feature branch consisting of check-ins C3 and C5 is +run concurrently with the main line in check-ins C4 and C6. Advocates +for rebase say that you should rebase the feature branch to the tip +of main in order to remove main-line development differences from +the feature branch's history: + +~~~ pikchr toggle center +# Duplicated below in section 5.0 +scale = 0.8 +circle "C0" fit fill white +arrow right 50% +circle same "C1" +arrow same +circle same "C2" +arrow same +circle same "C4" +arrow same +circle same "C6" +circle same "C3" at last arrow.width + C0.rad*2 heading 30 from C2 +arrow right 50% +circle same "C5" +arrow from C2 to C3 chop +C3P: circle same "C3'" at first arrow.width + C0.rad*2 heading 30 from C6 +arrow right 50% from C3P.e +C5P: circle same "C5'" +arrow from C6 to C3P chop + +box ht C3.y-C2.y wid C5P.e.x-C0.w.x+1.5*C1.rad with .w at 0.5*(first arrow.wid) west of C0.w \ + behind C0 fill 0xc6e2ff color 0xaac5df +box ht previous.ht wid previous.e.x - C2.w.x with .se at previous.ne \ + behind C0 fill 0x9accfc color 0xaac5df +~~~ + + +You could choose to collapse C3\' and C5\' into a single check-in +as part of this rebase, but that's a side issue we'll deal with +[separately](#collapsing). + +Because Fossil purposefully lacks rebase, the closest you can get to this same check-in +history is the following merge: + +~~~ pikchr toggle center +scale = 0.8 +circle "C0" fit fill white +arrow right 50% +circle same "C1" +arrow same +circle same "C2" +arrow same +circle same "C4" +arrow same +circle same "C6" +circle same "C3" at last arrow.width + C0.rad*2 heading 30 from C2 +arrow right 50% +circle same "C5" +arrow same +circle same "C7" +arrow from C2 to C3 chop +arrow from C6 to C7 chop + +box ht C3.y-C2.y wid C7.e.x-C0.w.x+1.5*C1.rad with .w at 0.5*(first arrow.wid) west of C0.w \ + behind C0 fill 0xc6e2ff color 0xaac5df +box ht previous.ht wid previous.e.x - C2.w.x with .se at previous.ne \ + behind C0 fill 0x9accfc color 0xaac5df +~~~ + + +Check-ins C5\' and C7 check-ins hold identical code. The only +difference is in their history. + +The argument from rebase advocates +is that with merge it is difficult to see only the changes associated +with the feature branch without the commingled mainline changes. +In other words, diff(C2,C7) shows changes from both the feature +branch and from the mainline, whereas in the rebase case +diff(C6,C5\') shows only the feature branch changes. + +But that argument is comparing apples to oranges, since the two diffs +do not have the same baseline. The correct way to see only the feature +branch changes in the merge case is not diff(C2,C7) but rather diff(C6,C7). + +<div align=center> + +| Rebase | Merge | What You See | +|---------------|-------------|----------------------------------------| +| diff(C2,C5\') | diff(C2,C7) | Commingled branch and mainline changes | +| diff(C6,C5\') | diff(C6,C7) | Branch changes only | + +</div> + +Remember: C7 and C5\' are bit-for-bit identical, so the output of the +diff is not determined by whether you select C7 or C5\' as the target +of the diff, but rather by your choice of the diff source, C2 or C6. + +So, to help with the problem of viewing changes associated with a feature +branch, perhaps what is needed is not rebase but rather better tools to +help users identify an appropriate baseline for their diffs. + +## <a id="siloing"></a>3.0 Rebase encourages siloed development + +The [golden rule of rebasing][golden] is that you should never do it +on public branches, so if you are using rebase as intended, that means +you are keeping private branches. Or, to put it another way, you are +doing siloed development. You are not sharing your intermediate work +with collaborators. This is not good for product quality. + +[Nagappan, et. al][nagappan] studied bugs in Windows Vista and found +that the best predictor of bugs is the distance on the org-chart between +the stake-holders. The bug rate is inversely related to the +amount of communication among the engineers. +Similar findings arise in other disciplines. Keeping +private branches does not prove that developers are communicating +insufficiently, but it is a key symptom of that problem. + +[Weinberg][weinberg] argues programming should be "egoless." That +is to say, programmers should avoid linking their code with their sense of +self, as that makes it more difficult for them to find and respond +to bugs, and hence makes them less productive. Many developers are +drawn to private branches out of sense of ego. "I want to get the +code right before I publish it." I sympathize with this sentiment, +and am frequently guilty of it myself. It is humbling to display +your stupid mistake to the whole world on an Internet that +never forgets. And yet, humble programmers generate better code. + +What is the fastest path to solid code? Is it to continue staring at +your private branch to seek out every last bug, or is it to publish it +as-is, whereupon the many eyeballs will immediately see that last stupid +error in the code? Testing and development are often done by separate +groups within a larger software development organization, because +developers get too close to their own code to see every problem in it. + +Given that, is it better for those many eyeballs to find your problems +while they're still isolated on a feature branch, or should that vetting +wait until you finally push a collapsed version of a private working +branch to the parent repo? Will the many eyeballs even see those errors +when they’re intermingled with code implementing some compelling new feature? + +## <a id="timestamps"></a>4.0 Rebase causes timestamp confusion + +Consider the earlier example of rebasing a feature branch: + +~~~ pikchr toggle center +# Copy of second diagram in section 2.2 above +scale = 0.8 +circle "C0" fit fill white +arrow right 50% +circle same "C1" +arrow same +circle same "C2" +arrow same +circle same "C4" +arrow same +circle same "C6" +circle same "C3" at last arrow.width + C0.rad*2 heading 30 from C2 +arrow right 50% +circle same "C5" +arrow from C2 to C3 chop +C3P: circle same "C3'" at first arrow.width + C0.rad*2 heading 30 from C6 +arrow right 50% from C3P.e +C5P: circle same "C5'" +arrow from C6 to C3P chop + +box ht C3.y-C2.y wid C5P.e.x-C0.w.x+1.5*C1.rad with .w at 0.5*(first arrow.wid) west of C0.w \ + behind C0 fill 0xc6e2ff color 0xaac5df +box ht previous.ht wid previous.e.x - C2.w.x with .se at previous.ne \ + behind C0 fill 0x9accfc color 0xaac5df +~~~ + +What timestamps go on the C3\' and C5\' check-ins? If you choose +the same timestamps as the original C3 and C5, then you have the +odd situation C3' is older than its parent C6. We call that a +"timewarp" in Fossil. Timewarps can also happen due to misconfigured +system clocks, so they are not unique to rebase, but they are very +confusing and so best avoided. The other option is to provide new +unique timestamps for C3' and C5' but then you lose the information +about when those check-ins were originally created, which can make +historical analysis of changes more difficult. It might also +complicate the legal defense of prior art claims. + +## <a id="lying"></a>5.0 Rebase misrepresents the project history + +By discarding parentage information, rebase attempts to deceive the +reader about how the code actually came together. + +Git’s rebase feature is more than just an +alternative to merging: it also provides mechanisms for changing the +project history in order to make editorial changes. Fossil shows that +you can get similar effects without modifying historical records, +allowing users to: + + 1. Edit check-in comments to fix typos or enhance clarity + 2. Attach supplemental notes to check-ins or whole branches + 3. Hide ill-conceived or now-unused branches from routine display + 4. Fix faulty check-in date/times resulting from misconfigured + system clocks + 5. Cross-reference check-ins with each other, or with + wiki, tickets, forum posts, and/or embedded documentation + +…and so forth. + +Fossil allows all of this not by removing or modifying existing +repository entries, but rather by adding new supplemental records. +Fossil keeps the original incorrect or unclear inputs and makes them +readily accessible, preserving the original historical record. Fossil +doesn’t make the user tell counter-factual “stories,” it only allows the +user to provide annotations to provide a more readable edited +presentation for routine display purposes. + +Git needs rebase because it lacks these annotation facilities. Rather +than consider rebase a desirable feature missing in Fossil, ask instead +why Git lacks support for making editorial changes to check-ins without +modifying history? Wouldn't it be better to fix the version control +tool rather than requiring users to fabricate a fictitious project +history? + +## <a id="collapsing"></a>6.0 Collapsing check-ins throws away valuable information + +One of the oft-cited advantages of rebasing in Git is that it lets you +collapse multiple check-ins down to a single check-in to make the +development history “clean.” The intent is that development appear as +though every feature were created in a single step: no multi-step +evolution, no back-tracking, no false starts, no mistakes. This ignores +actual developer psychology: ideas rarely spring forth from fingers to +files in faultless finished form. A wish for collapsed, finalized +check-ins is a wish for a counterfactual situation. + +The common counterargument is that collapsed check-ins represent a +better world, the ideal we're striving for. What that argument overlooks +is that we must throw away valuable information to get there. + +### <a id="empathy"></a>6.1 Individual check-ins support mutual understanding + +Ideally, future developers of our software can understand every feature +in it using only context available in the version of the code they start +work with. Prior to widespread version control, developers had no choice +but to work that way. Pre-existing codebases could only be understood +as-is or not at all. Developers in that world had an incentive to +develop software that was easy to understand retrospectively, even if +they were selfish people, because they knew they might end up being +those future developers! + +Yet, sometimes we come upon a piece of code that we simply cannot +understand. If you have never asked yourself, "What was this code's +developer thinking?" you haven't been developing software for very long. + +When a developer can go back to the individual check-ins leading up to +the current code, they can work out the answers to such questions using +only the level of personal brilliance necessary to be a good developer. To +understand such code using only the finished form, you are asking future +developers to make intuitive leaps that the original developer was +unable to make. In other words, you are asking your future maintenance +developers to be smarter than the original developers! That's a +beautiful wish, but there's a sharp limit to how far you can carry it. +Eventually you hit the limits of human brilliance. + +When the operation of some bit of code is not obvious, both Fossil and +Git let you run a [`blame`](/help/blame) on the code file to get +information about each line of code, and from that which check-in last +touched a given line of code. If you squash the check-ins on a branch +down to a single check-in, you throw away the information leading up to +that finished form. Fossil not only preserves the check-ins surrounding +the one that included the line of code you're trying to understand, its +[superior data model][sdm] lets you see the surrounding check-ins in +both directions; not only what lead up to it, but what came next. Git +can't do that short of crawling the block-chain backwards from the tip +of the branch to the check-in you’re looking at, an expensive operation. + +We believe it is easier to understand a line of code from the 10-line +check-in it was a part of — and then to understand the surrounding +check-ins as necessary — than it is to understand a 500-line check-in +that collapses a whole branch's worth of changes down to a single +finished feature. + +[sdm]: ./fossil-v-git.wiki#durable + +### <a id="bisecting"></a>6.2 Bisecting works better on small check-ins + +Git lets a developer write a feature in ten check-ins but collapse it +down to an eleventh check-in and then deliberately push only that final +collapsed check-in to the parent repo. Someone else may then do a bisect +that blames the merged check-in as the source of the problem they’re +chasing down; they then have to manually work out which of the 10 steps +the original developer took to create it to find the source of the +actual problem. + +An equivalent push in Fossil will send all 11 check-ins to the parent +repository so that a later investigator doing the same sort of bisect +sees the complete check-in history. That bisect will point the +investigator at the single original check-in that caused the problem. + +### <a id="comments"></a>6.3 Multiple check-ins require multiple check-in comments + +The more comments you have from a given developer on a given body of +code, the more concise documentation you have of that developer's +thought process. To resume the bisecting example, a developer trying to +work out what the original developer was thinking with a given change +will have more success given a check-in comment that explains what the +one check-in out of ten blamed by the "bisect" command was trying to +accomplish than if they must work that out from the eleventh check-in's +comment, which only explains the "clean" version of the collapsed +feature. + +### <a id="cherrypicking"></a>6.4 Cherry-picks work better with small check-ins + +While working on a new feature in one branch, you may come across a bug +in the pre-existing code that you need to fix in order for work on that +feature to proceed. You could choose to switch briefly back to the +parent branch, develop the fix there, check it in, then merge the parent +back up to the feature branch in order to continue work, but that's +distracting. If the fix isn't for a critical bug, fixing it on the +parent branch can wait, so it's better to maintain your mental working +state by fixing the problem in place on the feature branch, then check +the fix in on the feature branch, resume work on the feature, and later +merge that fix down into the parent branch along with the feature. + +But now what happens if another branch *also* needs that fix? Let us say +our code repository has a branch for the current stable release, a +development branch for the next major version, and feature branches off +of the development branch. If we rebase each feature branch down into +the development branch as a single check-in, pushing only the rebase +check-in up to the parent repo, only that fix's developer has the +information locally to perform the cherry-pick of the fix onto the +stable branch. + +Developers working on new features often do not care about old stable +versions, yet that stable version may have an end user community that +depends on that version, who either cannot wait for the next stable +version or who wish to put off upgrading to it for some time. Such users +want backported bug fixes, yet the developers creating those fixes have +poor incentives to provide those backports. Thus the existence of +maintenance and support organizations, who end up doing such work. +(There is [a famous company][rh] that built a multi-billion dollar +enterprise on such work.) + +This work is far easier when each cherry-pick transfers completely and +cleanly from one branch to another, and we increase the likelihood of +achieving that state by working from the smallest check-ins that remain +complete. If a support organization must manually disentangle a fix from +a feature check-in, they are likely to introduce new bugs on the stable +branch. Even if they manage to do their work without error, it takes +them more time to do the cherry-pick that way. + +[rh]: https://en.wikipedia.org/wiki/Red_Hat + +### <a id="backouts"></a>6.5 Back-outs also work better with small check-ins + +The inverse of the cherry-pick merge is the back-out merge. If you push +only a collapsed version of a private working branch up to the parent +repo, those working from that parent repo cannot automatically back out +any of the individual check-ins that went into that private branch. +Others must either manually disentangle the problematic part of your +merge check-in or back out the entire feature. + +## <a id="better-plan"></a>7.0 Cherry-pick merges work better than rebase + +Perhaps there are some cases where a rebase-like transformation +is actually helpful, but those cases are rare, and when they do +come up, running a series of cherry-pick merges achieves the same +topology with several advantages: + + 1. In Fossil, cherry-pick merges preserve an honest and clear record + of history. Fossil remembers where a cherry-pick came from, and + it shows this in its timeline, so other developers can understand + how a cherry-pick based commit came together. + + Git lacks the ability to remember the source of a cherry-pick as + part of the commit. This fact has no direct bearing on this + document’s thesis, but we can make a few observations. First, Git + forgets history in more cases than in rebasing. Second, if Git + remembered the source of cherry-picks in commits, Git users might + have a better argument for avoiding rebase, because they’d have an + alternative that *didn’t* lose history. + + 2. Fossil’s [test before commit philosophy][tbc] means you can test a + cherry-pick before committing it. Because Fossil allows multiple + cherry-picks in a single commit and it remembers them all, you can + do this for a complicated merge in step-wise fashion. + + Git commits cherry-picks straight to the repository, so that if it + results in a bad state, you have to do something drastic like + `git reset --hard` to repair the damage. + + 3. Cherry-picks keep both the original and the revised check-ins, + so both timestamps are preserved. + +[tbc]: ./fossil-v-git.wiki#testing + +## <a id="conclusion"></a>8.0 Summary and conclusion + +Rebasing is an anti-pattern. It is dishonest. It deliberately +omits historical information. It causes problems for collaboration. +And it has no offsetting benefits. + +For these reasons, rebase is intentionally and deliberately omitted +from the design of Fossil. + + +[golden]: https://www.atlassian.com/git/tutorials/merging-vs-rebasing#the-golden-rule-of-rebasing +[gitrebase]: https://git-scm.com/book/en/v2/Git-Branching-Rebasing +[nagappan]: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2008-11.pdf +[weinberg]: https://books.google.com/books?id=76dIAAAAMAAJ diff --git a/www/relatedwork.md b/www/relatedwork.md new file mode 100644 index 0000000..4592e8f --- /dev/null +++ b/www/relatedwork.md @@ -0,0 +1,93 @@ +# Related Work + +## Support Projects + +* [SQLite]: C-language library that implements a small, fast, + self-contained, high-reliability, full-featured, SQL database engine +* [pikchr]: PIC-like markup language for diagrams in technical + documentation +* [althttpd]: simple, secure, and low resource usage webserver + that has run the https://sqlite.org/ website since 2004 +* [Lemon Parser Generator][lemon]: re-entrant and thread-safe + LALR(1) parser with a less error-prone grammar syntax than YACC or BISON +* [Makeheaders]: automatically generate header files for C/C++ projects + +## Fossil Inspired Projects + +* [libfossil]: 3rd party Fossil SCM Library API +* [fnc]: interactive text-based user interface for Fossil +* [ChiselApp]: Free Fossil SCM hosting! +* [Inskinerator]: The Fossil Skin Generator +* [Fuel]: cross-platform GUI front-end for the excellent Fossil SCM +* [fsl]: Tcl/Expect wrapper script extending Fossil functionality + + +## Editor Plugins + +* [Emacs-Fossil][emacsfsl]: GNU Emacs VC backend for the Fossil version + control system +* [VS Code][vscode]: Integrated Fossil source control in Visual Studio Code +* [Qt Creator Plugin][qtfsl]: Fossil SCM plugin for the Qt Creator IDE +* [Jetbrains IDE Plugin][jetbrains]: Fossil SCM plugin for [CLion], [IntelliJ], + [GoLand], and more +* [NetBeans Plugin][netbeans]: NetBeans plugin module to use Fossil SCM + +## Version Control/Software Configuration Management + +* [Git]: Free and open source distributed version control system +* [Subversion]: Apache's open source version control system +* [Mercurial]: free, distributed source control management tool +* [Game of Trees][got]: version control which prioritizes ease of use and + simplicity over flexibility +* [Darcs]: free and open source, cross-platform version control system +* [Pijul]: patch-based distributed version control system +* [Sapling]: A Scalable, User-Friendly Source Control System + +## Podcasts + +* [Corecursive #066][corec66]: The Untold Story of SQLite +* [The Changelog #454][changelog454]: Richard Hipp returns +* [The Changelog #201][changelog201]: Why SQLite succeeded as a database +* [bsdtalk194][bsdtalk]: Interview with D. Richard Hipp +* [Two Weeks of Databases][db2w]: Richard Hipp interviewed by Federico Razzoli +* [Software Engineering Daily][swed]: SQLite with D. Richard Hipp +* [Floss Weekly 26][floss26]: Interview with D. Richard Hipp, creator of SQLite + +## Miscellany + +* [Tcl]: a simple-to-learn yet very powerful programming language + +[althttpd]: https://sqlite.org/althttpd/doc/trunk/althttpd.md +[bsdtalk]: https://bsdtalk.blogspot.com/2010/07/bsdtalk194-fossil-scm-with-d-richard.html +[changelog201]: https://changelog.com/podcast/201 +[changelog454]: https://changelog.com/podcast/454 +[ChiselApp]: https://chiselapp.com/ +[CLion]: https://www.jetbrains.com/clion/ +[corec66]: https://corecursive.com/066-sqlite-with-richard-hipp/ +[Darcs]: http://darcs.net/ +[db2w]: https://youtu.be/2eaQzahCeh4 +[emacsfsl]: https://chiselapp.com/user/venks/repository/emacs-fossil/doc/tip/README.md +[floss26]: https://twit.tv/shows/floss-weekly/episodes/26 +[fnc]: https://fnc.sh +[fsl]: http://fossil.0branch.com/fsl +[Fuel]: https://fuel-scm.org/fossil/index +[Git]: https://git-scm.com +[GoLand]: https://www.jetbrains.com/go/ +[got]: https://gameoftrees.org +[Inskinerator]: https://tangentsoft.com/inskinerator +[IntelliJ]: https://www.jetbrains.com/idea/ +[jetbrains]: https://plugins.jetbrains.com/plugin/7479-fossil-integration +[lemon]: https://www.hwaci.com/sw/lemon/ +[libfossil]: https://fossil.wanderinghorse.net/r/libfossil/wiki/home +[Makeheaders]: https://fossil-scm.org/home/doc/trunk/tools/makeheaders.html +[Mercurial]: https://www.mercurial-scm.org/ +[netbeans]: https://chiselapp.com/user/backendzeit/repository/netbeans-fossil-plugin/index +[Pijul]: https://pijul.org +[pikchr]: https://pikchr.org +[qtfsl]: https://code.qt.io/cgit/qt-creator/plugin-fossil-scm.git/ +[Sapling]: https://sapling-scm.com +[SQLite]: https://sqlite.org/index.html +[Subversion]: https://subversion.apache.org/ +[swed]: https://softwareengineeringdaily.com/2015/11/13/sqlite-with-d-richard-hipp/ +[Tcl]: https://core.tcl-lang.org/tcl/wiki?name=Index +[VSCode]: https://marketplace.visualstudio.com/items?itemName=koog1000.fossil diff --git a/www/reviews.wiki b/www/reviews.wiki new file mode 100644 index 0000000..db06398 --- /dev/null +++ b/www/reviews.wiki @@ -0,0 +1,138 @@ +<title>Reviews</title> +<b>External links:</b> + + * [https://www.nixtu.info/2010/03/fossil-dvcs-on-go-first-impressions.html | + Fossil DVCS on the Go - First Impressions] + +<b>See Also:</b> + + * [./quotes.wiki | Short Quotes on Fossil, Git, And DVCSes] + +<b>Daniel writes on 2009-01-06:</b> + +<div class="indent"> +The reasons I use fossil are that it's the only version control I +have found that I can get working through the VERY annoying MS +firewalls at work.. (albeit through an ntlm proxy) and I just love +single .exe applications! +</div> + + +<b>Joshua Paine on 2010-10-22:</b> + +<div class="indent"> +With one of my several hats on, I'm in a small team using git. Another +team member just checked some stuff into trunk that should have been on +a branch. Nothing else had happened since, so in fossil I would have +just edited that commit and put it on a new branch. In git that can't +actually be done without danger once other people have pulled, so I had +to create a new commit rolling back the changes, then branch and cherry +pick the earlier changes, then figure out how to make my new branch +shared instead of private. Just want to say thanks for fossil making my +life easier on most of my projects, and being able to move commits to +another branch after the fact and shared-by-default branches are good +features. Also not having a misanthropic command line interface. +</div> + +<b>Stephan Beal writes on 2009-01-11:</b> + +<div class="indent"> +Sometime in late 2007 I came across a link to fossil on <a +href="https://sqlite.org/">sqlite.org</a>. It was a good thing I +bookmarked it, because I was never able to find the link again (it +might have been in a bug report or something). The reasons I first +took a close look at it were (A) it stemmed from the sqlite project, +which I've held in high regards for years (e.g. I wrote bindings for +it for Mozilla's SpiderMonkey JavaScript engine), and (B) it could run +as a CGI. That second point might seem a bit archaic, but in practice +CGI is the only way most hosted sites can set up a shared source +repository with multiple user IDs. (i'm not about to give out my only +account password or SSH key for my hosted sites, no matter how much I +trust the other developers, and none of my hosters allow me to run +standalone servers or add Apache modules.) + +So I tried it out. The thing which bugged me most about it was having +to type "commit" or "com" instead of "ci" for checking in (as is +custom in all other systems I've used), despite the fact that fossil +uses "ci" as a filter in things like the timeline view. Looking back +now, I have used fossil for about about 95% of my work in the past +year, in over 15 source trees, and I now get tripped up when I have to +use svn or cvs. + +So, having got over typing "fossil com -m ...", here's why I love it so much... + +Point #1: CGI + +Again, this sounds archaic, but fossil has allowed me to share source +trees which I cannot justifiably host in other projects I work on +(they don't belong to those projects), which I cannot host in google +code (because google code doesn't allow/recognize Public Domain as a +license, and I refuse to relicense just to accommodate them), and for +which SourceForge is overkill (and way too slow). With fossil I can +create a new repo, have it installed on my hoster +(https://fossil.wanderinghorse.net), and be commiting code to it within +5 minutes. + +Point #2: Wiki + +I hate wikis. I really do. Always have. They all have a different +syntax and the content tends to get really disorganized really +quickly. Their nature makes it difficult to reorganize them without +replacing old pages with lots of "has been moved to +<nowiki>[TheNewPage]</nowiki>" +links. I'm one of those "code for tomorrow" coders (i.e., code such +that it'll be easy to reorganize/refactor later). I like to document +the same way, and wikis make that problematic. Then again, no +documentation system is really good in that regard. + +That said, fossil has made me love having a centralized, common +documentation platform. Whereas I used to document everything in the +API docs (header files) and often include an ODT file for a library +manual, fossil has become my preferred platform for non-API +documentation because it's just so easy to do. No matter where I am, I +can log in and write (I write a lot). The added ability to export my +wiki pages, edit them in xemacs, and re-import them just makes it +nicer, as I can tweak as much as I want without ending up with 10 +"updated wiki page SoAndSo" messages in the commit log. + + +Point #3: running a server locally + +Fossil runs not only as a CGI, but as a server. I don't WANT to host +my own server (and don't have the rights to on my hosters). I hate +server-side maintenance (a hate born from years of administering +systems). But the server has other uses. When working on the wiki, bug +reports, etc., the local server is *the* way to do it. It's blazingly +fast and much more productive. When you're done, just run "fossil +push" and everything's synced. + + +Point #4: the single-file repository + +Having all controlled content inside a single container file has been +a godsend when it comes to backups and copying/moving a repository. +There are no access or file ownership issues, which are often +problematic with other server-side systems (at least on the initial +install). For about 5 years I administered a CVS repo for a company +for, and every time someone added a directory to the (huge and +dynamic) source tree I had to log in and "chmod 4775" the directory +before others could commit to it. Fossil's model inherently eliminates +that type of problem, and I'm a *huge* fan of solutions which +inherently (that is, due to their very nature) avoid certain +foreseeable problems. The single-file repository model's flexibility +would seem to become more problematic for huge repos (a few hundred +MB+) with many users with differing levels of access (e.g. OpenOffice, +Firefox, or the Linux Kernel), but 99.9% of projects never reach +anywhere near that size or complexity. + + +In summary: + +I remember my first reaction to fossil being, "this will be an +excellent solution for small projects (like the dozens we've all got +sitting on our hard drives but which don't justify the hassle of +version control)." A year of daily use in over 15 source trees has +confirmed that, and I continue to heartily recommend fossil to other +developers I know who also have their own collection of "unhosted" pet +projects. +</div> diff --git a/www/scgi.wiki b/www/scgi.wiki new file mode 100644 index 0000000..b01431f --- /dev/null +++ b/www/scgi.wiki @@ -0,0 +1,27 @@ +<title>Fossil SCGI</title> + +To run Fossil using SCGI, start the [/help/server|fossil server] command +with the --scgi command-line option. You will probably also want to +specific an alternative TCP/IP port using --port. For example: + +<pre> +fossil server $REPOSITORY --port 9000 --scgi +</pre> + +Then configure your SCGI-aware web-server to send SCGI requests to port +9000 on the machine where Fossil is running. A typical configuration for +this in Nginx is: + +<pre> +location ~ ^/demo_project/ { + include scgi_params; + scgi_pass localhost:9000; + scgi_param SCRIPT_NAME "/demo_project"; + scgi_param HTTPS "on"; +} +</pre> + +Note that Nginx does not normally send either the PATH_INFO or SCRIPT_NAME +variables via SCGI, but Fossil needs one or the other. So the configuration +above needs to add SCRIPT_NAME. If you do not do this, Fossil returns an +error. diff --git a/www/selfcheck.wiki b/www/selfcheck.wiki new file mode 100644 index 0000000..4a4bbd7 --- /dev/null +++ b/www/selfcheck.wiki @@ -0,0 +1,107 @@ +<title>Fossil Repository Integrity Self-Checks</title> + +Fossil is designed with features to give it a high level +of integrity so that users can have confidence that content will +never be mangled or lost by Fossil. +This note describes the defensive measures that +Fossil uses to help prevent information loss due to bugs. + +Fossil has been hosting itself and many other projects for +years now. Many bugs have been encountered. But, thanks in large +part to the defensive measures described here, no data has been +lost. The integrity checks are doing their job well.</p> + +<h2>Atomic Check-ins With Rollback</h2> + +The Fossil repository is stored in an +<a href="http://www.sqlite.org/">SQLite</a> database file. +([./tech_overview.wiki | Additional information] about the repository +file format.) +SQLite is very mature and stable and has been in wide-spread use for many +years, so we are confident it will not cause repository +corruption. SQLite +databases do not corrupt even if a program or system crash or power +failure occurs in the middle of the update. If some kind of crash +does occur in the middle of a change, then all the changes are rolled +back the next time that the database is accessed. + +A check-in operation in Fossil makes many changes to the repository +database. But all these changes happen within a single transaction. +If something goes wrong in the middle of the commit, even if that something +is a power failure or OS crash, then the transaction +is rolled back and the database is unchanged. + +<h2>Verification Of Delta Encodings Prior To Transaction Commit</h2> + +The content files that comprise the global state of a Fossil repository +are stored in the repository as a tree. The leaves of the tree are +stored as zlib-compressed BLOBs. Interior nodes are deltas from their +descendants. A lot of encoding is going on. There is +zlib-compression which is relatively well-tested but still might +cause corruption if used improperly. And there is the relatively +new [./delta_encoder_algorithm.wiki | delta-encoding mechanism] designed expressly for Fossil. We want +to make sure that bugs in these encoding mechanisms do not lead to +loss of data. + +To increase our confidence that everything in the repository is +recoverable, Fossil makes sure it can extract an exact replica +of every content file that it changes just prior to transaction +commit. So during the course of check-in (or other repository +operation) many different files +in the repository might be modified. Some files are simply +compressed. Other files are delta encoded and then compressed. +While all this is going on, Fossil makes a record of every file +and the SHA1 or SHA3-256 hash of the original content of that +file. Then just before transaction commit, Fossil re-extracts +the original content of all files that were written, recomputes +the hash, and verifies that the recomputed hash still matches. +If anything does not match up, an error +message is printed and the transaction rolls back. + +So, in other words, Fossil always checks to make sure it can +re-extract a file before it commits a change to that file. +Hence bugs in Fossil are unlikely to corrupt the repository in +a way that prevents us from extracting historical versions of +files. + +<h2>Checksum Over All Files In A Check-in</h2> + +Manifest artifacts that define a check-in have two fields (the +R-card and Z-card) that record MD5 hashes of the manifest itself +and of all other files in the manifest. Prior to any check-in +commit, these checksums are verified to ensure that the check-in +agrees exactly with what is on disk. Similarly, +the repository checksum is verified after a checkout to make +sure that the entire repository was checked out correctly. +Note that these added checks use a different hash algorithm (MD5) +in order to avoid common-mode failures in the hash +algorithm implementation. + + +<h2>Checksums On Structural Artifacts And Deltas</h2> + +Every [./fileformat.wiki | structural artifact] in a Fossil repository +contains a "Z-card" bearing an MD5 checksum over the rest of the +artifact. Any mismatch causes the structural artifact to be ignored. + +The [./delta_format.wiki | file delta format] includes a 32-bit +checksum of the target file. Whenever a file is reconstructed from +a delta, that checksum is verified to make sure the reconstruction +was done correctly. + +<h2>Reliability Versus Performance</h2> + +Some version control systems make a big deal out of being "high performance" +or the "fastest version control system". Fossil makes no such claims and has +no such ambition. Indeed, profiling indicates that Fossil bears a +substantial performance cost for +doing all of the checksumming and verification outlined above. +Fossil takes the philosophy of the +<a href="http://en.wikipedia.org/wiki/The_Tortoise_and_the_Hare">tortoise</a>: +reliability is more important than raw speed. The developers of +Fossil see no merit in getting the wrong answer quickly. + +Fossil may not be the fastest versioning system, but it is <i>fast enough</i>. +Fossil runs quickly enough to stay out of the developer's way. +Most operations complete in milliseconds, faster than you can press +the "Enter" key. diff --git a/www/selfhost.wiki b/www/selfhost.wiki new file mode 100644 index 0000000..40bc179 --- /dev/null +++ b/www/selfhost.wiki @@ -0,0 +1,78 @@ +<title>Fossil Self-Hosting Repositories</title> + +Fossil has self-hosted since 2007-07-21. As of 2025-02-11 +there are three publicly accessible repositories for the Fossil source code: + + 1. [https://fossil-scm.org/] + 2. [https://www2.fossil-scm.org/] + 3. [https://www3.fossil-scm.org/] + +The canonical repository is (1). Repositories (2) and (3) automatically +stay in synchronization with (1) via a +<a href="http://en.wikipedia.org/wiki/Cron">cron job</a> that invokes +"fossil sync" at regular intervals. +Repository (2) also publishes a +[https://github.com/drhsqlite/fossil-mirror|GitHub mirror of Fossil] +as a demonstration of [./mirrortogithub.md|how that can be done]. + +Note that the two secondary repositories are more than just read-only mirrors. +All three servers support full read/write capabilities. +Changes (such as new tickets or wiki or check-ins) can be implemented +on any of the three servers and those changes automatically propagate to the +other two servers. + +Server (1) runs as a [./aboutcgi.wiki|CGI script] on a +<a href="http://www.linode.com/">Linode</a> located in Dallas, TX +- on the same virtual machine that +hosts <a href="http://www.sqlite.org/">SQLite</a> and over a +dozen other smaller projects. This demonstrates that Fossil can run on +a low-power host processor. +Multiple fossil-based projects can easily be hosted on the same machine, +even if that machine is itself one of several dozen virtual machines on +a single physical box. The CGI script that runs the canonical Fossil +self-hosting repository is as follows: + +<pre> +#!/usr/bin/fossil +repository: /fossil/fossil.fossil +</pre> + +Server (3) ran for 10 years as a CGI script on a shared hosting account at +<a href="http://www.he.net/">Hurricane Electric</a> in Fremont, CA. +This server demonstrated the ability of +Fossil to run on an economical shared-host web account with no +privileges beyond port 80 HTTP access and CGI. It is not necessary +to have a dedicated computer with administrator privileges to run Fossil. +As far as we are aware, +Fossil is the only full-featured configuration management system +that can run in +such a restricted environment. The CGI script that ran on the +Hurricane Electric server was the same as the CGI script shown above, +except that the pathnames are modified to suit the environment: + +<pre> +#!/home/hwaci/bin/fossil +repository: /home/hwaci/fossil/fossil.fossil +</pre> + +In recent years, virtual private servers have become a more flexible and +less expensive hosting option compared to shared hosting accounts. +So on 2017-07-25, server (3) was moved +onto a $5/month "droplet" [https://en.wikipedia.org/wiki/Virtual_private_server|VPS] +(update: $6/month now) +from [https://www.digitalocean.com|Digital Ocean] +located in San Francisco. + +Server (3) is synchronized with the canonical server (1) by running +a command similar to the following via cron: + +<pre> +/usr/local/bin/fossil all sync -u +</pre> + +Server (2) is a +<a href="http://www.linode.com/">Linode</a> located in Newark, NJ +and set up just like the canonical server (1) with the addition of a +cron job for synchronization. The same cron job also runs the +[/help/git|fossil git export] command after each sync in order to +[./mirrortogithub.md#ex1|mirror all changes to GitHub]. diff --git a/www/server/any/althttpd.md b/www/server/any/althttpd.md new file mode 100644 index 0000000..7797103 --- /dev/null +++ b/www/server/any/althttpd.md @@ -0,0 +1,42 @@ +# Serving via althttpd + +[Althttpd][althttpd] +is a light-weight web server that has been used to implement the SQLite and +Fossil websites for well over a decade. Althttpd strives for simplicity, +security, ease of configuration, and low resource usage. + +To set up a Fossil server as CGI on a host running the althttpd web +server, follow these steps. +<ol> +<li>Get the althttpd webserver running on the host. This is easily +done by following the [althttpd documentation][althttpd]. + +<li>Create a CGI script for your Fossil repository. The script will +be typically be two lines of code that look something like this: + +~~~ + #!/usr/bin/fossil + repository: /home/yourlogin/fossils/project.fossil +~~~ + +Modify the filenames to conform to your system, of course. The +CGI script accepts [other options][cgi] besides the +repository:" line. You can add in other options as you desire, +but the single "repository:" line is normally all that is needed +to get started. + +<li>Make the CGI script executable. + +<li>Verify that the fossil repository file and the directory that contains +the repository are both writable by whatever user the web server is +running and. +</ol> + +And you are done. Visit the URL that corresponds to the CGI script +you created to start using your Fossil server. + +*[Return to the top-level Fossil server article.](../)* + + +[althttpd]: https://sqlite.org/althttpd/ +[cgi]: ../../cgi.wiki diff --git a/www/server/any/cgi.md b/www/server/any/cgi.md new file mode 100644 index 0000000..79ede92 --- /dev/null +++ b/www/server/any/cgi.md @@ -0,0 +1,117 @@ +# Serving via CGI + +A Fossil server can be run from most ordinary web servers as a CGI +program. This feature allows Fossil to seamlessly integrate into a +larger website. The [self-hosting Fossil repository web +site](../../selfhost.wiki) is implemented using CGI. See the +[How CGI Works](../../aboutcgi.wiki) page for background information +on the CGI protocol. + +To run Fossil as CGI, create a CGI script (here called "repo") in the +CGI directory of your web server with content like this: + + #!/usr/bin/fossil + repository: /home/fossil/repo.fossil + +Adjust the paths appropriately. It may be necessary to set certain +permissions on this file or to modify an `.htaccess` file or make other +server-specific changes. Consult the documentation for your particular +web server. The following permissions are *normally* required, but, +again, may be different for a particular configuration: + +* The Fossil binary (`/usr/bin/fossil` in the example above) + must be readable/executable. + +* *All* directories leading up to the Fossil binary must be readable + by the process which executes the CGI. + +* The CGI script must be executable for the user under which it will + run, which often differs from the one running the web server. + Consult your site's documentation or the web server’s system + administrator. + +* *All* directories leading to the CGI script must be readable by the + web server. + +* The repository file *and* the directory containing it must be + writable by the same account which executes the Fossil binary. + (This might differ from the user the web server normally runs + under.) The directory holding the repository file(s) needs to be + writable so that SQLite can write its journal files. When using + another access control system, such as AppArmor or SELinux, it may + be necessary to explicitly permit that account to read and write + the necessary files. Also verify a possible _systemd_ sandboxing of + the web server service, especially the combination of _ProtectSystem_, + _ProtectHome_, and _ReadWriteDirectories_/_ReadWritePaths_. + +* Fossil must be able to create temporary files in a + [directory that varies by host OS](../../env-opts.md#temp). When the + CGI process is operating [within a chroot](../../chroot.md), + ensure that this directory exists and is readable/writeable by the + user who executes the Fossil binary. + +Once the CGI script is set up correctly, and assuming your server is +also set correctly, you should be able to access your repository with a +URL like: <b>http://mydomain.org/cgi-bin/repo</b> This is assuming you +are running a web server like Apache that uses a “`cgi-bin`” directory +for scripts like our “`repo`” example. + +To serve multiple repositories from a directory using CGI, use the +"directory:" tag in the CGI script rather than "repository:". You +might also want to add a "notfound:" tag to tell where to redirect if +the particular repository requested by the URL is not found: + + #!/usr/bin/fossil + directory: /home/fossil/repos + notfound: http://url-to-go-to-if-repo-not-found/ + +Once deployed, a URL like: <b>http://mydomain.org/cgi-bin/repo/XYZ</b> +will serve up the repository `/home/fossil/repos/XYZ.fossil` if it +exists. + +Additional options available to the CGI script are [documented +separately](../../cgi.wiki). + +#### CGI with Apache behind an Nginx proxy + +For the case where the Fossil repositories live on a computer, itself behind +an Internet-facing machine that employs Nginx to reverse proxy HTTP(S) requests +and take care of the TLS part of the connections in a transparent manner for +the downstream web servers, the CGI parameter `HTTPS=on` might not be set. +However, Fossil in CGI mode needs it in order to generate the correct links. + +Apache can be instructed to pass this parameter further to the CGI scripts for +TLS connections with a stanza like + + SetEnvIf X-Forwarded-Proto "https" HTTPS=on + +in its config file section for CGI, provided that + + proxy_set_header X-Forwarded-Proto $scheme; + +has been be added in the relevant proxying section of the Nginx config file. + +*[Return to the top-level Fossil server article.](../)* + +#### Apache mod_cgi and `CONTENT_LENGTH` + +At some point in its 2.4.x family, Apache's `mod_cgi` stopped relaying +the Content-Length header in the HTTP reply from CGIs back to clients. +However, Fossil clients prior to 2024-04-17 depended on seeing the +Content-Length header and were unable to handle HTTP replies without +one. The change in Apache behavior caused "fossil clone" and "fossil +sync" to stop working. There are two possible fixes to this problem: + + 1. Restore legacy behavior in Apache by adding + the following to the Apache configuration, scoped to the `<Directory>` + entry or entries in which fossil is being served via CGI: + <blockquote><pre> + <Directory ...> + SetEnv ap_trust_cgilike_cl "yes" + </Directory> + </pre></blockquote> + + 2. Upgrade your Fossil client to any trunk check-in after 2024-04-17, + as Fossil was upgraded to be able to deal with the missing + Content-Length field by + [check-in a8e33fb161f45b65](/info/a8e33fb161f45b65). diff --git a/www/server/any/http-over-ssh.md b/www/server/any/http-over-ssh.md new file mode 100644 index 0000000..8bc6fdb --- /dev/null +++ b/www/server/any/http-over-ssh.md @@ -0,0 +1,173 @@ +# Forcing Use of Fossil’s RBAC over SSH + +Andy Bradford posted a [clever solution][sshfc] to the problem of +Fossil’s RBAC system [being ignored](../../caps/#webonly) over `ssh://` +URLs: use OpenSSH’s `ForceCommand` feature to route the sync transfer +protocol data over `fossil http` rather than `fossil test-http`. + +The setup for this is complicated, but it’s a worthy option when you +need encrypted communications between the client and server, you already +have SSH set up, and [the HTTPS alternative](../../ssl.wiki) is +unworkable for some reason. + + +## 1. Force remote Fossil access through a wrapper script <a id="sshd"></a> + +Put something like the following into the `sshd_config` file on the +Fossil repository server: + +``` ssh-config +Match Group fossil + X11Forwarding no + AllowTcpForwarding no + AllowAgentForwarding no + ForceCommand /home/fossil/bin/wrapper +``` + +This file is usually found in `/etc/ssh`, but some OSes put it +elsewhere. + +The first line presumes that we will put all users who need to use our +Fossil repositories into the `fossil` group, as we will do +[below](#perms). You could instead say something like: + +``` ssh-config +Match User alice,bob,carol,dave +``` + +You have to list the users allowed to use Fossil in this case because +your system likely has a system administrator that uses SSH for remote +shell access, so you want to *exclude* that user from the list. For the +same reason, you don’t want to put the `ForceCommand` directive outside +a `Match` block of some sort. + +You could instead list the exceptions: + +``` ssh-config +Match User !evi +``` + +This would permit only [Evi the System Administrator][evi] to bypass this +mechanism. + +[evi]: https://en.wikipedia.org/wiki/Evi_Nemeth + +If you have a user that needs both interactive SSH shell access *and* +Fossil access, exclude that user from the `Match` rule and use Fossil’s +normal `ssh://` URL scheme for those cases. This user will bypass the +Fossil RBAC, but they effectively have Setup capability on those +repositories anyway by having full read/write access to the DB files via +the shell. + + +## 2. Rewrite the sync command with that wrapper <a id="wrapper"></a> + +When Fossil syncs over SSH, it attempts to launch a remote Fossil +instance with certain parameters in order to set up the HTTP-based sync +protocol over that SSH tunnel. We need to preserve some of this command +and rewrite other parts to make this work. + +Here is a simpler variant of Andy’s original wrapper script: + +``` sh +#!/bin/bash +set -- $SSH_ORIGINAL_COMMAND +while [ $# -gt 1 ] ; do shift ; done +export REMOTE_USER="$USER" +ROOT=/home/fossil +exec "$ROOT/bin/fossil" http "$ROOT/museum/$(/bin/basename "$1")" +``` + +The substantive changes are: + +1. Move the command rewriting bits to the start. + +2. Be explicit about executable paths. You might extend this idea by + using chroot, BSD jails, Linux containers, etc. + +3. Restrict the Fossil repositories to a single flat subdirectory under + the `fossil` user’s home directory. This scheme is easier to secure + than one allowing subdirectories, since you’d need to take care of + `../` and such to prevent a sandbox escape. + +4. Don’t take the user name via the SSH command; to this author’s mind, + the user should not get to override their Fossil user name on the + remote server, as that permits impersonation. The identity you + present to the SSH server must be the same identity that the Fossil + repository you’re working with knows you by. Since the users + selected by “`Match`” block above are dedicated to using only Fossil + in this setup, this restriction shouldn’t present a practical problem. + +The script’s shebang line assumes `/bin/sh` is POSIX-compliant, but that +is not the case everywhere. If the script fails to run on your system, +try changing this line to point at `bash`, `dash`, `ksh`, or `zsh`. Also +check the absolute paths for local correctness: is `/bin/basename` +installed on your system, for example? + +Under this scheme, you clone with a command like: + + $ fossil clone ssh://HOST/repo.fossil + +This will clone the remote `/home/fossil/museum/repo.fossil` repository +to your local machine under the same name and open it into a “`repo/`” +subdirectory. Notice that we didn’t have to give the `museum/` part of +the path: it’s implicit per point #3 above. + +This presumes your local user name matches the remote user name. Unlike +with `http[s]://` URLs, you don’t have to provide the `USER@` part to +get authenticated access +since this scheme doesn’t permit anonymous cloning. Only +if these two user names are different do you need to add the `USER@` bit to the +URL. + + +## 3. Set permissions <a id="perms"></a> + +This scheme assumes that the users covered by the `Match` rule can read +the wrapper script from where you placed it and execute it, and that +they have read/write access on the directory where the Fossil +repositories are stored. + +You can achieve all of this on a Linux box with: + +``` shell +sudo adduser fossil +for u in alice bob carol dave ; do + sudo adduser $u + sudo gpasswd -a fossil $u +done +sudo -i -u fossil +chmod 710 . +mkdir -m 750 bin +mkdir -m 770 museum +ln -s /usr/local/bin/fossil bin +``` + +You then need to copy the Fossil repositories into `~fossil/museum` and +make them readable and writable by group `fossil`. These repositories +presumably already have Fossil users configured, with the necessary +[user capabilities](../../caps/), the point of this article being to +show you how to make Fossil-over-SSH pay attention to those caps. + +You must also permit use of `REMOTE_USER` on each shared repository. +Fossil only pays attention to this environment variable in certain +contexts, of which “`fossil http`” is not one. Run this command against +each repo to allow that: + +``` shell +echo "INSERT OR REPLACE INTO config VALUES ('remote_user_ok',1,strftime('%s','now'));" | +fossil sql -R museum/repo.fossil +``` + +Now you can configure SSH authentication for each user. Since Fossil’s +password-saving feature doesn’t work in this case, I suggest setting up +SSH keys via `~USER/.ssh/authorized_keys` since the SSH authentication +occurs on each sync, which Fossil’s default-enabled autosync setting +makes frequent. + +Equivalent commands for other OSes should be readily discerned from the +script above. + +[sshfc]: forum:/forumpost/0d7d6c3df41fcdfd + +<div style="height:50em" id="this-space-intentionally-left-blank"></div> diff --git a/www/server/any/index.md b/www/server/any/index.md new file mode 100644 index 0000000..ccf2d94 --- /dev/null +++ b/www/server/any/index.md @@ -0,0 +1,11 @@ +# Portable Fossil Service Options + +- [Fossil as a standalone HTTP server](./none.md) +- [SCGI under your web server of choice](./scgi.md) +- [CGI under your web server of choice](./cgi.md) +- [CGI under `althttpd`](./althttpd.md) +- [Behind `stunnel` to get TLS encryption](./stunnel.md) +- [`inetd`](./inetd.md) +- [`xinetd`](./xinetd.md) + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/any/inetd.md b/www/server/any/inetd.md new file mode 100644 index 0000000..35766c7 --- /dev/null +++ b/www/server/any/inetd.md @@ -0,0 +1,52 @@ +# Serving via inetd + +A Fossil server can be launched on-demand by `inetd` by using the +[`fossil http`](/help/http) command. To do so, add a line like the +following to its configuration file, typically `/etc/inetd.conf`: + + 80 stream tcp nowait.1000 root /usr/bin/fossil /usr/bin/fossil http /home/fossil/repo.fossil + +In this example, you are telling `inetd` that when an incoming +connection appears on TCP port 80 that it should launch the program +`/usr/bin/fossil` with the arguments shown. Obviously you will need to +modify the pathnames for your particular setup. The final argument is +either the name of the fossil repository to be served or a directory +containing multiple repositories. + +If you use a non-standard TCP port on systems where the port +specification must be a symbolic name and cannot be numeric, add the +desired name and port to `/etc/services`. For example, if you want your +Fossil server running on TCP port 12345 instead of 80, you will need to +add: + + fossil 12345/tcp # fossil server + +and use the symbolic name “`fossil`” instead of the numeric TCP port +number (“12345” in the above example) in `inetd.conf`. + +Notice that we configured `inetd` to launch Fossil as root. See the +top-level section on “[The Fossil Chroot +Jail](../../chroot.md)” for the consequences of this and +alternatives to it. + +You can instead configure `inetd` to bind to a higher-numbered TCP port, +allowing Fossil to be run as a normal user. In that case, Fossil will +not put itself into a chroot jail, because it assumes you have set up +file permissions and such on the server appropriate for that user. + +The `inetd` daemon must be enabled for this to work, and it must be +restarted whenever its configuration file changes. + +This is a more complicated method than the [standalone HTTP server +method](./none.md), but it has the advantage of only using system +resources when an actual connection is attempted. If no one ever +connects to that port, a Fossil server will not (automatically) run. It +has the disadvantage of requiring "root" access, which may not be +available to you, either due to local IT policy or because of +restrictions at your shared Internet hosting service. + +For further details, see the relevant section in your system's +documentation. The FreeBSD Handbook covers `inetd` in [this +chapter](https://www.freebsd.org/doc/en/books/handbook/network-inetd.html). + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/any/none.md b/www/server/any/none.md new file mode 100644 index 0000000..2208091 --- /dev/null +++ b/www/server/any/none.md @@ -0,0 +1,56 @@ +# Standalone HTTP Server + +The easiest way to set up a Fossil server is to use either the +[`server`](/help/server) or [`ui`](/help/ui) command: + +* **fossil server** _REPOSITORY_ +* **fossil ui** _REPOSITORY_ + +The _REPOSITORY_ argument is either the name of the repository file or a +directory containing many repositories named “`*.fossil`”. Both of these +commands start a Fossil server, usually on TCP port 8080, though a +higher numbered port will be used instead if 8080 is already occupied. + +You can access these using URLs of the form **http://localhost:8080/**, +or if _REPOSITORY_ is a directory, URLs of the form +**http://localhost:8080/**_repo_**/** where _repo_ is the base name of +the repository file without the “`.fossil`” suffix. + +There are several key differences between “`ui`” and “`server`”: + +* “`ui`” always binds the server to the loopback IP address (127.0.0.1) + so that it cannot serve to other machines. + +* Anyone who visits this URL is treated as the all-powerful Setup + user, which is why the first difference exists. + +* “`ui`” launches a local web browser pointed at this URL. + +You can omit the _REPOSITORY_ argument if you run one of the above +commands from within a Fossil checkout directory to serve that +repository: + + $ fossil ui # or... + $ fossil server + +You can abbreviate Fossil sub-commands as long as they are unambiguous. +“`server`” can currently be as short as “`ser`”. + +You can serve a directory containing multiple `*.fossil` files like so: + + $ fossil server --port 9000 --repolist /path/to/repo/dir + +As a security precaution, when the “`server`” or +“`ui`” commands are run as `root` Fossil will drop into +a [chroot](../../chroot.md) with the directory where the repository is located +as the new root of the filesystem. + +There is an [example script](/file/tools/fslsrv) in the Fossil +distribution that wraps `fossil server` to produce more complicated +effects. Feel free to take it, study it, and modify it to suit your +local needs. + +See the [online documentation](/help/server) for more information on the +options and arguments you can give to these commands. + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/any/scgi.md b/www/server/any/scgi.md new file mode 100644 index 0000000..4c02158 --- /dev/null +++ b/www/server/any/scgi.md @@ -0,0 +1,72 @@ +# Serving via SCGI + +There is an alternative to running Fossil as a [standalone HTTP +server](./none.md), which is to run it in SimpleCGI (a.k.a. SCGI) mode, +which uses the same [`fossil server`](/help/server) command as for HTTP +service. Simply add the `--scgi` command-line option and the stand-alone +server will speak the SCGI protocol rather than raw HTTP. + +This can be used with a web server such as [nginx](http://nginx.org) +which does not support [Fossil’s CGI mode](./cgi.md). + +A basic nginx configuration to support SCGI with Fossil looks like this: + + location /code/ { + include scgi_params; + scgi_param SCRIPT_NAME "/code"; + scgi_pass localhost:9000; + } + +The `scgi_params` file comes with nginx, and it simply translates nginx +internal variables to `scgi_param` directives to create SCGI environment +variables for the proxied program; in this case, Fossil. Our explicit +`scgi_param` call to define `SCRIPT_NAME` adds one more variable to this +set, which is necessary for this configuration to work properly, because +our repo isn’t at the root of the URL hierarchy. Without it, when Fossil +generates absolute URLs, they’ll be missing the `/code` part at the +start, which will typically cause [404 errors][404]. + +The final directive simply tells nginx to proxy all calls to URLs under +`/code` down to an SCGI program on TCP port 9000. We can temporarily +set Fossil up as a server on that port like so: + + $ fossil server /path/to/repo.fossil --scgi --localhost --port 9000 & + +The `--scgi` option switches Fossil into SCGI mode from its default, +which is [stand-alone HTTP server mode](./none.md). All of the other +options discussed in that linked document — such as the ability to serve +a directory full of Fossil repositories rather than just a single +repository — work the same way in SCGI mode. + +The `--localhost` option is simply good security: we’re using nginx to +expose Fossil service to the outside world, so there is no good reason +to allow outsiders to contact this Fossil SCGI server directly. + +Giving an explicit non-default TCP port number via `--port` is a good +idea to avoid conflicts with use of Fossil’s default TCP service port, +8080, which may conflict with local uses of `fossil ui` and such. + +We characterized the SCGI service start command above as “temporary” +because running Fossil in the background like that means it won’t start +back up on a reboot of the server. A simple solution to that is to add +that command to `/etc/rc.local` on systems that have it. However, you +might want to consider setting Fossil up as an OS service instead, so +that you get the benefits of the platform’s service management +framework: + +* [Linux (systemd)](../debian/service.md) +* [Windows service](../windows/service.md) +* [macOS (launchd)](../macos/service.md) +* [xinetd](../any/xinetd.md) +* [inetd](../any/inetd.md) + +We go into more detail on nginx service setup with Fossil in our +[Debian/Ubuntu specific guide](../debian/nginx.md), which also +gets you TLS service. + +Similarly, our [OpenBSD specific guide](../openbsd/fastcgi.md) details how +to setup a Fossil server using httpd and FastCGI on OpenBSD. + +*[Return to the top-level Fossil server article.](../)* + +[404]: https://en.wikipedia.org/wiki/HTTP_404 diff --git a/www/server/any/stunnel.md b/www/server/any/stunnel.md new file mode 100644 index 0000000..c8824a7 --- /dev/null +++ b/www/server/any/stunnel.md @@ -0,0 +1,81 @@ +# Serving via stunnel + +[`stunnel`](https://www.stunnel.org/) is a TLS/SSL proxy for programs +that themselves serve only via HTTP, such as Fossil. (Fossil *can* speak +HTTPS, but only as a client.) `stunnel` decodes the HTTPS data from the +outside world as HTTP before passing it to Fossil, and it encodes the +HTTP replies from Fossil as HTTPS before sending them to the remote host +that made the request. + +You can run `stunnel` in one of two modes: socket listener — much like +in our [`inetd` doc](./inetd.md) — and as an HTTP reverse proxy. We’ll +cover both cases here, separately. + + +## <a id="sa"></a>Socket Activation + +The following `stunnel.conf` configuration configures it to run Fossil +in socket listener mode, launching Fossil only when an HTTPS hit comes +in, then shutting it back down as soon as the transaction is complete: + +```dosini + [fossil] + accept = 443 + TIMEOUTclose = 0 + exec = /usr/bin/fossil + execargs = /usr/bin/fossil http /home/fossil/ubercool.fossil --https + cert = /etc/letsencrypt/live/ubercool-project.org/fullchain.pem + key = /etc/letsencrypt/live/ubercool-project.org/privkey.pem + ciphers = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES128-SHA:DES-CBC3-SHA + options = CIPHER_SERVER_PREFERENCE +``` + +This configuration shows the TLS certificate generated by the [Let’s +Encrypt](https://letsencrypt.org) [Certbot](https://certbot.eff.org) in +[certonly mode](https://certbot.eff.org/lets-encrypt/debianbuster-other). +There are other ways to get TLS certificates, but this is a popular and +free option. + +You will need to adjust the site names and paths in this example. Where +this file goes varies by OS type, so check the man pages on your system +to find out where it should be locally. + +See the `stunnel` documentation for further details about this +configuration file. + +It is important that the [`fossil http`](/help/http) command in that +configuration include the `--https` option to let Fossil know to use +“`https://`” instead of “`http://`” in generated hyperlinks. + + + +## <a id="proxy"></a>Reverse Proxy + +You can instead have Fossil running in the background in [standalone +HTTP server mode](./none.md), bound to a high random TCP port number on +localhost via the `--localhost` and `--port` flags, then configure +`stunnel` to reverse proxy public HTTPS connections down to it via HTTP. + +The configuration is the same as the above except that you drop the +`exec` and `execargs` directives and add this instead: + +```dosini + connect = 9000 +``` + +That tells `stunnel` to connect to an already-running process listening +on the given TCP port number. + +There are a few advantages to this mode: + +1. At the cost of some server memory and a tiny bit of idle CPU time, + Fossil remains running so that hits can be served a smidge faster + than in socket listener mode, where the Fossil binary has to be + loaded and re-initialized on each HTTPS hit. + +2. The socket listener mode doesn’t work on all platforms that + `stunnel` runs on, particularly [on Windows](../windows/stunnel.md). + +*[Return to the top-level Fossil server article.](../)* + +<div style="height:50em" id="this-space-intentionally-left-blank"></div> diff --git a/www/server/any/xinetd.md b/www/server/any/xinetd.md new file mode 100644 index 0000000..9e93502 --- /dev/null +++ b/www/server/any/xinetd.md @@ -0,0 +1,27 @@ +# Serving via xinetd + +Some operating systems have replaced the old Unix `inetd` daemon with +`xinetd`, which has a similar mission but with a very different +configuration file format. + +The typical configuration file is either `/etc/xinetd.conf` or a subfile +in the `/etc/xinetd.d` directory. You need a configuration something +like this for Fossil: + + service http + { + port = 80 + socket_type = stream + wait = no + user = root + server = /usr/bin/fossil + server_args = http /home/fossil/repos/ + } + +This example configures Fossil to serve multiple repositories under the +`/home/fossil/repos/` directory. + +Beyond this, see the general commentary in our article on [the `inetd` +method](./inetd.md) as they also apply to service via `xinetd`. + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/debian/index.md b/www/server/debian/index.md new file mode 100644 index 0000000..220d177 --- /dev/null +++ b/www/server/debian/index.md @@ -0,0 +1,6 @@ +# Debian/Ubuntu Specific Fossil Service Options + +- [SCGI under nginx](./nginx.md) +- [`systemd`](./service.md) + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/debian/nginx.md b/www/server/debian/nginx.md new file mode 100644 index 0000000..0140b37 --- /dev/null +++ b/www/server/debian/nginx.md @@ -0,0 +1,395 @@ +# Serving via nginx on Debian and Ubuntu + +This document is an extension of [the platform-independent SCGI +instructions][scgii], which may suffice for your purposes if your needs +are simple. + +Here, we add more detailed information on nginx itself, plus details +about running it on Debian type OSes. This document was originally +written for and tested on Debian 10 (Buster) and Ubuntu 20.04, which +were common Tier 1 OS offerings for [virtual private servers][vps] +at the time. The same configuration appears to run on Ubuntu 22.04 +LTS without change. This material may not work for older OSes. It is +known in particular to not work as given for Debian 9 and older! + +We also cover [adding TLS](#tls) to the basic configuration, because several +details depend on the host OS and web stack details. Besides, TLS is +widely considered part of the baseline configuration these days. + +[scgii]: ../any/scgi.md +[vps]: https://en.wikipedia.org/wiki/Virtual_private_server + + +## <a id="benefits"></a>Benefits + +This scheme is considerably more complicated than the [standalone HTTP +server](../any/none.md) and [CGI options](../any/cgi.md). Even with the +benefit of this guide and pre-built binary packages, it requires quite a +bit of work to set it up. Why should you put up with this complexity? +Because it gives many benefits that are difficult or impossible to get +with the less complicated options: + +* **Power** — nginx is one of the most powerful web servers in the + world. The chance that you will run into a web serving wall that you + can’t scale with nginx is very low. + + To give you some idea of the sort of thing you can readily + accomplish with nginx, your author runs a single public web server + that provides transparent name-based virtual hosting for four + separate domains: + + * <p>One is entirely static, not involving any dynamic content or + Fossil integration at all.</p> + + * <p>Another is served almost entirely by Fossil, with a few select + static content exceptions punched past Fossil, which are handled + entirely via nginx.</p> + + * <p>The other two domains are aliases for one another — e.g. + `example.com` and `example.net` — with most of the content being + static. This pair of domains has several unrelated Fossil repo + proxies attached to various sections of the URI hierarchy.</p> + + By using nginx, I was able to do all of the above with minimal + repetition between the site configurations. + +* **Integration** — Because nginx is so popular, it integrates with +many different technologies, and many other systems integrate with it in +turn. This makes it great middleware, sitting between the outer web +world and interior site services like Fossil. It allows Fossil to +participate seamlessly as part of a larger web stack. + +* **Availability** — nginx is already in most operating system binary +package repositories, so you don’t need to go out of your way to get it. + + +## <a id="modes"></a>Fossil Service Modes + +Fossil provides four major ways to access a repository it’s serving +remotely, three of which are straightforward to use with nginx: + +* **HTTP** — Fossil has [a built-in HTTP server](../any/none.md). + While this method is efficient and it’s + possible to use nginx to proxy access to another HTTP server, we + don’t see any particularly good reason to make nginx reinterpret + Fossil’s own implementation of HTTP when we have a better option. + (But see [below](#http).) + +* **SSH** — This method exists primarily to avoid the need for HTTPS, + but we *want* HTTPS. (We’ll get to that [below](#tls).) + There is probably a way to get nginx to proxy Fossil to HTTPS via + SSH, but it would be pointlessly complicated. + +* **CGI** — This method is simple but inefficient, because it launches + a separate Fossil instance on every HTTP hit. + Since Fossil is a relatively small self-contained program, and it’s + designed to start up quickly, this method can work well in a + surprisingly large number of cases. + Nevertheless, we will avoid this option in this document because + we’re already buying into a certain amount of complexity here in + order to gain power. There’s no sense in throwing away any of that + hard-won performance on CGI overhead. + +* **SCGI** — The [SCGI protocol][scgip] provides the simplicity of CGI + without its performance problems. + +SCGI it is, then. + +[scgip]: https://en.wikipedia.org/wiki/Simple_Common_Gateway_Interface + + +## <a id="deps"></a>Installing the Dependencies + +The first step is to install some non-default packages we’ll need. SSH into +your server, then say: + + $ sudo apt install fossil nginx + +You can leave “`fossil`” out of that if you’re building Fossil from +source to get a more up-to-date version than is shipped with the host +OS. + + +## <a id="scgi"></a>Running Fossil in SCGI Mode + +For the following nginx configuration to work, it needs to contact a +background Fossil instance speaking the SCGI protocol. There are +[many ways](../) to set that up, such as [with `systemd`](./service.md) +on mainstream Linux distros. Another way is to [containerize][ctz] your +repository servers, then use the [`fslsrv` wrapper for Podman][fspm] to +generate `systemd` units for use by the front-end proxy. + +However you do it, you need to match up the TCP port numbers between it +and those in the nginx configuration below. + +[ctz]: ../../containers.md +[fspm]: https://tangentsoft.com/fossil/dir/bin + + +## <a id="config"></a>Configuration + +On Debian and Ubuntu systems the primary user-level configuration file +for nginx is `/etc/nginx/sites-enabled/default`. I recommend that this +file contain only a list of include statements, one for each site that +server hosts: + + include local/example.com + include local/foo.net + +Those files then each define one domain’s configuration. Here, +`/etc/nginx/local/example.com` contains the configuration for +`*.example.com` and its alias `*.example.net`; and `local/foo.net` +contains the configuration for `*.foo.net`. + +The configuration for our `example.com` web site, stored in +`/etc/nginx/sites-enabled/local/example.com` is: + +---- + + server { + server_name .example.com .example.net ""; + include local/generic; + include local/code; + + access_log /var/log/nginx/example.com-https-access.log; + error_log /var/log/nginx/example.com-https-error.log; + + # Bypass Fossil for the static documentation generated from + # our source code by Doxygen, so it merges into the embedded + # doc URL hierarchy at Fossil’s $ROOT/doc without requiring that + # these generated files actually be stored in the repo. This + # also lets us set aggressive caching on these docs, since + # they rarely change. + location /code/doc/html { + root /var/www/example.com/code/doc/html; + + location ~* \.(html|ico|css|js|gif|jpg|png)$ { + add_header Vary Accept-Encoding; + access_log off; + expires 7d; + } + } + + # Redirect everything under /code to the Fossil instance + location /code { + include local/code; + + # Extended caching for URLs that include unique IDs + location ~ "/(artifact|doc|file|raw)/[0-9a-f]{40,64}" { + add_header Cache-Control "public, max-age=31536000, immutable"; + include local/code; + access_log off; + } + + # Lesser caching for URLs likely to be quasi-static + location ~* \.(css|gif|ico|js|jpg|png)$ { + add_header Vary Accept-Encoding; + include local/code; + access_log off; + expires 7d; + } + } + } + +---- + +As you can see, this is a pure extension of [the basic nginx service +configuration for SCGI][scgii], showing off a few ideas you might want to +try on your own site, such as static asset proxying. + +You also need a `local/code` file containing: + + include scgi_params; + scgi_pass 127.0.0.1:12345; + scgi_param SCRIPT_NAME "/code"; + +We separate that out because nginx refuses to inherit certain settings +between nested location blocks, so rather than repeat them, we extract +them to this separate file and include it from both locations where it’s +needed. You see this above where we set far-future expiration dates on +files served by Fossil via URLs that contain hashes that change when the +content changes. It tells your browser that the content of these URLs +can never change without the URL itself changing, which makes your +Fossil-based site considerably faster. + +Similarly, the `local/generic` file referenced above helps us reduce unnecessary +repetition among the multiple sites this configuration hosts: + + root /var/www/$host; + + listen 80; + listen [::]:80; + + charset utf-8; + +There are some configuration directives that nginx refuses to substitute +variables into, citing performance considerations, so there is a limit +to how much repetition you can squeeze out this way. One such example +are the `access_log` and `error_log` directives, which follow an obvious +pattern from one host to the next. Sadly, you must tolerate some +repetition across `server { }` blocks when setting up multiple domains +on a single server. + +The configuration for `foo.net` is similar. + +See [the nginx docs](https://nginx.org/en/docs/) for more ideas. + + +## <a id="http"></a>Proxying HTTP Anyway + +[Above](#modes), we argued that proxying SCGI is a better option than +making nginx reinterpret Fossil’s own implementation of HTTP. If you +want Fossil to speak HTTP, just [set Fossil up as a standalone +server](../any/none.md). And if you want nginx to [provide TLS +encryption for Fossil](#tls), proxying HTTP instead of SCGI provides no +benefit. + +However, it is still worth showing the proper method of proxying +Fossil’s HTTP server through nginx if only to make reading nginx +documentation on other sites easier: + + location /code { + rewrite ^/code(/.*) $1 break; + proxy_pass http://127.0.0.1:12345; + } + +The most common thing people get wrong when hand-rolling a configuration +like this is to get the slashes wrong. Fossil is sensitive to this. For +instance, Fossil will not collapse double slashes down to a single +slash, as some other HTTP servers will. + + +## <a id="large-uv"></a> Allowing Large Unversioned Files + +By default, nginx only accepts HTTP messages [up to a +meg](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size) +in size. Fossil chunks its sync protocol such that this is not normally +a problem, but when sending [unversioned content][uv], it uses a single +message for the entire file. Therefore, if you will be storing files +larger than this limit as unversioned content, you need to raise the +limit. Within the `location` block: + + # Allow large unversioned file uploads, such as PDFs + client_max_body_size 20M; + +[uv]: ../../unvers.wiki + + +## <a id="fail2ban"></a> Integrating `fail2ban` + +One of the nice things that falls out of proxying Fossil behind nginx is +that it makes it easier to configure `fail2ban` to recognize attacks on +Fossil and automatically block them. Fossil logs the sorts of errors we +want to detect, but it does so in places like the repository’s admin +log, a SQL table, which `fail2ban` doesn’t know how to query. By putting +Fossil behind an nginx proxy, we convert these failures to log file +form, which `fail2ban` is designed to handle. + +First, install `fail2ban`, if you haven’t already: + + sudo apt install fail2ban + +We’d like `fail2ban` to react to Fossil `/login` failures. The stock +configuration of `fail2ban` only detects a few common sorts of SSH +attacks by default, and its included (but disabled) nginx attack +detectors don’t include one that knows how to detect an attack on +Fossil. We have to teach it by putting the following into +`/etc/fail2ban/filter.d/nginx-fossil-login.conf`: + + [Definition] + failregex = ^<HOST> - .*POST .*/login HTTP/..." 401 + +That teaches `fail2ban` how to recognize the errors logged by Fossil. + +Then in `/etc/fail2ban/jail.local`, add this section: + + [nginx-fossil-login] + enabled = true + logpath = /var/log/nginx/*-https-access.log + +The last line is the key: it tells `fail2ban` where we’ve put all of our +per-repo access logs in the nginx config above. + +There’s a [lot more you can do][dof2b], but that gets us out of scope of +this guide. + + +[dof2b]: https://www.digitalocean.com/community/tutorials/how-to-protect-an-nginx-server-with-fail2ban-on-ubuntu-14-04 + + +## <a id="tls"></a> Adding TLS (HTTPS) Support + +One of the [many ways](../../ssl.wiki) to provide TLS-encrypted HTTP +access (a.k.a. HTTPS) to Fossil is to run it behind a web proxy that +supports TLS. Because one such option is nginx, it’s best to delegate +TLS to it if you were already using nginx for some other reason, such as +static content serving, with only part of the site being served by +Fossil. + +The simplest way by far to do this is to use [Let’s Encrypt][LE]’s +[Certbot][CB], which can configure nginx for you and keep its +certificates up to date. You need but follow their [nginx on Ubuntu 20 +guide][CBU]. We had trouble with this in the past, but either Certbot +has gotten smarter or our nginx configurations have gotten simpler, so +we have removed the manual instructions we used to have here. + +You may wish to include something like this from each `server { }` +block in your configuration to enable TLS in a common, secure way: + +``` +# Tell nginx to accept TLS-encrypted HTTPS on the standard TCP port. +listen 443 ssl; +listen [::]:443 ssl; + +# Reference the TLS cert files produced by Certbot. +ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; +ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; + +# Load the Let's Encrypt Diffie-Hellman parameters generated for +# this server. Without this, the server is vulnerable to Logjam. +ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; + +# Tighten things down further, per Qualys’ and Certbot’s advice. +ssl_session_cache shared:le_nginx_SSL:1m; +ssl_protocols TLSv1.2 TLSv1.3; +ssl_prefer_server_ciphers on; +ssl_session_timeout 1440m; + +# Offer OCSP certificate stapling. +ssl_stapling on; +ssl_stapling_verify on; + +# Enable HSTS. +include local/enable-hsts; +``` + +The [HSTS] step is optional and should be applied only after due +consideration, since it has the potential to lock users out of your +site if you later change your mind on the TLS configuration. +The `local/enable-hsts` file it references is simply: + + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + +It’s a separate file because nginx requires that headers like this be +applied separately for each `location { }` block. We’ve therefore +factored this out so you can `include` it everywhere you need it. + +The [OCSP] step is optional, but recommended. + +You may find [Qualys’ SSL Server Test][QSLT] helpful in verifying that +you have set all this up correctly, and that the configuration is +strong. We’ve found their [best practices doc][QSLC] to be helpful. As +of this writing, the above configuration yields an A+ rating when run on +Ubuntu 22.04.01 LTS. + +[CB]: https://certbot.eff.org/ +[CBU]: https://certbot.eff.org/instructions?ws=nginx&os=ubuntufocal +[LE]: https://letsencrypt.org/ +[HSTS]: https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/ +[OCSP]: https://en.wikipedia.org/wiki/OCSP_stapling +[QSLC]: https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices +[QSLT]: https://www.ssllabs.com/ssltest/ + +<div style="height:50em" id="this-space-intentionally-left-blank"></div> + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/debian/service.md b/www/server/debian/service.md new file mode 100644 index 0000000..f056ff1 --- /dev/null +++ b/www/server/debian/service.md @@ -0,0 +1,259 @@ +# Serving via systemd on Debian and Ubuntu + +[`systemd`][sdhome] is the service management framework in all major +in-support versions of Linux. There are multiple ways to run Fossil +under `systemd`. + +[sdhome]: https://www.freedesktop.org/wiki/Software/systemd/ +[wpa]: https://en.wikipedia.org/wiki/Systemd#Adoption + + +## Containerized Service + +Two of the methods for running [containerized Fossil][cntdoc] integrate +with `systemd`, potentially obviating the more direct methods below: + +* If you take [the Podman method][podman] of running containerized + Fossil, it opens the `podman generate systemd` option for you, as + exemplified in [the `fslsrv` script][fslsrv] used on this author’s + public Fossil-based web site. That script pulls its container images + from [my Docker Hub repo][dhrepo] to avoid the need for my public + Fossil server to have build tools and a copy of the Fossil source + tree. You’re welcome to use my images as-is, or you may use these + tools to bounce custom builds up through a separate container image + repo you manage. + +* If you’re willing to give up [a lot of features][nsweak] relative to + Podman, and you’re willing to tolerate a lot more manual + administrivia, [the nspawn method][nspawn] has a lot less overhead, + being a direct feature of `systemd` itself. + +Both of these options provide [better security][cntsec] than running +Fossil directly under `systemd`, among [other benefits][cntdoc]. + +[cntdoc]: ../../containers.md +[cntsec]: ../../containers.md#security +[dhrepo]: https://hub.docker.com/r/tangentsoft/fossil +[fslsrv]: https://tangentsoft.com/fossil/dir?name=bin +[nspawn]: ../../containers.md#nspawn +[nsweak]: ../../containers.md#nspawn-weaknesses +[podman]: ../../containers.md#podman + + +## User Service + +A fun thing you can easily do with `systemd` that you can’t directly do +with older technologies like `inetd` and `xinetd` is to set a server up +as a “user” service. + +You can’t listen on TCP port 80 with this method due to security +restrictions on TCP ports in every OS where `systemd` runs, but you can +create a listener socket on a high-numbered (≥ 1024) TCP port, +suitable for sharing a Fossil repo to a workgroup on a private LAN. + +To do this, write the following in +`~/.local/share/systemd/user/fossil.service`: + +> ```dosini +[Unit] +Description=Fossil user server +After=network-online.target + +[Service] +WorkingDirectory=/home/fossil/museum +ExecStart=/home/fossil/bin/fossil server --port 9000 repo.fossil +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +Unlike with `inetd` and `xinetd`, we don’t need to tell `systemd` which +user and group to run this service as, because we’ve installed it +under the account we’re logged into, which `systemd` will use as the +service’s owner. + +The result is essentially [the standalone server method](../any/none.md) +coupled with an intelligent service manager that will start it +automatically in the background on system boot, perform automatic +service restarts with back-off logic, and more, making this much more +robust than the by-hand launches of `fossil` in the platform-independent +Fossil server instructions. The service will stay up until we +explicitly tell it to shut down. + +This scheme couples well with [the generic SCGI instructions][scgi] as +it requires a way to run the underlying repository server in the +background. Given that its service port is then proxied by SCGI, it +follows that it doesn’t need to run as a system service. A user service +works perfectly well for this. + +Because we’ve set this up as a user service, the commands you give to +manipulate the service vary somewhat from the sort you’re more likely to +find online: + + $ systemctl --user daemon-reload + $ systemctl --user enable fossil + $ systemctl --user start fossil + $ systemctl --user status fossil -l + $ systemctl --user stop fossil + +That is, we don’t need to talk to `systemd` with `sudo` privileges, but +we do need to tell it to look at the user configuration rather than the +system-level configuration. + +This scheme isolates the permissions needed by the Fossil server, which +reduces the amount of damage it can do if there is ever a +remotely-triggerable security flaw found in Fossil. + +On some `systemd` based OSes, user services only run while that user is +logged in interactively. This is common on systems aiming to provide +desktop environments, where this is the behavior you often want. To +allow background services to continue to run after logout, say: + + $ sudo loginctl enable-linger $USER + +You can paste the command just like that into your terminal, since +`$USER` will expand to your login name. + +[scgi]: ../any/scgi.md + + + +### System Service Alternative + +There are some common reasons that you’d have good cause to install +Fossil as a system-level service rather than the prior user-level one: + +* You’re using [the new `fossil server --cert` feature][sslsrv] to get + TLS service and want it to listen directly on port 443, rather than be + proxied, as one had to do before Fossil got the ability to act as a + TLS server itself. That requires root privileges, so you can’t run + it as a user-level service. + +* You’re proxying Fossil with [nginx](./nginx.md) or similar, allowing + it to bind to high-numbered ports, but because it starts as a system + service, you can’t get Fossil into the same dependency chain to + ensure things start up and shut down in the proper order unless it + *also* runs as a system service. + +* You want to make use of Fossil’s [chroot jail feature][cjail], which + requires the server to start as root. + +There are just a small set of changes required: + +1. Install the unit file to one of the persistent system-level unit + file directories. Typically, these are: + + /etc/systemd/system + /lib/systemd/system + +2. Add `User` and `Group` directives to the `[Service]` section so + Fossil runs as a normal user, preferably one with access only to + the Fossil repo files, rather than running as `root`. + +[sslsrv]: ../../ssl-server.md +[cjail]: ../../chroot.md + + +## Socket Activation + +Another useful method to serve a Fossil repo via `systemd` is via a +socket listener, which `systemd` calls “[socket activation][sa],” +roughly equivalent to [the ancient `inetd` method](../any/inetd.md). +It’s more complicated, but it has some nice properties. + +We first need to define the privileged socket listener by writing +`/etc/systemd/system/fossil.socket`: + +> ```dosini +[Unit] +Description=Fossil socket + +[Socket] +Accept=yes +ListenStream=80 +NoDelay=true + +[Install] +WantedBy=sockets.target +``` + +Note the change of configuration directory from the `~/.local` directory +to the system level. We need to start this socket listener at the root +level because of the low-numbered TCP port restriction we brought up +above. + +This configuration says more or less the same thing as the socket part +of an `inetd` entry [exemplified elsewhere in this +documentation](../any/inetd.md). + +Next, create the service definition file in that same directory as +`fossil@.service`: + +> ```dosini +[Unit] +Description=Fossil socket server +After=network-online.target + +[Service] +WorkingDirectory=/home/fossil/museum +ExecStart=/home/fossil/bin/fossil http repo.fossil +StandardInput=socket + +[Install] +WantedBy=multi-user.target +``` + +Notice that we haven’t told `systemd` which user and group to run Fossil +under. Since this is a system-level service definition, that means it +will run as root, which then causes Fossil to [automatically drop into a +`chroot(2)` jail](../../chroot.md) rooted at the `WorkingDirectory` +we’ve configured above, shortly after each `fossil http` call starts. + +The `Restart*` directives we had in the user service configuration above +are unnecessary for this method, since Fossil isn’t supposed to remain +running under it. Each HTTP hit starts one Fossil instance, which +handles that single client’s request and then immediately shuts down. + +Next, you need to tell `systemd` to reload its system-level +configuration files and enable the listening socket: + + $ sudo systemctl daemon-reload + $ sudo systemctl enable fossil.socket + +And now you can manipulate the socket listener: + + $ sudo systemctl start fossil.socket + $ sudo systemctl status -l fossil.socket + $ sudo systemctl stop fossil.socket + +Notice that we’re working with the *socket*, not the *service*. The fact +that we’ve given them the same base name and marked the service as an +instantiated service with the “`@`” notation allows `systemd` to +automatically start an instance of the service each time a hit comes in +on the socket that `systemd` is monitoring on Fossil’s behalf. To see +this service instantiation at work, visit a long-running Fossil page +(e.g. `/tarball`) and then give a command like this: + + $ sudo systemctl --full | grep fossil + +This will show information about the `fossil` socket and service +instances, which should show your `/tarball` hit handler, if it’s still +running: + + fossil@20-127.0.0.1:80-127.0.0.1:38304.service + +You can feed that service instance description to a `systemctl kill` +command to stop that single instance without restarting the whole +`fossil` service, for example. + +In all of this, realize that we’re able to manipulate a single socket +listener or single service instance at a time, rather than reload the +whole externally-facing network configuration as with the far more +primitive `inetd` service. + +[sa]: http://0pointer.de/blog/projects/socket-activation.html + + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/index.html b/www/server/index.html new file mode 100644 index 0000000..c15d42f --- /dev/null +++ b/www/server/index.html @@ -0,0 +1,316 @@ +<div class='fossil-doc' data-title="How To Configure A Fossil Server"> + +<style type="text/css"> + .doc > .content th.fep { + font-family: "Helvetica Neue", "Arial Narrow", "Myriad Pro", "Avenir Next Condensed"; + font-stretch: condensed; + min-width: 3em; + padding: 0.4em; + white-space: nowrap; + } + + .doc > .content th.host { + font-family: "Helvetica Neue", "Arial Narrow", "Myriad Pro", "Avenir Next Condensed"; + font-stretch: condensed; + padding: 0.4em; + text-align: right; + } + + .doc > .content td.doc { + text-align: center; + } +</style> + + +<h2>No Server Required</h2> + +<p>Fossil does not require a central server, but <a +href="whyuseaserver.wiki">a server can be useful</a>.</p> + +<p>A Fossil server does not require much memory, CPU, or disk space +and can run comfortably on a generic $5/month virtual host +or on a small device like a Raspberry Pi, or it can co-exist +on a host running other services without getting in the way. + +<p>This article is a quick-reference guide for setting up your own +Fossil server, with links to more detailed instructions specific to +particular systems, should you want extra help.</p> + + +<h2 id="prep">Repository Prep</h2> + +<p>Prior to serving a Fossil repository to others, consider running <a +href="$ROOT/help/ui"><tt>fossil ui</tt></a> locally and taking these +minimum recommended preparation steps:</p> + +<ol> + <li><p>Fossil creates only one user in a <a + href="$ROOT/help/new">new repository</a> and gives it the <a + href="../caps/admin-v-setup.md#apsu">all-powerful Setup capability</a>. + The 10-digit random password generated for that user is fairly strong + against remote attack, even without explicit password guess rate + limiting, but because that user has so much power, you may want to + give it a much stronger password under Admin → Users.</a></li> + + <li><p>Run the Admin → Security-Audit tool to verify that other + security-related permissions and settings are as you want them. + Consider clicking the “Take it private” link on that page to lock down + the security on that site to a level appropriate to a private + repository, even if you will eventually want some public service. It's + better to start from a secure position and open up service + feature-by-feature as necessary than it is to start from a fully open + position and lock down features one by one to achieve a secure + stance.</p></li> +</ol> + +<p>With the repository secured, it is safe to upload a copy of the +repository file to your server and proceed with server setup, below. +Further configuration steps can wait until <a href="#postsetup">after +the server is running</a>.</p> + + +<h2 id="methods">Activation Methods</h2> + +<p>There are basically five ways to run a Fossil server:</p> + +<ol> + <li><a href="any/cgi.md">CGI</a> + <li><a href="#slist">Socket listener</a> + <li><a href="any/none.md">Stand-alone HTTP server</a> + <li><a href="any/scgi.md">SCGI</a> + <li><a href="#ssh">SSH</a> +</ol> + +<p>All of these methods can serve either a single repository or a +directory hierarchy containing multiple repositories.</p> + +<p>You are not restricted to a single server setup. The same Fossil +repository can be served using two or more of the above techniques at +the same time. These methods use clean, well-defined, standard +interfaces (CGI, SCGI, and HTTP) which allow you to easily migrate from +one method to another in response to changes in hosting providers or +administrator preferences.</p> + +<h3 id="cgi">CGI</h3> + +<p>Most ordinary web servers can <a href="any/cgi.md">run Fossil as a +CGI script</a>. This method is known to work with Apache, +<tt>lighttpd</tt>, and <a +href="any/althttpd.md"><tt>althttpd</tt></a>. The Fossil server +administrator places a <a href="$ROOT/help/cgi">short CGI script</a> in +the web server's document hierarchy and when a client requests the URL +that corresponds to that script, Fossil runs and generates the +response.</p> + +<p>CGI is a good choice for merging Fossil into an existing web site, +particularly on hosts that have CGI set up and working. +The Fossil <a href="../selfhost.wiki">self-hosting repositories</a> are +implemented with CGI underneath <tt>althttpd</tt>.</p> + +<h3 id="slist">Socket Listener</h3> + +<p>Socket listener daemons such as +<a id="inetd" href="any/inetd.md"><tt>inetd</tt></a>, <a id="xinetd" +href="any/xinetd.md"><tt>xinetd</tt></a>, <a id="stunnel" +href="any/stunnel.md"><tt>stunnel</tt></a>, <a +href="macos/service.md"><tt>launchd</tt></a>, and <a +href="debian/service.md"><tt>systemd</tt></a> +can be configured to invoke the the +<a href="$ROOT/help/http"><tt>fossil http</tt></a> command to handle +each incoming HTTP request. The "<tt>fossil http</tt>" command reads +the HTTP request off of standard input, computes an appropriate +reply, and writes the reply on standard output. There is a separate +invocation of the "<tt>fossil http</tt>" command for each HTTP request. +The socket listener daemon takes care of relaying content to and from +the client, and (in the case of <a href="any/stunnel.md">stunnel</a>) +handling TLS decryption and encryption. + +<h3 id="standalone">Stand-alone HTTP Server</h3> + +<p>This is the <a href="any/none.md">easiest method</a>. +A stand-alone server uses the +<a href="$ROOT/help/server"><tt>fossil server</tt></a> command to run a +process that listens for incoming HTTP requests on a socket and then +dispatches a copy of itself to deal with each incoming request. You can +expose Fossil directly to the clients in this way or you can interpose a +<a href="https://en.wikipedia.org/wiki/Reverse_proxy">reverse proxy</a> +layer between the clients and Fossil.</p> + +<h3 id="scgi">SCGI</h3> + +<p>The Fossil standalone server can also handle <a href="any/scgi.md">SCGI</a>. +When the <a href="$ROOT/help/server"><tt>fossil server</tt></a> command is +run with the extra <tt>--scgi</tt> option, it listens for incoming +SCGI requests rather than HTTP requests. This allows Fossil to +respond to requests from web servers <a href="debian/nginx.md">such as +nginx</a> that don't support CGI. SCGI is a simpler protocol to proxy +than HTTP, since the HTTP doesn't have to be re-interpreted in terms of +the proxy's existing HTTP implementation, but it's more complex to set +up because you also have to set up an SCGI-to-HTTP proxy for it. It is +worth taking on this difficulty only when you need to integrate Fossil +into an existing web site already being served by an SCGI-capable web +server.</p> + +<h3 id="ssh">SSH</h3> + +<p>Fossil supports <tt>ssh://</tt> URLs, but there are <a +href="../caps/#webonly">practical limitations</a> with the default +behavior. You can get the full power of <a href="../caps/">Fossil's RBAC +system</a> over SSH <a href="any/http-over-ssh.md">with a bit of clever +configuration</a>.</p> + +<h2 id="matrix">Activation Tutorials</h2> + +<p>We've broken the configuration for each method out into a series of +sub-articles. Some of these are generic, while others depend on +particular operating systems or front-end software:</p> + +<div class="indent"><table> + <tr> + <th class="host">⇩ OS / Method ⇨</th> + <th class="fep">direct</th> + <th class="fep">inetd</th> + <th class="fep">stunnel</th> + <th class="fep">CGI</th> + <th class="fep">SCGI</th> + <th class="fep">FastCGI</th> + <th class="fep">althttpd</th> + <th class="fep">proxy</th> + <th class="fep">service</th> + </tr> + + <tr> + <th class="host"><a href="any/">Any</a></th> + <td class="doc"><a href="any/none.md">✅</a></td> + <td class="doc"><a href="any/inetd.md">✅</a></td> + <td class="doc"><a href="any/stunnel.md">✅</a></td> + <td class="doc"><a href="any/cgi.md">✅</a></td> + <td class="doc"><a href="any/scgi.md">✅</a></td> + <td class="doc">❌</td> + <td class="doc"><a href="any/althttpd.md">✅</a></td> + <td class="doc">❌</td> + <td class="doc">❌</td> + </tr> + + <tr> + <th class="host"><a href="debian/">Debian/Ubuntu</a></th> + <td class="doc"><a href="any/none.md">✅</a></td> + <td class="doc"><a href="any/inetd.md">✅</a></td> + <td class="doc"><a href="any/stunnel.md">✅</a></td> + <td class="doc"><a href="any/cgi.md">✅</a></td> + <td class="doc"><a href="any/scgi.md">✅</a></td> + <td class="doc">❌</td> + <td class="doc"><a href="any/althttpd.md">✅</a></td> + <td class="doc"><a href="debian/nginx.md">✅</a></td> + <td class="doc"><a href="debian/service.md">✅</a></td> + </tr> + + <tr> + <th class="host"><a href="macos/">macOS</a></th> + <td class="doc"><a href="any/none.md">✅</a></td> + <td class="doc">❌</td> + <td class="doc"><a href="any/stunnel.md">✅</a></td> + <td class="doc"><a href="any/cgi.md">✅</a></td> + <td class="doc"><a href="any/scgi.md">✅</a></td> + <td class="doc">❌</td> + <td class="doc"><a href="any/althttpd.md">✅</a></td> + <td class="doc">❌</td> + <td class="doc"><a href="macos/service.md">✅</a></td> + </tr> + + <tr> + <th class="host"><a href="openbsd/">OpenBSD</a></th> + <td class="doc"><a href="any/none.md">✅</a></td> + <td class="doc">❌</td> + <td class="doc"><a href="any/stunnel.md">✅</a></td> + <td class="doc"><a href="any/cgi.md">✅</a></td> + <td class="doc"><a href="any/scgi.md">✅</a></td> + <td class="doc"><a href="openbsd/fastcgi.md">✅</a></td> + <td class="doc"><a href="any/althttpd.md">✅</a></td> + <td class="doc">❌</td> + <td class="doc"><a href="openbsd/service.wiki">✅</a></td> + </tr> + + <tr> + <th class="host"><a href="windows/">Windows</a></th> + <td class="doc"><a href="windows/none.md">✅</a></td> + <td class="doc">❌</td> + <td class="doc"><a href="windows/stunnel.md">✅</a></td> + <td class="doc"><a href="windows/cgi.md">✅</a></td> + <td class="doc">❌</td> + <td class="doc">❌</td> + <td class="doc">❌</td> + <td class="doc"><a href="windows/iis.md">✅</a></td> + <td class="doc"><a href="windows/service.md">✅</a></td> + </tr> +</table></div> + +<p>Where there is a check mark in the "<b>Any</b>" row, the method for that is +generic enough that it works across OSes that Fossil is known to work +on. The check marks below that usually just link to this generic +documentation.</p> + +<p>The method in the "<b>proxy</b>" column is for the platform's default +web server configured as a <a +href="https://en.wikipedia.org/wiki/Reverse_proxy">reverse proxy</a> for +Fossil's built-in HTTP server: <a href="debian/nginx.md">nginx</a>, <a +href="windows/iis.md">IIS</a>, Apache, etc.</p> + +<p>We welcome <a href="../contribute.wiki">contributions</a> to fill gaps +(<font size="-2">❌</font>) in the table above.</p> +</noscript> + + +<h2 id="postsetup">Post-Activation Configuration</h2> + +<p>After the server is up and running, log into it as the Setup user and +visit the Admin menu to finish configuring that repository for +service:</p> + +<ol> + <li><p>Add user accounts for your other team members. Use <a + href="../caps/index.md#ucat">categories</a> to define access policies + rather than redundantly give each new user the same <a + href="../caps/index.md#ucap">individual capabilities</a>.</p></li> + + <li><p>Test access to the repository from each category of non-Setup + user that you created. You may have to give your user categories some + overlooked capabilities, particularly if you followed <a + href="#prep">our earlier advice</a> to take the repository private + prior to setting up the server.</p></li> + + <li><p>Modify the repository's look and feel by <a + href="../customskin.md">customizing the skin</a>.</p></li> + + <li><p>If the repository includes <a + href="../embeddeddoc.wiki">embedded documentation</a>, consider + activating the search feature (Admin → Search) so that visitors can do + full-text search on your documentation.</p></li> + + <li><p>Now that others can be making changes to the repository, + consider monitoring them via <a href="../alerts.md">email alerts</a> + or the <a href="$ROOT/help/www/timeline.rss">timeline RSS + feed</a>.</p></li> + + <li><p>Turn on the various logging features.</p></li> +</ol> + +<p>Reload the Admin → Security-Audit page occasionally during this +process to double check that you have not mistakenly configured the +server in a way that might expose information that you want to keep +private.</p> + + +<h2 id="more">Further Details</h2> + +<ul> + <li><a id="chroot" href="../chroot.md" >The Server Chroot Jail</a> + <li><a id="loadmgmt" href="../loadmgmt.md" >Managing Server Load</a> + <li><a id="bkofc" href="../backoffice.md" >The Backoffice</a> + <li><a id="tls" href="../ssl.wiki" >Securing a Repository with TLS</a> + <li><a id="ext" href="../serverext.wiki">CGI Server Extensions</a> + <li><a id="about" href="../aboutcgi.wiki" >How CGI Works In Fossil</a> + <li><a id="sync" href="../sync.wiki" >The Fossil Sync Protocol</a> +</ul> + +</div> diff --git a/www/server/macos/index.md b/www/server/macos/index.md new file mode 100644 index 0000000..52c9ef9 --- /dev/null +++ b/www/server/macos/index.md @@ -0,0 +1,5 @@ +# macOS Specific Fossil Service Options + +- [Running Fossil under `launchd`](./service.md) + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/macos/service.md b/www/server/macos/service.md new file mode 100644 index 0000000..b1358fa --- /dev/null +++ b/www/server/macos/service.md @@ -0,0 +1,181 @@ +# Serving via launchd on macOS + +[`launchd`][ldhome] is the default service management framework on macOS +[since the release of Tiger in 2005][wpa]. If you want a Fossil server +to launch in the background on a Mac, it’s the way Apple wants you to do +it. `launchd` is to macOS as `systemd` is to most modern Linux desktop +systems. (Indeed, `systemd` arguably reinvented the perfectly good, +pre-existing `launchd` wheel.) + +Unlike in [our `systemd` article](../debian/service.md), we’re not going +to show the per-user method here, because those so-called +[LaunchAgents][la] only start when a user is logged into the GUI, and +they stop when that user logs out. This does not strike us as proper +“server” behavior, so we’ll stick to system-level LaunchDaemons instead. + +However, we will still give two different configurations, just as in the +`systemd` article: one for a standalone HTTP server, and one using +socket activation. + +For more information on `launchd`, the single best resource we’ve found +is [launchd.info](https://launchd.info). The next best is: + + $ man launchd.plist + +[la]: http://www.grivet-tools.com/blog/2014/launchdaemons-vs-launchagents/ +[ldhome]: https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/CreatingLaunchdJobs.html +[wpa]: https://en.wikipedia.org/wiki/Launchd + + + +## Standalone HTTP Server + +To configure `launchd` to start Fossil as a standalone HTTP server, +write the following as `com.example.dev.FossilHTTP.plist`: + +```xml +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" + "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> +<plist version="1.0"> +<dict> + <key>Label</key> + <string>com.example.dev.FossilHTTP</string> + <key>ProgramArguments</key> + <array> + <string>/usr/local/bin/fossil</string> + <string>server</string> + <string>--port</string> + <string>9000</string> + <string>repo.fossil</string> + </array> + <key>WorkingDirectory</key> + <string>/Users/you/museum</string> + <key>KeepAlive</key> + <true/> + <key>RunAtLoad</key> + <true/> + <key>StandardErrorPath</key> + <string>/tmp/fossil-error.log</string> + <key>StandardOutPath</key> + <string>/tmp/fossil-info.log</string> + <key>UserName</key> + <string>you</string> + <key>GroupName</key> + <string>staff</string> + <key>InitGroups</key> + <true/> +</dict> +</plist> +``` + +In this example, we’re assuming your development organization uses the +domain name “`dev.example.org`”, that your short macOS login name is +“`you`”, and that you store your Fossils in “`~/museum`”. Adjust these +elements of the plist file to suit your local situation. + +You might be wondering about the use of `UserName`: isn’t Fossil +supposed to drop privileges and enter [a `chroot(2)` +jail](../../chroot.md) when it’s started as root like this? Why do we +need to give it a user name? Won’t Fossil use the owner of the +repository file to set that? All I can tell you is that in testing here, +if you leave the user and group configuration at the tail end of that +plist file out, Fossil will remain running as root! + +Install that file and set it to start with: + + $ sudo install -o root -g wheel -m 644 com.example.dev.FossilHTTP.plist \ + /Library/LaunchDaemons/ + $ sudo launchctl load -w /Library/LaunchDaemons/com.example.dev.FossilHTTP.plist + +Because we set the `RunAtLoad` key, this will also launch the daemon. + +Stop the daemon with: + + $ sudo launchctl unload -w /Library/LaunchDaemons/com.example.dev.FossilHTTP.plist + + +## Socket Listener + +Another useful method to serve a Fossil repo via `launchd` is by setting +up a socket listener: + +```xml +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" + "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> +<plist version="1.0"> +<dict> + <key>Label</key> + <string>com.example.dev.FossilSocket</string> + <key>ProgramArguments</key> + <array> + <string>/usr/local/bin/fossil</string> + <string>http</string> + <string>repo.fossil</string> + </array> + <key>Sockets</key> + <dict> + <key>Listeners</key> + <dict> + <key>SockServiceName</key> + <string>9001</string> + <key>SockType</key> + <string>stream</string> + <key>SockProtocol</key> + <string>TCP</string> + <key>SockFamily</key> + <string>IPv4</string> + </dict> + </dict> + <key>inetdCompatibility</key> + <dict> + <key>Wait</key> + <false/> + </dict> + <key>WorkingDirectory</key> + <string>/Users/you/museum</string> + <key>UserName</key> + <string>you</string> + <key>GroupName</key> + <string>staff</string> + <key>InitGroups</key> + <true/> +</dict> +</plist> +``` + +Save it as “`com.example.dev.FossilSocket.plist`” and install and load +it into `launchd` as above. + +This version differs in several key ways: + +1. We’re calling Fossil as `fossil http` rather than `fossil server` to + make it serve a single request and then shut down immediately. + +2. We’ve told `launchd` to listen on our TCP port number instead of + passing it to `fossil`. + +3. We’re running the daemon in `inetd` compatibility mode of `launchd` + with “wait” mode off, which tells it to attach the connected socket + to the `fossil` process’s stdio handles. + +4. We’ve removed the `Standard*Path` keys because they interfere with + our use of stdio handles for HTTP I/O. You might therefore want to + start with the first method and then switch over to this one only + once you’ve got the daemon launching debugged, since once you tie up + stdio this way, you won’t be able to get logging information from + Fossil via that path. (Fossil does have some internal logging + mechanisms, but you can’t get at them until Fossil is launching!) + +5. We’ve removed the `KeepAlive` and `RunAtLoad` keys because those + options aren’t appropriate to this type of service. + +6. Because we’re running it via a socket listener instead of as a + standalone HTTP server, the Fossil service only takes system + resources when it’s actually handling an HTTP hit. If your Fossil + server is mostly idle, this method will be a bit more efficient than + the first option. + + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/openbsd/fastcgi.md b/www/server/openbsd/fastcgi.md new file mode 100644 index 0000000..70db94b --- /dev/null +++ b/www/server/openbsd/fastcgi.md @@ -0,0 +1,302 @@ +# Serving via httpd on OpenBSD + +[`httpd`][httpd] is the default web server that is included in the base +install on OpenBSD. It's minimal and lightweight but secure and capable, +and provides a clean interface for setting up a Fossil server using +FastCGI. + +This article will detail the steps required to setup a TLS-enabled +`httpd` configuration that serves multiple Fossil repositories out of +a single directory within a chroot, and allow `ssh` access to create +new repositories remotely. + +**NOTE:** The following instructions assume an OpenBSD 6.7 installation. + +[httpd]: https://www.openbsd.org/papers/httpd-asiabsdcon2015.pdf + +## <a id="fslinstall"></a>Install Fossil + +Use the OpenBSD package manager `pkg_add` to install Fossil, making sure +to select the statically linked binary. + +```console +$ doas pkg_add fossil +quirks-3.325 signed on 2020-06-12T06:24:53Z +Ambiguous: choose package for fossil + 0: <None> + 1: fossil-2.10v0 + 2: fossil-2.10v0-static +Your choice: 2 +fossil-2.10v0-static: ok +``` + +This installs Fossil into the chroot. To facilitate local use, create a +symbolic link of the fossil executable into `/usr/local/bin`. + +```console +$ doas ln -s /var/www/bin/fossil /usr/local/bin/fossil +``` + +As a privileged user, create the file `/var/www/cgi-bin/scm` with the +following contents to make the CGI script that `httpd` will execute in +response to `fsl.domain.tld` requests; all paths are relative to the +`/var/www` chroot. + +```sh +#!/bin/fossil +directory: /htdocs/fsl.domain.tld +notfound: https://domain.tld +repolist +errorlog: /logs/fossil.log +``` + +The `directory` directive instructs Fossil to serve all repositories +found in `/var/www/htdocs/fsl.domain.tld`, while `errorlog` sets logging +to be saved to `/var/www/logs/fossil.log`; create the repository +directory and log file—making the latter owned by the `www` user, and +the script executable. + +```console +$ doas mkdir /var/www/htdocs/fsl.domain.tld +$ doas touch /var/www/logs/fossil.log +$ doas chown www /var/www/logs/fossil.log +$ doas chmod 660 /var/www/logs/fossil.log +$ doas chmod 755 /var/www/cgi-bin/scm +``` + +## <a id="chroot"></a>Setup chroot + +Fossil needs both `/dev/random` and `/dev/null`, which aren't accessible +from within the chroot, so need to be constructed; `/var`, however, is +mounted with the `nodev` option. Rather than removing this default +setting, create a small memory filesystem and then mount it on to +`/var/www/dev` with [`mount_mfs(8)`][mfs] so that the `random` and +`null` device files can be created. In order to avoid necessitating a +startup script to recreate the device files at boot, create a template +of the needed ``/dev`` tree to automatically populate the memory +filesystem. + +```console +$ doas mkdir /var/www/dev +$ doas install -d -g daemon /template/dev +$ cd /template/dev +$ doas /dev/MAKEDEV urandom +$ doas mknod -m 666 null c 2 2 +$ doas mount_mfs -s 1M -P /template/dev /dev/sd0b /var/www/dev +$ ls -l +total 0 +crw-rw-rw- 1 root daemon 2, 2 Jun 20 08:56 null +lrwxr-xr-x 1 root daemon 7 Jun 18 06:30 random@ -> urandom +crw-r--r-- 1 root wheel 45, 0 Jun 18 06:30 urandom +``` + +[mfs]: https://man.openbsd.org/mount_mfs.8 + +To make the mountable memory filesystem permanent, open `/etc/fstab` as +a privileged user and add the following line to automate creation of the +filesystem at startup: + +```console +swap /var/www/dev mfs rw,-s=1048576,-P=/template/dev 0 0 +``` + +The same user that executes the fossil binary must have writable access +to the repository directory that resides within the chroot; on OpenBSD +this is `www`. In addition, grant repository directory ownership to the +user who will push to, pull from, and create repositories. + +```console +$ doas chown -R user:www /var/www/htdocs/fsl.domain.tld +$ doas chmod 770 /var/www/htdocs/fsl.domain.tld +``` + +## <a id="httpdconfig"></a>Configure httpd + +On OpenBSD, [httpd.conf(5)][httpd] is the configuration file for +`httpd`. To setup the server to serve all Fossil repositores within the +directory specified in the CGI script, and automatically redirect +standard HTTP requests to HTTPS—apart from [Let's Encrypt][LE] +challenges issued in response to [acme-client(1)][acme] certificate +requests—create `/etc/httpd.conf` as a privileged user with the +following contents. + +[LE]: https://letsencrypt.org +[acme]: https://man.openbsd.org/acme-client.1 +[httpd.conf(5)]: https://man.openbsd.org/httpd.conf.5 + +```apache +server "fsl.domain.tld" { + listen on * port http + root "/htdocs/fsl.domain.tld" + location "/.well-known/acme-challenge/*" { + root "/acme" + request strip 2 + } + location * { + block return 301 "https://$HTTP_HOST$REQUEST_URI" + } + location "/*" { + fastcgi { param SCRIPT_FILENAME "/cgi-bin/scm" } + } +} + +server "fsl.domain.tld" { + listen on * tls port https + root "/htdocs/fsl.domain.tld" + tls { + certificate "/etc/ssl/domain.tld.fullchain.pem" + key "/etc/ssl/private/domain.tld.key" + } + hsts { + max-age 15768000 + preload + subdomains + } + connection max request body 104857600 + location "/*" { + fastcgi { param SCRIPT_FILENAME "/cgi-bin/scm" } + } + location "/.well-known/acme-challenge/*" { + root "/acme" + request strip 2 + } +} +``` + +[The default limit][dlim] for HTTP messages in OpenBSD’s `httpd` server +is 1 MiB. Fossil chunks its sync protocol such that this is not +normally a problem, but when sending [unversioned content][uv], it uses +a single message for the entire file. Therefore, if you will be storing +files larger than this limit as unversioned content, you need to raise +the limit as we’ve done above with the “`connection max request body`” +setting, raising the limit to 100 MiB. + +[dlim]: https://man.openbsd.org/httpd.conf.5#connection +[uv]: ../../unvers.wiki + +**NOTE:** If not already in possession of an HTTPS certificate, comment +out the `https` server block and proceed to securing a free +[Let's Encrypt Certificate](#letsencrypt); otherwise skip to +[Start `httpd`](#starthttpd). + + +## <a id="letsencrypt"></a>Let's Encrypt Certificate + +In order for `httpd` to serve HTTPS, secure a free certificate from +Let's Encrypt using `acme-client`. Before issuing the request, however, +ensure you have a zone record for the subdomain with your registrar or +nameserver. Then open `/etc/acme-client.conf` as a privileged user to +configure the request. + +```dosini +authority letsencrypt { + api url "https://acme-v02.api.letsencrypt.org/directory" + account key "/etc/acme/letsencrypt-privkey.pem" +} + +authority letsencrypt-staging { + api url "https://acme-staging.api.letsencrypt.org/directory" + account key "/etc/acme/letsencrypt-staging-privkey.pem" +} + +domain domain.tld { + alternative names { www.domain.tld fsl.domain.tld } + domain key "/etc/ssl/private/domain.tld.key" + domain certificate "/etc/ssl/domain.tld.crt" + domain full chain certificate "/etc/ssl/domain.tld.fullchain.pem" + sign with letsencrypt +} +``` + +Start `httpd` with the new configuration file, and issue the certificate +request. + +```console +$ doas rcctl start httpd +$ doas acme-client -vv domain.tld +acme-client: /etc/acme/letsencrypt-privkey.pem: account key exists (not creating) +acme-client: /etc/acme/letsencrypt-privkey.pem: loaded RSA account key +acme-client: /etc/ssl/private/domain.tld.key: generated RSA domain key +acme-client: https://acme-v01.api.letsencrypt.org/directory: directories +acme-client: acme-v01.api.letsencrypt.org: DNS: 172.65.32.248 +... +N(Q????Z???j?j?>W#????b???? H????eb??T??*? DNosz(???n{L}???D???4[?B] (1174 bytes) +acme-client: /etc/ssl/domain.tld.crt: created +acme-client: /etc/ssl/domain.tld.fullchain.pem: created +``` + +A successful result will output the public certificate, full chain of +trust, and private key into the `/etc/ssl` directory as specified in +`acme-client.conf`. + +```console +$ doas ls -lR /etc/ssl +-r--r--r-- 1 root wheel 2.3K Mar 2 01:31:03 2018 domain.tld.crt +-r--r--r-- 1 root wheel 3.9K Mar 2 01:31:03 2018 domain.tld.fullchain.pem + +/etc/ssl/private: +-r-------- 1 root wheel 3.2K Mar 2 01:31:03 2018 domain.tld.key +``` + +Make sure to reopen `/etc/httpd.conf` to uncomment the second server +block responsible for serving HTTPS requests before proceeding. + +## <a id="starthttpd"></a>Start `httpd` + +With `httpd` configured to serve Fossil repositories out of +`/var/www/htdocs/fsl.domain.tld`, and the certificates and key in place, +enable and start `slowcgi`—OpenBSD's FastCGI wrapper server that will +execute the above Fossil CGI script—before checking that the syntax of +the `httpd.conf` configuration file is correct, and (re)starting the +server (if still running from requesting a Let's Encrypt certificate). + +```console +$ doas rcctl enable slowcgi +$ doas rcctl start slowcgi +slowcgi(ok) +$ doas httpd -vnf /etc/httpd.conf +configuration OK +$ doas rcctl start httpd +httpd(ok) +``` + +## <a id="clientconfig"></a>Configure Client + +To facilitate creating new repositories and pushing them to the server, +add the following function to your `~/.cshrc` or `~/.zprofile` or the +config file for whichever shell you are using on your development box. + +```sh +finit() { + fossil init $1.fossil && \ + chmod 664 $1.fossil && \ + fossil open $1.fossil && \ + fossil user password $USER $PASSWD && \ + fossil remote-url https://$USER:$PASSWD@fsl.domain.tld/$1 && \ + rsync --perms $1.fossil $USER@fsl.domain.tld:/var/www/htdocs/fsl.domain.tld/ >/dev/null && \ + chmod 644 $1.fossil && \ + fossil ui +} +``` + +This enables a new repository to be made with `finit repo`, which will +create the fossil repository file `repo.fossil` in the current working +directory; by default, the repository user is set to the environment +variable `$USER`. It then opens the repository and sets the user +password to the `$PASSWD` environment variable (which you can either set +with `export PASSWD 'password'` on the command line or add to a +*secured* shell environment file), and the `remote-url` to +`https://fsl.domain.tld/repo` with the credentials of `$USER` who is +authenticated with `$PASSWD`. Finally, it `rsync`'s the file to the +server before opening the local repository in your browser where you can +adjust settings such as anonymous user access, and set pertinent +repository details. Thereafter, you can add files with `fossil add`, and +commit with `fossil ci -m 'commit message'` where Fossil, by default, +will push to the `remote-url`. It's suggested you read the +[Fossil documentation][documentation]; with a sane and consistent +development model, the system is much more efficient and cohesive than +`git`—so the learning curve is not steep at all. + +[documentation]: https://fossil-scm.org/home/doc/trunk/www/permutedindex.html + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/openbsd/index.md b/www/server/openbsd/index.md new file mode 100644 index 0000000..ea4f95f --- /dev/null +++ b/www/server/openbsd/index.md @@ -0,0 +1,6 @@ +# Debian/Ubuntu Specific Fossil Service Options + +- [Serving Fossil under OpenBSD’s `httpd` via FastCGI](./fastcgi.md) +- [Running Fossil as a service on OpenBSD](./service.wiki) + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/openbsd/service.wiki b/www/server/openbsd/service.wiki new file mode 100644 index 0000000..8729eb0 --- /dev/null +++ b/www/server/openbsd/service.wiki @@ -0,0 +1,118 @@ +<title>Serving via rc on OpenBSD</title> + +OpenBSD provides [https://man.openbsd.org/rc.subr.8|rc.subr(8)], +a framework for writing [https://man.openbsd.org/rc.8|rc(8)] scripts. + +<h2>Creating the daemon</h2> + +Create the file /etc/rc.d/fossil with contents like the following. + +<pre> +#!/bin/ksh +daemon="/usr/local/bin/fossil" # fossil executable +daemon_user="_fossil" # user to run fossil as +daemon_flags="server /home/_fossil/example --repolist --port 8888" # fossil command + +. /etc/rc.d/rc.subr +# pexp="$daemon server .*" # See below. +rc_reload=NO # Unsupported by Fossil; 'rcctl reload fossil' kills the process. +rc_bg=YES # Run in the background, since fossil serve does not daemonize itself +rc_cmd $1 +</pre> + +<h3>pexp</h3> + +You may need to uncomment the "pexp=". rc.subr typically +finds the daemon process based by matching the process name and argument list. +Without the "pexp=" line, rc.subr would look for this exact command: + +<pre> +/usr/local/bin/fossil server /home/_fossil/example --repolist --port 8888 +</pre> + +Depending on the arguments and their order, fossil may rewrite the arguments +for display in the process listing ([https://man.openbsd.org/ps.1|ps(1)]), +so rc.subr may fail to find the process through the default match. The example +above does not get rewritten, but the same commands in a different order can +be rewritten. +For example, when I switch the order of the arguments in "daemon_flags", + +<pre> +/usr/local/bin/fossil server --repolist --port 8888 /home/_fossil/example +</pre> + +the process command is changed to this. + +<pre> +/usr/local/bin/fossil server /home/_fossil/example /home/_fossil/example 8888 /home/_fossil/example +</pre> + +The commented "pexp=" line instructs rc.subr to choose the process whose +command and arguments text starts with this: + +<pre> +/usr/local/bin/fossil server +</pre> + +<h2>Enabling the daemon</h2> + +Once you have created /etc/rc.d/fossil, run these commands. + +<pre> +rcctl enable fossil # add fossil to pkg_scripts in /etc/rc.conf.local +rcctl start fossil # start the daemon now +</pre> + +The daemon should now be running and set to start at boot. + +<h2>Multiple daemons</h2> + +You may want to serve multiple fossil instances with different options. +For example, + + * If different users own different repositories, you may want different users + to serve different repositories. + * You may want to serve different repositories on different ports so you can + control them differently with, for example, HTTP reverse proxies or + [https://man.openbsd.org/pf.4|pf(4)]. + +To run multiple fossil daemons, create multiple files in /etc/rc.d, and +enable each of them. Here are two approaches for creating +the files in /etc/rc.d: Symbolic links and copies. + +<h3>Symbolic links</h3> + +Suppose you want to run one fossil daemon as user "user1" on port 8881 +and another as user "user2" on port 8882. Create the files with +[https://man.openbsd.org/ln.1|ln(1)], and configure them to run different +fossil commands. + +<pre> +cd /etc/rc.d +ln -s fossil fossil1 +ln -s fossil fossil2 +rcctl enable fossil1 fossil2 +rcctl set fossil1 user user1 +rcctl set fossil2 user user2 +rcctl set fossil1 flags 'server /home/user1/repo1.fossil --port 8881' +rcctl set fossil2 flags 'server /home/user2/repo2.fossil --port 8882' +rcctl start fossil1 fossil2 +</pre> + +<h3>Copies</h3> + +You may want to run fossil daemons that are too different to configure +just with [https://man.openbsd.org/rcctl.8|rcctl(8)]. +In particular, you can't change the "pexp" with rcctl. + +If you want to run fossil commands that are more different, +you may prefer to create separate files in /etc/rc.d. +Replace "ln -s" above with "cp" to accomplish this. + +<pre> +cp /etc/rc.d/fossil /etc/rc.d/fossil-user1 +cp /etc/rc.d/fossil /etc/rc.d/fossil-user2 +</pre> + +You can still use commands like "rcctl set fossil-user1 flags", but you +can also edit the "/etc/rc.d/fossil-user1" file. diff --git a/www/server/whyuseaserver.wiki b/www/server/whyuseaserver.wiki new file mode 100644 index 0000000..cdd9823 --- /dev/null +++ b/www/server/whyuseaserver.wiki @@ -0,0 +1,95 @@ +<title>Benefits of a Fossil Server</title> + +<h2>No Server Required</h2> + +Fossil does not require a central server. +Data sharing and synchronization can be entirely peer-to-peer. +Fossil uses +[https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type | +conflict-free replicated data types] +to ensure that (in the limit) all participating peers see the same content. + +<h2>But, a Server Can Be Useful</h2> + +Fossil does not require a server, but a server can be very useful. +Here are a few reasons to set up a Fossil server for your project: + + 1. <b>A server works as a complete project website.</b> + + Fossil does more than just version control. It also supports + [../tickets.wiki|trouble-tickets], + [../wikitheory.wiki|wiki], + a [../chat.md|developer chat room], and a [../forum.wiki|forum]. + The [../embeddeddoc.wiki|embedded documentation] + feature provides a great mechanism for providing project documentation. + The [../unvers.wiki|unversioned files] feature is a convenient way + to host builds and downloads on the project website. + + 2. <b>A server gives developers a common point of rendezvous for + syncing their work.</b> + + It is possible for developers to synchronize peer-to-peer but + that requires the developers coordinate the sync, which in turn + requires that the developers both want to sync at the same moment. + A server alleviates this time dependency by allowing each developer + to sync whenever it is convenient. For example, a developer may + choose to automatically sync + after each commit and before each update. Developers all stay + in sync with each other without having to interrupt each other + constantly to set up a peer-to-peer sync. + + 3. <b>A server provides project leaders with up-to-date status.</b> + + Project coordinators and BDFLs can click on a link or two at the + central Fossil server for a project and quickly tell what is + going on. They can do this from anywhere — even from their phones + — without needing to actually sync to the device they are using. + + 4. <b>A server provides automatic off-site backups.</b> + + A Fossil server is an automatic remote backup for all the work + going into a project. ([../backup.md | Within limits].) + You can even set up multiple servers at + multiple sites with automatic synchronization between them for + added redundancy. Such a setup means that no work is lost due + to a single machine failure. + + 5. <b>A server consolidates [https://www.sqlite.org/howtocorrupt.html + | SQLite corruption risk mitigation] to a single point.</b> + + The concerns in section 1 of that document assume you have direct + access to the central DB files, which isn't the case when the + server is remote and secure against tampering. + + Section 2 is about file locking, which concerns disappear when Fossil's + on the other side of an HTTP boundary and your server is set up + properly. + + Sections 3.1, 4 thru 6, and 8 apply to all Fossil configurations, + but setting up a server lets you address the risks + in a single place. Once a given commit is + sync'd to the server, you can be reasonably sure any client-side + corruption can be fixed with a fresh clone. Ultimately, this + is an argument for off-machine backups, which returns us to reason + #4 above. + + Sections 3.2 and the entirety of section 7 are no concern with + Fossil at all, since it's primarily written by the creator and + primary maintainer of SQLite, so you can be certain Fossil doesn't + actively pursue coding strategies known to risk database corruption. + + For another take on this topic, see the article + "[https://sqlite.org/useovernet.html | SQLite Over a Network, + Caveats and Considerations]". Fossil runs in rollback mode by + default per recommendation #3 at the end of that article, and a + Fossil server operates as a network proxy for the underlying + SQLite repository DB per recommendation #2. This <i>may</i> permit + you to safely switch it into WAL mode (<b>fossil rebuild --wal</b>) + depending on the underlying storage used by the server itself. + + 6. <b>A server allows [../caps/ | Fossil's RBAC system] to work.</b> + + The role-based access control (RBAC) system in Fossil only works + when the remote system is on the other side of an HTTP barrier. + ([../caps/#webonly | Details].) If you want its benefits, you need + a Fossil server setup of some kind. diff --git a/www/server/windows/cgi-bin-perm.png b/www/server/windows/cgi-bin-perm.png Binary files differnew file mode 100644 index 0000000..92ec090 --- /dev/null +++ b/www/server/windows/cgi-bin-perm.png diff --git a/www/server/windows/cgi-exec-perm.png b/www/server/windows/cgi-exec-perm.png Binary files differnew file mode 100644 index 0000000..5cfc2e3 --- /dev/null +++ b/www/server/windows/cgi-exec-perm.png diff --git a/www/server/windows/cgi-install-iis.png b/www/server/windows/cgi-install-iis.png Binary files differnew file mode 100644 index 0000000..ef6eb67 --- /dev/null +++ b/www/server/windows/cgi-install-iis.png diff --git a/www/server/windows/cgi-script-map.png b/www/server/windows/cgi-script-map.png Binary files differnew file mode 100644 index 0000000..bccc2fb --- /dev/null +++ b/www/server/windows/cgi-script-map.png diff --git a/www/server/windows/cgi.md b/www/server/windows/cgi.md new file mode 100644 index 0000000..768c76f --- /dev/null +++ b/www/server/windows/cgi.md @@ -0,0 +1,115 @@ +# Serving via IIS + CGI + +## This Is Not the Method You Are Looking For + +Setting up CGI service under IIS is surprisingly complicated compared to +running Fossil as a CGI under most other operating systems. We recommend +that you use the simpler [reverse proxying method](./iis.md) instead +unless there is some compelling reason why that method cannot work for +you, such as its dependence on non-stock IIS extensions. (Keep in mind +that both extensions it requires are by Microsoft, not third parties!) + +Once you’ve got this scheme working, it gives the same benefits as those +listed at the top of the linked-to document. + +There is a small benefit you get from using CGI over reverse proxying on +other OSes, which is that the Fossil program only runs briefly in order +to serve each HTTP hit. Once the request is done, that Fossil instance +shuts back down, releasing all of its resources. You don’t need to keep +a background Fossil HTTP server running full-time to provide CGI-based +Fossil service. + +You lose a lot of that benefit on Windows: + +1. It only matters to start with on servers that are highly RAM + constrained. (Roughly ≤ 128 MiB.) Our configuration steps below + assume you’re using the Windows and IIS GUIs, which have RAM + requirements well in excess of this, making Fossil’s resource + requirements a drop in the bucket next to them. On the [Azure + B1s][b1s] virtual machine I used to prepare these instructions, the + Windows Server Manager GUI kept filling the VM’s 1 GiB of RAM + during feature installation and crashing. I had to upgrade the VM’s + RAM to 2 GiB just to get useful work done! + +2. Process creation on Windows is [much more expensive][cp] than on the + other OSes Fossil runs on, so the benefits of firing up a Fossil + executable to process each HTTP request are partially swamped by the + overhead of doing so. + +Therefore, unless you’re willing to replace all of the GUI configuration +steps below with command line equivalents, or shut the GUI down entirely +after configuring IIS, CGI is a much less compelling option on Windows. + +**WARNING:** The following tutorial appears to fail with the current +(2019-08-17) version of Fossil, [apparently][fbug] due to an inability +of Fossil to detect that it’s being run in CGI mode. + +[b1s]: https://azure.microsoft.com/en-us/blog/introducing-b-series-our-new-burstable-vm-size/ +[cp]: https://stackoverflow.com/a/48244/142454 +[fbug]: https://fossil-scm.org/forum/forumpost/de18dc32c0 + + +## Install IIS with CGI Support + +The steps for this are identical to those for the [reverse proxying IIS +setup](./iis.md#install) except that you need to enable CGI in the last +step, since it isn’t installed by default. For Windows Server, the path +is: + + + +The path is similar on the consumer-focused versions of Windows, once +you get to that last step. + + +## Setup + +1. Install the Fossil executable to `c:\inetpub\wwwroot\bin` on the web + server. We can’t use an executable you might already have because IIS + runs under a separate user account, so we need to give that + executable special permissions, and that’s easiest to do under the + IIS tree: + +  + +2. In IIS Manager (a.k.a. `INETMGR`) drill down into the Sites folder + in the left-side pane and right-click your web site’s + configuration. (e.g. “Default Web Site”) + +3. On that menu say “Add Virtual Directory.” Give it the alias “`cgi`” + and point it at a suitable directory, such as + “`c:\inetpub\wwwroot\cgi`”. + +4. Double-click the “Handler Mappings” icon, then in the right-side + pane, click “Add Script Map...” Apply the following settings: + +  + + The Executable path must point to the path we set up in step 1, not + to some other `fossil.exe` you may have elsewhere on your system. + You will need to change the default “`*.dll`” filter in the Open + dialog to “`*.exe`” in order to see it when browsing via the “`...`” + button. + +5. Create a file called `repo.fslcgi` within the CGI directory you + chose in step 3, with a single line like this: + + repository: c:\Users\SOMEONE\museum\repo.fossil + + Give the actual path to the repository, of course. + +6. Up at the top level of IIS Manager, double-click the “ISAPI and CGI + Restrictions” icon, then click “Add...” in the right-side pane. + Give the script you just created permission to execute: + +  + +7. In the right-side pane, click “Restart” to apply this configuration, + then test it by visiting the newly-available URL in a browser: + + http://localhost/cgi/repo.fslcgi + +For more complicated setups such as “directory” mode, see [the generic +CGI instructions](../any/cgi.md). + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/windows/iis.md b/www/server/windows/iis.md new file mode 100644 index 0000000..9107da1 --- /dev/null +++ b/www/server/windows/iis.md @@ -0,0 +1,134 @@ +# Serving via IIS + +## Why Bother? + +The first part of the scheme below sets Fossil up as an HTTP server, so +you might be wondering why you wouldn’t just modify that to make it +listen on all network interfaces on TCP port 80, so you can avoid the +need for IIS entirely. For simple use cases, you can indeed do without +IIS, but there are several use cases where adding it is helpful: + +1. Proxying Fossil with IIS lets you [add TLS encryption][tls], which + [Fossil does not currently speak](../../ssl.wiki) in its server role. + +2. The URL rewriting we do below allows Fossil to be part of a larger + site already being served with IIS. + +3. You can have a mixed-mode site, with Fossil acting as a powerful + dynamic content management service and IIS as a fast static content + server. The pure-Fossil alternative requires that you check all of + your static content into Fossil as versioned or unversioned + artifacts. + +This article shows how you can get any combination of those benefits by +using IIS as a reverse proxy for `fossil server`. + +There are other ways to use IIS to serve Fossil, such as [via +CGI](./cgi.md). + + +## Background Fossil Service Setup + +You will need to have the Fossil HTTP server running in the background, +serving some local repository, bound to localhost on a fixed +high-numbered TCP port. For the purposes of testing, simply start it by +hand in your command shell of choice: + + fossil serve --port 9000 --localhost repo.fossil + +That command assumes you’ve got `fossil.exe` in your `%PATH%` and you’re +in a directory holding `repo.fossil`. See [the platform-independent +instructions](../any/none.md) for further details. + +For a more robust setup, we recommend that you [install Fossil as a +Windows service](./service.md), which will allow Fossil to start at +system boot, before anyone has logged in interactively. + + +## <a id="install"></a>Install IIS + +IIS might not be installed in your system yet, so follow the path +appropriate to your host OS. We’ve tested only the latest Microsoft +OSes as of the time of this writing, but the basic process should be +similar on older OSes. + + +### Windows Server 2019 + +1. Start “Server Manager” +2. Tell it you want to “Add roles and features” +3. Select “Role-based or feature-based installation” +4. Select your local server +5. In the Server Roles section, enable “Web Server (IIS)” + +### Windows + +1. Open Control Panel +2. Go to “Programs” +3. Select “Turn Windows features on or off” in the left-side pane +4. In the “Windows Features” dialog, enable “Internet Information + Services” + +The default set of IIS features there will suffice for this tutorial, +but you might want to enable additional features. + + +## Setting up the Proxy + +The stock IIS setup doesn’t have reverse proxying features, but they’re +easily added through extensions. You will need to install the +[Application Request Routing][arr] and [URL Rewrite][ure] extensions. In +my testing here, URL Rewrite showed up immediately after installing it, +but I had to reboot the server to get ARR to show up. (Yay Windows.) + +You can install these things through the direct links above, or you can +do it via the Web Platform Installer feature of IIS Manager (a.k.a. +`INETMGR`). + +Set these extensions up in IIS Manager like so: + +1. Double-click the “Application Request Routing Cache” icon. + +2. Right-click in the window that results, and select “Server Proxy + Settings...” + +3. Check the “Enable Proxy” box in the dialog. Click the “Apply” text + in the right-side pane. + +4. Return to the top server-level configuration area of IIS Manager and + double-click the “URL Rewrite” icon. Alternately, you might find + “URL Rewrite” in the right-side pane from within the ARR settings. + +5. Right click in the window that results, and click “Add Rule(s)...” + Tell it you want a “Blank rule” under “Inbound rules”. + +6. In the dialog that results, create a new rule called “Fossil repo + proxy.” Set the “Pattern” to “`^(.*)$`” and “Rewrite URL” set to + “`http://localhost:9000/{R:1}`”. That tells it to take everything in + the path part of the URL and send it down to localhost:9000, where + `fossil server` is listening. + +7. Click “Apply” in the right-side pane, then get back to the top level + configuration for the server, and click “Restart” in that same pane. + +At this point, if you go to `http://localhost/` in your browser, you +should see your Fossil repository’s web interface instead of the default +IIS web site, as before you did all of the above. + +This is a very simple configuration. You can do more complicated and +interesting things with this, such as redirecting only `/code` URLs to +Fossil by setting the Pattern in step 6 to “`^/code(.*)$`”. (You would +also need to pass `--baseurl http://example.com/code` in the `fossil +server` command to make this work properly.) IIS would then directly +serve all other URLs. You could also intermix ASP.NET applications in +the URL scheme in this way. + +See the documentation on [URL Rewrite rules][urr] for more ideas. + +*[Return to the top-level Fossil server article.](../)* + + +[arr]: https://www.iis.net/downloads/microsoft/application-request-routing +[tls]: https://docs.microsoft.com/en-us/iis/manage/configuring-security/understanding-iis-url-authorization +[ure]: https://www.iis.net/downloads/microsoft/url-rewrite +[urr]: https://docs.microsoft.com/en-us/iis/extensions/url-rewrite-module/creating-rewrite-rules-for-the-url-rewrite-module diff --git a/www/server/windows/index.md b/www/server/windows/index.md new file mode 100644 index 0000000..a30b00a --- /dev/null +++ b/www/server/windows/index.md @@ -0,0 +1,8 @@ +# Using Windows as a Fossil Server + +- [Fossil server command](./none.md) +- [Fossil as CGI (IIS)](./iis.md) +- [Fossil as a Service](./service.md) +- [Using stunnel with Fossil on Windows](./stunnel.md) + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/windows/none.md b/www/server/windows/none.md new file mode 100644 index 0000000..eab222d --- /dev/null +++ b/www/server/windows/none.md @@ -0,0 +1,69 @@ +# Serving as a Standalone Server on Windows + +On Windows, this method works more or less identically to how it’s +documented in [the generic instructions](../any/none.md). + +...but only while `fossil.exe` is actually running, which is the source +of much trouble on Windows. This problem has two halves: + + +## No App Startup Without Desktop + +The easy methods for starting a program in Windows at system start all +require an interactive desktop. There is no *easy* way to start an arbitrary +program on Windows at boot before anyone has logged in. In Unix +terms, Windows has no simple equivalent to [the `/etc/rc.local` file][rcl]. + +You can partially get around the first problem by setting your `fossil +server` call up as one of the user’s interactive startup items. Windows +10 has its own [idiosyncratic way of doing this][si10], and in older +systems you have [several alternatives to this][si7]. Regardless of the +actual mechanism, these will cause the Fossil standalone HTTP server to +start on an *interactive desktop login* only. While you’re sitting at +the Windows login screen, the Fossil server is *down*. + +[rcl]: http://nixdoc.net/man-pages/FreeBSD/man8/rc.local.8.html +[si10]: https://www.tenforums.com/tutorials/2944-add-delete-enable-disable-startup-items-windows-10-a.html +[si7]: https://www.wikihow.com/Change-Startup-Programs-in-Windows-7 + + + +## No Simple Background Mode + +Windows also lacks a direct equivalent of the Bourne shell’s “`&`” control operator to +run a program in the background, which you can give in Unix’s `rc.local` +file, which is just a normal Bourne shell script. + +By “background,” I mean +“not attached to any interactive user’s login session.” When the +`rc.local` script exits in Unix, any program it backgrounded *stays +running*. There is no simple and direct equivalent to this mechanism in +Windows. + +If you set `fossil server` to run on interactive login, as above, it +will shut right back down again when that user logs back out. + +With Windows 10, it’s especially problematic because you can no longer +make the OS put off updates arbitrarily: your Fossil server will go down +every time Windows Update decides it needs to reboot your computer, and +then Fossil service will *stay* down until someone logs back into that +machine interactively. + + +## Better Solutions + +Because of these problems, we only recommend setting `fossil server` up +on Windows this way when +you’re a solo developer or you work in a small office where everyone +arrives more or less at the same time each day, and everyone goes home +about the same time each day, so that one user can keep the Fossil +server up through the working day. + +If your needs go at all beyond this, you should expect proper “server” +behavior, which you can get on Windows by [registering Fossil as a +Windows service](./service.md), which solves the interactive startup and +shutdown problems above, at a bit of complexity over the Startup Items +method. You may also want to consider putting that service behind [an +IIS front-end proxy](./iis.md) to add additional web serving features. + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/windows/service.md b/www/server/windows/service.md new file mode 100644 index 0000000..e2cb3f2 --- /dev/null +++ b/www/server/windows/service.md @@ -0,0 +1,120 @@ +# Fossil as a Windows Service + +If you need Fossil to start automatically on Windows, it is suggested to install +Fossil as a Windows Service. + +## Assumptions + +1. You have Administrative access to a Windows 2012r2 or above server. +2. You have PowerShell 5.1 or above installed. + +## Place Fossil on Server + +However you obtained your copy of Fossil, it is recommended that you follow +Windows conventions and place it within `\Program Files\FossilSCM`, the +proper location for the official 64-bit binary. +This way Fossil is at an expected location and you will have minimal issues with +Windows interfering in your ability to run Fossil as a service. You will need +Administrative rights to place fossil at the recommended location. If you will +only be running Fossil as a service, you do not need to add this location to the +path, though you may do so if you wish. + +## Installing Fossil as a Service + +Luckily the hard work to use Fossil as a Windows Service has been done by the +Fossil team. We simply have to install it with the proper command line options. +Fossil on Windows has a command `fossil winsrv` to allow installing Fossil as a +service on Windows. This command is only documented on the windows executable +of Fossil. You must also run the command as administrator for it to be +successful. + +### Fossil winsrv Example + +The simplest form of the command is: + +``` +fossil winsrv create --repository D:/Path/to/Repo.fossil +``` + +This will create a windows service named 'Fossil-DSCM' running under the local +system account and accessible on port 8080 by default. `fossil winsrv` can also +start, stop, and delete the service. For all available options, please execute +`fossil help winsrv` on a windows install of Fossil. + +If you wish to serve a directory of repositories, the `fossil winsrv` command +requires a slightly different set of options vs. `fossil server`: + +``` +fossil winsrv create --repository D:/Path/to/Repos --repolist +``` + +### Choice of Directory Considerations + +When the Fossil server will be used at times that files may be locked +during virus scanning, it is prudent to arrange that its directory used +for temporary files is exempted from such scanning. Ordinarily, this +will be a subdirectory named "fossil" in the temporary directory given +by the Windows GetTempPath(...) API, [namely](https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-gettemppathw#remarks) +the value of the first existing environment variable from `%TMP%`, `%TEMP%`, +`%USERPROFILE%`, and `%SystemRoot%`; you can look for their actual values in +your system by accessing the `/test-env` webpage. +Excluding this subdirectory will avoid certain rare failures where the +fossil.exe process is unable to use the directory normally during a scan. + +### <a id='PowerShell'></a>Advanced service installation using PowerShell + +As great as `fossil winsrv` is, it does not have one to one reflection of all of +the `fossil server` [options](/help/server). When you need to use some of +the more advanced options, such as `--https`, `--skin`, or `--extroot`, you will +need to use PowerShell to configure and install the Windows service. + +PowerShell provides the [New-Service](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/new-service?view=powershell-5.1) +command, which we can use to install and configure Fossil as a service. The +below should all be entered as a single line in an Administrative PowerShell +console. + +```PowerShell +New-Service -Name fossil -DisplayName fossil -BinaryPathName '"C:\Program Files\FossilSCM\fossil.exe" server --port 8080 --repolist "D:/Path/to/Repos"' -StartupType Automatic +``` + +Please note the use of forward slashes in the repolist path passed to Fossil. +Windows will accept either back slashes or forward slashes in path names, but +Fossil has a preference for forward slashes. The use of `--repolist` will make +this a multiple repository server. If you want to serve only a single +repository, then leave off the `--repolist` parameter and provide the full path +to the proper repository file. Other options are listed in the +[fossil server](/help/server) documentation. + +The service will be installed by default to use the Local Service account. +Since Fossil only needs access to local files, this is fine and causes no +issues. The service will not be running once installed. You will need to start +it to proceed (the `-StartupType Automatic` parameter to `New-Service` will +result in the service auto-starting on boot). This can be done by entering + +```PowerShell +Start-Service -Name fossil +``` + +in the PowerShell console. + +Congratulations, you now have a base http accessible Fossil server running on +Windows. + +### Removing the Windows Service + +If you want to remove the Fossil service, execute the following from an +Administrative PowerShell or Command Prompt console: + +``` +sc.exe delete fossil +``` + +If you have Powershell version 6.0 or later, you can use: + +```PowerShell +Remove-Service -Name fossil +``` + +with the same effect. + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/server/windows/stunnel.md b/www/server/windows/stunnel.md new file mode 100644 index 0000000..5a7e8d1 --- /dev/null +++ b/www/server/windows/stunnel.md @@ -0,0 +1,156 @@ +# Using stunnel with Fossil on Windows + +While there are many ways to configure Fossil as a server using various web +servers (Apache, IIS, nginx, etc.), this document will focus on setting up a +minimal Fossil server using only Fossil's native [server +capabilities](../any/none.md) and [stunnel](https://www.stunnel.org/) +to provide a TLS proxy. It is recommended for public repositories to go to the +extra step of configuring stunnel to provide a proper HTTPS setup. + +## Assumptions + +1. You have Administrative access to a Windows 2012r2 or above server. +2. You have PowerShell 5.1 or above installed. +3. You have acquired a certificate either from a Public CA or an Internal CA. + +## Configure Fossil Service for https + +Due to the need for the `--https` option for successfully using Fossil with +stunnel, we will use [Advanced service installation using PowerShell](./service.md#PowerShell). +We will need to change the command to install the Fossil Service to configure +it properly for use with stunnel as an https proxy. Run the following: + +```PowerShell +New-Service -Name fossil-secure -DisplayName fossil-secure -BinaryPathName '"C:\Program Files\FossilSCM\fossil.exe" server --localhost --port 9000 --https --repolist "D:/Path/to/Repos"' -StartupType Automatic +``` + +The use of `--localhost` means Fossil will only listen for traffic on the local +host on the designated port - 9000 in this case - and will not respond to +network traffic. Using `--https` will tell Fossil to generate HTTPS URLs rather +than HTTP ones. + +`New-Service` does not automatically start a service on install, so you will +need to enter the following to avoid rebooting the server: + +```PowerShell +Start-Service -Name fossil-secure +``` + +To remove the service, run the following in a Powershell or cmd console: + +``` +sc.exe delete fossil +``` + +or (in a Powershell console) + +```PowerShell +Remove-Service -Name fossil +``` + +if your version of Powershell is 6.0 or above. + +## Install stunnel 5.55 + +Download stunnel from the [downloads](https://www.stunnel.org/downloads.html) +page. Select the latest stunnel windows package (at the time of writing this is +`stunnel-5.55-win64-installer.exe`). Execute the installer and make sure you +install openSSL tools when you install stunnel. You will need this to convert +your certificate from PFX to PEM format. + +Even though the installer says it is for win64, it installs stunnel by default +to `\Program Files (x86)\stunnel`. + +## Get your certificate ready for Stunnel + +Whether you use a Public Certificate Authority or Internal Certificate +Authority, the next step is exporting the certificate from Windows into a format +useable by Stunnel. + +### Export Certificate from Windows + +If your certificate is installed via Windows Certificate Management, you will +need to export the certificate into a usable format. You can do this either +using the Windows Certificate Management Console, or PowerShell. + +#### Certificate Management Console + +Start `mmc.exe` as an Administrator. Select 'File>Add/Remove Snapin', select +'Certificates' from the list, and click 'Add'. Select 'Computer Account', +'Next', 'Local Computer', and then 'Finish'. In the Console Root, expand +'Certificates', then 'Personal', and select 'Certificates'. In the middle pane +find and select your certificate. Right click the certificate and select +'All Tasks>Export'. You want to export as PFX the Private Key, include all +certificates in the certification path, and use a password only to secure the +file. Enter a path and file name to a working directory and complete the +export. + +Continue with [Convert Certificate from PFX to PEM](#convert). + +#### PowerShell + +If you know the Friendly +Name of the Certificate this is relatively easy. Since you need to export +the private key as well, you must run the following from an Administrative +PowerShell console. + +```PowerShell +$passwd = ConvertTo-SecureString -string "yourpassword" -Force -AsPlainText + +Get-ChildItem Cert:\LocalMachine\My | Where{$_.FriendlyName -eq "FriendlyName"} | +Export-PfxCertificate -FilePath fossil-scm.pfx -Password $passwd +``` + +You will now have your certificate stored as a PFX file. + +<a id="convert"></a> +### Convert Certificate from PFX to PEM + +For this step you will need the openssl tools that were installed with stunnel. + +```PowerShell +# Add stunnel\bin directory to path for this session. +$env:PATH += ";${env:ProgramFiles(x86)}\stunnel\bin" +# Export Private Key +openssl.exe pkcs12 -in fossil-scm.pfx -out fossil-scm.key -nocerts -nodes +# Export the Certificate +openssl.exe pkcs12 -in fossil-scm.pfx -out fossil-scm.pem -nokeys +``` + +Now move `fossil-scm.key` and `fossil-scm.pem` to your stunnel config directory +(by default this should be located at `\Program Files (x86)\stunne\config`). + +## stunnel Configuration + +Use the reverse proxy configuration given in the generic [Serving via +stunnel document](../any/stunnel.md#proxy). On Windows, the +`stunnel.conf` file is located at `\Program Files (x86)\stunnel\config`. + +You will need to modify it to point at the PEM and key files generated +above. + +After completing the above configuration restart the stunnel service in Windows +with the following: + +```PowerShell +Restart-Service -Name stunnel +``` + +## Open up port 443 in the Windows Firewall + +The following instructions are for the [Windows Advanced +Firewall](https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-firewall/windows-firewall-with-advanced-security). +If you are using a different Firewall, please consult your Firewall +documentation for how to open port 443 for inbound traffic. + +The following command should be entered all on one line. + +```PowerShell +New-NetFirewallRule -DisplayName "Allow Fossil Inbound" -Description "Allow Fossil inbound on port 443 using Stunnel as TLS Proxy." + -Direction Inbound -Protocol TCP -LocalPort 443 -Action Allow -Program "C:\Program Files (x86)\Stunnel\bin\stunnel.exe" +``` + +You should now be able to access your new Fossil Server via HTTPS. + + +*[Return to the top-level Fossil server article.](../)* diff --git a/www/serverext.wiki b/www/serverext.wiki new file mode 100644 index 0000000..b20f6a5 --- /dev/null +++ b/www/serverext.wiki @@ -0,0 +1,375 @@ +<title>CGI Server Extensions</title> + +<h2>1.0 Introduction</h2> + +If you have a [./server/|Fossil server] for your project, +you can add [./aboutcgi.wiki|CGI] +extensions to that server. These extensions work like +any other CGI program, except that they also have access to the Fossil +login information and can (optionally) leverage the "[./customskin.md|skins]" +of Fossil so that they appear to be more tightly integrated into the project. + +An example of where this is useful is the +[https://sqlite.org/src/ext/checklist|checklist application] on +the [https://sqlite.org/|SQLite] project. The checklist +helps the SQLite developers track which release tests have passed, +or failed, or are still to be done. The checklist program began as a +stand-alone CGI which kept its own private user database and implemented +its own permissions and login system and provided its own CSS. By +converting checklist into a Fossil extension, the same login that works +for the [https://sqlite.org/src|main SQLite source repository] also works +for the checklist. Permission to change elements of the checklist +is tied on permission to check-in to the main source repository. And +the standard Fossil header menu and footer appear on each page of +the checklist. + +<h2>2.0 How It Works</h2> + +CGI Extensions are disabled by default. +An administrator activates the CGI extension mechanism by specifying +an "Extension Root Directory" or "extroot" as part of the +[./server/index.html|server setup]. +If the Fossil server is itself run as +[./server/any/cgi.md|CGI], then add a line to the +[./cgi.wiki#extroot|CGI script file] that says: + +<pre> + extroot: <i>DIRECTORY</i> +</pre> + +Or, if the Fossil server is being run using the +"[./server/any/none.md|fossil server]" or +"[./server/any/none.md|fossil ui]" or +"[./server/any/inetd.md|fossil http]" commands, then add an extra +"--extroot <i>DIRECTORY</i>" option to that command. + +The <i>DIRECTORY</i> is the DOCUMENT_ROOT for the CGI. +Files in the DOCUMENT_ROOT are accessed via URLs like this: + +<pre> + https://example-project.org/ext/<i>FILENAME</i> +</pre> + +In other words, access files in DOCUMENT_ROOT by appending the filename +relative to DOCUMENT_ROOT to the [/help/www/ext|/ext] +page of the Fossil server. + + * Files that are readable but not executable are returned as static + content. + + * Files that are executable are run as CGI. + +<h3>2.1 Example #1</h3> + +The source code repository for SQLite is a Fossil server that is run +as CGI. The URL for the source code repository is [https://sqlite.org/src]. +The CGI script looks like this: + +<verbatim> +#!/usr/bin/fossil +repository: /fossil/sqlite.fossil +errorlog: /logs/errors.txt +extroot: /sqlite-src-ext +</verbatim> + +The "extroot: /sqlite-src-ext" line tells Fossil that it should look for +extension CGIs in the /sqlite-src-ext directory. (All of this is happening +inside of a chroot jail, so putting the document root in a top-level +directory is a reasonable thing to do.) + +When a URL like "https://sqlite.org/src/ext/checklist" is received by the +main webserver, it figures out that the /src part refers to the main +Fossil CGI script and so it runs that script. Fossil gets the remainder +of the URL to work with: "/ext/checklist". Fossil extracts the "/ext" +prefix and uses that to determine that this a CGI extension request. +Then it takes the leftover "/checklist" part and appends it to the +"extroot" to get the filename "/sqlite-src-ext/checklist". Fossil finds +that file to be executable, so it runs it as CGI and returns the result. + +The /sqlite-src-ext/checklist file is a +[https://wapp.tcl.tk|Wapp program]. The current source code to the +this program can be seen at +[https://www.sqlite.org/src/ext/checklist/3070700/self] and +recent historical versions are available at +[https://sqlite.org/docsrc/finfo/misc/checklist.tcl] with +older legacy at [https://sqlite.org/checklistapp/timeline?n1=all] + +There is a cascade of CGIs happening here. The web server that receives +the initial HTTP request runs Fossil as a CGI based on the +"https://sqlite.org/src" portion of the URL. The Fossil instance then +runs the checklist sub-CGI based on the "/ext/checklists" suffix. The +output of the sub-CGI is read by Fossil and then relayed on to the +main web server which in turn relays the result back to the original client. + +<h3>2.2 Example #2</h3> + +The [https://fossil-scm.org/home|Fossil self-hosting repository] is also +a CGI that looks like this: + +<verbatim> +#!/usr/bin/fossil +repository: /fossil/fossil.fossil +errorlog: /logs/errors.txt +extroot: /fossil-extroot +</verbatim> + +The extroot for this Fossil server is /fossil-extroot and in that directory +is an executable file named "fileup1" - another [https://wapp.tcl.tk|Wapp] +script. (The extension mechanism is not required to use Wapp. You can use +any kind of program you like. But the creator of SQLite and Fossil is fond +of [https://www.tcl.tk|Tcl/Tk] and so he tends to gravitate toward Tcl-based +technologies like Wapp.) The fileup1 script is a demo program that lets +the user upload a file using a form, and then displays that file in the reply. +There is a link on the page that causes the fileup1 script to return a copy +of its own source-code, so you can see how it works. + +<h3>2.3 Example #3</h3> + +For Fossil versions dated 2025-03-23 and later, the "--extpage FILENAME" +option to the [/help/ui|fossil ui] command is a short cut that treats +FILENAME as a CGI extension. When the ui command starts up a new web browser +pages, it points that page to the FILENAME extension. So if FILENAME is +a static content file (such as an HTML file or +[/md_rules|Markdown] or [/wiki_rules|Wiki] document), then the +rendered content of the file is displayed. Meanwhile, the user can be +editing the source text for that document in a separate window, and +periodically pressing "Reload" on the web browser to instantly view the +rendered results. + +For example, the author of this documentation page is running +"<tt>fossil ui --extpage www/serverext.wiki</tt>" while editing this +very paragraph, and presses Reload from time to time to view his +edits. + +The same idea applies when developing new CGI applications using a script +language (for example using [https://wapp.tcl.tk|Wapp]). Run the +command "<tt>fossil ui --extpage SCRIPT</tt>" where SCRIPT is the name +of the application script, while editing that script in a separate +window, then press Reload periodically on the web browser to test the +script. + +<h2 id="cgi-inputs">3.0 CGI Inputs</h2> + +The /ext extension mechanism is an ordinary CGI interface. Parameters +are passed to the CGI program using environment variables. The following +standard CGI environment variables are supplied: + + * AUTH_TYPE + * AUTH_CONTENT + * CONTENT_LENGTH + * CONTENT_TYPE + * DOCUMENT_ROOT + * GATEWAY_INTERFACE + * HTTPS + * HTTP_ACCEPT + * HTTP_ACCEPT_ENCODING + * HTTP_COOKIE + * HTTP_HOST + * HTTP_IF_MODIFIED_SINCE + * HTTP_IF_NONE_MATCH + * HTTP_REFERER + * HTTP_USER_AGENT + * PATH_INFO + * QUERY_STRING + * REMOTE_ADDR + * REMOTE_USER + * REQUEST_METHOD + * REQUEST_SCHEME + * REQUEST_URI + * SCRIPT_DIRECTORY + * SCRIPT_FILENAME + * SCRIPT_NAME + * SERVER_NAME + * SERVER_PORT + * SERVER_PROTOCOL + * SERVER_SOFTWARE + +Do a web search for +"[https://duckduckgo.com/?q=cgi+environment_variables|cgi environment variables]" +to find more detail about what each of the above variables mean and how +they are used. +Live listings of the values of some or all of these environment variables +can be found at links like these: + + * [https://fossil-scm.org/home/test-env] + * [https://sqlite.org/src/ext/checklist/top/env] + +In addition to the standard CGI environment variables listed above, +Fossil adds the following: + + * FOSSIL_CAPABILITIES + * FOSSIL_NONCE + * FOSSIL_REPOSITORY + * FOSSIL_URI + * FOSSIL_USER + +The FOSSIL_USER string is the name of the logged-in user. This variable +is missing or is an empty string if the user is not logged in. The +FOSSIL_CAPABILITIES string is a list of +[./caps/ref.html|Fossil capabilities] that +indicate what permissions the user has on the Fossil repository. +The FOSSIL_REPOSITORY environment variable gives the filename of the +Fossil repository that is running. The FOSSIL_URI variable shows the +prefix of the REQUEST_URI that is the Fossil CGI script, or is an +empty string if Fossil is being run by some method other than CGI. + +The [https://sqlite.org/src/ext/checklist|checklist application] uses the +FOSSIL_USER environment variable to determine the name of the user and +the FOSSIL_CAPABILITIES variable to determine if the user is allowed to +mark off changes to the checklist. Only users with check-in permission +to the Fossil repository are allowed to mark off checklist items. That +means that the FOSSIL_CAPABILITIES string must contain the letter "i". +Search for "FOSSIL_CAPABILITIES" in the +[https://sqlite.org/src/ext/checklist/top/self|source listing] to see how +this happens. + +If the CGI output is one of the forms for which Fossil inserts its own +header and footer, then the inserted header will include a +Content Security Policy (CSP) restriction on the use of javascript within +the webpage. Any <script>...</script> elements within the +CGI output must include a nonce or else they will be suppressed by the +web browser. The FOSSIL_NONCE variable contains the value of that nonce. +So, in other words, to get javascript to work, it must be enclosed in: + +<verbatim> +<script nonce='$FOSSIL_NONCE'>...</script> +</verbatim> + +Except, of course, the $FOSSIL_NONCE is replaced by the value of the +FOSSIL_NONCE environment variable. + +<h3>3.1 Input Content</h3> + +If the HTTP request includes content (for example if this is a POST request) +then the CONTENT_LENGTH value will be positive and the data for the content +will be readable on standard input. + + +<h2>4.0 CGI Outputs</h2> + +CGI programs construct a reply by writing to standard output. The first +few lines of output are parameters intended for the web server that invoked +the CGI. These are followed by a blank line and then the content. + +Typical parameter output looks like this: + +<verbatim> +Status: 200 OK +Content-Type: text/html +</verbatim> + +CGI programs can return any content type they want - they are not restricted +to text replies. It is OK for a CGI program to return (for example) +image/png. + +The fields of the CGI response header can be any valid HTTP header fields. +Those that Fossil does not understand are simply relayed back to up the +line to the requester. + +Fossil takes special action with some content types. If the Content-Type +is "text/x-fossil-wiki" or "text/x-markdown" then Fossil +converts the content from [/wiki_rules|Fossil-Wiki] or +[/md_rules|Markdown] into HTML, adding its +own header and footer text according to the repository skin. Content +of type "text/html" is normally passed straight through +unchanged. However, if the text/html content is of the form: + +<verbatim> +<div class='fossil-doc' data-title='DOCUMENT TITLE'> +... HTML content there ... +</div> +</verbatim> + +In other words, if the outer-most markup of the HTML is a <div> +element with a single class of "fossil-doc", +then Fossil will adds its own header and footer to the HTML. The +page title contained in the added header will be extracted from the +"data-title" attribute. + +Except for the three cases noted above, Fossil makes no changes or +additions to the CGI-generated content. Fossil just passes the verbatim +content back up the stack towards the requester. + +<h3>4.1 <tt>GATEWAY_INTERFACE</tt> and Recursive Calls to fossil</h3> + +Like many CGI-aware applications, if fossil sees the environment +variable <tt>GATEWAY_INTERFACE</tt> when it starts up, it assumes it +is running in a CGI environment and behaves differently than when it +is run in a non-CGI interactive session. If you intend to run fossil +itself from within an extension CGI script, e.g. to run a query +against the repository or simply fetch the fossil binary version, make +sure to <em>unset</em> the <tt>GATEWAY_INTERFACE</tt> environment +variable before doing so, otherwise the invocation will behave as if +it's being run in CGI mode. + +<h2>5.0 Filename Restrictions</h2> + +For security reasons, Fossil places restrictions on the names of files +in the extroot directory that can participate in the extension CGI +mechanism: + + 1. Filenames must consist of only ASCII alphanumeric characters, + ".", "_", and "-", and of course "/" as the file separator. + Files with names that includes spaces or + other punctuation or special characters are ignored. + + 2. No element of the pathname can begin with "." or "-". Files or + directories whose names begin with "." or "-" are ignored. + +If a CGI program requires separate data files, it is safe to put those +files in the same directory as the CGI program itself as long as the names +of the data files contain special characters that cause them to be ignored +by Fossil. + +<h2>6.0 Access Permissions</h2> + +CGI extension files and programs are accessible to everyone. + +When CGI extensions have been enabled (using either "extroot:" in the +CGI file or the --extroot option for other server methods) all files +in the extension root directory hierarchy, except special filenames +identified previously, are accessible to all users. Users do not +have to have "Read" privilege, or any other privilege, in order to +access the extensions. + +This is by design. The CGI extension mechanism is intended to operate +in the same way as a traditional web-server. + +CGI programs that want to restrict access +can examine the FOSSIL_CAPABILITIES and/or FOSSIL_USER environment variables. +In other words, access control is the responsibility of the individual +extension programs. + +<h3>6.1 Restricting Robot Access To Extensions</h3> + +If the "ext" tag is found in the [/help/robot-restrict|robot-restrict setting] +then clients are tested to see if they are robots before granting +access to any extension. If the "ext" tag is omitted but a tag +of the form "ext/PATH" is found on the robot-restrict setting, then +robots are restricted from the particular extension at PATH. + +<h2>7.0 Trouble-Shooting Hints</h2> + +Remember that the /ext will return any file in the extroot directory +hierarchy as static content if the file is readable but not executable. +When initially setting up the /ext mechanism, it is sometimes helpful +to verify that you are able to receive static content prior to starting +work on your CGIs. Also remember that CGIs must be +executable files. + +Fossil likes to run inside a chroot jail, and will automatically put +itself inside a chroot jail if it can. The sub-CGI program will also +run inside this same chroot jail. Make sure all embedded pathnames +have been adjusted accordingly and that all resources needed by the +CGI program are available within the chroot jail. + +If anything goes wrong while trying to process an /ext page, Fossil +returns a 404 Not Found error with no details. However, if the requester +is logged in as a user that has <b>[./caps/ref.html#D | Debug]</b> capability +then additional diagnostic information may be included in the output. + +If the /ext page has a "fossil-ext-debug=1" query parameter and if +the requester is logged in as a user with Debug privilege, then the +CGI output is returned verbatim, as text/plain and with the original +header intact. This is useful for diagnosing problems with the +CGI script. diff --git a/www/settings.wiki b/www/settings.wiki new file mode 100644 index 0000000..5e63b1c --- /dev/null +++ b/www/settings.wiki @@ -0,0 +1,61 @@ +<title>Fossil Settings</title> + +<h1>Using Fossil Settings</h1> + +Settings control the behaviour of fossil. They are set with the +<tt>fossil settings</tt> command, or through the web interface in +the Settings page in the Admin section. + +For a list of all settings, view the Settings page, or type +<tt>fossil help settings</tt> from the command line. + + +<h2 id="repo">1.0 Repository settings</h2> + +Settings are set on a per-repository basis. When you clone a repository, +a subset of settings are copied to your local repository. + +If you make a change to a setting on your local repository, it is not +synced back to the server when you <tt>push</tt> or <tt>sync</tt>. If +you make a change on the server, you need to manually make the change on +all repositories which are cloned from this repository. + +You can also set a setting globally on your local machine. The value +will be used for all repositories cloned to your machine, unless +overridden explicitly in a particular repository. Global settings can be +set by using the <tt>-global</tt> option on the <tt>fossil settings</tt> +command. + +<h2 id="versionable">2.0 "Versionable" settings</h2> + +Most of the settings control the behaviour of fossil on your local +machine, largely acting to reflect your preference on how you want to +use Fossil, how you communicate with the server, or options for hosting +a repository on the web. + +However, for historical reasons, some settings affect how you work with +versioned files. These are <tt>clean-glob</tt>, <tt>binary-glob</tt>, +<tt>crlf-glob</tt> (and its alias <tt>crnl-glob</tt>), <tt>empty-dirs</tt>, +<tt>encoding-glob</tt>, <tt>ignore-glob</tt>, <tt>keep-glob</tt>, +<tt>manifest</tt>, and <tt>mimetypes</tt>. The most important is +<tt>ignore-glob</tt> which specifies which files should be ignored when +looking for unmanaged files with the <tt>extras</tt> command. + +Because these options can change over time, and the inconvenience of +replicating changes, these settings are "versionable". As well as being +able to be set using the <tt>settings</tt> command or the web interface, +you can create versioned files in the <tt>.fossil-settings</tt> +subdirectory of the check-out root, named with the setting name. +Each file holds the value of a setting, and these files are checked in, +committed, merged, and so on, as with any other file. + +Where a setting is a list of values, such as <tt>ignore-glob</tt>, you +can use a newline as a separator as well as a comma. + +For example, to set the list of ignored files, create a +<tt>.fossil-settings/ignore-glob</tt> file where each line contains a +glob for ignored files. + +If you set the value of a setting using the <tt>settings</tt> command as +well as a versioned file, the versioned setting will take precedence. A +warning will be displayed. diff --git a/www/shunning.wiki b/www/shunning.wiki new file mode 100644 index 0000000..975fd0b --- /dev/null +++ b/www/shunning.wiki @@ -0,0 +1,163 @@ +<title>Deleting Content From Fossil</title> + +<h2>Good Reasons for Removing Content from a Fossil Repository</h2> + +Fossil is designed to keep all historical content forever. Fossil +purposely makes it difficult for users to delete content. Old content +is part of the project's <i>*ahem*</i> fossil record and should be +maintained indefinitely to maintain an accurate history of the project. + +Nevertheless, there may occasionally arise legitimate reasons for +deleting content. Such reasons include: + + * Spammers inserted inappropriate content into a wiki page, forum post, + or ticket. Fossil lets you easily hide or amend such content, but + since it is not a legitimate part of the project's history, there + is no value in keeping it, so it is best removed permanently. + + * A file that contains trade secrets or that is under someone else's + copyright was accidentally committed and needs to be backed out. + + * A malformed control artifact was inserted and is disrupting the + operation of Fossil. + + * A legitimate legal request was received requiring content to + be removed. This would most likely be related to the accidental + intellectual property error or spam cases listed above. Some countries + recognise software patents, and so allow legal claims targetting code + commits. Some countries can require publicly-available encryption + software to be taken down if it is committed to the DAG without + the correct government authorisation. + +<h2>Alternatives</h2> + +All of these are rare cases: Fossil is [./antibot.wiki | designed to +foil spammers up front], legally problematic check-ins should range from +rare to nonexistent, and you have to go way out of your way to force +Fossil to insert bad control artifacts. Therefore, before we get to +methods of permanently deleting content from a Fossil repo, let's give +some alternatives that usually suffice, which don't damage the project's +fossil record: + + * When a forum post or wiki article is "deleted," what actually + happens is that a new empty version is added to the Fossil repository. + The web interface interprets this + as "deleted," but the prior version remains available if you go + digging for it. + + * When you close a ticket, it's marked in a way that causes it + to not show up in the normal ticket reports. You usually want to + give it a Resolution such as "Rejected" when this happens, plus + possibly a comment explaining why you're closing it. This is all new + information added to the ticket, not deletion. + + * When you <tt>fossil rm</tt> a file, a new manifest is + checked into the repository with the same file list as for the prior + version minus the "removed" file. The file is still present in the + repository; it just isn't part of that version forward on that + branch. + + * If you make a bad check-in, you can shunt it off to the side + by amending it to put it on a different branch, then continuing + development on the prior branch: + <br><br> + <code>$ fossil amend abcd1234 --branch BOGUS --hide<br> + $ fossil up trunk</code> + <br><br> + The first command moves check-in ID <tt>abcd1234</tt> (and any + subsequent check-ins on that branch!) to a branch called + <tt>BOGUS</tt>, then hides it so it doesn't show up on the + timeline. You can call this branch anything you like, and you can + re-use the same name as many times as you like. No content is + actually deleted: it's just shunted off to the side and hidden away. + You might find it easier to do this from the Fossil web UI in + the "edit" function for a check-in. + <br><br> + The second command returns to the last good check-in on that branch + so you can continue work from that point. + + * When the check-in you want to remove is followed by good + check-ins on the same branch, you can't use the previous method, + because it will move the good check-ins, too. The solution is: + <br><br> + <tt>$ fossil merge --backout abcd1234</tt> + <br><br> + That creates a diff in the check-out directory that backs out the + bad check-in <tt>abcd1234</tt>. You then fix up any merge conflicts, + build, test, etc., then check the reverting change into the + repository. Again, nothing is actually deleted; you're just adding + more information to the repository which corrects a prior + check-in. + +<h2>Exception: Non-versioned Content</h2> + +It is normal and expected to delete data which is not versioned, such as +usernames and passwords in the user table. The [/help/scrub|fossil scrub] +command will remove all sensitive non-versioned data from a repository. + +The scrub command will remove user 'bertina', along with their password, +any supplied IP address, any concealed email address etc. However, in the +DAG, commits by 'bertina' will continue to be visible unchanged even though +there is no longer any such user in Fossil. + +<h2>Shunning</h2> + +Fossil provides a mechanism called "shunning" for removing content from +a repository. + +Every Fossil repository maintains a list of the hash names of +"shunned" artifacts. +Fossil will refuse to push or pull any shunned artifact. +Furthermore, all shunned artifacts (but not the shunning list +itself) are removed from the +repository whenever the repository is reconstructed using the +"rebuild" command. + +<h3>Shunning lists are local state</h3> + +The shunning list is part of the local state of a Fossil repository. +In other words, shunning does not propagate to a remote repository +using the normal "sync" mechanism. An artifact can be +shunned from one repository but be allowed to exist in another. The fact that +the shunning list does not propagate is a security feature. If the +shunning list propagated then a malicious user (or +a bug in the fossil code) might introduce a shun record that would +propagate through all repositories in a network and permanently +destroy vital information. By refusing to propagate the shunning list, +Fossil ensures that no remote user will ever be able to remove +information from your personal repositories without your permission. + +The shunning list does not propagate to a remote repository +by the normal "sync" mechanism, +but it is still possible to copy shuns from one repository to another +using the "configuration" command: + + <b>fossil configuration pull shun</b> <i>remote-url</i><br> + <b>fossil configuration push shun</b> <i>remote-url</i> + +The two commands above will pull or push shunning lists from or to +the <i>remote-url</i> indicated and merge the lists on the receiving +end. "Admin" privilege on the remote server is required in order to +push a shun list. In contrast, the shunning list will be automatically +received by default as part of a normal client "pull" operation unless +disabled by the "<tt>auto-shun</tt>" setting. + +Note that the shunning list remains in the repository even after the +shunned artifact has been removed. This is to prevent the artifact +from being reintroduced into the repository the next time it syncs with +another repository that has not shunned the artifact. + +<h3>Managing the shunning list</h3> + +The complete shunning list for a repository can be viewed by a user +with "admin" privilege on the "/shun" URL of the web interface to Fossil. +That URL is accessible under the "Admin" button on the default menu +bar. Items can be added to or removed from the shunning list. "Sync" +operations are inhibited as soon as the artifact is added to the +shunning list, but the content of the artifact is not actually removed +from the repository until the next time the repository is rebuilt. + +When viewing individual artifacts with the web interface, "admin" +users will usually see a "Shun" option in the submenu that will take +them directly to the shunning page and enable that artifact to be +shunned with a single additional mouse click. diff --git a/www/signing.md b/www/signing.md new file mode 100644 index 0000000..24071bd --- /dev/null +++ b/www/signing.md @@ -0,0 +1,124 @@ +# Signing Check-ins + +Fossil can sign check-in manifests. A basic concept in public-key +cryptography, signing can bring some advantages such as authentication and +non-repudiation. In practice, a serious obstacle is the public key +infrastructure – that is, the problem of reliably verifying that a given +public key belongs to its supposed owner (also known as _"signing is easy, +verifying is hard"_). + +Fossil neither creates nor verifies signatures by itself, instead relying on +external tools that have to be installed side-by-side. Historically, the tool +most employed for this task was [GnuPG](https://gnupg.org); recently, there has +been an increase in the usage of [OpenSSH](https://openssh.com) (the minimum +required version is 8.1, released on 2019-10-09). + +## Signing a check-in + +The `clearsign` setting must be on; this will cause every check-in to be signed +(unless you provide the `--nosign` flag to `fossil commit`). To this end, +Fossil calls the command given by the `pgp-command` setting. + +Fossil needs a non-detached signature that includes the rest of the usual +manifest. For GnuPG, this is no problem, but as of 2025 (version 9.9p1) OpenSSH +can create **and verify** only detached signatures; Fossil itself must +attach this signature to the manifest prior to committing. This makes the +verification more complex, as additional steps are needed to extract the +signature and feed it into OpenSSH. + +### GnuPG + +The `pgp-command` setting defaults to +`gpg --clearsign -o`. +(A possible interesting option to `gpg --clearsign` is `-u`, to specify the +user to be used for signing.) + +### OpenSSH + +A reasonable value for `pgp-command` is + +``` +ssh-keygen -q -Y sign -n fossilscm -f ~/.ssh/id_ed25519 +``` + +for Linux, and + +``` +ssh-keygen -q -Y sign -n fossilscm -f %USERPROFILE%/.ssh/id_ed25519 +``` + +for Windows, changing as appropriate `-f` to the path of the private key to be +used. + +The value for `-n` (the _namespace_) can be changed at will, but care has to be +taken to use the same value when verifying the signature. + +Fossil versions prior to 2.26 do not understand SSH signatures and +will treat artifacts signed this way as opaque blobs, not Fossil +artifacts. + + +## Verifying a signature + +Fossil does not provide an internal method for verifying signatures and +relies – like it does for signing – on external tools. + +### GnuPG + +Assuming you used the +default GPG command for signing, one can verify the signature using + +``` +fossil artifact <CHECK-IN> | gpg --verify +``` + +### OpenSSH + +The user and the key that was used to create the signature must be listed +together in the `ALLOWED_SIGNERS` file (see +[documentation](https://man.openbsd.org/ssh-keygen#ALLOWED_SIGNERS)). +Note that in that file, the "@DOMAIN" bit for the principal is only a +recommendation; you can (or even _should_) simply use your Fossil user name. + +As mentioned, for lack of an OpenSSH built-in non-detached signature mechanism, +the burden of extracting the relevant part of the signed check-in is on the +user. + +The following recipes are provided only as examples and can be easily extended +to fully-fledged scripts. + +#### For Linux: + +```bash +fsig=$(mktemp /tmp/__fsig.XXXXXX) && \ +fusr=$(fossil artifact tip \ + | awk -v m="${fsig}" -v s="${fsig}.sig" \ + '/^-----BEGIN SSH SIGNED/{of=m;next} \ + /^-----BEGIN SSH SIGNATURE/{of=s} \ + /^U /{usr=$2} \ + /./{if(!of){exit 42};print >> of} END{print usr}') && \ +ssh-keygen -Y verify -f ~/.ssh/allowed_signers -I ${fusr} -n fossilscm \ + -s "${fsig}.sig" < "${fsig}" || echo "No SSH signed check-in" && \ +rm -f "${fsig}.sig" "${fsig}" && \ +unset -v fsig fusr +``` + +#### For Windows (cmd): + +The following incantation makes use of `awk` and `dos2unix`, standard Unix +tools but requiring separate installation on Windows (for example,using +[BusyBox](https://frippery.org/busybox/#downloads)). The usage of `awk` can be +replaced with the Windows basic tool `findstr`, leading to a longer recipe. + +```bat +fossil artifact <CHECK-IN> | awk -v m="__fsig" -v s="__fsig.sig" ^ + "/^-----BEGIN SSH SIGNED/{of=m;next} /^-----BEGIN SSH SIGNATURE/{of=s} /./{if(!of){exit 42};print >> of}" +if %errorlevel% equ 42 (echo No SSH signed check-in) +REM ---Skip remaining lines if no SSH signed message--- +for /f "tokens=2" %i in ('findstr /b "U " __fsig') do set fusr=%i +dos2unix __fsig __fsig.sig +ssh-keygen -Y verify -f %USERPROFILE%\.ssh\allowed_signers -I "%fusr%" ^ + -n fossilscm -s __fsig.sig < __fsig +del __fsig __fsig.sig 2>nul & set "fusr=" +``` + diff --git a/www/ssl-server.md b/www/ssl-server.md new file mode 100644 index 0000000..7fd168c --- /dev/null +++ b/www/ssl-server.md @@ -0,0 +1,289 @@ +# SSL/TLS Server Mode + +## History + +Fossil has supported [client-side SSL/TLS][0] since [2010][1]. This means +that commands like "[fossil sync](/help/sync)" could use SSL/TLS when +contacting a server. But on the server side, commands like +"[fossil server](/help/server)" operated in clear-text only. To implement +an encrypted server, you had to put Fossil behind a web server or reverse +proxy that handled the SSL/TLS decryption/encryption and passed cleartext +down to Fossil. + +[0]: ./ssl.wiki +[1]: /timeline?c=b05cb4a0e15d0712&y=ci&n=13 + +Beginning in [late December 2021](/timeline?c=f6263bb64195b07f&y=a&n=13), +Fossil servers are now able to converse directly over TLS. Commands like + + * "[fossil server](/help/server)" + * "[fossil ui](/help/ui)", and + * "[fossil http](/help/http)" + +may now handle the encryption natively when suitably configured, without +requiring a third-party proxy layer. + +## <a id="usage"></a>Usage + +To put any of the Fossil server commands into SSL/TLS mode, simply +add the "`--cert`" command-line option: + + fossil ui --cert unsafe-builtin + +Here, we are passing the magic name "unsafe-builtin" to cause Fossil to +use a [hard-coded self-signed cert][hcssc] rather than one obtained from +a recognized [Certificate Authority][CA], or "CA". + +As the name implies, this self-signed cert is _not secure_ and should +only be used for testing. Your web browser is likely to complain +bitterly about it and will refuse to display the pages using the +"unsafe-builtin" cert until you placate it. The complexity of the +ceremony demanded depends on how paranoid your browser’s creators have +decided to be. It may require as little as clicking a single big "I know +the risks" type of button, or it may require a sequence be several +clicks designed to discourage the “yes, yes, just let me do the thing” +crowd lest they run themselves into trouble by disregarding well-meant +warnings. + +Our purpose here is to show you an alternate path that will avoid the +issue entirely, not weigh in on which browser handles self-signed +certificates best. + +[CA]: https://en.wikipedia.org/wiki/Certificate_authority +[hcssc]: /info/c2a7b14c3f541edb96?ln=89-116 + +## <a id="about"></a>About Certs + +The X.509 certificate system used by browsers to secure TLS connections +is based on asymmetric public-key cryptography. The methods for +obtaining one vary widely, with a resulting tradeoff we may summarize as +trustworthiness versus convenience, the latter characteristic falling as +the former rises.(^No strict correlation exists. CAs have invented +highly inconvenient certification schemes that offer little additional +real-world trustworthiness. Extreme cases along this axis may be fairly +characterized as [security theater][st]. We focus in this document on +well-balanced trade-offs between decreasing convenience and useful +levels of trustworthiness gained thereby.) + +The self-signed method demonstrated above offers approximately zero +trustworthiness, though not zero _value_ since it does still provide +connection encryption. + +More trustworthy methods are necessarily less convenient. One such is to +send your public key and the name of the domain you want to protect to a +recognized CA, which then performs one or more tests to convince itself +that the requester is in control of that domain. If the CA’s tests all +pass, it produces an X.509 certificate bound to that domain, which +includes assorted other information under the CA’s digital signature +attesting to the validity of the document’s contents. The result is sent +back to the requester, which may then use it to transitively attest to +these tests’ success: presuming one cannot fake the type of signature +used, the document must have been signed by the trusted, recognized CA. + +There is one element of the assorted information included with a +certificate that is neither supplied by the requester nor rubber-stamped +on it in passing by the CA. It also generates a one-time key pair and +stores the public half in the certificate. The cryptosystem this keypair +is intended to work with varies both by the CA and by time, as older +systems become obsolete. Details aside, the CA then puts this matching +private half of the key in a separate file, often encrypted under a +separate cryptosystem for security. + +SSL/TLS servers need both resulting halves to make these attestations, +but they send only the public half to the client when establishing the +connection. The client then makes its own checks to determine whether it +trusts the attestations being made. + +A properly written and administered server never releases the private +key to anyone. Ideally, it goes directly from the CA to the requesting +server and never moves from there; then when it expires, the server +deletes it permanently. + +[st]: https://en.wikipedia.org/wiki/Security_theater + +## <a id="startup"></a>How To Tell Fossil About Your Cert And Private Key + +As we saw [above](#usage), +if you do not have your own cert and private key, you can ask Fossil +to use "unsafe-builtin", which is a self-signed cert that is built into +Fossil. This is wildly insecure, since the private key is not really private; +it is [in plain sight][hcssc] in the Fossil +source tree for anybody to read. <b>Never add the private key that is +built into Fossil to your OS's trust store</b> as doing so will severely +compromise your computer.[^ssattack] This built-in cert is only useful for testing. +If you want actual security, you will need to come up with your own private +key and cert. + +Fossil wants to read certs and public keys in the +[PEM format](https://en.wikipedia.org/wiki/Privacy-Enhanced_Mail). +PEM is a pure ASCII text format. The private key consists of text +like this: + + -----BEGIN PRIVATE KEY----- + *base-64 encoding of the private key* + -----END PRIVATE KEY----- + +Similarly, a PEM-encoded cert will look like this: + + -----BEGIN CERTIFICATE----- + *base-64 encoding of the certificate* + -----END CERTIFICATE----- + +In both formats, text outside of the delimiters is ignored. That means +that if you have a PEM-formatted private key and a separate PEM-formatted +certificate, you can concatenate the two into a single file, and the +individual components will still be easily accessible. + +### <a id="cat"></a>Separate or Concatenated? + +Given a single concatenated file that holds both your private key and your +cert, you can hand it off to the "[fossil server](/help/server)" +command using the `--cert` option, like this: + + fossil server --port 443 --cert mycert.pem /home/www/myproject.fossil + +The command above is sufficient to run a fully-encrypted web site for +the "myproject.fossil" Fossil repository. This command must be run as +root, since it wants to listen on TCP port 443, and only root processes are +allowed to do that. This is safe, however, since before reading any +information off of the wire, Fossil will [put itself inside a chroot +jail](./chroot.md) at `/home/www` and drop all root privileges. + +This method of combining your cert and private key into a single big PEM +file carries risks, one of which is that the system administrator must +make both halves readable by the user running the Fossil server. Given +the chroot jail feature, a more secure scheme separates the halves so +that only root can read the private half, which then means that when +Fossil drops its root privileges, it becomes unable to access the +private key on disk. Fossil’s `server` feature includes the `--pkey` +option to allow for that use case: + + fossil server --port 443 --cert fullchain.pem --pkey privkey.pem /home/www/myproject.fossil + +[^ssattack]: ^How, you ask? Because the keys are known, they can be used + to provide signed certificates for **any** other domain. One foolish + enough to tell their OS’s TLS mechanisms to trust the signing + certificate is implicitly handing over all TLS encryption controls + to any attacker that knows they did this. Don’t do it. + +### <a id="chain"></a>Chains and Links + +The file name “`fullchain.pem`” used above is a reference to a term of +art within this world of TLS protocols and their associated X.509 +certificates. Within the simplistic scheme originally envisioned by the +creators of SSL — the predecessor to TLS — we were all expected to agree +on a single set of CA root authorities, and we would all agree to get +our certificates from one of them. The real world is more complicated: + +* The closest we have to universal acceptance of CAs is via the + [CA/Browser Forum][CAB], and even within its select membership there + is continual argument over which roots are trustworthy. (Hashing + that out is arguably this group’s key purpose.) + +* CAB’s decision regarding trustworthiness may not match that of any + given system’s administrator. There are solid, defensible reasons to + prune back the stock CA root set included with your browser, then to + augment it with ones CAB _doesn’t_ trust. + +* TLS isn’t limited to use between web browsers and public Internet + sites. Several common use cases preclude use of the process CAB + envisions, with servers able to contact Internet-based CA roots as + part of proving their identity. Different use cases demand different + CA root authority stores. + + The most common of these divergent cases are servers behind strict + firewalls and edge devices that never interact with the public + Internet. This class ranges from cheap home IoT devices to the + internal equipment managed by IT for a massive global corporation. + +Your private Fossil server is liable to fall into that last category. +This may then require that you generate a more complicated “chain” of +certificates for Fossil to use here, without which the client may not be +able to get back to a CA root it trusts. This is true regardless of +whether that client is another copy of Fossil or a web browser +traversing Fossil’s web UI, though that fact complicates matters by +allowing for multiple classes of client, each of which may have their +own rules for modifying the stock certificate scheme. + +This is distressingly common, in fact: Fossil links to OpenSSL to +provide its TLS support, but there is a good chance that your browser +uses another TLS implementation entirely. They may or may not agree on a +single CA root store. + +How you accommodate all this complexity varies by the CA and other +details. As but one example, Firefox’s “View Certificate” feature offers +_two_ ways to download a given web site’s certificate: the cert alone or +the “chain” leading back to the root. Depending on the use case, the +standalone certificate might suffice, or you might need some type of +cert chain. Complicating this is that the last link in the chain may be +left off when it is for a mutually trusted CA root, implicitly +completing the chain. + +[CAB]: https://en.wikipedia.org/wiki/CA/Browser_Forum + +## <a id="acme"></a>The ACME Protocol + +The [ACME Protocol][2] simplifies all this by automating the process of +proving to a recognized public CA that you are in control of a given +website. Without this proof, no valid CA will issue a cert for that +domain, as that allows fraudulent impersonation. + +The primary implementation of ACME is [certbot], a product of the Let’s +Encrypt organization. +Here is, in a nutshell, what certbot will do to obtain your cert: + + 1. It sends your "signing request" (the document that contains + your public key and your domain name) to the CA. + + 2. After receiving the signing request, the CA needs to verify that + you control the domain of the cert. One of several methods certbot has + for accomplishing this is to create a secret token and place it at + a well-known location, then tell the CA about it over ACME. + + 3. The CA then tries pulling that token, which if successful proves + that the requester is able to create arbitrary data on the server, + implicitly proving control over that server. This must be done + over the unencrypted HTTP protocol since TLS isn’t working yet. + + 4. If satisfied by this proof of control, the CA then creates the + keypair described above and bakes the public half into the + certificate it signs. It then sends this and the private half of + the key back to certbot. + + 5. Certbot stores these halves separately for the reasons sketched + out above. + + 6. It then deletes the secret one-time-use token it used to prove + domain control. ACME’s design precludes replay attacks. + +In order for all of this to happen, certbot needs to be able to create +a subdirectory named ".well-known", within a directory you specify, +then populate that subdirectory with a token file of some kind. To support +this, the "[fossil server](/help/server)" and +"[fossil http](/help/http)" commands have the --acme option. + +When specified, Fossil sees a URL where the path +begins with ".well-known", then instead of doing its normal processing, it +looks for a file with that pathname and returns it to the client. If +the "server" or "http" command is referencing a single Fossil repository, +then the ".well-known" sub-directory should be in the same directory as +the repository file. If the "server" or "http" command are run against +a directory full of Fossil repositories, then the ".well-known" sub-directory +should be in that top-level directory. + +Thus, to set up a project website, you should first run Fossil in ordinary +unencrypted HTTP mode like this: + + fossil server --port 80 --acme /home/www/myproject.fossil + +Then you create your public/private key pair and run certbot, giving it +a --webroot of /home/www. Certbot will create the sub-directory +named "/home/www/.well-known" and put token files there, which the CA +will verify. Then certbot will store your new cert in a particular file. + +Once certbot has obtained your cert, you may either pass the two halves +to Fossil separately using the `--pkey` and `--cert` options described +above, or you may concatenate them and pass that via `--cert` alone. + +[2]: https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment +[certbot]: https://certbot.eff.org diff --git a/www/ssl.wiki b/www/ssl.wiki new file mode 100644 index 0000000..f9d17cd --- /dev/null +++ b/www/ssl.wiki @@ -0,0 +1,328 @@ +<title>Securing a Repository with TLS</title> + +<h2>Using TLS-Encrypted Communications with Fossil</h2> + +If you are storing sensitive information in a repository accessible over +a network whose integrity you do not fully trust, you should use TLS to +encrypt all communications with it. This is most true for repositories +accessed over the Internet, especially if they will be accessed from +edge networks you do not control, since that admits of various forms of +[https://en.wikipedia.org/wiki/Man-in-the-middle_attack|man in the +middle attack]. + +TLS protects the credentials used to access the server, prevents +eavesdropping, prevents in-flight data modification, prevents server +identify spoofing, and more. + +There are two major aspects to this, both of which have to be addressed +in different ways. Those are the subjects of the next two major +sections. + + +<h2 id="client">Client-Side Configuration</h2> + +You can build Fossil against [https://www.openssl.org/ | +OpenSSL] to allow it to clone and sync with a remote +Fossil repository via <tt>https</tt> URIs. + + +<h3 id="openssl-bin">Building Against OpenSSL Automatically</h3> + +The <tt>configure</tt> script will attempt to find OpenSSL on your +system automatically. It first tries asking the <tt>pkg-config</tt> +system where the OpenSSL development files are, and if that fails, it +falls back to looking through a list of likely directories. + +If it can't find the files it needs, the most common solution is to +install the OpenSSL development package on your system via your OS's +package manager. Examples: + + * <b>RHEL & Fedora</b>: <tt>sudo dnf install openssl-devel</tt> + * <b>Debian & Ubuntu</b>: <tt>sudo apt install libssl-dev</tt> + * <b>FreeBSD</b>: <tt>su -c 'pkg install openssl'</tt> + * <b>macOS</b>: <tt>sudo brew install openssl</tt> + * <b>Cygwin</b>: Install <tt>openssl-devel</tt> via Cygwin's + <tt>setup-*.exe</tt> program + +The macOS case requires explanation. Apple last shipped OpenSSL +develpoment files in OS X 10.6 (Snow Leopard), choosing to deprecate it +from that point forward. (Apple wants you to use their proprietary +platform-specific encryption methods instead.) Since macOS has no +built-in package manager, a number have sprung up out of the FOSS world. +It is not known to this author whether Fossil's current build system can +find OpenSSL as installed with any of these other package managers, so +unless you have a particular reason to avoid it, we recomend that you +use [https://brew.sh|Homebrew] on macOS to install OpenSSL as above. +Fossil's build system will seek it out and use it automatically. + + +<h3 id="openssl-src">Building Against a Non-Platform Version of +OpenSSL</h3> + +The Fossil build system has a few other methods for finding OpenSSL when +the automatic methods fail or when you'd prefer that Fossil use a +different version of OpenSSL than the one Fossil's build system picks on +its own. + +A good reason to do this is when the Fossil build system finds a +functioning version of OpenSSL which is nevertheless unsuitable. One +common case is that your OS is sufficiently outdated that the platform +version of OpenSSL can no longer communicate with remote systems +adhering to the latest advice on secure communications. An old OpenSSL +might not support any of the +[https://en.wikipedia.org/wiki/Cipher_suite|cipher suites] the remote +Fossil repository's HTTPS proxy is willing to offer, for example, so +that even though both sides are speaking a variant of TLS/SSL, the peers +cannot come to an agreement on the cryptography. + +If you've installed the OpenSSL development files somewhere that +Fossil's build system cannot find on its own, you can clue it in by +passing the <tt>--with-openssl</tt> option to the <tt>configure</tt> +script. Type <tt>./configure --help</tt> for details. + +Another option is to download the source code to OpenSSL and build +Fossil against that private version of OpenSSL: + +<pre> +cd compat # relative to the Fossil source tree root +tar xf /path/to/openssl-*.tar.gz +ln -fs openssl-x.y.z openssl +cd openssl +./config # or, e.g. ./Configure darwin64-x86_64-cc +make -j11 +cd ../.. +./configure --with-openssl=tree +make -j11 +</pre> + +That will get you a Fossil binary statically linked to this in-tree +version of OpenSSL. + +Beware, taking this path typically opens you up to new problems, which +are conveniently covered in the next section! + + +<h3 id="certs">Certificates</h3> + +To verify the identify of a server, TLS uses +[https://en.wikipedia.org/wiki/X.509#Certificates|X.509 certificates], a +scheme that depends on a trust hierarchy of so-called +[https://en.wikipedia.org/wiki/Certificate_authority | Certificate +Authorities]. The tree of trust relationships ultimately ends in the +CA roots, which are considered the ultimate arbiters of who to trust in +this scheme. + +The question then is, what CA roots does Fossil trust? + +If you are using a self-signed certificate, Fossil will initially not +know that it can trust your certificate, so you'll be asked if you want +to accept the certificate the first time you communicate with the +server. Verify the certificate fingerprint is correct, then answer +"always" if you want Fossil to remember your decision. + +If you are cloning from or syncing to Fossil servers that use a +certificate signed by a well-known CA or one of its delegates, Fossil +still has to know which CA roots to trust. When this fails, you get an +error message that looks like this: + +<pre> +Unable to verify SSL cert from fossil-scm.org + subject: CN = sqlite.org + issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 + sha256: bf26092dd97df6e4f7bf1926072e7e8d200129e1ffb8ef5276c1e5dd9bc95d52 +accept this cert and continue (y/N)? +</pre> + +In older versions, the message was much longer and began with this line: + +<pre> +SSL verification failed: unable to get local issuer certificate +</pre> + +Fossil relies on the OpenSSL library to have some way to check a trusted +list of CA signing keys. There are two common ways this fails: + + # The OpenSSL library Fossil is linked to doesn't have a CA + signing key set at all, so that it initially trusts no certificates + at all. + + # The OpenSSL library does have a CA cert set, but your Fossil server's + TLS certificate was signed by a CA that isn't in that set. + +A common reason to fall into the second trap is that you're using +certificates signed by a local private CA, as often happens in large +enterprises. You can solve this sort of problem by getting your local +CA's signing certificate in PEM format and pointing OpenSSL at it: + +<pre> +fossil set --global ssl-ca-location /path/to/local-ca.pem +</pre> + +The use of <tt>--global</tt> with this option is common, since you may +have multiple repositories served under certificates signed by that same +CA. However, if you have a mix of publicly-signed and locally-signed +certificates, you might want to drop the <tt>--global</tt> flag and set +this option on a per-repository basis instead. + +A common way to run into the broader first problem is that you're on +FreeBSD, which does not install a CA certificate set by default, even as +a dependency of the OpenSSL library. If you're using a certificate +signed by one of the major public CAs, you can solve this by installing +the <tt>ca_root_nss</tt> package. That package contains the Mozilla NSS +certificate bundle, which gets installed in a location that OpenSSL +checks by default, so you don't need to change any Fossil settings. +(This is the same certificate set that ships with Firefox, by the way.) + +The same sort of thing happens with the Windows build of OpenSSL, but +for a different core reason: Windows does ship with a stock CA +certificate set, but it's not in a format that OpenSSL understands how +to use. Rather than try to find a way to convert the data format, you +may find it acceptable to use the same Mozilla NSS cert set. I do not +know of a way to easily get this from Mozilla themselves, but I did find +a [https://curl.se/docs/caextract.html | third party source] for the +<tt>cacert.pem</tt> file. I suggest placing the file into your Windows +user home directory so that you can then point Fossil at it like so: + +<pre> +fossil set --global ssl-ca-location %userprofile%\cacert.pem +</pre> + +This can also happen if you've linked Fossil to a version of OpenSSL +[#openssl-src|built from source]. That same <tt>cacert.pem</tt> fix can +work in that case, too. + +<blockquote> +OpenSSL 3.2.0 or greater is able to use the stock CA certificates +managed by Windows, and Fossil 2.25 (still in development as of +2024-07-15) takes advantage of this feature. This <em>possibly</em> +eliminates the need to manually install the Mozilla certificate package, +for example when connecting to Fossil servers secured by the widely-used +Let's Encrypt certificates. Run the following command to check if the +feature is supported: + +<pre> +fossil tls-config show -v +</pre> + +(See the "OpenSSL-winstore" section, requires Fossil 2.25 or greater.) +</blockquote> + +When you build Fossil on Linux platforms against the binary OpenSSL +package provided with the OS, you typically get a root cert store along +with the platform OpenSSL package, either built-in or as a hard +dependency. + + +<h4>Client-Side Certificates</h4> + +You can also use client side certificates to add an extra layer of +authentication, over and above Fossil's built in user management. If you +are particularly paranoid, you'll want to use this to remove the ability +of anyone on the internet from making any request to Fossil. Without +presenting a valid client side certificate, the web server won't invoke +the Fossil CGI handler. + +Configure your server to request a client side certificate, and set up a +certificate authority to sign your client certificates. For each person +who needs to access the repository, create a private key and certificate +signed with that CA. + +The PEM encoded private key and certificate should be stored in a single +file, simply by concatenating the key and certificate files. Specify the +location of this file with the <tt>ssl-identity</tt> setting, or the +<tt>--ssl-identity</tt> option to the <tt>clone</tt> command. + +If you've password protected the private key, the password will be +requested every time you connect to the server. This password is not +stored by fossil, as doing so would defeat the purpose of having a +password. + +If you attempt to connect to a server which requests a client +certificate, but don't provide one, fossil will show an error message +which explains what to do to authenticate with the server. + + +<h2 id="server">Server-Side Configuration</h2> + +Before Fossil's built-in HTTP server gained [./ssl-server.md | TLS support], +system administrators that wanted to add this +had to put it behind a reverse proxy that would do the translation. +Since advantages remain for delegating TLS to another layer in the +stack, instructions for doing so continue to be included in our +documentation, such as: + + * <a id="stunnel" href="./server/any/stunnel.md">Serving via stunnel</a> + * <a id="althttpd" href="./server/any/althttpd.md">Serving via stunnel + althttpd</a> + * <a id="nginx" href="./server/debian/nginx.md#tls">Serving via SCGI with nginx on Debian</a> + + +<h2 id="enforcing">Enforcing TLS Access</h2> + +To use TLS encryption in cloning and syncing to a remote Fossil +repository, be sure to use the <tt>https:</tt> URI scheme in +<tt>clone</tt> and <tt>sync</tt> commands. If your server is configured +to serve the repository via both HTTP and HTTPS, it's easy to +accidentally use unencrypted HTTP if you forget the all-important 's'. + +As of Fossil 2.9, using an <tt>http://</tt> URI with <tt>fossil +clone</tt> or <tt>sync</tt> on a site that forwards to HTTPS will cause +Fossil to remember the secure URL. However, there's a +[https://en.wikipedia.org/wiki/Trust_on_first_use | TOFU problem] with +this: it's still better to use <tt>https://</tt> from the start. + +As of Fossil 2.8, there is a setting in the Fossil UI under Admin → +Access called "Redirect to HTTPS," which is set to "Off" by default. +Changing this only affects web UI access to the Fossil repository. It +doesn't affect clones and syncs done via the <tt>http</tt> URI scheme. + +In Fossil 2.7 and earlier, there was a much weaker form of this setting +affecting the <tt>/login</tt> page only. If you're using this setting, +you should migrate to the new setting as soon as possible, because the +old setting allows multiple ways of defeating it. + +<b id="rloop">WARNING:</b> Enabling HTTPS redirects at the Fossil repo +level while running Fossil behind an HTTPS proxy can result in an +infinite redirect loop. It happens when the proxy mechanism presents +"<tt>http</tt>" URIs to Fossil, so Fossil issues a redirect, so the browser +fetches the page again, causing Fossil to see an "<tt>http</tt>" URI again, so +it issues a redirect...'round and 'round it goes until the web browser +detects it's in a redirect loop and gives up. This problem prevents you +from getting back into the Admin UI to fix it, but there are several +ways to fix it: + + # <b>Reset via CLI.</b> You can turn the setting back off from the + CLI with the command "<tt>fossil -R /path/to/repo.fossil set + redirect-to-https 0</tt>". (Currently doesn't work.) + + # <b>Backup first.</b> This setting is stored in the Fossil + repository, so if you make a backup first <i>on the server</i>, you + can restore the repo file if enabling this feature creates a + redirect loop. + + # <b>Download, fix, and restore.</b> You can copy the remote + repository file down to a local machine, use <tt>fossil ui</tt> to + fix the setting, and then upload it to the repository server + again. + +It's best to enforce TLS-only access at the front-end proxy level +anyway. It not only avoids the problem entirely, it can be significantly +more secure. The [./server/debian/nginx.md#tls | nginx-on-Debian proxy guide] shows one way +to achieve this. + + +<h2>Terminology Note</h2> + +This document is called <tt>ssl.wiki</tt> for historical reasons. The +TLS protocol was originally called SSL, and it went through several +revisions before being replaced by TLS. Years before this writing, SSL +finally became entirely obsolete due to weaknesses in the protocol fixed +in the later TLS series of protocols. + +Some people still use the term "SSL" when they actually mean "TLS," but +in the Fossil project, we always use "TLS" except when we must preserve +some sort of historical compatibility, as with this document's name in +order to avoid broken external URLs. The Fossil TLS-related settings +also often use "<tt>ssl</tt>" in their names for the same reason. + +This series of protocols is also called "HTTPS" after the URI scheme +used to specify "HTTP over TLS." diff --git a/www/stats.wiki b/www/stats.wiki new file mode 100644 index 0000000..285bb41 --- /dev/null +++ b/www/stats.wiki @@ -0,0 +1,149 @@ +<title>Fossil Performance</title> + +The questions will inevitably arise: How does Fossil perform? +Does it use a lot of disk space or bandwidth? Is it scalable? + +In an attempt to answer these questions, this report looks at several +projects that use fossil for configuration management and examines how +well they are working. The following table is a summary of the results. +(Last updated on 2018-06-04.) +Explanation and analysis follows the table. + +<table> +<tr> +<th>Project</th> +<th>Number Of Artifacts</th> +<th>Number Of Check-ins</th> +<th>Project Duration<br>(as of 2018-06-04)</th> +<th>Uncompressed Size</th> +<th>Repository Size</th> +<th>Compression Ratio</th> +<th>Clone Bandwidth</th> +</tr> + +<tr align="center"> +<td>[http://www.sqlite.org/src/timeline | SQLite] +<td>77492 +<td>20686 +<td>6580 days<br>18.02 years +<td>5.6 GB +<td>70.0 MB +<td>80:1 +<td>51.1 MB +</tr> + +<tr align="center"> +<td>[http://core.tcl.tk/tcl/timeline | TCL] +<td>161991 +<td>23146 +<td>7375 days<br>20.19 years +<td>8.0 GB +<td>222.0 MB +<td>36:1 +<td>150.5 MB +</tr> + +<tr align="center"> +<td>[/timeline | Fossil] +<td>39148 +<td>11266 +<td>3971 days<br>10.87 years +<td>3.8 GB +<td>42.0 MB +<td>90:1 +<td>27.4 MB +</tr> + +<tr align="center"> +<td>[http://www.sqlite.org/slt/timeline | SLT] +<td>2384 +<td>169 +<td>3474 days<br>9.51 years +<td>2.1 GB +<td>145.9 MB +<td>14:1 +<td>143.4 MB +</tr> + +<tr align="center"> +<td>[http://www.sqlite.org/th3.html | TH3] +<td>12406 +<td>3718 +<td>3539 days<br>9.69 years +<td>544 MB +<td>18.0 MB +<td>30:1 +<td>14.7 MB +</tr> + +<tr align="center"> +<td>[http://www.sqlite.org/docsrc/timeline | SQLite Docs] +<td>8752 +<td>2783 +<td>3857 days<br>10.56 years +<td>349.9 MB +<td>16.3 MB +<td>21:1 +<td>13.57 MB +</tr> + +</table> + +<h2>Measured Attributes</h2> + +In Fossil, every version of every file, every wiki page, every change to +every ticket, and every check-in is a separate "artifact". One way to +think of a Fossil project is as a bag of artifacts. Of course, there is +a lot more than this going on in Fossil. Many of the artifacts have meaning +and are related to other artifacts. But at a low level (for example when +synchronizing two instances of the same project) the only thing that matters +is the unordered collection of artifacts. In fact, one of the key +characteristics of Fossil is that the entire project history can be +reconstructed simply by scanning the artifacts in an arbitrary order. + +The number of check-ins is the number of times that the "commit" command +has been run. A single check-in might change 3 or 4 files, or it might +change dozens or hundreds of files. Regardless of the number of files +changed, it still only counts as one check-in. + +The "Uncompressed Size" is the total size of all the artifacts within +the repository assuming they were all uncompressed and stored +separately on the disk. Fossil makes use of delta compression between related +versions of the same file, and then uses zlib compression on the resulting +deltas. The total resulting repository size is shown after the uncompressed +size. + +On the right end of the table, we show the "Clone Bandwidth". This is the +total number of bytes sent from server back to the client. The number of +bytes sent from client to server is negligible in comparison. +These byte counts include HTTP protocol overhead. + +In the table and throughout this article, +"GB" means gigabytes (10<sup><small>9</small></sup> bytes) +not <a href="http://en.wikipedia.org/wiki/Gibibyte">gibibytes</a> +(2<sup><small>30</small></sup> bytes). Similarly, "MB" and "KB" +means megabytes and kilobytes, not mebibytes and kibibytes. + +<h2>Analysis And Supplemental Data</h2> + +Perhaps the two most interesting data points in the above table are SQLite +and SLT. SQLite is a long-running project with long revision chains. +Some of the files in SQLite have been edited over a thousand times. +Each of these edits is stored as a delta, and hence the SQLite project +gets excellent 80:1 compression. SLT, on the other hand, consists of +many large (megabyte-sized) SQL scripts that have one or maybe two +edits each. There is very little delta compression occurring and so the +overall repository compression ratio is much lower. Note also that +quite a bit more bandwidth is required to clone SLT than SQLite. + +For the first nine years of its development, SQLite was versioned by CVS. +The resulting CVS repository measured over 320MB in size. So, the +developers were surprised to see that the equivalent Fossil project (the +first nine years on SQLite) would clone with only 13MB of bandwidth. +The "sync" protocol +used by fossil has turned out to be surprisingly efficient. A typical +check-in on SQLite might use 3 or 4KB of network bandwidth. +For example, the [04eef9522386a59e] check-in used a single HTTP request +of 2099 bytes and got back a reply of 1116 bytes. +The sync protocol is efficient enough that, once cloned, +Fossil can easily be used over a dial-up connection. diff --git a/www/style.wiki b/www/style.wiki new file mode 100644 index 0000000..10b009e --- /dev/null +++ b/www/style.wiki @@ -0,0 +1,159 @@ +<title>Coding Style</title> + +Fossil source code should follow the style guidelines below. + +<em> The Fossil source tree includes a few files taken from external +sources +(examples: [https://github.com/antirez/linenoise|linenoise] and +[http://zlib.net/|zLib]) +and this externally sourced code might not comply with these style guidelines. +</em> + +<b>1. General points</b>: + +<ol> + <li value=10> No line of code exceeds 80 characters in length. (Occasional + exceptions are made for HTML text on @-lines.) + + <li> There are no tab characters. + + <li> Line terminators are \n only. Do not use a \r\n line terminator. + + <li> 2-space indentation is used. Remember: No tabs. + + <li> Comments contain no spelling or grammatical errors. (Abbreviations + and sentence fragments are acceptable when trying to fit a comment + on a single line as long as the meaning is clear.) + + <li> The tone of comments is professional and courteous. Comments + contain no profanity, obscenity, or innuendo. + + <li> All C-code conforms to ANSI C-89. + Three well-defined existing exceptions are: + <ol type="a"> + + <li> -Wno-overlength-strings: The Fossil build system converts (some of the) source code comments + into strings, which may exceed the 509 character limit defined by ANSI. + (example: bld/page_index.h) + + <li> -Wno-long-long: Fossil uses the 'long long' integer type, which is not strictly ANSI C-89 (defined in C99). + The use of 'long long' resolves many problems with 64-bit arithmetics, especially on 32-bit machines. + (http_ssl.c, sha3.c, shell.c, util.c) + + <li> alloca(): By default, sqlite3.c was compiled with the -DSQLITE_USE_ALLOCA flag to use the alloca() function. + This is no longer the case as of 20220119. alloca() is not considered ANSI C, and normally not + recommended due to portability issues, but performance and/or memory consumption + improvement may have been a stronger argument in favor of its usage. + (sqlite3.c) + </ol> + + <li> All comments and identifiers are in English. + + <li> The program is single-threaded. Do not use threads. + (One exception to this is the HTTP server implementation for Windows, + which we do not know how to implement without the use of threads.) + +</ol> + +<b>2. C preprocessor macros</b>: + +<ol> + + <li value=20> The purpose of every preprocessor macros is clearly explained in a + comment associated with its definition. + + <li> Every preprocessor macro is used at least once. + + <li> The names of preprocessor macros clearly reflect their use. + + <li> Assumptions about the relative values of related macros are + verified by asserts. Example: <tt>assert(READ_LOCK+1==WRITE_LOCK);</tt> + +</ol> + + +<b>3. Function header comments</b>: + +<ol> + <li value=30> Every function has a header comment describing the purpose and use + of the function. + + <li> A function header comment defines the behavior of the function in + sufficient detail to allow the function to be re-implemented from + scratch without reference to the original code. + + <li> Functions that perform dynamic memory allocation (either directly + or indirectly via subfunctions) say so in their header comments. + +</ol> + + +<b>4. Function bodies</b>: + +<ol> + <li value=40> The name of a function clearly reflects its purpose. + + <li> Automatic variables are small, not large objects or arrays. Avoid + excessive stack usage. + + <li> The check-list items for functions also apply to major subsections + within a function. + + <li> All code subblocks are enclosed in {...}. + + + <li> <b>assert() macros are used as follows</b>: + <ol type="a"> + + <li> Function preconditions are clearly stated and verified by asserts. + + <li> Invariants are identified by asserts. + </ol> + +</ol> + + +<b>5. Class (struct) declarations</b>: + +<ol> + <li value=50> The purpose and use of every class (a.k.a. structure) is clearly defined + in the header comment of its declaration. + + <li> The purpose and use of every class member is clearly defined either + in the header comment of the class declaration or when the member is + declared or both. + + <li> The names of class members clearly reflect their use. + + <li> Invariants for classes are clearly defined. + +</ol> + +<b>6. Variables and class instances</b>: + +<ol> + <li value=60> The purpose and use of every variable is defined by a comment at the + variable definition. + + <li> The names of variables clearly reflect their use. + + <li> Related variables have related names. (ex: aSavepoint and nSavepoint.) + + <li> Variables have minimum practical scope. + + <li> Automatic variables are small, not large objects or arrays. + + <li> Constants are "const". + + <li> Invariants on variables or groups of variables are defined and + tested by asserts. + + <li> When a variable that refers to the same value is used within + multiple scopes, the same name is used in all cases. + + <li> When variables refer to different values, different names are used + even when the names are in different scopes. + + <li> Variable names with wide scope are sufficiently distinctive to allow + searching for them using grep. +</ol> diff --git a/www/sync.wiki b/www/sync.wiki new file mode 100644 index 0000000..1518321 --- /dev/null +++ b/www/sync.wiki @@ -0,0 +1,1120 @@ +<title>The Fossil Sync Protocol</title> + +This document describes the wire protocol used to synchronize +content between two Fossil repositories. + +<h2>1.0 Overview</h2> + +The global state of a fossil repository consists of an unordered +[./fileformat.wiki|collection of artifacts]. Each artifact is +identified by a cryptographic +hash of its content, expressed as a lower-case hexadecimal string. +Synchronization is the process of sharing artifacts between +repositories so that all repositories have copies of all artifacts. Because +artifacts are unordered, the order in which artifacts are received +is unimportant. It is assumed that the hash names +of artifacts are unique - that every artifact has a different hash. +To a first approximation, synchronization proceeds by sharing lists +of hashes for available artifacts, then sharing the content of artifacts +whose names are missing from one side or the other of the connection. +In practice, a repository might contain millions of artifacts. The list of +hash names for this many artifacts can be large. So optimizations are +employed that usually reduce the number of hashes that need to be +shared to a few dozen. + +Each repository also has local state. The local state determines +the web-page formatting preferences, authorized users, ticket formats, +and similar information that varies from one repository to another. +The local state is not usually transferred during a sync. Except, +some local state is transferred during a [/help/clone|clone] +in order to initialize the local state of the new repository. Also, +an administrator can sync local state using +the [/help/configuration|config push] and +[/help/configuration|config pull] +commands. + +<h3 id="crdt">1.1 Conflict-Free Replicated Datatypes</h3> + +The "bag of artifacts" data model used by Fossil is apparently an +implementation of a particular +[https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type|Conflict-Free +Replicated Datatype (CRDT)] called a "G-Set" or "Grow-only Set". The +academic literature on CRDTs only began to appear in about 2011, and +Fossil predates that research by at least 4 years. But it is nice to +know that theorists have now proven that the underlying data model of +Fossil can provide strongly-consistent replicas using only +peer-to-peer communication and without any kind of central +authority. + +If you are already familiar with CRDTs and were wondering if Fossil +used them, the answer is "yes". We just don't call them by that name. + + +<h2 id="transport">2.0 Transport</h2> + +All communication between client and server is via HTTP requests. +The server is listening for incoming HTTP requests. The client +issues one or more HTTP requests and receives replies for each +request. + +The server might be running as an independent server +using the [/help/server|"fossil server" command], or it +might be launched from inetd or xinetd using the +[/help/http|"fossil http" command]. Or the server might +be [./server/any/cgi.md|launched from CGI] or from +[./server/any/scgi.md|SCGI]. +(See "[./server/|How To Configure A Fossil Server]" for details.) +The specifics of how the server listens +for incoming HTTP requests is immaterial to this protocol. +The important point is that the server is listening for requests and +the client is the issuer of the requests. + +A single [/help/push|push], +[/help/pull|pull], or [/help?cmd=sync|sync] +might involve multiple HTTP requests. +The client maintains state between all requests. But on the server +side, each request is independent. The server does not preserve +any information about the client from one request to the next. + +Note: Throughout this article, we use the terms "server" and "client" +to represent the listener and initiator of the interaction, respectively. +Nothing in this protocol requires that the server actually be a back-room +processor housed in a datacenter, nor does the client need to be a desktop +or handheld device. For the purposes of this article "client" simply means +the repository that initiates the conversation and "server" is the repository +that responds. Nothing more. + +<h4 id="https">2.0.1 HTTPS Transport</h4> + +HTTPS differs from HTTP only in that the HTTPS protocol is +encrypted as it travels over the wire. The underlying protocol +is the same. This document describes only the underlying, unencrypted +messages that go client to server and back to client. +Whether or not those messages are encrypted does not come into play +in this document. + +Fossil includes built-in +[./ssl-server.md|support for HTTPS encryption] in both client and server. + +<h4 id="ssh">2.0.2 SSH Transport</h4> + +When doing a sync using an "<code>ssh:…</code>" URL, the same HTTP transport protocol +is used. Fossil simply uses [https://en.wikipedia.org/wiki/Secure_Shell|ssh] +to start an instance of the [/help?cmd=test-http|fossil test-http] command +running on the remote machine. It then sends HTTP requests and gets back HTTP +replies over the SSH connection, rather than sending and receiving over an +internet socket. To see the specific "ssh" command that the Fossil client +runs in order to set up a connection, add either of the the "--httptrace" or +"--sshtrace" options to the "fossil sync" command line. + +This method is dependent on the remote <var>PATH</var> set by the SSH +daemon, which may not be the same as your interactive shell's +<var>PATH</var> on that same server. It is common to find +<var>$HOME/bin</var> in the latter but not the former, for instance, +leading to failures to sync over <code>ssh:…</code> URLs when you +install the <code>fossil</code> binary in a nonstandard location, as +with + +<verbatim>./configure --prefix=$HOME && make install</verbatim> + +The simpler of the two solutions to this problem is to install Fossil +where sshd expects to find it, but when that isn't an option, you can +instead give a URL like this: + +<verbatim>fossil clone ssh://myserver.example.com/path/to/repo.fossil?fossil=/home/me/bin/fossil</verbatim> + +That gives the local Fossil instance the absolute path to the binary on +the remote machine for use when calling that Fossil instance through the +SSH tunnel. + +<h4 id="file">2.0.3 FILE Transport</h4> + +When doing a sync using a "file:..." URL, the same HTTP protocol is +still used. But instead of sending each HTTP request over a socket or +via SSH, the HTTP request is written into a temporary file. The client +then invokes the [/help?cmd=http|fossil http] command in a subprocess +to process the request and and generate a reply. The client then reads +the HTTP reply out of a temporary file on disk, and deletes the two +temporary files. To see the specific "fossil http" command that is run +in order to implement the "file:" transport, add the "--httptrace" +option to the "fossil sync" command. + +<h3 id="srv-id">2.1 Server Identification</h3> + +The server is identified by a URL argument that accompanies the +push, pull, or sync command on the client. (As a convenience to +users, the URL can be omitted on the client command and the same URL +from the most recent push, pull, or sync will be reused. This saves +typing in the common case where the client does multiple syncs to +the same server.) + +The client modifies the URL by appending the method name "<b>/xfer</b>" +to the end. For example, if the URL specified on the client command +line is + +<pre>https://fossil-scm.org/home</pre> + +Then the URL that is really used to do the synchronization will +be: + +<pre>https://fossil-scm.org/home/xfer</pre> + +<h3 id="req-format">2.2 HTTP Request Format</h3> + +The client always sends a POST request to the server. The +general format of the POST request is as follows: + +<pre> +POST /home/xfer HTTP/1.0 +Host: fossil-scm.hwaci.com:80 +Content-Type: application/x-fossil +Content-Length: 4216 +</pre> + +<i><pre>content...</pre></i> + +In the example above, the pathname given after the POST keyword +on the first line is a copy of the URL pathname. The Host: parameter +is also taken from the URL. The content type is always either +"application/x-fossil" or "application/x-fossil-debug". The "x-fossil" +content type is the default. The only difference is that "x-fossil" +content is compressed using zlib whereas "x-fossil-debug" is sent +uncompressed. + +A typical reply from the server might look something like this: + +<pre> +HTTP/1.0 200 OK +Date: Mon, 10 Sep 2007 12:21:01 GMT +Connection: close +Cache-control: private +Content-Type: application/x-fossil; charset=US-ASCII +Content-Length: 265 +</pre> + +<i><pre>content...</pre></i> + +The content type of the reply is always the same as the content type +of the request. + +<h2 id="content">3.0 Fossil Synchronization Content</h2> + +A synchronization request between a client and server consists of +one or more HTTP requests as described in the previous section. This +section details the "x-fossil" content type. + +<h3 id="lines">3.1 Line-oriented Format</h3> + +The x-fossil content type consists of zero or more "cards". Cards +are separated by the newline character ("\n"). Leading and trailing +whitespace on a card is ignored. Blank cards are ignored. + +Each card is divided into zero or more space separated tokens. +The first token on each card is the operator. Subsequent tokens +are arguments. The set of operators understood by servers is slightly +different from the operators understood by clients, though the two +are very similar. + +<h3 id="login">3.2 Login Cards</h3> + +Every message from client to server begins with one or more login +cards. Each login card has the following format: + +<pre><b>login</b> <i>userid nonce signature</i></pre> + +The userid is the name of the user that is requesting service from the +server, encoded in "fossilized" form (exactly as described for <a +href="#error">the error card</a>). The nonce is the SHA1 hash of the +remainder of the message - all text that follows the newline character +that terminates the login card. The signature is the SHA1 hash of the +concatenation of the nonce and the users password. + +When receving a login card, the server looks up the user and verifies +that the nonce matches the SHA1 hash of the remainder of the message. +It then checks the signature hash to make sure the signature matches. +If everything checks out, then the client is granted all privileges of +the specified user. + +Only one login card is permitted. A second login card will trigger +a sync error. (Prior to 2025-07-21, the protocol permitted multiple +logins, treating the login as the union of all privileges from all +login cards. That capability was never used and has been removed.) + +As of version 2.27, Fossil supports transfering of the login card +externally to the request payload via a Cookie HTTP header: + +<verbatim> + Cookie: x-f-l-c=... +</verbatim> + +Where "..." is the URL-encoded login cookie. <code>x-f-l-c</code> is +short for X-Fossil-Login-Cookie. + + +<h3 id="file">3.3 File Cards</h3> + +Artifacts are transferred using either "file" cards, or "cfile" +or "uvfile" cards. +The name "file" card comes from the fact that most artifacts correspond to +files that are under version control. +The "cfile" name is an abbreviation for "compressed file". +The "uvfile" name is an abbreviation for "unversioned file". + +<h4 id="ordinary-fc">3.3.1 Ordinary File Cards</h4> + +For sync protocols, artifacts are transferred using "file" +cards. File cards come in two different formats depending +on whether the artifact is sent directly or as a +[./delta_format.wiki|delta] from some +other artifact. + +<pre> +<b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i> +<b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> +</pre> + +File cards are followed by in-line "payload" data. +The content of the artifact +or the artifact delta is the first <i>size</i> bytes of the +x-fossil content that immediately follow the newline that +terminates the file card. + + +The first argument of a file card is the ID of the artifact that +is being transferred. The artifact ID is the lower-case hexadecimal +representation of the name hash for the artifact. +The last argument of the file card is the number of bytes of +payload that immediately follow the file card. If the file +card has only two arguments, that means the payload is the +complete content of the artifact. If the file card has three +arguments, then the payload is a +[./delta_format.wiki|delta] and the second argument is +the ID of another artifact that is the source of the delta. + +File cards are sent in both directions: client to server and +server to client. A delta might be sent before the source of +the delta, so both client and server should remember deltas +and be able to apply them when their source arrives. + +<h4 id="compressed-fc">3.3.2 Compressed File Cards</h4> + +A client that sends a clone protocol version "3" or greater will +receive artifacts as "cfile" cards while cloning. This card was +introduced to improve the speed of the transfer of content by sending +the compressed artifact directly from the server database to the +client. In this case, the containing response body is <em>not</em> +compressed separately because the vast majority of the response is +already compressed in cfile cards. In practice, version "3" is +significantly faster than version "2". + +Compressed File cards are similar to File cards, sharing the same +in-line "payload" data characteristics and also the same treatment of +direct content or delta content. Cfile cards come in two different formats +depending on whether the artifact is sent directly or as a delta from +some other artifact. + +<pre> +<b>cfile</b> <i>artifact-id usize csize</i> <b>\n</b> <i>content</i> +<b>cfile</b> <i>artifact-id delta-artifact-id usize csize</i> <b>\n</b> <i>content</i> +</pre> + +The first argument of the cfile card is the ID of the artifact that +is being transferred. The artifact ID is the lower-case hexadecimal +representation of the name hash for the artifact. The second argument of +the cfile card is the original size in bytes of the artifact. The last +argument of the cfile card is the number of compressed bytes of payload +that immediately follow the cfile card. If the cfile card has only +three arguments, that means the payload is the complete content of the +artifact. If the cfile card has four arguments, then the payload is a +delta and the second argument is the ID of another artifact that is the +source of the delta and the third argument is the original size of the +delta artifact. + +Unlike file cards, cfile cards are only sent in one direction during a +clone from server to client for clone protocol version "3" or greater. + +<h4 id="private">3.3.3 Private artifacts</h4> + +"Private" content consist of artifacts that are not normally synced. +However, private content will be synced when the +the [/help?cmd=sync|fossil sync] command includes the "--private" option. + +Private content is marked by a "private" card: + +<pre><b>private</b></pre> + +The private card has no arguments and must directly precede a +file card that contains the private content. + +<h4 id="uv-fc">3.3.4 Unversioned File Cards</h4> + +Unversioned content is sent in both directions (client to server and +server to client) using "uvfile" cards in the following format: + +<pre><b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i></pre> + +The <i>name</i> field is the name of the unversioned file. The +<i>mtime</i> is the last modification time of the file in seconds +since 1970. The <i>hash</i> field is the hash of the content +for the unversioned file, or "<b>-</b>" for deleted content. +The <i>size</i> field is the (uncompressed) size of the content +in bytes. The <i>flags</i> field is an integer which is interpreted +as an array of bits. The 0x0004 bit of <i>flags</i> indicates that +the <i>content</i> is to be omitted. The content might be omitted if +it is too large to transmit, or if the sender merely wants to update the +modification time of the file without changing the files content. +The <i>content</i> is the (uncompressed) content of the file. + +The receiver should only accept the uvfile card if the hash and +size match the content and if the mtime is newer than any existing +instance of the same file held by the receiver. The sender will not +normally transmit a uvfile card unless all these constraints are true, +but the receiver should double-check. + +A server will only accept uvfile cards if the login user has +the "y" write-unversioned permission. + +Servers send uvfile cards in response to uvgimme cards received from +the client. Clients send uvfile cards when they determine that the server +needs the content based on uvigot cards previously received from the server. + +<h3 id="push" name="pull">3.4 Push and Pull Cards</h3> + +Among the first cards in a client-to-server message are +the push and pull cards. The push card tells the server that +the client is pushing content. The pull card tells the server +that the client wants to pull content. In the event of a sync, +both cards are sent. The format is as follows: + +<pre> +<b>push</b> <i>servercode projectcode</i> +<b>pull</b> <i>servercode projectcode</i> +</pre> + +The <i>servercode</i> argument is the repository ID for the +client. The <i>projectcode</i> is the identifier +of the software project that the client repository contains. +The projectcode for the client and server must match in order +for the transaction to proceed. + +The server will also send a push card back to the client +during a clone. This is how the client determines what project +code to put in the new repository it is constructing. + +The <i>servercode</i> argument is currently unused. + +<h3 id="clones">3.5 Clone Cards</h3> + +A clone card works like a pull card in that it is sent from +client to server in order to tell the server that the client +wants to pull content. The clone card comes in two formats. Older +clients use the no-argument format and newer clients use the +two-argument format. + +<pre> +<b>clone</b> +<b>clone</b> <i>protocol-version sequence-number</i> +</pre> + +<h4>3.5.1 Protocol 3</h4> + +The latest clients send a two-argument clone message with a +protocol version of "3". (Future versions of Fossil might use larger +protocol version numbers.) Version "3" of the protocol enhanced version +"2" by introducing the "cfile" card which is intended to speed up clone +operations. Instead of sending "file" cards, the server will send "cfile" +cards + +<h4>3.5.2 Protocol 2</h4> + +The sequence-number sent is the number +of artifacts received so far. For the first clone message, the +sequence number is 1. The server will respond by sending file +cards for some number of artifacts up to the maximum message size. + +The server will also send a single "clone_seqno" card to the client +so that the client can know where the server left off. + +<pre> +<b>clone_seqno</b> <i>sequence-number</i> +</pre> + +The clone message in subsequent HTTP requests for the same clone +operation will use the sequence-number from the +clone_seqno of the previous reply. + +In response to an initial clone message, the server also sends the client +a push message so that the client can discover the projectcode for +this project. + +<h4>3.5.3 Legacy Protocol</h4> + +Older clients send a clone card with no argument. The server responds +to a blank clone card by sending an "igot" card for every artifact in the +repository. The client will then issue "gimme" cards to pull down all the +content it needs. + +The legacy protocol works well for smaller repositories (50MB with 50,000 +artifacts) but is too slow and unwieldy for larger repositories. +The version 2 protocol is an effort to improve performance. Further +performance improvements with higher-numbered clone protocols are +possible in future versions of Fossil. + +<h3 id="igot">3.6 Igot Cards</h3> + +An igot card can be sent from either client to server or from +server to client in order to indicate that the sender holds a copy +of a particular artifact. The format is: + +<pre> +<b>igot</b> <i>artifact-id</i> ?<i>flag</i>? +</pre> + +The first argument of the igot card is the ID of the artifact that +the sender possesses. +The receiver of an igot card will typically check to see if +it also holds the same artifact and if not it will request the artifact +using a gimme card in either the reply or in the next message. + +If the second argument exists and is "1", then the artifact +identified by the first argument is private on the sender and should +be ignored unless a "--private" [/help?cmd=sync|sync] is occurring. + +The name "igot" comes from the English slang expression "I got" meaning +"I have". + +<h4>3.6.1 Unversioned Igot Cards</h4> + +Zero or more "uvigot" cards are sent from server to client when +synchronizing unversioned content. The format of a uvigot card is +as follows: + +<pre> +<b>uvigot</b> <i>name mtime hash size</i> +</pre> + +The <i>name</i> argument is the name of an unversioned file. +The <i>mtime</i> is the last modification time of the unversioned file +in seconds since 1970. +The <i>hash</i> is the SHA1 or SHA3-256 hash of the unversioned file +content, or "<b>-</b>" if the file has been deleted. +The <i>size</i> is the uncompressed size of the file in bytes. + +When the server sees a "pragma uv-hash" card for which the hash +does not match, it sends uvigot cards for every unversioned file that it +holds. The client will use this information to figure out which +unversioned files need to be synchronized. +The server might also send a uvigot card when it receives a uvgimme card +but its reply message size is already oversized and hence unable to hold +the usual uvfile reply. + +When a client receives a "uvigot" card, it checks to see if the +file needs to be transferred from client to server or from server to client. +If a client-to-server transmission is needed, the client schedules that +transfer to occur on a subsequent HTTP request. If a server-to-client +transfer is needed, then the client sends a "uvgimme" card back to the +server to request the file content. + +<h3 id="gimme">3.7 Gimme Cards</h3> + +A gimme card is sent from either client to server or from server +to client. The gimme card asks the receiver to send a particular +artifact back to the sender. The format of a gimme card is this: + +<pre> +<b>gimme</b> <i>artifact-id</i> +</pre> + +The argument to the gimme card is the ID of the artifact that +the sender wants. The receiver will typically respond to a +gimme card by sending a file card in its reply or in the next +message. + +The "gimme" name means "give me". The imperative "give me" is +pronounced as if it were a single word "gimme" in some dialects of +English (including the dialect spoken by the original author of Fossil). + +<h4>3.7.1 Unversioned Gimme Cards</h4> + +When synchronizing unversioned content, the client may send "uvgimme" +cards to the server. A uvgimme card requests that the server send +unversioned content to the client. The format of a uvgimme card is +as follows: + +<pre> +<b>uvgimme</b> <i>name</i> +</pre> + +The <i>name</i> is the name of the unversioned file found on the +server that the client would like to have. When a server sees a +uvgimme card, it normally responses with a uvfile card, though it might +also send another uvigot card if the HTTP reply is already oversized. + +<h3 id="cookie">3.8 Cookie Cards</h3> + +A cookie card can be used by a server to record a small amount +of state information on a client. The server sends a cookie to the +client. The client sends the same cookie back to the server on +its next request. The cookie card has a single argument which +is its payload. + +<pre> +<b>cookie</b> <i>payload</i> +</pre> + +The client is not required to return the cookie to the server on +its next request. Or the client might send a cookie from a different +server on the next request. So the server must not depend on the +cookie and the server must structure the cookie payload in such +a way that it can tell if the cookie it sees is its own cookie or +a cookie from another server. (Typically the server will embed +its servercode as part of the cookie.) + +<h3 id="reqconfig">3.9 Request-Configuration Cards</h3> + +A request-configuration or "reqconfig" card is sent from client to +server in order to request that the server send back "configuration" +data. "Configuration" data is information about users or website +appearance or other administrative details which are not part of the +persistent and versioned state of the project. For example, the "name" +of the project, the default Cascading Style Sheet (CSS) for the web-interface, +and the project logo displayed on the web-interface are all configuration +data elements. + +The reqconfig card is normally sent in response to the +"fossil configuration pull" command. The format is as follows: + +<pre> +<b>reqconfig</b> <i>configuration-name</i> +</pre> + +As of 2024-10-22, the configuration-name must be one of the +following values: + +<table border=0 align="center"> +<tr><td valign="top"> +<ul> +<li> css +<li> header +<li> mainmenu +<li> footer +<li> details +<li> js +<li> default-skin +<li> logo-mimetype +<li> logo-image +<li> background-mimetype +<li> background-image +<li> icon-mimetype +<li> icon-image +<li> timeline-block-markup +<li> timeline-date-format +<li> timeline-default-style +<ul></td><td valign="top"><ul> +<li> timeline-dwelltime +<li> timeline-closetime +<li> timeline-hard-newlines +<li> timeline-max-comment +<li> timeline-plaintext +<li> timeline-truncate-at-blank +<li> timeline-tslink-info +<li> timeline-utc +<li> adunit +<li> adunit-omit-if-admin +<li> adunit-omit-if-user +<li> default-csp +<li> sitemap-extra +<li> safe-html +<li> th1-hooks +<li> th1-uri-regexp +<li> project-name +<ul></td><td valign="top"><ul> +<li> short-project-name +<li> project-description +<li> index-page +<li> manifest +<li> binary-glob +<li> clean-glob +<li> ignore-glob +<li> keep-glob +<li> crlf-glob +<li> crnl-glob +<li> encoding-glob +<li> empty-dirs +<li> <s title="removed 2020-08, version 2.12.1">allow-symlinks</s> +<li> dotfiles +<li> parent-project-code +<li> parent-project-name +<ul></td><td valign="top"><ul> +<li> hash-policy +<li> comment-format +<li> mimetypes +<li> forbid-delta-manifests +<li> mv-rm-files +<li> ticket-table +<li> ticket-common +<li> ticket-change +<li> ticket-newpage +<li> ticket-viewpage +<li> ticket-editpage +<li> ticket-reportlist +<li> ticket-report-template +<li> ticket-key-template +<li> ticket-title-expr +<li> ticket-closed-expr +<ul></td><td valign="top"><ul> +<li> user-color-map +<li> xfer-common-script +<li> xfer-push-script +<li> xfer-commit-script +<li> xfer-ticket-script +<li> @reportfmt +<li> @user +<li> @concealed +<li> @shun +<li> @alias +<li> @subscriber +<li> @interwiki +</ul></td></tr> +</table> + +New configuration-names are likely to be added in future releases of +Fossil. If the server receives a configuration-name that it does not +understand, the entire reqconfig card is silently ignored. The reqconfig +card might also be ignored if the user lacks sufficient privilege to +access the requested information. + +The configuration-names that begin with an alphabetic character refer +to values in the "config" table of the server database. For example, +the "logo-image" configuration item refers to the project logo image +that is configured on the Admin page of the [./webui.wiki | web-interface]. +The value of the configuration item is returned to the client using a +"config" card. + +If the configuration-name begins with "@", that refers to a class of +values instead of a single value. The content of these configuration items +is returned in a "config" card that contains pure SQL text that is +intended to be evaluated by the client. + +The @user and @concealed configuration items contain sensitive information +and are ignored for clients without sufficient privilege. + +<h3 id="config">3.10 Configuration Cards</h3> + +A "config" card is used to send configuration information from client +to server (in response to a "fossil configuration push" command) or +from server to client (in response to a "fossil configuration pull" or +"fossil clone" command). The format is as follows: + +<pre> +<b>config</b> <i>configuration-name size</i> <b>\n</b> <i>content</i> +</pre> + +The server will only accept a config card if the user has +"Admin" privilege. A client will only accept a config card if +it had sent a corresponding reqconfig card in its request. + +The content of the configuration item is used to overwrite the +corresponding configuration data in the receiver. + +<h3 id="pragma">3.11 Pragma Cards</h3> + +The client may try to influence the behavior of the server by +issuing a pragma card: + +<pre> +<b>pragma</i> <i>name value...</i> +</pre> + +The "pragma" card has at least one argument which is the pragma name. +The pragma name defines what the pragma does. +A pragma might have zero or more "value" arguments +depending on the pragma name. + +New pragma names may be added to the protocol from time to time +in order to enhance the capabilities of Fossil. +Unknown pragmas are silently ignored, for backwards compatibility. + +The following are the known pragma names as of 2019-06-30: + +<ol> +<li><b>send-private</b> The send-private pragma instructs the server to send all of its +private artifacts to the client. The server will only obey this +request if the user has the "x" or "Private" privilege. + +<li><b>send-catalog</b> The send-catalog pragma instructs the server to transmit igot +cards for every known artifact. This can help the client and server +to get back in synchronization after a prior protocol error. The +"--verily" option to the [/help?cmd=sync|fossil sync] command causes +the send-catalog pragma to be transmitted. + +<li><b>uv-hash</b> <i>HASH</i> The uv-hash pragma is sent from client to server to provoke a +synchronization of unversioned content. The <i>HASH</i> is a SHA1 +hash of the names, modification times, and individual hashes of all +unversioned files on the client. If the unversioned content hash +from the client does not match the unversioned content hash on the +server, then the server will reply with either a "pragma uv-push-ok" +or "pragma uv-pull-only" card followed by one "uvigot" card for +each unversioned file currently held on the server. The collection +of "uvigot" cards sent in response to a "uv-hash" pragma is called +the "unversioned catalog". The client will used the unversioned +catalog to figure out which files (if any) need to be synchronized +between client and server and send appropriate "uvfile" or "uvgimme" +cards on the next HTTP request. + +If a client sends a uv-hash pragma and does not receive back +either a uv-pull-only or uv-push-ok pragma, that means that the +content on the server exactly matches the content on the client and +no further synchronization is required. + +<li><b>uv-pull-only</b></i> A server sends the uv-pull-only pragma to the client in response +to a uv-hash pragma with a mismatched content hash argument. This +pragma indicates that there are differences in unversioned content +between the client and server but that content can only be transferred +from server to client. The server is unwilling to accept content from +the client because the client login lacks the "write-unversioned" +permission. + +<li><b>uv-push-ok</b></i> A server sends the uv-push-ok pragma to the client in response +to a uv-hash pragma with a mismatched content hash argument. This +pragma indicates that there are differences in unversioned content +between the client and server and that content can be transferred +in either direction. The server is willing to accept content from +the client because the client login has the "write-unversioned" +permission. + +<li><b>ci-lock</b> <i>CHECKIN-HASH CLIENT-ID</i> A client sends the "ci-lock" pragma to the server to indicate +that it is about to add a new check-in as a child of the +CHECKIN-HASH check-in and on the same branch as CHECKIN-HASH. +If some other client has already indicated that it was also +trying to commit against CHECKIN-HASH, that indicates that a +fork is about to occur, and the server will reply with +a "ci-lock-fail" pragma (see below). Check-in locks +automatically expire when the check-in actually occurs, or +after a timeout (currently one minute but subject to change). + +<li><b>ci-lock-fail</b> <i>LOGIN MTIME</i> When a server receives two or more "ci-lock" pragma messages +for the same check-in but from different clients, the second a +subsequent ci-lock will provoke a ci-lock-fail pragma in the +reply to let the client know that it if continues with the +check-in it will likely generate a fork. The LOGIN and MTIME +arguments are intended to provide information to the client to +help it generate a more useful error message. + +<li><b>ci-unlock</b> <i>CLIENT-ID</i> A client sends the "ci-unlock" pragma to the server after +a successful commit. This instructs the server to release +any lock on any check-in previously held by that client. +The ci-unlock pragma helps to avoid false-positive lock warnings +that might arise if a check-in is aborted and then restarted +on a branch. + +<li><b>req-clusters</b> A client sends the "req-clusters" pragma +to the server to ask the server to reply with "igot" cards for +every [./fileformat.wiki#cluster|cluster artifact] that it holds. The +client typically does this when it thinks that it might be attempting +to pull a long chain of cluster artifacts. Sending the artifacts +all at once can dramatically reduce the number of round trip +messages needed to complete the synchronization. +</ol> + +<h3 id="comment">3.12 Comment Cards</h3> + +Any card that begins with "#" (ASCII 0x23) is a comment card and +is silently ignored. + +<h3 id="error">3.13 Message and Error Cards</h3> + +If the server discovers anything wrong with a request, it generates +an error card in its reply. When the client sees the error card, +it displays an error message to the user and aborts the sync +operation. An error card looks like this: + +<pre> +<b>error</b> <i>error-message</i> +</pre> + +The error message is English text that is encoded in order to +be a single token. +A space (ASCII 0x20) is represented as "\s" (ASCII 0x5C, 0x73). A +newline (ASCII 0x0a) is "\n" (ASCII 0x6C, x6E). A backslash +(ASCII 0x5C) is represented as two backslashes "\\". Apart from +space and newline, no other whitespace characters nor any +unprintable characters are allowed in +the error message. + +The server can also send a message card that also prints a +message on the client console, but which is not an error: + +<pre> +<b>message</b> <i>message-text</i> +</pre> + +The message-text uses the same format as an error message. + +<h3 id="unknown">3.14 Unknown Cards</h3> + +If either the client or the server sees a card that is not +described above, then it generates an error and aborts. + +<h2 id="phantoms" name="clusters">4.0 Phantoms And Clusters</h2> + +When a repository knows that an artifact exists and knows the ID of +that artifact, but it does not know the artifact content, then it stores that +artifact as a "phantom". A repository will typically create a phantom when +it receives an igot card for an artifact that it does not hold or when it +receives a file card that references a delta source that it does not +hold. When a server is generating its reply or when a client is +generating a new request, it will usually send gimme cards for every +phantom that it holds. + +A cluster is a special artifact that tells of the existence of other +artifacts. Any artifact in the repository that follows the syntactic rules +of a cluster is considered a cluster. + +A cluster is line oriented. Each line of a cluster +is a card. The cards are separated by the newline ("\n") character. +Each card consists of a single character card type, a space, and a +single argument. No extra whitespace and no trailing or leading +whitespace is allowed. All cards in the cluster must occur in +strict lexicographical order. + +A cluster consists of one or more "M" cards followed by a single +"Z" card. Each M card holds an argument which is an artifact ID for an +artifact in the repository. The Z card has a single argument which is the +lower-case hexadecimal representation of the MD5 checksum of all +preceding M cards up to and included the newline character that +occurred just before the Z that starts the Z card. + +Any artifact that does not match the specifications of a cluster +exactly is not a cluster. There must be no extra whitespace in +the artifact. There must be one or more M cards. There must be a +single Z card with a correct MD5 checksum. And all cards must +be in strict lexicographical order. + +<h3 id="unclustered">4.1 The Unclustered Table</h3> + +Every repository maintains a table named "<b>unclustered</b>" +which records the identity of every artifact and phantom it holds that is not +mentioned in a cluster. The entries in the unclustered table can +be thought of as leaves on a tree of artifacts. Some of the unclustered +artifacts will be other clusters. Those clusters may contain other clusters, +which might contain still more clusters, and so forth. Beginning +with the artifacts in the unclustered table, one can follow the chain +of clusters to find every artifact in the repository. + +<h2 id="strategies">5.0 Synchronization Strategies</h2> + +<h3 id="pull-strategy">5.1 Pull</h3> + +A typical pull operation proceeds as shown below. Details +of the actual implementation may vary slightly but the gist of +a pull is captured in the following steps: + +<ol> +<li>The client sends login and pull cards. +<li>The client sends a cookie card if it has previously received a cookie. +<li>The client sends gimme cards for every phantom that it holds. +<hr> +<li>The server checks the login password and rejects the session if +the user does not have permission to pull. +<li>If the number of entries in the unclustered table on the server is +greater than 100, then the server constructs a new cluster artifact to +cover all those unclustered entries. +<li>The server sends file cards for every gimme card it received +from the client. +<li>The server sends igot cards for every artifact in its unclustered +table that is not a phantom. +<hr> +<li>The client adds the content of file cards to its repository. +<li>The client creates a phantom for every igot card in the server reply +that mentions an artifact that the client does not possess. +<li>The client creates a phantom for the delta source of file cards when +the delta source is an artifact that the client does not possess. +</ol> + +These ten steps represent a single HTTP round-trip request. +The first three steps are the processing that occurs on the client +to generate the request. The middle four steps are processing +that occurs on the server to interpret the request and generate a +reply. And the last three steps are the processing that the +client does to interpret the reply. + +During a pull, the client will keep sending HTTP requests +until it holds all artifacts that exist on the server. + +Note that the server tries +to limit the size of its reply message to something reasonable +(usually about 1MB) so that it might stop sending file cards as +described in step (6) if the reply becomes too large. + +Step (5) is the only way in which new clusters can be created. +By only creating clusters on the server, we hope to minimize the +amount of overlap between clusters in the common configuration where +there is a single server and many clients. The same synchronization +protocol will continue to work even if there are multiple servers +or if servers and clients sometimes change roles. The only negative +effects of these unusual arrangements is that more than the minimum +number of clusters might be generated. + +<h3 id="push-stragegy">5.2 Push</h3> + +A typical push operation proceeds roughly as shown below. As +with a pull, the actual implementation may vary slightly. + +<ol> +<li>The client sends login and push cards. +<li>The client sends file cards for any artifacts that it holds that have +never before been pushed - artifacts that come from local check-ins. +<li>If this is the second or later cycle in a push, then the +client sends file cards for any gimme cards that the server sent +in the previous cycle. +<li>The client sends igot cards for every artifact in its unclustered table +that is not a phantom. +<hr> +<li>The server checks the login and push cards and issues an error if +anything is amiss. +<li>The server accepts file cards from the client and adds those artifacts +to its repository. +<li>The server creates phantoms for igot cards that mention artifacts it +does not possess or for file cards that mention delta source artifacts that +it does not possess. +<li>The server issues gimme cards for all phantoms. +<hr> +<li>The client remembers the gimme cards from the server so that it +can generate file cards in reply on the next cycle. +</ol> + +As with a pull, the steps of a push operation repeat until the +server knows all artifacts that exist on the client. Also, as with +pull, the client attempts to keep the size of the request from +growing too large by suppressing file cards once the +size of the request reaches 1MB. + +<h3 id="sync-strategy">5.3 Sync</h3> + +A sync is just a pull and a push that happen at the same time. +The first three steps of a pull are combined with the first five steps +of a push. Steps (4) through (7) of a pull are combined with steps +(5) through (8) of a push. And steps (8) through (10) of a pull +are combined with step (9) of a push. + +<h3 id="uv-strategy">5.4 Unversioned File Sync</h3> + +"Unversioned files" are files held in the repository +where only the most recent version of the file is kept rather than +the entire change history. Unversioned files are intended to be +used to store ephemeral content, such as compiled binaries of the +most recent release. + +Unversioned files are identified by name and timestamp (mtime). +Only the most recent version of each file (the version with +the largest mtime value) is retained. + +Unversioned files are synchronized using the +[/help?cmd=unversioned|fossil unversioned sync] command. + +A schematic of an unversioned file synchronization is as follows: + +<ol> +<li>The client sends a "pragma uv-hash" card to the server. The argument + to the uv-hash pragma is a hash of all filesnames, mtimes, and + content hashes for the unversioned files held by the client. + <hr> +<li>If the unversioned content hash from the client matches the unversioned + content hash on the server, then nothing needs to be done and the + server no-ops. But if the hashes are different, then the server + replies with either a uv-pull-only or a uv-push-ok pragma followed by + uvigot cards for all unversioned files held on the server. + <hr> +<li>The client examines the uvigot cards received from the server and + determines which unversioned files need to be exchanged in order + to bring the client and server into synchronization. The client + then sends appropriate "uvgimme" or "uvfile" cards back to the + server. + <hr> +<li>The server updates its unversioned file store with received "uvfile" + cards and answers "uvgimme" cards with "uvfile" cards in its reply. +</ol> + +The last two steps might be repeated multiple +times if there is more unversioned content to be transferred than will +fit comfortably in a single HTTP request. + +<h2 id="summary">6.0 Summary</h2> + +Here are the key points of the synchronization protocol: + +<ol> +<li>The client sends one or more PUSH HTTP requests to the server. + The request and reply content type is "application/x-fossil". +<li>HTTP request content is compressed using zlib. +<li>The content of request and reply consists of cards with one + card per line. +<li>Card formats are: + <ul> + <li> <b>login</b> <i>userid nonce signature</i> + <li> <b>push</b> <i>servercode projectcode</i> + <li> <b>pull</b> <i>servercode projectcode</i> + <li> <b>clone</b> + <li> <b>clone</b> <i>protocol-version sequence-number</i> + <li> <b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i> + <li> <b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> + <li> <b>cfile</b> <i>artifact-id size</i> <b>\n</b> <i>content</i> + <li> <b>cfile</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> + <li> <b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i> + <li> <b>private</b> + <li> <b>igot</b> <i>artifact-id</i> ?<i>flag</i>? + <li> <b>uvigot</b> <i>name mtime hash size</i> + <li> <b>gimme</b> <i>artifact-id</i> + <li> <b>uvgimme</b> <i>name</i> + <li> <b>cookie</b> <i>cookie-text</i> + <li> <b>reqconfig</b> <i>parameter-name</i> + <li> <b>config</b> <i>parameter-name size</i> <b>\n</b> <i>content</i> + <li> <b>pragma</b> <i>name</i> <i>value...</i> + <li> <b>error</b> <i>error-message</i> + <li> <b>message</b> <i>text-messate</i> + <li> <b>#</b> <i>arbitrary-text...</i> + </ul> +<li>Phantoms are artifacts that a repository knows exist but does not possess. +<li>Clusters are artifacts that contain IDs of other artifacts. +<li>Clusters are created automatically on the server during a pull. +<li>Repositories keep track of all artifacts that are not named in any +cluster and send igot messages for those artifacts. +<li>Repositories keep track of all the phantoms they hold and send +gimme messages for those artifacts. +</ol> + +<h2 id="troubleshooting">7.0 Troubleshooting And Debugging Hints</h2> + +If you run the [/help?cmd=sync|fossil sync] command +(or [/help?cmd=pull|pull] or [/help?cmd=push|push] or +[/help?cmd=clone|clone]) with the --httptrace option, Fossil +will keep a copy of each HTTP request and reply in files +named: +<ul> +<li> <tt>http-request-</tt><i>N</i><tt>.txt</tt> +<li> <tt>http-reply-</tt><i>N</i><tt>.txt</tt> +</ul> + +In the above, <i>N</i> is an integer that increments with each +round-trip. If you are having trouble on the server side, +you can run the "[/help?cmd=test-http|fossil test-http]" command in a +debugger using one the "http-request-N.txt" files as input and +single step through the processing performed by the server. + +The "--transport-command CMD" option on [/help?cmd=sync|fossil sync] +(and similar) causes the external program "CMD" to be used to move +the sync message to the server and retrieve the sync reply. The +CMD is given three arguments: +<ol> +<li> The URL of the server +<li> The name of a temporary file that contains the output-bound sync + protocol text, with the HTTP headers +<li> The name of a temporary file into which the CMD should write the + reply sync protocol text, again without any HTTP headers +</ol> + +In a complex debugging situation, you can run the command +"fossil sync --transport-command ./debugging_script" where +"debugging_script" is some script of your own that invokes +the anomalous behavior you are trying to debug. diff --git a/www/tech_overview.wiki b/www/tech_overview.wiki new file mode 100644 index 0000000..a46c9ff --- /dev/null +++ b/www/tech_overview.wiki @@ -0,0 +1,378 @@ +<title>A Technical Overview of Fossil's Design & Implementation</title> + +<h2>1.0 Introduction</h2> + +At its lowest level, a Fossil repository consists of an unordered set +of immutable "artifacts". You might think of these artifacts as "files", +since in many cases the artifacts are exactly that. +But other "structural artifacts" are also included in the mix. +These structural artifacts define the relationships +between artifacts - which files go together to form a particular +version of the project, who checked in that version and when, what was +the check-in comment, what wiki pages are included with the project, what +are the edit histories of each wiki page, what bug reports or tickets are +included, who contributed to the evolution of each ticket, and so forth. +This low-level file format is called the "global state" of +the repository, since this is the information that is synced to peer +repositories using push and pull operations. The low-level file format +is also called "enduring" since it is intended to last for many years. +The details of the low-level, enduring, global file format +are [./fileformat.wiki | described separately]. + +This article is about how Fossil is currently implemented. Instead of +dealing with vague abstractions of "enduring file formats" as the +[./fileformat.wiki | other document] does, this article provides +some detail on how Fossil actually stores information on disk. + +<h2>2.0 Three Databases</h2> + +Fossil stores state information in +[http://www.sqlite.org/ | SQLite] database files. +SQLite keeps an entire relational database, including multiple tables and +indices, in a single disk file. The SQLite library allows the database +files to be efficiently queried and updated using the industry-standard +SQL language. SQLite updates are atomic, so even in the event of +a system crashes or power failure the repository content is protected. + +Fossil uses three separate classes of SQLite databases: + +<ol> +<li>The configuration database +<li>Repository databases +<li>Checkout databases +</ol> + +The configuration database is a one-per-user database that holds +global configuration information used by Fossil. There is one +repository database per project. The repository database is the +file that people are normally referring to when they say +"a Fossil repository". The checkout database is found in the working +checkout for a project and contains state information that is unique +to that working checkout. + +Fossil does not always use all three database files. The web interface, +for example, typically only uses the repository database. And the +[/help/settings | fossil settings] command only opens the configuration database +when the --global option is used. But other commands use all three +databases at once. For example, the [/help/status | fossil status] +command will first locate the checkout database, then use the checkout +database to find the repository database, then open the configuration +database. Whenever multiple databases are used at the same time, +they are all opened on the same SQLite database connection using +SQLite's [http://www.sqlite.org/lang_attach.html | ATTACH] command. + +The chart below provides a quick summary of how each of these +database files are used by Fossil, with detailed discussion following. + +<table align="center"> +<tr valign="bottom"> +<th style="text-align:center">Configuration Database<br>"~/.fossil" or<br> +"~/.config/fossil.db" +<th style="text-align:center">Repository Database<br>"<i>project</i>.fossil" +<th style="text-align:center">Checkout Database<br>"_FOSSIL_" or ".fslckout" +<tr valign="top"> +<td><ul> +<li>Global [/help/settings |settings] +<li>List of active repositories used by the [/help/all | all] command +</ul></td> +<td><ul> +<li>[./fileformat.wiki | Global state of the project] + encoded using delta-compression +<li>Local [/help/settings|settings] +<li>Web interface display preferences +<li>User credentials and permissions +<li>Metadata about the global state to facilitate rapid + queries +</ul></td> +<td><ul> +<li>The repository database used by this checkout +<li>The version currently checked out +<li>Other versions [/help/merge | merged] in but not + yet [/help/commit | committed] +<li>Changes from the [/help/add | add], [/help/delete | delete], + and [/help/rename | rename] commands that have not yet been committed +<li>"mtime" values and other information used to efficiently detect + local edits +<li>The "[/help/stash | stash]" +<li>Information needed to "[/help/undo|undo]" or "[/help/redo|redo]" +</ul></td> +</tr> +</table> + +<h3 id="configdb">2.1 The Configuration Database</h3> + +The configuration database holds cross-repository preferences and a list of all +repositories for a single user. + +The [/help/settings | fossil settings] command can be used to specify various +operating parameters and preferences for Fossil repositories. Settings can +apply to a single repository, or they can apply globally to all repositories +for a user. If both a global and a repository value exists for a setting, +then the repository-specific value takes precedence. All of the settings +have reasonable defaults, and so many users will never need to change them. +But if changes to settings are desired, the configuration database provides +a way to change settings for all repositories with a single command, rather +than having to change the setting individually on each repository. + +The configuration database also maintains a list of repositories. This +list is used by the [/help/all | fossil all] command in order to run various +operations such as "sync" or "rebuild" on all repositories managed by a user. + +<h4 id="configloc">2.1.1 Location Of The Configuration Database</h4> + +On Unix systems, the configuration database is named by the following +algorithm: + +<table> +<tr><td>1. if environment variable FOSSIL_HOME exists + <td> → <td>$FOSSIL_HOME/.fossil +<tr><td>2. if file ~/.fossil exists + <td> →<td>~/.fossil +<tr><td>3. if environment variable XDG_CONFIG_HOME exists + <td> →<td>$XDG_CONFIG_HOME/fossil.db +<tr><td>4. if the directory ~/.config exists + <td> →<td>~/.config/fossil.db +<tr><td>5. Otherwise<td> →<td>~/.fossil +</table> + +Another way of thinking of this algorithm is the following: + + * Use "$FOSSIL_HOME/.fossil" if the FOSSIL_HOME variable is defined + * Use the XDG-compatible name (usually ~/.config/fossil.db) on XDG systems + if the ~/.fossil file does not already exist + * Otherwise, use the traditional unix name of "~/.fossil" + +This algorithm is complex due to the need for historical compatibility. +Originally, the database was always just "~/.fossil". Then support +for the FOSSIL_HOME environment variable was added. Later, support for the +[https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html|XDG-compatible configation filenames] +was added. Each of these changes needed to continue to support legacy +installations. + +On Windows, the configuration database is the first of the following +for which the corresponding environment variables exist: + + * %FOSSIL_HOME%/_fossil + * %LOCALAPPDATA%/_fossil + * %APPDATA%/_fossil + * %USERPROFILES%/_fossil + * %HOMEDRIVE%%HOMEPATH%/_fossil + +The second case is the one that usually determines the name. Note that the +FOSSIL_HOME environment variable can always be set to determine the +location of the configuration database. Note also that the configuration +database file itself is called ".fossil" or "fossil.db" on unix but +"_fossil" on windows. + +The [/help/info|fossil info] command will show the location of +the configuration database on a line that starts with "config-db:". + +<h3>2.2 Repository Databases</h3> + +The repository database is the file that is commonly referred to as +"the repository". This is because the repository database contains, +among other things, the complete revision, ticket, and wiki history for +a project. It is customary to name the repository database after the +name of the project, with a ".fossil" suffix. For example, the repository +database for the self-hosting Fossil repository is called "fossil.fossil" +and the repository database for SQLite is called "sqlite.fossil". + +<h4>2.2.1 Global Project State</h4> + +The bulk of the repository database (typically 75 to 85%) consists +of the artifacts that comprise the +[./fileformat.wiki | enduring, global, shared state] of the project. +The artifacts are stored as BLOBs, compressed using +[http://www.zlib.net/ | zlib compression] and, where applicable, +using [./delta_encoder_algorithm.wiki | delta compression]. +The combination of zlib and delta compression results in a considerable +space savings. For the SQLite project (when this paragraph was last +updated on 2020-02-08) +the total size of all artifacts is over 7.1 GB but thanks to the +combined zlib and delta compression, that content only takes less than +97 MB of space in the repository database, for a compression ratio +of about 74:1. The median size of all content BLOBs after delta +and zlib compression have been applied is 156 bytes. +The median size of BLOBs without compression is 45,312 bytes. + +Note that the zlib and delta compression is not an inherent part of the +Fossil file format; it is just an optimization. +The enduring file format for Fossil is the unordered +set of artifacts. The compression techniques are just a detail of +how the current implementation of Fossil happens to store these artifacts +efficiently on disk. + +All of the original uncompressed and un-delta'd artifacts can be extracted +from a Fossil repository database using +the [/help/deconstruct | fossil deconstruct] +command. Individual artifacts can be extracted using the +[/help/artifact | fossil artifact] command. +When accessing the repository database using raw SQL and the +[/help/sqlite3 | fossil sql] command, the extension function +"<tt>content()</tt>" with a single argument which is the SHA1 or +SHA3-256 hash +of an artifact will return the complete uncompressed +content of that artifact. + +Going the other way, the [/help/reconstruct | fossil reconstruct] +command will scan a directory hierarchy and add all files found to +a new repository database. The [/help/import | fossil import] command +works by reading the input git-fast-export stream and using it to construct +corresponding artifacts which are then written into the repository database. + +<h4>2.2.2 Project Metadata</h4> + +The global project state information in the repository database is +supplemented by computed metadata that makes querying the project state +more efficient. Metadata includes information such as the following: + + * The names for all files found in any check-in. + * All check-ins that modify a given file + * Parents and children of each check-in. + * Potential timeline rows. + * The names of all symbolic tags and the check-ins they apply to. + * The names of all wiki pages and the artifacts that comprise each + wiki page. + * Attachments and the wiki pages or tickets they apply to. + * Current content of each ticket. + * Cross-references between tickets, check-ins, and wiki pages. + +The metadata is held in various SQL tables in the repository database. +The metadata is designed to facilitate queries for the various timelines and +reports that Fossil generates. +As the functionality of Fossil evolves, +the schema for the metadata can and does change. +But schema changes do not invalidate the repository. Remember that the +metadata contains no new information - only information that has been +extracted from the canonical artifacts and saved in a more useful form. +Hence, when the metadata schema changes, the prior metadata can be discarded +and the entire metadata corpus can be recomputed from the canonical +artifacts. That is what the +[/help/rebuild | fossil rebuild] command does. + +<h4>2.2.3 Display And Processing Preferences</h4> + +The repository database also holds information used to help format +the display of web pages and configuration settings that override the +global configuration settings for the specific repository. All of +this information (and the user credentials and privileges too) is +local to each repository database; it is not shared between repositories +by [/help/sync | fossil sync]. That is because it is entirely reasonable +that two different websites for the same project might have completely +different display preferences and user communities. One instance of the +project might be a fork of the other, for example, which pulls from the +other but never pushes and extends the project in ways that the keepers of +the other website disapprove of. + +Display and processing information includes the following: + + * The name and description of the project + * The CSS file, header, and footer used by all web pages + * The project logo image + * Fields of tickets that are considered "significant" and which are + therefore collected from artifacts and made available for display + * Templates for screens to view, edit, and create tickets + * Ticket report formats and display preferences + * Local values for [/help/settings | settings] that override the + global values defined in the per-user configuration database. + +Though the display and processing preferences do not move between +repository instances using [/help/sync | fossil sync], this information +can be shared between repositories using the +[/help/config | fossil config push] and +[/help/config | fossil config pull] commands. +The display and processing information is also copied into new +repositories when they are created using +[/help/clone | fossil clone]. + +<h4>2.2.4 User Credentials And Privileges</h4> + +Just because two development teams are collaborating on a project and allow +push and/or pull between their repositories does not mean that they +trust each other enough to share passwords and access privileges. +Hence the names and emails and passwords and privileges of users are +considered private information that is kept locally in each repository. + +Each repository database has a table holding the username, privileges, +and login credentials for users authorized to interact with that particular +database. In addition, there is a table named "concealed" that maps the +SHA1 hash of each users email address back into their true email address. +The concealed table allows just the SHA1 hash of email addresses to +be stored in tickets, and thus prevents actual email addresses from falling +into the hands of spammers who happen to clone the repository. + +The content of the user and concealed tables can be pushed and pulled using the +[/help/config | fossil config push] and +[/help/config | fossil config pull] commands with the "user" and +"email" as the AREA argument, but only if you have administrative +privileges on the remote repository. + +<h4>2.2.5 Shunned Artifact List</h4> + +The set of canonical artifacts for a project - the global state for the +project - is intended to be an append-only database. In other words, +new artifacts can be added but artifacts can never be removed. But +it sometimes happens that inappropriate content is mistakenly or +maliciously added to a repository. The only way to get rid of +the undesired content is to [./shunning.wiki | "shun"] it. +The "shun" table in the repository database records the hash values for +all shunned artifacts. + +The shun table can be pushed or pulled using +the [/help/config | fossil config] command with the "shun" AREA argument. +The shun table is also copied during a [/help/clone | clone]. + +<h3 id="localdb">2.3 Checkout Databases</h3> + +Fossil allows a single repository +to have multiple working checkouts. Each working checkout has a single +database in its root directory that records the state of that checkout. +The checkout database is named "_FOSSIL_" or ".fslckout". +The checkout database records information such as the following: + + * The name of the repository database file. + * The version that is currently checked out. + * Files that have been [/help/add | added], + [/help/rm | removed], or [/help/mv | renamed] but not + yet committed. + * The mtime and size of files as they were originally checked out, + in order to expedite checking which files have been edited. + * Other check-ins that have been [/help/merge | merged] into the + working checkout but not yet committed. + * Copies of files prior to the most recent undoable operation - needed to + implement the [/help/undo | undo] and [/help/redo | redo] commands. + * The [/help/stash | stash]. + * State information for the [/help/bisect | bisect] command. + +For Fossil commands that run from within a working checkout, the +first thing that happens is that Fossil locates the checkout database. +Fossil first looks in the current directory. If not found there, it +looks in the parent directory. If not found there, the parent of the +parent. And so forth until either the checkout database is found +or the search reaches the root of the file system. (In the latter case, +Fossil returns an error, of course.) Once the checkout database is +located, it is used to locate the repository database. + +Notice that the checkout database contains a pointer to the repository +database but that the repository database has no record of the checkout +databases. That means that a working checkout directory tree can be +freely renamed or copied or deleted without consequence. But the +repository database file, on the other hand, has to stay in the same +place with the same name or else the open checkout databases will not +be able to find it. + +A checkout database is created by the [/help/open | fossil open] command. +A checkout database is deleted by [/help/close | fossil close]. The +fossil close command really isn't needed; one can accomplish the same +thing simply by deleting the checkout database. + +Note that the stash, the undo stack, and the state of the bisect command +are all contained within the checkout database. That means that the +fossil close command will delete all stash content, the undo stack, and +the bisect state. The close command is not undoable. Use it with care. + +<h2>3.0 See Also</h2> + + * [./makefile.wiki | The Fossil Build Process] + * [./contribute.wiki | How To Contribute Code To Fossil] + * [./adding_code.wiki | Adding New Features To Fossil] diff --git a/www/th1-hooks.md b/www/th1-hooks.md new file mode 100644 index 0000000..10845fd --- /dev/null +++ b/www/th1-hooks.md @@ -0,0 +1,134 @@ +TH1 Hooks +========= + +<big><big><big><font color="red">** DRAFT **</font></big></big></big> + +The **TH1 hooks** feature allows <a href="th1.md">TH1</a> scripts to be +configured that can monitor, create, alter, or cancel the execution of +Fossil commands and web pages. + +This feature requires the TH1 hooks feature to be enabled at compile-time. +Additionally, the "th1-hooks" repository setting must be enabled at runtime +in order to successfully make use of this feature. + +TH1 Hook Related User-Defined Procedures +---------------------------------------- + +In order to activate TH1 hooks, one or more of the following user-defined +procedures should be defined, generally from within the "th1-setup" script +(setting) for a repository. The following bullets summarize the available +TH1 hooks: + + * command\_hook -- _Called before execution of a command._ + * command\_notify -- _Called after execution of a command._ + * webpage\_hook -- _Called before rendering of a web page._ + * webpage\_notify -- _Called after rendering of a web page._ + +TH1 Hook Related Variables for Commands +--------------------------------------- + + * cmd\_name -- _Name of command being executed._ + * cmd\_args -- _Current command line arguments._ + * cmd\_flags -- _Bitmask of CMDFLAG values for the command being executed._ + +TH1 Hook Related Variables for Web Pages +---------------------------------------- + + * web\_name -- _Name of web page being rendered._ + * web\_args -- _Current web page arguments._ + * web\_flags -- _Bitmask of CMDFLAG values for the web page being rendered._ + +<a id="cmdReturnCodes"></a>TH1 Hook Related Return Codes for Commands +----------------------------------------------------------------------- + + * TH\_OK -- _Command will be executed, notification will be executed._ + * TH\_ERROR -- _Command will be skipped, notification will be skipped, + error message will be emitted._ + * TH\_BREAK -- _Command will be skipped, notification will be skipped._ + * TH\_RETURN -- _Command will be executed, notification will be skipped._ + * TH\_CONTINUE -- _Command will be skipped, notification will be executed._ + +For commands that are not included in the Fossil binary, allowing their +execution will cause the standard "unknown command" error message to be +generated, which will typically exit the process. Therefore, adding a +new command generally requires using the TH_CONTINUE return code. + +<a id="webReturnCodes"></a>TH1 Hook Related Return Codes for Web Pages +------------------------------------------------------------------------ + + * TH\_OK -- _Web page will be rendered, notification will be executed._ + * TH\_ERROR -- _Web page will be skipped, notification will be skipped, + error message will be emitted._ + * TH\_BREAK -- _Web page will be skipped, notification will be skipped._ + * TH\_RETURN -- _Web page will be rendered, notification will be skipped._ + * TH\_CONTINUE -- _Web page will be skipped, notification will be executed._ + +For web pages that are not included in the Fossil binary, allowing their +rendering will cause the standard "Not Found" error message to be generated, +which will cause an HTTP 404 status code to be sent. Therefore, adding a +new web page generally requires using the TH_CONTINUE return code. + +<a id="triggerReturnCodes"></a>Triggering TH1 Return Codes from a Script +-------------------------------------------------------------------------- + + * TH\_OK -- _This is the default return code, nothing special needed._ + * TH\_ERROR -- _Use the **error** command._ + * TH\_BREAK -- _Use the **break** command._ + * TH\_RETURN -- _Use the **return -code 5** command._ + * TH\_CONTINUE -- _Use the **continue** command._ + +<a id="command_hook"></a>TH1 command_hook Procedure +----------------------------------------------------- + + * command\_hook + +This user-defined procedure, if present, is called just before the +execution of a command. The name of the command being executed will +be stored in the "cmd\_name" global variable. The arguments to the +command being executed will be stored in the "cmd\_args" global variable. +The associated CMDFLAG value will be stored in the "cmd\_flags" global +variable. Before exiting, the procedure should trigger the return +<a href="#cmdReturnCodes">code</a> that corresponds to the desired action +to take next. + +<a id="command_notify"></a>TH1 command_notify Procedure +--------------------------------------------------------- + + * command\_notify + +This user-defined procedure, if present, is called just after the +execution of a command. The name of the command being executed will +be stored in the "cmd\_name" global variable. The arguments to the +command being executed will be stored in the "cmd\_args" global variable. +The associated CMDFLAG value will be stored in the "cmd\_flags" global +variable. Before exiting, the procedure should trigger the return +<a href="#cmdReturnCodes">code</a> that corresponds to the desired action +to take next. + +<a id="webpage_hook"></a>TH1 webpage_hook Procedure +----------------------------------------------------- + + * webpage\_hook + +This user-defined procedure, if present, is called just before the +rendering of a web page. The name of the web page being rendered will +be stored in the "web\_name" global variable. The arguments to the +web page being rendered will be stored in the "web\_args" global variable. +The associated CMDFLAG value will be stored in the "web\_flags" global +variable. Before exiting, the procedure should trigger the return +<a href="#webReturnCodes">code</a> that corresponds to the desired action +to take next. + +<a id="webpage_notify"></a>TH1 webpage_notify Procedure +--------------------------------------------------------- + + * webpage\_notify + +This user-defined procedure, if present, is called just after the +rendering of a web page. The name of the web page being rendered will +be stored in the "web\_name" global variable. The arguments to the +web page being rendered will be stored in the "web\_args" global variable. +The associated CMDFLAG value will be stored in the "web\_flags" global +variable. Before exiting, the procedure should trigger the return +<a href="#webReturnCodes">code</a> that corresponds to the desired action +to take next. diff --git a/www/th1.md b/www/th1.md new file mode 100644 index 0000000..97875cf --- /dev/null +++ b/www/th1.md @@ -0,0 +1,1024 @@ +TH1 Scripts +=========== + +TH1 is a very small scripting language used to help generate web-page +content in Fossil. + +Origins +------- + +TH1 began as a minimalist re-implementation of the Tcl scripting language. +There was a need to test the SQLite library on Symbian phones, but at that +time all of the test cases for SQLite were written in Tcl and Tcl could not +be easily compiled on the SymbianOS. So TH1 was developed as a cut-down +version of Tcl that would facilitate running the SQLite test scripts on +SymbianOS. + +Fossil was first being designed at about the same time. +Early prototypes of Fossil were written in pure Tcl. But as the development +shifted toward the use of C-code, the need arose to have a Tcl-like +scripting language to help with code generation. TH1 was small and +light-weight and used minimal resources and seemed ideally suited for the +task. + +The name "TH1" stands for "Test Harness 1", +since its original purpose was to serve as testing harness +for SQLite. + +Where TH1 Is Used In Fossil +--------------------------- + + * In the header and footer for [skins](./customskin.md) + text within `<th1>...</th1>` is run as a TH1 script. + ([example](/builtin/skins/default/header.txt)) + + * This display of [tickets](./bugtheory.wiki) is controlled by TH1 + scripts, so that the ticket format can be customized for each + project. Administrators can visit the <b>/tktsetup</b> page in + their repositories to view and customize these scripts. + ([example usage](./custom_ticket.wiki)) + +Overview Of The Tcl/TH1 Language +-------------------------------- + +TH1 is a string-processing language. All values are strings. Any numerical +operations are accomplished by converting from string to numeric, performing +the computation, then converting the result back into a string. (This might +seem inefficient, but it is faster than people imagine, and numeric +computations do not come up very often for the kinds of work that TH1 does, +so it has never been an issue.) + +A TH1 script consists of a sequence of commands. +Each command is terminated by the first *unescaped* newline or ";" character. +The text of the command (excluding the newline or semicolon terminator) +is broken into space-separated tokens. The first token is the command +name and subsequent tokens are the arguments. In this sense, TH1 syntax +is similar to the familiar command-line shell syntax. + +A token is any sequence of characters other than whitespace and semicolons. +Text inside double-quotes is a single token even if it includes +whitespace and semicolons. Text within {...} pairs is also a +single token, which is useful because curly braces are easier to “pair” +and nest properly than double-quotes. + +The nested {...} form of tokens is important because it allows TH1 commands +to have an appearance similar to C/C++. It is important to remember, though, +that a TH1 script is really just a list of text commands, not a context-free +language with a grammar like C/C++. This can be confusing to long-time +C/C++ programmers because TH1 does look a lot like C/C++, but the semantics +of TH1 are closer to FORTH or Lisp than they are to C. + +Consider the `if` command in TH1. + + if {$current eq "dev"} { + puts "hello" + } else { + puts "world" + } + +The example above is a single command. The first token, and the name +of the command, is `if`. +The second token is `$current eq "dev"` - an expression. (The outer {...} +are removed from each token by the command parser.) The third token +is the `puts "hello"`, with its whitespace and newlines. The fourth token +is `else` and the fifth and last token is `puts "world"`. + +The `if` command evaluates its first argument (the second token) +as an expression, and if that expression is true, it evaluates its +second argument (the third token) as a TH1 script. +If the expression is false and the third argument is `else`, then +the fourth argument is evaluated as a TH1 expression. + +So, you see, even though the example above spans five lines, it is really +just a single command. + +All of this also explains the emphasis on *unescaped* characters above: +the curly braces `{ }` are string quoting characters in Tcl/TH1, not +block delimiters as in C. This is how we can have a command that extends +over multiple lines. It is also why the `else` keyword must be cuddled +up with the closing brace for the `if` clause's scriptlet. The following +is invalid Tcl/TH1: + + if {$current eq "dev"} { + puts "hello" + } + else { + puts "world" + } + +If you try to run this under either Tcl or TH1, the interpreter will +tell you that there is no `else` command, because with the newline on +the third line, you terminated the `if` command. + +Occasionally in Tcl/TH1 scripts, you may need to use a backslash at the +end of a line to allow a command to extend over multiple lines without +being considered two separate commands. Here's an example from one of +Fossil's test scripts: + + return [lindex [regexp -line -inline -nocase -- \ + {^uuid:\s+([0-9A-F]{40}) } [eval [getFossilCommand \ + $repository "" info trunk]]] end] + +Those backslashes allow the command to wrap nicely within a standard +terminal width while telling the interpreter to consider those three +lines as a single command. + +<a id="taint"></a>Tainted And Untainted Strings +----------------------------------------------- + +Beginning with Fossil version 2.26 (circa 2025), TH1 distinguishes between +"tainted" and "untainted" strings. Tainted strings are strings that are +derived from user inputs that might contain text that is designed to subvert +the script. Untainted strings are known to come from secure sources and +are assumed to contain no malicious content. + +Beginning with Fossil version 2.26, and depending on the value of the +[vuln-report setting](/help/vuln-report), TH1 will prevent tainted +strings from being used in ways that might lead to XSS or SQL-injection +attacks. This feature helps to ensure that XSS and SQL-injection +vulnerabilities are not *accidentally* added to Fossil when +custom TH1 scripts for headers or footers or tickets are added to a +repository. Note that the tainted/untainted distinction in strings does +not make it impossible to introduce XSS and SQL-injections vulnerabilities +using poorly-written TH1 scripts; it just makes it more difficult and +less likely to happen by accident. Developers must still consider the +security implications of TH1 customizations they add to Fossil, and take +appropriate precautions when writing custom TH1. Peer review of TH1 +script changes is encouraged. + +In Fossil version 2.26, if the vuln-report setting is set to "block" +or "fatal", the [html](#html) and [query](#query) TH1 commands will +fail with an error if their argument is a tainted string. This helps +to prevent XSS and SQL-injection attacks, respectively. Note that +the default value of the vuln-report setting is "log", which allows those +commands to continue working and only writes a warning message into the +error log. <b>Future versions of Fossil may change the default value +of the vuln-report setting to "block" or "fatal".</b> Fossil users +with customized TH1 scripts are encouraged to audit their customizations +and fix any potential vulnerabilities soon, so as to avoid breakage +caused by future upgrades. <b>Future versions of Fossil might also +place additional restrictions on the use of tainted strings.</b> +For example, it is likely that future versions of Fossil will disallow +using tainted strings as script, for example as the body of a "for" +loop or of a "proc". + + +Summary of Core TH1 Commands +---------------------------- + +The original Tcl language (after which TH1 is modeled) has a very rich +repertoire of commands. TH1, as it is designed to be minimalist and +embedded has a greatly reduced command set. The following bullets +summarize the commands available in TH1: + + * array exists VARNAME + * array names VARNAME + * break + * catch SCRIPT ?VARIABLE? + * continue + * error ?STRING? + * expr EXPR + * for INIT-SCRIPT TEST-EXPR NEXT-SCRIPT BODY-SCRIPT + * foreach VARIABLE-LIST VALUE-LIST BODY-SCRIPT + * if EXPR SCRIPT (elseif EXPR SCRIPT)* ?else SCRIPT? + * info commands + * info exists VARNAME + * info vars + * lappend VARIABLE TERM ... + * lindex LIST INDEX + * list ARG ... + * llength LIST + * lsearch LIST STRING + * proc NAME ARG-LIST BODY-SCRIPT + * rename OLD NEW + * return ?-code CODE? ?VALUE? + * set VARNAME VALUE + * string compare STR1 STR2 + * string first NEEDLE HAYSTACK ?START-INDEX? + * string index STRING INDEX + * string is CLASS STRING + * string last NEEDLE HAYSTACK ?START-INDEX? + * string match PATTERN STRING + * string length STRING + * string range STRING FIRST LAST + * string repeat STRING COUNT + * string trim STRING + * string trimleft STRING + * string trimright STRING + * unset VARNAME + * uplevel ?LEVEL? SCRIPT + * upvar ?FRAME? OTHERVAR MYVAR ?OTHERVAR MYVAR? + +All of the above commands work as in the original Tcl. Refer to the +<a href="https://www.tcl-lang.org/man/tcl/contents.htm">Tcl documentation</a> +for details. + +Summary of Core TH1 Variables +----------------------------- + + * tcl\_platform(engine) -- _This will always have the value "TH1"._ + * tcl\_platform(platform) -- _This will have the value "windows" or "unix"._ + * th\_stack\_trace -- _This will contain error stack information._ + +TH1 Extended Commands +--------------------- + +There are many new commands added to TH1 and used to access the special +features of Fossil. The following is a summary of the extended commands: + + * [anoncap](#anoncap) + * [anycap](#anycap) + * [artifact](#artifact) + * [builtin_request_js](#bireqjs) + * [capexpr](#capexpr) + * [captureTh1](#captureTh1) + * [cgiHeaderLine](#cgiHeaderLine) + * [checkout](#checkout) + * [combobox](#combobox) + * [copybtn](#copybtn) + * [date](#date) + * [decorate](#decorate) + * [defHeader](#defHeader) + * [dir](#dir) + * [enable\_output](#enable_output) + * [encode64](#encode64) + * [getParameter](#getParameter) + * [glob\_match](#glob_match) + * [globalState](#globalState) + * [hascap](#hascap) + * [hasfeature](#hasfeature) + * [html](#html) + * [htmlize](#htmlize) + * [http](#http) + * [httpize](#httpize) + * [insertCsrf](#insertCsrf) + * [linecount](#linecount) + * [markdown](#markdown) + * [nonce](#nonce) + * [puts](#puts) + * [query](#query) + * [randhex](#randhex) + * [redirect](#redirect) + * [regexp](#regexp) + * [reinitialize](#reinitialize) + * [render](#render) + * [repository](#repository) + * [searchable](#searchable) + * [setParameter](#setParameter) + * [setting](#setting) + * [stime](#stime) + * [styleHeader](#styleHeader) + * [styleFooter](#styleFooter) + * [styleScript](#styleScript) + * [submenu](#submenu) + * [taint](#taintCmd) + * [tclEval](#tclEval) + * [tclExpr](#tclExpr) + * [tclInvoke](#tclInvoke) + * [tclIsSafe](#tclIsSafe) + * [tclMakeSafe](#tclMakeSafe) + * [tclReady](#tclReady) + * [trace](#trace) + * [untaint](#untaintCmd) + * [unversioned content](#unversioned_content) + * [unversioned list](#unversioned_list) + * [utime](#utime) + * [verifyCsrf](#verifyCsrf) + * [verifyLogin](#verifyLogin) + * [wiki](#wiki) + * [wiki_assoc](#wiki_assoc) + +Each of the commands above is documented by a block comment above their +implementation in the th\_main.c or th\_tcl.c source files. + +All commands starting with "tcl", with the exception of "tclReady", +require the Tcl integration subsystem be included at compile-time. +Additionally, the "tcl" repository setting must be enabled at runtime +in order to successfully make use of these commands. + +<a id="anoncap"></a>TH1 anoncap Command +----------------------------------------- + +Deprecated: prefer [capexpr](#capexpr) instead. + + * anoncap STRING... + +Returns true if the anonymous user has all of the capabilities listed +in STRING. + +<a id="anycap"></a>TH1 anycap Command +--------------------------------------- + +Deprecated: prefer [capexpr](#capexpr) instead. + + * anycap STRING + +Returns true if the current user user has any one of the capabilities +listed in STRING. + +<a id="artifact"></a>TH1 artifact Command +------------------------------------------- + + * artifact ID ?FILENAME? + +Attempts to locate the specified artifact and return its contents. An +error is generated if the repository is not open or the artifact cannot +be found. + + +<a id="bireqjs"></a>TH1 builtin_request_js Command +-------------------------------------------------- + + * builtin_request_js NAME + +NAME must be the name of one of the +[built-in javascript source files](/dir?ci=trunk&type=flat&name=src&re=js$). +This command causes that javascript file to be appended to the delivered +document. + + + +<a id="capexpr"></a>TH1 capexpr Command +----------------------------------------------------- + + * capexpr CAPABILITY-EXPR + +The capability expression is a list. Each term of the list is a +cluster of [capability letters](./caps/ref.html). +The overall expression is true if any +one term is true. A single term is true if all letters within that +term are true. Or, if the term begins with "!", then the term is true +if none of the terms are true. Or, if the term begins with "@" then +the term is true if all of the capability letters in that term are +available to the "anonymous" user. Or, if the term is "*" then it is +always true. + +Examples: + +``` +capexpr {j o r} True if any one of j, o, or r are available +capexpr {oh} True if both o and h are available +capexpr {@2 @3 4 5 6} 2 or 3 available for anonymous or one of + 4, 5 or 6 is available for the user +capexpr L True if the user is logged in +capexpr !L True if the user is not logged in +``` + +The `L` pseudo-capability is intended only to be used on its own or with +the `!` prefix for implementing login/logout menus via the `mainmenu` +site configuration option: + +``` +Login /login !L {} +Logout /logout L {} +``` + +i.e. if the user is logged in, show the "Logout" link, else show the +"Login" link. + +<a id="captureTh1"></a>TH1 captureTh1 Command +----------------------------------------------------- + + * captureTh1 STRING + +Executes its single argument as TH1 code and captures any +TH1-generated output as a string, which becomes the result of the +function call. e.g. any `puts` calls made from that block will not +generate any output, and instead their output will become part of the +result string. + + +<a id="cgiHeaderLine"></a>TH1 cgiHeaderLine Command +----------------------------------------------------- + + * cgiHeaderLine line + +Adds the specified line to the CGI header. + +<a id="checkout"></a>TH1 checkout Command +------------------------------------------- + + * checkout ?BOOLEAN? + +Return the fully qualified directory name of the current checkout or an +empty string if it is not available. Optionally, it will attempt to find +the current checkout, opening the configuration ("user") database and the +repository as necessary, if the boolean argument is non-zero. + +<a id="combobox"></a>TH1 combobox Command +------------------------------------------- + + * combobox NAME TEXT-LIST NUMLINES + +Generates and emits an HTML combobox. NAME is both the name of the +CGI parameter and the name of a variable that contains the currently +selected value. TEXT-LIST is a list of possible values for the +combobox. NUMLINES is 1 for a true combobox. If NUMLINES is greater +than one then the display is a listbox with the number of lines given. + +<a id="copybtn"></a>TH1 copybtn Command +----------------------------------------- + + * copybtn TARGETID FLIPPED TEXT ?COPYLENGTH? + +Output TEXT with a click-to-copy button next to it. Loads the copybtn.js +Javascript module, and generates HTML elements with the following IDs: + + * TARGETID: The `<span>` wrapper around TEXT. + * copy-TARGETID: The `<span>` for the copy button. + +If the FLIPPED argument is non-zero, the copy button is displayed after TEXT. + +The optional COPYLENGTH argument defines the length of the substring of TEXT +copied to clipboard: + + * <= 0: No limit (default if the argument is omitted). + * >= 3: Truncate TEXT after COPYLENGTH (single-byte) characters. + * 1: Use the "hash-digits" setting as the limit. + * 2: Use the length appropriate for URLs as the limit (defined at + compile-time by `FOSSIL_HASH_DIGITS_URL`, defaults to 16). + +<a id="date"></a>TH1 date Command +----------------------------------- + + * date ?-local? + +Return a string which is the current time and date. If the -local +option is used, the date appears using localtime instead of UTC. + +<a id="decorate"></a>TH1 decorate Command +------------------------------------------- + + * decorate STRING + +Renders STRING as wiki content; however, only links are handled. No +other markup is processed. + +<a id="defHeader"></a>TH1 defHeader Command +--------------------------------------------- + + * defHeader + +Returns the default page header. + +<a id="dir"></a>TH1 dir Command +--------------------------------- + + * dir CHECKIN ?GLOB? ?DETAILS? + +Returns a list containing all files in CHECKIN. If GLOB is given only +the files matching the pattern GLOB within CHECKIN will be returned. +If DETAILS is non-zero, the result will be a list-of-lists, with each +element containing at least three elements: the file name, the file +size (in bytes), and the file last modification time (relative to the +time zone configured for the repository). + +<a id="enable_output"></a>TH1 enable\_output Command +------------------------------------------------------ + + * enable\_output BOOLEAN + +Enable or disable sending output when the combobox, copybtn, puts, or wiki +commands are used. + +<a id="encode64"></a>TH1 encode64 Command +------------------------------------------- + + * encode64 STRING + +Encode the specified string using Base64 and return the result. + +<a id="getParameter"></a>TH1 getParameter Command +--------------------------------------------------- + + * getParameter NAME ?DEFAULT? + +Returns the value of the specified query parameter or the specified +default value when there is no matching query parameter. + +<a id="glob_match"></a>TH1 glob\_match Command +------------------------------------------------ + + * glob\_match ?-one? ?--? patternList string + +Checks the string against the specified glob pattern -OR- list of glob +patterns and returns non-zero if there is a match. + +<a id="globalState"></a>TH1 globalState Command +------------------------------------------------- + + * globalState NAME ?DEFAULT? + +Returns a string containing the value of the specified global state +variable -OR- the specified default value. The supported items are: + + 1. **checkout** -- _Active local checkout directory, if any._ + 1. **configuration** -- _Active configuration database file name, if any._ + 1. **executable** -- _Fully qualified executable file name._ + 1. **flags** -- _TH1 initialization flags._ + 1. **log** -- _Error log file name, if any._ + 1. **repository** -- _Active local repository file name, if any._ + 1. **top** -- _Base path for the active server instance, if applicable._ + 1. **user** -- _Active user name, if any._ + 1. **vfs** -- _SQLite VFS in use, if overridden._ + +Attempts to query for unsupported global state variables will result +in a script error. Additional global state variables may be exposed +in the future. + +<a id="hascap"></a>TH1 hascap Command +--------------------------------------- + +Deprecated: prefer [capexpr](#capexpr) instead. + + * hascap STRING... + +Returns true if the current user has all of the capabilities listed +in STRING. + +<a id="hasfeature"></a>TH1 hasfeature Command +----------------------------------------------- + + * hasfeature STRING + +Returns true if the binary has the given compile-time feature enabled. +The possible features are: + + 1. **ssl** -- _Support for the HTTPS transport._ + 1. **legacyMvRm** -- _Support for legacy mv/rm command behavior._ + 1. **execRelPaths** -- _Use relative paths with external diff/gdiff._ + 1. **th1Docs** -- _Support for TH1 in embedded documentation._ + 1. **th1Hooks** -- _Support for TH1 command and web page hooks._ + 1. **tcl** -- _Support for Tcl integration._ + 1. **useTclStubs** -- _Tcl stubs enabled in the Tcl headers._ + 1. **tclStubs** -- _Uses Tcl stubs (i.e. linking with stubs library)._ + 1. **tclPrivateStubs** -- _Uses Tcl private stubs (i.e. header-only)._ + 1. **json** -- _Support for the JSON APIs._ + 1. **markdown** -- _Support for Markdown documentation format._ + 1. **unicodeCmdLine** -- _The command line arguments are Unicode._ + 1. **dynamicBuild** -- _Dynamically linked to libraries._ + 1. **mman** -- _Uses POSIX memory APIs from "sys/mman.h"._ + 1. **see** -- _Uses the SQLite Encryption Extension._ + +Specifying an unknown feature will return a value of false, it will not +raise a script error. + +<a id="html"></a>TH1 html Command +----------------------------------- + + * html STRING + +Outputs the STRING literally. It is assumed that STRING contains +valid HTML, or that if STRING contains any characters that are +significant to HTML (such as `<`, `>`, `'`, or `&`) have already +been escaped, perhaps by the [htmlize](#htmlize) command. Use the +[puts](#puts) command to output text that might contain unescaped +HTML markup. + +**Beware of XSS attacks!** If the STRING value to the html command +can be controlled by a hostile user, then he might be able to sneak +in malicious HTML or Javascript which could result in a +cross-site scripting (XSS) attack. Be careful that all text that +in STRING that might come from user input has been sanitized by the +[htmlize](#htmlize) command or similar. In recent versions of Fossil, +the STRING value must be [untainted](#taint) or else the "html" command +will fail. + +<a id="htmlize"></a>TH1 htmlize Command +----------------------------------------- + + * htmlize STRING + +Escape all characters of STRING which have special meaning in HTML. +Returns the escaped string. + +<a id="http"></a>TH1 http Command +----------------------------------- + + * http ?-asynchronous? ?--? url ?payload? + +Performs an HTTP or HTTPS request for the specified URL. If a +payload is present, it will be interpreted as text/plain and +the POST method will be used; otherwise, the GET method will +be used. Upon success, if the -asynchronous option is used, an +empty string is returned as the result; otherwise, the response +from the server is returned as the result. Synchronous requests +are not currently implemented. + +<a id="httpize"></a>TH1 httpize Command +----------------------------------------- + + * httpize STRING + +Escape all characters of STRING which have special meaning in URI +components. Returns the escaped string. + +<a id="insertCsrf"></a>TH1 insertCsrf Command +----------------------------------------------- + + * insertCsrf + +While rendering a form, call this command to add the Anti-CSRF token +as a hidden element of the form. + +<a id="linecount"></a>TH1 linecount Command +--------------------------------------------- + + * linecount STRING MAX MIN + +Returns one more than the number of `\n` characters in STRING. But +never returns less than MIN or more than MAX. + +<a id="markdown"></a>TH1 markdown Command +------------------------------------------- + + * markdown STRING + +Renders the input string as markdown. The result is a two-element list. +The first element contains the body, rendered as HTML. The second element +is the text-only title string. + +<a id="nonce"></a>TH1 nonce Command +------------------------------------- + + * nonce + +Returns the value of the cryptographic nonce for the request being processed. + +<a id="puts"></a>TH1 puts Command +----------------------------------- + + * puts STRING + +Outputs STRING. Characters within STRING that have special meaning +in HTML are escaped prior to being output. Thus is it safe for STRING +to be derived from user inputs. See also the [html](#html) command +which behaves similarly except does not escape HTML markup. This +command ("puts") is safe to use on [tainted strings](#taint), but the "html" +command is not. + +<a id="query"></a>TH1 query Command +------------------------------------- + + * query ?-nocomplain? SQL CODE + +Runs the SQL query given by the SQL argument. For each row in the result +set, run CODE. + +In SQL, parameters such as $var are filled in using the value of variable +"var". Result values are stored in variables with the column name prior +to each invocation of CODE. The names of the variables in which results +are stored can be controlled using "AS name" clauses in the SQL. As +the database will often contain content that originates from untrusted +users, all result values are marked as [tainted](#taint). + +**Beware of SQL injections in the `query` command!** +The SQL argument to the query command should always be literal SQL +text enclosed in {...}. The SQL argument should never be a double-quoted +string or the value of a \$variable, as those constructs can lead to +an SQL Injection attack. If you need to include the values of one or +more TH1 variables as part of the SQL, then put \$variable inside the +{...}. The \$variable keyword will then get passed down into the SQLite +parser which knows to look up the value of \$variable in the TH1 symbol +table. For example: + +~~~ + query {SELECT res FROM tab1 WHERE key=$mykey} {...} +~~~ + +SQLite will see the \$mykey token in the SQL and will know to resolve it +to the value of the "mykey" TH1 variable, safely and without the possibility +of SQL injection. The following is unsafe: + +~~~ + query "SELECT res FROM tab1 WHERE key='$mykey'" {...} ;# <-- UNSAFE! +~~~ + +In this second example, TH1 does the expansion of `$mykey` prior to passing +the text down into SQLite. So if `$mykey` contains a single-quote character, +followed by additional hostile text, that will result in an SQL injection. + +To help guard against SQL-injections, recent versions of Fossil require +that the SQL argument be [untainted](#taint) or else the "query" command +will fail. + +<a id="randhex"></a>TH1 randhex Command +----------------------------------------- + + * randhex N + +Returns a string of N*2 random hexadecimal digits with N<50. If N is +omitted, use a value of 10. + +<a id="redirect"></a>TH1 redirect Command +------------------------------------------- + + * redirect URL ?withMethod? + +Issues an HTTP redirect to the specified URL and then exits the process. +By default, an HTTP status code of 302 is used. If the optional withMethod +argument is present and non-zero, an HTTP status code of 307 is used, which +should force the user agent to preserve the original method for the request +(e.g. GET, POST) instead of (possibly) forcing the user agent to change the +method to GET. + +<a id="regexp"></a>TH1 regexp Command +--------------------------------------- + + * regexp ?-nocase? ?--? exp string + +Checks the string against the specified regular expression and returns +non-zero if it matches. If the regular expression is invalid or cannot +be compiled, an error will be generated. + +See [fossil grep](./grep.md) for details on the regexp syntax. + +<a id="reinitialize"></a>TH1 reinitialize Command +--------------------------------------------------- + + * reinitialize ?FLAGS? + +Reinitializes the TH1 interpreter using the specified flags. + +<a id="render"></a>TH1 render Command +--------------------------------------- + + * render STRING + +Renders the TH1 template and writes the results. + +<a id="repository"></a>TH1 repository Command +----------------------------------------------- + + * repository ?BOOLEAN? + +Returns the fully qualified file name of the open repository or an empty +string if one is not currently open. Optionally, it will attempt to open +the repository if the boolean argument is non-zero. + +<a id="searchable"></a>TH1 searchable Command +----------------------------------------------- + + * searchable STRING... + +Return true if searching in any of the document classes identified +by STRING is enabled for the repository and user has the necessary +capabilities to perform the search. The possible document classes +are: + + 1. **c** -- _Check-in comments_ + 1. **d** -- _Embedded documentation_ + 1. **t** -- _Tickets_ + 1. **w** -- _Wiki_ + +To be clear, only one of the document classes identified by each STRING +needs to be searchable in order for that argument to be true. But all +arguments must be true for this routine to return true. Hence, to see +if ALL document classes are searchable: + + if {[searchable c d t w]} {...} + +But to see if ANY document class is searchable: + + if {[searchable cdtw]} {...} + +This command is useful for enabling or disabling a "Search" entry on the +menu bar. + +<a id="setParameter"></a>TH1 setParameter Command +--------------------------------------------------- + + * setParameter NAME VALUE + +Sets the value of the specified query parameter. + +<a id="setting"></a>TH1 setting Command +----------------------------------------- + + * setting name + +Gets and returns the value of the specified setting. + +<a id="stime"></a>TH1 stime Command +------------------------------------- + + * stime + +Returns the number of microseconds of CPU time consumed by the current +process in system space. + +<a id="styleHeader"></a>TH1 styleHeader Command +------------------------------------------------- + + * styleHeader TITLE + +Render the configured style header for the selected skin. + +<a id="styleFooter"></a>TH1 styleFooter Command +------------------------------------------------- + + * styleFooter + +Render the configured style footer for the selected skin. + +<a id="styleScript"></a>TH1 styleScript Command +------------------------------------------------- + + * styleScript + +Render the configured JavaScript for the selected skin. + +<a id="submenu"></a>TH1 submenu Command +----------------------------------------- + + * submenu link LABEL URL + +Add hyperlink to the submenu of the current page. + +<a id="taintCmd"></a>TH1 taint Command +----------------------------------------- + + * taint STRING + +This command returns a copy of STRING that has been marked as +[tainted](#taint). Tainted strings are strings which might be +controlled by an attacker and might contain hostile inputs and +are thus unsafe to use in certain contexts. For example, tainted +strings should not be output as part of a webpage as they might +contain rogue HTML or Javascript that could lead to an XSS +vulnerability. Similarly, tainted strings should not be run as +SQL since they might contain an SQL-injection vulerability. + +<a id="tclEval"></a>TH1 tclEval Command +----------------------------------------- + +**This command requires the Tcl integration feature.** + + * tclEval arg ?arg ...? + +Evaluates the Tcl script and returns its result verbatim. If a Tcl script +error is generated, it will be transformed into a TH1 script error. The +Tcl interpreter will be created automatically if it has not been already. + +<a id="tclExpr"></a>TH1 tclExpr Command +----------------------------------------- + +**This command requires the Tcl integration feature.** + + * tclExpr arg ?arg ...? + +Evaluates the Tcl expression and returns its result verbatim. If a Tcl +script error is generated, it will be transformed into a TH1 script +error. The Tcl interpreter will be created automatically if it has not +been already. + +<a id="tclInvoke"></a>TH1 tclInvoke Command +--------------------------------------------- + +**This command requires the Tcl integration feature.** + + * tclInvoke command ?arg ...? + +Invokes the Tcl command using the supplied arguments. No additional +substitutions are performed on the arguments. The Tcl interpreter +will be created automatically if it has not been already. + +<a id="tclIsSafe"></a>TH1 tclIsSafe Command +--------------------------------------------- + +**This command requires the Tcl integration feature.** + + * tclIsSafe + +Returns non-zero if the Tcl interpreter is "safe". The Tcl interpreter +will be created automatically if it has not been already. + +<a id="tclMakeSafe"></a>TH1 tclMakeSafe Command +------------------------------------------------- + +**This command requires the Tcl integration feature.** + + * tclMakeSafe + +Forces the Tcl interpreter into "safe" mode by removing all "unsafe" +commands and variables. This operation cannot be undone. The Tcl +interpreter will remain "safe" until the process terminates. The Tcl +interpreter will be created automatically if it has not been already. + +<a id="tclReady"></a>TH1 tclReady Command +------------------------------------------- + + * tclReady + +Returns true if the binary has the Tcl integration feature enabled and it +is currently available for use by TH1 scripts. + +<a id="trace"></a>TH1 trace Command +------------------------------------- + + * trace STRING + +Generates a TH1 trace message if TH1 tracing is enabled. + +<a id="untaintCmd"></a>TH1 untaint Command +------------------------------------------ + + * untaint STRING + +This command returns a copy of STRING that has been marked as +[untainted](#taint). Untainted strings are strings which are +believed to be free of potentially hostile content. Use this +command with caution, as it overwrites the tainted-string protection +mechanisms that are built into TH1. If you do not understand all +the implications of executing this command, then do not use it. + +<a id="unversioned_content"></a>TH1 unversioned content Command +----------------------------------------------------------------- + + * unversioned content FILENAME + +Attempts to locate the specified unversioned file and return its contents. +An error is generated if the repository is not open or the unversioned file +cannot be found. + +<a id="unversioned_list"></a>TH1 unversioned list Command +----------------------------------------------------------- + + * unversioned list + +Returns a list of the names of all unversioned files held in the local +repository. An error is generated if the repository is not open. + +<a id="utime"></a>TH1 utime Command +------------------------------------- + + * utime + +Returns the number of microseconds of CPU time consumed by the current +process in user space. + +<a id="verifyCsrf"></a>TH1 verifyCsrf Command +----------------------------------------------- + + * verifyCsrf + +Before using the results of a form, first call this command to verify +that the Anti-CSRF token is present and is valid. If the Anti-CSRF token +is missing or is incorrect, that indicates a cross-site scripting attack. +If the event of an attack is detected, an error message is generated and +all further processing is aborted. + +<a id="verifyLogin"></a>TH1 verifyLogin Command +------------------------------------------------- + + * verifyLogin + +Returns non-zero if the specified user name and password represent a +valid login for the repository. + +<a id="wiki"></a>TH1 wiki Command +----------------------------------- + + * wiki STRING + +Renders STRING as wiki content. + +<a id="wiki_assoc"></a>TH1 wiki_assoc Command +----------------------------------- + + * wiki_assoc STRING STRING + +Renders the special wiki. The first string refers to the namespace +(checkin, branch, tag, ticket). The second string specifies the +concrete wiki page to be rendered. + +Tcl Integration Commands +------------------------ + +When the Tcl integration subsystem is enabled, several commands are added +to the Tcl interpreter. They are used to allow Tcl scripts access to the +Fossil functionality provided via TH1. The following is a summary of the +Tcl commands: + + * th1Eval + * th1Expr + +<a id="th1Eval"></a>Tcl th1Eval Command +----------------------------------------- + +**This command requires the Tcl integration feature.** + + * th1Eval arg + +Evaluates the TH1 script and returns its result verbatim. If a TH1 script +error is generated, it will be transformed into a Tcl script error. + +<a id="th1Expr"></a>Tcl th1Expr Command +----------------------------------------- + +**This command requires the Tcl integration feature.** + + * th1Expr arg + +Evaluates the TH1 expression and returns its result verbatim. If a TH1 +script error is generated, it will be transformed into a Tcl script error. diff --git a/www/theory1.wiki b/www/theory1.wiki new file mode 100644 index 0000000..b6876f9 --- /dev/null +++ b/www/theory1.wiki @@ -0,0 +1,142 @@ +<title>Thoughts On The Design Of The Fossil DVCS</title> + +Two questions (or criticisms) that arise frequently regarding Fossil +can be summarized as follows: + + 1. Why is Fossil based on SQLite instead of a distributed NoSQL database? + + 2. Why is Fossil written in C instead of a modern high-level language? + +Neither question can be answered directly because they are both +based on false assumptions. We claim that Fossil is not based on SQLite +at all and that Fossil is not based on a distributed NoSQL database +because Fossil is a distributed NoSQL database. And, Fossil does use +a modern high-level language for its implementation, namely SQL. + +<h2>Fossil Is A NoSQL Database</h2> + +We begin with the first question: Fossil is not based on a distributed +NoSQL database because Fossil <u><i>is</i></u> a distributed NoSQL database. +Fossil is <u>not</u> based on SQLite. +The current implementation of Fossil uses +SQLite as a local store for the content of the distributed database and as +a cache for meta-information about the distributed database that is precomputed +for quick and easy presentation. But the use of SQLite in this role is an +implementation detail and is not fundamental to the design. Some future +version of Fossil might do away with SQLite and substitute a pile-of-files or +a key/value database in place of SQLite. +(Actually, that is very unlikely +to happen since SQLite works amazingly well in its current role, but the point +is that omitting SQLite from Fossil is a theoretical possibility.) + +The underlying database that Fossil implements has nothing to do with +SQLite, or SQL, or even relational database theory. The underlying +database is very simple: it is an unordered collection of "artifacts". +An artifact is a list of bytes - a "file" in the usual manner of thinking. +Many artifacts are simply the content of source files that have +been checked into the Fossil repository. Call these "content artifacts". +Other artifacts, known as +"control artifacts", contain ASCII text in a particular format that +defines relationships between other artifacts, such as which +content artifacts that go together to form a particular version of the +project. Each artifact is named by its SHA1 or SHA3-256 hash and is +thus immutable. +Artifacts can be added to the database but not removed (if we ignore +the exceptional case of [./shunning.wiki | shunning].) Repositories +synchronize by computing the union of their artifact sets. SQL and +relation theory play no role in any of this. + +SQL enters the picture only in the implementation details. The current +implementation of Fossil stores each artifact as a BLOB in an SQLite +database. +The current implementation also parses up each control artifact as it +arrives and stores the information discovered from that parse in various +other SQLite tables to facilitate rapid generation of reports such as +timelines, file histories, file lists, branch lists, and so forth. Note +that all of this additional information is derived from the artifacts. +The artifacts are canonical. The relational tables serve only as a cache. +Everything in the relational tables can be recomputed +from the artifacts, and in fact that is exactly what happens when one runs +the "fossil rebuild" command on a repository. + +So really, Fossil works with two separate databases. There is the +bag-of-artifacts database which is non-relational and distributed (like +a NoSQL database) and there is the local relational database. The +bag-of-artifacts database has a fixed format and is what defines a Fossil +repository. Fossil will never modify the file format of the bag-of-artifacts +database in an incompatible way because to do so would be to make something +that is no longer "Fossil". The local relational database, on the other hand, +is a cache that contains information derived from the bag-of-artifacts. +The schema of the local relational database changes from time to time as +the Fossil implementation is enhanced, and the content is recomputed from +the unchanging bag of artifacts. The local relational database is an +implementation detail which currently happens to use SQLite. + +Another way to think of the relational tables in a Fossil repository is +as an index for the artifacts. Without the relational tables, +to generate a report like a timeline would require scanning every artifact - +the equivalent of a full table scan. The relational tables hold pointers to +the relevant artifacts in presorted order so that generating a timeline +is much more efficient. So like an index in a relational database, the +relational tables in a Fossil repository do not add any new information, +they merely make the information in the artifacts faster and easier to +look up. + +Fossil is not "based" on SQLite. Fossil simply exploits SQLite as +a powerful tool to make the implementation easier. +And Fossil doesn't use a distributed +NoSQL database because Fossil is a distributed NoSQL database. That answers +the first question. + +<h2>SQL Is A High-Level Scripting Language</h2> + +The second concern states that Fossil does not use a high-level scripting +language. But that is not true. Fossil uses SQL (as implemented by SQLite) +as its scripting language. + +This misunderstanding likely arises because people fail +to appreciate that SQL is a programming language. People are taught that SQL +is a "query language" as if that were somehow different from a +"programming language". But they really are two different flavors of the +same thing. I find that people do better with SQL if they think of +SQL as a programming language and each statement +of SQL is a separate program. SQL is a peculiar programming language +in that one uses SQL to specify <i>what</i> to compute whereas in +most other programming languages one specifies <i>how</i> +to carry out the computation. +This difference means that SQL +is an extraordinary high-level programming language, but it is still +just a programming language. + +For certain types of problems, SQL has a huge advantage over other +programming languages because it is so high level and because it allows +programmers to focus more on the <i>what</i> and less on the <i>how</i> +of a computation. In other words, +programmers tend to think about problems at a much higher level when +using SQL; this can result in better applications. +SQL is also very dense. +In practice, this often means that a few +lines of SQL can often replace hundreds or thousands of lines of +procedural code, with a corresponding decrease in programming effort +and opportunities to introduce bugs. +Fossil happens to be one of those problems for which SQL is well suited. + +Much of the "heavy lifting" within the Fossil implementation is carried +out using SQL statements. It is true that these SQL statements are glued +together with C code, but it turns out that C works surprisingly well in +that role. Several early prototypes of Fossil were written in a scripting +language (TCL). We normally find that TCL programs are shorter than the +equivalent C code by a factor of 10 or more. But in the case of Fossil, +the use of TCL was actually making the code longer and more difficult to +understand. +And so in the final design, we switched from TCL to C in order to make +the code easier to implement and debug. + +Without the advantages of having SQLite built in, the design might well +have followed a different path. Most reports generated by Fossil involve +a complex set of queries against the relational tables of the repository +database. These queries are normally implemented in only a few dozen +lines of SQL code. But if those queries had been implemented procedurally +using a key/value or pile-of-files database, it +may have well been the case that a high-level scripting language such as +Tcl, Python, or Ruby may have worked out better than C. diff --git a/www/tickets.wiki b/www/tickets.wiki new file mode 100644 index 0000000..76b5071 --- /dev/null +++ b/www/tickets.wiki @@ -0,0 +1,199 @@ +<title>The Fossil Ticket System</title> + +<h2>1.0 File Format</h2> + +At its lowest level, the tickets of Fossil consist solely of +[./fileformat.wiki#tktchng | ticket change artifacts]. +Each ticket change artifact corresponds to a single change +to a ticket. The act of creating a ticket is considered a +change. + +Each ticket change artifact contains the following information: + +<ul> +<li>The ID of the ticket that was changed +<li>The time stamp for when the change occurred +<li>The user who made the change +<li>A list of key/value pairs that show what changed in the ticket +</ul> + +To determine the current state of a particular ticket, Fossil orders +the change artifacts for that ticket from oldest to most recent, +then applies each change in time stamp order. + +On each change artifact, there are one or more key/value pairs that +implement the change. The key corresponds to a field of the ticket +that is modified. The value may either replace the earlier value for +that key, or the value may be appended to the prior value. + +<h2>2.0 Ticket Tables</h2> + +The low-level artifact format for ticket content is tedious and +cumbersome to access in real time. To facilitate reporting and display +of tickets, the low-level artifact information is collected and +summarized in a pair of SQL tables in each local repository. Display +and reporting of tickets is accomplished by querying these two tables. + +Note that only the low-level ticket change artifacts are synced. The +content of the two ticket tables can always be reconstructed from the +ticket change artifacts. And, indeed, the reconstruction of the ticket +tables from low-level artifacts happens automatically whenever new +ticket change artifacts are received by the system. The important point +to remember is that display of tickets is accomplished using SQL tables +but that recording and syncing of ticket information is accomplished using +ticket change artifacts. + +<h3>2.1 Ticket Table Schema</h3> + +The two ticket tables are called TICKET and TICKETCHNG. +The default schema (as of this writing) for these two tables is shown +below: + +<verbatim> +CREATE TABLE ticket( + -- Do not change any column that begins with tkt_ + tkt_id INTEGER PRIMARY KEY, + tkt_uuid TEXT UNIQUE, + tkt_mtime DATE, + tkt_ctime DATE, + -- Add as many fields as required below this line + type TEXT, + status TEXT, + subsystem TEXT, + priority TEXT, + severity TEXT, + foundin TEXT, + private_contact TEXT, + resolution TEXT, + title TEXT, + comment TEXT +); +CREATE TABLE ticketchng( + -- Do not change any column that begins with tkt_ + tkt_id INTEGER REFERENCES ticket, + tkt_rid INTEGER REFERENCES blob, + tkt_mtime DATE, + -- Add as many fields as required below this line + login TEXT, + username TEXT, + mimetype TEXT, + icomment TEXT +); +CREATE INDEX ticketchng_idx1 ON ticketchng(tkt_id, tkt_mtime); +</verbatim> + +Generally speaking, there is one row in the TICKETCHNG table for each +change to each ticket. In other words, there is one row in the +TICKETCHNG table for each low-level ticket change artifact. The +TICKET table, on the other hand, contains a summary of the current +status of each ticket. + +Fields of the TICKET and TICKETCHNG tables that begin with "tkt_" are +used internally by Fossil. The logic inside of Fossil that converts +ticket change artifacts into row data for the two ticket tables expects +the "tkt_" fields to always be present. All of the other fields of the +TICKET and TICKETCHNG tables are "user defined" in the sense that they +can be anything the administrator of the system wants them to be. The +user-defined fields should correspond to keys in the key/value pairs of +the ticket change artifacts. + +The <b>tkt_id</b> fields of TICKET and TICKETCHNG are an integer key +used to uniquely identify the ticket to which the row belongs. These +keys are for internal use only and may change when doing a "fossil rebuild". + +The <b>tkt_uuid</b> field is the unique hexadecimal identifier for the ticket. +Ticket identifiers appear to be SHA1 hash strings, but they +are not really the hash of any identifiable artifact. They are +just random hexadecimal numbers. When creating a new ticket, Fossil uses +a (high-quality) pseudo-random number generator to create the ticket +number. The ticket numbers are large so that the chance of collision +between any two tickets is vanishingly small. + +The <b>tkt_mtime</b> field of TICKET shows the time (as a Julian day number) +of the most recent ticket change artifact for that ticket. The +<b>tkt_mtime</b> field of TICKETCHNG shows the timestamp on the ticket +change artifact that the TICKETCHNG row refers to. The +<b>tkt_ctime</b> field of TICKET is the time of the oldest ticket change +artifact for that ticket, thus holding the time that the ticket was +created. + +The <b>tkt_rid</b> field of TICKETCHNG is the integer primary key in the +BLOB table of the ticket change artifact that gave rise to the row in the +TICKETCHNG table. + +All the other fields of the TICKET and TICKETCHNG tables are available +for customization for individual projects. None of the remaining fields +are required, but all of them are needed in order to use the default +ticket creating, viewing, and editing scripts. It is recommended that +the other fields be retained and that customizations be restricted to +adding new fields above and beyond the default. + +<h3>2.2 Translating Artifacts To Tables</h3> + +Each row in the TICKETCHNG table corresponds to a single ticket change +artifact. The tkt_id field is the integer primary key of the TICKET +table entry for the corresponding ticket. The tkt_rid field is the +integer primary key for the BLOB table entry that contains the low-level +artifact text. The tkt_mtime field is the time stamp on the ticket +change artifact, expressed as a Julian day number. If the ticket +change artifact contains a key/value pair where the key is "login", +then the corresponding value is stored in the login field of the +TICKETCHNG table. The same it true for "username", "mimetype", and +"icomment" fields. Any time there is a key/value pair in the ticket +change artifact and the key corresponds to the name of a field in the +TICKETCHNG table, then the value of that key/value pair is stored in +the TICKETCHNG table. If the TICKETCHNG table has a field for which there +is no corresponding key/value pair in the artifact, then that field of +the TICKETCHNG table is NULL. If there are key/value pairs in the +artifact that have no corresponding field in the TICKETCHNG table, those +key/value pairs are silently ignored. + +Each row in the TICKET table records the overall status of a ticket. +The tkt_id field is a unique integer primary key for the ticket. +the tkt_uuid field is the global ticket identifier - a larger random +hexadecimal constant. The tkt_mtime and tkt_ctime fields hold the +times of the most recent and the oldest ticket change artifacts for +this ticket, respectively. + +To reconstruct the TICKET table, the ticket change +artifacts are visited in time stamp order. As each ticket change artifact is +visited, its key/value pairs are examined. For any key/value pair in +which the key is the same as a field in the TICKET table, the value +of that pair either replaces or is appended to the previous value +of the corresponding field in the TICKET table. Whether a value is +replaced or appended is determined by markings in the ticket change +artifact itself. Most fields are usually replaced. (For example, to change +the status from "Open" to "Fixed" would involve a key value pair +"status/Fixed" with the replace attribute set). The main exception +is the "comment" field, which is usually appended with new comment +text. + +Note that the replace-or-append mark on ticket change artifacts is +only used by the TICKET table. Since the initial value of all fields +in the TICKETCHNG table is NULL, the replace-or-append mark makes no +difference there. + +<h3>2.3 Old-Style versus New-Style Tickets</h3> + +Older versions of Fossil +(before [/timeline?c=2012-11-27T16:26:29 | 2012-11-27]) +only supported the TICKET table. +In this older style, new comments were added to tickets by using +the append-value feature on the comment field. Thus the TICKET.COMMENT +field contains the complete text of all user comments already appended +together and ready for display. + +A problem with the old approach is that all comment text had to +be in the same format. In other words, the all comment text had to be +either plaintext or wiki or HTML. It was not possible for some comments +to be in HTML and others to be plaintext. Some site administrators wanted the +ability to mix plaintext, wiki, and HTML comments and display each +comment according to its chosen format. Hence, Fossil was enhanced to +support the "new-style" tickets. + +The TICKETCHNG table was added to support new-style tickets. In the new +style, comment text is stored with the "icomment" (for "Incremental Comment") +key and appears separately, and with its own mimetype, in multiple rows +of the TICKETCHNG table. It then falls to the TH1 script code on the +View Ticket Page to query the TICKETCHNG table and extract and format +the various comments in time stamp order. diff --git a/www/title-test.md b/www/title-test.md new file mode 100644 index 0000000..5ea13d7 --- /dev/null +++ b/www/title-test.md @@ -0,0 +1,12 @@ +# Markdown Doc Title > & " ' < Test + +Test of unusual characters in the title of Markdown formatted documents. +The title should read: + +> Markdown Doc Title > & " ' < Test + +See also: + + * [](/doc/trunk/www/title-test.wiki) + * [](/wiki?name=Test+Wiki+>+%26+%22+%27+%3c+Title&p) + * [](/forumpost/481ab1f9) diff --git a/www/title-test.wiki b/www/title-test.wiki new file mode 100644 index 0000000..d4f8a36 --- /dev/null +++ b/www/title-test.wiki @@ -0,0 +1,14 @@ +<title>Wiki Doc Title > & " ' < Test</title> + +Test of unusual characters in the title of Fossil-wiki formatted documents. +The title should read: + +<big><b><verbatim> + Wiki Doc Title > & " ' < Test +</verbatim></b></big> + +See also: + + * [/doc/trunk/www/title-test.md] + * [/wiki?name=Test+Wiki+>+%26+%22+%27+%3c+Title&p] + * [/forumpost/481ab1f9] diff --git a/www/tls-nginx.md b/www/tls-nginx.md new file mode 100644 index 0000000..6e207b9 --- /dev/null +++ b/www/tls-nginx.md @@ -0,0 +1,3 @@ +# Proxying Fossil via HTTPS with nginx + +This document has [moved](./server/debian/nginx.md#tls). diff --git a/www/uitest.html b/www/uitest.html new file mode 100644 index 0000000..add91a8 --- /dev/null +++ b/www/uitest.html @@ -0,0 +1,129 @@ +<html> +<head> +<title>Fossil UI Test</title> +</head> +<body> +<script> + var aTest = [ +/////////////////////////////////////////////////////////////////////////// +/// Add pages to be tested below: +////////////////////////////////////////////////////////////////////////// +{ + url: "timeline", + desc: + "Simple timeline of most recent check-ins. Verify that all submenus work." +}, +{ + url: "timeline?n1=125", + desc: + "Timeline with 125 entries. Verify that submenus preserve the entry count." +}, +{ + url: "wiki", + desc: + "The wiki homepage" +} +////////////////////////////////////////////////////////////////////////////// +/// End of testing data +///////////////////////////////////////////////////////////////////////////// + ]; + var iTest = 0; + var nTest = aTest.length; + var totalTest = nTest; + var firstTest = aTest[0]; + function gebi(x){ return document.getElementById(x); } +</script> +<style type="text/css"> + a { + padding-top:5px; + padding-bottom:5px; + padding-left:10px; + padding-right:10px; + text-align: right; + color: #000; + background-color: #eef; + vertical-align:middle; + box-shadow: 0px 3px 4px #999; + border-radius: 10px; + } +</style> +<p>Test frame for the Fossil server at <span id="x1">???</span>.</p> +<ul> +<li> <span id="x2">0</span> of <span id="x3">0</span> pages checked so far. +<li> Current page: <b><span id="x5"></span></b> +<li> <span id="x6">Press "Begin" to begin testing</span> +</ul> +<a id="x-start" target="fossiltest" onclick="startTest()">Begin</a> +<a id="x-prev" target="fossiltest" onclick="prevTest()">Previous</a> +<a id="x-next" target="fossiltest" onclick="nextTest()">Next</a> +<a id="x-pass" target="fossiltest" onclick="passTest()">Test Passes</a> +<p id="x-done" style="color:green;">Testing Complete!</p> +<script> + var re = new RegExp("/doc/[^/]+/www/.*$"); + var baseURI = document.location.href.replace(re,"/"); + gebi("x1").innerHTML = '"' + baseURI + '"'; + gebi("x3").innerHTML = nTest; + var xprev = gebi("x-prev"); + var xnext = gebi("x-next"); + var xpass = gebi("x-pass"); + var xstart = gebi("x-start"); + gebi("x-done").hidden = 1; + function loadPage(){ + var x = aTest[iTest]; + gebi("x5").innerHTML = x.url; + gebi("x6").innerHTML = x.desc; + gebi("x2").innerHTML = totalTest-nTest; + xstart.hidden = 1; + xpass.hidden = 0; + if( iTest>0 ){ + var y = aTest[iTest-1]; + xprev.href = baseURI + y.url; + xprev.hidden = 0; + }else{ + xprev.hidden = 1; + } + if( iTest+1<nTest ){ + var z = aTest[iTest+1]; + xnext.href = baseURI + z.url; + xpass.href = baseURI + z.url; + xnext.hidden = 0; + }else{ + xnext.hidden = 1; + if( nTest>1 ) xpass.href = xprev.href; + } + } + gebi("x3").innerHTML = nTest; + xprev.hidden = 1; + xnext.hidden = 1; + xpass.hidden = 1; + xstart.hidden = 0; + xstart.href = baseURI + aTest[0].url; + function startTest(){ + setTimeout(loadPage,1); + } + function prevTest(){ + if( iTest<=0 ) return false; + iTest--; + setTimeout(loadPage,1); + } + function nextTest(){ + if( iTest+1>=nTest ) return false; + iTest++; + setTimeout(loadPage,1); + } + function passTest(){ + if( nTest==1 ){ + xpass.hidden = 1; + xnext.hidden = 1; + xprev.hidden = 1; + gebi("x2").innerHTML = totalTest; + gebi("x-done").hidden = 0; + return false; + }else{ + aTest.splice(iTest, 1); + nTest--; + if( iTest>=nTest ) iTest = nTest-1; + setTimeout(loadPage,1); + } + } +</script> diff --git a/www/unvers.wiki b/www/unvers.wiki new file mode 100644 index 0000000..9c581a3 --- /dev/null +++ b/www/unvers.wiki @@ -0,0 +1,115 @@ +<title>Unversioned Content</title> + +"Unversioned content" or "unversioned files" are +files stored in a Fossil repository without history, meaning +it retains the newest version of each such file, and that alone. +Unversioned files may have an optional directory name prefix. + +Though it omits history, Fossil does sync unversioned content between +repositories. In the event of a conflict during a sync, it retains +the most recent version of each unversioned file, discarding +older versions. + +Unversioned files are useful for storing ephemeral content such as +builds or frequently changing web pages. We store the +[https://fossil-scm.org/home/uv/download.html|download] page of the +self-hosting Fossil repository as unversioned content, for +example. + +<h2>Accessing Unversioned Files</h2> + +Unversioned files are <u>not</u> a part of a check-out. +Unversioned files are intended to be accessible as web pages using +URLs of the form: "<tt>https://example.com/cgi-script/<b>uv</b>/<i>FILENAME</i></tt>". +In other words, the URI method "<b>uv</b>" (short for "unversioned") +followed by the name of the unversioned file will retrieve the content +of the file. The MIME type is inferred from the filename suffix. + +The content of unversioned files can also be retrieved using the +[/help/unversioned|fossil unvers cat <i>FILENAME...</i>] +or [/help/unversioned|fossil unvers export <i>FILENAME</i>] commands. + +A list of all unversioned files on a server can be seen using +the [/help/www/uvlist|/uvlist] URL. ([/uvlist|example].) + +<h2>Syncing Unversioned Files</h2> + +Unversioned content does not sync between repositories by default. +One must request it via commands such as: + +<pre> +fossil sync <b>-u</b> +fossil clone <b>-u</b> <i>URL local-repo-name</i> +fossil unversioned sync +</pre> + +The [/help/sync|fossil sync] and [/help/clone|fossil clone] +commands will synchronize unversioned content if and only if they're +given the "-u" (or "--unversioned") command-line option. The +[/help/unversioned|fossil unversioned sync] command synchronizes the +unversioned content without synchronizing anything else. + +Notice that the "-u" option does not work on +[/help/push|fossil push] or [/help?cmd=pull|fossil pull]. +The "-u" option is only available on "sync" and "clone". +A rough equivalent of an unversioned pull would be the +[/help?cmd=unversioned|fossil unversioned revert] command. The +"unversioned revert" +command causes the unversioned content on the local repository to be +overwritten by the unversioned content found on the remote repository. + +Beware that because unversioned file sync is an uncommonly dangerous +capability — there being no history to revert to in the case of human +error — even the all-powerful Fossil "setup" user does not get +unversioned file sync capability by default. See +[./caps/admin-v-setup.md#dcap | this] for the full details, but the +short-and-sweet takeaway is that a user needs the "y" capability on the +remote before they can sync unversioned content. Until then, you're +allowed to add such files to your local repo, but they will not sync. + +<h2>Implementation Details</h2> + +<i>(This section outlines the current implementation of unversioned +files. This is not an interface spec and hence subject to change.)</i> + +Unversioned content is stored in the repository in the +"unversioned" table: + +<pre> +CREATE TABLE unversioned( + uvid INTEGER PRIMARY KEY AUTOINCREMENT, -- unique ID for this file + name TEXT UNIQUE, -- Name of the file + rcvid INTEGER, -- From whence this file was received + mtime DATETIME, -- Last change (seconds since 1970) + hash TEXT, -- SHA1 or SHA3-256 hash of uncompressed content + sz INTEGER, -- Size of uncompressed content + encoding INT, -- 0: plaintext 1: zlib compressed + content BLOB -- File content +); +</pre> + +Fossil does not create the table ahead of need. +If there are no unversioned files in the repository, the +"unversioned" table will not exist. Consequently, +one simple way to purge all unversioned content from a repository +is to run: + +<pre> +fossil sql "DROP TABLE unversioned; VACUUM;" +</pre> + +Lacking history for unversioned files, Fossil does not attempt delta +compression on them. + +Fossil servers exchange unversioned content whole; it does not attempt +to "diff" your local version against the remote and send only the +changes. We point this out because one use-case for unversioned content +is to send large, frequently-changing files. Appreciate the consequences +before making each change. + +There are two bandwidth-saving measures in "<tt>fossil uv sync</tt>". +The first is the regular HTTP payload compression step, done on all +syncs. The second is that Fossil sends hash exchanges to determine +when it can avoid sending duplicate content over the wire unnecessarily. +See the [./sync.wiki|synchronization protocol documentation] for further +information. diff --git a/www/userlinks.wiki b/www/userlinks.wiki new file mode 100644 index 0000000..abed234 --- /dev/null +++ b/www/userlinks.wiki @@ -0,0 +1,52 @@ +<title>Links For Fossil Users:</title> + + * [./relatedwork.md | Related Work]: projects and links related to + Fossil and version control. + * [./permutedindex.html | Documentation index] with [/search?c=d | full text search]. + * [./reviews.wiki | Testimonials] from satisfied Fossil users and + [./quotes.wiki | Quotes] about Fossil and other DVCSes. + * [./faq.wiki | Frequently Asked Questions] + * The [./concepts.wiki | concepts] behind Fossil. + [./glossary.md | Another viewpoint]. + * [./quickstart.wiki | Quick Start] guide to using Fossil. + * [./qandc.wiki | Questions & Criticisms] directed at Fossil. + * [./build.wiki | Compiling and Installing] + * Fossil supports [./embeddeddoc.wiki | embedded documentation] + that is versioned along with project source code. + * Fossil uses an [./fileformat.wiki | enduring file format] that is + designed to be readable, searchable, and extensible by people + not yet born. + * A tutorial on [./branching.wiki | branching], what it means and how + to do it using Fossil. + * The [./selfcheck.wiki | automatic self-check] mechanism + helps insure project integrity. + * Fossil contains a [./wikitheory.wiki | built-in wiki]. + * An [./event.wiki | Event] is a special kind of wiki page associated + with a point in time rather than a name. + * [./settings.wiki | Settings] control the behaviour of Fossil. + * [./ssl.wiki | Use SSL] to encrypt communication with the server. + * The [https://fossil-scm.org/forum|Fossil forum] is, as of late-2024, + the project's central communication channel. The + [https://www.mail-archive.com/fossil-users@lists.fossil-scm.org + | read-only mailing list archives] house discussions spanning Fossil's + first decade. + * [./stats.wiki | Performance statistics] taken from real-world projects + hosted on Fossil. + * How to [./shunning.wiki | delete content] from a Fossil repository. + * How Fossil does [./password.wiki | password management]. + * On-line [/help | help]. + * List of [./th1.md | TH1 commands provided by Fossil itself] that expose + its key functionality to TH1 scripts. + * List of [./th1-hooks.md | TH1 hooks exposed by Fossil] that enable + customization of commands and web pages. + * A free hosting server for Fossil repositories is available at + [http://chiselapp.com/]. + * How to [./server/ | set up a server] for your repository. + * Customizing the [./custom_ticket.wiki | ticket system]. + * Methods to [./checkin_names.wiki | identify a specific check-in]. + * [./inout.wiki | Import and export] from and to Git. + * [./fossil-v-git.wiki | Fossil versus Git]. + * [./fiveminutes.wiki | Up and running in 5 minutes as a single user] + (contributed by Gilles Ganault on 2013-01-08). + * [./antibot.wiki | How Fossil defends against abuse by spiders and bots]. + * [./wsl_caveats.wiki | Using Fossil on WSL], caveats and guidance. diff --git a/www/webpage-ex.md b/www/webpage-ex.md new file mode 100644 index 0000000..fc0dde1 --- /dev/null +++ b/www/webpage-ex.md @@ -0,0 +1,127 @@ +Web-Page Examples +================= + +Here are just a few examples of the many web pages supported +by Fossil. Follow hyperlinks on the examples below to see many +other examples. + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?y=ci&n1=100'>(Example)</a> → + 100 most recent check-ins. + + * <a target='_blank' class='exbtn' + href='$ROOT/finfo?name=src/file.c'>(Example)</a> → + All changes to the <b>src/file.c</b> source file. + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?n1=200&uf=0c3c2d086a'>(Example)</a> → + All check-ins using a particular version of the <b>src/file.c</b> + source file. + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?n1=11&y=ci&c=2014-01-01'>(Example)</a> → + Check-ins proximate to an historical point in time (2014-01-01). + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?df=release&y=ci'>(Example)</a> → + All check-ins derived from the most recent release. + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?n1=11&y=ci&c=2014-01-01&v=1'>(Example)</a> → + The previous example augmented with file changes. + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?n1=25&y=ci&a=1970-01-01'>(Example)</a> → + First 25 check-ins after 1970-01-01. (The first 25 check-ins of + the project.) + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?n1=200&r=svn-import'>(Example)</a> → + All check-ins of the "svn-import" branch together with check-ins + that merge with that branch. + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?n1=200&t=svn-import'>(Example)</a> → + All check-ins of the "svn-import" branch only. + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?n1=100&y=ci&ubg'>(Example)</a> → + 100 most recent check-ins color coded by committer rather than by branch. + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?from=version-1.27&to=version-1.28'>(Example)</a> → + All check-ins on the most direct path from + version-1.27 to version-1.28 + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?namechng'>(Example)</a> → + Show check-ins that contain file name changes + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?u=drh&c=2014-01-08&y=ci'>(Example)</a> → + Show check-ins circa 2014-01-08 by user "drh". + + * <a target='_blank' class='exbtn' + href='$ROOT/timeline?from=version-1.34&to=version-1.35&chng=src/timeline.c,src/doc.c'>(Example)</a> → + Show all check-ins between version-1.34 and version-1.35 that make + changes to either of the files src/timeline.c or src/doc.c. + + <big><b>→</b></big> (Hint: In the pages above, click the graph nodes + for any two check-ins or files to see a diff.) + <big><b>←</b></big> + + * <a target='_blank' class='exbtn' + href='$ROOT/search?s=interesting+pages'>(Example)</a> → + Full-text search for "interesting pages". + + * <a target='_blank' class='exbtn' + href='$ROOT/tree?ci=daff9d20621&type=tree'>(Example)</a> → + All files for a particular check-in (daff9d20621480) + + * <a target='_blank' class='exbtn' + href='$ROOT/tree?ci=trunk&type=tree&mtime=1'>(Example)</a> → + All files for the latest check-in on a branch (trunk) sorted by + last modification time. + + * <a target='_blank' class='exbtn' + href='$ROOT/fileage?name=svn-import'>(Example)</a> → + Age of all files in the latest checking for branch "svn-import". + + * <a target='_blank' class='exbtn' + href='$ROOT/brlist'>(Example)</a> → + Table of branches. (Click on column headers to sort.) + + * <a target='_blank' class='exbtn' + href='$ROOT/stat'>(Example)</a> → + Overall repository status. + + * <a target='_blank' class='exbtn' + href='$ROOT/reports?type=ci&view=byuser'>(Example)</a> → + Number of check-ins per committer. + + * <a target='_blank' class='exbtn' + href='$ROOT/reports?view=byfile'>(Example)</a> → + Number of check-ins for each source file. + (Click on column headers to sort.) + + * <a target='_blank' class='exbtn' + href='$ROOT/blame?checkin=5260fbf63287&filename=src/rss.c&limit=-1'> + (Example)</a> → + Most recent change to each line of a particular source file in a + particular check-in. + + * <a target='_blank' class='exbtn' + href='$ROOT/taglist'>(Example)</a> → + List of tags on check-ins. + + * <a target='_blank' class='exbtn' + href='$ROOT/bigbloblist'>(Example)</a> → + The largest objects in the repository. + + * <a target='_blank' class='exbtn' + href='$ROOT/hash-collisions'>(Example)</a> → + Hash prefix collisions + + * <a target='_blank' class='exbtn' + href='$ROOT/sitemap'>(Example)</a> → + The "sitemap" containing links to many other pages diff --git a/www/webui.wiki b/www/webui.wiki new file mode 100644 index 0000000..f0126d7 --- /dev/null +++ b/www/webui.wiki @@ -0,0 +1,181 @@ +<title>The Fossil Web Interface</title> + +One of the innovative features of Fossil is its built-in web interface. +This web interface provides everything you need to run a software +development project: + + * [./bugtheory.wiki | Ticketing and bug tracking] + * [./wikitheory.wiki | Wiki] + * [./embeddeddoc.wiki | On-line documentation] + * [./event.wiki | Technical notes] + * [./forum.wiki | Forum] + * [./chat.md | Chatroom] + * Timelines + * Full text search over all of the above + * Status information + * Graphs of revision and branching history + * File and version lists and differences + * Download historical versions as ZIP archives + * Historical change data + * Add and remove tags on check-ins + * Move check-ins between branches + * Revise check-in comments + * Manage user credentials and access permissions + * And so forth... (some [./webpage-ex.md|examples]) + +You get all of this, and more, for free when you use Fossil. +There are no extra programs to install or setup. +Everything you need is already pre-configured and built into the +self-contained, stand-alone Fossil executable. + +As an example of how useful this web interface can be, +the entire [./index.wiki | Fossil website], +including the document you are now reading, +is rendered using the Fossil web interface, with no enhancements, +and little customization. + +<div class="indent"> +<b>Key point:</b> <i>The Fossil website is just a running instance +of Fossil! +</div> + +Note also that because Fossil is a distributed system, you can run +the web interface on your local machine while off network (for example, +while on an airplane) including +making changes to wiki pages and/or trouble ticket, then synchronize with your +co-workers after you reconnect. When you clone a Fossil repository, you +don't just get the project source code, you get the entire project +management website. + +<h2>Very Simple Startup</h2> + +To start using the built-in Fossil web interface on an existing Fossil +repository, simply type this: + +<pre>fossil ui existing-repository.fossil</pre> + +Substitute the name of your repository, of course. +The "ui" command will start a web server running (it figures out an +available TCP port to use on its own) and then automatically launches +your web browser to point at that server. If you run the "ui" command +from within an open check-out, you can omit the repository name: + +<pre>fossil ui</pre> + +The latter case is a very useful short-cut when you are working on a +Fossil project and you want to quickly do some work with the web interface. +Notice that Fossil automatically finds an unused TCP port to run the +server on and automatically points your web browser to the correct +URL. So there is never any fumbling around trying to find an open +port or to type arcane strings into your browser URL entry box. +The interface just pops right up, ready to run. + +The Fossil web interface is also very easy to setup and run on a +network server, as either a CGI program or from inetd, or as an +SCGI server. Details on how +to do that are described further below. + +<h2>Things To Do Using The Web Interface</h2> + +You can view <b>timelines</b> of changes to the project. The default +"Timeline" link on the menu bar takes you to a page that shows the 20 +most recent check-ins, wiki page edits, ticket/bug-report changes, +and/or blog entries. +This gives a very useful snapshot of what has been happening lately on the +project. You can click to go further back in time, if needed. Or +follow hyperlinks to see details, including diffs and annotated diffs, +of individual check-ins, wiki page edits, ticket changes, and +blog edits. + +You can view and edit <b>tickets and bug reports</b> by following the +"Tickets" link on the menu bar. +Fossil is backed by an SQL database, so users with appropriate permissions +can write new ticket report formats based on SQL query statements. +Fossil is careful to prevent ticket report formats from doing any mischief +on the database (it only allows SELECT statements to run) and it restricts +access to sensitive data such as user passwords. So it is actually safe +to let anonymous users on the internet write their own ticket formats if +you like. In addition to viewing and/or creating report formats, you can +also create new tickets or look at summaries or complete histories of +existing tickets. Any changes you make will automatically merge with +changes from your co-workers the next time your repository is synchronized. + +You can view and edit <b>wiki</b> by following the "Wiki" link on the +menu bar. Fossil has its own easy-to-remember +[/wiki_rules | markup rules], or if you prefer, it also +supports [/md_rules | Markdown]. And, as with tickets, all of +your edits will automatically merge with those of your co-workers when +your repository synchronizes. + +You can view summary reports of <b>branches</b> in the +check-in graph by visiting the "Branches" link on the +menu bar. From those pages you can follow hyperlinks to get additional +details. These screens allow you to easily keep track of what is going +on with separate sub-teams within your project team. + +The "Files" link on the menu allows you to browse through the <b>file +hierarchy</b> of the project and to view complete changes histories on +individual files, with hyperlinks to the check-ins that made those +changes, and with diffs and annotated diffs between versions. + +The web interface supports [./embeddeddoc.wiki | embedded documentation] +files (usually in wiki format) that are checked into the project as +part of the source tree. Such files can be viewed as if they were +ordinary web pages. This document that you are now reading is an +example of embedded documentation. + +<h2>Customizing The Web Interface Appearance</h2> + +Users with appropriate permissions can customize the look and feel of +the web interface using the "Admin" link on the main menu of the web +interface. Templates +for the header and footer of each page can be edited, as can the CSS +for the entire page. You can even change around the main menu. +Timeline display preferences can be edited. The page that is brought +up as the "Home" page can be changed. It is often useful to set the +"Home" page to be a wiki page or an embedded document. The built-in +pages <b>/home</b> and <b>/index</b> can be used as the "Home" page. +They have identical effect, which is to instruct Fossil to find and +display a wiki page with the same name as the project, or if that +does not exist, <b>/README.md</b> or <b>/index.wiki</b>. + +An embedded document link such as <b>doc/trunk/README.md</b> can be +used for the "Home" page. If you specify one of the built-in keywords +<b>/home</b> or <b>/index</b>, the page will not be treated as an +embedded document. + +<h2>Installing On A Network Server</h2> + +When you create a new Fossil project and after you have configured it +like you want it using the web interface, you can make the project +available to a distributed team by simply copying the single +repository file up to a web server that supports CGI or SCGI. To +run Fossil as CGI, just put the +<b>sample-project.fossil</b> file in a directory where CGI scripts +have both read and write permission on the file and the directory that +contains the file, then add a CGI script that looks something like this: + +<verbatim>#!/usr/local/bin/fossil +repository: /home/www/sample-project.fossil</verbatim> + +Adjust the script above so that the paths are correct for your system, +of course, and also make sure the Fossil binary is installed on the server. +But that is <u>all</u> you have to do. You now have everything you need to host +a distributed software development project in less than five minutes using a +two-line CGI script. + +Instructions for setting up an SCGI server are +[./scgi.wiki | available separately]. + +You don't have a CGI- or SCGI-capable web server running on your +server machine? +Not a problem. The Fossil interface can also be launched via inetd or +xinetd. An inetd configuration line sufficient to launch the Fossil +web interface looks like this: + +<verbatim>80 stream tcp nowait.1000 root /usr/local/bin/fossil \ +/usr/local/bin/fossil http /home/www/sample-project.fossil</verbatim> + +As always, you'll want to adjust the pathnames to whatever is appropriate +for your system. The xinetd setup uses a different syntax but follows +the same idea. diff --git a/www/whyallinone.md b/www/whyallinone.md new file mode 100644 index 0000000..c71cf00 --- /dev/null +++ b/www/whyallinone.md @@ -0,0 +1,235 @@ +# Why Add Forum, Wiki, and Web Software To Your DVCS? + +One notable feature of Fossil is that it bundles +[bug tracking](./bugtheory.wiki), +[wiki](./wikitheory.wiki), +[forum](./forum.wiki), +[chat](./chat.md), and +[technotes](./event.wiki) +with distributed version control to give you an +all-in-one software project management system. + +A commenter on [Hacker News](https://news.ycombinator.com/item?id=27437895) +takes exception to this idea, writing: + +> *I don't want forum/web software built into my dvcs.* +> *I don't see how this improves over git.* + +This commenter may hold whatever opinions he wishes, of course. +However, there are many good reasons why bundling other project management +features with the DVCS might be useful for a given project: + + 1. There is a single software package to install and manage for the + project website. + The alternative is to select, install, configure, learn about, + manage, and maintain separate DVCS, wiki, + ticketing, forum, + chat, documentation, and whatever other software packages your project needs. + Less time spent on project administration details means more + time available to spend on the project itself. + + 2. Fossil’s autosync feature gives you an implicit backup of the + wiki, tickets, forum, and so forth simply by cloning the + repository to another machine and using that clone regularly. + Since the typical Fossil usage pattern is to stand the repo up on a + central server and have the developers clone that repository down + to their personal machines, if the server falls over, the last + developer to do anything that resulted in an autosync has a + functional and up-to-date backup. + + There are [limitations to relying on Fossil’s autosync feature for + backup purposes](./backup.md), but that document gives two methods + for more complete backups, both of which are easily automated. The + Fossil project itself is distributed across three data centers in + this manner via cron. + + 3. Remote workers get more than just the source code: + they get the entire website including versioned documentation, + wiki articles, tickets, forum posts, and so forth. This supports + off-network development when traveling, when riding out Internet + service failures, and when workers must sync with multiple remote + servers, as when working alternately from home and in some central + office. + + Feature-competitive Fossil alternatives typically solve this same + problem with centralization, which generally means that only the + DVCS piece still works in these situations where the developer is + unable to contact the central server. Why accept the limitation of + having a distributed clone of the code repo alone? + + Centralization doesn’t work for every project. If you enjoy the + benefits of truly distributed (read: non-centralized) version + control, you may also benefit from distributed forums, distributed + ticket tracking, distributed wiki article publishing, and so + forth. + + 4. Integration of all of these features allows easy hyperlinks between + check-in comments, wiki pages, forum posts, and tickets. More, + because the software sees both sides of the link, referrer and + referent, it can provide automatic back-references. + + A common situation in a Fossil project is that: + + * a forum post refers to a versioned project document that shows + that the software isn’t behaving as documented; + * a developer triages that forum report as a verified bug, filing + a ticket to prioritize and track the resolution; + * developers chat about the problem, referring to the ticket and + thereby indirectly referring to the forum post, plus perhaps to + other Fossil-managed resources such as a wiki document giving + design principles that guide the proper fix; and finally + * the commit message resolving the ticket includes a reference to + the ticket it resolves. + + Since Fossil sees that the commit refers to a ticket, the ticket + page automatically also refers back to the commit, closing the + loop. A latecomer may arrive at the ticket via a web search, and + from that see that it was closed following a commit. They can + follow the link from the initial ticket message to the forum + thread to catch up on the discussion leading to the fix and likely + find a follow-up post from the initial reporting user saying + whether the fix worked for them. If further work was needed, the + latecomer can likely find it from that forum thread. + + This works even in a remote off-network clone: the developer can + pull up the project web site via an `http://localhost` link and + follow these links around the loop. + + Fossil allows breaking some of these project facilities out into + separate repositories, as when the public forum is kept separate + from the actual software development repository for administration + reasons. By using Fossil’s [interwiki link + feature](./interwiki.md), you can get this same internal linking + from ticket to commit to forum to wiki even across these + administrative boundaries, even with remote off-network clones, + simply by adjusting the interwiki map to match the remote clone’s + network configuration. + + 5. Bundling all of these services gives [single sign-on][SSO] (SSO) for all + aspects of the project. The same username/password works for code, + wiki, forum, tickets, and chat. + + If you choose to administratively separate some of these features + by setting up multiple cooperating Fossil repositories, its [login + groups feature](./caps/login-groups.md) allows asymmetric SSO + across these administrative boundaries so that, for example, users + allowed to commit to the code repository also get a forum + repository login, but self-registered forum users don’t + automatically get the ability to commit to the code repo. + + 6. Bundling all of these features reduces the number of external + dependencies for the project. + + Take the first two points above: standing up a Fossil repo backup + on a new server may be as simple as copying the backup to the new + server and [configuring its stock HTTP server to point at the + backup repository via CGI](./server/any/cgi.md). + + Consider: If you had good backups for all of the elements in a + Git + [Jira] + [Discord] + [MediaWiki] + [Sphinx] lash-up, how long + would it take you to stand up a replacement? That lash-up + certainly has more features combined than Fossil alone, but are + they worth the administration and hosting costs they impose? + Fossil’s feature set suffices for the SQLite project it was + created to serve, as well as for many others; is your project + sufficiently more complex, such that it *needs* all of those extra + features and their concomitant complexity? + + Considerations such as these push many into centralized hosting + services such as GitHub, GitLab, Bitbucket, and so forth, but that + just takes you back to point 3 above. + + 7. Hosting all of these elements within a single service gives a + consistent look-and-feel across all aspects of the project. + + Skinning independent software packages’ web interfaces to make + them appear unified is more work than [skinning] everything once, as + in Fossil, and even then, you can’t make independently-developed + software look like it was produced by a single entity without + resorting to heroic levels of customization. If you use a separate + DVCS web front end, chat system, forum manager, documentation + system, ticket tracker, and so on, you are likely to be relegated + to simply matching colors and fonts; you *might* also get the + ability to add a common logo to the header of all of these + independent pieces. The pieces won’t look unified, because they + weren’t developed that way. + + The Fossil project’s + skinning system lets you affect all of its elements globally from the + single skin editor. + + Or not: there’s a feature in Fossil that lets skin customizations + apply to only *some* Fossil features. The initial impetus behind + this feature was that one of our users wanted Markdown to be + rendered with different indentation in forum posts than in + [embedded documentation][edoc] owing to the inherent differences between + the two presentation modalities. + + A user taking advantage of this per-feature CSS capability who + wishes to change a UI element common to all Fossil features — say, + to change the font for literal code blocks — may still make such a + change globally. Opting into this per-feature CSS doesn’t fork all + skinning efforts: UI elements not explicitly reskinned on a + per-feature basis inherit the global skinning. + + But it goes futher. Fossil has a feature for [project-specific + extensions](./serverext.wiki), which backs the [SQLite Release + Checklist][srckl], for instance. You wouldn’t know by looking at + that page that it’s produced by software that isn’t actually part + of Fossil: the extension only delivers the core of the page, + and Fossil’s skining wraps it in a way that lets it inherit all of + the project-level skinning customizations. + + 8. Unifying all of these features within Fossil + means we have a single Markdown interpreter common to all + elements. If you lash multiple software systems together, even if + they can all agree on Markdown as a common document markup + language — hardly a given, as shown by the MediaWiki and Sphinx + elements in point 6’s example above — they’re likely to render your text + using different — possibly even incompatibly-different — Markdown + dialects. + + This costs you in mental gear-switching when moving from the code + repository to the documentation system to the forums to the ticket + tracker. + + More than that, though, a developer might write a forum post that later gets + promoted to a wiki article or to an embedded version-controlled + project document. A developer on a Fossil-backed project may simply copy-paste the forum post + text into the new document and save it, not needing to carefully + check that it still renders properly under the second Markdown + rendering engine. Similarly, if a user reports a potential bug via + the forum, the developer can copy interesting pieces of the + Markdown from the post into a ticket comment, again without + needing to fiddle with dialect incompatibilities. + + 9. Fossil is [free, open-source software](../COPYRIGHT-BSD2.txt), + through and through. Git-backed lash-ups tend to incorporate + either proprietary add-ons or proprietary hosting systems that + produce vendor lock-in. Fossil gives you the freedom to take your + complete backup (point 2) of the project including its + idiosyncratic customizations and stand it up elsewhere on + commodity hardware and software stacks. + +All of this having been said, the non-DVCS features of Fossil are +optional. Its forum and chat features are disabled by default, and you +can disable the ticket-tracking and wiki features with a quick +configuration change to its [role-based access control system](./caps/). +When you’re ready to turn these additional features on, you can do so +with a few mouse clicks. + +Because Fossil is web-native out of the box, if you’ve delegated these +features to outside systems to flesh out Git’s DVCS-only nature, you are +free to do the same with Fossil. One of the many things the [skinning] +facility allows is replacing the built-in links to the wiki, forum, +ticket system, etc. with links to external systems. How easy those +systems make it to link back into Fossil is up to their developers. + +[Discord]: https://discord.com/ +[edoc]: ./embeddeddoc.wiki +[Jira]: https://www.atlassian.com/software/jira +[MediaWiki]: https://www.mediawiki.org/ +[skinning]: ./customskin.md +[Sphinx]: https://www.sphinx-doc.org/en/master/ +[SSO]: https://en.wikipedia.org/wiki/Single_sign-on +[srckl]: https://www.sqlite.org/src/ext/checklist/top/index diff --git a/www/whyusefossil.wiki b/www/whyusefossil.wiki new file mode 100644 index 0000000..7cbc8aa --- /dev/null +++ b/www/whyusefossil.wiki @@ -0,0 +1,196 @@ +<title>Why You Should Use Fossil</title> + +<h4>(Or if not Fossil, at least some kind of modern +version control such as Git, Mercurial, or Subversion.)</h4> + +<h5>I. Benefits of Version Control</h5> + + <ol type='A'> + <li><p><b>Immutable file and version identification</b> + <ol type='i'> + <li>Simplified and unambiguous communication between developers + <li>Detect accidental or surreptitious changes + <li>Locate the origin of discovered files + </ol> + <li><p><b>Parallel development</b> + <ol type='i'> + <li>Multiple developers on the same project + <li>Single developer with multiple subprojects + <li>Experimental features do not contaminate the main line + <li>Development/Testing/Release branches + <li>Incorporate external changes into the baseline + </ol> + <li><p><b>Historical record</b> + <ol type='i'> + <li>Exactly reconstruct historical builds + <li>Locate when and by whom faults were injected + <li>Find when and why content was added or removed + <li>Team members see the big picture + <li>Research the history of project features or subsystems + <li>Copyright and patent documentation + <li>Regulatory compliance + </ol> + <li><p><b>Automatic replication and backup</b> + <ol type='i'> + <li>Everyone always has the latest code + <li>Failed disk-drives cause no loss of work + <li>Avoid wasting time doing manual file copying + <li>Avoid human errors during manual backups + </ol> + </ol> + +<h5 id="definitions">II. Definitions</h5> + +<div class="indent">Moved to [./glossary.md | a separate document].</div> + +<h5>III. Basic Fossil commands</h5> + + <ul> + <li><p><b>clone</b> → + Make a copy of a repository. The original repository + is usually (but not always) on a remote machine and the copy is on + the local machine. The copy remembers the network location from + which it was copied and (by default) tries to keep itself synchronized + with the original. + <li><p><b>open</b> → + Create a new check-out from a repository on the local machine. + <li><p><b>update</b> → + Modify an existing check-out so that it is derived from a + different version of the same project. + <li><p><b>commit</b> → + Create a new version (a new check-in) of the project that + is a snapshot of the current check-out. + <li><p><b>revert</b> → + Undo all local edits on a check-out. Make the check-out + be an exact copy of its associated check-in. + <li><p><b>push</b> → + Copy content found in a local repository over to a remote + repository. (Fossil usually does this automatically in response to + a "commit" and so this command is seldom used, but it is important + to understand it.) + <li><p><b>pull</b> → + Copy new content found in a remote repository into a local + repository. A "pull" by itself does not modify any check-out. The + "pull" command only moves content between repositories. However, + the "update" command will (often) automatically do a "pull" before + attempting to update the local check-out. + <li><p><b>sync</b> → + Do both a "push" and a "pull" at the same time. + <li><p><b>add</b> → + Add a new file to the local check-out. The file must already be + on disk. This command tells Fossil to start tracking and managing + the file. This command affects only the local check-out and does + not modify any repository. The new file is inserted into the + repository at the next "commit" command. + <li><p><b>rm/mv</b> → + Short for 'remove' and 'move', these commands are like "add" + in that they specify pending changes to the structure of the check-out. + As with "add", no changes are made to the repository until the next + "commit". + </ul> + +<h5>IV. The history of a project is a Directed Acyclic Graph (DAG)</h5> + + <ul> + <li><p>Fossil (and other distributed VCSes like Git and Mercurial, but + not Subversion) represent + the history of a project as a directed acyclic graph (DAG). + <ul> + <li><p>Each check-in is a node in the graph + <li><p>If check-in Y is derived from check-in X then there is + an arc in the graph from node X to node Y. + <li><p>The older check-in (X) is call the "parent" and the newer + check-in (Y) is the "child". The child is derived from + the parent. + </ul> + <li><p>Two users (or the same user working in different check-outs) + might commit different changes against the same check-in. This + results in one parent node having two or more children. + <li><p>Command: <b>merge</b> → + combines the work of multiple check-ins into + a single check-out. That check-out can then be committed to create + a new check-in that has two (or more) parents. + <ul> + <li><p>Most check-ins have just one parent, and either zero or + one child. + <li><p>When a check-in has two or more parents, one of those parents + is the "primary parent". All the other parent nodes are "secondary" + or "merge" parents. + Conceptually, the primary parent shows the main line of + development. Content from the merge parents is added + into the main line. + <li><p>The "direct children" of a check-in X are all children that + have X as their primary parent. + <li><p>A check-in node with no direct children is sometimes called + a "leaf". + <li><p>The "merge" command changes only the check-out. + The "commit" command must be run subsequently to make the merge + a permanent part of project. + </ul> + <li><p>Definition: <b>branch</b> → + a sequence of check-ins that are all linked + together in the DAG through the primary parent. + <ul> + <li><p>Branches are often given names which propagate to direct children. + The tradition in Fossil is to call the main branch "trunk". In + Git, it's called "master" by default, though some call it + something else, like "main". + <li><p>It is possible to have multiple branches with the same name. + Fossil has no problem with this, but it can be confusing to + humans, so best practice is to give each branch a unique name. + <li><p>The name of a branch can be changed by adding special tags + to the first check-in of a branch. The name assigned by this + special tag automatically propagates to all direct children. + </ul> + </ul> + +<h5>V. Why version control is important (reprise)</h5> + + <ol> + <li><p>Every check-in and every individual file has a unique name - its + SHA1 or SHA3-256 hash. Team members can unambiguously identify + any specific + version of the overall project or any specific version of an + individual file. + <li><p>Any historical version of the whole project or of any individual + file can be easily recreated at any time and by any team member. + <li><p>Accidental changes to files can be detected by recomputing their + cryptographic hash. + <li><p>Files of unknown origin can be identified using their hash. + <li><p>Developers are able to work in parallel, review each others work, + and easily merge their changes together. External revisions to + the baseline can be easily incorporated into the latest changes. + <li><p>Developers can follow experimental lines of development, then + revert back to an earlier stable version if the experiment does + not work out. Creativity is enhanced by allowing crazy ideas to + be investigated without destabilizing the project. + <li><p>Developers can work on several independent subprojects, flipping + back and forth from one subproject to another at will, and merge + patches together or back into the main line of development as they + mature. + <li><p>Older changes can be easily backed out of recent revisions, for + example if bugs are found long after the code was committed. + <li><p>Enhancements in a branch can be easily copied into other branches, + or into the trunk. + <li><p>The complete history of all changes is plainly visible to + all team members. Project leaders can easily keep track of what + all team members are doing. Check-in comments help everyone to + understand and/or remember the reason for each change. + <li><p>New team members can be brought up-to-date with all of the historical + code, quickly and easily. + <li><p>New developers, interns, or inexperienced staff members who still + do not understand all the details of the project or who are otherwise + prone to making mistakes can be assigned significant subprojects to + be carried out in branches without risking main line stability. + <li><p>Code is automatically synchronized across all machines. No human + effort is wasted copying files from machine to machine. The risk + of human errors during file transfer and backup is eliminated. + <li><p>A hardware failure results in minimal lost work because + all previously committed changes will have been automatically + replicated on other machines. + <li><p>The complete work history of the project is conveniently archived + in a single file, simplifying long-term record keeping. + <li><p>A precise historical record is maintained which can be used to + support copyright and patent claims or regulatory compliance. + </ol> +</ol> diff --git a/www/wikitheory.wiki b/www/wikitheory.wiki new file mode 100644 index 0000000..7273db2 --- /dev/null +++ b/www/wikitheory.wiki @@ -0,0 +1,90 @@ +<title>Wiki In Fossil</title> +<h2>Introduction</h2> + +Fossil uses [/wiki_rules | Fossil wiki markup] and/or +[/md_rules | Markdown markup] for many things: + + * Stand-alone wiki pages. + * Description and comments in [./bugtheory.wiki | bug reports]. + * Check-in comments. (For historical reasons, these must + currently be in fossil-wiki text format.) + * [./embeddeddoc.wiki | Embedded documentation] files whose + name ends in ".wiki" or ".md" or ".markdown". + * [./event.wiki | Technical notes]. + * [./forum.wiki | Forum messages]. + * Auxiliary notes on check-ins and branches. + +The [/wiki_rules | formatting rules for fossil wiki] +are designed to be simple and intuitive. The idea is that wiki provides +paragraph breaks, numbered and bulleted lists, and hyperlinking for +simple documents together with a safe subset of HTML for more complex +formatting tasks. + +The [/md_rules | Markdown formatting rules] are more complex, but +are also more widely known, and are thus provided as an alternative. + +<h2>Stand-alone Wiki Pages</h2> + +Each wiki page has its own revision history which is independent of +the sequence of check-ins. Wiki pages can branch and merge +just like check-ins, though as of this writing (2008-07-29) there is +no mechanism in the user interface to support branching and merging. +The current implementation of the wiki shows the version of the wiki +page that has the most recent time stamp. + +In other words, if two users make unrelated changes to the same wiki +page on separate repositories and those repositories are synced, +the wiki page will fork. The web interface will display whichever edit +was checked in last. The other edit can be found in the history. The +file format will support merging the branches back together, but there +is no mechanism in the user interface (yet) to perform the merge. + +Every change to a wiki page is a separate +[./fileformat.wiki | control artifact] +of type [./fileformat.wiki#wikichng | "Wiki Page"]. + +<h2>Embedded Documentation</h2> + +Files in the source tree that use the ".wiki", ".md", or ".markdown" suffixes +can be accessed and displayed using special URLs to the fossil server. +This allows +project documentation to be stored in the source tree and accessed +online. (Details are described [./embeddeddoc.wiki | separately].) + +Some projects prefer to store their documentation in wiki. There is nothing +wrong with that. But other projects prefer to keep documentation as part +of the source tree, so that it is versioned along with the source tree and +so that only developers with check-in privileges can change it. +Embedded documentation serves this latter purpose. Both forms of documentation +use the exact same markup. Some projects may choose to +use both forms of documentation at the same time. Because the same +format is used, it is trivial to move a file from wiki to embedded documentation +or back again as the project evolves. + +<h2>Bug-reports and check-in comments and Forum messages</h2> + +The comments on check-ins, forum posts, and the text in the +descriptions of bug reports both use wiki formatting. Exactly the +same set of formatting rules apply. There is never a need to learn +one formatting language for documentation and a different markup for +bugs or for check-in comments. + +Minor caveat: check-in messages are currently limited to the +fossil-wiki format. + +<h2 id="assocwiki">Auxiliary notes attached to check-ins or branches</h2> + +Stand-alone wiki pages with special names "branch/<i>BRANCHNAME</i>" +or "checkin/<i>HASH</i>" are associated with the corresponding +branch or check-in. The wiki text appears in an "About" section of +timelines and info screens. Examples: + + * [/timeline?r=graph-test-branch] shows the text of the + [/wiki?name=branch/graph-test-branch&p|branch/graph-test-branch] + wiki page at the top of the timeline + * [/info/19c60b7fc9e2] shows the text of the + [/wiki?name=checkin/19c60b7fc9e2400e56a6f938bbad0e34ca746ca2eabdecac10945539f1f5e8c6&p|checkin/19c60b7fc9e2...] + wiki page in the "About" section. + +These special wiki pages are very useful for recording historical +notes. diff --git a/www/wsl_caveats.wiki b/www/wsl_caveats.wiki new file mode 100644 index 0000000..2a8e3c4 --- /dev/null +++ b/www/wsl_caveats.wiki @@ -0,0 +1,176 @@ +<title>Caveats and Precautions for Fossil Usage with Windows Subsystem for Linux</title> + +<h2>When These Issues Matter</h2> + +The discussion following is relevant to those who: + + * Are using the Windows Subsystem for Linux (aka "WSL"); + * Create a Fossil checkout in a directory accessible from WSL and Windows; + * Use both Linux Fossil and Windows tools to modify files in a checkout; + * Desire to or must preserve execute permissions set for repository files; + * and Use Linux Fossil to commit changes made within the checkout. + +Note that these criteria apply conjunctively; if any are not met, +then the consequences of the issues below are at worst annoying +and otherwise harmless or absent. + +<h2>What Can Go Wrong (Why It Matters)</h2> + +The most readily seen manifestation of the problem occurs when +"<tt>fossil status</tt>" or "<tt>fossil changes</tt>" is run, +using Linux Fossil from WSL after Windows tools (including +fossil.exe) have been used to modify files within a checkout. +Unless filter options block it, those subcommands will tag +some (and often many) checkout files with <b>EXECUTABLE</b> +or <b>NOEXEC</b>, indicating that the file's user execute permission +has been altered such that it differs from what is recorded +in the repository for that version of the file. + +This "user execute permission" is referred to as "the X-bit" +hereafter, referring to either the recorded version state +or a checkout file attributes state. + +This is merely annoying and distracting if the altered X-bit +will never be committed using Linux Fossil. It can be quite +distracting because those tags tend to mask the presence or +absence of other changes whose detection is the usual +reason for using Fossil's changes or status subcommands. + +However, in the problematic usage scenario, those tags will +most often represent inadvertant toggling of the X-bit on the +affected file. The X-bit is kept in the repository for good +reason (usually), and arbitrary changes to it by means of a +commit when that change is not intended is virtually always +a bad result. (At best, the change causes useless churn; at +worst it frustrates the intended purpose of having an X-bit.) + +<h2>Technical Discusion of the Problem</h2> + +The genesis of altered X-bits, while not obvious at first glance, +involves obvious facts. The Windows OS does not deal with the +triple of user/group/other executable permissions the way that +Unix and similar operating systems do. Hence, tools which run +on Windows, including Fossil built for Windows, do not manage +the X-bit; it may not even exist yet for files which have not +had their permissions set by any Linux program running in WSL. +When such tools modify a file which has had its X-bit set (or +cleared) by a program in WSL, an existing X-bit value may not +be preserved depending upon how the modification is effected. + +The WSL infrastructure (or virtual system) compensates for the +absence of an X-bit in Windows filesystems with two stratagems: +(1) Establishing a default for its value when no Linux program +has yet set it; and (2) stashing Linux "mode" bits in a "special" +place for each file once it has been subject to a chmod() call. +That default's default can be changed by way of /etc/wsl.conf +content. However, this default cannot be right for files which +are tracked in a Fossil repository as having the other value. +And Windows tools generally are not written to deal with "mode" +bits in that "special" place. (They are kept in a NTFS extended +file attribute named $LXMOD, not accessible through the WIN32 +API; the OS layer below WIN32 must be used to get at them.) +Hence, inadvertant X-bit changes are unavoidable, or avoided +only by luck, in the general usage case noted above. + +<h2>Problematic Usage Scenarios</h2> + +<h3>A Simple Example</h3> + + * Open a checkout in Windows (using fossil.exe) from a project +whose files have a mixture of executable and non-executable files. +Use a checkout directory visible when running under WSL. + + * Navigate to the same directory in a Linux shell on WSL, then +run "fossil status". + + * Depending upon /etc/wsl.conf content (or defaults in its absence), +the status ouput will tag checkout files as EXECUTABLE or NOEXEC. + +<h3>Continuation of Simple Example</h3> + + * In the same checkout as above "Simple Example", on WSL, +run "fossil revert" to correct all those errant X-bit values. + + * Run "fossil status" again in WSL to verify absence of toggled X-bits. + + * Run "ls -l" from WSL to find two files, one with its X-bit set +and the other with it clear. + + * From Windows, perform these steps on each of those files:<br> + (1) read the_file content into a buffer<br> + (2) rename the_file the_file.bak<br> + (3) write buffer content to new file, the_file<br> + (4) del the_file.bak (or leave it)<br> + (Note that this sequence is similar to what many editors do when +a user modifies a file then uses undo to reverse the changes.) + + * Run "fossil status" again in WSL and observe that one of the +two files has had its X-bit toggled. + +<h3>A Fossil-Only Example</h3> + + * In the another (different) checkout of the same version, +somehow cause "legitimate" X-bit toggles of two files whose +X-bits differ. (This "somehow" probably will involve WSL to +toggle file bits and fossil on WSL to commit the toggles.) + + * In the Simple Example checkout, use fossil.exe on Windows +to update the checkout, ostensibly bringing the X-bit toggles +into the affected checkout files. + + * In the Simple Example checkout, use fossil on WSL to run +"fossil status", and observe at least one X-bit discrepancy. + +<h2>Recommended Workflow</h2> + +There are two simple approaches for dealing with this issue when +one wishes to continue using the same checkout directory from +Windows and WSL. Either one is effective. These are: + + * Do not use fossil on WSL for any operations which will modify +the repository. Instead, block those operations in some manner. + + * Do not use any tools on Windows, (including certain subcommands +of fossil.exe,) which may modify the X-bits on files within the +shared checkout, instead restricting use of Windows tools to those +which are known to only (and actually) modify file content in place +while preserving X-bit values. (The "actually" proviso emphasizes +that tools which only simulate in-place file modification, but do +so via create combined with delete and rename, are to be avoided. +A simulation which works flawlessly on Windows may not preserve +the WSL X-bit.) + +There are more complex ways to deal with this issue, involving +use of fossil on WSL to fix (or revert) toggled X-bits prior +to any commit, together with actions needed to preserve all +intended changes to the checkout as fossil revert is done. +Such methods are overly clever or fragile for elaboration here. + +Another way to deal with this issue is to correct any toggled +X-bits within a checkout before using "fossil commit" on WSL +by means other than "fossil revert". + +<h2>Corrective Measures or Mitigation</h2> + +It is possible, by either manual or automated means, to perform +a pre-commit check and/or correction for mis-toggled X-bits. + +The X-bit states are available from the repository for whatever +versions it has stored. And several Linux tools are able to read +or alter the X-bit state of files. With these components, a tool +can be readily built to aid avoidance of a commit (via fossil on +WSL) that would record mis-toggled X-bits into the repository. + +Fossil itself on WSL will detect mis-toggled X-bits for files +which have not been otherise modified, but altered file content +masks such detection, and it is just such modification that is +among the problematic scenarios. So Fossil alone cannot yet +reliably do the detection or correction needed to avoid or +remedy the mis-toggled X-bit commit problem. + +It is also feasible to detect or correct the mis-toggled X-bit +problem within Windows with a special-purpose tool which can +read, create or modify the X-bits stored by WSL for any file +which has been subject to the Linux chmod(...) system call. + +Creation of these tools is beyond the scope of this document. diff --git a/www/xkcd-git.gif b/www/xkcd-git.gif Binary files differnew file mode 100644 index 0000000..c0d525d --- /dev/null +++ b/www/xkcd-git.gif |
