diff options
Diffstat (limited to 'www/makefile.wiki')
| -rw-r--r-- | www/makefile.wiki | 327 |
1 files changed, 327 insertions, 0 deletions
diff --git a/www/makefile.wiki b/www/makefile.wiki new file mode 100644 index 0000000..8cccfb0 --- /dev/null +++ b/www/makefile.wiki @@ -0,0 +1,327 @@ +<title>The Fossil Build Process</title> + +<h1>1.0 Introduction</h1> + +The build process for Fossil is tricky in that the source code +needs to be processed by three different preprocessor programs +before it is compiled. Most users will download a +[https://fossil-scm.org/home/uv/download.html | precompiled binary] +so this is of no consequence to them, and even those who +want to compile the code themselves can use one of the +[./build.wiki | existing makefiles]. +So most people do not need to be concerned with the +build complexities of Fossil. But hard-core developers who desire +a deep understanding of how Fossil is put together can benefit +from reviewing this article. + +<h1 id="srctour">2.0 Source Code Tour</h1> + +The source code for Fossil is found in the +[/dir?ci=trunk&name=src | src/] subdirectory of the +source tree. The src/ subdirectory contains all code, including +the code for the separate preprocessor programs. + +Each preprocessor program is a separate C program implemented in +a single file of C source code. The three preprocessor programs +are: + + 1. [/file/tools/mkindex.c | mkindex.c] + 2. [/file/tools/translate.c | translate.c] + 3. [/file/tools/makeheaders.c | makeheaders.c] + +Fossil uses [http://www.sqlite.org/ | SQLite] for on-disk +storage. The SQLite implementation is contained in three source +code files that do not participate in the preprocessing steps. +These three files that implement SQLite are: + + 4. sqlite3.c + 5. sqlite3.h + 6. shell.c + +All three SQLite source files are byte-for-byte copies of files by +the same name in the +standard [http://www.sqlite.org/amalgamation.html | amalgamation]. +The sqlite3.c file implements the database engine. The shell.c file +implements the command-line shell, which is accessed in fossil using +the [/help/sqlite3 | fossil sql] command. + +The shell.c command-line shell uses the [https://github.com/antirez/linenoise | +linenoise] library to implement line editing. linenoise comprises two +source files which were copied from the upstream repository with only +very minor portability edits: + + 7. linenoise.c + 8. linenoise.h + +The TH1 script engine is implemented using files: + + 9. th.c + 10. th.h + +The preprocessing steps are omitted for all of these imported +files. + +The VERSION.h header file is generated from other information sources +using a small program called: + + 11. [/file/tools/mkversion.c | mkversion.c] + +The builtin_data.h header file contains the definitions of C-language +byte-array constants that contain various resources such as scripts and +images. The builtin_data.h header file is generated from the original +resource files using a small program called: + + 12 [/file/tools/mkbuiltin.c | mkbuiltin.c] + +Examples of built-in resources include the [/file/src/diff.tcl | diff.tcl] +script used to implement the --tk option to [/help/diff| fossil diff], +the [/file/src/markdown.md | markdown documentation], and the various +CSS scripts, headers, and footers used to implement built-in skins. New +resources files are added to the "extra_files" variable in +[/file/tools/makemake.tcl | makemake.tcl]. + +The src/ subdirectory also contains documentation about the +makeheaders preprocessor program: + + 13. [../tools/makeheaders.html | makeheaders.html] + +Click on the link to read this documentation. In addition there is +a [http://www.tcl-lang.org/ | Tcl] script used to build the various makefiles: + + 14. makemake.tcl + +Running this Tcl script will automatically regenerate all makefiles. +In order to add a new source file to the Fossil implementation, simply +edit makemake.tcl to add the new filename, then rerun the script, and +all of the makefiles for all targets will be rebuilt. + +There is an optional code verification step implemented using + + 15. [/file/tools/codecheck1.c | codecheck1.c] + +This file implements a small utility program ("codecheck1") +that scans other Fossil source files looking for errors in printf-style +format strings. +The codecheck1 utility detects missing or surplus arguments on +printf-like functions and dangerous uses of "%s" that might +permit SQL injection or cross-site scripting attacks. This code +check step is run automatically on each build of Fossil, and can +also be run separately by typing "make codecheck". Note that the +built-in printf format checking of GCC does not function for Fossil +since Fossil implements its own printf (in the +[/file/src/printf.c | printf.c] source file) that includes special +features and formatting letters that are useful to Fossil. The +codecheck1 utility can be seen as an enhanced application-specific +replacement for the GCC printf format checker. + +Finally, there is one of the makefiles generated by makemake.tcl: + + 16. main.mk + +The main.mk makefile is invoked from the Makefile in the top-level +directory. The main.mk is generated by makemake.tcl and should not +be hand edited. Other makefiles generated by makemake.tcl are in +other subdirectories (currently all in the win/ subdirectory). + +All the other files in the src/ subdirectory (79 files at the time of +this writing) are C source code files that are subject to the +preprocessing steps described below. In the sequel, we will call these +other files "src.c" in order to have a convenient name. The reader +should understand that whenever "src.c" or "src.h" is used in the text +that follows, we really mean all (79) other source files other than +the exceptions described above. + +<h1>3.0 Automatically generated files</h1> + +The "VERSION.h" header file contains some C preprocessor macros that +identify the version of Fossil that is to be built. The VERSION.h file is +generated automatically from information extracted from the "manifest", +"manifest.uuid", and "VERSION" source files in the root directory of the +source tree. +(The "manifest" and "manifest.uuid" files are automatically generated and +updated by Fossil itself. See the [/help/setting | fossil set manifest] +command for additional information.) + +The VERSION.h header file is generated by +a C program: tools/mkversion.c. +To run the VERSION.h generator, first compile the tools/mkversion.c + source file into a command-line program (named "mkversion.exe") +then run: + +<pre> +mkversion.exe manifest.uuid manifest VERSION >VERSION.h +</pre> + +The pathnames in the above command might need to be adjusted to get the +directories right. The point is that the manifest.uuid, manifest, and +VERSION files +in the root of the source tree are the three arguments and +the generated VERSION.h file appears on standard output. + +The builtin_data.h header file is generated by a C program: tools/mkbuiltin.c. +The builtin_data.h file contains C-language byte-array definitions for +the content of resource files used by Fossil. To generate the +builtin_data.h file, first compile the mkbuiltin.c program, then run: + +<pre> +mkbuiltin.exe diff.tcl <i>OtherFiles...</i> >builtin_data.h +</pre> + +At the time of this writing, the "diff.tcl" script (a Tcl/Tk script used +to generate implement --tk option on the diff command) is the only resource +file processed using mkbuiltin.exe. However, new resources will likely be +added using this facility in future versions of Fossil. + +<h1 id="preprocessing">4.0 Preprocessing</h1> + +There are three preprocessors for the Fossil sources. The mkindex +and translate preprocessors can be run in any order. The makeheaders +preprocessor must be run after translate. + +<h2>4.1 The mkindex preprocessor</h2> + +The mkindex program scans the "src.c" source files looking for special +comments that identify routines that implement various Fossil commands, +web interface methods, and help text comments. The mkindex program +generates some C code that Fossil uses in order to dispatch commands and +HTTP requests and to show on-line help. Compile the mkindex program +from the mkindex.c source file. Then run: + +<pre> +./mkindex src.c >page_index.h +</pre> + +Note that "src.c" in the above is a stand-in for the (79) regular source +files of Fossil - all source files except for the exceptions described in +section 2.0 above. + +The output of the mkindex program is a header file that is #include-ed by +the main.c source file during the final compilation step. + +<h2>4.2 The translate preprocessor</h2> + +The translate preprocessor looks for lines of source code that begin +with "@" and converts those lines into string constants or (depending on +context) into special "printf" operations for generating the output of +an HTTP request. The translate preprocessor is a simple C program whose +sources are in the translate.c source file. The translate preprocess +is run on each of the other ordinary source files separately, like this: + +<pre> +./translate src.c >src_.c +</pre> + +In this case, the "src.c" file represents any single source file from the +set of ordinary source files as described in section 2.0 above. Note that +each source file is translated separately. By convention, the names of +the translated source files are the names of the input sources with a +single "_" character at the end. But a new makefile can use any naming +convention it wants - the "_" is not critical to the build process. + +After being translated, the output files (the "src_.c" files) should be +used for all subsequent preprocessing and compilation steps. + +<h2>4.3 The makeheaders preprocessor</h2> + +For each C source module "src.c", there is an automatically generated +header module "src.h" that contains all of the datatype and procedure +declarations needed by the source module. These header files are generated +automatically by the makeheaders program. The sources to makeheaders +are contained in a single file "makeheaders.c". Additional documentation +on makeheaders can be found in [../tools/makeheaders.html | tools/makeheaders.html]. + +The makeheaders program is run once. It scans all inputs source files and +generates header files for each one. Note that the sqlite3.c and shell.c +source files are not scanned by makeheaders. Makeheaders only runs over +"ordinary" source files, not the exceptional source files. However, +makeheaders also uses some extra header files as input. The general format +is like this: + +<pre> +makeheaders src_.c:src.h sqlite3.h th.h VERSION.h +</pre> + +In the example above the "src_.c" and "src.h" names represent all of the +(79) ordinary C source files, each as a separate argument. + +<h1>5.0 Compilation</h1> + +After all generated files have been created and all ordinary source files +have been preprocessed, the generated and preprocessed files can be +combined into a single executable using a C compiler. This can be done +all at once, or each preprocessed source file can be compiled into a +separate object code file and the resulting object code files linked +together in a final step. + +Some files require special C-preprocessor macro definitions. +When compiling sqlite.c, the following macros are recommended: + + * -DSQLITE_OMIT_LOAD_EXTENSION=1 + * -DSQLITE_ENABLE_DBSTAT_VTAB=1 + * -DSQLITE_ENABLE_FTS4=1 + * -DSQLITE_ENABLE_LOCKING_STYLE=0 + * -DSQLITE_LIKE_DOESNT_MATCH_BLOBS=1 + * -DSQLITE_THREADSAFE=0 + * -DSQLITE_DEFAULT_FILE_FORMAT=4 + * -DSQLITE_ENABLE_EXPLAIN_COMMENTS=1 + +The first three symbol definitions above are required; the others are merely +recommended. Extension loading is omitted as a security measure. The dbstat +virtual table is needed for the [/help/www/repo-tabsize|/repo-tabsize] page. +FTS4 is needed for the search feature. Fossil is single-threaded so mutexing +is disabled in SQLite as a performance enhancement. The +SQLITE_ENABLE_EXPLAIN_COMMENTS option makes the output of "EXPLAIN" queries +in the "[/help/sqlite3|fossil sql]" command much more readable. + +When compiling the shell.c source file, these macros are required: + + * -Dmain=sqlite3_main + * -DSQLITE_OMIT_LOAD_EXTENSION=1 + +The "main()" routine in the shell must be changed into sqlite3_main() +to prevent it from colliding with the real main() in Fossil, and to give +Fossil an entry point to jump to when the +[/help/sqlite3 | fossil sql] command is invoked. + +All the other source code files can be compiled without any special +options. + +<h1>6.0 Linkage</h1> + +Fossil needs to be linked against [http://www.zlib.net | zlib]. If +the HTTPS option is enabled, then it will also need to link against +the appropriate SSL implementation. And, of course, Fossil needs to +link against the standard C library. No other libraries or external +dependences are used. + +<h1>7.0 Debugging</h1> + +Debug mode is controlled via a FOSSIL_DEBUG preprocessor macro. This can be +set explicitly with the make command for the target platform. + +However, in practice it is instead recommended to add a respective configure +option for the target platform and then perform a clean build. This way the +Debug flags are consistently applied across the whole build process. For +example, use these Debug flags in addition to other flags passed to the +configure scripts: + +On Linux, *NIX and similar platforms: +<pre> +./configure --fossil-debug +</pre> + +On Windows: +<pre> +win\buildmsvc.bat FOSSIL_DEBUG=1 +</pre> + +The resulting fossil binary can then be loaded into a platform-specific +debugger. Source files displayed in the debugger correspond to the ones +generated from the translation stage of the build process, as that is what +was actually compiled into the respective object files that make up the +fossil binary. + +<h1>8.0 See Also</h1> + + * [./tech_overview.wiki | A Technical Overview Of Fossil] + * [./adding_code.wiki | How To Add Features To Fossil] |