diff options
Diffstat (limited to 'www/antibot.wiki')
| -rw-r--r-- | www/antibot.wiki | 276 |
1 files changed, 276 insertions, 0 deletions
diff --git a/www/antibot.wiki b/www/antibot.wiki new file mode 100644 index 0000000..c38cfec --- /dev/null +++ b/www/antibot.wiki @@ -0,0 +1,276 @@ +<title>Defense Against Robots</title> + +A typical Fossil website can have billions and billions of pages, +and many of those pages (for example diffs and annotations and tarballs) +can be expensive to compute. +If a robot walks a Fossil-generated website, +it can present a crippling bandwidth and CPU load. +A "robots.txt" file can help, but in practice, most robots these +days ignore the robots.txt file, so it won't help much. + +A Fossil website is intended to be used +interactively by humans, not walked by robots. This article +describes the techniques used by Fossil to try to welcome human +users while keeping out robots. + +<h2>Defenses Are Enabled By Default</h2> + +In the latest implementations of Fossil, most robot defenses are +enabled by default. You can probably get by with standing up a +public-facing Fossil instance in the default configuration. But +you can also customize the defenses to serve your particular needs. + +<h2>Customizing Anti-Robot Defenses</h2> + +Admin users can configure robot defenses on the +"Robot Defense Settings" page (/setup_robot). +That page is accessible (to Admin users) from the default menu bar +by click on the "Admin" menu choice, then selecting the +"Robot-Defense" link from the list. + +<h2>The Hyperlink User Capability</h2> + +Every Fossil web session has a "user". For random passers-by on the internet +(and for robots) that user is "nobody". The "anonymous" user is also +available for humans who do not wish to identify themselves. The difference +is that "anonymous" requires a login (using a password supplied via +a CAPTCHA) whereas "nobody" does not require a login. +The site administrator can also create logins with +passwords for specific individuals. + +Users without the <b>[./caps/ref.html#h | Hyperlink]</b> capability +do not see most Fossil-generated hyperlinks. This is +a simple defense against robots, since [./caps/#ucat | the "nobody" +user category] does not have this capability by default. +Users must log in (perhaps as +"anonymous") before they can see any of the hyperlinks. A robot +that cannot log into your Fossil repository will be unable to walk +its historical check-ins, create diffs between versions, pull zip +archives, etc. by visiting links, because there are no links. + +A text message appears at the top of each page in this situation to +invite humans to log in as anonymous in order to activate hyperlinks. + +But requiring a login, even an anonymous login, can be annoying. +Fossil provides other techniques for blocking robots which +are less cumbersome to humans. + +<h2>Automatic Hyperlinks Based on UserAgent and Javascript</h2> + +Fossil has the ability to selectively enable hyperlinks for users +that lack the <b>Hyperlink</b> capability based on their UserAgent string in the +HTTP request header and on the browsers ability to run Javascript. + +The UserAgent string is a text identifier that is included in the header +of most HTTP requests that identifies the specific maker and version of +the browser (or robot) that generated the request. Typical UserAgent +strings look like this: + +<ul> +<li> Mozilla/5.0 (Windows NT 6.1; rv:19.0) Gecko/20100101 Firefox/19.0 +<li> Mozilla/4.0 (compatible; MSIE 8.0; Windows_NT 5.1; Trident/4.0) +<li> Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) +<li> Wget/1.12 (openbsd4.9) +</ul> + +The first two UserAgent strings above identify Firefox 19 and +Internet Explorer 8.0, both running on Windows NT. The third +example is the robot used by Google to index the internet. +The fourth example is the "wget" utility running on OpenBSD. +Thus the first two UserAgent strings above identify the requester +as human whereas the second two identify the requester as a robot. +Note that the UserAgent string is completely under the control +of the requester and so a malicious robot can forge a UserAgent +string that makes it look like a human. But most robots want +to "play nicely" on the internet and are quite open +about the fact that they are a robot. And so the UserAgent string +provides a good first-guess about whether or not a request originates +from a human or a robot. + +The [/help/auto-hyperlink|auto-hyperlink] setting, shown as +"<b>Enable hyperlinks based on User-Agent and/or Javascript</b>" on +the Robot Defense Settings page, +can be set to "HTTP Header Only" or "HTTP Header and Javascript" or "off". +If the UserAgent string looks like a human and not a robot, then +Fossil will enable hyperlinks even if the <b>Hyperlink</b> capability +is omitted from the user permissions. This setting gives humans easy +access to the hyperlinks while preventing robots +from walking the billions of pages on a typical Fossil site. + +If the setting is "HTTP Header Only" (2), then the hyperlinks are simply +enabled and that is all. But if the setting is "HTTP Header and Javascript" (1), +then the hyperlinks are not enabled directly. +Instead, the HTML code that is generated contains anchor tags ("<a>") +with "href=" attributes that point to [/honeypot] rather than the correct +link. JavaScript code is added to the end of the page that goes back and +fills in the correct "href=" attributes of +the anchor tags with the true hyperlink targets, thus enabling the hyperlinks. +This extra step of using JavaScript to enable the hyperlink targets +is a security measure against robots that forge a human-looking +UserAgent string. Most robots do not bother to run JavaScript and +so to the robot the empty anchor tag will be useless. But all modern +web browsers implement JavaScript, so hyperlinks will show up +normally for human users. +(UPDATE 2026-08-06: The previous two sentences +were true when this document was first written in 2013, but they are +not true any more. Most robots these days have normal-looking UserAgent +strings and do run Javascript.) + +If the [/help/auto-hyperlink|"auto-hyperlink"] setting is (2) +"<b>Enable hyperlinks using HTTP-Header and/or Javascript</b>", +then there are now two additional sub-settings that control when +hyperlinks are enabled. + +The first new sub-setting is a delay (in milliseconds) before setting +the "href=" attributes on anchor tags. The default value for this +delay is 10 milliseconds. The idea here is that a robots will try to +interpret the links on the page immediately, and will not wait for delayed +scripts to be run, and thus will never enable the true links. + +The second sub-setting waits to run the +JavaScript that sets the "href=" attributes on anchor tags until after +at least one "mousedown" or "mousemove" event has been detected on the +<body> element of the page. The thinking here is that robots will not be +simulating mouse motion and so no mouse events will ever occur and +hence the hyperlinks will never become enabled for robots. + +See also [./loadmgmt.md|Managing Server Load] for a description +of how expensive pages can be disabled when the server is under heavy +load. + +<h2>Do Not Allow Robot Access To Certain Pages</h2> + +The [/help/robot-restrict|robot-restrict setting] is a comma-separated +list of GLOB patterns for pages for which robot access is prohibited. +The default value is: + +<blockquote><pre> +timelineX,diff,annotate,fileage,file,finfo,reports,tree,hexdump,download +</pre></blockquote> + +Each entry corresponds to the first path element on the URI for a +Fossil-generated page. If Fossil does not know for certain that the +HTTP request is coming from a human, then any attempt to access one of +these pages brings up a javascript-powered captcha. The user has to +click the accept button the captcha once, and that sets a cookie allowing +the user to continue surfing without interruption for 15 minutes or so +before being presented with another captcha. + +To disable this feature (not recommended for public-facing websites) +change this setting to "off". If robots become particularly troublesome, +change this setting to "*" (the GLOB wildcard) which restricts robots +from accessing any page of the Fossil webpage other than "/login". +Robots have gotten so bad lately (as of 2026) that the use of "*" is +advisable for any public-facing website that is running on a low-power +platform such as a RaspberryPI or a $6/month VPS, or any platform with +limited bandwidth. + +Some path elements have special meanings: + + * <b>timelineX →</b> + This means a subset of /timeline/ pages that are considered + "expensive". The exact definition of which timeline pages are + expensive and which are not is still the subject of active + experimentation and is likely to change by the time you read this + text. The idea is that anybody (including robots) can see a timeline + of the most recent changes, but timelines of long-ago change or that + contain lists of file changes or other harder-to-compute values are + prohibited. + + * <b>zip →</b> + The special "zip" keyword also matches "/tarball/" and "/sqlar/". + + * <b>zipX →</b> + This is like "zip" in that it restricts access to "/zip/", "/tarball"/ + and "/sqlar/" but with exceptions:<ol type="a"> + <li><p> If the [/help/robot-zip-leaf|robot-zip-leaf] setting is + true, then tarballs of leaf check-ins are allowed. This permits + URLs that attempt to download the latest check-in on trunk or + from a named branch, for example. + <li><p> If a check-in has a tag that matches the GLOB list in + [/help/robot-zip-tag|robot-zip-tag], then tarballs of that + check-in are allowed. This allow check-ins tagged with + "release" or "allow-robots" (for example) to be downloaded + without restriction. + </ol> + The "zipX" restriction is not in the default robot-restrict setting. + This is something you might want to add, depending on your needs. + + * <b>diff →</b> + This matches /vdiff/ and /fdiff/ and /vpatch/ and any other page that + is primarily about showing the difference between two check-ins or two + file versioons. + + * <b>annotate →</b> + This also matches /blame/ and /praise/. + +Other special keywords may be added in the future. + +The default [/help/robot-restrict|robot-restrict] +setting has been shown (circa 2026) to do a reasonable job of keeping +robots from consuming all available CPU and bandwidth while will +still allowing humans access to the full power of the site without +having to be logged in. + +One possible enhancement is to add "zipX" to the +[/help/robot-restrict|robot-restrict] setting, +and enable [help?cmd=robot-zip-leaf|robot-zip-leaf] +and configure [help?cmd=robot-zip-tag|robot-zip-tag]. +Do this if you find that robots downloading lots of +obscure tarballs is causing load issues on your site. + +<h2>Anti-robot Exception RegExps</h2> + +The [/help/robot-exception|robot-exception setting] under the name +of <b>Exceptions to anti-robot restrictions</b> is a list of +[/re_rules|regular expressions], one per line, that match +URIs that will bypass the captcha and allow robots full access. The +intent of this setting is to allow automated build scripts +to download specific tarballs of project snapshots. + +The recommended value for this setting allows robots to use URIs of the +following form: + +<blockquote> +<b>https://</b><i>DOMAIN</i><b>/tarball/release/</b><i>HASH</i><b>/</b><i>NAME</i><b>.tar.gz</b> +</blockquote> + +The <i>HASH</i> part of this URL can be any valid +[./checkin_names.wiki|check-in name]. The link works as long as that +check-in is tagged with the "release" symbolic tag. In this way, +robots are permitted to download tarballs (and ZIP archives) of official +releases, but not every intermediate check-in between releases. Humans +who are willing to click the captcha can still download whatever they +want, but robots are blocked by the captcha. This prevents aggressive +robots from downloading tarballs of every historical check-in of your +project, once per day, which many robots these days seem eager to do. + +For example, on the Fossil project itself, this URL will work, even for +robots: + +<blockquote> +https://fossil-scm.org/home/tarball/release/version-2.27/fossil-scm.tar.gz +</blockquote> + +But the next URL will not work for robots because check-in 3bbd18a284c8bd6a +is not tagged as a "release": + +<blockquote> +https://fossil-scm.org/home/tarball/release/3bbd18a284c8bd6a/fossil-scm.tar.gz +</blockquote> + +The second URL will work for humans, just not robots. + +<h2>The Ongoing Struggle</h2> + +Fossil currently does a good job of providing easy access to humans +while keeping out troublesome robots. However, robots +continue to grow more sophisticated, requiring ever more advanced +defenses. This "arms race" is unlikely to end. The developers of +Fossil will continue to try improve the robot defenses of Fossil so +check back from time to time for the latest releases and updates. + +Readers of this page who have suggestions on how to improve the robot +defenses in Fossil are invited to submit your ideas to the Fossil Users +forum: +[https://fossil-scm.org/forum]. |