Automating web content deployment with Fossil

12 August 2016

Update Mar 2021 - This blog now uses Fossil's newer hook mechanism. The HOWTO described in this post is obsolete.

I've migrated this site to Pharo 5 from Pharo 3. One notable behind-the-scene change is that content is now deployed using Fossil, the "simple, high-reliability, distributed software configuration management system". Fossil is usually compared with DVCS such as Git and Mercurial.

This server runs the published version of this site. (Of course.) The development version is on my laptop. Now, when I wish to publish changes from my development version, say, when I have written a new blog post, I 'fossil commit', then 'fossil push', whereupon the new blog post and associated artefacts such as images are deployed into the published version on this server.

There are any number of articles on the web describing using Git hooks to automate deploying web content. In this post, I describe using Fossil to do the same. Some familiarity with Fossil is assumed.


To begin, build Fossil with TH1 hooks and Tcl integration. Deploy it both locally and on the server, by copying the single Fossil executable to appropriate directories.

Create a new repostory locally. Take note of the admin user ID and generated password. Let's assume that the admin user ID is 'laptop-user'. Be aware that Fossil has its own user IDs, passwords and roles, distinct from the underlying operating system's.

laptop% mkdir ~/repo
laptop% cd ~/repo
laptop% fossil init webtree.fossil
project-id: 3c05c3016eeabf8e87816ee218c6a86d3c87b950
server-id:  ff42bc86dba1a26b1d94b64685f7c09d02581617
admin-user: laptop-user (initial password is "1fe2ff")

Check out the repository locally:

laptop% mkdir ~/webtree
laptop% cd ~/webtree
laptop% fossil open ~/repo/webtree.fossil

Add new content:

laptop% cd ~/webtree
laptop% echo "Hello, world, from Fossil." > helloworld.txt
laptop% fossil add helloworld.txt 
laptop% fossil commit -m "First commit." 

Upon commit, the repository, ~/repo/webtree.fossil, is updated. Note that the repository is a single file that can be copied, renamed, moved to other computers, etc. Let's scp it to our server and set up the following directory structure there. Here I use /x to keep path names short for this post.

    fossil         <= The Fossil executable.
    fossilupdate   <= Shell script, described below.
  webtree/         <= Checked out tree of webtree.fossil.
    webtree.fossil <= The repository.

To populate /x/webtree:

server% cd /x/webtree
server% fossil open /x/repo/webtree.fossil
server% ls -a
./  ../  .fslckout  helloworld.txt

The file .fslckout is Fossil's checkout database.

Create the shell script /x/bin/fossilupdate:

server% cd /x/bin
server% cat > fossilupdate
cd /x/webtree
/x/bin/fossil update
server% chmod 755 fossilupdate

Run the Fossil server. Here I use port 8081.

server% cd /x/webtree
server% /x/bin/fossil server --port 8081
Listening for HTTP requests on TCP port 8081

With your web browser, navigate to the Fossil server and login using the admin user ID and password you noted down earlier. Go to Admin > Settings. Disable autosync, enable tcl, enable th1-hooks, and click 'Apply Changes'. Go to Admin > Transfers > Push. Enter below command into the text box and click 'Apply Changes'.

tclInvoke exec /x/bin/fossilupdate & 

This sets up the Fossil transfer hook, which will fire after your Fossil server processes a 'fossil push' request.

Back on the laptop, commit another file:

laptop% cd ~/webtree
laptop% echo "Hello again." > take2.txt
laptop% fossil add take2.txt
laptop% fossil commit -m "Take 2."
laptop% fossil push http://laptop-user@server:8081/
password for laptop-user: 
remember password (Y/n)? n
Round-trips: 2   Artifacts sent: 2  received: 0
Push done, send: 1075  received: 1199  ip: <server>

The push from the laptop will trigger the abovementioned transfer hook configured in the server's Fossil server instance, which will update /x/webtree on the server. Viola!

Automatically starting the Fossil server

It is often desirable to start the Fossil server automatically upon server startup. On my server, I use daemontools. However, for some reason daemontools' setuidgid program is unable to run 'fossil server' in the correct directory. So I use Ubuntu's start-stop-daemon instead, and this is the daemontools run file:

exec /sbin/start-stop-daemon --start \
  -c cms:cms -d /x/webtree \
  --exec /x/bin/fossil \
  -- server --port 8081

Create a new user, say, 'cms', and set up ownership of /x:

User:Group   Dir/File
root:root    /x/
root:root      bin/
root:root        fossil
root:root        fossilupdate
cms:cms        webtree/
cms:cms        repo/
cms:cms          webtree.fossil
root:root        run <= daemontools run file

Link /x/repo into daemontools, and 'fossil server' runs as the 'cms' user in /x/webtree.


/x/webtree/.fslckout is Fossil's checkout database. If your web server serves content directly from /x/webtree, you should configure it to not serve the .fslckout file.

An alternate practice (according to those Git hook articles) is to rsync the content of /x/webtree to another directory, and it is this second directory that the web server reads from. In such a case, it is still necessary to avoid rsync'ing the .fslckout file.

Users in Fossil

As mentioned, Fossil maintains its own user IDs, passwords, and roles. In this post, I assumed that the Fossil admin user is called 'laptop-user' and used it for 'fossil push'. It is preferable to set up a separate lower privileged Fossil user and use that instead.

Now Has Dynamic Content

6 April 2012

This blog began life as a set of static pages, generated by a home-grown content management system written in Smalltalk, imaginatively called SmallCMS1.

I've now rewritten SmallCMS1 to serve content dynamically, to support tag linking, like this: SQLite.

Each blog post page now has forward and backward navigational links just above the blog post title.

Rendering code now uses Seaside. More than a year ago, I blogged on that. Seaside now has a cleaner way to render static HTML, or maybe that previous blog post got it wrong. Anyhow, here's how SmallCMS1 uses Seaside's HTML rendering engine:

^ WAHtmlCanvas builder
    fullDocument: true;
    rootBlock: [ :root | self renderSiteRootOn: root ];
    render: [ :html | self renderContentOn: html ].

Similarly, RSS is rendered thusly:

^ RRRssRenderCanvas builder
    fullDocument: true;
    render: [ :rss | self renderRssOn: rss ]).

Seaside and static files with Comanche

10 July 2011

I've had this for a while. Putting it up here in case this is helpful to others. To use, create "htdocs" in "FileDirectory default". Web-wise, "htdocs" is known as "/static". I serve my CSS and JS files from there.

WAComancheAdaptor subclass: #WAComancheStaticAdaptor
    instanceVariableNames: ''
    classVariableNames: ''
    poolDictionaries: ''
    category: 'Seaside-Adaptors-Comanche'

    | contentPath dirPath  svc ma |

    contentPath := 'htdocs'.
    dirPath := FileDirectory default fullNameFor: contentPath.
    svc := (HttpService on: self port)
        name: 'seaside-' , self port greaseString.
    ma := ModuleAssembly core.
    ma alias: '/static' to: [
        ma serverRoot: dirPath.
        ma documentRoot: dirPath.
        ma directoryIndex: 'index.html index.htm'.
        ma serveFiles ].
    ma addPlug: self.
    svc plug: ma rootModule.
    ^ svc

Web Content Management in Smalltalk

9 April 2011

This site is made up of static pages, managed by a living web content management system written using Pharo Smalltalk, imaginatively called SmallCMS1. Content is written in Markdown syntax, and transformed into HTML using Discount. Templates are written in Smalltalk a la Seaside, but using HttpView2's html generator.

There are presently two main templates in SmallCMS1, for "regular" HTML pages and blog posts respectively. The template for regular pages invokes Discount to "HTML-ize" a page's content, and renders said content into a full-fledged HTML page. The blog post template does the same, and additionally generates a list of (currently non-hyperlinked) tags and Disqus Javascript. SmallCMS1 also generates a static RSS feed file of the blog posts.

So far, for the small number of pages that make up this site, SmallCMS1 works well. Even with just three templates - the two mentioned above, plus the one that generates index.html - there has been a fair bit of refactoring of the templating code. Here's the current "driver" method:

renderTemplateFor: aFileNode on: html with: aBlock

html html: [
    self renderSiteHeadFor: aFileNode on: html.
    html body: [
        self renderPageHeaderOn: html.
        html div id: 'wrap'; with: [
            html div class: 'inner'; with: [
                html div id: 'content'; with: [
                    aBlock value: aFileNode value: html ]].
            html text: '<br class="clear" />' ].
        self renderFooterOn: html ]] 

Here's how the blog post template calls the above method:

renderBlogPost: aFileNode on: html

    self updateMenu: aFileNode.

    self renderTemplateFor: aFileNode 
        on: html 
        with: [ :aNode :aCanvas | self renderBlogPostBody: aNode on: aCanvas ] 

And the index.html template calls it thusly:

renderSiteIndexFor: aFileNode on: html to: aStream

    self renderTemplateFor: aFileNode 
        on: html 
        with: [ :aNode :aCanvas |
            self renderIndexBodyOn: aCanvas.
            self renderSideBarOn: aCanvas ].

    aStream nextPutAll: html render 

Now I wish to generate an iPhone-friendly version of this site and have obtained suitable HTML templates and CSS. For each content file, SmallCMS1 generates a corresponding HTML file. D'ingTSTTCPW, I decided to generate an "iPhone-optimized" version of each HTML file, prefixing "m_" to each file name.

I began to modify the Smalltalk-coded templates, by sprinkling "isMobile" code forks here and there, expecting to refactor the template code later. The code started to look messy quite quickly though.

So I tried another approach: For each generated HTML file, parse it to extract the content fragment, and render this content fragment through an iPhone-specific template to generate the corresponding "m_" HTML file.

How that is done is the topic for the next post.