Pharo headless VM

First, build the Pharo headless VM, following the instructions in its README. When done, the VM files are in build/vm.

Pharo application image

Next, prepare the Pharo application image. Basically, start from a fresh Pharo 8.0 image and load the EmbeddedSupport code from Pablo's repository. The following snippet works, assuming you have cloned Pablo's repository locally; adjust the gitlocal path in the snippet, of course.

Metacello new
	baseline: 'EmbeddedSupport';
	repository: 'gitlocal:///home/pierce/src/st/pharo-vm-embedded-example/smalltalk-src';
	load.

NoChangesLog install.
NoPharoFilesOpener install.
PharoCommandLineHandler forcePreferencesOmission: true.

SmalltalkImage current snapshot: true andQuit: true

Windows resource

Then, embed the Pharo application image into a Windows resource file.

Here's a simplified version of resources.rc. This assumes you have named the image to be embedded Pharo.image.

300 RCDATA "Pharo.image"

Building a Windows resource file on Linux requires windres. On Ubuntu, this program is part of the package binutils-mingw-w64-x86-64.

Place the application Pharo.image and resources.rc in the same directory. Then,

% /usr/bin/x86_64-w64-mingw32-windres -i resources.rc -o resources.res
% ls -l
total 113488
-rw-r--r-- 2 pierce pierce 58098672 Jan 23 14:10 Pharo.image
-rw-r--r-- 2 pierce pierce       25 Jan 23 14:01 resources.rc
-rw-r--r-- 2 pierce pierce 58098736 Jan 23 14:10 resources.res

The output resources.res is the compiled resources file.

Pascal host program

Finally we are ready to build the Pascal host program. Move resources.res to this repository's embedded-cli directory. Also create a link to libPharoVMCore.so or make a copy of it. The directory's content is now as follows:

% ls -l
total 58736
-rwxr-xr-x 2 pierce pierce  3344760 Jan 23 09:26 libPharoVMCore.so*
-rw-r--r-- 2 pierce pierce 58098736 Jan 23 14:10 resources.res
-rw-r--r-- 4 pierce pierce     1945 Jan 23 12:55 runPharoEmbedded.pas
-rw-r--r-- 4 pierce pierce     3824 Jan 23 12:54 ulibEmbeddedImage.pas
-rw-r--r-- 5 pierce pierce     1045 Jan 22 15:42 ulibPharoVM.pas

At the start of runPharoEmbedded.pas, the line {$R resources.res} is the Pascal compiler directive to incorporate resources.res into the executable that is being built:

program runPharoEmbedded;

{$mode objfpc}{$H+}
{$R resources.res} { <= compiler directive to incorporate resources.res }
{$linklib m}

Build the host program - the Pascal compiler is its own make system and knows to compile the necessary source files. (The Pascal compiler also knows how to invoke windres to compile resources.rc into resources.res when so configured. I've done that part by hand since this is a HOWTO.)

% fpc runPharoEmbedded.pas
Free Pascal Compiler version 3.0.4 [2018/10/29] for x86_64
Copyright (c) 1993-2017 by Florian Klaempfl and others
Target OS: Linux for x86-64
Compiling runPharoEmbedded.pas
Compiling ulibPharoVM.pas
Compiling ulibEmbeddedImage.pas
Compiling resource runPharoEmbedded.or
Linking runPharoEmbedded
/usr/bin/ld: warning: link.res contains output sections; did you forget -T?
232 lines compiled, 0.4 sec

% ls -l runPharoEmbedded
-rwxr-xr-x 1 pierce pierce 58884328 Jan 23 14:22 runPharoEmbedded*

Much of the size of the executable is due to the embedded Pharo.image.

We'll run runPharoEmbedded in the headless VM build directory because Pharo's baked-in library lookup currently requires this. (More on this in a later post.) So move the program over.

% mv runPharoEmbedded ~/src/st/opensmalltalk-vm-pharo/build/vm
% cd ~/src/st/opensmalltalk-vm-pharo/build/vm
% ls -T 30
libB2DPlugin.so*             libgit2.so.0.25.1             libSDL2-2.0.so.0*      libssl.so*
libBitBltPlugin.so*          libgit2.so.25                 libSDL2-2.0.so.0.7.0*  libssl.so.1.0.0*
libcrypto.so.1.0.0*          libIA32ABI.so*                libSDL2.so*            libSurfacePlugin.so*
libDSAPrims.so*              libJPEGReaderPlugin.so*       libSecurityPlugin.so*  libTestLibrary.so*
libffi.so*                   libJPEGReadWriter2Plugin.so*  libSocketPlugin.so*    libUnixOSProcessPlugin.so*
libffi.so.7*                 libLargeIntegers.so*          libSqueakFFIPrims.so*  libUUIDPlugin.so*
libffi.so.7.1.0*             libLocalePlugin.so*           libSqueakSSL.so*       pharo*
libFileAttributesPlugin.so*  libMiscPrimitivePlugin.so*    libssh2.so*            runPharoEmbedded*
libFilePlugin.so*            libPharoVMCore.so*            libssh2.so.1*
libgit2.so                   libPThreadedPlugin.so*        libssh2.so.1.0.1*

Set up LD_LIBRARY_PATH. The first path segment is for the Pharo VM. The second is for libcairo2.so needed by the embedded Pharo.image - on Ubuntu, it lives in /usr/lib/x86_64-linux-gnu, which isn't in Pharo 8's current hardcoded lookup path. Then run the executable:

% export LD_LIBRARY_PATH=`pwd`:/usr/lib/x86_64-linux-gnu
% uname -a
Linux Otrus 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
% ./runPharoEmbedded
lala

Ta da! "lala" is printed by Pharo.

Recently Pablo Tesone published an example of embedding a Pharo image as a Windows resource inside a program written in C.

Well, the excellent and, like Smalltalk, vastly underrated Free Pascal programming system has cross-platform support for handling Windows resources. With it, I have reimplemented Pablo's example: One source code building executables that run on Ubuntu 18.04 and MacOS Mojave. Not tested on Windows yet, but I'm sure it'll work there too. The three Pascal source files, including blank lines and comments, total 209 lines:

• ulibPharoVM.pas, FFI to libPharoVMCore, comprising one data type and two functions.

• ulibEmbeddedImage.pas, implementing the functions called by libPharoVMCore to access the Pharo image that is embedded as a Windows resource.

• runPharoEmbedded.pas, the main program which is pretty much a straight port of Pablo's C program.

Repository on GH. More details to come.

I've pushed another update to libffidemo, adding the class DemoFFITypeMappedThing and tests. With Pharo v50647 image's UFFI enhancements, all DemoFFI tests pass.

DemoFFITypeMappedThing allows using the C typedef demo_thing in #ffiCall: function signatures:

DemoFFILibrary>>apiAllocByMappedPointerPointer: handle
    "int alloc_by_pointer_pointer(demo_thing**)"
    ^ self ffiCall: #(int alloc_by_pointer_pointer (demo_thing **handle))

The C and Smalltalk signatures match. The ability to specify #ffiCall: function signatures very much like the actual C function signatures should make it easier to create bindings for UFFI.

The magic of DemoFFITypeMappedThing lies in FFIOpaqueObject, and,

DemoFFILibrary class>>initializeTypeMap
    TypeMap := Dictionary newFromPairs: #(
        demo_thing    FFIOpaqueObject
    )

With the above, UFFI maps (a pointer to) demo_thing in #ffiCall: signature to FFIOpaqueObject.

Compare and contrast with the earlier versions of DemoFFI:

DemoFFILibrary>>apiAllocByPointerPointer: handle
    "int alloc_by_pointer_pointer(demo_thing**)"
    ^ self ffiCall: #(int alloc_by_pointer_pointer (DemoFFIExternalObject *handle))

I've pushed an update to libffidemo, adding the class DemoFFIAutoThing and tests.

DemoFFIAutoThing sends #autoRelease to its handle when said handle is filled by libffidemo with a proper pointer. To support auto-release, DemoFFIExternalObject implements the following class-side method:

finalizeResourceData: resourceData
    DemoFFILibrary uniqueInstance
        ffiCall: #(int free_thing (ExternalAddress resourceData))

Note the C function signature, which is different from DemoFFILibrary>>apiFree:'s:

apiFree: handle
    "int free_thing(demo_thing*)"
    ^ self ffiCall: #(int free_thing (DemoFFIExternalObject handle))

DemoFFIAutoThing>>letGo becomes a no-op. Making it send "super letGo", which effectively calls DemoFFILibrary>>apiFree:, crashes the VM.

I've published libffidemo, a simple C source file meant to be built as a shared library, and DemoFFI, a Smalltalk package that calls libffidemo to, well, demonstrate Pharo's new Unified FFI. DemoFFI was originally published on Smalltalkhub; Damien Pollet kindly repackaged the Smalltalk code, wrote a Makefile for the C source, etc and now libffidemo is a single self-contained repository with all necessary source files. The unit tests serve as the examples. Known working on Linux and OS X.

Periodically, I build the CogVM from source on FreeBSD. I've prevously mentioned that I had encountered some oddity with FFI on that platform. Still there today.

The following code crashes the VM/image:

| db |
db := SQLiteConnection fileNamed: '/tmp/x.db'.
db withResultForQuery: 'select * from a;'
  do: [ :result |
    result do: [ :row |
      Transcript show: row asString; cr ]].
db close.

The oddity is that one particular version of the SqueakFFIPrims plugin works. Several other versions that I've built all fail. Overwriting the failed versions of the plugin with the known-working one allows that particular crashing VM's FFI to work.

-rwxr-xr-x  1 root  wheel  77035 Oct 21  2012 ./4.0-2562/SqueakFFIPrims
-rwxr-xr-x  1 root  wheel  77035 Oct 21  2012 ./4.0-2586/SqueakFFIPrims
-rwxr-xr-x  1 root  wheel  74637 Nov 11  2012 ./4.0-2586/SqueakFFIPrims.broken
-rwxr-xr-x  1 root  wheel  77035 Oct 21  2012 ./4.0-2779/SqueakFFIPrims

The working one appears here three times. I'm running FreeBSD 8.3.