I've started to update Glorp and GlorpSQLite for Pharo 8. This post lists the stuff to be handled.

Changes in Pharo

First, changes in Pharo from version to version. Glorp's TimedProxyReaper uses a weak-valued dictionary to hold TimedProxy instances. In Pharo 6, WeakValueDictionary>>at:put: essentially does the following:

WeakValueAssociation key: key value: anObject

In Pharo 7, that became:

WeakValueAssociation key: key value: anObject asSetElement

This required TimedProxy to implement #asSetElement.

In Pharo 8, #asSetElement is deprecated in favour of #asCollectionElement.

WeakValueAssociation key: key value: anObject asCollectionElement

So TimedProxy now also needs #asCollectionElement.

New SQLite Binding

The Pharo community has consolidated around Pharo-SQLite3 as the definitive SQLite binding going forward. GlorpSQLite uses the now-legacy UDBC-SQLite binding currently. This change should be straightforward.

Glorp Changes

Todd Blanchard has been working on Ruby on Rails-style ActiveRecord for Glorp, and testing the changes with PostgreSQL.

CI

With independently evolving drivers for SQLite, PostgreSQL and MySQL, and the ActiveRecord work changing Glorp itself, the time has come to set up CI for Glorp.

GlorpSQLite works on Pharo 7!

Take a fresh Pharo 7 alpha image; as of yesterday's download that is 5f13ae8. Launch it and run the following snippet in a Playground:

Metacello new
  baseline: 'GlorpSQLite';
  repository: 'github://PierceNg/glorp-sqlite3:pharo7dev';
  load.

Run the Glorp tests in TestRunner. The result should be green, with all 891 tests passed and 12 tests skipped. The database file is sodbxtestu.db in your image directory. Tested on 32- and 64-bit Ubuntu 18.04.

I've updated ConfigurationOfGlorp for Pharo 6 and added catalog methods to ConfigurationOfGlorpSQLite.

Take a fresh Pharo 60365 image, the latest as of yesterday's download. Launch it, open the Catalog Browser, and install GlorpSQLite from there.

Run the Glorp tests in TestRunner. The result should be green, with all 889 tests passed and 12 tests skipped. The database file is sodbxtestu.db in your image directory.

(On Pharo 5, Glorp runs and passes total of 953 tests. Something to look into.)

I've created ConfigurationOfGlorpSQLite on STH.

Take a fresh Pharo 5 image. Make a script like the following and run it on the image:

% cat loadGlorpSQLite.sh
#!/bin/sh
MCREPO=http://www.smalltalkhub.com/mc/DBXTalk/Glorp/main/
pharo $1.image config $MCREPO ConfigurationOfGlorpSQLite --install=stable

% ./loadGlorpSQLite.sh Pharo-50757

When done, fire up the image again, and run the Glorp tests in TestRunner. The result should be green, with all 953 tests passed and 12 tests skipped. The database file is sodbxtestu.db in your image directory.

I'm pleased to announce the release of Glorp-SQLite3 for Pharo 5.

Developed and tested on Linux. Known working on Windows 7. Your Pharo 5 VM needs to be able to find libsqlite3.so or the Windows equivalent.

Take a fresh Pharo 5 image. Run the following:

Gofer it
    smalltalkhubUser: 'TorstenBergmann' project: 'UDBC';
    configuration;
    load.
(Smalltalk at: #ConfigurationOfUDBC) loadBleedingEdge.

Gofer it
    smalltalkhubUser: 'DBXTalk' project: 'Glorp';
    configurationOf: 'Glorp';
    load.
#ConfigurationOfGlorp asClass project stableVersion load.

Gofer it
    smalltalkhubUser: 'DBXTalk' project: 'Glorp';
    package: 'Glorp-SQLite3';
    load.

GlorpSQLite3CIConfiguration new configureSqlite3.
GlorpDemoTablePopulatorResource invalidateSetup.

Run the Glorp tests in TestRunner. All tests should pass, with 12 tests skipped. The database file is sodbxtestu.db in your image directory.

Using the Pharo v40592 image with which I had verified NBSQLite3 for Glorp, in this blog post I go through doing the same with the PostgresV2 pure-Smalltalk database driver.

Outside of Smalltalk, create the database 'sodbxtest', user 'sodbxtest' with password 'sodbxtest':

# su postgres -c psql
postgres=# create role sodbxtest with password 'sodbxtest' login;
CREATE ROLE 
postgres=# create database sodbxtest;
CREATE DATABASE
postgres=# \q
#

In Smalltalk, firstly, install PostgresV2:

Gofer it
    smalltalkhubUser: 'PharoExtras' 
    project: 'PostgresV2';
    package: 'ConfigurationOfPostgresV2';
    load.
((Smalltalk at: #ConfigurationOfPostgresV2) project version: '2.4') load.

Open Test Runner and runs the PostgresV2 tests. On my Linux Mint machine, using a vanilla PostgreSQL 9.3 installation, 23 of 24 tests passed, and TestPGConnection>>#testNotify2 erred.

Now that we know the PostgresV2 driver can talk to our database, using the Monticello browser, open the PostgresV2 repository and load the package GlorpDriverPostgreSQL. Here I had to edit NativePostgresDriver>>connectionArgsFromCurrentLogin: to comment out the second last line:

connectionArgs clientEncoding: aLogin encodingStrategy asSymbol

This is because GlorpDatabaseLoginResource class>defaultPostgreSQLLocalLogin does not specify encodingStrategy, meaning it is nil and will respond to #asSymbol with DNU.

Next, in a playground, execute the following:

GlorpDemoTablePopulatorResource invalidateSetup.
GlorpDatabaseLoginResource
    defaultLogin: GlorpDatabaseLoginResource defaultPostgreSQLLocalLogin

Open Test Runner and run the Glorp tests.

Tested on Linux Mint 17.

I've integrated NBSQLite3 into Glorp on the current Pharo 4.0 v40592 beta image.

Firstly, install NBSQLite3 (TorstenBergmann.7) and then Glorp (TorstenBergmann.42) from the config browser.

Then, using the Monticello browser, open the NBSQLite3 repository and load the packages NBSQLite3-Glorp and NBSQLite3-Test-Glorp. Next, open the Glorp repository and load the packages Glorp-PierceNg.97.mcz and GlorpTests-PierceNg.44.mcz.

In a workspace/playground, execute the following:

GlorpDemoTablePopulatorResource invalidateSetup.
GlorpDatabaseLoginResource
    defaultLogin: GlorpDatabaseLoginResource defaultNBSQLite3LocalLogin

Open Test Runner and run the Glorp tests.

Tested on Linux Mint 17 and OSX Mavericks. 2 fewer tests passed on Linux.

Curiously, #testLargeBlob, #testLargeClob and #testLargeText passed on the Pharo 3 image that I wrote this code on.

The database file created by the tests is sodbxtest.db.

% sqlite3 sodbxtest.db
sqlite> .tables
AIRLINE                  GALLERY_LINK             PERISHABLE_ITEM        
AIRLINE_MEAL             GLORP_IMAGE              PERSON                 
ATTACHMENT               GLORP_IMAGE_FILE         POULTRY                
ATTACHMENTBYTES          GLORP_JOB                PUBLISHER_EMP          
BANK_ACCT                GLORP_OWNER              PUBLISHER_TITLE        
BANK_TRANS               GLORP_SLAVE              PUBLISHER_TITLE2       
BOOK                     GLORP_TAG                PUB_EMP_LINK           
BOOK_CUSTOMER            GLORP_TAGS               PUB_TITLES_STOCK       
COMPRESSED_MONEY_TABLE   GLORP_WORKER             PUB_TITLE_LINK         
CUSTOMER_ACCT_LINK       GLORP_WORKER_JOB_LINK    RECORD_WITH_UPDATE     
CUSTOMER_BOOK_LINK       GR_ADDRESS               RESERVATION            
DEFAULTABLE_THING        GR_CUSTOMER              STUFF                  
DOCUMENT                 GR_FOLDER                TAX                    
EMAIL_ADDRESS            GR_MESSAGE               TRANSFORMED_TIME       
EMPLOYEE                 GR_PUBLISHER             TREE_NODE              
ENCYC                    GR_THINGONE              TREE_NODE_LINK         
ENCYC_BIO                GR_THINGWITHCOLLECTIONS  UNASSEMBLED_ITEM       
ENCYC_ENTRY              GR_THING_LINK            VIDEO_CREDIT_STATUS    
ENCYC_ENTRY_LINK         GR_TRAVEL_AGENT          VIDEO_PURCHASE         
FKADDRESS                GR_USER                  VIDEO_PURCHASE_LINK    
FKCONTACT                IMAGETAGS                VIDEO_RENTAL           
FLIGHT                   ITINERARY                VIDEO_STORE            
FLIGHT_PASS              NONPERISHABLE_ITEM       WAREHOUSE              
FREQUENT_FLYER           OFFICE                   WAREHOUSE_ITEM_LINK    
GALLERY                  PASSENGER                WORKING_STIFF          
sqlite> 

Made really good progress with NBSQLite3 for Glorp.

On the failed tests:

The entire GlorpOptimisticLockingTest and GlorpTimestampTest suites are skipped, because some of the tests fail, and foobars the Pharo-SQLite interface, causing many subsequent tests to fail, requiring restart of the Pharo image. Still need to look into these.

Making good progress with NBSQLite3 for Glorp.

Sven van Caekenberghe has written a very nice tutorial implementing a Reddit clone in Pharo using Seaside, Glorp and PostgreSQL. Sven also makes available a prebuilt image containing the application.

Seeing that the image contains Glorp working with the PostgresV2 driver, I set about integrating NBSQLite3 with Glorp. After about an afternoon's work, I now have Reddit.st working with Glorp+NBSQLite3.

$ sqlite3 reddit.db
SQLite version 3.8.2 2013-12-06 14:53:30
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select * from reddit_links;
1|http://www.pharo.org|Pharo Smalltalk|2014-09-22 22:46:53|1
2|http://planet.smalltalk.org|Planet Smalltalk|2014-09-22 22:47:18|1
6|http://www.world.st/|The World of Smalltalk|2014-09-22 22:58:50|0
sqlite> 

There is still much to be done to get Glorp fully working with NBSQLite3: Some tests apparently expect Glorp proxies, but are getting OrderedCollections, and one particular test rendered my Linux X session non-responsive to mouse and keyboard, except for screen brightness key chords!

I'm reading the book Seven Databases in Seven Weeks by Eric Redmond and Jim Wilson. The databases covered are PostgreSQL, Riak, HBase, MongoDB, CouchDB, Neo4J and Redis.

There are several Pharo/Squeak libraries for PostgreSQL:

• PostgresV3 on SqueakSource.
• PostgresV2 on SqueakSource.
• Squeak_PostgreSQL by Jim Menard. Last updated in 2006.
• DBXTalk, using OpenDBX, which wraps libpq. Also has GlorpDBX, which is an ORM that uses DBXTalk or the PostgresV2 driver.

Riak:

• Runar Jordahl's EpigentRiakInterface.
• Goran Krampe's fork, Phriak.

HBase runs on JVM. It supports a RESTful HTTP API, Thrift, and a Java API. The first is probably the easiest way to write a Smalltalk interface.

MongoDB:

• Estaban Lorenzano's Voyage is an object persistence layer that works with MongoDB.
• MongoTalk, by Kent Beck and others.

CouchDB:

• SCouchDB by Igor Stasenko.
• SCouchDBViewServer, by Jesus Mari Aguirre, a fork and superset that includes other functionality.

Neo4J is a graph database. It is provides a RESTful API. I've not played with Neo4J, but I'd imagine the Smalltalk environment, and by extension any Smalltalk object persistence mechanism, make up a graph database. Probably speaking from ignorance here, but I'm not sure what interest a Smalltalk programmer will have in a graph database written in Java. :-)

Finally, there is Redis Client by Mike Hale and others.

I haven't finished the book, but so far I haven't seen any discussion on authentication or security of these HTTP-speaking NoSQL databases. If the database is lacking authentication or SSL, and if your threat model covers that, probably the easiest is to put these behind a proxy. And, for database and other such connectivity from the Smalltalk client, I suggest SpsSplitPasswordStore.

GOODS "is an object oriented fully distrbuted database management system using an active client model." It is also described as a "language-neutral object database" with client interfaces for C++, Java and Perl. GOODS is written by Konstantin Knizhnik.

Avi Bryant developed a Squeak client for GOODS that allows transparent storage of Smalltalk objects. The client is now maintained for Pharo, Squeak and VisualWorks by David Shaffer and is hosted on SqueakSource3.

GOODS works over TCP and Unix domain sockets. However, the GOODS documentation doesn't actually describe how to configure for the latter. There is, however, a hint in the main configuration file "goodsrv.cfg":

# Enable or disable acceptance of remote connections by GOODS server. 
# Disabling remote connections avoid any problem with firewall. In this case
# GOODS server is able to handle only local connections (Unix socket, Win32 local socket, process socket) 
server.remote.connections=1 # 1 or 0

In addition to the main configuration file, there is database-specific configuration file, which I've named "test.cfg", that looks like this:

1
0:localhost:60060

This matches the configuration format:

<number of storages = N>
<storage identifier 0>: <hostname>:<port>

Meaning, for test.cfg, I'm specifying one storage server, and it listens on TCP port 60060 on localhost.

But what about the Unix domain socket path? To find out, I set the server.remote.connections parameter to 0 and try it out:

% goodsrv test 0 -
Read the whole database size to page pool
16:52.59 28-JUL-2013: Checkpoint 37 finished
GOODS server started...
server is up...
>

What happened? Looking around, it is found that GOODS has created a Unix domain socket at "/tmp/localhost:60060". Yes, ":60060" is part of the socket's path name. Cute. Trying to use a more descriptive name like "goodsock" or whatever fails with "bad address". Hitting the GOODS source, unisock.cxx shows that that unix_socket_dir is hardcoded to "/tmp/", and that the file name format has to be "string:number".

Oh well. I've abstracted the procedure to obtain a Unix domain socket address from a path string from my previous post as follows:

NetNameResolver class>>addressForSocketPath: socketPath
  | size sa |

  NetNameResolver primGetAddressInfoHost: '' service: socketPath flags: 0 family: 1 type: 0 protocol: 0.
  size := NetNameResolver primGetAddressInfoSize.
  sa := SocketAddress new: size withAll: 0.
  NetNameResolver primGetAddressInfoResult: sa.
  ^ sa

However, asking for a Unix domain socket address for "/tmp/localhost:60060" causes primGetAddressInfoHost:blah:blah: to fail. Bummer.

Okay, it is easier to modify GOODS since I've been browsing its source, then to get well-acquainted with SocketPlugin. So, at line 107 of unisock.cxx, make this change:

//sprintf(u.name + offsetof(sockaddr,sa_data), "%s%s", unix_socket_dir, address);
sprintf(u.name + offsetof(sockaddr,sa_data), "%s%s", unix_socket_dir, hostname);

The commented out line is the original. "hostname" is "address" minus the colon and port number that comes after it.

Rebuild GOODS, change the path in test.cfg to say "goodserver:60060", restart, and we see that the Unix domain socket is now called "/tmp/goodserver" and "NetNameResolver addressForSocketPath: '/tmp/goodserver'" duly returns a SocketAddress instance.

Next, load the Squeak/Pharo GOODS client from SS3. Subclass KKSqueakTCPSocketTransport as KKPharoIPCSocketTransport, with the single method:

KKPharoIPCSocketTransport>>initializeSocketAddress: aSocketPath
  "Create and connect to specified Unix domain socket address."

  socket := Socket newIPC connectTo: (NetNameResolver addressForSocketPath: aSocketPath).
  (Delay forMilliseconds: 10) wait.

Some other corresponding modifications are needed, such as in KKDatabase and KKConnection. Finally, in workspace, run this:

| db |
KKDatabase defaultTransportClass: KKPharoIPCSocketTransport.
db := KKDatabase onSocketPath: '/tmp/goodserver'.
db root: Dictionary new.
db commit.
db explore.

And it works!

% goodsrv test 0 -
Read the whole database size to page pool
17:32.38 28-JUL-2013: Checkpoint 45 finished
GOODS server started...
server is up...
>
17:32.44 28-JUL-2013: Open session for client 'squeak16r38E5541'
Send class 'Dictionary' to client

Running "self logout" in the KKDatabase instance explorer from above results in additional output from GOODS:

17:39.23 28-JUL-2013: Client 'squeak16r38E5541' send logout request
17:39.23 28-JUL-2013: Disconnect 'squeak16r38E5541'
Server agent 'squeak16r38E5541' terminated