Databases and Persistence
Last updated at 10:27 pm UTC on 10 November 2023
is the master page which summarizes the various databases and other forms of persistense supported by Squeak. See also Persistence.
Summary
This began as a rewrite of a Seaside mailing list post. Though its been edited since.
Just save the image
See below. This is sometimes practical, but you end up saving a lot that isn't your apps data.
SMFileDatabase
Minimal amount of code, not ACID compliant, but gives you about 1/2 of Prevalyer transparently (no need to use Command pattern). One step up from the image, from Ramon Leon: SMFileDatabase
OmniBase
Discard if you intend to use the free version with Squeak.
Magma
Excellent code integration, probably the most elegant of all.
Can run as server and client images separately.
Has a great query language that no one else has
Has special classes for large collections that you can create indexes to specific attributes
Has programmable ReadStrategies
Has excellent support from its creator Chris Muller, and the list.
The bad thing: It is a bit too slow. Both for inserting and for querying, even on mid sized collections (a few thousand elements) with indexes and optimal read strategies can still be pretty slow. (to check if this is still the case)
More see Magma
Glorp / Postgres
see GLORP: Generic Lightweight Object-Relational Persistence
Background
SqueakImage-based persistence
FileDictionary
ImageSegments
OmniBase
MinneStore
GemSqueak
Kats
MySQL
PostgreSQL
SqueakDBX
GOODS
ODBC
SQLite
SIXX
BerkeleyDBPlugin
Magma
Glorp
Tantalus
SPrevayler
J2SBridge
CouchDB
Squeak Image
Squeak (as all Smalltalk implementations worthy of that name) implements persistency in a way which is (almost) unseen in any other computational environment. It does not need databases, XML or any similar superfluous stuff to be persistent. Just try: Smalltalk snapshot: true andQuit: false.
You can also create a process which saves the image periodically instead of you.
[[true] whileTrue:
[(Delay forSeconds: 1000) wait.
Smalltalk snapshot: true andQuit: false ]]
forkAt: 49.
But keep in mind that this type of persistence is not ACID. You may lose data. In cases where it is not acceptable to lose data, some of us think databases are not superfluous at all (Gerald Zincke). It also involves shutting down and restarting all those objects that involve external resources; see the various implementors of #shutDown etc.
File Dictionary
The File Dictionary is an extremely simple database structure. It allows a large (larger than memory) file to be treated as a Dictionary by keeping an in-memory index. It uses native Squeak serialization (DataStream and ReferenceStream version available). It does not support transactions/concurrency/etc. For details see FileDictionary: Tiny Squeak Database
ImageSegments
ImageSegments may also be used as a simple kind to store objects while at the same time keeping the footprint of your main image low.
Details see : Introduction to ImageSegments.
OmniBase
OmniBase is a production-strength object database with gargabe collection and on-line backup support originally written by David Gorisek. Unfortunately his website is no longer online. OmniBase is written 100% in Smalltalk and runs on Squeak, Dolphin, VW, VAST and ST/X. Database files can be written and read from any Smalltalk dialect. There is still a VW/CinCom page about it at https://www.cincomsmalltalk.com/main/community/product-portal/contributed/development-license-components/omnibase/. There is a Pharo port mentioned on https://github.com/pharo-nosql/OmniBase. Those two pages seem to report quite different licensing statuses, so maybe it is more complex.
A version of OmniBase is available for Squeak 3.4 or thereabouts (2005-ish) on SqueakMap, maybe it is still usable. Cees de Groot was porting it to Squeak, but his site no longer exists.
There are a lot of hits for 'omnibase' these days and several seem to refer to quite different data systems.
CouchDB
For more details, see CouchDB
MinneStore
MinneStore is a free object database that has been ported to Squeak. Information about MinneStore can be found at: http://minnestore.sourceforge.net/ The latest files are in http://sourceforge.net/projects/minnestore/
Now it works in 3.2.
Minnestore has been around for along time and seen a fair bit of use. It feels somewhat like using a relational DB - you have to declare type and relationship info for your model. In return, you get automatic indexing and query facilities. It supports transactions.
Questions:
- What is the state of concurrent access?. Not finished, still can be two users locked tryng to change the same object.
- Does it has GC? Not by now (02/25/2003)
GemSqueak
Squeak Client for GemStone/S. GemStone is an industrial-strength (and not inexpensive) object-oriented database based on Smalltalk. Please visit the GemSqueak page to learn more about it.
Kats
Kats is a transaction service for smalltalk that is used in conjunction with persistency and ORB services. It is used to help manage the synchronization of object state with external systems. See Kats for more details.
MySQL
Bolot Kerimbaev is maintaining the Squeak port of Josh Miller's VisualWorks MySQL driver. See the MySQL page for more details.
SqueakDBX
The aim of this project is to build an OpenDBX (http://www.linuxnetworks.de/doc/index.php/OpenDBX) plugin which will allow users to perform relational database operations (DDL, DML and SQL) through an open source library.
Through this feature, the squeak community will hopefully be able to interact with major database engines, such as oracle and mssql, besides those which are open source, like postgresql and mysql. Moreover, by integrating this plugin with GLORP (http://www.glorp.org/), will allow us to generate a complete and open source solution to relational data base access.
To make this possible, the Squeak Foreign Function Interface (http://wiki.squeak.org/squeak/1414/) will be used. FFI is used to call functions located in shared libraries (OpenDBX in this case) which are not part of the Squeak VM nor its plugins. Don't worry, queries are asynchronous, so the time of the FFI blocks is very low and unimportant.
For more details, see SqueakDBX
PostgreSQL
The latest postgresql interface can be found at http://www.squeaksource.com/PostgresV3.html The version currently (2023-11-10) on SqueakMap is not this version, which may be confusing.
GOODS
Avi Bryant wrote the beginnings of a Squeak interface to the GOODS object database.
ODBC
ODBC for Squeak: Very simple framework for ODBC support in Squeak
SQLite
SIXX - Smalltalk Instance eXchange in XML
BerkeleyDB Plugin
Actually, this is more than just a plugin...it's an object database implementation using BerkeleyDB. See: http://spair.swiki.net/27
Magma
Magma is a multi-user object database designed for Squeak systems from the ground up. It provides transaction support and transparent access.
See Magma. Magma FAQs
Glorp
Nevin Pratt ported Glorp to Squeak. His porting "diary" can be found at:
http://home.earthlink.net/~nevinpratt/GlorpPort.doc
The Squeak changeset can be found at:
http://www.smalltalkpro.com/squeakextras.html
Or, you can get it from SqueakMap.
J2Sbridge
A JDBC bridge for Squeak. Up to now very simple, supports only Strings, Integers and
Binary objects (blobs).
More here: J2SBridge
Tantalus
Tantalus is an experimental object-relational mapping framework. Its purpose is to provide a mechanism for storing Smalltalk objects in a relational database as painlessly as possible, without having to embed SQL in the application logic. It allows an application to define how various classes are to be stored, and then provides methods for storing and retrieving objects.
more
Additional notes from the author: The difference between Tantalus and GLORP is the strategy used to ensure the integrity of the database. GLORP uses database transactions; Tantalus uses atomic updates.
SPrevayler
This is Prevayler (http://www.prevayler.com) for Squeak.
Follow: SPrevayler for more information.
See also ReferenceStream
Answers from the list (unedited) hjh – 17-Feb-2003
Date: Sun, 16 Feb 2003
From: Hannes Hirzel
Subject: Implemeting a memory OO database in Squeak (was Re: port of prevayler)
To: The general-purpose Squeak developers list
Hi Frank
Frank Sergeant wrote:
> Marco Paga wrote:
> > Is anybody out there who wants to port prevyler to squeak? I have > > a version aviable, but that one is badly designed. You can't
> >it in any way.
> > So the next version NEEDS to be better.
> > If you are interested please let me know.
>
> How is your implementation licensed? From the heading of your 13 Oct
> 2002 changeset, it looks like you are using GPL. If so, I think this is
> a terrible idea. If it is something for Squeak, why not use Squeak-L or
> something even more permissive? Is your implementation derived from the
> actual Prevayler code (which is LGPL)? I do not see how I can use GPL
> or LPGL.
Marco is planning to rewrite the whole thing. He likes to do it
test-driven and is looking for somebody to work with him. (SUnit !)
As prevalyer is written in Java and Marco is writing it in Squeak there is no need to choose GPL and LPGL for the Squeak version.
It is the code of Prevalyer in Java which is GPL licensed, not the
ideas.
The ideas are straightforward and generally known. You may use them in
any context and write software and put the software under any license you like.
Regards
Hannes Hirzel
P.S. There are already some OO and non OO database efforts for Squeak
see Databases and Persistence
- File Dictionary
- MinneStore
- GemSqueak
- Kats
- MySQL
- PostgreSQL
- GOODS
- ODBC
- SQLite
- SIXX - Smalltalk object XML serializer/deserializer
- BerkeleyDB Plugin
- Magma
- Glorp
- Tantalus
- SPrevayler
Did somebody do an evaluation of the concepts / quality of these
implementations? If yes, please post remarks there.
Date: Sun, 16 Feb 2003
From: Avi Bryant
Subject: Re: Implemeting a memory OO database in Squeak (was Re: port of prevayler)
To: The general-purpose Squeak developers list
On Sun, 16 Feb 2003, Hannes Hirzel wrote:
> P.S. There are already some OO and non OO database efforts for Squeak
> see Databases and Persistence
> Did somebody do an evaluation of the concepts / quality of these
> implementations?
> If yes, please post remarks there.
Object Databases:
> - File Dictionary
[snip] (comments of Avi Bryant moved up)
> - MinneStore
[snip] (comments of Avi Bryant moved up)
> - Kats
Provides transactional semantics for in memory smalltalk objects. Uses
a modified compiler to trap inst var reads/writes so that the same
instance can look different to different processes. No persistence (but
watch out for Stephen's Chango, which should integrate Kats with a
transparent persistence layer using a modified VM).
> - GOODS
GOODS is a public domain distributed OODB written in C++. This is a not
quite finished client for it - the protocol and serialization code is
stable, but the transactional level stuff needs a fair bit of work. All
GC, crash recovery, concurrency, locking, schema evolution, etc is handled
nicely by the C++ server. Transparent storage of any object (but can
provide more efficient storage with optional type info).
(If anyone has a strong interest in using this, I'd be happy to finish the
client - particularly if there's money involved ;).
> - Magma
Pure Squeak client/server OODB. Somewhat immature (no class schema
evolution, no transaction logging) and not yet tested in the Real World,
but very promising. Allows transparent, transactional storage of any
Squeak objects.
Relational Databases:
> - MySQL
Port of Josh Miller's socket-level VW MySQL driver. Stable enough for
production work (at least with Colin Putney's patches, not sure if these
have been merged yet). Maintains type information.
> - PostgreSQL
Socket-level Postgres driver. Seems stable in recent releases. Doesn't
yet maintain type info (all columns are retured as strings).
> - ODBC
Solid FFI interface to ODBC. Works on Linux (with unixodbc), Windows, and
Mac OS X (with some effort - bug Colin for details). Maintains type info,
and I've tried it successfully with drivers for MySQL, Postgresql, and
SQLite.
> - SQLite
An FFI interface to the SQLite library (small, in-process, fast relational
DB). Less overhead (particularly configuration overhead) than going
through ODBC. Simple and stable.
Object Relational Mapping:
> - Glorp
Object/Relational mapping framework, based on TOPLink. Maps rows into
objects, to-one relationships into references, to-many relationships into
collections. Currently only works with PostgreSQL. Not in production use
AFAIK, but has extensive tests and is probably stable.
> - Tantalus
Another O/R mapping layer. Designed specifically to be used with MySQL,
so implements transactions on the client side instead of using DB
transactions, but has also been used with Postgres and SQLite. Otherwise
fairly similar to GLORP in philosophy. In production use, but still
immature.
Both GLORP and Tantalus can be adapted to use Apple's EOModel files for
mapping metadata.
Not Evaluated:
> - GemSqueak
> - SIXX - Smalltalk object XML serializer/deserializer
> - BerkeleyDB Plugin
> - SPrevayler
Date: Mon, 17 Feb 2003 01:36:44 +0100
From: tblanchard@mac.com
Subject: Re: Implemeting a memory OO database in Squeak (was Re: port of prevayler)
To: The general-purpose Squeak developers list
reply-to: The general-purpose Squeak developers list
On Monday, February 17, 2003, at 12:44 AM, Avi Bryant wrote:
>> - PostgreSQL
>
> Socket-level Postgres driver. Seems stable in recent releases.
> Doesn't
> yet maintain type info (all columns are retured as strings).
This is no longer true - the latest pgsql client provided with glorp
does return typed data (if you ask the row for typedData).
You can install new type conversion blocks into the connection object
if you like. They're just single argument blocks keyed by type oid.
Core types not supported by default are TimeStamp and Time with Time
Zone (for the simple reason that squeak has no concept of time zones
out of the box). If you use the TimeStampWithTimeZoneDatabase, on
SqueakMap then its easy to add type converters for these.
> Object Relational Mapping:
>
>> - Glorp
>
> Object/Relational mapping framework, based on TOPLink. Maps rows into
> objects, to-one relationships into references, to-many relationships
> into
> collections. Currently only works with PostgreSQL. Not in production
> use
> AFAIK, but has extensive tests and is probably stable.
The version on squeakmap is still pretty immature. Its missing code to
generate primary keys - works fine for reads but not writes. I have a
changeset that fixes this. I'm waiting for Alan to finish his current
release cycle to work with him to integrate the changes. The next
version released should be pretty decent.
>
> Both GLORP and Tantalus can be adapted to use Apple's EOModel files for
> mapping metadata.
I have a set of extensions to glorp to support eomodel files. A few
people have seen them but I would like more testing. email me if you
want em. I'm giving them back to glorp as well.
Date: Mon, 17 Feb 2003 09:56:55 -0500
From: "Stephen Pair"
Subject: RE: Implemeting a memory OO database in Squeak (was Re: port of prevayler)
To: "'The general-purpose Squeak developers list'"
reply-to: The general-purpose Squeak developers list
> > Did somebody do an evaluation of the concepts / quality of these
> > implementations? If yes, please post remarks there.
>
> —
> Object Databases:
...snip...
> > - BerkeleyDB Plugin
The BDB plugin simply provides an interface to BerkeleyDB. It currently
does not implement the entire BerkeleyDB API, but it gives you enough to
be useful. Most of what's missing are APIs that are concerned with
scalability and fault tolerance. What is there is stable and clean.
[Q]Is the code doing something just the same as "save image" does?
Smalltalk snapshot: true andQuit: false.
Well, how if I'm dealing with a very huge Object Database? Is it still ok using image as database?
rememberMeForDocumentation
