Databases and Persistence
Last updated at 1:35 pm UTC on 9 May 2018
is the master page which summarizes the various databases and other forms of persistense supported by Squeak. See also Persistence.
Summary
This began as a rewrite of a Seaside mailing list post. Though its been edited since.
Just save the image
See below. This is sometimes practical, but you end up saving a lot that isn't your apps data.
SMFileDatabase
Minimal amount of code, not ACID compliant, but gives you about 1/2 of Prevalyer transparently (no need to use Command pattern). One step up from the image, from Ramon Leon: SMFileDatabase
OmniBase
Discard if you intend to use the free version with Squeak.
Magma
Excellent code integration, probably the most elegant of all.
Can run as server and client images separately.
Has a great query language that no one else has
Has special classes for large collections that you can create indexes to specific attributes
Has programmable ReadStrategies
Has excellent support from its creator Chris Muller, and the list.
The bad thing: It is a bit too slow. Both for inserting and for querying, even on mid sized collections (a few thousand elements) with indexes and optimal read strategies can still be pretty slow. (to check if this is still the case)
More see Magma
Glorp / Postgres
see GLORP: Generic Lightweight Object-Relational Persistence
Background
SqueakImage-based persistence
FileDictionary
ImageSegments
OmniBase
MinneStore
GemSqueak
Kats
MySQL
PostgreSQL
SqueakDBX
GOODS
ODBC
SQLite
SIXX
BerkeleyDBPlugin
Magma
Glorp
Tantalus
SPrevayler
J2SBridge
CouchDB
Squeak Image
Squeak (as all Smalltalk implementations worthy of that name) implements persistency in a way which is (almost) unseen in any other computational environment. It does not need databases, XML or any similar superfluous stuff to be persistent. Just try:
Smalltalk snapshot: true andQuit: false.
You can also create a process which saves the image periodically instead of you.
[[true] whileTrue:
[(Delay forSeconds: 1000) wait.
Smalltalk snapshot: true andQuit: false
]
] forkAt: 49.
But keep in mind that this type of persistence is not ACID. You may lose data. In cases where it is not acceptable to lose data, some of us think databases are not superfluous at all (Gerald Zincke).
File Dictionary
The File Dictionary is an extremely simple database structure. It allows a large (larger than memory) file to be treated as a Dictionary by keeping an in-memory index. It uses native Squeak serialization (DataStream and ReferenceStream version available). It does not support transactions/concurrency/etc. For details see FileDictionary: Tiny Squeak Database
ImageSegments
ImageSegments may also be used as a simple kind to store objects while at the same time keeping the footprint of your main image low.
Details see : Introduction to ImageSegments.
OmniBase
OmniBase is a production-strength object database with gargabe collection and on-line backup support. OmniBase is written 100% in Smalltalk and runs on Squeak, Dolphin, VW, VAST and ST/X. Database files can be written and read from any Smalltalk dialect.
A fully functional version of OmniBase is available for Squeak running on Windows, for Linux there is currently only a single-user (db connection) version available (which is still suitable for e.g. a web server). Technical details can be found on the OmniBase wiki support site.
Cees de Groot is porting it to Squeak for Linux, details of his port can be found on the OmniBase Swiki.
CouchDB
For more details, see CouchDB
MinneStore
MinneStore is a free object database that has been ported to Squeak. Information about MinneStore can be found at: http://minnestore.sourceforge.net/ The latest files are in http://sourceforge.net/projects/minnestore/
Now it works in 3.2.
Minnestore has been around for along time and seen a fair bit of use. It feels somewhat like using a relational DB - you have to declare type and relationship info for your model. In return, you get automatic indexing and query facilities. It supports transactions.
Questions:
- What is the state of concurrent access?. Not finished, still can be two users locked tryng to change the same object.
- Does it has GC? Not by now (02/25/2003)
GemSqueak
Squeak Client for GemStone/S. GemStone is an industrial-strength (and not inexpensive) object-oriented database based on Smalltalk. Please visit the GemSqueak page to learn more about it.
Kats
Kats is a transaction service for smalltalk that is used in conjunction with persistency and ORB services. It is used to help manage the synchronization of object state with external systems. See Kats for more details.
MySQL
Bolot Kerimbaev is maintaining the Squeak port of Josh Miller's VisualWorks MySQL driver. See the MySQL page for more details.
SqueakDBX
The aim of this project is to build an OpenDBX (http://www.linuxnetworks.de/doc/index.php/OpenDBX) plugin which will allow users to perform relational database operations (DDL, DML and SQL) through an open source library.
Through this feature, the squeak community will hopefully be able to interact with major database engines, such as oracle and mssql, besides those which are open source, like postgresql and mysql. Moreover, by integrating this plugin with GLORP (http://www.glorp.org/), will allow us to generate a complete and open source solution to relational data base access.
To make this possible, the Squeak Foreign Function Interface (http://wiki.squeak.org/squeak/1414/) will be used. FFI is used to call functions located in shared libraries (OpenDBX in this case) which are not part of the Squeak VM nor its plugins. Don't worry, queries are asynchronous, so the time of the FFI blocks is very low and unimportant.
For more details, see SqueakDBX
PostgreSQL
A "PostgreSQL Client for Squeak" can be found at http://members.rogers.com/yanni/pgsqueak
I'm finding the name unwieldy, so future versions will call it "pgsqueak".
The page has moved to http://ca.geocities.com/zazu@rogers.com/pgsqueak, where version 0.9.2 is available. Also, pgsqueak appears to suppport only the "password" method of authentication, not the "crypt" or "md5" methods, which I didn't see documented anywhere. – Eric Entzel (Jan 13, 2006)
Latest release 0.9 is available. The same code runs on VAST. – Yanni Jew (Apr 26, 2003).
I updated the links above (a newer 0.8.0 release is available). – Yanni Jew (Dec 14, 2002).
How about "pgsqueakl"? –Pens v. Mice
It took me a minute to get it. Maybe "pgSQueakL" would be more obvious. –Yanni Jew (Apr 14, 2002)
I've just released a Squeak PostgreSQL interface. See http://www.io.com/~jimm/projects/squeak_postgresql/. Please let me know about any problems (jimm@io.com). –Jim Menard (Sep 13, 2002)
GOODS
Avi Bryant wrote the beginnings of a Squeak interface to the GOODS object database.
ODBC
ODBC for Squeak: Very simple framework for ODBC support in Squeak
SQLite
SIXX - Smalltalk Instance eXchange in XML
BerkeleyDB Plugin
Actually, this is more than just a plugin...it's an object database implementation using BerkeleyDB. See: http://spair.swiki.net/27
Magma
Magma is a multi-user object database designed for Squeak systems from the ground up. It provides transaction support and transparent access.
See Magma. Magma FAQs
Glorp
Nevin Pratt ported Glorp to Squeak. His porting "diary" can be found at:
http://home.earthlink.net/~nevinpratt/GlorpPort.doc
The Squeak changeset can be found at:
http://home.earthlink.net/~nevinpratt/glorp.cs
Don't use the above. Instead use my newer port at:
http://www.smalltalkpro.com/squeakextras.html
Or, you can get it from SqueakMap.
J2Sbridge
A JDBC bridge for Squeak. Up to now very simple, supports only Strings, Integers and
Binary objects (blobs).
More here: J2SBridge
Tantalus
Tantalus is an experimental object-relational mapping framework. Its purpose is to provide a mechanism for storing Smalltalk objects in a relational database as painlessly as possible, without having to embed SQL in the application logic. It allows an application to define how various classes are to be stored, and then provides methods for storing and retrieving objects.
more
Additional notes from the author: The difference between Tantalus and GLORP is the strategy used to ensure the integrity of the database. GLORP uses database transactions; Tantalus uses atomic updates.
SPrevayler
This is Prevayler (http://www.prevayler.com) for Squeak.
Follow: SPrevayler for more information.
See also ReferenceStream
Answers from the list (unedited) hjh – 17-Feb-2003
Date: Sun, 16 Feb 2003
From: Hannes Hirzel
Subject: Implemeting a memory OO database in Squeak (was Re: port of prevayler)
To: The general-purpose Squeak developers list
Hi Frank
Frank Sergeant wrote:
> Marco Paga wrote:
> > Is anybody out there who wants to port prevyler to squeak? I have > > a version aviable, but that one is badly designed. You can't
> >it in any way.
> > So the next version NEEDS to be better.
> > If you are interested please let me know.
>
> How is your implementation licensed? From the heading of your 13 Oct
> 2002 changeset, it looks like you are using GPL. If so, I think this is
> a terrible idea. If it is something for Squeak, why not use Squeak-L or
> something even more permissive? Is your implementation derived from the
> actual Prevayler code (which is LGPL)? I do not see how I can use GPL
> or LPGL.
Marco is planning to rewrite the whole thing. He likes to do it
test-driven and is looking for somebody to work with him. (SUnit !)
As prevalyer is written in Java and Marco is writing it in Squeak there is no need to choose GPL and LPGL for the Squeak version.
It is the code of Prevalyer in Java which is GPL licensed, not the
ideas.
The ideas are straightforward and generally known. You may use them in
any context and write software and put the software under any license you like.
Regards
Hannes Hirzel
P.S. There are already some OO and non OO database efforts for Squeak
see Databases and Persistence
- File Dictionary
- MinneStore
- GemSqueak
- Kats
- MySQL
- PostgreSQL
- GOODS
- ODBC
- SQLite
- SIXX - Smalltalk object XML serializer/deserializer
- BerkeleyDB Plugin
- Magma
- Glorp
- Tantalus
- SPrevayler
Did somebody do an evaluation of the concepts / quality of these
implementations? If yes, please post remarks there.
Date: Sun, 16 Feb 2003
From: Avi Bryant
Subject: Re: Implemeting a memory OO database in Squeak (was Re: port of prevayler)
To: The general-purpose Squeak developers list
On Sun, 16 Feb 2003, Hannes Hirzel wrote:
> P.S. There are already some OO and non OO database efforts for Squeak
> see Databases and Persistence
> Did somebody do an evaluation of the concepts / quality of these
> implementations?
> If yes, please post remarks there.
Object Databases:
> - File Dictionary
[snip] (comments of Avi Bryant moved up)
> - MinneStore
[snip] (comments of Avi Bryant moved up)
> - Kats
Provides transactional semantics for in memory smalltalk objects. Uses
a modified compiler to trap inst var reads/writes so that the same
instance can look different to different processes. No persistence (but
watch out for Stephen's Chango, which should integrate Kats with a
transparent persistence layer using a modified VM).
> - GOODS
GOODS is a public domain distributed OODB written in C++. This is a not
quite finished client for it - the protocol and serialization code is
stable, but the transactional level stuff needs a fair bit of work. All
GC, crash recovery, concurrency, locking, schema evolution, etc is handled
nicely by the C++ server. Transparent storage of any object (but can
provide more efficient storage with optional type info).
(If anyone has a strong interest in using this, I'd be happy to finish the
client - particularly if there's money involved ;).
> - Magma
Pure Squeak client/server OODB. Somewhat immature (no class schema
evolution, no transaction logging) and not yet tested in the Real World,
but very promising. Allows transparent, transactional storage of any
Squeak objects.
Relational Databases:
> - MySQL
Port of Josh Miller's socket-level VW MySQL driver. Stable enough for
production work (at least with Colin Putney's patches, not sure if these
have been merged yet). Maintains type information.
> - PostgreSQL
Socket-level Postgres driver. Seems stable in recent releases. Doesn't
yet maintain type info (all columns are retured as strings).
> - ODBC
Solid FFI interface to ODBC. Works on Linux (with unixodbc), Windows, and
Mac OS X (with some effort - bug Colin for details). Maintains type info,
and I've tried it successfully with drivers for MySQL, Postgresql, and
SQLite.
> - SQLite
An FFI interface to the SQLite library (small, in-process, fast relational
DB). Less overhead (particularly configuration overhead) than going
through ODBC. Simple and stable.
Object Relational Mapping:
> - Glorp
Object/Relational mapping framework, based on TOPLink. Maps rows into
objects, to-one relationships into references, to-many relationships into
collections. Currently only works with PostgreSQL. Not in production use
AFAIK, but has extensive tests and is probably stable.
> - Tantalus
Another O/R mapping layer. Designed specifically to be used with MySQL,
so implements transactions on the client side instead of using DB
transactions, but has also been used with Postgres and SQLite. Otherwise
fairly similar to GLORP in philosophy. In production use, but still
immature.
Both GLORP and Tantalus can be adapted to use Apple's EOModel files for
mapping metadata.
Not Evaluated:
> - GemSqueak
> - SIXX - Smalltalk object XML serializer/deserializer
> - BerkeleyDB Plugin
> - SPrevayler
Date: Mon, 17 Feb 2003 01:36:44 +0100
From: tblanchard@mac.com
Subject: Re: Implemeting a memory OO database in Squeak (was Re: port of prevayler)
To: The general-purpose Squeak developers list
reply-to: The general-purpose Squeak developers list
On Monday, February 17, 2003, at 12:44 AM, Avi Bryant wrote:
>> - PostgreSQL
>
> Socket-level Postgres driver. Seems stable in recent releases.
> Doesn't
> yet maintain type info (all columns are retured as strings).
This is no longer true - the latest pgsql client provided with glorp
does return typed data (if you ask the row for typedData).
You can install new type conversion blocks into the connection object
if you like. They're just single argument blocks keyed by type oid.
Core types not supported by default are TimeStamp and Time with Time
Zone (for the simple reason that squeak has no concept of time zones
out of the box). If you use the TimeStampWithTimeZoneDatabase, on
SqueakMap then its easy to add type converters for these.
> Object Relational Mapping:
>
>> - Glorp
>
> Object/Relational mapping framework, based on TOPLink. Maps rows into
> objects, to-one relationships into references, to-many relationships
> into
> collections. Currently only works with PostgreSQL. Not in production
> use
> AFAIK, but has extensive tests and is probably stable.
The version on squeakmap is still pretty immature. Its missing code to
generate primary keys - works fine for reads but not writes. I have a
changeset that fixes this. I'm waiting for Alan to finish his current
release cycle to work with him to integrate the changes. The next
version released should be pretty decent.
>
> Both GLORP and Tantalus can be adapted to use Apple's EOModel files for
> mapping metadata.
I have a set of extensions to glorp to support eomodel files. A few
people have seen them but I would like more testing. email me if you
want em. I'm giving them back to glorp as well.
Date: Mon, 17 Feb 2003 09:56:55 -0500
From: "Stephen Pair"
Subject: RE: Implemeting a memory OO database in Squeak (was Re: port of prevayler)
To: "'The general-purpose Squeak developers list'"
reply-to: The general-purpose Squeak developers list
> > Did somebody do an evaluation of the concepts / quality of these
> > implementations? If yes, please post remarks there.
>
> —
> Object Databases:
...snip...
> > - BerkeleyDB Plugin
The BDB plugin simply provides an interface to BerkeleyDB. It currently
does not implement the entire BerkeleyDB API, but it gives you enough to
be useful. Most of what's missing are APIs that are concerned with
scalability and fault tolerance. What is there is stable and clean.
[Q]Is the code doing something just the same as "save image" does?
Smalltalk snapshot: true andQuit: false.
Well, how if I'm dealing with a very huge Object Database? Is it still ok using image as database?
rememberMeForDocumentation
