Magma 1.5 Release Notes
Last updated at 3:46 am UTC on 18 August 2015
Version 1.5 of Magma coincides with the release of Squeak 5, as well as many other improvements and fixes over Magma 1.4.
Transparent Cog and Spur Compatibility
Magma 1.5 is transparently compatible with Spur's new object memory format and VM. Persistent CompiledMethod headers are transparently converted back-and-forth as necessary depending on what kind of client (a V4 Cog or V5 Spur) is accessing.
Under Cog, the server can only grow to about 1.8GB before running out of memory and once it grows that large, it never gives that memory back to the OS. Using the aforementioned multicore batch processing capability, I was able to increase the number of simultaneous forked clients so that the server could grow to nearly to 4GB! Once the large task was completed and clients signed out, the server actually recovered the memory and went back down to consuming only about 60MB from the OS. Magma thrives on the extra speed and memory management and breathing-room provided by Spur.
A New Concurrent Statistical Accumulator Type
A MagmaCounter is a legacy Magma type which allows multiple clients to commit deltas to a synchronized accumulator. 1.5 introduces MagmaStat, an extension to MagmaCounter adds a synchronized #sum, #min, #max and #last. This was created to support a multidimensional data-aggregator. It allows multiple clients to simultaneously accumulate into a single accumulator, without commit-conflict, and the accumulator will always report the latest known values.
New Object-Level Meta Information
Meta-information can now be retrieved for any object as long as the commit-log file in which it was committed is still available to the database. Since these files are not explicitly required to be kept (it depends on ones backup and roll-forward needs), so it depends on your server administration.
- magmaOid – The unique integer oid of this object.
- magmaCommitNumber – The commit number which this object was last committed. The first commit to a new database is commit number 1, every commit increments it from there.
- #magmaLastUpdated – The DateAndTime this object was last committed.
- #magmaLastUpdaterSessionId – The sessionId (UUID) of the last session that last committed this object.
- #magmaLastUpdaterUserId – The userId of the last session that committed this object.
- #magmaCommitPackage – The package of objects which were part of the same commit as this objects last commit.
- #isDirtyInMagma – Answer whether Magma currently considers this object as changed and intending to update it in the DB upon next commit.
If the commit-log record is no longer available, nil will be answered for all of the above (except magmaOid and isDirtyInMagma, of course). Each Sessions #definition can now answer the collection of persistent #classes in the DB. #dirtyGraphDo: allows all currently dirty objects to be enumerated.
Experimental Multicore Batch Processing Tool
A new tool employs the client-server framework in a new way – to manage batch processing tasks using multiple CPU cores. The user creates a collection of Blocks which are then forked and run concurrently in their own separate VM process. The user specifies how many CPU cores to allocate to the task so that the framework will maintain exactly that many processes running concurrently. If 100 blocks of work are created but only a 4-core processor, then only 4 at a time will run, but as any block finishes and exits its image, the framework automatically detects this and replace that slot by the launch of the next block, in sequence, until all the blocks are done. The number of cores can adjusted dynamically after the process has started.
Although Ma Client Server framework is a standalone framework (completely independent of Magma), typical usage of this multi-core feature is for each forked block to collaborate on the same Magma database(s). A variety of configurations are possible supporting several use-cases. The client-sever framework provides the inter-proces communication between the calling process and the forked block processes to support supervisory use-cases (remote progress bars) over all of the forked processes from the single launching image, with pause, resume, etc.).
A New Data-Repair Tool
MagmaDataRepair is used to perform schema upgrades or other "repairs" on Magma models in a controlled and reliable fashion. It can take advantage of the multi-core processing, if necessary.
This is a great tool, the user simply creates the first-class MagmaDataRepair object using the #at:start:enumerate:check:repair: constructor message. A MagmaDataRepair instance is returned which supports the API of the repair:
- #check – Answer a boolean whether there is any data needing repair.
- #count – Answer the number of objects needing repair.
- #identify – Answer a collection of the objects needing repair.
- #improve – commit repairs as they are made, allowing some but possibly not all objects to be repaired, an "improvement".
- #repair – Repair all objects but only commit if all repairs are verified successful.
Graphical Status Monitor
Magma has kept track of various timings and counts of its operations for years. The objects are in there, but not easily viewed until now. Some rudimentary charts show various stats by 5-minute periods over the last 24 hours (configurable). This can provide insight into, for example, the number and size of reads vs. commits by an application, which might be interesting for, i.e., application performance tuning.
Better Linux Server Integration
- Magma servers now respond to SIGTERM signals to shutdown the server, close the DB and exit the image gracefully.
- A new tool lets the user select an existing Magma DB and, with one-click, create a deployment zip file with not only those DB files, but every other piece necessary to deploy it on a fresh Linux server and run it as secure service under daemontools. Every script needed is included, including Linux system configuration scripts, vm installation, and daemontools configuration. It even produces a README describing the exact steps for doing it. It's been tested on Ubuntu 14.04 Server as small as an AWS t2.micro.
- Script 'configsys' configures a new system for the first time for running a Magma server (installs 32-bit libs and Squeak VM).
- Script 'configdbservice' is a one-time setup for a specific Magma repository, separated in case to run multiple servers on one instance.
- Script 'run' is the actual script invoked by daemontools.
- Magma servers deployed this way include Squeak's RFB. This affords console interaction afforded by headless mode, without losing the ability to look and interact with the running image if necessary.
Performance and Robustness
- The independent Ma Client Server networking framework utilized by Magma has had several improvements in speed and robustness.
- Every request/response interaction is now counted via a sequenceNumber which is matched by the server against its expectation for each client, ensuring missed or duplicate messages are handled appropriately.
- The client-server framework now employs SocketStream instead of its own separate internal ByteArray's and Sockets, allowing slightly less garbage to be created I think.
- Although sessions will timeout after 30 minutes of inactivity, they will reconnect automatically upon the next user activity, updating their image with changes by other sessions since. During that update, it is now possible to receive a MagmaTooFarBehindConflict, the same as a MagmaCommitConflictError, which applications already must handle.
- Improvement to MaDateAndTimeIndex; by changing the default precision from 1 nanoSeconds to 1 minute, the number of required bits was reduced from 72 to 32! Please rebuild your DateAndTime indexes!
- The server will allow up to 180 simulataneously-open files by default instead of only 60. This is just a resource balance / performance tune for managing strict VPS hosts. Magma manages all opening and closing files of individual files automatically, the user only is concerned with opening/closing the db as a whole.
- A real-world application performance demand has exposed several opportunities for optimization in the client and server, along with a long release time between 1.4 and 1.5, resulting in the most-heavily tested version of Magma ever released. During this time, the server was pounded with as many as 30 simultaneous sessions each with their own CPU core (7 physical computers) doing rapid-fire reads and commits of all sizes, with commit-conflicts handled correctly by the application. Admin initiated database backups run well even under load, and restore function fully tested.
Local-Conflict Detection
At last, Magma will now detect a local-conflict when a session must reconnect and catch-up to the repository. During this process, if any objects are refreshed which were already locally changed by that session, a MagmaTooFarBehindConflict is signaled.
The idea is it can be handled the same as a commit-conflict, because the new MagmaTooFarBehindConflict inherits the 'result', a FailedCommitResult.
Compatibility Notes
FloatIndex was revamped so that negatives are treated properly as less than positives in their indexing sequence. Any applications using FloatIndex should rebuild them.