SystemTracer
Last updated at 3:22 pm UTC on 19 June 2020
System Tracer is a tool for creating a new, clean copy of the objects in a Squeak image file. It evolved from a process invented by Ted Kaehler to clone early copies of Smalltalk.
Squeaks saves its objects in an image file and the source code in a separate sources file. While modifying objects in a live session, any changes made to global variables or method sources are saved in a separate file called the change log. After a while, this log file can become very large, so it is necessary to condense changes into a new image. This is accomplished by using the System Tracer tool.
SystemTracer was part of Squeak until the 4.x releases. From Squeak 4.x series onwards, it is available as a separate package at http://www.squeaksource.com/SystemTracing.html.
In this package, you will find three variants of SystemTracer.
- SystemTracer - the original version suited for memory constrained environments. See class comments for details
- SystemTracer2 - redesigned to convert V3 images into block closure images. Easier to use but memory intensive.
- SystemTracer64 - designed to write images for 64-bit machines. See class comments for details.
Operation
The SystemTracer runs through the system tracing every accessible object, and writing a copy of each object onto a disk image. It does this carefully enough that the resulting clone can be started up and run, just like a snapshot. Great care must also be taken that the tracer never sees (due to clamping) any of the structures created dynamically while writing the clone. If this happens an infinite recursion can result.
Tracing is a slow process and will take hours. One has to learn to be patient.
The tracing process is started (depending on the variant) with:
SystemTracer writeClone
SystemTracer2 writeImage: 'clone-32bit.image'
SystemTracer64 writeImage: 'clone-64bit.image'
This method creates a new tracer instance and then initiates tracing (see doIt) objects and saving them into an image file named "clone.image". The new image will resume in the above method and then trigger a startup chain in the new image. Doing a "save and quit" will then save an image which resumes in the normal resume path.
The tracer has built-in support for clamping (i.e. skipping unwanted or unneeded) objects out of the system. It also allows fundamental changes to, eg, object header format, compiledMetho4 format, number formats, etc. The current version is fairly simple, as it just copies what is already in the system. This version does, however, reassign object hashes in order to illustrate how one must remap all the hashed structures accordingly.
Squeak currently only has 12 bits of object hash. A normaldictionary, used to map from old oop to new oop, would bog down in a system of, say, 50k objects. This tracer has a special structure that is optimized for this purpose. See the comment in initOopMap. If we ever extend the object hash, this structure could be replaced by a simpleIdentityDictionary.
When you first start up the cloned.changes file, you will probably get a "cannotReturn" error, a vestige of the old tracing process. This can simply be closed (or maybe you can figure out how to fix the tracers so it doens't happen ;-).
Further comment fromTim Rowledge's message about replacing classes...
The cloner already does a great job of allowing one to leave out classes, individual instances, change representation of particular objects etc.
However, when doing the ncm stuff I needed a way to build the new class in the image and test it before cloning. Thus at clone-time I needed to:
a) replace each compiled method with a new compiled method
b) replace the classCompiledMethod with NewCompiledMethod
c) get rid of the now-redundant CompiledMethod class
Doing a) is easy, that is the stock in trade of the cloner.
Doing b) & c) is a little more complicated since I wanted to catch every
reference to CM and write it out as a reference to NCM and also needed to
'clamp' CM.
To do this we need to make sure that when a ref to CompiledMethod is found we not only write out the replacement reference to NewCompiledMethod (NCM) instead, but that we log both CM and NCM as having been dealt with by putting _both_ into the oopmap with the same new-oop. Thus when we find a reference to NCM (for example when the subclass list of its superclass is scanned) it will already have been 'dealt with'. The system also has to cope with arbitrary order of finding either of the involved objects; you cannot guarantee that one will be spotted before the other and rely on that.
Take a look at SystemTracer>replace:withNew:class:length:trace:write: to see the diferences from the normal #new:class:length:trace:write:. One also needs to check Associations in order to 'correct' the value instvar if it refers to a replaced class.
History
iFrom the class comment (version 3.3a-4769)/i
NOTE: The SystemTracer was invented byTed Kaehler. It has gone through many variations. I have simplified it and adapted it to work with Squeak's limited Object hashing. It has written runnable Mac images on a Mac. Some changes may be needed for it to produce runnable images on a bigEndian machine. – Dan I. 9/29/97
It evolved from a process called VMem-write that was used to clone early Smalltalk images. See section 23.4 in theRed Book for historical details.
