Squeak
  links to this page:    
View this PageEdit this PageUploads to this PageHistory of this PageTop of the SwikiRecent ChangesSearch the SwikiHelp Guide
ImageSegment
Last updated at 4:03 pm UTC on 1 April 2017

Definition

http://forum.world.st/4-6-5-0-how-to-help-users-manage-different-formats-td4811303.html#a4811856

An ImageSegment is a WordArray whose contents are a sequence of heap objects in the VM's native object format. A segment is loaded by invoking a primitive that swizzles the pointers in the word array so that they become valid objects, and then shortening the word array, leaving the objects behind in memory. Since the contents of the WordArray are specific to a given VM architecture one cannot use the primitive loading mechanism to load segments from VMs with different object representations. Here different means

But there is no reason why Smalltalk code could not parse the contents of the WordArray and hence load objects by creating them based on the data in the WordArray. This would allow image segments to be loaded in any format the Smalltalk code supported (and it would be interesting to compare performance and see how much slower "synthesized" loading would be). This would provide legacy support for segments from "old" versions, and maybe even provide a oath to adopting a more neutral format such as Fuel.

To implement such a system one would have to understand the object formats of each system, using the VM source code as a guide, both the object representation and the load and save primitives themselves.

More: Object Swapping Challenges - an Evaluation of ImageSegment

2006

ImageSegments are a very fast binary object storage mechanism.
From the class comment - 3.3a-4769)
I represent a segment of Squeak address space. I am created from an array of root objects. After storing, my segment contains a binary encoding of every object accessible from my roots but not otherwise accessible from anywhere else in the system. My segment contains outward pointers that are indices into my table of outPointers.
When a segment is written out onto a file, it goes in a folder called image name_segs.
If your image is called "Squeak2.6.image", the folder "Squeak2.6_segs" must accompany the image whenever your move, copy, or rename it.
Whenever a Class is in arrayOfRoots, its class (aClass class) must also be in the arrayOfRoots.

See also Introduction to ImageSegments

There are two kinds of image segments.

Normal image segments are a piece of a specific Squeak image, and can only be read back into that image. The image holds the array of outPointers that are necessary to turn the bits in the file into objects.
To put out a normal segment that holds a Project (not the current project), execute (Project named: 'xxx') storeSegment.

The second kind of image segment is an Export Segment. It can be read into a different Squeak image. It does not have stay with your Squeak image. To create one:

 (ImageSegment new copyFromRootsForExport: (Array with: Baz with: Baz class))
		writeForExport: 'myFile.extSeg'.

To create one for a project:
	(Project named: 'Play With Me - 3') exportSegment.

To read it into another image:

It will install its classes automatically. If you need to see the roots array, it is temporarily stored in (SmartRefStream scannedObject).


state that an ImageSegment may exist in...

  1. activeCopy (has been copied, with the intent to become active)
arrayOfRoots, segment, and outPointers have been created by copyFromRoots:. The tree of objects has been encoded in the segment, but those objects are still present in the Squeak system.

  1. active (segment is actively holding objects)
The segment is now the only holder of tree of objects. Each of the original roots has been transmuted into an ImageSegmentRootStub that refers back to this image segment. The original objects in the segment will all be garbageCollected.

  1. onFile
The segment has been written out to a file and replaced by a file pointer. Only ImageSegmentRootStubs and the array of outPointers remains in the image. To get this far:
(ImageSegment new copyFromRoots: (Array with: Baz with: Baz class))
writeToFile: 'myFile.seg'.

  1. inactive
The segment has been brought back into memory and turned back into objects. rootsArray is set, but the segment is invalid.

  1. onFileWithSymbols
The segment has been written out to a file, along with the text of all the symbols in the outPointers array, and replaced by a file pointer. This reduces the size of the outPointers array, and also allows the system to reclaim any symbols that are not referred to from elsewhere in the image. The specific format used is that of a literal array as follows:
#(symbol1 symbol2 # symbol3 symbol4 'symbolWithSpaces' # symbol5).
In this case, the original outPointers array was 8 long, but the compacted table of outPointers retains only two entries. These get inserted in place of the #'s in the array of symbols after it is read back in. Symbols with embedded spaces or other strange characters are written as strings, and converted back to symbols when read back in. The symbol # is never written out.
NOTE: All IdentitySets or dictionaries must be rehashed when being read back from this format. The symbols are effectively internal. (No, not if read back into same image. If a different image, then use #imported. -tk)

  1. imported
The segment is on an external file or just read in from one. The segment and outPointers are meant to be read into a foreign image. In this form, the image segment can be read from a URL, and installed. A copy of the original array of root objects is constructed, with former outPointers bound to existing objects in the host system.
(Any Class inside the segment MUST be in the arrayOfRoots. This is so its association can be inserted into Smalltalk. The class's metaclass must be in roots also. Methods that are in outPointers because blocks point at them, were found and added to the roots.
All IdentitySets and dictionaries are rehashed when being read back from exported segments.)


To discover why only some of the objects in a project are being written out, try this (Destructive Test). This breaks lots of backpointers in the target project, and puts up an array of suspicious objects, a list of the classes of the outPointers, and a debugger.
"Close any transcripts in the target project"
 World currentHand objectToPaste ifNotNil: [
	self inform: 'Hand is holding a Morph in its paste buffer:\' withCRs,
		World currentHand objectToPaste printString].
 PV _ Project named: 'xxxx'.
 (IS _ ImageSegment new) findRogueRootsImSeg: 
	(Array with: PV world presenter with: PV world).
 IS findOwnersOutPtrs.	"Optionally: write a file with owner chains"
 "Quit and DO NOT save"

When an export image segment is brought into an image, it is like an image starting up. Certain startUp messages need to be run. These are byte and word reversals for nonPointer data that comes from a machine of the opposite endianness. #startUpProc passes over all objects in the segment, and:
The first time an instance of class X is encountered, (msg _ X startUpFrom: anImageSegment) is sent. If msg is nil, the usual case, it means that instances of X do not need special work. X is included in the IdentitySet, noStartUpNeeded. If msg is not nil, store it in the dictionary, startUps (aClass -> aMessage).
When a later instance of X is encountered, if X is in noStartUpNeeded, do nothing. If X is in startUps, send the message to the instance. Typically this is a message like #swapShortObjects.
Every class that implements #startUp, should see if it needs a parallel implementation of #startUpFrom:.


Reference:
Mariano Martinez Peck 1, Noury Bouraqadi, Stéphane Ducasse, Luc Fabresse

Inria open archive
Object Swapping Challenges - an Evaluation of ImageSegment

ACM digital library
Object swapping challenges: An evaluation of imageSegment

Eliot Miranda Thu, Sep 22, 2016 at 9:07 PM
Reply-To: Squeak Virtual Machine Development Discussion
To: The general-purpose Squeak developers list
Cc: Squeak Virtual Machine Development Discussion , Pharo Development List

Hi Ron, Hi All,

On Thu, Sep 22, 2016 at 11:58 AM, Ron Teitelbaum wrote:

Hi All,



Sorry I’m joining this conversation late.



We are using ImageSegment for transferring croquet island state. Is this something covered in the changes from Bert or are we talking about dropping ImageSegment altogether?


I suppose I should explain. At ESUG Bert and others announced support for Toys in Squeak 5.1. As part of this work, ImageSegment loading and saving was moved from the VM up into pure Smalltalk code. This allows Squeak 5.1 (Spur) to load image segments saved on pre-Spur VMs. Consequent;ly we no longer need VM support for ImageSegment. Instead we can use the pure image-level code Bert has kindly written. The question is then at what point can we nuke the Spur VM support, since it is now superfluous.

Bert, am I jumping the gun? Do we still need to write image-level support for writing image segments? If this is the case, Igor Stasenko described a primitive for collecting the set of objects to be written out. It is the initial part of the segment-writing primitive that collects the transitive closure of objects reachable only from those root objects.

Max Leske Mon, Sep 26, 2016 at 9:43 PM
Reply-To: The general-purpose Squeak developers list
To: The general-purpose Squeak developers list
Reply | Reply to all | Forward | Print | Delete | Show original
Hi Bert,

No, we use ImageSegment directly (we’ve subclassed ImageSegment). Here’s the short version:

1. we copy the roots:

self
copyFromRoots: (Array with: anObject)
sizeHint: self fileSize // 2
areUnique: true.

2. we write the segment onto a MutliByteFileStream (which we’ve subclassed):

NSSegmentStream
forceNewFileNamed: fileName
do: [ :stream | stream writeObject: self ].

where #writeObject: is implemented as:

writeObject: anObject
(SmartRefStream on: self) nextPutObjOnly: anObject


We load the segment back by:

1. opening a MultiByteFileStream and sending the installation message:

NSSegmentStream
readOnlyFileNamed: (aDirectory fullNameFor: filename)
do: [ :stream |
self
installSegmentFrom: stream
andDo: aBlock ]

2. #installSegmentFrom:andDo: basically sends #readObject to the file stream:

readObject
"This is a hack to allow for the old segments (mixed code and objects)
to be loaded without the #inform: in #nextAndClose opening a UI element."
^ self binary peek = ReferenceStream versionCode
ifTrue: [ (SmartRefStream on: self) nextAndClose ]
ifFalse: [
| object |
self
ascii;
fileIn.
object := SmartRefStream scannedObject.
SmartRefStream scannedObject: nil.
object ]

3. and once we have the segment we send #install to it


We’ve made some modifications here and there (e.g. to suppress UI elements) but the above is the gist of it. I can give you the full source if you need it.
It’s possible that this is all overkill or parallels functionality found in other places. I’ve inherited the code and haven’t spent the time exploring improvements in that area.

Cheers,
Max


Eliot Miranda Mon, Oct 20, 2014 at 9:41 PM
Reply-To: Squeak Virtual Machine Development Discussion
To: Pharo Development List
Cc: Squeak Virtual Machine Development Discussion , The general-purpose Squeak developers list
Reply | Reply to all | Forward | Print | Delete | Show original



On Mon, Oct 20, 2014 at 8:26 AM, stepharo wrote:

While I as a big fan of imageSegment and proposed to Mariano to work on imageSegment2 (it was the original idea for his phd)
he convinced us that imagesegment were not worth their complexity.


I absolutely agree.


So why do you want to have imageSegment?


Because of backwards-compatibility. If Spur does not provide image segments then the barrier to entry for Terf, eToys and Squeak may be too high. Spur is supposed to be a plug-in replacement for Cog, not something that requires lots of effort to port to.


Stef


stepharo Mon, Oct 20, 2014 at 9:55 PM
Reply-To: Pharo Development List
To: Pharo Development List

Sorry for breaking the stream. I could not find the email with thunderbird :(

Just for the record, you see what netstyle people did: they use image segment and they had to kill everything (like processes and others) because of
possible pointers so after saving a segment the image was basically dead because some escaping pointers to make a kind of memory leak.
Now they migrated to Fuel.

So now I think that people should better use Fuel. IS were nice at first but not when you look carefully to them, especially when you have
an object not in the roots pointing to an object inside the graph. Then you have to do a GC…. Mariano spent a year fighting with that
and his PhD is really nice.

Less magic, more stability, simpler VMs.

Because all the energy that you will put in something from the past will not be put in things for the future(not counting the bug hunting).
Stef



On Mon, Oct 20, 2014 at 8:26 AM, stepharo wrote:
While I as a big fan of imageSegment and proposed to mariano to work on imageSegment2 (it was the original idea for his phd)
he convinced us that imagesegment were not worth their complexity.

I absolutely agree.

So why do you want to have imageSegment?

Because of backwards-compatibility. If Spur does not provide image segments then the barrier to entry for Terf, eToys and Squeak may be too high. Spur is supposed to be a plug-in replacement for Cog, not something that requires lots of effort to port to