Last updated at 9:49 am UTC on 11 April 2019
Smalltalk sources for the VMMaker generator (VM)
Repository of generated code and platform specific code is on https://github.com/OpenSmalltalk/opensmalltalk-vm
The VMMaker.oscog branch is the branch from which the opensmalltalk-vm code its generated and constitutes the standard, release VM for Squeak, Pharo, Cuis, Terf and Newspeak.
See the image subdirectory in the opensmalltalk-vm repository for scripts to build complete VMMaker.oscog development images.
By Tim Rowledge, 2013
Some history, a description of what VMMaker is, what it does and how. Followed by a short guide on its usage for the brave
In the distant past it was a major labour to create a set of VM code from Squeak. Several do-its to be done. Files to be moved. Did you do the right things in the right order? Will the plugins you want to be internal to the VM be internal? Will the external ones be external – or even created?
In that same misty time long ago, we also had to worry about keeping the code (stored in string literals scattered about the image!) up to date with what was actually compiled to make production VMs. Oh, and it was not in a form suitable for storing in something like CVS or BitKeeper or SCCS.
At last a tool to help the oppressed masses. Or at least, those of us that build VMs on a frequent basis. Don't forget to read the page on the UI for VMMaker as well.
There are a lot of VM configurations in use. We have four major platforms in use; unix land includes at least dozens of variations of display (X11, frame buffer, Quartz, etc) and sound (ALSA, dsp, SUN, etc) and whatever; OS X has Cocoa and Carbon and of course very old PPC machines as well as the intel models; Windows has Cthulhu alone knows how many variants; RISC OS is still in there fighting for all that is right and decent. Then we have a bunch of less active platforms that have been developed over the years like OS/2. Without tools to try to keep this under control we would have no chance.
To fight this problem I wanted a tool that would gather all the VM building jobs together in one reasonably convenient place. It should be both scriptable and interactive.
What is needed in a VM making tool?
The most important capability of a VMMaker is that it must produce the correct sourcecode for your desired VM configuration. An important major feature is handling the platform-specific files that are not generated from the Slang code in the image; a useful touch would be some way to maintain the code for all platforms in a uniform, integrated, directory tree. Ideally, it would be able to read and save configurations so that we do not always have to manually hit little buttons on the screen. Of course, it would be nice to have those little buttons in order to establish the configuration.
Essentially, the entire code production process can be reduced to two tasks; get theSlangcode generated correctly for the chosen configuration and get the handwritten platform files inserted in the proper places in the code tree.
There are three subparts of the system written in Slang
- the interpreter and object memory
- the platform independent parts of plugins, both internal and external
- the entirely system generated list of plugin functions
We also have several files that might seem to be generated when translating Slang, but are actually stored as literal strings within the image.
Once we have decided which plugins should be translated for internal use, which for external and which not translated at all (some plugins may simply be innapropriate, or not supportable on a particular platform) along with the various options such as whether to inline the code, produce code suited to being a web-browser plugin, we can tell the various classes to perform their translations. The files should be written to logical places so that there is a reasonable structure for the makefiles or other compilation tools to work with. During the development of VMMaker there was a lot of discussion about what that structure ought to be and the compromise reached was not entirely tidy.
Handwritten platform files
The core VM requires a number of support functions in order to interface to the host OS; these are typically found in files such as sqXWindow.c, sqRPCEvents.c, etc.
Many plugins require platform support files (FilePlugin and SocketPlugin are two examples) whilst others are complete unto themselves (Klatt and LargeIntegers among them). One or two are entirely handwritten.
These files need to be placed somewhere convenient for the makefile. Of course, it is possible to write makefiles that look just about anywhere for the source files, but generally it is considered useful to gather files together in sensible groups, if only to make it easy to find them when trying to work out why the compile failed. Another useful aspect of sensible grouping is that tools such as the unix 'autoconf' can make use of the grouping to work out which files go together into the VM and plugins.
Sourcecode tree layout
To make life simpler for autoconf-like tools and indeed human minds, we want a source tree that includes just what is needed for the desired configuration; no more and certainly no less. We do not want platform specific files intended for unused plugins to be sitting around. At the same time, we do want all the files for our platform kept somewhere safe and it would be nice if the same arrangement could keep the files for other platforms handy for reference purposes. Note that such an arrangement is very convenient for the basis of a SVN or similar archive; we only really need to archive the files that are handwritten since they can be so easily lost.
VMMaker uses a separate 'platforms' branch for these files and copies the appropriate ones into the tree generated by the Slang translation process. By default VMMaker will use "Smalltalk platformName" to find the type of machine on which it is running and will expect to find a subdirectory matching the result:-
- /Mac OS/
Note the 'Cross' platform; this is where all the files shared between platforms (such as sq.h) are maintained.
Within each platform's directory there should be a subdirectory for each supported plugin that needs platform support - and obviously it should contain the relevant files. Note that the VMMaker only checks for a directory - so if you are starting work on a new plugin you need only create the directory to ensure the platform independent code is produced, ready for you to break. We also need a directory containing the support files for the core vm (called, inventively, 'vm') and optionally another directory called 'misc' in which assorted resources and makefiles etc may be kept. For example:-
The source tree built by VMMaker is arranged in a similar fashion; all the generated core VM files are gathered in one directory, with subdirectories for each internal plugin and then external plugins each have their own place in a similar tree. Any files found in the 'misc/ToCopy' directory above are copied to the root of the source tree. Thus we might find:
...at the end of the processing. This clear separation of all the components makes it very easy to work out which files need to be compiled in which groupings.
How VMMaker works
It should now be obvious that the VMMaker has two basic functions.
- keep track of the configuration - which plugins are translated which way, and what option values have been chosen
- arrange the generation of the files derived from Slang or otherwise created by the image
There are three varieties of plugin we can support, each of which can be used as internal or external code:-
- purely Slang generated code, such as LargeIntegers
- Slang combined with handwritten platform specific code such as FilePlugin
- purely handwritten code such as Unix's Profile plugin (though these typically are not hooked into the named primitives mechanism, nor indeed compiled internally).
To establish a configuration we first have to find out which plugins are known in the image. This is done by listing all the subclasses of InterpreterPlugin and excluding all those that return false when sent #shouldBeTranslated. Some subclasses such as SmartSyntaxInterpreterPlugin are abstract parents of other plugins and do not get translated.
Next we have to decide which plugins to generate for internal use and which for external use - and which to leave out. Plugins needing platform support code that cannot be found within the relevant platforms directory tree must be left out since the compilation would fail; VMMaker will automatically avoid these. When scripting the build process you can use the #generateAllInternal or #generateAllExternal messages for the simple cases, or simply inspect the result of 'VMMaker default' and use methods such as #initializeAllExternalBut: #(ThisPlugin ThatPugin). Note that the lists contain the names of the plugin classes rather than the classes themselves. This reduces problems of dangling references should the image be shrunk in any manner.
Once we have a configuration to our liking we can move on and generate the interpreter code. This is handled in VMMaker>generateMainVM and #generateExternalPlugins, where you can see the process laid out in easy steps.
- #generateIntepreterFile simply delegates to the Intepreter class, specifying the required directory (typically 'src/vm/')
- #processFilesForCoreVM and
- #processAssortedFiles copies any files from misc/ToCopy to the root of the target dir - such as a readme file to explain how to compile the vm.
- #generateInternalPlugins enumerates all the plugins intended for internal compilation, delegating to the proper class just as for the Interpreter. If a plugin requires platform support and none can be found we generate an error.
- All the exported functions from each internal plugin and the vm are added to the exports collection for later writing out as sqNamedPrims.h in the #generateExports method.
- #generateExternalPlugins is similar to #generateInternalPlugins with two differences. No function exports need to be recorded and a missing platform support directory means we simply skip over the plugin.
I will assume you already have a suitable 'platforms' directory for your platform. Exemplar directories exist on thehttp://squeakvm.org/svn/squeak/trunk/platformsSubversion site for you to download. You will need a suitable client for your platform - see http://subversion.tigris.org for more info.
The system has a nice UI tool described in VMMakerTool.
The simplest scripting usage of VMMaker is:
VMMaker default initializeAllInternal generateEntire
which will make a source tree ready to compile with all plugins (excluding those that lack needed platform support) generated for internal linking.
If you have a slightly more complex configuration you want to use, perhaps with Socket and Serial support external (because for your case they are rarely used and saving the space has some value) then you could try
(VMMaker default initializeAllInternalBut: #(SocketPlugin SerialPlugin) generateEntire
More complex still would be
(VMMaker default initializeInternal: #(BitBltPlugin MiscPrimsPlugin FilePlugin) external: #(SocketPlugin ZipPlugin B2DPlugin)
which allows you to precisely list all the plugins to use. WARNING If you miss out a plugin you need, it won't be there. This message is really best suited to use by a UI - see later.
To save a configuration for later use, you need to send #saveConfiguration to an active instance of VMMaker.
And now, on to the VMMakerTool the GUI for VM builders everywhere. Well, in Squeak land, anyway.
VMMaker is available via the Monticello package system from the http://source.squeak.org repository.
The same repository also has VMMakerInbox to accept fixes.