SUnit Test Runner in 3.8
Last updated at 7:33 pm UTC on 11 December 2006
Three major issues were raised about SUnit Test Runner. 1)Should we back out the change that has tests running in background? 2) Should we backout the change that excludes abstract tests? 3) should we move to some other interface.
Andreas Raab November 02, 2004 I don't know how many people out there use 3.8 but given the behavior of TestRunner I cannot imagine that many people use it for daily work. In moving some code from 3.6 to 3.8 I immediately noticed that some important things have been broken, others removed, as if it were the goal to make it harder to run tests.
- As has been pointed out http://lists.squeakfoundation.org/pipermail/squeak-dev/2004-March/074876.html TestRunner is STILL broken if you run any UI tests with it.
- For some odd reason, someone must have decided that a nicer look of TestRunner is more important than the work it does. So what we got in this process is a beautifully anti-aliased progress bar ... and what we lost is the ability to easily run multiple tests at once.
In case this hasn't been clear to everyone, abstract test cases run all of their "sub-tests" if executed. In other words, in the 3.6 TestRunner you can have a test structure that looks like:
and with AllMyTests being an abstract superclass you only need to select it and run all of my three tests at once. In the 3.8 TestRunner you don't get to see abstract test cases in the list - but rather, you need to select EACH AND EVERY single test manually in this list. Which, besides, is broken if you try to do a selection by dragging through with the mouse but anyway.
So if we were interested in BOTH good look and good usability we really ought to do a number of things, including:
a) get rid of the single multi-selection list and replace it with a tree widget that shows the structure of trees, after this has been done
b) get rid of all of the "run one" and "run all" button and have only a single "run" button (which automatically runs the hierarchy of the tests selected)
c) get rid of the "select all", "deselect all", "toggle selection", and "filter" stuff
At which point we have a very nice, simple, good looking and actually usable UI with a clean structure of tests that may actually be of help to someone who is trying to get work done.
Torsten Bergmann November 02, 2004 Squeak now behaves like all the other SUnit implementations. Abstract means there are no instances and therefore no test cases to run. So I would expect it to behave like it is now - not to show or run abstract classes.
We should use abstract test superclasses to implement common behavior and not to run the tests together. If we have what you describe there is the danger that people change their systems design just to get the tests to run together.
So it's mainly the problem of using the simple standard TestRunner UI. I use the TestBrowser. [Try downloading http://www.squeaksource.com/TestBrowser/TestBrowser-tbn.115.mcz Note that it installs a preference #testBrowserRunsAbstractClasses - so it's up to the user how it should behave regarding abstract classes. More info: TestBrowser ] ..If you have used it for a while I'm sure you will like it and throw the simple TestRunner away. I use it in Sq3.6 up to Sq3.8 images and it's rock solid !!! It's nice since you can run tests by category, by class or run single test methods. Especially when you use Monticello and special class categories for tests this is a really helpfull and clearly arranged tool. I would vote for replacing the simple TestRunner with the TestBrowser in the standard image and provide the old TestRunner as an optional package. Later we want to convert it to use OmniBrowser as it's base architecture.
Romain Robbes If all your tests are in a single package (a PackageInfo instance), you could use SUnit Test Runner in 3.8 to run them at once.
Stefan Matthias Aust Ok, yes, I should have put [the fix for UI testing] into the SUnit package sooner... I will soon.
I was 100% sure that the idea of an abstract testcase was that they are supposed to be abstract classes. (Do not show up in Testrunner, test are inherited by the children). I never had the idea to use them for grouping tests... so with that mental model, I added the fixes that removed the abstract tests from the testrunner that were posted to the list.(Having Abstract classes in the TestRunner has the problem that "run all" then runs tests multiple times).
For grouping, I wanted to add a pane with the sytemcategories to the testrunner. But I havn't yet come around to do that.
Another thing I should look at again is Romain's BrowseUnit. I used an older version a lot.
Colin Putney I was probably the person that objected to [the fix for UI tests] the most. In hindsight I think I was wrong - this isn't the "best" solution, but it definitely has the best utility/effort ratio. Let's get this
I was also the one that submitted [the change to exclude abstract test cases], but I stand by this one. The problem with the old behavior is that it also causes you to have to select each and every test manually. You can't just use the 'Filter' button to select some tests and run them all, because the abstract classes will cause tests to get run twice - or more than twice if you have several levels of abstract classes in your test hierarchy.
I like your idea of revamping the test runner UI - we need a better way to group tests into suites. I'll take a stab at doing something with OmniBrowser and post when I have something to play with.
Bert Freudenberg Wouldn't it be trivial to avoid these duplicate tests? Like if you only collect test methods from the class that actually implements them? Am I missing something obvious?
Stefan Matthias Aust [re: Am I missing something obvious?] But there are testsuites that use Abstract Tests in the sense of
abstract classes: e.g. the ClassTestCase implements a bunch of tests that should be run on all tests that are subclasses of ClassTestCase.
I really don't like the idea of grouping by inheritance... and having something named Abstract that you can run is strange, somehow.
Goran Krampe [re: having something named Abstract that you can run is strange, somehow]it seems to me that suites should be used for grouping. As always, object composition seems more appropriate than inheritance.
Andreas Raab [Marcus on showing abstract tests] This is in clear contradiction to the code which can be found in, e.g., TestCase class>>buildSuite which specifically builds a test suite of all the sub tests if invoked on an "abstract" superclass. So the intent here is obviously that it's perfectly okay to "run abstract test cases".
I mean, how do you run all of Squeak's tests programmatically at once? The way I do it is simply executing "TestCase buildSuite run" and off you go. What do you do?
[Having Abstract classes in the TestRunner has the problem that "run all" then runs tests multiple times)]
I might not have been clear on this issue - it is the whole idea that you run these tests "multiple times". Consider the following situation: Let us assume we have a set of invariants for a class which must hold true for all subclasses. Such as "when the mouse moves over [any subclass of] button its state changes from normal to over". Yes, we can rely on copying the test for each and every subclass of Button. Or, alternatively, we put it in the superclass (abstract or not) with the specific intent that this test is an invariant for the (abstract or not) class and all of its subclasses. So that if you make a new button subclass these invariants can be automatically validated without you much doing about it.
Also, it is likely that situations in which this is relevant (lots of tests which should be grouped together) will expose some common behavior which is best put into a common superclass.
[Bert: Wouldn't it be trivial to avoid these duplicate tests?] That's quite right. There are various other ways in which the UI could make it much simpler to specify individual tests and groups (and assemble the "right" set of tests to be run). And really, all I'm asking for is ONE solution - and the last one available was taken away with the latest changes
[Markus: there are testsuites that use Abstract Tests in the sense of abstract classes: ]And there is absolutely no problem with this. If ClassTestCase is abstract then (to repeat) "ClassTestCase buildSuite" will do The Right Thing and that's precisely what I am asking for.
[Goran: object composition seems more appropriate than inheritance] And again, all I can point to is TestCass class>buildSuite which does precisely what I am asking for, no more, no less. It seems that the author(s) of SUnit and I are in agreement about "good behavior" here and all I am asking for is a UI which makes that behavior accessible.Colin Putney[Re: Andreas Raab ] I completely agree about the utility of abstract test cases. I use them a lot. Consider the situation you mentioned in your first mail, something like this:
FooAbstractTest has a single test method #testBar, which gets run in slightly different contexts in FooTestA, FooTestB and FooTestC. Now, suppose I use the filtering in the test runnner to select all the Foo-related tests with the pattern 'Foo', and click Run All. This is what gets run:
I have no problem with #testBar being run 3 times. As you say, that's the whole point. But I don't want it to run 6 times, as having it fail twice doesn't tell me any more than if it had failed once. Ultimately this boils down to different mechanisms used for building suites. I use names and filters to group tests, while you use the inheritance hierarchy.
Stefan Matthias Aust I'l add a preference to the next version of TestRunner to turn off filtering of abstract testcases.
How to save a parts bin [Re problem running user tests] What I think should happen is that there should be a TestResource that can be used by those few tests that actually require the UI. This TestResource would ensure that the tests run in the foreground.Or perhaps the resource could be able to run individual tests in the foreground.
Andreas Raab Or, perhaps, the resource would be able to run individual tests in the background. ...Seems to me that the default ought to be to run tests in the foreground unless otherwise specified.
Marcus Denker So which should I include? It seems that running in the foreground will be default, as there is no way to tell the side effects (e.g. even adding/changing a method when having a MC Browser open has a graphical sideeffect).
So Andreas' fix seems to be better for that, as it does not block UI interaction completely....