Implementing Apache-Test Support into Simple Test

I have successfully gotten Simple Test to test itself in Apache-Test.
It wasn’t that hard really. I have added very basic Apache-Test support
by just subclassing the SimpleReporter class in SimpleTest. The new
class name is TAPReporter. It basically just allows a test to create TAP compliant (see this post)
output which is then read by Apache-Test. There are a few things which
in my opinion could be changed in SimpleTest in order to make TAP
complancy much better and it would actually improve some of the
functionality of SimpleTest apart from Apache-Test. I am somewhat
hesitant to make those changes right now however as the code is not
mine and really should be discussed with the creator of SimpleTest.
(Which I do plan on doing.) I will outline later in this articles the
changes I think would be useful. For now I would just like to review
what I did do and how to run the tests.


All of the current Apache-Test support resides in this one file. The clas in this file is TAPReporter. It is a subclass of SimpleReporter and is really quite simple in nature. I’ll post the stub and documentation of the class below:

If you were to see the code inside you would realize how simple it
really was to add the basic support. All praise for that goes to Marcus
Baker and the other original developers of the project. It’s been a
while since I have come accross such an easily extensible open source

What it Does

Like I have said before this new file adds just the basic syntax
required by Apache-Test. It was the best I could do without having to
modify some of the already existant classes. In my opinion it is a
great starting point to implement full TAP functionality and in the
process complete Apache-Test support. Here is a list of what the frame
work does:

Test Plan

Test plans had to be held off for display at the end of each test.
This isn’t necessarily the best way to do it but it is somewhat
mandated to be done that way barring some changes in the main
SimpleTest API and underlying code.

Test Lines

Almost all of the test line support is there. Test numbers are
calculated and displayed and test descriptions are given. The only
thing missing here is the ability to add directives to tests. This
again would take some heavier additions and modifications to SimpleTest.


Diagnostic lines are implemented via the TAPReporter::paintFormattedMessage() and TAPReporter::paintMessage() functions. Any information passed to these two functions will be printed out as diagnostics. Some automatic
diagnostics can probably be considered, things such as printing out the
test name and what not. I however did not implement anything of that
sort this time around.

Bail out!

There is no support for Bail out! I am not currently familiar if
there are any functions in the Simple Test API that support this
feature of TAP. If there is I do not know what they are, so if you do,
tell me :).

Possible Changes for Simple Test

Now following are some ways that I could see changing SimpleTest
such that it would be able to more fully support all of the features of

Better Plan Support

As I have already mentioned, there is no way in SimpleTest currently
to forecast the number of tests that will be made. While there are no
real ill side effects of this, I do think it would be nice to offer the
ability. The best route I could see in this is to add a function to SimpleTestCase
called plan. This function could be passed a number that should be
representative of the number of tests that will be performed. This
would by no means be a function that users would be required to use.
However if they would like to have better management over the number of
subtests they would like to run in the script they would have that
ability. The GroupTest class can handle these plans
by simply adding up the number of plans each added test has. If any of
the added tests do not have a set plan, then the GroupTest‘s
plan would be automatically unset and the script would revert to the
current behaviour of establishing and displaying the plan after the
test has ended.

Directive Support

The TAP format has two different directives allowed for each test. TODO and SKIP.
The TODO directive is used to denote portions of the application
being tested that are incomplete. The SKIP directive is used if there
is no need to
complete the given test. There isn’t a really easy way to integrate
directives into the tests unfortunately. The best way that I can think
of is by adding two more parameters to the various assert*() functions found in the SimpleTestCase class and its descendants. The first parameter would be something like $directive and would contain a constant (either SIMPLE_TEST_DIRECTIVE_TODO or SIMPLE_TEST_DIRECTIVE_SKIP) indicating which directive to use. The second parameter to be added would be $directivemsg
and would be used to pass an optional description with the directive.
If both parameters are ommitted then the behaviour would be as it is
now and no directives would be printed out. These two parameters would
also have to be existant in the following functions (and any overwrites
of the functions):

  • SimpleTestCase::pass()
  • SimpleTestCase::fail()
  • SimpleScorer::paintPass()
  • SimpleScorer::paintFail()

The last change that would have to be made would be in the TAPReporter class’s version of paintPass() and paintFail(). The only change there would be to simply add the two parameters and display them in the output (in the appropriate place.

bail out!

As I mentioned before, I do not know of any functions in the
SimpleTest API that allow you to bail out of a test. My thought
currently is to overide the SimpleScorer::paintError() function in TAPReporter
to display the error as a bailout. The only thing I have against that
is whether or not there will be instances when a bail out is not
necessary. The workaround is to of course introduce a new function into
SimpleTestCase called bailout($msg). This function would then call a method that would be added to SimpleRunner called paintBailout($msg) which would in turn call the same method in SimpleScorer. The default implementation of SimpleScorer::paintBailout() could be to just simply exit; out of the script. TAPReporter will of course extend it to print out the bail out! <msg> line before exiting.

Download the Work So Far

I am sure you would all like to see this all in action so I have
created a new Tarball of simple test containing apache test. Before
attempting to run the apache-tests I recommend you read through
simpletest/apache-test/INSTALL-APACHE-TEST. After you have ensured that
Apache-Test is installed properly just use the following commands in
the simpletest/apache-test directory.

  1. perl – This will ask you the
    location of your httpd and apxs file. The location’s mentioned should
    be right, if they are not you can probably just find them using which apxs or find / -name apxs
  2. make test

and that’s all there is to it.

*EDIT* I have updated tap-reporter.php since this post, for a brief overview of the changes go here. The link below has been updated so download away :).


Leave a Reply

Your email address will not be published. Required fields are marked *