As some of you know I am currently working on a project to allow Simple Test to be Apache-Test
compliant. The process to take in doing so is to enable Simple Test to
produce TAP compliant output. To do so obviously requires a healthy
knowledge of the different aspects of the TAP protocol. The best way
for me to get a grasp of something is of course to write about it. I
gleaned pretty much all of this information from CPAN’s TAP format documentation.
The general format of TAP is really quite straightforward. An example TAP file is shown below:
# Test Input File
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
not ok 4 - Summarized correctly # TODO Not written yet
ok 5 - Test for file contents # SKIP File not world Readable
You notice that there our 3 different types of lines. The first one you encounter is 1..4. This is called the plan.
Basically it is used to tell the testing harness how many tests you are
expecting to perform. This line will usually be the first line of
output. There may be times however where you cannot know the total
number of tests you are going to perform until all tests are complete.
If that is the case you ARE allowed to place the line at the end of the
file. This is going to be the case for any test currently in SimpleTest
as there is not yet any built in functionality to count the number of
tests before they are performed. This line will always be in this
format: 1..n. Where n is the number
of tests you plan to or already have performed. So basically, the only
important thing to remeber about this line is that it has to go either
at the begining or end of the output and it must contain a count of the
number of tests in the current test run.
The Test Lines
The test line is probably the most common and most useful line in
the test file. It tells the test harness the results of each test. The
structure of the test line is fairly simple: <ok|not ok> [testnumber] [description] [# <directive>]. If the test passes then the first part of the line will read ok. If the test fails the first part of the line will read not ok.
The next part of the line is the test number. This is of course a value
that will increase by 1 with each test. It is not required by the TAP
format and will be automatically generated if it is not provided. The
next part is a description. This is where you would put a reason as to
why the test failed or passed. It is not necessary and harnesses are
allowed to do whatever they would like. Most harnesses should have a
‘verbose’ mode that could be enabled to see these descriptions. One
thing to be careful of is to not start a description with a number as
that could confuse the harness into thinking the description is the
testnumber. As a habit, I start all of my descriptions with a ‘-’. The
last part of the test line is called the directive.
Test Line Directives
There are two possible directives. The first of these is the TODO
directive. This directive is used to denote portions of the application
being tested that are incomplete. In other words, tests marked as todo
are expected to fail and therefore will not be counted when
determining if the application as a whole passes. You can also pass an
extra description after the TODO directive if it is followed by space.
This directive can be used to help you manage which features are and
are not implemented in a program. For instance, if you have a TODO test
that all of a sudden starts passing you will know that the feature has
since been completed. You can then remove the ‘TODO’ directive from the
test and add it to you feature list .
The second directive is SKIP. This is used if there is no need to
complete the given test. For instance, if you have a test that will
pass or fail based on a certain known state of the program (and this is
not a bug), you can just elect to document the test as being skipped
instead of it registering as an error.
These are the lines that begin with a ‘#’. (Not to
be confused with test line directives.) They will have no effect on the
test results but if the harness chooses, it can display them. They can
be used to pass extra information back to the user running the test.
(If the harness allows it.)
There is one line that I haven’t yet mentioned: Bail out!
This is used to immediatly halt a test. This could come in useful if a
majority of your test relies on one component to work properly. If that
component tests falsely (has bugs) then your script can just print Bail out! to halt the test and prevent a bevy of errors showing up that are most likely all being caused by the same component.
So, that is the pure basics of the TAP format. I have actually
successfully implemented a TAP compliant reporter for SimpleTest, but I
am unfortunately out of blogging time today, so that’ll have to wait