TAP — Test Anything Protocol
Years ago, when I first heard about the Test Anything Protocol (TAP), I gave it little thought. I wrote tests, I ran them, and I saw if they all passed or not. Whatever programming language I used, the testing harness — test runner, framework, data, the tests and surrounding tools — usually produced some result that could be easily interpreted by humans and machines, most of the time. For all I wanted from my tests back then, it was enough.
Over time, different needs and expectations arose. Using many test frameworks, switching between languages, and creating tooling that works on top of the test runner output; a lot can change with projects maintained over a long period of time.
And with that ever-changing landscape came the longing for stability and reusability — to not again write a tool for the next test runner, not reinterpret test output, or be able to get the same output format for tests written in different languages.
TAP is a text-based interface, offering a unified way of not only reporting test results but also how tools should handle the output, as a stream, describing the communication from a TAP producer to a TAP consumer. This decouples the test reporting from the presentation and further processing.
TAP producers can be many things, from unit and integration testing frameworks to visual testing software, lint tooling, build systems, etc. Support exists in many languages, but there are also language-agnostic tools.
TAP consumers are programs that take the input and transform, process, or format it. This can range from adding color, full reformatting, transforming to different output formats such as JUnit or HTML, running analysis and statistics on it, integrating it into CI/CD like Jenkins, or even getting desktop notifications, if so desired.
A core philosophy is to not wait for the full output, but rather process the data line by line, or chunk by chunk, as a stream. The reason for the "data chunks" is that TAP can contain YAML blocks, that can potentially only make sense as a whole block.
The full specification for the current version, TAP14, can be found on the official TAP website. Taken from this website, this is an example of what a test output can look like for four tests, where two pass and two don't.
The specification also describes how consumers should handle new versions, with the goal of being forward-compatible.
In general, the official website is very readable and holistically describes TAP, how to implement the protocol, how to upgrade one TAP version to another, the ideas behind it, language and tooling support, and much more. It is well worth a read if the use of TAP is considered.
Getting giddy about a protocol
In a full test harness, neither the producer nor consumer need to be written in the same language. Tooling from one context can be used in another, resulting in different projects sharing more tooling and me spending less time reworking systems to fit different needs.
That was what really sold me on TAP. It is minimal and readable*, tools can be reused; and it is language agnostic, being nothing more than a text-based interface, an interface that has served me very well over the years.
Whenever I have the option to use TAP, I will. Without further ado, let's take a look at some examples of how to produce TAP with different languages and tools.
Getting TAP as a reporting format is a straight-forward process, usually setting a "reporter" flag. I picked a collection of possible producers to showcase the setup.
C++ with Catch2
comes with TAP as a reporter built-in. To get a minimal example
with CMake going, the following CMake
CMakeLists.txt will fetch
Catch2 and set up an example project.
Test.cpp looks like the following, taken from the
Building the project with CMake through the command line.
And finally, running the test using TAP as a reporter.
Lua with LuaUnit
The popular Lua testing framework LuaUnit has TAP built-in as a reporter. To set up an environment to run tests with LuaRocks, I can refer to an earlier article of mine on setting up projects with LuaRocks.
With the environment ready and a project folder capable of running LuaRocks, a
rockspec could look like the following.
Given a test file.
The LuaRocks test command can be executed, passing the output option for TAP to LuaUnit.
testing library. With Node.js installed, to get a minimal setup going, the
package.json can describe a project.
A small sample test file shows two tests passing and one failing.
Installing the node-tap dependency for the project setup.
And running the tests. By default, node-tap will produce TAP output.
Python with Tappy
Tests can be written like any other in Python; here is a test file taken from the "unittest" documentation.
Running this test via Python will produce TAP.
CSS with StyleLint
The CSS linting tool
does support TAP as a reporter — Node.js with NPM needs to
be installed. For a minimal setup, the following
describe the project.
npm run lint command will find all CSS files in the
project and run the linter on them. As an example, two different CSS files
will demonstrate the test, one with a problem and the other without.
Running the lint command will produce TAP for the lint report.
PostgreSQL with pgTAP
pgTAP for PostgreSQL is a collection of database functions to truly write unit tests for a database while emitting TAP. It requires PostgreSQL 9.1 or higher and needs to be installed on the host that runs the database server.
installation of pgTAP, the
pgtap extension can be added to a PostgreSQL database by
running the following as the superuser for the database that should be tested.
The following example test is taken from the official documentation of pgTAP.
And finally, running that test against a PostgreSQL database, producing TAP.
I will need to point to the official documentation for consumers, as it contains a multitude of options for different languages, with quite a few producers also providing options for consumption. Additionally, there is always the option to search the internet for more — there always is, more I mean.
The best part here is that, thanks to TAP, consumers can work with any producer, no matter the language or context.
I have stronger feelings about TAP than I initially thought I would have, and this article can only give a small glimpse into them. It may not look like it at first, but this is a love letter to TAP and hopefully an encouragement for others to give it a shot.
All the producer examples can be found in the companion repository on GitHub.
Until then 👋🏻