Martin Helmut Fieber

How to test your Lua

Test all the things!

After the first article of this series, setting up a project with Lua and LuaRocks, and a deep dive into managing Lua modules, it is time to look at how to test Lua code in this part of the series.

The article will start by showing how to quickly iterate over code and run small snippets — manually, without external dependencies. After that; unit testing, first without and then with an external module, LuaUnit; how to create mocks for tests; and finally using continuous integration.

All the code shown in this article is also available in the companion repository on GitHub, branch part-3.

About versions

The latest Lua version as of writing is 5.4 (released June 2020), though everything in this article series will work with Lua 5.1 (released February 2006) and up. If this deviates, a callout like the following will signal the required version.

An overview of all versions and main their features can be found on the official Lua version history page.

Quick setup

Besides Lua, LuaRocks will be used for the base test setup. I covered setting up both in the first article of this series, showing how to install and run Lua and LuaRocks for macOS, Linux, and Windows.

With both installed, running the following command inside a project folder will create a new LuaRocks projects.

$ luarocks init \
    --lua-versions "5.1,5.2,5.3,5.4" \
    my-project 1.0.0

This will set up the package "my-project" with LuaRocks. More about managing projects with LuaRocks in the second article of this series.

Running tests

With the project setup done, it is now possible to run ./luarocks test. Without further parameters, this will run a test.lua file from the project root if found.

Alternatively, the test property of the project .rockspec file can be used to define a different location or file for the test entry.

-- my-project-1.0.0-1.rockspec
-- other properties ...

test = {
  type = "command",
  script = "src/test.lua"
}

Besides using a Lua file as the test entry, a shell command can be used by changing script to command.

-- my-project-1.0.0-1.rockspec
-- other properties ...

test = {
  type = "command",
  command = "lua src/test.lua"
}

For now, the test entry file will only print that no tests are defined.

-- src/test.lua
print("No tests defined, yet.")

All options for setting up the test command can be found in the test command specification on the LuaRocks GitHub wiki.

Iterative work

Even with a way to run the test entry point via LuaRocks, there is also a simpler way to test code without LuaRocks or even tests — running and verifying the code manually.

I'm absolutely in favor of test automation, but for small scripts and snippets, manually testing, as with Lua's interactive mode, can be a viable way for quick iterations.

Running the lua CLI without any arguments will enter interactive mode.

$ lua
Lua 5.4.6  Copyright (C) 1994-2023 Lua.org, PUC-Rio
>

To test how to iterate on code, the following small piece of code, called chunk in Lua, will be defined inside a new file factorial.lua.

-- factorial.lua
function factorial(n)
  if (n == 0) then
    return 1
  else
    return n * factorial(n - 1)
  end
end

print(factorial(4))  --> 24

The idea is to use the dofile function for quick tests.

$ lua
Lua 5.4.6  Copyright (C) 1994-2023 Lua.org, PUC-Rio
> dofile("factorial.lua")
24

After making changes to the code, dofile can be executed again from the console to re-run the code. Using this approach will enable quick iterations on small chunks via the command-line.

Unit testing via assert

Before diving into LuaUnit, for small tests, Lua's assert function can be used in combination with pcall. To keep the setup minimal, all that is needed is a test helper function.

-- src/examples/assert.lua
function test(fn)
  local status, err = pcall(fn)
  if not status then
    print(err)
  end
end

The function will take an anonymous function as an argument, call it via pcall, and in the case of an error, print the result.

Using this approach, tests for the factorial function could look like the following.

-- src/examples/assert.lua
local factorial = require("examples/factorial")

test(function ()
  assert(
    factorial(1) == 1,
    "Factor of 1 should be 1")
end)

test(function ()
  assert(
    factorial(4) == 24,
    "Factor of 4 should be 24")
end)

A slightly more complex example would be to create a test runner table to handle how to exit the test on failure.

-- src/examples/test-runner.lua
local Runner = {
  hasErrors = false
}

function Runner:test(fn)
  local status, err = pcall(fn)
  if not status then
    self.hasErrors = true
    print(err)
  end
end

function Runner:evaluate()
  if self.hasErrors then
    print("Test suite exited with errors")
    os.exit(1)
  else
    print("All tests successful")
  end
end

return Runner

This will use the same test function but set a property hasErrors when a test fails. Via the evaluate function, called after all tests, a proper exit code and status message can be emitted.

The same test suite for the factorial function with this small runner could look like the following.

-- src/examples/factorial-test-2.lua
local tr = require("examples/test-runner")
local factorial = require("examples/factorial")

tr:test(function ()
  assert(
    factorial(1) == 1,
    "Factor of 1 should be 1")
end)

tr:test(function ()
  assert(
    factorial(4) == 24,
    "Factor of 4 should be 24")
end)

tr:evaluate()

Used as a base, one can already get quite far with this simple test runner, which can even be further extended.

For example, instead of directly calling defined test functions, gather them all and run them when calling evaluate. This would then enable one to pretty print the test output or even shuffle tests to ensure test isolation.

Unit testing via LuaUnit

The LuaUnit module installed via LuaRocks is a very popular unit testing framework for Lua that comes packed with features.

Setup LuaUnit

LuaUnit needs to be added to the test dependencies of the project. Dependencies defined under the test_dependencies property in the projects .rockspec file will be automatically installed when running ./luarocks test.

-- my-project-1.0.0-1.rockspec
-- other properties ...

test_dependencies = {
  "luaunit >= 3.4"
}

The .rockspec test definition also needs some adjustments, namely running the entry test file with Lua directly to be able to pass command-line options. As I really like TAP I will use it as the output format for all tests.

-- my-project-1.0.0-1.rockspec
-- other properties ...

test = {
  type = "command",
  command = "lua src/test.lua -o TAP"
}

Test conventions

For LuaUnit tests are functions or tables of functions that all start with the word test or Test and get executed through the root test file.

-- src/test.lua
local lu = require("luaunit")

-- Define tests ...

os.exit(lu.LuaUnit.run())

A single test

Taking the example function from earlier, a single test case can be created by defining a function starting with Test.

-- src/examples/factorial.lua
local function factorial(n)
  if (n == 0) then
    return 1
  else
    return n * factorial(n - 1)
  end
end

return factorial

-- src/test.lua
local lu = require("luaunit")
local factorial = require("examples/factorial")

function Test1()
  lu.assertEquals(factorial(4), 24)
end

os.exit(lu.LuaUnit.run())

Group tests

Multiple test cases can be organized or grouped in a table. This will increase readability when running the tests.

-- src/test.lua
local lu = require("luaunit")
local factorial = require("examples/factorial")

TestFactorial = {}

function TestFactorial:test1()
  lu.assertEquals(factorial(0), 1)
end

function TestFactorial:test2()
  lu.assertEquals(factorial(1), 1)
end

function TestFactorial:test3()
  lu.assertEquals(factorial(4), 24)
end

os.exit(lu.LuaUnit.run())

Multiple files

A growing test suite needs to be organized for better maintainability — splitting tests in multiple files can be one way. To enable this pattern, a test module needs to return the defined test table; the entry test file will then require all defined tests and execute them.

To showcase this, a new example function will be created.

-- src/examples/round.lua
local function round(num, numDecimalPlaces)
  local mult = 10^(numDecimalPlaces or 0)
  return math.floor(num * mult + 0.5) / mult
end

return round

The tests for this new example will be in a separate file, returning the local table with the test cases.

-- src/examples/round-test.lua
local lu = require("luaunit")
local round = require("examples/round")

local TestRound = {}

function TestRound:test1()
  lu.assertEquals(round(3.44), 3)
end

function TestRound:test2()
  lu.assertEquals(round(3.44, 1), 3.4)
end

return TestRound

Same for the factorial tests, moving them into a separate file and exporting the test table.

-- src/examples/factorial-test.lua
local lu = require("luaunit")
local factorial = require("examples/factorial")

local TestFactorial = {}

function TestFactorial:test1()
  lu.assertEquals(factorial(0), 1)
end

-- other test cases ...

return TestFactorial

I like keeping tests close to the implementation, which is why tests are in a file with the same name and the postfix -test in the same folder — having a factorial-test.lua test file for the factorial.lua implementation. Mind you, this is my personal preference. Another typical pattern is having a single test folder containing all test files.

Now with dedicated test files, the entry file will require all tests and run LuaUnit.

-- src/test.lua
TestFactorial = require("examples/factorial-test")
TestRound = require("examples/round-test")

local lu = require("luaunit")
os.exit(lu.LuaUnit.run())

Running ./luarocks test will execute the test suite.

$ ./luarocks test

1..5
# Started on Fri Jul 12 20:15:00 2023
# Starting class: TestFactorial
ok     1  TestFactorial.test1
ok     2  TestFactorial.test2
ok     3  TestFactorial.test3
# Starting class: TestRound
ok     4  TestRound.test1
ok     5  TestRound.test2
# Ran 5 tests in 0.000 seconds, 5 successes, 0 failures

Setup and teardown

LuaUnit provides a way to call code before and after a set of grouped tests — often called "setup" and "teardown" or "beforeAll" and "afterAll" — via the special functions setUp and tearDown.

-- src/examples/logger-test.lua
local lu = require("luaunit")
local Logger = require("examples/logger")

local TestLogger = {}

function TestLogger:setUp()
  self.fname = "mytmplog.log"
  -- Ensure file does not already exist.
  os.remove(self.fname)
end

function TestLogger:testLoggerCreatesFile()
  Logger.setup(self.fname)
  Logger.log("Hello?")

  f = io.open(self.fname, "r")
  lu.assertNotNil(f)
  f:close()
end

function TestLogger:tearDown()
  -- Cleanup log file.
  os.remove(self.fname)
end

return TestLogger

Pass arguments to LuaUnit

Arguments passed to LuaUnit through ./luarocks test should be separated by --. For example, using LuaUnit's pattern option to select a subset of tests to run.

$ ./luarocks test -- -p TestFactorial

This will only execute tests matching the pattern TestFactorial.

Mocking

There are many ways to isolate code and replace external dependencies for testing, called mocking. In my time using Lua, I found no test-mock library that won my heart; instead, I use a combination of different techniques to mock code for testing.

Testability of code

Increasing testability should always be the first step, avoiding any workarounds to mock code altogether. Instead of accessing a dependency inside a function, pass it as an argument; initialize an object with needed dependencies; or make default values accessible to override via a function or property.

Here is an example of moving dependencies outside a function.

function getEmail(name)
  local user = UserService:query(name)
  return user.email
end

getEmail("Martin")

function getEmail(service, name)
  local user = service:query(name)
  return user.email
end

getEmail(UserService, "Martin")

Another example is having a default dependency with the option to override it on object creation.

-- Defining a default logger
Player = { logger = { log = print } }
function Player:new(o)
  o = o or {}
  setmetatable(o, self)
  self.__index = self
  return o
end

function Player:say(what)
  self.logger.log(what)
end

-- Using the default logger
p1 = Player:new()
p1:say("Hello")  --> "Hello"

-- Overriding the default logger
p2 = Player:new({
  logger = {
    log = function () print("Mocked") end
  }
})
p2:say("Hello")  --> "Mocked"

Basic mock

A basic mock can be anything that replaces a dependency in a chunk of code. What this value does or how it is shaped depends on the goal of the test; e.g., an empty function can be passed to skip or remove a call, check if a method actually got called, or simulate values on a table to fully isolate a test.

To showcase this, the following createUser function will be tested; the callback will be mocked.

function createUser(name, callback)
  -- User creation code ...
  callback(user)
end

The goal is to test if the user gets created successfully; therefore, an empty callback function is of no use. Only if the callback is called does the user get created.

One way to test this is to create a mock function that toggles a boolean when called and asserts the value.

local lu = require("luaunit")

function Test1()
  local hasBeenCalled = false
  createUser("Martin", function ()
    hasBeenCalled = true
  end)
  lu.assertTrue(hasBeenCalled)
end

os.exit(lu.LuaUnit.run())

Mock exported functions and values

Mocking a function or value exported by a module is best done inside LuaUnit's setUp and tearDown functions. The exported function or value gets replaced with a mock in setUp, keeping a reference, and restoring the original value in tearDown.

-- src/examples/mock-exported-test.lua
local lu = require("luaunit")
local someModule = require("some-module")

local TestSomeModule = {}

function TestSomeModule:setUp()
  -- Backup the function that will be mocked
  self.backupFn = someModule.fn

  -- Mock function for tests
  someModule.fn = function ()
    return true
  end
end

function TestSomeModule:test1()
  lu.assertTrue(someModule.fn())
end

function TestSomeModule:tearDown()
  -- Restore mocked function
  someModule.fn = self.backupFn
end

os.exit(lu.LuaUnit.run())

Mock globals

As Lua keeps all global variables in a table called _G, the environment, global functions and values can be mocked for tests in the same way as exported values.

To demonstrate this, the goal for the next example is to mock the print function and test if it has been called at least once.

function log(text)
  print(text)
end

Inside LuaUnit's setUp a reference to the global print function will be kept and restored inside tearDown.

function TestGlobal:setUp()
  self.backupPrint = _G.print
end

function TestGlobal:tearDown()
  _G.print = self.backupPrint
end

The actual test unit will define the mock function with a boolean to check if it has been called.

function TestGlobal:test1()
  local hasBeenCalled = false
  _G.print = function ()
    hasBeenCalled = true
  end

  log("Hello World")
  lu.assertTrue(hasBeenCalled)
end

This is how the whole example looks.

-- src/examples/mock-global-test.lua
local lu = require("luaunit")

local function log(text)
  print(text)
end

TestGlobal = {}

function TestGlobal:setUp()
  self.backupPrint = _G.print
end

function TestGlobal:test1()
  local hasBeenCalled = false
  _G.print = function ()
    hasBeenCalled = true
  end
  log("Hello World")
  lu.assertTrue(hasBeenCalled)
end

function TestGlobal:tearDown()
  _G.print = self.backupPrint
end

os.exit(lu.LuaUnit.run())

Mock required modules

Let's take the following scenario: a function to be tested has a dependency to a module added via LuaRocks. To properly test the function, the required module needs to be mocked.

To better visualize this, a function to retrieve an asset path from the user's system depends on the module lua-path.

-- src/examples/mock-module/assets.lua
local path = require("path")

local Assets = {}
function Assets:getPath()
  local userHome = path.user_home()
  return userHome .. path.DIR_SEP .. "assets"
end

return Assets

The function getPath needs to be tested, and the values path.user_home and path.DIR_SEP mocked, to isolate the test from the user environment.

As require is a global function, it can be mocked the same way as other global functions: by storing a reference inside LuaUnit's setUp and restoring it in tearDown.

-- src/examples/mock-module/assets-test.lua
function TestModule:setUp()
  self.actualRequire = _G.require
end

function TestModule:tearDown()
  _G.require = self.actualRequire
end

Inside setUp the global require can be overridden, listening for the dependency to be mocked, and returning a mocked value in place. The actualRequire is used to get any non-mocked module.

-- src/examples/mock-module/assets-test.lua
function TestModule:setUp()
  self.actualRequire = _G.require

  -- Override global require
  _G.require = function (modname)
    if modname == "path" then
      -- Return mocked version of "path".
      return {
        user_home = function ()
          return ""
        end,
        DIR_SEP = "/"
      }
    else
      -- For anything else, return actual.
      return self.actualRequire(modname)
    end
  end
end

The to-be-tested module will be required inside the test unit.

-- src/examples/mock-module/assets-test.lua

-- Other test code ...

function TestAssets:test1()
  local assets = require(
    "examples/mock-module/assets")
  lu.assertEquals(assets.getPath(), "/assets")
end

Requiring the code for testing inside the actual test needs to be done to ensure the require function is mocked first in setUp.

Here is the full example how to mock require.

local lu = require("luaunit")

local TestAssets = {}

function TestAssets:setUp()
  self.actualRequire = _G.require

  _G.require = function (modname)
    if modname == "path" then
      return {
        user_home = function ()
          return ""
        end,
        DIR_SEP = "/"
      }
    else
      return self.actualRequire(modname)
    end
  end
end

function TestAssets:test1()
  local assets = require(
    "examples/mock-module/assets")
  lu.assertEquals(assets.getPath(), "/assets")
end

function TestAssets:tearDown()
  _G.require = self.actualRequire
end

return TestAssets

Continuous integration

Tests should be run often, and adding continuous integration will help spot issues early. With a repository on GitHub, GitHub Actions is a convenient way to enable this.

I will use two amazing actions created by Leafo, leafo/gh-actions-lua and leafo/gh-actions-luarocks.

Creating a new workflow called test, running on every push.

# .github/workflows/test.yml
name: test
on: [push]

Defining the test job using the GitHub actions.

# .github/workflows/test.yml

# Other settings ...

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: leafo/gh-actions-lua@v10
      - uses: leafo/gh-actions-luarocks@v4

Finally, installing all dependencies with LuaRocks and running the test suite. To run all tests, the environment variable LUA_INIT needs to be set with the path to the setup file, letting Lua know where to find modules installed with LuaRocks.

# .github/workflows/test.yml

# Other settings ...

    - name: install
        run: |
          luarocks install \
            --deps-only lua-series-1.1.0-1.rockspec
      - name: test
        run: luarocks test
        env:
          LUA_INIT: "@src/setup.lua"

The whole workflow file, enabling continuous execution of the test suite, looks as follows.

# .github/workflows/test.yml
name: test
on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: leafo/gh-actions-lua@v10
      - uses: leafo/gh-actions-luarocks@v4
      - name: install
        run: |
          luarocks install \
            --deps-only lua-series-1.1.0-1.rockspec
      - name: test
        run: luarocks test
        env:
          LUA_INIT: "@src/setup.lua"

What comes next

As said in the beginning, all the code shown in this article is also available in the companion repository on GitHub, branch part-3.

The next article will focus on debugging and profiling Lua code — really digging into fixing bugs, detecting performance bottlenecks, and even writing a small profiling tool.

Until then 👋🏻