ReferenceTests.jl is a Julia package that adds a couple of
additional macros to your testing toolbox. In particular, it
focuses on functionality for testing values against reference
files, which in turn the package can help create and update if
need be. ReferenceTests.jl is build on top of
FileIO.jl and designed
to be used alongside Base.Test.
| Package Status | Package Evaluator | Build Status |
|---|---|---|
  |
 |
  |
It is very common for Julia packages to test the functionality of
their exported functions against known input-to-output
combinations. We will refer to such kind of tests as reference
tests. In most cases these will be quite simple; something along
the line of @test f(x) == y, where f is a function of the
user package and x is some interesting input value for which
the desired output y is known.
For testing the output of more complex functions, for which the
expected output is more complicated (e.g. anything image
processing related), using @test can be a little cumbersome to
work with. To that end this package provides the
@test_reference macro, which expects a filename (relative to
the file that invokes the macro) and an expression that evalutes
to the value of interest.
using ReferenceTests
@test_reference "stringtest1.txt" collect(1:20)If you put the above code into your test/runtests.jl and
execute the file in an interactive julia session (i.e. with
include), then it will trigger an interactive dialog if the
results don't match. This dialog allows the user to update the
reference files. If you do not want to be prompted, just
delete the reference data before running the tests.
The given file stringtest1.txt is assumed to be the relative
path to the file that contains the macro invocation. This likely
means that the path is relative to the test/ folder of your
package.
The file-extension of the filename (here txt), as well as the
type of the result of evaluating the expression (here String),
determine how the actual value is compared to the reference
value. The default implementation will do a simple equality check
with the result of FileIO.load. This means that it is the
user's responsibility to have the required IO package installed.
Colorant arrays (i.e.) receive special treatment. If the
extension of the filename is txt then the package
ImageInTerminal.jl
will be used to create a string-based crude approximation of the
image. This will have low storage requirements and also allows to
view the reference file in a simple terminal using cat.
using ReferenceTests, TestImages
@test_reference "imagetest1.txt" testimage("cameraman")
Note that while a text-based storage of reference images can be
convenient, proper image formats (e.g. png) are also supported
by the package. Those, however, will require the proper FileIO
backends to be installed.
Another special file extension is sha256 which will cause the
hash of the result of the given expression to be stored and
compared as plain text. This is useful for a convenient
low-storage way of making sure that the return value doesn't
change for selected test cases.
Reference tests are typically used within a package's test/runtests.jl
test suite. These tests are easy to run via pkg> test but
the child process used within pkg> test is non-interactive, so the
update prompt will not show if there are mismatches.
To update references within a package test suite, there are three options:
- Set the environment variable
JULIA_REFERENCETESTS_UPDATEto"true"and runpkg> test, which will force update any non-matches. You can then check changes to any git-tracked reference images before commit. - Delete any reference images you wish to update and run
pkg> test, given that missing references are created automatically. - Run the
test/runtests.jlinteractively. This may be easier using theTestEnv.jlpackage, given that the test environment used bypkg> testis a merge of thesrc/Project.tomlandtest/Project.tomlenvironments.
Check out the stable documentation or dev documentation.
Additionally, you can make use of Julia's native docsystem.
The following example shows how to get additional information
on @test_reference within Julia's REPL:
?@test_referenceThis code is free to use under the terms of the MIT license.