Units test facility¶
|Maintainer:||Masatake YAMATO <firstname.lastname@example.org>|
Exuberant ctags has a test facility. The test case were Test directory. So Here I call it Test.
Main aim of the facility is detecting regression. All files under Test directory are given as input for old and new version of ctags commands. The output tags files of both versions are compared. If any difference is found the check fails. Test expects the older ctags binary to be correct.
This expectation is not always met. Consider that a parser for a new language is added. You may want to add a sample source code for that language to Test. An older ctags version is unable to generate a tags file for that sample code, but the newer ctags version does. At this point a difference is found and Test reports failure.
The units test facility(Units) I describe here takes a different approach. An input file and an expected output file are given by a contributor of a language parser. The units test facility runs ctags command with the input file and compares its output and the expected output file. The expected output doesn’t depend on ctags.
If a contributor sends a patch which may improve a language parser, and if a reviewer is not familiar with that language, s/he cannot evaluate it.
Unit test files, the pair of input file and expected output file may be able to explain the intent of patch well; and may help the reviewer.
How to write a test case¶
The test facility recognizes an input file and an expected output file by patterns of file name. Each test case should have its own directory under Units directory.
Input file name must have a input as basename. TEST part should explain the test case well.
Units/TEST/input-[0-9].* Units/TEST/input-[0-9]-.** optional
Optional input file names. They are put next to input.* in testing command line.
Expected output file must have a name expected.tags. It should be the same directory of the input file.
If this file is not given, the exit status of ctags process is just checked; the output is ignored.
If you want to test etags output (specified with
-e), Use .tags-e as suffix instead of .tags. In such a case you don’t have to write
args.ctags. The test facility sets
If you want to test cross reference output (specified with
-x), Use .tags-x as suffix instead of .tags. In such a case you don’t have to write
args.ctags. The test facility sets
If you want to test json output (specified with
--output-format=json), Use .tags-json as suffix instead of .tags. In such a case you don’t have to write
features. The test facility sets the option and the feature automatically.
-o -is used as default optional argument when running a unit test ctags. If you want to add more options, enumerate options in args.ctags file. This file is an optional.
Remember you have to put one option in one line; don’t put multiple options to one line. Multiple options in one line doesn’t work.
You can rearrange the output of ctags with this command before comparing with executed.tags. This command is invoked with no argument. The output ctags is given via stdin. Rearrange data should be written to stdout.
If a unit test case requires special features of ctags, enumerate them in this file line by line. If a target ctags doesn’t have one of the features, the test is skipped.
If a file line is started with
!, the effect is inverted; if a target ctags has the feature specified with
!, the test is skipped.
All features built-in can be listed with passing
If a unit test case requires that language parsers are enabled/available, enumerate them in this file line by line. If one of them is disabled/unavailable, the test is skipped.
language parsers enabled/available can be checked with passing
Used in spell checking. See cspell for more details.
Note for importing a test case from Test directory¶
I think all test cases under Test directory should be converted to Units.
If you convert use following TEST name convention.
- use .t instead of .d as suffix for the name
Here is an example:
This should be:
With this name convention we can track which test case is converted or not.
Example of files¶
See Units/c-sample/input.c and Units/c-sample/expected.
How to run unit tests¶
test make target:
$ make units
The result of unit tests is reported by lines. You can specify
test cases with
An example to run vim-command.d only:
$ make units UNITS=vim-command
Another example to run vim-command.d and parser-python.r/bug1856363.py.d:
$ make units UNITS=vim-command,bug1856363.py
During testing OUTPUT.tmp, EXPECTED.tmp and DIFF.tmp files are generated for each test case directory. These are removed when the unit test is passed. If the result is FAILED, it is kept for debugging. Following command line can clean up these generated files at once:
$ make clean-units
Other than FAILED and passed two types of result are defined.
means running the test case is skipped in some reason.
failed (KNOWN bug)
mean the result if failed but the failure is expected. See “Gathering test cases for known bugs”.
Example of running¶
Running unit tests for specific languages¶
You can run only the tests for specific languages by setting
LANGUAGES to parsers as reported by
make units LANGUAGES=PHP,C
Multiple languages can be selected using a comma separated list.
Gathering test cases for known bugs¶
When we met a bug, making a small test case that triggers the bug is important development activity. Even the bug cannot be fixed in soon, the test case is an important result of work. Such result should be merged to the source tree. However, we don’t love FAILED message, too. What we should do?
In such a case, merge as usually but use .b as suffix for the directory of test case instead of .d.
Unix/css-singlequote-in-comment-issue2.b is an example of .b suffix usage.
When you run test.units target, you will see:
Testing c-sample as C passed Testing css-singlequote-in-comment as CSS failed (KNOWN bug) Testing ctags-simple as ctags passed
Suffix .i is a variant of .b. .i is for merging/gathering input which lets ctags process enter an infinite loop. Different from .b, test cases marked as .i are never executed. They are just skipped but reported the skips:
Testing ada-ads as Ada passed Testing ada-function as Ada skipped (may cause an infinite loop) Testing ada-protected as Ada passed ... Summary (see CMDLINE.tmp to reproduce without test harness) ------------------------------------------------------------ #passed: 347 #FIXED: 0 #FAILED (unexpected-exit-status): 0 #FAILED (unexpected-output): 0 #skipped (features): 0 #skipped (languages): 0 #skipped (infinite-loop): 1 ada-protected ...
Running under valgrind and timeout¶
VG=1 is given, each test cases are run under valgrind.
If valgrind detects an error, it is reported as:
$ make units VG=1 Testing css-singlequote-in-comment as CSS failed (valgrind-error) ... Summary (see CMDLINE.tmp to reproduce without test harness) ------------------------------------------------------------ ... #valgrind-error: 1 css-singlequote-in-comment ...
In this case the report of valgrind is recorded to
/bin/bash is needed to report the result. You can specify a shell
running test with SHELL macro like:
$ make units VG=1 SHELL=/bin/bash
TIMEOUT=N is given, each test cases are run under timeout
command. If ctags doesn’t stop in
N second, it is stopped
by timeout command and reported as:
$ make units TIMEOUT=1 Testing css-singlequote-in-comment as CSS failed (TIMED OUT) ... Summary (see CMDLINE.tmp to reproduce without test harness) ------------------------------------------------------------ ... #TIMED-OUT: 1 css-singlequote-in-comment ...
TIMEOUT=N is given, .i test cases are run. They will be
reported as TIMED-OUT.
With .r suffix, you can put test cases under a sub directory
Units/parser-ada.r is an example. If misc/units
test harness, the sub directory is called a category.
is the name category in the above example.
CATEGORIES macro of make is for running units in specified categories.
Following command line is for running units in
$ make units CATEGORIES='parser-sh,parser-ada'
Finding minimal bad input¶
When a test case is failed, the input causing
FAILED result is
passed to misc/units shrink. misc/units shrink tries to make the
shortest input which makes ctags exits with non-zero status. The
result is reported to
useful to debug.