run_tests

run_tests is a small Python script that can be found in every MOOSE-based application and in the framework itself (under moose/test). The run_tests script will find tests and run them with your compiled binary for your app. There is also a run_tests script in the moose/scripts which you may consider adding to your system PATH to conveniently run tests in any directory containing test specification files.

Basic Usage

To run the tests for any application, make sure the executable is built (generally by running make) and then do:

./run_tests -j 8

There are many options for run_tests, but the -j option shown above is by far the most widely used. It tells the run_tests script how many processors to utilize on your local machine for running tests concurrently. Put the correct number there (instead of 8). The script will then go find and run your tests and display the results. More detail on concurrency can be found below.

More Options

To see more options for run_tests you can invoke it with -h. There are many options to look through, but some of the important ones are:

• --failed-tests: Runs the tests that just failed. Note: as long as you keep using this option the set of failed tests will not change.

• --n-threads <n>: Causes the tests to run with # of (OpenMP/Pthread/TBB) threads.

• -p <n>: Causes the tests to run with # MPI processes. Useful for guaranteeing that you get the same result in parallel!

• --valgrind: Run all of the tests with the Valgrind utility. Does extensive memory checking to catch segfaults and uninitialized memory errors.

• --recover: Run all of the tests in "recovery" mode. This runs the test halfway, stops, then attempts to "recover" it and run until the end. This is very rigorous testing that tests whether everything in your application can work with restart/recover.

• --opt (The default) Builds an optimized executable suitable for running calculations.

• --dbg An executable meant for use in a debugger (like gdb or lldb). Will be very slow and not meant to be used on large problems

• --oprof Not normally used. Can be used for "code profiling": running your code with a utility like oprof, gprof, vtune, etc.

• --pro An alias to --oprof

• --devel Something in-between opt and dbg. Attempts to be faster but also does some more run-time checks (asserts and mooseAsserts)

testroot

The testroot file is a small configuration file that is formatted like a MOOSE input file. It is read by the run_tests script. It should be placed in the root of your application directory (i.e. right next to where the binary is). Some things you can set in that file are:

• app_name: A unique, short name for your application.

• allow_warnings: true by default, set this to false to make all warnings from running tests be _errors_ instead.

• allow_override and allow_unused: true by default if set to false then syntax errors in your test input files will be treated as errors.

The one thing we do not recommend is enforcing that the use of deprecated code should be treated like an error. That is entirely too rigid of a requirement and impedes the normal flow of development. Instead, developers should periodically run their tests with --error-deprecated to see if any of their tests are using deprecated code / parameters and then fix them up at that point. The MOOSE team is not responsible for fixing deprecated code.

Test Specifications

Tests are controlled by creating "test specification" files. By default, the TestHarness searches a directory tree for files named "tests" (no extension). These files use the standard HIT syntax that MOOSE uses. An example is given below:

[Tests]
  [test]
    type = 'Exodiff'
    input = 'simple_diffusion.i'
    exodiff = 'simple_diffusion_out.e'

    issues = '#1493'
    design = 'kernels/Diffusion.md'
    requirement = 'The system shall run a simple 2D linear diffusion problem with Dirichlet boundary conditions on a regular mesh.'

    # Enables running the limited HPC tests on CIVET on all events
    group = 'hpc'
  []
[]

Test Evaluation and Custom Evaluators

MOOSE has various configurations for evaluating whether a created test is successful or not. Depending on what type of Tester you specify in the "test specification" file, these can range from comparing the output of JSON, XML, CSV, or Exodus files, or matching a pattern in the output of the test using a regular expression.

Sometimes, it may not be possible to properly evaluate a test with the built-in checks MOOSE provides. In this situation, MOOSE has functionality for evaluating a test using a custom, user-supplied evaluation function.

To do this, create a Python script in the folder containing your test. The script should be blank aside from a single function named custom_evaluation(output):

def custom_evaluation(output):
  #Do your evaluation logic here
  if output == "foo":
    return True
  return False

output is a string containing the test's entire output and can be manipulated as one sees fit. The function should return a boolean representing whether the test passed or not.

After creating the .py file, add custom_evaluation_script = '[filename].py' to your test specification file. For example:

[Tests]
  [my_test]
    type = RunApp
    input = my_input.i
    custom_evaluation_script = 'my_custom_eval.py'
  []
[]

Parallel Test Execution

The TestHarness is designed to schedule several tests (jobs) concurrently. The -j <n> option to the run_tests script determines the numbers of available "slots" that may be used for testing. For maximum utilization of a given machine, this number should be set to the number of cores available on the system. The TestHarness will not oversubscribe a machine. When running with different combinations of parallel or threaded testing, each job will consume multiple slots (e.g. A test running with 2 MPI processes and 2 threads would consume 4 jobs slots).

To reduce the chance of tests in the same directory from writing to one or more of the same output or checkpoint files concurrently, the TestHarness normally only schedules a single job from a single directory (test specification) at a time. Concurrency is achieved by scheduling several tests from different directories at the same time. It is possible however to tell the TestHarness that tests from the same directory should be scheduled at the same time by using the parallel_scheduling syntax (shown below). Care should be taken when using this parameter to avoid race conditions with file output. Extensive use of this parameter is discouraged. It should be used in places where splitting up tests into separate folders is cumbersome, or splits of several related tests.

[Tests]
  parallel_scheduling = True

  [test1]
     ...
  []
  [test2]
     ...
  []
  [test3]
     ...
  []
[]