The directory gccsrcdir/libstdc++-v3/testsuite contains the individual test cases organized in sub-directories corresponding to clauses of the C++ standard (detailed below), the DejaGnu test harness support files, and sources to various testsuite utilities that are packaged in a separate testing library.

All test cases for functionality required by the runtime components of the C++ standard (ISO 14882) are files within the following directories:

    17_intro
    18_support
    19_diagnostics
    20_util
    21_strings
    22_locale
    23_containers
    24_iterators
    25_algorithms
    26_numerics
    27_io
    28_regex
    29_atomics
    30_threads
   

In addition, the following directories include test files:

Some directories don't have test files, but instead contain auxiliary information:

Within a directory that includes test files, there may be additional subdirectories, or files. Originally, test cases were appended to one file that represented a particular section of the chapter under test, and was named accordingly. For instance, to test items related to 21.3.6.1 - basic_string::find [lib.string::find] in the standard, the following was used:

    21_strings/find.cc 

However, that practice soon became a liability as the test cases became huge and unwieldy, and testing new or extended functionality (like wide characters or named locales) became frustrating, leading to aggressive pruning of test cases on some platforms that covered up implementation errors. Now, the test suite has a policy of one file, one test case, which solves the above issues and gives finer grained results and more manageable error debugging. As an example, the test case quoted above becomes:

    21_strings/basic_string/find/char/1.cc
    21_strings/basic_string/find/char/2.cc
    21_strings/basic_string/find/char/3.cc
    21_strings/basic_string/find/wchar_t/1.cc
    21_strings/basic_string/find/wchar_t/2.cc
    21_strings/basic_string/find/wchar_t/3.cc

All new tests should be written with the policy of "one test case, one file" in mind.

In addition, there are some special names and suffixes that are used within the testsuite to designate particular kinds of tests.

You can check the status of the build without installing it using the DejaGnu harness, much like the rest of the gcc tools, i.e. make check in the libbuilddir directory, or make check-target-libstdc++-v3 in the gccbuilddir directory.

These commands are functionally equivalent and will create a 'testsuite' directory underneath libbuilddir containing the results of the tests. Two results files will be generated: libstdc++.sum, which is a PASS/FAIL summary for each test, and libstdc++.log which is a log of the exact command-line passed to the compiler, the compiler output, and the executable output (if any) for each test.

Archives of test results for various versions and platforms are available on the GCC website in the build status section of each individual release, and are also archived on a daily basis on the gcc-testresults mailing list. Please check either of these places for a similar combination of source version, operating system, and host CPU.

There are several options for running tests, including testing the regression tests, testing a subset of the regression tests, testing the performance tests, testing just compilation, testing installed tools, etc. In addition, there is a special rule for checking the exported symbols of the shared library.

To debug the DejaGnu test harness during runs, try invoking with a specific argument to the variable RUNTESTFLAGS, like so:

    make check-target-libstdc++-v3 RUNTESTFLAGS="-v"

or

    make check-target-libstdc++-v3 RUNTESTFLAGS="-v -v"

To run a subset of the library tests, you can either generate the testsuite_files file (described below) by running make testsuite_files in the libbuilddir/testsuite directory, then edit the file to remove the tests you don't want and then run the testsuite as normal, or you can specify a testsuite and a subset of tests in the RUNTESTFLAGS variable.

For example, to run only the tests for containers you could use:

    make check-target-libstdc++-v3 RUNTESTFLAGS="conformance.exp=23_containers/*"

When combining this with other options in RUNTESTFLAGS the testsuite.exp=testfiles options must come first.

There are two ways to run on a simulator: set up DEJAGNU to point to a specially crafted site.exp, or pass down --target_board flags.

Example flags to pass down for various embedded builds are as follows:

  --target=powerpc-eabisim (libgloss/sim)
  make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=powerpc-sim"

  --target=calmrisc32 (libgloss/sid)
  make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=calmrisc32-sid"

  --target=xscale-elf (newlib/sim)
  make check-target-libstdc++-v3 RUNTESTFLAGS="--target_board=arm-sim"

Also, here is an example of how to run the libstdc++ testsuite for a multilibed build directory with different ABI settings:

    make check-target-libstdc++-v3 RUNTESTFLAGS='--target_board \"unix{-mabi=32,,-mabi=64}\"'

You can run the tests with a compiler and library that have already been installed. Make sure that the compiler (e.g., g++) is in your PATH. If you are using shared libraries, then you must also ensure that the directory containing the shared version of libstdc++ is in your LD_LIBRARY_PATH, or equivalent. If your GCC source tree is at /path/to/gcc, then you can run the tests as follows:

    runtest --tool libstdc++ --srcdir=/path/to/gcc/libstdc++-v3/testsuite

The testsuite will create a number of files in the directory in which you run this command,. Some of those files might use the same name as files created by other testsuites (like the ones for GCC and G++), so you should not try to run all the testsuites in parallel from the same directory.

In addition, there are some testing options that are mostly of interest to library maintainers and system integrators. As such, these tests may not work on all CPU and host combinations, and may need to be executed in the libbuilddir/testsuite directory. These options include, but are not necessarily limited to, the following:

We are interested in any strange failures of the testsuite; please email the main libstdc++ mailing list if you see something odd or have questions.

The tests will be compiled with a set of default compiler flags defined by the libbuilddir/scripts/testsuite_flags file, as well as options specified in individual tests. You can run the tests with different options by adding them to the output of the --cxxflags option of that script, or by setting the CXXFLAGS variable when running make, or via options for the DejaGnu test framework (described below). The latter approach uses the --target_board option that was shown earlier, but requires DejaGnu version 1.5.3 or newer to work reliably, so that the dg-options in the test aren't overridden. For example, to run the tests with -O1 -D_GLIBCXX_ASSERTIONS you could use:

    make RUNTESTFLAGS=--target_board=unix/-O1/-D_GLIBCXX_ASSERTIONS

The --target_board option can also be used to run the tests multiple times in different variations. For example, to run the entire testsuite three times using -O3 but with different -std options:

    make 'RUNTESTFLAGS=--target_board=unix/-O3\"{-std=gnu++98,-std=gnu++11,-std=gnu++14}\"'

N.B. that set of variations could also be written as unix/-O3\"{-std=gnu++98,-std=gnu++11,}\" so that the third variation would use the default for -std (which is -std=gnu++14 as of GCC 6).

To run the libstdc++ test suite under the debug mode, use make check-debug. Alternatively, edit libbuilddir/scripts/testsuite_flags to add the compile-time flag -D_GLIBCXX_DEBUG to the result printed by the --cxxflags option. Additionally, add the -D_GLIBCXX_DEBUG_PEDANTIC flag to turn on pedantic checking. The libstdc++ test suite should produce the same results under debug mode that it does under release mode: any deviation indicates an error in either the library or the test suite. Note, however, that the number of tests that PASS may change, because some test cases are skipped in normal mode, and some are skipped in debug mode, as determined by the dg-require-support directives described below.

The parallel mode can be tested using make check-parallel, or in much the same manner as the debug mode, substituting -D_GLIBCXX_PARALLEL for -D_GLIBCXX_DEBUG in the previous paragraph.

Or, just run the testsuite -D_GLIBCXX_DEBUG or -D_GLIBCXX_PARALLEL in CXXFLAGS or RUNTESTFLAGS.

Example 1: Testing compilation only:

// { dg-do compile }

Example 2: Testing for expected warnings on line 36, which all targets fail:

// { dg-warning "string literals" "" { xfail *-*-* } 36 }

Example 3: Testing for expected warnings on line 36:

// { dg-warning "string literals" "" { target *-*-* } 36 }

Example 4: Testing for compilation errors on line 41:

// { dg-do compile }
// { dg-error "no match for" "" { target *-*-* } 41 }

Example 5: Testing with special command line settings, or without the use of pre-compiled headers, in particular the stdc++.h.gch file. Any options here will override the DEFAULT_CXXFLAGS and PCH_CXXFLAGS set up in the normal.exp file:

// { dg-options "-O0" { target *-*-* } }

Example 6: Compiling and linking a test only for C++14 and later, and only if Debug Mode is active:

// { dg-do link { target c++14 } }
// { dg-require-debug-mode "" }

Example 7: Running a test only on x86 targets, and only for C++11 and later, with specific options, and additional options for 32-bit x86:

// { dg-options "-fstrict-enums" }
// { dg-additional-options "-march=i486" { target ia32 } }
// { dg-do run { target { ia32 || x86_64-*-* } } }
// { dg-require-effective-target "c++11" }

More examples can be found in the libstdc++-v3/testsuite/*/*.cc files.

In addition to the usual Variants of dg-require-support several more directives are available for use in libstdc++ tests, including the following:

Underlying details of testing for conformance and regressions are abstracted via the GNU DejaGnu package. This is similar to the rest of GCC.

This is information for those looking at making changes to the testsuite structure, and/or needing to trace DejaGnu's actions with --verbose. This will not be useful to people who are "merely" adding new tests to the existing structure.

The first key point when working with DejaGnu is the idea of a "tool". Files, directories, and functions are all implicitly used when they are named after the tool in use. Here, the tool will always be "libstdc++".

The lib subdir contains support routines. The lib/libstdc++.exp file ("support library") is loaded automagically, and must explicitly load the others. For example, files can be copied from the core compiler's support directory into lib.

Some routines in lib/libstdc++.exp are callbacks, some are our own. Callbacks must be prefixed with the name of the tool. To easily distinguish the others, by convention our own routines are named "v3-*".

The next key point when working with DejaGnu is "test files". Any directory whose name starts with the tool name will be searched for test files. (We have only one.) In those directories, any .exp file is considered a test file, and will be run in turn. Our main test file is called normal.exp; it runs all the tests in testsuite_files using the callbacks loaded from the support library.

The config directory is searched for any particular "target board" information unique to this library. This is currently unused and sets only default variables.

The testsuite directory also contains some files that implement functionality that is intended to make writing test cases easier, or to avoid duplication, or to provide error checking in a way that is consistent across platforms and test harnesses. A stand-alone executable, called abi_check, and a static library called libtestc++ are constructed. Both of these items are not installed, and only used during testing.

These files include the following functionality: