One-stop shop for any C and C++ compiler and library testing, validation and safety services.
SuperTest
SuperTest Compiler Test and Validation Suite for C and C++
SuperTest is the test and validation suite for C and C++ compilers and libraries that has tracked the (ISO) language specifications for more than 30 years. It is a complete validation environment with:
• A huge collection of hand-written test
• A large body of generated tests
• A configurable test driver to easily set up and run validations
• A report generator that provides comprehensive overviews of validation results in multiple formats, including HTML and JUnit XML, and will immediately identify regressions of the latest compiler and library version compared to a designated reference
• SuperTest comes with a bundle of ~24K hand-written files generating Millions lines of code to achieve all standard testing.
SuperTest includes many more unique features that are not found in other test suites:
• Very easy and quick installation and configuration
• Can be used to verify deeply embedded freestanding C implementations on memory constraint devices (down to 4K memory)
• Clear and easy reporting in text, HTML, JUnit XML and CSV
• The CGtrainer (Code Generator trainer), a suite of nearly one thousand tests aimed at bootstrapping new compiler development
• Powerful testing of the code generator/back-end
• The Depth Suite is a collection of generated tests that exhaustively verifies arithmetic correctness relative to the target machine’s data model. Depth Suites for many data models are included, but if your data model is not there, we will create one for you free of charge
• Parallel testing automatically splits validation runs into a preconfigured numbers of threads to speed up the validation
• Testing of libraries
• A production rule driven test generator to create pseudo-random variations of a particular test
• Easy addition of your own specific tests
• Excellent professional support
• Multi-processing to execute tests faster
• Supports bare metal targets
• Traceability to the sections of the ISO standards
• Runs on Windows and Linux
• Flexible test generators
• Full control over the test sets
• Python Configuration plugins
• Reporting tools: SuperTest generates high quality reports that are easy to read and analyze
ISO Standard Conformance Testing
SuperTest can be used to qualify compilers and libraries for functional safety based on standards like ISO 26262, IEC 61508, EN 50128, IEC 62304. Even if you are a compiler end-user and can’t fix the C/C++ compiler, SuperTest will help you know what C/C++ syntax to avoid so you don’t have a runtime issue.
The SuperTest suite covers the following ISO specifications:
ISO/IEC 9899:1990 C90 standard
ISO/IEC 9899:1999 C99 standard
ISO/IEC 9899:2011 C11 standard
ISO/IEC 9899:2018 C18 standard
ISO/IEC TR 18037:2008 Embedded C extension
ISO/IEC 14882:2003 C++03
ISO/IEC 14882:2011 C++11
ISO/IEC 14882:2014 C++14
ISO/IEC 14882:2017 C++17
ISO/IEC 14882:2020 C++20
The test suite is systematically structured so that it is clear what section of which standard is tested by each test. Therefore, it is also easy to configure which language version should be tested when executing the test suite.
Additional SuperTest Features
• Optimization correctness
• C and C++ Standard Library Testing
• Code Generator development
• Data Model specific arithmetic
• Calling convention testing
• In-depth arithmetic testing
SuperTest Can Qualify Any C or C++ Compiler
SuperTest can be configured to qualify any C or C++ compiler toolchain for any execution environment. We provide example configurations for many of the compiler toolchains and targets listed below. Using an example configuration, getting started with using SuperTest for the qualification of the compiler toolchain and target that you are using is easy. SuperTest targets open source based compilers such as GCC (all versions), LLVM (Clang, all versions), STM32CubeIDE, TI Code Composer, Renesas 22 Studio; and most popular proprietary C/C++ compilers in the market. Email info@emprog.com to check if you compiler has a ready-to-use SuperTest for it.
SuperTest Testimonials
“With SuperTest, we can now reliably tell our customers that our compiler verification covers the entire C and C++ language standard. SuperTest includes tests for the optimizations that we often develop for our customers in order to meet their performance or code size requirements. As a result, we can run tests that target every part of the compiler,” Customer’s testimony after trying to develop their own test suit based on the LLVM and GCC Torture test suites.
“Before, when customers asked us how do we really know our compilers work, we didn’t always have a very good answer. After using SuperTest, now we can say with a very high level of confidence that they do.” Customer’s testimony
SuperGuard
SuperGuard Library Safety Qualification Suite
The SuperGuard Library Safety Qualification Suite is a requirements-based test suite for the C and C++ standard libraries with full traceability between the requirements derived from the ISO C and C++ language definitions and the individual library tests. It is designed to support the qualification of implementations of the C and C++ standard libraries for safety-critical applications, both for third-party (COTS) and for self-developed or maintained library implementations.
SuperGuard qualification suite focuses on ensuring the safety of standard libraries for C and C++ and ensures that the standard library meets its requirements.
Requirements-Based Testing
SuperGuard provides a detailed breakdown of the ISO C and C++ library specifications into the requirements that must be met by an implementation of the library. These requirements are linked to test specifications that describe how a test verifies the requirements. In turn, the test specification is linked to tests in SuperGuard. In this way, a detailed path is created from the specification to the tests that is easy to comprehend and verify, so that it can be used to create confidence in the compliance of the C and C++ library implementations with their specifications.
High Structural Code Coverage
SuperGuard is built to achieve high structural code coverage of the target library implementation. In addition to requirements traceability, this provides a second path to demonstrate the completeness of the test suite.
Reporting Tool
The suite comes with a software tool that reports the requirements that are met, and which are not met, by C and C++ library implementations as verified by a run of the test suite for a specific use case or configuration.
SuperGuard includes many more unique features that are not found in other qualification suites:
• Compliance with the requirements of functional safety standards
• Very easy and quick installation and configuration
• Parallel testing automatically splits validation runs into a preconfigured number of threads to speed up the validation
• Full control over test sets
• Easy addition of your own specific tests
• Excellent professional support
SuperTest Code Generator Trainer (add-on)
Developing a C/C++ compiler for a new (or old) target processor is one of the greatest and most rewarding software engineering projects imaginable. It is also challenging and daunting. This is where the Code Generator Trainer comes in.
The assumption is that this compiler is not written from scratch. It will be using an existing compiler development platform, either open source or proprietary, that already has a front-end, optimizations and a more or less retargetable code generator. What has to be done is writing a code generator for the new target processor. The Code Generator Trainer suite is going to make this project run smooth, efficiently and with a lot of fun on top of that.
The Code Generator Trainer consists of nearly one thousand test files split over 25 carefully constructed levels aimed at boot-strapping new compiler development. Each level addresses a specific capability of the code generator. Level 0 is there to set up the minimal interface that is needed to run the tests. The next levels introduce simple integer arithmetic, conversions, pointers, addressing modes for structures, register allocation, control flow and calling conventions. Beyond that are levels dealing with characters and shorts, long long arithmetic and varargs.
Each level is constructed to build upon the groundwork of the previous levels. This way, the code generator development progresses one logical step at a time. It gets you a better compiler, faster.
By the time the compiler passes the final level, it is ready to take on SuperTest’s big guns: the conformance suites that validate the front-end, optimization correctness and more nitty gritty details of code generation. There will be more work to do: add more features, improve efficiency, etc. You can be sure that this work builds on the solid foundations laid out by the Code Generator Trainer.
Want to get your hands on the Code Generator Trainer and start developing your code generator now? Get the SuperTest demo by clicking the button below. You will find the first few levels included for free.
Functional Safety Overview
Functional safety is one of the most important aspects of many embedded systems today, especially in sectors such as automotive, railway, industrial automation and medical. SuperTest and SuperGuard can be used to qualify compilers and libraries for functional safety.
Compiler qualification is the process that can be used to develop the required confidence in the compiler. Compiler testing against the language specification is the proper method for this. SuperTest, with its traceability from the C and C++ language standards to the test, is the best test suite for compiler qualification.
Library qualification is even more critical because the library contains code that is linked into the application and loaded onto the target device. If a library component is defective, the functional safety of the entire application is therefore jeopardized. SuperGuard can be used to support qualification of standard library implementations for safety critical applications.
There are different functional safety standards for which you can use SuperTest and SuperGuard to qualify your compiler and library.
ISO 26262
ISO 26262 is the functional safety standard for electrical and electronic systems in series production passenger cars and other automotive categories. It requires qualification of all (electrical and electronic) components with respect to safety. The qualification process is described in the standard.
IEC 61508
IEC 61508 is the international umbrella standard for functional safety. This standard and other standards derived from it are used in all kinds of industries with requirements on reliability and safety, for example process industries, nuclear power plants and railway control systems.
EN 50128
EN 50128 is a European standard for safety related software in railway applications.
IEC 62304
IEC 62304 is the international standard which specifies life cycle requirements for the development of medical software and medical devices.