Methods for the generation of autonomous driving simulation scenarios and internal error regulation

The future has already arrived. (Partially) autonomous cars are already on our roads today with automated systems such as braking or lane departure warning systems. As a central vehicle component, the software of these systems must continuously and reliably meet high-quality criteria. Franz Wotawa from the Institute of Software Technology at TU Graz and his team in close collaboration with the cyber-physical system testing team of AVL are dedicated to the great challenges of this future technology: the guarantee of safety through the automatic generation of extensive test scenarios for simulations and system-internal error compensation by means of an adaptive control method.

Ontologies instead of test kilometers

Test drives alone do not provide sufficient evidence for the accident safety of autonomous driving systems, explains Franz Wotawa: "Autonomous vehicles would have to be driven around 200 million kilometers to prove their reliability—especially for accident scenarios. That is 10,000 times more test kilometers than are required for conventional cars." However, critical test scenarios with danger to life and limb cannot be reproduced in real test drives. Autonomous driving systems must therefore be tested for their safety in simulations.

"Although the tests so far cover many scenarios, the question always remains whether this is sufficient and whether all possible accident scenarios have been considered," says Wotawa. Mihai Nica from the AVL underlines this statement: "In order to test highly autonomous system, it is required to re-think how the automotive industry must validate and certify Advanced Driver Assistance Systems and Autonomous Driving systems. Therefore, AVL participates with TU Graz to develop a unique and highly efficient method and workflow based on simulation and test case generation for prove fulfillment of Safety Of The Intended Functionality (SOTIF), quality and system integrity requirements of the autonomous systems".

Together the project team is working on innovative methods with which far more test scenarios can be simulated than before. The researchers' approach is as follows: instead of driving millions of kilometers, they use ontologies to describe the environment of autonomous vehicles. Ontologies are knowledge bases for the exchange of relevant information within a machine system. For example, interfaces, behavior and relationships of individual system units can communicate with each other. In the case of autonomous driving systems, these would be "decision making," "traffic description," or "autopilot." The Graz researchers worked with basic detailed information about environments in driving scenarios and fed the knowledge bases with details about the construction of roads, intersections and the like, which AVL provided. From this, driving scenarios can be derived, by using AVL's world leading test case generation algorithm, that test the behavior of the automated driving systems in simulations.

Additional weaknesses uncovered

As part of the EU AutoDrive project, researchers have used two algorithms to convert these ontologies into input models for combinatorial testing that can subsequently be executed using simulation environments. "In initial experimental tests we have discovered serious weaknesses in automated driving functions. Without these automatically generated test scenarios, the vulnerabilities would not have been detected so quickly: nine out of 319 test cases investigated have led to accidents." For example, in one test scenario, a brake assistance system failed to detect two people coming from different directions at the same time and one of them was badly hit due to the initiated braking maneuver. "This means that with our method, you can find test scenarios that are difficult to test in reality and that you might not even be able to focus on," says Wotawa.

This work by Franz Wotawa et al was also presented in the journal Information and Software Technology at the beginning of 2020.

Adaptive compensation of internal errors

Autonomous systems and in particular autonomous driving systems must be able to correct themselves in the event of malfunctions or changed environmental conditions and reliably reach given target states at all times. "When we look at semi-automated systems already in use today, such as cruise control, it quickly becomes clear that in the case of errors, the driver can and will always intervene. With fully autonomous vehicles, this is no longer an option, so the system itself must be able to act accordingly," explains Franz Wotawa.

In a new publication for the Software Quality Journal, Franz Wotawa and his Ph.D. student Martin Zimmermann present a control method that can adaptively compensate for internal errors in the software system. The presented method selects alternative actions in such a way that predetermined target states can be achieved, while providing a certain degree of redundancy. Action selection is based on weighting models that are adjusted over time and measure the success rate of specific actions already performed. In addition to the method, the researchers also present a Java implementation and its validation using two case studies motivated by the requirements of the autonomous driving range.