New ISO 34504 provides guidance for safety assessment of Automated Driving Systems

Scenario-based testing for ADS

For the safety assessment of Automated Driving Systems (ADSs), the automotive industry widely endorses scenario-based testing. It is anticipated that a wide array of scenario formats will be employed to conduct the validation and verification process of ADSs, alongside various forms of categorisation.

Henk Goossens, Innovation Manager for Safety Assessment of Automated Driving at TNO, remarks, ‘One of the greatest challenges in (virtual) safety assessment is organising the vast array of scenarios. Instead of striving for consensus among all stakeholders on a unified set of scenarios, it is much more efficient to establish agreement on the underlying categorisation structure.’

Achieving a comprehensive standard

The objective of the new ISO 34504 standard is precisely to achieve this. ‘Our goal is to establish alignment in categorising test scenarios for automated vehicles and driving systems,’ explains TNO scientist Erwin de Gelder, who has chaired the development of this standard since its inception in 2018. To create a comprehensive standard, TNO has collaborated closely with all stakeholders: OEMs, testing facilitators, research and regulatory authorities, representing a diverse range of nations including the EU, US, and Japan.

Erwin de Gelder elaborates, ‘We have made extensive efforts to engage all these stakeholders in the standard's design process, incorporating their specific requirements. This includes considerations for varying road conditions and situations, as well as existing structures and definitions for scenario description. Additionally, we aimed to ensure that the new standard aligns with the broader family of 3450X ISO standards.’

Robust safety argumentation

The introduction of the new ISO standard is poised to bolster the development of a robust safety argumentation framework for ADSs grounded in safety evidence.

Erwin de Gelder explains, ‘Ultimately, ADS developers must demonstrate that they have considered every conceivable scenario, a feat that's mathematically unattainable given the limitless variety of scenarios. However, through a scenario-category approach, safety assessment can be conducted by ensuring representation across all scenario categories, with an ample number of scenarios per category to encompass the entire spectrum of driving scenarios.’

An important question that requires additional effort is the definition of the right set of scenario categories. Henk Goossens: ‘Getting this right is a trade-off: on the one hand, the number of scenario categories must be limited in order to keep the safety assessment manageable. On the other hand, it is important that the scenario categories cover the complete Operational Design Domain (ODD) with all the details that are relevant for the ADS’s performance.’

Scientific categorisation framework

TNO has developed a categorisation methodology rooted in scientific principles, underpinned by solid technological rationale. Henk Goossens elaborates, ‘The effectiveness of the StreetWise methodology has been demonstrated through numerous projects and publications, providing valuable insights into the characteristics of scenarios and the requirements for safe vehicle behaviour. This validated methodology now forms the foundation of the new ISO standard.’

Drawing on this methodology, a foundational structure was established to categorise scenarios using tags. Erwin de Gelder elaborates, ‘These tags are structured in a format that can be used for digital modelling, mirroring the process we employ when extracting scenarios from real-world driving data. By discerning which tags impact vehicle behaviour and which do not, scenarios can be grouped into primary and subcategories.’

This approach provides a powerful tool for the definition of the right set of scenario categories for each specific AD system.

Establishing a universal framework

The objective is to comprehensively outline a scenario using tags. ‘Through the consistent application and structured organisation of these tags, we establish a shared vocabulary for scenario-based safety assessment for both AD developers and assessors,’ explains De Gelder.

‘Emphasising dynamic vehicle behaviour, insights from the StreetWise methodology have guided our approach. Moreover, input from the stakeholders in the standardisation group on vehicle types, driving environments, and various conditions such as light, weather, and road conditions has been instrumental in refining the tags. Despite multiple rounds of review, the core concept of the standard remains unchanged.’

While meticulously crafted and extensively reviewed, the standard retains some degree of flexibility. Erwin de Gelder elaborates, ‘For instance, when we mention 'A vehicle is moving faster or slower,' we refrain from defining the exact speed differential. We deliberately omitted such details from the standard, as their significance varies depending on the circumstances and context. It's essential to allow for flexibility in interpretation to enable effective and practical application.’

More transparency in safety assessment

How will the new ISO standard enhance the development process of ADSs? Erwin de Gelder explains, ‘Primarily, it will greatly enhance transparency in the safety assessment process for all stakeholders. Developers must clearly outline the scenarios their systems must navigate. This standard is really helpful to specify these scenarios. And for regulators it helps in eliminating all ambiguity from safety assessment.’

‘Moreover’, Henk Goossens adds, ‘within OEMs, various teams handle different safety assessment related tasks like data acquisition, simulations, data analytics, and safety argumentation. When everyone adheres to the same definitions and structure, it minimises inefficiencies and speeds up collaboration.’

Harmonising the implementation

Finally, what can the industry expect from the new ISO standard? Erwin de Gelder: ‘Although it’s still early days, authorities may reference the standard in upcoming regulations concerning scenario-based safety assessment of automated vehicles. Given the broad collaboration among major stakeholders, backed by industry and authorities, there's a strong likelihood that this standard will see widespread adoption for safety assessments of automated vehicles.’

Additionally, De Gelder has a clear vision for the future: ‘The standard provides examples rather than instructions on its application and tags. It would be beneficial to strive towards a universally accepted harmonised methodology for implementation as well.’