The Validation Suite Approach Approach to safety ...

The Validation Suite Approach to Safety Qualification of Tools Automotive Safety and Security 2008 Stuttgart, 19. Nov. 2008.

Automotive - Safety & Security 2008 19-20 November 2008, Stuttgart VASE 28.10.2008 Seite 1

BMW AG, Dr. Stefan-Alexander Schneider PMFS IT Consulting, Pierre R. Mai TÜV-Nord IFM, Dr. Tomislav Lovric

Automotive - Safety & Security 2008 19-20 November 2008, Stuttgart

The Validation Suite Approach

Tool Qualification Introduction

VASE 28.10.2008 Seite 2

Model-Based Software Development Advantages + Efficient Software Design and Development + Early Validation of Executable Specifications +… Challenges – Testability of large Models (needs to be assured) – Models too abstract for Target – complex translation steps - details not known – translation integrity level - not known – resulting code quality - not known a priori –…



Tool Qualification Introduction

VASE 28.10.2008 Seite 3

Safety-Related Model-Based Software Development: Due Diligence of Developer to assure: – Language is suitable for safety – Developed models are suitable for safety (diagnosis) – Translation steps are suitable for safety – Translation Results are suitable … Systematic validation of translator suitability needed (see 61508/26262…) - tool validation difficult for individual projects -> Do this reproducibly and thoroughly: Validation Suite Approach

VS EW



Agenda

VASE 28.10.2008 Seite 4

1. Tool Qualification Introduction 2. Validation Suite Approach to safety Qualification of Tools 3. Validation Suite Requirements 4. Assessment steps of the Validation Suite 5. Experience with validation suite and tool validation 6. Outlook



Approach to safety Qualification of Tools Basic Idea for Validation Suite EWt: Transformation of a model m by development tool EW to a binary Bt Sm: Set of stimuli for a model m ∈ M (= suitable models (i.e. EW Inputs)) Em: Set of possible results em for evaluations of model m evalx :evaluation function for an execution environment x

To be shown by VS for all Models and Stimuli: distance d(em, em’) = 0

(∀ em ∈ Em; em’ ∈ Em’)

• Function d defines “Correctness” (here: Bit-Equality of all results). • “Teststrategy” defines required models and stimuli for the VS



Approach to safety Qualification of Tools

VASE 28.10.2008 Seite 6

Central to BMW for VS Strategy: 1. Automated testing and result evaluation of many test • database with various models and stimuli • powerful automated test environment • efficient evaluation of result correctness • validity of validation results (incl. “golden results”) 2. State-Of-The-Art validation strategy: • evaluation of safety standards requirements • independent derivation: requirements for tools • independent derivation: requirements for VS • include tool requirements into VS • implement VS requirements



Approach to safety Qualification of Tools Basic structure of VS-TL-0001: State of the art requirements for VS

TVS-Input: EW-TL-xxxx

TVS User Manual (with Checklists) TVS Maintenance Manual

TVS

Review-Part Review Manual Test-Part Test cases (Models, Stimuli, Configurations) Test scripts Test Environment

TVS-Operating Environment HW (Evalboard, PC), OS, Tools

TVS-Out: Qualification Report



VASE 28.10.2008 Seite 8

Define TVS-Input early in Project

Approach to safety Qualification of Tools Fix basic configuration of tool to be validated: 1. 2. 3. 4.

Operating Environment, Code generator incl. relevant Hooks, Compiler with Source files (if req.) Development guidelines

e.g. Windows, JAVA, MATLAB, TargetLink, Windriver, and Modelling Guidelines Document “EW-Def” (Tool Definition)



Approach to safety Qualification of Tools Basic structure of VS-TL-0001: State of the art requirements for VS

TVS-Input: EW-TL-xxxx

TVS User Manual (with Checklists) TVS Maintenance Manual

TVS

Review-Part Review Manual Test-Part Test cases (Models, Stimuli, Configurations) Test scripts Test Environment

TVS-Operating Environment HW (Evalboard, PC), OS, Tools

TVS-Out: Qualification Report



Approach to safety Qualification of Tools Test Environment:

– thorough testing and analysis, certified in advance

Referenz Interpreter Model +

References

Simulator

MIL results C-Code Generator

?

Stimu li PC

C code +

SIL results

Target

PIL results ...0100110

001

101010010001001001001001001111100...

Result Analysis:

Bit-by-Bit comparison



Approach to safety Qualification of Tools

Test Environment:

VASE 28.10.2008 Seite 11

– Derivation of EW specific characteristics EW

Adaptions of TVS and/or RefInt

test generation (manual, automated, …)

Loop…

...until all NOKs are assigned as Adaption of TVS, RefInt or EW

TVS-Developer Assignemet

EW Developer Independent Analysis

VASE Specifics EW

Results NOK PIL / RefInt deviate

ERR TVS-Developer Analysis

Qualification Report

Specifiied tests TVS-Test environment

Abbreviations

TVS … TargetLink-VS NOK … not OK ERR … Abort during Codegeneration

TestResults

OK

Most critical scenario: Erroneous code generated, despite: 1. 2. 3. 4.

Conformance to Modelling-Guidelines Confirmation by ES-Model-checkers Model preparation via TL-Tool chain Consideration of all notes from Log-file

EW



Agenda

VASE 28.10.2008 Seite 12



The Validation Suite Approach.

Validation Suite Requirements

VASE 28.10.2008 Seite 13

Development Tools are complex, • potentially error prone • without evidence of integrity • without „proven history“ • they need validation prior use in safety related projects (e.g. ISO CD 26262 ) • No specifix test criteria exist in standards Task: Derive requirements • top-down-filtering process • derive and refinement existing requirements: • currently: “VS-Anforderungen V2.0” Result: 113 categorized requirements • functional • non-functional 39

IEC 61508

7

ISO 26262 51

BMW Grob PTB EXP

12

4



Validation Suite Requirements.

VASE 28.10.2008 Seite 14

VS-Requirements V2.0 VS-Requirements

Derivation & Use Guideline

General structure of requirements: • •

… for VS for VS Main Test (= main part for test based VS) … for Pre Test of EW (= Annex; general suitability criteria for language and tool)



Validation Suite Requirements

VASE 28.10.2008 Seite 15

Structure of VS Requirements, with requirements on … 1. …Functionality of VS, e.g. test execution, logging of Configuration data 2. …Documentation and Report generation e.g. Generation of a tool Qualification Report 3. …Structure and Constituents of VS e.g. test database; different test categories with systematic tests test of language 4. …Validation of correct transformation of EW-Input e.g. for modular EW-Input 5. …Validation of intermediate code representations e.g. readability 6. …other Functional checks of the tool e.g. integration of other modules into the code 7. … Verification and Validation of VS itself e.g. structured and systematically planned V&V activities 8. … Qualität of VS itself e.g. Config. management; planning and execution of QA measures Structure of Requirements for pre test of tools: 1. Support on and facilitation of suitable tool-Input by tool 2. Basic properties of intermediate code representations (C) and tool-Output 3. Other functional properties of the tool 4. Tool quality



Agenda

VASE 28.10.2008 Seite 16




Assessment steps of the VS

VASE 28.10.2008 Seite 17

Structured stepwise procedure for TVS-Assessment Test-Phase

Test object

Test aim

1. Pre-Test (general suitability of tool)

EW Definition EW + Documentation EW-Developer (Audits)

e.g. tool allows suitable sw development (e.g. modularity)?

2. Test of VS Test-Engine (TAU) (suitability of VS Testenvironment)

TAU + Documentation TAU-Developer (Audits)

e.g. TAU logs all relevant data?

3. Test of VS Testcases (suitability of VS test case planning)

VS-Test strategy VS-Test plan VS-Test specification

Test categories for each language feature? …; for modular progr. building blocks?

4. Main VS Test (suitability of VS as a whole)

VS-Documentation VS-Development, V&V VS-Developer Audits

Test cases generation and executions for language features; VS-V&V plan / Rep. …



Assessment steps of the VS

VASE 28.10.2008 Seite 18

Derivation of compliance level for each defined VS requirement by TN Overall results of assessment activities: VS-TL-0001 complies satisfactory to the VS Requirements •

The VS requirements • could be traced into the BMW TVS design • were refined for the TargetLink VS • Have been successfully implemented to a large extend

•

The TVS has been successfully applied to a TargetLink / WindRiver based tool chain EW-TL-0001 Several potential issues have been identified and suitably handled Nevertheless, overall the results have confirmed the suitability of the tool chain and maturity of the contained tools for intended use

• •



Agenda

VASE 28.10.2008 Seite 26




Summary / Outlook

VASE 28.10.2008 Seite 27

Especially for Model Based development •

Tools have many useful functionalities … allow various design improvements

•

Tools are complex (>>100.000 LOC)

•

Tools are combined differently (use of different tool chains, tool usage guidelines)

•

Tools “take over” design tasks previously performed “manually”

•

Tools may translate “similar looking” blocks differently

Tools need systematic validation if used for safety Many parts of Validation may be automated by VS Approach



Summary / Outlook

VASE 28.10.2008 Seite 28

Use of a Validation Suite is recommended in •

IEC 61508-3, 2nd Edition, note 3 on 7.4.4.11

•

ISO WD 26262-8, 10.4.5 with tables 10.1 und 10.2.

The documents “VS-Anforderungen” 1. Herleitung und Leitfaden zur Anwendung 2. Anforderungen an eine Validierungssuite für Entwicklungs-werkzeuge •

are systematically derived from existing and upcoming standards (ISO 26262)

•

have been already used in practice and improved

•

close a normative gap for development tool validation suites

•

the requirements define and summarise current state-of-the-art

•

can be used by anyone to improve

•

will be further maintained, translated, published and used



Summary / Outlook

VASE 28.10.2008 Seite 29

BMW •

has successfully used the TVS • for qualification of TargetLink based Toolchains • for safety related series developments projects

•

plans use of VS for further general qualification of tools • for further QA improvement reasons • for other tool chains (other targets, other tools)

•

has gained positive experience from TVS use by • revealing potential difficulties otherwise not seen early • understanding potential and limits of tool chains in detail • improving guidelines for modelling and coding • improving quality of software and tools • Improving awareness by developers of potential tool limits

Is VS Approach applicable to your tool based project?

Improve it !

Mobilität sicher genießen



Finish: THANK YOU !

VASE 28.10.2008 Seite 30

Thanks to all supporting parties, especially • VALIDAS • • • Audi Electronic Venture Thank you very much for your attention Contact Dr. Stefan-Alexander Schneider, BMW AG [email protected] +49 (0) 89 382 - 54566 Dr. Tomislav Lovric, TÜV NORD [email protected] +49 (0) 201 825 - 4101 Herr Pierre Mai, PMSF IT Consulting [email protected] +49 (0) 8332 93669 – 13