Critical I&C software quality assessment: verification technologies diversification. … 1 Critical I&C software quality assessment: diversification of software verification technologies on the basis of numerical and semantic invariants. Authors: Boris Konorev, Juriy Alekseev, Juriy Manzhos, Vladimir Sergienko, Vyacheslav Kharchenko, Georgiy Chertkov I&C Certification Center State center of supplies and services quality State committee of nuclear regulation of Ukraine. Ukraine 61070, Kharkov, p/o 9871, Akademika Prockury str, 1. Tel/fax: (057) , Director Georgiy Chertkov Kharkov, 2005

Critical I&C software quality assessment: verification technologies diversification. … 2 Problems: Profiling of regulatory normative software requirements in frameworks of critical I&C life cycle. Verification technologies diversification: I&C software quality assessment models and methods for a composition of the diverse verification technologies. Concept, technical requirements and development of static analysis utilities as the bases of tool equipment of test laboratories. Realization of source software instrumentation procedures, measurements of attributes (invariants), calibrations of software defects of various types sensitivity and degrees of verification methods variety. Purpose: improvement of quality of I&C software conformity expertise to regulatory requirements on reliability, completeness, cost parameters on a basis of verification technologies diversification; reduction of emergencies risks related to critical I&C software residual defects; creation of normative-methodical and tool equipment for I&C development organization and test laboratories which carry out critical I&C software expertise and independent verification. Scope Software independent verification, which carried out by software development organization on different phases of critical I&C software life cycle; Certifications and licensing critical I&C software in the framework of safety, supply and services quality State regulation Urgency: I&C systems are the key factor of NPP safety provision in long-term programs of atomic engineering development; I&C safe application directly depends on software quality; software residual defects are risk factors of I&C aberrant behavior and emergencies appearance; the tendency is the growth of amount of software based NPP critical I&C functions. Normative-methodical and tool equipment of I&C software independent verification and expertise processes strictly define real opportunities of provision of necessary safety level and I&C quality as a whole. Existing practice of software quality assessment: a significant degree of the subjectivity, insufficient completeness and reliability, high labour-intensiveness of expert assessment of I&C software conformity to regulatory requirements. The basic direction of I&C software quality assessment reliability increasing is verification technologies diversification. Critical I&C software quality assessment: verification technologies diversification.

Critical I&C software quality assessment: verification technologies diversification. … 3 SW Normative regulation and licensing bodies Area of application technical documen- tation Normative-methodical provision Tools - software expertise support utilities SW Licences, Certificates, Permissions Organization, which carries out software expertise and independent verification ? SW conformity assessment Expert group Expert's report. Software quality indicator value The general circuit of risk-informed critical software licensing: an assessment of software conformity to certification and expertise requirements taking into account software residual defects risks. PROBLEM: Regulation object: software-based functions in critical systems with intensive software use. Normative regulation I&C software quality and safety.

Critical I&C software quality assessment: verification technologies diversification. … 4

5 … Stand-alone tests Code Architectural design Integration tests Software requirements System tests Acceptance test (validation) Detailed project and production Software quality assessment during critical I&C life cycle. System requirements Software modular structure Software physical model Software logical model Requirements specification ContractSupply validation verification Qualification tests. Validation. Certification Experts report Acceptance document Acceptance test Preliminary tests (field test). Internal and external quality and quality in use characteristics measurement. Conformity expertise. Independent verification. Certification. Internal and external quality and quality in use characteristics measurement in the coordinated volume, including actual operating conditions. Acceptance tests (preliminary and final field acceptance). Quality in use characteristics measurement by operational testing results. maintenance Software lifecycle "Split" V-model - orientation to verification processes (process approach). Family of V-model right branches corresponds to software lifecycle stages as a whole: preliminary tests, qualifying tests, acceptance tests, commissioning.

Critical I&C software quality assessment: verification technologies diversification. … 6 Software quality measurement model and scheme Software specificationSoftware integration Real platform Is defined for Определяется для Internal quality External quality Quality in use Are measured Internal quality attributes and metrics А1 External quality attributes and metrics А2 Quality in use attributes and metrics А3 affects depends on characteristic sub-characteristic … Attribute … Metric … Method and scale … Primitives (software prime attributes- invariants) … Software quality measurement scheme: А1 А2 А3 Software quality analysis and assessment base of represents superposition of software attributes sets А1, А2, А3 (physical or abstract software properties which can be measured with the help of corresponding metrics), determining characteristics of internal, external and qualities in use. Quality models diversification is intended for software qualities assessment in different realization environments and at different I&C software life cycle stages

Critical I&C software quality assessment: verification technologies diversification. … 7 Methodology elements of diversification of software verification technologies. 1 The aim of diversification of software verification technologies is the increase of fullness and reliability of software quality assessments and obtaining of improved assessment due to variety principle realization. The task of each diverse technology is measurement of various software attributes metrics using various methods.. 3 Concept: possible variants of defects detection by diverse methods d1 and d2 for each type of program defect: 4 Technologies diversification effect depends from: а) Coverage of controlled (estimated) attributes for a diverse technologies composition. б) Sensitivity (checking ability) each of diverse technologies at detection of various types program defects; в) Actual (not terminological) degrees of technologies variety on various types defects sensitivity in concrete software project conditions 5 Offered decision – verification diverse technologies realization software invariant basis: а) technology on the basis of logical-numerical analysis (invariant – variables numerical value in view of interval restrictions, accuracy of representation and logic of calculation б) technology on the basis of semantic analysis (invariant –variables physical value). 2 Software fault - the key concept at solving diversification technologies problems. It is considered: а) at specification level (source software text) – software attribute incorrect value (anomaly) (in terms of programming languages); b) at software address field level - display (projection) of anomaly at software specification level in an incorrect cells composition of software address field (in terms of an executable code). 6 Assessment criteria – saving of invariants (permanent software properties) in different I&C operational conditions. d1d2Diverse method composition result + +Defect confirmation by both diverse methods +- Diverse method effective variety -+Secondary analysis to resolve the contradiction --Insensibility of both diverse methods

Critical I&C software quality assessment: verification technologies diversification. … 8 Software verification technologies diversification: software assessment on the basis of numerical and semantic invariants D' 1 U D' 2 – defects set, detected by 1- st or 2-nd methods (in software address field) D\D' 1 U D' 2 - defects set, undetected by 1-st or 2-nd methods (both methods insensibility) А1- software attributes set on the basis of 1-st invariant and appropriate metrics (method1+scale1) А 2 - software attributes set on the basis of 2-st invariant and appropriate metrics (method2+scale2) Initial set of defects D D'1D'1 D' 2 D1D1 D2D2 А1А1 А2А2 Diverse assessments comparative analysis А1А1 А2А2 А3А3 U Ai – software characteristics analysis and assessment base for diverse technologies composition (superposition of software attributes sets and internal, external quality and quality in use corresponding characteristics) is the for technologies diversification ) Дефект ПО –representation (projection) on software address filed of incorrect attribute value (measured physial or abstract software property). Method sensitivity (Checking ability) – specific type of software defect detection probability Possible variants: d1 d D' 1 и D' 2 – software specification anomalies on programming language level D 1 and D 2 Diverse technologies composition checking ability: D' 1 U D' 2 – defects set, detected by 1- st or 2-nd methods (in software address field) Software address field

Critical I&C software quality assessment: verification technologies diversification. … 9 Common case: M1 M2 Ø Theоretical-plural model of software residual defects for diverse verification technologies composition. A – program address space М – Initial defects set Variants: pessimistic- M1 M2 optimistic - M1 M2 = Ø A М М2 М1 M\M1 U M2 М1 –sub-set of residual defects, undetected at verification M1 M2– sub-set of residual defects, undetected for verification and independent verification composition М2 – sub-set of residual defects, undetected at independent verification M\M1 U M2 – sub-set of detected defects for verification and independent verification composition Verification effectiveness is defined by software residual defects probability Software residual defects model represents sub-sets М i superposition for composition of verification diverse methods with different sensitivity (checking ability) M1 M2

Critical I&C software quality assessment: verification technologies diversification. … 10 maximum possible benefits value from independent verification can make Р(М1) Benefit Assessment of verification technologies diversification efficiency. M1 \ М1 М2 software defects absence probability after verification Рбд = 1-Р(М1) after independent verification Рбд1.2= 1-Р(М1 М2) indicator of software residual defects risks decrease for verification and independent verification composition Р(М1 М2) В= Р(М1) - –––––––––––– Р(М1)= Р(М1) (1 – Р(М2М1) Р(М1) А М М1М1 М2М2 М1 М2

Critical I&C software quality assessment: verification technologies diversification. … 11 Problems: Diverse technologies realization on the basis of numerical and semantic software invariants. Definition of quantitative value of residual defects risks decrease indicator Source software Source software front- end processing. Variables overriding Compilation. Tool version formation. Check points CP arrangement Recursive interpretation Assessment according to CP Project database Operation «mix» of source software Defects profile formation Defects profile matrix Source software tool version Diverse technologies calibration by defects «crop» method. Primary data registration Defects profile Statistic processing and assessment of : а)sensitivity and variety degree of diverse technologies б) indicator of software residual defects risks decrease value Calibration results by program defects injection methods Independent verification preliminary report Source software residual defects risks decrease conclusion Report Source software with class «type+checkpoint» Functions supported by tool system: source software grammatic analysis source software instrumentation; arrangement of control points - probes source software attributes measurement in recursive interpretation mode; formation of software project quality integrated assessment calibration is an experimental definition of various types software defects sensitivity and variety degrees of diverse technologies using method of software defects drop injection according to the profiles which are determined taking into account concrete software project statistical invariant «mix of operations ». Flow model of the static analysis and an software attributes assessment. Diverse technologies realization on the basis of static analysis of source software

Critical I&C software quality assessment: verification technologies diversification. … 12 Expertise object normalization Expertise scenario formation Functional model of full scenario of expertise critical I&C software independent verification Заявка Technique, utility of organizational type Discrepancies report Pattern-form Technique, utility of informational type Requirements Measurement scheme Technique, utility of informational type Project documentation manual analysis on the basis of expert maps Static analysis of source software Software characteristic assessment verification of milestone and scenario at whole Experts report Technique Technique, utility of analytic type Experts report Technique, utility of analytic type Scenario (reference monitor) Нормализован- ный объект экспертизы Specification: Software attributes CP-probes Expert map of manual analysis. Measurement scheme: CP arrangement Coverage assessment Methods sensitivity Project normative profile Attributes measurement result at manual analysis Attributes measurement result at static analysis Technique, analysis group ОЭ 1 ОЭ 2 … ОЭn Project documentation Place and part of source software static analysis.

Critical I&C software quality assessment: verification technologies diversification. … 13 Conclusions 3. A basis of normative-methodical equipment of test laboratories are normative profiles of software requirements of various statements statuses including processes, methods, metrics and I&C software assessment procedures regulatory requirements. Appropriate normative requirements profiles to software are the necessary condition of quality and safety critical I&C application achievement. 2. The basis of I&C software expertise and independent verification tool support is the static analysis of software source texts. Technologies diversification is based on measurement of semantic, interval and accuracy source software invariants. Diverse technologies realization on the basis of the source software static analysis provides a controllable degree of technologies variety at the assessment of I&C software base characteristics - "Functionality", "Reliability", "Maintainability", etc., with use of metrics "Semantics", "Interval", "Accuracy" of software variables. 1. The concept of normative-methodical and tool equipment of independent verification and I&C software expertise processes is the use of a variety principle (technological diversity) as target means of expert assessment quality improvement.. 4. Offered approach provides the opportunities in frameworks of risk-informed regulations of NPP safety to estimate quantitatively and control the value of decrease of I&C software residual defects risks probability in diapason from 0 to 100% for a composition of diverse verification technologies.. 5. Prospect of long-term operation and development of test laboratories and critical I&C software development organizations tool equipment on the basis of static analysis is provided due to application of open architecture principle and advanced web- technologies (the web-services, software-based web-applications, communication protocols).