Software Testing in Latvia: Lessons Learned Dr.math. Anda Adamsone, SWH Technology Dr.hab.sc.comp. Juris Borzovs, RITI M.Sc.comp. Martins Gills, RITI M.Sc.comp. Sanda Linde, IT Alise M.Sc.comp. Janis Plume, RITI 1
Testing in Latvia: a look into history Early research - Automatic construction of test cases - theoretical and practical approach. Courses on software testing are included into undergraduate computer curricula of Latvian Universities for more than a decade. Recent and current research in testing: • • • •
software test tools universal symbolic interpretation software process improvement; testing issues practical manual methods of software testing
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
2
Testing in Latvia: major publications 1. J.M.Barzdins, J.J.Bicevskis, A.A.Kalnins. Construction of Complete Sample Systems for Correctness Testing. 1975 2. J.M.Barzdin, J.J.Bicevskis, A.A.Kalninsh. Automatic Construction of Complete Sample Systems for Program Testing. 1977 3. A.Auzins, J.Barzdins, J.Bicevskis, K.Cerans, A.Kalnins. Automatic Construction of the Test Sets: Theoretical Approach.1991 4. J.Bicevskis, J.Borzovs, U.Straujums, A.Zarins, E.F.Miller,jr. SMOTL - A System to Construct Samples for Data Processing Program Debugging. 1979 5. J.Borzovs, A.Kalnins, I.Medvedis. Automatic Construction of Test Sets: Practical Approach. 1991 6. Z.Bicevska, J.Bicevskis, J.Borzovs. Regression Testing of Software System Specifications and Computer Programs. 1995 ICSTEST 2000
Software Testing in Latvia: Lessons Learned
3
IT in Latvia: a snapshot 1991 - First international development project in by Latvian software designers to set up an information system for the social insurance of artists in the German state of Bremen. The successful result gave the basis for the local software industry. 1999 - Latvia's two largest software producers - DATI and SWH Technology - exported products worth a total of 7 million DEM. 2000 - Up to 100 local companies are engaged in SW development. Three major of them, DATI, SWH Technology and IT Alise, employ more than 650 employees, IT professionals. 2000 - 5 computer manufacturers and 2 software companies in Latvia have obtained the ISO 9001 certificate.
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
4
The quest for quality Initial stage - any computer program is more or less OK Current move - software as a company’s business process backbone. The outmost priority is quality. Customer Satisfaction
Efficient Work
Product Quality
Profit
Competitiveness ICSTEST 2000
Software Testing in Latvia: Lessons Learned
5
Awareness of testing Customers require quality
Quality related activities within the software development life-cycle
• • • • • • • •
identification, documentation, tractability, verification, validation, reviews, audits, testing, problem resolution ...
What is customer going to pay for? They are interested in the least expensive, but at the same for time the most evident and efficient activities. Historically, the most widespread and the most acceptable is testing. Although developers understand its importance, only a minor part of customers are aware. ICSTEST 2000
Software Testing in Latvia: Lessons Learned
6
IT standards: origin 1993-1995 Leading IT company DATI (formerly SIS) launches the initiative to examine internationally used software engineering standards. Goals: • Searching for world-known and the most informative standards • Assess the experience from IT projects • Guidelines, standards, templates Output: • List of industry standards • IT terminology in Latvian • Company standards, guidelines; mostly based on IEEE • Changes in development process management ICSTEST 2000
Software Testing in Latvia: Lessons Learned
7
National IT standards Most of Latvian National IT standards were approved in 1996, adopting DATI internal standards based on IEEE standard family. Areas covered: • • • • • • •
Software Quality Assurance, configuration management, reviews and audits Testing, verification and validation Software documentation SW requirements specification, Operational concept description Guidelines for Software design descriptions Project management plans User Documentation ICSTEST 2000
Software Testing in Latvia: Lessons Learned
8
IT standards: industry practice For couple of years in Latvia, large business software development, especially documentation, is based on IEEE J-STD-016-1995 Software development: Acquirer-supplier agreement standard. Main features: • • • •
All software development project activities specified Major document templates given Includes guidance for tailoring to a certain project Built-in quality assurance mechanism
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
9
Software development processes ISO/IEC 12207 defines 5 primary, 8 supporting and 4 organizational life cycle processes. Industry uses a reduced version of ISO/IEC 12207. For example, DATI has adopted the standard identifying the following processes: 3 primary processes: Development, Reengineering and Maintenance. 3 supporting processes: Documentation, Configuration management, Quality Assurance 2 organizational processes: Management and Training
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
10
Who performs testing? (1) Testing laboratory / department
Project A Testing function is provided by a separate testing laboratory
Project B ...
Advantages: United methodology, pool of testing professionals, independence.
Project Z
Disadvantages: Testing mostly impossible at unit or integration level. ICSTEST 2000
Software Testing in Latvia: Lessons Learned
11
Who performs testing? (2) Each project has a small testing group Advantages: Testing at all levels, high knowledge about software under test. Disadvantages: Reduced independence, may not be testing professionals but rather programmers, designers or analysts.
ICSTEST 2000
Project A Testing group
...
Project B Testing group
Project Z Testing group Software Testing in Latvia: Lessons Learned
12
Demand for testing professionals What qualifies a testing professional?
Typical project mistake - testing is given to person who is bad at programming, designing or system analysis. Testing professional is a software engineer specialized in testing. His/her main task is to have a different look at development activities. How to become a testing professional? Currently a course on testing is included as a part in the curriculum of Latvian SWEPE (Software Engineering Professional Education) project.
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
13
Tester’s body of knowledge Initiated by SWEPE project, a certain analysis has started to define a knowledge area for the testing professional. Key areas: • Basic concepts and definitions of testing • Analysis of test levels • Test techniques • Test automation • Test related measures
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
14
Qualification levels of the tester Level 1
2
3
4
Tasks Execution of previously specified tests and other tasks that do not require special knowledge or training Tasks of 1st level + Test description according to system documentation. Tasks of 2nd level + Test planning and managing of test team Tasks of 3rd level + Training of other testers and development of testing methods
ICSTEST 2000
Required skills Computer usage skills - advanced used level and ability to understand specified tests. Skills of 1st level + Ability to analyze system documentation and to write a testing documentation. Skills of 2nd level + Ability to identify required tests, skills in team and resource management. Skills of 3rd level + Tuition skills and expert knowledge in testing theory
Software Testing in Latvia: Lessons Learned
15
Tester’s role Test engineer is not just a resource! Test engineer =
Functional tests Non-functional tests Test tool development
This is a person specialized on testing in a categories of development platforms, test types and system application areas. ICSTEST 2000
Software Testing in Latvia: Lessons Learned
16
Collaboration among the developers
System analyst / designer
Requirements
Problem reports Requirements
Tester
Code
Programmer Problem reports
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
17
Test levels and performers
Test level
Structural tests
Unit testing Integration testing System testing Qualification testing Acceptance testing
+ +
ICSTEST 2000
Functional tests
+ + + +
Performed by developers developers developers, independent testers developers, independent testers customer, independent testers
Software Testing in Latvia: Lessons Learned
18
Tasks of tester Type of software development activity System analysis, specification of requirements Design Coding
Testing
Prior to delivery (preparation for the acceptance)
ICSTEST 2000
Tasks for tester Verify whether requirements specification and other documents are suitable for testing. Plan the testing: its tasks and schedules. Plan in detail the tests: type, coverage. Make test descriptions. Refine plans and test descriptions. Execute tests. Make automation and regression tests. Thoroughly implement the system, installation and documentation testing. Do regression testing. Check-up the corrections. Run the regression tests.
Software Testing in Latvia: Lessons Learned
19
Well defined testing process Test planning
Test identification
Test execution Review of tests and their results ICSTEST 2000
Software Testing in Latvia: Lessons Learned
20
The absolute minimum of testing What to do when no testing is planned? Problem: customers refuse to pay for testing activities (assuming that product will be perfect). When the original contract agreement does not provide enough resources for testing, DATI practices the allocation of the extra staff to the project to keep the expected quality of the product.
Project A
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
21
Educating the customer Commonly asked customer questions are: What is testing? Why is it necessary for the particular project? Why it costs so much?, etc. How does the developer benefit from a well informed and competent customer? Main results: • It is possible to develop an appropriate testing strategy for the project • Better collaboration between acquirer and supplier in the reporting and solving the problems • Customer can do the acceptance testing ICSTEST 2000
Software Testing in Latvia: Lessons Learned
22
A case study (1): Testing process for porting project Task: to port software from 16bit to 32bit environment
Test cases
Test coverage extension
Test operation
Initial state for testing: test cases cover 25% of functionality. Recordplayback regression testing is done.
Test results review Test coverage maintenance
ICSTEST 2000
Problem registration
Problem reports
Software Testing in Latvia: Lessons Learned
23
A case study (2) measurements 6 4
Testing
3
Testing makes up 40% of the total project effort.
Coding
2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Months
After 15 months there was no evidence found that 20% source code (modules) contains 80% problems. The defects were very evenly disseminated across the source code. ICSTEST 2000
Lines of code in module
Persons
5
100000 90000 80000 70000 60000 50000 40000 30000 20000 10000 0 0
50
100
150
200
250
Updates in module
Software Testing in Latvia: Lessons Learned
24
Future vision The goal for the next 10-20 years The Baltic States - a "second IBM" - a unified "concern" with 120,000 highly qualified specialists. • Baltic States - exporter of software services • Design and maintenance of information systems and software products - a "trademark" for the Baltic States
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
25
Latvian software export income mill. Ls 5,000 4,073 4,000 3,000 2,000
1,142 7
1,000 7 1999
153 25 2005
78 2010
181 2020
year
Income from Latvia’s exports in the sector if it is not developed Income from Latvia’s exports in the sector if it is developed
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
26
Employment in IT sector People employed in the export-oriented software sector
Number of people
25,000 22,760 20,000 15,000 8,743
10,000 5,000
225
2,460 398
1,012
597
0 1999
2005
If the sector is not developed If the sector is developed
2010
2020
years
Students in software design
No. of people
6,000 4,850
5,000
4,850
4,850
4,000 3,000 2,000 1,000
1,400
1,400
1,400
1,400
1999
2005
2010
2020
0 years
If the sector is not developed If the sector is developed
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
27
Expected role of testing (1) In a context of a planned increase of remote development projects in the Baltic region, DATI has analyzed its current experience. Activities that suit best the remote development are: • • • • •
design coding testing maintenance reengineering and porting
Testing is among the top ones, mainly because of its task to be an independent verifying activity.
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
28
Expected role of testing (2) Main factors that define the increasing role of testing: • • • •
Increasing complexity of the software Dependence from the software in almost every area Increasing number of interfaces between various programs A large number of reengineering and porting tasks
Software testing - Latvian IT industry sector to boom
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
29
Questions ? Contact information: Riga Information Technology Institute Kuldigas iela 45 LV 1083 Riga Latvia E-mail:
[email protected] http://www.riti.lv tel.: + 371 761 15 22 fax: + 371 761 95 73
ICSTEST 2000
Software Testing in Latvia: Lessons Learned
30