Title is Arial Bold 14 centered - Semantic Scholar

9 downloads 353 Views 322KB Size Report
Managing Software Defects: Defect analysis and traceability. Craig Henderson ... The expectation of having defects is so great that all companies producing ..... [4] Boehm, B. and Basili, V. (2001): Software Defect Reduction Top 10 List. In.
ACM SIGSOFT Software Engineering Notes

Page 1

July 2008 Volume 33 Number 4

Managing Software Defects: Defect analysis and traceability Craig Henderson http://www.craighenderson.co.uk [email protected] Build Defects Defects Known KLOC Defect Abstract Reported Fixed Defects Density This paper describes a mechanism for presenting software defect Baseline --460 78 5.9 metrics to aid analysis. A graphical representation of the history of 0 26 78 408 84 4.9 software builds is presented, that records software build quality in 1 14 31 391 84 4.7 a way that cannot be displayed in a single numerical table, and is 2 5 28 368 85 4.3 visually more appealing and more easily digestible than a series of 3 2 9 361 85 4.2 related tables. The radial analysis charts can be used to represent 47 146 derivative information in a two-dimensional form and is demonstrated with practical examples of Defect Analysis and Root Cause Table 1 Analysis. The columns are defined as: Keywords: defect management, bugs, software quality, metrics, Build. The build number is an incremental number that uniquely radial charts, software builds, defect analysis identifies the software build. In this example, the build number

Introduction An irony of Software Engineering is that it is an inexact science using very exact technology. Nothing could be more precise than a unit of two states, 0 or 1; yet creating the sequence of states that can get a machine to do something useful or interesting – at least to do it correctly and consistently – is very difficult. Defects will exist within software for as long as software itself exists, and that is set to be a very long time. So we can resign ourselves to the inevitable doom of failure, or we can accept that the software we produce will contain defects and manage the situation as best we can. I am a realist – some may say a pessimist – but also a perfectionist. Accepting failure is difficult for me to do, so I look to the alternative of managing the situation and looking for improved methods of doing so. Defect Management The expectation of having defects is so great that all companies producing software commercially will run some form of defect management system. There is a vast choice of commercial and freeware products available to choose from, and I’m not going to condemn or condone any here. I have used a variety over the years and all have served the purpose – it’s just a case of efficiency. I would summarize the key requirements of a useful defect management system to be: •

Defect lifetime management



User and Role based workflow



Defect segregation (multiple products supported)



Defect cross-referencing – preferably across segregation

starts at zero and increments by one each time the software is built. This build number can be used in a variety of ways to add further value. For example, one company I worked for used odd number builds internally within the software engineering team and even numbers to release a software build to the test team. This segregation of purpose helps to control the distribution of software internally within the company. The numbering scheme also extended naturally so that General Availability (GA) releases are only ever an even build number. Defects Reported identifies how many defects have been reported to date (see Managing the Statistics) in each build of software. This figure should be a very simple metric to collect via a query from the defect management system in use in the organization. Defects Fixed identifies how many defects were claimed fixed (see Fixed or not) in each build of software. This figure should also be a very simple metric to collect via a query from the defect management system. Known Defects is calculated from the number of Known Defects in the previous build, plus the number of Defects Reported in the build less the number of Defects Fixed in the build. For example, for Build 1 in Table 1, 408+14-31=391. KLOC. Thousands of lines of code in the module being analyzed (see Scope). Defect Density. This is a very widely used metric computed by dividing the number of defects by the KLOC to yield an index figure. How this figure is interpreted and used is then a much wider field, and interpretation must be defined in drawing conclusions using the defect density. The figure standing alone is near useless, and can even be dangerous if its purpose and meaning is left to the reader to define.



Root cause analysis (incomplete/inaccurate requirements, Each row in Table 1 represents a build of software. The top row is coding error, unit testing, system testing) labeled Baseline and the purpose of this is to relate the analysis within this table back to the previous round of analysis. The baseline row is useful to enable segregation of defect analysis between Defect Analysis Table 1 shows a tabular analysis of defect statistics collected dur- software releases. ing a short release cycle of four software builds DOI: 10.1145/1384139.1384141

http://doi.acm.org/10.1145/10.1145/1384139.1384141

ACM SIGSOFT Software Engineering Notes

Page 2

Scope To get most out of the analysis, it is important to define the scope at a manageable and appropriate level, and to identify metrics that will add value to the development process when analyzed [1]. In a complex software project consisting of many applications with shared components for example, collecting and analyzing metrics across the entire product will yield little benefit as the defect density will be diluted and it will be impossible to quantify the quality of each application. Defining the scope on a granularity that is too fine will cause an excessive amount of information to be generated and the analysis will become too time-consuming to be beneficial, and the process will become disproportionate to the value gained from doing it. A fair compromise is to calculate and analyze the Defect Density at the module level, but the scope will vary from one project to the next.

July 2008 Volume 33 Number 4

and the defect density figures are inaccurate. I therefore now insist that defects are never re-opened, but a new defect must be raised to cover the remaining problem, or new problems that have been introduced, and if I can get a metric of defects claimed fixed but proved not fixed, then that helps to identify areas for improvement within my software engineering team. ANALYSIS CHART Table 1 is sufficient in providing the detail for each build and a trend can be seen across the builds by analyzing the column contents, but the information is flat and difficult to comprehend. Even in this four-build example, the numbers are beginning to blur and lose any impact. To try to breathe some life into the statistics and to avoid the number blindness of staring at a table of figures, I have devised a radial analysis chart representation of the data, as shown in Figure 1. The intention with this chart is to provide information about the build quality in a way that cannot be presented in a two-dimensional table. Each software build is represented by a circular band – a build band. Thin black lines are drawn outwards from the centre point for each statistic element (column) from Table 1, and where the line passes through the build band, the numeric statistic is written adjacent to the line.

Managing the Statistics The statistics cannot be frozen once they are published. As testing continues with previous builds, or as customers report issues with releases, the figures for Defects Reported and Defect Density need to be updated. For a General Availability software release, it is important to be able to capture the defect analysis before the release and periodically after it. This will be helpful in analyzing the trend of the defect density pre- and post-release, which will give This in itself is fairly unremarkable and simply an alternative repan indication to the efficacy of the system testing within the soft- resentation of the information. However, with a careful design in ware production lifecycle. placing the statistic elements, I can begin to add more data in a third dimension of the graph. Fixed or Not Fixed? In nearly every organization in which I have worked, there has been a lot of debate over what to do with a defect report if system testing demonstrates that the defect is not fixed, or is only partially fixed. The fundamental contentious issue is whether the defect should be reopened and re-assigned to the software engineering team, or if a new defect should be raised to describe the defect again, or the remaining issues not addressed by the original issue. During my career, I have changed my opinion over which is the best way to address such an issue several times. From a developer’s perspective, re-opening the defect is preferable, as it keeps all the detail together and the discussion through the defect report can help to document the decisions made and the final conclusion over what the fix should be. There can be occasions where a very subjective defect is discussed to such an extent that the defect becomes confused, in which case I’d advocate closing the original defect as a non-issue and create a new defect report (against the same build number) to summarize the discussion. Cross-references between the two defects – in both directions – is an absolute must though, to ensure traceability. However, from a management perspective, my view is different. While I still appreciate the benefits of the developer’s preference, the overall picture of software quality becomes skewed if defects are re-opened – particularly if the defect is partially fixed, or the fix has introduced further problems. If a defect is only partially fixed, then re-opening the defect will cause the partial fix to be omitted from the release information for the next build, and the regression test and subsequent system testing will suffer. Where new defects have been introduced by a fix, then if the original defect is simply re-opened and re-assigned to the software engineering team, then the new defects are missed from the statistics, DOI: 10.1145/1384139.1384141

Figure 1: Radial analysis chart A key piece of information missing from Table 1 is a traceability of the build numbers where defects are reported and fixed. It gives us totals of Defects Reported and Defects Fixed per build, but there is no correlation between the two. This additional information is however represented in the radial analysis chart in Figure 1. By orienting Defects Reported vertically at the top of the chart, we can introduce a Defects Fixed statistic which represents the build where defects are fixed and relate them to where they are reported. Note how the Defects Fixed statistic from Table 1 has been labeled Defects Fixed per Build in the chart for clarity. The vertical Defects Reported statistic remains unchanged from http://doi.acm.org/10.1145/10.1145/1384139.1384141

ACM SIGSOFT Software Engineering Notes

Page 3

July 2008 Volume 33 Number 4

Table 1 and records the number of defects that are reported in a given build number. From the per-build value at the intersection of the build band and the vertical statistic bar, I have drawn a horizontal statistic bar which crosses the build bands for subsequent builds. At each intersection I have entered the number of those defects that were fixed in each build. This gives a statistic of quality improvement through the build/test cycle prior to a software release to a customer. At the end of each horizontal bar is a total representing the number of defects that were reported in each build and fixed in subsequent builds. The sum of these totals will then equal the sum of Defects Fixed per Build; 119 + 11 + 13 + 3 = 78 + 31 + 28 + 9 = 146 ROOT CAUSE ANALYSIS Collecting defect statistics is invaluable for assessing the efficiency of the development team and can help to identify areas for improvement. With the addition of the traceability statistic in Figure 1, we begin to get real value by seeing a trend of figures through the release cycle. A logical addition to the statistics already discussed is to be able identify where in the software development lifecycle a defect was introduced; the root cause. Identifying the root cause can be a controversial and subjective issue. The idea is not to cause rifts between members of the development team, or even between the development team and other teams that contribute to the software development lifecycle, but to build a long-term picture of the performance of the group and to be able to identify where attention could be concentrated for the biggest improvement. I have extended the radial analysis chart that I devised for defect analysis for use in Root Cause Analysis (Figure 2). The basic principle is the same, but this time each radial band represents an element to which a defect can be attributed. In my example, I have chosen to use these stages; Requirements, Architecture, Design, Implementation, Testing.

Figure 2: Root Cause Analysis

in the requirements and the design, but the knock-on defects caused by these few can be significant [2, 3]. For instance, it has been claimed that finding and fixing a software problem after delivery can be 100 times more expensive than finding and fixing it during the requirements and design phase [4]. CONCLUSION The radial charts presented here can be used to represent derivative information in a two-dimensional form. I have used defect analysis and root cause analysis to demonstrate their use in presenting complex numeric information in an easy to understand form. This form if representation can be useful in a number of situations where there are dependencies or contributing values which influence subsequent values. Without any specific time line requirement, the dependencies are captured and represented such that influences can be identified addressed to reduce the knock-on effects.

The vertical statistic in this chart is the Defects Attributed figure, which represents the number of defects that have been attributed References to each element (band). The horizontal statistic bar coming off of [1] Fodeh, J. (2007): What’s on your Dashboard? In Better Software magazine, each of these figures crosses subsequent elements in the software October 2007, pp26-32 development lifecycle and the intersection of this bar with each [2] Nakajo, T., Sasabuchi, K., Akiyama, T. (1989): A structured approach to band shows a figure for the number of defects later in the lifecycle software defect analysis. In Hewlett-Packard Journal, April 1989, pp50-56 that were ultimately caused by a particular defect; knock-on de[3] Subramaniam , B. (1999): Effective Software Defect Tracking - Reducing fects. The totals at the end of the horizontal statistic bars summa- Project Costs and Enhancing Quality. In CrossTalk; A Journal of Defense Software rize the number of defects that were cause. Engineering, April 1999, pp3-9 The diagram shown in Figure 2 demonstrates what I would consider to be a typical trend. A low number of defects were present

DOI: 10.1145/1384139.1384141

[4] Boehm, B. and Basili, V. (2001): Software Defect Reduction Top 10 List. In IEEE Computer, Vol. 34, No 1, January 2001, pp135-137

http://doi.acm.org/10.1145/10.1145/1384139.1384141