1
The virtual logging meeting: a web-based solution to resource problems in software inspection Ilkka Tervonen, Lasse Harjumaa and Juha Iisakka University of Oulu, Finland Summary: Face-to-face meetings typically provide a natural way of negotiating and collecting opinions. The problem is that they also cause resource problems, i.e. they waste time and are difficult to arrange. A shift to more flexible implementations of meetings is thus an understandable alternative. The virtual logging meeting is the most interesting of these, because it provides at the same time both a disciplined and a flexible way of inspecting a document. Logging may be implemented at the same time but in a different place or at a different time and in a different place. The present paper defines the major steps of the virtual logging meeting and introduces a supporting tool called WiT (Web inspection Tool), the main features of which are its ability to inspect any HTML document and the support it gives for organising the meeting, removing duplicate issues and collecting inspection data. Ilkka Tervonen, Lasse Harjumaa and Juha Iisakka Department of Information Processing Science University of Oulu, P.O.Box 333, FIN-90571, Oulu, Finland Fax: +358 8 553 1890, E-mail: {tervo, marcus, isac}@rieska.oulu.fi
1. Introduction It is largely accepted that formal inspection is the most effective means of finding defects, so that it plays an important role in the software development process. Preparation for the inspection (individual inspection) and the logging meeting form the major phases in the conventional inspection process. Face-to-face meetings typically provide a natural way of negotiating and collecting opinions. The problem is that these meetings also cause resource problems, i.e. they waste time and are difficult to arrange. A shift to more flexible implementation of meetings is thus an understandable alternative. Flexibility is thus the motivation for finding new types of inspection meetings, such as the virtual logging meeting, limited logging meeting, pair inspection and no-logging meeting inspection. The virtual logging meeting is the most interesting of these, because it provides at the same time both a disciplined and a flexible way of inspecting a document. Logging may be implemented at the same time but in a different place or at a different time and in a different place, for example. This physical and temporal distribution reduces the mutual time required, makes reconciliation of participants' timetables easier and provides flexibility. Flexible and efficient on-line logging requires support, and this can be provided by web and network technology. The existing web-based prototypes support inspection by means of checklists, collect defect data, provide links from code lines to these data and allow for the production of readable summary reports. Web technology facilitates the collaborative aspects of inspection, because the web is global and working is not limited to working hours. This means that it is easy to distribute the artefacts for inspection. The web also supports the collection of inspection results. Thus the inspection leader may decide whether a face-to-face meeting is necessary or not. Based on our experiences with web inspection tools, we are now developing
2
a more comprehensive inspection tool, the main feature of which will be the ability to inspect any HTML document. In this paper we first motivate our approach and explain what advantages can be gained from a virtual logging meeting. We then define the major characteristics of virtual logging meetings, and finally introduce the WiT (Web inspection Tool) tool use in them. 2. Why a virtual logging meeting? In our earlier papers (Tervonen and Oinas-Kukkonen, 1996; Iisakka and Tervonen, 1998) we discussed the requirements for a reorganised inspection process, focusing especially on reducing the number of logging meetings. Philip Johnson discusses a similar reorganisation of inspection in his recent article (Johnson, 1998), considering the reorganisation problem from a broader viewpoint and introducing seven recommendations for the future of FTR (formal technical review, i.e. inspection): (1) provide tighter interaction between FTR and the development method, (2) minimise meetings and maximise asynchronicity in FTR, (3) shift the focus from defect removal to improved developer quality, (4) build organisational knowledge bases on the review, (5) outsource the review and insource the review knowledge, (6) investigate computer-mediated review technology, and (7) break the boundaries of review group size. We evaluated these recommendations from the discipline and flexibility viewpoints (Tervonen, Iisakka and Harjumaa, 1998), and found guidelines for the further development of inspection, such as (1) inspection should be more tailorable, (2) inspection should be more independent of time and place, (3) inspection should include more comprehensive metrics for measuring success, (4) inspection should actively focus on process improvement, e.g. evaluate the relevance of checklists, (5) the inspection team should hire external consultants to participate in inspections and buy style guides, for example, (6) the inspection tools should be based on network and web technology and utilise their benefits, and (7) in some cases the number of inspection participants may be over the limit of 6. As depicted in Figure 1, we have three principles for implementing discussion during inspections. The conventional logging meeting is based on face-to-face discussion, i.e. recording issues and questions in the same place and at the same time and allowing brief discussion. Public inspection uses asynchronous discussion and provides a way to distribute the meeting by means of web or network technology, whereas individual inspection supports only the collection of issues without discussion. We have defined four new types of inspection (Iisakka and Tervonen, 1998): virtual logging meeting, limited logging meeting, pair inspection and no-logging meeting, which are based on these three types of discussion and differ in their degree of discipline and flexibility. Face-toface meetings typically provide a natural way of negotiating and collecting opinions, but they also waste time and are difficult to arrange. A shift to more flexible implementations is thus an understandable alternative, although these raise some new problems such as insufficient understanding of the material and communication difficulties. This means that face-to-face meetings are always less flexible than virtual meetings, although from another viewpoint we may argue that face-to-face meetings are more flexible because they involve human resources. We will now introduce the major characteristics of the virtual logging meeting and compare it with the other three types of inspection.
3
1
No discussion
No logging meeting Collected issues
requires a trained moderator
Collected issues
Virtual logging meeting
2 Asynchronous discussion
Shared issues
Limited logging meeting limited number of participants
if controversy is evident Controversial issues
Controversial issues
Pair inspection a couple checks material and finally makes a certificate
3
Face to face discussion
Certificate
Figure 1. New types of inspection based on different discussion principles 2.1. Virtual logging meeting A virtual logging meeting is a combination of all three discussion principles, although asynchronous discussion is the aspect emphasised. This means that a logging meeting may be implemented at the same time but in a different place or at a different time and in a different place. This physical and temporal distribution of the inspection reduces the mutual time required and provides flexibility. After individual inspection and collection of issues, the logging and recording of agreed, unique issues is one responsibility of the moderator (usually the inspection leader; Gilb and Graham, 1993). During asynchronous discussion, support is required for flexible communication between participants. In order to manage distributed inspectors and their comments, the virtual logging meeting needs a fairly fixed protocol for acting. The moderator's duties in particular are strictly defined. If controversy is evident or discussion is required, a face-to-face meeting may be called, mainly to continue the discussion of open issues, i.e. topics on which the participants disagree. The inspection leader will steer the discussion of controversial issues, the group will discuss unresolved issues and negotiate over them, and their opinions will be recorded. 2.2. Other types of inspection
4
The three other types of inspection are the limited logging meeting, pair inspection and nologging meeting inspection. We will now briefly compare these with the virtual logging meeting. 2.2.1 No logging meeting The traditional inspection style demands a trained moderator who has a lot of work to do in one inspection, whereas the participants should be motivated to find errors anyway, without anyone to guide them. The inspectors can send their issues and comments to the author privately and he/she can check them alone and decide whether there is any need for action or not. There is no need for a meeting at which a whole group can together say what is wrong with the material. One risk factor in this approach is the competence of the inspectors. Borrajo (1994) suggests that the improvement proposals given by the inspectors should also be evaluated by the authors. This would guarantee efficiency on the part of the inspectors. Inspection without a logging meeting is a special case of a virtual logging meeting, where in any case the inspectors do not necessarily meet each other face-to-face. Normally they just send their issues to the common "notice board" where the other inspectors are able to study them and comment on them. If the comments from the other inspectors are irrelevant and the author merely checks the issues, the virtual logging meeting has become a no logging meeting inspection. Because there is no virtual meeting, there is no need for the readers to comment on each other, and therefore it is not so disciplined as a virtual logging meeting. 2.2.2 Pair inspection Pair inspection is a special type of peer inspection in which a software designer participates in the inspection and guides the process. In pair inspection two employees make a designerinspector couple and change roles after a while so that they act as inspectors for each other. As they form a "permanent" couple and each inspects the other's work, and we can assume that they take it more seriously than when participating in a normal inspection with many other people. The inspection practices of such a pair will evolve step by step during discussions, and thus it is meaningless to formulate any strict rules of action. This means that pair inspection is a very flexible and undisciplined way of inspecting. The recording of "temporary" issues (defects) found in informal meetings is not required, although a pair inspection needs a form or template of its own in the final (formal) meeting on which both the author and the inspector guarantee that the material has passed the pair inspection. This final document is the only disciplined aspect. 2.2.3 Limited logging meeting: Inspectors ought in any case to write issues down before the meeting, as this helps the scribe, and as the issues themselves may be very complicated. In this way they can be sent to the author by electronic mail to be collected and brought to the meeting. If it is hard to find a suitable meeting time for all the participants, only the most important and technically most competent employees may need to participate in the meeting, check all the issues and decide what to do. The limited logging meeting is a specialisation of the virtual logging meeting in the same way as the no logging meeting is, but in this case the common notice board is replaced by a smaller group of inspectors. In principle, the limited logging meeting is a "normal" inspection meeting, and it should therefore be quite disciplined, but it can be a specialised form of pair inspection. When only a couple of people -- the author and another person (e.g. the chief designer) -- check the issues together, they can work iteratively, focusing more on issues than on the document to be inspected. In this sense, it is sometimes a very flexible way of inspecting the material
5
3. The major steps of virtual logging meeting The virtual logging meeting is at the same time both a disciplined and a flexible way of inspecting a document. We can find three roles for participants in it, the inspection leader, the inspectors and the author. As we note in Figure 2, the role of inspection leader is the most important one, because he/she makes the decisions on how to proceed and sets the deadlines. The inspection process starts with a kickoff meeting, as normal, but this may be skipped if all the participants have earlier experience of inspection. Then, as depicted in Figure 2, the inspection leader selects the participants and when ready sets a deadline for individual inspection. Individual inspection should be completed by the deadline. The inspection leader decides, based on the results of the inspection, when to proceed to the next phase. If necessary he/she may ask some participants to recheck by a new deadline. After individual inspection and collection of issues, the logging and recording of agreed, unique issues is one responsibility of the inspection leader (in a logging meeting this role is usually called that of moderator). During asynchronous discussion, support is required for flexible communication between participants. In order to manage distributed inspectors and their comments, the virtual logging meeting needs rules for acting. When the leader is satisfied with the results, he removes duplicate issues and starts the public inspection by distributing the list of unique issues to all participants. The leader also sets a new deadline for the commenting phase. As in the individual inspection, the leader may ask for further comments when necessary. The comments of the earlier cycle may be shared for public inspection if the leader thinks that they are valuable enough. If participants fail to agree with all the issues, the leader may call a face-to-face meeting, mainly to continue the discussion of open issues. The leader will steer the discussion of controversial issues, the group will discuss unresolved issues and negotiate over them, and their opinions will be recorded. The inspection leader walks through all the comments collected during asynchronous discussion and face-to-face discussion. He also decides which comments to add to the list of issues. When the commenting phase is ready, the author (editor) walks through the list of issues and classifies them. This means deciding which ones are defects and which are caused by misunderstanding. The author must deal with every issue logged. After that the leader checks the status classification, starts studying and edit phases concurrently and sets deadlines for them. In studying phase the inspectors may now check the comments of the author, i.e. whether he/she accepts a given issue as a defect. This step is important for further learning of inspectors. The comments recorded and the classification of them may also be used to collect a company-wise organisational memory. The deadline defines how long time the results of classification are visible. In addition, concurrently with studying and edit, the inspection leader also provides summary reports of inspection so far. Although not defined here, the edit and follow-up phases are important to guarantee that defects will be corrected. The list of defects forms an agenda for the editor (author) and the person in charge of correcting.
6
Inspectors
Inspection leader
Selecting participants No
Ready for inspection?
Author
Legend: = setting deadline
Yes Checking and on-line recording
Rechecking
No
Ready for cleaning? Yes
Removing duplicate issues Sharing issues
Commenting and on-line recording
Further commenting (may include face-to-face meeting)
No
Ready for classification? Yes Classification of defects
No Ready to proceed? Yes
Studying
Reporting results of inspection
Edit & follow-up
Figure 2. The major steps in a virtual logging meeting
4. A web tool for virtual logging meetings Compared to earlier paper-based inspection tools, the on-line inspection tools go one stage further and carry out either the whole process or major parts of it on-line. Web technology facilitates the collaborative aspects of inspection, because the web is global and work is not limited to working hours. This not only brings flexibility to the inspection meetings (in the
7
form of asynchronicity and geographical diversion), but also enables easy and manageable distribution of the artefacts for inspection, including the document to be inspected, checklists, or any other related documents. Even the early review/inspection tools, such as CSRS (Collaborative Software Review System) (Johnson and Tjahjono, 1993), supported the collection and recording of review data. Likewise, the web-based prototypes such as Code Review (Rodgers, 1998), developed at the University of Arizona, and WiP (Harjumaa and Tervonen, 1998), developed at the University of Oulu, support inspection of text documents with WWW browsers. The on-line commenting facilities which they provide are convenient and easy. These tools are similar in many aspects, as they both support inspection by means of checklists, collect defect data, provide links from code lines to these data and allow readable summary reports to be produced. 4.1 Support for individual and public inspection Based on experiences with WiP, we have developed a more comprehensive inspection tool called WiT (Web inspection Tool) (Harjumaa, 1998), the main feature of which is the ability to inspect any HTML document. As depicted in Figure 3, the inspector may mark a suspect item with a vertical line at the left, which follows the metaphor of correcting paper documents.
Figure 3. Marking a suspect item in the WiT tool The most popular word processors today can easily convert documents into HTML, so that virtually all written material produced with the aid of computers becomes available for inspection. Instead of reading a document through in an applet or application, which is embedded into a web page, we now are able to inspect the web page itself. This makes the organisation of the client side of the tool much more convenient and complies better with the web metaphor which has proved to be easy to use and has been much appreciated by millions
8
of World Wide Web users. The checklist in Figure 3 is tailored to the design phase and its language can be chosen freely by the inspection team. The annotation window allows recording of the issue (comment) and its classification as a major issue, minor issue or question. The WiT tool also supports the inspection leader in planning the inspection process and in cleaning the list of issues. The step in which the leader walks through suggested issues and decides whether to approve, recheck or remove each is illustrated in Figure 4. Removal (ignoring) typically means removing duplicates.
Figure 4. Cleaning the list of issues (by the inspection leader) 4.2 Evaluation of the tool Macdonald and Miller (1998) evaluated a number of on-line inspection tools under four categories: document handling, individual preparation, meeting support and data collection. We use the same categories for evaluating the WiT tool. We also report on our first experiments with the tool in which evaluation was focused on usability aspects. Document handling: Since most documents are produced by computer, it is natural to allow browsing of documents on-line. This guarantees access to the latest version of the document and cross-reference between documents. Computer support allows on-line annotation of the documents, with annotations linked to the parts of the documents to which they refer. They can then be made available for all inspectors during public inspection. In the WiT tool the standard web browser allows documents to be viewed and annotated. The annotations can be classified and have checklist references associated with them. Individual preparation: Inspectors generally make use of checklists and other supporting documentation during preparation. These are kept on-line for easy cross-referencing. Inspectors can browse through the document being inspected and make annotations to it as they
9
progress, consulting on-line checklists at any moment. Suspect items are marked effortlessly by drawing a vertical line at the appropriate locations. Meeting support: Prior to the public inspection and face-to-face meeting, computer support may be used to monitor inspector effort. The inspection leader (moderator) can use this information to decide the best time to move from the preparation stage to the meeting, taking account of the amount of preparation done by each inspector. There is a large overhead involved in setting up each meeting, including finding a mutually suitable time, a room to hold the meeting in and so forth. By allowing a distributed meeting to be held using web technology, it may be easier for team members to attend the meeting using a suitably equipped workstation. The WiT tool supports asynchronous meetings by providing shared issues and later the author’s classification for commenting on. Comments from an earlier commenting cycle may also be shared if the inspection leader approves them. Data collection: An important part of an inspection is the collection of data which can be used to provide feedback to improve the inspection process. The WiT tool collects the time spent inspecting the document and the number of issues generated and calculates the inspection rate and issues per thousand lines. In our early experiments we evaluated the usability of the WiT tool. The participants liked the idea of a vertical line for marking and the constantly visible checklist for inspection-guided checking. The platform dependence (on the latest version of Netscape) caused some problems at first, but once installed, the WiT tool worked well. The support for the inspection leader was satisfactory, and the three alternatives (approval, rechecking, removal) guided the leader when browsing the list of issues. 5. Conclusions Face-to-face meetings typically provide a natural way of negotiating and collecting opinions. The problem is that these meetings also cause resource problems, i.e. they waste time and are difficult to arrange. A shift to more flexible implementations of meetings is thus an understandable alternative. The virtual logging meeting is the most interesting of these, because it provides at the same time both a disciplined and a flexible way of inspecting a document. The present paper defines the major steps of the virtual logging meeting and introduces a supporting tool called WiT (Web inspection Tool), which allows inspection of any HTML document. In addition, it supports the tasks of the inspection leader, such as organising the meetings (by collecting relevant inspection data) and removing duplicate issues. In our early experiments we evaluated the usability of the WiT tool. The participants liked characteristics such as the vertical line for marking, the checklist for guiding and the support given to the inspection leader. Large-scale experiments will take place during the years 19982000, in a project financed by the government and three Finnish software companies which will focus on improving inspection practices in those companies by tailoring an appropriate inspection method (e.g. from among those introduced in part 2) for use with the appropriate inspection tools.
10
References Borrajo Iniesta J.: A Tool and A Set of Metrics to Support Technical Reviews, in Ross M., Brebbia C.A., Staples G., and Stapleton J.(eds.), Software Quality Management II vol.2: Building Quality into Software, Computational Mechanics Publication, Southampton, U.K., 1994, ISBN 1-85312-353-6, pp. 579-594 Gilb T., and Graham D.: Software Inspection, Addison-Wesley, Wokingham, England, 1993, ISBN 0-201-63181-4 Harjumaa L.: The URL of WiT demo is http://rieska.oulu.fi/~wit, user account from Lasse Harjumaa,
[email protected], 1998 Harjumaa L., and Tervonen I.: A WWW-based Tool for Software Inspection, Proceedings of the 31st HICSS Conference, vol III, IEEE Computer Society Press, Los Alamitos, CA, 1998, ISBN 0-8186-8239-6, pp. 379-388 Iisakka J., and Tervonen I.: Painless Improvements to the Review Process, Software Quality Journal, 7, 1998, pp. 11-20 Johnson P.M.: Reengineering Inspection, Communications of the ACM, vol 41, no 2, 1998, pp. 49-52 Johnson P.M., and Tjahjono D.: Improving Software Quality through Computer Supported Collaborative Review, in De Michelis G., Simone C., and Schmidt K. (eds.), Proceedings of the Third European Conference on Computer Supported Cooperative Work, Kluwer Academic Publishers, Dortrecht, Netherlands, 1993, pp. 61-76 Macdonald F., and Miller J.: A Comparison of Computer Support Systems for Software Inspection, submitted for publication, 1998 Rodgers T.: The URL of Code Review demo is http://cmi118.cmi.arizona.edu/CodeReview. html, user account from Tom Rodgers,
[email protected], 1998 Tervonen I., and Oinas-Kukkonen H.: Reorganizing the Inspection Process, Software Process - Improvement and Practice, 2, 1996, pp. 97-110 Tervonen I., Iisakka J., and Harjumaa L.: Software Inspection – a blend of discipline and flexibility, in Kusters R., Cowderoy A., Heemstra F., and Trienekens J. (eds.), Project Control for 2000 and beyond, (Proceedings of the ESCOM-ENCRESS 98 Conference), Shaker Publishing B.V., The Netherlands, ISBN 90-423-0038-8, pp. 157-166
