Release Criteria: Good to Go - CiteSeerX

WHITE PAPER

Release Criteria: Good to Go by Johanna Rothman

Release Criteria: Good to Go

Release Criteria: Good to Go Part 1: Using Release Criteria to Check Your Software for “Doneness” by Johanna Rothman For any project, the big question is, “Is the software ready to release yet?” Or more specifically, “When is the development and testing part of the project done?” You

Is It Done Yet?

can't know if you’re ready to release unless you know what “done” means.

When is the development and testing part of the project done? You

Why Release Criteria?

can't know if you’re ready to release,

It’s certainly possible to release before you are done. In the mid-1980s, I worked

unless you know what “done”

as a test lead on a major operating system release that included a brand-new,

means.

window-based GUI. At the time, most operating systems had command line interfaces, so our customers couldn't tell us what they wanted in a graphical interface. For many months we worked on it until management decided it was time to release. When senior management mandated the release date, we didn’t have an easy way of knowing if the software would be good enough for our customers by the desired date. Despite many internal objections(from developers and testers), the system was released … and it was a disaster. Our customers hated the way the GUI worked and wanted to get rid of it altogether. We hastily planned an incremental release, fixing the obvious defects. Our customers liked the initial point release and were thrilled about the next point release, when everything in the system finally worked in a way they could use. Now, we were essentially done. When you don't know what “done” means, you may also go on testing after the software is ready enough. A few years ago, Fast Track had already released

Villanova University | World Leader in Online Education ©2007 Johanna Rothman and Villanova University | All rights reserved | www.VillanovaU.com

Release Criteria: Good to Go a successful client/server product and needed to transition to the Web quickly. During the Web transition, a few more features were added to the product and a number of defects were fixed. The project was on an aggressive schedule, and the VP was under serious pressure from the rest of the management team to release the product. For the last four weeks of the project, the VP interrogated everyone daily on the project team, asking if they were done yet. At the project retrospective, the VP described what he thought was necessary for the release to be done. A senior developer said, “If I'd known that was all I had to do, I could have been

Getting on the Same Page

done a month ago.”

At the project retrospective, the VP described what he thought

The VP looked furious.

was necessary for the release to be done. A senior developer said, “If

Fast Track didn’t have release criteria; they only had a release date. Because they

I'd known that was all I had to do, I

had no way to assess how good each product component had to be, the developers

could have been done a month ago.”

and testers determined each component’s doneness themselves. Another release problem occurs when someone who doesn't actually know what's going on decides to release the software. Deb, a QA manager, found herself in a tough spot. Development had turned over part of the software, and her team had finished preliminary planning and exploratory testing. They had a short testing cycle and felt like they had too much to do in too little time. Deb called her technical leads into a meeting to assess risks and figure out how to best allocate their scarce resources to testing the product. During the meeting, one of the testers ran in and said, “Don’t even bother. They shipped the release at noon.” Deb asked her VP why they’d already shipped the software. The VP said, “You weren’t going to be able to tell me anything about the software in less than a month, and the VP of Sales convinced our CEO that we had to ship. I was vetoed.” Fortunately, Deb saw this as a potential opportunity, and said, “Ok, let me hire more testers.” The VP said, “If I give you more people, can we release the software faster?” Deb said, “I don’t know if we’ll be able to release it faster, but we’ll know earlier if it’s good enough to release. We can make release decisions without you being vetoed.” Maybe you've run into problems like the ones I've described. Maybe, as the test lead or test manager, you have found yourself in the role of gatekeeper, personally responsible for the content and quality of the release. Or perhaps your organization makes the release decision based on a gut feeling, without the data to back it up. How do you know when testing is done? How do you know when the product is ready to release? Does the decision to release seem similar to deals done in dark,


Release Criteria: Good to Go smoke-filled rooms, or as if there are rules of the game that you don’t know? Does the release decision sometimes seem completely arbitrary? If you're finding it difficult to make rational decisions about when to release the software, developing and using release criteria can help. Release criteria are objective measurements of the critical attributes of the product or project. Listing and referring to the criteria allow you to know whether the product is ready to release. They can help you make rational decisions about when

Defining Project Success

to ship the software.

If you find out at the beginning that not everyone agrees on a definition

Developing Release Criteria

of success, then you won’t be able to

When you develop release criteria, first decide what’s critically important to

agree on what makes the software

this specific project—Why is the company doing this project? Why would the

ready to release.

customers buy this product? If you don’t know what success means for this project, start by defining success for this project. Then quantify how you would recognize success during the project, and gain the agreement from the project and senior management teams on the criteria you generated that will measure the doneness of the release.

1. Define Success Not every project has a clear definition of success. If you’re in the fuzzy frontend of a project, you may find it easier to interview the project manager and other stakeholders first, before trying to define what’s important to the project or drafting release criteria. I define what success means by understanding: • The problem(s) this project is trying to solve • The project’s requirements (What are your managers paying you to deliver? How good does it have to be? When do they want it? Are there other constraints?) • The product’s requirements (What do the customers want out of the project?) You’ll notice that success has nothing to do with the number of defects found, or thenumber of tests defined or run. Success is about what the customers will be able to do with the product when you’re done with it. If your project manager doesn’t initiate projects this way, you can learn what success means by interviewing the PM and other managers in the organization. If you find out at the beginning that not everyone agrees on a definition of success, then you won’t be able to agree on what makes the software ready to release. It’s better to discover any disconnect at the beginning of a project, before you plan


Release Criteria: Good to Go your approach to testing. If not everyone agrees on what success means, make sure the PM knows. If you're the PM, make sure your project sponsor knows. Then, get help reconnecting everyone's definition of success. If necessary, use a professional facilitator to help with this reconnection. After her management’s preemptive release, Deb realized that defining success

What’s Really Important?

would help her plan which testing to do when, for future releases. Defining

Customers don’t buy products based

success would provide enough information needed to plan a product’s release.

on the number or lack of defects— they buy products because they

When Deb started trying to define success, she found her questions weren’t always

solve some problem the customer

welcome. “Not everyone was happy that I wanted to pin down what success

needs solved.

means,” she said. “The VP Sales was unhappy, because it was clear that if we all knew what success was, he couldn’t make the decision to release or not by himself … he wouldn't be able to blame engineering for his salespeople's inability to sell the product.” Release criteria are not about shifting blame from one group to another; they are to provide some objective measurement about when a product is ready to release. Release criteria can also help you build whole-product responsibility into the product release—Can the salespeople sell the product that meets these criteria? Can the support staff support the product? Can the trainers develop and deliver training? When I work with people to define what success looks like, they realize not only are they accountable for their part, but they’re pointing me towards project success as well. Once you know what success means, you can define what’s most important for this project.

2. Learn What’s Important For This Project What’s critically important to this project is the combination of what the company needs and what the customers need. Customers don’t buy products based on the number or lack of defects—they buy products because they solve some problem the customer needs solved. When you define what's most important, don't stop at the number or type of defects. The customer model isn’t just important to commercial software organizations. If you’re a non-profit or an in-house organization, a customer-oriented business imperative still applies to you. Non-profits still need revenue; they just don’t distribute the excess profit back to shareholders. In-house organizations create products because their users think they can create a product more cheaply than


Release Criteria: Good to Go an outside organization. An in-house service exists more to save time and money for the rest of the company, not to specifically create revenue. Whether you sell products or work for an in-house organization, we’re all solving problems for our customers. Sometimes the date is most important. Sometimes, it’s a particular feature or set of functionality. More often, it’s a combination of schedule, features, and low-enough defects. It all depends on your customers and their expectations. (See Part 2, “What

More than Defects and Release Dates

Does Success Look Like?,” at the end of this paper.)

“...we didn’t realize how much

Rita worked at a startup company driven by cash flow. The company was not fully

demand we would have for this

funded, so they had a tremendous incentive to ship product early in order to get

product, and we now realize we

enough revenue to continue the company’s existence. For the first three releases,

can’t use only the date or some

the only release criterion was the date the release had to go to customers, so the

defect or test numbers to assess the

company could legally recognize revenue. Once they made it past the first couple

state of the release.”

of years and were adequately funded, the company developed other criteria including defect counts, test progress, and the states of “code freeze” (the times when developers would stop working on the release). Rita described it this way: “When we were a startup, we just needed to keep our heads above water. Our initial customers wanted pretty much what we gave them, but were willing to work with us. We were all blown away though, with last year’s release. All of a sudden, the customers cared about defects, more than they ever had before. My management demanded to know what kind of a QA group I was running, and I felt completely besieged. The only thing that saved my sanity was knowing that I’d checked with the entire project team and senior management in advance, to know that the release criteria we chose were what we needed. Unfortunately, we didn’t realize how much demand we would have for this product, and we now realize we can’t use only the date or some defect or test numbers to assess the state of the release. We need a much bigger picture to know when we’re ready to release the software.”

3. Draft Release Criteria I find it useful to draft a set a strawman release criteria, so we have a vehicle to start the discussion. I try to create a balanced representation of time to market, what our customers want, along with defect, performance, and reliability levels, if that’s appropriate. I don’t have to guess all the criteria correctly the first time, I just need to have something we can discuss as a group. Rita first drafted a set of release criteria as a starting place to discuss the release with the project manager and the rest of the project team. These were Rita’s draft criteria:


Release Criteria: Good to Go 1. All code must compile and build for all platforms 2. Zero high-priority bugs 3. For all open bugs, documentation in release notes with workarounds 4. All planned QA tests run, at least ninety-eight percent pass 5. Number of open defects decreasing for last three weeks 6. Feature x unit tested by developers, system tested by QA, verified with customers A, B before release 7. Ready to release by June 1

Get the Whole Story

8. All open defects evaluated by cross-functional team

Rita knew from her previous projects at the company that low

However, once she started talking to the PM, Rita realized that her initial criteria

defects were just part of the story

were not exactly what the customers or the company needed. Yes, the customers

when making a release decision.

were concerned with defects, but not to the same extent that Rita was. In fact, if a couple of the major customers were satisfied with the release, then chances were good that it was good enough for the rest of the customers. Initially, Rita’s PM was surprised that she had tried to look at the whole release from the customers’ perspective and come up with a balanced idea of what would make a complete release. He’d expected that Rita would be much more concerned with a traditional test manager’s perspective on quality—low defects. But, Rita knew from her previous projects at the company that low defects were just part of the story when making a release decision. Rita, along with the PM and the rest of the project team, revised the release criteria: 1. All code must compile and build for all platforms 2. Zero high-priority bugs 3. For all open bugs, documentation in release notes with workarounds 4. All planned QA tests run, at least ninety percent pass 5. Number of open defects decreasing for last three weeks. 6. Feature x unit tested by developers, system tested by QA, verified with customers A, B before release 7. Ready to release by June 1 These criteria don’t address everything for Rita’s release, but they cover what’s critically important for this release: ship date, good-enough software, and a specific feature tested and found working by two specific customers. Rita was disappointed that only ninety percent of the planned QA tests had to pass—she thought there was too much risk with such a low passing number. But, after hearing what everyone else said, she was willing to go along with the rest of the criteria because the release date was so critical. Rita was also concerned about removing the criterion about the cross-functional team evaluating the open defects at the end of the release. However, the product manager reassured Rita that he’d


Release Criteria: Good to Go discussed the issues with the PM, and that the PM could speak for the marketing and support staff.

4. Make Release Criteria SMART When you draft the release criteria, make sure they can be answered by anyone on the project team in the same way. I use the acronym SMART—Specific, Measurable, Attainable, Relevant, Trackable—to test that I have reasonable and objective criteria.

Avoiding Last-minute Surprises When you make criteria trackable,

Each criterion should be specific for this product at this point in its lifecycle.

you can evaluate the state of the

When you make a criterion measurable, you're ensuring that you can evaluate the

criteria during the project, not just

software against the criteria. Release criteria are not the place for stretch goals, so

during the last week.

make each criterion attainable. Make sure your criteria are relevant by evaluating this product against what the customer wants and what management wants. When you make criteria trackable, you can evaluate the state of the criteria during the project, not just during the last week. “No high priority open defects” is an objective and measurable criterion. You can look at the open defects and verify that none of them deserves to be high priority. “All open defects reviewed by cross-functional team” is another example of an objective and measurable criterion. Either the cross-functional team reviewed the open defects or they didn’t. “Fast performance” is an ambiguous criterion. To change this into an objective and measurable criterion, make it something like this: “Performance scenario A (corresponding to use case A) to complete in under ten seconds”. Here, you name the specific scenario, so people can refer to it, and give the performance criterion to verify that you met that performance.

5. Gain Consensus on Release Criteria Now that you have reasonable release criteria, it’s time to gain consensus on what you’ve developed. If people react negatively to your draft criteria, (“No, we couldn’t possibly do that”), then learn why they are concerned. Generating release criteria reveals assumptions and fears about the project and the product. I've found these questions helpful when generating or gaining consensus on release criteria: 1. Must we meet this criterion before the release date? 2. What happens if we do not meet this criterion before the release date? 3. Do we put our product or company at risk by not meeting this criterion? 4. Do we negatively affect people’s safety if we don’t meet this criterion? These questions generally help the entire project team stay focused on what's needed for this release.


Release Criteria: Good to Go As the test lead, I’ve gained consensus on release criteria by either of two ways: 1) drafting the release criteria, discussing them and coming to an agreement at a project team meeting, or 2) drafting them with the project manager and discussing them at a project team meeting. I prefer generating release criteria with the entire project team, because then the team owns the release criteria, not just the PM or me. However, if you’re working on a large project, or have never used release criteria before and want people to understand what they’re working towards, you may be better off generating release criteria with the PM in advance.

Adjusting Your Release Criteria “I can't get performance that good

No matter how you gain consensus, make sure you evaluate the release criteria and

between now and the time we have

the product's readiness for release—not how the developers and tester achieved

to release. When we discussed the

the release criteria.

release criteria, I thought it was possible, but now I realize it's not

Using Release Criteria

going to happen.”

I use release criteria to evaluate the state of the project throughout its duration. I ask my PMs to use the release criteria as part of the project team meeting, so that the entire team is able to assess the project's state of doneness during the entire project. When we get to the formal system test part of the project, I use the criteria as part of the testing status report. I also find that release criteria are an early warning sign that you're not going to make the release. Manny was a PM, halfway through a six-month project. Manny looked at the progress to date and was concerned that the team was not going to make the ship date. He decided to make the release more real to the project team by generating release criteria with them. Manny then used the release criteria each week during the project team meeting, to verify the project was making progress. It worked for a couple of weeks until one week during the release criteria evaluation, one of the engineers said, “I'm not going to make it. I've tried and tried, and I'm just not going to make that criterion for our ship date.” Manny said, “OK, I need to go back to management and see what we need to do. Before I do that, does anyone else think they're going to have problems meeting any of the release criteria?” Another engineer said, “I can't get performance that good between now and the time we have to release. When we discussed the release criteria, I thought it was possible, but now I realize it's not going to happen.” Manny was able to get a much earlier heads-up about his project’s progress with release criteria. For this team, realizing they weren't going to meet the release criteria two-thirds of the way through the project instead of at the end was a relief. The PM knew what the project reality was and could work with management to see which tradeoffs made the most sense. In this case, the PM was able to renegotiate the release date, so that the product could meet the release criteria.


Release Criteria: Good to Go If all goes well, you’ll evaluate the release criteria as you go, and you'll meet the criteria when the project is supposed to end. However, projects don’t always go well, and you won’t always be able to meet the release criteria. When that happens, make sure you’re honest about what’s happening. For me, release criteria are either met or not met. We aren’t partway to meeting a criterion—we haven’t met it. I find that this binary approach helps me more when I’m discussing the state of the software with senior management. If I say we’re

Progress with Peace of Mind

partway there, they hear that we’re done. If I say we haven’t met the criterion, they

Use the criteria as you progress

hear we’re still working.

through the project and up to the final release. Then you can say,

I generally don’t change the release criteria during a project. There are at least two

“Release it!” with pride.

circumstances when it’s reasonable to change release criteria: 1) when we learn more about what “done” means for this project, and 2) if we realize we can’t meet all the release criteria by the desired release date. If you learn more about what “done” means, then ask yourself the questions above about the release criteria: Must we meet this criterion? What happens if we don’t meet the criterion? If you’re working on a project and can’t meet the release criteria, say so. Have your management say something like: “We thought these other criteria were important, but we realize now that the date is even more important than we thought. We’re going to release the product, even though we haven’t met all the release criteria.” If this is true, you can have them add, “We’re going to determine what prevented us from meeting our criteria this time, and create the next project so that we don’t miss our release criteria.” It’s possible to create a software product your customers will want to use, and to know when you’re done creating it. You don't have to play the “when-can-werelease-the product” game. When you use release criteria to know when a project is done, you have taken potentially hidden decisions and made them public and clear. Make your release criteria objective and measurable, so everyone on the project knows what they’re working towards. Use the criteria as you progress through the project and up to the final release. Then you can say, “Release it!” with pride.

References 1. Moore, Geoffrey. Crossing the Chasm. New York: Harperbusiness, 1995. 2. Gause, Don and Gerald M. Weinberg. Exploring Requirements, Quality Before Design. New York: Dorset House Publishing, 1988. 3. Grady, Robert. Practical Software Metrics for Project Management and Process Improvement. New York: Prentice Hall, 1992.

10


Release Criteria: Good to Go

Part 2: What Does Success Look Like? Geoffrey Moore[1], discusses a model of high-tech marketing, what your customers expect and when, from the introduction of a product, through the growing acceptance, through the product’s end of life.

Managing Your Goals

Bob Grady[3] described the three possible goals of any software project: Meeting

You only have one primary goal for

a specific time to market, some feature set that would make the customers happy,

a given project, and then manage the

and keeping defects low. You only have one primary goal for a given project, and

other goals within the constraints of

then manage the other goals within the constraints of that goal. (See “What to Do

that goal.

When Your Project Slips,” http://www.jrothman.com/Papers/Whattodowithslips. html). Table 1 describes the intersection of where your market is with your possible project goals. If you’re just starting out with a product, you have technology enthusiasts as your primary customers. They want the software fast. The software has to do something, and not be too shabby, but the enthusiasts don’t need a lot of product, it just has to do something well. Once you’ve increased your market to include the visionaries, you have slightly different concerns. Although frequently, the priorities are similar, your customers are early adopters. They have a specific problem, and they want your software to fix that problem. Unfortunately, it seems as if each early adopter has their own problem, and that the problems are sufficiently different that you may be able to use a core product, but you have several one-off solutions to address each of your early adopters’ problems. If you’ve seen the product competition appear like a game of leapfrog, then you’ve seen the early adopters in action. Table 1: Project priorities based on where you are in the product's lifecycle Market description

Introduction

Early Adopters

Mainstream

Late Majority

End of Life

Who buys the software

Technology Enthusiasts

Visionaries

Pragmatists

Conservatives

Skeptics

Time, features, defects, ranked by importance

1. Time to Market 2. Feature Set 3. Low Defects

1. Time to Market 2. Feature Set 3. Low Defects

1. Low Defects 2. Time to Market 3. Feature Set

1. Low Defects 2. Feature Set 3. Time to Market

1. Low Defects 2. Feature Set 3. Time to Market

Once you hit the Mainstream, the pragmatists take over. Pragmatists want your software to work. They care about upgrade availability, but not as much about installing upgrades or new versions as you do. (Because your new software costs them money and time, and they have perfectly good software to use until they

11


Release Criteria: Good to Go choose to upgrade.) In fact, once your software has reached the late majority, even though it’s still in the mainstream, your customers don’t want releases from you any more often than they absolutely have to take them. The skeptics may or may not buy your software, but nothing says they’ll actually use the software. Let’s take word processing software as an example. Back in the 80’s, there were several successful and competing word processing software packages. However, once PCs were available for the work and home markets, people wanted to use

Who are Your Customers?

word processing software that was easier than using a markup language under

Are you selling to early adopters

Unix. Many companies, including Wang, Xerox, Corel, Microsoft, and Lotus,

or the late majority? Is it some

made word processing software. During the late 80’s and early 90’s, each of these

combination of markets? Use those

companies released numerous versions of their products. In my opinion, the word

customer definitions to determine

processing wars were won with Microsoft’s release of Word 5.1. At the time, it had

your project goals.

fewer obvious defects than its competitors, and it made its release date ahead of the competition. After Word 5.1, the other companies still released their products, but Microsoft became what Moore calls the market “gorilla”. Now, with the most recent versions of Office, Word is firmly in the late majority. Those of you who use Word don’t want to upgrade any more than necessary. We discover new problems from the ones we’re used to, and most of us prefer to stick with our known problems. When you decide what’s important to your project, define where your product is in its lifecycle. Are you selling to early adopters or the late majority? Is it some combination of markets? Use those customer definitions to determine your project goals. Even when you define what’s most important for your customers, your company will still have corporate goals it needs to accomplish with a particular release. Many project managers (PMs) talk about the trade offs between cost, schedule, and quality, but that’s too simplistic for what PMs actually do. In reality, PMs trade off among the things the customers value: time to market, the feature set, and the defect levels, and the things the company values: cost to market, the number of people working on the project, and the working environment of the project. Release criteria help make these tradeoffs obvious. As testers, and especially as test leads, or test managers, we’re more frequently aware of potential risks than other people—that’s our job. You can’t change the corporate needs that drive the release. After all, if you need revenue this quarter, you still need revenue. However, you can help illuminate the tradeoff decisions by discussing potential risks with the project manager. You can use the tradeoffs between the six project attributes (time to market, feature set, defect levels, cost to market, people, working environment) to best employ the test people and best use the test time to the project’s advantage. How do you ferret out this information?

12


Release Criteria: Good to Go I interview the various project stakeholders: the project sponsor or senior management, marketing, support, training, whoever has a stake in the project’s done-ness.

Use Context-Free Questions I use context free questions [Gause and Weinberg] as my primary interviewing technique for discovering what success means. These are some context free questions:

Who are Your Customers? Are you selling to early adopters

1. What does success look like?

or the late majority? Is it some

2. Why are these results desirable?

combination of markets? Use those

3. What is the solution worth to you?

customer definitions to determine

4. What problems does this system solve?

your project goals.

5. What problems could this system create? These questions all ask why we’re doing the project, without actually asking why. Sometimes, when we use the word “why,” we sound as if we’re interrogating others, or asking them to defend their positions. Interrogations or putting people on the defensive are rarely a good way to gather information about the project, and retain a good working relationship with other people. You’ll notice that the questions also don’t ask “how”. Sometimes, when we ask how, we start designing things. You don’t want to design the project or the product; you want information about the product, to know how to evaluate the product’s state of completion throughout the project. Once you’ve collected information about the product and the project, you can start to draft release criteria. As a nice side-benefit, you’ll learn where to put your scarce testing resources and time.

About the Author Johanna Rothman is a well-known consultant, speaker and author on managing high-technology product development. She is a frequent contributor for Fast Company’s online career center, Software Development, Computerworld.com and StickyMinds.com. She is the author of the highly acclaimed Hiring the Best Knowledge Workers, Techies & Nerds, and is the coauthor of Corrective Action for the Software Industry. She holds a Master of Science in Systems (Software) Engineering from Boston University, a Bachelor of Science in Computer Science and a Bachelor of Arts in English Literature from the University of Vermont.

13


PROFESSIONAL EDUCATION ONLINE www.VillanovaU.com | 866-588-5263