PowerPoint Presentation - Good Practices - PNSQC

32 downloads 105 Views 2MB Size Report
o On managerial aspects as against tool technicalities. Focus ... Complexity and cross-impacts grow with ... Market expectations and stock price impacts ...
PNSQC 2013

Administering QA for Large Programs Sreeram Gopalakrishnan PMP, CMST Sr. Manager – Quality Engineering & Assurance Cognizant Technology Solutions

All rights reserved. The information contained herein is subject to change without notice.

©2013, Cognizant Technology Solutions

Agenda

o Large program characteristics o Administering key QA facets o Nuances & challenges o Best practices & tools

Focus 1

o o

On nuances as against fundamentals On managerial aspects as against tool technicalities

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Large Programs – What Is Different? Vendor Methodologies

Span & Integration •

Multi year, incremental capability releases



Multiple business units impacted



Different methodologies used by vendors pose problems during integration



Different contractual terms guiding vendor behaviours



Discrete projects in early stages, integration later



Complexity and cross-impacts grow with integration



Mutual dependencies between vendor teams



High executive visibility due to:



Competitiveness impacts co-operation



Large budgets and spends



Challenge of building a badgeless team



Market expectations and stock price impacts

Vendor Dynamics 2

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

Executive Visibility ©2013, Cognizant Technology Solutions

Organizing For Efficiency, Integration & Independence For objective assessment of quality & risks, and avoid bias in reporting To counter constant flux, disjointed views, communication gaps

Independence

Integration

Efficiency • QA connect with business crucial – not just at operational level, but at tactical / strategic • Program QA integration with production support / operations vital for feedback and course corrections. 3

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

Grouping functions for maximum collaboration & productivity

©2013, Cognizant Technology Solutions

Managing Risks & Dependencies

Objectives

Interdependency Meetings

o o

Get earliest visibility to changes, risks and dependencies Influence stakeholders positively to remedy risks and issues

Integrated Weekly Status Meetings

Forum to discuss changes to the solution or schedule

Forum to discuss status with Program Leadership

Analyze impacts across all projects or application areas of the program

Program Leadership, IT PMs, Solution Leads, QA PMs attend

IT PMs, Solution Leads, Business Leads, QA PMs attend

Integrated Weekly Status Report is the prime tool

Risk & Issue Trackers Key tool to bring leadership attention to risks and issues across the program

Integrated Program Plan is the prime tool

4

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Illustration: Integrated Program Plan: GANTT View Integrated Program Plan

Rel#1-March Design Rel#1-March Build (incl FUT) Rel#1-March Peformance Execution Rel#1-March SIT & Regression Testing Rel#1-March Training Rel#1-March UAT & Regression Testing Rel#1-March Go Live & Warranty Rel#1-April Design Rel#1-April Build Rel#1-April FUT Rel#1-April SIT/ Regression Rel#1-April Peformance Execution Rel#1-April UAT & Regression Testing Rel#1-April Go Live & Warranty Rel#2-May Design Rel#2-May Build (incl FUT) Rel#2-May Performance Execution Rel#2-May SIT & Regression Testing Rel#2-May Training Rel#2-May UAT & Regression Testing Rel#2-May Go Live & Warranty

5

Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

6-Jan 20-Jan 11-Mar 17-Feb 4-Mar 24-Mar 29-Mar 14-Jan 18-Feb 1-Apr 8-Apr 15-Apr 15-Apr 21-Apr 1-Feb 22-Feb 29-Apr 10-Apr 22-Apr 12-May 19-May

18-Jan 22-Feb 28-Mar 28-Mar 15-Mar 28-Mar 26-Apr 29-Mar 5-Apr 9-Apr 16-Apr 19-Apr 19-Apr 17-May 29-Mar 17-Apr 17-May 10-May 4-May 17-May 20-Jun

30-Jun

23-Jun

16-Jun

9-Jun

2-Jun

26-May

19-May

12-May

5-May

28-Apr

21-Apr

7-Apr

14-Apr

31-Mar

24-Mar

17-Mar

10-Mar

3-Mar

24-Feb

17-Feb

10-Feb

Q1 Q2 Q3 2 3 4 5 6 7 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 3-Feb

End date

2013

27-Jan

E Start Status x Date e

13-Jan

Milestone Description

6-Jan

Program Milestones

30-Dec

Year Quarter Period 1 Week 1 2 3

20-Jan

Version - IPP_1.0

Rel#1 DESIGN Rel#1 BUILD Rel#1 PERF Rel#1 SIT/ REGRESSION TRAINING Rel#1 UAT  Rel#1 GO-LIVE Rel#1- APR DESIGN (SAP/ BI) Rel#1- APR BUILD Rel#1- APR FUT Rel#1- APR SIT/ REGRESSION Rel#1- APR PERF Rel#1- APR UAT/ REG  Rel#1 - April Go-LIVE Rel#2 - MAY DESIGN Rel#2 BUILD/ FUT Rel#2 PERF Rel#2 SIT/ REGRESSION TRAINING Rel#2 UAT  Rel#2 GO LIVE

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Illustration: Integrating Program & QA Project Plans

6

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Illustration: Integrating Program & QA Project Plans

7

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Balance between Standardization & Flexibility

Standardization of Test Nomenclature

Flexibility In Governance of Processes

8

o

Large programs have several test phases & test types

o

Several of them non-standard & specific to the program

o

Vendor methodology differences add to the issue

o

Confusion results once program integration starts

o

Well defined processes with ETVX essential

o

Knowing the fine line between process adherence and excessive rigidity

o

Defining tolerance limits and deviation approval mechanisms

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Illustration: Definition of Test Phases & Types

9

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Status Reporting Speed Accuracy Consistency Lack of Bias

10

o o

Due to high executive visibility that large programs carry Challenges: • Different details for different levels of audience • Different timings of individual reports & co-ordination challenge • Team or vendor biases clouding accuracy

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Illustration: Status Reporting Cadence

QA WSR created

Fri

QA WSR review with QA Leadership

Mon

QA WSR consolidated into Program WSR + delta updates

Tue

Program WSR review with Program Mgr

Wed

Thu

Next week’s QA WSR created

Fri

QA Monthly Operational Report published

Monthend

Daily Status Reports

11

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Illustration: Program-Level Weekly Status Report

12

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Illustration: QA Weekly Status Report

13

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Environment Planning & Management Single-most biggest dependency for QA is generally the readiness of test environments Integration with Release Mgmt

Environment Reservation



Multiple releases in testing phase at the same time, hence right code versions to be ensured



Environments shared across multiple releases within a program, or with other projects



Collaboration between Environment Mgmt and Release Mgmt functions to be well defined: o RM accountable & responsible (OR) o RM accountable, EM responsible



Code version / data conflicts on shared environments



Environment reservation calendar as a tool to manage conflicts



Synchronized data across applications in a landscape for SIT / UAT is a real challenge



Not factoring in smoke test in test schedules is an oft repeated mistake



Defining the right data to fulfill test scenarios is the critical activity o Data lead staffed with the best resource o Integrated data planning sessions



Smoke test of application environments separate from smoke test of integration



Different layers of the solution owned by different teams makes fixes time consuming



War-room for smoke test is a successful option



Data management tools to automate mechanical tasks Data Synchronization & Refresh

14

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

Smoke Test ©2013, Cognizant Technology Solutions

Illustration: Environment Reservation Calendar

15

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Illustration: Data Refresh Plan

16

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Defect Management Challenges

o o o

Triage during SIT Defining defect turnaround SLAs Usage of Severity and Priority

Triage during SIT

Defect Turnaround SLAs

Large number of teams makes triage unwieldy and inefficient

Fix turnaround impacts QA’s execution productivity – thus a vital factor

Focused triage as a best practice

SLAs may have contractual connotations to dev vendors, hence

o Application / business area-wise triage meetings

o Dedicated defect managers

17

Turnaround “guidelines” coupled with defect aging KPI’s can be an effective method

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

Usage of Severity & Priority Standard definitions o Severity – business criticality o Priority – urgency to tester Confusion over which of the two should drive defect resolution. Recommendation: o Priority – in early stages o Severity – in later stages

©2013, Cognizant Technology Solutions

Regression Approach Regression testing is testing done to ensure code has not regressed due to changes

strategy to be made specific w.r.t

Scope of Regression

coverage is a balancing act

Ownership of Regression

Coverage of functionality of previous releases versus functionality of current release

Dedicated regression team o Greater focus & higher cost o Knowledge transition challenge

Impact-based (focused) regression versus broad-based critical scenario regression

SIT team doing regression o Cost efficient but lesser focus o Knowledge transition challenge

18

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

Methodology of Regression Number of regression cycles planned Overlap with SIT / Performance / UAT cycles Coverage in each cycle o Core (critical) scenarios o Impact or defect based scenarios

©2013, Cognizant Technology Solutions

Illustration: Regression Methodology

19

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Integrating Performance Testing Integrating Performance

20

o o o

Performance does not get attention till it comes in the critical path Early integration of Performance Testing is an issue One-Face QA model as a best practice • QA manager as a SPoC for all testing tracks – SIT, Regression, Performance, UAT – regardless of separate ownership by vendors / teams

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Application & End-to-End Performance Requirements

Performance Requirements Challenges

21

o

Perception of performance being secondary to functional

o

Rudimentary requirements gathering processes

o

Lack of historical data on transactions / user volumes

o

Solution architects / business resort to trail-and-error on SLAs

o

E2E performance requirements need collaboration across multiple IT teams

o

Performance Requirements Workshop as a best practice • Soln Architect, Business and Dev leads present • Systematic, framework driven approach • Production performance monitoring to provide supporting data

| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.

©2013, Cognizant Technology Solutions

Questions ? [email protected]

All rights reserved. The information contained herein is subject to change without notice.

©2013, Cognizant Technology Solutions

Thank You

All rights reserved. The information contained herein is subject to change without notice.

©2013, Cognizant Technology Solutions