o On managerial aspects as against tool technicalities. Focus ... Complexity and
cross-impacts grow with ... Market expectations and stock price impacts ...
PNSQC 2013
Administering QA for Large Programs Sreeram Gopalakrishnan PMP, CMST Sr. Manager – Quality Engineering & Assurance Cognizant Technology Solutions
All rights reserved. The information contained herein is subject to change without notice.
©2013, Cognizant Technology Solutions
Agenda
o Large program characteristics o Administering key QA facets o Nuances & challenges o Best practices & tools
Focus 1
o o
On nuances as against fundamentals On managerial aspects as against tool technicalities
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Large Programs – What Is Different? Vendor Methodologies
Span & Integration •
Multi year, incremental capability releases
•
Multiple business units impacted
•
Different methodologies used by vendors pose problems during integration
•
Different contractual terms guiding vendor behaviours
•
Discrete projects in early stages, integration later
•
Complexity and cross-impacts grow with integration
•
Mutual dependencies between vendor teams
•
High executive visibility due to:
•
Competitiveness impacts co-operation
•
Large budgets and spends
•
Challenge of building a badgeless team
•
Market expectations and stock price impacts
Vendor Dynamics 2
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
Executive Visibility ©2013, Cognizant Technology Solutions
Organizing For Efficiency, Integration & Independence For objective assessment of quality & risks, and avoid bias in reporting To counter constant flux, disjointed views, communication gaps
Independence
Integration
Efficiency • QA connect with business crucial – not just at operational level, but at tactical / strategic • Program QA integration with production support / operations vital for feedback and course corrections. 3
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
Grouping functions for maximum collaboration & productivity
©2013, Cognizant Technology Solutions
Managing Risks & Dependencies
Objectives
Interdependency Meetings
o o
Get earliest visibility to changes, risks and dependencies Influence stakeholders positively to remedy risks and issues
Integrated Weekly Status Meetings
Forum to discuss changes to the solution or schedule
Forum to discuss status with Program Leadership
Analyze impacts across all projects or application areas of the program
Program Leadership, IT PMs, Solution Leads, QA PMs attend
IT PMs, Solution Leads, Business Leads, QA PMs attend
Integrated Weekly Status Report is the prime tool
Risk & Issue Trackers Key tool to bring leadership attention to risks and issues across the program
Integrated Program Plan is the prime tool
4
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Illustration: Integrated Program Plan: GANTT View Integrated Program Plan
Rel#1-March Design Rel#1-March Build (incl FUT) Rel#1-March Peformance Execution Rel#1-March SIT & Regression Testing Rel#1-March Training Rel#1-March UAT & Regression Testing Rel#1-March Go Live & Warranty Rel#1-April Design Rel#1-April Build Rel#1-April FUT Rel#1-April SIT/ Regression Rel#1-April Peformance Execution Rel#1-April UAT & Regression Testing Rel#1-April Go Live & Warranty Rel#2-May Design Rel#2-May Build (incl FUT) Rel#2-May Performance Execution Rel#2-May SIT & Regression Testing Rel#2-May Training Rel#2-May UAT & Regression Testing Rel#2-May Go Live & Warranty
5
Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y
6-Jan 20-Jan 11-Mar 17-Feb 4-Mar 24-Mar 29-Mar 14-Jan 18-Feb 1-Apr 8-Apr 15-Apr 15-Apr 21-Apr 1-Feb 22-Feb 29-Apr 10-Apr 22-Apr 12-May 19-May
18-Jan 22-Feb 28-Mar 28-Mar 15-Mar 28-Mar 26-Apr 29-Mar 5-Apr 9-Apr 16-Apr 19-Apr 19-Apr 17-May 29-Mar 17-Apr 17-May 10-May 4-May 17-May 20-Jun
30-Jun
23-Jun
16-Jun
9-Jun
2-Jun
26-May
19-May
12-May
5-May
28-Apr
21-Apr
7-Apr
14-Apr
31-Mar
24-Mar
17-Mar
10-Mar
3-Mar
24-Feb
17-Feb
10-Feb
Q1 Q2 Q3 2 3 4 5 6 7 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 3-Feb
End date
2013
27-Jan
E Start Status x Date e
13-Jan
Milestone Description
6-Jan
Program Milestones
30-Dec
Year Quarter Period 1 Week 1 2 3
20-Jan
Version - IPP_1.0
Rel#1 DESIGN Rel#1 BUILD Rel#1 PERF Rel#1 SIT/ REGRESSION TRAINING Rel#1 UAT Rel#1 GO-LIVE Rel#1- APR DESIGN (SAP/ BI) Rel#1- APR BUILD Rel#1- APR FUT Rel#1- APR SIT/ REGRESSION Rel#1- APR PERF Rel#1- APR UAT/ REG Rel#1 - April Go-LIVE Rel#2 - MAY DESIGN Rel#2 BUILD/ FUT Rel#2 PERF Rel#2 SIT/ REGRESSION TRAINING Rel#2 UAT Rel#2 GO LIVE
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Illustration: Integrating Program & QA Project Plans
6
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Illustration: Integrating Program & QA Project Plans
7
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Balance between Standardization & Flexibility
Standardization of Test Nomenclature
Flexibility In Governance of Processes
8
o
Large programs have several test phases & test types
o
Several of them non-standard & specific to the program
o
Vendor methodology differences add to the issue
o
Confusion results once program integration starts
o
Well defined processes with ETVX essential
o
Knowing the fine line between process adherence and excessive rigidity
o
Defining tolerance limits and deviation approval mechanisms
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Illustration: Definition of Test Phases & Types
9
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Status Reporting Speed Accuracy Consistency Lack of Bias
10
o o
Due to high executive visibility that large programs carry Challenges: • Different details for different levels of audience • Different timings of individual reports & co-ordination challenge • Team or vendor biases clouding accuracy
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Illustration: Status Reporting Cadence
QA WSR created
Fri
QA WSR review with QA Leadership
Mon
QA WSR consolidated into Program WSR + delta updates
Tue
Program WSR review with Program Mgr
Wed
Thu
Next week’s QA WSR created
Fri
QA Monthly Operational Report published
Monthend
Daily Status Reports
11
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Illustration: Program-Level Weekly Status Report
12
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Illustration: QA Weekly Status Report
13
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Environment Planning & Management Single-most biggest dependency for QA is generally the readiness of test environments Integration with Release Mgmt
Environment Reservation
•
Multiple releases in testing phase at the same time, hence right code versions to be ensured
•
Environments shared across multiple releases within a program, or with other projects
•
Collaboration between Environment Mgmt and Release Mgmt functions to be well defined: o RM accountable & responsible (OR) o RM accountable, EM responsible
•
Code version / data conflicts on shared environments
•
Environment reservation calendar as a tool to manage conflicts
•
Synchronized data across applications in a landscape for SIT / UAT is a real challenge
•
Not factoring in smoke test in test schedules is an oft repeated mistake
•
Defining the right data to fulfill test scenarios is the critical activity o Data lead staffed with the best resource o Integrated data planning sessions
•
Smoke test of application environments separate from smoke test of integration
•
Different layers of the solution owned by different teams makes fixes time consuming
•
War-room for smoke test is a successful option
•
Data management tools to automate mechanical tasks Data Synchronization & Refresh
14
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
Smoke Test ©2013, Cognizant Technology Solutions
Illustration: Environment Reservation Calendar
15
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Illustration: Data Refresh Plan
16
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Defect Management Challenges
o o o
Triage during SIT Defining defect turnaround SLAs Usage of Severity and Priority
Triage during SIT
Defect Turnaround SLAs
Large number of teams makes triage unwieldy and inefficient
Fix turnaround impacts QA’s execution productivity – thus a vital factor
Focused triage as a best practice
SLAs may have contractual connotations to dev vendors, hence
o Application / business area-wise triage meetings
o Dedicated defect managers
17
Turnaround “guidelines” coupled with defect aging KPI’s can be an effective method
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
Usage of Severity & Priority Standard definitions o Severity – business criticality o Priority – urgency to tester Confusion over which of the two should drive defect resolution. Recommendation: o Priority – in early stages o Severity – in later stages
©2013, Cognizant Technology Solutions
Regression Approach Regression testing is testing done to ensure code has not regressed due to changes
strategy to be made specific w.r.t
Scope of Regression
coverage is a balancing act
Ownership of Regression
Coverage of functionality of previous releases versus functionality of current release
Dedicated regression team o Greater focus & higher cost o Knowledge transition challenge
Impact-based (focused) regression versus broad-based critical scenario regression
SIT team doing regression o Cost efficient but lesser focus o Knowledge transition challenge
18
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
Methodology of Regression Number of regression cycles planned Overlap with SIT / Performance / UAT cycles Coverage in each cycle o Core (critical) scenarios o Impact or defect based scenarios
©2013, Cognizant Technology Solutions
Illustration: Regression Methodology
19
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Integrating Performance Testing Integrating Performance
20
o o o
Performance does not get attention till it comes in the critical path Early integration of Performance Testing is an issue One-Face QA model as a best practice • QA manager as a SPoC for all testing tracks – SIT, Regression, Performance, UAT – regardless of separate ownership by vendors / teams
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Application & End-to-End Performance Requirements
Performance Requirements Challenges
21
o
Perception of performance being secondary to functional
o
Rudimentary requirements gathering processes
o
Lack of historical data on transactions / user volumes
o
Solution architects / business resort to trail-and-error on SLAs
o
E2E performance requirements need collaboration across multiple IT teams
o
Performance Requirements Workshop as a best practice • Soln Architect, Business and Dev leads present • Systematic, framework driven approach • Production performance monitoring to provide supporting data
| All ©2010, Technology Solutions rights Cognizant reserved. The information contained herein is subject toConfidential change without notice.
©2013, Cognizant Technology Solutions
Questions ?
[email protected]
All rights reserved. The information contained herein is subject to change without notice.
©2013, Cognizant Technology Solutions
Thank You
All rights reserved. The information contained herein is subject to change without notice.
©2013, Cognizant Technology Solutions