Slides in pdf format

4/14/2008

Two Strategies Using Execution Feedback in Test Case Generation CMSC 737 – Presentation Bao Nguyen [email protected]

y Static plan y Category Partition y Data flow analysis (path, branch, def-use, etc) y Predicate based: BOR – BRO > Try to guess ahead!!! =>

y Dynamic plan y Execution information as feedback y Generating test cases on the fly

What’s next…

What’s next…

y Test case generation based on execution feedback


y Case study: two recent papers in ICSE’07


y What I’m looking at


y Conclusion

y Conclusion

Why Static Plans Are Not Enough?

General Framework

y Software is so complex: y Empirical products y Subjective solutions y Human actuators y Intelligent products y Dynamic environments

=> Difficult to formalize => Difficult to use => Difficult to manage => Difficult to predict => Difficult to address

Hard to predict ahead !!! Hard to exhaustedly test !!!

Required Quality Qr

Specifications Program Execution

Test case Engine

Feedback

Qo

Observed Quality

-

e =Q −Qo r

f(e)

(Adapted from “Software Cybernetics: Progress and Challenges” - Aditya P. Mathur )

1

4/14/2008

General Framework

General Framework Required Quality Qr

Specifications Program Execution

Test case Engine

Feedback

Qo

Observed Quality

-

e =Q −Qo r

Program Execution

Test case Engine

f(e)


Required Quality Qr

Specifications

Feedback

Qo

Observed Quality

-

e =Q −Qo r

f(e)


What’s next…

What’s next…

y Execution feedback based test case generation

y Execution feedback based test case generation



y OOP Testing: Pacheco, et al. “Feedback-Directed Random Test

y OOP Testing: Pacheco, et al. “Feedback-Directed Random Test

Generation” y GUI Testing: Yuan and Memon. “Using GUI Run-Time State as Feedback to Generate Test Cases”

y GUI Testing: Yuan and Memon. “Using GUI Run-Time State as

Generation” Feedback to Generate Test Cases”



y Conclusion

y Conclusion

Motivations of the paper Using GUI Run-Time State as Feedback to Generate Test Cases

y Previous work y 1-way: Crash Test y 2-way: Smoke Test

y Longer test cases detected

Xun Yuan and Atif Memon ICSE’07

additional faults y Unable to run multi-way

test coverage y 2-way run for months

[TSE’05] Try to prune edge

2

4/14/2008

Key idea

Key idea

y Use GUI states as feedback to identify

y Use GUI states as feedback to identify

“important” edges

“important” edges

y Called Event Semantic Interaction Edges

y Generate new longer test cases covering those edges

y Called Event Semantic Interaction Edges

y Generate new longer test cases covering those edges

ESI

ESI

ESI ESI

Event Semantic Interaction y Heuristic: Two events executed together results differently

than executed in isolation => semantic interaction Select Eclipse

Drag

ESI

Six predicates for modeless windows Predicate 1: ∃w ∈ W; p ∈ P; v ∈ V; v’ ∈V; s.t: ((v ≠ v’) ∧ ((w, p, v) ∈ {S0 ∩ e1(S0) ∩ e2(S0)}) ∧ ((w, p, v’) ∈ e2(e1(S0)))); Predicate 2: ∃w ∈ W; p ∈ P; v ∈ V; v’ ∈V; v’’ ∈V; s t: ((v ≠ v s.t: v’)) ∧ (v (v’ ≠ v v’’)) ∧((w, ∧((w p p, v) ∈ {S0 ∩ e2(S0)}) ∧ ((w, p, v’) ∈ e1(S0)) ∧ ((w, p, v’’) ∈ e2(e1(S0)))); Predicate 3: ∃w ∈ W; p ∈ P; v ∈ V; v’ ∈V; v’’ ∈V; s.t: ((v ≠ v’) ∧ (v’ ≠ v’’) ∧((w, p, v) ∈ {S0 ∩ e1(S0)}) ∧ ((w, p, v’) ∈ e2(S0)) ∧ ((w, p, v’’) ∈ e2(e1(S0))));

Predicate 4: ∃w ∈ W; p ∈ P; v ∈ V; v’ ∈V; v’’ ∈V; v ∈V; s.t: ((v ≠ v’) ∧ (v ≠ v’’) ∧ (v’’ ≠ v ) ∧((w, p, v) ∈ S0) ∧((w, p, v’) ∈ e1(S0)}) ∧ ((w, p, v’’) ∈ e2(S0)}) ∧((w, p, v) ∈ e2(e1(S0)))); Predicate 5: ∃w ∈ W; p ∈ P; v ∈ V; vv’ ∈V; vv’’ ∈V; s.t: ((v’ ≠ v’’) ∧(((w, *, *) ∉ S0) ∧ (((w, p, v) ∈ e1(S0)) ∨ ((w, p, v’) ∈ e2(S0))) ∧ ((w, p, v’’) ∈ e2(e1(S0)))); Predicate 6: ∃w ∈ W; ENABLED ∈ P; TRUE ∈ V; FALSE ∈V; s.t: (((w, ENABLED, FALSE) ∈ S0 ) ∧ ((w, ENABLED, TRUE) ∈ e1(S0)) ∧ EXEC(e2, w));

(More details refer to “A comprehensive framework for testing graphical user interfaces” Atif M. Memon.Ph.D. dissertation, 2001)

Three contexts for events wrt windows

Experiments

y Context 1 : events in modeless window

y Subject applications: three OSS

y Context 2 : events in same modal window y ex(S): the GUI state after executing , x= 1, 2 y e2(e1(S)): the GUI state after executing sequence

y Context 3 : events in parent and child modal window y e1(S): the GUI state after executing y e2(e1(S)): the GUI state after executing sequence

y CrosswordSage 0.3.5 y FreeMind 0.8.0 y GanttProject 2.0.1 y JMSN 0.9.9b2

y Test T t oracle l y Program crashes

3

4/14/2008

Result - Test case reduction

Result - Faults detection

Num ber of F aults

9 8 7 6 5 4 3 2 1 0

2-way

3-way 99.78%

4-way 99.97%

5-way

CrosswordS age

FreeMind

GanttProject

JMS N

99.99%

All-edges Covering Test Cases

3,4,5-way Covering Test Cases

Table 1: Percentages of test case reduction

Lessons learned

Conclusion of this paper

y Event handlers are implemented in multiple classes

y Contributions

y A large input space is needed y Crash because the permutations of events

=> Need longer test cases???

y A new GUI model for test-case generation y A new relationships among GUI events (i.e. context) y A utilization of GUI state as feedback y A fully automatic end-to-end GUI testing process y A demonstration

y Future work y Simplify 6 predicates and 3 contexts y Identify and classify events dominating ESI y Minimize number of test cases y Apply feedback technique to objects outside GUI

What’s next…

What I’m looking at


y Push test case generation and test case execution closer


y Generate new test cases during the execution


y Utilize the feedback immediately

y Conclusion

4

4/14/2008

A case study

Conclusion

y Adaptive test oracles: the QoS idea

y Software is dynamic so we need a dynamic approach

Test Case =

Oracle Information =

S0

e1

e2 S1

… S2

FIND_NEXT

…

y Using feedback in software testing is feasible

… en-1 en

Si-1

Si

y Somewhat related to control theories …

Sn-1

Sn

(i.e. software cybernetic) y Drawback: Like hill climbing

=> local optimization y Can mutants (like in GA) overcome this? y Systematically vs. Randomly

“Using Transient/Persistent Errors to develop Automated Test Oracles for Event-Driven Software” Atif M. Memon and Qing Xie. – ASE’04

Questions y What does “Event Semantic Interaction” in section 4

mean? y What are the threats to validity and what are the

weaknesses in Xun’s experiments?

5