Controlled Random Tests - Springer Link

ISSN 0005-1179, Automation and Remote Control, 2012, Vol. 73, No. 10, pp. 1704–1714. © Pleiades Publishing, Ltd., 2012. Original Russian Text © S.V. Yarmolik, V.N. Yarmolik, 2012, published in Avtomatika i Telemekhanika, 2012, No. 10, pp. 142–165.

SAFETY, PERSISTENCE, RELIABILITY, TECHNICAL DIAGNOSTICS

Controlled Random Tests S. V. Yarmolik and V. N. Yarmolik Belarussian State University of Informatics and Radioelectronics, Minsk, Belarus Received February 7, 2011

Abstract—Random testing and its different modifications are considered. Some criteria of controlled generation of random tests are analyzed; an optimal controlled random test is synthesized on the basis of the above criteria. Finally, a universal efficiency estimate is suggested for controlled random tests, with an application to controlled random tests of small length. DOI: 10.1134/S0005117912100104

1. INTRODUCTION An indubitable benefit of trivial tests [1] used to implement exhaustive testing consists in their maximal efficiency for detecting failures of digital devices and software errors [1–3]. However, due to considerable time complexity of such tests, engineers involve their different approximations, in the first place, pseudoexhaustive tests [4, 5]. The latter represent a real alternative to exhaustive tests and proceed from generating a set of test patterns T (N, k) ensuring 2k feasible bit combinations for any k from N inputs of a tested object. A specific feature of pseudoexhaustive tests T (N, k) is that the number of their test patterns R(T (N, k)) appears substantially smaller than the number of test patterns of a trivial test. For instance, consider the test T (6, 2) = {000000, 000011, 011100, 101101, 110110, 111011} studied in [6]; the number of test patterns R(T (6, 2)) makes up 6, thus being appreciably smaller than the corresponding quantity for a trivial test (under N = 6, it constitutes 2N = 26 = 64). In the general case, the number of test patterns R(T (N, k)) in pseudoexhaustive tests T (N, k) is estimated by the inequality 2k ≤ R(T (N, k)) ≤ 2N [3]. Random testing, as well as its numerous modifications based on the black box principle, provide another approximation of exhaustive testing [7–17]. All modifications of random testing are united by the controllability principle for test pattern generation. Indeed, many types of random testing such as antirandom testing [8], fast antirandom (FAR) testing [9], adaptive random testing [10–12], good random testing [13], mirror random testing [14], orderly random testing [15], evolutionary random testing [16] (and other similar types of testing) evaluate certain characteristics of controlled generation of a next random test pattern. The necessity of going over potential candidates for test patterns and computing their characteristics significantly increases computational complexity of generation of controlled random tests (as against classical random tests, see [8–16]). 2. ANALYSIS OF CONTROLLED RANDOM TESTS In the sequel, we understand controlled random testing as constructing random test sequences, where a next test pattern is generated taking into account previously formed patterns. A key difference of controlled generation of random test patterns consists in that information is extracted from previously generated patterns in the form of some characteristics (metrics) and used to form a next test pattern [8–16]. Let us introduce the following definition of controlled random testing methods (note such testing serves for checking of digital devices and software). 1704

CONTROLLED RANDOM TESTS

1705

Definition 1. A controlled random test CRT = {T0 , T1 , T2 , . . . , Tq−1 } is a test T which includes randomly generated test patterns Ti = ti,N −1 ti,N −2 . . . ti,2 ti,1 ti,0 , where ti,l ∈ {0, 1}, i ∈ {0, 1, 2, . . . , q − 1} and N means the size of a pattern (in bits), such that Ti satisfies some criterion or criteria obtained on the basis of previous test patterns T0 , T1 , T2 , . . . , Ti−1 . Generally, a test T represents a certain set of input patterns Ti meeting a given testing quality criterion. In the case of controlled random tests CRT, the criteria used to include a next test pattern in a test is a characteristic obtained on the basis of previous patterns [8–16]. As a rule, a next test pattern Ti of a controlled random test is generated to be maximally differing from previously formed patterns. Here one accepts the hypothesis that the number of identified failures (errors) is minimal for two test patterns having the minimal difference. And vice versa, the error identification capacity reaches its maximum for maximally different test patterns [8–16]. The Hamming distance and Cartesian distance [8–10] often act as difference measures for a test pattern Ti being compared with previous patterns T0 , T1 , T2 , . . . , Ti−1 . These measures apply to binary test patterns Ti and Tj . The Hamming distance HD(Ti , Tj ) is computed as the weight w(Ti ⊕ Tj ) of the vector Ti ⊕ Tj according to the formula HD(Ti , Tj ) = w(Ti ⊕ Tj ) =

N −1

(ti,l ⊕ tj,l ).

(1)

l=0

To estimate the difference between two test patterns Ti = ti,N −1 ti,N −2 . . . ti,2 ti,1 ti,0 and Tj = tj,N −1 tj,N −2 . . . tj,2 tj,1tj,0, one defines the characteristics Szg (Ti , Tj ) for i = j ∈ {0, 1, 2, . . . , q − 1} and z, g ∈ {0, 1}. Notably, they are evaluated as the number of paired components (tik , tjk ), k ∈ {0, 1, 2, . . . , N − 1} in test patterns Ti and Tj such that tik = z and tjk = g. There exist four characteristics Szg (Ti , Tj ), viz., S00 (Ti , Tj ), S01 (Ti , Tj ), S10 (Ti , Tj ), and S11 (Ti , Tj ) (they are widely used for identifying the difference or similarity of binary vectors [18]). We underline that HD(Ti , Tj ) = S01 (Ti , Tj ) + S10 (Ti , Tj ). The Cartesian distance CD (Ti , Tj ) is given by

CD(Ti , Tj ) =

HD(Ti , Tj ).

(2)

Example. Consider Ti = 0000 and Tj = 1100; consequently, S00 = 2, S √01 = 2, S10 = 0, and S11 = 0. This yields HD(Ti , Tj ) = S01 + S10 = 2 + 0 = 2 and CD (Ti , Tj ) = 2 = 1.414. To form a set Ti , where i > 2, one uses the total values of the distances between Ti and previous patterns T0 , T1 , T2 , . . . , Ti−1 [8, 9]. Thus, for a next pattern Ti , the total value of the distances with respect to T0 , T1 , T2 , . . . , Ti−1 , constitutes THD(Ti ) =

i−1

HD(Ti , Tj );

TCD(Ti ) =

j=0

i−1

CD(Ti , Tj ).

(3)

j=0

Here THD(Ti ) and TCD(Ti ) stand for the total Hamming distance and total Cartesian distance, respectively [8, 9]. With an attempt to overview the existing modifications of controlled random tests, we should emphasize the uniformity of their procedures used to generate a next test pattern Ti [8–16]. All above methods find a (globally or locally) best solution based on elementary metrics (distances) involving the greedy algorithm (i.e., accepting locally optimal solutions at each stage). Using the considered distance metrics and following the design procedure for controlled random tests, let us synthesize an optimal controlled random test OCRT . At step 1, one involves the black AUTOMATION AND REMOTE CONTROL

Vol. 73

No. 10

2012

1706

S.V. YARMOLIK, V.N. YARMOLIK

box principle to generate a random test pattern T0 consisting of N bits. Without loss of generality, suppose that T0 = 000 . . . 0. According to the above-stated procedures [8–16], the optimal value of the second test pattern T1 is the inverse value T0 , i.e., T1 = T0 . Consequently, the optimal controlled random test which includes two (q = 2) patterns takes the form OCRT = {T0 , T1 }, where T1 = T0 . For T0 = 000 . . . 0 one has the pattern T1 = T 0 = 000 . . . 0 = 111 . . . 1. The optimal choice of the second test pattern T1 as the inversion of the first one (T0 ) is confirmed by maximal values√of all the metrics (1)–(3). Indeed, HD(T0 , T1 ) = THD(T1 ) = N , and CD(T0 , T1 ) = TCD(T1 ) = N . The characteristics HD(Ti , Tj ) and CD(Ti , Tj ) appear inadmissible to derive the third test pattern T2 of the test OCRT. The reason lies in the fact that maximizing their values (HD(T2 , T1 ) and CD (T2 , T1 )) yields a contradiction, viz., T2 = T0 . Similar conclusion applies to the metrics THD(Ti ). For OCRT = {T0 , T1 , T2 }, where T1 = T0 , any third test pattern T2 enables maximization of this characteristic. Really, for arbitrary T2 : THD(T2 ) = N . The presented analysis leads to the following outcome. A candidate for the third pattern T2 of the test OCRT may be a test pattern T2 satisfying the conditions T2 = T0 and T2 = T1 . Assume that T2 is chosen such that the characteristic HD(T1 , T2 ) = N − HD(T0 , T2 ). The HD(Ti , Tj ) possesses the value HD(T0 , T2 ); consequently, characteristic TCD (Ti ) = CD(T0 , T2 ) + CD(T1 , T2 ) = HD(T0 , T2 ) + N − HD(T0 , T2 ), while the HD(T0 , T2 ) = N/2 representing a solution to maximal value TCD(Ti ) is attained for the quantity H(T0 , T2 ) + N − H(T0 , T2 ) /∂ (H(T0 , T2 )) = 0. the equation ∂ Recall the example discussed above and suppose that N is even. Then the optimal test pattern T2 for the vectors T0 = 000 . . . 0000 . . . 0 and T1 = 111 . . . 1111 . . . 1 is the one consisting of N/2 bits having value 0 and of N/2 bits having value 1; rewriting it formally, T2 = 000 . . . 0111 . . . 1. Therefore, under q = 3 one obtains OCRT = {T0 , T1 , T2 } = {000. . . 0000. . . 0, 111. . . 1111. . . 1, 000. . . 0111. . . 1}. Note that synthesizing an optimal controlled test OCRT = {T0 , T1 , . . . , Tq−1 } by the greedy algorithm gives a rather trivial test, which includes q = 2(log 2 N + 1) test patterns and satisfies the condition HD(Ti , Tj ) ≥ N/2, where i = j ∈ {0, 1, . . . , q − 1}. For instance, in the case N = 4 we have OCRT = {0000, 1111, 0011, 1100, 0101, 1010}. A similar result easily follows from the classical binary search algorithm [19] (being generalized to the case of an arbitrary integer number N ). Meanwhile, OCRT can be constructed by applying well-known generation methods for controlled random tests (e.g., the method of antirandom test generation [8], the procedure of fast antirandom test formation [9], the procedure of orderly random test formation [15] or their numerous modifications). We emphasize that OCRT is the best result of the mentioned methods (it can be obtained along with other nonoptimal solutions). The major disadvantage of all controlled random tests is their bounded length q; actually, the latter decreases as requirements to such sequences get tougher. For instance, the larger is the minimal distance minCD(Ti , Tj ) (used as a criterion to include Ti in a test), the smaller is the number of candidates Q for the test. This is immediate from the Hamming √ bound [20]. In the cases minHD(Ti , Tj ) = 2r + 1, where r is an integer, and minCD(Ti , Tj ) = 2r + 1, this bound can be expressed by the inequality 2N . Q≤ r l CN

(4)

i−0

Under N =√15 and minHD(Ti , Tj ) = 5 = 2 × 2 + 1, the Cartesian distance meets the inequality 28 feasible candidates for tests with CD (Ti , Tj ) ≥ 5. Then according to (4) there exist at most √ the Cartesian distance CD (Ti , Tj ), which is above or equal to 5. Taking into account the inverse values, the maximal length of √ the controlled random test does not exceed q = 29 . In fact, increasing the Cartesian distance up to 7 would reduce q down to 26 . AUTOMATION AND REMOTE CONTROL

Vol. 73

No. 10

2012


1707

3. UNIVERSAL METRIC FOR CONTROLLED RANDOM TESTS A pseudoexhaustive test T (N, k) is known to exhaustively cover all possible k from N subspaces of an N -dimensional space [2, 3]. For instance, the test T (4, 2) = {0000, 0111, 1011, 1101, 1110} forms all possible binary combinations (2k = 22 ) for all k = 2 from N = 4 bits of test patterns Ti = ti,3 ti,2 ti,1 ti,0 . This fact follows from Table 1.

x00x x11x x01x x10x xxxx

0x0x 0x1x 1x1x 1x0x xxxx

00xx 01xx 10xx 11xx xxxx

0x00 0x11 1x11 1x01 1x10

00x0 01x1 10x1 11x1 11x0

3 ti ,2 t i,1

3 ti ,2 t i,0

3 ti ,1 t i,0

2 ti ,1 t i,0

x000 x111 x011 x101 x110

ti,

0xx0 0xx1 1xx1 xxxx 1xx0

ti,

x0x0 x1x1 x0x1 xxxx x1x0

ti,

xx00 xx11 xxxx xx01 xx10

ti,

3 ti ,2

ti,

3 ti ,1

ti,

2 ti ,1

ti,

3 ti ,0

2 ti ,0

1 ti ,0

ti,

0000 0111 1011 1101 1110

ti,

T0 T1 T2 T3 T4

ti,

ti,

Ti

3 ti ,2 t i,1 t i,0

Table 1. The estimation of covering completeness degree for the test T (4,2)

000x 011x 101x 110x 111x

On the other hand, for k = 3 the test T (4, 2) generates 5 from 2k = 23 = 8 binary combinations (for k = 4, it generates the same number of combinations from 16 possible ones, see Table 1). By its definition, a pseudoexhaustive test T (N, k) guarantees covering completeness degree of 100% for a specific value k < N (i.e., it forms all 2k binary combinations for all possible k from N inputs of a tested object) and covering completeness degree of 100% for all possible r ≤ k from its N inputs [3, 4, 6]. In the case r > k, the efficiency E(T (N, k), r) of a pseudoexhaustive test T (N, k) (in percentage) can be estimated as the number of combinations C(T (N, k), r) from r bits, being formed by the test over all possible r from N bits of test patterns (inputs of a tested object), divided by the total number of possible combinations. Notably, E(T (N, k), r) =

C(T (N, k), r) 100%. r 2r CN

(5)

Let us get back to the example in Table 1. Formula (5) gives E(T (4, 2), 3) = ((5 + 5 + 5 + 5)/(8 × 4))100% = 62.5% and E(T (4, 2), 4) = 5/(16 × 1)100% = 31.25%. On the other part, E(T (4, 2), 1) = E(T (4, 2), 2) = 100%. The efficiency index (5) of a pseudoexhaustive test T (N, k) may serve for assessing the efficiency of controlled random tests. Such characteristic is introduced for a next test pattern Ti and a certain test including CRT. Definition 2. Consider a next test pattern Ti (i ∈ {1, . . . , q − 1}) in a test T = {T0 , T1 , T2 , . . . , Tq−1 }. The metric E(Ti , r) of the test pattern Ti is the additional number of binary combinations over all possible r from N bits, being generated by this test pattern, with respect to binary combinations that have been generated by the previous test patterns T0 , T1 , T2 , . . . , Ti−1 . Definition 3. The metric E(T, r) of a test T = {T0 , T1 , T2 , . . . , Tq−1 } is the number of binary combinations over all possible r from N bits, being generated by the test T . The above-described metrics act as quality criteria of pseudoexhaustive testing [3, 4, 6]. By analogy to (5), the both metrics, E(Ti , r) and E(T, r), may have their weighted estimates in percentage representation, WE (Ti , r) and WE (T, r), respectively. For instance, take a random test RT consisting of q random patterns RT0 , RT1 , . . . , RTq−1 . By supposing that r N and using Definition 3, AUTOMATION AND REMOTE CONTROL

Vol. 73

No. 10

2012

1708


we obtain [17]

WE (RT, r) = 1 − 1 −

1 2r

q

100%.

(6)

For a next (actually, ith) test pattern of RT , the metric WE (RTi , r) is defined by

WE (RTi , r) =

1 1− 1− r 2 1 = r 2

i+1

1 100% − 1 − 1 − r 2

1 1− r 2

i

100% (7)

i

100%.

The values of WE (RT, r) for small r are combined in Table 2. Table 2. Numerical values of WE (RT, r) for the random test RT r T0 T0 ,T1 T0 ,T1 ,T2 ,T3 T0 , . . . , T5 T0 , . . . , T7 T0 , . . . , T9 T0 , . . . , T11 T0 , . . . , T13 2 25.0 43.7 68.3 82.2 89.9 94.3 96.8 98.2 41.3 55.1 65.6 73.7 79.8 84.5 3 12.5 23.4 4 6.25 12.1 22.7 32.1 40.3 47.6 53.9 59.4 11.9 17.3 22.4 27.2 31.7 35.8 5 3.12 6.15

Analyzing the expressions (6) and (7) and numerical values (see Table 2) indicates the following. The maximal covering completeness degree of the random test RT (in fact, 100%) is attained only as q → ∞. Under fixed values of q, covering completeness degree of the random test can approach 100% only for small values of r [17]. As against RT , controlled random tests CRT enable increasing the efficiency of forming various binary combinations over any r from N bits of generated test patterns. Consider the case of optimal controlled random tests (OCRT ) and assume that N 2 N and r N . Consequently, one can show that WE (OCRT , r) is evaluated using the formula ⎛

WE (OCRT , r) = ⎝1 −

2r−1 − 1 2r−1

q/2 ⎞ ⎠ 100%;

q = 2, 4, 6, . . . .

(8)

The presented expression follows from the estimates derived in [2, 21] for the case when OCRT consists of an even number q = 2(log2 N + 1) of patterns and includes successive inverse pairs of patterns Ti = T i−1 , i ∈ {1, 3, 5, . . . , q − 1}. The weighted number of additional binary combinations WE (OCRT (Ti−1 , Ti ), r) for any r from N bits, generated by a pair of test patterns Ti−1 , Ti of the OCRT , is given by ⎛

WE (OCRT (Ti−1 , Ti ), r) = ⎝1 − ⎛

− ⎝1 −

2r−1 − 1 2r−1

(i−1)/2 ⎞

2r−1 − 1 2r−1 1

⎠ 100% = 2r−1

(i+1)/2 ⎞ ⎠ 100%

2r−1 − 1 2r−1

(9)

(i−1)/2

100%.

Evidently, due to the inverse relation between Ti and Ti−1 (i ∈ {1, 3, 5, . . . , q − 1}), one obtains 1 WE (OCRT (Ti ), r) = WE (OCRT (Ti−1 ), r) = r 2

2r−1 − 1 2r−1

(i−1)/2

AUTOMATION AND REMOTE CONTROL

100%. Vol. 73

No. 10

(10) 2012


1709

% 7 6 5

k=2

4

k=3

3

k=4

2

k=5

1 0 –1

1

2

3

4

5

6

i

7

8

9

10

11

12

The comparative efficiency estimate WE of the test OCRT against the test RT .

Table 3 provides numerical values of WE (OCRT , r). For q ∈ {1, 3, 5, . . .}, the comparative efficiency of the OCRT against the RT is estimated by the quantity (11), whose numerical values are illustrated in the figure. 1 WE = WE (OCRT q , r) − WE (RTq , r) = r 2

1−

(q−1)/2

1 2r−1

1 − 1− r 2

q

100%.

(11)

The above results demonstrate high efficiency of the test OCRT (in comparison with the random test RT ) merely for small values of k and tests with a small number q of test patterns (see the data in Tables 2, 3 and the figure). Therefore, the analysis leads to the following conclusion. Controlled Table 3. Numerical values of WE (OCRT , r) for OCRT with q = 2, 4, . . . , 14 r T0 ,T1 T0 ,T1 ,T2 ,T3 T0 , . . . , T5 T0 , . . . , T7 T0 , . . . , T9 T0 , . . . , T11 T0 , . . . , T13 2 50.0 75.00 87.5 93.75 96.87 98.43 99.2 43.75 57.8 68.35 76.26 82.20 86.65 3 25.0 23.43 33.0 41.4 48.71 55.1 60.73 4 12.5 12.10 17.6 22.8 27.6 32.1 36.35 5 6.25

random tests possess limited advantages over traditional random tests. Indeed, in the figure one can see that OCRT excels RT only for small q (not exceeding 10). Note that OCRT is the best optimal controlled random test. 4. OPTIMAL CONTROLLED RANDOM TESTS OF SMALL LENGTH The previous section has demonstrated the following. The characteristics E(Ti , r) and E(T, r) (similarly to their weighted percentage-type analogs WE (Ti , r) and WE (T, r)) enable deriving a unique estimate of the efficiency of each candidate for controlled random tests. Nevertheless, in comparison with metrics used for well-known controlled random tests [8–16], the universal characteristics E(Ti , r) and E(T, r) require higher amounts of computations (for each candidate for tests). Only in the case of a fixed structure of controlled random tests (e.g., RT and OCRT ) one succeeds in obtaining analytical estimates for the above-mentioned characteristics. In Section 2 we have noted that OCRT satisfies the inequality HD(Ti , Tj ) ≥ N/2. Consequently, in the case of OCRT we have minHD(Ti , Tj ) = N/2. At the same time, this condition fails for RT . Hence, it AUTOMATION AND REMOTE CONTROL

Vol. 73

No. 10

2012

1710


seems natural to suppose that increasing the efficiency of controlled random tests requires maximizing minHD(Ti , Tj ) (under the assumption that an increase in minHD(Ti , Tj ) reduces the length q of CRT ). Reaching the maximal value minHD(Ti , Tj ) also maximizes the characteristics THD(Ti ) and TCD (Ti ) (3), as well as their sums THD(T ) and TCD(T ) [8–10]. To generate optimal controlled random tests of small length (their number of patterns q do not exceed 6; denote such tests by SOCRT ), let us first consider codes with the maximal minimal Hamming distance [20]. Plotkin’s theorem defines the maximal possible number q of code words in a binary code of a given length N under a minimal code distance d. Moreover, the Plotkin bound yields the upper limit for this number [20]. Theorem 1 (the Plotkin bound). If d ≥ N/2, then q meets the inequality q≤

⎧ ⎨

2d for 2d − N > 0 2d − N ⎩ 4d for 2d − N = 0.

(12)

In the case 2d − N >0, the following formula holds true: d ≤ qN/(2(q − 1)).

(13)

Based on Theorem 1, introduce a new characteristic to construct SOCRT (q, d) = {T0 , T1 , T2 , . . . , Tq−1 } with a bounded number q of test patterns and a bounded distance d. Definition 4. The minimal Hamming distance minHD(Ti , Tj ) is the minimal distance between any two patterns Ti and Tj of the test SOCRT = {T0 , T1 , T2 , . . . , Tq−1 }. Consider the test SOCRT = {T0 , T1 , T2 , . . . , Tq−1 } with the maximal minimal distance (max minHD(Ti , Tj )) between any two test patterns Ti and Tj ; a higher number of binary combinations in arbitrary sets from k < N bits of test vectors is definitely formed in this test. The expression (13) enables determining the maximal minimal distance d = max minHD(Ti , Tj ) for different values q ∈ {2, 3, . . .}. Under q = 2, formula (13) implies max minHD(Ti , Tj ) ≤ N . Recall the previous section; to construct OCRT for q = 2, we have adopted the inversion of the first pattern T0 as the second test pattern T1 (i.e., T1 = T0 ). In this case, the maximal value HD(T0 , T1 ) = N is achieved and SOCRT (2, N ) = OCRT = {T0 , T 0 }. Under q = 3, formula (13) yields max minHD(Ti , Tj ) ≤ 3N/4. The closest optimal solution can be obtained only for max minHD(Ti , Tj ) = 2N/3, where 2N/3 < 3N/4. Let us show that, if q = 3, the quantity max minHD(Ti , Tj ) would note exceed 2N/3. Theorem 2. For SOCRT = {T0 , T1 , T2 } one obtains max minHD(Ti , Tj ) = 2N/3. The test SOCRT (3, 2N/3) consisting of three patterns takes the form T0 = t0,N −1 , t0,N −2 , . . . , t0,2N/3 , t0,2N/3−1 , . . . , t0,N/3 , t0,N/3−1 , . . . , t0,2 , t0,1 , t0,0 ; T1 = t1,N −1 , t1,N −2 , . . . , t1,2N/3 , t1,2N/3−1 , . . . , t1,N/3 , t0,N/3−1 , . . . , t1,2 , t1,1 , t1,0 ;

(14)

T2 = t2,N −1 , t2,N −2 , . . . , t2,2N/3 , t2,2N/3−1 , . . . , t2,N/3 , t2,N/3−1 , . . . , t2,2 , t2,1 , t2,0 . For N = 6, the corresponding SOCRT (3, 4) = {T0 , T1 , T2 } = {000000, 111100, 001111}. Next, in the case q = 4, formula (13) leads to max minHD(Ti , Tj ) ≤ 2N/3. The optimal test SOCRT (3, 2N/3) has max minHD(Ti , Tj ) = 2N/3; actually, this follows from Theorem 3. Theorem 3. For SOCRT = {T0 , T1 , T2 , T3 }, one obtains max minHD(Ti , Tj ) = 2N/3. Proofs of Theorems 2–3 could be found in the Appendix. AUTOMATION AND REMOTE CONTROL

Vol. 73

No. 10

2012


1711

The test SOCRT (4, 2N/3) has the form T0 = t0,N −1 , t0,N −2 , . . . , t0,2N/3 , t0,2N/3−1 , . . . , t0,N/3 , t0,N/3−1 , . . . , t0,2 , t0,1 , t0,0 ; T1 = t1,N −1 , t1,N −2 , . . . , t1,2N/3 , t1,2N/3−1 , . . . , t1,N/3 , t0,N/3−1 , . . . , t1,2 , t1,1 , t1,0 ; T2 = t2,N −1 , t2,N −2 , . . . , t2,2N/3 , t2,2N/3−1 , . . . , t2,N/3 , t2,N/3−1 , . . . , t2,2 , t2,1 , t2,0 ;

(15)

T3 = t3,N −1 , t3,N −2 , . . . , t3,2N/3 , t3,2N/3−1 , . . . , t3,N/3 , t3,N/3−1 , . . . , t3,2 , t3,1 , t3,0 . For N = 6, we get SOCRT (4, 4) = {T0 , T1 , T2 , T3 } = {000000, 111100, 001111, 110011}. An important aspect should be emphasized here. The examples of SOCRT (4, 4) and SOCRT (3, 4) are not unique solutions in the case N = 6. In addition, under q = 4 there may exist other solutions for the values max minHD(Ti , Tj ) being smaller than 2N/3, but also close to the optimal values defined by (13). For instance, one of such solutions is the test SOCRT (4, 5N/8) with max minHD(Ti , Tj ) = 5N/8 < 2N/3, described by the following expressions: T0 = t0,N −1 , t0,N −2 , . . . , t0,3N/8 , t0,3N/8−1 , . . . , t0,3N/8 , t0,3N/8−1 , . . . , t0,1 , t0,0 ; T1 = t1,N −1 , t1,N −2 , . . . , t1,3N/8 , t1,3N/8−1 , . . . , t1,2 , t1,1 , t1,0 ; T2 = t2,N −1 , t2,N −2 , . . . , t2,5N/8 , t2,5N/8−1 , . . . , t2,2 , t2,1 , t2,0 ;

(16)

T3 = t3,N −1 , t3,N −2 , . . . , t3,5N/8 , t3,5N/8−1 , . . . , t3,3N/8 , t3,3N/8−1 , . . . , t3,1 , t3,0 . Due to HD(T0 , T1 ) = HD(T0 , T2 ) = HD(T1 , T3 ) = HD(T2 , T3 ) = 5N/8 and HD(T0 , T3 ) = HD(T1 , T2 ) = 6N/8, we obtain max minHD(Ti , Tj ) = 5N/8 for (16). Note that implementing the stated test requires divisibility of N by 8. Evidently, this appears more practical in comparison with divisibility of N by 3 (the latter is necessary to use (14) and (15)). Possible examples of SOCRT (4, 5N/8) (16) are SOCRT (4, 5) = {T0 , T1 , T2 , T3 } = {00000000, 00011111, 11111000, 11100111} and SOCRT (4, 10) = {T0 , T1 , T2 , T3 } = {0000000000000000, 0000001111111111, 1111111111000000, 1111110000111111} for N = 8 and N = 16, respectively. The above couple of examples (15) and (16), SOCRT (4, 2N/3) and SOCRT (4, 5N/8), are apparently the best solutions that correspond to the conditions of Theorem 1 and formula (13) in the case q = 4. At the same time, if q = 5, formula (13) gives max minHD(Ti , Tj ) ≤ 5N/8; still, the best solution is obviously the test T0 = t0,N −1 , t0,N −2 , . . . , t0,3N/7 , t0,3N/7−1 , . . . , t0,N/3+1 , t0,N/3 , . . . , t0,1 , t0,0 ; T1 = t1,N −1 , t1,N −2 , . . . , t1,3N/7 , t1,3N/7−1 , . . . , t1,2 , t1,1 , t1,0 ; T2 = t2,N −1 , t2,N −2 , . . . , t2,4N/7 , t2,4N/7−1 , . . . , t2,2 , t2,1 , t2,0 ; T3 = t3,N −1 , t3,N −2 , . . . , t3,5N/7 , t3,5N/7−1 , . . . , t3,2N/7 , t3,2N/7−1 , . . . , t3,1 , t3,0 ;

(17)

T4 = t4,N −1 , t4,N −2 , . . . , t4,6N/7 , t4,6N/7−1 , . . . , t4,5N/7 , t4,5N/7−1 , . . . . . . , t4,4N/7 , t4,4N/7−1 , . . . , t4,3N/7 , t4,3N/7−1 , . . . , t4,2N/7 , t4,2N/7−1 , . . . . . . , t4,N/7 , t4,N/7−1 , . . . , t1,0 . For the test SOCRT (5, 4N/7) = {T0 , T1 , T2 , T3 , T4 }, the quantity max minHD(Ti , Tj ) = 4N/7 < 5N/8. An example of the test SOCRT (5, 4N/7) under N = 7 is SOCRT (5, 4) = {T0 , T1 , T2 , T3 , T4 } = {0000000, 1111000, 0001111, 1100011, 1010101}. By analogy, one can extend the construction procedure of tests SOCRT (q, d) for small values of q. However, in each concrete case, a corresponding test represents a heuristic result to-be-studied in a greater detail. AUTOMATION AND REMOTE CONTROL

Vol. 73

No. 10

2012

1712


5. EFFICIENCY ANALYSIS FOR CONTROLLED RANDOM TESTS The major hypothesis of the present paper states that optimal controlled random tests of small length SOCRT , possessing the maximal value of the minimal Hamming distance (max minHD(Ti , Tj )), are characterized by higher covering completeness degrees. For verifying the hypothesis experimentally, we have studied pattern-sensitive failures P P SF k of RAM in the case k = 3 [22, 23]. To develop a software model of bit-oriented RAM with the capacity of N bits, various P P SF 3 have been generated; using MATS+ of similar tests, we have tested RAM for different SOCRT under q = 3 and q = 4. Table 4 illustrates the results obtained for four SOCRT consisting of three patterns (q = 3). For each SOCRT , we have evaluated covering completeness degree for P P SF 3 (as the percentage of the number of failures detected by the test P P SF 3 in the total number of failures in RAM with the capacity of N = 8 bits). Table 4. Covering completeness degree of failures P P SF 3 for tests SOCRT (q = 3) SOCRT {00000000, 11111000, 00011111} {00000000, 11111111, 11110000} {00000000, 11110000, 00001111} {00000000, 11111111, 00000001}

minHD(Ti , Tj ) 5 4 4 1

F C(P P SF 3) 37.05 35.71 35.71 28.68

According to this table, the maximal covering completeness degree F C(P P SF 3) for patternsensitive failures is attained in the case of test, where the value minHD(Ti , Tj ) achieves its maximum. Table 5 combines the results for four tests SOCRT consisting of four patterns (q = 4) that have been used in testing of RAM with the capacity of N = 9 bits. Table 5. Covering completeness degree of failures P P SF 3 for tests SOCRT (q = 4) SOCRT minHD (Ti , Tj ) {000000000, 111111000, 000111111, 111000111} 6 {000000000, 111111111, 000000111, 110000011} 3 {000000000, 111111111, 000000111, 000001100} 2 {000000000, 000001111, 000000111, 000000001} 1

F C(P P SF 3) 49.1 42.3 40.2 28.1

Evidently, the maximal covering completeness degree F C(P P SF 3) for pattern-sensitive failures is ensured by the test SOCRT (4, 2N/3) (see formula (15) in the previous section) with the maximal possible value of minHD(Ti , Tj ), i.e., 2N/3 = 2 × 9/3 = 6. 6. CONCLUSION This paper has analyzed controlled random tests and shown the identity of their procedures used to generate a next test pattern. By means of the given procedure, an optimal controlled random test has been synthesized; moreover, its advantage has been established in the case of small test patterns. Finally, the synthesis problem for optimal tests of small length has been considered and their efficiency has been verified experimentally (testing of pattern-sensitive failures of RAM has served as an example). AUTOMATION AND REMOTE CONTROL

Vol. 73

No. 10

2012


1713

APPENDIX Proof of Theorem 2. Let us provide a reductio ad absurdum proof. Suppose that for the integer 2N/3 : max minHD(Ti , Tj ) = 2N/3 + Δ, where Δ > 0, i.e., max minHD(Ti , Tj ) > 2N/3. Consequently, for SOCRT = {T0 , T1 , T2 } the distances HD(T0 , T1 ), HD(T0 , T2 ), and HD(T1 , T2 ) between the test patterns must be above or equal to 2N/3 + Δ. Assume that the distance HD(T0 , T1 ) = S01 (T0 , T1 ) + S10 (T0 , T1 ) = 2N/3 + Δ; hence, S00 (T0 , T1 ) + S11 (T0 , T1 ) = N − 2N/3 − Δ = N/3 − Δ. The maximal difference between the third pattern T2 and the first one T0 (and the second one T1 ) is achieved when the former varies from the latter in positions with identical symbols 0 or 1, viz., in S00 (T0 , T1 ) + S11 (T0 , T1 ) = N/3 − Δ positions. Then HD(T0 , T2 ) = N/3 − Δ + Q and, taking into account that S01 (T0 , T1 ) + S10 (T0 , T1 ) = 2N/3 + Δ, one obtains HD(T1 , T2 ) = N/3 − Δ + (2N/3 + Δ − Q) = N − Q. To meet the inequality HD(T0 , T2 ) ≥ 2N/3 + Δ, the value of Q must satisfy the formula Q ≥ N/3 + 2Δ. In this case, HD(T1 , T2 ) = N − Q ≤ N − (N/3 + 2Δ) = 2N/3 − 2Δ. This quantity is smaller than 2N/3 + Δ, and the assumption that max minHD(Ti , Tj ) > 2N/3 is wrong. Proof of Theorem 3. According to Theorem 2, for q = 3 we have max minHD(Ti , Tj ) = 2N/3. As the result, it is impossible to derive max minHD(Ti , Tj ) > 2N/3 for the test SOCRT consisting of q = 4 test patterns T0 , T1 , T2 , and T3 . This means that the best solution for SOCRT with the patterns T0 , T1 , T2 , and T3 lies in three corresponding patterns of SOCRT (3, 2N/3) and the pattern T3 being equidistant from the first three ones T0 , T1 , and T2 (the distance equals HD(T0 , T3 ) = HD(T1 , T3 ) = HD(T2 , T3 ) = 2N/3). In the case of SOCRT (3, 2N/3), T0 and T1 possess S01 (T0 , T1 ) + S10 (T0 , T1 ) = 2N/3 different bits. The third pattern T2 is formed using half of these bits from the pattern T0 and another half of bits from T1 , as well as using the inversion of all S00 (T0 , T1 ) + S11 (T0 , T1 ) = N/3 identical bits for T0 and T1 . The fourth pattern T3 is constructed in the following way. From S01 (T0 , T1 ) + S10 (T0 , T1 ) = 2N/3 different bits of patterns T0 and T1 one chooses other halves (as compared with T2 ) and the inversion of S00 (T0 , T1 ) + S11 (T0 , T1 ) = N/3 bits from the patterns T0 and T1 . The described generation procedure for T2 and T3 makes it possible to conclude that in S01 (T0 , T1 ) + S10 (T0 , T1 ) = 2N/3 positions with different bits for T0 and T1 the patterns T2 and T3 have inverse values of bits. And so, HD(T2 , T3 ) = 2N/3, which yields HD(T0 , T1 ) = HD(T0 , T2 ) = HD(T0 , T3 ) = HD(T1 , T2 ) = HD(T1 , T3 ) = HD(T2 , T3 ) = 2N/3. REFERENCES 1. Kulyamin, V.V., Combinatorics of Words and Construction of Test Sequences, Tr. Inst. Sist. Program. Ross. Akad. Nauk, 2004, no. 8(1), pp. 25–40. 2. Das, D. and Karpovsky, M.G., Exhaustive and Near-Exhaustive Memory Testing Techniques and Their BIST Implementations, J. Electron. Test., 1997, vol. 10, pp. 215–229. 3. Tang, D.T., Iterative Exhaustive Pattern Generation for Logic Testing, IBM J. Res. Develop., 1984, vol. 28, no. 2, pp. 212–219. 4. Hartman, A. and Raskin, L., Problems and Algorithms for Covering Arrays, Discret. Math., 2004, no. 284, pp. 149–156. 5. Yarmolik, S.V. and Yarmolik, V.N., The Repeated Nondestructive March Tests with Variable Address Sequences, Autom. Remote Control, 2007, vol. 68, no. 4, pp. 688–698. 6. Karpovsky, M.G. and Yarmolik, V.N., Transparent Memory BIST, Proc. IEEE Workshop on Memory Technology Design and Testing, San Jose, 1994, pp. 106–111. 7. Malaiya, Y.K., Mayrhauser, A., and Srimani, P.K., The Coverage Problem for Random Testing, Proc. ITC, Philadelphia, 1984, pp. 237–242. 8. Wu, S.H., Jandhyala, S., Malaiya, Y.K., and Jayasumana, A.P., Antirandom Testing: A Distance-based Approach, VLSI Design, 2008, no. 2, pp. 1–9. AUTOMATION AND REMOTE CONTROL

Vol. 73

No. 10

2012

1714


9. Mayrhause, A., Chen, T., Hajjar, A., Bai, A., and Anderson, C., Fast Antirandom (FAR) Test Generation, Proc. 3rd IEEE Int. High-Assurance Syst. Eng. Symp., 1998, pp. 262–269. 10. Zhou, Z.Q., Using Coverage Information to Guide Test Case Selection in Adaptive Random Testing, Proc. 34 IEEE Comput. Software Appl. Conf., 2010, pp. 208–213. 11. Chen, T.Y., Leung, H., and Mak, I.K., Adaptive Random Testing, Proc. 9 Asian Comput. Sci. Conf. (ASIAN 2004), 2004, pp. 320–329. 12. Jiang, B., Zhang, Z., Chang, W.K., and Tse, T.H., Adaptive Random Test Case Prioritization, Proc. IEEE/ACM Int. Conf. Automat. Software Eng., 2009, pp. 233–244. 13. Chan, K.P., Chen, T.Y., and Towey, D., Good Random Testing, Proc. 9 Ada-Eur. Int. Conf. Reliable Software Technolog. (LNCS), 2004, pp. 200–212. 14. Kuo, F.C., An In-depth Study of Mirror Adaptive Random Testing, Proc. 14th Eur. Conf. Software Quality, 2009, pp. 51–58. 15. Shiyi, Xu., Orderly Random Testing for Both Hardware and Software, Proc. Pacific Rim Int. Sympos. Dependable Comput., 2008, pp. 160–167. 16. Tappenden, A.F. and Miller, J., A Novel Evolutionary Approach for Adaptive Random Testing, IEEE Trans. Reliabil., 2009, vol. 58, no. 4, pp. 619–632. 17. Yarmolik, S.V. and Yarmolik, V.N., Iterative Almost Pseudoexhaustive Random Tests, Informatika, 2010, no. 2(26), pp. 66–75. 18. Tubbs, J.D., A Note on Binary Template Matching, Patt. Rec., 1989, vol. 22, no. 4, pp. 359–366. 19. Knuth, D.E., The Art of Computer Programming, vol. 3: Sorting and Searching, Reading, Massachusetts: Addison-Wesley, 1998, 2nd ed. 20. Peterson, W.W. and Weldon, E.J., Error-correction Codes, Cambridge, Massachusetts: MIT Press, 1972. 21. Yarmolik, S.V., Address Sequences and Backgrounds with Different Hamming Distance for Multiple Run March Tests, IEEE Int. J. Appl. Math. Comput. Sci., 2008, vol. 18, no. 3, pp. 329–339. 22. Goor, A.J., Testing Semiconductor Memories: Theory and Practice, Chichester: Wiley, 1991. 23. Yarmolik, S.V., Zankovich, A.P., and Ivanyuk, A.A., Marshevye testy dlya samotestirovaniya OZU (March Tests for RAM Self-testing), Yarmolik, V.N., Ed., Minsk: Beloruss. Gos. Univ., 2009.

This paper was recommended for publication by P.P. Parkhomenko, a member of the Editorial Board

AUTOMATION AND REMOTE CONTROL

Vol. 73

No. 10

2012