Scalable Non-Blocking Networks with Fixed Size ... - Semantic Scholar

0 downloads 0 Views 553KB Size Report
Keywords— Nonblocking networks, Clos-networks. Abstract— The generation of non-blocking Clos-networks from custom routing chips is studied. In particular ...
1

Scalable Non-Blocking Networks with Fixed Size Routers Tor Skeie, Olav Lysne and Geir Horn Keywords— Nonblocking networks, Clos-networks Abstract— The generation of non-blocking Clos-networks from custom routing chips is studied. In particular we approach the question of how to construct non-blocking networks with P ports using as few routers as possible. Unlike previous attempts we study this problem for fixed size routers, and for any P , not only those values arising from straightforward recursion of the Clos scheme. We describe a practical way to circumvent the shortcomings of the developed theory for non-blocking networks when the size of the router cannot be manipulated.

I. Introduction In a nonblocking circuit switching network any connection between an idle input port and an idle output port can be realized without interfering with any of the other existing connections. The original application area for such networks was telephone switching, but due to their low blocking probability nonblocking networks have also been used for multiprocessor interconnection [1], and they have been advocated for ATM switching [2]. In 1950 Shannon gave a theoretical lower bound for the number of crosspoints needed in any rearrangeable or nonblocking network with N -inputs and N -outputs [3]. Shannon did, however, not show how such networks should be constructed, and to this end we have seen a large body of research. In 1953 Clos published his seminal paper which presented a construction of strict sense nonblocking networks that subsided previous results significantly in the need of crosspoints [4]. Since then we have seen a series of refinements and analyzes, e.g. [5], [6], [7], [8], [9], [10]. A construction for rearrangeable non-blocking networks that realizes the lower bound given by Shannon was first presented by Margulis [11]. After the ground-breaking work of Clos his result has been extended to different traffic requirements like multirate traffic sources [12], [13]. Likewise there is a body of results that address the issue of path-choosing for new connections, see e.g. [14] and [15]. This paper concentrates on complexity of strict sense non-blocking networks, but our focus is different from previous work in five respects: This work is supported by Esprit under the projects OMI´ and OMI-ARCHES MACRAME Tor Skeie, who will present the paper, is a research fellow at the Department of Informatics, University of Oslo, Box 1080, Blindern, N-0316 OSLO,Norway, E-mail: torsk@ifi.uio.no, Phone: +47 22 06 73 43, Fax: +47 22 85 24 01 Olav Lysne is a researcher at the Department of Informatics, University of Oslo, Box 1080, Blindern, N-0316 OSLO,Norway, E-mail: olavly@ifi.uio.no, Phone: +47 22 85 24 28, Fax: +47 22 85 24 01 Geir Horn is a research scientist at SINTEF, Box 124 Blindern, N0314 Oslo, NORWAY, E-mail: [email protected], Phone: +47 22 06 73 00, Fax: +47 22 06 73 50

Most of the previous work assumes that the networks can be built using crossbars of arbitrary size. In modern digital switching systems, however, the elements of the networks are custom integrated circuits of standard size called routers, thus we focus on the construction of nonblocking networks from fixed size routers. • Previous results only tell you how to build networks of size P for only very few values of P when your router is of fixed size. We aim at answering the question of how to build low complexity nonblocking networks of size P with routers of size p for any P and p. • Instead of using the number of crosspoints as complexity measure we focus on the number of routers that is needed in order to build a network. • Previous work on complexity for Clos-networks present results that are valid as the size of the networks grow towards infinity. This paper focuses on network sizes in the range that they are actually built today. • Most of the work addressing the non-blocking issue, has communication schemes where all the input ports are located in one end-stage and the output ports are found in the other. However, in the communication environment we are investigating the ports can serve both as input and output channels, also concurrently, using bidirectional links. Therefore we do not distinguish between input- and output ports. •

II. Construction steps The general idea behind Clos networks is to organize the routers in three stages, such that between the stages each pair of routers is connected via a link. Strict nonblockingness of such networks may be guaranteed by ensuring that there are sufficiently many midstage routers – if a connection is requested between two ports, there is always a mid stage router that can be reached from both of the ports using idle links. If m is the number of midstage routers, and pe is the number of external ports on each end-stage router, then the requirement for nonblockingness is m ≥ 2pe − 1. In the full paper we prove that the bidirectional communication scheme has no implication on the number of midstage elements needed for the network to be strictly non-blocking. Assume now that the size of the router is 8. This means that each mid stage router can only connect to 8 end stage routers, thus the largest Clos network that can be generated straightforwardly is one with 24 ports using 13 routers, see figure 1. In order to build a nonblocking network with arbitrary many ports, we make use of 5 different construction steps: Recursion: In the above example we built a Clos network of with 24 ports from a router with 8 ports. We may now

2

E1

E2

E1

Fig. 1. An example of a nonblocking Clos-network with routers of size 8.

use recursively use the 24 port network as a building block in another Clos-networks. From now on we shall refer to such building blocks as elements (E). E.g. if we use the 24 port Clos as the midstage element and the 8 port routers as end stage elements (Mid stage Recursion), we can construct a 72 port network (see figure 2). The proof that such recursive networks will also be non-blocking is given by Clos in [4]. Abandoning: From any nonblocking network with P ports we can create one with P − 1 ports by abandoning one of the ports. The result will trivially be nonblocking. Pruning: An end-stage element on which all external ports are abandoned can be removed, and all of the internal links that become external ports by doing this should become abandoned. The result will trivially be nonblocking if the network was nonblocking before the pruning. Midstage Removal: If the number of midstage elements in a (sub)-Clos exceeds 2max(pe ) − 1 where max(pe ) is the maximal number of non-abandoned external links in an end-stage element, then one midstage element can be removed, and all of the internal links that are freed become abandoned. Due to the argument of Clos, the resulting network will be non-blocking. Element-reuse: Assume that we have an element that is not pruned, and has n abandoned ports. Then clearly these n ports can by themselves be seen as the external ports of an isolated non-blocking network. These n ports can therefore replace any element with n or fewer ports in other places of the network. Previous work in this area has only been using Recursion, and as demonstrated above this gives a rather coarse grained scalability. In this paper we demonstrate different strategies using Recursion, Abandoning, Pruning, Midstage Removal and Element reuse in order to build nonblocking networks with any number of ports using as few routers as possible. The issue of path finding at channel setup, and the effect of the different construction steps on the path lengths are left to the full paper.

E1

Fig. 2. This is an illustration of Midstage Recursion. The nonblocking network E2 is built using networks of type E1 as midstage elements.

III. The Strategies In this section we present some strategies for using the above constructions. Each strategy results in a function (R) that returns the number of routers needed for any given number of network ports (P ). As mentioned the Clos networks constructible from fixed size elements without use of Recursion is relatively limited. This limitation can be stated by the following inequalities: m ≥ 2pe − 1 m + pe ≤ p

(1) (2)

Inequality (1) stems from Clos’ result on nonblockingness, and inequality (2) states the obvious fact that the ports of an end-stage element that connects to m midstagerouters and has pe external ports must have at least m + pe ports in total. Solving the above inequalities with some given router size p gives us the number of midstage routers (m) and the number of external input/output ports per end-stage router (pe ), respectively. Let p be the size of the router. Using the fact that each midstage element must connect to all end stage elements, we find that the total number of

3

input/output ports available on a three-stage Clos component is: Ptot = p · pe

(3)

Above we have showed that with p = 8 we obtain a Clos network with 24 ports without Recursion, and one with 72 ports by one application of Midstage Recursion. Alas, Recursion does not answer the question of how to build a nonblocking network with e.g. 32 ports using routers of size 8. As the number of requested ports increase, the answers given by Recursion alone becomes even more sparse. In the following we shall use the five constructions from section II to solve this problem. We propose four strategies, and we shall call the corresponding functions that give number of routers needed as a function of the number of ports P for RI , . . . , RIV respectively. Depending on P and the recursion level these strategies will alternately give the best solution, thus we shall concentrate on the function R given by: R(P ) = min(RI (P ), ..., RIV (P )) A. Strategy I Assume that we need a nonblocking network of size P . Strategy I consists of performing Midstage Recursion until a network is reached with at least P ports. Then ports are abandoned until P is reached, and in such a way that the abandoned links are spread out on the end stage routers as evenly as possible. Then we perform Midstage Removal as many times as possible, and finally element reuse to the extent that abandoned ports on the mid stage elements can be used to implement end stage elements. Below we formulate the strategy in an algorithmic way: 1. Perform Midstage Recursion until we have a network with size greater than P . 2. Abandon ports from the end-stage elements of the network until the correct number of ports is attained, and in such a way that the difference between max(pe ) and min(pe ) is less than or equal to 1. 3. Prune the endstage elements that now have become obsolete (if any). 4. Compute the number of necessary midstage-elements mnew = 2 · max(pe ) − 1, and perform Midstage Removal until there are mnew midstage elements left. 5. Count eventually abandoned ports in the midstage elements. 6. While there are midstage elements that have sufficiently many abandoned ports to implement an end stage element, perform element reuse. 7. While there are endstage elements that have sufficiently many abandoned ports to implement another endstage element, perform element reuse. Correctness of the method is easily verified from the correctness of the construction steps. We may now define RI as the function that takes a number of ports P and returns with the number of routers needed by strategy I.

B. Strategy II In this strategy we extensively reuse components generated earlier by one of the other strategies. Knowing the router size p we may compute the number of external ports on each end stage router from the inequalities (1) and (2). E.g. with a router size of 8, the end stage routers can have at most 3 external ports. From this number we calculate the number of end stage routers needed to implement P ports in a nonblocking network. Having the number of end-stage routers we can compute the needed size of the midstage elements. We now obtain a network with P ports by using the optimum network of needed size generated by any of the other strategies as midstage element. A more precise formulation of strategy II: 1. Compute the number of end-stage routers necessary to implement the P ports network. Notice that this number gives the size, PmidStage, of the midstage elements as well. 2. Obtain the optimal previously computed network which has PmidStage ports, and use this network as the midstage element in a Recursion step, reaching a nonblocking network of size P . Again we define RII to be the function that gives the number of routers needed by strategy II for any P . The correctness of this strategy does also follow trivially from the fact that the networks are built using the constructions from section II. C. Strategy III This strategy has similarities with strategy I, in that it first performs Midstage Recursion, then abandoning and finally element reuse. In this case, however, the abandoning is performed in such a way that all ports on one end stage router are abandoned before any other router is affected. Then the useless endstage routers are pruned, and element reuse is performed on the midstage elements. Strategy III: 1. Perform Midstage Recursion until we have a network with size greater than P . 2. Pick the endstage routers one by one and for each of them abandon all of its external ports until a network with P ports is reached. 3. Perform pruning of all the endstage routers whose external links are all abandoned. 4. As long as one midstage element has enough abandoned ports to implement one of the other midstage elements, perform element reuse on the midstage elements in question. (Note that the number of logical midstage elements needed is not altered, but each physical midstage element can implement several logical ones using its abandoned ports.) 5. As long as one midstage element has enough abandoned ports to implement one of the endstage routers, perform element reuse on the midstage element and the endstage router. Both the function RIII (P ) and correctness of the strategy follow in the same way as before.

4

D. Strategy IV In this strategy we perform Recursion not only in the midstage, but on all stages of a 3-stage Clos-network. We use expressions (1), (2) and (3) to compute the needed size of the standard router in order to reach P ports in one Recursion step. Then the best nonblocking network found by any of the strategies is used in place of this “standard router”. Strategy IV formulated more precisely: 1. Let P = Ptot in equation (3), and solve the expressions (1)-(3) with respect to p, pe and m. 2. Find the optimal network with p ports computed by any of the strategies. 3. Use this network as a building block in a regular three stage Clos structure (three stage Recursion). 4. Abandon/prune in the same way as in strategy III in until the correct number of ports is reached. RIV (P ) is defined as the function that returns the number of routers needed by this strategy to construct a nonblocking network with P ports. Correctness of the strategy follows in the same way as previously.

each of the p! states of the first router. Hence, the total theoretical number of supported states for the network of two routers is p! · p!. Extending this argument to a network with R routers, we get the theoretical maximum on the number of states supported: (p!)R . For the network to be able to support all states inequality (4) must be satisfied (p!)R ≥ P !

(4)

We may solve this inequality with respect to the least number of routers needed to support a given number of ports.

R (P ) ≥ logp! (P !)

(5)

= logp! (P (P − 1)(P − 2) · · · 1) = logp! (P ) + logp! (P − 1) + · · · logp! (1) =

P 

logp! (i)

(6) (7)

i=1

IV. Optimality considerations In order to evaluate the strategies we developed above, we shall compare them with a theoretical lower bound. A similar theoretical lower bound with respect to the number of crosspoints has been developed by Shannon [3], and our method for determining the theoretical lower bound for routers is inspired by his work. The basic idea is to derive the number of routers needed to uniquely represent all possible transmission states attained by the set of ports in the network. First consider a nonblocking network with P bidirectional ports. A state for this network is a set of connections between all P inputs to all of the P outputs such that all inputs and outputs participate in exactly one connection. A port on the network is taken to be passive if its input channel is connected to its output channel; i.e. the state contains a connection of the form a → a where a is the port identifier. Consequently, if we start with the first input channel, say a, it has the full set of P alternative destinations. However, the second input channel cannot select the same output channel as a, and have only P − 1 possible output channels. For the third input channel neither of the two used output channels are selectable which leaves it with only P − 2 possible chooses. Thus for each of the P destination allowed by the first input channel, the next has P − 1 alternative output channels yielding P (P − 1) different combinations. After adding the third connection, we have P (P − 1)(P − 2) possible combinations or states the router may attend. Continuing this argument shows that the network must be capable of supporting P ! different states. By the same argument we have that a router with p ports can support p! different states. The number of states supported by the first router in the network is p!, and adding the second router to the network it allows p! new states for

By letting R∗ (P ) denote the function that gives the least number of routers needed to satisfy equation (4), we easily obtain a recursive relation in this lower bound in equation (9) which gives an increasing increment in the numbers of routers as the number of ports increases.

R∗ (P ) =

P 

logp! (i)

(8)

i=1

R∗ (P + 1) =

P +1 

logp! (i)

i=1

= logp! (P + 1) +

P 

logp! (i)

i=1 ∗

= logp! (P + 1) + R (P )

(9)

Observe that this optimum number of routers, R∗ (P ) is significantly less than the asymptotic values of O (P log(P )) obtained by Shannon and Pippenger for the number of crosspoints [3], [8]. This is evident by noting that each of the element in the sum of equation (6) is less or equal to logp! (P ), thus logp! (P ) + logp! (P − 1) + · · · logp! (1) < logp! (P ) + logp! (P ) + · · · logp! (P ) = P logp! (P ) The optimum number of crosspoints given by Shannon has been reached by a construction for rearrangeable nonblocking networks only [11]. It is therefore doubtful whether the lower bound we have found for routers can be implemented by a construction of nonblocking networks in the strict sense.

5 Best strategy

Routers 80000

Midstage recursion only

Strategy IV 70000 R(P)

60000

Strategy III

50000 40000 30000

Strategy II 20000 10000 Ports 1000

2000

3000

Strategy I Ports

4000

Fig. 3. This figure shows the number of routers needed as a function of the number of ports in a nonblocking network. The “staircase” curve denotes the performance of the Recursion construction that has been studied by previous work, whereas the other curve presents the combined optimum of our four strategies using Abandoning, Pruning, Midstage Removal and Element Reuse as well as Recursion.

1000

2000

3000

4000

Fig. 4. This figure shows at which ranges the various four different strategies outperform the others. Note that in some areas this oscillates between several strategies in so small intervals that the plot is not able to present it. R(P) vs. lower bound 25

V. Performance of the strategies The size of the router is mainly irrelevant to the general results on the performance of the strategies. For practical reasons we have, however, chosen a router size of 8 for the plots we present. This corresponds to the size of e.g. the IEEE 1355 based R-cube router [17]. Consult [16] for details on IEEE 1355. In all the plots of this section we have assumed a router size of 8, but the results we present would have given the same general picture for any chosen router size. In figure 3 we have plotted the combined minimum of our four strategies, i.e.

R(P ) = min(RI (P ), ..., RIV (P )) against a strategy using Midstage Recursion only. As we can see the staircase effect of using Recursion only, has to a great extent been smoothened out by our approach. Therefore we can indeed claim that the straitjacket of having to stick to the network sizes given by Recursion and the size of the given router has been removed. There are, however, still some significant “steps” left that corresponds to the points where another Recursion step is needed. We believe that it should be possible to find other strategies for using our constructions that make these steps even smaller, but it is not likely that one can make them disappear. With a router size of 8 it takes one router to build a network of size 8, whereas we require at least 6 routers to build a network of size 9. In order to demonstrate that all of our strategies outperform the others in some areas we have in figure 4 given a presentation of which strategy is the best in which P interval. Here we can see that strategy IV dominates as the size of the network increase, whereas strategies I, II and III performs the best for low P values. At this point it should be said that the gaps between the different strategies in general is huge, thus there are none of the four strategies alone that performs even close to the combined

20

15

10

5

Ports 1000

2000

3000

4000

Fig. 5. This figure plots the ratio between R(P ) and the theoretical lower bound.

optimum. Whether there actually exist a strategy that in itself is optimal in all intervals does, however, remain an open question. In figure 5 we have plotted the ratio between R(P ) and the theoretical lower bound of equation (8) developed in section IV. As we can see the factor exhibits a staircase effect within the domain we have studied. It is reason to believe that this staircase effect continues as P increases. The question of how to close the gap between the construction and the lower bound remains as a subject of future work. The constructions we have presented are, however, general enough to serve as a basis for a search for better strategies. Furthermore it should be possible to improve the theoretical lower bound by taking more information on the character of nonblocking networks into consideration. VI. Conclusion In this paper we have attacked the problem of how to build strictly nonblocking Clos-like networks of arbitrary size with fixed size routers. The aim has been to use as few routers as possible. We have given a set of construction steps that can be combined in different ways, and presented four different strategies for using them. We have developed a theoretical lower bound for the number of routers needed, and we have compared the performance of our strategy with this lower bound. Our strategies have to a large extent solved the problem

6

of the coarse grained scalability of nonblocking networks, although they might not be optimal. We do, however, believe that the construction steps we have presented are general enough to be used as a basis of further study of this problem. We have concentrated on developing methods for networks that are non-blocking in the strict sense. An interesting path for further research would be to develop similar constructions and strategies for wide-sense nonblocking networks, rearrangeable nonblocking networks and networks that operate in the multirate environment. References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]

[14] [15] [16] [17]

J. Beetem, M. Denneau, and D. Weingarten, “The GF11 supercomputer,” in Proceedings 12th Annnual International Symposium on Computer Architecture, 1985, pp. 108–115. P. Coppo, M. D’Ambrosio, and R. Melen, “Optimal cost/performance design of ATM switches,” IEEE/ACM Transactions on Networking, vol. 1, no. 5, pp. 566–575, 1993. C. E. Shannon, “Memory requirements in a telephone exchange,” Bell Syst. Tech. J., vol. 29, pp. 343–349, 1950. C. Clos, “A study of non-blocking switching networks,” Bell Syst. Tech. J., vol. 32, pp. 406–424, 1953. V. E. Bene˘s, Mathematical Theory of Connecting Networks and Telephone Traffic, Academic Press, New York, 1965. D. G. Cantor, “On non-blocking switching networks,” Networks, vol. 1, pp. 367–377, 1971. A. Bassalygio and M. S. Pinsker, “Complexity of an optimum nonblocking switching network without reconnections,” Problems Inform. Transmission, vol. 9, pp. 64–66, 1973. N. Pippenger, “A new lower bound for the number of switches in rearrangeable networks,” SIAM J. Alg. Disc. Meth., vol. 1, no. 2, pp. 164–167, 1980. N. Pippenger, “Telephone switching networks,” in Proceedings of Symposia in Applied Mathematics, 1982, pp. 101–133. N. Pippenger, “Analysis of a recurrence arising from a construction for nonblocking networks,” SIAM Journal of Discreet Mathematics, vol. 8, no. 2, pp. 322–345, 1995. G. A. Margulis, “Explicit constructions of concentrators,” Problems Inform. Transmission, vol. 9, pp. 325–332, 1973. R. Melen and J. S. Turner, “Nonblocking networks for fast packet switching,” in IEEE Infocom’89, April 24-27, 1989, 1989, pp. 548–557. F. K. Liotopoulos and S. Chalasani, “Semi-rearrangeably nonblocking operation of clos networks in the multirate environment,” IEEE/ACM Transactions on Networking, vol. 4, no. 2, pp. 281–291, Apr. 1996. D. M. Koppelmann and A. Y. Oru¸c, “The complexity of routing in clos permutation networks,” IEEE Transactions on Information Theory, vol. 40, no. 1, pp. 278–285, 1994. S. Arora, F. T. Leighton, and B. M. Maggs, “On-line algorithms for path selection in a nonblocking network,” SIAM Journal of Computing, vol. 25, no. 3, pp. 600–625, 1996. IEEE working group P1355, “IEEE standard for Heterogeneous InterConnect (HIC) (Low cost, low latency scalable serial interconnect for parallel system construction),” 1995. B. Zerrouk, V. Reibaldi, F. Potter, A. Greiner, and A. Derieux, “RCube: A gigabit serial links low latency adaptive router,” in Hot Interconnects Symposium IV, Standford Univ., California, USA, Aug. 1996.