A Silicon photonic WDM network for high performance ...

Invited Paper

A Silicon photonic WDM network for high performance macrochip communications *

*

+

+

Xuezhe Zheng, Pranay Koka , Herb Schwetman , Jon Lexau , Ron Ho , Ivan Shubin, John E. Cunningham, and Ashok V. Krishnamoorthy Sun Microsystems, Microelectronics Physical Sciences Center, San Diego, CA 92121, USA +

SUN Laboratories, Menlo Park, CA 94025, USA *

SUN Laboratories, Austin, TX 78727, USA [email protected]

Abstract: We introduce a novel approach to interconnect multiple chips together with a silicon photonic WDM point-to-point network enabled by optical proximity communications to act as a single large piece of logical silicon much larger than a single reticle limit. We call this structure a macrochip. This non-blocking network provides all-to-all low-latency connectivity while maximizing bisection bandwidth, making it ideal for multi-core and multi-processor interconnections. We envision bisection bandwidth up to TBps for an 8x8 macrochip design. And a 5-6x improvement in latency can be achieved when compared to a purely electronic implementation. We also observe better overall performance over other optical network architectures. Keywords: Si photonics, WDM, Point-to-point, Network, Optical proximity communication, Optical interconnect 1. Introduction Since the invention of the integrated circuits in 1958, the number of transistors that can be placed inexpensively on a silicon chip has increased exponentially, described as Moore’s law [1]. Transistors shrinking in size also allow circuits to run faster. With more transistors integrated on chip, running at faster clock rate, designers have been able to improve the system performance effectively over the past few decades. Unfortunately, the design complexity, power consumption, and yield limits associated with this simple transistor count and clock rate scaling are making it more and more difficult to meet the performance scaling demands generated by new applications from military and commercial sectors. Parallelism through multi-core processing has become the technology choice to continue processor performance scaling to avoid hitting power, bandwidth and latency “walls”. Processors with 8 cores and 64 threads are commercially available [2], and an 80core experimental processor has been reported [3]. This trend is expected to continue to hundreds of cores on a chip. However, yield and lithographic reticle limits will eventually constrain the number of processing cores on a single silicon chip. New system topologies containing multiple chips, each with multiple-cores and multi-threading would represent a major advancement. The emerging challenge here is to provide a low-latency, low power, and high bandwidth network to effectively interconnect the multi-chip system. Recent advancements in silicon photonics (SiP) bring the hope of integrating complete, functional photonic systems with electronics using standard VLSI technologies, providing unique advantages in low latency and high bandwidth while maintaining minimal power dissipation [4]. In this work, we introduce a wavelength division multiplexing (WDM) point-to-point networking architecture for multi-processor interconnection enabled by novel optical proximity communication (OPxC) technologies. It offers strictly non-blocking all-to-all connectivity for the processors while maximizing bisection bandwidth. It also provide scalability to interconnect multiple chips together Photonics Packaging, Integration, and Interconnects IX, edited by Alexei L. Glebov, Ray T. Chen, Proc. of SPIE Vol. 7221, 72210E · © 2009 SPIE · CCC code: 0277-786X/09/$18 · doi: 10.1117/12.813420

Proc. of SPIE Vol. 7221 72210E-1

which can thus act as a single large piece of logical silicon with a physical size beyond a single reticle limit. Our envisioned layout of this network is reported within along with a discussion of its physical layer considerations. Finally, we present a performance evaluation of the proposed network architecture. 2. Photonically interconnected Macrochip To scale beyond single reticle limit by integrating many chips together as a logical large chip requires high bandwidth density and low latency I/O for electronic chips. It’s very challenging for traditional electrical signaling methods such as SerDes to meet these requirements because it consumes significant chip real estate and power. Electrical Proximity communication (PxC) using capacitive coupling is a more promising approach [5]. As shown in Figure 1a, capacitive coupling between micropads on chips can be used to achieve low power, high bandwidth density communication without having to escape the package as is typically the case in traditional electrical signaling. Aggregate bandwidth density of fifty GBps/mm2, and power efficiency of 3.6pJ/bit has been reported for PxC [6].

Cbip2 Chip 1

Transmit

Receive

I Trnsmir

Figure 1 Chip to chip electrical proximity communication through aligned micro metal pads.

With capacitive PxC, integration of many chips into a logically continuous piece of silicon larger than the reticle limit can be implemented, as shown in Figure 2. In such systems, electronic chips with PxC interfaces are tiled into a 2D array as island chips, and connected through bridge chips with matching PxC I/Os. Chip-to-chip I/O is through aligned tiny metal PxC pads, effectively extending highly-dense on-chip wires across a chip-to-chip gap. However, by keeping all chip-tochip I/O within on-chip wires, a large multi-chip system would suffer from high message latency: on-chip wires propagate at only 1/10th or 1/20th the speed of light [7]. This will limit the scalability of such a chip grid to a small number of chips, and hence will also limit its performance benefits.


Figure 2 Logical integration of large continuous piece of silicon with electrical proximity communications (PxC) by using island chips and bridge chips.

Optical signaling based on Silicon photonics, on the other hand, could potentially alleviate the off-chip bandwidth bottleneck as well as provide low latency communication. One envisioned implementation is shown as Figure 3. Electronic chips containing processor cores and memories are equipped with Si photonic interface, and arranged into a 2D array. Instead of simply replacing the electrical PxC with optical signaling, two optical routing layers are added to minimize the number of in/out chip couplings between the source and destination chips. The bottom routing layer facilitates the routing waveguides along the row direction, while the top routing layer (strips) houses the routing waveguides going along the column direction. With this two layer routing structure, waveguide crossing can be avoided completely. A sample signal path for the upper left corner chip to communicate with the bottom right corner chip is shown in Figure 3 with the red-arrow lines. Essentially, the source chip sends out the data optically via its photonic interface to the bottom routing layer first. The optical data is then routed by the waveguides to its destination column, coupled to the top routing layer waveguide, further routed to its destination chip, coupled back to the bottom routing layer for transition, and finally coupled back and received by the destination chip via its photonic interface. Taking advantage of this system-wide optical interconnect with low latency optical links, array of chips can be integrated into a logically-contiguous piece of silicon. We call this structure a macrochip.


Optical proximity coupling

Chip with cor caches

Si photonic inter1ce Slllcc'n r'I".tng layer 2 Integrated Si veguides

Silicon routing layer I

Figure 3 Macrochip enabled by Si photonic interconnects.

3. OPxC and Macrochip interconnection network One of the key components needed to enable the optically interconnected macrochip is the OPxCs that connects the chips with photonic interface to the bottom routing layer, and the bottom routing layer to the top routing layer respectively. So essentially what OPxC does is to optically couple waveguide signals between two silicon chips placed face to face. One OPxC method is to use grating couplers [8]. With waveguide gratings on both chips facing each other, light from one chip can be coupled directly into the other chip when they are accurately aligned. A different OPxC approach uses reflecting pits and employs a pair of parallel reflecting mirrors to transfer optical data signals in one chip into another parallel chip. As shown in Figure 5a, a silicon waveguide that carriers an optical signal on a SOI chip is terminated at an etch pit created by a silicon micromachining technique. One side of this etch-pit has a straight wall facet to terminate the waveguide to air, while the other side is a tilted surface. The tilt angle is the facet angle of 54.7 degrees, which appears when etching through a (100) surface to the (111) plane. The surface of this tilted facet is Silicon Substrate Al coated mirror 54 7

Si Waveguide

\547.

Si waveguide

Si Waveguide

3 Gap

54

BOX

Silicon Substrate (a)

Silicon Substrate

(b)

Figure 4: OPxC with reflecting pit.. (a) Silicon chip (SOI) with waveguide terminated by reflecting pit. (b) Two SOI chips are placed face to face, and interconnected by reflecting pit OPxC.


metal coated and hence act as a reflecting mirror. With a second chip equipped with a similar structure and the two chips placed face to face, light from one chip can be coupled to the other, as shown in Figure 4b.

ftdupI (a)

(b)

Figure 5 (a) Chip layout with waveguides and OPxC reflecting pits integrated. (b) SEM image of the reflecting pit integration with waveguide.

We fabricated such OPxC structure on silicon on insulator (SOI) substrate that monolithically integrated a reflecting pit to waveguides on the SOI platform as has been previously engineered for a variety of application including turning light in waveguides towards the surface [11-18]. Single mode transport waveguides were built to have ridge widths of 8μm and slab heights of 6μm on an SOI wafer consisting of a 12 micron thick silicon layer above a 0.4 micron box. The transport waveguide ridge width is tapered from 8μm to 13μm and then terminated onto a reflecting pit. The reflecting facets are etched, followed by Al metal coating the sidewall of the reflecting pit. An antireflection dielectric coating was deployed to minimize back-reflections to the waveguides. On the chip in Fig. 5a the center waveguides contain the reflecting mirror. On the opposing side of the chip the waveguides were terminated for coupling to a fiber array. The waveguides features above and below the reflecting pit waveguides in Figure 5a correspond to different experiments that are not reported here. The waveguides in Figure 5a are 1.3 cm long. Fig. 5b shows an SEM of the integrated reflecting pit and waveguide. The footprint of the reflectors is about 20x7 square microns.


2ø?

£.4d

143i.1Sfl.

-

8:IT

-SSZ

1J4

r 31 06:37 :F1