0.12 Dsd_chapter 5_supplement_optical tera bus

DSD_Chapter 5_ Supplement_Optical Fiber Inteconnects- A Novel Way of overcoming von-Neumann Bottlenecks.

[“Get on the Optical Bus”, IBM’s light powered links overcome the greatest speed bump in supercomputing: interconnect bandwidth, by Clint Schow, Fuad Doany and Jeffrey Kash, September 2010, IEEE SPECTRUM, pp 30-35]

As already discussed, in the classical von-Neumann Architecture there is a very serious bottleneck in the availability of data for processing. The origin of this problem is the metallic interconnects which act as Data Bus as well as Instruction bus. As the Clock Speed go up, the problem is being compounded. In giga-Hz clock range, metallic interconnects face skin effect and eddy current losses. Skin effect leads to excessive resistance of the conducting wires and consequent high heat dissipation. Also oscillating signal on the buses at very high frequency induces stray eddy current in the board’s conducting part leading to heavy eddy current energy losses. With increase in Clock Rate, attenuation along the copper buses increases exponentially. At 2 GHz clock there is 50% attenuation and at 10GHz there is 98% attenuation. At few GHz several resonances occur which cause signal to be reflected at VIAS. Vias are vertical conductors that connect two level components on Printed Circuit Board(PCB).

Apart from this there is severe cross-talk among the metallic interconnects leading to excessive bit-error rate (BER). At 10 gigabits per second bit rates, cross talk blurs the signal after 1 meter of propogation down the copper bus. With the increase in processing speed the problem of attenuation and cross-talk is becoming more acute. Several methods were adopted to overcome the bandwidth bottleneck problem. The foremost method amongst these is storing the current data in Cache Memory. Multigigahertz Clock rate has been achieved within the microprocessor between processing core and on-chip cache memories but the data exchange between the chip and external components ( say external memories) has been one order of magnitude slower.

This bandwidth gap between the processor and the buses will continue to widen as processor performance improves with scaling and improved processor architecture.

The problems of signal-loss and cross-talk is particularly severe in super-computers( massively parallel machine) where we put together multi-chip modules together. Here the modules may be at two ends of the PCB or may be in different racks of equipment.

Optical buses removes the problems of signal-loss, cross-talk problems and bandwidth bottleneck and make the assembly of a supercomputer technically viable. Programming become simpler in supercomputers using optically powered links because severe communication delays among processors donot have to be compensated while writing the programs.

On 25 ^th September 1956 TAT-1, transoceanic voice-grade coaxial cables were laid down and trans-Atlantic telephone calls became possible for the first time.

In 1988, TAT-8 optical fiber cables were laid across the bottom of Atlantic Ocean for trans-oceanic voice-video-data cable transmission.

By 1990s, fiber-optic links were being used in local area networks (LANS) and in Storage Area Networks, interconnecting systems hundreds of meter apart. Optically linked LANS made Video-conferencing possible within a campus.

In 2003 and 2004, fastest supercomputers were NEC Earth Simulator(5,120 processors running at a cost of 17.5GFLOPS per million dollars) and IBM’s Blue Gene L(65,536 processors with 10TFlOPS capability at a cost of 2.8TFLOPS per million dollars).Their peak performances reached 36 TFLOPS capability using metallic buses.

In 2008 IBM introduced RoadRunner using 40,000 optical fiber link buses. This reached Peta FLOPS capability. Peta means 10 ¹⁵ . Hence Road runner had almost 2 orders of magnitude of performance improvement over that of Blue Gene L.

In IBM Blue Waters machine 1 million optical interconnects are being used. This is expected to reach 10 PFLOPS capability. To achieve 1000-PFLOPS we will need 400 million optical links.

Judging from past performance we can say that Supercomputer processing rate will increase ten-fold every four years.

Projected growth is shown in Table 1.

Table 1. Growth in calculation capability of Suprcomputers.

TERA BUS Program was launched in 2003 as a collaborative program of IBM and Agilent.

The architecture of TERA BUS is given in Figure 1.

TERA BUS is essentially optical polymer waveguides interconnecting any number of modules of the supercomputer. The module of the supercomputer contains an opto-chip which interfaces the electrical processor to optical TERA Bus. The back-side of the opto-chip receives

Figure 1. Architecture of TERA BUS using optical fiber links.

the optical signals from the optical BUS and launches the optical signals into the optical BUS. The opto-chip is shown in Figure 1. At the front side of the opto-chip is CMOS Transceiver which receives electrical signals from and transmits electrical signals to the Processor of the parallel machine.

In future we see that electrical connections will supply the power to the processor, it will provide the ground and it will provide the control signals. Data will always move in and out of the processor as photons and they will travel at the speed of light down the TERA BUS connecting the numerous processors.