How Ethernet Will Get to 400Gbps
The IEEE 802.3bs standard for 400Gbps is on track to be ratified and released late this year. Part of the secret is PAM4, a clever encoding method that instantly doubles throughput.
Ethernet is now on target to see 400Gbps ratification by the end of the year, using fiber optics. Over the past three to four years, the IEEE 802.3 (Ethernet) group has been energetically innovating, working on as many as 16 projects at one point. Ethernet works. Over 70 billion meters of Cat5e and 6 were sold between 2000 and 2014, and with so many projects in the works, Ethernet is about to burst onto the market with an unprecedented number of solutions.
The IEEE 802.3bs standard for 400Gbps is on track to be ratified and released late this year. Higher speed technologies tend to get driven to adoption as soon as they are available. In 2004, 10Gbps was the leading edge. In 2010 40Gbps Ethernet and 100Gbps were introduced. How did we get this far, so fast? The present group is leveraging a parallel lane structure to get to 400Gbps. For electrical interfaces the fastest speeds in the spec will be 50Gbps. When discussing optical fiber transmission, then the variation depends on the distance that one requires.
John D’Ambrosia, Chair of the IEEE 802.3bs 400GbE Task Force, says, “The decision in 2007 to break away from doing these leaps in increments of just 10x has been the motivation, the driver for success thus far. So, with the 40Gbps project back in 2010, and some other revelations, Ethernet started introducing new speeds.” The 50 Gbps serializer/deserializer (SERDES) rate became the basis for developing 400G, as 8 channels of 50Gbps gives you an aggregate of 400Gbps.
Technically, 400Gbps is not possible without switching away from non-return-to-zero modulation (also known as NRZ-type) encoding, the encoding scheme that everyone thinks of when they visualize Ethernet communication and other serial data transmission schemes. NRZ data is encoded into a binary pattern with fixed voltage levels. A binary 0 is represented by the lower voltage level; the higher voltage level indicates binary 1. In 1000base-T Ethernet, the stream of 0s and 1s is driven at a 1000 bits per second (1Gbps) transmission rate. At present, the physical “wall” of streaming 0s and 1s for single lane electrical interfaces is 25 Gbps, found in the standards as 802.3bj across backplanes and cables, and 802.3bm across chip-to-chip and chip-to-module interfaces. In May 2016, an IEEE 802.3 task force formed to develop a single-lane 50 Gbps Ethernet standard. The 802.3bs standard, which defines 400Gbps in aggregate, will use an encoding scheme called PAM4 (4-Level Pulse Amplitude Modulation) to reach 50Gbps per channel. PAM4 is an encoding scheme that doubles the bit rate by providing four signal levels in the space of the two that NRZ presently provides. PAM4 cleverly divides the least significant bit (LSB) signal level in half and adds it to the signal of the most significant bit (MSB).
However, there are several options for delivering 400Gbps that allow implementers to choose based on distance requirements and whether they want to use multi-mode fiber (MMF) or single mode fiber (SMF).
PAM4 has actually been around for a while. There are some older physical layer specs for Ethernet that do use PAM already, but not in the high-speed space that it is leveraged for today. PAM4 is now leveraged for 50G, across the electrical interfaces as well as single mode fiber interfaces.
The use of PAM4 is not ubiquitous in 803.2bs, however; it depends on what is used to conduct the signal. In the case of electrical interfaces, there are two versions: the first version is 16 lanes of 25Gbps that use NRZ signaling. The industry has invested in 25Gbps signaling infrastructure that will be in place for some time, and the 16 x 25G interface leverages that infrastructure. An 8 x 50G PAM4 variant is also available to deliver the aggregate of 400Gbps. According D’Ambrosia, “You will see a pretty spectacular demo on 400G from the Ethernet Alliance,” referring to an upcoming announcement on 400G.
Table 1 shows the variants for 802.3bs. For fiber interfaces, the new 802.3bs spec to be released in December 2017 includes four modes of operation for 400Gbps Ethernet. The first variant includes 16 lanes of 25Gbps transmitting in each direction for a total of 32 fibers over 100 meters. At the time that 400GBASE-SR16 was conceived, Vertical-Cavity Surface-Emitting Lasers (VCSELs), the basic technology for multimode fiber, weren’t ready to run 50Gbps PAM4. Thus, the 16 x 25Gbps approach was taken. The 500-meter 400GBASE-DR4 variant uses four parallel single mode fibers (SMF) in each direction, and each fiber is running a 100Gbps PAM4 signal. The 400GBASE-FR8 port type uses Wavelength-Division Multiplexing (WDM ). Based on 8×50G, 400GBASE-FR8 will take the signal as far as 2 kilometers away. The 10km version,400GBASE-LR8, is also a WDM solution, based on 8×50G. The 400GBASE-DR4 variant is not a WDM solution.
Another variation in the upcoming release covers 200Gbps speeds (see Table 2). The 200Gbps options include three distances serviced with single mode fiber; 500 meters, 2 kilometers, and 10 kilometers. The 200Gbps and 400Gbps optical interface specs as well as the electrical 400Gbps and 200Gbps interface specs (based on 25Gbps and 50Gbps signaling) are all planned for release in Dec 2017.
Why so many variants when they all accomplish 400G?
The main reason why there are so many variants in the 802.3bs standard is due to the many different markets that are served, such as hyperscale data centers, carriers, and enterprise data centers. Some applications benefit from using single mode fiber whereas others do better with multi-mode fiber, so both are served. The application spaces that would be interested in a 200Gbps variant are the most cost-sensitive, as there are fewer lanes required to support a lower bandwidth of 200Gbps versus 400Gbps. Not all applications require top speeds, nevertheless, as D’Ambrosia indicated, “People are anticipating it in much the same way that it was with 40Gbps and 100Gbps. And perhaps the best way to put it is that we allow the industry to gain from its investment by having multiple opportunities to leverage that technology, thus there’s more than one way to arrive at the same destination. There may be a potential cost savings with 200Gbps over 400Gbps that some are anticipating.”
Existing infrastructure has to be a part of the equation, since the entirety of the world’s existing cabling cannot be uprooted and replaced all at once. “The big message here is that we are moving very quickly here in the Ethernet world, introducing tons of new projects, at one point we had 16 projects going on simultaneously. And a lot of this is leveraging common technology across multiple projects,” says D’Ambrosia.
The 50 Gbps rate may have started out as the basis for 400G, however, 50Gbps signaling itself was then leveraged across other efforts, as a number of decisions were made by the 802.3 working group that led to generating additional projects. With the 400Gbps project underway, the additional projects have been able to leverage a lot of the technologies and architectures that were originally created as a result of the 802.3bs project (the project for 400G). Thus, over the course of the next 18 months, Ethernet will introduce 50Gbps for a number of different physical layers. The public will see 100Gbps and new variants based on 100G, as well as off-shoots for 200Gbps and 400Gbps. All these technologies are simultaneously hitting the market, leading to the internal nickname of “the Next Ethernet Era” within the Ethernet community, because it is perhaps the largest build-out the group has ever seen.
D’Ambrosia is a native New Yorker, an Italian, and talks easily about the technology and the possibilities of Ethernet. Having been involved with the Ethernet Alliance for over 20 years, D’Ambrosia, who is also the Chairman of the Board of Directors for the Ethernet Alliance, points out that not everyone is a speed demon. The Ethernet community has members working anywhere from DC up to 400Gbps and in the future, 800Gbps. Plenty of Ethernet applications exist with alternate and evolving requirements that focus on size, weight, distance, and cost over speed. Automotive requirements for Ethernet can focus more on the number of connections rather than lightning speed. Industrial, Power over Ethernet (PoE), 2.5Gbps, and 5Gbps all have a place in the Ethernet community.
All these technologies are simultaneously hitting the market, leading to the internal nickname of “the Next Ethernet Era” as dubbed by the Ethernet Alliance, because it is perhaps the largest build-out the group has ever seen.
States D’Ambrosia, “The world is fascinating now from the Ethernet perspective. We have big application spaces that are well-funded across the entire gambit of Ethernet. Ethernet is embracing applications now. Fifteen or twenty years ago, it was: ‘Here’s an Ethernet solution, how do we make it work in this application space?’ Now it’s, ‘Here’s an application space, how do we make Ethernet work in it?’ It’s been amazing for me to watch this evolve over my career.”
Lynnette Reese is Editor-in-Chief, Embedded Intel Solutions and Embedded Systems Engineering, and has been working in various roles as an electrical engineer for over two decades. She is interested in open source software and hardware, the maker movement, and in increasing the number of women working in STEM so she has a greater chance of talking about something other than football at the water cooler.