PCI Express Standard Continues to Cover New Ground



While the PCI Express Gen 3 specification is seeing record-breaking up-take in the server space and for graphics-heavy applications, embedded computing experts expect slower adoption for most applications. But there’s still lots of activity around previous generations, and plenty of interesting new developments underway. We talked to experts from a range of companies, including those developing test and measurement equipment, PCI Express switches,  and full lines of embedded boards and systems to get their views. Respondents include John Wiedemeier, senior product marketing manager at LeCroy Corporation; Jeff Much, chief technology officer for Ampro ADLINK Technology, Inc., chair of the AdvancedTCA Subcommittee and interim chairman of the PICMG COM Express Plug-and-Play Subcommittee; Akber Kazmi, product marketing director for PCI Express switches, PLX Technology; and Matt Ferraro, a senior hardware engineer and project manager at Connect Tech Inc. who is leading product development in embedded systems and FPGA computing solutions.

EE Catalog: PCIe Gen 3, with its high bandwidth and optimized power, is seeing fast adoption in the server space and in graphics-heavy applications. What role do you see it taking in embedded applications?

John WiedemeierJohn Wiedemeier, LeCroy: PCI Express 3.0 has much to offer the embedded development community. The specification describes operating link speeds up to 8GT/s, doubling performance of the PCI Express 2.0 specification. New enhanced signaling and data-integrity features can help increase data throughput and performance in embedded applications. However, the trend has always been that embedded applications’ I/O technology adoption lags a couple of years behind high-performance I/O applications’ deployment schedules. This time factor benefits the embedded community with proven I/O silicon solutions at lower prices.

Jeff Much, Ampro ADLINK Technology, Inc.: At this point, PCIe Gen 1 meets the data transfer rates of most embedded applications. There might be future applications in high-end video for the military, but that is down the road.Jeff_Munch_ADLINK

AkberKazmi-PLXAkber Kazmi, PLX Technology: Embedded applications will follow what is being done by server, storage and graphics markets, i.e., take advantage the 2x bandwidth offered by PCI Express Gen 3, while maintaining existing distances and form factors. Signal conditioning such as decision feedback equalization (DFE) and continuous time-linear equalization (CTLE) in PCI Express Gen 3 SerDes (such as what PLX uses) will allow embedded system designs to drive the signal longer distances over backplanes and large blades.

Matt Ferraro Connect TechMatt Ferraro, Connect Tech: Initially PCIe Gen 3 adoption will be slow. Many embedded applications take advantage of processor/chipset combinations which feature a high-end, built-in GPU such as the Nvidia Tegra2 and Texas Instruments OMAP-4. Given the performance of these devices, there will only be a need for an external graphics peripheral or an external graphics ASIC in very high-end graphics applications. PCIe Gen 3 will most likely have a large role in communications and data acquisition; for example where a processor is paired with an FPGA, which has a high-end analog-to-digital conversion front end. Additionally, embedded storage solutions, such as ruggedized SSDs, will take advantage of the higher bandwidth that PCIe Gen 3 provides.

EE Catalog: What are the adoption trends you are seeing in embedded applications for the different generations of PCI Express?

Wiedemeier, LeCroy: It is clear that the PCI Express 3.0 market segment includes high-performance I/O add-in-cards and server system manufacturers. These types of products benefit from the higher throughput and performance that the specification delivers. Other types of market segments are in some cases using PCIe 1.1, 2.0 and 3.0 as product differentiators, especially in the graphics and workstation markets. PCIe 1.0 is still mostly relegated to the embedded community as the replacement technology for PCI and PCIe bridged applications.

Much, Ampro ADLINK: PCIe is being used as a primary interface for FPGA and other types of programmable I/O. The move to PCIe for general I/O is slow – we still see demand for ISA bus. The embedded market moves at a much slower pace.

Kazmi, PLX Technology: Adoption of PCI Express Gen 1 was slow but was much quicker by the VME/VITA forums when Gen 2 was introduced. We are hearing many users of VME and VITA buses want to use PCI Express Gen 3 rather than go to the next-generation of VITA.

Ferraro, Connect Tech: PCI Express is slowly becoming a core feature of several embedded standards, examples of which include PCIe/104, PCI-104/Express and Qseven. While these standards focus mainly on general PCIe support, in practice, single-board computers and computer-on-modules only support Gen 1, often with single lane links. The main advantage to engineers is the relative simplicity of designing with PCIe when compared with PCI or PCI-X; specifically where PCB layout is concerned. Additionally, there is a substantial cost savings in using newer PCIe-enabled silicon versus legacy PCI silicon which is increasingly harder to come by.

EE Catalog: As the PCI-SIG and other companies work on new optional features above and beyond the base standard, what are some of the most interesting new directions you are seeing?

Wiedemeier, LeCroy: The PCI Express I/O virtualization and the address translation services extensions are potentially going to have a big impact on the data enterprise market. This technology is being used to enhance I/O performance by virtualized system consolidation by sharing I/O resources, which can drive down the total cost of ownership. The second direction is the new NVM Express specification for SSD technology. The NVM Express specification was developed by the more than 80 members of the Non-Volatile Memory Host Controller Interface (NVMHCI) Workgroup. The goal was to define an optimized register interface, command set and feature set for high-performance PCI Express-based solid-state drives (SSDs). This interface is scalable and can apply to enterprise and client applications.

Much, Ampro ADLINK: Additional capabilities to determine channel quality are important in determining the performance of a channel.

Kazmi, PLX Technology: PCI Express is now becoming a household name. My son wants to buy the latest PCI Express solid-state drive, not to mention a PCI Express x16 GPU card from Nvidia. Now that the PCI Express interfaces are being offered on x86 CPUs and all the major embedded CPUs, we see a lot more activity in PCI Express at the consumer and embedded levels. Also, new interconnect technology named Thunderbolt (TB), introduced by Intel, utilizes some of the PCI Express features. And with PCI Express embedded inside the TB port, you will a lot more excitement about PCI Express in the consumer space.

Ferraro, Connect Tech: One of the intriguing areas of development is the expanded use of the PCI Express external cabling sub-specification. While this cabling specification has been around for a couple years, it has not been widely adopted. In most cases, the external cabling has been used in conjunction with engineering test and measurement equipment or in manufacturing validation systems. However, we see a potential market for its use as an I/O expansion mechanism. While the electronics world is moving towards the exclusive use of high-speed serial interfaces (PCIe, SATA, USB, XAUI, Infiniband) those interfaces are focused primarily on storage and network infrastructure – not on real-world data acquisition and control.

One of the main challenges for integrators is the physical problem of connecting many channels of high bandwidth analog data to a system. The space required to fit connectors is limited by standard computer or embedded chassis. An interesting solution would be the development of an I/O expansion module; a standalone piece of equipment which would hold the application-specific interface circuitry and connectors (i.e., 16 high-bandwidth ADC channels) and connecting them to the host system via a 16-lane PCI Express cable. Given the size of connectors, the alternative would be 8×1-lane PCIe expansion boards (whether it is standard PCIe or embedded PCIe-104) – which is not practical in any situation.

EE Catalog: What trends are you seeing in the adoption of PCI Express-interfaced solid-state drives (SSDs) in embedded applications?

Wiedemeier, LeCroy: At the 2011 Intel Developer Forum, many SSD manufactures revealed roadmaps showing PCI Express as a potential replacement for existing storage protocols. Features such as high performance (5x the performance of SAS/SATA SSDs) lower power, lower latency and smaller foot print to implement a SSD and others make this a compelling technology.

Much, Ampro ADLINK: None yet

Kazmi, PLX Technology: Recently, the NVMe standard for non-volatile memories was introduced by Intel and other major enterprise-system vendors, using PCI Express as the host interface. New form-factor specifications are being developed to support PCI Express and SATA interfaces on a single connector. This will give a major boost to PCI Express use in SSD subsystems for the embedded space and other applications.

Ferraro, Connect Tech: Currently most designs we are seeing use either an external SATA SSD or an internal SATA SSD ASIC. Many of the embedded processor chipsets, such as the Intel Atom E600 family, provide a direct SATA interface. Given the limited PCIe resources of embedded processors, designers may still choose this approach. However, while the robustness of chip-down ASIC design is essential to many applications, the ability to easily swap out a failing SSD is also a must for many applications. In many cases, having a SATA-to-cable-to-HDD solution is not appropriate for embedded environments, so storages devices that leverage the Mini PCI Express form factor are a great way to add high-capacity storage on a small footprint to any PCI Express system.

EE Catalog: As high-end embedded applications adopt PCI Express on the backplane, how are developers addressing debugging challenges such as probing?

Wiedemeier, LeCroy: Test equipment vendors provide an important role in helping high-end embedded applications. However, connectivity is a major challenge when trying to reach into a backplane and provide access to PCI Express signals. There are three probing approaches used in the case of protocol analysis. The first is the mid-bus probe, which is a special cabled connector that is attached to an industry-specified footprint laid out on the backplane ahead of time. This method is sometimes intrusive to board real estate.

The second probe type is the multi-lead probe, which is configurable from x1 to x16 PCIe lanes depending on the test setup. This probing solution is highly configurable but is very time consuming to set up and remove.

The third probe type is the interposer board, which is a custom PCB board that sits between a card/module that plugs into the I/O connector on the chassis backplane. This interposer provides an almost electrically transparent probing point in the system. The advantage is that the interposer can be inserted or removed without having to worry about placing or removing large numbers of multi-lead probes on the backplane.

Connectivity is an important consideration when purchasing a protocol analysis solution.

Much, Ampro ADLINK: This is where we need additional channel quality information from the silicon. The problem is that PCIe can work with a severely out-of-spec channel. Just because there is a PCIe link between two devices does not mean that the link width and quality of the link are as expected.

Kazmi, PLX Technology: PCI Express switches are becoming such an integral part of the system that they have to provide significant value by helping designers with debug and diagnostic capabilities. Features such as visibility of internal data paths, Rx eye measurements, error injection and error counts are becoming must-haves by major server, storage and embedded OEMs.

Ferraro, Connect Tech: Validation of PCIe can be quite challenging with any design. It is somewhat easier with a standard PCIe Express PC card, where the PCI-SIG compliance boards can be used to qualify a design. Providing the designer has access to the proper high-bandwidth oscilloscope and SMA cables, the setup and analysis is trivial. However, with an embedded design, using the PCI-SIG compliance boards is not possible unless an adapter fixture is available – for example a fixture that will convert PCI Express to PCIe/104. Otherwise the PCB layout engineer must give sufficient consideration to the design-for-test (DFT) methodology. Careful consideration must be given to the placement of measurement (test) points, in order not to adversely affect the signal itself. A common technique is to place a testpoint on a via, which connects to a pad on the PCIe ASIC. Once valid measurement technique is established, debugging can be facilitated through many software add-ins available for oscilloscopes. These add-ins can decode a PCI Express stream and provide detailed information on the PCIe messages contained within.

cheryl_coupe_optCheryl Berglund Coupé is editor of EECatalog. com. Her articles have appeared in EE Times, Electronic Business, Microsoft Embedded Review and Windows Developer’s Journal and she has developed presentations for the Embedded Systems Conference and ICSPAT. She has held a variety of production, technical marketing and writing positions within technology companies and agencies in the Northwest.

Share and Enjoy:
  • Digg
  • Sphinn
  • del.icio.us
  • Facebook
  • Mixx
  • Google
  • TwitThis