How Modular Fog Servers Advance Industry 4.0
The future of Industry 4.0 systems across all sectors lies in the combination of fog server and real-time technology.
Robots, machines and manufacturing systems that are integrated in Industry 4.0 applications generate an immense volume of communication—both horizontal and vertical. Horizontally, other machines, material handling and also vision systems must be connected and increasingly form a unit. Vertical integration of automated production from batch size 1 is necessary to allow constant monitoring. Additional IoT services, such as big data analytics and predictive maintenance, or simple access to digital user guides and helpdesks—customized for the specific machine—are integrated directly into the system. Access is granted to everyone who needs it—in customized form and, if required, also via the cloud and smartphone apps. Any third-party supplier that needs to be integrated into the overall system gets specific interfaces for Industry 4.0 connectivity as a matter of course. What’s more, the original control task of the individual Industry 4.0 components remains, of course, an essential part of the solution and an important communication task.
However, they are not positioned ‘somewhere’ in the cloud but directly in production and must be able to communicate in real time.
Why Fog Servers Are Key to Connected Applications
Real-time fog servers are tipped to become the key enabler of such connected IoT and Industry 4.0 applications. Their design is similar to that of cloud servers. However, they are not positioned ‘somewhere’ in the cloud but directly in production and must be able to communicate in real time. They also integrate everything that is needed on site: Control, collaboration with other machines and the connected material handling technology, big data capture, and many other customized cloud services—including an interface for software upgrades. They bridge machines and business IT and are essential enablers of vertical integration—if this image is still valid in a fully connected world. Even the GUI of the machine can be connected to the fog server via thin clients and can so reside on the machine itself as well as being accessible for other users at quite different locations. What is ‘up’ and ‘down’, or whether a function is remote or local, is not always clearly defined, hence the beauty of fully connected Industry 4.0 applications.
Deploying tasks on a real-time fog server first and foremost requires a real-time hypervisor that securely separates the different tasks on a single embedded server via virtualization and assigns each component of the ecosystem its own dedicated resources. Second, the virtualized subsystems demand synchronization in real time. In most modern Industry 4.0 applications, this requires real-time communication over Ethernet.
Intra-System Communication in Real Time
The real-time hypervisor of Real-Time Systems GmbH (RTS), which is widely used in industrial automation, provides all the necessary requirements for the development of real-time fog servers that seamlessly connect the real-time control infrastructure with cloud services. It is currently available in version R4.6 and runs on many different platforms, from the Intel Atom processor to the Intel Xeon® processor, and supports all processor generations, including the sixth generation of Intel Core technology. Seventh generation support, launched at the beginning of January, is slated to follow soon.
The RTS real-time hypervisor allows multiple real-time operating systems to run simultaneously, such as Wind River VxWorks, QNX Neutrino RTOS, Microware OS-9, On Time RTOS-32 and T-Kernel, or Windows Embedded Compact. Of course, it is also possible to run a mix of real-time and classic multi-purpose operating systems—like the Microsoft Windows family including Windows 10 IoT and Linux RedHawk—on a single x86 multicore processor. In any case, all virtualized (real-time) systems run separately and can even be rebooted without affecting each other or impacting the operation of the others. One partition, for instance, can be implemented with a Linux firewall specially designed for security tasks to provide additional security. This not only consolidates the hardware, but also provides a complete and, above all, secure subsystem for the IoT connection.
The real-time communication between the individual virtual machines is achieved via a virtual network provided by the RTS hypervisor or, for lowest latency, through shared memory. For this shared memory area, it is possible to assign individual write and read rights to ensure that access is limited to authorized OS partitions, which increases security.
The RTS hypervisor further provides special memory management that allows real-time operating systems to be moved freely within the memory without affecting performance. Thanks to hardware-based virtualization technology users can now freely move guest operating systems. For example, a 32-bit operating system can address memory beyond the theoretical limit of 4GB, or several instances of operating systems, despite linking to the same physical address, can operate simultaneously and deterministically on the same system in hard real-time.
The RTS hypervisor’s security features, such as the rights managements API for starting, stopping or monitoring individual operating systems, suits real-time capable Industry 4.0 fog servers and embedded applications where determinism, hard real-time and/or security play a key role.
Inter-System Communication in Real Time
When real-time capable hypervisor technology for fog servers is paired with real-time communication protocols such as DDS or OPC UA and a deterministic Ethernet with time-sensitive networking (TSN) support, users and fog servers of additional machines can get access to the latest machine data at any time. And this at a scale and speed that was previously impossible. In addition, RTS offers an off-the-shelf master/slave software stack for Windows developed in compliance with the IEEE 1588 Precision Time Protocol specification. This enables highly accurate synchronization of distributed devices over Ethernet with a fog server. The maximum jitter for standard Intel Network Interface Controllers (NICs) is in the high-precision nanosecond range. Such technologies have been available to developers for more than a year as a suitable software and processor basis.
What developers of real-time fog servers have so far been missing is an appropriate, IEEE 1588 compliant hardware platform with which to adapt the Intel® Xeon® D processors—which are ideally suited for high-performance fog servers—as efficiently as possible to the specific application requirements. With the pre-release of the Type 7 pinout in 2016 by the PICMG COM Express specification, in which congatec was actively involved as an editor, this bottleneck has finally been removed. The COM Express Type 7 pinout offers IEEE 1588 compliant, standardized 10GbE interfaces on a standardized embedded module for the first time, defining a new category of embedded computing form factors: Server-on-Modules. The COM Express Type 7 specification also differs from the current COM Express Type 6 specification for classic embedded computer systems in that it removes the need for any video interfaces. Instead, its 32 lanes support significantly more PCIe interfaces that can be used for mass storage, dedicated graphics cards and additional server interfaces. By pairing the real-time hypervisor technology and real-time Ethernet communication with COM Express Basic modules providing 10GbE support (currently 2 ports, up to 4 ports in the future), it is possible to develop extremely compact fog servers that are hardly larger than the 125 x 95 mm Server-on-Modules.
Support for Copper and Fiber Optic Cables
The 10GbE interfaces are designed as 10GBASE-KR single backplane lanes (see also IEEE 802.3 / 49) to avoid being constrained to predefined physical interfaces. The PHY, which defines the physical transmission layer, is not on the module, but must be implemented on the carrier board. Only the implementation on the carrier board defines whether the data is transmitted via copper or fiber optic cables. For even more flexibility, the PHY can be implemented as exchangeable SFP+ modules, which makes it possible to delay the decision whether to transfer via copper or fiber optics to installation on site. Such high bandwidths are required, for example, in industrial image processing applications where full HD or 4K camera signals with high color depths and frame rates are transmitted. It is also possible to combine the performance of multiple 10GbE signals. Four 10GBASE-KR lanes can be bundled into one PHY for 40GBASE-KR4.
To ensure compliance with IEEE 1588, the feature set of the COM Express 10GBASE-KR interfaces also includes a software defined pin for each of the four interfaces. This physical pin can be configured as input or output, and is controlled by the corresponding Ethernet controller. This enables the implementation of a hardware-based timing protocol in accordance with IEEE 1588 for high-performance real-time applications.
In addition, the Server-on-Modules follow the trend towards fast storage media. Since fast storage is best natively connected via Non-Volatile Memory Express (NVMe) interfaces that are based on PCIe technology, the new Server-on-Modules support more PCIe lanes but only two SATA ports. In this context, the new Intel Optane™ Memory deserves a mention. It is based on 3D Xpoint technology, which offers significantly lower latency compared to NAND SSDs but can handle the same size of data packets. With a latency of just 10μs—more than a thousand times smaller than that of standard HDDs—the boundaries between memory and storage become fluid. The carrier boards from congatec already support this fast storage technology, which is particularly suitable for virtualization, data storage, big data processing, medical imaging and many other applications.
A big benefit of the new Server-on-Modules is the option to scale systems across processor socket generations by replacing COM Express modules. This optimally supports performance bandwidth, upgradeability and long-term availability. Another advantage of computer modules is the comprehensive software support, which significantly shortens time-to-market as well as development costs, compared to developing boards from scratch. For example, advanced Server-on-Modules support powerful server-class tools to manage distributed IoT, M2M, and Industry 4.0 applications, matching the needs of devices connected via edge or fog servers. If they further provide server-class remote management technologies and integrated board management controllers with watchdog timers and power loss control on board, the modules also suit remote monitoring, administration and maintenance tasks. The Type 7 modules also integrate an NC-SI interface for the out-of-band management that is required for this purpose. All that needs to be done to realize highly reliable industrial fog servers with out-of-band management is the implementation of a matching board controller on the carrier board.
Now available are platforms based on the COM Express Type 7 specification. They include comprehensive support of the RTS hypervisor and deterministic Ethernet with time-sensitive networking (TSN), are headless designs, and are available in 10 different server processor variants, starting from the 16-core Intel Xeon Processor D-1577 to the Intel Pentium® Processor D-1519 for the industrial temperature range (-40°C to +85°C). On the memory front, the conga-B7XD modules offer up to 48 GB super-fast 2400 DDR4 memory with or without error correction (ECC) depending on the customer’s requirements. The absence of an integrated graphics unit is server typical, but also benefits the real-time capability of the overall system since the determinism of the processor cores can be secured even more easily.
A special feature of the new congatec Server-on-Modules is their high network performance with 2x 10 Gigabit Ethernet. They also support up to 24 PCIe Gen 3.0 lanes and 8x PCIe Gen 2.0 lanes for the connection of powerful system extensions including flash memory.
Conventional storage media can be connected via 2x SATA 6G. Further I/O interfaces include 4x USB 3.0, 4x USB 2.0, LPC (to be replaced by eSPI, as already specified in COMe 3.0), SPI, I2C bus and 2x UART. Operating system support is offered for all common Linux distributions and Microsoft Windows versions, Microsoft Windows 10 IoT included. The support of remote management technologies completes the feature set.
Because the new Server-on-Modules have a power consumption of up to 65 watts, system developers must pay particular attention to efficient cooling, especially with a view to increased reliability and chip life. A good cooling concept also supports the turboboost features of current processors that provide additional computing power. Turboboost allows overclocking the processors only if they remain cool enough. To make thermal management as straightforward as possible for developers, the COM Express specification defines a heat spreader as a thermal interface. Its flat surface is easy to integrate into industrial fog server applications and allows for fast technology upgrades without the need to change the mechanical and/or electrical system architecture. This makes it much easier to keep up with the constantly changing roadmaps of the chip manufacturers. The COM Express specification also defines the I2C bus through which several temperature sensors can be connected for comprehensive system monitoring. However, a specification alone does not make a heat spreader, so consider a supplier who offers standardized heat spreaders and heat sinks to match all the different modules with a view to making the integration of the new Server-on Modules particularly easy for developers. For the evaluation of the new Server-on-Modules, congatec provides an evaluation carrier board. The circuit diagrams and layout are freely available and can be used as a basis for the customer’s own designs.
Shorter Development Time
In addition to this hardware-oriented support, congatec also offers comprehensive firmware and middleware support for IoT applications, due to be expanded at Embedded World with a dedicated embedded IoT API to standardize the IoT interface to the embedded hardware. This makes developing industrial IoT edge and fog server applications even faster. As congatec also offers Embedded Design & Manufacturing Services for its Server-on-Modules, even complete customer-specific system designs can be implemented quickly. Application developers do not have to worry about the details of hardware design. They can immediately test the RTS hypervisor as well as the entire embedded firmware and middleware with the congatec starter kits. The first prototypes, which can be used immediately for first field tests, won’t be long behind.
Dan Demers is the Director of Marketing – Americas at congatec, Inc. He holds a B.B.S degree in International Business from Grand Valley State University, Grand Rapids, Michigan and an M.B.A. from Ashford University, Clinton, Iowa. Mr. Demers has over 19 years of experience in embedded computing having worked with Fortune 500 companies in the Industrial, Medical, and Communications markets.