Accelerating OS Bring-up And Software Debug across the Spectrum of Electronics Systems

Insights from a virtual platform methodologies case study: OS bring-up and testing on an Altera Cyclone V SoC FPGA (with Arm Cortex-A9).

What do applications such as advanced driver assistance systems (ADAS) and autonomous vehicles, IoT, networking, industrial automation, security, video analytics, and machine learning have in common?

They often depend on heterogeneous multiprocessor SoCs, with multiple homogeneous or heterogeneous clusters of CPUs, GPUs, and/or hardware accelerator units that work together on a common set of data. These architectures help optimize the resources on the SoC for different tasks, e.g. an application processor plus “minion” processors for power management, communication management, etc.

The popular Linux operating system for embedded systems, and other operating systems, must be modified for these heterogeneous multi-cluster architectures to support coherence between clusters, as well as differences such as number of processors, processor type, and other features. Vendors often use an open source Linux or other OS distribution, then customize for the specific SoC, including peripheral drivers, other customizations, and unique features. Such customization gets quite complex, and the need to port, customize and bring up the OS on these heterogeneous SoCs requires significant engineering effort.

So how can developers be more efficient? What are the best practices for OS/ Linux porting and bring up on heterogeneous multi-cluster/multiprocessor SoCs?

As software complexity is increasing exponentially, companies must adopt better ways to address problems, as eventually the existing methods will no longer be sufficient. And, one serious failure changes everything for your business and your career. One lesson to be learned from SoC design and verification:  A structured methodology makes execution predictable and reduces risk, benefits that argue for a more formalized approach within the embedded software development domain.

Software Challenges Posed by Heterogeneous SoC System Architectures
Processor IP companies have now developed processor configurations that are similar enough to allow a common Linux OS to run. Originally this was ARM big.LITTLE, with quad core Cortex-A15 plus quad core Cortex-A7, for power optimization. These different processors did not run simultaneously; but the operating system switched automatically between the quad core processors depending on application load. While big.LITTLE was originally architected as equal numbers of big and LITTLE cores, there are now several big.LITTLE architectures, as Figure 1 indicates.

Figure 1: (image courtesy ARM)

Recently Imagination Technologies MIPS I6500 and ARM DynamIQ extended the heterogeneous compute paradigm, enabling fully configurable architectures. Multiple heterogeneous cores per cluster and clusters per SoC mean Linux and architecture support for more than1,000 computing elements.

Figure 2: (Image Courtesy Imagination Technologies)

The OS complexity increases with full heterogeneity. During boot, Linux needs to probe each core to determine characteristics so that correct hardware routines are installed. Also, some Linux modules need to be updated. For example, cache initialization/handling previously assumed homogeneous cache size but now needs to accommodate potentially different cache sizes for different cores/clusters.

Comparison of Hardware-based and Virtual Platform-based Methodologies
Traditional hardware-based software development has timing/cycle accuracy; JTAG-based debug and trace. Traditional breadboard/emulation based testing suffers from limited physical system availability, external test access (controllability) and internal visibility. To get around these limitations, software must be modified (for example, debug versions of OS kernels, and instrumentation for specific analytical tools, e.g. code coverage, profiling). This modified software may not have the same behavior as clean (un-instrumented) source code.

Virtual platform-based software development offers instruction-accurate simulation and runs actual hardware executables, e.g. MIPS code on x86 PC (Figure 3). These virtual prototypes, or models, require only the functionality the software needs, while fast performance enables quick turnaround and comprehensive testing. Simulation-based development provides visibility and controllability not available from the hardware, with improved debugger access. Technical advantages of the virtual platform:

·         Full controllability, from both external ports and internal nodes, and repeatability: Can corner cases be tested? Can an error be made to happen?>
·         Full visibility: If an error occurs, will it be observed by the test environment?

Business advantages of virtual platforms include earlier system availability and giving the entire team access to the virtual platform of the simulated SoC(s) or system. It is also easy to replicate the platform and test environment to support regression testing on compute farms.

Figure 3: (Image Courtesy Imperas)

Virtual platforms—software simulation—complement current hardware-based software development and test methodologies, which have significant limitations.

Linux is a complex, heavy-weight operating system, to get to the boot prompt takes almost a billion instructions. Android takes 5 or 6 billion instructions. With both, the data is overwhelming. As a result, simulation is even more important. In fact, industry is at the breaking point with embedded software and needs to adopt more disciplined approaches! Hardware design procedures are very structured, while in the past, software methods have been much more old-school and chaotic. People often think they can just fix their code after the product has shipped, but that’s no longer true. There’s a new wave of more rigorous, more automated methodologies for developing, testing, and integrating embedded software—not just to get it working, but to get it working fast, with high quality. OS booting, and the drivers, are important and challenging. This is where Imperas comes in.

Building a virtual platform is straightforward. This platform, or prototype, is a set of models that reflects the hardware on which the hardware-dependent software will execute: a single SoC, multiple SoCs, a board, a system; there are no physical limitations to complexity. Models are not an obstacle; and are typically written in C or SystemC. They are functionally accurate, such that the software does not know that it is not running on the hardware; the target binaries run unmodified. Models for individual components (interrupt controller, UART, Ethernet, etc.) are connected as in the hardware. Peripheral components can be connected to the real world by using the host workstation resources:  keyboard, mouse, screen, Ethernet, USB, etc. (Figure 4).

Figure 4: (Image Courtesy Imperas)

Case Study:  OS Bring Up and Testing on Altera Cyclone V SoC FPGA
One example of a program for OS bring-up is the Altera Cyclone V SoC FPGA, including Arm Cortex-A9MPx2 (Figure 5).

Figure 5: (Image Courtesy Imperas and Altera)

In this case study, the top-level Cyclone V SoC FPGA virtual platform was built using Open Virtual Platforms (OVP) ( models and APIs. The ARM Cortex-A9MPx2 processor core model and most peripheral component models were already available in the OVP Library, others were developed using OVP APIs, written in the C language. The Imperas simulation engine provided virtual platform performance.

Linux bring-up and testing on the Altera Cyclone V SoC FPGA followed incremental steps 1, 2, 3:

1.       Linux boot on single core ARM Cortex-A9 with minimal peripheral models.

2.       SMP Linux boot on dual core ARM Cortex-A9 with minimal peripheral models.

3.       Peripheral models added for Cyclone V SoC FPGA.

First, the Linux (OS from Altera) booted on a single core ARM Cortex-A9 using default configurations and device trees. Immediately, a bug was found in the Linux kernel pre-emptive scheduling. (Running multiple applications under Linux is part of standard Imperas bring-up testing, and it was found that Linux booted and ran, but did not switch tasks properly.) It took approximately two weeks of engineering effort to build the virtual platform able to boot Linux, which then ran to the boot prompt in under five seconds on a standard PC (Windows or Linux).

The multicore-capable OS-aware tools were used to find the bug: OS tracing [task, execve, schedule, context, etc.] traced at the OS, not the instruction level. With about 1 billion instructions to boot SMP Linux, instruction tracing to find OS problems would be painfully slow and complicated. With about 700 tasks to boot Linux, the Imperas task tracing capability provides a much more efficient starting point for debugging OS problems during bring-up.

These OS-aware software scheduling, tracing, and analysis tools also achieved debug in hours once the bug was observed, with simulation overhead of less than 10%. Task tracing was non-intrusive with no instrumentation or modification of source code, and provided tracing at appropriate levels of abstraction and granularity. It enabled in-depth monitoring and analysis, even before a console was available.

In the second step, a Symmetric Multiprocessing (SMP) Linux boot was run on a dual-core ARM Cortex-A9, Altera Linux was again used with default configurations and device trees; peripherals not yet modeled. Results: a bug was found in Linux accesses of Generic Interrupt Controller (GIC) registers. It also ensured that operating systems did not access forbidden memory segments. Virtual platform debug took two days versus two weeks on a hardware platform: factor of 5x improvement.

Other useful debug capabilities included non-intrusive trace/callbacks on changes/events in the system hardware/OS/software. Building on both an OS-aware layer and a CPU-aware layer in the solution, additional C code monitored and checked activity via protocols, rules, and assertions. A custom memory access monitor (just C code, less than 350 lines, loaded into the simulation environment) accelerated platform debug; during simulation, this monitor produced a warning if memory access rules were violated. Outputs included runtime analysis, checkers, and reports.

Tracing Linux instructions—a billion-instruction process, for instance—with Imperas tools, you can trace the Linux task calls, of which there are only 700 or so. And this tracing is graphical, not in terms of instructions. You can actually see the processor come to life, graphically; how the operating system is running; and when something’s not right.

Keep in mind that just one example case study has been described here. Although many programs use standard Linux, there are 10 or 12 other popular operating systems, including Android, of course. All can profit from accelerated bring-up, debug, and test in the same way: different versions of Linux, using different Ethernets, different USBs, etc.

Virtual platforms providing software simulation helps achieve higher quality software, reduces development schedules, increases software project predictability, and reduces delivery risk. These platforms complement hardware-based testing of embedded software. Linux (and other OS) bring-up on virtual platforms can be done incrementally for productivity, while OS-aware tools add efficiency.

Simon Davidmann is CEO of Imperas and initiator of Open Virtual Platforms, and has been working on simulators and EDA products since 1978. Prior to founding Imperas, Davidmann founded Co-Design Automation, the developer of SystemVerilog, and he has also been an executive or European GM with 5 EDA startups including Chronologic and Ambit. Davidmann is a Visiting Professor of Digital Systems at Queen Mary, University of London.


·         To view slides from a tutorial on this topic, presented at DAC 2017, see:

·         Follow Imperas: LinkedIn, Twitter, and YouTube



Share and Enjoy:
  • Digg
  • Sphinn
  • Facebook
  • Mixx
  • Google
  • TwitThis