Tag Archives: Antmicro

Open source and CI-driven RTL testing and verification for Caliptra’s RISC-V VeeR core

As part of CHIPS Alliance’s mission to enable a software-driven approach to silicon, Google, Antmicro and other CHIPS members have been developing and improving a growing number of open source tools to enable effective, CI-driven silicon development.

Fully reproducible and scalable workflows based on open source tooling are especially beneficial for efforts spanning across multiple industrial and academic actors such as Caliptra, a Root of Trust project driven by Google, AMD, NVIDIA and Microsoft which joined CHIPS in order to host the ongoing development and provide the necessary structure, working environment and support for the reference implementation of the standard, originally hosted by Open Compute Project.

In this blog post, we describe Antmicro and Google’s collaborative effort focused on introducing a Continuous Integration (CI) based code quality checks, code indexing, coverage and functional testing pipeline into the RISC-V VeeR core family, as used within the Caliptra project.

CI-driven testing and verification for Caliptra/VeeR
Open source RTL CI testing and verification for Caliptra/VeeR

Caliptra and VeeR

VeeR (Very Efficient & Elegant RISC-V) is an open source production-grade RISC-V core family hosted by CHIPS Alliance and comes in three variants:

Caliptra’s hardware block structure with an EL2 VeeR CPU core includes the following elements:

CI-driven testing for Veer flow diagram
Caliptra’s hardware block structure

As can be seen in the diagram, VeeR EL2 plays a central role in the implementation and, while it is a mature and well-tested technology, keeping both the core itself and its integration with Caliptra consistently tested is important.

Advanced code processing with Verible and Kythe

Many of Antmicro’s efforts focus around building not only the end products but the scalable CI solutions for collaborative hardware development environments that power them. Caliptra’s needs for establishing a more open process tie in perfectly with Antmicro’s and all of CHIPS Alliance’s open source-based approach to tooling.

One of the core parts of this effort involved Verible, an open source SystemVerilog parser developed by Google in collaboration with Antmicro within CHIPS Alliance, offering a number of code processing functionalities, including linting, formatting, indexing and producing a Kythe schema. Verible comes with a Language Server Protocol which enables integration with popular text editors such as VS Code, Vim, Neovim, Emacs, Sublime, Kakoune and Kate, described in detail in a separate article on Antmicro's blog.

Antmicro’s work for Google around Caliptra involved [adding the Verible formatter to the VeeR CI which marks non-compliant formatting changes and uses the reviewdog bot to add comments in the Pull Request Discussion with suggested fixes. Furthermore, we added a Verible linting Action that helps developers maintain good coding practices by providing lint rules for continuous validation of the code, before it even reaches the compilation phase. Notably, the provided lint rules are flexible and can be adjusted based on the project's requirements, or even turned off completely through creation of a waiver-file or by an inline directive.

CI-driven testing for VeeR lint error
Verible linting example

Thanks to Verible’s ability to output a Kythe schema, besides linting and formatting code changes we can also provide an indexed overview of the entire codebase, viewable online. The Kythe Verible Indexer, using Verible Indexing Action, enables the user to select multiple repositories to create a set of indexed webpages. The source code is available in the Verible Indexer GitHub repo.

CI-driven testing for Veer: Verible Indexer landing page
Verible Indexer landing page

The workflow also checks if a newer revision is available for any of the defined repositories and, if needed, performs indexing. The indexed code browser webpages were deployed for Cores-VeeR-EL2 and Caliptra-rtl.

Example from Caliptra-RTL Code Navigator demonstrating visual representation of the project structure (left pane), code browser (right pane) and usage of active references (bottom pane)
Example from Caliptra-RTL Code Navigator demonstrating visual representation of the project structure (left pane), code browser (right pane) and usage of active references (bottom pane)

Putting riscv-dv to use

The riscv-dv framework is another tool hosted by CHIPS Alliance helping address the complexities of SoC design and verification. It is an SV/UVM based open source instruction generator for RISC-V processors, originally developed for Google’s own needs but currently in use by a wide array of organizations and companies working with verification of RISC-V cores.

The riscv-dv framework generates random instruction chains to exercise certain core features. These instructions are then simultaneously executed by the core (through RTL simulation) and by a reference RISC-V ISS (instruction set simulator), for example Spike or Renode, Antmicro’s open source simulation framework.

Core states of both are then compared after each executed instruction and an error is reported in case of a mismatch.

CI-driven testing for VeeR risc-dv flow diagram
riscv-dv flow

While working on the Caliptra project, Antmicro introduced the riscv-dv framework for testing the VeeR-EL2 core as well as a GH Actions CI flow which builds or downloads all the dependencies (Verilator, Spike, VeeR-ISS and Renode) and runs the tests. For the purpose of using riscv-dv with VeeR we had to write a VeeR-specific execution trace log parser. The task of this parser is to translate the log to the format understandable by the riscv-dv framework.

As an interesting detail, VeeR implements division (div) and remainder (rem) instructions in a way it delegates the calculations to the division logic and proceeds with the execution of the program. Once the division core ends, the result is written back to the div/rem instruction result register. This flow takes into account the situation where any instruction following div/rem requires the div/rem result. In such cases the pipeline is stalled until the result is available. If any instruction following div/rem overwrites the result register before division logic finishes, the division operation is canceled.

To handle the case where the division results are available after a few other instructions were executed we’ve developed a lazy parsing method of the VeeR trace log to be able to catch the result register update even if it is not immediate. The second case - cancellation of the division calculation has been handled by adding a code post-processing script. It can detect a situation where a cancellation would happen and prevent it by injecting a number of the NOP instructions (allowing the division core to finish).

Custom GitHub Actions runners for greater scalability and more flexibility to mix tools

Much like a large part of the industry, the Caliptra project uses Universal Verification Methodology (UVM) as its verification methodology. While Antmicro’s ongoing work on enabling fully open source UVM support in Verilator should ultimately enable completely open source verification, today UVM testbenches or tools like RISC-V DV cannot be run using open source tools only.

Fortunately, this problem already has a solution, also developed within the CHIPS Alliance - custom GitHub Actions Runners that are already in use by a large number of CHIPS projects.

A custom runner setup, currently in development for Caliptra, allows mixing and matching open and closed source tools for CI testing purposes, exposing only the results (such as pass/fail or coverage) with fine-grained control.

What is more, given that RTL design testing and verification of RISC-V based cores and SoCs often require long, memory-consuming and computationally demanding simulations, the custom runners will play another very important role in Caliptra. While GitHub is the obvious choice for hosting the reference RTL, the processing power and throughput of the CI machines available in GitHub Actions is simply not enough to cover the needs of simulation of complex designs, especially in a highly dynamic, collaborative environment with lots of CI angles.

In order to enable public-facing yet secure CI, and improve the flexibility and scalability of Caliptra’s/VeeR’s pipelines, the custom runners will be deployed for the respective repositories. This setup will enable us to precisely select machines to be used for specific workloads (i.e. the architecture, virtual CPU count, memory size and disk space) but also to use tools stored on an external cloud disk that can be attached to a virtual machine running the job workload.

Seeding other verification methods

The Caliptra SoC is meant as a macro for use in a variety of chip designs, big and small. Various teams adopting Caliptra/VeeR as their Root of Trust solution will need to plug it into a larger ecosystem of tools used in their organization (of course hopefully using Caliptra as a good reference and role model).

As part of the project, on top of the original Caliptra test suite we implemented more specific tests around the VeeR integration in cocotb, a co-simulation testbench library that enables connecting Python coroutines with your HDL simulator of choice. We prepared a cocotb testbench that is able to not only run programs from the generator, but also apply dedicated stimuli and monitor the results in a Python coroutine.

Furthermore, for projects who prefer a more UVM-like testing methodology but need an open source option today, we also provide some example tests using pyuvm, a Pythonic library that mirrors the industry accepted SystemVerilog implementation. We have implemented a minimal UVM Agent for the programmable interrupt controller of the VeeR-EL2 Core, which will be used to verify handling of the interrupt service routines triggered by external or local-to-core timer interrupts. The verification environment is expected to grow as more test cases could be added, covering the DMA controller, close coupled memory buses or the debug interface.

For system level tests we decided to connect to an interactive simulation of the complete design via JTAG with commonly used tools: Open On-Chip Debugger OpenOCD and the GNU Project Debugger (GDB). The simulation exposes a virtual JTAG port, which is used to establish a connection with OpenOCD. Then the OpenOCD instance connects to the GNU debugger. Finally, test scripts are run in GDB, which verify core registers content, memory access and peripheral accesses.

With this testing methodology we exposed an actual problem in the design which prevented accessing system peripherals via JTAG. As it turned out the issue was caused by the side AHB bus of the debug core being disconnected.

CI-driven testing for veer waveform
Incorrect waveform - no system bus activity

Once a connection of the side bus had been made it became possible to access all the peripherals. A 2-to-1 AHB multiplexer was used to join the system and side AHB master ports and forward requests to the peripherals.

CI-driven testing for veer waveform pass
Correct waveform - correct system bus activity

To verify the effectiveness of all kinds of tests, both ISS and RTL level, and help ensure that all design states are properly tested, we use coverage analysis. While open source tools and frameworks have some support for gathering and presenting these metrics, e.g. Verilator supports line, toggle and functional coverage, some additional work needs to be done to integrate all of those and present them in a comprehensive, visual form, which will be part of our future efforts.

Transparent and open source-driven hardware ecosystem

In addition to the efforts described in this article, there are other interesting developments within CHIPS Alliance, including improving Verilator to better handle large designs and verification tasks, which are helping bring more open source-driven development and verification solutions to the Caliptra project and the entire open hardware ecosystem.

To learn more about the Caliptra project, watch a recording from a joint talk by Google and Antmicro given at this year’s RISC-V Summit Europe. You can also join the contributors at the upcoming 2023 OCP Global Summit for several talks about the latest developments in the project and future plans.

If you are interested in the work of CHIPS Alliance, keep an eye out for updates during their next Technology Update coinciding with the RISC-V Summit this Fall.

By Michael Gielda – Antmicro

ChromeOS EC testing suite in Renode for consumer products

Besides main application cores that are directly exposed to the users, many industrial and consumer devices include embedded controllers, which, although fairly invisible to the user, perform critical system tasks such as power management, receiving and processing user input, or signals from sensors like thermal. Given their role in the system, those MCUs need to be rigorously tested in CI. This is why the ChromeOS team has collaborated with Antmicro to simulate the ChromeOS FPMCU (Fingerprint Firmware) module, based on the ChromeOS EC (Embedded Controller) firmware in Antmicro’s Renode open source simulation framework.

This enables automated testing of embedded controllers in CI at scale, in a deterministic manner, and with complete observability. It also streamlines the developer feedback loop for faster development of microcontroller firmware that ChromeOS uses to drive peripherals, such as fingerprint readers or touchpads. To make this possible, we have improved the simulation capabilities for two of the microcontrollers used in FPMCU modules, popular in consumer electronics like Chromebooks and wearables, but also in many industrial applications: STM32F412 and STM32H743.

Testing consumer-grade products with Renode
Testing consumer-grade products with Renode

Continuous testing for embedded systems

The project required implementing continuous testing of the FPMCU module against tens of binaries that test the controller in the most common operations and scenarios to ensure maximum reliability at all times. A traditional approach would require reflashing the physical microcontroller memory with each binary, which is time-consuming and error-prone. To scratch that itch we developed the CrOS EC Tester, which runs all EC tests in a Renode simulation and uses GitHub Actions to handle building and executing test binaries for a truly automated workflow—useful both in CI and in an interactive development environment.

Renode has broad support for architectures such as (but not limited to) RISC-V, ARM Cortex-M, (recently added) Cortex-R or Cortex-A, and runs binary-compatible software. Thus, it is not limited to testing embedded controllers but entire multi-CPU systems. You can easily add Renode to an existing workflow without any major changes for testing in real-life scenarios. By moving all testing efforts into an interactive and deterministic environment of Renode, you can implement a fully CI-driven testing approach in your projects and benefit from advanced debugging, tracing, and prototyping capabilities.

Comprehensive simulation of STM32 microcontrollers

The Renode models of the STM32F412 and STM32H743 microcontrollers give you access to a broad range of peripherals, allowing you to run various scenarios you’d typically test on hardware. As a result of our collaboration with Google, we have added or improved models of ST peripherals like UART, EXTI, GPIO, DMA, ADC, SPI, flash controllers, timers, watchdogs, and more.

The need for in-depth testing has led to the introduction of many enhancements to ARM Cortex-M support in general, such as the MPU (Memory Protection Unit), which allows you to protect certain memory areas from unauthorized modification or access or FPU interrupts. These features can now be used by other Cortex-M-based projects to further extend their test coverage with Renode.

Renode for rapid, interactive prototyping

One of the tests from our test suite used the microcontroller's MPU module to test address space security. When you run the test-rollback test case, you can see that the MPU simulated in Renode successfully protected the OS from unauthorized memory access:

Testing consumer-grade products with Renode
Testing consumer-grade products with Renode

Another Renode feature that allowed us to increase our test coverage of the EC ecosystem is support for dummy SPI and I2C devices. While Renode supports a recently added advanced framework for time-controlled feeding of sensor data, many scenarios require much simpler interaction with the external device. For this purpose, we developed a dummy SPI device that simply returns pre-programmed data to the controller, which allowed us to pass initialization tests for a sensor without modeling the sensor itself. From the functional point of view of the simulation, the dummy sensor data is identical to the real data, which is useful when the specific component is difficult to model or lacks documentation.

Build a CI-driven test workflow with Renode

Renode is a powerful tool for automating and simplifying the test workflow in the project at any stage of development, even pre-silicon. It helps you reduce the tedium typically associated with embedded software testing by providing a fully controllable environment that can lead to fewer bugs and vulnerabilities, which is naturally important for mass-market products such as Chromebooks.

By Michael Gielda – Antmicro

Co-simulating ML with Springbok using Renode

The landscape of Machine Learning software libraries and models is evolving rapidly, and to satisfy the ever-increasing demand for memory and compute while managing latency, power and security considerations, hardware must be developed in an iterative process alongside the workloads it is meant to run.

With its open architecture, custom instructions support and flexible vector extensions, the RISC-V ISA offers an unprecedented capacity for such co-design. And by energizing the open hardware ecosystem, RISC-V has supercharged research and innovation into how to improve chipmaking itself to better leverage the methods and suit the needs of software. Initiatives such as Google’s OpenMPW Shuttle show how a more open and software-focused approach to building hardware, are key to enabling a new wave of more powerful and transparent ML-focused solutions.

A RISC-V-based ML accelerator with a HW/SW co-design flow

In the past months, Google Research has joined efforts with Antmicro to work on a silicon project that can serve as a template for efficient hardware-software co-design. For their secure ML solution, the Google Research team supported by Antmicro has been developing a completely open source, rapid pre-silicon ML development flow using Renode, Antmicro’s open source simulation framework.

This builds on the result of cooperation from last year in which Antmicro implemented Renode support for RISC-V Vector extensions, which are used in the Google team’s RISC-V based ML accelerator codenamed Springbok. To allow a more well-rounded developer experience, as part of the project Antmicro is also working on improving the support for the underlying SoC and a large number of user oriented features such as OS-aware debugging, performance optimizations, payload profiling and performance measurement capabilities.

Springbok is part of Google’s AmbiML project that aims to create an open source ML development ecosystem centered on privacy and security. By using the RISC-V Vector extensions, the Google Research team has a standard but flexible way to parallelize the matrix multiply and accumulate operations that are universal in ML payloads. And thanks to Renode, the team can make informed choices as to how exactly to leverage RISC-V’s flexibility by analyzing tradeoffs between speed, complexity and specialization in a practical, iterative fashion using data generated by Renode and the text-based configuration capabilities that let them play around with hardware composition and functionality in a matter of minutes, not days.

Diagram of A RISC-V-based ML accelerator with a HW/SW co-design flow

On the ML software side, the ecosystem revolves around IREE—Google’s research project developing an open source ML compiler and runtime for constrained devices, based on LLVM MLIR.

IREE allows you to load models from typical ML frameworks such as TensorFlow or TensorFlow Lite and then convert them to Intermediate Representation (MLIR), which later goes through optimizations on graph level and then through an LLVM compilation flow to get the best-fitted runtime for a specific target. When it comes to deploying models on target devices, IREE provides APIs for both the C and Python programming languages as well as a TFLite C API which provides the same convention as TFLite for model loading, tensor management and inference invoking.

Using these runtimes, the model can be deployed and tested, debugged, benchmarked and executed on the target device or in a simulation environment like Renode.

Demoing the flow at Spring 2022 RISC-V Week

In the build up to the Spring 2022 RISC-V Week in Paris, the first such large open hardware meeting in years, an initial version of the AmbiML bare metal ML flow was released as open source. This includes both the ability to run interactively and an example CI using Antmicro’s GitHub Renode Action showing how such a workflow can be tested automatically on each commit. As a Google Cloud partner, Antmicro is currently working with Google Cloud to make Renode available for massive scale CI testing and deployments for scenarios similar to this one.

In a joint talk at the Paris event, Antmicro and Google presented the software co-development flow, together with a demo of a heterogeneous multi-core solution, with one core running the AmbiML Springbok payload and another core running Zephyr.

In the presented scenario the Springbok core, acting as a ML compute offload unit to the main CPU, executed inference on the MobileNetv1 network and reported the work done to the application core via a RISC-V custom instruction. Adding and modifying custom instructions is trivial in Renode, either via a single line of Python, C#, or even co-simulated in RTL.

Renode helps ML developers and silicon designers not only to run and test their solutions, but also to learn more about what their software is actually doing. As part of the Paris demonstration, Antmicro and Google showed how you can count executed instructions and how often specific opcodes are used to measure how well your solution is performing. These features, accompanied by execution metrics analysis, executed functions logging, and recently developed execution trace generation, give you great insight into every detail of your emulated ML environment.

These capabilities join the wide arsenal of hardware/software co-development solutions in Renode, such as RTL co-simulation which Antmicro has been developing with Microchip and support for verilated custom instructions developed with another ML-focused Google team responsible for RISC-V Custom Function Units and also used in the EU-funded VEDLIoT project.

Future plans

This is just the beginning of a wider activity from the Google Research team Antmicro is working with to release software and hardware components as well as tools supporting a collaborative co-design ecosystem for secure ML development. If you think Renode, RISC-V and co-development could help in building your next ML-focused product, go ahead and try the AmbiML flow yourself!

Visit the iree-rv32-springbok repository on GitHub, clone it locally and follow the instructions from README.md.

Renode Repository

You can also grab Renode from the official repository and start playing with the available demos, or head to the Renode documentation to read up on features helpful for ML acceleration development such as Verilator co-simulation.

By Peter Zierhoffer – Antmicro

DC-SCM compatible FPGA based open source BMC hardware platform

Open source software is omnipresent in the server and cloud world, and is giving rise to impressive successes in the SaaS space where useful products can be rapidly created from open source components: operating systems, container runtimes, frameworks for device management, monitoring and data pipelining, workload execution, etc.

Mirroring this trend in software, cloud providers and users are increasingly looking at building their servers using open source hardware, collaborating in initiatives such as the Open Compute Project or OpenPOWER.

With Google, Antmicro has developed two open source hardware FPGA-based Baseboard Management Controller (BMC) platforms compliant with OCP’s DC-SCM ver 1.0 specification to help increase the security, configurability and flexibility of server management and monitoring infrastructure. These have since been adopted by OpenPOWER’s LibreBMC workgroup as the base hardware platform.

The DC-SCM spec

The Datacenter-ready Secure Control Module (DC-SCM) specification aims to move common server management, security and control features from a typical motherboard into a module designed in a normalized form factor, which can be used across various datacenter platforms.

Currently rolling out in first DC-SCM compliant servers, the spec will help Cloud providers share costs, risks and increase reuse for the critical BMC component. Coupled with a fully open source implementation based on popular, inexpensive FPGA platforms will not only allow for more configurability and a tighter integration between hardware and software, but also tap into the momentum behind the broader open source hardware community via groups like CHIPS Alliance, OpenPOWER and RISC-V.

The hardware

Antmicro has developed two implementations of the DC-SCM-compatible BMC. Both designs meet the Open Compute Project specification for a Horizontal Form Factor 90x120 mm DC-SCM ver 1.0.

The BMCs role is central to the server’s faultless operation, responsible for monitoring the system while preventing and mitigating failures. Essentially, acting as an external watchdog.

To be able to provide this functionality, according to the requirements, the module offers a feature-packed Secure Control Interface for communication with the host platform, including:
  • PCI Express
  • USB
  • QSPI
  • NCSI
  • multiple I2C, I3C and UART channels
The unique property of Antmicro’s implementation of the standard is merging DC-SCM’s central Baseband Management Controller and the usual programmable SCM CPLD block into one powerful FPGA. This solution was chosen to enable a greater flexibility of the design, allowing remote updates of DC-SCM peripherals, and most notably, placing OpenPOWER or RISC-V IP cores as central processing units of the module.

One of our designs is based on Xilinx Artix-7, whereas the other one features a Lattice ECP5 - both low-cost and open source friendly FPGAs supported by the open source F4PGA toolchain project.

The FPGA is complemented by 512MB of DDR3 memory, 16GB of eMMC flash, as well as a dedicated Gigabit-Ethernet interface. To ensure the security of the Datacenter Control Module, external cryptographic modules: Root of Trust (RoT) and Trusted Platform Module (TPM) can be connected. This will allow future integration with e.g. open hardware Root of Trust projects such as OpenTitan and implementing various boot flow and authentication approaches.

Use in LibreBMC / OpenPOWER

Open source, configurable hardware platforms based on FPGA, open tooling and standards can make BMC more flexible for tomorrow's challenges. Antmicro’s DC-SCM boards have been adopted by the LibreBMC Workgroup, operating under the umbrella of the OpenPOWER Foundation, in a push to build a complete, fully transparent BMC solution. The workgroup, with participation from IBM, Google and Antmicro, among others, will be involved with creating FPGA gateware and software needed to make the hardware fully operational in real-world server solutions.

Variants involving both Linux (in its default open source BMC- distribution, OpenBMC), and Zephyr RTOS, as well as with both POWER and RISC-V cores are planned and thanks to the flexibility of FPGA all of those options will be just one gateware update away. Of course, both the gateware and software will be open source as well.

If you’re looking to develop a secure and transparent DC-SCM spec-compatible BMC solution, reach out to [email protected]. See how you can collaborate with partners such as Antmicro, Google, and IBM around the open source FPGA-based hardware platform.

By Peter Katarzynski – Antmicro