Every server has a small computer inside it that most people never think about. The Baseboard Management Controller — the BMC — is responsible for monitoring hardware health, controlling fans, reading temperatures, managing power, and providing remote access. It runs its own operating system, has its own network stack, and it boots before the host CPU even powers on. If you have ever used iLO, iDRAC, or any other out-of-band management interface, you have talked to a BMC.
The problem is that this critical piece of infrastructure is almost entirely proprietary. The firmware running on your BMC is a black box, written by your server vendor, with no source code, no ability to audit, and no way to fix bugs yourself. You get what the vendor ships, on their schedule, with their priorities.
We think that needs to change.
Canopy is an open-source BMC firmware distribution built on OpenBMC. We take upstream OpenBMC, add long-term support, hardware CI testing on real servers, and the integration work needed to run on production platforms.
Our first target is the HPE ProLiant Gen11 platform — specifically the DL360, DL380, and related SKUs running on HPE's GXP SoC. We chose this platform because it is widely deployed, the hardware is capable, and there is real demand for an open firmware alternative.
We are not a fork. We track OpenBMC upstream weekly, rebase continuously, and contribute fixes back. The goal is to stay as close to upstream as possible while providing what upstream alone does not: tested releases, long-term maintenance, and production readiness for specific hardware.
Canopy is running on real hardware. Not in simulation, not "coming soon" — on actual HPE ProLiant Gen11 servers in our lab. Here is what works today:
gxp-psu hwmon driver we wrote from scratch — 8 sensor channels per power supply (voltage, current, power, temperature, fan speed)All of this is upstream-aligned. We use entity-manager for hardware discovery, dbus-sensors for monitoring, and the standard OpenBMC service stack. Where we had to write kernel drivers — like the PSU monitoring driver and the POST code capture driver — we followed upstream coding standards and ran checkpatch.pl --strict before every commit.
The work so far covers the basics. Servers boot, sensors report, fans spin at the right speed, and you can manage the machine over Redfish. But there is more to do:
Further out, we want to expand to additional server platforms and processor architectures. The meta-layer approach makes this practical — the platform-specific configuration lives in JSON and device tree overlays, not in code forks.
BMC firmware is one of those areas where the gap between "how things actually work" and "what is publicly documented" is enormous. If you have ever tried to understand how a baseboard management controller discovers hardware, how fan PID loops are tuned, or how the BMC talks to the host BIOS during POST, you know the pain. The information either does not exist, is locked behind vendor NDAs, or is scattered across mailing list archives.
This blog is where we will share what we learn — real code, real configs, and the engineering decisions behind them. You can follow along via RSS.
We are building Canopy because we believe server firmware should be open, auditable, and fixable. If that matters to you too, stick around.
BMC firmware has a testing problem. The software runs on constrained embedded hardware, interacts with dozens of physical sensors and buses, and controls safety-critical systems like fan speed and host power. But most OpenBMC development still follows the same pattern: build an image, flash it onto hardware, SSH in, poke around, and hope nothing regressed. There is no npm test. There is no CI that catches a broken Redfish route or a failed user authentication before it reaches the lab. OpenBMC has the Robot Framework test suite, but it is not integrated into upstream CI — most developers never see it run on their patches.
We built Canopy alongside FirmwareCI, so automated testing was not optional — it was built in from day one. Every commit to Canopy tests against a QEMU-emulated AST2600 running the full firmware image, and an HPE DL320 Gen11 in our lab getting its flash re-imaged on every merge to main. This post describes how that works.
Testing exclusively on physical hardware has obvious limitations. Hardware is scarce, flashing is slow, and the feedback loop is measured in hours rather than seconds. Worse, many regressions have nothing to do with hardware at all. A misconfigured systemd unit, a broken Redfish RBAC rule, or a D-Bus service that fails to register — these are software problems that happen to run on a BMC. You should not need a server rack to catch them.
At the same time, testing exclusively in emulation misses an entire class of problems. QEMU does not have real I2C buses, real PECI interfaces, or real power supply PMBus devices. You cannot test whether your entity-manager JSON correctly discovers an HPE GXP SoC's temperature sensors in QEMU, because those sensors do not exist there. Fan PID control is meaningless without real PWM outputs and real tachometer feedback. But you can test Redfish RBAC enforcement, user authentication, and WebUI accessibility — problems that have nothing to do with hardware.
We need both. So we built both.
The canopybmc repository defines two machine configurations. The first is hpe-proliant-g11, which is the production target — the actual HPE ProLiant Gen11 platform running on HPE's GXP baseboard management controller. The second is canopy-qemu, a machine configuration that targets the AST2600 EVB in QEMU:
# meta-canopy/conf/machine/canopy-qemu.conf
KERNEL_DEVICETREE = "aspeed/aspeed-ast2600-evb.dtb"
UBOOT_MACHINE = "ast2600_openbmc_spl_defconfig"
UBOOT_DEVICETREE = "ast2600-evb"
FLASH_SIZE = "65536"
PREFERRED_PROVIDER_virtual/kernel = "linux-aspeed"
The QEMU machine deliberately excludes the HPE GXP layers. It uses the upstream Aspeed kernel, the standard AST2600 EVB device tree, and a 64 MB SPI flash layout. This is intentional — the QEMU target is not trying to emulate the HPE hardware. It is testing the Canopy distribution layer: our distro configuration, our security hardening, our user management policies, our Redfish interface, our service stack. Everything that is not platform-specific.
Both machines share the same canopy distro configuration:
# meta-canopy/conf/distro/canopy.conf
DISTRO = "canopy"
DISTRO_NAME = "CanopyBMC (based on Phosphor OpenBMC)"
DISTROOVERRIDES .= ":canopy"
This means the QEMU image and the hardware image are built from the same codebase, the same package versions, and the same distro policies. When we test user RBAC enforcement in QEMU, the result applies to hardware too, because the Redfish stack, the user manager, and the access control configuration are identical.
Every push to main and every pull request triggers two GitHub Actions workflows. Each workflow builds the firmware image on a self-hosted runner (we call these Hydra nodes), then hands the resulting .static.mtd binary off to FirmwareCI for testing.
The QEMU workflow is straightforward — build, upload the artifact, trigger the test pipeline:
# .github/workflows/build-canopy-qemu.yml
jobs:
build:
runs-on: [self-hosted, Hydra, Large]
steps:
- uses: actions/checkout@v6
with:
submodules: recursive
- uses: ./.github/actions/build
with:
board: canopy-qemu
firmwareci:
needs: build
steps:
- uses: actions/download-artifact@v8
with:
name: obmc-phosphor-image-canopy-qemu.static.mtd
- uses: docker://firmwareci/action:v5.2
with:
WORKFLOW_NAME: $
BINARIES: firmware=obmc-phosphor-image-canopy-qemu.static.mtdThe hardware workflow is similar but the build is more involved. HPE ProLiant Gen11 systems use a GXP SoC with a secure boot chain that requires signed firmware. The build step injects a signing key and appends the GXP bootblock, producing a complete 32 MB flash image directly. The resulting binary is uploaded to FirmwareCI without any post-build processing.
The hardware pipeline also has a firmwareci-main job that only runs on pushes to main (not on PRs). This triggers a more comprehensive test suite that includes boot timing regressions, fan control validation, and VUART console verification — tests that take longer and exercise the physical server more aggressively.
FirmwareCI runs the tests. GitHub Actions builds the images, uploads them to FirmwareCI, and FirmwareCI handles device control, test execution, and result reporting. We define two device-under-test (DUT) configurations in the repository: dut-canopy-qemu and dut-hpe-dl320.
The QEMU DUT starts a qemu-system-arm process with an AST2600 EVB, forwards SSH to port 2222 and Redfish to port 2443, waits 100 seconds for boot, then runs the test suite.
The hardware DUT uses dutctl to control a physical HPE DL320 Gen11 in the lab. The test sequence: power off, write the firmware image to a flash emulator connected to the server's SPI bus, power on, wait for the "Phosphor OpenBMC" boot banner on the serial console (10-minute timeout), then run the test suite. This is hardware-in-the-loop testing — the BMC boots real firmware on a real GXP SoC.
The QEMU test suite validates everything that does not require real hardware. Every test connects via SSH or Redfish.
Boot, D-Bus, networking. Clean boot (zero failed units), core D-Bus services registered (Inventory.Manager, Logging, Settings), bmcweb and IPMI running, eth0 configured.
User management. Six tests covering CRUD operations via Redfish, RBAC enforcement (ReadOnly/Operator restrictions), password policy, session management, account limits, and root account disable/re-enable.
Event logging and web interface. Event creation/retrieval/deletion at all severity levels, logging service restart resilience, HTTPS connectivity, TLS validation, Redfish authentication enforcement.
The hardware CI test suite runs on every pull request.
Service health and D-Bus. 24 systemd services must start (EntityManager, CHIF, sensors, power control, bmcweb, FRU, logging, etc.). Platform-specific D-Bus names registered (GxpChif, Smbios.MDR_V2, fan/PSU/CPU sensors).
Inventory and power control. Host powers on via Redfish, POST completes, inventory populated (20+ sensors, CPU/DIMM/PSU data, system identity). Full power cycle: On → verify state → ForceOff → On again, console output validated at each step.
KVM. Video capture driver (/dev/video0), UDC driver, and HID gadgets (/dev/hidg0, /dev/hidg1). HID devices only appear when a VNC client connects, so the test opens a TCP connection to port 5900, waits 3 seconds for gadget binding, then checks for the devices. Userspace validation: obmc-ikvm service args, VNC port listening, ConfigFS configuration, bmcweb WebSocket endpoint.
CHIF. I2C proxy and PlatDef download handlers. Verifies BMC journal shows PlatDef extraction and I2C segment mapping at startup, host console log has zero CHIF response-format errors after POST.
Boot time regression. Reflash, power on, assert boot banner within 80 seconds and SSH within 25 seconds after that. Baseline: 74s to banner, 19s to SSH. Catches service dependency or kernel changes that slow boot.
Fan control and user persistence. PID control active, 5 PWM channels reporting 5-100% range. User creation via Redfish survives BMC power cycle (catches filesystem persistence issues).
VUART console stack. Full console pipeline: obmc-console-server, udev symlinks, Unix socket, SSH console on port 2200, log file. Power cycle host, verify log grew and contains POST output.
VUART kernel and data path. Driver-level validation: ttyS3 registered as 16550A, sysfs attributes correct, raw data capture from /dev/ttyS3 during boot (stops obmc-console, uses dd, verifies bytes + interrupt count + POST strings). TX path: write to /dev/ttyVUART0 and obmc-console-client, confirm /proc/tty/driver/serial TX counter increments.
SMBIOS and CHIF stability. After POST, verify D-Bus objects for TPM, PCIe slots, BIOS, DIMM, CPU. Then 3 host reboot cycles via SSH, checking for zero CHIF errors, zero PlatDef errors, CHIF service never restarted.
In practice, the two-tier approach catches different failure modes at different stages.
The QEMU tier catches packaging errors (a recipe dropped from the image), configuration errors (a D-Bus service file with the wrong bus name), Redfish API regressions (a bmcweb change that breaks authentication), and security regressions (RBAC rules that stopped enforcing). These are typically caught on PRs, before the code ever touches hardware.
The hardware tier catches integration failures: an entity-manager JSON that does not match the actual I2C topology, a kernel driver that fails to probe on the real SoC, a fan control configuration that produces out-of-range PWM values, a VUART driver that works in isolation but breaks the console pipeline. These are caught either on PRs (the CI suite) or on merge to main (the deep suite).
The boot time regression test is worth calling out specifically. OpenBMC boot time tends to creep upward as features are added — a new service here, a new dependency there. The 80-second hard limit forces us to notice and address boot time regressions as they happen, rather than discovering months later that the BMC now takes two minutes to boot.
We are expanding the QEMU sensor model to include more of the emulated I2C devices so we can test entity-manager discovery and dbus-sensors without hardware. We are also integrating the DMTF Redfish Service Validator into the pipeline — the storage configuration for it is already in the repository — to catch Redfish schema compliance issues automatically.
The longer-term goal is to make every test that runs on hardware also runnable in QEMU for the cases where the test is checking software behavior rather than hardware integration. The less often an engineer has to wait for a hardware flash cycle to find out their change broke something, the faster we all move.
The test definitions are all in the canopybmc repository under .firmwareci/. If you are evaluating Canopy for your platform, they serve as a concrete reference for what we validate on every commit.