Progress Report: Linux 7.2 - Asahi Linux

Linux 7.2 has been released! That was fast. Let’s dive in to yet another Asahi Linux progress report. We’ve got a lot of interesting developments for you today, so make yourself a cuppa and enjoy.Think Different… againThe Apple Silicon platform’s power management infrastructure is complicated. Responsibilities are divided between multiple hardware blocks, including the SMC, PMGR, and PMP, all of which have featured before on this blog. While supporting these blocks is important for power use, one of the biggest obstacles to improving battery life has been the application cores themselves.There are multiple ways to “sleep” a CPU core, and each one should be used in a specific context. The most basic way to sleep an ARM CPU core is to use a Wait For Interrupt (WFI) instruction. This tells the core to stop doing things until it is woken up by an interrupt from an interrupt source. While this does save power by virtue of stopping the core from executing any code, the core stays powered up and retains enough state for it to resume work extremely quickly. As such, WFI is typically only used for parking a core on a running system. Apple cores include a “deep” WFI mode, which shuts down more of the core at the expense of losing its state. Our downstream cpuidle driver operates by setting WFI up in this mode, saving the core’s state, then issuing a WFI loop.Vendor-specific power management oddities like this are quite common. Thankfully for kernel maintainers, there is a standard way to deal with them: the Power State Coordination Interface. PSCI defines a standard iterface that allows an operating system to call into a defined set of CPU core power management functions implemented by system firmware, including to prepare them for sleep.To avoid a proliferation of vendor-specific power management hacks inside the Linux kernel, the maintainers of the arm64 arch-specific code have mandated that all upstream hardware must use PSCI for power management. As such, we are not able to upstream our Apple-specific cpuidle driver. Why are we still using it then?PSCI defines “conduits” through which calls to firmware are to be dispatched from the kernel. The two currently supported conduits in the kernel are the SMC (Secure Monitor Call) and HVC (Hypervisor Call) instructions, which are used to yield execution to a higher Exception Level. The Linux kernel expects to be running at EL2, which means that its PSCI calls must yield to firmware running in EL3. Except Apple’s cores do not implement EL3…With the kernel already running in EL2 and no firmware running in EL3 to talk to, we are a bit stuck. Linux cannot issue an SMC or HVC instruction since there is no EL3 to yield execution to, which means we cannot make use of PSCI. Being able to properly power-manage the CPU cores is vital for battery life and efficiency, so the status quo simply will not do. One quick and dirty solution would be to have m1n1 load the kernel into EL1, and host a PSCI implementation in EL2. While this would theoretically work, it would also break a lot of architectural features, such as virtualisation. There must be something else we can do…If you think about it, m1n1 is almost like our own firmware for Apple Silicon. mBoot (formerly iBoot) starts it in EL2, it does its job, then jumps to whatever payload is attached to it. m1n1 does not reserve any memory for itself and does not have any code that must stay resident, so its payload is free to reclaim and overwrite that memory.On production Asahi Linux systems, m1n1 loads U-Boot rather than the kernel directly. We do this to make use of U-Boot’s UEFI implementation, allowing distros and users to utilise whichever standard UEFI bootloader (GRUB, systemd-boot, etc.) they want. UEFI also provides another feature, Runtime Services. Much as the BIOS interrupts of old did, UEFI Runtime Services provide a way for the operating system to access code originating in system firmware.Reading the PSCI standard as published by Arm, one will notice that it deliberately defines the API without reference to any specific conduit, and only lists SMC and HVC as examples. If we take a broad interpretation of this, we could conclude that this means other conduits are allowed by the spec…To this end, Sven has been working on implementing a UEFI Runtime Service based PSCI conduit. With m1n1’s memory region carved out like other firmware regions, this will allow the kernel to call back into it for PSCI services, even though it is running at the same Exception Level. Sven has already modified m1n1 to reserve its memory and leave behind a PSCI implementation, and the patches to the kernel enabling its use are already on the mailing list as an RFC!Please stop Thinking DifferentGiven that the cpuidle situation saw no progress until very recently, one might assume that some event has catalysed work in this space. One would be correct.The ARM specification mandates that cores in WFI loops should preserve all state. This is not the default mode on Apple Silicon. On M1 through M3 series SoCs, state retention can be configured on a per-core basis using chicken bits.Due to a number of reasons that are not worth mentioning, Apple now sets each core’s chicken bits in mBoot and then locks down the registers controlling them starting with the M4 series. This makes our life a little easier as m1n1 now has marginally less work to do, however it also means that we cannot fine tune low level CPU behaviour. This is an issue on M4 particularly, as calling WFI causes the core to lose its state and crash whatever was running on it.Yurkea noticed this while doing M4 bringup work, and added a kernel command line parameter to make idle loop behaviour configurable. The parameter allows us to tell the kernel how it should park cores in idle loops, including by doing a basic no-op loop. This prevents M4 machines from crashing during early kernel initialisation, before our cpuidle driver has been loaded. Once the driver takes over, it saves the the lost state before issuing WFI. The patches to enable this are already in linux-next.They won’t stop Thinking DifferentApple takes its reputation for platform security very seriously. As such, a lot of engineering effort goes in to features that make exploiting vulnerabilities in their ecosystem infeasible for all but the most sophisticated of attackers. One such feature is the Secure Page Table Monitor, or SPTM.Traditionally, the operating system kernel has been directly responsible for managing memory. This includes handling memory allocations, virtual mappings to physical addresses, and MMU/IOMMU management. A vulnerability in the code responsible for these operations could give an attacker access to the platform’s entire address space. In other words, you’re cooked.Naturally this makes memory management code a very common target for attackers, and given that its job is fundamentally insecure (applications may want arbitrary allocations for just about anything) it is incredibly difficult to lock down correctly.Many years ago, Apple introduced the Page Protection Layer to XNU. PPL uses Apple’s hardware security features to isolate pagetable management from the rest of the kernel at the hardware level. This works very well, however attackers eventually caught up and found ways inside PPL that allowed them full access to the system.Apple has had similar issues with IOMobileFramebuffer in the past. As mentioned on previous blog posts, Apple (mostly) solved the IOMFB problem by putting it inside DCP’s firmware, behind an IOMMU. Nothing in macOS userspace nor in XNU can reach it, except via a defined set of IPC functions. Taking inspiration from this approach, PPL eventually morphed into SPTM. SPTM takes PPL and places it inside Apple’s Guarded Execution Framework (GXF), a set of Exception Levels that run parallel to the standard ARM64 EL1 and EL2 (there is no GL0). GXF also comes with SPRR, a custom pagetable permissions system used when the CPU is running code in GL1 or GL2.When a modern Apple Silicon device starts up, iBoot or mBoot will detect if the configured boot payload is an XNU image. If it is, it will first load SPTM into GL2. There, SPTM sets up pagetables and memory management and locks down control of said features to GL2. XNU then runs, using yet another IPC protocol to talk to SPTM. If XNU cannot successfully contact SPTM, it will panic very early in init and halt the system.This is problematic for us. If we try to run XNU under the m1n1 hypervisor, it will crash as SPTM has not been loaded. If we configure mBoot to treat m1n1 as an XNU binary, m1n1 will crash as it will be unable to manage memory in the way it needs to.Given that SPTM is mandatory for XNU on M4 and above, the hypervisor was rendered entirely nonfunctional on these machines. We don’t know when to quit however, which is why that’s was and not is!SPTM itself is not particularly special. It is an ARM64 Mach-O binary that lives inside the same preboot directory as the OS-specific firmware blobs for the various coprocessors. That means we could theoretically have m1n1 load it already under the hypervisor, load XNU next to it, then watch both of them! But SPTM must be run inside GL2 with SPRR enabled. If only we knew how both of them worked…Thanks to Sven’s work reverse engineering SPRR and GXF way back when Asahi Linux was first getting started, he was recently able to teach the m1n1 hypervisor how to emulate them! This allows us to load Apple’s SPTM blob in exactly the manner XNU expects, do some quick surgery on the XNU binary, load it, and then monitor MMIO accesses just as we could from M1 to M3! The gory details of how Sven achieved this means that tracing is slower on these machines, however not unusably so. This will allow us to continue bringing up new hardware