A Week of Bug Reporting

If you use software regularly, and have your wits about you, you will often realise that you’ve encountered a bug — in other words, that the software has done (or not) something that it shouldn’t (or should have). When this happens, the vast majority of people moan to the nearest bystander about how incompetent the people behind the software are and then carry on with whatever they were doing — often stumbling into the same bug repeatedly. Only a minority of people think of reporting bugs and, if they do think of it, they often fail to do so either because they think it’s too time-consuming or that they might end up looking silly. It’s unfortunate that so few people report bugs. First, surprisingly few bugs affect a single person: by the time someone does report a bug, it can have affected hundreds or thousands of other people. We can’t all be freeloaders — someone has to take the plunge and report the bug. Second, bugs frequently appear in contexts unfamiliar to the software’s developers1. Most obviously, we all have slightly different software installations, and those variations can cause odd effects. Less obviously, users often use software in ways unimagined by its creators, such that some previously untested combinations of operations work incorrectly2. Either way, we cannot assume that the software’s developers will encounter the same bugs that we do: in many cases, if we don’t report bugs, the developers simply won’t know they exist. Reporting bugs in someone else’s software is different from finding and fixing bugs in one’s own code, because we understand those other systems much less well. The sense of confusion I have with bugs in my own code is magnified when using other systems. I’m less confident that I’m actually seeing unexpected behaviour; less sure what the cause might be; and have lower trust than normal in any fix I propose. That means that we must acknowledge an important trade-off: a full understanding of the cause of a bug in an unfamiliar system is often beyond any reasonable investment of time; but if we just say “it doesn’t work”, no-one can be expected to replicate, let alone fix, the bug. Somewhere in the middle is a sweet-spot where the bug reporter invests the minimum amount of time to generate a good enough bug report that a developer can use to fix the bug. While I wouldn’t exactly say that reporting bugs gets me out of bed each morning, I do feel something of a duty to do so. I have benefited hugely from the existence of open-source software, and reporting bugs is one of my ways of contributing a little back. Now that I’ve been doing so for years, reporting bugs holds many fewer fears, in terms of time or reputation, than it once did. In this post I’m going to look at the bugs I reported in the first week of August on various pieces of open-source software. This is hardly a normal week, because I started it with a new laptop, which pretty much guaranteed that I’d hit more bugs than normal — but, as you’ll soon see, that was not the only source of bugs I encountered. Some of my reports have led to bugs being fixed, some haven’t. While I obviously hope for the former outcome, open-source developers aren’t under any obligation to investigate, let alone fix, my bug reports. I hope it’s useful to see what happens when someone like me, who sometimes, but definitely not always, knows some details about the software I’m using, runs into what I believe are bugs. I’ve tried to record as much of the confusion that I went through as I can remember, because that seems an inevitable part of the process. Unsurprisingly, a lot of what I’ve written relates to my attempts to understand the effects of a bug and investigate their cause. I hope you also get a sense of how much my interactions with developers played a role in my and their understanding of a bug. Not only did my bug reports vary substantially in quality, but I was quite often wrong in my initial guess of the cause of a bug or what a good fix might be. I’m going to present these bugs in roughly the order that I encountered them, because as you’ll see, there are more overlaps than might otherwise be apparent. A new laptop After 5 years of use, my old Thinkpad had reached the end of its useful life. The battery life had declined by around half, and more than once I ran out of power at an inconvenient time, damaging the battery further. Furthermore, its 2 physical cores were no longer up to the demands modern software places on them, causing me to twiddle my thumbs increasingly often. I’d been using Thinkpads for 19 years3 and until recently assumed I would continue doing so. However, Framework’s modular approach to laptops, which seems to me a more environmentally sustainable approach to hardware, appeared to be both equivalently good, and more worthy of my support, so I ordered one with my hard-earned cash. I knew that the first generation Framework was supported under OpenBSD (my operating system of choice) and crossed my fingers that the newly released second generation (which I’d bought) would work too. However, I was aware from experience that newly released hardware tends not to fully work on OpenBSD for a little while. Video and keyboard oddities Once I’d turned off “secure boot” in the BIOS, the OpenBSD installer booted and functioned correctly. However, the installer is text only and places relatively few demands on hardware — I knew that the real test would happen when I rebooted into the “full” installation. Fortunately, that seemed to go fairly well, and I started installing all the packages I’m used to. I soon realised that the keys I was pressing on the UK keyboard I’d ordered weren’t always corresponding with what I was seeing on screen. I don’t remember that ever happening with OpenBSD before, so it took me a little while to realise that wsconsctl tells me the keyboard encoding. On the Framework I saw: $ doas wsconsctl keyboard.encoding keyboard.encoding=unknown_0 On my desktop, in contrast, keyboard.encoding=uk. I then tried to force the keyboard encoding to UK and immediately hit a kernel panic: $ doas wsconsctl keyboard.encoding uvm_fault(0xfffffd87b4e6a780, 0x8, 0, 1) -> e kernel: page fault trap, code=0 Stopped at wskbd_load_keymap+0x33: movl 0x8(%rax),%ecx ... That’s not good news: it means the kernel has crashed. I was rather surprised, so rebooted and hit a kernel panic before I’d even typed anything in. I rebooted again, got to a terminal prompt and tried setting the keyboard encoding again — immediately causing another crash. I rebooted yet again, and then hit another kernel panic after a few seconds of usage (without so much as trying to set the keyboard encoding). At this point I was a bit bamboozled, as I seemed to be hitting at least two separate bugs. Setting a keyboard encoding reliably caused an immediate kernel panic. But I was hitting kernel panics for some other reason that I couldn’t fathom. I then deliberately caused another panic after setting the keyboard encoding, took a photo of the kernel’s output, and filed a bug report to OpenBSD’s bugs mailing list, the main body of which is: I’ve just got my sticky mits on a Framework 12th Gen Intel laptop. It somewhat works, but regularly hits kernel panics and other oddities on -current. I haven’t worked out why yet, but uvm_faults seem to be frequent. For example setting the keyboard encoding causes a uvm_fault. $ doas wsconsctl keyboard.encoding=uk uvm_fault(0xfffffd87b4e6a780, 0x8, 0, 1) -> e Screenshot of this particular panic at https://postimg.cc/nM5rLGY6 I suspect that it’s simply that some hardware isn’t yet fully supported. There are a few more “not configured“s than normal in the dmesg and it even contains this, which is a new one on me (presumably PCI related?): 0:31:5: mem address conflict 0xfe010000/0x1000 It’s a deliberately short bug report. First, since I was quite possibly the first person to try OpenBSD on a second generation Framework, I knew that I shouldn’t expect things to work perfectly. Second, I noted down the one way that I could reliably cause a panic (setting a keyboard encoding) while noting vaguely that there seemed to be other causes too. I made a possible suggestion for the latter (which has later turned out to be irrelevant). But I then waited to see what the OpenBSD developers said, since they might have context about the likely status of OpenBSD on the Framework that I lacked. Within three hours, Jonathan Gray, a prolific OpenBSD developer, had replied, though he wasn’t able to immediately solve the panics I was seeing. In the meantime, by continually rebooting, I’d started to spot an intermittent correlation between graphics corruption and panics, which I used to update my report: I’m also starting to understand a bit more about some of the random panics: they seem to happen very soon after X starts. Sometimes the mouse appears as a two inch square of weird colours (!) – things only last for a few seconds after that. Only once have I got a visible panic out of this which said solely: uvm_fault(0xffffffff822f0400, 0xfff80001660014, 0, 1) -> e drm: Global state not read locked drm: Global state not read locked Jonathan again quickly replied with a suggested fix. I applied the fix (which requires some basic knowledge about how to build OpenBSD from source), and decided that if I couldn’t trigger panics after 10 boots, I’d consider the problem probably fixed. 10 successful reboots later I was able to report: A kernel with this patch has now survived 10 reboots (previously I was getting a panic perhaps 1 time in 3), so I think it’s quite likely that the effect is real. Thanks! The fix was duly committed to OpenBSD less than 24 hours after my initial bug report! Unfortunately the keyboard encoding problem has turned out to be rather harder to fix. Miod Vallat, another frequent OpenBSD developer, suggested a mechanism for preventing the keyboard encoding panic but noted that the problem should never have gotten that far. A