4-Way Comparison of Compilation Performance on OpenBSD

Article Update Information:

Initial publication on 2021-10-05

I recently bought the Asus ExpertBook B9400CEA. Buying newer hardware when using an operating system with a limited number of kernel developers like OpenBSD is always a bit of a gamble when it comes to how much will work as expected. This laptop looked like it was worth it. It seems to hit the sweet spot between portability, battery life, performance, and customizability (in my case especially multibooting with 2 separate M.2 SSD drives).

See my initial setup and impressions here

In terms of performance, I was expecting to come at least reasonably close to the performance of my desktop i7-10700 CPU. The i7-1165G7 in the ExpertBook only powers 4 cores and is a chip designed for ultra-portable, low-power laptops (Evo/Ultrabook family). However, the newer 10mm Tiger Lake architecture, the high peak CPU speeds (4.7GHz), and the decent scores on CPU benchmarks suggest this one is not a total slouch.

PassMark assessment and comparison of the i7-1165G7

Noticeable Issues with Performance on OpenBSD

After my initial rather positive impressions with the new Asus ExpertBook B9400CEA, I now have a bit more experience on it and some of the more subtle observations led me to this article. Specifically, I noticed unexpected issues with performance in a couple of situations:

Browser, especially when playing back video and opening/closing browser tabs (event with only 1-3 tabs open)

Compilation (unexpected long duration of compiling ports)

Gaming (Cruelty Squad) - more noticeable stutters than expected

Cruelty Squad recording, showing stuttering for example at 2:26, 2:32, 3:16, 3:20 ... (note that you need to log in/confirm age to watch because of graphical violence in this game)

All of this together is why I set out to try a more objective way to compare the performace with other OpenBSD machines that I have around.

The Test Machines

Here is what I have available, as a line-up of OpenBSD machines that I've used in the last ~5 years. They are all amd64 architecture.

I have been using the desktop i7-10700 for much of 2020 to 2021 and it has been a fairly reliable OpenBSD work horse. When I bought the ExpertBook with the i7-1165G7, I was hoping for a mild to moderate improvement in graphical/gaming performance, and at least getting close to the raw CPU performance of the i7-10700.

AMD Ryzen CPUs have been snatching market dominance from Intel over the last years, so I was expecting a fairly decent result from the Ryzen 7 PRO 3700U. It was hard to place based on gut feeling. Let's say, I wouldn't have been surprised if this one ended up outperforming most other CPUs.

Finally, the Latitude E5570 is by far the oldest machine in this collection. At the time, it was one of the beefiest laptop CPUs I could find at an affordable price, but that was about 5 years ago and I bought it used at the time. In addition, what I picked up from discussion around the mitigations for Intel's Spectre and Meltdown-related issues seemed to indicate that Skylake CPUs (like the i7-6820HQ) would be the most affected in their performance because of the problems with the architecture.

The Comparison Testing

I've made a few preparations to try to ensure a comparison that's as even as feasible:

SMT disabled on all machines (as is the default on OpenBSD for security reasons after Spectre/Meltdown)

MAKE_JOBS=4 on all machines (including the octacore i7-10700)

`make patch` has already been done, so that really only build time is measured

softraid disk encryption active on all machines

Here is how I went about my testing:

Note that the first run on the i7-10700 was done with all 8 MAKE_JOBS (and is unsurprisingly the fastest result).

Results

First run (with `apm -A`)

Second run (with `apm -H`)

And as a visual representation of the second run where all ran with MAKE_JOBS=4 (shorter is better):

Conclusion: Some Surprises

It was my gut feeling that something about the performance of the ExpertBook's i7-1165G7 wasn't meeting expectations, and this was the initial incentive to do this comparison testing. The overall result of this is surprising:

The i7-1165G7 is at the bottom of the comparison which is unexpected based on the documented benchmark performance (of course not done on OpenBSD).

Despite its age and the Skylake architecture, the i7-6820HQ performs surprisingly well and is closer to the winning performance of the i7-10700 than the rest.

Like with the i7-1165G7, the performance of the Ryzen 7 PRO 3700U is disappointing. Note both share that they are designed for low wattage, energy-efficient systems.

Note that for someone who spends quite a bit of time on ports, the magnitude of the difference is striking. This means that when using all cores, it takes MORE THAN 4 TIMES AS LONG to complete compilation on the i7-1165G7, compared to the i7-10700. Even when leveling the playing field by using the same number of MAKE_JOBS, it still takes 3 times as long. This means less productivity and longer wait times.

In addition, as I use OpenBSD heavily as my daily driver even while compiling, for example by browsing the internet, sometimes with many browser tabs open, reading, watching videos, etc., the poor performance also limits usability during compilation. Casually browsing a few web pages while letting the compiler do its work is rather painful on the i7-1165G7, as there is lots of stuttering and lag.

On top of that, even in normal, relatively limited use scenarios like _just_ using the browser (without concurrent compute-intensive tasks) or playing a game, there is a noticeable slowness to the system.

Of course, this is far from a rigorous assessment of objective performance. There are some differences in other components that may affect the results, like size and type of RAM (note no swapping or memory exhaustion was noted when the process was monitored with htop), I/O speed of the disk, etc. Between several runs of compiling games/godot, the compile time was quite consistent and differed by less than 1 minute in most cases (note the i7-10700 was initially run with 8 MAKE_JOBS).

Overall, the components like mostly NVME drives and the size and type of RAM make me think that it is unlikely that those had a significant influence on the results.

Instead, I would raise the following questions:

Is OpenBSD more prone to reaching thermal/voltage limits of the CPUs than other operating systems?

Does the i7-1165G7 suffer from the relative novelty of the Tiger Lake architecture? Is there work to be done to unlock the full potential of the architecture on OpenBSD?

I wonder if in the current state of OpenBSD, the performance (with a multiprocess kernel) is very much dependent on the multi-threaded performance, as opposed to single-core performance. This could partially explain the results, as the performance testing of the i7-1165G7 suggests great single-threaded performance, but relatively underwhelming results in multithreaded loads.

In the end, the difference in performance is so drastic that I've decided that for now I will go back to doing my main work on the desktop, which runs the smoothest and fastest. I will probably use the Skylake i7-6820HQ for a while longer, given the surprisingly good results. I am curious if performance of the Tiger Lake CPUs will see some more improvements. This may be a hard problem to diagnose/solve - I can't tell how much (if any) of the performance issues are CPU-related, or if the thermals are a bigger factor. I hope to test the laptop again later on, maybe in half a year or so, to see if there have been more improvements. For now, this laptop is back to just running Windows when I need it (not that often..., I hope).

DMESGs

Desktop i7-10700

Asus ExpertBook B9400CEA

Lenovo ThinkPad X395

Dell Latitude E5570