Real World Apple Silicon Benchmarks Part 6: .NET Performance Suite

The last benchmark of the .NET Platform that I have is the benchmarking suite that the .NET team put out here . It is literally thousands of tests covering all parts of the CLR. Nothing could be more thorough. As I wrote in this previous post I’m doing a series of benchmarks of .NET and JVM on Apple Silicon. While there are impressive native benchmarks the fact it will be some time before the .NET runtime has native support. I have to factor in the potential hit and problems with Rosetta. How much of a performance hit is there and will it be enough that applications targeting it will have problems? All code and results are published here .

The .NET performance benchmarks themselves fall into three major categories. The biggest set are the “micro benchmarks” which consist of over 3300 benchmarks in total, again covering every part of the CLR. There are then some “real world” benchmarks that can be run. The two in particular I chose was the compiler benchmark, which studies the performance of the Roslyn compiler, and a machine learning benchmark.

I was very excited to see how well such a thorough set did but became disappointed quickly. Three times in a row, but not in the exact same place in the test suite, the M1 MBP spontaneously rebooted. This was not a problem I had on the Intel MBP. The good news is that others have flagged this issue, see the GitHub Issue tracker here . The bad news is that there isn’t a fix yet and it’s not clear when that will be out or if it will be rolled out in a .NET 5.0 (or even 3.1) release once it is fixed. Officially support on Apple Silicon will begin with .NET 6, at least according to this Epic . Because this bug seems to be pretty pervasive I imagine that this will be addressed sooner than a .NET 6 release. In the meantime though I’m not sure how this manifests as other stability issues elsewhere. Could this be what caused other stability problems I saw? Maybe, maybe not.

Setting aside the microbenchmarks for the time being, I was able to generate results for the two real world benchmarks. So let’s explore those. First let’s look at the compiler benchmark, full results can be found with tabular information here .

As we can see the total aggregated benchmark performance was about 25% slower on Rosetta on M1 than it was on the Intel MBP. If you dive into the detailed charts you’ll see that a couple were break even, but with Rosetta slightly slower. The rest had a pretty consistent 30% slowdown on Rosetta. That’s pretty good. That does make me wonder why the Avalonia compilation was 3x slower though. It could be something else in the build process that’s causing extra drag. Is there a bootstrapping time for each of the dotnet processes kicked off for each build (many are created during the build process) which in aggregate really cause a slow down? That’s literally just a WAG on my part. The good news is that it isn’t intrinsic to the compiler itself. Let’s look now at the machine learning benchmark, detailed results with tabular information here .

In this case the Rosetta on M1 system was about 30% faster than the Intel one. If you look at the detailed charts you will see a much bigger mixed bag than on the compiler benchmark. Yes, overall the suite ran faster, yet there were some tests were over 2.5 times slower. Other tests were slightly faster, and the rest were essentially washes. This goes back to a conclusion I had from the Avalonia tests that the .NET under Rosetta could swing anywhere from 30% slower to 30% faster than on native Intel depending on the functions.

By far the most disconcerting thing from this batch of tests was a fatal error which literally causes the system to force restart. It feels like the olden days of running on systems without protected memory systems. Thankfully I still compulsively hit the save key sequence very frequently from being burned in that era. There is still no excuse for it though. It’s one thing for there to be a fatal bug that crashes the dotnet CLR, it’s another to take down the whole computer. Hopefully between Apple and the .NET team it’s fixed relatively soon. In the mean time assuming your code avoids it one can expect reasonable but not eye watering performance on Apple Silicon with the .NET runtime.