C# JSON Deserialization Performance Basics

Shortly before .NET Core 3.0 came out, I was getting quite excited about System.Text.Json, specifically about the performance claims from the blog post. However, those performance claims were measured in requests per second on an ASP.NET Core web application. One of the design goals of System.Text.Json was to design it with async in mind, which is of benefit to high throughput, IO-bound workloads, such as server applications. But async is a theoretical performance penalty when the number of tasks scheduled on the Thread Pool not not exceed the number of CPU cores available, or for CPU-bound work. I was working on the NuGet client at the time, so I was curious what the client performance characteristics were. Here's a look back into what I learned, with a quick investigation into .NET 5 improvements.

What was tested

Looking at the code I wrote for the benchmark, I see that I never finished several things I wanted to test. I wrote implementations/benchmarks for:

• Automatic conversion with Newtonsoft.Json
• Automatic conversion with System.Text.Json
• Custom JsonConverter with Newtonsoft.Json
• Custom JsonConverter with System.Text.Json
• An unnecessarily complicated case that mimicked the NuGet client as best as I could figure out, but in reality it's just parsing the JSON stream into Newtonsoft.Json's JObject, and then converting that into the strongly typed model.

I think these were the first JsonConverters I ever wrote, so I'm sure that I didn't write them in the ideal way. Originally, I wanted to spend time optimizing them, but I never got around to that. I wrote them before these docs on writing custom converters existed, which might have tips on better ways to implement than what I wrote. Additionally, I wanted to attempt writing implementations using only JsonReader and Utf8JsonReader, to avoid the perceived overhead of the JsonSerializer needing to find a JsonConverter to use. Honestly, I don't believe it would make a significant impact, but without benchmarks it's just speculation. I would have liked to implement custom converters that use a string cache, similar to what .NET string interning does. If two different search results use the same version string, or any other string, that string should be allocated once in memory at the end of the deserialization, not once per search result, each of which has only a single reference. Finally, rather than using a switch statement (or a series of if-else blocks) to determine which property value is being read, I would have liked to try using a radix tree, to see if this can speed up property name lookups, at least for classes with a large number of properties.

The benchmarks test one the NuGet server API's resource types, search. I chose this because at the time .NET Core's CI infrastructure was publishing packages to a static NuGet feed on Azure's Blob Storage. Being a static feed, this meant that search queries would always return the same static result set, but more importantly would return every package in the feed. It was exploiting the fact that the official NuGet client, and therefore Visual Studio, would show all results from the search query. However, as the dotnet-core feed was getting new packages on every CI build (of release/merged branches, not PR builds), the feed was growing quickly, and was huge. While memory usage of a JToken is proportional to the length of the JSON text is represents, the in-memory model uses an order of magnitude more memory, at least during deserialization. The 220 MB JSON file could use over 2 GB of memory when deserialized into a JToken. Since Visual Studio is a 32-bit process, a large enough search result could cause Visual Studio to exceed 3.5 GB of memory, and therefore crash being out of memory. This is why my benchmarks are in a class named FullResultBenchmarks.cs, as I had intended on writing additional benchmarks that would read the first x results, and ignore the rest of the file. Since then, that's exactly what we implemented in the official NuGet client, so Visual Studio will no longer crash, regardless of how many search results the NuGet feed returns.

So, I had intended to optimize my JsonConverter implementations, add more implementations, and benchmark different scenarios. However, like many other people, I have more things I want to do than time to do it. I was distracted for long enough that I eventually forgot about this plan. If I hadn't decided to write this blog post about it, I probably never would have remembered. Maybe I'll eventually get around to it, but this is my first ever blog post, and given the amount of effort it took to decide on a blog platform, domain name, and get everything set up, I wanted a relatively low effort first post. Hence I'm writing about something I did in the past.

Methodology and Results

I use BenchmarkDotNet for my benchmarks, although I manually created the table below, as the default output didn't show exactly what I wanted to highlight. Anyway, it's a great library which I highly recommend to all .NET developers trying to benchmarks anything. It avoids several pitfalls/mistakes easy to make if trying to write your own benchmark runner and analysis. I made some changes to the benchmark app to make the JToken benchmark more fair, and to test multiple runtimes. So, if you look at the commit timestamp, it's recent, but it was just infrastructure changes. The business logic (converters) was written well over a year ago. Unless explicitly mentioned otherwise, all the results are from the nuget-org.json tests. It's a 195 KB file, compared to dotnet-core.json, which is a 226 MB file, so over 1000 times as large. It made each run of the benchmarks last 36 minutes, rather than 5 minutes.

I tested 3 versions of .NET, .NET 5, .NET Framework 4.8 and .NET Core 3.1. .NET 5 is the current version of .NET at the time I'm writing this. Although I compiled my benchmarks against .NET Framework 4.7.2, since I have .NET Framework 4.8 installed, that's the version of the runtime that was tested. Finally, since .NET 5 is not a Long Term Support (LTS) release, I also tested .NET Core 3.1 which is the latest LTS release. For .NET Core 3.1 and .NET 5, I used the bundled version of System.Text.Json. For .NET Framework, I used package version 5.0.0, unless stated otherwise.

Method	.NET Framework 4.8	.NET Core App 3.1	.NET 5
'System.Text.Json JsonSerializer'	1.52	1.39	1.00
'System.Text.Json with JsonConverters'	1.59	1.02	1.00
'Newtonsoft.Json JsonSerializer'	1.88	1.78	1.74
'Newtonsoft.Json with JsonConverters'	1.25	1.21	1.16
'Newtonsoft.Json with JTokens'	4.71	4.53	4.26

Maybe at a later time I'll update this post to add a chart, but here's a list of interesting things I see.

• System.Text.Json is 50-60% slower on .NET Framework than it is on .NET 5.
• My custom System.Text.Json converter has the same performance on .NET Core 3.1 and .NET 5, but the default converter was 40% slower on .NET Core 3.1, whereas it has the same performance on .NET 5. On .NET Framework, my custom converter has the same performance as the default converter, taking into account the .NET Framework performance decrease from the point above.
• Improvements to the runtime benefit Newtonsoft.Json. All 3 methods showed improvements going from .NET Framework 4.8 to .NET Core 3.1, and more improvements again going to .NET 5. There's about a 10% performance gain by having the exact same code, same library version, just different runtime.
• Using a custom converter with Newtonsoft.Json is very beneficial. Around about 33% faster on all 3 runtimes.
• As expected, parsing to JToken, and then converting to strongly typed classes is much slower than parsing directly into the class. With this specific schema and data size, it's around 2.7 times slower than using custom converters, which was the highest performance Newtonsoft.Json implementation. There were about 4 times as many gen 0 and gen 1 garbage collections, and 4 times as much memory allocated. Interestingly there were no gen 2 garbage collections in any of the benchmarks.
• Newtonsoft.Json with custom converters are faster than any System.Text.Json implementation on .NET Framework, despite the higher number of garbage collections and memory allocated.

I don't want to create a lot of tables of data, so I'll just say that using the System.Text.Json 5.0.0 package on .NET Core 3.1 closes about half of the performance gap between the custom and default converters, compared to migrating to the .NET 5 runtime instead. Referencing System.Text.Json version 4.7.2 (the version that ships with .NET Core 3.1) does not downgrade what gets used at runtime on .NET 5, so the performance improvement remained on .NET 5.

Regarding deserializing the 200 MB dotnet-core.json, rather than the 200 KB nuget-org.json, everything was roughly in line with the smaller input file, except my custom System.Text.Json converters seemed to be a little bit worse, and the performance gap between Newtonsoft.Json and System.Text.Json increased a bit. With the smaller nuget-org.json tests, sometimes when I ran it, the .NET 5 results showed my custom converters 2% faster than the default converter, sometimes 5% slower. But on the large dotnet-core.json file, it's consistently 5-8% slower.

Conclusions

I tried to keep the results above as factual as possible, so here is my commentary:

• System.Text.Json 5.0.0's default converters are as fast as a basic custom converter. So it's a waste of your time to implement a custom converter yourself. Instead, upgrade your runtime to .NET 5 if you're not already using it. Especially if you're already on .NET Core, the effort to upgrade the runtime is probably less than writing custom converters, and you'll get a whole lot of other performance gains, not just JSON deserialization.
• On the other hand, if you're using Newtonsoft.Json, you should write custom converters for all your classes where you care about performance.
• While the System.Text.Json team claim that System.Text.Json is about 2x faster than Newtonsoft.Json, for this schema the smaller nuget-org.json was only 15% faster, and the larger dotnet-core.json was 40% faster. It's significant, but much less than what their blog claims.
• If you can't upgrade your app's runtime for some reason, then you may get non-trivial performance gains by referencing the latest versions of the packages, at least if you're using the default converters, but it still won't be as much as upgrading your runtime version. There doesn't appear to be any harm if you upgrade to a newer runtime and forget to update the package version.
  • Personally, I'd recommend removing any PackageReferences that you can. It helps avoid any NU1605 package downgrade issues, and it's fewer things to check for upgrades. When you multi-target a project and you need to reference a package or assembly for some target frameworks, I suggest using conditions, as I did in my project file.
• While both of my System.Text.Json benchmarks were faster than all Newtonsoft.Json benchmarks on .NET Core and .NET 5, I found it very interesting that on .NET Framework that using custom converters with Newtonsoft.Json was the fastest method. Newtonsoft.Json still had about double the number of garbage collections and memory allocations, yet overall System.Text.Json was still 20% slower.
  • I didn't look into performance counters or ETW traces to determine whether or not the System.Text.Json methods were completing synchronously or if they were scheduling async completions on the thread pool, but even if it was doing async work it still feels like a higher performance cost than I would have expected. Not that I have any experience measuring sync vs async methods. Maybe this is totally expected.
  • Having said that, I know that System.Text.Json makes use of ValueTask, whereas that never existed on the .NET Framework. While ValueTask is a package that can be used in projects targeting the .NET Framework, allowing the C# to be the same across different target frameworks, maybe the .NET Core runtime is more optimized to understand it at a primitive level, while the .NET Framework doesn't. Or maybe ValueTask is purely a compile-time concern, with no runtime optimizations even possible. I don't know, I'm not an expert on that topic and never thought about it until I started writing this paragraph.

I think this all reinforces the suggestion that I commonly see on the internet, and hear in person, where the first step to any performance improvement is taking measurements. Even with the evidence that that System.Text.Json causes fewer garbage collections and allocations, when using a custom converter, it's faster than System.Text.Json on the .NET Framework. This goes against general advice that fewer allocations is faster. So, even if you know a guideline which "everyone" says improves performance, you still should measure. Particularly since computers have multiple dimensions of what performance means: Total memory usage, latency, and throughput. Optimizing any one of these can harm the performance of the other two.

In the end, when I took a performance trace of doing a NuGet restore on the NuGet.Client repo, I found that Newtonsoft.Json was in the CPU stack just 0.5% of the time. Half of one percent. Even if we could get a 5x performance gain, going from half of one percent to one tenth of one percent of CPU time is not worth the effort, when I could be doing more impactful work instead. Furthermore, my initial investigation to see if it's worthwhile to migrate from Newtonsoft.Json to System.Text.Json appears to have shown that as long as NuGet runs on .NET Framework, no, it is not worthwhile. Although if JSON deserialization used more than 0.5% of CPU time, then we should at least write our own converters. However, when I recently had some spare time, and knew it wouldn't take much effort, I still created a pull request to stop (de)serializing via JTokens.