Microsoft Teams' journey to .NET Core

Microsoft Teams "MiddleTier" is an internal service that powers various scenarios in Microsoft Teams. It's one of the largest services consisting of 700+ APIs that are maintained by 10+ teams at Microsoft. Over the last two years, 50+ projects (libraries, tests, applications) under this service have been converted to .NET Standard 2.0 and .NET Core 3.1, having functional and performance equivalence (or better), and are now almost entirely running on .NET Core 3.1 in production, and looking to move to .NET 6 next. Before this migration, the service was running on .NET Framework 4.6.2 using ASP.NET Core 2.2 MVC pipeline. It runs on Azure Service Fabric with deployments in 35 Azure datacenters.

The scope of this migration was large because MiddleTier is an extra-large service in terms of the functionality that it provides with hundreds of developers that work on it every day.

Motivation for migration

The team was motivated to move to .NET Core 3.1 for the following reasons:

• Performance and cost-efficiency improvements
• .NET Framework 4.6.2 is likely reaching end of life soon
• Cross-platform support
• Move to a modern framework for better developer experience

Benefits after migrating to .NET Core 3.1

After migrating to .NET Core 3.1, the team has noticed the following improvements:

• 25% CPU improvement
• Around 25% reduction in infrastructure cost
• Improved thread pool usage
• Reduced technical debt and efforts in moving to annual .NET releases

The following charts show comparisons between .NET Framework and .NET Core. In the future, with .NET 6, we should see even further improvements.

Approach

The overall migration was divided into three stages:

They also chose to multi-target the application to .NET Framework and .NET Core so that they would have both binaries available, and they could continue to roll out .NET Core slowly.

Learnings

OData and other REST APIs can't share route prefix

Their service has few OData endpoints as well along with many REST endpoints. These two shared the same routing prefix for the endpoints. This used to work fine in .NET Framework but because of routing changes, this stopped working in .NET Core. They had to move OData APIs to a different route prefix to resolve this.

Performance issues with OData client libraries

The HttpWebRequest pattern with OData clients to make calls to downstream OData APIs results in higher latency compared to .NET Framework. This was due to a regression in .NET Core in which the framework doesn't cache connections. This is already resolved in newer versions of .NET.

Issues with Azure Service Bus SDKs

The Azure Service Bus SDK had to be upgraded as part of this migration since the old version isn't .NET Standard compatible. The latest version of the Azure Service Bus SDK sends the request payload in JSON format whereas the older SDK sends the payload in XML format. To continue using XML payload, they had to use the DataContractSerializer.

Issues in Service Fabric project for multi-targeting

The Service Fabric Project (sfproj) doesn't support multi-targeting inherently. They had to do workarounds in the build pipeline to produce Service Fabric packages for both target frameworks.

Issues with the older version of MimeKit NuGet

The older version of MimeKit can have issues with double byte characters, thus language specific validation is advised in this scenario. They uncovered similar issues when they rolled out to deployments located in Asian geography.

Classic ASP.NET quirks

• Had to remove usage of some of the .NET Framework classes that were marked internal in .NET Core.
• MVC async suffix dropped from action names as mentioned in the ASP.NET Core breaking changes for versions 3.0 and 3.1 article. If any of the code paths are dependent on the action name, it can cause a change in behavior.
• Synchronous IO operations are by default turned off on all servers starting in .NET Core 3.0 as mentioned in the dotnet/aspnetcore#7644 GitHub issue.
• Content-Length header not set in Content.Headers when sending StreamContent as HTTP request content. It can result in errors from downstream calls.
• .NET framework produces stable hash code for a string, but .NET Core doesn't.
• The Required attribute from the System.ComponentModel.DataAnnotations namespace behaves differently in .NET Core. On .NET Framework, this attribute doesn't do any model validation for non-null fields but on .NET Core it does.

Future

Every new release of .NET comes with tremendous productivity and performance improvements that continue to help accomplish our goals to build resilient, scalable, performant, and secure services. The team will continue to leverage the improvements made in .NET by upgrading to .NET 6 next.