Latency performance improvement after moving your legacy system to the cloud

This story is not about the journey of cloud migration, it is more about stabilization your application after migration. Most people would like to run their applications in the cloud but not all of them would like to spend a lot of time and money on preparation stuff before migration. The easiest way is to push the application to the cloud “as-is” (using the “lift and shift” approach). But overall, you should spend some time optimizing and adapting solution performance in the end. That is exactly what happened to me when I was involved in one project and I would like to share my personal experience on it.

Step back, we had a large legacy system based on ASP Net MVC running on IIS Windows Server's latest version. So, it was a traditional architecture application with MS SQL and some customization on front end (JavaScript). Everything was working locally on the same machine.

As you might see in the picture above, we have not done so many changes in the legacy system, just put the existing solution to the windows container and deploy it to the AKS service in Azure. For the presentation layer, we used Azure Front Door service and for the data hosting, we used two types of storage: Azure SQL Database and Storage Account with File Shares. We mounted File Shares as additional space for keeping application files and SMB was the protocol of communication.

After several UI performance testing, we have seen not enough performance, especially with the client-side. You can see it below,

The same issues we found here https://github.com/Azure/AKS/issues/85.

Well, we did change nothing with business logic, only infrastructure was changed. We identified a few areas where we believed something should be improved:

1. Front Door communication.
2. Ingress communication.
3. SQL communication.
4. File Share communication.

Front Door Communication

So, let’s move on. After analyzing Azure Front Door we have seen some dependencies on where the client is located. Since we used basic Front Door, it means the service is global, no change to set it up near the client.

To improve Azure Front Door we have at least two options:

The first one is using cache — https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/frontdoor/front-door-caching.md

The second one is to change Front Door Plan to use Standard/Premium — https://docs.microsoft.com/en-us/azure/frontdoor/standard-premium/create-front-door-portal.

Ingress Communication

The next step is ingress communication between Nginx and containers. Also, a few things we found to improve:

• It’s HTTP request headers that need to be used when you should configure routing by using hosts in ingress. So, in the backend pool of Front Door you need to out such configuration for every backend component.

• Do not forget about pod liveness and readiness. https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/. This is so important to have a running application 24/7 without downtime.

SQL Communication

The next area to tune is the communication container with Azure SQL database. We did not have really big problems with that but to analyze such performance I recommend enabling external events to get all events and traces. https://techcommunity.microsoft.com/t5/azure-database-support-blog/extended-events-capture-step-by-step-walkthrough/ba-p/369013. One more note of using Azure SQL Database is choosing the right data migration plan. A nice tool would be Azure Migration Tool https://docs.microsoft.com/en-us/sql/dma/dma-migrateonpremsqltosqldb?view=sql-server-ver15

File Share Communication

Last but not least is communication with Storage Account. As I mentioned before we used File Share in Premium Tier of Storage Account to mount as disks to containers and use it for keeping application files. After enabling log analytics we saw the cause of our problems with latency. Firstly, our storage account had another regional location, and it was affected. Another thing that we saw,

So many errors and so many transactions were affected our client performance. And this is not a part of infrastructure. This is a part of the business logic of the application itself. One related question is why it does not appear in the local environment? The answer is simple, locally the whole application was hosted in one machine and we did not see any network dependencies. And of course, we can add more IOPS and enable multichannel feature (https://docs.microsoft.com/en-us/azure/storage/files/storage-files-smb-multichannel-performance) but in our case, it did not help because needs to be redesigned application business logic. BTW, a good article about storage performance you can find here https://docs.microsoft.com/en-us/azure/storage/files/storage-troubleshooting-files-performance

To sum up my story, I would like to say that if you are planning or at least thinking about cloud migration, my recommendation is do not to choose “lift and shift” approach as a permanent solution. You need to be aware that you might see the same issues in the end.

Happy coding.