So lately I had to speed up the server with my colleagues at work so it could process requests faster. In our case we have a lot logic that needs to be executed which takes a bit of processing power, it's not a normal web server that just renders a simple page. There are tens of thousands of lines of code that need to be executed and not a couple of hundreds or a few thousands in the case of a normal web server.
And to successfully process a request a lot of times the server needs to put the request on-hold until it receives all information about that request freeing that thread to process another request. This is the first place where performance issues can arise because of context switching. Once a request is put on-hold and after that reprocessed, all the context must be restored. This can involve loading heavy stuff from the database. Ideally you should make sure that you load the context from the database once a request is actually ready to be processed and once all the required information has been received for it. If you load all the context for that request from the database only to see that you do not have all the information to process it then you would have wasted a lot of database calls.
As a general rule, you should do as much work as possible once you load all your data from your database. In our case after optimization it takes around 70 milliseconds to restore the request context from the database. Which is pretty considerable. Sometimes a request has to be put on-hold 3 times in a row only to be processed in the end. Only load as much data as you need and then make maximum use of it.
Secondly, exceptions are a bad as they slow down the code by a considerable amount. In the CLR virtual machine there are more than 10 thousand lines of code which are executed if an exception is thrown. Each throw exception takes at least 30 milliseconds to process in general. If you throw one or a couple of them while you process a request your server won't be able to handle more than 10 or 20 requests per second.
Thirdly, insert or delete operations in the database can be quite costly especially for complicated table structures with restrictive isolation levels. You should minimize them. If you do a lot of insert/delete operations when you process a request it will have a big negative impact on the performance. In our case all the insert/delete operations takes around 50 milliseconds on average. Try to figure out if you really need that many insert operations. If possible you can do them asynchronously after you process the request. Honestly having bigger tables here with more rows helps. Adding indexes or materialized views in the database can increase the insert and delete times which can directly slow down the server without you changing a line of code in your application.
When you need to get some data or wait for more information to finish processing a request you should do everything in parallel as much as possible and once they are finished you can start processing it. This way the request processing will be delayed as much as the longest retrieval operation instead of the sum of all the retrieval operations. Or if you need to wait for more information, then wait for all the individual pieces of information at once instead of waiting for each piece at a time.
If you have a tree like structure in a database between some tables then it is a very bad idea to do more than 3 or 4 joins between those tables. Especially if root node has a lot of children.
Finally, too much logging in your code can have a big impact too. Especially in big loops or repeatable actions. Here is the tricky part, too little logging information and you won't be able to fix bugs that appear at clients. Too much logging and you will slow down your server. And too much log entries actually makes debugging much harder because you will have much more irrelevant information. I ran a test and without logging I could process in around 6.5 minutes the amount of requests that I could process with logging enabled in around 7 minutes. So it's a 30 second difference. But the test wasn't perfect and there may have been other factors.
Those are pretty much all the lessons that I learned.
And to successfully process a request a lot of times the server needs to put the request on-hold until it receives all information about that request freeing that thread to process another request. This is the first place where performance issues can arise because of context switching. Once a request is put on-hold and after that reprocessed, all the context must be restored. This can involve loading heavy stuff from the database. Ideally you should make sure that you load the context from the database once a request is actually ready to be processed and once all the required information has been received for it. If you load all the context for that request from the database only to see that you do not have all the information to process it then you would have wasted a lot of database calls.
As a general rule, you should do as much work as possible once you load all your data from your database. In our case after optimization it takes around 70 milliseconds to restore the request context from the database. Which is pretty considerable. Sometimes a request has to be put on-hold 3 times in a row only to be processed in the end. Only load as much data as you need and then make maximum use of it.
Secondly, exceptions are a bad as they slow down the code by a considerable amount. In the CLR virtual machine there are more than 10 thousand lines of code which are executed if an exception is thrown. Each throw exception takes at least 30 milliseconds to process in general. If you throw one or a couple of them while you process a request your server won't be able to handle more than 10 or 20 requests per second.
Thirdly, insert or delete operations in the database can be quite costly especially for complicated table structures with restrictive isolation levels. You should minimize them. If you do a lot of insert/delete operations when you process a request it will have a big negative impact on the performance. In our case all the insert/delete operations takes around 50 milliseconds on average. Try to figure out if you really need that many insert operations. If possible you can do them asynchronously after you process the request. Honestly having bigger tables here with more rows helps. Adding indexes or materialized views in the database can increase the insert and delete times which can directly slow down the server without you changing a line of code in your application.
When you need to get some data or wait for more information to finish processing a request you should do everything in parallel as much as possible and once they are finished you can start processing it. This way the request processing will be delayed as much as the longest retrieval operation instead of the sum of all the retrieval operations. Or if you need to wait for more information, then wait for all the individual pieces of information at once instead of waiting for each piece at a time.
If you have a tree like structure in a database between some tables then it is a very bad idea to do more than 3 or 4 joins between those tables. Especially if root node has a lot of children.
Finally, too much logging in your code can have a big impact too. Especially in big loops or repeatable actions. Here is the tricky part, too little logging information and you won't be able to fix bugs that appear at clients. Too much logging and you will slow down your server. And too much log entries actually makes debugging much harder because you will have much more irrelevant information. I ran a test and without logging I could process in around 6.5 minutes the amount of requests that I could process with logging enabled in around 7 minutes. So it's a 30 second difference. But the test wasn't perfect and there may have been other factors.
Those are pretty much all the lessons that I learned.
Comments
Post a Comment