In recent months, the number of our users and the size of their data has been growing. We value this trust and strive to ensure fluency of Evolio , we started to increase your performance every morning so that you could work comfortably. We have always wanted and want to be honest, that is why we are writing this article. Last Thursday (November 12) we saw a significant slowdown in the whole of Evolio, due to the fact that the increase in performance did not go well. Learn why it happened, how we fixed it, and how we look forward to it in the future.
Explanation in a nutshell
- Thursday 12 November at 6:12 a.m. we noticed a significant slowdown in server response
- We started looking for the cause and the way to fix it
- Routine repair methods did not work
- The tracks led to a script that runs additional servers
- Cause: The additional server did not actually start
- After two hours, the system runs at full power
- We increase overall performance and add preventive measures
More detailed explanation
Each law firm has its (separate) database on our servers. There are already over 400 of them. Cloud performance is automatically added based on the current server load. Figuratively speaking, the auxiliary motors (servers) are switched on. And they didn’t turn on, and Evolio drove “in slow motion” for less than two hours.
As we found out
We automatically monitor the operation of the system, among others. and the processing time of the request. As a requirement, imagine sending a data message, clicking on an order or saving a work report. This should happen in a tenth of a second. Suddenly, however, the request processing time jumped to 2 seconds. This triggered an automatic alert and we started to deal with it.
What was the problem?
According to server monitoring, all services worked normally. Both database and web server. Two “additional” servers were also launched. BUT the main server was busy at 90% and the additional barely at 1% performance. As if they were running empty.
And there was a mistake. Our “power-on” script correctly assessed the need to add performance and turned on the servers. They looked on, but didn’t actually run. It was caused by a specific bug in MS Cloud (we reported it and it is being fixed).
What we have done now and in the future
- We’ve modified the startup script. Newly, it not only checks the response of the additional server, but also sends the request to the database and verifies its execution.
- We start the additional server when the main load is 65 %.
- We have also increased the basic performance of the main server.
- During this week, we will add a separate server that will only process user requests.
Just to give you an idea of what the system normally does. For example, we only update data on 150,000 insolvency proceedings from the insolvency register every day. Or we generate and send 10,000 payment orders at once. And normally no user when working with legal agenda does not record.
We are sorry for the slowdown and we apologize for it.
Traps of the home Internet at the Home Office
Of course, how your internet connection works is important for the good functioning of Evolia. If you are at home at the Home Office, you should know that most household connections are shared. You can be alone at home, but when a neighbor’s children start Netflix and they go to YouTube, it also affects the state of the Internet at home.
You can check the speed of your internet with services that measure its speed. For example, you can use Czech rychlost.cz or the world’s most popular speedtest.net . You can use both for comparison, just don’t run them at the same time 🙂
