One Click LCA Platform - Incident Report - March 8th, 2021

SUMMARY

Between 22 February and 8 March 2021, One Click LCA has had intermittent performance degradation leading to significantly longer page loading times. Following our investigations, the root cause of this was the added load of malfunctioning report generation feature which slowed down the software. This issue has now been resolved. In addition of addressing the issue, we have also initiated a number of measures to avoid future bottlenecks. We have also doubled our server capacity. We apologize for the inconvenience caused.

Timeline

09:00 GMT - 9th of March

A patch has been introduced which resolves the issue with report generation.

16:00 GMT  - 8th of March

We've temporarily disabled the results report generation in the RICS tool, which lead to stabilized server performance. 

13:00 GMT - 8th of March

We are currently investigating performance issues on the One Click LCA platform. This is currently leading to longer loading times for all users.

22nd of February - 8th of March

Occasional occurrences of performance issues have been reported.

POST-MORTEM

Root Cause Analysis

One Click LCA has had intermittent performance degradation leading to significantly longer page loading times. Following our investigations, the root cause of this was the added load of malfunctioning report generation feature which significantly slowed down the software. 

Resolution

To help with resolving the issue, we initially disabled the results report generation on the RICS tool. A patch has been introduced to the One Click LCA server which resolved the issue. 

• Disabling the results report generation stabilized the server performance. 
• A patch was required to in order to turn the results report generation functionality in the RICS tool back on.

At 16:00 GMT on March 8th, server performance was restored.

At 09:00 GMT on March 9th, full service availability was restored. 

Remediation Items

In addition of addressing the issue, we have also taken a number of measures to avoid future bottlenecks.

1. We have introduced several performance optimization measures.
2. We have doubled our server capacity.