We have resolved all outstanding issues. The analytics were successfully restored this morning and the product is back to normal operations.
Posted Mar 26, 2020 - 18:51 CET
Update
We are continuing to work on the analytics issue. To provide some more technical details we are currently working on replaying events from our source of truth to the service we use to produce the dashboard analytics. We temporarily disabled the process that creates daily analytics last night so that it would not incorrectly aggregate yesterday and todays results. We expect to complete this task today and will re-enable this batch process this evening.
Posted Mar 25, 2020 - 14:35 CET
Monitoring
We have applied several performance fixes to our Swiss datacenter and are monitoring closely. Currently all metrics report green and datacenter is within normal parameters.
We are working on resolving the issue with our analytics but the numbers for 25.02.2020 may remain incomplete until tomorrow.
Posted Mar 25, 2020 - 10:20 CET
Update
The high load on the datacenter introduced errors in the Dashboard Analytics. The numbers for the 25.02.2020 can be off. Past data is not affected by this. We are currently running a script to solve the issues in the Analytics. We expect the numbers to be correct again in the next few hours.
Posted Mar 25, 2020 - 09:43 CET
Update
Today we saw a significant increase in usage. This highlighted several performance issues within our service. We are currently adding additional instrumentation to help identify the root cause of the issues. Currently the service performance is returning to normal due to a decrease in day time traffic. We will continue working on implementing measures to identify root cause and mitigate side effects.
Posted Mar 24, 2020 - 17:25 CET
Update
We are still investigating the issue and will provide updates as soon as possible.
Posted Mar 24, 2020 - 16:24 CET
Investigating
The site is currently experiencing a higher than normal amount of load, and may be causing pages to be slow or unresponsive. We're investigating the cause and will provide an update as soon as possible.
We will provide an additional update in 60 minutes.
Posted Mar 24, 2020 - 15:22 CET
This incident affected: Switzerland Datacenter (Data Center (Switzerland)).