Resolved -
We have been monitoring the datacenter and are happy to announce that we believe that the issue is now resolved.
Apr 12, 11:17 CEST
Update -
We continued to monitor the datacenter performance throughout the night. We did not detect any more performance anomalies during the night. We believe that the issue is resolved and will close this incident after 3 more hours of monitoring.
Apr 12, 07:17 CEST
Monitoring -
Our provider has exchanged the underlying infrastructure of our database servers. We believe that this was the root cause the issue. We are proactively working with our provider to identify any other hardware that could be impacted by this issue. We are now monitoring the situation to verify that the systems are fully operational
Apr 11, 18:33 CEST
Update -
We have started replacing the hardware. We must put the system into maintenance mode to continue
Apr 11, 16:46 CEST
Identified -
We believe that we have identified the issue. We are seeing disk degradation on the database servers. We are proceeding to exchanging the hardware on the database servers
Apr 11, 16:03 CEST
Update -
We have partitioned the network to handle different customers. In addition, we have added more metrics & monitoring into our infrastructure. The system is currently operating with a slightly degraded performance. We will continue investigating the situation, but we predict that we will have further stability issues until we find the root cause
Apr 11, 11:53 CEST
Update -
We continue to see degraded performance in our Swiss datacenter. We have been changing low level kernel settings on the system to no avail. We will start partitioning the network so that each customer will be handled speciality. We are scaling up our infrastructure to provide that additional layer service
Apr 11, 07:27 CEST
Update -
Performance is currently restored in the Swiss datacenter. However, we do not believe that we have yet identified the root cause of the issue. We will continue to investigate throughout the night to ensure that this issue does not resurface
Apr 10, 21:58 CEST
Update -
We are investigating any possible product changes that could have caused this issue. We still have not found the root cause of the performance degradation / outage
Apr 10, 20:13 CEST
Investigating -
We are still seeing availability issues after we have applied several fixes. We continue to investigate
Apr 10, 18:12 CEST
Monitoring -
The fix seems to have resolved most of the performance issues. We are now monitoring the situation
Apr 10, 14:35 CEST
Update -
We have scaled our persistent storage layer in an attempt to mitigate the situation. We believe that this will have a positive impact on the situation. We continue to monitor the situation and see if it has brought the desired results
Apr 10, 12:43 CEST
Update -
We are continuing to investigate this issue.
Apr 10, 10:41 CEST
Investigating -
We are still having performance issues in the Swiss Datacenter. The first remedies helped the situation temporarily and we are still facing degraded performance.
We will update as soon as we have identified the root cause of the issue.
Apr 10, 09:18 CEST
Monitoring -
We have applied an immediate fix to remedy the issue. The system is available again and we are monitoring the situation
Apr 9, 12:57 CEST
Investigating -
We are currently experiencing an outage in the Switzerland datacenter. The system is currently not available for customers who have their data stored in the Swiss datacenter
Apr 9, 12:28 CEST
