We have been monitoring the datacenter and are happy to announce that we believe that the issue is now resolved.
Posted Apr 12, 2019 - 11:17 CEST
Update
We continued to monitor the datacenter performance throughout the night. We did not detect any more performance anomalies during the night. We believe that the issue is resolved and will close this incident after 3 more hours of monitoring.
Posted Apr 12, 2019 - 07:17 CEST
Monitoring
Our provider has exchanged the underlying infrastructure of our database servers. We believe that this was the root cause the issue. We are proactively working with our provider to identify any other hardware that could be impacted by this issue. We are now monitoring the situation to verify that the systems are fully operational
Posted Apr 11, 2019 - 18:33 CEST
Update
We have started replacing the hardware. We must put the system into maintenance mode to continue
Posted Apr 11, 2019 - 16:46 CEST
Identified
We believe that we have identified the issue. We are seeing disk degradation on the database servers. We are proceeding to exchanging the hardware on the database servers
Posted Apr 11, 2019 - 16:03 CEST
Update
We have partitioned the network to handle different customers. In addition, we have added more metrics & monitoring into our infrastructure. The system is currently operating with a slightly degraded performance. We will continue investigating the situation, but we predict that we will have further stability issues until we find the root cause
Posted Apr 11, 2019 - 11:53 CEST
Update
We continue to see degraded performance in our Swiss datacenter. We have been changing low level kernel settings on the system to no avail. We will start partitioning the network so that each customer will be handled speciality. We are scaling up our infrastructure to provide that additional layer service
Posted Apr 11, 2019 - 07:27 CEST
Update
Performance is currently restored in the Swiss datacenter. However, we do not believe that we have yet identified the root cause of the issue. We will continue to investigate throughout the night to ensure that this issue does not resurface
Posted Apr 10, 2019 - 21:58 CEST
Update
We are investigating any possible product changes that could have caused this issue. We still have not found the root cause of the performance degradation / outage
Posted Apr 10, 2019 - 20:13 CEST
Investigating
We are still seeing availability issues after we have applied several fixes. We continue to investigate
Posted Apr 10, 2019 - 18:12 CEST
Monitoring
The fix seems to have resolved most of the performance issues. We are now monitoring the situation
Posted Apr 10, 2019 - 14:35 CEST
Update
We have scaled our persistent storage layer in an attempt to mitigate the situation. We believe that this will have a positive impact on the situation. We continue to monitor the situation and see if it has brought the desired results
Posted Apr 10, 2019 - 12:43 CEST
Update
We are continuing to investigate this issue.
Posted Apr 10, 2019 - 10:41 CEST
Investigating
We are still having performance issues in the Swiss Datacenter. The first remedies helped the situation temporarily and we are still facing degraded performance.
We will update as soon as we have identified the root cause of the issue.
Posted Apr 10, 2019 - 09:18 CEST
Monitoring
We have applied an immediate fix to remedy the issue. The system is available again and we are monitoring the situation
Posted Apr 09, 2019 - 12:57 CEST
Investigating
We are currently experiencing an outage in the Switzerland datacenter. The system is currently not available for customers who have their data stored in the Swiss datacenter
Posted Apr 09, 2019 - 12:28 CEST
This incident affected: Switzerland Datacenter (Data Center (Switzerland)).