Performance Degradation within the product
Incident Report for Beekeeper
Resolved
We have re-enabled stream deletion for the majority of our customers. Those customers still impacted have received direct notification.

In terms of performance we have continued to closely monitor the situation and I am happy to inform you that there have been no performance regressions since our last update on June 29th.

We thank you for your patience and apologise for any inconvenience caused by disabling this functionality (stream deletion).
Posted Aug 12, 2021 - 21:42 CEST
Update
We are continuing to monitor the situation and configuration changes closely to minimize any disruptions.
We will keep stream deletion temporarily disabled and expect to re-enable it mid-July. In the interim for urgent requests regarding stream deletion please contact support.
Posted Jun 29, 2021 - 17:53 CEST
Update
We are continuing to monitor the situation for any further issues over the next 24 hours.
Posted Jun 28, 2021 - 21:08 CEST
Monitoring
We have applied a fix for the specific performance issue observed. We are continuing to monitor for any further issues. Stream deletion is still temporarily disabled and search results can still be outdated.
Posted Jun 24, 2021 - 23:38 CEST
Identified
The site is again experiencing a higher than normal amount of load, and may be causing pages to be slow or unresponsive. We're investigating the cause and will provide an update as soon as possible.

We will provide an additional update in 30 minutes.
Posted Jun 24, 2021 - 23:05 CEST
Update
We are continuing to monitor for any further issues. Stream deletion has been temporarily disabled and search results may be outdated.
Posted Jun 24, 2021 - 14:52 CEST
Update
We are continuing to monitor for any further issues.
Posted Jun 24, 2021 - 14:37 CEST
Update
We are continuing to monitor for any further issues.
Posted Jun 24, 2021 - 14:07 CEST
Update
We have implemented a mitigation for the issue and expect performance to return to normal. We are monitoring the situation.
Posted Jun 24, 2021 - 13:30 CEST
Monitoring
Database restart has completed. The system should now be returning to normal operations. We are monitoring the situation
Posted Jun 24, 2021 - 03:23 CEST
Update
We are forced to perform a database reboot. This will cause a short outage on the entire system. We will attempt to bring up the DB as soon as possible
Posted Jun 24, 2021 - 03:15 CEST
Update
We have applied multiple fixes to the performance issues. Currently, most of the system behaves well, with some exceptional cases. The system will be fully restored to normal operations within the next 1.5 hours. Unfortunately, we are unable to speed up the process as we require our database to fully recover from the performance problem it encountered.
Posted Jun 24, 2021 - 01:08 CEST
Identified
Unfortunately, the performance degradation still persist. We believe to have identified the issue and work on restoring normal operations.

We will provide an additional update in 60 minutes.
Posted Jun 23, 2021 - 23:38 CEST
Update
Datacenter is performing normally. We will continue to monitor throughout the day to ensure that it remains fully operational as we head towards peak traffic.

Will provide further updates if we see any regressions.
Posted Jun 23, 2021 - 13:52 CEST
Update
Remaining issues should be resolved. Conversations, campaigns and surveys should be working normally again.

We will provide an additional update in 180 minutes.
Posted Jun 23, 2021 - 08:25 CEST
Update
Currently our mitigation has restored performance of our datacenter to normal levels.

However some functionality is still impacted by the underlying issue. Currently it is not possible to create new chats. All existing chats should work. This also impacts campaigns and surveys which may fail to send.

We expect this issue to be resolved in the next couple of hours once our configuration changes are fully applied to the database.

We will provide an additional update in 7 hours or as situation evolves.
Posted Jun 23, 2021 - 02:57 CEST
Monitoring
We have implemented a mitigation for the issue and expect performance to return to normal. There may be some delays in certain functionality around analytics and push notifications as we work off the queue backlog.

We will provide an additional update in 8 hours or as situation evolves.

We will also conduct a post-mortem to evaluate how to avoid such issues in future.
Posted Jun 23, 2021 - 01:26 CEST
Identified
We have identified the cause responsible for the performance degradation and are working on providing a solution.

We will provide an additional update in 120 minutes.
Posted Jun 23, 2021 - 00:13 CEST
Investigating
The site is currently experiencing a higher than normal amount of load, and may be causing pages to be slow or unresponsive. We're investigating the cause and will provide an update as soon as possible.

We will provide an additional update in 60 minutes.
Posted Jun 22, 2021 - 22:51 CEST
This incident affected: US Datacenter (Data Center (Oregon)).