Scheduled Tasks on @Muse - An update on reliability

We deployed a fix to Muse to improve scheduled task and cron recovery and reliability especially, and have hardened our monitoring and alerting of the cron systems in the VMs.

Over the last few days the Muse team has been deploying a few changes to:
Improve the KV cache hit rate after increasing the context window size of the underlying model
Improve our VM scheduling algorithm to reduce downtime during deployments
Deploy a new authentication token format across the fleet for more fine-grained access.

During these changes we had a few SEVs (outages) where the inference tier was unresponsive, and the VM scheduling resulted in VMs not being available for longer periods of time during deployment and being unresponsive to clients.

We observed after we recovered from these outages that several crons and scheduled tasks failed to resume / heal properly when the outage was done, resulting in several users receiving reports from their Muse that the crons have been failing for a while.

We apologise for this incident, and know how much this disrupts your flow especially since many users rely on muse to reliably deliver outcomes for them. We take these outages seriously, and are running a full audit of our recovery systems and tools, and are adding more dashboards and alerts so you can continue to receive the high quality service we want all users to have.