HuggingFace download issues

This incident has been resolved.

Still seeing signs of improvement, most models are able to spin up with a few still waiting their turn from HuggingFace. Models should spin up after enough tries if they are failing for HuggingFace related reasons.

We have seen most HuggingFace calls start succeeding, except for a small subset that are being downloaded through [cas-bridge.xethub.hf.co](http://cas-bridge.xethub.hf.co "cas-bridge.xethub.hf.co"). Most models are now succeeding to spin up

HuggingFace is reporting recovery, which we are also seeing on our end. However, we still see minor degradation, which we believe is due to thundering herd once the main issue was resolved. We are still keeping an eye out, but believe we are recovering

We are aware of 504s being returned by models that reach out to HuggingFace during setup. We believe this is likely related to the disruption visible at [https://status.huggingface.co/](https://status.huggingface.co/ "https://status.huggingface.co/"). We are monitoring and will update when we can identify the HuggingFace CDN is back up/