Thomas Wolf· @Thom_Wolf · X·· 3 小时前AI 评分47
AI 导读
来自 @microagi 关于传输 PB 级机器人数据的文章里有个有趣的数据: 他们向 Hugging Face 重新上传了一个 239 GB 的数据集,只做了小幅修改。结果只有 2.6 MB 走了网络,大约一分钟就完成了(!) 而且这在规模化时持续见效,因为机器人录制数据高度重复(相同的场景、相同的设备)。然后 @Cloudflare workers 把它分发到任何云,超过 1 PB 只需约 500 美元 团队写的这篇文章很棒:https://www.microagi.ai/articles/delivery-platform
正文 · 原文
fun stat from @microagi’s write-up on shipping petabytes of robot data:
they re-uploaded a 239 GB dataset with small changes to Hugging Face. only 2.6 MB went over the network, done in about a minute (!)
and it keeps paying off at scale, because robot recordings are super repetitive (same scenes, same rigs). then @Cloudflare workers fan it all out to any cloud, >1 PB for ~$500
great write-up by the team: https://www.microagi.ai/articles/delivery-platform
As our customers scale to training robot models using millions of hours of our recordings, we needed to build a new kind of data infrastructure. Terabytes of data delivered in minutes, and the capacity to deliver over a Petabyte per day to each of our customers, to multiple customers at the same time. That’s more than 10 Gigabytes a second, delivered into any cloud, wherever our customers need it. Using our customer portal, customers select precisely the data they want and it starts arriving in minutes at their destination, including AWS, GCP, Cloudflare, Hugging Face and any cloud provider who can handle the ingest speed. We’ve written about how we worked with @huggingface and @Cloudflare to build this secure, scalable infrastructure. Link in comments.在 X 查看被引用的帖子
来源:Thomas Wolf · x.com