Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps. In this paper, we revisit Marigold, a set of techniques for repurposing modern image generation and editing models, powered by the diffusion transformer (DiT) architecture, into state-of-the-art monocular depth estimators. Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run. We analyze the artifacts of naive training and identify two effective remedies: aligning the model's internal representations with semantic features extracted from ground-truth, and adopting a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D. Qualitatively, our model resolves fur, foliage, and hair-thin edges that have eluded prior models. Furthermore, Marigold V2 achieves state-of-the-art results when applied to other dense regression tasks, such as surface normals estimation and intrinsic image decomposition. Project website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web
Marigold V2:用扩散Transformer重做单目深度估计
AI 导读
Marigold V2 提出基于扩散Transformer(DiT)架构的单目深度估计新方法,支持从预训练多步流匹配模型单步推理,并可按需量化以降低运行成本。通过将模型内部表征与真实语义特征对齐、采用基于新型Sinkhorn损失的两阶段微调协议,模型在KITTI和ETH3D上的AbsRel较此前最优结果提升16-26%,并能清晰还原皮毛、 foliage 等细薄边缘。
HuggingFace Daily Papers(社区热门论文)
45
AI 编辑部评分,满分 100Marigold V2:用扩散Transformer重做单目深度估计
Marigold V2 提出基于扩散Transformer(DiT)架构的单目深度估计新方法,支持从预训练多步流匹配模型单步推理,并可按需量化以降低运行成本。通过将模型内部表征与真实语义特征对齐、采用基于新型Sinkhorn损失的两阶段微调协议,模型在KITTI和ETH3D上的AbsRel较此前最优结果提升16-26%,并能清晰还原皮毛、 foliage 等细薄边缘。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org