Support of "depth-anything/Depth-Anything-V2-Metric-*-*-hf"

[Disk-MTH]

Model description

Hello everyone,

I have a use case where I'd like to use a depth estimation model, not just a relative one, but an absolute one. I saw that the relative model onnx-community/depth-anything-v2-small is a conversion of depth-anything/Depth-Anything-V2-Small, so I thought I'd just convert the model myself using the tools you provide.

So I tried using your online conversion tool: https://huggingface.co/spaces/onnx-community/convert-to-onnx

But I keep getting an error:

Please refer to https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/export_a_model#custom-export-of-transformers-models for an example on how to export custom models. Please open an issue at https://github.com/huggingface/optimum/issues if you would like the model type depth-anything to be natively supported in the ONNX export.

Then, I tried with script/convert.py (I ran venv, installed the dependencies...) and still got the same error.

I spent the afternoon trying everything, searching through issues, testing all the LLMs hoping for a sensible answer, but to avoid smashing my screen, I had to resort to opening an issue here.

I don't understand how you exported onnx-community/depth-anything-v2-small and how to reproduce it. Furthermore, the example in your documentation is not functional (or at least I can't get it to work: python -m scripts.convert --quantize --model_id bert-base-uncased:

ModuleNotFoundError: No module named 'onnxscript'

Thanks for your work for the open-source community

Prerequisites

• The model is supported in Transformers (i.e., listed here)
• The model can be exported to ONNX with Optimum (i.e., listed here)

Additional information

No response

Your contribution

N/A

[xenova]

Hi there 👋 Could you try use the latest version of optimum (now under the optimum-onnx repo) directly? https://github.com/huggingface/optimum-onnx

The onnxscript error is because of the new torch exporter which uses the dynamo exporter, so you may need to install that if you want to export in that way.

[Disk-MTH]

Thanks for you quick response 😊

I just try that :
python -m venv .venv (activate)
python -m pip install --upgrade pip
pip install "optimum-onnx[onnxruntime]" hf_xet
optimum-cli export onnx --model depth-anything/Depth-Anything-V2-Metric-Indoor-Small-hf onnx_export/

And faced this output:
Warn:

Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.

Error:

File "xxx\.venv\Lib\site-packages\optimum\exporters\onnx\convert.py", line 944, in onnx_export_from_model
    raise ValueError(
        f"Trying to export a {model_type} model, that is a custom or unsupported architecture, but no custom onnx configuration was passed as `custom_onnx_configs`. Please refer to https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/export_a_model#custom-export-of-transformers-models for an example on how to export custom models. Please open an issue at https://github.com/huggingface/optimum/issues if you would like the model type {model_type} to be supported natively in the ONNX export."
    )
ValueError: Trying to export a depth_anything model, that is a custom or unsupported architecture, but no custom onnx configuration was passed as `custom_onnx_configs`. Please refer to https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/export_a_model#custom-export-of-transformers-models for an example on how to export custom models. Please open an issue at https://github.com/huggingface/optimum/issues if you would like the model type depth_anything to be supported natively in the ONNX export.

I don't know what I'm doing wrong in this case 🧐

[Disk-MTH]

Little addition on for yesterday post. I've tried to export other models like google-bert/bert-base-uncased and it works fine, but every time I try a depth-estimation model (depth-anything, depth-pro, ...), the export fails

[xenova]

ValueError: Trying to export a depth_anything model, that is a custom or unsupported architecture, but no custom onnx configuration was passed as custom_onnx_configs. Please refer to https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/export_a_model#custom-export-of-transformers-models for an example on how to export custom models. Please open an issue at https://github.com/huggingface/optimum/issues if you would like the model type depth_anything to be supported natively in the ONNX export.

Ah I see, it looks like depth_anything hasn't been added to Optimum, even though it should literally just be a few lines of code needed. Something like https://github.com/huggingface/optimum-onnx/blob/682988bc2fd972f4594fb66725fa7e7c6c2b628d/optimum/exporters/onnx/model_configs.py#L1035-L1037. Would you be interested in opening a PR?

[Disk-MTH]

I just tried the suggested code (I'll submit a pull request in the day when everything will work), but I'm still having a problem: it doesn't work with WebGPU.

@register_tasks_manager_onnx("depth_anything", *["feature-extraction", "depth-estimation"])
class DepthAnythingOnnxConfig(DptOnnxConfig):
    pass

My export command: optimum-cli export onnx --model depth-anything/Depth-Anything-V2-Metric-Indoor-Small-hf model/
(I also tried with --dtype fp32 and --dtype fp16 and that actually change something in the config.json)

With the code:

pipe = await pipeline('depth-estimation', './model', {
    device: 'webgpu',
    dtype: 'fp32', //or fp16
});

I still get these console errors:

An error occurred during model execution: "Error: failed to call OrtRun(). ERROR_CODE: 2, ERROR_MESSAGE: Unexpected input data type. Actual: (tensor(float)) , expected: (tensor(float16))".

Inputs given to model: 
{pixel_values: {…}}
pixel_values:  {type: 'float32', dims: Array(4), location: 'cpu', data: Float32Array(456876)}

whereas if I use this code:

pipe = await pipeline('depth-estimation', './model');

I just get the fallback warning in dtype q8 (atrociously slow and poor quality) but it works.

It seems I have a problem with the expected Tensor format (some quantization to do or something like that I suppose). The remaining code on my page hasn't changed when I used onnx-community/depth-anything-v2-small so I don't know what detail I'm missing to make it work

EDIT: That was a cache issue (even if I disabled cache from my browser)...

[Disk-MTH]

I opened a PR for depth_anything model type support in optimum-onnx

Now I have another issue : quality

This is a picture from onnx-community/depth-anything-v2-small, the export made from depth-anything/Depth-Anything-V2-Small-hf

And that an picture when I export it:

Behind that is exactly the same code, no changes at all, I don't understand 😭
My export command optimum-cli export onnx --model depth-anything/Depth-Anything-V2-Small-hf --dtype fp32 model/

I tried also to use your custom script (with some code correction - PR - and my new optimum-onnx version), but I have two problems:

• preprocessor_config.json is not generated (whereas it is with the command optimum-onnx)
• It does not fix the quality issue (it looks exactly the same but I have all the versions generated with the folder structure)

Thanks for your support

[xenova]

During export do you notice any warnings? For example, there may be a portion of the traced model which is incorrectly marked as static (e.g., positional embeddings), and then during inference, it produces incorrect results when running on an image of a different size.

[Disk-MTH]

Hi,

I have some warnings yes.

With optimum-cli export onnx --model depth-anything/Depth-Anything-V2-Small-hf --dtype fp32 model/

`torch_dtype` is deprecated! Use `dtype` instead!

Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.

/.../.venv/lib/python3.12/site-packages/transformers/models/dinov2/modeling_dinov2.py:143: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if num_channels != self.num_channels:

/.../.venv/lib/python3.12/site-packages/transformers/models/depth_anything/modeling_depth_anything.py:159: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if hidden_state.shape != residual.shape:

/.../.venv/lib/python3.12/site-packages/transformers/models/depth_anything/modeling_depth_anything.py:312: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  (int(patch_height * self.patch_size), int(patch_width * self.patch_size)),

And with the script python -m scripts.convert --quantize --model_id depth-anything/Depth-Anything-V2-Small-hf

Same as before and a lot of this type:

/.../transformers.js/scripts/float16.py:73: UserWarning: the float32 number 6.384451634033894e-09 will be truncated to 1e-07