Add support for Depth Anything

[xenova]

Possible now that huggingface/transformers#28654 is merged.

Example: Depth estimation with Xenova/depth-anything-small-hf.

import { pipeline } from '@xenova/transformers';

// Create depth-estimation pipeline
const depth_estimator = await pipeline('depth-estimation', 'Xenova/depth-anything-small-hf');

// Predict depth map for the given image
const url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/bread_small.png';
const output = await depth_estimator(url);
// {
//   predicted_depth: Tensor {
//     dims: [350, 518],
//     type: 'float32',
//     data: Float32Array(181300) [...],
//     size: 181300
//   },
//   depth: RawImage {
//     data: Uint8Array(271360) [...],
//     width: 640,
//     height: 424,
//     channels: 1
//   }
// }

You can visualize the output with:

output.depth.save('depth.png');

cc @NielsRogge

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[kungfooman]

Very nice, I wanted to test it directly in the browser and came up with this:

const depth_estimator = await pipeline('depth-estimation', 'Xenova/depth-anything-small-hf');
const url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/bread_small.png';
const output = await depth_estimator(url);
function rawImageToCanvas({width, height, data}) {
  const canvas = document.createElement('canvas');
  canvas.width = width;
  canvas.height = height;
  const imageData = new ImageData(width, height);
  const {length} = data;
  let j = 0;
  for (let i = 0; i < length; i++) {
    const color = data[i];
    imageData.data[j++] = color;
    imageData.data[j++] = color;
    imageData.data[j++] = color;
    imageData.data[j++] = 255; // make sure it's opaque
  }
  canvas.getContext('2d').putImageData(imageData, 0, 0);
  return canvas;
}
const canvas = rawImageToCanvas(output.depth);
document.body.prepend(canvas); // add to top for quick testing without scrolling

Output:

(first I tried RawImage#toCanvas but it didn't show anything when I appended it to document.body besides its text description)

[xenova]

(first I tried RawImage#toCanvas but it didn't show anything when I appended it to document.body besides its text description)

I believe that's because it's an OffscreenCanvas element, and you will need to create a normal Canvas from it.

[kungfooman]

I believe that's because it's an OffscreenCanvas element, and you will need to create a normal Canvas from it.

Good point, just tested it and it works:

const {depth} = output;
const canvas = document.createElement('canvas');
canvas.width = depth.width;
canvas.height = depth.height;
const offscreenCanvas = output.depth.toCanvas();
const imageData = offscreenCanvas.getContext("2d").getImageData(0, 0, depth.width, depth.width);
canvas.getContext('2d').putImageData(imageData, 0, 0);
document.body.prepend(canvas);