create tensor of float16 as input using the Java API

[andyguo007]

Hi team,

In order to run a fp16 model, do we have a way to create a tensor of float16 as input using the Java API? e.g.,
OnnxTensor.createTensor(env, FloatBuffer.wrap(pixels), shape, ***);

Thanks,

[Craigacp]

There is no way to do that in the Java API at the moment. It supports the outbound transformation (i.e. a model can produce a fp16 output and it will be converted into a FloatBuffer or float array), but the inbound transformation is not allowed.

What's your fp16 input type in Java? Are you storing them in short or do you want to downcast a float into fp16?

[andyguo007]

There is no way to do that in the Java API at the moment. It supports the outbound transformation (i.e. a model can produce a fp16 output and it will be converted into a FloatBuffer or float array), but the inbound transformation is not allowed.

What's your fp16 input type in Java? Are you storing them in short or do you want to downcast a float into fp16?

The input type in Java is float. I converted the model from fp32 to fp16, wondering if there is a way to use the "accelerated(fp16) model".

[Craigacp]

So you'd like to be able to pass in a float[] or FloatBuffer and have ORT downcast the floats into fp16 before storing them in the OnnxTensor?

[andyguo007]

So you'd like to be able to pass in a float[] or FloatBuffer and have ORT downcast the floats into fp16 before storing them in the OnnxTensor?

Yes.

Since the fp16 model expects fp16 input, currently I am getting this error:
ai.onnxruntime.OrtException: Error code - ORT_INVALID_ARGUMENT - message: Unexpected input data type. Actual: (N11onnxruntime17PrimitiveDataTypeIfEE) , expected: (N11onnxruntime17PrimitiveDataTypeINS_9MLFloat16EEE)

[Craigacp]

Ok. If you don't need to persist fp16 values in Java, and are fine with storing floats on the Java side then it's easier to implement, but still will require a bunch of changes to the Java side of ORT. I'll put it on the list of things to do.

[MankaranSingh]

Hi, Any updates on this ?

[Craigacp]

The support is still not available in the ORT Java API, though Java 20 will likely have conversion methods fp32 <-> fp16 which will make it easier to implement (https://download.java.net/java/early_access/jdk20/docs/api/java.base/java/lang/Float.html#floatToFloat16(float)).

For the time being you can take an ONNX model in fp16 and add a fp32 -> fp16 cast node to the start of it using the ONNX python tooling (or edit the protobuf in java).

[MankaranSingh]

Hey @Craigacp, thanks for quick reply. I have a float16 model and would like to add a node to convert the float32 input to float16, Is there any sample code available ? My model has multiple named input nodes and would like to preserve the naming.
Thanks!

[MankaranSingh]

On a side note, is it possible to implement a method that just takes byte-buffer as inputs and let user specify what type it is? The conversion will take place in the c/c++ side in JNI layer.

[Craigacp]

I don't have sample code, but you can load the protobuf and add cast nodes. Preserving the names will be trickier as you'll need to rename the existing input layer and that tends to ripple.

We could implement something that accepted a byte buffer and a type, but you'd still have to prepare the fp16 values in Java to put into the byte buffer, and it would be significantly easier to mess things up by accidentally specifying the wrong type, or getting the endian-ness wrong.

[MankaranSingh]

Ended up running this script in reverse to convert whole model to float32. Needed that anyways because most onnx to 'other format' model converters dont support Cast operator.

[Xiao-Nine]

Sry, I noticed that this issue has been closed，but how to create tensor of float16 as input? I haven't found any interfaces for me to use in onnxruntime_gpu 1.17.3. An exception (java.lang.ClassCastException: class java.nio.HeapByteBuffer cannot be cast to class java.nio.ShortBuffer) has occurred when I try to use OnnxTensor.createTensor(this.env, inputBuffer, INPUT_SHAPE, OnnxJavaType.FLOAT16)

[Craigacp]

There's an example in the tests - https://github.com/microsoft/onnxruntime/blob/main/java/src/test/java/ai/onnxruntime/OnnxTensorTest.java#L298 which shows creating a direct byte buffer, taking a short buffer view of it and then writing in fp16 values to it. Alternatively you can use this method to prepare a suitable ShortBuffer from a FloatBuffer - https://github.com/microsoft/onnxruntime/blob/main/java/src/main/jvm/ai/onnxruntime/platform/Fp16Conversions.java#L67. What's the stack trace of the exception?

[Xiao-Nine]

Thank you for your reply.
My envs:

• JDK17
• onnxruntime_gpu 1.17.3

I try to create a OnnxTensor by ByteBuffer like https://github.com/microsoft/onnxruntime/blob/main/java/src/test/java/ai/onnxruntime/OnnxTensorTest.java#L298.
but an exception occurred:

java.lang.ClassCastException: class java.nio.HeapByteBuffer cannot be cast to class java.nio.ShortBuffer (java.nio.HeapByteBuffer and java.nio.ShortBuffer are in module java.base of loader 'bootstrap')

	at ai.onnxruntime.OrtUtil.prepareBuffer(OrtUtil.java:524)
	at ai.onnxruntime.OnnxTensor.createTensor(OnnxTensor.java:754)
	at ai.onnxruntime.OnnxTensor.createTensor(OnnxTensor.java:610)
	at ai.onnxruntime.OnnxTensor.createTensor(OnnxTensor.java:589)
	at com.example.demo.DemoApplicationTests.modelTest(DemoApplicationTests.java:58)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[Craigacp]

You need to pass in a direct ByteBuffer allocated with ByteBuffer.allocateDirect then call asShortBuffer() on it. Otherwise it tries to cast the buffer you pass in back as a ShortBuffer to copy it into a direct ByteBuffer and fails. I'll work at improving the checks so it accepts byte buffers of any kind and does the copy, but for the moment you need to prepare it appropriately.