Hello ORT(OnnxRumTime)

An ONNX(Open Neural Network Exchange) Runtime Demo

About ORT

• ORT DOC: https://onnxruntime.ai/docs
• ORT Github: https://github.com/microsoft/onnxruntime
• ORT Maven Repo: https://mvnrepository.com/artifact/com.microsoft.onnxruntime
• ORT Architecture:
• ORT Workflow:
  • TensorRT
  • PyTorch
  • TensorFlow
  • onnx runtime
  • OpenVino

About dependencies

1 CUDA

Microarchitecture

1. Tesla 特斯拉
2. Fermi 费米
3. Kepler 开普勒
4. Maxwell 麦克斯韦
5. Pascal 帕斯卡
6. Volta 伏打
7. Turing 图灵
8. Ampere 安培
9. Ada Lovelace 阿达·洛芙莱斯
10. Hopper 格蕾丝·赫柏
11. Blackwell 戴维·布莱克维尔

Requirements

• https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements
• https://docs.nvidia.com/deeplearning/cudnn/latest/installation/overview.html

	ONNX Runtime	CUDA(Compute Unified Device Architecture)	cuDNN(CUDA Deep Neural Network library)
url	https://github.com/microsoft/onnxruntime/tags	https://developer.nvidia.com/cuda-toolkit-archive	https://developer.nvidia.com/rdp/cudnn-archive
version	1.18	CUDA Toolkit 11.8.0 (October 2022)	cuDNN v8.9.2 (June 1st, 2023), for CUDA 11.x
version	1.18	CUDA Toolkit 12.5.0 (May 2024)	cuDNN v8.9.7 (December 5th, 2023), for CUDA 12.x

CUDA_PATH=D:\garden\cuda12.5
CUDNN_PATH=D:\garden\cudnn8.9.7.29_cuda12
PATH=%CUDA_PATH%\bin;%CUDA_PATH%\libnvvp;%CUDNN_PATH%\bin;...

cp "C:\Program Files\NVIDIA Corporation\Nsight Systems 2024.2.3\host-windows-x64\zlib.dll" D:\garden\cuda12.5\bin

env | grep CUDA

CUDA Driver API 版本

nvidia-smi

CUDA Runtime API 版本

nvcc -V

每隔5秒刷新一次GPU使用情况

watch -n 5 nvidia-smi

2 CoreML

https://developer.apple.com/documentation/coreml

Which devices have an ANE?

https://github.com/hollance/neural-engine/blob/master/docs/supported-devices.md

MORE…

Issues solved

UnsatisfiedLinkError on MacOS

UnsatisfiedLink no onnxruntime in java.library.path

reload the dependency jar file and check it:

no cudart in java.library.path

-Djava.library.path=D:\\coding\\cuda\\11.8\\lib\\x64

A Deep Dive into ONNX & ONNX Runtime

• https://medium.com/becoming-human/a-deep-dive-into-onnx-onnx-runtime-part-1-874517c66ffc
• https://medium.com/becoming-human/a-deep-dive-into-onnx-onnx-runtime-part-2-785b523e0cca