[ONNXRuntimeError] How to compare the precision layer by layer with engine(for example, tensorRT) if I have a custom operator in my onnx (and a corresponding plugin in TensorRT)?

[MyraYu2022]

Describe the issue

Description

I used to check my engine(tensorRT) layer by layer with onnx using polygraphy tool. But I have a custom operator writen by myself in my onnx, and a corresponding plugin in my tensorRT now and how can I compare them layer by layer using polygraphy?

I write the .sh script like this:

 polygraphy run my_model_with_custom_operator.onnx  \
                --onnxrt --trt  \
                --trt-outputs mark all  \
                --onnx-outputs mark all  \
                --atol 1e-3  --rtol 1e-3  \
                --fail-fast  \
                --val-range [0,1]  \
                --verbose 

And I get error as below:
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Fatal error: customPlugin(-1) is not a registered function/op

Thank you very much.

Environment

TensorRT Version:
8.4.0.6

NVIDIA Driver Version:
535.183.01

CUDA Version:
11.3

CUDNN Version:
8.3.2.44

Operating System:

Python Version (if applicable):
3.9.7

PyTorch Version (if applicable):
1.13.0

Urgency

No response

Target platform

onnxruntime

Build script

 polygraphy run my_model_with_custom_operator.onnx  \
                --onnxrt --trt  \
                --trt-outputs mark all  \
                --onnx-outputs mark all  \
                --atol 1e-3  --rtol 1e-3  \
                --fail-fast  \
                --val-range [0,1]  \
                --verbose 

Error / output

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Fatal error: customPlugin(-1) is not a registered function/op

Visual Studio Version

No response

GCC / Compiler Version

No response

[skottmckay]

I don't know what polygraphy is, but you need to register the custom operator library in the session options before creating the inference session.

https://onnxruntime.ai/docs/api/python/api_summary.html#onnxruntime.SessionOptions.register_custom_ops_library