Streamlined model checkpoint run slower than my Brevitas model

[Phu1234567]

I recently went over the CNV end-to-end tutorial notebook and tried to follow its process for my custom model. I managed to streamlined and partitioned the HW layers, but when i run validation on the FPGA, the output does not change no matter what input I use (basically 0% accuracy compared to 94% accuracy of the brevtias model before converting to onnx). I then debug it by following the end-to-end verification. The accuracy of the model right after streamlining was very close to expected (still 94%), but it was significantly slower than when validating the brevitas model.

This is how I validate the streamlined model:

import numpy as np
from qonnx.core.modelwrapper import ModelWrapper
from IPython.display import clear_output
import finn.core.onnx_exec as oxe
import time

num_resnet_blocks = 5
kernel_size = 5, 1
brevitas_m = QuantizedResNet(num_resnet_blocks=num_resnet_blocks, 
                        kernel_size=kernel_size,
                        num_classes=len(mods))
brevitas_m.load_state_dict(torch.load('quantized_model/brevitas_model_checkpoint.pt'),strict=False)

correct_sim=0
correct_bre=0
total=0
changes=0
raw_i = X_test
raw_o=Y_test

model_for_sim = ModelWrapper(f"{build_dir}_streamlined.onnx")
bt=0  #timer for brevitas
srt=0 #timer for streamline
for i in range(300):
    print(i)
    inp=np.array(raw_i[i,:,:,:])[np.newaxis,...]
    inp = inp.astype(np.float32)
    input_dict = {"global_in": inp}
    input_brevitas = torch.from_numpy(inp).float()

    # brevitas model
    t=time.time()
    brevitas_m.eval()
    # print(raw_i.shape)
    output_golden = brevitas_m.forward(input_brevitas).detach().numpy()
    bt+=(time.time()-t)

   
    # streamlined model
    t=time.time()
    output_dict = oxe.execute_onnx(model_for_sim, input_dict, return_full_exec_context=False)
    srt+=time.time()-t

    output_pysim = output_dict[list(output_dict.keys())[0]]
    total+=1
    if np.argmax(raw_o[i])==output_pysim[0][0]:
        correct_sim+=1
    if np.argmax(raw_o[i])==np.argmax(output_golden):
        correct_bre+=1
    if output_pysim[0][0]!=np.argmax(output_golden):
        changes+=1
clear_output(wait=True)

print(f"Streamlined model accuracy: {correct_sim/total},   Total time: {srt}s")
print(f"Brevitas model accuracy: {correct_bre/total},   Total time: {bt}s")
print(f"%Differences between brevitas and streamlined model: {changes/total}")

The output:

Streamlined model accuracy: 0.9366666666666666,   Total time: 1564.9453701972961s
Brevitas model accuracy: 0.94,   Total time: 5.599371671676636s
%Differences between brevitas and streamlined model: 0.0033333333333333335

My question is:

1. Is this normal for the forward passing to be this long for the streamlined model even if the accuracy still relatively the same?
2. If the accuracy is still almost identical but the accuracy on the FPGA is 0%, does that mean that the problem is somewhere after streamlining?