Would improved loss functions like focal loss further boost the performance?

[LUPIN11]

Thank you for your great work!
I notice that Segmenter just uses unweighted ce loss for training. However, ADE20K is not a dataset with balanced classes. I wonder whether we can boost current performance by improved loss functions like focal loss to compete with MaskFormer-like architectures.
I tried to do this but get extermely bad performance.
Have the authors or others attempted the same? Discussion is welcome!
My code for focal loss/dice loss is copied from:
https://github.com/qubvel-org/segmentation_models.pytorch/tree/main/segmentation_models_pytorch/losses