[Backport 2.x] Fix inference logic and standardize config index mapping

[Backport 2.x] Fix inference logic and standardize config index mapping #1453