Run time error in prediction stage with pytorch-fpn

[NoaMillsUSDA-ARS]

I am building a semantic segmentation model that uses four bands of satellite data. I am running rastervision in a singularity container on a compute cluster. Per my previous discussion post, I have cloned the pytorch-fpn repo to my working directory on the compute cluster, and use the ExternalModuleConfig function to link the fpn model as in the isprs_potsdam.py example. When I run the model, I am able to get through the train and bundle stages, but I get the following error at the prediction stage:

2022-09-12 12:13:00:rastervision.core.rv_pipeline.rv_pipeline: INFO - Making predictions for scene
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/src/rastervision_pipeline/rastervision/pipeline/cli.py", line 248, in <module>
    main()
  File "/opt/conda/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/opt/src/rastervision_pipeline/rastervision/pipeline/cli.py", line 240, in run_command
    runner=runner)
  File "/opt/src/rastervision_pipeline/rastervision/pipeline/cli.py", line 217, in _run_command
    command_fn()
  File "/opt/src/rastervision_core/rastervision/core/rv_pipeline/rv_pipeline.py", line 184, in predict
    for s in dataset.validation_scenes
  File "/opt/src/rastervision_core/rastervision/core/rv_pipeline/rv_pipeline.py", line 178, in _predict
    labels = self.predict_scene(scene, self.backend)
  File "/opt/src/rastervision_core/rastervision/core/rv_pipeline/rv_pipeline.py", line 219, in predict_scene
    predict_batch(batch_chips, batch_windows)
  File "/opt/src/rastervision_core/rastervision/core/rv_pipeline/rv_pipeline.py", line 204, in predict_batch
    batch_labels = backend.predict(scene, chips, windows)
  File "/opt/src/rastervision_pytorch_backend/rastervision/pytorch_backend/pytorch_semantic_segmentation.py", line 93, in predict
    batch_out = self.learner.numpy_predict(chips, raw_out=raw_out)
  File "/opt/src/rastervision_pytorch_learner/rastervision/pytorch_learner/semantic_segmentation_learner.py", line 162, in numpy_predict
    out = self.predict(x, raw_out=True)
  File "/opt/src/rastervision_pytorch_learner/rastervision/pytorch_learner/semantic_segmentation_learner.py", line 145, in predict
    out = self.model(x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/data/testrun_4b/train/modules/fpn/fpn/layers.py", line 54, in forward
    return sum(inps)
RuntimeError: The size of tensor a (64) must match the size of tensor b (50) at non-singleton dimension 3
/opt/data/testrun_4b/Makefile:9: recipe for target '1' failed
make: *** [1] Error 1```

[AdeelH]

What version/branch of RV are you using?

Can you share your get_config() function?

[NoaMillsUSDA-ARS]

I'm using version 0.13. Here's my get_config function. I set multiband=True at the command line.

def get_config(runner, root_uri, learn_rate, epoch, cyc, multiband: bool = False) -> SemanticSegmentationConfig:
    #output_uri = '/opt/data/output/'
    input_uri = '/opt/data/input_data'

    image_1_uri = f'{input_uri}/linn_raster.tif'
    label_1_uri = f'{input_uri}/linn_vector.geojson'
    image_2_uri = f'{input_uri}/polk_raster.tif'
    label_2_uri = f'{input_uri}/polk_vector.geojson'
    image_3_uri = f'{input_uri}/scott_raster.tif'
    label_3_uri = f'{input_uri}/scott_vector.geojson'

    if multiband:
        # use all 4 channels
        channel_order = [0, 1, 2, 3]
        channel_display_groups = {'RGB': (0, 1, 2), 'IR': (3, )}
        #aug_transform = example_multiband_transform
    else:
        # use infrared, red, & green channels only
        channel_order = [3, 0, 1]
        channel_display_groups = None
        #aug_transform = example_rgb_transform #would only need if performing data aug
    
    class_config = ClassConfig(
        names=['agricultural', 'non-agricultural'], colors=['red', 'black'])

    def make_scene(scene_id: str, image_uri: str,
                   label_uri: str) -> SceneConfig:
        """
        - The GeoJSON does not have a class_id property for each geom,
          so it is inferred as 0 (ie. building) because the default_class_id
          is set to 0.
        - The labels are in the form of GeoJSON which needs to be rasterized
          to use as label for semantic segmentation, so we use a RasterizedSource.
        - The rasterizer set the background (as opposed to foreground) pixels
          to 1 because background_class_id is set to 1.
        """
        raster_source = RasterioSourceConfig(
            uris=[image_uri], channel_order=channel_order)
        vector_source = GeoJSONVectorSourceConfig(
            uri=label_uri, default_class_id=0, ignore_crs_field=True)
        label_source = SemanticSegmentationLabelSourceConfig(
            raster_source=RasterizedSourceConfig(
                vector_source=vector_source,
                rasterizer_config=RasterizerConfig(background_class_id=1)))
        return SceneConfig(
            id=scene_id,
            raster_source=raster_source,
            label_source=label_source)

    scene_dataset = DatasetConfig(
        class_config=class_config,
        train_scenes=[
            make_scene('linn', image_1_uri, label_1_uri),
            make_scene('polk', image_2_uri, label_2_uri)
        ],
        validation_scenes=[
            make_scene('scott', image_3_uri, label_3_uri)
        ])

    # Use the PyTorch backend for the SemanticSegmentation pipeline.
    chip_sz = 200
    class_config.ensure_null_class()
    backend = PyTorchSemanticSegmentationConfig(
        data=SemanticSegmentationGeoDataConfig(
            scene_dataset=scene_dataset,
            channel_display_groups=channel_display_groups,
            window_opts=GeoDataWindowConfig(
                method=GeoDataWindowMethod.random,
                size=chip_sz,
                size_lims=(chip_sz, chip_sz + 1),
                max_windows=10)),
        model=SemanticSegmentationModelConfig(
                external_def=ExternalModuleConfig(
                    uri="/opt/data/src/pytorch-fpn",
                    name="fpn",
                    entrypoint="make_fpn_resnet",
                    entrypoint_kwargs={
                        'name':'resnet50',
                        'fpn_type':'panoptic',
                        'num_classes':len(class_config),
                        'in_channels':len(channel_order),
                        'out_size':(chip_sz, chip_sz)
                    })),
        solver=SolverConfig(lr=learn_rate, num_epochs=epoch, batch_sz=8, one_cycle = cyc), log_tensorboard = True)

    return SemanticSegmentationConfig(
        root_uri=root_uri, #input at command line, ie /opt/data/output1
        dataset=scene_dataset,
        backend=backend,
        train_chip_sz=chip_sz,
        predict_chip_sz=chip_sz)

[AdeelH]

I believe this is caused by a discrepancy between your chip_sz and img_sz. I think any of the following will fix it:

1. Pass img_sz=chip_sz to SemanticSegmentationGeoDataConfig.
2. Change 'out_size': (chip_sz, chip_sz) to 'out_size': (256, 256) (256 is the default value of img_sz).
3. Set chip_sz = 256.

You will need to retrain your model.