no accuracy metrics while training

[cpoptic]

Under model.py we see the keras model compile

        self.keras_model.compile(
            optimizer=optimizer,
            loss=[None] * len(self.keras_model.outputs),
            metrics=['accuracy'])

And then the metrics for the Mask RCNN metrics are added:

        # Add metrics for losses
        for name in loss_names:
            if name in self.keras_model.metrics_names:
                continue
            layer = self.keras_model.get_layer(name)
            self.keras_model.metrics_names.append(name)
            loss = (
                tf.reduce_mean(layer.output, keepdims=True)
                * self.config.LOSS_WEIGHTS.get(name, 1.))
            self.keras_model.metrics_tensors.append(loss)
        print("metrics: ", self.keras_model.metrics_names)

But when training the model, neither train_accuracy nor val_accuracy are reported. How to add these metrics and report them with each epoch, alongside the mrcnn metrics?

[VtlNmnk]

Try using custom_callbacks. Counting accuracy takes a lot of time, so don't use it every epoch.

mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model,
model_inference, dataset_val, calculate_map_at_every_X_epoch=5, verbose=1)

model.train(dataset_train, dataset_val,
learning_rate=config.LEARNING_RATE,
epochs=100,
layers='heads',
custom_callbacks=[mean_average_precision_callback])

[VtlNmnk]

also add this somewhere in the model.py, for example at the end of the file

############################################################
#  Custom Callbacks
############################################################

class MeanAveragePrecisionCallback(Callback):
    def __init__(self, train_model: MaskRCNN, inference_model: MaskRCNN, dataset: Dataset,
                 calculate_map_at_every_X_epoch=5, dataset_limit=None,
                 verbose=1):
        super().__init__()
        self.train_model = train_model
        self.inference_model = inference_model
        self.dataset = dataset
        self.calculate_map_at_every_X_epoch = calculate_map_at_every_X_epoch
        self.dataset_limit = len(self.dataset.image_ids)
        if dataset_limit is not None:
            self.dataset_limit = dataset_limit
        self.dataset_image_ids = self.dataset.image_ids.copy()

        if inference_model.config.BATCH_SIZE != 1:
            raise ValueError("This callback only works with the bacth size of 1")

        self._verbose_print = print if verbose > 0 else lambda *a, **k: None

    def on_epoch_end(self, epoch, logs=None):

        if epoch > 2 and (epoch+1)%self.calculate_map_at_every_X_epoch == 0:
            self._verbose_print("Calculating mAP...")
            self._load_weights_for_model()

            mAPs = self._calculate_mean_average_precision()
            mAP = np.mean(mAPs)

            if logs is not None:
                logs["val_mean_average_precision"] = mAP

            self._verbose_print("mAP at epoch {0} is: {1}".format(epoch+1, mAP))

        super().on_epoch_end(epoch, logs)

    def _load_weights_for_model(self):
        last_weights_path = self.train_model.find_last()
        self._verbose_print("Loaded weights for the inference model (last checkpoint of the train model): {0}".format(
            last_weights_path))
        self.inference_model.load_weights(last_weights_path,
                                          by_name=True)

    def _calculate_mean_average_precision(self):
        mAPs = []

        # Use a random subset of the data when a limit is defined
        np.random.shuffle(self.dataset_image_ids)

        for image_id in self.dataset_image_ids[:self.dataset_limit]:
            image, image_meta, gt_class_id, gt_bbox, gt_mask = load_image_gt(self.dataset, self.inference_model.config,
                                                                             image_id, use_mini_mask=False)
            molded_images = np.expand_dims(mold_image(image, self.inference_model.config), 0)
            results = self.inference_model.detect(molded_images, verbose=0)
            r = results[0]
            # Compute mAP - VOC uses IoU 0.5
            AP, _, _, _ = utils.compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"],
                                           r["class_ids"], r["scores"], r['masks'])
            mAPs.append(AP)

        return np.array(mAPs)

[nayzen]

Hello guys, thanks for sharing the code to have accuracy metrics. I try to implement the code but I have errors :(
I pasted the class "MeanAveragePrecisionCallbacks" at the end of the model.py code.

Then, in train.py I defined the variable like you did @VtlNmnk:
mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, model_inference, dataset_val, calculate_map_at_every_X_epoch=5, verbose=1)

But when I launch my training, I have this error:

Traceback (most recent call last): File "Taraudage.py", line 372, in <module> train(model) File "Taraudage.py", line 196, in train mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(train_model=model, inference_model=model_inference, NameError: name 'model_inference' is not defined

What variable do you expect for model inference ?

Thanks for you time,
Regards,
Antoine

[VtlNmnk]

Then, in train.py I defined the variable like you did @VtlNmnk:
mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, model_inference, dataset_val, calculate_map_at_every_X_epoch=5, verbose=1)

I did not write to you to change anything in the train.py )
This line needs to be added to the script from which you access the train.py.
Personally, I use google collab from guys, but with many of my changes.

[VtlNmnk]

Here is an example of how this works for me

[ben975]

Hi,

I am sorry but I am trying to implement your code but the dataset: Dataset in the def init part when defining the callback comes up as "Dataset not defined".

Thank you for any suggestions :)

[mhtarora39]

Hi @ben975,

You can use below class to calculate MAP, precision, recall for each image

from mrcnn.utils import compute_ap
class EvalImage():
  def __init__(self,dataset,model,cfg):
    self.dataset = dataset
    self.model   = model
    self.cfg     = cfg

 
 
  def evaluate_model(self , len = 50):
    APs = list()
    precisions_dict = {}
    recall_dict     = {}
    for index,image_id in enumerate(self.dataset.image_ids):
      if(index > len):
         break; 
      # load image, bounding boxes and masks for the image id
      image, image_meta, gt_class_id, gt_bbox, gt_mask = modellib.load_image_gt(self.dataset, self.cfg,image_id, use_mini_mask=False)
      # convert pixel values (e.g. center)
      #scaled_image = modellib.mold_image(image, self.cfg)
      # convert image into one sample
      sample = np.expand_dims(image, 0)
     # print(len(image))
      # make prediction
      yhat = self.model.detect(sample, verbose=1)
      # extract results for first sample
      r = yhat[0]
      # calculate statistics, including AP
      AP, precisions, recalls, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
      precisions_dict[image_id] = np.mean(precisions)
      recall_dict[image_id] = np.mean(recalls)
      # store
      APs.append(AP)

    # calculate the mean AP across all images
    mAP = np.mean(APs)
    return mAP,precisions_dict,recall_dict

Usage :

prepare data set:

dataset_val = Dataset() 
dataset_val.load_data_set("dataset name","val_data")
dataset_val.prepare()

Create object of config and load model

config = DataConfig()
eval = EvalImage(dataset_val,model,config)

evel.evaluate_model()
you will get the results

[ben975]

Wow thank for the really fast reply!
I am currently training but it will be done soon and will have a go! I have searched ages for this question to be answered. Will report back shortly! :)

[ben975]

Hi @ben975,

You can use below class to calculate MAP, precision, recall for each image

class EvalImage():
  def __init__(self,dataset,model,cfg):
    self.dataset = dataset
    self.model   = model
    self.cfg     = cfg

 
 
  def evaluate_model(self , len = 50):
    APs = list()
    precisions_dict = {}
    recall_dict     = {}
    for index,image_id in enumerate(self.dataset.image_ids):
      if(index > len):
         break; 
      # load image, bounding boxes and masks for the image id
      image, image_meta, gt_class_id, gt_bbox, gt_mask = modellib.load_image_gt(self.dataset, self.cfg,image_id, use_mini_mask=False)
      # convert pixel values (e.g. center)
      #scaled_image = modellib.mold_image(image, self.cfg)
      # convert image into one sample
      sample = np.expand_dims(image, 0)
     # print(len(image))
      # make prediction
      yhat = self.model.detect(sample, verbose=1)
      # extract results for first sample
      r = yhat[0]
      # calculate statistics, including AP
      AP, precisions, recalls, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
      precisions_dict[image_id] = np.mean(precisions)
      recall_dict[image_id] = np.mean(recalls)
      # store
      APs.append(AP)

    # calculate the mean AP across all images
    mAP = np.mean(APs)
    return mAP,precisions_dict,recall_dict

Usage :

prepare data set:

dataset_val = Dataset() 
dataset_val.load_data_set("onion","val_data")
dataset_val.prepare()

Create object of config and load model

config = DataConfig()
eval = EvalImage(dataset_val,model,config)

evel.evaluate_model()
you will get the results

Hi @mhtarora39 im sorry but when I run it I seem to get:

File "", line 1, in
eval.evaluate_model()

File "", line 28, in evaluate_model
AP, precisions, recalls, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])

NameError: name 'compute_ap' is not defined

Thanks again for your help I am new to mask rcnns

[mhtarora39]

Sorry I didn't mentioned please import compute_ap as "from mrcnn.utils import compute_ap". I am also updating above code with import let me know if you encounter any other issue i will update code accordingly

[ben975]

seems to be running good thank you so much!

[hardikmanek]

Here is an example of how this works for me

Hi @VtlNmnk I have made the changes that were mentioned by you and was able to run the code without any errors, but when I start training mAP is not getting printed.
Training command: python custom.py train --dataset =customImages --weights=coco

Please find the output:

[VtlNmnk]

Hi, @hardikmanek! Can you show the code how you initialize the callback?
Without additional information, I can’t help you.

[hardikmanek]

Hi @VtlNmnk Please find the attached screenshot.

I am passing the training command as a command-line argument

This is the initial code of the function copied in the last part of model.py

By the way, the complete execution command for the program is:
python custom.py train --dataset=customImages --weights=coco

I got the mAP value after 9th Epoch which is 0.3, not sure what's wrong

Thank you so much.

[dreichCSL]

Can anyone in this thread explain how to get the loss output logging that we see in hardikmanik's screenshot? I'm talking about the various losses logged to stdout - I'm only seeing the total loss and none of the rest (like rpn_class_loss, rpn_box_loss, etc..).

[zrc9627]

hi,@VtlNmnk .i used your code but have the error
Traceback (most recent call last): File "D:/study/Mask_RCNN-2.0/train_test.py", line 16, in <module> import model as modellib File "D:\study\Mask_RCNN-2.0\model.py", line 2769, in <module> import train_test File "D:\study\Mask_RCNN-2.0\train_test.py", line 225, in <module> mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, AttributeError: module 'model' has no attribute 'MeanAveragePrecisionCallback'

model.py:
`from keras.callbacks import Callback
import train_test
Dataset=train_test.DrugDataset()
class MeanAveragePrecisionCallback(Callback):
def init(self, train_model: MaskRCNN, inference_model=MaskRCNN, dataset=Dataset,
calculate_map_at_every_X_epoch=1, dataset_limit=None,
verbose=1):
super().init()
self.train_model = train_model
self.inference_model = inference_model
self.dataset = dataset
self.calculate_map_at_every_X_epoch = calculate_map_at_every_X_epoch
self.dataset_limit = len(self.dataset.image_ids)
if dataset_limit is not None:
self.dataset_limit = dataset_limit
self.dataset_image_ids = self.dataset.image_ids.copy()

    if inference_model.config.BATCH_SIZE != 1:
        raise ValueError("This callback only works with the bacth size of 1")

    self._verbose_print = print if verbose > 0 else lambda *a, **k: None

def on_epoch_end(self, epoch, logs=None):

    if epoch > 2 and (epoch+1)%self.calculate_map_at_every_X_epoch == 0:
        self._verbose_print("Calculating mAP...")
        self._load_weights_for_model()

        mAPs = self._calculate_mean_average_precision()
        mAP = np.mean(mAPs)

        if logs is not None:
            logs["val_mean_average_precision"] = mAP

        self._verbose_print("mAP at epoch {0} is: {1}".format(epoch+1, mAP))

    super().on_epoch_end(epoch, logs)

def _load_weights_for_model(self):
    last_weights_path = self.train_model.find_last()
    self._verbose_print("Loaded weights for the inference model (last checkpoint of the train model): {0}".format(
        last_weights_path))
    self.inference_model.load_weights(last_weights_path,
                                      by_name=True)

def _calculate_mean_average_precision(self):
    mAPs = []

    # Use a random subset of the data when a limit is defined
    np.random.shuffle(self.dataset_image_ids)

    for image_id in self.dataset_image_ids[:self.dataset_limit]:
        image, image_meta, gt_class_id, gt_bbox, gt_mask = load_image_gt(self.dataset, self.inference_model.config,
                                                                         image_id, use_mini_mask=False)
        molded_images = np.expand_dims(mold_image(image, self.inference_model.config), 0)
        results = self.inference_model.detect(molded_images, verbose=0)
        r = results[0]
        # Compute mAP - VOC uses IoU 0.5
        AP, _, _, _ = utils.compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"],
                                       r["class_ids"], r["scores"], r['masks'])
        mAPs.append(AP)

    return np.array(mAPs)`

my train.py:
model = modellib.MaskRCNN(mode="training", config=config, model_dir=MODEL_DIR) model_inference = modellib.MaskRCNN(mode="inference", config=config, model_dir=MODEL_DIR) mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, model_inference, dataset=dataset_val, calculate_map_at_every_X_epoch=1, verbose=1)
model.train(dataset_train, dataset_val, learning_rate=config.LEARNING_RATE, epochs=1, layers='heads', custom_callbacks=[mean_average_precision_callback])

thanks a lot!!

[VtlNmnk]

how to get the loss output logging that we see in hardikmanik's screenshot?

set verbose = 1

[VtlNmnk]

from keras.callbacks import Callback
import train_test
Dataset=train_test.DrugDataset()
class MeanAveragePrecisionCallback(Callback):
def init(self, train_model: MaskRCNN, inference_model=MaskRCNN, dataset=Dataset,
calculate_map_at_every_X_epoch=1, dataset_limit=None,
verbose=1):

you showed your code in the model.py that the markup is not visible )
I think either invalid path to the model.py file is specified, or the wrong indentation in the model.py file itself.

[VtlNmnk]

I got the mAP value after 9th Epoch which is 0.3, not sure what's wrong

Sorry for the long reply. Well, the mAP was calculated, but not after the third epoch. Perhaps there were no saved models to count it earlier? Check how often your models are saved.

[WillianaLeite]

I customized the "https://github.com/matterport/Mask_RCNN.git" repository to train with my own data set, for object detection, ignoring the mask segmentation part. Now I am evaluating my results, I can calculate the MAP, but I cannot calculate the F1 score. I have this function: compute_ap, from "https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/utils.py" which returns the "mAP, details, memories, overlays" for each image. The point is that I cannot apply the F1 score formula, because the variables "precision" and "recalls" are lists.

def compute_ap(gt_boxes, gt_class_ids, gt_masks,
               pred_boxes, pred_class_ids, pred_scores, pred_masks,
               iou_threshold=0.5):

    # Get matches and overlaps
    gt_match, pred_match, overlaps = compute_matches(
        gt_boxes, gt_class_ids, gt_masks,
        pred_boxes, pred_class_ids, pred_scores, pred_masks,
        iou_threshold)

    # Compute precision and recall at each prediction box step
    precisions = np.cumsum(pred_match > -1) / (np.arange(len(pred_match)) + 1)
    recalls = np.cumsum(pred_match > -1).astype(np.float32) / len(gt_match)

    # Pad with start and end values to simplify the math
    precisions = np.concatenate([[0], precisions, [0]])
    recalls = np.concatenate([[0], recalls, [1]])

    # Ensure precision values decrease but don't increase. This way, the
    # precision value at each recall threshold is the maximum it can be
    # for all following recall thresholds, as specified by the VOC paper.
    for i in range(len(precisions) - 2, -1, -1):
        precisions[i] = np.maximum(precisions[i], precisions[i + 1])

    # Compute mean AP over recall range
    indices = np.where(recalls[:-1] != recalls[1:])[0] + 1
    mAP = np.sum((recalls[indices] - recalls[indices - 1]) *
                 precisions[indices])

    return mAP, precisions, recalls, overlaps

[rupa1118]

Can anyone in this thread explain how to get the loss output logging that we see in hardikmanik's screenshot? I'm talking about the various losses logged to stdout - I'm only seeing the total loss and none of the rest (like rpn_class_loss, rpn_box_loss, etc..).

I have tried all the solutions I'm facing the same issue ...... any solution?

[rupa1118]

Hi @VtlNmnk Please find the attached screenshot.

I am passing the training command as a command-line argument

This is the initial code of the function copied in the last part of model.py

By the way, the complete execution command for the program is:
python custom.py train --dataset=customImages --weights=coco

I got the mAP value after 9th Epoch which is 0.3, not sure what's wrong

Thank you so much.

Hello, I have tried this and I'm getting mAP value as 0 after every 3 epochs.... any suggestion why is the value of mAP zero????

[dreichCSL]

@rupa1118 I couldn't get the loss outputs. At this point I'm suspecting it might have something to do with the tensorflow/keras version you're using - maybe some older version prints these values? However, I'm not going to change the version just for printing the loss. You could also try writing a custom callback to print these numbers, I guess (I haven't tried that - instead I put some tf.Print lines in the code to see the losses for debugging (which is ugly, but was fast)).

[VtlNmnk]

@rupa1118 Does the balloon example work for you? I would try to first add the mAP to a known working example, and add the changes one at a time. Perhaps you have something wrong with the masks, as your val_loss also looks strange.

[yoya93]

Hello, I'm trying to reproduce your @VtlNmnk VtlNmnk code.
When I start training I get the error:

Using TensorFlow backend.
Traceback (most recent call last):
File "/content/Mask_RCNN/samples/metal_blanco/metal_blanco.py", line 45, in
from mrcnn import model as modellib, utils
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 656, in _load_unlocked
File "", line 626, in _load_backward_compatible
File "/usr/local/lib/python3.6/dist-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py", line 2876, in
NameError: name 'Callback' is not defined

This is my model.py file

############################################################

Custom Callbacks

############################################################

from keras.callbacks import Callback

class MeanAveragePrecisionCallback(Callback):
def init(self, train_model: MaskRCNN, inference_model: MaskRCNN, dataset: Dataset,
calculate_map_at_every_X_epoch=3, dataset_limit=None,
verbose=1):
super().init()
self.train_model = train_model
self.inference_model = inference_model
self.dataset = dataset

############################################################
This is my file that starts training:
############################################################
def train(model):
"""Train the model."""
# Training dataset.
dataset_train = metal_blancoDataset()
dataset_train.load_metal_blanco(args.dataset, "train")
dataset_train.prepare()

# Validation dataset
dataset_val = metal_blancoDataset()
dataset_val.load_metal_blanco(args.dataset, "val")
dataset_val.prepare()

model_inference = modellib.MaskRCNN(mode="inference", config=config,
                              model_dir=args.logs)

mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model,
model_inference, dataset_val, calculate_map_at_every_X_epoch=3, verbose=1)

# *** This training schedule is an example. Update to your needs ***
# Since we're using a very small dataset, and starting from
# COCO trained weights, we don't need to train too long. Also,
# no need to train all layers, just the heads should do it.
print("Training network heads")
model.train(dataset_train, dataset_val,
            learning_rate=config.LEARNING_RATE,
            epochs=30,
            layers='heads',
            custom_callbacks=[mean_average_precision_callback])

############################################################

Any help will be appreciated

[rupa1118]

Hello, I'm trying to reproduce your @VtlNmnk VtlNmnk code.
When I start training I get the error:

Using TensorFlow backend.
Traceback (most recent call last):
File "/content/Mask_RCNN/samples/metal_blanco/metal_blanco.py", line 45, in
from mrcnn import model as modellib, utils
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 656, in _load_unlocked
File "", line 626, in _load_backward_compatible
File "/usr/local/lib/python3.6/dist-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py", line 2876, in
NameError: name 'Callback' is not defined

This is my model.py file

############################################################

Custom Callbacks

############################################################

from keras.callbacks import Callback

class MeanAveragePrecisionCallback(Callback):
def init(self, train_model: MaskRCNN, inference_model: MaskRCNN, dataset: Dataset,
calculate_map_at_every_X_epoch=3, dataset_limit=None,
verbose=1):
super().init()
self.train_model = train_model
self.inference_model = inference_model
self.dataset = dataset

############################################################
This is my file that starts training:
############################################################
def train(model):
"""Train the model."""

Training dataset.

dataset_train = metal_blancoDataset()
dataset_train.load_metal_blanco(args.dataset, "train")
dataset_train.prepare()

# Validation dataset
dataset_val = metal_blancoDataset()
dataset_val.load_metal_blanco(args.dataset, "val")
dataset_val.prepare()

model_inference = modellib.MaskRCNN(mode="inference", config=config,
                              model_dir=args.logs)

mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model,
model_inference, dataset_val, calculate_map_at_every_X_epoch=3, verbose=1)

# *** This training schedule is an example. Update to your needs ***
# Since we're using a very small dataset, and starting from
# COCO trained weights, we don't need to train too long. Also,
# no need to train all layers, just the heads should do it.
print("Training network heads")
model.train(dataset_train, dataset_val,
            learning_rate=config.LEARNING_RATE,
            epochs=30,
            layers='heads',
            custom_callbacks=[mean_average_precision_callback])

############################################################

Any help will be appreciated

Hi @yoya93,Try checking train function in model.py and if custom_callbacks = None then please remove that None as this function will be accepting callback as parameter.

[yoya93]

Hello @VtlNmnk thank you very much for your response. I am currently facing another problem. Training throws me this error:

Traceback (most recent call last):
File "/content/Mask_RCNN/samples/metal_blanco/metal_blanco.py", line 371, in
train(model)
File "/content/Mask_RCNN/samples/metal_blanco/metal_blanco.py", line 195, in train
inference_model=model_inference, dataset=dataset_val, calculate_map_at_every_X_epoch=5, verbose=1)
File "/usr/local/lib/python3.6/dist-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py", line 2893, in init
ValueError: This callback only works with the bacth size of 1

For this reason configure my config.py like this:

class Config(object):
"""Base configuration class. For custom configurations, create a
sub-class that inherits from this one and override properties
that need to be changed.
"""
# Name the configurations. For example, 'COCO', 'Experiment 3', ...etc.
# Useful if your code needs to do things differently depending on which
# experiment is running.
NAME = None # Override in sub-classes

# NUMBER OF GPUs to use. When using only a CPU, this needs to be set to 1.
GPU_COUNT = 1

# Number of images to train with on each GPU. A 12GB GPU can typically
# handle 2 images of 1024x1024px.
# Adjust based on your GPU memory and image sizes. Use the highest
# number that your GPU can handle for best performance.
IMAGES_PER_GPU = 1

.
.
.
def init(self):
"""Set values of computed attributes."""
# Effective batch size
self.BATCH_SIZE = self.IMAGES_PER_GPU * self.GPU_COUNT

This should be the bacth size = 1 but when I print it on the screen it continues with the value 2

And if you freeze the value of bacth size at 1 like this

def init(self):
"""Set values of computed attributes."""
# Effective batch size
self.BATCH_SIZE = 1

Print the error:

ValueError: slice index 1 of dimension 0 out of bounds. for 'ROI/strided_slice_12' (op: 'StridedSlice') with input shapes: [1,261888,4], [1], [1], [1] and with computed input tensors: input[1] = <1>, input[2] = <2>, input[3] = <1>.

If I ignore the line that check the BATCH_SIZE

if inference_model.config.BATCH_SIZE! = 1:

The system starts training but throws me another error when calculating the mAP

thank you very much in advance a greeting

[Juuustin]

Try using custom_callbacks. Counting accuracy takes a lot of time, so don't use it every epoch.

mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model,
model_inference, dataset_val, calculate_map_at_every_X_epoch=5, verbose=1)

model.train(dataset_train, dataset_val,
learning_rate=config.LEARNING_RATE,
epochs=100,
layers='heads',
custom_callbacks=[mean_average_precision_callback])

Thank you for sharing this code, and I have a small question: I am wondering why is there only dataset_val in the mean_average_precision_callback,

[Juuustin]

Hey @Juuustin. I divide the complete dataset into 3 sets: train, test, and val(60%-30%-10%). The first two are used when training the network, and the third is used to understand the accuracy on the data that the network does not see during training.

Hey, @VtlNmnk, is it available to get the accuracy during training, it would be better if we could get the accuracy during training:)

[rupa1118]

Hey @Juuustin. I divide the complete dataset into 3 sets: train, test, and val(60%-30%-10%). The first two are used when training the network, and the third is used to understand the accuracy on the data that the network does not see during training.

Hello @VtlNmnk can you explain how the validation works during the training process??

and do we need to change the mean values of the image in the configuration file??

[yoya93]

Hello and thanks for your previous answers.
@VtlNmnk @rupa1118

When executed it behaves like this:

And I have fluctuations just like @rupa1118

I got a better answer from val_loss ??
How did you improve it?
Greetings and thank you very much for your great help

[VtlNmnk]

How did you improve it?

It’s hard to say without additional information.
How many images are in the dataset? And how many classes? How did you split the data?
Do you have augmentation? If so, which one?

[TheScottishninja]

Can anyone in this thread explain how to get the loss output logging that we see in hardikmanik's screenshot? I'm talking about the various losses logged to stdout - I'm only seeing the total loss and none of the rest (like rpn_class_loss, rpn_box_loss, etc..).

I have tried all the solutions I'm facing the same issue ...... any solution?

I found I had the same using running on Colab. I think it's due to the Keras version they use (2.3.1 at the time of writing this). It seems the newer versions have a model.add_metric() method for adding a tensor to the list of metrics:

model = KM.Model(inputs, outputs, name='mask_rcnn')

# add metrics
            
model.add_metric(mask_loss, name="mask_loss")
model.add_metric(bbox_loss, name="bbox_loss")
model.add_metric(class_loss, name="class_loss")
model.add_metric(rpn_bbox_loss, name="rpn_bbox_loss")
model.add_metric(rpn_class_loss, name="rpn_class_loss")

Note that this is immediately after the model=KM.Model() in the build method. After adding these lines, the loss values appear while training:

12/100 [==>...........................] - ETA: 3:08 - loss: 1.3889 - mask_loss: 0.0680 - bbox_loss: 0.0478 - class_loss: 0.1614 - rpn_bbox_loss: 0.3475 - rpn_class_loss: 0.3476

[Romuns-Nicole]

还将其添加到model.py中的某个位置，例如在文件末尾

############################################################
#  Custom Callbacks
############################################################

class MeanAveragePrecisionCallback(Callback):
    def __init__(self, train_model: MaskRCNN, inference_model: MaskRCNN, dataset: Dataset,
                 calculate_map_at_every_X_epoch=5, dataset_limit=None,
                 verbose=1):
        super().__init__()
        self.train_model = train_model
        self.inference_model = inference_model
        self.dataset = dataset
        self.calculate_map_at_every_X_epoch = calculate_map_at_every_X_epoch
        self.dataset_limit = len(self.dataset.image_ids)
        if dataset_limit is not None:
            self.dataset_limit = dataset_limit
        self.dataset_image_ids = self.dataset.image_ids.copy()

        if inference_model.config.BATCH_SIZE != 1:
            raise ValueError("This callback only works with the bacth size of 1")

        self._verbose_print = print if verbose > 0 else lambda *a, **k: None

    def on_epoch_end(self, epoch, logs=None):

        if epoch > 2 and (epoch+1)%self.calculate_map_at_every_X_epoch == 0:
            self._verbose_print("Calculating mAP...")
            self._load_weights_for_model()

            mAPs = self._calculate_mean_average_precision()
            mAP = np.mean(mAPs)

            if logs is not None:
                logs["val_mean_average_precision"] = mAP

            self._verbose_print("mAP at epoch {0} is: {1}".format(epoch+1, mAP))

        super().on_epoch_end(epoch, logs)

    def _load_weights_for_model(self):
        last_weights_path = self.train_model.find_last()
        self._verbose_print("Loaded weights for the inference model (last checkpoint of the train model): {0}".format(
            last_weights_path))
        self.inference_model.load_weights(last_weights_path,
                                          by_name=True)

    def _calculate_mean_average_precision(self):
        mAPs = []

        # Use a random subset of the data when a limit is defined
        np.random.shuffle(self.dataset_image_ids)

        for image_id in self.dataset_image_ids[:self.dataset_limit]:
            image, image_meta, gt_class_id, gt_bbox, gt_mask = load_image_gt(self.dataset, self.inference_model.config,
                                                                             image_id, use_mini_mask=False)
            molded_images = np.expand_dims(mold_image(image, self.inference_model.config), 0)
            results = self.inference_model.detect(molded_images, verbose=0)
            r = results[0]
            # Compute mAP - VOC uses IoU 0.5
            AP, _, _, _ = utils.compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"],
                                           r["class_ids"], r["scores"], r['masks'])
            mAPs.append(AP)

        return np.array(mAPs)

Hello, thank you very much for your code. Your code is useful to me. How can I save the mAP value from training?

[milanzdr]

Can anyone in this thread explain how to get the loss output logging that we see in hardikmanik's screenshot? I'm talking about the various losses logged to stdout - I'm only seeing the total loss and none of the rest (like rpn_class_loss, rpn_box_loss, etc..).

In model.py, replace self.keras_model.fit_generator(..) with h=self.keras_model.fit_generator(..) and return h (Keras history object) as a result of train method of model.py. Then in your script, after h=model.train(...), refer to h.history['loss'], h.history['mrcnn_class_loss'], etc. for plotting loss diagrams.

[mhtarora39]

By Default matterport added tensorboard callbacks during training.In your log directory you will get event files along with your weights files. You can refer that file for tensorboard or you can directly run tensorboard on that directly you will get the all the losses.

Tensorboard command : tensorboard --logdir "path/to/logs" --port 8888(or your opened port)

[tomjordan]

hi @mhtarora39

I am trying to use your code to calculate map, precision, and recall but get the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-23-a2f082d763b6> in <module>
     64 config = InferenceConfig()
     65 evel = EvalImage(dataset_vall,modelinf,config)
---> 66 evel.evaluate_model()
     67 

<ipython-input-23-a2f082d763b6> in evaluate_model(self, len)
     51       r = yhat[0]
     52       # calculate statistics, including AP
---> 53       AP, precisions, recalls, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
     54       precisions_dict[image_id] = np.mean(precisions)
     55       recall_dict[image_id] = np.mean(recalls)

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_ap(gt_boxes, gt_class_ids, gt_masks, pred_boxes, pred_class_ids, pred_scores, pred_masks, iou_threshold)
    728         gt_boxes, gt_class_ids, gt_masks,
    729         pred_boxes, pred_class_ids, pred_scores, pred_masks,
--> 730         iou_threshold)
    731 
    732     # Compute precision and recall at each prediction box step

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_matches(gt_boxes, gt_class_ids, gt_masks, pred_boxes, pred_class_ids, pred_scores, pred_masks, iou_threshold, score_threshold)
    680 
    681     # Compute IoU overlaps [pred_masks, gt_masks]
--> 682     overlaps = compute_overlaps_masks(pred_masks, gt_masks)
    683 
    684     # Loop through predictions and find matching ground truth boxes

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_overlaps_masks(masks1, masks2)
    113 
    114     # intersections and union
--> 115     intersections = np.dot(masks1.T, masks2)
    116     union = area1[:, None] + area2[None, :] - intersections
    117     overlaps = intersections / union

<__array_function__ internals> in dot(*args, **kwargs)

ValueError: shapes (5,1048576) and (3136,6) not aligned: 1048576 (dim 1) != 3136 (dim 0)

This is my implementation:

from mrcnn.utils import compute_ap

class EvalImage():
  def __init__(self,dataset,model,cfg):
    self.dataset = dataset
    self.model   = model
    self.cfg     = cfg


  def evaluate_model(self , len = 50):
    APs = list()
    precisions_dict = {}
    recall_dict     = {}
    for index,image_id in enumerate(self.dataset.image_ids):
      if(index > len):
         break; 
      # load image, bounding boxes and masks for the image id
      image, image_meta, gt_class_id, gt_bbox, gt_mask = modellib.load_image_gt(self.dataset, self.cfg,image_id)
      # convert pixel values (e.g. center)
      #scaled_image = modellib.mold_image(image, self.cfg)
      # convert image into one sample
      sample = np.expand_dims(image, 0)
    
     # print(len(image))
      # make prediction
      yhat = self.model.detect(sample, verbose=1)
      # extract results for first sample
      r = yhat[0]
      # calculate statistics, including AP
      AP, precisions, recalls, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
      precisions_dict[image_id] = np.mean(precisions)
      recall_dict[image_id] = np.mean(recalls)
      # store
      APs.append(AP)

    # calculate the mean AP across all images
    mAP = np.mean(APs)
    return mAP,precisions_dict,recall_dict


dataset_vall = CocoLikeDataset()
dataset_vall.load_data('D:/Desktop/aktwelve_mask_rcnn/datasets/Acacia dataset', 'val')
dataset_vall.prepare()

config = InferenceConfig()

modelinf = modellib.MaskRCNN(mode="inference", 
                          config=config,
                          model_dir=MODEL_DIR)

modelinf.load_weights(os.path.join(ROOT_DIR, "mask_rcnn_acacia-stp73-epch150_0075.h5"), by_name=True)

evel = EvalImage(dataset_vall,modelinf,config)
evel.evaluate_model()

Any suggestions would be appreciated

[morganaribeiro]

Can anyone give me a suggestion on how to solve this problem after the -------- ? I'm trying to run a Mask R-CNN code.
PLEASE.
`Configurations:
BACKBONE resnet101
BACKBONE_STRIDES [4, 8, 16, 32, 64]
BATCH_SIZE 1
BBOX_STD_DEV [0.1 0.1 0.2 0.2]
COMPUTE_BACKBONE_SHAPE None
DETECTION_MAX_INSTANCES 100
DETECTION_MIN_CONFIDENCE 0.95
DETECTION_NMS_THRESHOLD 0.3
FPN_CLASSIF_FC_LAYERS_SIZE 1024
GPU_COUNT 1
GRADIENT_CLIP_NORM 5.0
IMAGES_PER_GPU 1
IMAGE_MAX_DIM 1024
IMAGE_META_SIZE 14
IMAGE_MIN_DIM 704
IMAGE_MIN_SCALE 0
IMAGE_RESIZE_MODE square
IMAGE_SHAPE [1024 1024 3]
LEARNING_MOMENTUM 0.9
LEARNING_RATE 0.001
LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0}
MASK_POOL_SIZE 14
MASK_SHAPE [28, 28]
MAX_GT_INSTANCES 30
MEAN_PIXEL [123.7 116.8 103.9]
MINI_MASK_SHAPE (56, 56)
NAME shapes
NUM_CLASSES 2
POOL_SIZE 7
POST_NMS_ROIS_INFERENCE 1000
POST_NMS_ROIS_TRAINING 2000
ROI_POSITIVE_RATIO 0.33
RPN_ANCHOR_RATIOS [0.5, 1, 2]
RPN_ANCHOR_SCALES (48, 96, 192, 384, 768)
RPN_ANCHOR_STRIDE 1
RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2]
RPN_NMS_THRESHOLD 0.7
RPN_TRAIN_ANCHORS_PER_IMAGE 256
STEPS_PER_EPOCH 3500
TOP_DOWN_PYRAMID_SIZE 256
TRAIN_BN False
TRAIN_ROIS_PER_IMAGE 300
USE_MINI_MASK True
USE_RPN_ROIS True
VALIDATION_STEPS 300
WEIGHT_DECAY 0.0001

train_data/labelme_json/DSCN46592_json/img.png
train_data/labelme_json/DSCN46601_json/img.png
train_data/labelme_json/DSCN46602_json/img.png
train_data/labelme_json/DSCN46611_json/img.png
train_data/labelme_json/DSCN46612_json/img.png
train_data/labelme_json/DSCN46615_json/img.png
train_data/labelme_json/DSCN46621_json/img.png
train_data/labelme_json/DSCN46622_json/img.png
train_data/labelme_json/DSCN46625_json/img.png
train_data/labelme_json/DSCN46631_json/img.png
train_data/labelme_json/DSCN46632_json/img.png
train_data/labelme_json/DSCN46635_json/img.png
train_data/labelme_json/DSCN46641_json/img.png
train_data/labelme_json/DSCN46642_json/img.png
train_data/labelme_json/DSCN46645_json/img.png
train_data/labelme_json/DSCN46651_json/img.png
train_data/labelme_json/DSCN46652_json/img.png
train_data/labelme_json/DSCN46655_json/img.png
train_data/labelme_json/DSCN46661_json/img.png
train_data/labelme_json/DSCN46662_json/img.png
train_data/labelme_json/DSCN46665_json/img.png
train_data/labelme_json/DSCN46671_json/img.png
train_data/labelme_json/DSCN46672_json/img.png
train_data/labelme_json/DSCN46675_json/img.png
train_data/labelme_json/DSCN46681_json/img.png
train_data/labelme_json/DSCN46682_json/img.png
train_data/labelme_json/DSCN46685_json/img.png
train_data/labelme_json/DSCN46691_json/img.png
train_data/labelme_json/DSCN46692_json/img.png
train_data/labelme_json/DSCN46695_json/img.png
train_data/labelme_json/DSCN46701_json/img.png
train_data/labelme_json/DSCN46702_json/img.png
train_data/labelme_json/DSCN46705_json/img.png
train_data/labelme_json/DSCN46711_json/img.png
train_data/labelme_json/DSCN46712_json/img.png
train_data/labelme_json/DSCN46713_json/img.png
train_data/labelme_json/DSCN46715_json/img.png
train_data/labelme_json/DSCN46721_json/img.png
train_data/labelme_json/DSCN46722_json/img.png
train_data/labelme_json/DSCN46723_json/img.png
train_data/labelme_json/DSCN46725_json/img.png
train_data/labelme_json/DSCN46731_json/img.png
train_data/labelme_json/DSCN46732_json/img.png
train_data/labelme_json/DSCN46733_json/img.png
train_data/labelme_json/DSCN46735_json/img.png
train_data/labelme_json/DSCN46741_json/img.png
train_data/labelme_json/DSCN46742_json/img.png
train_data/labelme_json/DSCN46743_json/img.png
train_data/labelme_json/DSCN46745_json/img.png
train_data/labelme_json/DSCN46751_json/img.png
train_data/labelme_json/DSCN46752_json/img.png
train_data/labelme_json/DSCN46753_json/img.png
train_data/labelme_json/DSCN46755_json/img.png
train_data/labelme_json/DSCN46761_json/img.png
train_data/labelme_json/DSCN46762_json/img.png
train_data/labelme_json/DSCN46763_json/img.png
train_data/labelme_json/DSCN46765_json/img.png
train_data/labelme_json/DSCN46771_json/img.png
train_data/labelme_json/DSCN46772_json/img.png
train_data/labelme_json/DSCN46773_json/img.png
train_data/labelme_json/DSCN46775_json/img.png
train_data/labelme_json/DSCN46781_json/img.png
train_data/labelme_json/DSCN46782_json/img.png
train_data/labelme_json/DSCN46783_json/img.png
train_data/labelme_json/DSCN46785_json/img.png
train_data/labelme_json/DSCN46791_json/img.png
train_data/labelme_json/DSCN46792_json/img.png
train_data/labelme_json/DSCN46793_json/img.png
train_data/labelme_json/DSCN46795_json/img.png
train_data/labelme_json/DSCN46801_json/img.png
train_data/labelme_json/DSCN46802_json/img.png
train_data/labelme_json/DSCN46803_json/img.png
train_data/labelme_json/DSCN46805_json/img.png
train_data/labelme_json/DSCN46811_json/img.png
train_data/labelme_json/DSCN46812_json/img.png
train_data/labelme_json/DSCN46813_json/img.png
train_data/labelme_json/DSCN46815_json/img.png
train_data/labelme_json/DSCN46821_json/img.png
train_data/labelme_json/DSCN46822_json/img.png
train_data/labelme_json/DSCN46823_json/img.png
train_data/labelme_json/DSCN46825_json/img.png
train_data/labelme_json/DSCN46831_json/img.png
train_data/labelme_json/DSCN46832_json/img.png
train_data/labelme_json/DSCN46833_json/img.png
train_data/labelme_json/DSCN46835_json/img.png
train_data/labelme_json/DSCN46841_json/img.png
train_data/labelme_json/DSCN46842_json/img.png
train_data/labelme_json/DSCN46843_json/img.png
train_data/labelme_json/DSCN46845_json/img.png
train_data/labelme_json/DSCN46851_json/img.png
train_data/labelme_json/DSCN46852_json/img.png
train_data/labelme_json/DSCN46853_json/img.png
train_data/labelme_json/DSCN46855_json/img.png
train_data/labelme_json/DSCN46861_json/img.png
train_data/labelme_json/DSCN46862_json/img.png
train_data/labelme_json/DSCN46863_json/img.png
train_data/labelme_json/DSCN46865_json/img.png
train_data/labelme_json/DSCN46871_json/img.png
train_data/labelme_json/DSCN46872_json/img.png
train_data/labelme_json/DSCN46873_json/img.png
train_data/labelme_json/DSCN46875_json/img.png
train_data/labelme_json/DSCN46881_json/img.png
train_data/labelme_json/DSCN46882_json/img.png
train_data/labelme_json/DSCN46883_json/img.png
train_data/labelme_json/DSCN46885_json/img.png
train_data/labelme_json/DSCN46891_json/img.png
train_data/labelme_json/DSCN46892_json/img.png
train_data/labelme_json/DSCN46893_json/img.png
train_data/labelme_json/DSCN46895_json/img.png
train_data/labelme_json/DSCN46901_json/img.png
train_data/labelme_json/DSCN46902_json/img.png
train_data/labelme_json/DSCN46903_json/img.png
train_data/labelme_json/DSCN46905_json/img.png
train_data/labelme_json/DSCN46911_json/img.png
train_data/labelme_json/DSCN46912_json/img.png
train_data/labelme_json/DSCN46913_json/img.png
train_data/labelme_json/DSCN46915_json/img.png
train_data/labelme_json/DSCN46921_json/img.png
train_data/labelme_json/DSCN46922_json/img.png
train_data/labelme_json/DSCN46923_json/img.png
train_data/labelme_json/DSCN46925_json/img.png
train_data/labelme_json/DSCN46931_json/img.png
train_data/labelme_json/DSCN46932_json/img.png
train_data/labelme_json/DSCN46933_json/img.png
train_data/labelme_json/DSCN46935_json/img.png
train_data/labelme_json/DSCN46941_json/img.png
train_data/labelme_json/DSCN46942_json/img.png
train_data/labelme_json/DSCN46943_json/img.png
train_data/labelme_json/DSCN46945_json/img.png
train_data/labelme_json/DSCN46951_json/img.png
train_data/labelme_json/DSCN46952_json/img.png
train_data/labelme_json/DSCN46953_json/img.png
train_data/labelme_json/DSCN46955_json/img.png
train_data/labelme_json/DSCN46961_json/img.png
train_data/labelme_json/DSCN46962_json/img.png
train_data/labelme_json/DSCN46963_json/img.png
train_data/labelme_json/DSCN46965_json/img.png
train_data/labelme_json/DSCN46971_json/img.png
train_data/labelme_json/DSCN46972_json/img.png
train_data/labelme_json/DSCN46973_json/img.png
train_data/labelme_json/DSCN46975_json/img.png
train_data/labelme_json/DSCN46981_json/img.png
train_data/labelme_json/DSCN46982_json/img.png
train_data/labelme_json/DSCN46983_json/img.png
train_data/labelme_json/DSCN46985_json/img.png
train_data/labelme_json/DSCN46991_json/img.png
train_data/labelme_json/DSCN46992_json/img.png
train_data/labelme_json/DSCN46993_json/img.png
train_data/labelme_json/DSCN46995_json/img.png
train_data/labelme_json/DSCN47001_json/img.png
train_data/labelme_json/DSCN47002_json/img.png
train_data/labelme_json/DSCN47003_json/img.png
train_data/labelme_json/DSCN47005_json/img.png
train_data/labelme_json/DSCN47011_json/img.png
train_data/labelme_json/DSCN47012_json/img.png
train_data/labelme_json/DSCN47013_json/img.png
train_data/labelme_json/DSCN47015_json/img.png
train_data/labelme_json/DSCN47021_json/img.png
train_data/labelme_json/DSCN47022_json/img.png
train_data/labelme_json/DSCN47023_json/img.png
train_data/labelme_json/DSCN47025_json/img.png
train_data/labelme_json/DSCN47031_json/img.png
train_data/labelme_json/DSCN47032_json/img.png
train_data/labelme_json/DSCN47033_json/img.png
train_data/labelme_json/DSCN47035_json/img.png
train_data/labelme_json/DSCN47041_json/img.png
train_data/labelme_json/DSCN47042_json/img.png
train_data/labelme_json/DSCN47043_json/img.png
train_data/labelme_json/DSCN47045_json/img.png
train_data/labelme_json/DSCN47051_json/img.png
train_data/labelme_json/DSCN47052_json/img.png
train_data/labelme_json/DSCN47053_json/img.png
train_data/labelme_json/DSCN47055_json/img.png
train_data/labelme_json/DSCN47061_json/img.png
train_data/labelme_json/DSCN47062_json/img.png
train_data/labelme_json/DSCN47063_json/img.png
train_data/labelme_json/DSCN47065_json/img.png
train_data/labelme_json/DSCN47071_json/img.png
train_data/labelme_json/DSCN47072_json/img.png
train_data/labelme_json/DSCN47073_json/img.png
train_data/labelme_json/DSCN47075_json/img.png
train_data/labelme_json/DSCN47081_json/img.png
train_data/labelme_json/DSCN47082_json/img.png
train_data/labelme_json/DSCN47083_json/img.png
train_data/labelme_json/DSCN47085_json/img.png
train_data/labelme_json/DSCN47091_json/img.png
train_data/labelme_json/DSCN47092_json/img.png
train_data/labelme_json/DSCN47093_json/img.png
train_data/labelme_json/DSCN47095_json/img.png
train_data/labelme_json/DSCN47101_json/img.png
train_data/labelme_json/DSCN47102_json/img.png
train_data/labelme_json/DSCN47103_json/img.png
train_data/labelme_json/DSCN47105_json/img.png
train_data/labelme_json/DSCN47111_json/img.png
train_data/labelme_json/DSCN47112_json/img.png
train_data/labelme_json/DSCN47113_json/img.png
train_data/labelme_json/DSCN47115_json/img.png
train_data/labelme_json/DSCN47121_json/img.png
train_data/labelme_json/DSCN47122_json/img.png
train_data/labelme_json/DSCN47123_json/img.png
train_data/labelme_json/DSCN47125_json/img.png
train_data/labelme_json/DSCN47131_json/img.png
train_data/labelme_json/DSCN47132_json/img.png
train_data/labelme_json/DSCN47133_json/img.png
train_data/labelme_json/DSCN47135_json/img.png
train_data/labelme_json/DSCN47141_json/img.png
train_data/labelme_json/DSCN47142_json/img.png
train_data/labelme_json/DSCN47143_json/img.png
train_data/labelme_json/DSCN47145_json/img.png
train_data/labelme_json/DSCN47151_json/img.png
train_data/labelme_json/DSCN47152_json/img.png
train_data/labelme_json/DSCN47153_json/img.png
train_data/labelme_json/DSCN47155_json/img.png
train_data/labelme_json/DSCN47161_json/img.png
train_data/labelme_json/DSCN47162_json/img.png
train_data/labelme_json/DSCN47163_json/img.png
train_data/labelme_json/DSCN47165_json/img.png
train_data/labelme_json/DSCN47171_json/img.png
train_data/labelme_json/DSCN47172_json/img.png
train_data/labelme_json/DSCN47173_json/img.png
train_data/labelme_json/DSCN47175_json/img.png
train_data/labelme_json/DSCN47181_json/img.png
train_data/labelme_json/DSCN47182_json/img.png
train_data/labelme_json/DSCN47183_json/img.png
train_data/labelme_json/DSCN47185_json/img.png
train_data/labelme_json/DSCN47191_json/img.png
train_data/labelme_json/DSCN47192_json/img.png
train_data/labelme_json/DSCN47193_json/img.png
train_data/labelme_json/DSCN47195_json/img.png
train_data/labelme_json/DSCN47201_json/img.png
train_data/labelme_json/DSCN47202_json/img.png
train_data/labelme_json/DSCN47203_json/img.png
train_data/labelme_json/DSCN47205_json/img.png
train_data/labelme_json/DSCN47211_json/img.png
train_data/labelme_json/DSCN47212_json/img.png
train_data/labelme_json/DSCN47213_json/img.png
train_data/labelme_json/DSCN47215_json/img.png
train_data/labelme_json/DSCN47221_json/img.png
train_data/labelme_json/DSCN47222_json/img.png
train_data/labelme_json/DSCN47223_json/img.png
train_data/labelme_json/DSCN47225_json/img.png
train_data/labelme_json/DSCN47231_json/img.png
train_data/labelme_json/DSCN47232_json/img.png
train_data/labelme_json/DSCN47233_json/img.png
train_data/labelme_json/DSCN47235_json/img.png
train_data/labelme_json/DSCN47241_json/img.png
train_data/labelme_json/DSCN47242_json/img.png
train_data/labelme_json/DSCN47243_json/img.png
train_data/labelme_json/DSCN47245_json/img.png
train_data/labelme_json/DSCN47251_json/img.png
train_data/labelme_json/DSCN47252_json/img.png
train_data/labelme_json/DSCN47253_json/img.png
train_data/labelme_json/DSCN47255_json/img.png
train_data/labelme_json/DSCN47261_json/img.png
train_data/labelme_json/DSCN47262_json/img.png
train_data/labelme_json/DSCN47263_json/img.png
train_data/labelme_json/DSCN47265_json/img.png
train_data/labelme_json/DSCN47271_json/img.png
train_data/labelme_json/DSCN47272_json/img.png
train_data/labelme_json/DSCN47273_json/img.png
train_data/labelme_json/DSCN47275_json/img.png
train_data/labelme_json/DSCN47281_json/img.png
train_data/labelme_json/DSCN47282_json/img.png
train_data/labelme_json/DSCN47283_json/img.png
train_data/labelme_json/DSCN47285_json/img.png
train_data/labelme_json/DSCN47291_json/img.png
train_data/labelme_json/DSCN47292_json/img.png
train_data/labelme_json/DSCN47293_json/img.png
train_data/labelme_json/DSCN47295_json/img.png
train_data/labelme_json/DSCN47301_json/img.png
train_data/labelme_json/DSCN47302_json/img.png
train_data/labelme_json/DSCN47303_json/img.png
train_data/labelme_json/DSCN47305_json/img.png
train_data/labelme_json/DSCN47311_json/img.png
train_data/labelme_json/DSCN47312_json/img.png
train_data/labelme_json/DSCN47313_json/img.png
train_data/labelme_json/DSCN47315_json/img.png
train_data/labelme_json/DSCN47321_json/img.png
train_data/labelme_json/DSCN47322_json/img.png
train_data/labelme_json/DSCN47323_json/img.png
train_data/labelme_json/DSCN47325_json/img.png
train_data/labelme_json/DSCN47331_json/img.png
train_data/labelme_json/DSCN47332_json/img.png
train_data/labelme_json/DSCN47333_json/img.png
train_data/labelme_json/DSCN47335_json/img.png
train_data/labelme_json/DSCN47341_json/img.png
train_data/labelme_json/DSCN47342_json/img.png
train_data/labelme_json/DSCN47343_json/img.png
train_data/labelme_json/DSCN47345_json/img.png
train_data/labelme_json/DSCN47351_json/img.png
train_data/labelme_json/DSCN47352_json/img.png
train_data/labelme_json/DSCN47353_json/img.png
train_data/labelme_json/DSCN47355_json/img.png
train_data/labelme_json/DSCN47361_json/img.png
train_data/labelme_json/DSCN47362_json/img.png
train_data/labelme_json/DSCN47363_json/img.png
train_data/labelme_json/DSCN47365_json/img.png
train_data/labelme_json/DSCN47371_json/img.png
train_data/labelme_json/DSCN47372_json/img.png
train_data/labelme_json/DSCN47373_json/img.png
train_data/labelme_json/DSCN47375_json/img.png
train_data/labelme_json/DSCN47381_json/img.png
train_data/labelme_json/DSCN47382_json/img.png
train_data/labelme_json/DSCN47383_json/img.png
train_data/labelme_json/DSCN47385_json/img.png
train_data/labelme_json/DSCN47391_json/img.png
train_data/labelme_json/DSCN47392_json/img.png
train_data/labelme_json/DSCN47393_json/img.png
train_data/labelme_json/DSCN47395_json/img.png
train_data/labelme_json/DSCN47401_json/img.png
train_data/labelme_json/DSCN47402_json/img.png
train_data/labelme_json/DSCN47403_json/img.png
train_data/labelme_json/DSCN47405_json/img.png
train_data/labelme_json/DSCN47411_json/img.png
train_data/labelme_json/DSCN47412_json/img.png
train_data/labelme_json/DSCN47413_json/img.png
train_data/labelme_json/DSCN47415_json/img.png
train_data/labelme_json/DSCN47421_json/img.png
train_data/labelme_json/DSCN47422_json/img.png
train_data/labelme_json/DSCN47423_json/img.png
train_data/labelme_json/DSCN47425_json/img.png
train_data/labelme_json/DSCN47431_json/img.png
train_data/labelme_json/DSCN47432_json/img.png
train_data/labelme_json/DSCN47433_json/img.png
train_data/labelme_json/DSCN47435_json/img.png
train_data/labelme_json/DSCN47441_json/img.png
train_data/labelme_json/DSCN47442_json/img.png
train_data/labelme_json/DSCN47443_json/img.png
train_data/labelme_json/DSCN47445_json/img.png
train_data/labelme_json/DSCN47451_json/img.png
train_data/labelme_json/DSCN47452_json/img.png
train_data/labelme_json/DSCN47453_json/img.png
train_data/labelme_json/DSCN47455_json/img.png
train_data/labelme_json/DSCN47461_json/img.png
train_data/labelme_json/DSCN47462_json/img.png
train_data/labelme_json/DSCN47463_json/img.png
train_data/labelme_json/DSCN47465_json/img.png
train_data/labelme_json/DSCN47471_json/img.png
train_data/labelme_json/DSCN47472_json/img.png
train_data/labelme_json/DSCN47475_json/img.png
train_data/labelme_json/DSCN47491_json/img.png
train_data/labelme_json/DSCN47492_json/img.png
train_data/labelme_json/DSCN47495_json/img.png
train_data/labelme_json/DSCN47501_json/img.png
train_data/labelme_json/DSCN47502_json/img.png
train_data/labelme_json/DSCN47505_json/img.png
train_data/labelme_json/DSCN47511_json/img.png
train_data/labelme_json/DSCN47512_json/img.png
train_data/labelme_json/DSCN47515_json/img.png
train_data/labelme_json/DSCN47521_json/img.png
train_data/labelme_json/DSCN47522_json/img.png
train_data/labelme_json/DSCN47525_json/img.png
train_data/labelme_json/DSCN47531_json/img.png
train_data/labelme_json/DSCN47532_json/img.png
train_data/labelme_json/DSCN47535_json/img.png
train_data/labelme_json/DSCN47541_json/img.png
train_data/labelme_json/DSCN47542_json/img.png
train_data/labelme_json/DSCN47545_json/img.png
train_data/labelme_json/DSCN47551_json/img.png
train_data/labelme_json/DSCN47552_json/img.png
train_data/labelme_json/DSCN47555_json/img.png
train_data/labelme_json/DSCN47561_json/img.png
train_data/labelme_json/DSCN47562_json/img.png
train_data/labelme_json/DSCN47565_json/img.png
train_data/labelme_json/DSCN47571_json/img.png
train_data/labelme_json/DSCN47572_json/img.png
train_data/labelme_json/DSCN47575_json/img.png
train_data/labelme_json/DSCN47581_json/img.png
train_data/labelme_json/DSCN47582_json/img.png
train_data/labelme_json/DSCN47585_json/img.png
train_data/labelme_json/DSCN47591_json/img.png
train_data/labelme_json/DSCN47592_json/img.png
train_data/labelme_json/DSCN47595_json/img.png
train_data/labelme_json/DSCN47601_json/img.png
train_data/labelme_json/DSCN47602_json/img.png
train_data/labelme_json/DSCN47605_json/img.png
train_data/labelme_json/DSCN47611_json/img.png
train_data/labelme_json/DSCN47612_json/img.png
train_data/labelme_json/DSCN47615_json/img.png
train_data/labelme_json/DSCN47621_json/img.png
train_data/labelme_json/DSCN47622_json/img.png
train_data/labelme_json/DSCN47625_json/img.png
train_data/labelme_json/DSCN47631_json/img.png
train_data/labelme_json/DSCN47632_json/img.png
train_data/labelme_json/DSCN47635_json/img.png
train_data/labelme_json/DSCN47641_json/img.png
train_data/labelme_json/DSCN47642_json/img.png
train_data/labelme_json/DSCN47645_json/img.png
train_data/labelme_json/DSCN47651_json/img.png
train_data/labelme_json/DSCN47652_json/img.png
train_data/labelme_json/DSCN47655_json/img.png
train_data/labelme_json/DSCN47661_json/img.png
train_data/labelme_json/DSCN47662_json/img.png
train_data/labelme_json/DSCN47665_json/img.png
train_data/labelme_json/DSCN47671_json/img.png
train_data/labelme_json/DSCN47672_json/img.png
train_data/labelme_json/DSCN47675_json/img.png
train_data/labelme_json/DSCN47681_json/img.png
train_data/labelme_json/DSCN47682_json/img.png
train_data/labelme_json/DSCN47685_json/img.png
train_data/labelme_json/DSCN46592_json/img.png
train_data/labelme_json/DSCN46601_json/img.png
train_data/labelme_json/DSCN46602_json/img.png
train_data/labelme_json/DSCN46611_json/img.png
train_data/labelme_json/DSCN46612_json/img.png
train_data/labelme_json/DSCN46615_json/img.png
train_data/labelme_json/DSCN46621_json/img.png

ValueError                                Traceback (most recent call last)
<ipython-input-1-50b8c26e8d29> in <module>
    214 # Create model in training mode
    215 model = modellib.MaskRCNN(mode="training", config=config,
--> 216                           model_dir=MODEL_DIR)
    217 
    218 # Which weights to start with?

~\Fish-characteristic-measurement\Complete code\mrcnn\model.py in __init__(self, mode, config, model_dir)
   1830         self.model_dir = model_dir
   1831         self.set_log_dir()
-> 1832         self.keras_model = self.build(mode=mode, config=config)
   1833 
   1834     def build(self, mode, config):

~\Fish-characteristic-measurement\Complete code\mrcnn\model.py in build(self, mode, config)
   1927             anchors = np.broadcast_to(anchors, (config.BATCH_SIZE,) + anchors.shape)
   1928             # A hack to get around Keras's bad support for constants
-> 1929             anchors = KL.Lambda(lambda x: tf.Variable(anchors), name="anchors")(input_image)
   1930         else:
   1931             anchors = input_anchors

~\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in __call__(self, *args, **kwargs)
    920                     not base_layer_utils.is_in_eager_or_tf_function()):
    921                   with auto_control_deps.AutomaticControlDependencies() as acd:
--> 922                     outputs = call_fn(cast_inputs, *args, **kwargs)
    923                     # Wrap Tensors in `outputs` in `tf.identity` to avoid
    924                     # circular dependencies.

~\Anaconda3\lib\site-packages\tensorflow\python\keras\layers\core.py in call(self, inputs, mask, training)
    887         variable_scope.variable_creator_scope(_variable_creator):
    888       result = self.function(inputs, **kwargs)
--> 889     self._check_variables(created_variables, tape.watched_variables())
    890     return result
    891 

~\Anaconda3\lib\site-packages\tensorflow\python\keras\layers\core.py in _check_variables(self, created_variables, accessed_variables)
    914           Variables.'''
    915       ).format(name=self.name, variable_str=variable_str)
--> 916       raise ValueError(error_str)
    917 
    918     untracked_used_vars = [

ValueError: 
The following Variables were created within a Lambda layer (anchors)
but are not tracked by said layer:
  <tf.Variable 'anchors/Variable:0' shape=(1, 261888, 4) dtype=float32>
The layer cannot safely ensure proper Variable reuse across multiple
calls, and consquently this behavior is disallowed for safety. Lambda
layers are not well suited to stateful computation; instead, writing a
subclassed Layer is the recommend way to define layers with
Variables.
`

[sohinimallick]

hi @mhtarora39

I am trying to use your code to calculate map, precision, and recall but get the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-23-a2f082d763b6> in <module>
     64 config = InferenceConfig()
     65 evel = EvalImage(dataset_vall,modelinf,config)
---> 66 evel.evaluate_model()
     67 

<ipython-input-23-a2f082d763b6> in evaluate_model(self, len)
     51       r = yhat[0]
     52       # calculate statistics, including AP
---> 53       AP, precisions, recalls, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
     54       precisions_dict[image_id] = np.mean(precisions)
     55       recall_dict[image_id] = np.mean(recalls)

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_ap(gt_boxes, gt_class_ids, gt_masks, pred_boxes, pred_class_ids, pred_scores, pred_masks, iou_threshold)
    728         gt_boxes, gt_class_ids, gt_masks,
    729         pred_boxes, pred_class_ids, pred_scores, pred_masks,
--> 730         iou_threshold)
    731 
    732     # Compute precision and recall at each prediction box step

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_matches(gt_boxes, gt_class_ids, gt_masks, pred_boxes, pred_class_ids, pred_scores, pred_masks, iou_threshold, score_threshold)
    680 
    681     # Compute IoU overlaps [pred_masks, gt_masks]
--> 682     overlaps = compute_overlaps_masks(pred_masks, gt_masks)
    683 
    684     # Loop through predictions and find matching ground truth boxes

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_overlaps_masks(masks1, masks2)
    113 
    114     # intersections and union
--> 115     intersections = np.dot(masks1.T, masks2)
    116     union = area1[:, None] + area2[None, :] - intersections
    117     overlaps = intersections / union

<__array_function__ internals> in dot(*args, **kwargs)

ValueError: shapes (5,1048576) and (3136,6) not aligned: 1048576 (dim 1) != 3136 (dim 0)

This is my implementation:

from mrcnn.utils import compute_ap

class EvalImage():
  def __init__(self,dataset,model,cfg):
    self.dataset = dataset
    self.model   = model
    self.cfg     = cfg


  def evaluate_model(self , len = 50):
    APs = list()
    precisions_dict = {}
    recall_dict     = {}
    for index,image_id in enumerate(self.dataset.image_ids):
      if(index > len):
         break; 
      # load image, bounding boxes and masks for the image id
      image, image_meta, gt_class_id, gt_bbox, gt_mask = modellib.load_image_gt(self.dataset, self.cfg,image_id)
      # convert pixel values (e.g. center)
      #scaled_image = modellib.mold_image(image, self.cfg)
      # convert image into one sample
      sample = np.expand_dims(image, 0)
    
     # print(len(image))
      # make prediction
      yhat = self.model.detect(sample, verbose=1)
      # extract results for first sample
      r = yhat[0]
      # calculate statistics, including AP
      AP, precisions, recalls, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
      precisions_dict[image_id] = np.mean(precisions)
      recall_dict[image_id] = np.mean(recalls)
      # store
      APs.append(AP)

    # calculate the mean AP across all images
    mAP = np.mean(APs)
    return mAP,precisions_dict,recall_dict


dataset_vall = CocoLikeDataset()
dataset_vall.load_data('D:/Desktop/aktwelve_mask_rcnn/datasets/Acacia dataset', 'val')
dataset_vall.prepare()

config = InferenceConfig()

modelinf = modellib.MaskRCNN(mode="inference", 
                          config=config,
                          model_dir=MODEL_DIR)

modelinf.load_weights(os.path.join(ROOT_DIR, "mask_rcnn_acacia-stp73-epch150_0075.h5"), by_name=True)

evel = EvalImage(dataset_vall,modelinf,config)
evel.evaluate_model()

Any suggestions would be appreciated

Hey I have the exact same issue. Did you find a solution to this error??

[nerdijoe]

@yoya93 and other guys, advice on any project. First achieve a well-known working condition. Then add the changes one at a time, and each time check whether it works or not.
With the repetition of the Mask-RCNN project from Matterport, you first need to repeat their project as it is and get the same result. Then, when there is a working project, copy this project, change the input data to your own, change the number of classes, etc. And after each change, check that nothing is broken.
Regarding your specific question: the source code of the "Config" class does not need to be changed. It is enough to access it from your code correctly. Here is an example of my call to the class. Naturally, you first need to import
from mrcnn.config import Config

Then you can see the properties of your configuration for inference
config = _InfConfig()
config.display()

Hi @VtlNmnk , how did you get the save_each_n_epoch argument? I couldn't find it in model.py

I managed to run the balloon training with the custom callback MeanAveragePrecisionCallback, however the mAP calculation was never printed out during the training. Did I initialize the callback correctly?

class CustomConfig(BalloonConfig):
    NAME = "ballon_custom"
    IMAGES_PER_GPU = 1
    GPU_COUNT = 1
    IMAGE_RESIZE_MODE = "square"
    DETECTION_MIN_CONFIDENCE = 0.0
    NUM_CLASSES = 1 + 1
    STEPS_PER_EPOCH = 100
    USE_RPN_ROIS = False


def train(model):
    """Train the model."""
    # Training dataset.
    dataset_train = BalloonDataset()
    dataset_train.load_balloon(args.dataset, "train")
    dataset_train.prepare()

    # Validation dataset
    dataset_val = BalloonDataset()
    dataset_val.load_balloon(args.dataset, "val")
    dataset_val.prepare()


    model_inference = modellib.MaskRCNN(mode="inference", config=CustomConfig(),
                                model_dir=args.logs)

    mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, 
        model_inference, dataset_val, calculate_map_at_every_X_epoch=10, verbose=1)

    model.train(dataset_train, dataset_val,
                learning_rate=config.LEARNING_RATE,
                epochs=30,
                layers='heads', # heads or all
                custom_callbacks=[mean_average_precision_callback])

[mallyagirish]

also add this somewhere in the model.py, for example at the end of the file

############################################################
#  Custom Callbacks
############################################################

class MeanAveragePrecisionCallback(Callback):
    def __init__(self, train_model: MaskRCNN, inference_model: MaskRCNN, dataset: Dataset,
                 calculate_map_at_every_X_epoch=5, dataset_limit=None,
                 verbose=1):
        super().__init__()
        self.train_model = train_model
        self.inference_model = inference_model
        self.dataset = dataset
        self.calculate_map_at_every_X_epoch = calculate_map_at_every_X_epoch
        self.dataset_limit = len(self.dataset.image_ids)
        if dataset_limit is not None:
            self.dataset_limit = dataset_limit
        self.dataset_image_ids = self.dataset.image_ids.copy()

        if inference_model.config.BATCH_SIZE != 1:
            raise ValueError("This callback only works with the bacth size of 1")

        self._verbose_print = print if verbose > 0 else lambda *a, **k: None

    def on_epoch_end(self, epoch, logs=None):

        if epoch > 2 and (epoch+1)%self.calculate_map_at_every_X_epoch == 0:
            self._verbose_print("Calculating mAP...")
            self._load_weights_for_model()

            mAPs = self._calculate_mean_average_precision()
            mAP = np.mean(mAPs)

            if logs is not None:
                logs["val_mean_average_precision"] = mAP

            self._verbose_print("mAP at epoch {0} is: {1}".format(epoch+1, mAP))

        super().on_epoch_end(epoch, logs)

    def _load_weights_for_model(self):
        last_weights_path = self.train_model.find_last()
        self._verbose_print("Loaded weights for the inference model (last checkpoint of the train model): {0}".format(
            last_weights_path))
        self.inference_model.load_weights(last_weights_path,
                                          by_name=True)

    def _calculate_mean_average_precision(self):
        mAPs = []

        # Use a random subset of the data when a limit is defined
        np.random.shuffle(self.dataset_image_ids)

        for image_id in self.dataset_image_ids[:self.dataset_limit]:
            image, image_meta, gt_class_id, gt_bbox, gt_mask = load_image_gt(self.dataset, self.inference_model.config,
                                                                             image_id, use_mini_mask=False)
            molded_images = np.expand_dims(mold_image(image, self.inference_model.config), 0)
            results = self.inference_model.detect(molded_images, verbose=0)
            r = results[0]
            # Compute mAP - VOC uses IoU 0.5
            AP, _, _, _ = utils.compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"],
                                           r["class_ids"], r["scores"], r['masks'])
            mAPs.append(AP)

        return np.array(mAPs)

Many thanks for this! I was able to get this working straightaway; the only change I made was to calculate the mAP at the end of every epoch for my purposes.

Just a word of caution. I don't know if it's just me, but when viewing the plot on Tensorboard (which considers the epoch numbers to be 0-based as indicated by the steps on the X axes), the plot for val_mean_average_precision is shifted to the right by 1. That is, assuming the mAP is calculated at the end of every epoch, the mAP from the 0th epoch is shown at Step 1 of the plot, the mAP from the 1st epoch is at Step 2, and so on. This causes the mAP from the final epoch to be absent from the plot (because the Step would be out of the bounds of the X axis).

It looks like this is due to the sequence in which the callbacks are called. In MaskRCNN.train() in model.py, the custom callbacks are added to the end of the callback list while the Tensorboard callback is the first one.

Therefore, when the MeanAveragePrecisionCallback gets called for the n'th epoch, Tensorboard would have already finished preparing the summary for that epoch. So, it doesn't see the mAP value from the n'th epoch until it is preparing the summary for the (n+1)'th epoch, at which point that value is erroneously considered to be part of the (n+1)'th epoch.

I changed the sequence of the callbacks, to make the Tensorboard callback the last one. That corrected the problem, and I now see the mAP values at the correct step in the plot.

# The callback to save the model checkpoint
callbacks = [keras.callbacks.ModelCheckpoint(self.checkpoint_path, verbose=0, save_weights_only=True)]

# Add custom callbacks to the list
if custom_callbacks:
  callbacks += custom_callbacks

# The tensorboard callback is last so that any metric logged by the custom callbacks would be picked up as part of the same epoch
callbacks.append(keras.callbacks.TensorBoard(log_dir=self.log_dir, histogram_freq=0, write_graph=True, write_images=False))

[nomurakeiya]

Hello @VtlNmnk thank you very much for your response. I am currently facing another problem. Training throws me this error:

Traceback (most recent call last):

File "/content/Mask_RCNN/samples/metal_blanco/metal_blanco.py", line 371, in
train(model)
File "/content/Mask_RCNN/samples/metal_blanco/metal_blanco.py", line 195, in train
inference_model=model_inference, dataset=dataset_val, calculate_map_at_every_X_epoch=5, verbose=1)
File "/usr/local/lib/python3.6/dist-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py", line 2893, in init
ValueError: This callback only works with the bacth size of 1
For this reason configure my config.py like this:

class Config(object):
"""Base configuration class. For custom configurations, create a
sub-class that inherits from this one and override properties
that need to be changed.
"""

Name the configurations. For example, 'COCO', 'Experiment 3', ...etc.

Useful if your code needs to do things differently depending on which

experiment is running.

NAME = None # Override in sub-classes

# NUMBER OF GPUs to use. When using only a CPU, this needs to be set to 1.
GPU_COUNT = 1

# Number of images to train with on each GPU. A 12GB GPU can typically
# handle 2 images of 1024x1024px.
# Adjust based on your GPU memory and image sizes. Use the highest
# number that your GPU can handle for best performance.
IMAGES_PER_GPU = 1

.
.
.
def init(self):
"""Set values of computed attributes."""

Effective batch size

self.BATCH_SIZE = self.IMAGES_PER_GPU * self.GPU_COUNT

This should be the bacth size = 1 but when I print it on the screen it continues with the value 2

And if you freeze the value of bacth size at 1 like this

def init(self):
"""Set values of computed attributes."""

Effective batch size

self.BATCH_SIZE = 1

Print the error:

ValueError: slice index 1 of dimension 0 out of bounds. for 'ROI/strided_slice_12' (op: 'StridedSlice') with input shapes: [1,261888,4], [1], [1], [1] and with computed input tensors: input[1] = <1>, input[2] = <2>, input[3] = <1>.

If I ignore the line that check the BATCH_SIZE

if inference_model.config.BATCH_SIZE! = 1:

The system starts training but throws me another error when calculating the mAP

thank you very much in advance a greeting

Hello @VtlNmnk,
I still have this same error "This callback only works with the bacth size of 1".
Now my BATCH_SIZE is '2'
How can I fix it?

Below this what I did.

1. I added code to coco.py
   

2. I added code to model.py
   

[VtlNmnk]

Hello @VtlNmnk thank you very much for your response. I am currently facing another problem. Training throws me this error:

Traceback (most recent call last):

File "/content/Mask_RCNN/samples/metal_blanco/metal_blanco.py", line 371, in
train(model)
File "/content/Mask_RCNN/samples/metal_blanco/metal_blanco.py", line 195, in train
inference_model=model_inference, dataset=dataset_val, calculate_map_at_every_X_epoch=5, verbose=1)
File "/usr/local/lib/python3.6/dist-packages/mask_rcnn-2.1-py3.6.egg/mrcnn/model.py", line 2893, in init
ValueError: This callback only works with the bacth size of 1

Hey! You need to use 2 configurations, created on the basis of the base one, which can be imported like this
from mrcnn.config import Config

And for the inference mode, a separate configuration must be created.

You use the same configuration for both training and inference modes.

[nomurakeiya]

thanks for reply!

After changing class CocoConfig(Config), BATCH_SIZE changed to "1"
thanks for good explain. I overlooked the code here.

coco.py

result

error

but an error has occurred.
It works correctly when IMAGE_PER_GPU is returned to "2".

[VtlNmnk]

It works correctly when IMAGE_PER_GPU is returned to "2".

I wrote to you, and wrote about this earlier: use a separate instance of the configuration with BATCH_SIZE = 1 for Inference and another, separate instance, with a different name, for training. And of course, it will have BATCH_SIZE = 2 or how much your GPU can handle.
Look at https://github.com/matterport/Mask_RCNN/blob/master/samples/balloon/balloon.py

[nomurakeiya]

I looked misplace! thanks
Configuration is already separated for 'train' and 'inference'.

I'm confused.
When 'train' how much GPU_COUNT I should use?
I think should use IMAGE_PER_GPU = 1 because callback only works with BATCH_SIZE=1.
But now, I'm using IMAGE_PER_GPU = 2.
It configured from line 441 → line 71~87.
Should I change line 81 to 'IMAGE_PER_GPU = 1' ?

When 'inference' how much IMAGE_PER_GPU I should use?
now I'm using IMAGE_PER_GPU=1.
It configured from line 443 ~ 448.

Thank you very much for your help.

[zaiternouara]

Hello guys, thanks for sharing the code to have accuracy metrics. I try to implement the code but I have errors :(
I pasted the class "MeanAveragePrecisionCallbacks" at the end of the model.py code.

Then, in train.py I defined the variable like you did @VtlNmnk:
mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, model_inference, dataset_val, calculate_map_at_every_X_epoch=5, verbose=1)

But when I launch my training, I have this error:

Traceback (most recent call last): File "Taraudage.py", line 372, in <module> train(model) File "Taraudage.py", line 196, in train mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(train_model=model, inference_model=model_inference, NameError: name 'model_inference' is not defined

What variable do you expect for model inference ?

Thanks for you time,
Regards,
Antoine

I have the same issue , how did you solve it please !

[VtlNmnk]

Hello guys, thanks for sharing the code to have accuracy metrics. I try to implement the code but I have errors :(
I pasted the class "MeanAveragePrecisionCallbacks" at the end of the model.py code.

Then, in train.py I defined the variable like you did @VtlNmnk:
mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, model_inference, dataset_val, calculate_map_at_every_X_epoch=5, verbose=1)

But when I launch my training, I have this error:
Traceback (most recent call last): File "Taraudage.py", line 372, in <module> train(model) File "Taraudage.py", line 196, in train mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(train_model=model, inference_model=model_inference, NameError: name 'model_inference' is not defined
What variable do you expect for model inference ?
Thanks for you time,
Regards,
Antoine

I have the same issue , how did you solve it please !

model_inference = modellib.MaskRCNN(mode="inference", config=_InfConfig(), model_dir=MODEL_DIR)

[zaiternouara]

Hello guys, thanks for sharing the code to have accuracy metrics. I try to implement the code but I have errors :(
I pasted the class "MeanAveragePrecisionCallbacks" at the end of the model.py code.

Then, in train.py I defined the variable like you did @VtlNmnk:
mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, model_inference, dataset_val, calculate_map_at_every_X_epoch=5, verbose=1)

But when I launch my training, I have this error:
Traceback (most recent call last): File "Taraudage.py", line 372, in <module> train(model) File "Taraudage.py", line 196, in train mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(train_model=model, inference_model=model_inference, NameError: name 'model_inference' is not defined
What variable do you expect for model inference ?
Thanks for you time,
Regards,
Antoine

I have the same issue , how did you solve it please !

model_inference = modellib.MaskRCNN(mode="inference", config=_InfConfig(), model_dir=MODEL_DIR)

Thank you ,i tried this :

def train(dataset_train, dataset_val, model):
    """Train the model."""
    # Training dataset.
    dataset_train.prepare()
 
    # Validation dataset
    dataset_val.prepare()
    model_inference = modellib.MaskRCNN(mode="inference", config=InferenceConfig(), model_dir=DEFAULT_LOGS_DIR)
    
    mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model,model_inference, dataset_val,calculate_map_at_every_X_epoch=1, verbose=1)
        
    # *** This training schedule is an example. Update to your needs ***
    print("Training network heads")
    model.train(dataset_train, dataset_val,
                learning_rate=config.LEARNING_RATE,
                epochs=300,
                layers='heads',
                custom_callbacks=[mean_average_precision_callback])

But still the same error

[zaiternouara]

Hello guys, thanks for sharing the code to have accuracy metrics. I try to implement the code but I have errors :(
I pasted the class "MeanAveragePrecisionCallbacks" at the end of the model.py code.

Then, in train.py I defined the variable like you did @VtlNmnk:
mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model, model_inference, dataset_val, calculate_map_at_every_X_epoch=5, verbose=1)

But when I launch my training, I have this error:
Traceback (most recent call last): File "Taraudage.py", line 372, in <module> train(model) File "Taraudage.py", line 196, in train mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(train_model=model, inference_model=model_inference, NameError: name 'model_inference' is not defined
What variable do you expect for model inference ?
Thanks for you time,
Regards,
Antoine

I have the same issue , how did you solve it please !

model_inference = modellib.MaskRCNN(mode="inference", config=_InfConfig(), model_dir=MODEL_DIR)

This is my costumTrain :

The Script is modified to train on custom Labelme Data.

import os
import sys
import json
import wget
import datetime
import numpy as np
import skimage.draw

Root directory of the project

ROOT_DIR = os.path.dirname(os.path.abspath(os.path.abspath(file)))
print(ROOT_DIR)
from mrcnn.utils import compute_ap
from mrcnn.model import load_image_gt
from mrcnn.model import mold_image
from mrcnn.config import Config
from mrcnn import model as modellib
from mrcnn import utils
from mrcnn import visualize
from numpy import expand_dims
from numpy import mean

In case of configuration

git clone the repo and append the path Mask_RCNN repo to use the custom configurations

git clone https://github.com/matterport/Mask_RCNN

sys.path.append("path_to_mask_rcnn")

maskrrr=sys.path.append("/content/drive/MyDrive/business_card_extract_information/buisness_card_extract_information/maskrcnn/Mask-RCNN-Implementation")

Download Link

https://github.com/matterport/Mask_RCNN/releases/download/v2.0/mask_rcnn_coco.h5

if not os.path.isfile("mask_rcnn_coco.h5"):
url = "https://github.com/ayoolaolafenwa/PixelLib/releases/download/1.2/mask_rcnn_coco.h5"
print("Downloading the mask_rcnn_coco.h5")
filename = wget.download(url)
print(f"[INFO]: Download complete >> {filename}")

Path to trained weights file

COCO_WEIGHTS_PATH = os.path.join(ROOT_DIR, "mask_rcnn_coco.h5")

Directory to save logs and model checkpoints, if not provided

through the command line argument --logs

DEFAULT_LOGS_DIR = os.path.join(ROOT_DIR, "logs")

Change it for your dataset's name

source="Dataset2021"
############################################################

My Model Configurations (which you should change for your own task)

############################################################

class ModelConfig(Config):
"""Configuration for training on the toy dataset.
Derives from the base Config class and overrides some values.
"""
# Give the configuration a recognizable name
NAME = "ImageSegmentationMaskRCNN"

# We use a GPU with 12GB memory, which can fit two images.
# Adjust down if you use a smaller GPU.
IMAGES_PER_GPU = 2 # 1

# Number of classes (including background)
NUM_CLASSES = 1 + 1 # Background,
# typically after labeled, class can be set from Dataset class
# if you want to test your model, better set it corectly based on your trainning dataset

# Number of training steps per epoch
STEPS_PER_EPOCH = 100

# Skip detections with < 90% confidence
DETECTION_MIN_CONFIDENCE = 0.9

class InferenceConfig(ModelConfig):
# Set batch size to 1 since we'll be running inference on
# one image at a time. Batch size = GPU_COUNT * IMAGES_PER_GPU
GPU_COUNT = 1
IMAGES_PER_GPU = 1

############################################################

Dataset (My labelme dataset loader)

############################################################

class LabelmeDataset(utils.Dataset):
# Load annotations
# Labelme Image Annotator v = 3.16.7
# different version may have different structures
# besides, labelme's annotation setup one file for
# one picture not for all pictures
# and this annotations are all dicts after Python json load
# {
# "version": "3.16.7",
# "flags": {},
# "shapes": [
# {
# "label": "balloon",
# "line_color": null,
# "fill_color": null,
# "points": [[428.41666666666674, 875.3333333333334 ], ...],
# "shape_type": "polygon",
# "flags": {}
# },
# {
# "label": "balloon",
# "line_color": null,
# "fill_color": null,
# "points": [... ],
# "shape_type": "polygon",
# "flags": {}
# },
# ],
# "lineColor": [(4 number)],
# "fillColor": [(4 number)],
# "imagePath": "10464445726_6f1e3bbe6a_k.jpg",
# "imageData": null,
# "imageHeight": 2019,
# "imageWidth": 2048
# }
# We mostly care about the x and y coordinates of each region
def load_labelme(self, dataset_dir, subset):
"""
Load a subset of the dataset.
source: coustomed source id, exp: load data from coco, than set it "coco",
it is useful when you ues different dataset for one trainning.(TODO)
see the prepare func in utils model for details
dataset_dir: Root directory of the dataset.
subset: Subset to load: train or val
"""
assert subset in ["train", "val"]
dataset_dir = os.path.join(dataset_dir, subset)
# Train or validation dataset
filenames = os.listdir(dataset_dir)
jsonfiles,annotations=[],[]
for filename in filenames:
if filename.endswith(".json"):
jsonfiles.append(filename)
annotation = json.load(open(os.path.join(dataset_dir,filename)))
# Insure this picture is in this dataset
imagename = annotation['imagePath']
if not os.path.isfile(os.path.join(dataset_dir,imagename)):
continue
if len(annotation["shapes"]) == 0:
continue
# you can filter what you don't want to load
annotations.append(annotation)

    print("In {source} {subset} dataset we have {number:d} annotation files."
        .format(source=source, subset=subset,number=len(jsonfiles)))
    print("In {source} {subset} dataset we have {number:d} valid annotations."
        .format(source=source, subset=subset,number=len(annotations)))

    # Add images and get all classes in annotation files
    # typically, after labelme's annotation, all same class item have a same name
    # this need us to annotate like all "ball" in picture named "ball"
    # not "ball_1" "ball_2" ...
    # we also can figure out which "ball" it is refer to.
    labelslist = []
    for annotation in annotations:
        # Get the x, y coordinaets of points of the polygons that make up
        # the outline of each object instance. These are stores in the
        # shape_attributes (see json format above)
        shapes = [] 
        classids = []

        for shape in annotation["shapes"]:
            # first we get the shape classid
            label = shape["label"]
            if labelslist.count(label) == 0:
                labelslist.append(label)
            classids.append(labelslist.index(label)+1)
            shapes.append(shape["points"])
        
        # load_mask() needs the image size to convert polygons to masks.
        width = annotation["imageWidth"]
        height = annotation["imageHeight"]
        self.add_image(
            source,
            image_id=annotation["imagePath"],  # use file name as a unique image id
            path=os.path.join(dataset_dir,annotation["imagePath"]),
            width=width, height=height,
            shapes=shapes, classids=classids)

    print("In {source} {subset} dataset we have {number:d} class item"
        .format(source=source, subset=subset,number=len(labelslist)))

    for labelid, labelname in enumerate(labelslist):
        self.add_class(source,labelid,labelname)

def load_mask(self,image_id):
    """
    Generate instance masks for an image.
   Returns:
    masks: A bool array of shape [height, width, instance count] with one mask per instance.
    class_ids: a 1D array of class IDs of the instance masks.
    """
    # If not the source dataset you want, delegate to parent class.
    image_info = self.image_info[image_id]
    if image_info["source"] != source:
        return super(self.__class__, self).load_mask(image_id)

    # Convert shapes to a bitmap mask of shape
    # [height, width, instance_count]
    info = self.image_info[image_id]
    mask = np.zeros([info["height"], info["width"], len(info["shapes"])], dtype=np.uint8)
    #printsx,printsy=zip(*points)
    for idx, points in enumerate(info["shapes"]):
        # Get indexes of pixels inside the polygon and set them to 1
        pointsy,pointsx = zip(*points)
        rr, cc = skimage.draw.polygon(pointsx, pointsy)
        mask[rr, cc, idx] = 1
    masks_np = mask.astype(np.bool)
    classids_np = np.array(image_info["classids"]).astype(np.int32)
    # Return mask, and array of class IDs of each instance. Since we have
    # one class ID only, we return an array of 1s
    return masks_np, classids_np

def image_reference(self,image_id):
    """Return the path of the image."""
    info = self.image_info[image_id]
    if info["source"] == source:
        return info["path"]
    else:
        super(self.__class__, self).image_reference(image_id)

def train(dataset_train, dataset_val, model):
"""Train the model."""
# Training dataset.
dataset_train.prepare()

# Validation dataset
dataset_val.prepare()
model_inference = modellib.MaskRCNN(mode="inference", config=InferenceConfig(), model_dir=DEFAULT_LOGS_DIR)

mean_average_precision_callback = modellib.MeanAveragePrecisionCallback(model,model_inference, dataset_val,calculate_map_at_every_X_epoch=1, verbose=1)
    
# *** This training schedule is an example. Update to your needs ***
print("Training network heads")
model.train(dataset_train, dataset_val,
            learning_rate=config.LEARNING_RATE,
            epochs=300,
            layers='heads',
            custom_callbacks=[mean_average_precision_callback])

def test(model, image_path = None, video_path=None, savedfile=None):
assert image_path or video_path

 # Image or video?
if image_path:
    # Run model detection and generate the color splash effect
    print("Running on {}".format(args.image))
    # Read image
    image = skimage.io.imread(args.image)
    # Detect objects
    r = model.detect([image], verbose=1)[0]
    # Colorful
    import matplotlib.pyplot as plt
    
    _, ax = plt.subplots()
    visualize.get_display_instances_pic(image, boxes=r['rois'], masks=r['masks'], 
        class_ids = r['class_ids'], class_number=model.config.NUM_CLASSES,ax = ax,
        class_names=None,scores=None, show_mask=True, show_bbox=True)
    # Save output
    if savedfile == None:
        file_name = "test_{:%Y%m%dT%H%M%S}.png".format(datetime.datetime.now())
    else:
        file_name = savedfile
    plt.savefig(file_name)
    #skimage.io.imsave(file_name, testresult)
elif video_path:
    pass
print("Saved to ", file_name)

############################################################

Training and Validating

############################################################

if name == 'main':
import argparse

# Parse command line arguments
parser = argparse.ArgumentParser(description='Train Mask R-CNN to detect balloons.')
parser.add_argument("command",
                    metavar="<command>",
                    help="'train' or 'test'")
parser.add_argument('--dataset', required=False,
                    metavar="/path/to/dataset/",
                    help='Directory of your dataset')
parser.add_argument('--weights', required=True,
                    metavar="/path/to/weights.h5",
                    help="Path to weights .h5 file or 'coco', 'last' or 'imagenet'")
parser.add_argument('--logs', required=False,
                    default=DEFAULT_LOGS_DIR,
                    metavar="/path/to/logs/",
                    help='Logs and checkpoints directory (default=./logs/)')
parser.add_argument('--image', required=False,
                    metavar="path or URL to image",
                    help='Image to test and color splash effect on')
parser.add_argument('--video', required=False,
                    metavar="path or URL to video",
                    help='Video to test and color splash effect on')
parser.add_argument('--classnum', required=False,
                    metavar="class number of your detect model",
                    help="Class number of your detector.")
args = parser.parse_args()

# Validate arguments
if args.command == "train":
    assert args.dataset, "Argument --dataset is required for training"
elif args.command == "test":
    assert args.image or args.video or args.classnum, \
        "Provide --image or --video and  --classnum of your model to apply testing"


print("Weights: ", args.weights)
print("Dataset: ", args.dataset)
print("Logs: ", args.logs)

# Configurations
if args.command == "train":
    config = ModelConfig()
    dataset_train, dataset_val = LabelmeDataset(), LabelmeDataset()
    dataset_train.load_labelme(args.dataset,"train")
    dataset_val.load_labelme(args.dataset,"val")
    config.NUM_CLASSES = len(dataset_train.class_info)
elif args.command == "test":
    config = InferenceConfig()
    config.NUM_CLASSES = int(args.classnum)+1 # add backgrouond
 


config.display()

# Create model
if args.command == "train":
    model = modellib.MaskRCNN(mode="training", config=config,model_dir=args.logs)
else:
    model = modellib.MaskRCNN(mode="inference", config=config, model_dir=args.logs)

# Select weights file to load
if args.weights.lower() == "coco":
    weights_path = COCO_WEIGHTS_PATH
elif args.weights.lower() == "last":
    # Find last trained weights
    weights_path = model.find_last()
else:
    weights_path = args.weights

# Load weights
print("Loading weights ", weights_path)
if args.command == "train":
    if args.weights.lower() == "coco":
        # Exclude the last layers because they require a matching
        # number of classes if we change the backbone?
        model.load_weights(weights_path, by_name=True, exclude=[
            "mrcnn_class_logits", "mrcnn_bbox_fc",
            "mrcnn_bbox", "mrcnn_mask"])
    else:
        model.load_weights(weights_path, by_name=True)
    # Train or evaluate
    train(dataset_train, dataset_val, model)
elif args.command == "test":
    # we test all models trained on the dataset in different stage
    print(os.getcwd())
    filenames = os.listdir(args.weights)
    for filename in filenames:
        if filename.endswith(".h5"):
            print(f"Load weights from {filename}")
            model.load_weights(os.path.join(args.weights,filename),by_name=True)
            savedfile_name = os.path.splitext(filename)[0] + ".jpg"
            test(model, image_path=args.image,video_path=args.video, savedfile=savedfile_name)
else:
    print("'{}' is not recognized.Use 'train' or 'test'".format(args.command))    

[kecyarchana]

I incorporated mean average precision as a callback (as mentioned above) for my validation dataset during training in every 5 epochs but I am getting nan values and a runtime warning as below.

I cannot figure out what might be the cause as my validation dataset has only the images with the objects in it and ground truth files as shapefiles . Any help would be greatly appreciated.

Thanks

[vineet131]

hi @mhtarora39

I am trying to use your code to calculate map, precision, and recall but get the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-23-a2f082d763b6> in <module>
     64 config = InferenceConfig()
     65 evel = EvalImage(dataset_vall,modelinf,config)
---> 66 evel.evaluate_model()
     67 

<ipython-input-23-a2f082d763b6> in evaluate_model(self, len)
     51       r = yhat[0]
     52       # calculate statistics, including AP
---> 53       AP, precisions, recalls, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
     54       precisions_dict[image_id] = np.mean(precisions)
     55       recall_dict[image_id] = np.mean(recalls)

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_ap(gt_boxes, gt_class_ids, gt_masks, pred_boxes, pred_class_ids, pred_scores, pred_masks, iou_threshold)
    728         gt_boxes, gt_class_ids, gt_masks,
    729         pred_boxes, pred_class_ids, pred_scores, pred_masks,
--> 730         iou_threshold)
    731 
    732     # Compute precision and recall at each prediction box step

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_matches(gt_boxes, gt_class_ids, gt_masks, pred_boxes, pred_class_ids, pred_scores, pred_masks, iou_threshold, score_threshold)
    680 
    681     # Compute IoU overlaps [pred_masks, gt_masks]
--> 682     overlaps = compute_overlaps_masks(pred_masks, gt_masks)
    683 
    684     # Loop through predictions and find matching ground truth boxes

D:/Desktop/aktwelve_mask_rcnn\mrcnn\utils.py in compute_overlaps_masks(masks1, masks2)
    113 
    114     # intersections and union
--> 115     intersections = np.dot(masks1.T, masks2)
    116     union = area1[:, None] + area2[None, :] - intersections
    117     overlaps = intersections / union

<__array_function__ internals> in dot(*args, **kwargs)

ValueError: shapes (5,1048576) and (3136,6) not aligned: 1048576 (dim 1) != 3136 (dim 0)

This is my implementation:

from mrcnn.utils import compute_ap

class EvalImage():
  def __init__(self,dataset,model,cfg):
    self.dataset = dataset
    self.model   = model
    self.cfg     = cfg


  def evaluate_model(self , len = 50):
    APs = list()
    precisions_dict = {}
    recall_dict     = {}
    for index,image_id in enumerate(self.dataset.image_ids):
      if(index > len):
         break; 
      # load image, bounding boxes and masks for the image id
      image, image_meta, gt_class_id, gt_bbox, gt_mask = modellib.load_image_gt(self.dataset, self.cfg,image_id)
      # convert pixel values (e.g. center)
      #scaled_image = modellib.mold_image(image, self.cfg)
      # convert image into one sample
      sample = np.expand_dims(image, 0)
    
     # print(len(image))
      # make prediction
      yhat = self.model.detect(sample, verbose=1)
      # extract results for first sample
      r = yhat[0]
      # calculate statistics, including AP
      AP, precisions, recalls, _ = compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
      precisions_dict[image_id] = np.mean(precisions)
      recall_dict[image_id] = np.mean(recalls)
      # store
      APs.append(AP)

    # calculate the mean AP across all images
    mAP = np.mean(APs)
    return mAP,precisions_dict,recall_dict


dataset_vall = CocoLikeDataset()
dataset_vall.load_data('D:/Desktop/aktwelve_mask_rcnn/datasets/Acacia dataset', 'val')
dataset_vall.prepare()

config = InferenceConfig()

modelinf = modellib.MaskRCNN(mode="inference", 
                          config=config,
                          model_dir=MODEL_DIR)

modelinf.load_weights(os.path.join(ROOT_DIR, "mask_rcnn_acacia-stp73-epch150_0075.h5"), by_name=True)

evel = EvalImage(dataset_vall,modelinf,config)
evel.evaluate_model()

Any suggestions would be appreciated

Hey I have the exact same issue. Did you find a solution to this error??

@sohinimallick @tomjordan and anyone else having this issue:

I just had this same issue today, and was stuck on it for a while. Turns out, something was overriding the default value of False passed to use_mini_mask at this line (in my case, the code had been changed to fetch it from elsewhere):

image, image_meta, gt_class_id, gt_bbox, gt_mask = load_image_gt(self.dataset, self.inference_model.config, image_id, use_mini_mask=False)

So just pass False and it should be fixed. Although now I'm getting an error that my mAP is NaN, which I need to check.

[filmonmehari]

@VtlNmnk That's awsome idea but can u help me to plot the accuracy and losses seen on the output arguments. I have have a topic to submit for such parameters after 3 weeks.

thanks in advance!

[VtlNmnk]

@VtlNmnk That's awsome idea but can u help me to plot the accuracy and losses seen on the output arguments. I have have a topic to submit for such parameters after 3 weeks.

thanks in advance!
Hi, @filmonmehari! Why don't you use the tensorboard?
#2714

[WangariKimotho]

Hello @VtlNmnk and all, I tried implementing the code, made the necesary changes for my workflow but I still cannot seem to get the mask and rpn class losses etc

Also I get the feeling that part of the reason lies in that the compile function predefines the loss var as None on lines 2188-2190.... correct me if I'm wrong. I appreciate any insight moving forward