ValueError: Tried to convert 't' to a tensor and failed.

[harshilpatel312]

System information

• What is the top-level directory of the model you are using: models/research/object-detection
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04.4 LTS
• TensorFlow installed from (source or binary): pip install tensorflow-gpu
• TensorFlow version (use command below): 1.8.0
• GPU model and memory: GeForce GTX 980 Ti
• CUDA/cuDNN version: N/A
• Bazel version: N/A
• Exact command to reproduce: python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_resnet101_coco.config --num_clones=2 --ps_tasks=1

Describe the problem

Not able to train the model. It gives a 'ValueError' as shown below.

Things I tried:

1. I found out a solution from #3705 which doesn't work for me. Ran python setup.py build and python setup.py install from the 'models/research/' directory too. Still doesn't work.

2. I commented out Line 169 (+added random print statements), built and installed setup.py, and ran the 'train.py' command again, and it still shows the traceback as show below (notice that it still calls Line 169). I think the problem is it doesn't apply the changes I make to the 'utils/learning_schedules.py' file. Can someone tell me how to apply these changes or if possible, the solution to this entire problem?

Traceback

Traceback (most recent call last):
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 510, in _apply_op_helper
    preferred_dtype=default_dtype)
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1104, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 235, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 214, in constant
    value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py", line 441, in make_tensor_proto
    _GetDenseDimensions(values)))
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 524, in _apply_op_helper
    values, as_ref=input_arg.is_ref).dtype.name
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1104, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 235, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 214, in constant
    value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py", line 441, in make_tensor_proto
    _GetDenseDimensions(values)))
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 184, in <module>
    tf.app.run()
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 126, in run
    _sys.exit(main(argv))
  File "train.py", line 180, in main
    graph_hook_fn=graph_rewriter_fn)
  File "/home/smr/tensorflow/models/research/object_detection/trainer.py", line 284, in train
    train_config.optimizer)
  File "/home/smr/tensorflow/models/research/object_detection/builders/optimizer_builder.py", line 50, in build
    learning_rate = _create_learning_rate(config.learning_rate)
  File "/home/smr/tensorflow/models/research/object_detection/builders/optimizer_builder.py", line 109, in _create_learning_rate
    learning_rate_sequence, config.warmup)
  File "/home/smr/tensorflow/models/research/object_detection/utils/learning_schedules.py", line 169, in manual_stepping
    [0] * num_boundaries))
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 2681, in where
    return gen_math_ops.select(condition=condition, x=x, y=y, name=name)
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 6699, in select
    "Select", condition=condition, t=x, e=y, name=name)
  File "/home/smr/tf/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 528, in _apply_op_helper
    (input_name, err))
ValueError: Tried to convert 't' to a tensor and failed. Error: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
Bazel version
CUDA/cuDNN version

[harshilpatel312]

Here are the changes (updated the post too):
Have I written custom code: No
CUDA/cuDNN version: N/A
Bazel version: N/A

[austinmw]

What's your batch_size?

[harshilpatel312]

I tried multiple batch sizes [2, 4, 8, 16] but I usually keep it at 2. Didn't work for either of them.

[austinmw]

No idea if your model or data is the problem but you could try validating your TFRecord file to eliminate that possibility: https://gist.github.com/ed-alertedh/9f49bfc6216585f520c7c7723d20d951

[Alexchandriyaa]

same error for me also

[coenvalk]

Getting the same error here. I was able to replicate the error while using the pets_examples.record, so the issue is probably not with the record file.

[tensorflowbutler]

Hi There,
We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.

[kumariko]

@harshilpatel312 we are checking to see if you still need help on this issue ? Could you please have a look on the link1 , link2 and let us know if it helps? Thanks!

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No