-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NotImplementedError: While importing/Loading tfds plant_leaves dataset #5416
Comments
Hi, thank you for reporting! Workaround: add the following arg to your tfds.load(..., download_and_prepare_kwargs={'file_format': tfds.core.FileFormat.ARRAY_RECORD}) We'll look on how to update the code and update on the bug. |
It's still giving error. import tensorflow_datasets as `tfds`
plant_leaves_data, plant_leaves_info = tfds.load('plant_leaves', split='train', shuffle_files=True, download_and_prepare_kwargs={'file_format': tfds.core.FileFormat.ARRAY_RECORD}) Gives Downloading and preparing dataset 6.56 GiB (download: 6.56 GiB, generated: 6.81 GiB, total: 13.37 GiB) to /root/tensorflow_datasets/plant_leaves/0.1.1...
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
[<ipython-input-3-608b46b22c6c>](https://localhost:8080/#) in <cell line: 4>()
2 #plant_leaves = tfds.load('plant_leaves', split='train', shuffle_files=True)
3 #plant_leaves_data, plant_leaves_info = tfds.load('plant_leaves', split='train', shuffle_files=True, as_data_source=True)
----> 4 plant_leaves_data, plant_leaves_info = tfds.load('plant_leaves', split='train', shuffle_files=True, download_and_prepare_kwargs={'file_format': tfds.core.FileFormat.ARRAY_RECORD})
5 frames
[/usr/local/lib/python3.10/dist-packages/tensorflow_datasets/core/logging/__init__.py](https://localhost:8080/#) in __call__(self, function, instance, args, kwargs)
167 metadata = self._start_call()
168 try:
--> 169 return function(*args, **kwargs)
170 except Exception:
171 metadata.mark_error()
[/usr/local/lib/python3.10/dist-packages/tensorflow_datasets/core/load.py](https://localhost:8080/#) in load(name, split, data_dir, batch_size, shuffle_files, download, as_supervised, decoders, read_config, with_info, builder_kwargs, download_and_prepare_kwargs, as_dataset_kwargs, try_gcs)
645 try_gcs,
646 )
--> 647 _download_and_prepare_builder(dbuilder, download, download_and_prepare_kwargs)
648
649 if as_dataset_kwargs is None:
[/usr/local/lib/python3.10/dist-packages/tensorflow_datasets/core/load.py](https://localhost:8080/#) in _download_and_prepare_builder(dbuilder, download, download_and_prepare_kwargs)
504 if download:
505 download_and_prepare_kwargs = download_and_prepare_kwargs or {}
--> 506 dbuilder.download_and_prepare(**download_and_prepare_kwargs)
507
508
[/usr/local/lib/python3.10/dist-packages/tensorflow_datasets/core/logging/__init__.py](https://localhost:8080/#) in __call__(self, function, instance, args, kwargs)
167 metadata = self._start_call()
168 try:
--> 169 return function(*args, **kwargs)
170 except Exception:
171 metadata.mark_error()
[/usr/local/lib/python3.10/dist-packages/tensorflow_datasets/core/dataset_builder.py](https://localhost:8080/#) in download_and_prepare(self, download_dir, download_config, file_format)
679 # to generate the files.
680 if file_format:
--> 681 self.info.set_file_format(file_format, override=True)
682
683 # Create a tmp dir and rename to self.data_dir on successful exit.
[/usr/local/lib/python3.10/dist-packages/tensorflow_datasets/core/dataset_info.py](https://localhost:8080/#) in set_file_format(self, file_format, override)
470 )
471 if override and self._fully_initialized:
--> 472 raise RuntimeError(
473 "Cannot override the file format "
474 "when the DatasetInfo is already fully initialized!"
RuntimeError: Cannot override the file format when the DatasetInfo is already fully initialized! |
Same errors on refcoco dataset. |
Anyway, one thing I do to solve this is add the following line:
|
builder = tfds.builder('oxford_iiit_pet') dataset, info = tfds.load('oxford_iiit_pet:4.0.0', download_and_prepare_kwargs={'file_format': tfds.core.FileFormat.ARRAY_RECORD}) also erroers: `NotImplementedError Traceback (most recent call last) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/logging/init.py:176, in _FunctionDecorator.call(self, function, instance, args, kwargs) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/load.py:673, in load(name, split, data_dir, batch_size, shuffle_files, download, as_supervised, decoders, read_config, with_info, builder_kwargs, download_and_prepare_kwargs, as_dataset_kwargs, try_gcs) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/logging/init.py:176, in _FunctionDecorator.call(self, function, instance, args, kwargs) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/dataset_builder.py:1026, in DatasetBuilder.as_dataset(self, split, batch_size, shuffle_files, decoders, read_config, as_supervised) File /usr/local/lib/python3.12/dist-packages/tree/init.py:428, in map_structure(func, *structures, **kwargs) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/dataset_builder.py:1044, in DatasetBuilder._build_single_dataset(self, split, batch_size, shuffle_files, decoders, read_config, as_supervised) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/dataset_builder.py:1498, in FileReaderBuilder._as_dataset(self, split, decoders, read_config, shuffle_files) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/reader.py:430, in Reader.read(self, instructions, split_infos, read_config, shuffle_files, disable_shuffling, decode_fn) File /usr/local/lib/python3.12/dist-packages/tree/init.py:428, in map_structure(func, *structures, **kwargs) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/reader.py:422, in Reader.read.._read_instruction_to_ds(instruction) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/reader.py:462, in Reader.read_files(self, file_instructions, read_config, shuffle_files, disable_shuffling, decode_fn) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/reader.py:302, in _read_files(file_instructions, read_config, shuffle_files, disable_shuffling, file_format) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/data/ops/dataset_ops.py:2534, in DatasetV2.interleave(self, map_func, cycle_length, block_length, num_parallel_calls, deterministic, name) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/data/ops/interleave_op.py:49, in _interleave(input_dataset, map_func, cycle_length, block_length, num_parallel_calls, deterministic, name) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/data/ops/interleave_op.py:119, in _ParallelInterleaveDataset.init(self, input_dataset, map_func, cycle_length, block_length, num_parallel_calls, buffer_output_elements, prefetch_input_elements, deterministic, name) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/data/ops/structured_function.py:265, in StructuredFunctionWrapper.init(self, func, transformation_name, dataset, input_classes, input_shapes, input_types, input_structure, add_to_graph, use_legacy_function, defun_kwargs) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:1251, in Function.get_concrete_function(self, *args, **kwargs) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:1221, in Function._get_concrete_function_garbage_collected(self, *args, **kwargs) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:696, in Function._initialize(self, args, kwds, add_initializers_to) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py:178, in trace_function(args, kwargs, tracing_options) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py:283, in _maybe_define_function(args, kwargs, tracing_options) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py:310, in _create_concrete_function(function_type, type_context, func_graph, tracing_options) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/framework/func_graph.py:1059, in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, create_placeholders) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:599, in Function._generate_scoped_tracing_options..wrapped_fn(*args, **kwds) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/data/ops/structured_function.py:231, in StructuredFunctionWrapper.init..trace_tf_function..wrapped_fn(*args) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/data/ops/structured_function.py:161, in StructuredFunctionWrapper.init..wrapper_helper(*args) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/autograph/impl/api.py:690, in convert..decorator..wrapper(*args, **kwargs) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/autograph/impl/api.py:352, in converted_call(f, args, kwargs, caller_fn_scope, options) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/autograph/impl/api.py:331, in converted_call(f, args, kwargs, caller_fn_scope, options) File /usr/local/lib/python3.12/dist-packages/tensorflow/python/autograph/impl/api.py:459, in _call_unconverted(f, args, kwargs, options, update_cache) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/reader.py:69, in _get_dataset_from_filename(instruction, do_skip, do_take, file_format, add_tfds_id, override_buffer_size) File /usr/local/lib/python3.12/dist-packages/tensorflow_datasets/core/file_adapters.py:267, in ArrayRecordFileAdapter.make_tf_data(cls, filename, buffer_size) NotImplementedError: |
/!\ PLEASE INCLUDE THE FULL STACKTRACE AND CODE SNIPPET
Short description
tfds plant_leaves is not getting loaded successfully. It's throwing NotImplementedError. Tried on May 16, 2024
Environment information
Operating System: Windows 11
Python version: 3.10.12
tensorflow-datasets
/tfds-nightly
version: 4.9.4tensorflow
/tf-nightly
version: version: 2.15.0Does the issue still exists with the last
tfds-nightly
package (pip install --upgrade tfds-nightly
) ? YupReproduction instructions
Gives:
Expected behavior
To load dataset successfully.
The text was updated successfully, but these errors were encountered: