Per segment chunks #8272
Per segment chunks #8272
Conversation
zhiltsov-max
requested review from
azhavoro,
bsekachev and
Marishka17
as code owners
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThis update introduces several enhancements across various components, particularly focusing on improved media handling, frame processing, and task management. Notable features include new configuration options for disabling local media storage, refined frame prefetching logic, and updated test frameworks for better reliability. Memory management issues have been addressed, ensuring smoother operations during video processing, while modular code improvements enhance overall maintainability. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Server
participant MediaCache
participant FrameProvider
User->>Server: Request Media
Server->>MediaCache: Check Cache
MediaCache-->>Server: Return Media (if available)
Server->>FrameProvider: Retrieve Frames
FrameProvider-->>Server: Return Frames
Server-->>User: Deliver Media
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Outside diff range, codebase verification and nitpick comments (1)
cvat/apps/engine/media_extractors.py (1)
Ensure all instantiations of
VideoReaderandVideoReaderWithManifesthandle the newallow_threadingparameter.The following instances do not include the
allow_threadingparameter:
cvat/apps/engine/migrations/0024_auto_20191023_1025.pycvat/apps/engine/cache.pyPlease update these instances to correctly handle the
allow_threadingparameter.Analysis chain
Line range hint
509-524:
LGTM! But verify the new parameter usage.The addition of the
allow_threadingparameter enhances flexibility in video processing.However, ensure that all usages of
VideoReadercorrectly handle the new parameter.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify all usages of `VideoReader` handle the new `allow_threading` parameter. # Test: Search for the class usage. Expect: Only occurrences with the new parameter. rg --type python -A 5 $'VideoReader'Length of output: 66
Script:
#!/bin/bash # Description: Verify all usages of `VideoReader` handle the new `allow_threading` parameter. # Test: Search for the class usage. Expect: Only occurrences with the new parameter. rg --type py -A 5 $'VideoReader'Length of output: 5816
Script:
#!/bin/bash # Description: Verify all instantiations of `VideoReader` handle the new `allow_threading` parameter. # Check for instantiations of `VideoReader` and include context lines to verify parameter usage. rg --type py 'VideoReader(' -A 3Length of output: 109
Script:
#!/bin/bash # Description: Verify all instantiations of `VideoReader` handle the new `allow_threading` parameter. # Check for instantiations of `VideoReader` and include context lines to verify parameter usage. rg --type py 'VideoReader\(' -A 3Length of output: 852
Script:
#!/bin/bash # Description: Verify all instantiations of `VideoReader` and `VideoReaderWithManifest` handle the new `allow_threading` parameter. # Check for instantiations of `VideoReader` and `VideoReaderWithManifest` and include context lines to verify parameter usage. rg --type py 'VideoReader\(' -A 3 rg --type py 'VideoReaderWithManifest\(' -A 3Length of output: 1164
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (13)
- cvat-core/src/frames.ts (10 hunks)
- cvat-data/src/ts/cvat-data.ts (4 hunks)
- cvat/apps/dataset_manager/bindings.py (4 hunks)
- cvat/apps/dataset_manager/formats/cvat.py (3 hunks)
- cvat/apps/engine/cache.py (3 hunks)
- cvat/apps/engine/frame_provider.py (1 hunks)
- cvat/apps/engine/media_extractors.py (15 hunks)
- cvat/apps/engine/models.py (2 hunks)
- cvat/apps/engine/pyproject.toml (1 hunks)
- cvat/apps/engine/task.py (8 hunks)
- cvat/apps/engine/views.py (10 hunks)
- cvat/apps/lambda_manager/views.py (2 hunks)
- dev/format_python_code.sh (1 hunks)
Files skipped from review due to trivial changes (2)
- cvat/apps/engine/pyproject.toml
- dev/format_python_code.sh
Additional comments not posted (69)
cvat-data/src/ts/cvat-data.ts (7)
103-103: Initialize the newstartFrameparameter.The
startFrameparameter is correctly added to the class properties.
109-109: Include the newstartFrameparameter in the constructor.The
startFrameparameter is correctly included in the constructor.
123-123: Set thestartFrameparameter in the constructor.The
startFrameparameter is correctly set in the constructor.
209-209: Integrate thestartFrameparameter in theframemethod.The
startFrameparameter is correctly integrated into the calculation ofchunkNumber.
268-268: Integrate thestartFrameparameter in thestartDecodemethod.The
startFrameparameter is correctly integrated into the calculation ofchunkNumber.
Line range hint
1-1: Ensure proper error handling indecodeContextImages.The function correctly handles errors by releasing the mutex and rejecting the promise.
Line range hint
1-1: Ensure proper resource management indecodeContextImages.The function correctly manages resources by using a mutex to control access.
cvat/apps/engine/cache.py (11)
61-62: Ensure correct checksum computation inget_checksum.The method correctly computes a checksum using
zlib.crc32.
64-99: Ensure correct cache item handling in_get_or_set_cache_item.The method correctly handles cache item creation and retrieval, including checksum validation.
100-107: Ensure correct cache retrieval in_get.The method correctly retrieves an item from the cache and handles potential errors.
111-118: Ensure correct segment chunk handling inget_segment_chunk.The method correctly retrieves or prepares a segment chunk.
121-128: Ensure correct selective job chunk handling inget_selective_job_chunk.The method correctly retrieves or prepares a selective job chunk.
131-134: Ensure correct segment preview handling inget_or_set_segment_preview.The method correctly retrieves or prepares a segment preview.
137-138: Ensure correct cloud preview handling inget_cloud_preview.The method correctly retrieves a cloud preview.
140-143: Ensure correct cloud preview handling inget_or_set_cloud_preview.The method correctly retrieves or prepares a cloud preview.
146-149: Ensure correct frame context image handling inget_frame_context_images.The method correctly retrieves or prepares frame context images.
152-232: Ensure correct raw frame reading in_read_raw_frames.The method correctly reads raw frames based on task and frame IDs.
234-286: Ensure correct segment chunk preparation inprepare_segment_chunk.The method correctly prepares a segment chunk based on the segment type.
cvat/apps/engine/frame_provider.py (6)
41-73: Ensure correct implementation of_ChunkLoaderclass.The class correctly defines an abstract interface for loading chunks of media data.
76-94: Ensure correct implementation of_FileChunkLoaderclass.The class correctly handles file-based chunk loading.
96-108: Ensure correct implementation of_BufferChunkLoaderclass.The class correctly handles buffer-based chunk loading.
128-206: Ensure correct implementation ofIFrameProviderclass.The class correctly defines an abstract interface for frame providers.
209-377: Ensure correct implementation ofTaskFrameProviderclass.The class correctly handles task-based frame provision.
379-524: Ensure correct implementation ofSegmentFrameProviderclass.The class correctly handles segment-based frame provision.
cvat-core/src/frames.ts (7)
28-28: Typo correction: RenamedprefetchAnalizertoprefetchAnalyzer.This change corrects a typographical error, improving code readability.
211-211: Added private field#startFrametoPrefetchAnalyzer.This field is necessary for the new functionality involving the
startFrameparameter.
213-216: Updated constructor to acceptstartFrameparameter.This change is essential for initializing the new
#startFramefield.
224-224: UpdatedcurrentChunkcalculation to account forstartFrame.This change ensures that the prefetching logic correctly considers the starting frame.
230-230: Updated condition to use modified formula forcurrentChunk.This change ensures that the prefetching logic correctly considers the starting frame.
272-272: UpdatedchunkNumbercalculation to account forstartFrame.This change ensures that the chunk number calculation correctly considers the starting frame.
277-280: UpdatednextChunkNumbercalculation to account forstartFrame.This change ensures that the next chunk number calculation correctly considers the starting frame.
cvat/apps/engine/models.py (5)
259-267: Updated_get_chunk_nameto includesegment_idand return descriptive format.This change improves the naming conventions and provides clearer context for the chunk names.
269-270: Updated_get_compressed_chunk_nameto includesegment_id.This change ensures consistency with the updated chunk naming conventions.
272-273: Updated_get_original_chunk_nameto includesegment_id.This change ensures consistency with the updated chunk naming conventions.
275-277: Renamedget_original_chunk_pathtoget_original_segment_chunk_pathand addedsegmentparameter.This change aligns with the updated method signatures and provides clearer context for the parameters.
279-281: Renamedget_compressed_chunk_pathtoget_compressed_segment_chunk_pathand addedsegmentparameter.This change aligns with the updated method signatures and provides clearer context for the parameters.
cvat/apps/lambda_manager/views.py (4)
35-35: Updated import statement to replaceFrameProviderwithFrameQualityandTaskFrameProvider.This change indicates a shift towards a more structured approach in managing frame quality attributes.
483-485: Updated logic for determiningqualityto useFrameQuality.This change enhances clarity and maintainability by encapsulating quality definitions within a dedicated class.
492-492: Updated instantiation of frame provider toTaskFrameProvider(db_task).This change suggests a refactoring that may provide better context or functionality tied to the task rather than the data alone.
485-485: Updated condition to referenceFrameQuality.ORIGINALandFrameQuality.COMPRESSED.This change improves the semantic accuracy of the code.
cvat/apps/engine/media_extractors.py (8)
55-57: LGTM!The
FrameQualityenumeration is well-defined and categorizes frame quality effectively.
104-137: LGTM!The
RandomAccessIteratorclass is well-implemented, allowing random access to elements in an iterable and managing resources effectively.
144-189: LGTM!The
CachingMediaIteratorclass is well-implemented, extendingRandomAccessIteratorand adding effective caching capabilities.
533-573: LGTM!The
_make_frame_iteratormethod is well-implemented, encapsulating the logic for frame iteration and handling optional filtering and rotation based on metadata.
589-605: LGTM!The
_read_av_containermethod ensures proper cleanup of resources, addressing potential memory leaks.
645-667: LGTM!The
get_frame_countmethod is well-implemented, providing the total frame count in the video.
670-679: LGTM!The
ImageReaderWithManifestclass is well-implemented, providing manifest-based access to images.
680-754: LGTM!The
VideoReaderWithManifestclass is well-implemented, providing manifest-based access to videos and handling threading and frame iteration effectively.cvat/apps/engine/task.py (5)
Line range hint
125-173:
LGTM!The
_generate_segment_paramsfunction is well-implemented, generating segment parameters with named parameters and improving readability and maintainability.
176-200: LGTM!The
_create_segments_and_jobsfunction is well-implemented, focusing on segment and job creation and streamlining the process.
1134-1266: LGTM!The
_create_static_chunksfunction is well-implemented, encapsulating the logic for creating static chunks and improving modularity.
77-79: LGTM!The
SegmentParamsclass is well-implemented, providing a clearer structure for segment parameters with the addition of thetypefield and optionalframesparameter.
1140-1154: LGTM!The
update_progressfunction is well-implemented, enhancing user feedback during long-running operations.cvat/apps/dataset_manager/formats/cvat.py (2)
1374-1375: LGTM! Using the factory function improves modularity.The use of
make_frame_providerinstead of directly instantiatingFrameProviderenhances modularity and flexibility.
1380-1385: Verify the correctness of the frame retrieval process.Ensure that the
iterate_framesmethod correctly retrieves frames with the specified parameters.cvat/apps/dataset_manager/bindings.py (5)
34-34: Import changes approved.The import statement has been updated to reflect the new frame provider and enums.
1351-1361: Frame quality and output type changes approved.The method now uses
FrameQuality.ORIGINALandFrameOutputType.NUMPY_ARRAYfor video frames andFrameQuality.ORIGINALandFrameOutputType.BUFFERfor image frames, aligning with the new enums.
1369-1369: Frame provider update approved.The method now uses
TaskFrameProviderinstead ofFrameProvider, aligning with the new class introduced.
1505-1505: Frame extension update approved.The method now uses
TaskFrameProvider.VIDEO_FRAME_EXTto set the frame extension for video tasks, aligning with the new class and its properties.
1596-1596: Frame extension update approved.The method now uses
TaskFrameProvider.VIDEO_FRAME_EXTto set the frame extension for video tasks, aligning with the new class and its properties.cvat/apps/engine/views.py (9)
631-641: Method Implementation: Ensure_TaskDataGetterusageThe
previewmethod uses_TaskDataGetterfor data retrieval. Ensure that_TaskDataGetteris correctly used and implemented.Verification successful
Method Implementation: Ensure
_TaskDataGetterusageThe
_TaskDataGetterclass is correctly defined and used in thepreviewmethod ofProjectViewSet. The class extends_DataGetterand includes the necessary parameters.
_TaskDataGetteris defined incvat/apps/engine/views.py.- It is used multiple times in the same file, including in the
previewmethod.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify that `_TaskDataGetter` is correctly used and implemented. # Test: Search for `_TaskDataGetter` usage. Expect: Correct usage and implementation. rg --type python '_TaskDataGetter'Length of output: 65
Script:
#!/bin/bash # Verify that `_TaskDataGetter` is correctly used and implemented. # Test: Search for `_TaskDataGetter` usage. Expect: Correct usage and implementation. rg '_TaskDataGetter' --glob '*.py' -A 5Length of output: 1385
714-727: Constructor: Ensuredb_taskis a valid Task model instanceThe constructor of
_TaskDataGetteracceptsdb_task. Ensure thatdb_taskis a valid instance of theTaskmodel.Verification successful
Constructor: Ensure
db_taskis a valid Task model instanceThe constructor of
_TaskDataGetteracceptsdb_task. Based on the extensive usage ofself._db_taskacross the codebase, it is consistently assigned a validTaskmodel instance, particularly highlighted inbackup.pywhere it is assigned from aTaskmodel object.
cvat/apps/engine/backup.py: self._db_task = models.Task.objects.create(**self._manifest, organization_id=self._org_id)cvat/apps/engine/backup.py: self._db_task = models.Task.objects.prefetch_related(...).get(pk=pk)Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify that `db_task` is a valid instance of the `Task` model. # Test: Search for `db_task` usage. Expect: Valid instance of `Task`. rg --type python 'self\._db_task'Length of output: 66
Script:
#!/bin/bash # Verify that `db_task` is a valid instance of the `Task` model. # Test: Search for `db_task` usage. Expect: Valid instance of `Task`. rg --glob '*.py' 'self\._db_task'Length of output: 6864
661-664: Constructor: Ensuredata_numtype consistencyThe constructor of
_DataGetteracceptsdata_numasOptional[Union[str, int]]. Ensure thatdata_numis consistently handled as an integer within the class.Verification successful
Constructor: Ensure
data_numtype consistencyThe constructor of
_DataGetteracceptsdata_numasOptional[Union[str, int]]. The search results confirm thatdata_numis consistently converted to an integer within the class, ensuring type consistency.
self.number = int(data_num) if data_num is not None else NoneScripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify that `data_num` is consistently handled as an integer within `_DataGetter`. # Test: Search for `data_num` usage. Expect: Consistent handling as an integer. rg --type python 'self\.number'Length of output: 64
Script:
#!/bin/bash # Verify that `data_num` is consistently handled as an integer within `_DataGetter`. # Test: Search for `data_num` usage. Expect: Consistent handling as an integer. rg --type py 'self\.data_num'Length of output: 31
Script:
#!/bin/bash # Verify that `data_num` is consistently handled as an integer within `_DataGetter`. # Test: Search for `data_num` usage in the entire file. Expect: Consistent handling as an integer. rg 'data_num' cvat/apps/engine/views.pyLength of output: 848
730-743: Constructor: Ensuredb_jobis a valid Job model instanceThe constructor of
_JobDataGetteracceptsdb_job. Ensure thatdb_jobis a valid instance of theJobmodel.Verification successful
Constructor: Ensure
db_jobis a valid Job model instanceThe constructor of
_JobDataGetteracceptsdb_job. It has been verified thatdb_jobis a valid instance of theJobmodel.
db_jobis assigned usingJob.objects.select_related,Job.objects.get, andmodels.Job.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify that `db_job` is a valid instance of the `Job` model. # Test: Search for `db_job` usage. Expect: Valid instance of `Job`. rg --type python 'self\._db_job'Length of output: 65
Script:
#!/bin/bash # Verify that `db_job` is a valid instance of the `Job` model. # Test: Search for `self._db_job` usage. Expect: Valid instance of `Job`. rg -t py 'self\._db_job'Length of output: 1377
Script:
#!/bin/bash # Verify the initialization or type assignment of `db_job` to ensure it is a valid instance of the `Job` model. # Search for `db_job` assignments in the codebase to verify its type. rg -t py 'db_job\s*='Length of output: 1908
1997-2000: Method Implementation: Ensure_JobDataGetterusageThe
datamethod uses_JobDataGetterfor data retrieval. Ensure that_JobDataGetteris correctly used and implemented.Verification successful
Method Implementation: Ensure
_JobDataGetterusageThe
_JobDataGetterclass is correctly used and implemented in thedatamethod ofJobViewSet. The class is properly defined with an__init__method that initializes necessary parameters and a_get_frame_providermethod that encapsulates the frame provider logic.
_JobDataGetterclass definition and methods:
__init__method initializes withdb_job,data_type,data_quality, anddata_num._get_frame_providermethod returns aJobFrameProviderinstance.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify that `_JobDataGetter` is correctly used and implemented. # Test: Search for `_JobDataGetter` usage. Expect: Correct usage and implementation. rg --type python '_JobDataGetter'Length of output: 64
Script:
#!/bin/bash # Verify that `_JobDataGetter` is correctly used and implemented. # Test: Search for `_JobDataGetter` usage. Expect: Correct usage and implementation. rg --type-add 'py:*.py' '_JobDataGetter'Length of output: 229
Script:
#!/bin/bash # Retrieve the definition and methods of `_JobDataGetter` from `cvat/apps/engine/views.py`. # Search for the class definition and its methods. ast-grep --lang python --pattern 'class _JobDataGetter($_) { $$$ }'Length of output: 524
Script:
#!/bin/bash # Retrieve the definition and methods of `_JobDataGetter` from `cvat/apps/engine/views.py`. # Search for the class definition and its methods. ast-grep --lang python --pattern 'class _JobDataGetter($_): $$$'Length of output: 932
742-743: Method Implementation: EnsureJobFrameProviderusageThe
_get_frame_providermethod returns an instance ofJobFrameProvider. Ensure thatJobFrameProvideris correctly used and implemented.Verification successful
JobFrameProvider is correctly used and implemented
The
JobFrameProviderclass is correctly implemented and used within the codebase. It inherits fromSegmentFrameProviderand is instantiated properly in themake_frame_providerfunction.
cvat/apps/engine/frame_provider.py:
- Definition at line 537
- Usage at line 546
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify that `JobFrameProvider` is correctly used and implemented. # Test: Search for `JobFrameProvider` usage. Expect: Correct usage and implementation. rg --type python 'JobFrameProvider'Length of output: 66
Script:
#!/bin/bash # Find all Python files and search for `JobFrameProvider` usage and implementation. fd -e py -x grep -Hn 'JobFrameProvider' {}Length of output: 423
Script:
#!/bin/bash # Extract lines around the definition and usage of `JobFrameProvider` in `frame_provider.py`. grep -A 10 -B 10 'class JobFrameProvider' ./cvat/apps/engine/frame_provider.py grep -A 10 -B 10 'JobFrameProvider(data_source)' ./cvat/apps/engine/frame_provider.pyLength of output: 1575
2093-2098: Method Implementation: Ensure_JobDataGetterusageThe
previewmethod uses_JobDataGetterfor data retrieval. Ensure that_JobDataGetteris correctly used and implemented.Verification successful
Method Implementation: Ensure
_JobDataGetterusageThe
previewmethod uses_JobDataGetterfor data retrieval. The_JobDataGetterclass is correctly implemented and used in thepreviewmethod.
_JobDataGetterclass is defined incvat/apps/engine/views.py(line 730).- It inherits from
_DataGetterand correctly implements necessary methods.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify that `_JobDataGetter` is correctly used and implemented. # Test: Search for `_JobDataGetter` usage. Expect: Correct usage and implementation. rg --type python '_JobDataGetter'Length of output: 64
Script:
#!/bin/bash # Verify that `_JobDataGetter` is correctly used and implemented. # Test: Search for `_JobDataGetter` usage. Expect: Correct usage and implementation. rg --type py '_JobDataGetter'Length of output: 218
Script:
#!/bin/bash # Extract the implementation of `_JobDataGetter` from `cvat/apps/engine/views.py`. # Use `ast-grep` to get the class definition and its methods. ast-grep --lang python --pattern $'class _JobDataGetter($_):\n $$$' cvat/apps/engine/views.pyLength of output: 962
Script:
#!/bin/bash # Extract the implementation of `_DataGetter` from `cvat/apps/engine/views.py`. # Use `ast-grep` to get the class definition and its methods. ast-grep --lang python --pattern $'class _DataGetter($_):\n $$$' cvat/apps/engine/views.pyLength of output: 4104
1266-1269: Method Implementation: Ensure_TaskDataGetterusageThe
datamethod uses_TaskDataGetterfor data retrieval. Ensure that_TaskDataGetteris correctly used and implemented.Verification successful
Method Implementation: Ensure
_TaskDataGetterusageThe
datamethod inTaskViewSetcorrectly uses_TaskDataGetterfor data retrieval. The_TaskDataGetterclass is properly implemented with necessary methods.
_TaskDataGetterclass is defined incvat/apps/engine/views.pyand inherits from_DataGetter.- It includes an
__init__method and a_get_frame_providermethod.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify that `_TaskDataGetter` is correctly used and implemented. # Test: Search for `_TaskDataGetter` usage. Expect: Correct usage and implementation. rg --type python '_TaskDataGetter'Length of output: 65
Script:
#!/bin/bash # Verify that `_TaskDataGetter` is correctly used and implemented. # Test: Search for `_TaskDataGetter` usage. Expect: Correct usage and implementation. rg '_TaskDataGetter' --glob '*.py' -A 5Length of output: 1385
Script:
#!/bin/bash # Extract the full definition of `_TaskDataGetter` and its methods from `cvat/apps/engine/views.py`. # Search for the class definition and its methods. ast-grep --lang python --pattern $'class _TaskDataGetter(_DataGetter) { $$$ }' cvat/apps/engine/views.pyLength of output: 574
Script:
#!/bin/bash # Extract the full definition of `_TaskDataGetter` and its methods from `cvat/apps/engine/views.py`. # Search for the class definition and its methods. ast-grep --lang python --pattern 'class _TaskDataGetter(_DataGetter):' cvat/apps/engine/views.py -A 50Length of output: 104
Script:
#!/bin/bash # Extract the full definition of `_TaskDataGetter` and its methods from `cvat/apps/engine/views.py`. # Search for the class definition and its methods. rg 'class _TaskDataGetter' cvat/apps/engine/views.py -A 50Length of output: 1637
726-727: Method Implementation: EnsureTaskFrameProviderusageThe
_get_frame_providermethod returns an instance ofTaskFrameProvider. Ensure thatTaskFrameProvideris correctly used and implemented.Verification successful
TaskFrameProvider is correctly used and implemented
The
TaskFrameProviderclass is defined and implemented in thecvat/apps/engine/frame_provider.pyfile. The class includes methods for validating frame and chunk numbers, indicating that it adheres to the expected interface and functionality.
cvat/apps/engine/frame_provider.py:class TaskFrameProvider(IFrameProvider): def __init__(self, db_task: models.Task) -> None: self._db_task = db_task def validate_frame_number(self, frame_number: int) -> int: start = self._db_task.data.start_frame stop = self._db_task.data.stop_frame if frame_number not in range(start, stop + 1, self._db_task.data.get_frame_step()): raise ValidationError( f"Invalid frame '{frame_number}'. " f"The frame number should be in the [{start}, {stop}] range" ) return frame_number def validate_chunk_number(self, chunk_number: int) -> int: last_chunk = math.ceil(self._db_task.data.size / self._db_task.data.chunk_size) - 1 if not (0 <= chunk_number <= last_chunk): raise ValidationError( f"Invalid chunk number '{chunk_number}'. " f"The chunk number should be in the [0, {last_chunk}] range" )This confirms that the
TaskFrameProvideris correctly used and implemented in the codebase.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Verify that `TaskFrameProvider` is correctly used and implemented. # Test: Search for `TaskFrameProvider` usage. Expect: Correct usage and implementation. rg --type python 'TaskFrameProvider'Length of output: 67
Script:
#!/bin/bash # Find all Python files and search for `TaskFrameProvider` usage and implementation. # Locate all Python files and search for `TaskFrameProvider`. fd -e py -x rg 'TaskFrameProvider' {}Length of output: 841
Script:
#!/bin/bash # Extract the implementation details of `TaskFrameProvider`. # Search for the definition and implementation of `TaskFrameProvider`. ast-grep --lang python --pattern $'class TaskFrameProvider(IFrameProvider) { $$$ }'Length of output: 558
Script:
#!/bin/bash # Extract the implementation details of `TaskFrameProvider`. # Search for the definition and implementation of `TaskFrameProvider`. ast-grep --lang python --pattern 'class TaskFrameProvider(IFrameProvider):'Length of output: 77
Script:
#!/bin/bash # Search for the definition of `TaskFrameProvider` class. # Use `rg` to find the class definition of `TaskFrameProvider`. rg 'class TaskFrameProvider' -A 20Length of output: 1730
| @@ -0,0 +1,16 @@ | |||
| # Copyright (C) 2024 CVAT.ai Corporation | |||
There was a problem hiding this comment.
Why do we need the file? Why don't define the settings in the global settings file?
There was a problem hiding this comment.
You can redefine these settings in the global settings file if you need it, this file defines the default values and guarantees the required settings are there.
|
@zhiltsov-max, Mostly LGTM, but I've found 2 small bugs:
|
…ation and retrieval
Fixed
Fixed |
| # TODO: find a way to use prefetched results, if provided | ||
| db_images = ( | ||
| db_data.images.order_by("frame") | ||
| .filter(frame__gte=frame_ids[0], frame__lte=frame_ids[-1]) |
There was a problem hiding this comment.
Probably we should set chunk_size max limit and use here frame__in (in that case we do not need to iterate over requested_frame_iter and compare a frame_id with next_requested_frame_id). What do you think? (can be changed in a separate PR)
There was a problem hiding this comment.
I don't really like using __in, as it can result in very big queries, which are processed slowly. And it is possible here, as chunk_size is only limited by the task size now. I think it can be done once we introduce a limit on the chunk_size, which is planned to be done in another PR.
There was a problem hiding this comment.
I don't really like using __in, as it can result in very big queries, which are processed slowly.
I agree, but I also said that frame__in should be used only when we add a chunk_size limit.
There was a problem hiding this comment.
Ok, I thought you're speaking about order by + limit in the resulting sql.
|
Motivation and context
importworker or the server process on process shutdownCVAT_ALLOW_STATIC_CACHE(boolean) to enable and disable static cache support. The option is disabled by default (it's changed from the previous behavior)indexparameter inGET /api/jobs/{id}/data/?type=chunkto simplify indexingWhen this update is applied to the server, there will be a data storage setting migration for the tasks. Existing tasks using static chunks (
task.data.storage_method == FILE_SYSTEM) will be switched to the dynamic cache (i.e. to== CACHE)). The remaining files should be removed manually, there will be a list of such tasks in the migration log file.After this update, you'll have an option to enable or disable static cache use during task creation. This allows, in particular, prohibit new tasks using the static cache. With this option, any tasks using static cache will use the dynamic cache instead on data access.
User-observable changes:
use_cache = falseorstorage_method = filesystemparameters in task creation can be ignored by the serverHow has this been tested?
Checklist
developbranch(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)
License
Feel free to contact the maintainers if that's a concern.
Summary by CodeRabbit
Summary by CodeRabbit
New Features
startFrameparameter for improved chunk calculations.data_original_chunk_type, for enhanced job differentiation in the metadata.Bug Fixes
prefetchAnalyzer.Documentation
Refactor
Chores