ViT-B classifier added with first 30 labels #7842

omerferhatt · 2024-05-02T22:03:04Z

Motivation and context

Since there is no classifier model in the repo, I added it to make it easier for general use and to be used as a draft. I think this can be useful for testing too.

#3896 (comment)

How has this been tested?

Checklist

I submit my changes into the develop branch
I have created a changelog fragment
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues (see GitHub docs)
I have increased versions of npm packages if it is necessary
(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.

Summary by CodeRabbit

New Features
- Implemented a serverless function using a Vision Transformer (ViT-B) model for inferring labels from images.
- Added necessary imports and declarations for processing images and running inference.
Enhancements
- Enhanced model deployment specifications, runtime environment, and dependencies.
- Introduced restart policies and storage mount settings for improved resource management.
Functionality
- Developed a serverless function to process input images, run inference using the ViT-B model, and return results in JSON format.

coderabbitai · 2024-05-02T22:03:12Z

Walkthrough

The recent updates introduce a serverless deployment of a PyTorch-based Vision Transformer (ViT-B) model using Nuclio. The changes encompass enhanced configurations, new function definitions, and a class for model handling, streamlining the process for image classification tasks in a cloud environment.

Changes

File Path	Change Summary
`.../function-gpu.yaml`	Configuration for GPU-based deployment, including metadata, runtime, resources, and HTTP trigger.
`.../function.yaml`	Similar to `function-gpu.yaml` but tailored for standard deployment. Includes similar metadata, runtime, and HTTP trigger settings.
`.../main.py`	Implementation of serverless function for image processing and inference.
`.../model_handler.py`	Introduction of `ModelHandler` class for initializing and running the ViT model.

🐰✨
In the land of code and cloud,
A rabbit hopped, feeling proud.
For the models now deploy with ease,
Amongst the servers, a gentle breeze.
"Hop, skip, a serverless feat,
With PyTorch and Nuclio, none can compete!"

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 3

Out of diff range and nitpick comments (1)

serverless/pytorch/omerferhatt/vit-b/nuclio/function-gpu.yaml (1)

1-88: Ensure GPU resource allocation is optimized.

The configuration specifies the allocation of one GPU. It is important to monitor the utilization of GPU resources to ensure they are being used efficiently. Over-provisioning can lead to unnecessary costs, while under-provisioning can affect performance. Consider implementing monitoring tools to track GPU usage and adjust the allocation based on actual usage patterns.

serverless/pytorch/omerferhatt/vit-b/nuclio/model_handler.py

serverless/pytorch/omerferhatt/vit-b/nuclio/main.py

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

bsekachev · 2024-05-07T09:05:21Z

Hello,

I am not sure we may use models from torchvision.
As this specific model was trained on ImageNet, but ImageNet is non-commercial dataset.

Sources:

omerferhatt · 2024-05-28T15:16:34Z

@bsekachev I can update the model without weights, and it will be used as a template for classifier serverless models since there are none.

Regarding the torchvision usage, I don't think there will be a problem since all the architectures published under torchvision are open-source. But things can be different for weights, as you say.

bsekachev · 2024-05-07T08:56:29Z

serverless/pytorch/omerferhatt/vit-b/nuclio/model_handler.py

+
+
+class ModelHandler:
+    weights = tv.models.ViT_B_16_Weights.DEFAULT


Let's explicitly specify what model will be used: IMAGENET1K_V1

bsekachev · 2024-05-07T08:58:54Z

serverless/pytorch/omerferhatt/vit-b/nuclio/function-gpu.yaml

@@ -0,0 +1,88 @@
+metadata:


Path should be serverless/pytorch/torchvision/vit-b/nuclio as this model is a part of torchvision

bsekachev · 2024-05-30T09:42:06Z

serverless/pytorch/omerferhatt/vit-b/nuclio/main.py

+    context.logger.info("Init context...100%")
+
+def handler(context, event):
+    context.logger.info("Run ViT-B model")


I do not think we need these logs on production

bsekachev · 2024-05-30T09:42:29Z

serverless/pytorch/omerferhatt/vit-b/nuclio/main.py

+        "confidence": str(score),
+        "label": context.user_data.labels[class_id],
+        "type": "tag",
+        "objectType": "tag",


Not necessary field, as discussed in another pull request

bsekachev · 2024-05-30T09:43:33Z

serverless/pytorch/omerferhatt/vit-b/nuclio/model_handler.py

+            class_id = prediction.argmax().item()
+            score = prediction[class_id].item()
+            category_name = self.weights.meta["categories"][class_id]
+        return (class_id, category_name, score)


Please, put return inside the context manager

bsekachev · 2024-05-30T09:46:56Z

@omerferhatt

We may put this code to our repository, but we will need to add a license file to this serverless function, mentioning its license.
Unfortunately, it will not be possible to deploy it on app.cvat.ai, as it is non-commercial, but it would be a good example on how to write serverless classifiers. And, maybe it will be useful for someone in research purposes.

bsekachev · 2024-06-19T05:47:37Z

Hello @omerferhatt

Do you have any plans regarding the pull request?

bsekachev · 2024-06-24T08:35:55Z

I will close the pull request now.
Please, do not hesitate to reopen if you are ready to finish it.

omerferhatt · 2024-07-02T08:37:34Z

I'm going to try to finish this week, thanks.

omerferhatt added 3 commits May 3, 2024 00:59

ViT-B classifier added with first 30 labels

6bbbc77

Typo fixes

766ac2c

EOF added

3c9d0f6

coderabbitai bot reviewed May 2, 2024

View reviewed changes

serverless/pytorch/omerferhatt/vit-b/nuclio/model_handler.py Show resolved Hide resolved

serverless/pytorch/omerferhatt/vit-b/nuclio/main.py Show resolved Hide resolved

serverless/pytorch/omerferhatt/vit-b/nuclio/main.py Outdated Show resolved Hide resolved

omerferhatt changed the title ~~[WiP] ViT-B classifier added with first 30 labels~~ ViT-B classifier added with first 30 labels May 2, 2024

omerferhatt and others added 3 commits May 3, 2024 13:42

Update serverless/pytorch/omerferhatt/vit-b/nuclio/main.py

5db441a

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

Merge branch 'develop' into feature/vit-b-classifier

02ed0e5

Merge branch 'develop' into feature/vit-b-classifier

ad33560

bsekachev reviewed May 30, 2024

View reviewed changes

bsekachev closed this Jun 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ViT-B classifier added with first 30 labels #7842

ViT-B classifier added with first 30 labels #7842

omerferhatt commented May 2, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented May 2, 2024 •

edited

Loading

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (`.coderabbit.yaml`)

Documentation and Community

coderabbitai bot left a comment

bsekachev commented May 7, 2024

omerferhatt commented May 28, 2024

bsekachev May 7, 2024

bsekachev May 7, 2024

bsekachev May 30, 2024

bsekachev May 30, 2024

bsekachev May 30, 2024

bsekachev commented May 30, 2024

bsekachev commented Jun 19, 2024

bsekachev commented Jun 24, 2024

omerferhatt commented Jul 2, 2024



		class ModelHandler:
		weights = tv.models.ViT_B_16_Weights.DEFAULT

ViT-B classifier added with first 30 labels #7842

ViT-B classifier added with first 30 labels #7842

Conversation

omerferhatt commented May 2, 2024 • edited by coderabbitai bot Loading

Motivation and context

How has this been tested?

Checklist

License

Summary by CodeRabbit

coderabbitai bot commented May 2, 2024 • edited Loading

Walkthrough

Changes

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configration File (.coderabbit.yaml)

Documentation and Community

coderabbitai bot left a comment

Choose a reason for hiding this comment

bsekachev commented May 7, 2024

omerferhatt commented May 28, 2024

bsekachev May 7, 2024

Choose a reason for hiding this comment

bsekachev May 7, 2024

Choose a reason for hiding this comment

bsekachev May 30, 2024

Choose a reason for hiding this comment

bsekachev May 30, 2024

Choose a reason for hiding this comment

bsekachev May 30, 2024

Choose a reason for hiding this comment

bsekachev commented May 30, 2024

bsekachev commented Jun 19, 2024

bsekachev commented Jun 24, 2024

omerferhatt commented Jul 2, 2024

omerferhatt commented May 2, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented May 2, 2024 •

edited

Loading

CodeRabbit Configration File (`.coderabbit.yaml`)