-
Notifications
You must be signed in to change notification settings - Fork 262
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: improve the use case doc (v1) (#928)
* Image classification * Image similarity search * Q&A application
- Loading branch information
Showing
4 changed files
with
293 additions
and
15 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,104 @@ | ||
Implementing a Image Classification Pipeline using EvaDB on a Video | ||
==== | ||
|
||
Assume the database has loaded a video ``mnist_video``. | ||
|
||
1. Connect to EvaDB | ||
---- | ||
|
||
.. code-block:: python | ||
import evadb | ||
cursor = evadb.connect().cursor() | ||
2. Register Image Classification Model as a Function in SQL | ||
---- | ||
|
||
Create an image classification function from python source code. | ||
|
||
.. code-block:: python | ||
query = cursor.query(""" | ||
CREATE UDF IF NOT EXISTS MnistImageClassifier | ||
IMPL 'evadb/udfs/mnist_image_classifier.py'; | ||
""").execute() | ||
3. Execute Image Classification through SQL | ||
---- | ||
|
||
After the function is registered to EvaDB system, it can be directly called and used in SQL query. | ||
|
||
.. code-block:: python | ||
query = cursor.table("mnist_video").select("MnistImageClassifier(data).label") | ||
.. note:: | ||
|
||
SQL statement | ||
|
||
.. code-block:: sql | ||
SELECT MnistImageClassifier(data).label FROM mnist_video | ||
Get results in a ``DataFrame``. | ||
|
||
.. code-block:: python | ||
query.df() | ||
The result contains a projected ``label`` column, which indicates the digit of a particular frame. | ||
|
||
.. code-block:: | ||
+------------------------------+ | ||
| mnistimageclassifier.label | | ||
|------------------------------| | ||
| 6 | | ||
| 6 | | ||
| 6 | | ||
| 6 | | ||
| 6 | | ||
| 6 | | ||
| 4 | | ||
| 4 | | ||
... ... | ||
4. Optional: Process Only Segments of Videos based on Conditions | ||
---- | ||
|
||
Like normal SQL, you can also specify conditions to filter out some frames of the video. | ||
|
||
.. code-block:: python | ||
query = cursor.table("mnist_video") \ | ||
.filter("id < 2") \ | ||
.select("MnistImageClassifier(data).label") | ||
.. note:: | ||
|
||
SQL statement | ||
|
||
.. code-block:: sql | ||
SELECT MnistImageClassifier(data).label FROM mnist_video | ||
WHERE id < 2 | ||
Return results in a ``DataFrame``. | ||
|
||
.. code-block:: python | ||
query.df() | ||
Now, the ``DataFrame`` only contains 2 rows after filtering. | ||
|
||
.. code-block:: | ||
+------------------------------+ | ||
| mnistimageclassifier.label | | ||
|------------------------------| | ||
| 6 | | ||
| 6 | | ||
+------------------------------+ | ||
Check out our `Jupyter Notebook <https://github.com/georgia-tech-db/evadb/blob/master/tutorials/01-mnist.ipynb>`_ for working example. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,92 @@ | ||
Q&A Application on Videos | ||
==== | ||
|
||
1. Connect to EvaDB | ||
---- | ||
|
||
.. code-block:: python | ||
import evadb | ||
cursor = evadb.connect().cursor() | ||
2. Register Functions | ||
---- | ||
|
||
Whisper | ||
**** | ||
|
||
.. code-block:: python | ||
cursor.query(""" | ||
CREATE UDF SpeechRecognizer | ||
TYPE HuggingFace | ||
'task' 'automatic-speech-recognition' | ||
'model' 'openai/whisper-base'; | ||
""").execute() | ||
.. note:: | ||
|
||
EvaDB allows users to register any model in HuggingFace as a function. | ||
|
||
ChatGPT | ||
**** | ||
|
||
.. code-block:: python | ||
cursor.query(""" | ||
CREATE UDF ChatGPT | ||
IMPL 'evadb/udfs/chatgpt.py' | ||
""").execute() | ||
# Set OpenAI token | ||
import os | ||
os.environ["OPENAI_KEY"] = "sk-..." | ||
.. note:: | ||
|
||
ChatGPT function is a wrapper around OpenAI API call. You can also switch to other LLM models that can run locally. | ||
|
||
3. Summarize Video in Text | ||
---- | ||
|
||
Create a table with text summary of the video. | ||
Text summarization is generated by running audio-to-text ``Whisper`` model from ``HuggingFace``. | ||
|
||
.. code-block:: python | ||
cursor.query(""" | ||
CREATE TABLE text_summary AS | ||
SELECT SpeechRecognizer(audio) FROM ukraine_video; | ||
""").execute() | ||
This results a table shown below. | ||
|
||
.. code-block:: | ||
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ||
| text_summary.text | | ||
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| The war in Ukraine has been on for 415 days. Who is winning it? Not Russia. Certainly not Ukraine. It is the US oil companies. US oil companies have reached $200 billion in pure profits. The earnings are still on. They are still milking this war and sharing the spoils. Let us look at how Exxon mobile has been doing. In 2022, the company made $56 billion in profits. Oil companies capitalized on instability and they are profiting from pain. American oil companies are masters of this art. You may remember the war in Iraq. The US went to war in Iraq by selling a lie. The Americans did not find any weapons of mass destruction but they did find lots of oil. And in the year since, American officials have admitted this. And this story is not over. It's repeating itself in Ukraine. They are feeding another war and filling the coffers of US oil companies. | | ||
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ||
4. Q&A using ChatGPT | ||
---- | ||
|
||
We can now embed the ChatGPT prompt inside SQL with text summary from the table as its knowledge base. | ||
|
||
.. code-block:: python | ||
cursor.query(""" | ||
SELECT ChatGPT('Is this video summary related to Ukraine russia war', text) | ||
FROM text_summary; | ||
""").df() | ||
This query returns a projected ``DataFrame``. | ||
|
||
.. code-block:: | ||
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ||
| chatgpt.response | | ||
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| Based on the provided context, it seems that the video summary is related to the Ukraine-Russia war. It discusses how US oil companies are allegedly profiting from the war in Ukraine, similar to how they allegedly benefited from the war in Iraq. | | ||
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
Implementing a Similar Image Search Pipeline using EvaDB on Images | ||
==== | ||
|
||
In this use case, we want to search similar images based on an image provided by the user. | ||
To implement this use case, we leverage EvaDB's capability of easily expressing feature extraction pipeline. | ||
Additionaly, we also leverage EvaDB's capability of building a similarity search index and searching the index to | ||
locate similar images through ``FAISS`` library. | ||
|
||
For this use case, we use a reddit image dataset that can be downloaded from `Here <https://www.dropbox.com/scl/fo/fcj6ojmii0gw92zg3jb2s/h\?dl\=1\&rlkey\=j3kj1ox4yn5fhonw06v0pn7r9>`_. | ||
We populate a table in the database that contains all images. | ||
|
||
1. Connect to EvaDB | ||
---- | ||
|
||
.. code-block:: python | ||
import evadb | ||
cursor = evadb.connect().cursor() | ||
2. Register SIFT as Function | ||
---- | ||
|
||
.. code-block:: python | ||
cursor.query(""" | ||
CREATE UDF IF NOT EXISTS SiftFeatureExtractor | ||
IMPL 'evadb/udfs/sift_feature_extractor.py' | ||
""").execute() | ||
3. Search Similar Images | ||
---- | ||
|
||
To locate images that have similar appearance, we will first build an index based on embeddings of images. | ||
Then, for the given image, EvaDB can find similar images by searching in the index. | ||
|
||
Build Index using ``FAISS`` | ||
**** | ||
|
||
The below query creates a new index on the projected column ``SiftFeatureExtractor(data)`` from the ``reddit_dataset`` table. | ||
|
||
.. code-block:: python | ||
cursor.query(""" | ||
CREATE INDEX reddit_sift_image_index | ||
ON reddit_dataset (SiftFeatureExtractor(data)) | ||
USING FAISS | ||
""").execute() | ||
Search Index for a Given Image | ||
**** | ||
|
||
EvaDB leverages the ``ORDER BY ... LIMIT ...`` SQL syntax to retrieve the top 5 similar images. | ||
In this example, ``Similarity(x, y)`` is a built-in function to calculate distance between ``x`` and ``y``. | ||
In current version, ``x`` is a single tuple and ``y`` is a column that contains multiple tuples. | ||
By default EvaDB does pairwise distance calculation between ``x`` and all tuples from ``y``. | ||
In this case, EvaDB leverages the index that we have already built. | ||
|
||
.. code-block:: python | ||
query = cursor.query(""" | ||
SELECT name FROM reddit_dataset ORDER BY | ||
Similarity( | ||
SiftFeatureExtractor(Open('reddit-images/g1074_d4mxztt.jpg')), | ||
SiftFeatureExtractor(data) | ||
) | ||
LIMIT 5 | ||
""") | ||
query.df() | ||
The ``DataFrame`` contains the top 5 similar images. | ||
|
||
.. code-block:: | ||
+---------------------------------+ | ||
| reddit_dataset.name | | ||
|---------------------------------| | ||
| reddit-images/g1074_d4mxztt.jpg | | ||
| reddit-images/g348_d7ju7dq.jpg | | ||
| reddit-images/g1209_ct6bf1n.jpg | | ||
| reddit-images/g1190_cln9xzr.jpg | | ||
| reddit-images/g1190_clna2x2.jpg | | ||
+---------------------------------+ | ||
Check out our `Jupyter Notebook <https://github.com/georgia-tech-db/evadb/blob/master/tutorials/11-similarity-search-for-motif-mining.ipynb>`_ for working example. | ||
We also demonstrate more complicated features of EvaDB for similarity search. |