Provider: justtakeitfree.com #2793

obulat · 2023-08-07T17:03:04Z

Fixes

Description

Adds the script to get all the media from justtakeitfree.com. Currently, there are 167 images.

To collect the filesize, this script makes head requests for individual media items, which make the script slower than expected. But it should be okay considering the number of available images.
Image dimensions are not available, so they will need to be collected separately in the future.

Testing Instructions

Run the justtakeitfree DAG and check that it succeeds.
See what it saves using

just catalog/pgcli
select count(*) from image where source='justtakeitfree';

(should return 167).
Check that tags, filesize creator, creator url are saved correctly:

select tags, filesize, creator, creator_url from image where source='justtakeitfree';

Checklist

My pull request has a descriptive title (not a vague title likeUpdate index.md).
My pull request targets the default branch of the repository (main) or a parent feature branch.
My commit messages follow best practices.
My code follows the established code style of the repository.
I added or updated tests for the changes I made (if applicable).
I added or updated documentation (if applicable).
I tried running the project locally and verified that there are no visible errors.
I ran the DAG documentation generator (if applicable).

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

krysal · 2023-08-31T16:45:31Z

@obulat What is this new DAG missing for being ready for review?

stacimc

Looks fantastic, no blocking concerns! I see 167 results, data all looks great. Just confirming, were we interested in using the preview_url from the API for thumbnails?

catalog/dags/providers/provider_workflows.py

catalog/tests/dags/providers/provider_api_scripts/test_justtakeitfree.py

obulat · 2023-09-08T07:16:04Z

Looks fantastic, no blocking concerns! I see 167 results, data all looks great. Just confirming, were we interested in using the preview_url from the API for thumbnails?

That's a great suggestions, the preview_url seems to be 800px wide, so the quality is good for thumbnails. Added in 34921aa

krysal

Looks good! I was able to get the images using the API key and with a few tweaks. There is only one blocking request for it that is related to the last-minute added thumbnails.

catalog/dags/providers/provider_api_scripts/justtakeitfree.py

catalog/tests/dags/providers/provider_api_scripts/test_justtakeitfree.py

krysal · 2023-09-08T21:02:02Z

catalog/dags/providers/provider_api_scripts/justtakeitfree.py

+    def get_file_info(self, url) -> int | None:
+        """Get the image size in bytes."""
+        resp = self.delayed_requester.head(url)
+        if resp:
+            filesize = int(resp.headers.get("Content-Length", 0))
+            return filesize if filesize != 0 else None


Can we maybe get the sizes in the same/a similar way? 🤔

I couldn't find the sizes in any response 😞

krysal · 2023-09-08T21:05:09Z

catalog/dags/providers/provider_api_scripts/justtakeitfree.py

+Output:                 TSV file containing the media and the
+                        respective meta-data.
+
+Notes:                  https://justtakeitfree.com/api/api.php


Can we leave a note that it needs authentication? Just linking to the GH issue would help as well.

I looked at the other DAG notes, and we usually don't say whether the provider needs authentication or not. Do you think we should add it for every provider?
Do you suggest adding something like:

This API requires an API key. For more details, see #2793

Yes, I believe that would be helpful to know when running, especially because this comment gets displayed in the Airflow UI as well!

Co-authored-by: Krystle Salazar <[email protected]>

…eitfree.py Co-authored-by: Krystle Salazar <[email protected]>

Co-authored-by: Krystle Salazar <[email protected]>

krysal · 2023-09-11T15:20:55Z

catalog/dags/providers/provider_api_scripts/justtakeitfree.py

+Output:                 TSV file containing the media and the
+                        respective meta-data.
+
+Notes:                  https://justtakeitfree.com/api/api.php


Yes, I believe that would be helpful to know when running, especially because this comment gets displayed in the Airflow UI as well!

catalog/dags/providers/provider_api_scripts/justtakeitfree.py

catalog/DAGs.md

github-actions bot added the 🧱 stack: catalog Related to the catalog and Airflow DAGs label Aug 7, 2023

openverse-bot added 🟩 priority: low Low priority and doesn't need to be rushed 🌟 goal: addition Addition of new feature 💻 aspect: code Concerns the software code in the repository labels Aug 7, 2023

obulat force-pushed the add/just_take_it_free branch from 4cda04e to 68a860f Compare August 7, 2023 17:13

obulat force-pushed the add/just_take_it_free branch from 7178faa to c149723 Compare August 18, 2023 15:31

obulat self-assigned this Aug 23, 2023

obulat force-pushed the add/just_take_it_free branch 2 times, most recently from 50b6f82 to 4fc5a90 Compare August 30, 2023 03:56

obulat force-pushed the add/just_take_it_free branch from 4fc5a90 to e2a8a7b Compare September 5, 2023 10:04

obulat marked this pull request as ready for review September 5, 2023 10:13

obulat requested a review from a team as a code owner September 5, 2023 10:13

obulat requested review from krysal and stacimc September 5, 2023 10:13

obulat force-pushed the add/just_take_it_free branch from e2a8a7b to e94fbea Compare September 5, 2023 10:13

stacimc approved these changes Sep 6, 2023

View reviewed changes

catalog/dags/providers/provider_workflows.py Show resolved Hide resolved

catalog/tests/dags/providers/provider_api_scripts/test_justtakeitfree.py Outdated Show resolved Hide resolved

krysal requested changes Sep 8, 2023

View reviewed changes

obulat and others added 8 commits September 11, 2023 15:27

Add justtakeitfree as an image provider

7d44c53

Add DAG docs

0aa79e3

Add thumbnail

7a38672

Remove comments from test

e55ba81

Add start date for documentation purposes

cd27282

Update catalog/dags/providers/provider_api_scripts/justtakeitfree.py

96ce13f

Co-authored-by: Krystle Salazar <[email protected]>

Update catalog/tests/dags/providers/provider_api_scripts/test_justtak…

e332627

…eitfree.py Co-authored-by: Krystle Salazar <[email protected]>

Update catalog/dags/providers/provider_api_scripts/justtakeitfree.py

8b90085

Co-authored-by: Krystle Salazar <[email protected]>

obulat force-pushed the add/just_take_it_free branch from b817704 to 8b90085 Compare September 11, 2023 12:27

obulat requested a review from krysal September 11, 2023 12:36

krysal approved these changes Sep 11, 2023

View reviewed changes

obulat commented Sep 12, 2023

View reviewed changes

catalog/dags/providers/provider_api_scripts/justtakeitfree.py Show resolved Hide resolved

Update catalog/dags/providers/provider_api_scripts/justtakeitfree.py

efe5c55

obulat commented Sep 12, 2023

View reviewed changes

catalog/DAGs.md Outdated Show resolved Hide resolved

Update catalog/DAGs.md

47293e5

obulat merged commit 5b9db79 into main Sep 12, 2023
42 checks passed

obulat deleted the add/just_take_it_free branch September 12, 2023 04:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provider: justtakeitfree.com #2793

Provider: justtakeitfree.com #2793

obulat commented Aug 7, 2023 •

edited

Loading

krysal commented Aug 31, 2023

stacimc left a comment

obulat commented Sep 8, 2023

krysal left a comment

krysal Sep 8, 2023

obulat Sep 9, 2023

krysal Sep 8, 2023

obulat Sep 11, 2023 •

edited

Loading

krysal Sep 11, 2023

krysal Sep 11, 2023

Provider: justtakeitfree.com #2793

Provider: justtakeitfree.com #2793

Conversation

obulat commented Aug 7, 2023 • edited Loading

Fixes

Description

Testing Instructions

Checklist

Developer Certificate of Origin

krysal commented Aug 31, 2023

stacimc left a comment

Choose a reason for hiding this comment

obulat commented Sep 8, 2023

krysal left a comment

Choose a reason for hiding this comment

krysal Sep 8, 2023

Choose a reason for hiding this comment

obulat Sep 9, 2023

Choose a reason for hiding this comment

krysal Sep 8, 2023

Choose a reason for hiding this comment

obulat Sep 11, 2023 • edited Loading

Choose a reason for hiding this comment

krysal Sep 11, 2023

Choose a reason for hiding this comment

krysal Sep 11, 2023

Choose a reason for hiding this comment

obulat commented Aug 7, 2023 •

edited

Loading

obulat Sep 11, 2023 •

edited

Loading