Add implementation plan for documenting the API media properties

[dhruvkb]

Fixes

Fixes #1996 by @obulat

Description

This PR adds the implementation plan for documenting the media properties in the API. The IP uses introspection as the primary approach along with a documentation generation plan similar to that for the catalog.

Reviewers

I have requested reviews from @obulat and @krysal. The reason for the selection is because @obulat was involved in the IP and implementation for the corresponding project at the catalog layer and because IIRC @krysal has done prior work in the area of media properties documentation.

Testing Instructions

Please read the IP and comment your opinions. You can also see the implementation of this plan at #3966.

Decision-making process

This discussion is following the Openverse decision-making process. Information about this process can be found on the Openverse documentation site. Requested reviewers or participants will be following this process. If you are being asked to give input on a specific detail, you do not need to familiarise yourself with the process and follow it.

Current round

This discussion is currently in the Clarification round. The deadline for review of this round is 5 April 2024.

Checklist

• My pull request has a descriptive title (not a vague title likeUpdate index.md).
• My pull request targets the default branch of the repository (main) or a parent feature branch.
• My commit messages follow best practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no visible errors.
• I ran the DAG documentation generator (if applicable).

Developer Certificate of Origin

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

[github-actions]

Full-stack documentation: https://docs.openverse.org/_preview/3965

Please note that GitHub pages takes a little time to deploy newly pushed code, if the links above don't work or you see old versions, wait 5 minutes and try again.

You can check the GitHub pages deployment action list to see the current status of the deployments.

New files ➕:

• https://docs.openverse.org/_preview/3965/projects/proposals/document_all_media_properties/20240313-implementation_plan_document_media_properties_api.html

Changed files 🔄:

• https://docs.openverse.org/_preview/3965/catalog/guides/quickstart.html
• https://docs.openverse.org/_preview/3965/frontend/guides/analytics.html
• https://docs.openverse.org/_preview/3965/frontend/reference/playwright_tests.html
• https://docs.openverse.org/_preview/3965/meta/maintenance/elasticsearch_cluster.html

[krysal]

I'm reviewing this today. In the meantime, @dhruvkb, could you add a note about the revision round, like in #3650, for example?

[krysal]

@dhruvkb This is looking like a solid document! I don't anticipate more questions or comments than the ones I left here.

[krysal]

Thanks for updating the details, and writing this comprehensive proposal! Everything looks good to me, so shortcuting the rounds, I approve it 👍

[obulat]

It's great that Django has all of the capabilities to allow for documenting the properties! Love the plan and the linked PR with documentation.

I would love to see more detail on the Future improvements point on jsonb fields, especially after the incident caused by #3930. Do you think this should be done in a separate project, or as a separate part of this project? Would we add separate serializers to allow us to use introspection, or is there another way available to auto-generate documentation?

I wouldn't consider the "Fully document media properties" project completed without documenting the JSONFields because they are the ones that can cause problems in the future.

[obulat]

Will this work with help_text that is located in a separate file and is added inside __init__, as the source ones?

openverse/api/api/serializers/media_serializers.py

Lines 310 to 311 in 12d7162

	self.fields["source"].help_text = SOURCE_HELP_TEXT.format(**variables)
	self.fields["excluded_source"].help_text = EXCLUDED_SOURCE_HELP_TEXT.format(

[dhruvkb]

These help texts are taken from the model and not from the serializer. That's because for a specific model a number of serializers may be defined that can have different help_texts as per the requirements of the endpoint.

So for source, we show this

openverse/api/api/models/media.py

Lines 56 to 58 in 12d7162

	help_text="The source of the data, meaning a particular dataset. "
	"Source and provider can be different. Eg: the Google Open "
	"Images dataset is source=openimages, but provider=flickr.",

[dhruvkb]

You can visit https://docs.openverse.org/_preview/3966/meta/media_properties/api.html to see the the generated docs output.

[dhruvkb]

I wouldn't consider the "Fully document media properties" project completed without documenting the JSONFields because they are the ones that can cause problems in the future.

I totally agree with this sentiment and I tried earnestly to include them. But the documentation of JSONFields in their current state poses a number of issues and the alternative of documenting them from the serializers produces documentation that is based on our "expectation" during serialization and not reality of the DB or models.

I have documented these issues in f9dd8c9 and noted some steps we can take to enforce validity in our JSON data, which can then be used to generate documentation.

[openverse-bot]

Based on the medium urgency of this PR, the following reviewers are being gently reminded to review this PR:

@obulat
This reminder is being automatically generated due to the urgency configuration.

Excluding weekend¹ days, this PR was ready for review 6 day(s) ago. PRs labelled with medium urgency are expected to be reviewed within 4 weekday(s)².

@dhruvkb, if this PR is not ready for a review, please draft it to prevent reviewers from getting further unnecessary pings.

Footnotes

1. Specifically, Saturday and Sunday. ↩

2. For the purpose of these reminders we treat Monday - Friday as weekdays. Please note that the operation that generates these reminders runs at midnight UTC on Monday - Friday. This means that depending on your timezone, you may be pinged outside of the expected range. ↩

[obulat]

Thank you for adding information on the json fields and possible solutions.
The plan looks good, and I'm short-circuiting this discussion to approve it.