Merge branch 'main' into soft_lock

api/api/admin/media_report.py

@@ -325,6 +325,7 @@ def changelist_view(self, request, extra_context=None):
             for item in lock_set
         )
 
+<<<<<<< soft_lock -- Incoming Change
         extra_context = extra_context or {}
         extra_context |= {"locked_reports": locked_reports}
 
@@ -338,6 +339,8 @@ def render_change_form(
             request, context, add=add, change=change, form_url=form_url, obj=obj
         )
 
+=======
+>>>>>>> main -- Current Change
 
 class ImageReportAdmin(MediaReportAdmin):
     media_type = "image"

catalog/justfile

@@ -34,7 +34,7 @@ export CATALOG_AIRFLOW_VERSION := `grep '^apache-airflow' requirements-prod.txt
 # Create a virtual environment using the project Python version
 venv:
     # Invokes `python<version>`, like `python3.10`
-    bash -c "python$CATALOG_PY_VERSION -m venv venv/"
+    bash -c "python$CATALOG_PY_VERSION -m venv .venv/"
 
 # Check that the active Python version matches the required Python version
 check-py-version:

documentation/catalog/guides/quickstart.md

@@ -52,8 +52,8 @@ To set up the local python environment along with the pre-commit hook, run:
 <!-- vale Vale.Repetition = NO -->
 
 ```shell
-python3 -m venv venv
-source venv/bin/activate
+just venv
+source .venv/bin/activate
 just catalog/install
 ```
 

documentation/changelogs/api/2024.05.23.15.02.00.md

@@ -0,0 +1,12 @@
+# 2024.05.23.15.02.00
+
+## Internal Improvements
+
+- Add 'revoked' field to ThrottledApplication to enable easily revoking access
+  to client applications violating openverse TOS
+  ([#4334](https://github.com/WordPress/openverse/pull/4334)) by @madewithkode
+
+## Bug Fixes
+
+- Change search query approach to include only available providers
+  ([#4238](https://github.com/WordPress/openverse/pull/4238)) by @krysal

documentation/meta/maintenance/elasticsearch_cluster.md

@@ -336,6 +336,69 @@ etc. The main focus should be understanding the new configuration and
 determining whether it's possible to test it locally. Otherwise, follow the
 steps for deploying a new cluster and switching the live services over to it.
 
+## Node Replacements
+
+Occasionally, the underlying EC2 instance for one of our nodes will undergo a
+scheduled decomission ("retirement") and need to be replaced beforehand.
+
+This procedure describes the process of replacing a single node, but could be
+extrapolated to multiple nodes with relative ease.
+
+If a data refresh is scheduled during the timeframe of this procedure, it is
+best to pause it for performance reasons.
+
+### Steps
+
+1. Use the Instance ID provided in the retirement notification to find the
+   "Private IPv4 address" of the instance. This value is the "Name" of the
+   ElasticSearch node.
+1. Identify where the node is stored in our Terraform state. This will be useful
+   for manual state changes later in the process. Check each node until you find
+   the matching private IP. In a terminal, navigate to the appropriate
+   environment directory and run:
+
+   ```bash
+   terraform <env> state show "module.<env>-elasticsearch-8-8-2.aws_instance.datanodes[0]
+   terraform <env> state show "module.<env>-elasticsearch-8-8-2.aws_instance.datanodes[1]
+   # ...continue as needed until the correct node is returned
+   ```
+
+   Replace the `<env>` with the correct name for the current environment. Record
+   the final `module.<env>-elasticsearch-8-8-2.aws_instance.datanodes[x]` index
+   for later use.
+
+1. In Terraform, increase the `data_node_count` for the relevant Elasticsearch
+   module by one. Here is an
+   [example commit](https://github.com/WordPress/openverse-infrastructure/pull/894/commits/4a827b786b1460aa89931d474db119d835784727)
+   with this change in production. Apply this change and wait for a new instance
+   to be provisioned and connected to the cluster. Record the "Public IPv4 DNS"
+   record of the instance for use in the next step.
+1. Configure the created instance by running our ansible playbook to sync
+   Elasticsearch nodes. Supply the correct environment name to the command and
+   pass the DNS record from the previous step to the limit flag like so:
+   `just ansible/playbook <env> elasticsearch/sync_config.yml -e apply=true -l <public_ipv4_dns>`
+1. Use `just jh es <env>` to connect to the cluster, and send
+   `{ "transient":{ "cluster.routing.allocation.exclude.name": "<IP_ADDRESS>" } }`
+   to the `/_cluster/settings` endpoint (in a GUI like Elasticvue or via `curl`)
+   to deallocate shards from the bad node. Be sure to replace `<IP_ADDRESS>`
+   with the private IPv4 address identified in step 1.
+1. The cluster will now relocate shards to the new node and from the retired
+   node. Wait for the cluster health to return to green.
+1. Manually terminate the retired instance in the AWS console.
+1. In Terraform:
+   - Decrease the `data_node_count` for the Elasticsearch module down by one. Do
+     not apply this yet.
+   - Remove the retired node from the Terraform state. This is a prerequisite to
+     moving the new node into the retired node’s position in the state. Use the
+     following command, replacing "x" with the index found in step 2.
+     `j tf prod state rm module.elasticsearch.aws_instance.elasticsearch-ec2-datanodes[x]`
+   - Move the newest node added in step 3 to the position of the retired node we
+     just removed. If the newest node was in position 3 and the old node was in
+     position 0, for example:
+     `j tf <env> state mv module.elasticsearch.aws_instance.elasticsearch-ec2-datanodes[3] module.elasticsearch.aws_instance.elasticsearch-ec2-datanodes[0]`
+1. Apply the Terraform changes, which will cleanup the dangling DNS records of
+   the retired node.
+
 ## Potential improvements
 
 - Pass node configuration at the module level rather than generating it inside