fix: Search bots behavior

[bprusinowski]

Fixes #1676

It looks like we had a legacy environmental variable – ALLOW_SEARCH_BOTS – that was not passed via the Dockerfile and couldn't be picked up during build. The exact rule we had in the code was "if ALLOW_SEARCH_BOTS is not 'true'", then disable the indexing, which resulted in the problem described in #1676.

I think it makes more sense to enable indexing by default, and disable it conditionally; that's why there's a new, PREVENT_SEARCH_BOTS variable to disable this behavior where needed.

Todo

• Add PREVENT_SEARCH_BOTS=true env variable to Vercel
• Add PREVENT_SEARCH_BOTS=true env variable to GitLab TEST pipeline
• (after PR is approved): Reach out to Abraxas to add PREVENT_SEARCH_BOTS=true variable to INT pipeline (see ticket here)
• PROD doesn't need a variable at all, once we deploy the behavior should be right due to reversed condition

cc @adintegra

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
visualization-tool	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 6, 2024 1:35pm

[noahonyejese]

LGTM 👍