Introduce project-level environment variables

[gtsiolis]

Problem to solve

The product currently supports only user-level environment variables set in user settings, where you use a scope syntax to specify for which projects the variable should be available to use. See /variables and relevant docs.

With the upcoming Teams & Projects features, it will be able to set up a project and share this project configuration with other team members under a team.

Proposal

Given there's also a plan to add project and team settings, it could make sense to add support for project-level environment variables or even team-level environment variables. This could also protect variables from being accidentally deleted or changed, and more as these could be read-only to non-owner members.

TBD (To be discussed)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ntziolis]

Any updates?

[gtsiolis]

Hey @ntziolis! FYI, We've added this feature request on the roadmap[1] for the next quarter. 🗺️

[ntziolis]

Great to hear. This is currently blocking us from adopting the gitpod way of doing things.

Since we cannot use prebuilds (private npm registry), each dev currently creates a single pinned workspace per project and reuses it. This is because creating a new workspace per issue would take too long without prebuilds.

[shaal]

+1 for this feature.
@gtsiolis Adding a team member (and giving them access to secret), happens through access to the repo on Github/Gitlab/Bitbucket? or I need to add a team member to the repo, and then additional steps in the Projects interface inside Gitpod website?

[ntziolis]

I would prefer to keep gitpod teams and github teams separate. We have third party devs that are working on code that doesn't need access to the secrets and we would like them to continue to not have access to them.

That said, I could see this being solved by

• making this auto-add configurable (meaning one can turn it off when this is not wanted)
• extending the gitpod browser extension that adds an additional button to the add team memeber section on github/gitlab etc (add to github and gitpod team)
• and/or via a notification to the gitpod admin whenever there is a new github team member added so he can take action

[AlCalzone]

It would already be immensely helpful to be able to define (public) env variables in .gitpod.yml without any UI whatsover.

Our repo is using yarn to manage dependencies. The repo is configured to use a global yarn cache, but this does not work with gitpod prebuilds because only /workspace is persisted.

So we're changing the yarn config in the prebuilds to use a local (persisted) cache, but without extra steps does not carry over to workspaces spawned from the prebuilds. To avoid changing the config files, we need to set environment variables for each workspace. This can be done with startup scripts, but is currently very repetitive. With two startup tasks, we need to do this:

tasks:
  - name: "Task 1"
    before: |
      # set env variables for this terminal
      export YARN_CACHE_FOLDER=".yarn/cache"
      export YARN_ENABLE_GLOBAL_CACHE="false"

      # Persist the env variables for other terminals
      echo "export YARN_CACHE_FOLDER=\".yarn/cache\"" >> ~/.bashrc
      echo "export YARN_ENABLE_GLOBAL_CACHE=\"false\"" >> ~/.bashrc
    # ... rest of task 1

  - name: "Task 2"
    openMode: split-right
    before: |
      # set env variables for this terminal
      export YARN_CACHE_FOLDER=".yarn/cache"
      export YARN_ENABLE_GLOBAL_CACHE="false"
    # ... rest of task 2

[SilvanCodes]

Hey @ntziolis! FYI, We've added this feature request on the roadmap[1] for the next quarter. 🗺️

Looking forward to see this land! 👍🏼

[louis030195]

+1 for this feature 👍

[flauc]

+1

[andreafalzetti]

I can't wait to have this feature 👀 I need to run prebuilds and be able to install private NPM packages

[XBeg9]

meanwhile, what will be the recommendation to use a private registry for prebuilds? is env vars injected into custom docker build phase?

[shaal]

@andreafalzetti @XBeg9 I am using my user environment variables during prebuild here -
https://github.com/shaal/DrupalPod/blob/main/.gitpod/drupal/envs-prep/prepare-environments-gcs.sh#L8

[XBeg9]

@shaal does it mean that every single user should put this env into config until this feature will be done?

Just out of curiosity, when prebuild triggers, specifically for team, which envs will be injected into the build pipeline?

[ntziolis]

@shaal Are you sure this works? @gtsiolis Does prebuild have access to user env variables? If so which user? the user that connected the repo? This could be a great workaround if thats the case. So far and based on my tests I could not confirm this.

[XBeg9]

@ntziolis same here, if envs are injected into prebuild phase even from project / team level, this is gonna be a security issue

[jgallucci32]

As a workaround try creating the file .gitpod/Dockerfile then specifying inside of it

FROM gitpod/workspace-full:latest
ENV YOURENV=yourvalue

Then in your .gitpod.yml file you would add this entry

image:
  file: .gitpod/Dockerfile

This will allow the env vars to be configured per-project and stored in source code for the project.

[XBeg9]

this way you are exposing the secret, right? @jgallucci32

[jgallucci32]

If you had sensitive data in the env var then yes. But the examples above were using all non-sensitive env vars so this workaround would be OK in those cases.

[jankeromnes]

Okay, we've decided to move forward with Project-level environment variables, and I'll build a first iteration.

We'll likely implement this as a new feature under a Project's settings, and worry about aligning or deprecating the current User-level variables later. (This way, we can make it 100% clear upfront where/when exactly the new Project-level variables are exposed, and there is no overlap with the current User-level variables, thus we avoid unexpected/surprise variable leaks by design.)

Type	Defined In	In Prebuilds?	In Workspaces?
User-level env vars	/variables	No	In user's workspaces
Project-level env vars	/:team/:project/variables	In all :project prebuilds	In all :project workspaces

Next up, I'll try to clarify the full Project-level env var access matrix, in order to make sure that we cover all desired use cases safely. (Spoilers: Both Gitpod Team membership and GitHub/GitLab/Bitbucket repository access can give you access to Project-level variables. This is likely okay for private repositories, but raises questions for public repositories, for which anyone can consume prebuilds.)

[shaal]

@jankeromnes Thank you for all the work on this!
Did I understand your last comment correctly (ie: GitHub/GitLab/Bitbucket repository access can give you access to Project-level variables) ?

If I have an open-source project, hosted on Github, that will use Gitpod's future project-level environment variables, anyone that open that project in Gitpod, will have access to the project's environment variables?

If that is true, would you consider changing it to give the access to project's environment variables, only to people with "write" access to that repo?
(so also Prebuilds can run, and in my case with DrupalPod, access Google Cloud Storage with secret keys)

[ntziolis]

Thank you for the update. I would suggest to make the following disabled by default.

GitHub/GitLab/Bitbucket repository access can give you access to Project-level variables

And i agree with @shaal, that if there is such implicit access to secrets it should generally be limited to folks that have a certain trust level. Write access to the source seems like the right path here.

[jankeromnes]

Thanks @shaal and @ntziolis for your feedback!

Did I understand your last comment correctly [...] If I have an open-source project, hosted on Github, that will use Gitpod's future project-level environment variables, anyone that open that project in Gitpod, will have access to the project's environment variables?

Yes, this is the current idea, but I'm happy to further discuss it so that we can work towards a better solution that addresses all use cases safely.

The arguments going for "project variables available in prebuild are also available in (any) workspace" go as follows:

• We want Prebuilds and Workspaces to remain consistent for the same Project/repository, i.e. your init tasks should give the same result in a Prebuild or in an interactive Workspace. However, if init relies on a variable that's present in Prebuild but not in Workspace, it might surprisingly fail in a Workspace.
• Prebuilds of public repositories are inherently accessible to the public (as currently designed). If you prebuild a public repo, anyone opening the same commit will get that prebuild. If the prebuild contains sensitive data (e.g. secret environment variables, or cached secrets from tools that authenticate via env vars, etc) you might accidentally "leak" them even if we explicitly don't include env variables from Prebuilds in Workspaces.

(so also Prebuilds can run, and in my case with DrupalPod, access Google Cloud Storage with secret keys)

I understand that "provide environment variables to prebuilds of a public repository without sharing them with users of that public repository" is a use case that isn't covered by our current design. (Note: this limitation only applies to public repositories -- for private repositories, you can only use them in Gitpod if you already have access to them, so environment variables are effectively only accessible to project members.)

Currently, we're here:

Env Variable Type	Team can read?	Team can write?	In Prebuilds?	In Workspaces?
User-level	No	No	No	Only for user
Project-level	No?	Yes	Yes	Yes?

However, thinking again, maybe "In Workspace?" could actually be a configuration option per Project-level environment variable. E.g. a checkbox when you create a new variable:

New Environment Variable for {Project}
--------------------------------------
Name: _________
Value: _________
Visible in Prebuilds: [x] (always checked + disabled)
Visible in Workspaces: [ ] (configurable)

But for consistency, if we allow hiding some Project variables from Workspaces, we also need to do a good job of protecting the value from "leaking" outside of Prebuilds. This means doing as good a job at it as other CI services, e.g. by doing:

• Prebuild log censoring (e.g. if you accidentally print your secret in the logs, we could automatically censor it before the logs get sent/displayed anywhere)
• Limiting who can trigger new Prebuilds for your Project (e.g. you may want to disable Prebuilds for pullRequestsFromForks -- luckily, that's already the default, but we'll need to verify it's indeed the default for all Git providers, and also make the security implications of enabling this very clear & visible)
• Or, more drastically, maybe we could allow restricting Prebuilds to only Team members... but this starts undoing the primary benefit of Prebuilds, which is making the workspace initialization faster for everyone

[jankeromnes]

However, thinking again, maybe "In Workspace?" could actually be a configuration option per Project-level environment variable. E.g. a checkbox when you create a new variable:

New Environment Variable for {Project}
--------------------------------------
Name: _________
Value: _________
Visible in Prebuilds: [x] (always checked + disabled)
Visible in Workspaces: [ ] (configurable)

Alternatively, a more fine-grained option than a checkbox to control the visibility of Project environment variables could be a visibility selector with options such as:

Who can access the value of this environment variable?
- Noone (accessible during Prebuilds only, censored out from Workspaces and Prebuild logs)
- Everyone at <TEAM> (accessible during Prebuilds and in Team Workspaces, censored out in non-Team Workspaces and Prebuild logs)
- Everyone on the Internet (accessible during Prebuilds, in all Workspaces, and visible in Prebuild logs)

However, for simplicity's sake, let's keep it as a on/off checkbox during initial implementation. This option could be a follow-up enhancement in case of strong interest.

[jankeromnes]

Update: This feature is now deployed! 🎉

Here is how to use it:

1. Navigate to your Project in Gitpod's dashboard
2. Go to Settings
3. Click on Variables in the left menu

The UI allows to define new environment variables that will be visible during your Project's Prebuilds. 📝

[jankeromnes]

Also FYI, I'll give a short demo of this feature in today's open office hours on Discord, in just under 1 hour. 🙂

[devxpy]

Can this be done via the cli? The UI is too hard to use for 20 env vars

[axonasif]

Hi @devxpy, not yet but could you please react with a " 👍 " on the following issue?

• Support altering project-level environment variables with gp env #12208

If the CLI support was there for projects, you could use such a method to quickly add env vars in bulk.

• Introduce gp command to add env from .env file #12243 (comment), and this of course is another feature request to have a built-in command instead of doing it ourselves.