No diff when adding packages to pypi_packages of gcp.composer.Environment

[ringods]

What happened?

After adding an additional package to the pypi_packages list of a gcp.composer.Environment resource, this resource is marked for Update on pulumi up, but no diff is shown for this property. The change is applied correctly though.

Example

Start with this sample program:

import pulumi
import pulumi_gcp as gcp

gcp_config = pulumi.Config('gcp')
config = pulumi.Config()

service_account = gcp.serviceaccount.Account(
    "my-service-account",
    args=gcp.serviceaccount.AccountArgs(
        account_id="mycomposer", display_name="Service Account for Composer nodes"
    ),
)

permissions_composer_worker = gcp.projects.IAMMember(
    "my-composer-node-permissions",
    args=gcp.projects.IAMMemberArgs(
        project=gcp_config.require('project'),
        member=pulumi.Output.concat("serviceAccount:", service_account.email),
        role="roles/composer.worker",
    ),
)

composer_service_account = gcp.serviceaccount.IAMMember(
    "my-composer-service-account",
    args=gcp.serviceaccount.IAMMemberArgs(
        service_account_id=service_account.id,
        role="roles/composer.ServiceAgentV2Ext",
        member=f"serviceAccount:service-{config.require('projectNumber')}@cloudcomposer-accounts.iam.gserviceaccount.com",
    ),
)

# Create a GCP Composer Environment that includes 'dqops' as an additional PyPI package
composer_environment = gcp.composer.Environment(
    "my-composer-environment",
    args=gcp.composer.EnvironmentArgs(
        config=gcp.composer.EnvironmentConfigArgs(
            software_config=gcp.composer.EnvironmentConfigSoftwareConfigArgs(
                # Include 'dqops' in the list of PyPI packages to be installed in the environment
                pypi_packages={
                    "dqops": "",
                    # "datachart": ""
                },
            ),
            # Define the machine type and other node configurations here
            node_config=gcp.composer.EnvironmentConfigNodeConfigArgs(
                service_account=composer_service_account.service_account_id
            ),
        )
    ),
)

pulumi.export("composer_environment_name", composer_environment.name)
pulumi.export(
    "composer_environment_airflow_uri", composer_environment.config.airflow_uri
)

Define the following stack config:

config:
  gcp:project: "<your-project-name>"
  gcp:region: <your preferred region>
  projectNumber: "<your-project-number>" # Number must match to the project name above

Run pulumi up and have an initial Composer (Apache Airflow) environment.

Now uncomment the second package so the pypi_packages list reads like this:

                pypi_packages={
                    "dqops": "",
                    "datachart": ""
                },

At preview time, the following details are reported:

  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:ringo::4725::pulumi:pulumi:Stack::4725-ringo]
    ~ gcp:composer/environment:Environment: (update)
        [id=projects/<your-project-name>/locations/europe-west1/environments/my-composer-environment-0bd68a2]
        [urn=urn:pulumi:ringo::4725::gcp:composer/environment:Environment::my-composer-environment]
        [provider=urn:pulumi:ringo::4725::pulumi:providers:gcp::default_7_9_0::53525f9e-1093-421f-b296-eca7f1d1eb39]
        config    : {
            nodeConfig    : {
                serviceAccount: "projects/<your-project-name>/serviceAccounts/mycomposer@<your-project-name>.iam.gserviceaccount.com"
            }
            softwareConfig: {
                pypiPackages: {
                    dqops: ""
                }
            }
        }
        name      : "my-composer-environment-0bd68a2"

The second package that is added to the sources but is not shown as an addition in the diff, but when applying the changes, the second package is added to the setup.

Output of pulumi about

pulumi about
CLI          
Version      3.106.0
Go Version   go1.22.0
Go Compiler  gc

Plugins
NAME    VERSION
gcp     7.9.0
python  unknown

Host     
OS       darwin
Version  14.2.1
Arch     arm64

This project is written in python: executable='/Users/ringods/Projects/pulumi-customers/4725/venv/bin/python3' version='3.11.7'

Current Stack: team-ce/4725/ringo

TYPE                                    URN
pulumi:pulumi:Stack                     urn:pulumi:ringo::4725::pulumi:pulumi:Stack::4725-ringo
pulumi:providers:gcp                    urn:pulumi:ringo::4725::pulumi:providers:gcp::default_7_9_0
gcp:serviceaccount/account:Account      urn:pulumi:ringo::4725::gcp:serviceaccount/account:Account::my-service-account
gcp:serviceaccount/iAMMember:IAMMember  urn:pulumi:ringo::4725::gcp:serviceaccount/iAMMember:IAMMember::my-composer-service-account
gcp:projects/iAMMember:IAMMember        urn:pulumi:ringo::4725::gcp:projects/iAMMember:IAMMember::my-composer-node-permissions
gcp:composer/environment:Environment    urn:pulumi:ringo::4725::gcp:composer/environment:Environment::my-composer-environment


Found no pending operations associated with team-ce/ringo

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/v-ringo-pulumi-corp
User           v-ringo-pulumi-corp
Organizations  v-ringo-pulumi-corp, team-ce
Token type     personal

Dependencies:
NAME        VERSION
pip         24.0
pulumi_gcp  7.9.0
setuptools  69.1.0
wheel       0.42.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[iwahbe]

Hey @ringods. Thanks for letting us know.

[VenelinMartinov]

I suspect this is fixed due to pulumi/pulumi-terraform-bridge#1785 but have been unable to run the repro. It keeps failing with:

  gcp:composer:Environment (my-composer-environment):
    error:   sdk-v2/provider2.go:509: sdk.helper_schema: googleapi: Error 400: Composer API Service Agent service account (service-921927215178@cloudcomposer-accounts.iam.gserviceaccount.com) does not have required permissions set. Cloud Composer API Service Agent role is missing. Please bind roles/composer.serviceAgent back on service-921927215178@cloudcomposer-accounts.iam.gserviceaccount.com., failedPrecondition: [email protected]
    error: 1 error occurred:
    	* googleapi: Error 400: Composer API Service Agent service account (service-921927215178@cloudcomposer-accounts.iam.gserviceaccount.com) does not have required permissions set. Cloud Composer API Service Agent role is missing. Please bind roles/composer.serviceAgent back on service-921927215178@cloudcomposer-accounts.iam.gserviceaccount.com., failedPrecondition

@ringods can you please retry with the newest GCP and let us know if this is still an issue?

[VenelinMartinov]

@ringods has confirmed this is no longer an issue with v7