Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataproc module cleanup & fixes #2141

Merged
merged 6 commits into from
Mar 11, 2024
Merged

Dataproc module cleanup & fixes #2141

merged 6 commits into from
Mar 11, 2024

Conversation

wiktorn
Copy link
Collaborator

@wiktorn wiktorn commented Mar 8, 2024

  • added end-to-end tests for dataproc module, regrettably - GCE example proved to be flaky in tests - so it is disabled (the depends_on is not always enough for the cluster creation to complete successfully, sometimes manual retry is required)
  • included in the examples necessary components (firewall rules, permissions)
  • added fixture for GKE cluster
  • removed unused variable service_account - properly managing service account within module is left as excessive for future reader / self
  • removed variable prefix as it doesn't make much sense in this context
  • removed output variables as due to bugs in the terraform provider they are not returned and result in terraform plan errors after apply

Error on GCE dataproc cluster apply:

│ Error: Error waiting for creating Dataproc cluster: Error code 13, message: 
│ Failed to initialize node my-cluster-m: Failed to create event log dir with 403 HTTP response code and 22 exit code.  Please check if Service Account has permission to create directory in dataproc-temp-*** GCS bucket. See output in: gs://***/google-cloud-dataproc-metainfo/d12144b5-b1b5-4b60-8cea-26ac5faa941f/my-cluster-m/dataproc-startup-script_output
│ 
│   with module.processing-dp-cluster.google_dataproc_cluster.cluster,
│   on ***/cloud-foundation-fabric/modules/dataproc/main.tf line 19, in resource "google_dataproc_cluster" "cluster":
│   19: resource "google_dataproc_cluster" "cluster" {

Second apply (recreates cluster) works without error.

E2E Run

See: #2127


Checklist

I applicable, I acknowledge that I have:

  • Read the contributing guide
  • Ran terraform fmt on all modified files
  • Regenerated the relevant README.md files using tools/tfdoc.py
  • Made sure all relevant tests pass

@wiktorn wiktorn force-pushed the wiktorn-dataproc-fixes branch from d5e0248 to 1d2255a Compare March 8, 2024 19:02
@wiktorn wiktorn requested a review from lcaggio March 8, 2024 19:10
Copy link
Collaborator

@ludoo ludoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work Wiktor :)

@wiktorn wiktorn enabled auto-merge (rebase) March 11, 2024 09:54
@wiktorn wiktorn merged commit 7753b1f into master Mar 11, 2024
13 checks passed
@wiktorn wiktorn deleted the wiktorn-dataproc-fixes branch March 11, 2024 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants