GCS loader fills hardisk causing dataflow job crash.

[jrluis]

Hi,

While trying to load 150GB of events from PubSub to GCS the hard disk fills up causing the job to crash.

The error seems to be on level db. This db is used by dataflow to store information locally as is being pulled from the datasource.

The error message is:

E 2019-09-05T07:08:37.383218Z Opening leveldb /windmill/gcs-loader-for-enrichment-09040756-trux-harness-disk-9/S0-b333333333333000-ccccccccccccd000 failed with error: IO error: /windmill/gcs-loader-for-enrichment-09040756-trux-harness-disk-9/S0-b333333333333000-ccccccccccccd000/000007.ldb: No space left on device. Retrying in 1s. 
E 2019-09-05T07:08:37.383218Z Opening leveldb /windmill/gcs-loader-for-enrichment-09040756-trux-harness-disk-9/S0-b333333333333000-ccccccccccccd000 failed with error: IO error: /windmill/gcs-loader-for-enrichment-09040756-trux-harness-disk-9/S0-b333333333333000-ccccccccccccd000/000007.ldb: No space left on device. Retrying in 1s. 
E 2019-09-05T07:08:37.383218Z Opening leveldb /windmill/gcs-loader-for-enrichment-09040756-trux-harness-disk-9/S0-b333333333333000-ccccccccccccd000 failed with error: IO error: /windmill/gcs-loader-for-enrichment-09040756-trux-harness-disk-9/S0-b333333333333000-ccccccccccccd000/000007.ldb: No space left on device. Retrying in 1s. 
I 2019-09-05T07:08:37.383319Z E0905 07:08:37.383218      22 persistent_state_leveldb.cc:1734] Opening leveldb /windmill/gcs-loader-for-enrichment-09040756-trux-harness-disk-9/S0-b333333333333000-ccccccccccccd000 failed with error: IO error: /windmill/gcs-loader-for-enrichment-09040756-trux-harness-disk-9/S0-b333333333333000-ccccccccccccd000/000007.ldb: No space left on device. Retrying in 1s. 
I 2019-09-05T07:08:38.383379Z Opening leveldb /windmill/gcs-loader-for-enrichment-09040756-trux-harness-disk-9/S0-b333333333333000-ccccccccccccd000 for S0 [ b333333333333000_ .. ccccccccccccd000_ ) 

Regards