Kafka IO fixes

[tims]

fixes for #22

[tims]

@baskaranz

[baskaranz]

@tims The GCP credentials is not required to be set explicitly when using Flink Runner? I know Dataflow Runner wouldn't need it explicitly but I wasn't sure If Flink Runner would work without setting it that way.

[tims]

No it shouldn't. Anything that makes use of GCP resources like BQ or GCS should pick up credentials from the environment automatically.

For Flink in particular, we expect it's mostly likely to be used when you aren't using GCP at all. So in that case you definitely don't need to set it.

[tims]

@baskaranz @woop can you review this when you get a chance so we can merge?

[baskaranz]

@tims Looks good from my end

[baskaranz]

@tims

Thought of checking on earlier comment to make myself clear

For Flink in particular, we expect it's mostly likely to be used when you aren't using GCP at all. So in that case you definitely don't need to set it.

The use case we would be using flink-runner is, when ingesting data from kafka topic(s) to BQ warehouse store and redis serving store, so flink-runner wouldn't need (or use) the GCP credentials to write to BQ in this scenario?

[woop]

lgtm

[tims]

@tims

Thought of checking on earlier comment to make myself clear

For Flink in particular, we expect it's mostly likely to be used when you aren't using GCP at all. So in that case you definitely don't need to set it.

The use case we would be using flink-runner is, when ingesting data from kafka topic(s) to BQ warehouse store and redis serving store, so flink-runner wouldn't need (or use) the GCP credentials to write to BQ in this scenario?

If you want to write to BQ as a warehouse then yes you would need gcp credentials.

I think my answer was confusing. What I meant was, you don't need to explicitly call getApplicationCredentials, because all the GCP Beam IOs already do that.

They implicitly make of the class GcpCredentialFactory .