Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scala Common Enrich: update CloudfrontAccessLogAdapter to support newer 26 field format #3816

Closed
mdemri opened this issue Jun 20, 2018 · 2 comments
Assignees

Comments

@mdemri
Copy link

mdemri commented Jun 20, 2018

While running EMR ETL runner with Cloudfront collector,
The enrichment phase could not parse the logs, since it contains 26 fields.

Log file format actually contains 26 fields, https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html#LogFileFormat

But the common enrich code doesn't support it (supports only 24 fields)

The last two fields in the CF logs are redundent for this case.

@mdemri mdemri changed the title Spark enrich failed while processing Cloudfront Logs Spark (common) enrich failed while processing Cloudfront Logs Jun 20, 2018
@alexanderdean alexanderdean changed the title Spark (common) enrich failed while processing Cloudfront Logs Scala Common Enrich: update CloudfrontAccessLogAdapter to support newer 26 field format Jun 20, 2018
@alexanderdean
Copy link
Member

Updated ticket name as there is a newer format. But @mdemri - you are using the wrong configuration setting for the CloudFront collector. Please raise a Discourse thread and we can help:

https://discourse.snowplowanalytics.com/

@BenFradet
Copy link
Contributor

Thanks for the report @mdemri 👍 , feel free to open a PR 👍

@BenFradet BenFradet self-assigned this Jul 9, 2018
BenFradet added a commit that referenced this issue Jul 13, 2018
BenFradet added a commit that referenced this issue Jul 31, 2018
BenFradet added a commit that referenced this issue Jul 31, 2018
BenFradet added a commit that referenced this issue Aug 8, 2018
BenFradet added a commit that referenced this issue Aug 14, 2018
BenFradet added a commit that referenced this issue Aug 14, 2018
BenFradet added a commit that referenced this issue Aug 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants