Skip to content

Commit

Permalink
es-feed: ignore invalid files
Browse files Browse the repository at this point in the history
We currently have about 1% (28 out of 2756) of our build logs that have
invalid JSON files. They are all about a `-profile` file being
incomplete, and since those files represent a single JSON object we
can't do smarter things like filtering invalid individual lines.

I haven't looked deeply into _why_ we create invalid files, but this
should let our ingestion process make some progress in the meantime.

CHANGELOG_BEGIN
CHANGELOG_END
  • Loading branch information
garyverhaegen-da committed Jul 7, 2021
1 parent c92c678 commit d241f2a
Showing 1 changed file with 16 additions and 11 deletions.
27 changes: 16 additions & 11 deletions infra/es_cluster.tf
Original file line number Diff line number Diff line change
Expand Up @@ -797,24 +797,29 @@ bulk_upload() {
}
push() {
local job
local job f
job="$1"
for cmd in "build" "test"; do
e="$job/$cmd-events.json"
if [[ -f "$e" ]]; then
echo "$job: pushing $cmd-events.json"
emit_build_events "$job" "$cmd" "$e" | bulk_upload "$job"
else
f="$job/$cmd-events.json"
if ! [[ -f "$f" ]]; then
echo "$job: no $cmd-events.json"
elif ! jq . >/dev/null 2>&1 < $f; then
echo "$job: $cmd-events.json exists but is not valid json, skipping"
else
echo "$job: pushing $cmd-events.json"
emit_build_events "$job" "$cmd" "$f" | bulk_upload "$job"
fi
p="$job/$cmd-profile.json"
if [[ -f "$p" ]]; then
echo "$job: pushing $cmd-trace.json"
emit_trace_events "$job" "$cmd" "$p" | bulk_upload "$job"
else
f="$job/$cmd-profile.json"
if ! [[ -f "$f" ]]; then
echo "$job: no $cmd-profile.json"
elif ! jq . >/dev/null 2>&1 < $f; then
echo "$job: $cmd-profile.json exists but is not valid json, skipping"
else
echo "$job: pushing $cmd-profile.json"
emit_trace_events "$job" "$cmd" "$f" | bulk_upload "$job"
else
fi
done
}
Expand Down

0 comments on commit d241f2a

Please sign in to comment.