Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce transform cannot handle keys with spaces #21983

Closed
dekelpilli opened this issue Dec 9, 2024 · 1 comment · Fixed by #21989
Closed

Reduce transform cannot handle keys with spaces #21983

dekelpilli opened this issue Dec 9, 2024 · 1 comment · Fixed by #21989
Labels
meta: regression This issue represents a regression transform: reduce Anything `reduce` transform related type: bug A code related bug.

Comments

@dekelpilli
Copy link
Contributor

dekelpilli commented Dec 9, 2024

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

If a log comes in with an (unquoted) key with a space, it will result in an error. With the below config:

Input:

{"a b":1}
{"a b":2}

Console output:

2024-12-09T02:04:43.802329Z ERROR transform{component_kind="transform" component_id=red component_type=reduce}: vector::internal_events::reduce: Event field could not be reduced. path=KeyString("a b") error=InvalidPathSyntax { path: "a b" } error_type="condition_failed" stage="processing" internal_log_rate_limit=true
2024-12-09T02:04:43.802549Z ERROR transform{component_kind="transform" component_id=red component_type=reduce}: vector::internal_events::reduce: Internal log [Event field could not be reduced.] is being suppressed to avoid flooding.
{"path":"/","source_type":"http_server","timestamp":"2024-12-09T02:04:43.802069Z","timestamp_end":"2024-12-09T02:04:43.802084Z"}

This does not happen if the key comes in quoted. It is worth noting that logs that come in with quoted keys have those keys unquoted. I'm not sure if this is an intended consequence of #21201, but I don't believe this behaviour is documented. Regardless, this does "fix" the space issue. With the below config:
Input:

{"\"a b\"":1}
{"\"a b\"":2}

Output:

{"a b":3,"path":"/","source_type":"http_server","timestamp":"2024-12-09T02:06:58.705281Z","timestamp_end":"2024-12-09T02:06:58.705292Z"}

Note that the output a b key is not quoted.

Configuration

{
  "sources": {
    "m": {
      "type": "http",
      "acknowledgements": false,
      "address": "0.0.0.0:1234",
      "encoding": "ndjson",
      "inputs": []
    }
  },
  "transforms": {
    "red": {
      "type": "reduce",
      "inputs": ["m"],
      "expire_after_ms": 1000
    }
  },
  "sinks": {
    "serr": {
      "encoding": {
        "codec": "json"
      },
      "inputs": ["red"],
      "target": "stderr",
      "type": "console"
    }
  }
}

Version

0.43.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

@dekelpilli dekelpilli added the type: bug A code related bug. label Dec 9, 2024
@jszwedko jszwedko added the transform: reduce Anything `reduce` transform related label Dec 9, 2024
@pront pront added this to the Vector v0.43.1 milestone Dec 9, 2024
@pront
Copy link
Member

pront commented Dec 9, 2024

I think this is a regression from #21323. But it is just a misconfiguration issue, the functionality is there already. I am working on #21989 to fix this.

@pront pront added the meta: regression This issue represents a regression label Dec 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
meta: regression This issue represents a regression transform: reduce Anything `reduce` transform related type: bug A code related bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants