built-in functions bug

[howan36]

Component(s)

processor/transform

Describe the issue you're reporting

expression: - set(attributes["code"], Substring(body,indexOf(body,"code"),indexOf(body,"message")))

error:
Error: invalid configuration: processors::transform: unable to parse OTTL statement "set(attributes["code"], Substring(body,indexOf(body,"code"),indexOf(body,"message")))": error while parsing arguments for call to "set": invalid argument at position 1: error while parsing arguments for call to "Substring": invalid argument at position 1: no value field set. This is a bug in the OpenTelemetry Transformation Language

[github-actions]

Pinging code owners:

• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bacherfl]

Hi @howan36!

To make this easier to reproduce, can you please share the full collector config yaml, along with an example input and expected outcome?
Also, which version of the collector are you using?

Using the provided example statement, i do get an error as well, however a different one, since the indexOf function used in the example is not part of OTTL - when replacing that with int literals, the Substring function was parsed correctly

[github-actions]

Pinging code owners for pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[howan36]

transform:
error_mode: ignore
log_statements:

- context: log

      #   conditions:
      #     - IsString(body) == true
      #   statements:
      #     # - set(attributes["trace.id"], Substring(body,body.indexOf("traceId"),body.indexOf("spanId")))
      #     - set(attributes["code"], Substring(body,indexOf(body,"code"),indexOf(body,"message")))

[howan36]

i saw ottl support to use this golang built-in function?
collector version: 0.107.0

[howan36]

the int literals you mean like 2,3 numbers? or if you some other method to have the same function

[bacherfl]

@howan36 just checked again - indexOf does not seem to be a supported function in OTTL. However, can you post an example of a log record body? Then I will try to find an alternative way to extract the code as an attribute

[howan36]

response:
--------------------- request end ---------------------------->
200 http://sssss.xxxxx.svc.cluster.local/v2.0/ciam/xxxx/get/profiles/by/partyUidList

{"code":"0","message":null,"data":[{"customerId":xxxxx,"customerName":null,"gender":0,"phone":"xxxxxx","email":null,"birthYear":null,"birthMonth":null,"birthDay":null,"partyUid":"Cxxxxxxx","customerAddress":null,"wechatOpenId":"oPq0xxxxxxx","wechatUnionid":"ombktxxxxx","customerStatus":"AC","preferStoreName":"xxxxx","preferStoreCode":"549","registerStoreName":"xxxxx","registerStoreCode":"549","registerSource":"{"content":"SHOPPABLE_MINI_P="}","ctgProvinceId":null,"ctgProvinceName":null,"ctgCityId":null,"ctgCityName":null,"ctgDistrictId":null,"ctgDistrictName":null,"ctgSubDistrictId":0,"ctgSubDistrictName":null,"postalCode":null,"idCardNumber":null,"registerTime":1730166345903,"childStatus":null,"cdmCustomerChildDtoList":[],"cdmCustomerPreferenceDto":{"decorationPlan":null,"decorationPlanFlag":null,"outboundSmsFlag":true},"ikeaFamilySpaceIdList":[],"cdmMembershipDto":{"cardNumber":"627598033xxxxx5","wechatCardCode":null,"wechatOfficialAccountOpenId":null},"cardNumber":"6275xxxxxxxx","wechatCardCode":null,"wechatOfficialAccountOpenId":null,"houseOwnerShip":null,"livingStatus":{"status":null,"livingStatusList":[]},"migrationStatus":"N/A"}]}

[howan36]

it's unstructed logs, for example, i want to replace the phone field with ****

[howan36]

and maybe i will also need to extract its value to attributes

[bacherfl]

thanks for the example, I will try to see if there is a way of handling this kind of unstructured log data

[bacherfl]

@howan36 maybe something like the following config might be of help to handle unstructured log data like from the example you posted above:

extensions:
  health_check:
    endpoint: ${env:MY_POD_IP}:13133
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  batch:
    timeout: 10s
  transform:
    log_statements:
      - context: log
        statements:
          - set(attributes["parsedBody"], Split(body, "\n"))
          - set(attributes["parsedBody"], ParseJSON(attributes["parsedBody"][4]))
          - set(attributes["code"], attributes["parsedBody"]["code"])

exporters:
  debug:
    verbosity: detailed

service:
  extensions:
    - health_check
  pipelines:
    logs:
      receivers: [otlp]
      processors:
        - transform
      exporters: [debug]

This config first splits the log body lines into a slice, and then takes the line containing the json object and passes that line to the ParseJSON function. From there on, properties such as code or phone can be accessed and modified.
This config does make some assumptions regarding the format of the log entries though, so in this case it relies on the json payload to be in line 4.

[howan36]

hi @bacherfl , thanks a lot for your response, so the functions that transform processor supported are all in this page:https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs, and other go-lang built-in functions won't be supported?

[bacherfl]

hi @bacherfl , thanks a lot for your response, so the functions that transform processor supported are all in this page:https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs, and other go-lang built-in functions won't be supported?

Exactly, the functions listed in those docs are the ones that are supported. Executing Golang functions directly is not possible

[evan-bradley]

@howan36 Would the ExtractPatterns function work for your use case?