[Ingest Pipelines] Invalid error shown for grok processor

[alisonelizabeth]

Kibana version:
7.16.3

Describe the bug:
User is shown the error "Invalid JSON string" after entering a valid grok pattern in the Ingest Pipelines UI. This bug appears to have been introduced via #94689.

Steps to reproduce:

1. Navigate to Ingest Pipelines --> Create pipeline
2. Add a grok processor with the following grok pattern:

%{IPORHOST:ip}%{SPACE}-%{SPACE}%{DATA:user_name}%{SPACE}\[%{HTTPDATE:timestamp}\]%{SPACE}\"%{WORD:http_method}%{SPACE}%{DATA:url}%{SPACE}HTTP/%{NUMBER:http_version}\"%{SPACE}%{NUMBER:response_code}%{SPACE}%{NUMBER:body_sent_bytes}%{SPACE}\"%{DATA:referrer}\"%{SPACE}\"%{DATA:agent}\"%{SPACE}\"%{IPORHOST:client_ip}\

3. Note the field error "Invalid JSON string" appears, even though this is a valid Grok pattern. This is caused by the \ character in the following part of the expression: \[%{HTTPDATE:timestamp}\].

Screenshots:

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[jloleysens]

The original thinking was that users are expected to edit a stringified string, they will be able to work with whitespace chars more easily in the context of the dissect processor. It looks like this same assumption does not need to hold for the grok processor. I think we can get the behavior we want with the following changes:

diff --git i/x-pack/plugins/ingest_pipelines/public/application/components/pipeline_editor/components/processor_form/processors/grok.tsx w/x-pack/plugins/ingest_pipelines/public/application/components/pipeline_editor/components/processor_form/processors/grok.tsx
index ae2d341c58c..7ff779e1398 100644
--- i/x-pack/plugins/ingest_pipelines/public/application/components/pipeline_editor/components/processor_form/processors/grok.tsx
+++ w/x-pack/plugins/ingest_pipelines/public/application/components/pipeline_editor/components/processor_form/processors/grok.tsx
@@ -49,7 +49,6 @@ const patternsValidation: ValidationFunc<any, string, ArrayItem[]> = ({ value, f
 
 const patternValidations: Array<ValidationFunc<any, string, string>> = [
   emptyField(valueRequiredMessage),
-  isJSONStringValidator,
 ];
 
 const fieldsConfig: FieldsConfig = {
@@ -58,8 +57,7 @@ const fieldsConfig: FieldsConfig = {
     label: i18n.translate('xpack.ingestPipelines.pipelineEditor.grokForm.patternsFieldLabel', {
       defaultMessage: 'Patterns',
     }),
-    deserializer: flow(String, to.escapeBackslashes),
-    serializer: from.unescapeBackslashes,
+    deserializer: String,
     helpText: i18n.translate('xpack.ingestPipelines.pipelineEditor.grokForm.patternsHelpText', {
       defaultMessage:
         'Grok expressions used to match and extract named capture groups. Uses the first matching expression.',

This should give us how the grokdebugger behaves.

[EDIT]
Did not test for all cases!

[biasettigrulex]

Hi All,
after several tests i found a way to bypass the issue.

I'm using this grok code and works:
%{IPORHOST:ip}%{SPACE}-%{SPACE}%{DATA:user_name}%{SPACE}.%{HTTPDATE:timestamp}.%{SPACE}\"%{WORD:http_method}%{SPACE}%{NOTSPACE:url_path}%{SPACE}%{SPACE}HTTP/%{NUMBER:http_version}\"%{SPACE}%{NUMBER:response_code}%{SPACE}%{NUMBER:body_sent_bytes}%{SPACE}\"%{NOTSPACE:referrer}\"%{SPACE}\"%{DATA:client_agent}\"%{SPACE}\"%{IPORHOST:client_ip}\"%{SPACE}%{GREEDYDATA:notparsed}

[JsonBlack09]

Hi All,
after several tests i found a way to bypass the issue. Look at [(] as escape method. I need to escape ( like \ (, but it not worked, common patterns like [(] - worked.

(?:-|PhoneCall: %{BASE16NUM:id}) - (?:-|%{WORD:answer.status}). BadRequest from Beeline: (?:-|%{WORD:error}) - (?:-|%{DATA:message}): (?< trash >[(])(?:-|%{INT:answer.code})(?< trash >[)]) (?:-|%{GREEDYDATA:answer.text}).

[bzarbalthazar]

Any movement on this? This still exists in 7.17.3. After testing in the Grok Parser in Dev Tools my grok pattern works, but in the ingest pipeline it breaks with this error.

[JsonBlack09]

Up

[cjcenizal]

@bzarbalthazar Could you please share the pattern you're using that's not working?

@GexogenSG1 Could you also please share a pattern that's being rejected by the Ingest Pipelines UI?

[koterin]

[

A glass of vine to this sir! Until this issue gets resolved, please use workarounds as advised above:

• Don't use spaces explicitly: only %{SPACE}
• Don't use symbols like [ even with escape characters. \[ won't work. Use . - as any single symbol

[alisonelizabeth]

Hi @koterin! This issue should have been resolved via #137245.

It is available in v8.3.4 and up.

[remixmabix]

Hello it's behaving the same in Agent Policy, processor block too (Nginx 1.7.0) or it is not related ?