[FEA] JSON parsing is not handling escaped single quote the same as Spark

[revans2]

Is your feature request related to a problem? Please describe.
This is really odd and documented at NVIDIA/spark-rapids#10596

But essentially if we want to enable normalization of single quoted values, we also then want to allow backslash escaping of single quoted values.

I am fine if we end up with a separate config for this, but it does make thinks a lot more difficult. Especially because this fits in with the validation too

#15222

[GregoryKimball]

Thank you @revans2 for documenting this. I'm a bit surprised that libcudf is returning a null for this string.

"""{"data": "\'TESTING\'"}"""

I would expect this to be valid with and without single-quote normalization. Would you please help me understand what I'm missing?

[shrshi]

Hello @revans2, escaping of double quotes is expected behaviour when single quote normalization is enabled to handle the case of double quotes being present within single quotes. For example

{'TE"ST'} -> {"TE\"ST"}

If there are escaped single quotes within a single quoted string in the input, then the current behaviour of the quote normalization FST is to remove the backslash escapes since we no longer need them. For example

{'TE\'ST'} -> {"TE'ST"}

If I understand correctly, the quote normalizer is expected to retain the backslash i.e. {'TE\'ST'} -> {"TE\'ST"}
Is this correct?

[revans2]

@shrshi

I am sorry. I think I have been moving too quickly for my own good. I should have written some tests as repro cases for this before filing the issue.

I think there is a mismatch between the escape handling in the parser and the single quote pre-processing.

TEST_F(JsonReaderTest, EscapedSingleQuotes)
{
  const std::string fname = temp_env->get_temp_dir() + "EscapedSingleQuotes.json";
  std::ofstream outfile(fname, std::ofstream::out);
  outfile << "{\"data\": \"\\'TESTING\\'\"}\n{\"data\": \"'TESTING'\"}";
  outfile.close();

  cudf::io::json_reader_options in_options =
    cudf::io::json_reader_options::builder(cudf::io::source_info{fname})
      .lines(true)
      .recovery_mode(cudf::io::json_recovery_mode_t::RECOVER_WITH_NULL)
      .mixed_types_as_string(true)
      .normalize_whitespace(true)
      .keep_quotes(true)
      .normalize_single_quotes(false);

  cudf::io::table_with_metadata result = cudf::io::read_json(in_options);

  EXPECT_EQ(result.tbl->num_columns(), 1);
  EXPECT_EQ(result.tbl->num_rows(), 2);

  EXPECT_EQ(result.tbl->get_column(0).type().id(), cudf::type_id::STRING);
  EXPECT_EQ(result.metadata.schema_info[0].name, "data");

  cudf::test::print(result.tbl->get_column(0));

  cudf::test::strings_column_wrapper const strings(
      {"", "\"'TESTING'\""},
      {false, true});
  CUDF_TEST_EXPECT_COLUMNS_EQUAL(result.tbl->get_column(0), strings);
}

TEST_F(JsonReaderTest, EscapedSingleQuotesNormalize)
{
  const std::string fname = temp_env->get_temp_dir() + "EscapedSingleQuotesNormalize.json";
  std::ofstream outfile(fname, std::ofstream::out);
  outfile << "{\"data\": \"\\'TESTING\\'\"}\n{\"data\": \"'TESTING'\"}";
  outfile.close();

  cudf::io::json_reader_options in_options =
    cudf::io::json_reader_options::builder(cudf::io::source_info{fname})
      .lines(true)
      .recovery_mode(cudf::io::json_recovery_mode_t::RECOVER_WITH_NULL)
      .mixed_types_as_string(true)
      .normalize_whitespace(true)
      .keep_quotes(true)
      .normalize_single_quotes(true);

  cudf::io::table_with_metadata result = cudf::io::read_json(in_options);

  EXPECT_EQ(result.tbl->num_columns(), 1);
  EXPECT_EQ(result.tbl->num_rows(), 2);

  EXPECT_EQ(result.tbl->get_column(0).type().id(), cudf::type_id::STRING);
  EXPECT_EQ(result.metadata.schema_info[0].name, "data");

  cudf::test::print(result.tbl->get_column(0));

  cudf::test::strings_column_wrapper const strings{"\"'TESTING'\"", "\"'TESTING'\""};
  CUDF_TEST_EXPECT_COLUMNS_EQUAL(result.tbl->get_column(0), strings);
}

The first test passes, which is what Spark would expect to see when single quote support is disabled, but when it is enabled (which is the default in Spark) nothing changes. I think that the problem is that anything in double quotes is not being processed by the quote normalization code. {"TE\'ST"} -> {"TE\'ST"}, when what I think I want is {"TE\'ST"} -> {"TE'ST"}.

So this was a missed requirement on my part.

[shrshi]

Thank you for the detailed explanation, @revans2.
I've opened #15324 to address this.

[GregoryKimball]

I believe this is closed by #15324