String generation from complex regex in integration tests

[thirtiseven]

Closes #8593
Closes #8607
Contributes to #8447

This PR :

• Fixed StringGen in integration tests to correctly generate long and complex strings from a regex. StringGen can't do this when the state space of a regex is very large, as described in [FEA] Generation of long strings properly with StringGen in integration tests #8593.

• Implemented a decimal generation function to speed up the generation of decimals that were previously generated by sre_yield.

• Updated _cache_repr of UniqueLongGen and RepeatSeqGen to make them simpler.

Some small changes in test cases that affected by the StringGen change:

1. In test_regexp_extract_all_idx_out_of_bounds, the error message is raised only if the given string matches the pattern, so it can't raise the error with mk_str_gen('[abcd]{0,3}') after StringGen changes because of bad luck. This behavior is matched between cpu and gpu, so I just change the gen to mk_str_gen('[a-d]{1,2}.{0,1}[0-9]{1,2}').

2. One regex used by test_json_tuple and test_get_json_object will generate numbers with leading zeros, which is not compatible between spark and plugin, as described in [BUG] Mismatch cases in json_tuple function when json items have leading zeroes #8607. I changed it to not generate such numbers.

3. When processing string to sql in _convert_to_sql, there is a edge case that if there is a \' in string before replace, the result will be r\\', which will make the ' escape. So I replaced all r\ with r\\ first to prevent it.

4. [UPDATED] test_hash_groupby_collect_set_on_nested_type and test_hash_reduction_collect_set_on_nested_type faile after this PR, but they also fail when selecting different seeds, as described in [BUG] test_hash_groupby_collect_set_on_nested_type and test_hash_reduction_collect_set_on_nested_type failed #8716. I don't think it's related to this PR, so I temporarily marked them as XFAIL.

[UPDATED] Some test results:
For DecimalGen, this change brings some performance improvement. I tested this one from current IT:

def test_special_decimal_division():
    for precision in range(1, 39):
        for scale in range(-3, precision + 1):
            print("PRECISION " + str(precision) + " SCALE " + str(scale))
            data_gen = DecimalGen(precision, scale)
            assert_gpu_and_cpu_are_equal_collect(
                    lambda spark : two_col_df(spark, data_gen, data_gen).select(
                        f.col('a') / f.col('b')))

Before this PR: 0:07:05, after this PR: 0:04:44

For StringGen, if we use ''.join(rand.choice(sre_yield.CHARSET + ['\n']) for _ in range(30)) to generate default strings, the speed is even slower, so I leave it unchanged. I tested this case:

@pytest.mark.parametrize('data_gen', [string_gen]*100, ids=idfn)
def test_string_gen_perf(data_gen):
    assert_gpu_and_cpu_are_equal_collect(
            lambda spark: unary_op_df(spark, data_gen))

Before this PR: 52.04s, First commit of this PR: 52.49s. Other string-related tests are also a bit slower. Thus, I will not close the sre_yield performance issue #8477.

The overall performance improvement will be small because it only affects DecimalGen.

Note

• Now the DecimalGen will only generating numbers with the given precision rather than (1, precision) because it may cause precision limitations in some cases, leading to CPU/GPU mismatch.

• test_str_to_map_expr_fixed_pattern_input and test_initcap are failed when using random() * length as index but passed now. I don't know why and I will investigate them later. Currently they don't block anything.

[thirtiseven]

build

[thirtiseven]

build

[revans2]

Why is this change being made? The original regexp was made to match closely with the regexp_extract_all pattern below.

[thirtiseven]

Sorry for the late reply.

The test is failed after this PR because the error message "Regex group count is 2, but the specified group index is 3" will only raised when data matches the pattern.

In the previous code, what triggers this error message is actually the '.{0,10}' as a special case in mk_str_gen. So the test was PASSED because of good luck.

Now I change the pattern to '[a-d]{1,2}.{0,1}[0-9]{1,2}' to make sure they can match the pattern below.

[thirtiseven]

build

[revans2]

Generally the change looks good. I just want to check that you changed the generation pattern to make it so it would always match regular expression below.