Add IMDB(JOB) Benchmark [2/N] (imdb queries)

[austin362667]

Which issue does this PR close?

Partially closes #12311.

1. After generating IMDB dataset (.csv, .parquet).

   ./benchmarks/bench.sh data imdb
   

2. Users can benchmark IMDB queries through following script against imdb parquet files.

   ./benchmarks/bench.sh run imdb
   

   Or just benchmarking single query, for example, query_id 5 indicates query 2a,

   cargo run --bin imdb benchmark datafusion --iterations 1 --path ./arrow-datafusion/benchmarks/data/imdb --prefer_hash_join true --format csv -o ./arrow-datafusion/benchmarks/results/heads_doupache_imdb-data/imdb.json --query 5 --debug
   

   Returning

    === Logical plan ===
    Projection: min(t.title) AS movie_title
      Aggregate: groupBy=[[]], aggr=[[min(t.title)]]
        Filter: cn.country_code = Utf8("[de]") AND k.keyword = Utf8("character-name-in-title") AND cn.id = mc.company_id AND mc.movie_id = t.id AND t.id = mk.movie_id AND mk.keyword_id = k.id AND mc.movie_id = mk.movie_id
    
    (...)
   
    +-------------+
    | movie_title |
    +-------------+
    | 'Doc'       |
    +-------------+
    Query 5 iteration 0 took 3222.3 ms and returned 1 rows
    Query 5 avg time: 3222.30 ms
   

3. And verify SQL results via SQL Logic Test against imdb csv files

   INCLUDE_IMDB=true cargo test --test sqllogictests -- imdb
   

Rationale for this change

1. We download IMDB queries from https://db.in.tum.de/~leis/qo/job.tgz and benchmark them with the helps of Add JOB benchmark dataset [1/N] (imdb dataset) #12497.
2. Ensuring correctness by imdb.slt, just like what we did to tpch.slt.

Unlike TPC-H, IMDB dataset is not generated and it's fixed sized, so no scaling factor and we don't need another docker container to generate data and answers.
I have also cross-checked answers in csv files from https://github.com/duckdb/duckdb/tree/main/benchmark/imdb/answers .

What changes are included in this PR?

IMDB(JOB) queries don't have incremental query_id, so I hard-coded the benchmark runner query_id (1,2,3,4, ... 113 in integer) to actual IMDB query name (1a, 1b, 1c, 1d, 2a, ... 33c in string, there is no pattern) mapping via lots of if.

Currently, I've only add SLT for:

• 1a, 1b, 1c, 1d
• 2a, 2b, 2c, 2d
• 3a, 3b, 3c
• 4a, 4b, 4c
• 5a, 5b, 5c
• 6a, 6b, 6c, 6d, 6e, 6f
• ...
• 33a, 33b, 33c

Are these changes tested?

Yes, please check test_files/imdb for details.

Are there any user-facing changes?

No.

[austin362667]

I'll rebase on @doupache 's PR #12497 while it's merged

[doupache]

I have test all table in imdb dataset , there is no "negative" id

SELECT * FROM aka_name 
WHERE id < 0 LIMIT 5

SELECT * FROM aka_title 
WHERE  id < 0 LIMIT 5

SELECT * FROM cast_info  
WHERE   id < 0 LIMIT 5

SELECT * FROM char_name  
WHERE   id < 0 LIMIT 5

SELECT * FROM comp_cast_type  
WHERE   id < 0 LIMIT 5

SELECT * FROM company_name  
WHERE   id < 0 LIMIT 5

SELECT * FROM company_type  
WHERE   id < 0 LIMIT 5

SELECT * FROM complete_cast  
WHERE   id < 0 LIMIT 5

SELECT * FROM info_type  
WHERE   id < 0 LIMIT 5

SELECT * FROM keyword  
WHERE   id < 0 LIMIT 5

SELECT * FROM kind_type  
WHERE   id < 0 LIMIT 5

SELECT * FROM link_type  
WHERE   id < 0 LIMIT 5

SELECT * FROM movie_companies  
WHERE   id < 0 LIMIT 5

SELECT * FROM movie_info  
WHERE   id < 0 LIMIT 5

SELECT * FROM movie_info_idx  
WHERE   id < 0 LIMIT 5

SELECT * FROM movie_keyword  
WHERE   id < 0 LIMIT 5

SELECT * FROM movie_link  
WHERE   id < 0 LIMIT 5

SELECT * FROM name  
WHERE id < 0 LIMIT 5

SELECT * FROM person_info  
WHERE   id < 0 LIMIT 5

SELECT * FROM role_type  
WHERE   id < 0 LIMIT 5

SELECT * FROM title  
WHERE   id < 0 LIMIT 5

[jayzhan211]

👍

[alamb]

Thank you @austin362667

My only concern is that the newly added "verify results" test takes 20 minutes

I don't think it is practical to run such queries on each CI run 🤔 it seems quite wasteful and we can get the same coverage using more targeted testing

You can see we only use a tiny amount of data for the tpch benchmark tests. Maybe we can split the addition to bench.sh from the slt files so we can discuss how to test these queries in a different PR?

[austin362667]

Sure, that makes sense to me. I'll remove that part first. Thank you @alamb
This way could also avoid unethically sending too much requests to Peter Boncz's personal page. 😅

[alamb]

Thanks @austin362667 -- let's get this one in and iterate on it

[alamb]

Given JOB seems to identify queries by alphanumeric (like 1a instead of 1) it might make sense simply to have this be an Option<String> and avoid having the mapping required below