Fix Count_Distinct in Snowflake

[radeusgd]

Pull Request Description

• Closes Fix Count_Distinct on a Boolean column in Snowflake #10611

Important Notes

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

• The documentation has been updated, if necessary.
• Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
• All code follows the
  Scala,
  Java,
  TypeScript,
  and
  Rust
  style guides. In case you are using a language not listed above, follow the Rust style guide.
• Unit tests have been written where possible.

[radeusgd]

[radeusgd]

I wanted to see if the COUNT DISTINCT hacks we do affects performance of the query.
I tried the following code:

from Standard.Base import all
import Standard.Base.Errors.Illegal_State.Illegal_State

from Standard.Table import all
from Standard.Database import all
from Standard.Snowflake import all

my_snowflake =
    account_name = Environment.get "ENSO_SNOWFLAKE_ACCOUNT"
    if account_name.is_nothing then Nothing else
        get_var name =
            Environment.get name if_missing=(Panic.throw (Illegal_State.Error "ENSO_SNOWFLAKE_ACCOUNT is set, but "+name+" is not. Please set all required environment variables."))
        user = get_var "ENSO_SNOWFLAKE_USER"
        password = get_var "ENSO_SNOWFLAKE_PASSWORD"
        database = get_var "ENSO_SNOWFLAKE_DATABASE"
        schema = Environment.get "ENSO_SNOWFLAKE_SCHEMA" if_missing="PUBLIC"
        warehouse = Environment.get "ENSO_SNOWFLAKE_WAREHOUSE" if_missing=""

        resolved_password = if password.starts_with "enso://" then Enso_Secret.get password else password
        credentials = Credentials.Username_And_Password user resolved_password
        Snowflake_Details.Snowflake account_name credentials database schema warehouse

main =
    c = Database.connect my_snowflake
    n = 50000
    in_memory_table = Table.new [["X", (0.up_to n . map v-> if v%5 == 1 then Nothing else (v%5))]]
    in_memory_table.column_info.print
    in_memory_table.print
    table_name = "My_Big_Table"
    IO.println <| Duration.time_execution <|
        Panic.rethrow <| in_memory_table.select_into_database_table c table_name primary_key=[] temporary=True
    
    runs = 20
    measure_query query = 
        IO.println "Running "+query
        results = 0.up_to runs . map _->
            r = Duration.time_execution <|
                Panic.rethrow <| c.read (..Raw_SQL query)
            r.second.print
            r.first
        results.each t-> IO.println t.to_display_text

    IO.println "SIMPLE"
    measure_query 'SELECT COUNT(DISTINCT "X") FROM "'+table_name+'"'

    rnd = Random.uuid
    IO.println "old replaced"
    measure_query 'SELECT COUNT(DISTINCT COALESCE("X", {\'enso-null-replacement-marker\':\''+rnd+'\'}::variant)) FROM "'+table_name+'"'

    IO.println "new replaced+casted"
    measure_query 'SELECT COUNT(DISTINCT COALESCE("X"::variant, {\'enso-null-replacement-marker\':\''+rnd+'\'}::variant)) FROM "'+table_name+'"'

The biggest problem was that uploading only 50k rows took me... 5 minutes! And that's far too little rows to show any performance difference:

[GregoryTravis]

Does this not give a compile-time error, having too many arguments?

[radeusgd]

No, such stuff is only currently resolved at runtime.

Once #9812 is implemented, we'd get a warning if static type analysis were enabled.

[radeusgd]

I've redone the analysis by manually uploading a 10M row dataset to Snowflake through the web panel and then running the queries on it. Here we can see a difference:

The 'Simple' flavour is COUNT(DISTINCT X) without the COALESCE, Old is what we had right now and New is the one with the fix for boolean columns. I don't see any significant difference between Old and New, so I think the fix is totally OK.

We can see however that we do pay a price for the COALESCE workaround that gives us the correct null behaviour. Thankfully, if we are doing distinct by a single column with ignore_nothing=True (non-default setting), we will get the efficient Simple query. Unfortunately, with the default ignore_nothing=True OR if we have more than one column, we will get the less efficient one.