nif_panicked when using kino_explorer and DF.summarise

[lkarthee]

I tried to find users who are linked to same wallet_id using group_by and summarise. It results in a nif_panic. If I disable kino_explorer it does not result in exception. I am not sure where to log this bug in kino_explorer or explorer - I am logging it here as the stacktrace is from Explorer code.

Also the error occurs when user_ids: user_id is included in DF.summarise.

Mix.install([
  {:explorer, "~> 0.7.0"},
  {:kino, "~> 0.12.0"},
  {:kino_explorer, "~> 0.1.12"},
  {:kino_vega_lite, "~> 0.1.0"},
  {:tucan, "~> 0.2.1"},
  {:jason, "~> 1.4"}
])

DF.filter(user_df, is_not_nil(wallet_id)) 
|> DF.group_by("wallet_id") 
|> DF.summarise(count: count(wallet_id), user_ids: user_id)
|> DF.arrange(desc: count)

10:07:34.160 [error] ** (ArgumentError) could not start Kino.JS.Live for Elixir.Kino.Table, an exception was raised:
    ** (ErlangError) Erlang error: :nif_panicked
        (explorer 0.7.2) Explorer.PolarsBackend.Native.s_frequencies(#Explorer.PolarsBackend.Series<
  shape: (1_610,)
  Series: 'user_ids' [list[str]]
  [
    ["1d35f6f5-3151-4099-b21c-78aeaa2cd140", "0d6f5599-b3d5-4b4c-bfc5-0108bce16195", … "658824d7-83e8-4610-9f91-7c5853a88561"]
    ["8ae77ad7-f8a7-4a03-ace2-b91b3e8c2454", "d35eb16e-5b94-410c-9c7c-3a4d1e1bfc44", … "51f28964-ce35-4d0e-a69a-664be6941a64"]
    ["e7eceedf-ca2c-4fb0-8873-a224bd0449a6", "661dc869-6043-4dcd-8a45-c5389ce679d1", … "48280b44-e550-45dc-874e-79195cfbec30"]
    ["90d26406-6f5b-463d-a4c2-9da342237f2e", "8c6b6dee-858e-47ae-a223-20bc50fb8073", "7f73a507-e534-498e-83a0-39bd3d0a1a8f"]
    ["e43b6e61-70cd-471f-897d-3944643b741e", "6ebfb479-0632-45c9-9c6b-330987524e1d", "38b8fb1e-586d-4f21-b3c9-5a37d9de99c9"]
    ["16c30f2c-5e28-4666-b65f-b1bb8540d935", "5dc7533b-d0fb-4cfd-a8f6-55d0036b349f"]
    ["7478bbab-25cb-40c9-b1be-33ad184be6a8", "f21861a7-826b-4aca-8a01-c319208a2f7a"]
    ["a1c7967c-17a9-4c3b-9fe8-ea0b11aad7e6", "b6b8b1ff-d920-4c9f-b150-da5d823e3cc7"]
    ["59c37020-f1c3-46ae-a393-6ead70d8b46b", "3d1f9842-b98e-485e-a0e1-2ed9991a9bfd"]
    ["f726a379-71bc-478b-a527-b421c1c12a57", "98a3ae63-c740-493f-a2c0-4809622dea88"]
    ["c48f93f4-28ed-4178-9b70-e2dcc4a88fc8", "fbd1d615-64a3-407a-b5e9-040a2cef76c4"]
    ["3e1bc501-8a2e-44eb-b777-853b7001af49", "a1e18e09-f994-45aa-a0c1-6b3c91452fcf"]
    …
    ["1ae045b4-3528-4745-912c-1923e7f494bf"]
    ["56cad56d-af18-41de-b3e9-c76fdfd9517c"]
    ["9e9de7eb-2070-4d65-9306-9c57091dae9b"]
    ["ac2120fe-656d-4b30-b16f-8dace9da3650"]
    ["a2656866-e7c0-4cb4-af14-eed3527ce7ed"]
    ["21e13b0b-25af-4895-bcdf-6f9b51cf11bc"]
    ["bb76cdfd-1bc5-4fb3-b0b0-872008541ade"]
    ["8acd3191-6178-405a-a320-f86eda624462"]
    ["790860a6-d523-41d6-9814-db3cdfef4f48"]
    ["68651ccc-5ee2-41b0-b671-db0d33cd8acf"]
    ["e5afad89-8ecc-485f-baf3-63d88a44191d"]
    ["2377a815-18c7-4c67-b6f8-a4207fd00560"]
    ["01a6e3eb-ff19-47b4-acae-efa30af0a52e"]
  ]
>)
        (explorer 0.7.2) lib/explorer/polars_backend/shared.ex:15: Explorer.PolarsBackend.Shared.apply/2
        (explorer 0.7.2) lib/explorer/polars_backend/series.ex:430: Explorer.PolarsBackend.Series.frequencies/1
        (kino_explorer 0.1.13) lib/kino/explorer.ex:180: Kino.Explorer.most_frequent/1
        (kino_explorer 0.1.13) lib/kino/explorer.ex:163: anonymous fn/4 in Kino.Explorer.summaries/2
        (stdlib 5.1.1) maps.erl:416: :maps.fold_1/4
        (kino_explorer 0.1.13) lib/kino/explorer.ex:145: Kino.Explorer.summaries/2
        (kino_explorer 0.1.13) lib/kino/explorer.ex:64: Kino.Explorer.init/1
    (kino 0.12.0) lib/kino/js/live.ex:351: Kino.JS.Live.new/3
    (kino_explorer 0.1.13) lib/kino_explorer.ex:3: Kino.Render.Explorer.DataFrame.to_livebook/1
    lib/livebook/runtime/evaluator/formatter.ex:62: Livebook.Runtime.Evaluator.Formatter.to_output/1
    lib/livebook/runtime/evaluator.ex:469: Livebook.Runtime.Evaluator.continue_do_evaluate_code/6
    lib/livebook/runtime/evaluator.ex:331: Livebook.Runtime.Evaluator.loop/1
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3

[lkarthee]

Below results in nif_panic

Series.from_list([[1,2], [1]]) |> Series.frequencies()

some observations

> df.dtypes["user_ids"] == df["user_ids"].dtype # => 
false
> df["user_ids"].dtype
{:list, :string}
> df.dtypes["user_ids"]
:string

> df.dtypes["wallet_id"] == df["wallet_id"].dtype
true

[philss]

@lkarthee I'm investigating this. It looks like Series.frequencies/1 does not work with {:list, any()} natively. If we activate Polars' feature group_by_list, we can take frequencies of :list series that contains numeric dtypes inside - like {:list, {:s, 64}.

But this won't work for you case. I think we would need to treat this at kino_explorer level for now.
WDYT @cristineguadelupe ?

[lkarthee]

I have tried to use explorer from git url - it fails. I am suspecting its due to path issues w.r.to how livebook resolves paths ? Anyway I can use git version ?

Mix.install([
  # {:explorer, "~> 0.7.0"},
  {:explorer, git: "https://github.com/elixir-explorer/explorer.git", override: true},
])

==> explorer
Compiling 25 files (.ex)
== Compilation error in file lib/explorer/polars_backend/native.ex ==
** (ErlangError) Erlang error: :enoent
    (elixir 1.15.7) lib/system.ex:1092: System.cmd("cargo", ["metadata", "--format-version=1"], [cd: "native/explorer"])
    (rustler 0.29.1) lib/rustler/compiler/config.ex:83: Rustler.Compiler.Config.metadata!/1
    (rustler 0.29.1) lib/rustler/compiler/config.ex:70: Rustler.Compiler.Config.build/1
    (rustler 0.29.1) lib/rustler/compiler.ex:9: Rustler.Compiler.compile_crate/2
    lib/explorer/polars_backend/native.ex:32: (module)
could not compile dependency :explorer, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile explorer --force", update it with "mix deps.update explorer" or clean it with "mix deps.clean explorer"

[philss]

@lkarthee to install from source you need to have Rust installed. Please check the instructions here: https://github.com/elixir-explorer/explorer#contributing

But just to clarify, the problem still there. It's just with a better error message from the Explorer side. We need to avoid using Series.frequencies/1 from kino_explorer for list series that are not numeric.

[lkarthee]

@philss I have rust setup on my mac. If I do git clone explorer and do a mix compile and iex -S mix, it works. If I include it another mix application it works. But if I include it as git dependency in livebook it fails with :enoent.

> git clone https://github.com/elixir-explorer/explorer.git
> mix deps.get
> mix compile
> ies -S mix
iex(1)> alias Explorer.Series
Explorer.Series
iex(2)> Series.from_list([[1,2], [1]]) |> Series.frequencies()
#Explorer.DataFrame<
  Polars[2 x 2]
  values list[s64] [[1, 2], [1]]
  counts s64 [1, 1]
>

[lkarthee]

@philss @cristineguadelupe sent a pull request addressing kino_explorer fix. please review

[josevalim]

If you are using Livebook Desktop, you need to go to settings and configure PATH so it finds the rustc and friends. :) Closing this one in favor of the PR!