ci: Support Python 3.13

[dangotbanned]

Resolves #3587

Minimal/fallback support for 3.13.

Ideally I'd like to minimise hard dependencies we have on typing_extensions before October 2024. However, this would be the simpler option if it passes CI

• WHL: Python 3.13 pyproj4/pyproj#1439
• [Python][Packaging] Support Python 3.13 and upload wheels apache/arrow#43519
• WIP: [Release] Verify release-18.0.0-rc0 apache/arrow#44440
• ci: use duckdb instead of ibis to test interchange-only support #3672

[mattijn]

I think it is noble to wait for support for python 3.13 for all our upstream soft dependencies. But I'm also not against releasing before all our soft dependencies provide support (in my understanding pyarrow). What you think @dangotbanned?

[dangotbanned]

I think it is noble to wait for support for python 3.13 for all our upstream soft dependencies. But I'm also not against releasing before all our soft dependencies provide support (in my understanding pyarrow). What you think @dangotbanned?

@mattijn nothing noble here 😄

I'm happy with releasing before pyarrow>=18.0.0 in principle.
The issue is we'd need to remove pyarrow from altair[all] (https://github.com/vega/altair/actions/runs/11448782729/job/31853033909?pr=3591)

Also maybe do some decoupling in tests/., if there are any imports that assume pyarrow is always installed.

Doing either of those is fine with me if you'd like to give it a try?

---

Personally though, I've got a few issues/PRs that I'd rather have ready for 5.5.0 (+ some others that are less pressing):

• Wide support for datetime.(date|datetime) #3650
  • feat: Support datetime.(date|datetime) in Expression(s) #3654
• docs: Promote when-then-otherwise in User Guide #3544
• Improvements to alt.themes #3519
  • docs: Add ThemeConfig section to User Guide #3645
  • refactor: alt.themes -> alt.theme #3618
  • PluginRegistry persists unregistered plugins #3619
  • Make it easier for downstream libraries to *safely* contribute themes #3586
• Update Chart.transform_filter() to mirror alt.when() #3657

I think there's more value in delivering some/all of these vs a slightly earlier 5.5.0, if that makes sense?

[mattijn]

Hmpf. Decoupling altair[all] is not something I was aiming for. Let's wait.

[dangotbanned]

Hmpf. Decoupling altair[all] is not something I was aiming for. Let's wait.

No worries @mattijn

Also, IIRC pyarrow was blocking for geopandas - which I think is relevant to you?

[mattijn]

Yeah, I assume that geospatial users of Altair are the minority, so if that's causing the delay, also feel a bit responsible for that.

[dangotbanned]

Yeah, I assume that geospatial users of Altair are the minority, so if that's causing the delay, also feel a bit responsible for that.

Its not on you @mattijn!
just giving you a heads up if you were eager to use 3.13 is all

[dangotbanned]

I'm all out of ideas on this one 😞

@mattijn @jonmmease @joelostblom @binste
Pinging in case any of you guys can make sense of the logs

[jonmmease]

It looks like pip is first installing pyarrow 18, but then installs 17, which it tries to build from source since there aren't wheel published for Python 3.13. I think the issue is that ibis 9.5.0 has an upper bound on pyarrow of <18. See https://github.com/ibis-project/ibis/blob/9.5.0/pyproject.toml. This was relaxed on their man branch, but a new version hasn't been published yet.

I don't recall offhand which of our tests need ibis with the polars backend, but we could see if theres a way to rework them without it.

[dangotbanned]

Thanks @jonmmease

I don't recall offhand which of our tests need ibis with the polars backend, but we could see if theres a way to rework them without it.

If that is the issue, these tests are the one's that need it:

altair/tests/vegalite/v5/test_api.py

Lines 1614 to 1656 in 211f7c5

	def test_ibis_with_date_32():
	ibis.set_backend("polars")
	df = pl.DataFrame(
	{"a": [1, 2, 3], "b": [date(2020, 1, 1), date(2020, 1, 2), date(2020, 1, 3)]}
	)
	tbl = ibis.memtable(df)
	result = alt.Chart(tbl).mark_line().encode(x="a", y="b").to_dict()
	assert next(iter(result["datasets"].values())) == [
	{"a": 1, "b": "2020-01-01T00:00:00"},
	{"a": 2, "b": "2020-01-02T00:00:00"},
	{"a": 3, "b": "2020-01-03T00:00:00"},
	]
	@pytest.mark.skipif(
	Version("1.5") > PANDAS_VERSION,
	reason="A warning is thrown on old pandas versions",
	)
	@pytest.mark.xfail(
	sys.platform == "win32", reason="Timezone database is not installed on Windows"
	)
	def test_ibis_with_vegafusion(monkeypatch: pytest.MonkeyPatch):
	ibis.set_backend("polars")
	df = pl.DataFrame(
	{
	"a": [1, 2, 3],
	"b": [datetime(2020, 1, 1), datetime(2020, 1, 2), datetime(2020, 1, 3)],
	}
	)
	tbl = ibis.memtable(df)
	# "poison" `arrow_table_from_dfi_dataframe` to check that it does not get called
	# if we use the vegafusion transformer
	monkeypatch.setattr(
	"altair.utils.data.arrow_table_from_dfi_dataframe", lambda x: 1 / 0
	)
	tbl = ibis.memtable(df)
	with alt.data_transformers.enable("vegafusion"):
	result = alt.Chart(tbl).mark_line().encode(x="a", y="b").to_dict(format="vega")
	assert next(iter(result["data"]))["values"] == [
	{"a": 1, "b": "2020-01-01T00:00:00.000"},
	{"a": 2, "b": "2020-01-02T00:00:00.000"},
	{"a": 3, "b": "2020-01-03T00:00:00.000"},
	]

I had to change those a bit when dropping 3.8 to fix #3647 (comment)

Interesting if they cause problems in both cases

[dangotbanned]

@MarcoGorelli
I was wondering if you had any ideas on if/how we could move this forward, prior to ibis>=10.0.0?

I'd been following the 3.13 PR you had (narwhals-dev/narwhals#1094) but I must say I'm feeling a bit out of my depth on this one

[MarcoGorelli]

hey - I'd suggest removing Ibis from CI, I think we can use duckdb for that test, I'll make a PR

here you go #3672

(note that we continue to test Ibis in CI in Narwhals, so support for it should stay strong)

[dangotbanned]

hey - I'd suggest removing Ibis from CI, I think we can use duckdb for that test, I'll make a PR

here you go #3672

(note that we continue to test Ibis in CI in Narwhals, so support for it should stay strong)

@MarcoGorelli looks to have done the job, thanks again 🙏

[mattijn]

Indeed thanks @MarcoGorelli! Appreciated👍