Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: lazily import pyiceberg and unity catalog if available #3565

Merged
merged 1 commit into from
Dec 14, 2024

Conversation

jaychia
Copy link
Contributor

@jaychia jaychia commented Dec 13, 2024

memray reports that a not-insignificant amount of memory is being taken by our catalog module at import-time

image

This PR makes those imports lazy

@github-actions github-actions bot added the perf label Dec 13, 2024
@jaychia jaychia requested a review from colin-ho December 13, 2024 09:38
Copy link

codspeed-hq bot commented Dec 13, 2024

CodSpeed Performance Report

Merging #3565 will degrade performances by 27.38%

Comparing jay/lazy-imports-catalogs (9b964b5) with main (35ed63c)

Summary

⚡ 1 improvements
❌ 2 regressions
✅ 24 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark main jay/lazy-imports-catalogs Change
test_count[1 Small File] 3.3 ms 4 ms -17.14%
test_iter_rows_first_row[100 Small Files] 175.2 ms 241.2 ms -27.38%
test_show[100 Small Files] 19 ms 15.9 ms +19.48%

Copy link

codecov bot commented Dec 13, 2024

Codecov Report

Attention: Patch coverage is 13.33333% with 13 lines in your changes missing coverage. Please review.

Project coverage is 77.79%. Comparing base (35ed63c) to head (9b964b5).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
daft/catalog/__init__.py 13.33% 13 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3565      +/-   ##
==========================================
- Coverage   77.81%   77.79%   -0.02%     
==========================================
  Files         716      716              
  Lines       87987    87991       +4     
==========================================
- Hits        68463    68455       -8     
- Misses      19524    19536      +12     
Files with missing lines Coverage Δ
daft/catalog/__init__.py 30.55% <13.33%> (-25.70%) ⬇️

... and 1 file with indirect coverage changes

@jaychia jaychia merged commit 95a61d2 into main Dec 14, 2024
41 of 43 checks passed
@jaychia jaychia deleted the jay/lazy-imports-catalogs branch December 14, 2024 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants