Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Added a generic FileStream (still in active development!) #2654

Merged
merged 45 commits into from
Oct 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
a45d34d
Initial impl
edgarrmondragon Sep 6, 2024
20b8edc
Merge branch 'main' into 2648-feat-add-a-generic-filestream-interface
edgarrmondragon Sep 10, 2024
914520c
Implement a simple local file and directory
edgarrmondragon Sep 10, 2024
3ebea11
Fix types
edgarrmondragon Sep 10, 2024
bf66f7f
Fix order
edgarrmondragon Sep 10, 2024
bd5c138
Creation and modification time
edgarrmondragon Sep 10, 2024
e1ffcc0
Implement S3 file and directory
edgarrmondragon Sep 11, 2024
6c9a423
Implement filesystem root
edgarrmondragon Sep 11, 2024
fba831c
Read bytes
edgarrmondragon Sep 11, 2024
fca4f2d
Use a different caching approach
edgarrmondragon Sep 11, 2024
8ea6652
Deptry DEP001
edgarrmondragon Sep 11, 2024
79707cc
Change S3 list_contents implementation
edgarrmondragon Sep 11, 2024
939e544
Merge branch 'main' into 2648-feat-add-a-generic-filestream-interface
edgarrmondragon Sep 11, 2024
8881efa
Add boto3 as an extra dep
edgarrmondragon Sep 11, 2024
ce600f6
Simpler interface
edgarrmondragon Sep 12, 2024
97b27d7
Merge branch 'main' into 2648-feat-add-a-generic-filestream-interface
edgarrmondragon Sep 17, 2024
0fd556a
Use fsspec
edgarrmondragon Sep 17, 2024
5a24cad
Start abstracting away a `FileStream`
edgarrmondragon Sep 17, 2024
742524c
Move implementation upstream
edgarrmondragon Sep 18, 2024
83c6754
Use preferred fsspec abstraction
edgarrmondragon Sep 18, 2024
ecabf25
Use fsspec to list directory contents
edgarrmondragon Sep 18, 2024
b85739e
Test incremental extraction
edgarrmondragon Sep 18, 2024
33c8c04
Fix types
edgarrmondragon Sep 18, 2024
982b6d8
Clean up
edgarrmondragon Sep 18, 2024
3d08c62
Increase coverage
edgarrmondragon Sep 18, 2024
2aabcc4
Move tap logic upstream
edgarrmondragon Sep 18, 2024
8795b7a
Fix imports
edgarrmondragon Sep 18, 2024
e82da7b
Add no cover comment
edgarrmondragon Sep 18, 2024
f0b2707
Configure filesystem name
edgarrmondragon Sep 18, 2024
1b9f4ed
Update settings docs
edgarrmondragon Sep 18, 2024
8144fe2
CSV-specific settings
edgarrmondragon Sep 18, 2024
22b4fbe
Test custom delimiter
edgarrmondragon Sep 18, 2024
a01449a
Merge branch 'main' into 2648-feat-add-a-generic-filestream-interface
edgarrmondragon Sep 23, 2024
92f1522
Merge branch 'main' into 2648-feat-add-a-generic-filestream-interface
edgarrmondragon Sep 26, 2024
e5a4a4f
Merge branch 'main' into 2648-feat-add-a-generic-filestream-interface
edgarrmondragon Sep 26, 2024
95d8b82
Implement schema discovery
edgarrmondragon Sep 26, 2024
cbf69c9
Add pragma: no cover
edgarrmondragon Sep 26, 2024
9fd33fe
Move ReadMode
edgarrmondragon Sep 26, 2024
3b11ac4
Implement FTP and SFTP configurations
edgarrmondragon Sep 27, 2024
e20567f
Make deptry happy
edgarrmondragon Sep 27, 2024
4628468
Link to docs
edgarrmondragon Sep 27, 2024
47ba446
Merge branch 'main' into 2648-feat-add-a-generic-filestream-interface
edgarrmondragon Oct 2, 2024
d6c4cc9
Update pyproject.toml
edgarrmondragon Oct 2, 2024
6f2e30c
Merge branch 'main' into 2648-feat-add-a-generic-filestream-interface
edgarrmondragon Oct 2, 2024
e527cee
Implement primary keys
edgarrmondragon Oct 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,002 changes: 1,001 additions & 1,001 deletions fixtures/csv/customers.csv

Large diffs are not rendered by default.

102 changes: 51 additions & 51 deletions fixtures/csv/employees.csv
Original file line number Diff line number Diff line change
@@ -1,51 +1,51 @@
id,first_name,last_name,email,ip_address
1,Tobye,Tallach,[email protected],159.80.54.64
2,Bret,Auchterlonie,[email protected],63.179.228.179
3,Chester,Leban,[email protected],109.23.123.220
4,Weston,Venny,[email protected],80.78.0.69
5,Alejoa,Hassen,[email protected],193.70.126.231
6,Otes,Ioselevich,[email protected],55.238.240.160
7,Dolley,Mc Ilwrick,[email protected],225.224.151.67
8,Cliff,Druitt,[email protected],216.35.85.142
9,Alfreda,Parysiak,[email protected],234.124.93.69
10,Alfonso,Wotherspoon,[email protected],34.94.1.132
11,Jemmy,Gavriel,[email protected],69.13.142.245
12,Ezechiel,Binion,[email protected],85.203.127.191
13,Burk,Blowfelde,[email protected],74.133.42.177
14,Danette,Brealey,[email protected],249.85.157.243
15,Brent,Collcutt,[email protected],68.202.67.52
16,Filbert,Wane,[email protected],51.190.146.189
17,Amory,Brewers,[email protected],147.155.225.194
18,Giraud,Reen,[email protected],134.254.177.66
19,Burtie,Siebert,[email protected],47.194.48.217
20,Adam,Maddick,[email protected],165.16.248.228
21,Callean,Vernall,[email protected],243.145.198.197
22,Olympie,Itzakovitz,[email protected],40.55.240.15
23,Jacky,Emney,[email protected],216.72.80.81
24,Isidoro,Novello,[email protected],153.171.11.150
25,Kora,Liversedge,[email protected],131.126.97.242
26,Salaidh,McMenamie,[email protected],228.231.31.219
27,Corey,Dowdeswell,[email protected],203.73.30.64
28,Brodie,Holwell,[email protected],64.200.225.25
29,Trudey,Ungerer,[email protected],122.82.88.41
30,Doralin,Maxted,[email protected],37.78.14.199
31,Maurie,Marklin,[email protected],22.181.178.6
32,Hermann,Voase,[email protected],155.126.157.84
33,Fanchette,Callaway,[email protected],121.161.80.246
34,Sara-ann,Birdall,[email protected],35.64.166.83
35,Harriot,Clipsham,[email protected],78.103.253.219
36,Bonita,Woolway,[email protected],70.114.50.135
37,Arleyne,MacComiskey,[email protected],80.22.221.216
38,Ethelbert,Covill,[email protected],52.66.186.124
39,Irita,Knee,[email protected],239.247.34.120
40,Naoma,Janca,[email protected],189.63.152.60
41,Kayne,Mizzen,[email protected],84.133.236.10
42,Estell,Stuckford,[email protected],246.168.153.22
43,Larine,Stack,[email protected],197.176.195.68
44,Rikki,Newbold,[email protected],27.245.43.243
45,Romonda,Charer,[email protected],137.144.236.93
46,Letizia,Monksfield,[email protected],209.47.5.147
47,Sinclare,McAreavey,[email protected],251.250.216.206
48,Athene,Haysham,[email protected],81.227.231.240
49,Gale,Tracy,[email protected],93.138.226.205
50,Dareen,O'Shields,[email protected],27.226.127.240
id first_name last_name email ip_address
1 Tobye Tallach [email protected] 159.80.54.64
2 Bret Auchterlonie [email protected] 63.179.228.179
3 Chester Leban [email protected] 109.23.123.220
4 Weston Venny [email protected] 80.78.0.69
5 Alejoa Hassen [email protected] 193.70.126.231
6 Otes Ioselevich [email protected] 55.238.240.160
7 Dolley Mc Ilwrick [email protected] 225.224.151.67
8 Cliff Druitt [email protected] 216.35.85.142
9 Alfreda Parysiak [email protected] 234.124.93.69
10 Alfonso Wotherspoon [email protected] 34.94.1.132
11 Jemmy Gavriel [email protected] 69.13.142.245
12 Ezechiel Binion [email protected] 85.203.127.191
13 Burk Blowfelde [email protected] 74.133.42.177
14 Danette Brealey [email protected] 249.85.157.243
15 Brent Collcutt [email protected] 68.202.67.52
16 Filbert Wane [email protected] 51.190.146.189
17 Amory Brewers [email protected] 147.155.225.194
18 Giraud Reen [email protected] 134.254.177.66
19 Burtie Siebert [email protected] 47.194.48.217
20 Adam Maddick [email protected] 165.16.248.228
21 Callean Vernall [email protected] 243.145.198.197
22 Olympie Itzakovitz [email protected] 40.55.240.15
23 Jacky Emney [email protected] 216.72.80.81
24 Isidoro Novello [email protected] 153.171.11.150
25 Kora Liversedge [email protected] 131.126.97.242
26 Salaidh McMenamie [email protected] 228.231.31.219
27 Corey Dowdeswell [email protected] 203.73.30.64
28 Brodie Holwell [email protected] 64.200.225.25
29 Trudey Ungerer [email protected] 122.82.88.41
30 Doralin Maxted [email protected] 37.78.14.199
31 Maurie Marklin [email protected] 22.181.178.6
32 Hermann Voase [email protected] 155.126.157.84
33 Fanchette Callaway [email protected] 121.161.80.246
34 Sara-ann Birdall [email protected] 35.64.166.83
35 Harriot Clipsham [email protected] 78.103.253.219
36 Bonita Woolway [email protected] 70.114.50.135
37 Arleyne MacComiskey [email protected] 80.22.221.216
38 Ethelbert Covill [email protected] 52.66.186.124
39 Irita Knee [email protected] 239.247.34.120
40 Naoma Janca [email protected] 189.63.152.60
41 Kayne Mizzen [email protected] 84.133.236.10
42 Estell Stuckford [email protected] 246.168.153.22
43 Larine Stack [email protected] 197.176.195.68
44 Rikki Newbold [email protected] 27.245.43.243
45 Romonda Charer [email protected] 137.144.236.93
46 Letizia Monksfield [email protected] 209.47.5.147
47 Sinclare McAreavey [email protected] 251.250.216.206
48 Athene Haysham [email protected] 81.227.231.240
49 Gale Tracy [email protected] 93.138.226.205
50 Dareen O'Shields [email protected] 27.226.127.240
Loading
Loading