Skip to content

Commit

Permalink
refactor: Added a generic FileStream (still in active development!) (…
Browse files Browse the repository at this point in the history 
…#2654)

* Initial impl

* Implement a simple local file and directory

* Fix types

* Fix order

* Creation and modification time

* Implement S3 file and directory

* Implement filesystem root

* Read bytes

* Use a different caching approach

* Deptry DEP001

* Change S3 list_contents implementation

* Add boto3 as an extra dep

* Simpler interface

* Use fsspec

* Start abstracting away a `FileStream`

* Move implementation upstream

* Use preferred fsspec abstraction

* Use fsspec to list directory contents

* Test incremental extraction

* Fix types

* Clean up

* Increase coverage

* Move tap logic upstream

* Fix imports

* Add no cover comment

* Configure filesystem name

* Update settings docs

* CSV-specific settings

* Test custom delimiter

* Implement schema discovery

* Add pragma: no cover

* Move ReadMode

* Implement FTP and SFTP configurations

* Make deptry happy

* Link to docs

* Update pyproject.toml

* Implement primary keys
  • Loading branch information
@edgarrmondragon
edgarrmondragon authored Oct 2, 2024
1 parent 5c68b09 commit 1442536
Show file tree
Hide file tree
Showing 12 changed files with 2,468 additions and 1,248 deletions.
2,002 changes: 1,001 additions & 1,001 deletions fixtures/csv/customers.csv
View file

Large diffs are not rendered by default.

102 changes: 51 additions & 51 deletions fixtures/csv/employees.csv
View file
Original file line number Diff line number Diff line change
@@ -1,51 +1,51 @@
id,first_name,last_name,email,ip_address
1,Tobye,Tallach,[email protected],159.80.54.64
2,Bret,Auchterlonie,[email protected],63.179.228.179
3,Chester,Leban,[email protected],109.23.123.220
4,Weston,Venny,[email protected],80.78.0.69
5,Alejoa,Hassen,[email protected],193.70.126.231
6,Otes,Ioselevich,[email protected],55.238.240.160
7,Dolley,Mc Ilwrick,[email protected],225.224.151.67
8,Cliff,Druitt,[email protected],216.35.85.142
9,Alfreda,Parysiak,[email protected],234.124.93.69
10,Alfonso,Wotherspoon,[email protected],34.94.1.132
11,Jemmy,Gavriel,[email protected],69.13.142.245
12,Ezechiel,Binion,[email protected],85.203.127.191
13,Burk,Blowfelde,[email protected],74.133.42.177
14,Danette,Brealey,[email protected],249.85.157.243
15,Brent,Collcutt,[email protected],68.202.67.52
16,Filbert,Wane,[email protected],51.190.146.189
17,Amory,Brewers,[email protected],147.155.225.194
18,Giraud,Reen,[email protected],134.254.177.66
19,Burtie,Siebert,[email protected],47.194.48.217
20,Adam,Maddick,[email protected],165.16.248.228
21,Callean,Vernall,[email protected],243.145.198.197
22,Olympie,Itzakovitz,[email protected],40.55.240.15
23,Jacky,Emney,[email protected],216.72.80.81
24,Isidoro,Novello,[email protected],153.171.11.150
25,Kora,Liversedge,[email protected],131.126.97.242
26,Salaidh,McMenamie,[email protected],228.231.31.219
27,Corey,Dowdeswell,[email protected],203.73.30.64
28,Brodie,Holwell,[email protected],64.200.225.25
29,Trudey,Ungerer,[email protected],122.82.88.41
30,Doralin,Maxted,[email protected],37.78.14.199
31,Maurie,Marklin,[email protected],22.181.178.6
32,Hermann,Voase,[email protected],155.126.157.84
33,Fanchette,Callaway,[email protected],121.161.80.246
34,Sara-ann,Birdall,[email protected],35.64.166.83
35,Harriot,Clipsham,[email protected],78.103.253.219
36,Bonita,Woolway,[email protected],70.114.50.135
37,Arleyne,MacComiskey,[email protected],80.22.221.216
38,Ethelbert,Covill,[email protected],52.66.186.124
39,Irita,Knee,[email protected],239.247.34.120
40,Naoma,Janca,[email protected],189.63.152.60
41,Kayne,Mizzen,[email protected],84.133.236.10
42,Estell,Stuckford,[email protected],246.168.153.22
43,Larine,Stack,[email protected],197.176.195.68
44,Rikki,Newbold,[email protected],27.245.43.243
45,Romonda,Charer,[email protected],137.144.236.93
46,Letizia,Monksfield,[email protected],209.47.5.147
47,Sinclare,McAreavey,[email protected],251.250.216.206
48,Athene,Haysham,[email protected],81.227.231.240
49,Gale,Tracy,[email protected],93.138.226.205
50,Dareen,O'Shields,[email protected],27.226.127.240
id first_name last_name email ip_address
1 Tobye Tallach [email protected] 159.80.54.64
2 Bret Auchterlonie [email protected] 63.179.228.179
3 Chester Leban [email protected] 109.23.123.220
4 Weston Venny [email protected] 80.78.0.69
5 Alejoa Hassen [email protected] 193.70.126.231
6 Otes Ioselevich [email protected] 55.238.240.160
7 Dolley Mc Ilwrick [email protected] 225.224.151.67
8 Cliff Druitt [email protected] 216.35.85.142
9 Alfreda Parysiak [email protected] 234.124.93.69
10 Alfonso Wotherspoon [email protected] 34.94.1.132
11 Jemmy Gavriel [email protected] 69.13.142.245
12 Ezechiel Binion [email protected] 85.203.127.191
13 Burk Blowfelde [email protected] 74.133.42.177
14 Danette Brealey [email protected] 249.85.157.243
15 Brent Collcutt [email protected] 68.202.67.52
16 Filbert Wane [email protected] 51.190.146.189
17 Amory Brewers [email protected] 147.155.225.194
18 Giraud Reen [email protected] 134.254.177.66
19 Burtie Siebert [email protected] 47.194.48.217
20 Adam Maddick [email protected] 165.16.248.228
21 Callean Vernall [email protected] 243.145.198.197
22 Olympie Itzakovitz [email protected] 40.55.240.15
23 Jacky Emney [email protected] 216.72.80.81
24 Isidoro Novello [email protected] 153.171.11.150
25 Kora Liversedge [email protected] 131.126.97.242
26 Salaidh McMenamie [email protected] 228.231.31.219
27 Corey Dowdeswell [email protected] 203.73.30.64
28 Brodie Holwell [email protected] 64.200.225.25
29 Trudey Ungerer [email protected] 122.82.88.41
30 Doralin Maxted [email protected] 37.78.14.199
31 Maurie Marklin [email protected] 22.181.178.6
32 Hermann Voase [email protected] 155.126.157.84
33 Fanchette Callaway [email protected] 121.161.80.246
34 Sara-ann Birdall [email protected] 35.64.166.83
35 Harriot Clipsham [email protected] 78.103.253.219
36 Bonita Woolway [email protected] 70.114.50.135
37 Arleyne MacComiskey [email protected] 80.22.221.216
38 Ethelbert Covill [email protected] 52.66.186.124
39 Irita Knee [email protected] 239.247.34.120
40 Naoma Janca [email protected] 189.63.152.60
41 Kayne Mizzen [email protected] 84.133.236.10
42 Estell Stuckford [email protected] 246.168.153.22
43 Larine Stack [email protected] 197.176.195.68
44 Rikki Newbold [email protected] 27.245.43.243
45 Romonda Charer [email protected] 137.144.236.93
46 Letizia Monksfield [email protected] 209.47.5.147
47 Sinclare McAreavey [email protected] 251.250.216.206
48 Athene Haysham [email protected] 81.227.231.240
49 Gale Tracy [email protected] 93.138.226.205
50 Dareen O'Shields [email protected] 27.226.127.240
Loading

0 comments on commit 1442536

Please sign in to comment.