Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Combiner script is accidentally quadratic #1

Open
jtsymon opened this issue Jul 30, 2016 · 0 comments
Open

Combiner script is accidentally quadratic #1

jtsymon opened this issue Jul 30, 2016 · 0 comments

Comments

@jtsymon
Copy link
Member

jtsymon commented Jul 30, 2016

🐌 🐌 🐌
For each area, the script is searching all the data files for that area.
So 2000 areas * 7 data (so far) = 14000 data files, so 2000 * 14000 = roughly 28,000,000 stat() calls to combine the files.
jtsymon referenced this issue Jul 30, 2016
@jtsymon
Fixed quadratic behaviour
a5adb11 
Now enumerate all the input files, sorts them by their basename (to
group the same basenames together), then combines all the files with the
same basename.

Unfortunately this doesn't seem to provide any speedup, probably because
I'm running it through cygwin which does weird stuff with file I/O.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jtsymon