Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

readthedocs build / documentation build time #4147

Closed
keewis opened this issue Jun 11, 2020 · 1 comment · Fixed by #4149
Closed

readthedocs build / documentation build time #4147

keewis opened this issue Jun 11, 2020 · 1 comment · Fixed by #4149

Comments

@keewis
Copy link
Collaborator

keewis commented Jun 11, 2020

It seems that after the merge of #3818, the RTD builds started to time out while the docs CI take about 37 minutes.

As a reference, before #3818 our docs CI completed in about 6 minutes.

I'm not sure if that is due to #3818 or because of updated dependencies (dask?), but I think we should try not to take these 30 minutes.

@keewis keewis changed the title readthedocs build readthedocs build / documentation build time Jun 11, 2020
@keewis
Copy link
Collaborator Author

keewis commented Jun 11, 2020

It seems that's because adding (potentially big) Dataset / DataArray objects using

npargs = np.empty((len(iterable),), dtype=np.object)
npargs[:] = iterable
is really slow and the stacktrace from sending KeyboardInterrupt says that _iter is called (if I understand it correctly, that is iterating over all values of the DataArray / Dataset using a python loop).

Looking at the code of map_blocks, the object arrays (npargs ) is only used because of its indexing and we might be able to use a list / tuple and itertools to get the same result, but without iterating over values. Am I missing something? cc @dcherian

@keewis keewis mentioned this issue Jun 12, 2020
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant