You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I realized that there is a simpler way to avoid the Out of memory error. The merge step in which it happens is unnecessary because geo2topo can accept ndjson when run with -n (see this issue).
I was experiencing an out of memory error when trying to process the US counties shapefile from census.gov with the following workflow:
ndjson-reduce
is obviously not streaming but loading everything in memory and that causes the overflow.I could work around the issue by replacing the
ndjson-reduce
step with my ownndjson-merge
utility as below:This is my implementation of
ndjson-merge
:Would it make sense to add a command like this one to
ndjson-cli
? If so, I'd try to put together a proper PR.The text was updated successfully, but these errors were encountered: