You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Should we support distributed copy into from s3 distributed=true in the future?
If we use copy into from s3 purge=true to import data simultaneously in many query-node, we must very careful to set s3-prefix or pattern so that different nodes use different directories to avoid repeatedly importing.
like this:
#query1
copy into xx from 's3://query1/' pattern ='.*[.]tsv' purge=true;
#query2
copy into xx from 's3://query2/' pattern ='.*[.]tsv' purge=true;
#query3
copy into xx from 's3://query3/' pattern ='.*[.]tsv' purge=true;
If we support distributed copy into, things will be easier.
we just need to use:
copy into xx from 's3://import/' pattern ='.*[.]tsv' purge=true distributed=true;
The text was updated successfully, but these errors were encountered:
Summary
Should we support distributed
copy into from s3 distributed=true
in the future?If we use
copy into from s3 purge=true
to import data simultaneously in many query-node, we must very careful to sets3-prefix or pattern
so that different nodes use different directories to avoid repeatedly importing.like this:
If we support distributed
copy into
, things will be easier.we just need to use:
The text was updated successfully, but these errors were encountered: