Directory Organization

[rubenssoto]

Today whale pull table information and organize in the same directory, so we have a mix of tables of different databases, I think that could be more organized if we have a directory for each database. It make senses for you?

Thank you!

[yosupmoon]

Which ones would you like prioritized? e.g. Snowflake, BigQuery, AWS S3, etc. A comprehensive collection that would make you happy?

[rubenssoto]

Hi Joseph, how are you?

I used only aws glue catalog, what do you mean with comprehensive collection?

[rsyi]

Yeah this makes sense, @rubenssoto. Just to be clear, I'll walk through how I'm thinking of changing this:

For one of my tables, the present format is:

glue/sample-client-pg-1.postgres_public_comments

which corresponds to

{name}/{instance}.{catalog}_{schema}_{table}

Is this what you'd want instead?

{name}/{instance}/{catalog}_{schema}_{table}

I.e. isolate the metadata based on instance name into different folders?

The only tricky thing is that this would affect the backwards compatibility of the documentation, so I'll have to build in a small migration script, but I think that's no big deal. :)

[rubenssoto]

Yeah, I think exactly this

{name}/{instance}/{catalog}{schema}{table}

Instance is the same as database right?

[rubenssoto]

I don't need backward compatibility.