This repository has been archived by the owner on Sep 3, 2022. It is now read-only.
Use GCS prefix+delimiter for listing buckets #270
Comments
rajivpb
pushed a commit
to rajivpb/datalab
that referenced
this issue
Jun 16, 2017
…oogledatalab#270) This PR fixes the Query constructor to take dependencies (subqueries, udfs, or external datasources) in the form of either a list or dictionary. If a list is given, it should be a list of strings, where items are names of objects defined in the environment object (user can build their own object or pass the notebook_environment() as parameter). If a dictionary is provided, it should be a mapping between dependency names and objects. All of the below examples now work: q1 = bq.Query('sql') q2 = bq.Query('sql', subqueries=['q1'], env={'q1': q1}) q2 = bq.Query('sql', subqueries=['q1'], env=utils.commands.notebook_environment()) q2 = bq.Query('sql', subqueries={'q1': q1})
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
GCS supports returning "sub-directories" (i.e., common object prefixes) when listing a bucket and specifying a prefix (e.g., "/") within the query. The sub-directories are returned as CommonPrefixes within the response, but are not currently returned by the gcloud-node library being used.
Current work-around is listing all objects within the bucket (expensive) and then filtering the response.
Depends upon gcloud-node issue resolution:
googleapis/google-cloud-node#514
The text was updated successfully, but these errors were encountered: