Change gcloud package to google.cloud #2223

dhermes · 2016-08-30T18:42:57Z

Also use namespace packages throughout.

The commit messages will be quite useful, as almost all of them contain bash scripts used to generated the commit.

Fixes #1978

Currently the PR has problems (e.g. imports don't work for tox), but I wanted to get it out there as a start since it is such a monster.

google/__init__.py

+    pkg_resources.declare_namespace(__name__)
+except ImportError:
+    import pkgutil
+    __path___ = pkgutil.extend_path(__path__, __name__)


daspecster · 2016-08-30T20:46:00Z

@dhermes, I'm sure this will break json-docs as well.
I can start playing with that from this branch though.

dhermes · 2016-08-30T20:48:35Z

@daspecster Definitely. I was planning on asking for your help at some point. Trying to get the basic tox envs working first.

Done via: $ mkdir google $ git mv gcloud google/cloud

Done via: $ git grep -l 'import gcloud' | \ > xargs sed -i s/'import gcloud'/'import google.cloud'/g

Done via: $ git grep -l 'from gcloud import' | \ > xargs sed -i s/'from gcloud import'/'from google.cloud import'/g

Done via: $ git grep -l '>>> from gcloud.' | \ > xargs sed -i s/'>>> from gcloud.'/'>>> from google.cloud.'/g

This was done entirely via sed. First checked that the remaining imports were at the start of their respective lines: $ git grep 'from gcloud.*import' > anywhere.txt $ git grep '^[[:space:]]*from gcloud.*import' > line_start.txt $ diff -s anywhere.txt line_start.txt Files anywhere.txt and line_start.txt are identical $ rm -f anywhere.txt line_start.txt Then put all the imports in a text file $ git grep -h '^[[:space:]]*from gcloud.*import' > imports.txt Then parsed that file to get a list of the modules: $ cat << EOF > get_mods.py > def get_import(line): > a, b, c, d = line.strip().split(None, 3) > assert a == 'from' > assert c == 'import' > try: > e, f, g = b.split('.', 2) > except ValueError: > e, f = b.split('.') > assert e == 'gcloud' > return f > > with open('imports.txt', 'rb') as file_obj: > imported = sorted(set(map(get_import, file_obj.readlines()))) > > for mod in imported: > print(mod) > > EOF $ python get_mods.py > mods.txt $ rm -f imports.txt get_mods.py Then using the list of modules to rewrite the imports $ cat << 'EOF' > all-sed.sh > for mod in $(cat mods.txt); do > git grep -l "from gcloud.${mod}" | \ > xargs sed -i s/"from gcloud.${mod}"/"from google.cloud.${mod}"/g > done > > EOF $ /bin/bash all-sed.sh $ rm -f all-sed.sh mods.txt

Done via: $ git grep -l 'gcloud-python' | \ > xargs sed -i s/gcloud-python/google-cloud-python/g

Done via: $ git grep -l 'hack-on-gcloud' | \ > xargs sed -i s/hack-on-gcloud/hack-on-google-cloud-python/g

Done via: $ git grep -l 'automodule:: gcloud' | \ > xargs sed -i s/'automodule:: gcloud'/'automodule:: google.cloud'/g

Done via: $ git grep -l '<gcloud.' | \ > xargs sed -i s/'<gcloud.'/'<google.cloud.'/g $ git grep -l '`gcloud\.' | \ > xargs sed -i s/'`gcloud\.'/'`google.cloud.'/g

Done via: $ git grep -l '"gcloud/' | \ > xargs sed -i s/'"gcloud\/'/'"google\/cloud\/'/g

Many usages were ad-hoc so this was done by hand rather than via a sed script.

Done via: $ git grep -l 'GCLOUD_' | \ > xargs sed -i s/'GCLOUD_'/'GOOGLE_CLOUD_'/g

Done via: $ git grep -l GCloudError | \ > xargs sed -i s/GCloudError/GoogleCloudError/g

Done via: $ git grep -l '~gcloud' | \ > xargs sed -i s/'~gcloud'/'~google.cloud'/g

commit 6faf161 Author: Danny Hermes <[email protected]> Date: Tue Aug 30 12:39:46 2016 -0700 Converting almost all packages to namespace packages. As a result, the module objects for those packages can no longer contain members. Hence the following is no longer possible: >>> from google.cloud import storage >>> client = storage.Client() The "_generated" packages have not been converted to namespace packages and neither have "google.cloud.logging.handlers" nor "google.cloud.logging.handlers.transports". In addition, some members of "google.cloud.logging" have been moved into "google.cloud.logging.client" in order to be importable. Finally, the unit tests all pass at this commit, but `tox` fails to run them: the namespace imports appear broken to `tox` even though they aren't broken to the `virtualenv` for the given `tox` environment.

CONTRIBUTING.rst

@@ -326,7 +326,7 @@ instead of
 ``https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/CONTRIBUTING.rst``)
 may cause problems creating links or rendering the description.

-.. _description on PyPI: https://pypi.python.org/pypi/gcloud
+.. _description on PyPI: https://pypi.python.org/pypi/google-cloud



dhermes · 2016-09-06T22:02:10Z

@tseaver Renames that are also a concern:

gcloud-python-expenses-demo
google-cloud-python.readthedocs.org

google/cloud/datastore/batch.py

@@ -290,7 +290,7 @@ def _assign_entity_to_pb(entity_pb, entity):

    Helper method for ``Batch.put``.

-    :type entity_pb: :class:`google.cloud.datastore._generated.entity_pb2.Entity`
+    :type entity_pb: :class:`.datastore._generated.entity_pb2.Entity`


dhermes · 2016-09-06T23:27:13Z

@tseaver Other than the path changes on docstrings, anything else for me to do?

tseaver · 2016-09-07T15:53:54Z

@dhermes Nope, I think we have to trust our tests here.

dhermes · 2016-09-07T17:09:51Z

@tseaver merging now to stop blocking other PRs and to avoid having a long Travis wait time. Will send the docstring fixes in a new PR ASAP. Also going to rename the repo immediately after merging

dhermes · 2016-09-07T17:20:52Z

FYI AppVeyor seems happy: https://ci.appveyor.com/project/dhermes/gcloud-python/build/1.0.631.master

Follow-up to PR googleapis#2223. The Sphinx mechanism of using a `.` to mean "go find this class" can be confusing, so we make sure the `.` also is a relative path.

tseaver · 2016-09-07T17:52:25Z

@dhermes System tests for pubsub and logging are failing, due to the namespace packages.

tseaver · 2016-09-07T17:52:59Z

I'm working on a fix for them.

dhermes · 2016-09-07T17:53:17Z

Just filed #2259 for the overall fix. Something more specific busted for logging and pubsub?

tseaver · 2016-09-07T17:54:03Z

Namespace packages break the client construction for pubsub and logging.

Follow-up to PR googleapis#2223. The Sphinx mechanism of using a `.` to mean "go find this class" can be confusing, so we make sure the `.` also is a relative path.

dhermes added packaging do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Aug 30, 2016

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Aug 30, 2016

theacodes reviewed Aug 30, 2016
View reviewed changes

google/__init__.py

pkg_resources.declare_namespace(__name__)

except ImportError:

import pkgutil

__path___ = pkgutil.extend_path(__path__, __name__)

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

dhermes force-pushed the namespace-change branch from 45b46c8 to 102b781 Compare August 30, 2016 19:41

theacodes mentioned this pull request Aug 30, 2016

Running and bundling tests with namespace packages #2225

Closed

dhermes force-pushed the namespace-change branch from 4acb8ef to a30295d Compare September 3, 2016 04:41

dhermes added 21 commits September 6, 2016 09:30

Renaming gcloud/ directory to google/cloud/.

261c969

Done via: $ mkdir google $ git mv gcloud google/cloud

Renaming "import gcloud" statements as google.cloud.

e55a1d8

Done via: $ git grep -l 'import gcloud' | \ > xargs sed -i s/'import gcloud'/'import google.cloud'/g

Renaming "from gcloud import *" statements as google.cloud.

416071b

Done via: $ git grep -l 'from gcloud import' | \ > xargs sed -i s/'from gcloud import'/'from google.cloud import'/g

Renaming ">>> from gcloud.foo.bar import *" statements as google.cloud.

835578f

Done via: $ git grep -l '>>> from gcloud.' | \ > xargs sed -i s/'>>> from gcloud.'/'>>> from google.cloud.'/g

Updating gcloud->google.cloud imports in gRPC rewrite scripts.

d05a9d4

Renaming mentions of "gcloud-python" as "google-cloud-python".

ab18ce8

Done via: $ git grep -l 'gcloud-python' | \ > xargs sed -i s/gcloud-python/google-cloud-python/g

Renaming mentions of "hack-on-gcloud" as "hack-on-google-cloud-python".

c13d2ec

Done via: $ git grep -l 'hack-on-gcloud' | \ > xargs sed -i s/hack-on-gcloud/hack-on-google-cloud-python/g

Renaming mentions of "gcloud" docs automodules as "google.cloud".

f29999a

Done via: $ git grep -l 'automodule:: gcloud' | \ > xargs sed -i s/'automodule:: gcloud'/'automodule:: google.cloud'/g

Renaming docs mentions of "gcloud" as "google-cloud".

f12fee4

Done via: $ git grep -l '<gcloud.' | \ > xargs sed -i s/'<gcloud.'/'<google.cloud.'/g $ git grep -l '`gcloud\.' | \ > xargs sed -i s/'`gcloud\.'/'`google.cloud.'/g

Renaming docs path in JSON docs (gcloud->google/cloud).

ae89f77

Done via: $ git grep -l '"gcloud/' | \ > xargs sed -i s/'"gcloud\/'/'"google\/cloud\/'/g

Replacing usage of gcloud with google-cloud in docs config.

381d236

Replacing gcloud with google-cloud in scripts directory.

3649f3a

Many usages were ad-hoc so this was done by hand rather than via a sed script.

Renaming all GCLOUD_* env. vars. as GOOGLE_CLOUD_*.

ab59f8d

Done via: $ git grep -l 'GCLOUD_' | \ > xargs sed -i s/'GCLOUD_'/'GOOGLE_CLOUD_'/g

Renaming all GCloudError as GoogleCloudError.

e12cc95

Done via: $ git grep -l GCloudError | \ > xargs sed -i s/GCloudError/GoogleCloudError/g

Renaming all ~gcloud references as google.cloud.

faab61c

Done via: $ git grep -l '~gcloud' | \ > xargs sed -i s/'~gcloud'/'~google.cloud'/g

Cleaning up remaining renames of gcloud->google-cloud.

5910b22

Adding __init__.py to make google/ a package.

e6139ac

Fixing lint issues after package rename.

adf5e21

Fixing docs issues after package rename.

e03a7f8

tseaver reviewed Sep 6, 2016
View reviewed changes

dhermes mentioned this pull request Sep 6, 2016

Add 'Client.list_projects'. #2254

Merged

dhermes merged commit e407bfb into googleapis:master Sep 7, 2016

dhermes deleted the namespace-change branch September 7, 2016 17:09

dhermes removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Sep 7, 2016

dhermes mentioned this pull request Sep 7, 2016

Updating some Sphinx class links. #2257

Merged

dhermes mentioned this pull request Sep 7, 2016

Travis build broken due to GCLOUD --> GOOGLE_CLOUD rename #2259

Closed

This was referenced Sep 7, 2016

Fix logging/pubub system tests #2260

Merged

Shorten instance ID to match restrictions. #2261

Merged

tseaver mentioned this pull request Sep 8, 2016

Revert change to User-Agent header #2271

Closed

This was referenced Sep 16, 2016

Removing pubsub and logging as namespace packages. #2334

Merged

Cut 0.19.0 release. #2270

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change gcloud package to google.cloud #2223

Change gcloud package to google.cloud #2223

dhermes commented Aug 30, 2016

This comment was marked as spam.

This comment was marked as spam.

daspecster commented Aug 30, 2016

dhermes commented Aug 30, 2016

This comment was marked as spam.

This comment was marked as spam.

dhermes commented Sep 6, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

dhermes commented Sep 6, 2016

tseaver commented Sep 7, 2016

dhermes commented Sep 7, 2016

dhermes commented Sep 7, 2016

tseaver commented Sep 7, 2016 •

edited

Loading

tseaver commented Sep 7, 2016

dhermes commented Sep 7, 2016

tseaver commented Sep 7, 2016

Change gcloud package to google.cloud #2223

Change gcloud package to google.cloud #2223

Conversation

dhermes commented Aug 30, 2016

This comment was marked as spam.

This comment was marked as spam.

daspecster commented Aug 30, 2016

dhermes commented Aug 30, 2016

This comment was marked as spam.

This comment was marked as spam.

dhermes commented Sep 6, 2016

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

dhermes commented Sep 6, 2016

tseaver commented Sep 7, 2016

dhermes commented Sep 7, 2016

dhermes commented Sep 7, 2016

tseaver commented Sep 7, 2016 • edited Loading

tseaver commented Sep 7, 2016

dhermes commented Sep 7, 2016

tseaver commented Sep 7, 2016

tseaver commented Sep 7, 2016 •

edited

Loading