Change in behavior of %%bigquery magic specifying destination dataframe

[fedorov]

Describe the current behavior

%%bigquery magic expects destination dataframe to follow the magic command as in

%%bigquery destination_df --project=my-project

Describe the expected behavior

Apparently, sometime around mid-November (Nov 11, 2022, perhaps, per https://colab.research.google.com/notebooks/relnotes.ipynb#scrollTo=cJxFZHsR7HAZ) google-cloud-bigquery was updated. Prior to this update, the syntax was

%%bigquery --project=my-project destination_df

This breaking change invalidated notebooks that used the former syntax. The runtime error has no indications about the actual cause of the error. Colab release notes do not have any mention of this breaking change. Users of Colab (I used to be a paying subscriber) did not get any notifications about this breaking change.

As a user of Colab, I would like to understand why there was no communication about this breaking change, and whether backwards compatibility was considered when upgrading the package? Are there any regression tests Colab developers are performing to identify this kind of issues and inform users?

Additional context

It is trivial to reproduce the error triggered by the original syntax using the examples above.

[colaboratory-team]

Sorry about the breakage. We indeed upgraded many of the google-cloud packages recently, and one of the changes that google-cloud-bigquery introduced was a change in the syntax for the %%bigquery cell magic. The var name now has to be specified before the project flag, as in %%bigquery destination_df --project=my-project. This was a change to the library itself, which Colab team does not own.

Colab installs many packages by default, and we upgrade many of these each week. We generally do not announce package upgrades ahead of time unless its a major change, such as our upcoming Python 3.8 upgrade, as this would not be sustainable for a team of our size. If you want to ensure that a package version stays the same indefinitely, we recommend putting a pip install to the version you want at the start of your notebook.

[michelle-carou]

Hi @fedorov Thanks so much for raising this, it helped me realize what my error is after 1 whole day of trying to scour the internet for reasons why my df is facing parsing error! :<
Btw, is there any place that better documents magic commands other than this sad-looking page..?