TFDV and TFMA visualizations crash in 1.0rc2

[Bobgy]

What steps did you take:

Ran tfx taxi example
Look at statistics gen or other steps with TFDV/TFMA visualization.

What happened:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-27-a59281919cdf> in <module>
     14 
     15 import base64
---> 16 import tensorflow_data_validation as tfdv
     17 from IPython.display import display
     18 from IPython.display import HTML

/usr/local/lib/python3.6/dist-packages/tensorflow_data_validation/__init__.py in <module>
     22 
     23 # Import stats API.
---> 24 from tensorflow_data_validation.api.stats_api import GenerateStatistics
     25 
     26 # Import validation API.

/usr/local/lib/python3.6/dist-packages/tensorflow_data_validation/api/stats_api.py in <module>
     46 
     47 import random
---> 48 import apache_beam as beam
     49 import pyarrow as pa
     50 from tensorflow_data_validation import constants

/usr/local/lib/python3.6/dist-packages/apache_beam/__init__.py in <module>
     94 
     95 from apache_beam import coders
---> 96 from apache_beam import io
     97 from apache_beam import typehints
     98 from apache_beam import version

/usr/local/lib/python3.6/dist-packages/apache_beam/io/__init__.py in <module>
     37 try:
     38   from apache_beam.io.gcp.bigquery import *
---> 39   from apache_beam.io.gcp.pubsub import *
     40   from apache_beam.io.gcp import gcsio
     41 except ImportError:

/usr/local/lib/python3.6/dist-packages/apache_beam/io/gcp/pubsub.py in <module>
     46 
     47 try:
---> 48   from google.cloud import pubsub
     49 except ImportError:
     50   pubsub = None

/usr/local/lib/python3.6/dist-packages/google/cloud/pubsub.py in <module>
     18 from __future__ import absolute_import
     19 
---> 20 from google.cloud.pubsub_v1 import PublisherClient
     21 from google.cloud.pubsub_v1 import SubscriberClient
     22 from google.cloud.pubsub_v1 import types

/usr/local/lib/python3.6/dist-packages/google/cloud/pubsub_v1/__init__.py in <module>
     16 
     17 from google.cloud.pubsub_v1 import types
---> 18 from google.cloud.pubsub_v1 import publisher
     19 from google.cloud.pubsub_v1 import subscriber
     20 

/usr/local/lib/python3.6/dist-packages/google/cloud/pubsub_v1/publisher/__init__.py in <module>
     15 from __future__ import absolute_import
     16 
---> 17 from google.cloud.pubsub_v1.publisher.client import Client
     18 
     19 

/usr/local/lib/python3.6/dist-packages/google/cloud/pubsub_v1/publisher/client.py in <module>
     27 from google.cloud.pubsub_v1 import _gapic
     28 from google.cloud.pubsub_v1 import types
---> 29 from google.cloud.pubsub_v1.gapic import publisher_client
     30 from google.cloud.pubsub_v1.gapic.transports import publisher_grpc_transport
     31 from google.cloud.pubsub_v1.publisher._batch import thread

/usr/local/lib/python3.6/dist-packages/google/cloud/pubsub_v1/gapic/publisher_client.py in <module>
     73 
     74 
---> 75 class PublisherClient(object):
     76     """
     77     The service that an application uses to manipulate topics, and to send

/usr/local/lib/python3.6/dist-packages/google/cloud/pubsub_v1/gapic/publisher_client.py in PublisherClient()
    243         message_storage_policy=None,
    244         kms_key_name=None,
--> 245         retry=google.api_core.gapic_v1.method.DEFAULT,
    246         timeout=google.api_core.gapic_v1.method.DEFAULT,
    247         metadata=None,

AttributeError: module 'google.api_core' has no attribute 'gapic_v1'

What did you expect to happen:

visualization shows

Environment:

How did you deploy Kubeflow Pipelines (KFP)?
standalone

KFP version:
1.0.0-rc.2

Anything else you would like to add:

[Miscellaneous information that will assist in solving the issue.]

/kind bug

[Bobgy]

/cc @rmgogogo @numerology @jingzhang36

This is 1.0 blocking.

/assign @Bobgy
assign myself, but if any of you already know how to fix it, please go ahead

[Bobgy]

Probably related to https://stackoverflow.com/questions/62474473/breaking-change-for-google-api-python-client-1-9-3-attributeerror-module-goo

[numerology]

I think according to the stackoverflow, it might worth trying pinning the version of google-api-core in backend/src/apiserver/visualization/requirements.txt

Another question is that shall we consider upgrading tfma/tfdv version in viz server? The latest version should be 0.22.2

[Bobgy]

Thanks @numerology, yes I've been tracking upgrade dependencies in #4084 issue

[Bobgy]

This might be solved by #4084, we should try upgrading all versions first.

[Bobgy]

/reopen

@jingzhang36 I cherry picked this to rc3 and deployed it, but the error still exists.

[jingzhang36]

tfx 0.22.0 pulls in google-api-core 1.21.0, which suffers a regression googleapis/python-pubsub#115.
Moreover, google-api-core 1.17.0 suffers another regression googleapis/python-pubsub#115 (comment)
Therefore, we'll need to use google-api-core 1.16.0 and figure out a tfx version to work with it

[subodh101]

Why is that I am getting the same error?
Using:

gcr.io/ml-pipeline/visualization-server:1.0.0

tried 1.0.1, didn't see the change.

[Bobgy]

@subodh101 do you use full kubeflow 1.1?
There are visualization servers in user namespaces, you should update them instead

[subodh101]

@Bobgy Thanks! Yes, you are right. I was able to fix that.
However, there were two problems first one was the image version by default 1.0.0-rc.3 and the second one is kubeflow/website#2265.

[Bobgy]

Hi @subodh101, the version problem has been fixed in kubeflow/manifests#1488, but it hasn't been released yet.