Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kedro to Kubeflow pipeline iris AttributeError: 'NoneType' object has no attribute 'id' #211

Open
cpereir1 opened this issue Feb 6, 2023 · 10 comments
Open

Kedro to Kubeflow pipeline iris AttributeError: 'NoneType' object has no attribute 'id' #211

cpereir1 opened this issue Feb 6, 2023 · 10 comments

Comments

@cpereir1
Copy link

cpereir1 commented Feb 6, 2023

Hi, I am trying to turn the kedro iris sample project into a kubeflow pipelines deployment.

I have the following in place:

  1. Local installation of kedro and successful execution of the iris project
  2. Cloud deployment of kubeflow (with pipelines) to which I want to push the kedro-iris project
  3. The kedro project containerized and available at a remote container registry following the instructions here. All steps seem to work fine until:

Running kedro kubeflow upload-pipeline which gives me the following error:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/homebrew/bin/kedro:8 in <module>                                                            │
│                                                                                                  │
│   5 from kedro.framework.cli import main                                                         │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/kedro/framework/cli/cli.py:211 in main                │
│                                                                                                  │
│   208 │   """                                                                                    │
│   209 │   _init_plugins()                                                                        │
│   210 │   cli_collection = KedroCLI(project_path=Path.cwd())                                     │
│ ❱ 211 │   cli_collection()                                                                       │
│   212                                                                                            │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1130 in __call__                        │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/kedro/framework/cli/cli.py:139 in main                │
│                                                                                                  │
│   136 │   │   )                                                                                  │
│   137 │   │                                                                                      │
│   138 │   │   try:                                                                               │
│ ❱ 139 │   │   │   super().main(                                                                  │
│   140 │   │   │   │   args=args,                                                                 │
│   141 │   │   │   │   prog_name=prog_name,                                                       │
│   142 │   │   │   │   complete_var=complete_var,                                                 │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1055 in main                            │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1657 in invoke                          │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1657 in invoke                          │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1404 in invoke                          │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:760 in invoke                           │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/click/decorators.py:26 in new_func                    │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/kedro_kubeflow/cli.py:200 in upload_pipeline          │
│                                                                                                  │
│   197 │   context_helper = ctx.obj["context_helper"]                                             │
│   198 │   config = context_helper.config.run_config                                              │
│   199 │                                                                                          │
│ ❱ 200 │   context_helper.kfp_client.upload(                                                      │
│   201 │   │   pipeline_name=pipeline,                                                            │
│   202 │   │   image=image if image else config.image,                                            │
│   203 │   │   image_pull_policy=config.image_pull_policy,                                        │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/kedro_kubeflow/kfpclient.py:112 in upload             │
│                                                                                                  │
│   109 │   │   │   version_id = self._upload_pipeline_version(pipeline, pipeline_id)              │
│   110 │   │   │   self.log.info("New version of pipeline created: %s", version_id)               │
│   111 │   │   else:                                                                              │
│ ❱ 112 │   │   │   (pipeline_id, version_id) = self._upload_pipeline(                             │
│   113 │   │   │   │   pipeline, full_pipeline_name                                               │
│   114 │   │   │   )                                                                              │
│   115 │   │   │   self.log.info("Pipeline created")                                              │
│                                                                                                  │
│ /opt/homebrew/lib/python3.10/site-packages/kedro_kubeflow/kfpclient.py:146 in _upload_pipeline   │
│                                                                                                  │
│   143 │   │   │   │   description=self.pipeline_description,                                     │
│   144 │   │   │   │   _request_timeout=10000,                                                    │
│   145 │   │   │   )                                                                              │
│ ❱ 146 │   │   │   return (pipeline.id, pipeline.default_version.id)                              │
│   147 │                                                                                          │
│   148 │   def _ensure_experiment_exists(self, experiment_name, experiment_namespace):            │
│   149 │   │   try: 

AttributeError: 'NoneType' object has no attribute 'id'

This is my kubeflow.yaml:

host: https://<host>/pipelines/?ns=<namespace>
run_config:
  image: <image_in_remote_container_url>
  image_pull_policy: Always #IfNotPresent
  experiment_name: test-experiment
  run_name: iris-test-run
  scheduled_run_name: 
  description: "Test kedro2kubeflow iris training pipeline"
  wait_for_completion: False
  ttl: 604800
  store_kedro_outputs_as_kfp_artifacts: False
  volume:
    storageclass:
    size: 1Gi
    access_modes: [ReadWriteMany]
    skip_init: False
    owner: 0
    keep: False
  resources:
    split_data:
      memory: 1Gi
    make_predictions:
      cpu: 1
      memory: 1Gi
    report_accuracy:
      cpu: 0.5
      memory: 1Gi

    __default__:
      cpu: 1
      memory: 1Gi

  retry_policy:
    wait_for_partition_availability:
      num_retries: 90
      backoff_duration: 5m
      backoff_factor: 1

    __default__:
      num_retries: 4
      backoff_duration: 60s
      backoff_factor: 2

  tolerations:
    __default__:
    - key: "dedicated"
      operator: "Equal"
      value: "ml-ops"
      effect: "NoSchedule"

What could I be doing wrong?

Local Kedro version: 0.18.4

Thank you!
@szczeles
Copy link
Contributor

szczeles commented Feb 8, 2023

Hey @cpereir1! That's interesting issue, let's investigate together. Could you please provide me a bit more of a context?

  1. Please run the kedro kubeflow compile and attach the generated yaml -> I will check it on my testbed.
  2. Attach the result of pip freeze so I can ensure I use the same libraries versions as you.

@cpereir1
Copy link
Author

cpereir1 commented Feb 8, 2023

Hi @szczeles ! Thank you so much for your reply :)

Running kedro kubeflow compile does not generate a yaml for me? It generates the following warning in the terminal:

[02/08/23 10:33:50] WARNING  /opt/homebrew/lib/python3.10/site-packages/kedro_viz/models warnings.py:109
                             /experiment_tracking.py:18: MovedIn20Warning:                              
                             [31mDeprecated API features detected! These feature(s) are                 
                             not compatible with SQLAlchemy 2.0. [32mTo prevent                         
                             incompatible upgrades prior to updating applications,                      
                             ensure requirements files are pinned to "sqlalchemy<2.0".                  
                             [36mSet environment variable SQLALCHEMY_WARN_20=1 to show                  
                             all deprecation warnings.  Set environment variable                        
                             SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this                          
                             message.[0m (Background on SQLAlchemy 2.0 at:                              
                             https://sqlalche.me/e/b8d9)                                                
                               Base = declarative_base()                                                
                                                                                                        
[02/08/23 10:34:03] WARNING  Failed to send data to Heap. Exception of type 'SSLError' was plugin.py:213
                             raised.                                                                    
[02/08/23 10:34:05] WARNING  Failed to send data to Heap. Exception of type 'SSLError' was plugin.py:213
                             raised.                                                                    
[02/08/23 10:34:06] WARNING  /opt/homebrew/lib/python3.10/site-packages/kfp/components/_ warnings.py:109
                             data_passing.py:227: UserWarning: Missing type name was                    
                             inferred as "Float" based on the value "0.8".                              
                               warnings.warn(                                                           
                                                                                                        
                    WARNING  /opt/homebrew/lib/python3.10/site-packages/kfp/components/_ warnings.py:109
                             data_passing.py:227: UserWarning: Missing type name was                    
                             inferred as "Integer" based on the value "3".                              
                               warnings.warn(

Running kedro kubeflow init <host> does generate a yaml (attached in .txt so it is allowed by github). Running kedro kubeflow compile after does not change the yaml file.

pipfreeze.txt
kubeflowyaml.txt

@szczeles
Copy link
Contributor

szczeles commented Feb 8, 2023

If you use the latest plugin version, kedro kubeflow compile should generate file pipeline.yml in the root directory: https://github.com/getindata/kedro-kubeflow/blob/0.7.3/kedro_kubeflow/cli.py#L162

@cpereir1
Copy link
Author

cpereir1 commented Feb 8, 2023

Oops, apologies. Yes it does. Here it goes attached.
pipeline_yml.txt

@szczeles
Copy link
Contributor

szczeles commented Feb 8, 2023

Hm, all looks good, both pipeline yaml and the packages you have installed. Maybe the exception is a result of bad handling of connectivity issue. Can you please run the following?

$ kedro kubeflow list-pipelines

@cpereir1
Copy link
Author

cpereir1 commented Feb 8, 2023

Hmm, I guess that is the issue. How could I correctly configure the connectivity or authenticate to the Kubeflow instance from kedro/kedro kubeflow?

This is the output kedro kubeflow list-pipelines

[02/08/23 16:11:29] WARNING  /opt/homebrew/lib/python3.10/site-packages/kedro_vi warnings.py:109
                             z/models/experiment_tracking.py:18:                                
                             MovedIn20Warning: [31mDeprecated API features                      
                             detected! These feature(s) are not compatible with                 
                             SQLAlchemy 2.0. [32mTo prevent incompatible                        
                             upgrades prior to updating applications, ensure                    
                             requirements files are pinned to "sqlalchemy<2.0".                 
                             [36mSet environment variable SQLALCHEMY_WARN_20=1                  
                             to show all deprecation warnings.  Set environment                 
                             variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to                      
                             silence this message.[0m (Background on SQLAlchemy                 
                             2.0 at: https://sqlalche.me/e/b8d9)                                
                               Base = declarative_base()                                        
                                                                                                
[02/08/23 16:11:35] WARNING  Failed to send data to Heap. Exception of type        plugin.py:213
                             'SSLError' was raised.                                             
                    WARNING  Failed to send data to Heap. Exception of type        plugin.py:213
                             'SSLError' was raised.                                             
                    WARNING  Retrying (Retry(total=2, connect=None,        connectionpool.py:812
                             read=None, redirect=None, status=None)) after                      
                             connection broken by                                               
                             'NewConnectionError('<urllib3.connection.HTTP                      
                             SConnection object at 0x12f125090>: Failed to                      
                             establish a new connection: [Errno 8]                              
                             nodename nor servname provided, or not                             
                             known')': /apis/v1beta1/healthz                                    
                    WARNING  Retrying (Retry(total=1, connect=None,        connectionpool.py:812
                             read=None, redirect=None, status=None)) after                      
                             connection broken by                                               
                             'NewConnectionError('<urllib3.connection.HTTP                      
                             SConnection object at 0x12f124e50>: Failed to                      
                             establish a new connection: [Errno 8]                              
                             nodename nor servname provided, or not                             
                             known')': /apis/v1beta1/healthz                                    
                    WARNING  Retrying (Retry(total=0, connect=None,        connectionpool.py:812
                             read=None, redirect=None, status=None)) after                      
                             connection broken by                                               
                             'NewConnectionError('<urllib3.connection.HTTP                      
                             SConnection object at 0x12f124f40>: Failed to                      
                             establish a new connection: [Errno 8]                              
                             nodename nor servname provided, or not                             
                             known')': /apis/v1beta1/healthz                                    
╭───────────────────────────── Traceback (most recent call last) ──────────────────────────────╮
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connection.py:174 in _new_conn            │
│                                                                                              │
│   171 │   │   │   extra_kw["socket_options"] = self.socket_options                           │
│   172 │   │                                                                                  │
│   173 │   │   try:                                                                           │
│ ❱ 174 │   │   │   conn = connection.create_connection(                                       │
│   175 │   │   │   │   (self._dns_host, self.port), self.timeout, **extra_kw                  │
│   176 │   │   │   )                                                                          │
│   177                                                                                        │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/util/connection.py:72 in                  │
│ create_connection                                                                            │
│                                                                                              │
│    69 │   │   │   LocationParseError(u"'%s', label empty or too long" % host), None          │
│    70 │   │   )                                                                              │
│    71 │                                                                                      │
│ ❱  72 │   for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):             │
│    73 │   │   af, socktype, proto, canonname, sa = res                                       │
│    74 │   │   sock = None                                                                    │
│    75 │   │   try:                                                                           │
│                                                                                              │
│ /opt/homebrew/Cellar/[email protected]/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python │
│ 3.10/socket.py:955 in getaddrinfo                                                            │
│                                                                                              │
│   952 │   # We override this function since we want to translate the numeric family          │
│   953 │   # and socket type values to enum constants.                                        │
│   954 │   addrlist = []                                                                      │
│ ❱ 955 │   for res in _socket.getaddrinfo(host, port, family, type, proto, flags):            │
│   956 │   │   af, socktype, proto, canonname, sa = res                                       │
│   957 │   │   addrlist.append((_intenum_converter(af, AddressFamily),                        │
│   958 │   │   │   │   │   │    _intenum_converter(socktype, SocketKind),                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

╭───────────────────────────── Traceback (most recent call last) ──────────────────────────────╮
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:703 in urlopen          │
│                                                                                              │
│    700 │   │   │   │   self._prepare_proxy(conn)                                             │
│    701 │   │   │                                                                             │
│    702 │   │   │   # Make the request on the httplib connection object.                      │
│ ❱  703 │   │   │   httplib_response = self._make_request(                                    │
│    704 │   │   │   │   conn,                                                                 │
│    705 │   │   │   │   method,                                                               │
│    706 │   │   │   │   url,                                                                  │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:386 in _make_request    │
│                                                                                              │
│    383 │   │                                                                                 │
│    384 │   │   # Trigger any extra validation we need to do.                                 │
│    385 │   │   try:                                                                          │
│ ❱  386 │   │   │   self._validate_conn(conn)                                                 │
│    387 │   │   except (SocketTimeout, BaseSSLError) as e:                                    │
│    388 │   │   │   # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout.     │
│    389 │   │   │   self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)           │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:1042 in _validate_conn  │
│                                                                                              │
│   1039 │   │                                                                                 │
│   1040 │   │   # Force connect early to allow us to validate the connection.                 │
│   1041 │   │   if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`      │
│ ❱ 1042 │   │   │   conn.connect()                                                            │
│   1043 │   │                                                                                 │
│   1044 │   │   if not conn.is_verified:                                                      │
│   1045 │   │   │   warnings.warn(                                                            │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connection.py:358 in connect              │
│                                                                                              │
│   355 │                                                                                      │
│   356 │   def connect(self):                                                                 │
│   357 │   │   # Add certificate verification                                                 │
│ ❱ 358 │   │   self.sock = conn = self._new_conn()                                            │
│   359 │   │   hostname = self.host                                                           │
│   360 │   │   tls_in_tls = False                                                             │
│   361                                                                                        │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connection.py:186 in _new_conn            │
│                                                                                              │
│   183 │   │   │   )                                                                          │
│   184 │   │                                                                                  │
│   185 │   │   except SocketError as e:                                                       │
│ ❱ 186 │   │   │   raise NewConnectionError(                                                  │
│   187 │   │   │   │   self, "Failed to establish a new connection: %s" % e                   │
│   188 │   │   │   )                                                                          │
│   189                                                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x12f125db0>: Failed to 
establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

╭───────────────────────────── Traceback (most recent call last) ──────────────────────────────╮
│ /opt/homebrew/bin/kedro:8 in <module>                                                        │
│                                                                                              │
│   5 from kedro.framework.cli import main                                                     │
│   6 if __name__ == '__main__':                                                               │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                     │
│ ❱ 8 │   sys.exit(main())                                                                     │
│   9                                                                                          │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kedro/framework/cli/cli.py:211 in main            │
│                                                                                              │
│   208 │   """                                                                                │
│   209 │   _init_plugins()                                                                    │
│   210 │   cli_collection = KedroCLI(project_path=Path.cwd())                                 │
│ ❱ 211 │   cli_collection()                                                                   │
│   212                                                                                        │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1130 in __call__                    │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kedro/framework/cli/cli.py:139 in main            │
│                                                                                              │
│   136 │   │   )                                                                              │
│   137 │   │                                                                                  │
│   138 │   │   try:                                                                           │
│ ❱ 139 │   │   │   super().main(                                                              │
│   140 │   │   │   │   args=args,                                                             │
│   141 │   │   │   │   prog_name=prog_name,                                                   │
│   142 │   │   │   │   complete_var=complete_var,                                             │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1055 in main                        │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1657 in invoke                      │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1657 in invoke                      │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1404 in invoke                      │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:760 in invoke                       │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/click/decorators.py:26 in new_func                │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kedro_kubeflow/cli.py:53 in list_pipelines        │
│                                                                                              │
│    50 def list_pipelines(ctx):                                                               │
│    51 │   """List deployed pipeline definitions"""                                           │
│    52 │   context_helper = ctx.obj["context_helper"]                                         │
│ ❱  53 │   click.echo(context_helper.kfp_client.list_pipelines())                             │
│    54                                                                                        │
│    55                                                                                        │
│    56 @kubeflow_group.command()                                                              │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kedro_kubeflow/context_helper.py:91 in kfp_client │
│                                                                                              │
│    88 │   def kfp_client(self):                                                              │
│    89 │   │   from .kfpclient import KubeflowClient                                          │
│    90 │   │                                                                                  │
│ ❱  91 │   │   return KubeflowClient(                                                         │
│    92 │   │   │   self.config,                                                               │
│    93 │   │   │   self.project_name,                                                         │
│    94 │   │   │   self.context,                                                              │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kedro_kubeflow/kfpclient.py:39 in __init__        │
│                                                                                              │
│    36 │   │   │   │   "cookies": f"authservice_session={dex_authservice_session}"            │
│    37 │   │   │   }                                                                          │
│    38 │   │   self.host = config.host                                                        │
│ ❱  39 │   │   self.client = Client(host=self.host, **client_params)                          │
│    40 │   │                                                                                  │
│    41 │   │   self.project_name = project_name                                               │
│    42 │   │   self.pipeline_description = config.run_config.description                      │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kfp/_client.py:196 in __init__                    │
│                                                                                              │
│    193 │   │   │   api_client)                                                               │
│    194 │   │   self._healthz_api = kfp_server_api.api.healthz_service_api.HealthzServiceApi( │
│    195 │   │   │   api_client)                                                               │
│ ❱  196 │   │   if not self._context_setting['namespace'] and self.get_kfp_healthz(           │
│    197 │   │   ).multi_user is True:                                                         │
│    198 │   │   │   try:                                                                      │
│    199 │   │   │   │   with open(Client.NAMESPACE_PATH, 'r') as f:                           │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kfp/_client.py:410 in get_kfp_healthz             │
│                                                                                              │
│    407 │   │   │   │   │   'Failed getting healthz endpoint after {} attempts.'.format(      │
│    408 │   │   │   │   │   │   max_attempts))                                                │
│    409 │   │   │   try:                                                                      │
│ ❱  410 │   │   │   │   response = self._healthz_api.get_healthz()                            │
│    411 │   │   │   │   return response                                                       │
│    412 │   │   │   # ApiException, including network errors, is the only type that may       │
│    413 │   │   │   # recover after retry.                                                    │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/api/healthz_service_api.py:63 in   │
│ get_healthz                                                                                  │
│                                                                                              │
│    60 │   │   :rtype: ApiGetHealthzResponse                                                  │
│    61 │   │   """                                                                            │
│    62 │   │   kwargs['_return_http_data_only'] = True                                        │
│ ❱  63 │   │   return self.get_healthz_with_http_info(**kwargs)  # noqa: E501                 │
│    64 │                                                                                      │
│    65 │   def get_healthz_with_http_info(self, **kwargs):  # noqa: E501                      │
│    66 │   │   """Get healthz data.  # noqa: E501                                             │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/api/healthz_service_api.py:134 in  │
│ get_healthz_with_http_info                                                                   │
│                                                                                              │
│   131 │   │   # Authentication setting                                                       │
│   132 │   │   auth_settings = ['Bearer']  # noqa: E501                                       │
│   133 │   │                                                                                  │
│ ❱ 134 │   │   return self.api_client.call_api(                                               │
│   135 │   │   │   '/apis/v1beta1/healthz', 'GET',                                            │
│   136 │   │   │   path_params,                                                               │
│   137 │   │   │   query_params,                                                              │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/api_client.py:364 in call_api      │
│                                                                                              │
│   361 │   │   │   then the method will return the response directly.                         │
│   362 │   │   """                                                                            │
│   363 │   │   if not async_req:                                                              │
│ ❱ 364 │   │   │   return self.__call_api(resource_path, method,                              │
│   365 │   │   │   │   │   │   │   │      path_params, query_params, header_params,           │
│   366 │   │   │   │   │   │   │   │      body, post_params, files,                           │
│   367 │   │   │   │   │   │   │   │      response_type, auth_settings,                       │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/api_client.py:181 in __call_api    │
│                                                                                              │
│   178 │   │                                                                                  │
│   179 │   │   try:                                                                           │
│   180 │   │   │   # perform request and return response                                      │
│ ❱ 181 │   │   │   response_data = self.request(                                              │
│   182 │   │   │   │   method, url, query_params=query_params, headers=header_params,         │
│   183 │   │   │   │   post_params=post_params, body=body,                                    │
│   184 │   │   │   │   _preload_content=_preload_content,                                     │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/api_client.py:389 in request       │
│                                                                                              │
│   386 │   │   │   │   _request_timeout=None):                                                │
│   387 │   │   """Makes the HTTP request using RESTClient."""                                 │
│   388 │   │   if method == "GET":                                                            │
│ ❱ 389 │   │   │   return self.rest_client.GET(url,                                           │
│   390 │   │   │   │   │   │   │   │   │   │   query_params=query_params,                     │
│   391 │   │   │   │   │   │   │   │   │   │   _preload_content=_preload_content,             │
│   392 │   │   │   │   │   │   │   │   │   │   _request_timeout=_request_timeout,             │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/rest.py:230 in GET                 │
│                                                                                              │
│   227 │                                                                                      │
│   228 │   def GET(self, url, headers=None, query_params=None, _preload_content=True,         │
│   229 │   │   │   _request_timeout=None):                                                    │
│ ❱ 230 │   │   return self.request("GET", url,                                                │
│   231 │   │   │   │   │   │   │   headers=headers,                                           │
│   232 │   │   │   │   │   │   │   _preload_content=_preload_content,                         │
│   233 │   │   │   │   │   │   │   _request_timeout=_request_timeout,                         │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/rest.py:208 in request             │
│                                                                                              │
│   205 │   │   │   │   │   raise ApiException(status=0, reason=msg)                           │
│   206 │   │   │   # For `GET`, `HEAD`                                                        │
│   207 │   │   │   else:                                                                      │
│ ❱ 208 │   │   │   │   r = self.pool_manager.request(method, url,                             │
│   209 │   │   │   │   │   │   │   │   │   │   │     fields=query_params,                     │
│   210 │   │   │   │   │   │   │   │   │   │   │     preload_content=_preload_content,        │
│   211 │   │   │   │   │   │   │   │   │   │   │     timeout=timeout,                         │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/request.py:74 in request                  │
│                                                                                              │
│    71 │   │   urlopen_kw["request_url"] = url                                                │
│    72 │   │                                                                                  │
│    73 │   │   if method in self._encode_url_methods:                                         │
│ ❱  74 │   │   │   return self.request_encode_url(                                            │
│    75 │   │   │   │   method, url, fields=fields, headers=headers, **urlopen_kw              │
│    76 │   │   │   )                                                                          │
│    77 │   │   else:                                                                          │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/request.py:96 in request_encode_url       │
│                                                                                              │
│    93 │   │   if fields:                                                                     │
│    94 │   │   │   url += "?" + urlencode(fields)                                             │
│    95 │   │                                                                                  │
│ ❱  96 │   │   return self.urlopen(method, url, **extra_kw)                                   │
│    97 │                                                                                      │
│    98 │   def request_encode_body(                                                           │
│    99 │   │   self,                                                                          │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/poolmanager.py:376 in urlopen             │
│                                                                                              │
│   373 │   │   if self._proxy_requires_url_absolute_form(u):                                  │
│   374 │   │   │   response = conn.urlopen(method, url, **kw)                                 │
│   375 │   │   else:                                                                          │
│ ❱ 376 │   │   │   response = conn.urlopen(method, u.request_uri, **kw)                       │
│   377 │   │                                                                                  │
│   378 │   │   redirect_location = redirect and response.get_redirect_location()              │
│   379 │   │   if not redirect_location:                                                      │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:815 in urlopen          │
│                                                                                              │
│    812 │   │   │   log.warning(                                                              │
│    813 │   │   │   │   "Retrying (%r) after connection broken by '%r': %s", retries, err, ur │
│    814 │   │   │   )                                                                         │
│ ❱  815 │   │   │   return self.urlopen(                                                      │
│    816 │   │   │   │   method,                                                               │
│    817 │   │   │   │   url,                                                                  │
│    818 │   │   │   │   body,                                                                 │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:815 in urlopen          │
│                                                                                              │
│    812 │   │   │   log.warning(                                                              │
│    813 │   │   │   │   "Retrying (%r) after connection broken by '%r': %s", retries, err, ur │
│    814 │   │   │   )                                                                         │
│ ❱  815 │   │   │   return self.urlopen(                                                      │
│    816 │   │   │   │   method,                                                               │
│    817 │   │   │   │   url,                                                                  │
│    818 │   │   │   │   body,                                                                 │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:815 in urlopen          │
│                                                                                              │
│    812 │   │   │   log.warning(                                                              │
│    813 │   │   │   │   "Retrying (%r) after connection broken by '%r': %s", retries, err, ur │
│    814 │   │   │   )                                                                         │
│ ❱  815 │   │   │   return self.urlopen(                                                      │
│    816 │   │   │   │   method,                                                               │
│    817 │   │   │   │   url,                                                                  │
│    818 │   │   │   │   body,                                                                 │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:787 in urlopen          │
│                                                                                              │
│    784 │   │   │   elif isinstance(e, (SocketError, HTTPException)):                         │
│    785 │   │   │   │   e = ProtocolError("Connection aborted.", e)                           │
│    786 │   │   │                                                                             │
│ ❱  787 │   │   │   retries = retries.increment(                                              │
│    788 │   │   │   │   method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]       │
│    789 │   │   │   )                                                                         │
│    790 │   │   │   retries.sleep()                                                           │
│                                                                                              │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/util/retry.py:592 in increment            │
│                                                                                              │
│   589 │   │   )                                                                              │
│   590 │   │                                                                                  │
│   591 │   │   if new_retry.is_exhausted():                                                   │
│ ❱ 592 │   │   │   raise MaxRetryError(_pool, url, error or ResponseError(cause))             │
│   593 │   │                                                                                  │
│   594 │   │   log.debug("Incremented Retry for (url='%s'): %r", url, new_retry)              │
│   595                                                                                        │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
MaxRetryError: HTTPSConnectionPool(host='<host>', port=443): Max retries exceeded with url: 
/apis/v1beta1/healthz (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object 
at 0x12f125db0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided,
or not known'))

@szczeles
Copy link
Contributor

szczeles commented Feb 9, 2023

It looks you didn't provide a valid Kubeflow Pipelines URL when doing kedro kubeflow init, it can be fixed by editing the generated kubeflow.yml file in conf directory. It should look like this:

host: https://<your-kubeflow-base-url>/pipeline

Also, if you use Dex for authentication, you need additional setup, as described in the docs

@cpereir1
Copy link
Author

Hi @szczeles thank you. Regarding the Dex authentication additional setup: we would like to avoid hardcoding a static user & password, and we disabled this by design choice. Would there be other, more dynamic ways, of authenticating a kedro kubeflow plugin user (from local environment) to the kubeflow pipelines (living in a cloud kubernetes cluter behind dex)?

@cpereir1
Copy link
Author

Also, could you confirm that, if the kubeflow pipelines url is correct, but the error persists, the reason should be inability to authenticate? Thank you!

@szczeles
Copy link
Contributor

@cpereir1 Unfortunately, at the time, Dex doesn't allow programmatic authentication, see: kubeflow/kfctl#140. Ideally, we would just add ID Token in the headers and Dex would check its validity and return a local token (exchanged for session cookie) for kubeflow services interaction. Unfortunately, the feature request is still open since 2019...

And yes, if kubeflow pipelines URL is correct but you still see an exception, it's likely due to an authentication issue (with no DEX_* env vars set the plugin doesn't try to authenticate at all).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@szczeles @cpereir1