tests.system.aiplatform.test_model_monitoring.TestModelDeploymentMonitoring: test_mdm_two_models_one_valid_config failed #1672

flaky-bot · 2022-09-16T11:13:21Z

Note: #1629 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

commit: 95855a2
buildURL: Build Status, Sponge
status: failed

Test output

self = 
def test_mdm_two_models_one_valid_config(self):
    """
    Enable model monitoring on two existing models deployed to the same endpoint.
    """
    # test model monitoring configurations
    job = aiplatform.ModelDeploymentMonitoringJob.create(
        display_name=self._make_display_name(key=JOB_NAME),
        logging_sampling_strategy=sampling_strategy,
        schedule_config=schedule_config,
        alert_config=alert_config,
        objective_configs=objective_config,
        create_request_timeout=3600,
        project=e2e_base._PROJECT,
        location=e2e_base._LOCATION,
        endpoint=self.endpoint,
        predict_instance_schema_uri="",
        analysis_instance_schema_uri="",
    )
    assert job is not None

    gapic_job = job._gca_resource
    assert (
        gapic_job.logging_sampling_strategy.random_sample_config.sample_rate
        == LOG_SAMPLE_RATE
    )
    assert (
        gapic_job.model_deployment_monitoring_schedule_config.monitor_interval.seconds
        == MONITOR_INTERVAL * 3600
    )
    assert (
        gapic_job.model_monitoring_alert_config.email_alert_config.user_emails
        == [USER_EMAIL]
    )
    assert gapic_job.model_monitoring_alert_config.enable_logging

    gca_obj_config = gapic_job.model_deployment_monitoring_objective_configs[
        0
    ].objective_config

    expected_training_dataset = (
        gca_model_monitoring.ModelMonitoringObjectiveConfig.TrainingDataset(
            bigquery_source=gca_io.BigQuerySource(input_uri=DATASET_BQ_URI),
            target_field=TARGET,
        )
    )
    assert gca_obj_config.training_dataset == expected_training_dataset
    assert (
        gca_obj_config.training_prediction_skew_detection_config
        == skew_config.as_proto()
    )
    assert (
        gca_obj_config.prediction_drift_detection_config == drift_config.as_proto()
    )

    job_resource = job._gca_resource.name

    # test job update and delete()
    timeout = time.time() + 3600
    new_obj_config = model_monitoring.ObjectiveConfig(skew_config)

    while time.time() < timeout:
        if job.state == gca_job_state.JobState.JOB_STATE_RUNNING:


          job.update(objective_configs=new_obj_config)


tests/system/aiplatform/test_model_monitoring.py:167:

google/cloud/aiplatform/jobs.py:2486: in update

ModelDeploymentMonitoringJob._parse_configs(

cls = <class 'google.cloud.aiplatform.jobs.ModelDeploymentMonitoringJob'>

objective_configs = <google.cloud.aiplatform.model_monitoring.objective.ObjectiveConfig object at 0x7fa9e6b0b0a0>

endpoint = 'projects/580378083368/locations/us-central1/endpoints/8289570005524152320'

deployed_model_ids = None
@classmethod
def _parse_configs(
    cls,
    objective_configs: Union[
        model_monitoring.ObjectiveConfig,
        Dict[str, model_monitoring.ObjectiveConfig],
    ],
    endpoint: "aiplatform.Endpoint",
    deployed_model_ids: Optional[List[str]] = None,
) -> List[
    gca_model_deployment_monitoring_job_compat.ModelDeploymentMonitoringObjectiveConfig
]:
    """Helper function for matching objective configs with their corresponding models.

    Args:
        objective_configs (Union[model_monitoring.objective.ObjectiveConfig,
            Dict[str, model_monitoring.objective.ObjectiveConfig]):
            Required. A single config if it applies to all models, or a dictionary of
            model_id: model_monitoring.objective.ObjectiveConfig if
            different model IDs have different configs.
        endpoint (aiplatform.Endpoint):
            Required. A valid instance of aiplatforn.Endpoint to launch the MDM job on.
        deployed_model_ids (Optional[List[str]]):
            Optional. A list of deployed model IDs to apply the objective config to.
            Note that a model will have a deployed_model_id that is different from the
            uploaded model ID, and IDs in this list should consist of deployed model IDs
            on the same endpoint passed in the argument. If `objective_configs` is a dictionary,
            then this parameter is ignored. If `objective_configs` is an instance of
            `model_monitoring.ObjectiveConfig` and `deployed_model_ids` is a non-empty
            list of valid IDs, then the same objective config will apply to all models in this list.

    Returns:
        A List of ModelDeploymentMonitoringObjectiveConfig objects.

    Raises:
        ValueError, when the model IDs given are invalid.
        RuntimeError, when XAI is enabled on a model that doesn't have XAI parameters configured.
    """
    all_models = []
    xai_enabled = []


  for model in endpoint.list_models():


E       AttributeError: 'str' object has no attribute 'list_models'
google/cloud/aiplatform/jobs.py:2107: AttributeError

The text was updated successfully, but these errors were encountered:

flaky-bot · 2022-09-17T02:55:06Z

commit: 9a506ee
buildURL: Build Status, Sponge
status: failed

Test output

self = 
def test_mdm_two_models_one_valid_config(self):
    """
    Enable model monitoring on two existing models deployed to the same endpoint.
    """
    # test model monitoring configurations
    job = aiplatform.ModelDeploymentMonitoringJob.create(
        display_name=self._make_display_name(key=JOB_NAME),
        logging_sampling_strategy=sampling_strategy,
        schedule_config=schedule_config,
        alert_config=alert_config,
        objective_configs=objective_config,
        create_request_timeout=3600,
        project=e2e_base._PROJECT,
        location=e2e_base._LOCATION,
        endpoint=self.endpoint,
        predict_instance_schema_uri="",
        analysis_instance_schema_uri="",
    )
    assert job is not None

    gapic_job = job._gca_resource
    assert (
        gapic_job.logging_sampling_strategy.random_sample_config.sample_rate
        == LOG_SAMPLE_RATE
    )
    assert (
        gapic_job.model_deployment_monitoring_schedule_config.monitor_interval.seconds
        == MONITOR_INTERVAL * 3600
    )
    assert (
        gapic_job.model_monitoring_alert_config.email_alert_config.user_emails
        == [USER_EMAIL]
    )
    assert gapic_job.model_monitoring_alert_config.enable_logging

    gca_obj_config = gapic_job.model_deployment_monitoring_objective_configs[
        0
    ].objective_config

    expected_training_dataset = (
        gca_model_monitoring.ModelMonitoringObjectiveConfig.TrainingDataset(
            bigquery_source=gca_io.BigQuerySource(input_uri=DATASET_BQ_URI),
            target_field=TARGET,
        )
    )
    assert gca_obj_config.training_dataset == expected_training_dataset
    assert (
        gca_obj_config.training_prediction_skew_detection_config
        == skew_config.as_proto()
    )
    assert (
        gca_obj_config.prediction_drift_detection_config == drift_config.as_proto()
    )

    job_resource = job._gca_resource.name

    # test job update and delete()
    timeout = time.time() + 3600
    new_obj_config = model_monitoring.ObjectiveConfig(skew_config)

    while time.time() < timeout:
        if job.state == gca_job_state.JobState.JOB_STATE_RUNNING:


          job.update(objective_configs=new_obj_config)


tests/system/aiplatform/test_model_monitoring.py:167:

google/cloud/aiplatform/jobs.py:2486: in update

ModelDeploymentMonitoringJob._parse_configs(

cls = <class 'google.cloud.aiplatform.jobs.ModelDeploymentMonitoringJob'>

objective_configs = <google.cloud.aiplatform.model_monitoring.objective.ObjectiveConfig object at 0x7f2fc5c315b0>

endpoint = 'projects/580378083368/locations/us-central1/endpoints/8289570005524152320'

deployed_model_ids = None
@classmethod
def _parse_configs(
    cls,
    objective_configs: Union[
        model_monitoring.ObjectiveConfig,
        Dict[str, model_monitoring.ObjectiveConfig],
    ],
    endpoint: "aiplatform.Endpoint",
    deployed_model_ids: Optional[List[str]] = None,
) -> List[
    gca_model_deployment_monitoring_job_compat.ModelDeploymentMonitoringObjectiveConfig
]:
    """Helper function for matching objective configs with their corresponding models.

    Args:
        objective_configs (Union[model_monitoring.objective.ObjectiveConfig,
            Dict[str, model_monitoring.objective.ObjectiveConfig]):
            Required. A single config if it applies to all models, or a dictionary of
            model_id: model_monitoring.objective.ObjectiveConfig if
            different model IDs have different configs.
        endpoint (aiplatform.Endpoint):
            Required. A valid instance of aiplatforn.Endpoint to launch the MDM job on.
        deployed_model_ids (Optional[List[str]]):
            Optional. A list of deployed model IDs to apply the objective config to.
            Note that a model will have a deployed_model_id that is different from the
            uploaded model ID, and IDs in this list should consist of deployed model IDs
            on the same endpoint passed in the argument. If `objective_configs` is a dictionary,
            then this parameter is ignored. If `objective_configs` is an instance of
            `model_monitoring.ObjectiveConfig` and `deployed_model_ids` is a non-empty
            list of valid IDs, then the same objective config will apply to all models in this list.

    Returns:
        A List of ModelDeploymentMonitoringObjectiveConfig objects.

    Raises:
        ValueError, when the model IDs given are invalid.
        RuntimeError, when XAI is enabled on a model that doesn't have XAI parameters configured.
    """
    all_models = []
    xai_enabled = []


  for model in endpoint.list_models():


E       AttributeError: 'str' object has no attribute 'list_models'
google/cloud/aiplatform/jobs.py:2107: AttributeError

rosiezou · 2022-09-20T00:53:15Z

Will be fixed when #1671 lands

flaky-bot · 2022-09-21T03:37:11Z

commit: 1cda4b4
buildURL: Build Status, Sponge
status: failed

Test output

args = (parent: "projects/ucaip-sample-tests/locations/us-central1" model_deployment_monitoring_job { display_name: "temp_e... user_emails: "" } enable_logging: true } sample_predict_instance { null_value: NULL_VALUE } } ,) kwargs = {'metadata': [('x-goog-request-params', 'parent=projects/ucaip-sample-tests/locations/us-central1'), ('x-goog-api-client', 'model-builder/1.17.1 gl-python/3.8.13 grpc/1.47.0 gax/1.32.0 gapic/1.17.1')], 'timeout': 3600}

@six.wraps(callable_)
def error_remapped_callable(*args, **kwargs):
    try:

      return callable_(*args, **kwargs)

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py:67:

self = <grpc._channel._UnaryUnaryMultiCallable object at 0x7fb7ed1c2550>
request = parent: "projects/ucaip-sample-tests/locations/us-central1"
model_deployment_monitoring_job {
display_name: "temp_e2...
user_emails: ""
}
enable_logging: true
}
sample_predict_instance {
null_value: NULL_VALUE
}
}

timeout = 3600
metadata = [('x-goog-request-params', 'parent=projects/ucaip-sample-tests/locations/us-central1'), ('x-goog-api-client', 'model-builder/1.17.1 gl-python/3.8.13 grpc/1.47.0 gax/1.32.0 gapic/1.17.1')]
credentials = None, wait_for_ready = None, compression = None

def __call__(self,
             request,
             timeout=None,
             metadata=None,
             credentials=None,
             wait_for_ready=None,
             compression=None):
    state, call, = self._blocking(request, timeout, metadata, credentials,
                                  wait_for_ready, compression)

  return _end_unary_response_blocking(state, call, False, None)

.nox/system-3-8/lib/python3.8/site-packages/grpc/_channel.py:946:

state = <grpc._channel._RPCState object at 0x7fb7ec0e94f0>
call = <grpc._cython.cygrpc.SegregatedCall object at 0x7fb7ec1105c0>
with_call = False, deadline = None

def _end_unary_response_blocking(state, call, with_call, deadline):
    if state.code is grpc.StatusCode.OK:
        if with_call:
            rendezvous = _MultiThreadedRendezvous(state, call, None, deadline)
            return state.response, rendezvous
        else:
            return state.response
    else:

      raise _InactiveRpcError(state)

E grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
E status = StatusCode.FAILED_PRECONDITION
E details = "ModelMonitoring config [endpoint: 8289570005524152320
E model_monitoring_schedule_config {
E monitoring_interval {
E seconds: 3600
E }
E monitoring_window {
E seconds: 3600
E }
E }
E logging_sampling_strategy {
E random_sample_config {
E sample_rate: 0.8
E }
E }
E model_monitoring_alert_config {
E email_alert_config {
E user_emails: ""
E }
E enable_logging: true
E }
E bigquery_tables {
E log_source: SERVING
E log_type: PREDICT
E bigquery_table_path: "bq://ucaip-sample-tests.model_deployment_monitoring_8289570005524152320.serving_predict"
E }
E is_enabled: true
E predict_instance_schema_uri: "gs://cloud-ai-platform-62f88573-082b-416d-a1b7-e47dcc520767/instance_schemas/job-8376379472194043904/predict"
E predict_instance_schema {
E type_url: "type.googleapis.com/cloud.ai.platform.openapi.Schema"
E value: "\b\006\032\031\n\023cnt_user_engagement\022\002\b\002\032\r\n\acountry\022\002\b\001\032 \n\032cnt_spend_virtual_currency\022\002\b\002\032"\n\034cnt_level_complete_quickplay\022\002\b\002\032\016\n\blanguage\022\002\b\001\032\034\n\026cnt_challenge_a_friend\022\002\b\002\032\037\n\031cnt_level_start_quickplay\022\002\b\002\032\031\n\023cnt_use_extra_steps\022\002\b\002\032\023\n\rcnt_ad_reward\022\002\b\002\032\017\n\tjulianday\022\002\b\002\032\v\n\005month\022\002\b\002\032\037\n\031cnt_level_reset_quickplay\022\002\b\002\032\035\n\027cnt_level_end_quickplay\022\002\b\002\032\017\n\tdayofweek\022\002\b\002\032\026\n\020operating_system\022\002\b\001\032\024\n\016user_pseudo_id\022\002\b\001\032\034\n\026cnt_completed_5_levels\022\002\b\002\032\024\n\016cnt_post_score\022\002\b\0022\324\004*\321\004\n \n\023cnt_user_engagement\022\t\021\000\000\000\000\236j"A\n\024\n\acountry\022\t\032\aDenmark\n'\n\032cnt_spend_virtual_currency\022\t\021\000\000\000\000\000\000\000\000\n)\n\034cnt_level_complete_quickplay\022\t\021\000\000\000\000\000\000\b@\n\023\n\blanguage\022\a\032\005da-dk\n#\n\026cnt_challenge_a_friend\022\t\021\000\000\000\000\000\000\000\000\n&\n\031cnt_level_start_quickplay\022\t\021\000\000\000\000\000\000\030@\n \n\023cnt_use_extra_steps\022\t\021\000\000\000\000\000\000\000\000\n\032\n\rcnt_ad_reward\022\t\021\000\000\000\000\000\000\000\000\n\026\n\tjulianday\022\t\021\000\000\000\000\000\300o@\n\022\n\005month\022\t\021\000\000\000\000\000\000"@\n&\n\031cnt_level_reset_quickplay\022\t\021\000\000\000\000\000\000\000@\n$\n\027cnt_level_end_quickplay\022\t\021\000\000\000\000\000\000\024@\n\026\n\tdayofweek\022\t\021\000\000\000\000\000\000\b@\n\031\n\020operating_system\022\005\032\003IOS\n4\n\016user_pseudo_id\022"\032 104B0770BAE16E8B53DF330C95881893\n#\n\026cnt_completed_5_levels\022\t\021\000\000\000\000\000\000\360?\n\033\n\016cnt_post_score\022\t\021\000\000\000\000\000\000A@"
E }
E tfdv_instance_schema_uri: "gs://cloud-ai-platform-62f88573-082b-416d-a1b7-e47dcc520767/instance_schemas/job-8376379472194043904/analysis"
E tfdv_instance_schema {
E type_url: "type.googleapis.com/cloud.ai.platform.openapi.Schema"
E value: "\b\006\032\031\n\023cnt_user_engagement\022\002\b\002\032\r\n\acountry\022\002\b\001\032 \n\032cnt_spend_virtual_currency\022\002\b\002\032"\n\034cnt_level_complete_quickplay\022\002\b\002\032\016\n\blanguage\022\002\b\001\032\034\n\026cnt_challenge_a_friend\022\002\b\002\032\037\n\031cnt_level_start_quickplay\022\002\b\002\032\031\n\023cnt_use_extra_steps\022\002\b\002\032\023\n\rcnt_ad_reward\022\002\b\002\032\017\n\tjulianday\022\002\b\002\032\v\n\005month\022\002\b\002\032\037\n\031cnt_level_reset_quickplay\022\002\b\002\032\035\n\027cnt_level_end_quickplay\022\002\b\002\032\017\n\tdayofweek\022\002\b\002\032\026\n\020operating_system\022\002\b\001\032\024\n\016user_pseudo_id\022\002\b\001\032\034\n\026cnt_completed_5_levels\022\002\b\002\032\024\n\016cnt_post_score\022\002\b\0022\324\004*\321\004\n \n\023cnt_user_engagement\022\t\021\000\000\000\000\236j"A\n\024\n\acountry\022\t\032\aDenmark\n'\n\032cnt_spend_virtual_currency\022\t\021\000\000\000\000\000\000\000\000\n)\n\034cnt_level_complete_quickplay\022\t\021\000\000\000\000\000\000\b@\n\023\n\blanguage\022\a\032\005da-dk\n#\n\026cnt_challenge_a_friend\022\t\021\000\000\000\000\000\000\000\000\n&\n\031cnt_level_start_quickplay\022\t\021\000\000\000\000\000\000\030@\n \n\023cnt_use_extra_steps\022\t\021\000\000\000\000\000\000\000\000\n\032\n\rcnt_ad_reward\022\t\021\000\000\000\000\000\000\000\000\n\026\n\tjulianday\022\t\021\000\000\000\000\000\300o@\n\022\n\005month\022\t\021\000\000\000\000\000\000"@\n&\n\031cnt_level_reset_quickplay\022\t\021\000\000\000\000\000\000\000@\n$\n\027cnt_level_end_quickplay\022\t\021\000\000\000\000\000\000\024@\n\026\n\tdayofweek\022\t\021\000\000\000\000\000\000\b@\n\031\n\020operating_system\022\005\032\003IOS\n4\n\016user_pseudo_id\022"\032 104B0770BAE16E8B53DF330C95881893\n#\n\026cnt_completed_5_levels\022\t\021\000\000\000\000\000\000\360?\n\033\n\016cnt_post_score\022\t\021\000\000\000\000\000\000A@"
E }
E log_ttl {
E }
E job_id: 8376379472194043904
E ] already exitsts in Endpoint projects/580378083368/locations/us-central1/endpoints/8289570005524152320, please update the existing ModelMonitoringJob."
E debug_error_string = "{"created":"@1663718443.425050684","description":"Error received from peer ipv4:173.194.202.95:443","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"ModelMonitoring config [endpoint: 8289570005524152320\nmodel_monitoring_schedule_config {\n monitoring_interval {\n seconds: 3600\n }\n monitoring_window {\n seconds: 3600\n }\n}\nlogging_sampling_strategy {\n random_sample_config {\n sample_rate: 0.8\n }\n}\nmodel_monitoring_alert_config {\n email_alert_config {\n user_emails: ""\n }\n enable_logging: true\n}\nbigquery_tables {\n log_source: SERVING\n log_type: PREDICT\n bigquery_table_path: "bq://ucaip-sample-tests.model_deployment_monitoring_8289570005524152320.serving_predict"\n}\nis_enabled: true\npredict_instance_schema_uri: "gs://cloud-ai-platform-62f88573-082b-416d-a1b7-e47dcc520767/instance_schemas/job-8376379472194043904/predict"\npredict_instance_schema {\n type_url: "type.googleapis.com/cloud.ai.platform.openapi.Schema"\n value: "\b\006\032\031\n\023cnt_user_engagement\022\002\b\002\032\r\n\acountry\022\002\b\001\032 \n\032cnt_spend_virtual_currency\022\002\b\002\032"\n\034cnt_level_complete_quickplay\022\002\b\002\032\016\n\blanguage\022\002\b\001\032\034\n\026cnt_challenge_a_friend\022\002\b\002\032\037\n\031cnt_level_start_quickplay\022\002\b\002\032\031\n\023cnt_use_extra_steps\022\002\b\002\032\023\n\rcnt_ad_reward\022\002\b\002\032\017\n\tjulianday\022\002\b\002\032\v\n\005month\022\002\b\002\032\037\n\031cnt_level_reset_quickplay\022\002\b\002\032\035\n\027cnt_level_end_quickplay\022\002\b\002\032\017\n\tdayofweek\022\002\b\002\032\026\n\020operating_system\022\002\b\001\032\024\n\016user_pseudo_id\022\002\b\001\032\034\n\026cnt_completed_5_levels\022\002\b\002\032\024\n\016cnt_post_score\022\002\b\0022\324\004*\321\004\n \n\023cnt_user_engagement\022\t\021\000\000\000\000\236j"A\n\024\n\acountry\022\t\032\aDenmark\n'\n\032cnt_spend_virtual_currency\022\t\021\000\000\000\000\000\000\000\000\n)\n\034cnt_level_complete_quickplay\022\t\021\000\000\000\000\000\000\b@\n\023\n\blanguage\022\a\032\005da-dk\n#\n\026cnt_challenge_a_friend\022\t\021\000\000\000\000\000\000\000\000\n&\n\031cnt_level_start_quickplay\022\t\021\000\000\000\000\000\000\030@\n \n\023cnt_use_extra_steps\022\t\021\000\000\000\000\000\000\000\000\n\032\n\rcnt_ad_reward\022\t\021\000\000\000\000\000\000\000\000\n\026\n\tjulianday\022\t\021\000\000\000\000\000\300o@\n\022\n\005month\022\t\021\000\000\000\000\000\000"@\n&\n\031cnt_level_reset_quickplay\022\t\021\000\000\000\000\000\000\000@\n$\n\027cnt_level_end_quickplay\022\t\021\000\000\000\000\000\000\024@\n\026\n\tdayofweek\022\t\021\000\000\000\000\000\000\b@\n\031\n\020operating_system\022\005\032\003IOS\n4\n\016user_pseudo_id\022"\032 104B0770BAE16E8B53DF330C95881893\n#\n\026cnt_completed_5_levels\022\t\021\000\000\000\000\000\000\360?\n\033\n\016cnt_post_score\022\t\021\000\000\000\000\000\000A@"\n}\ntfdv_instance_schema_uri: "gs://cloud-ai-platform-62f88573-082b-416d-a1b7-e47dcc520767/instance_schemas/job-8376379472194043904/analysis"\ntfdv_instance_schema {\n type_url: "type.googleapis.com/cloud.ai.platform.openapi.Schema"\n value: "\b\006\032\031\n\023cnt_user_engagement\022\002\b\002\032\r\n\acountry\022\002\b\001\032 \n\032cnt_spend_virtual_currency\022\002\b\002\032"\n\034cnt_level_complete_quickplay\022\002\b\002\032\016\n\blanguage\022\002\b\001\032\034\n\026cnt_challenge_a_friend\022\002\b\002\032\037\n\031cnt_level_start_quickplay\022\002\b\002\032\031\n\023cnt_use_extra_steps\022\002\b\002\032\023\n\rcnt_ad_reward\022\002\b\002\032\017\n\tjulianday\022\002\b\002\032\v\n\005month\022\002\b\002\032\037\n\031cnt_level_reset_quickplay\022\002\b\002\032\035\n\027cnt_level_end_quickplay\022\002\b\002\032\017\n\tdayofweek\022\002\b\002\032\026\n\020operating_system\022\002\b\001\032\024\n\016user_pseudo_id\022\002\b\001\032\034\n\026cnt_completed_5_levels\022\002\b\002\032\024\n\016cnt_post_score\022\002\b\0022\324\004*\321\004\n \n\023cnt_user_engagement\022\t\021\000\000\000\000\236j"A\n\024\n\acountry\022\t\032\aDenmark\n'\n\032cnt_spend_virtual_currency\022\t\021\000\000\000\000\000\000\000\000\n)\n\034cnt_level_complete_quickplay\022\t\021\000\000\000\000\000\000\b@\n\023\n\blanguage\022\a\032\005da-dk\n#\n\026cnt_challenge_a_friend\022\t\021\000\000\000\000\000\000\000\000\n&\n\031cnt_level_start_quickplay\022\t\021\000\000\000\000\000\000\030@\n \n\023cnt_use_extra_steps\022\t\021\000\000\000\000\000\000\000\000\n\032\n\rcnt_ad_reward\022\t\021\000\000\000\000\000\000\000\000\n\026\n\tjulianday\022\t\021\000\000\000\000\000\300o@\n\022\n\005month\022\t\021\000\000\000\000\000\000"@\n&\n\031cnt_level_reset_quickplay\022\t\021\000\000\000\000\000\000\000@\n$\n\027cnt_level_end_quickplay\022\t\021\000\000\000\000\000\000\024@\n\026\n\tdayofweek\022\t\021\000\000\000\000\000\000\b@\n\031\n\020operating_system\022\005\032\003IOS\n4\n\016user_pseudo_id\022"\032 104B0770BAE16E8B53DF330C95881893\n#\n\026cnt_completed_5_levels\022\t\021\000\000\000\000\000\000\360?\n\033\n\016cnt_post_score\022\t\021\000\000\000\000\000\000A@"\n}\nlog_ttl {\n}\njob_id: 8376379472194043904\n] already exitsts in Endpoint projects/580378083368/locations/us-central1/endpoints/8289570005524152320, please update the existing ModelMonitoringJob.","grpc_status":9}"
E >

.nox/system-3-8/lib/python3.8/site-packages/grpc/_channel.py:849: _InactiveRpcError

The above exception was the direct cause of the following exception:

self = <tests.system.aiplatform.test_model_monitoring.TestModelDeploymentMonitoring object at 0x7fb7ef9a1a00>

def test_mdm_two_models_one_valid_config(self):
    """
    Enable model monitoring on two existing models deployed to the same endpoint.
    """
    # test model monitoring configurations

  job = aiplatform.ModelDeploymentMonitoringJob.create(

        display_name=self._make_display_name(key=JOB_NAME),
        logging_sampling_strategy=sampling_strategy,
        schedule_config=schedule_config,
        alert_config=alert_config,
        objective_configs=objective_config,
        create_request_timeout=3600,
        project=e2e_base._PROJECT,
        location=e2e_base._LOCATION,
        endpoint=self.endpoint,
        predict_instance_schema_uri="",
        analysis_instance_schema_uri="",
    )

tests/system/aiplatform/test_model_monitoring.py:110:

google/cloud/aiplatform/jobs.py:2355: in create
self._gca_resource = self.api_client.create_model_deployment_monitoring_job(
google/cloud/aiplatform_v1/services/job_service/client.py:3111: in create_model_deployment_monitoring_job
response = rpc(
.nox/system-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py:145: in call
return wrapped_func(*args, **kwargs)
.nox/system-3-8/lib/python3.8/site-packages/google/api_core/timeout.py:102: in func_with_timeout
return func(*args, **kwargs)
.nox/system-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py:69: in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)

value = None
from_value = <_InactiveRpcError of RPC that terminated with:
status = StatusCode.FAILED_PRECONDITION
details = "ModelMonitoring c...ocations/us-central1/endpoints/8289570005524152320, please update the existing ModelMonitoringJob.","grpc_status":9}"

???
E google.api_core.exceptions.FailedPrecondition: 400 ModelMonitoring config [endpoint: 8289570005524152320
E model_monitoring_schedule_config {
E monitoring_interval {
E seconds: 3600
E }
E monitoring_window {
E seconds: 3600
E }
E }
E logging_sampling_strategy {
E random_sample_config {
E sample_rate: 0.8
E }
E }
E model_monitoring_alert_config {
E email_alert_config {
E user_emails: ""
E }
E enable_logging: true
E }
E bigquery_tables {
E log_source: SERVING
E log_type: PREDICT
E bigquery_table_path: "bq://ucaip-sample-tests.model_deployment_monitoring_8289570005524152320.serving_predict"
E }
E is_enabled: true
E predict_instance_schema_uri: "gs://cloud-ai-platform-62f88573-082b-416d-a1b7-e47dcc520767/instance_schemas/job-8376379472194043904/predict"
E predict_instance_schema {
E type_url: "type.googleapis.com/cloud.ai.platform.openapi.Schema"
E value: "\b\006\032\031\n\023cnt_user_engagement\022\002\b\002\032\r\n\acountry\022\002\b\001\032 \n\032cnt_spend_virtual_currency\022\002\b\002\032"\n\034cnt_level_complete_quickplay\022\002\b\002\032\016\n\blanguage\022\002\b\001\032\034\n\026cnt_challenge_a_friend\022\002\b\002\032\037\n\031cnt_level_start_quickplay\022\002\b\002\032\031\n\023cnt_use_extra_steps\022\002\b\002\032\023\n\rcnt_ad_reward\022\002\b\002\032\017\n\tjulianday\022\002\b\002\032\v\n\005month\022\002\b\002\032\037\n\031cnt_level_reset_quickplay\022\002\b\002\032\035\n\027cnt_level_end_quickplay\022\002\b\002\032\017\n\tdayofweek\022\002\b\002\032\026\n\020operating_system\022\002\b\001\032\024\n\016user_pseudo_id\022\002\b\001\032\034\n\026cnt_completed_5_levels\022\002\b\002\032\024\n\016cnt_post_score\022\002\b\0022\324\004*\321\004\n \n\023cnt_user_engagement\022\t\021\000\000\000\000\236j"A\n\024\n\acountry\022\t\032\aDenmark\n'\n\032cnt_spend_virtual_currency\022\t\021\000\000\000\000\000\000\000\000\n)\n\034cnt_level_complete_quickplay\022\t\021\000\000\000\000\000\000\b@\n\023\n\blanguage\022\a\032\005da-dk\n#\n\026cnt_challenge_a_friend\022\t\021\000\000\000\000\000\000\000\000\n&\n\031cnt_level_start_quickplay\022\t\021\000\000\000\000\000\000\030@\n \n\023cnt_use_extra_steps\022\t\021\000\000\000\000\000\000\000\000\n\032\n\rcnt_ad_reward\022\t\021\000\000\000\000\000\000\000\000\n\026\n\tjulianday\022\t\021\000\000\000\000\000\300o@\n\022\n\005month\022\t\021\000\000\000\000\000\000"@\n&\n\031cnt_level_reset_quickplay\022\t\021\000\000\000\000\000\000\000@\n$\n\027cnt_level_end_quickplay\022\t\021\000\000\000\000\000\000\024@\n\026\n\tdayofweek\022\t\021\000\000\000\000\000\000\b@\n\031\n\020operating_system\022\005\032\003IOS\n4\n\016user_pseudo_id\022"\032 104B0770BAE16E8B53DF330C95881893\n#\n\026cnt_completed_5_levels\022\t\021\000\000\000\000\000\000\360?\n\033\n\016cnt_post_score\022\t\021\000\000\000\000\000\000A@"
E }
E tfdv_instance_schema_uri: "gs://cloud-ai-platform-62f88573-082b-416d-a1b7-e47dcc520767/instance_schemas/job-8376379472194043904/analysis"
E tfdv_instance_schema {
E type_url: "type.googleapis.com/cloud.ai.platform.openapi.Schema"
E value: "\b\006\032\031\n\023cnt_user_engagement\022\002\b\002\032\r\n\acountry\022\002\b\001\032 \n\032cnt_spend_virtual_currency\022\002\b\002\032"\n\034cnt_level_complete_quickplay\022\002\b\002\032\016\n\blanguage\022\002\b\001\032\034\n\026cnt_challenge_a_friend\022\002\b\002\032\037\n\031cnt_level_start_quickplay\022\002\b\002\032\031\n\023cnt_use_extra_steps\022\002\b\002\032\023\n\rcnt_ad_reward\022\002\b\002\032\017\n\tjulianday\022\002\b\002\032\v\n\005month\022\002\b\002\032\037\n\031cnt_level_reset_quickplay\022\002\b\002\032\035\n\027cnt_level_end_quickplay\022\002\b\002\032\017\n\tdayofweek\022\002\b\002\032\026\n\020operating_system\022\002\b\001\032\024\n\016user_pseudo_id\022\002\b\001\032\034\n\026cnt_completed_5_levels\022\002\b\002\032\024\n\016cnt_post_score\022\002\b\0022\324\004*\321\004\n \n\023cnt_user_engagement\022\t\021\000\000\000\000\236j"A\n\024\n\acountry\022\t\032\aDenmark\n'\n\032cnt_spend_virtual_currency\022\t\021\000\000\000\000\000\000\000\000\n)\n\034cnt_level_complete_quickplay\022\t\021\000\000\000\000\000\000\b@\n\023\n\blanguage\022\a\032\005da-dk\n#\n\026cnt_challenge_a_friend\022\t\021\000\000\000\000\000\000\000\000\n&\n\031cnt_level_start_quickplay\022\t\021\000\000\000\000\000\000\030@\n \n\023cnt_use_extra_steps\022\t\021\000\000\000\000\000\000\000\000\n\032\n\rcnt_ad_reward\022\t\021\000\000\000\000\000\000\000\000\n\026\n\tjulianday\022\t\021\000\000\000\000\000\300o@\n\022\n\005month\022\t\021\000\000\000\000\000\000"@\n&\n\031cnt_level_reset_quickplay\022\t\021\000\000\000\000\000\000\000@\n$\n\027cnt_level_end_quickplay\022\t\021\000\000\000\000\000\000\024@\n\026\n\tdayofweek\022\t\021\000\000\000\000\000\000\b@\n\031\n\020operating_system\022\005\032\003IOS\n4\n\016user_pseudo_id\022"\032 104B0770BAE16E8B53DF330C95881893\n#\n\026cnt_completed_5_levels\022\t\021\000\000\000\000\000\000\360?\n\033\n\016cnt_post_score\022\t\021\000\000\000\000\000\000A@"
E }
E log_ttl {
E }
E job_id: 8376379472194043904
E ] already exitsts in Endpoint projects/580378083368/locations/us-central1/endpoints/8289570005524152320, please update the existing ModelMonitoringJob.

:3: FailedPrecondition

flaky-bot · 2022-09-23T01:59:46Z

commit: 2a906c8
buildURL: Build Status, Sponge
status: failed

Test output

args = (name: "projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320"
,)
kwargs = {'metadata': [('x-goog-request-params', 'name=projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320'), ('x-goog-api-client', 'model-builder/1.17.1 gl-python/3.8.13 grpc/1.47.0 gax/1.32.0 gapic/1.17.1')]}
@six.wraps(callable_)
def error_remapped_callable(*args, **kwargs):
    try:


      return callable_(*args, **kwargs)


.nox/system-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py:67:

self = <grpc._channel._UnaryUnaryMultiCallable object at 0x7f59637d2220>

request = name: "projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320"
timeout = None

metadata = [('x-goog-request-params', 'name=projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320'), ('x-goog-api-client', 'model-builder/1.17.1 gl-python/3.8.13 grpc/1.47.0 gax/1.32.0 gapic/1.17.1')]

credentials = None, wait_for_ready = None, compression = None
def __call__(self,
             request,
             timeout=None,
             metadata=None,
             credentials=None,
             wait_for_ready=None,
             compression=None):
    state, call, = self._blocking(request, timeout, metadata, credentials,
                                  wait_for_ready, compression)


  return _end_unary_response_blocking(state, call, False, None)


.nox/system-3-8/lib/python3.8/site-packages/grpc/_channel.py:946:

state = <grpc._channel._RPCState object at 0x7f59637d2d30>

call = <grpc._cython.cygrpc.SegregatedCall object at 0x7f59637dd280>

with_call = False, deadline = None
def _end_unary_response_blocking(state, call, with_call, deadline):
    if state.code is grpc.StatusCode.OK:
        if with_call:
            rendezvous = _MultiThreadedRendezvous(state, call, None, deadline)
            return state.response, rendezvous
        else:
            return state.response
    else:


      raise _InactiveRpcError(state)


E           grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

E           	status = StatusCode.NOT_FOUND

E           	details = "Endpoint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found."

E           	debug_error_string = "{"created":"@1663881225.766168768","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Endpoint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found.","grpc_status":5}"

E           >
.nox/system-3-8/lib/python3.8/site-packages/grpc/_channel.py:849: _InactiveRpcError
The above exception was the direct cause of the following exception:
self = <tests.system.aiplatform.test_model_monitoring.TestModelDeploymentMonitoring object at 0x7f597343c7f0>
def test_mdm_two_models_one_valid_config(self):
    """
    Enable model monitoring on two existing models deployed to the same endpoint.
    """
    # test model monitoring configurations


  job = aiplatform.ModelDeploymentMonitoringJob.create(


        display_name=self._make_display_name(key=JOB_NAME),
        logging_sampling_strategy=sampling_strategy,
        schedule_config=schedule_config,
        alert_config=alert_config,
        objective_configs=objective_config,
        create_request_timeout=3600,
        project=e2e_base._PROJECT,
        location=e2e_base._LOCATION,
        endpoint=self.endpoint,
        predict_instance_schema_uri="",
        analysis_instance_schema_uri="",
    )

tests/system/aiplatform/test_model_monitoring.py:110:

google/cloud/aiplatform/jobs.py:2330: in create

mdm_objective_config_seq = cls._parse_configs(

google/cloud/aiplatform/jobs.py:2107: in _parse_configs

for model in endpoint.list_models():

google/cloud/aiplatform/models.py:1708: in list_models

self._sync_gca_resource()

google/cloud/aiplatform/base.py:643: in _sync_gca_resource

self._gca_resource = self._get_gca_resource(resource_name=self.resource_name)

google/cloud/aiplatform/base.py:673: in resource_name

self._assert_gca_resource_is_available()

google/cloud/aiplatform/models.py:231: in _assert_gca_resource_is_available

self._sync_gca_resource_if_skipped()

google/cloud/aiplatform/models.py:223: in _sync_gca_resource_if_skipped

self._gca_resource = self._get_gca_resource(

google/cloud/aiplatform/base.py:636: in _get_gca_resource

return getattr(self.api_client, self._getter_method)(

google/cloud/aiplatform_v1/services/endpoint_service/client.py:746: in get_endpoint

response = rpc(

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py:145: in call

return wrapped_func(*args, **kwargs)

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/retry.py:286: in retry_wrapped_func

return retry_target(

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/retry.py:189: in retry_target

return target()

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py:69: in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

value = None

from_value = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.NOT_FOUND

details = "Endpoint projects/ucaip-sam...oint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found.","grpc_status":5}"



???

E   google.api_core.exceptions.NotFound: 404 Endpoint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found.

:3: NotFound

flaky-bot · 2022-09-27T00:13:34Z

commit: 926d0b6
buildURL: Build Status, Sponge
status: failed

Test output

args = (name: "projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320"
,)
kwargs = {'metadata': [('x-goog-request-params', 'name=projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320'), ('x-goog-api-client', 'model-builder/1.17.1 gl-python/3.8.13 grpc/1.47.0 gax/1.32.0 gapic/1.17.1')]}
@six.wraps(callable_)
def error_remapped_callable(*args, **kwargs):
    try:


      return callable_(*args, **kwargs)


.nox/system-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py:67:

self = <grpc._channel._UnaryUnaryMultiCallable object at 0x7fa865fc52e0>

request = name: "projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320"
timeout = None

metadata = [('x-goog-request-params', 'name=projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320'), ('x-goog-api-client', 'model-builder/1.17.1 gl-python/3.8.13 grpc/1.47.0 gax/1.32.0 gapic/1.17.1')]

credentials = None, wait_for_ready = None, compression = None
def __call__(self,
             request,
             timeout=None,
             metadata=None,
             credentials=None,
             wait_for_ready=None,
             compression=None):
    state, call, = self._blocking(request, timeout, metadata, credentials,
                                  wait_for_ready, compression)


  return _end_unary_response_blocking(state, call, False, None)


.nox/system-3-8/lib/python3.8/site-packages/grpc/_channel.py:946:

state = <grpc._channel._RPCState object at 0x7fa8674aeb50>

call = <grpc._cython.cygrpc.SegregatedCall object at 0x7fa86753b280>

with_call = False, deadline = None
def _end_unary_response_blocking(state, call, with_call, deadline):
    if state.code is grpc.StatusCode.OK:
        if with_call:
            rendezvous = _MultiThreadedRendezvous(state, call, None, deadline)
            return state.response, rendezvous
        else:
            return state.response
    else:


      raise _InactiveRpcError(state)


E           grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

E           	status = StatusCode.NOT_FOUND

E           	details = "Endpoint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found."

E           	debug_error_string = "{"created":"@1664222798.342272635","description":"Error received from peer ipv4:74.125.195.95:443","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Endpoint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found.","grpc_status":5}"

E           >
.nox/system-3-8/lib/python3.8/site-packages/grpc/_channel.py:849: _InactiveRpcError
The above exception was the direct cause of the following exception:
self = <tests.system.aiplatform.test_model_monitoring.TestModelDeploymentMonitoring object at 0x7fa86ddd1100>
def test_mdm_two_models_one_valid_config(self):
    """
    Enable model monitoring on two existing models deployed to the same endpoint.
    """
    # test model monitoring configurations


  job = aiplatform.ModelDeploymentMonitoringJob.create(


        display_name=self._make_display_name(key=JOB_NAME),
        logging_sampling_strategy=sampling_strategy,
        schedule_config=schedule_config,
        alert_config=alert_config,
        objective_configs=objective_config,
        create_request_timeout=3600,
        project=e2e_base._PROJECT,
        location=e2e_base._LOCATION,
        endpoint=self.endpoint,
        predict_instance_schema_uri="",
        analysis_instance_schema_uri="",
    )

tests/system/aiplatform/test_model_monitoring.py:110:

google/cloud/aiplatform/jobs.py:2330: in create

mdm_objective_config_seq = cls._parse_configs(

google/cloud/aiplatform/jobs.py:2107: in _parse_configs

for model in endpoint.list_models():

google/cloud/aiplatform/models.py:1708: in list_models

self._sync_gca_resource()

google/cloud/aiplatform/base.py:643: in _sync_gca_resource

self._gca_resource = self._get_gca_resource(resource_name=self.resource_name)

google/cloud/aiplatform/base.py:673: in resource_name

self._assert_gca_resource_is_available()

google/cloud/aiplatform/models.py:231: in _assert_gca_resource_is_available

self._sync_gca_resource_if_skipped()

google/cloud/aiplatform/models.py:223: in _sync_gca_resource_if_skipped

self._gca_resource = self._get_gca_resource(

google/cloud/aiplatform/base.py:636: in _get_gca_resource

return getattr(self.api_client, self._getter_method)(

google/cloud/aiplatform_v1/services/endpoint_service/client.py:746: in get_endpoint

response = rpc(

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py:145: in call

return wrapped_func(*args, **kwargs)

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/retry.py:286: in retry_wrapped_func

return retry_target(

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/retry.py:189: in retry_target

return target()

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py:69: in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

value = None

from_value = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.NOT_FOUND

details = "Endpoint projects/ucaip-sam...oint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found.","grpc_status":5}"



???

E   google.api_core.exceptions.NotFound: 404 Endpoint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found.

:3: NotFound

flaky-bot · 2022-09-29T08:49:21Z

commit: 876fb2a
buildURL: Build Status, Sponge
status: failed

Test output

args = (name: "projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320"
,)
kwargs = {'metadata': [('x-goog-request-params', 'name=projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320'), ('x-goog-api-client', 'model-builder/1.17.1 gl-python/3.8.13 grpc/1.47.0 gax/1.32.0 gapic/1.17.1')]}
@six.wraps(callable_)
def error_remapped_callable(*args, **kwargs):
    try:


      return callable_(*args, **kwargs)


.nox/system-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py:67:

self = <grpc._channel._UnaryUnaryMultiCallable object at 0x7f3f0702af10>

request = name: "projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320"
timeout = None

metadata = [('x-goog-request-params', 'name=projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320'), ('x-goog-api-client', 'model-builder/1.17.1 gl-python/3.8.13 grpc/1.47.0 gax/1.32.0 gapic/1.17.1')]

credentials = None, wait_for_ready = None, compression = None
def __call__(self,
             request,
             timeout=None,
             metadata=None,
             credentials=None,
             wait_for_ready=None,
             compression=None):
    state, call, = self._blocking(request, timeout, metadata, credentials,
                                  wait_for_ready, compression)


  return _end_unary_response_blocking(state, call, False, None)


.nox/system-3-8/lib/python3.8/site-packages/grpc/_channel.py:946:

state = <grpc._channel._RPCState object at 0x7f3f067fe2b0>

call = <grpc._cython.cygrpc.SegregatedCall object at 0x7f3f0d8cfe40>

with_call = False, deadline = None
def _end_unary_response_blocking(state, call, with_call, deadline):
    if state.code is grpc.StatusCode.OK:
        if with_call:
            rendezvous = _MultiThreadedRendezvous(state, call, None, deadline)
            return state.response, rendezvous
        else:
            return state.response
    else:


      raise _InactiveRpcError(state)


E           grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

E           	status = StatusCode.NOT_FOUND

E           	details = "Endpoint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found."

E           	debug_error_string = "{"created":"@1664427812.995626160","description":"Error received from peer ipv4:74.125.195.95:443","file":"src/core/lib/surface/call.cc","file_line":966,"grpc_message":"Endpoint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found.","grpc_status":5}"

E           >
.nox/system-3-8/lib/python3.8/site-packages/grpc/_channel.py:849: _InactiveRpcError
The above exception was the direct cause of the following exception:
self = <tests.system.aiplatform.test_model_monitoring.TestModelDeploymentMonitoring object at 0x7f3f0d739160>
def test_mdm_two_models_one_valid_config(self):
    """
    Enable model monitoring on two existing models deployed to the same endpoint.
    """
    # test model monitoring configurations


  job = aiplatform.ModelDeploymentMonitoringJob.create(


        display_name=self._make_display_name(key=JOB_NAME),
        logging_sampling_strategy=sampling_strategy,
        schedule_config=schedule_config,
        alert_config=alert_config,
        objective_configs=objective_config,
        create_request_timeout=3600,
        project=e2e_base._PROJECT,
        location=e2e_base._LOCATION,
        endpoint=self.endpoint,
        predict_instance_schema_uri="",
        analysis_instance_schema_uri="",
    )

tests/system/aiplatform/test_model_monitoring.py:110:

google/cloud/aiplatform/jobs.py:2330: in create

mdm_objective_config_seq = cls._parse_configs(

google/cloud/aiplatform/jobs.py:2107: in _parse_configs

for model in endpoint.list_models():

google/cloud/aiplatform/models.py:1708: in list_models

self._sync_gca_resource()

google/cloud/aiplatform/base.py:643: in _sync_gca_resource

self._gca_resource = self._get_gca_resource(resource_name=self.resource_name)

google/cloud/aiplatform/base.py:673: in resource_name

self._assert_gca_resource_is_available()

google/cloud/aiplatform/models.py:231: in _assert_gca_resource_is_available

self._sync_gca_resource_if_skipped()

google/cloud/aiplatform/models.py:223: in _sync_gca_resource_if_skipped

self._gca_resource = self._get_gca_resource(

google/cloud/aiplatform/base.py:636: in _get_gca_resource

return getattr(self.api_client, self._getter_method)(

google/cloud/aiplatform_v1/services/endpoint_service/client.py:746: in get_endpoint

response = rpc(

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py:145: in call

return wrapped_func(*args, **kwargs)

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/retry.py:286: in retry_wrapped_func

return retry_target(

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/retry.py:189: in retry_target

return target()

.nox/system-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py:69: in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

value = None

from_value = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.NOT_FOUND

details = "Endpoint projects/ucaip-sam...oint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found.","grpc_status":5}"



???

E   google.api_core.exceptions.NotFound: 404 Endpoint projects/ucaip-sample-tests/locations/us-central1/endpoints/8289570005524152320 is not found.

:3: NotFound

flaky-bot · 2022-09-29T13:28:20Z

Test passed for commit 186872d (Build Status, Sponge)! Closing this issue.

product-auto-label bot added the api: vertex-ai Issues related to the googleapis/python-aiplatform API. label Sep 16, 2022

rosiezou self-assigned this Sep 20, 2022

flaky-bot bot closed this as completed Sep 29, 2022

flaky-bot bot mentioned this issue Nov 18, 2022

tests.system.aiplatform.test_model_monitoring.TestModelDeploymentMonitoring: test_mdm_two_models_one_valid_config failed #1802

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests.system.aiplatform.test_model_monitoring.TestModelDeploymentMonitoring: test_mdm_two_models_one_valid_config failed #1672

tests.system.aiplatform.test_model_monitoring.TestModelDeploymentMonitoring: test_mdm_two_models_one_valid_config failed #1672

flaky-bot bot commented Sep 16, 2022

flaky-bot bot commented Sep 17, 2022

rosiezou commented Sep 20, 2022

flaky-bot bot commented Sep 21, 2022

flaky-bot bot commented Sep 23, 2022

flaky-bot bot commented Sep 27, 2022

flaky-bot bot commented Sep 29, 2022

flaky-bot bot commented Sep 29, 2022

tests.system.aiplatform.test_model_monitoring.TestModelDeploymentMonitoring: test_mdm_two_models_one_valid_config failed #1672

tests.system.aiplatform.test_model_monitoring.TestModelDeploymentMonitoring: test_mdm_two_models_one_valid_config failed #1672

Comments

flaky-bot bot commented Sep 16, 2022

flaky-bot bot commented Sep 17, 2022

rosiezou commented Sep 20, 2022

flaky-bot bot commented Sep 21, 2022

flaky-bot bot commented Sep 23, 2022

flaky-bot bot commented Sep 27, 2022

flaky-bot bot commented Sep 29, 2022

flaky-bot bot commented Sep 29, 2022