[Recovery Services] Vague error messages when updating a policy

[katbyte]

Bug Report

• import path of package in question, e.g. .../services/compute/mgmt/2018-06-01/compute
"github.com/Azure/azure-sdk-for-go/services/recoveryservices/mgmt/2016-06-01/backup"

• SDK version e.g. master, latest, 18.1.0

{
			"checksumSHA1": "XiQW2U5GBOhOfPTwFlFeM2thDCs=",
			"path": "github.com/Azure/azure-sdk-for-go/services/recoveryservices/mgmt/2016-06-01/backup",
			"revision": "2935c0241c74bd8549b843978dd6fc1be6f48b4a",
			"revisionTime": "2018-08-31T14:25:13Z",
			"version": "=v20.1.0",
			"versionExact": "v20.1.0"
		},

• output of go version
go version go1.10.3 darwin/amd64

• What happened?
  I am trying to update a recovery service policy retention days via this new terraform resource. Creating the policy works just fine for all configurations, however updating from daily -> weekly, or changing the retention days for monthly or yearly gives a vague error:

* azurerm_recovery_services_protection_policy_vm.test: Error creating/updating Recovery Service Protection Policy "acctest-7026298197833106544" (Resource Group "acctestRG-7026298197833106544"): backup.ProtectionPoliciesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BMSUserErrorInvalidPolicyInput" Message="Input for create or update policy is not in proper format\r\nPlease check format of parameters like schedule time, schedule days, retention time and retention days "

The error message does not help, and I cannot see why this would create just fine, yet fail to update. Other values will update just fine.

• What did you expect or want to happen?
  the policy be updated or a more detailed error message describing what is actually wrong.

• How can we reproduce it?
  run the updateDailyToWeekly test in that PR.

• Anything we should know about your environment.

[jhendrixMSFT]

@katbyte curious if you can try the 2016-12-01 API version. I mention this because this API version contains examples in the swagger which might help. In the meantime I'll track down the service owner so they can chime in.

[jhendrixMSFT]

@sumitmal can the error message be improved?

[katbyte]

@jhendrixMSFT,

I tried both the 2017-07-01 and 1016-12-01 versions of he API and am getting the same error.

[katbyte]

@marstr, would you be able to follow up on this?

[katbyte]

👋 @jhendrixMSFT, any updates on this issue?

[jhendrixMSFT]

Sorry for the delay, I can follow up with the service team about this. Do you have a sample JSON body of the failing request that I can provide to them?

[katbyte]

No, but there are some failing tf acc tests that it could be gathered from. Next time i work on that resource I'll try and grab some, but its pretty easy to trigger by changing the retention types.

[AsgharGhori]

Hi, I am getting the same error when I try to create a weekly policy. My code is:

resource "azurerm_recovery_services_protection_policy_vm" "backup_policy" {
name = "${var.RG4VM}-bkp-policy"
resource_group_name = "${var.RG4VM}"
recovery_vault_name = "${azurerm_recovery_services_vault.backup_vault.name}"
depends_on = ["azurerm_recovery_services_vault.backup_vault"]

backup {
frequency = "Weekly"
time = "18:00"
}

retention_weekly {
count = 5
weekdays = ["Friday"]
}
}

Error Message:

• azurerm_recovery_services_protection_policy_vm.backup_policy: 1 error(s) occurred:

• azurerm_recovery_services_protection_policy_vm.backup_policy: Error creating/updating Recovery Service Protection Policy "Terraform-Linux-test-RG-bkp-policy" (Resource Group "Terraform-Linux-test-RG"): backup.ProtectionPoliciesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BMSUserErrorInvalidPolicyInput" Message="Input for create or update policy is not in proper format\r\nPlease check format of parameters like schedule time, schedule days, retention time and retention days "

Appreciate any help.... Asghar

[pvrk]

We have made improvements in the error messages such that the relevant piece/section which is invalid is pointed out. Can you please retry and see if these are helpful? @AsgharGhori, @katbyte

[pvrk]

#customer-response-expected

[pvrk]

Closing this issue since we already made some improvements and customer can always re-open the issue if the improvements are not enough.
#please-close

[katbyte]

Hi @pvrk, @jhendrixMSFT

This is still an issue as of today and the error message we get is the exact same:

------- Stdout: -------
=== RUN   TestAccAzureRMRecoveryServicesProtectionPolicyVm_updateWeeklyToPartial
=== PAUSE TestAccAzureRMRecoveryServicesProtectionPolicyVm_updateWeeklyToPartial
=== CONT  TestAccAzureRMRecoveryServicesProtectionPolicyVm_updateWeeklyToPartial
--- FAIL: TestAccAzureRMRecoveryServicesProtectionPolicyVm_updateWeeklyToPartial (110.41s)
    testing.go:568: Step 1 error: errors during apply:
        
        Error: Error creating/updating Recovery Service Protection Policy "acctest-190607005853824275" (Resource Group "acctestRG-190607005853824275"): backup.ProtectionPoliciesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BMSUserErrorInvalidPolicyInput" Message="Input for create or update policy is not in proper format\r\nPlease check format of parameters like schedule time, schedule days, retention time and retention days "
        
          on /opt/teamcity-agent/temp/buildTmp/tf-test217175629/main.tf line 16:
          (source code not available)
        
        
FAIL

[caotruonghoang]

Hello,

I have the same error today.
Could you please fix it ? Thanks,

Terraform v0.12.1

• provider.azurerm v1.30.1

Error: Error creating/updating Recovery Service Protection Policy "weekly" (Resource Group "testcao"): backup.ProtectionPoliciesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BMSUserErrorInvalidPolicyInput" Message="Input for create or update policy is not in proper format\r\nPlease check format of parameters like schedule time, schedule days, retention time and retention days "

[horpek]

Hi,
Our code works well since Feb and the last run was 5 Jun that ended OK. We however experienced this error when we run it in 2 new builds yesterday.

We are using the older provider.azurerm v1.21.0 but it serves us well so far. We are open to using newer provider if it helps but like to understand if this is the cause or something else.

Appreciate any info if you have.

[caotruonghoang]

Hello,

Thank you for your feedback.

I currently use terraform v.0.12.1. And this version is not compatible with the azure provider v.1.21.0.
I also cannot roll back to older version of terraform because the older version have some bugs for example: bug on the list value.
Hope that you can correct this issue on the recents Azure provider.

Thanks,
Best regards,
Cao

[cmpl-giedriusk]

Hi,

What are your retention policies? We have just found out that for whatever reason, $retPol.DailySchedule.DurationCountInDays that is less than 7 will not work.
This has worked perfectly fine for 3-4 years, and suddenly stopped a week or so ago.

[horpek]

Hi @gkakanauskas , You are right. We opened a case with Azure support today and they came back with this:
_[Initial analysis]
From the latest update, "Retention range" needs to be set more than or equal to 7 days.
Could you check whether the creating new backup policy would work if you use more than or equal to 7 days?

If you have backup policies that have been set less than 7 days, it works. But you cannot change the "Retention range" less than 7 days._

I tried bump up the retention to 8 days and it just went through.

And you are also right that we have been living with 2 days retention for dev systems for past 1 year. This should be a rather recent change. I asked the support to provide the detail of this change and pending their reply.

[caotruonghoang]

Hello,

I had this error when I tried to create a weekly backup policy. The daily backup policy work well.
Here is my terraform script for the creation of this weekly policy.

resource "azurerm_recovery_services_protection_policy_vm" "weekly" {
name = "${var.recovery_weekly_policy_vm_name}"
resource_group_name = "${var.rg_name}"
recovery_vault_name = "${module.azother.recovery_services_name}"

timezone = "${var.backup_timezone}"

backup {
frequency = "${var.weekly_backup_frequency}"
time = "${var.backup_time}"
}

retention_weekly {
count = "${var.backup_retention_weekly_count}"
weekdays = "${var.backup_retention_weekdays}"
}
}

With the variables values likes :
recovery_weekly_policy_vm_name = "weekly-backup"
weekly_backup_frequency = "Weekly"
backup_time = "00:00"
ackup_timezone = "Romance Standard Time"
backup_retention_weekly_count = "1"
backup_retention_weekdays = ["Sunday"]

Hope that is useful to identify and correct the issue.

Thanks,

Best regards,
Cao

[katbyte]

@gkakanauskas, @horpek, the tests failing are 7 or more days/weeks for retention. Really what the service needs to do is have actionable/detailed error messages returned.

@jhendrixMSFT, this is still an issue as of today for us:

 Error: Error creating/updating Recovery Service Protection Policy "acctest-190703070257400167" (Resource Group "acctestRG-190703070257400167"): backup.ProtectionPoliciesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BMSUserErrorInvalidPolicyInput" Message="Input for create or update policy is not in proper format\r\nPlease check format of parameters like schedule time, schedule days, retention time and retention days "

[jhendrixMSFT]

@pvrk can you please take a look?

[pvrk]

Sorry for the delay. Will take a look and come back.

[metrue]

Met the same issue, here is my dumped data. @gkakanauskas and days is bigger than 7.

{
  "location": "westeurope",
  "properties": {
    "backupManagementType": "AzureIaasVM",
    "instantRpRetentionRangeInDays": 10,
    "retentionPolicy": {
      "dailySchedule": {
        "retentionTimes": [
          "2019-08-06T19:19:24+08:00"
        ],
        "retentionDuration": {
          "count": 10,
          "durationType": "Days"
        }
      },
      "retentionPolicyType": "LongTermRetentionPolicy"
    },
    "schedulePolicy": {
      "schedulePolicyType": "SimpleSchedulePolicy",
      "scheduleRunFrequency": "Daily",
      "scheduleRunTimes": [
        "2019-08-06T19:19:24+08:00"
      ],
      "scheduleWeeklyFrequency": 0
    },
    "timeZone": "UTC"
  },
  "tags": {
    "mccp_vm_backup": "mccp_vm_backup"
  }
}

error message is:

backup.ProtectionPoliciesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BMSUserErrorInvalidPolicyInput" Message="Input for create or update policy is not in proper format\r\nPlease check format of parameters like schedule time, schedule days, retention time and retention days "

[katbyte]

@pvrk @jhendrixMSFT, any progress on this?

[jeffpatton1971]

Hey, so this is still a thing! I'm using arm templates and am getting this error message, I know that someone said they've improved the error message, i can only assume they mean this

Please check format of parameters like schedule time, schedule days, retention time and retention days

This is not an improvement, I'm sure this is a common pain point because times need to be in the crazy time format, but my times all check out. I have a feeling it's something wrong with layout of the resource itself, and the biggest issue there is that the documentation is lacking and i'm being incredibly nice here.

VERBOSE: Performing the operation "Creating Deployment" on target "ptech-testing".
VERBOSE: 3:32:30 PM - Template is valid.
VERBOSE: 3:32:31 PM - Create template deployment 'azurevmworkload-test'
VERBOSE: 3:32:31 PM - Checking deployment status in 5 seconds
VERBOSE: 3:32:36 PM - Checking deployment status in 5 seconds
New-AzureRmResourceGroupDeployment : 3:32:41 PM - Resource Microsoft.RecoveryServices/vaults/backupPolicies 'RSV-01/AzureVmWorkloadProtectionPolicy' failed with message '{
  "error": {
    "code": "BMSUserErrorInvalidPolicyInput",
    "message": "Input for create or update policy is not in proper format. Please check format of parameters like schedule time, schedule days, retention time and retention days "
  }
}'

[jeffpatton1971]

Forgot to paste in the template

{
   "type": "Microsoft.RecoveryServices/vaults/backupPolicies",
   "apiVersion": "2021-12-01",
   "name": "RSV-01/AzureVmWorkloadProtectionPolicy",
   "location": "centralus",
   "tags": { "TagName": "TagValue" },
   "properties": {
    "backupManagementType": "AzureWorkload",
    "makePolicyConsistent": true,
    "settings": {
     "isCompression": false,
     "issqlcompression": false,
     "timeZone": "Eastern Standard Time"
    },
    "subProtectionPolicy": [
     {
      "policyType": "Incremental",
      "retentionPolicy": {
       "retentionPolicyType": "LongTermRetentionPolicy",
       "dailySchedule": {
        "retentionTimes": [ "2021-03-15T02:30:00Z" ],
        "retentionDuration": {
         "count": 15,
         "durationType": "Days"
        }
       },
       "weeklySchedule": {
        "daysOfTheWeek": [ "Sunday" ],
        "retentionDuration": {
         "count": 7,
         "durationType": "Weeks"
        },
        "retentionTimes": [ "2021-03-15T02:30:00Z" ]
       },
       "monthlySchedule": {
        "retentionDuration": {
         "count": 120,
         "durationType": "Months"
        },
        "retentionScheduleDaily": {},
        "retentionScheduleFormatType": "Weekly",
        "retentionScheduleWeekly": {
         "daysOfTheWeek": [ "Sunday" ],
         "weeksOfTheMonth": [ "First" ]
        },
        "retentionTimes": [ "2021-03-15T02:30:00Z" ]
       },
       "yearlySchedule": {
        "monthsOfYear": [ "January" ],
        "retentionDuration": {
         "count": 5,
         "durationType": "Years"
        },
        "retentionScheduleDaily": {},
        "retentionScheduleFormatType": "Weekly",
        "retentionScheduleWeekly": {
         "daysOfTheWeek": [ "Sunday" ],
         "weeksOfTheMonth": [ "First" ]
        },
        "retentionTimes": [ "2021-03-15T02:30:00Z" ]
       }
      },
      "schedulePolicy": {
       "schedulePolicyType": "SimpleSchedulePolicy",
       "scheduleRunDays": [],
       "scheduleRunFrequency": "Daily",
       "scheduleRunTimes": [ "2021-03-15T02:30:00Z" ],
       "scheduleWeeklyFrequency": 0
      }
     }
    ],
    "workLoadType": "SystemState"
   }
  }

[adityabalaji-msft]

Hi @jeffpatton1971 - thanks for reaching out and apologies for the inconvenience. We will look into the template that you shared. Just to clarify the scenario, are you looking to backup the system state of windows server using MARS agent backup solution (via ARM template)?

[jeffpatton1971]

@adityabalaji-msft honestly, I am just attempting to sort through the combinations and determine what is actually supported via the template. The list of WorkloadTypes does not sync with what is available via the portal, so there is some confusion within the documentation. It's also not really covered where and under what circumstances you could use each of the workloads. Of the 15 types available it appears that only AzureFileShare, SapHanaDatabase, and SqlDatabase work.

[jeffpatton1971]

For clarification, you can see in this documentation that there are several different ways to configure the backup policies, the AzureIaasVm works for sure, but when we move into other scenarios that involve workload type, most of these fail. I've attempted several different combinations in both ARM and Bicep in a vain attempt to get a better error message that would clarify what the actual issue may be. If these are all unsupported scenarios what can we do to get the documentation above corrected?

[adityabalaji-msft]

Thanks for the details. Am checking internally with the team on this and will get back with an update

[adityabalaji-msft]

Hi @jeffpatton1971 - sincere apologies for the delay as this was being investigated. To answer your question, currently many of the workload types shown in the above auto-generated doc are not actively supported today (these likely came up because of certain resources in our non-production environment - we will work on fixing our help texts etc. so that these types don't show up in the future).

The following workload types are supported via ARM template:

1. VM (workload type: VM, BackupManagementType: AzureIaasVM)
2. AzureFileShare (workload type: AzureFileShare, BackupManagementType: AzureStorage)
3. SQLDataBase (workload type: SQLDataBase, BackupManagementType: AzureWorkload)
4. SAPHANADatabase (workload type: SAPHANADatabase, BackupManagementType: AzureWorkload)

[ghost]

Hi @katbyte. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[ghost]

Hi @katbyte, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.