[BUG] Deleting a deleted model causes a 500 illegal_state_exception

[dblock]

What is the bug?

Attempting to delete a model that has just been deleted causes illegal_state_exception.

Use models.yaml test from opensearch-project/opensearch-api-specification#453. It creates a model. Then delete the model twice quickly. The first one succeeds with 200, the second is a 500.

[INFO] => DELETE /_plugins/_ml/models/ZVP1CJEBQj8N2F2ZqulZ ({}) [application/json] | undefined
[INFO] <= 200 (application/json; charset=UTF-8) | {
  "_index": ".plugins-ml-model",
  "_id": "ZVP1CJEBQj8N2F2ZqulZ",
  "_version": 3,
  "result": "deleted",
  "forced_refresh": true,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 58,
  "_primary_term": 1
}

[INFO] => DELETE /_plugins/_ml/models/ZVP1CJEBQj8N2F2ZqulZ ({}) [application/json] | undefined
[INFO] <= 500 (application/json) | {
  "root_cause": [
    {
      "type": "illegal_state_exception",
      "reason": "Model is not all cleaned up, please try again. Model ID: ZVP1CJEBQj8N2F2ZqulZ"
    }
  ],
  "type": "illegal_state_exception",
  "reason": "Model is not all cleaned up, please try again. Model ID: ZVP1CJEBQj8N2F2ZqulZ",
  "caused_by": {
    "type": "status_exception",
    "reason": "Failed to delete all model chunks, Bulk failure while deleting model of ZVP1CJEBQj8N2F2ZqulZ"
  }
}

What is the expected behavior?

The second call should either return a 404 that the model no longer exists, or a 200 that it has been deleted.

What is your host/environment?

• OS: M1 OSX
• Version OpenSearch 2.15

[zane-neo]

This is caused by not adding refresh policy when deleting model chunks, fix PR above.

[zane-neo]

This is fixed by the PR above, closing this now.