Tests for PATCH endpoints do not run test steps

[savvyintegrations]

Describe the bug

Tests for PATCH endpoints do not run test steps. (Or so it appears.)

We have a set of tests that contain GET, POST and PATCH operations. All of these automated tests are green when we run them against a Microcks instance started via Docker Compose. If we instead run them against a Microcks instance running in testcontainers, the PATCH operations fail with:

[
    {
      "success": false,
      "elapsedTime": -1,
      "operationName": "PATCH /jobs/{jobId}/executions",
      "testStepResults": []
    }
  ]

I have also duplicated this by spinning up a Microcks server that is long running (i.e., not spun up by jest), and then used the Test API via curl, e.g., the following, the with same result (green against Docker Compose server, red against testcontainers server):

export TEST_RESULT_ID=$(curl $MICROCKS_URL/api/tests -d '{
  "serviceId": "nRF Cloud Device Job Service:0.0.1",
  "runnerType": "OPEN_API_SCHEMA",
  "testEndpoint": "'$TEST_ENDPOINT'",
  "timeout": 10000,
  "operationsHeaders": {"globals":[{"name":"x-auth-context","values":"eyJ0ZWFtSWQiOiI5NGJlZTE3MC0xN2IzLTQxN2ItOTQxMS05ZDUzMDI0YzBlMzUiLCJpc05vcmRpY2FuVGVhbSI6ZmFsc2UsInNvdXJjZUFwaVR5cGUiOiJyZXN0IiwicGxhblR5cGUiOiJQcm8iLCJjcmVhdGVkQXQiOiIyMDI0LTA2LTI3VDIzOjE1OjQ0LjY0M1oiLCJhdXRoZW50aWNhdGlvblRva2VuVHlwZSI6ImFwaV9rZXkiLCJjb3JyZWxhdGlvbklkIjoiYjE1MmFhNjktZjk4NS00MWQ5LTlmMjQtMjY0MDA5NTMzZDU5IiwicmVxdWVzdGVySWQiOiIyZDk3ZDc0Ny1hZjM3LTQ4NTgtYjQ4OS0xMzI4ZWNlZjhkMDciLCJyZXF1ZXN0ZXJUeXBlIjoiVGVhbU1lbWJlciIsInVzZXJBY2NvdW50SWQiOiI1YTg3NmJmNy1mMDIxLTQyYTYtOTNiMC02YjhhY2VmOTgxOTIiLCJyb2xlIjoiQWRtaW4iLCJkZXZpY2VHcm91cHMiOltdfQ=="}]},
  "filteredOperations": [
    "PATCH /jobs/{jobId}/executions"
  ]
}' -H "Content-Type: application/json" | jq -r '.id')

Expected behavior

All operations test the same, whether run against a Docker Compose-based server or testcontainers-based.

Actual behavior

See above.

How to Reproduce?

Not sure about the easiest way to test this, but here are two endpoints involved in the tests:

    "/jobs/{jobId}/executions": {
      "get": {
        "operationId": "listJobExecutions",
        "description": "Get a list of job executions for a device job\n",
        "tags": [
          "Job execution"
        ],
        "parameters": [
          {
            "name": "jobId",
            "in": "path",
            "required": true,
            "schema": {
              "$ref": "#/components/schemas/Uuid"
            },
            "examples": {
              "1145cf7d-500f-4147-86eb-6a3b840c9d6a": {
                "value": "1145cf7d-500f-4147-86eb-6a3b840c9d6a"
              }
            }
          },
          {
            "name": "status",
            "in": "query",
            "required": false,
            "schema": {
              "$ref": "#/components/schemas/JobExecutionStatus"
            },
            "examples": {
              "inProgress": {
                "value": "IN_PROGRESS"
              }
            }
          },
          {
            "name": "pageLimit",
            "in": "query",
            "required": false,
            "schema": {
              "$ref": "#/components/schemas/PageLimit"
            },
            "examples": {
              "limit100": {
                "value": 100
              }
            }
          },
          {
            "name": "pageNextToken",
            "in": "query",
            "required": false,
            "schema": {
              "$ref": "#/components/schemas/PageToken"
            },
            "examples": {
              "nextTokenExample": {
                "value": "def456eubdkw9038kdd93l"
              }
            }
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "$ref": "#/components/schemas/JobExecutionCollection"
                },
                "examples": {
                  "1145cf7d-500f-4147-86eb-6a3b840c9d6a": {
                    "description": "Example response for getting a list of job executions for jobId 1145cf7d-500f-4147-86eb-6a3b840c9d6a.",
                    "value": {
                      "items": [
                        {
                          "id": "52eb33a0-101b-4034-a515-dfbf43be754e",
                          "jobId": "1145cf7d-500f-4147-86eb-6a3b840c9d6a",
                          "deviceInfo": {
                            "deviceId": "d44ff5ee-c3eb-4ea7-ab14-c238793810a9"
                          },
                          "status": "IN_PROGRESS",
                          "retryable": true,
                          "createdAt": "2024-04-16T12:45:00Z"
                        }
                      ],
                      "total": 1,
                      "pageNextToken": "def456"
                    }
                  }
                }
              }
            }
          }
        }
      },
      "patch": {
        "operationId": "bulkUpdateJobExecutionStatus",
        "description": "This operation only affects job executions that are in the following states: `QUEUED`, `TIMED_OUT`, `FAILED`.\nIf no device IDs are specified in the request body, all job executions will be updated.\n",
        "parameters": [
          {
            "name": "jobId",
            "in": "path",
            "required": true,
            "schema": {
              "$ref": "#/components/schemas/Uuid"
            },
            "examples": {
              "66a2afc3-ab5b-4437-ae7c-0bf0d79c72ca": {
                "value": "66a2afc3-ab5b-4437-ae7c-0bf0d79c72ca"
              }
            }
          }
        ],
        "tags": [
          "Job execution"
        ],
        "requestBody": {
          "required": true,
          "content": {
            "application/merge-patch+json": {
              "schema": {
                "$ref": "#/components/schemas/BulkUpdateJobExecutionStatusRequestBody"
              },
              "examples": {
                "66a2afc3-ab5b-4437-ae7c-0bf0d79c72ca": {
                  "value": {
                    "status": "CANCELED",
                    "deviceIds": [
                      "device1",
                      "device2"
                    ]
                  }
                }
              }
            }
          }
        },
        "responses": {
          "202": {
            "description": "Request to bulk update job execution statuses was accepted",
            "x-microcks-refs": [
              "66a2afc3-ab5b-4437-ae7c-0bf0d79c72ca"
            ]
          }
        }
      }
    },

Aside from the two different HTTP methods, could the issue be related to the use of application/merge-patch+json for PATCH operations?

I see nothing in the Microcks testcontainer logs.

Microcks version or git rev

No response

Install method (docker-compose, helm chart, operator, docker-desktop extension,...)

No response

Additional information

No response

[github-actions]

👋 @savvyintegrations

Welcome to the Microcks community! 💖

Thanks and congrats 🎉 for opening your first issue here! Be sure to follow the issue template or please update it accordingly.

📢 If you're using Microcks in your organization, please add your company name to this list. 🙏 It really helps the project to gain momentum and credibility. It's a small contribution back to the project with a big impact.

If you need to know why and how to add yourself to the list, please read the blog post "Join the Microcks Adopters list and Empower the vibrant open source Community 🙌"

Hope you have a great time there!

[lbroudoux]

Hi @savvyintegrations,

I did a quick test and cannot reproduce it. I think I'd need further information to investigate on this...

What I've done so far:

• Put your operations description into an OpenAPI file: nordic.json
• Start a local Microcks instance using docker run -p 8585:8080 -p 8686:9090 -it --rm quay.io/microcks/microcks-uber:1.10.0-native and load the above OpenAPI file in it. This will be used as the backend under test
• Test the endpoint itself from this instance to see if test is passing. I'm using http://host.docker.internal:8585/rest/Nordic+API/2.0.0 as the URL so that the container can reach itself. Got this result:

• Wrote a Jest test that launches another instance of Microcks (using the latest version of the lib so poped-up container version is 1.10.0 as well) and launch a test on my backend running on port 8585.

import * as path from "path";

import { TestContainers } from "testcontainers";
import { MicrocksContainer, TestRequest, TestRunnerType } from "./microcks-container";

describe("NordicAPI", () => {
  jest.setTimeout(180_000);

  const resourcesDir = path.resolve(__dirname, "..", "test-resources");

  // start and contract test {
  it("should start, load artifacts and contract tets mock", async () => {
    await TestContainers.exposeHostPorts(8585);

    const container = await new MicrocksContainer().start();

    await container.importAsMainArtifact(path.resolve(resourcesDir, "nordic.json"));
    
    var testRequest: TestRequest = {
      serviceId: "Nordic API:2.0.0",
      runnerType: TestRunnerType.OPEN_API_SCHEMA,
      testEndpoint: "http://host.testcontainers.internal:8585/rest/Nordic+API/2.0.0",
      timeout: 2000,
      filteredOperations: [
        "PATCH /jobs/{jobId}/executions"
      ]
    }

    var testResult = await container.testEndpoint(testRequest);

    console.log('testResult: ' + JSON.stringify(testResult));
    expect(testResult.success).toBe(true);
  });
});

Got the final result where test is a success:

$ node 'node_modules/.bin/jest' '/Users/laurent/Development/github/microcks-testcontainers-node/src/nordic.test.ts' -t 'NordicAPI should start, load artifacts and contract tets mock'   
=== OUTPUT ===
  console.log
    testResult: {"id":"66b9c697d1d67157accb05bd","version":2,"testNumber":1,"testDate":1723451031074,"testedEndpoint":"http://host.testcontainers.internal:8585/rest/Nordic+API/2.0.0","serviceId":"66b9c696d1d67157accb05b6","timeout":2000,"elapsedTime":46,"success":true,"inProgress":false,"runnerType":"OPEN_API_SCHEMA","testCaseResults":[{"success":true,"elapsedTime":46,"operationName":"PATCH /jobs/{jobId}/executions","testStepResults":[{"success":true,"elapsedTime":46,"requestName":"66a2afc3-ab5b-4437-ae7c-0bf0d79c72ca","message":""}]}]}

      at Object.<anonymous> (src/nordic.test.ts:33:13)

 PASS  src/nordic.test.ts (7.733 s)
  NordicAPI
    ✓ should start, load artifacts and contract tets mock (6957 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        7.784 s

Can you help to confirm I understood the logic, and by providing additional information like versions and distros of the different Microcks instances?

[savvyintegrations]

Thank you very for taking valuable time to investigate this, and so thoroughly. I will look into this further at our end ASAP and get back with you. For now I think it's safe to say that we are doing something wrong.

[savvyintegrations]

It appears to have been our error. Many thanks again for your prompt attention, and apologies for the time waste.