bug: Paladin node errors during performance test: Error persisting state distribution acknowledgements

[francescomiliani]

What happened?

Hi, while performing performance/stress tests on Noto, I noticed a series of errors on the logs in the paladin node.

I ran a test with 10 req/s of Mint type, based on Bond issuance example.

Test info: 10 users, 30 secs of duration. Once submitted a request, rest for 1 second.
log.zip

Attached log file of Paladin node 1 once test was started.

Here an error example:

[2024-12-11T10:41:52.424Z] ERROR Error persisting state distribution acknowledgements: ERROR: insert or update on table "state_distribution_acknowledgments" violates foreign key constraint "state_distribution_acknowledgments_state_distribution_fkey" (SQLSTATE 23503) job=writer_0skW8RQJ_0002 pid=1

below is the code used for the mint on Noto.

==========

const cashIssuer = "bank1@node1";
const investor = "bank3@node3";
console.info("Deploying Noto cash token...");
const notoFactory = new paladin.NotoFactory(paladin1, "noto");
notoCash = await notoFactory.newNoto(cashIssuer, {
  notary: cashIssuer,
  restrictMinting: true,
});
if (notoCash === undefined) {
  throw new Error("Failed to deploy Noto cash token");
}
console.info(`Noto Cash token deployed at address: ${notoCash.address}`);

function mint() {
    notoCash.mint(cashIssuer, {
      to: investor,
      amount: 100000000000,
      data: "0x",
    })
    .then((receipt) => {
      if (receipt === undefined) {
        console.error("Failed to Mint CBDC. "+
          `requestUUID: ${requestUUID}, ` +
          `requestNumber: ${requestNumber}, ` +
          `time: ${(new Date().getTime() - start_time) / 1000} s`);
      }
      else {
        console.info(
          config.logs.colors.success,
          "Mint successful! transaction_hash:", receipt.transactionHash, 
          "requestUUID:", requestUUID,
          "requestNumber:", requestNumber,
          "time:", (new Date().getTime() - start_time) / 1000, "s"
        );
      }
    })
    .catch((e) => {
      console.error("Failed to Mint CBDC: "+ e + ". "+
        `requestUUID: ${requestUUID}, ` +
        `requestNumber: ${requestNumber}, ` +
        `time: ${(new Date().getTime() - start_time) / 1000} s`);
    });
}

What did you expect to happen?

A correct processing of all requests

How can we reproduce it (as minimally and precisely as possible)?

By creating a simple express server and calling a /noto/mint endpoint executing mint function, once first deployed the notoCash token, with a performance test tool like postman, or Apache jmeter or Grafana k6.

Anything else we need to know?

No response

OS version

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
BuildNumber  Caption                   OSArchitecture  Version
22631        Microsoft Windows 11 Pro  64-bit          10.0.22631

Hardware specs:
Device name BLK28689
Full device name BLK28689.replynet.prv
Processor AMD Ryzen 7 PRO 5850U with Radeon Graphics 1.90 GHz
Installed RAM 16.0 GB (14.8 GB usable)
Device ID C016081C-E8CE-4F20-B91F-2B99B2115215
Product ID 00330-54210-83290-AAOEM
System type 64-bit operating system, x64-based processor
Pen and touch No pen or touch input is available for this display

[hosie]

@francescomiliani thanks for raising this. I think the error Error persisting state distribution acknowledgements is possibly a red herring. I see that same error in the logs in success cases. It is just a timing issue that usually gets reconciled by automatic retries in the code. Although, I am not completely ruling that out at this point

Looking at your logs, I do see that some transactions are not being assembled and am looking into why that could be the case. I'll update when I learn more.