getting service error without explanation when using aggregation service

[TomerGoldstein]

Hi
i have enrolled and managed to deploy the aggregation service and it looks ok (i see metrics logs and everything)
i do however have some questions:

1. i have some reports (with and without clear text option) i got a domain file, i took both of them and tried to use them in the local testing tool - everything looked ok- got a good output i have used the non-encrypted output
   then, i took the reports and the domain file and use them with the deployed aggregation service (of course this time encrypted since the local tool doesnt accept encrypted files)
   i got the following error (in this example i have sent only 1 report but also when i sent 200 i get the same error but with 200 as count):
   **"result_info": {
   "return_code": "SUCCESS_WITH_ERRORS",
   "return_message": "Aggregation job successfully processed but some reports have errors.",
   "error_summary": {
   "error_counts": [
   {
   "category": "SERVICE_ERROR",
   "count": 1,
   "description": "Internal error occurred during operation."
   },
   {
   "category": "NUM_REPORTS_WITH_ERRORS",
   "count": 1,
   "description": "Total number of reports that had an error. These reports were not considered in aggregation. See additional error messages for details on specific reasons."
   }
   ],
   "error_messages": []
   }, **
   the output report contains all the keys from the domain file but all of the metrics are just pure noise.. also i dont understand this error message, and i think it will be great to get more elaborate error message (since it is empty and doesnt give anymore info)
   im using the latests 2.4.2 version
   im really stuck right now can you help?

2. looking at the auto scailing group for the Aggregation Service i saw that no policy for auto scaling has been created after deploy (im looking in AWS->auto scaling groups -> my service -> automatic scaling or instance manager, however when sending some jobs i did see that the instnace number went up and then down.. what am i missing?
   do i need to create those policies myself? do you have a recommendations on what metrics to use for auto scaling? including thresholds?
   thanks!!

[TomerGoldstein]

colsed

[TomerGoldstein]

closed by mistake
we have a new error :
when sending the request:
"result_info": {
"return_code": "PRIVACY_BUDGET_AUTHENTICATION_ERROR",
"return_message": "com.google.aggregate.adtech.worker.exceptions.AggregationJobProcessException: Aggregation service is not authenticated to call privacy budget service. This could happen due to a misconfiguration during enrollment. Please contact support for resolution. \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.consumePrivacyBudgetUnits(ConcurrentAggregationProcessor.java:465) \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.process(ConcurrentAggregationProcessor.java:342) \n com.google.aggregate.adtech.worker.WorkerPullWorkService.run(WorkerPullWorkService.java:143)\nThe root cause is: com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngine$TransactionEngineException: PRIVACY_BUDGET_CLIENT_UNAUTHENTICATED \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.createTransactionEngineException(TransactionEngineImpl.java:213) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.proceedToNextPhase(TransactionEngineImpl.java:68) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.executeDistributedPhase(TransactionEngineImpl.java:206)",
"error_summary": {
"error_counts": [],
"error_messages": []
},
- we are using 2.5.0 version (newest one)
- we are using the new ARN you have sent us during our meeting (which seems to solve the SERVICE_ERROR issue)
- we have sent you the relevant files (main.tf, release_params.auto.tfvars,{env}.auto.tfvars) via mail
- we use the new ami 2.5.0 (pre built and copied to our region with our AWS ID as our ami_owner)
i do see that you hae a record of this error here
but we are not sure what to do with this

[pisutpon]

We got the same issue when we upgraded the aggregation service to the version 2.5. The return code was PRIVACY_BUDGET_AUTHENTICATION_ERROR

[yuki-tsuchiya]

We are experiencing the same issue when we upgraded the aggregation service to version 2.5. If anyone has a solution or workaround for this problem, we would greatly appreciate it.

[TomerGoldstein]

we have downgraded to version 2.4.3 and it works for us @yuki-tsuchiya @pisutpon

[nlrussell]

@TomerGoldstein @yuki-tsuchiya @pisutpon Thanks for making us aware of the issue. We're investigating it and will provide an update when we have one. In the meantime, please continue to use version 2.4.3.

[maybellineboon]

Hi @TomerGoldstein ,

Would you be able to provide the full getJob response or timing of the request.

[TomerGoldstein]

@maybellineboon hey!
this is a response that i got when using version 2.5.0:

    "result_info": {
        "return_code": "PRIVACY_BUDGET_AUTHENTICATION_ERROR",
        "return_message": "com.google.aggregate.adtech.worker.exceptions.AggregationJobProcessException: Aggregation service is not authenticated to call privacy budget service. This could happen due to a misconfiguration during enrollment. Please contact support for resolution. \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.consumePrivacyBudgetUnits(ConcurrentAggregationProcessor.java:465) \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.process(ConcurrentAggregationProcessor.java:342) \n com.google.aggregate.adtech.worker.WorkerPullWorkService.run(WorkerPullWorkService.java:143)\nThe root cause is: com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngine$TransactionEngineException: PRIVACY_BUDGET_CLIENT_UNAUTHENTICATED \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.createTransactionEngineException(TransactionEngineImpl.java:213) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.proceedToNextPhase(TransactionEngineImpl.java:68) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.executeDistributedPhase(TransactionEngineImpl.java:206)",
        "error_summary": {
            "error_counts": [],
            "error_messages": []
        },
        "finished_at": "2024-06-12T17:18:07.491437708Z"
    }

please let me know if i can do anything else to help :)

also this is a response for the same exact inputs (chang the name of directories but these are the same files):

    "result_info": {
        "return_code": "PRIVACY_BUDGET_AUTHENTICATION_ERROR",
        "return_message": "com.google.aggregate.adtech.worker.exceptions.AggregationJobProcessException: Aggregation service is not authenticated to call privacy budget service. This could happen due to a misconfiguration during enrollment. Please contact support for resolution. \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.consumePrivacyBudgetUnits(ConcurrentAggregationProcessor.java:465) \n com.google.aggregate.adtech.worker.aggregation.concurrent.ConcurrentAggregationProcessor.process(ConcurrentAggregationProcessor.java:342) \n com.google.aggregate.adtech.worker.WorkerPullWorkService.run(WorkerPullWorkService.java:143)\nThe root cause is: com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngine$TransactionEngineException: PRIVACY_BUDGET_CLIENT_UNAUTHENTICATED \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.createTransactionEngineException(TransactionEngineImpl.java:213) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.proceedToNextPhase(TransactionEngineImpl.java:68) \n com.google.scp.operator.cpio.distributedprivacybudgetclient.TransactionEngineImpl.executeDistributedPhase(TransactionEngineImpl.java:206)",
        "error_summary": {
            "error_counts": [],
            "error_messages": []
}

[maybellineboon]

Hi @TomerGoldstein ,

Thank you for sharing the information. We are working on the fix and will update everyone once the fix has been rolled out.

[nlrussell]

Hi all, a fix for the issue with Aggregation Service 2.5.0 has been identified, and we expect it to be rolled out by Monday, June 24. Please refer to the Public Status Dashboard for real-time updates: https://status.privacysandbox.com/

[nlrussell]

Hello, we have resolved the issue with Aggregation Service version 2.5.0. All users can now upgrade to 2.5.0 following the instructions for AWS or GCP. Thank you for your patience and continued support.