feat: Integrate AnonCreds with W3C VCDI Format Support in ACA-Py

[sarthakvijayvergiya]

Summary

Pull request introduces comprehensive support for AnonCreds within the W3C VCDI formats. The changes include the addition of new models, methods, and protocols that enable the Aries Cloud Agent Python to issue, hold, and verify credentials in alignment with W3C's VCDI standards.

Changes Overview

• AnonCreds Integration: Updated the anoncreds module to handle VCDI-compliant credential offers, requests, and credentials.
• Indy Models: Adapted indy/models to map onto VCDI credential.
• Issue Credential Protocol v2.0: Incorporated VCDI support into the existing issue_credential protocol
• VCDI Credential Format Handling: Introduced a new vc_di module within formats to handle the specificities of the VCDI credential format.
• Updated alice/faber demo: Introduced handling of multiple cred demo with cred-type argument

Detailed Changes

• Added handler.py in protocols/issue_credential/v2_0/formats/vc_di to manage VCDI credential operations.
• Created test_handler.py in the same directory to ensure robustness and reliability through comprehensive testing.
• Introduced vc_di.py within models/detail for detailed VCDI model definitions.
• Updated test_routes.py to test the integration of VCDI routes within the agent's service layer.
• Revised routes.py to handle API endpoints related to VCDI credential operations.
• Updated faber.py
• Update alice.py
• Updated agent_container.py
• Updated performance.py

Additional Notes

Link to the detailed documentation and specifications: W3C VCDI Integration

[ianco]

Does this replace the other PR? (In which case we can close it)

Also please make sure to include in the description of this PR a ... description of what the PR includes (and if there is any work still in progress) to help us to know what to review.

Thanks

[sarthakvijayvergiya]

Does this replace the other PR? (In which case we can close it)

Also please make sure to include in the description of this PR a ... description of what the PR includes (and if there is any work still in progress) to help us to know what to review.

Thanks

Thank you for your comments. Indeed, this replaces the previous PR and will close that one. I will update the detaild description as soon as I complete the modification and testing.

[gvelez17]

Looks good @sarthakvijayvergiya make sure as you wrap up to check all the comments in #2830 as well

[gvelez17]

Does this replace the other PR? (In which case we can close it)

Also please make sure to include in the description of this PR a ... description of what the PR includes (and if there is any work still in progress) to help us to know what to review.

Thanks

yes, closed the other. this one is not fully ready for review yet, @sarthakvijayvergiya however can you keep the description up to date on what parts are complete?

[ianco]

Overall looks good. I added some comments - there are some cleanup tasks I think we need before we can merge the PR, but also a bunch of stuff we can cleanup later (which I think we should add the TODO comments now so we don't lose track of them).

The demo "issue credential" works with the vc_di format but the proof request fails (not sure why I haven't looked into it yet).

[ianco]

Also ... I think for the demo we should add an option where the user can change Faber's credential_type "on the fly" (i.e. specify on startup but then change while faber is running). Eventually we need to test issuing a credential using "vc_di" and then testing a "legacy" proof request, or issuing a "legacy" credential and then requesting a "vc_di" proof.

Again not necessary for this PR ...

[ianco]

I'm still doing some testing, but overall looks good! Just a few minor comments ...

A couple of things we need:

• updates to documentation - a reference in the demo docs that --cred-type vc_di is now an option
• maybe in docs/features ... reference to vc_di as a supported format for issuing credentials (@swcurran where should this be documented?)
• at least one integration test

For the last one, it should be a simple matter of adding a new scenario to 0453-issue-credential.feature, like:

    @GHA @WalletType_Askar_AnonCreds @BasicTest
    Examples:
       | Acme_capabilities                          | Bob_capabilities              | Schema_name    | Credential_data          | Acme_extra | Bob_extra |
       | --public-did --wallet-type askar-anoncreds | --wallet-type askar-anoncreds | driverslicense | Data_DL_NormalizedValues |            |           |
       | --public-did --wallet-type askar-anoncreds --cred-type vc_di | --wallet-type askar-anoncreds | driverslicense | Data_DL_NormalizedValues |            |           |

[swcurran]

I would really like to see at least an interactive run of using Credo-TS as a holder, and using ACA-Py as an issuer and verifier. Crucial for the CWU is to see the interop. Thoughts on that? We can engage the devs from Animo in helping with that.

[ianco]

I would really like to see at least an interactive run of using Credo-TS as a holder, and using ACA-Py as an issuer and verifier. Crucial for the CWU is to see the interop. Thoughts on that? We can engage the devs from Animo in helping with that.

Could we do this using AATH? (Or AMTH?)

[swcurran]

AATH would be awesome. I don’t know the effort, but we would need someone that knows AATH to help out. The scope is too much for this work, unfortunately. My big concern is the current state of the Credo Backchannel. It needs work.

[ianco]

I ran a quick test of the alice/faber demo and it failed. I did the ./run_demo build ... on each agent and then:

Faber: $ AGENT_PORT_OVERRIDE=8010 ./run_demo run faber --wallet-type askar-anoncreds --cred-type vc_di

Alice: $ ./run_demo run alice --wallet-type askar-anoncreds

Then connected and tried to issue a credential:

[1/2/3/4/T/X] 1                                                                                                                                                                                                    

#13 Issue credential offer to X
Faber      | Credential: state = offer-sent, cred_ex_id = d5d5afa6-bb30-4b35-a085-96b935cbb9cc
Faber      | 2024-04-29 16:19:42,627 aries_cloudagent.messaging.models.base ERROR V20CredRequest message validation error:
Faber      | Traceback (most recent call last):
Faber      |   File "/home/aries/aries_cloudagent/messaging/models/base.py", line 196, in deserialize
Faber      |     schema.loads(obj) if isinstance(obj, str) else schema.load(obj),
Faber      |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 723, in load
Faber      |     return self._do_load(
Faber      |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 910, in _do_load
Faber      |     raise exc
Faber      | marshmallow.exceptions.ValidationError: {'binding_proof': {'anoncreds_link_secret': {'entropy': ['Value did:peer:4zQmetubuCeDizQRVaJWscDcGxrz65Cyv7RyFMAmCCFpQk22 is not an indy decentralized identifier (DID)']}}}
Faber      | 2024-04-29 16:19:42,630 aries_cloudagent.core.dispatcher ERROR Message parsing failed: Error deserializing message: V20CredRequest schema validation failed, sending problem report

Tagging @swcurran and @dbluhm because it looks like we're using a did:peer:4 here for some reason and the marshmallow schema isn't accepting it ...

[ianco]

... also when I run the integration tests they all report as "pass" but when I look through the logs there are errors:

$ AGENT_PORT_OVERRIDE=8030 ./run_bdd -t @T003-RFC0453
...
    When "Bob" requests a credential with data Data_DL_NormalizedValues from "Acme" it fails # features/steps/0453-issue-credential.py:127
Bob.agent  | 2024-04-29 16:41:48,606 aries_cloudagent.admin.server ERROR Handler error with exception: Unprocessable Entity
Bob.agent  | Traceback (most recent call last):
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/webargs/asyncparser.py", line 90, in parse
Bob.agent  |     result = schema.load(parsed)
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 723, in load
>>> after_scenario activated
>>> shutting down active agents ...
    shutting down: Acme
Shutting down agent ...
Bob.agent  |     return self._do_load(
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 910, in _do_load
Bob.agent  |     raise exc
Bob.agent  | marshmallow.exceptions.ValidationError: {'filter': {'ld_proof': ['Missing data for required field.']}}
Bob.agent  | 
Bob.agent  | During handling of the above exception, another exception occurred:
Bob.agent  | 
Bob.agent  | Traceback (most recent call last):
Bob.agent  |   File "/home/aries/aries_cloudagent/admin/server.py", line 181, in ready_middleware
Bob.agent  |     return await handler(request)
Bob.agent  |   File "/home/aries/aries_cloudagent/admin/server.py", line 218, in debug_middleware
Bob.agent  |     return await handler(request)
Bob.agent  |   File "/home/aries/aries_cloudagent/admin/server.py", line 386, in check_multitenant_authorization
Bob.agent  |     return await handler(request)
Bob.agent  |   File "/home/aries/aries_cloudagent/admin/server.py", line 451, in setup_context
Bob.agent  |     return await task
Bob.agent  |   File "/usr/local/lib/python3.9/asyncio/futures.py", line 284, in __await__
Bob.agent  |     yield self  # This tells Task to wait for completion.
Bob.agent  |   File "/usr/local/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
Bob.agent  |     future.result()
Bob.agent  |   File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result
Bob.agent  |     raise self._exception
Bob.agent  |   File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step
Bob.agent  |     result = coro.send(None)
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/aiohttp_apispec/middlewares.py", line 33, in validation_middleware
Bob.agent  |     data = await request.app["_apispec_parser"].parse(
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/webargs/asyncparser.py", line 94, in parse
Bob.agent  |     await self._on_validation_error(
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/webargs/asyncparser.py", line 108, in _on_validation_error
Bob.agent  |     await error_handler(error, req, schema, error_status_code, error_headers)
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/webargs/aiohttpparser.py", line 163, in handle_error
Bob.agent  |     raise error_class(
Bob.agent  | aiohttp.web_exceptions.HTTPUnprocessableEntity: Unprocessable Entity

[ianco]

Running the alice/faber demo with the --events flag I can see that alice receives the credential offer and sends the credential request, and faber gets the error receiving the request. Here is the cred request, I'm not sure how the did:peer:4 gets in there or why faber cares what is the format of the "entropy":

        "cred_request": {
            "vc_di": {
                "data_model_version": "2.0",
                "binding_proof": {
                    "anoncreds_link_secret": {
                        "entropy": "did:peer:4zQmSz9RPHAXYyKe8RNEaH3DWz6jAzYVADZWt82h88URnYTg",
                        "cred_def_id": "7JcY3HVEaUwXdoXeSeFSSi:3:CL:176:faber.agent.degree_schema",
                        "blinded_ms": {
                            "u": "98157803620660872170402354543798974200462544452245591862217725500504386281160961655563679393987978770632568196480064042932031478725476063915670091399243123044122923246815807721802098890962420986125428117490105853961326051702005632977344987905700321953571139659140865989527824169498631840709995380507817419826662772277722439467289687048146883920558627162230071386049697776277292986334401664583542979134445423113364037510563284891531520997746722867719148651795256773203744802404879619935812531852155599459431918862100233592485693666170250989538356501445054894228211868579282658727052977550795345340387507015924540012548",
                            "ur": null,
                            "hidden_attributes": [
                                "master_secret"
                            ],
                            "committed_attributes": {}
                        },
                        "blinded_ms_correctness_proof": {
                            "c": "59433802809596338271128470279603183486702814236484143165828684066161975073261",
                            "v_dash_cap": "2291688135408765057454102999713290079468470806026404345010878504526654291637245987562467966244405580205292137949428093919085198310059447159324212870553877066735257526420053820932460859921179097316828804442130780534924326501078304785117073706502754035170388518062719574873475329608590008271662112168037204411383802356689765012068442072241086717499759364113588149218843455030220001906292019130107467050776769753277284854618996023764841503608868017026452558261583504715139095668493430196979432551095789861656950035970995288498080778896408155269698695357277285620211683342360165406497987029591209902977936516027482604268823342975636845320416001178746420147803174461964012518294044516186535247995738909218895238190583877815",
                            "m_caps": {
                                "master_secret": "22460859598887235898916958953680270931050220445132084890048716866094989991233598532889415785637083539883931553700153352400740539022798084603405103364810530827905738895125862866024"
                            },
                            "r_caps": {}
                        },
                        "nonce": "595869006883776147466000"
                    },
                    "didcomm_signed_attachment": {
                        "attachment_id": "test"
                    }
                }
            }
        }

[swcurran]

I ran a quick test of the alice/faber demo and it failed. I did the ./run_demo build ... on each agent and then:

Faber: $ AGENT_PORT_OVERRIDE=8010 ./run_demo run faber --wallet-type askar-anoncreds --cred-type vc_di

Alice: $ ./run_demo run alice --wallet-type askar-anoncreds

Then connected and tried to issue a credential:

[1/2/3/4/T/X] 1                                                                                                                                                                                                    

#13 Issue credential offer to X
Faber      | Credential: state = offer-sent, cred_ex_id = d5d5afa6-bb30-4b35-a085-96b935cbb9cc
Faber      | 2024-04-29 16:19:42,627 aries_cloudagent.messaging.models.base ERROR V20CredRequest message validation error:
Faber      | Traceback (most recent call last):
Faber      |   File "/home/aries/aries_cloudagent/messaging/models/base.py", line 196, in deserialize
Faber      |     schema.loads(obj) if isinstance(obj, str) else schema.load(obj),
Faber      |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 723, in load
Faber      |     return self._do_load(
Faber      |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 910, in _do_load
Faber      |     raise exc
Faber      | marshmallow.exceptions.ValidationError: {'binding_proof': {'anoncreds_link_secret': {'entropy': ['Value did:peer:4zQmetubuCeDizQRVaJWscDcGxrz65Cyv7RyFMAmCCFpQk22 is not an indy decentralized identifier (DID)']}}}
Faber      | 2024-04-29 16:19:42,630 aries_cloudagent.core.dispatcher ERROR Message parsing failed: Error deserializing message: V20CredRequest schema validation failed, sending problem report

Tagging @swcurran and @dbluhm because it looks like we're using a did:peer:4 here for some reason and the marshmallow schema isn't accepting it ...

I created issue #2923 for this — I think it is an urgent one for 0.12.1. An qualified DID is being used by Alice, being sent in the request message, and is incorrectly being verified and required to be an unqualified (or did:sov) peer DID. The verification needs to be removed from wherever it is happening.

[ianco]

I created issue #2923 for this — I think it is an urgent one for 0.12.1. An qualified DID is being used by Alice, being sent in the request message, and is incorrectly being verified and required to be an unqualified (or did:sov) peer DID. The verification needs to be removed from wherever it is happening.

I suspect this is just an issue with the new vc_di but I will do a bit of digging

[ianco]

Still a couple of issues to resolve. The validation for the "entropy" should be straightforward. Not sure exactly what is happening with the integration tests.

[ianco]

These integration tests need to work with the sample data provided, which (in this test) is in this file: https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/features/data/cred_data_schema_driverslicense.json

The data is loaded and the passed through to the integration test. The data is loaded here: https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/bdd_support/agent_backchannel_client.py#L205

... and the code for the integration test step is here:

https://github.com/hyperledger/aries-cloudagent-python/blob/main/demo/features/steps/0453-issue-credential.py#L56

... which calls the faber agent to issue the credential.

Somewhere along the line a value is getting dropped (see the other comment re integration test errors).

...

[sarthakvijayvergiya]

@ianco Do we need to do anything here? I checked the status of integration tests and it works fine.

[ianco]

I'm taking a look at the integration tests now ...

[gvelez17]

@ianco when you take a look at this, my thought is if the code is correct and passing the current integration tests, we should merge this PR before it gets more outdated, and start a new one for any additional integration tests and also the one for the DIF format (see questions there)

Let me know if you think differently.

[gvelez17]

One question - if in the DIF pr they are calling it di_vc should we change all references in here to di_vc ? @swcurran was there a decision made about what the name is, of Veriable Credentials with Data Integrity, 'vc_di' or 'di_vc' ?

[ianco]

See comment re calling the process method on the legacy credential.

To duplicate, run the alice/faber demo (commands are in a previous comment somewhere), issue a credential, change cred type to indy (option "1a") and then ask for a proof. The proof should validate.

[ianco]

@ianco when you take a look at this, my thought is if the code is correct and passing the current integration tests, we should merge this PR before it gets more outdated, and start a new one for any additional integration tests and also the one for the DIF format (see questions there)

Let me know if you think differently.

Yes once this is working we should merge, and then we can deal with outstanding functionality in separate PR's.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

[ianco]

LGTM, tagging @jamshale for a review.

[jamshale]

Small spelling mistake on islolat

[ianco]

Good catch we can fix in the next PR tho ...

[jamshale]

I'm a bit confused what this is doing? If it is processing a cred_w3c credential wouldn't it have, and use, a return value?

[ianco]

We receive a W3C credential, but then we have to convert to an Indy credential to store. So, we have to process the W3C credential before we can convert to Indy format, but we don't store the W3C version of the credential explicitely.

[jamshale]

One tiny spelling mistake that might as well get fixed, and one question.

The code is clean and looks well tested!