🐛 datcore-adapter (Pennsieve API): filter out deleted files

[elisabettai]

What do these changes do?

When querying dataset packages, e.g. with:

GET https://api.pennsieve.io/datasets/N:dataset:a20b0a68-11be-4c39-a65a-15903de5edc2/packages

There might be deleted files in the response, e.g.:

{
      "content": {
        "id": 3706023,
        "name": "__DELETED__N:package:268f6590-c5d3-4efd-9538-8e6c3da77f54_submission.xlsx",
        "packageType": "Unsupported",
        "datasetId": 2998,
        "datasetNodeId": "N:dataset:a20b0a68-11be-4c39-a65a-15903de5edc2",
        "ownerId": 832,
        "state": "DELETED",
        "createdAt": "2023-12-28T19:28:06.491464Z",
        "updatedAt": "2023-12-28T19:32:38.143404Z",
        "nodeId": "N:package:268f6590-c5d3-4efd-9538-8e6c3da77f54"
      },
      "properties": [
        {
          "category": "Pennsieve",
          "properties": [
            {
              "key": "subtype",
              "value": "MS Excel",
              "dataType": "string",
              "fixed": false,
              "hidden": true,
              "display": "MS Excel"
            },
            {
              "key": "icon",
              "value": "Excel",
              "dataType": "string",
              "fixed": false,
              "hidden": true,
              "display": "Excel"
            }
          ]
        }
      ],
      "children": [],
      "extension": "xlsx"
    },

Once we then try to get those files, a 404 error is raised.
At the moment, the Pennsieve API does not have a flag to exclude those files, so we need to filter them out on our end (this was confirmed by Pennsieve devs).

Related issue/s

Fixes ITISFoundation/osparc-issues#1231
(Original support request came via the oSPARC Support Google form, see details here https://z43.fogbugz.com/f/cases/192158)

How to test

Manually

1. Use the oSPARC UI to navigate the datcore data tree.
2. Before this fix: the dataset "Influence of vagus nerve stimulation on cardiac activity in pigs" will load forever
3. After the fix: the dataset "Influence of vagus nerve stimulation on cardiac activity in pigs" works

Unit tests
Run tests with your API credentials and --dataset=dddbce01-bae0-4e2b-85fd-dafb47182afa

Dev Checklist

• No ENV changes or I properly updated ENV (read the instruction)

DevOps Checklist

Not applicable (at least for this change). 😉

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c0e329d) 57.3% compared to head (c0f72b8) 65.6%.

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #5254     +/-   ##
========================================
+ Coverage    57.3%   65.6%   +8.3%     
========================================
  Files         538     561     +23     
  Lines       22563   27433   +4870     
  Branches      281     202     -79     
========================================
+ Hits        12944   18018   +5074     
+ Misses       9566    9365    -201     
+ Partials       53      50      -3     

Flag	Coverage Δ
integrationtests	65.1% <ø> (+23.4%)	⬆️
unittests	92.5% <100.0%> (+1.5%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...mcore_service_datcore_adapter/modules/pennsieve.py	93.0% <100.0%> (+0.1%)	⬆️

... and 507 files with indirect coverage changes

[pcrespov]

thx!

[sonarqubecloud]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[sanderegg]

perfect! thanks!!