feat: add file-based access permissions for SharePoint ingest

[ahmetmeleq]

This PR:

• defines rbac_data as a SourceMetadata field,
• manages connections to an external api for obtaining rbac data with ConnectorRBAC class,
• serializes rbac data and saves it to the disk,
• matches the rbac_data in the disk to each IngestDoc, using a common field,
• forwards rbac data to Elements, via the partition() function

To test the changes, run examples/ingest/sharepoint/ingest.sh with the relevant rbac & connector credentials

[ahmetmeleq]

Potential improvements before merging:

• improve rbac data to ingestdoc matching logic (currently checking filenames in a naive way)
• better folder management for writing down the rbac data (currently writes all rbac data directly into the output_dir)
• obtaining rbac_data field as a python object and serializing later, (currently obtains it as a str)
• handling errors caused by upstream connection issues
  • ie requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: ...

[ahmetmeleq]

Note: Reading permissions data as an object rather than a str causes proper formatting in the outputted elements.json file (rather than keeping the permissions data in one line), and results in permission data dominating / polluting the file. Repeating it for each element adds to the problem. Open to suggestions about this

[ahmetmeleq]

Note: Sharepoint seems to be having trouble with our parallel requests, giving:

Error: {"error":"invalid_request",
"error_description":"AADSTS900023: Specified tenant identifier 'none' is neither a valid DNS name, nor a valid external domain.
...}

or

office365.runtime.client_request_exception.ClientRequestException: (None, None, "503 Server Error: Service Unavailable for url: 

To address this, I've reduced num-processes in the example .sh to 1, and it seems like this helps

[ahmetmeleq]

Example file for testing:

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
cd "$SCRIPT_DIR"/../../.. || exit 1

PYTHONPATH=. ./unstructured/ingest/main.py \
    sharepoint \
    --client-id $SHAREPOINT_CLIENT_ID \
    --client-cred $SHAREPOINT_CRED \
    --permissions-application-id $SHAREPOINT_RBAC_CLIENT_APPLICATION_ID \
    --permissions-client-cred $SHAREPOINT_RBAC_CLIENT_SECRET \
    --site $SHAREPOINT_SITE \
    --permissions-tenant $SHAREPOINT_RBAC_TENANT \
    --output-dir sharepoint-ingest-output \
    --num-processes 1 \
    --path "Shared Documents" \
    --verbose

[ahmetmeleq]

Test fixtures update PR on the way as well

are these still on route?

We're there now 👍

[ryannikolaidis]

couple nits on docs and a comment / followup on CliPermissionsConfig

[ryannikolaidis]

lgtm! nice work, I know this was a huge lift.