Module boto3 - Invalid Model Issue

[giusepperaffa]

I have been trying to use Pysa (Ubuntu 20.04 + virtual environment + Python 3.8) to perform a data flow analysis of a function that relies on the boto3 module. The relevant lines of code are below. The list_objects call should be the source of my analysis.

s3 = boto3.client('s3')
list = s3.list_objects(Bucket=media_bucket)['Contents']

I do not seem to be able to implement a valid model for the above method call. Here is a summary of my attempts:

Attempt 1
I implemented the following model by considering the list_objects method documentation for the boto3 S3 Client:

def boto3.S3.Client.list_object(**kwargs) -> TaintSource[UserControlled]: …

The execution of the analysis is unsuccessful, and the error message is:

Module ‘boto3’ does not define boto3.S3.Client.list_object

Attempt 2
After checking the type of the S3 client object (Python interpreter session within my virtual environment + the type command), which is botocore.client.S3, I have tried the following models:

def botocore.client.S3.list_objects(**kwargs) -> TaintSource[UserControlled]: …
def boto3.botocore.client.S3.list_objects(**kwargs) -> TaintSource[UserControlled]: …

The error messages are essentially the same as Attempt 1:

Module ‘botocore’ does not define… / Module ‘boto3’ does not define...

Attempt 3
I noticed that the resource file boto3_clients_sources_sinks.pysa in this repository suggests using mypy-related resources. I then tried the following model:

def mypy_boto3_s3.client.S3Client.list_objects() -> TaintSource[UserControlled]: ...

The error message is different in this case:

mypy_boto3_s3.client.S3Client.list_objects is not part of the environment, no module mypy_boto3_s3 in search path

Note that my virtualenv's site-packages folder is included in the search_path of the Pysa configuration file, as suggested in #279.

@dark I have read your comment in #64. However, that dates back to 2018, and I hoped that the issue had been solved by now.

Please let me know if you need any additional information.

Thank you very much.

[r0rshark]

Hi @giusepperaffa thanks for the detailed issue! Attempt number 3 should be the right approach (boto3 is extremely dynamic and Pysa need the mypy stubs definitions in order to correctly work). Given the error message it looks like pyre is not able to pick up the mypy_boto3_s3 module:

1. Is the module installed in the virtualenv (pip install mypy-boto3-s3)? Are you able to see the folder mypy_boto3_s3 in the virtualenv location <venv_folder>\lib/python3.X/site-packages ?
2. Can you try manually adding the module to the search_path using the approach shown in pyre venv (virtualenv) support #316 (comment) ?

[giusepperaffa]

Hi @r0rshark - Thank you very much for your help:

1. You were correct, the module mypy_boto3_s3 was not installed in my virtualenv. The execution of the command pip install mypy-boto3-s3 solved the issue, and I can now see the folder mypy_boto3_s3 within the site-packages folder of my virtualenv.
2. Yes, I have manually updated my configuration file as suggested here. Pysa now sees the model as valid and usable.

However, the results of the analysis are not what I expected. This is probably due to the fact that I have to learn more about the module mypy_boto3_s3. I am currently looking at its documentation.

I think that this issue can now be closed. Let me know if you wish me to do it.

If the problems with the results of the analysis persist, I will then open another issue.

Thank you very much again.