Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address Warnings, Remove Redundant Python setup/requirements #330

Merged
merged 6 commits into from
Feb 19, 2023

Conversation

ryanohoro
Copy link
Collaborator

Describe the change

Updates build/python/backend/Dockerfile too use src/python/setup.py and src/python/requirements.txt

Removes build/python/backend/setup.py and build/python/backend/requirements.txt

Removes a user-defined setuptools option that was throwing:

/usr/lib/python3.10/distutils/dist.py:274: UserWarning: Unknown distribution option: 'namespace'

Forces the logging level of the msoffcrypto module to INFO to prevent passwords from being logged.

backend_1      | 2023-02-18 01:33:37 - [DEBUG] msoffcrypto.format.ooxml [ooxml.__init__]: OOXMLFile.type: standard
backend_1      | 2023-02-18 01:33:38 - [DEBUG] msoffcrypto.method.ecma376_standard [ecma376_standard.makekey_from_password]: ['aaa', '0x660e', '0x8004', '0x18', 128, 16, b'\x93\x8b\x91"\xb9\xed^\x12s\x885q\xed\x8b6\xfb']
backend_1      | 2023-02-18 01:33:38 - [DEBUG] msoffcrypto.method.ecma376_standard [ecma376_standard.makekey_from_password]: b'V\x126\xb8JK\x86\x8fR\xe0\x8e\xe8\xa3\x01\\\xdf'
backend_1      | 2023-02-18 01:33:38 - [DEBUG] msoffcrypto.method.ecma376_standard [ecma376_standard.decrypt]: totalSize: 32042

Disables caching module in Dockerfile pytest to remove warnings (cache isn't useful in a container anyway, pytest is running unprivileged):

../../usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433
  /usr/local/lib/python3.10/dist-packages/_pytest/cacheprovider.py:433: PytestCacheWarning: cache could not write path /strelka/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

Replaces fromstring() with frombuffer() in scan_lsb.py and scan_nf.py to remove warnings:

backend_1      | /usr/local/lib/python3.10/dist-packages/strelka-0.0.0-py3.10.egg/strelka/scanners/scan_lsb.py:12: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
backend_1      | /usr/local/lib/python3.10/dist-packages/strelka-0.0.0-py3.10.egg/strelka/scanners/scan_nf.py:25: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead

Describe testing procedures

docker-compose -f build/docker-compose.yaml build

...

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /strelka
plugins: mock-3.10.0, unordered-0.5.2
collected 122 items

tests/test_required_for_scanner.py .
tests/test_scan_base64.py .
tests/test_scan_base64_pe.py .
tests/test_scan_batch.py .
tests/test_scan_bmp_eof.py .
tests/test_scan_bzip2.py .
tests/test_scan_capa.py ...
tests/test_scan_ccn.py .
tests/test_scan_delay.py .
tests/test_scan_dmg.py ...
tests/test_scan_docx.py .
tests/test_scan_elf.py .
tests/test_scan_email.py .
tests/test_scan_encrypted_doc.py ....
tests/test_scan_encrypted_zip.py ..
tests/test_scan_entropy.py .
tests/test_scan_exception.py .
tests/test_scan_exiftool.py ..
tests/test_scan_footer.py ..
tests/test_scan_gif.py .
tests/test_scan_gzip.py .
tests/test_scan_hash.py .
tests/test_scan_header.py ..
tests/test_scan_html.py ..
tests/test_scan_ini.py .
tests/test_scan_iso.py .
tests/test_scan_javascript.py ..
tests/test_scan_jpeg.py ..
tests/test_scan_json.py .
tests/test_scan_libarchive.py ......
tests/test_scan_lnk.py .
tests/test_scan_lzma.py .
tests/test_scan_macho.py .
tests/test_scan_manifest.py .
tests/test_scan_msi.py .
tests/test_scan_nf.py ....
tests/test_scan_ocr.py ...
tests/test_scan_ole.py ....
tests/test_scan_onenote.py ..
tests/test_scan_pcap.py ..
tests/test_scan_pdf.py .
tests/test_scan_pe.py .
tests/test_scan_pgp.py ....
tests/test_scan_plist.py .
tests/test_scan_png_eof.py ...
tests/test_scan_qr.py ...
tests/test_scan_rar.py .
tests/test_scan_seven_zip.py .....
tests/test_scan_strings.py .
tests/test_scan_tar.py .
tests/test_scan_tlsh.py .
tests/test_scan_transcode.py ...................
tests/test_scan_upx.py .
tests/test_scan_url.py ..
tests/test_scan_vhd.py ..
tests/test_scan_x509.py ..
tests/test_scan_xl4ma.py .
tests/test_scan_xml.py .
tests/test_scan_yara.py .
tests/test_scan_zip.py ..

====================== 122 passed, 24 warnings in 43.55s =======================
./strelka-oneshot -l - -f src/python/strelka/tests/fixtures/test.png

{"file":{"depth":0,"flavors":{"mime":["image/png"],"yara":["png_file"]},"name":"test.png","scanners":["ScanEntropy","ScanExiftool","ScanFooter","ScanHash","ScanHeader","ScanLsb","ScanNf","ScanOcr","ScanPngEof","ScanQr","ScanTlsh","ScanYara"],"size":539355,"tree":{"node":"9caf4c19-b2d9-416c-b4ef-9515a5eef139","root":"9caf4c19-b2d9-416c-b4ef-9515a5eef139"}},"request":{"attributes":{"filename":"test.png"},"client":"go-oneshot","id":"9caf4c19-b2d9-416c-b4ef-9515a5eef139","source":"ubuntu","time":1676695140},"scan":{"entropy":{"elapsed":0.000409,"entropy":7.941353347991658},"exiftool":{"elapsed":0.122625,"keys":[{"key":"ImageWidth","value":1236},{"key":"ImageHeight","value":891}]},"footer":{"backslash":"K\\x00\\xba\\x80\\x86\\xb1A33\\x93\\x1c\\x00\\x00@\\xa0\\x1b\\xc1\\xf2\\x06A\\xd2\\x0c-IF\\x81`\\x03\\xf8\\x7f\\xd0\\x9aA\\xba\\x9b'F\\x97\\x00\\x00\\x00\\x00IEND\\xaeB`\\x82","elapsed":0.000058,"footer":"K\u0000����A33�\u001c\u0000\u0000@�\u001b��\u0006A�\u000c-IF�`\u0003��КA��'F�\u0000\u0000\u0000\u0000IEND�B`�"},"hash":{"elapsed":0.013075,"md5":"8d39d685063ed37f21bc13a91276c2ca","sha1":"7c65103b65df7c3669569332c5a4a089cbadf792","sha256":"fbc5d27a8839e5d3298bb0c04ba2c92bf91d08912409fc69f06363ce8c1f1afa","ssdeep":"12288:837ZtoNwdQCzsLhGxMO1QdbKhGtiNEqdXIHy:27ZeKdQCw9GxX15miaqMy","tlsh":"T19EB42376E3871A85F09A989F8C1C0B14ADD795201B4C0CE5F4FEB92E3DE4D45CE0A9B6"},"header":{"backslash":"\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\x04\\xd4\\x00\\x00\\x03{\\x08\\x00\\x00\\x00\\x00\\xa8\\xce\\x98\\xe0\\x00\\x00\\x00\\tpHYs\\x00\\x00\\x0b6\\x00\\x00\\x0b6\\x01","elapsed":0.000036,"header":"�PNG\r\n\u001a\n\u0000\u0000\u0000\rIHDR\u0000\u0000\u0004�\u0000\u0000\u0003{\u0008\u0000\u0000\u0000\u0000�Θ�\u0000\u0000\u0000\tpHYs\u0000\u0000\u000b6\u0000\u0000\u000b6\u0001"},"lsb":{"elapsed":0.015532,"lsb":true},"nf":{"elapsed":0.015968,"noise_floor":true,"percentage":0,"threshold":0.25},"ocr":{"elapsed":0.167427},"png_eof":{"elapsed":0.000041,"flags":["no_trailer"]},"qr":{"elapsed":0.088679},"tlsh":{"elapsed":0.00479},"yara":{"elapsed":0.002191,"matches":["test"]}}}

Sample output

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of and tested my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings

Copy link
Contributor

@phutelmyer phutelmyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ryanohoro ryanohoro marked this pull request as ready for review February 19, 2023 04:09
@phutelmyer phutelmyer merged commit ce66caf into target:master Feb 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants