Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

'Extract Files' operation #440

Merged
merged 25 commits into from
Mar 11, 2019
Merged

'Extract Files' operation #440

merged 25 commits into from
Mar 11, 2019

Conversation

n1474335
Copy link
Member

@n1474335 n1474335 commented Dec 14, 2018 •
edited
Loading

Adds a new 'Extract Files' operation which detects files within a blob of data, parses them, and then extracts them.

Currently supports

  • JPG
  • EXE
  • ZIP
  • PDF
  • PNG
  • BMP
  • FLV
  • RTF
  • DOCX, XLSX, PPTX
  • EPUB
  • GZIP
  • ZLIB
  • ELF, BIN, AXF, O, PRX, SO

A Stream class has been added in src/core/lib/ to help parsing binary protocols.

TODO

  • Refactor file type detection to include more powerful search methods. I expect to address Support magic files for filetype detection #369 with this.
  • Refactor 'Scan for embedded files' to run much more efficiently. Pass offset to magic number check rather than reslicing the blob every time.
  • Add 'Move to input' button to displayFilesAsHTML function
  • Update 'Untar' operation to use the Stream lib.
  • ELF file format extractor

@n1474335 n1474335 self-assigned this Dec 14, 2018
@n1474335 n1474335 mentioned this pull request Dec 19, 2018
Merged
@n1474335 n1474335 mentioned this pull request Dec 30, 2018
Closed
@n1474335
Merge branch 'master' into feature-extract-files
fd07b89
@n1474335
Untar operation now uses lib/Stream library
3ae225a
@n1474335
Added PNG and BMP extractors
ede7553
@n1474335
Refactored scanning for file types to be more than twice as fast.
4c285bc
@n1474335
Significantly improved performance when scanning for embedded files b…
a56f92c 
…y implementing a fastcheck algorithm.
@n1474335
File scan now uses bytesMatch() instead of signatureMatches(), reduci…
cd0c86e 
…ng call stack size
@n1474335
Added FLV extractor.
0449c46
@n1474335
Added extractor for MS Office 2007+ files
7d8d80c
@n1474335
Fixed FLV previous tag size error
0d2cb02
@n1474335
Added RTF extractor
19b7957
@n1474335
Began implementing GZIP/DEFLATE extraction. Unfinished.
2a6db47
@n1474335
Stream.readBits() method implemented. Unfinished.
c077b22
@n1474335
Completed GZIP extraction
4e57b4b
@n1474335
Added Zlib extraction
2307325
@n1474335
Added ELF extractor. You can now specific which categories to search …
cd2c807 
…for in file type operations.
@mattnotmitt mattnotmitt mentioned this pull request Jan 23, 2019
Closed
@n1474335 n1474335 merged commit cd2c807 into master Mar 11, 2019
@n1474335 n1474335 mentioned this pull request Mar 11, 2019
Closed
@n1474335 n1474335 changed the title WIP: 'Extract Files' operation 'Extract Files' operation Mar 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@n1474335 n1474335

Labels
operation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@n1474335