tighten rule pre-selection

[williballenthin]

closes #2074
ref #2063, particularly "tighten rule pre-selection" and "lots of time spent in instancecheck"

Stacked on #1950, so I've marked this as a PR onto that branch so the diff is sensible. I think we can probably rebase onto master, though, if necessary.

---

This PR implements the "tighten rule pre-selection" algorithm described here: #2063 (comment) . In summary:

Rather than indexing all features from all rules, we should pick and index the minimal set (ideally, one) of features from each rule that must be present for the rule to match. When we have multiple candidates, pick the feature that is probably most uncommon and therefore "selective".

This seems to work pretty well. Total evaluations when running against mimikatz drop from 19M to 1.1M (wow!) and capa seems to match around 3x more functions per second (wow wow). I did not expect such a good result - in fact, although the capa matches seem the be the same, I still wonder if something is broken 🤔. More tests needed.

label	count(evaluations)	time
[before any optimization, prehistoric]	104,496,193	86.62s
8858537 pep8 [before this PR]	19,939,632	25.74s
a66524a rules: match: better debug paranoid matching	1,157,514	8.10s

TODO:

• add some tests for the feature indexer, if only to show a human how it works
• namespace matching
• prove that it matches exactly the same as before, just faster
• xfail the tests and document the unsupported constructs
• inline documentation explaining the algorithm better
• wall clock performance numbers

[williballenthin]

for backwards compatibility. during a major version, we can probably remove these with preference to rules_by_scope.

[mr-tz]

nice work, we should do extensive tests comparing the results before and after to ensure everything works as expected. the speedup looks promising!

[williballenthin]

we should do extensive tests comparing the results before and after to ensure everything works as expected.

I plan to run this implementation side by side with the ceng.match implementation and assert the results are precisely the same across a wide range of samples. There should be no leaks of abstraction or details in the new one, it should just be faster.

CHANGELOG updated or no update needed, thanks! 😄

[mike-hunhoff]

Can this be optimized? We're looping and calling isinstance on every feature three times.

[williballenthin]

let me try and then run some benchmarks. I agree it looks wasteful, but I'm not sure if it has a real world effect.

[mike-hunhoff]

This looks great @williballenthin - I'm pumped about the improved efficiency. The logic and code that you've implemented here appears sound. Let's get this merged pending successful paranoid invocation across a wide range of samples

[mike-hunhoff]

Should we rebase this on top of master so that it doesn't depend on BinExport2?

I'm inclined to say "yes" although we lose the intermediate history. This would allow us to do a minor release and get the optimizations out there.

Yes let's rebase on master so we can get this to our users ASAP

[mr-tz]

amazing work! noted a few minor things I've noticed and the paranoid run will provide a lot of value

[mr-tz]

add TODOs for these notes?

[williballenthin]

yeah, and i'll spin off the original issue comments into dedicated issues we can use to track the idea.

[fariss]

Very good improvmenets, I just have question below.

---

Unrelated to this PR, I think we can replace

capa/capa/rules/__init__.py

Line 459 in 960ee86

b = codecs.decode(s.replace(" ", "").encode("ascii"), "hex")

with:

b = bytes.fromhex(s)

https://docs.python.org/3/library/stdtypes.html#bytes.fromhex

[williballenthin]

paranoid linting succeeded!

❯ time python scripts/lint.py rules/ --thorough
INFO:lint:collecting potentially referenced samples
                                                                                                                                                                                                     
encrypt data using RC4 via SystemFunction033                                                                                                                                                         
FAIL: referenced example doesn't exist: Add the referenced example to samples directory ($capa-root/tests/data or supplied via --samples)                                                         
                                                                                                                                                                                                        
(nursery)  linked against hp-socket                                                                                                                                                                   
WARN: referenced example doesn't exist: Add the referenced example to samples directory ($capa-root/tests/data or supplied via --samples)                                                                                                                                                                                                                                                         rules with WARN:                                                                                                                                                                                      - linked against hp-socket

rules with FAIL:
  - encrypt data using RC4 via SystemFunction033

________________________________________________________
Executed in  125.20 mins    fish           external
   usr time  124.04 mins   66.00 micros  124.04 mins
   sys time    0.98 mins  898.00 micros    0.98 mins

	time
paranoid	125 minutes
master	62 minutes
this PR	44 minutes

So, this improves the performance of capa (with the vivisect backend) by about 30%. When using the BinExport2 backend, I think the performance improvement will be closer to 2-3x, since less time is spent doing analysis.

[mr-tz]

awesome, big performance improvement!

[williballenthin]

new PR that's rebased against master: #2125