Invalid rule in recent debugger-detection commit

[adamprescott91]

When downloading and running capa explorer on todays ruleset the following error is generated:

[ERROR] Failed to load rules from .../capa-rules-master (error: invalid rule: .../capa-rules-master\anti-analysis\anti-debugging\debugger-detection\check-for-kernel-debugger-via-shared-user-data-structure.yml: invalid rule: unexpected statement: instruction). (form:load_capa_rules)
[ERROR] Make sure your file directory contains properly formatted capa rules. You can download the standard collection of capa rules from https://github.com/mandiant/capa-rules. (form:load_capa_rules)
[INFO] Analysis failed. (form:analyze_program)

[adamprescott91]

There is also an issue with:

anti-analysis\anti-debugging\debugger-detection\check-for-peb-ntglobalflag-flag.yml

[adamprescott91]

And:

anti-analysis\anti-debugging\debugger-detection\check-for-software-breakpoints.yml

[adamprescott91]

And:

anti-analysis\anti-debugging\debugger-detection\check-for-trap-flag-exception.yml

[adamprescott91]

And:

anti-analysis\anti-disasm\64-bit-execution-via-heavens-gate.yml
collection\credit-card\parse-credit-card-information.yml
communication\http\client\check-http-status-code.yml
data-manipulation\checksum\adler32\compute-adler32-checksum.yml
data-manipulation\checksum\crc32\hash-data-with-crc32.yml
data-manipulation\compression\decompress-data-using-aplib.yml
data-manipulation\encryption\elliptic-curve\encrypt-data-using-curve25519.yml
data-manipulation\encryption\elliptic-curve\encrypt-data-using-curve25519.yml
data-manipulation\encryption\rc4\encrypt-data-using-rc4-ksa.yml
data-manipulation\encryption\sosemanuk\encrypt-data-using-sosemanuk.yml
data-manipulation\hashing\djb2\hash-data-using-djb2.yml
data-manipulation\hashing\murmur\hash-data-using-murmur3.yml
host-interaction\file-system\exists\check-if-file-exists.yml
host-interaction\network\connectivity\check-internet-connectivity-via-wininet.yml
host-interaction\os\version\check-os-version.yml
host-interaction\process\terminate\terminate-process-via-fastfail.yml
host-interaction\recycle-bin\empty-recycle-bin-quietly.yml
lib\calculate-modulo-256-via-x86-assembly.yml
lib\peb-access.yml
lib\validate-payment-card-number-using-luhn-algorithm-with-lookup-table.yml
lib\validate-payment-card-number-using-luhn-algorithm-with-no-lookup-table.yml
load-code\shellcode\execute-shellcode-via-copyfile2.yml
nursery\check-thread-yield-allowed.yml
nursery\generate-random-numbers-using-the-delphi-lcg.yml
nursery\hash-data-using-murmur2.yml
nursery\log-keystrokes-via-raw-input-data.yml
nursery\search-for-credit-card-data.yml

This leads to

error: invalid rule: rule "get process heap flags" depends on missing rule "PEB access"

And other related ones...

For reference
Env info
Windows 10
Python 3.8.5 64-bit
IDA 7.7 x86 version
flare-capa 3.2.0

[adamprescott91]

Running this version: https://github.com/mandiant/capa-rules/tree/82308c4109c97688e8a2e3f346520b064fb868e7
Works with my setup with no issues

[mr-tz]

Yep, that's expected although a little confusing. The commit you reference is what's supported by the latest release.

Besides using the rules at that point you could use the capa source from master. We'll also do a new release in the next few weeks.

Does that work for you?

[adamprescott91]

Ah I see. I may have missed it but it feels like there is some missing documentation on how to download the Capa rules repo then.

From the guides in the capa IDA repo it just says to

Download the standard collection of capa rules

And when I got to that repo I didn't see any links for "release" versions or documentation in the readme that says the master branch is not the one to download...
Now looking back at it I'm assuming that the "tagged" branches are meant to be the ones that are stable for release?

[williballenthin]

Yeah, this is our fault in 1) not really documenting exactly what to download, and 2) not making it easy to download the ruleset associated with each capa release.

In the short term, you'll want to download the rules from tag v3.2.0, like here: https://github.com/mandiant/capa-rules/tree/v3.2.0

Some things I think we should consider @mr-tz @mike-hunhoff @Ana06:

• add releases with zip files to capa-rules
• update capa documentation to explain versioning and link to appropriate release
• update capa-rules documentation right at the top

I wonder if we should also add a utility/mode to capa that can download the most up-to-date rules for a given release. It might autodetect when rules are not present locally and prompt the user to fetch the ruleset. Maybe also to update the rules? It's quite a bit more code, but the installation and configuration is rather confusion.

[williballenthin]

@adamprescott91 if you're ok with it, as we make these changes, we'll ask you to review to get your input and feedback. Obviously we have a poor perspective on what its like to get up and running the first time.

[mr-tz]

I think all of the mentioned points are good ideas.

Not sure, we really need an auto-update feature, but adding more details on failing to load rules and some common gotchas, like the one here, would be helpful.

[williballenthin]

two issues about this in two days: perhaps we should revert main and put our changes in a develop branch until our documentation is straightened out.

[mr-tz]

or do a beta release, whatever is easier?

[adamprescott91]

happy to test, but I may be infrequently available so I wouldn't wait on me for anything 😬

[williballenthin]

i've reverted the breaking changes in master and the rules now work with capa v3.2.0:

i've reverted the reverts in https://github.com/mandiant/capa-rules/tree/v4-dev. we should merge this branch into master when we go to release v4. further discussion in #556. updated capa documentation in mandiant/capa#1006

[williballenthin]

thanks for your patience @adamprescott91 and for reporting the issue!