Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add project discovery Nuclei boefje #518

Merged
merged 12 commits into from
Mar 30, 2023
Merged

add project discovery Nuclei boefje #518

merged 12 commits into from
Mar 30, 2023

Conversation

RiieCco
Copy link
Contributor

@RiieCco RiieCco commented Mar 17, 2023

Changes

This commit adds the Project discovery Nuclei scan as a boefje
More info about the scanner

Issue ticket number and link

Please paste a link to the issue on the project board here. Alternatively, if there was no submitted issue prior to this PR, you may add this PR to the project board directly.

Proof

openkat1

openkat2

openkat3

Extra instructions for others

This section may be skipped or omitted. Uncomment and answer the below questions if relevant.

Checklist for author(s):

  • All the commits in this PR are properly PGP-signed and verified;
  • This PR comes from a feature or hotfix branch, in line with our git branching strategy;
  • This PR is "bite-sized" and only focuses on a single issue, problem, or feature;
  • I am not reinventing the wheel: there is no high-quality library that already has this feature;
  • I have changed the example .env files if I added, removed, or changed any config options, and I have informed others that they need to modify their .env files if required;
  • I have performed a self-review of my own code;
  • I have commented my code, particularly in hard-to-understand areas;
  • I have made corresponding changes to the documentation, if necessary;
  • I have written unit, integration, and end-to-end tests for the change that I made;

If a non-trivial PR:

  • This PR is part of a milestone and has appropriate labels;
  • This PR is properly linked to the project board (either directly or via an issue);
  • I have added screenshots or some other proof that my code does what it is supposed to do;
## Checklist for functional reviewer(s):
- [ ] If a non-trivial PR: This PR is properly linked to an issue on the project board;
- [ ] I have checked out this branch, and successfully ran `make kat`;
- [ ] I have ran `make test-rf` and all end-to-end Robot Framework tests pass;
- [ ] I confirmed that the PR's advertised `feature` or `hotfix` works as intended;
- [ ] I confirmed that there are no unintended functional regressions in this branch;

### What works:
* _bullet point + screenshot (if useful) per tested functionality_

### What doesn't work:
* _bullet point + screenshot (if useful) per tested functionality_

### Bug or feature?:
* _bullet point + screenshot (if useful) if it is unclear whether something is a bug or an intended feature._
## Checklist for code reviewer(s):
- [ ] The code passes the CI tests and linters;
- [ ] The code does not bypass authentication or security mechanisms;
- [ ] The code does not introduce any dependency on a library that has not been properly vetted;
- [ ] The code does not violate Model-View-Template and our other architectural principles;
- [ ] The code contains docstrings, comments, and documentation where needed;
- [ ] The code prioritizes readability over performance where appropriate;
- [ ] The code conforms to our agreed coding standards.

Signed-off-by: Riccardo ten Cate <[email protected]>
@RiieCco RiieCco requested a review from a team as a code owner March 17, 2023 12:16
@CLAassistant
Copy link

CLAassistant commented Mar 17, 2023

CLA assistant check
All committers have signed the CLA.

@underdarknl
Copy link
Contributor

awesome, we'll have a look asap.

Copy link
Contributor

@ammar92 ammar92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice addition! I still have to test this out, but I do already have a few small remarks to improve it even more 👍

boefjes/boefjes/plugins/kat_nuclei/boefje.json Outdated Show resolved Hide resolved
boefjes/boefjes/plugins/kat_nuclei/normalizer.json Outdated Show resolved Hide resolved
boefjes/boefjes/plugins/kat_nuclei/normalize.py Outdated Show resolved Hide resolved
@ammar92
Copy link
Contributor

ammar92 commented Mar 23, 2023

Tested and seems to work for me!

@tobiasBDO
Copy link
Contributor

I see this boefje currently ingests the Hostname, which gets used as a URL in this boefje. What about the HostnameHTTPURL? because this could be more customizable due to software running on different ports than default 80 and 443.

@RiieCco
Copy link
Contributor Author

RiieCco commented Mar 24, 2023

I see this boefje currently ingests the Hostname, which gets used as a URL in this boefje. What about the HostnameHTTPURL? because this could be more customizable due to software running on different ports than default 80 and 443.

That is achieved by making just a change in the "boefje.json" right?

@ammar92
Copy link
Contributor

ammar92 commented Mar 24, 2023

Indeed, and in the boefje itself since the input is different because Hostname and HostnameHTTPURL are different models

@RiieCco
Copy link
Contributor Author

RiieCco commented Mar 24, 2023

@ammar92, can you make a suggestion for me as to how that should look like? :-)
I am not sure what/where to look for exactly.

@RiieCco RiieCco requested a review from ammar92 March 27, 2023 09:35
@ammar92
Copy link
Contributor

ammar92 commented Mar 28, 2023

@ammar92, can you make a suggestion for me as to how that should look like? :-)
I am not sure what/where to look for exactly.

So in general it would look like this:

  • Update the boefje.json to consume both Hostname and HostnameHTTPURL
  • In the boefje code (main.py) check wether you're dealing with a hostname or HostnameHTTPURL
    • If it's a HostnameHTTPURL, you can construct a URL from that using the netloc.name and port properties
    • If it's a Hostname, you can construct a URL from just the name property (meaning default http scheme and port 80)
    • To distinguish between the two input types, you could check for the object_type in boefje_meta.arguments["input"]["object_type"], this should either be Hostname or HostnameHTTPURL

# if the input object is HostnameHTTPURL then the hostname is located in netloc
if "netloc" in input and "name" in input["netloc"]:
netloc_name = input["netloc"]["name"]
port = input.get("port")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could also return a None, which makes the formatted string look like <hostname>:None. It can be avoided by giving a default port, e.g. port = input.get("port", 80)

But since the port property is required and therefore guaranteed to exist, you can simply access that element directly: port = input["port"]

Suggested change
port = input.get("port")
port = input["port"]

Copy link
Contributor Author

@RiieCco RiieCco Mar 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ammar92 Woeps i am not sure what i just did exactly haha, sorry about that!

ammar92
ammar92 previously approved these changes Mar 28, 2023
Copy link
Contributor

@ammar92 ammar92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! One small suggestion left, but consider it approved 👍

@dekkers dekkers merged commit f890d07 into minvws:main Mar 30, 2023
jpbruinsslot added a commit that referenced this pull request Mar 30, 2023
* main:
  Add sudo in Debian install manual (#153)
  Fix report translation (#609)
  Feature/add organization dashboard (#481)
  [CRT_SH] 🐛 don't treat invalid certificates from 3rd party as active (#541)
  Use PostgreSQL 15 in a single container (#546)
  [CRT_SH] 🐛 don't treat invalid certificates from 3rd party as active (#610)
  Update mula design documentation (#478)
  pass on underscore domains (#550)
  Feature: Add Subfinder Boefje (#516)
  add project discovery Nuclei boefje (#518)
  Remove loop from get random objects (#558)
jpbruinsslot added a commit that referenced this pull request Mar 30, 2023
* main:
  Add sudo in Debian install manual (#153)
  Fix report translation (#609)
  Feature/add organization dashboard (#481)
  [CRT_SH] 🐛 don't treat invalid certificates from 3rd party as active (#541)
  Use PostgreSQL 15 in a single container (#546)
  [CRT_SH] 🐛 don't treat invalid certificates from 3rd party as active (#610)
  Update mula design documentation (#478)
  pass on underscore domains (#550)
  Feature: Add Subfinder Boefje (#516)
  add project discovery Nuclei boefje (#518)
  Remove loop from get random objects (#558)
  Bump urllib3 from 1.26.14 to 1.26.15 (#576)
  Bump black from 22.3.0 to 23.1.0 (#578)
  Bump iniconfig from 1.1.1 to 2.0.0 (#579)
  Add indices for Bytes (#600)
  refactor(organization lists): unify access to organizations by user (#528)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants