feat(headless): add cdp-endpoint option

[dwisiswant0]

Proposed changes

Close #5692

How has this been tested?

# tty1
/path/to/chrome --headless --remote-debugging-port=9222 --ignore-certificate-errors --ignore-ssl-errors

# tty2
go run cmd/nuclei/main.go -headless -t headless-template-1.yaml -u http://scanme.sh -cdp-endpoint "CHROME_DEVTOOLS_ENDPOINT_URL"

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[dwisiswant0]

But I’m unable to connect to the CDP URL provided by finic:

[FTL] Could not create runner: websocket bad handshake: 400 Bad Request. Failed to open a WebSocket connection: invalid Sec-WebSocket-Key header: nil; Incorrect padding.

Not sure if this is an issue with go-rod or with finic. It needs further investigation, or maybe even a hacky workaround. I’ll try testing it via Playwright - if the issue can’t be reproduced there, I’ll go ahead and raise it as an issue in go-rod.

[dwisiswant0]

--flag=true eq to --flag=1 eq to --flag

[dwisiswant0]

Works with browserless.

Seems like the issue is within finic.

[ehsandeep]

$ ./nuclei -u http://scanme.sh -cdp-endpoint "ws://localhost:3000" -headless -pt headless

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.3.5

		projectdiscovery.io

[INF] Current nuclei version: v3.3.5 (latest)
[INF] Current nuclei-templates version: v10.0.3 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 116
[INF] Templates loaded for current scan: 18
[INF] Executing 18 signed templates from projectdiscovery/nuclei-templates
[INF] Targets loaded for current scan: 1
[extract-urls] [headless] [info] http://scanme.sh [""]
[piratebay] [headless] [info] https://thepiratebay.org/search.php?q=user:{{user}}
[INF] Screenshot successfully saved at /Users/geekboy/Github/nuclei/screenshots/http:__scanme.sh.png
panic: invalid character 'T' looking for beginning of value

goroutine 72 [running]:
github.com/go-rod/rod/lib/utils.glob..func2({0x103f959c0?, 0x140181cc570?})
	/Users/geekboy/go/pkg/mod/github.com/go-rod/[email protected]/lib/utils/utils.go:68 +0x24
github.com/go-rod/rod/lib/utils.E(...)
	/Users/geekboy/go/pkg/mod/github.com/go-rod/[email protected]/lib/utils/utils.go:74
github.com/go-rod/rod/lib/cdp.(*Client).consumeMessages(0x1400043f680)
	/Users/geekboy/go/pkg/mod/github.com/go-rod/[email protected]/lib/cdp/client.go:148 +0x104
created by github.com/go-rod/rod/lib/cdp.(*Client).Start in goroutine 1
	/Users/geekboy/go/pkg/mod/github.com/go-rod/[email protected]/lib/cdp/client.go:76 +0x88

[dwisiswant0]

@ehsandeep - If we look at the trace, it doesn't seem related to our source code - the underlying issue is in go-rod. Also, this panic wouldn't happen if we were using the system CDP.

$ go run cmd/nuclei/main.go -u http://scanme.sh -cdp-endpoint "ws://*********:9222/devtools/browser/************************************" -headless -pt headless

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.3.5

		projectdiscovery.io

[INF] Current nuclei version: v3.3.5 (latest)
[INF] Current nuclei-templates version: v10.0.3 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 116
[INF] Templates loaded for current scan: 18
[INF] Executing 18 signed templates from projectdiscovery/nuclei-templates
[INF] Targets loaded for current scan: 1
[piratebay] [headless] [info] https://thepiratebay.org/search.php?q=user:{{user}}
[extract-urls] [headless] [info] http://scanme.sh [""]

[dwisiswant0]

We either need to tweak the Browserless container or there's something on our end that needs to be handled - but we need to figure out the root cause (template) that's triggering that panic.