Returned status: Forbidden #56

kristianrl · 2019-01-17T08:17:14Z

I received this error when running the plugin, after clicking OK in the settings prompt for the plugin. I haven't previously used the plugin, so I do not know if this is a temporary issue, but https://haveibeenpwned.com/ is working. I believe it could be an HTTP 403.

Let me know if I can provide more relevant information.

SigHunter · 2019-01-17T08:32:17Z

happens to me aswell, tried it for the first time today because of the "Collection #1" list

NathanGloyn · 2019-01-17T08:44:32Z

I fired up fiddler to see what the response was and it appears that the plugin has breached the acceptable use policy (html returned below) but it isn't apparent if this is a rate limiting issue or if it is too many requests from single IP

<head>
<title>Request Blocked</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<h1>You have been blocked from accessing this resource on Have I Been Pwned.</h1>

<p>This may be due to violating one or more of <a href="https://haveibeenpwned.com/API/v2#AcceptableUse">the acceptable use terms of the API</a>.</p>

<p>It may also be due to your traffic patterns being similar to other users who may have violated the acceptable use terms.</p>

<p>Tips to avoid requests being blocked include:</p>
<ol>
<li>Stick well within the published rate limit</li>
<li>Don't distribute requests over multiple IP addresses in an attempt to circumvent the rate limit</li>
<li>Only query the email addresses of people who have a reasonable expectation that you should do so</li>
<li>Avoid prolonged querying of the API over an extended period of time</li>
</ol>

<div class="cf-error-details cf-error-1020">
  <h1>Access denied</h1>
  <p>This website is using a security service to protect itself from online attacks.</p>
  <ul class="cferror_details">
    <li>Ray ID: 49a780168d393542</li>
    <li>Timestamp: 2019-01-17 08:37:31 UTC</li>
    <li>Your IP address: 79.68.4.48</li>
    <li class="XXX_no_wrap_overflow_hidden">Requested URL: haveibeenpwned.com/api/v2/breachedaccount/Nathan </li>
    <li>Error reference number: 1020</li>
    <li>Server ID: FL_21F109</li>
    <li>User-Agent: KeePass HIBP Checker/2.40.0.0</li>
  </ul>
</div>

</body>
</html>```

NathanGloyn · 2019-01-17T08:59:24Z

Having checked the Pwnd Password docs here I don't believe its anything to do with rate limiting as that should return a 429 but instead I'm seeing a 403

andrew-schofield · 2019-01-17T11:23:19Z

The plugin should always be rate limited in its requests to the API, so I wonder if it had accidentally triggered the "multiple IP addresses" check because obviously there will be a lot of requests from different IPs.
The plugin sends a versioned user agent string too, so I might reach out to Troy to find out what's going wrong.

vdenisov · 2019-01-17T17:49:06Z

Could this be due to the sheer number of checks from a single IP? When I tried running it, it went through ~half my username list, then started getting an error for every entry, even if I waited for several seconds between attempts. Also - it's impossible to gracefully cancel plugin execution is this case, had to terminate KeePass process to abort.

vdenisov · 2019-01-17T17:58:39Z

One more thing I'd noticed: if progress indicator is to be believed, it definitely checks faster then once every 1.6s (I'd say, at least couple entries per second).

kapitanluffy · 2019-01-17T18:37:13Z

I was accessing the API manually earlier and got the same forbidden page. I think there is something wrong with hibp's API for now. Accessing other endpoints (pastes. pwned-pws) are accessible though.

I also tweeted haveibeenpwnd about it to make sure.

janKir · 2019-01-17T23:31:03Z

Although the problem might be API related, there should be the possiblity to cancel the process. These error messages pop up over and over again ...

Sanaki · 2019-01-18T03:17:54Z

Just ran site/service and username checks and had no issues myself. Either it's a temporary hiccup that resolved or my username count is severely low compared to yours.

marcelser · 2019-01-18T06:18:24Z

Works fine for me

peterpan123123 · 2019-01-18T08:01:49Z

Yep, works again since today.

NathanGloyn · 2019-01-18T08:20:27Z

I can reproduce the error when checking site or password by selecting the option "Check all supported breaches" without this option checked it runs successfully.

When trying to run the check via username even without the "Check all supported breaches" I'm getting the forbidden error but it appears that it is trying to get breaches so guessing its related.

marcofranssen · 2019-01-18T08:43:36Z

Same here Unable to check haveibeenpwned.com (return Status: Forbidden) See https://httpstatuses.com/403. Should this plugin be updated with credentials to authenticate against haveibeenpwned.com?

Even more anoying is that you have to dismiss the dialog for every single entry in your keepass file. That kept me busy for 15 minutes. :(

internationils · 2019-01-18T10:06:45Z

I'm stuck on this as well, under Ubuntu (see #58 ) ...
Hint: you can close the progress dialog, and close KeePass to stop the dialog flood...

p3l4h0 · 2019-01-18T11:55:12Z

Same Error only with usernames
@NathanGloyn

Having checked the Pwnd Password docs here I don't believe its anything to do with rate limiting as that should return a 429 but instead I'm seeing a 403

I'm not sure where to get the Error code 403 but this is about: no user agent:
https://haveibeenpwned.com/API/v2#UserAgent

andrew-schofield · 2019-01-18T12:23:21Z

Same Error only with usernames
@NathanGloyn

Having checked the Pwnd Password docs here I don't believe its anything to do with rate limiting as that should return a 429 but instead I'm seeing a 403

I'm not sure where to get the Error code 403 but this is about: no user agent:
https://haveibeenpwned.com/API/v2#UserAgent

That shouldn't be the issue, as the plugin always sets the user agent appropriately

keepass2-haveibeenpwned/HaveIBeenPwned/HaveIBeenPwnedExt.cs

Line 46 in d78ade5

    
           client.DefaultRequestHeaders.UserAgent.ParseAdd(string.Format("KeePass HIBP Checker/{0}", Application.ProductVersion));

However if this is being stripped from the request for whatever reason, there's not much I can do.

strayge · 2019-01-18T14:08:13Z

However if this is being stripped from the request for whatever reason, there's not much I can do.

It's not stripped. Proper user-agent showed in cloudflare response.

Also same error happens for me by test link (from api page) opened in browser.

andrew-schofield · 2019-01-18T14:16:27Z

However if this is being stripped from the request for whatever reason, there's not much I can do.

It's not stripped. Proper user-agent showed in cloudflare response.

Also same error happens for me by test link (from api page) opened in browser.

Yeah, the direct link returns a 403 for me too.

duichwer · 2019-01-18T21:40:24Z

Requests to the breaches and pastes APIs are limited to one per every 1500 milliseconds each from any given IP address (an address may request both APIs within this period).

As mentioned on the Api-Website

And you have already added Delay but only for usernameChecks and not for the other checks.
Maybe this causes this issue.

keepass2-haveibeenpwned/HaveIBeenPwned/BreachCheckers/HaveIBeenPwnedUsername/HaveIBeenPwnedUsernameChecker.cs

Line 102 in d78ade5

await Task.Delay(1600);

andrew-schofield · 2019-01-18T22:18:40Z

Requests to the breaches and pastes APIs are limited to one per every 1500 milliseconds each from any given IP address (an address may request both APIs within this period).

As mentioned on the Api-Website

And you have already added Delay but only for usernameChecks and not for the other checks.
Maybe this causes this issue.

keepass2-haveibeenpwned/HaveIBeenPwned/BreachCheckers/HaveIBeenPwnedUsername/HaveIBeenPwnedUsernameChecker.cs

Line 102 in d78ade5

await Task.Delay(1600);

Unless Troy has changed something and not updated the docs, I don't think this is the issue.

The API page specifically says

There is no rate limit on the Pwned Passwords API.

Additionally the site/service check only does 1 call to retrieve the entire list of breaches, and then compares entries locally.

NathanGloyn · 2019-01-22T10:03:58Z

I have installed latest release today and can confirm that all of the searches are working correctly (regardless of if "Check all breaches" is checked or unchecked 👍

p3l4h0 · 2019-01-22T10:42:32Z

Unfortunately not for me:
Update reinstalled installed today, and with both ways, checked and uncheked

andrew-schofield · 2019-02-01T22:51:13Z

Is this still an issue for people?

BenTen · 2019-02-02T02:27:42Z

I just check with Keepass 2.41 and the plugin 1.3.1, but the issue is not resolved ("Returned status: Forbidden")

andrew-schofield · 2019-02-02T21:38:38Z

Right, I've fired off an email to Troy about this. Hopefully he can offer some insight into why this is happening.

modelrockettier · 2019-02-07T03:00:50Z

I just got ran into this last night, here's what shows up when I manually attempt to query the API (example URL)

You have been blocked from accessing this resource on Have I Been Pwned

This may be due to violating one or more of the acceptable use terms of the API or for not complying with the API specifications. It may also be due to your traffic patterns being similar to other users who may have violated the acceptable use terms.

Tips to avoid requests being blocked include:

Stick well within the published rate limit

Don't distribute requests over multiple IP addresses in an attempt to circumvent the rate limit

Only query the email addresses of people who have a reasonable expectation that you should do so

Avoid prolonged querying of the API over an extended period of time

Clearly identify your app in the user agent string per the API docs.

If you believe your request meets these requirements and was still blocked, please send this entire response body along with any communication you send regarding the error.

Access denied

This website is using a security service to protect itself from online attacks.

Ray ID: 4a5297ee9b01c99f

Timestamp: 2019-02-07 02:58:12 UTC

Your IP address: 75.172.3.60

Requested URL: haveibeenpwned.com/api/v2/breachedaccount/[email protected]

Error reference number: 1020

Server ID: FL_28F150

User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

modelrockettier · 2019-02-08T03:31:52Z

So I've done a bit more investigation in my case, and the plugin is still popping up an error message saying Unable to check haveibeenpwned.com (returned Status: Forbidden).

All I have to do to reproduce it is check for breaches based on username and uncheck all boxes.

The error page from my previous comment seems to have been caused by the haveibeenpwned API rejecting requests from web browser user agents (as is documented in the API docs).

The above API URL does work over curl though (with the user agent KeePass HIBP Checker/1.3.1).

andrew-schofield · 2019-02-08T09:30:01Z

If you can reliably reproduce the error from within keepass, can you try capturing the error response by using fiddler (https://www.telerik.com/fiddler) or a similar tool?

Troy needs to see the exact response returned by cloudflare in order to debug this.

You're right that using a browser is not a valid test, as cloudflare rejects this based on the user agent, but it is interesting that the request fails from within keepass, but works via curl.

I haven't been able to reproduce the error on any of my systems (using at least 3 different IP addresses from different locations), either using the plugin, or with postman etc.

p3l4h0 · 2019-02-08T10:34:40Z

Downloaded and installed and "learned" Fiddler.
Question

GET https://haveibeenpwned.com/api/v2/breachedaccount/****** HTTP/1.1
User-Agent: KeePass HIBP Checker/2.41.0.0
Accept: application/json
Host: haveibeenpwned.com
Cookie: __cfduid=dd4a76c0ce61505da0a219e7c1d168ae01549621013
Connection: Keep-Alive

First answer:

HTTP/1.1 403 Forbidden
Date: Fri, 08 Feb 2019 10:20:12 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Cache-Control: max-age=15
Expires: Fri, 08 Feb 2019 10:20:27 GMT
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Content-Type-Options: nosniff
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 4a5d5cc34e15c4b0-DUS
Content-Length: 1951

<!DOCTYPE html>
<head>
<title>Request Blocked</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<h1>You have been blocked from accessing this resource on Have I Been Pwned</h1>

<p>This may be due to violating one or more of <a href="https://haveibeenpwned.com/API/v2#AcceptableUse">the acceptable use terms of the API</a> or for not complying with <a href="https://haveibeenpwned.com/API/v2">the API specifications</a>. It may also be due to your traffic patterns being similar to other users who may have violated the acceptable use terms.</p>

<p>Tips to avoid requests being blocked include:</p>
<ol>
<li>Stick well within the published rate limit</li>
<li>Don't distribute requests over multiple IP addresses in an attempt to circumvent the rate limit</li>
<li>Only query the email addresses of people who have a reasonable expectation that you should do so</li>
<li>Avoid prolonged querying of the API over an extended period of time</li>
<li>Clearly identify your app in the user agent string <a href="https://haveibeenpwned.com/API/v2#UserAgent">per the API docs</a>.</li>
</ol>
<p>If you believe your request meets these requirements and was still blocked, please send this entire response body along with any communication you send regarding the error.</p>
<div class="cf-error-details cf-error-1020">
  <h1>Access denied</h1>
  <p>This website is using a security service to protect itself from online attacks.</p>
  <ul class="cferror_details">
    <li>Ray ID: 4a5d5cc34e15c4b0</li>
    <li>Timestamp: 2019-02-08 10:20:12 UTC</li>
    <li>Your IP address: *********</li>
    <li class="XXX_no_wrap_overflow_hidden">Requested URL: haveibeenpwned.com/api/v2/breachedaccount/****** </li>
    <li>Error reference number: 1020</li>
    <li>Server ID: FL_48F23</li>
    <li>User-Agent: KeePass HIBP Checker/2.41.0.0</li>
  </ul>
</div>

</body>
</html>

Reconnect to provider some times to get new Ip adress -> Same answer

Start VPN it works

andrew-schofield · 2019-02-08T10:46:12Z

Thanks, I've forwarded the details on to Troy. It does look like cloudflare using an over enthusiastic IP range blocker that is causing this.

modelrockettier · 2019-02-09T22:43:24Z

So for me, this turned out to be an IP blocking issue.

I had previously been doing some testing with IE over a VPN and forgot to reset its proxy settings, and it looks like Keepass picks up the IE proxy settings, so all of the plugin's traffic was going over the VPN whereas curl, etc. did not.

Trying the same curl commands on the other end of the VPN failed, and after clearing the IE proxy settings, the keepass plugin started working again.

Braintelligence · 2019-02-23T13:25:24Z

Same problem. Fresh install of keepass and this plugin. No proxy involved. No other users on this IP even looking at haveibeenpwned.

mykey · 2019-03-01T12:08:18Z

I got the same error here when letting the plugin search for usernames and ticking the box
"Check all breaches". I don't get the error when I search for passwords. I have fiddler and burp on the box and might set up a proxy later to help debug (but helas today and this weekend will not be that time. Way too much work).

andrew-schofield · 2019-03-01T15:22:17Z

OK everyone, I've chatted to Troy about this, and he says the only way for him to get to the bottom of this is for him to get copies of the error response returned by cloudflare, including the IP address (if this is missing, it's useless to him).

Since this isn't easily accessible when using KeePass (unless you're willing to run Fiddler, or another inspecting proxy), I'd like to have a show of hands of who would be willing to test a version of keepass that specifically saves these cloudflare error messages so we can send them to Troy.

Please 👍 this message if you're happy to take part so I know it's worth my time creating a debug build.

SoyRA · 2019-03-02T16:59:11Z

(...) is for him to get copies of the error response returned by cloudflare, including the IP address (...)

You mean this? Do I share the result here or what? 😅
* If you're saying you'll give us a version that will send the data automatically...I accept.

I've KeePass and Plugin updated, and I always have the same problem. :c!

andrew-schofield · 2019-03-02T17:50:51Z

@SoyRA No, that link will always fail because of the browser user-agent. We need the response from a valid request from KeePass.
I'll work on getting a debug version created.

mykey · 2019-03-03T21:33:22Z

Thanks for your work. Happy to help out if needed.

Firehawke · 2019-03-09T21:00:24Z

This is hitting me as well, and I'd be willing to run a beta plugin for testing purposes if it'll help solve this once and for all.

andrew-schofield · 2019-03-10T22:09:56Z

I've attached a "rough-and-ready" debug version of the plugin which will store the cloudflare error responses returned from HIBP.com

This will dump a txt file into %TEMP%\KeePassHIBP and you can either attach the responses here, or email them to [email protected]
I'll collate the responses and send them off to Troy so he can take a look.

Debug download: HaveIBeenPwned.zip

BenTen · 2019-03-10T22:42:48Z

I sent an email with debug file.

vdenisov · 2019-03-11T07:04:19Z

Here's two of my error response. Received when trying to run username check. Both site and password checks have worked.

srihwf1e.dj1.txt
bcqq0wzq.k3x.txt

mykey · 2019-03-18T22:09:30Z

I've ran the beta plugin and also sent you an email with a number of debug files.

As stated previously I can only get errors by searching on username.

andrew-schofield · 2019-09-22T21:03:34Z

I believe this should be fixed now that HIPB API v3 has been released, however I'd be interested in knowing if people are still seeing this issue with >= v1.3.4 of the plugin.

YogSottot · 2019-09-23T05:23:58Z

Tested with 1.3.4
This problem has been resolved.

SoyRA · 2019-09-23T15:01:26Z

Let's see, everything works. The only "problem" is trying to use (...) based on username because it asks me for the API Key...But taking that out, everything's fine. :P
* KeePass and this Plugins updated to the latest version.

Sanaki mentioned this issue Jan 18, 2019

Breach date always current date #44

Closed

internationils mentioned this issue Jan 18, 2019

Stuck on "Getting HaveIBeenPwned breach list..." #58

Closed

strayge mentioned this issue Jan 18, 2019

allow break usernames check on error #62

Merged

duichwer mentioned this issue Jan 18, 2019

Add to List on HIBP #66

Open

thewhiteh4t mentioned this issue Apr 28, 2019

Every Time I tried thewhiteh4t/pwnedOrNot#22

Closed

andrew-schofield added the bug label Sep 22, 2019

Returned status: Forbidden #56

Returned status: Forbidden #56

Comments

kristianrl commented Jan 17, 2019

SigHunter commented Jan 17, 2019

NathanGloyn commented Jan 17, 2019

NathanGloyn commented Jan 17, 2019

andrew-schofield commented Jan 17, 2019

vdenisov commented Jan 17, 2019

vdenisov commented Jan 17, 2019

kapitanluffy commented Jan 17, 2019 • edited Loading

janKir commented Jan 17, 2019 • edited Loading

Sanaki commented Jan 18, 2019

marcelser commented Jan 18, 2019

peterpan123123 commented Jan 18, 2019

NathanGloyn commented Jan 18, 2019

marcofranssen commented Jan 18, 2019 • edited Loading

internationils commented Jan 18, 2019

p3l4h0 commented Jan 18, 2019

andrew-schofield commented Jan 18, 2019 • edited Loading

strayge commented Jan 18, 2019

andrew-schofield commented Jan 18, 2019

duichwer commented Jan 18, 2019

andrew-schofield commented Jan 18, 2019

NathanGloyn commented Jan 22, 2019

p3l4h0 commented Jan 22, 2019

andrew-schofield commented Feb 1, 2019

BenTen commented Feb 2, 2019

andrew-schofield commented Feb 2, 2019

modelrockettier commented Feb 7, 2019

You have been blocked from accessing this resource on Have I Been Pwned

Access denied

modelrockettier commented Feb 8, 2019

andrew-schofield commented Feb 8, 2019

p3l4h0 commented Feb 8, 2019 • edited Loading

andrew-schofield commented Feb 8, 2019

modelrockettier commented Feb 9, 2019

Braintelligence commented Feb 23, 2019

mykey commented Mar 1, 2019 • edited Loading

andrew-schofield commented Mar 1, 2019

SoyRA commented Mar 2, 2019 • edited Loading

andrew-schofield commented Mar 2, 2019

mykey commented Mar 3, 2019

Firehawke commented Mar 9, 2019

andrew-schofield commented Mar 10, 2019

BenTen commented Mar 10, 2019

vdenisov commented Mar 11, 2019

mykey commented Mar 18, 2019

andrew-schofield commented Sep 22, 2019

YogSottot commented Sep 23, 2019

SoyRA commented Sep 23, 2019

kapitanluffy commented Jan 17, 2019 •

edited

Loading

janKir commented Jan 17, 2019 •

edited

Loading

marcofranssen commented Jan 18, 2019 •

edited

Loading

andrew-schofield commented Jan 18, 2019 •

edited

Loading

p3l4h0 commented Feb 8, 2019 •

edited

Loading

mykey commented Mar 1, 2019 •

edited

Loading

SoyRA commented Mar 2, 2019 •

edited

Loading