High Bandwidth Usage

[captn3m0]

We crossed our 1TB limit on the OSS Plan for October-November billing period (by 300GB), resulting in a 55x3 = $165 charge. Netlify sends me a warning when we cross 75% and 90% bandwidth usage every month, but this was the first time where we blew past.

While our impressions have only gone up by 10-20%, our bandwidth usage has more than doubled over the same period:

The bandwidth usage is only on the primary domain. We haven't had any recent changes to configuration. Everything is static content, which ought to be well-cached, but Netlify clears the cache on each new deploy, but our number of builds/deploys haven't changed at all:

In the absence of logs, it is hard to figure out if we can actually do anything about this bandwidth bump.

I've opened a Support ticket with Netlify for now (319294) to see if we can get a Bandwidth upgrade on our OSS Plan.

[captn3m0]

This is the mail I sent to Netlify Support:

Our Team: endoflife-date is on the OSS Plan, and we are a fairly popular website, with an API. All our content is static, and very well cached.

Yet, our bandwidth usage has been increasing with our popularity, and we crossed the 1TB bandwidth limit this month to 1.3TB. Seeing our current numbers (Published at #382), and the growth from the last few months, I do not expect to see the bandwidth usage going down.

We have an overage of 3*100 GB resulting in a $165 charge. I specifically want to check for two things:

1. Is it possible to upgrade the bandwidth on our OSS Plan to something around 1.5-2TB? While we do have some donations I can put to use towards paying for bandwidth, as a non-profit open-source project, paying $200-500 every month is going to be unfeasible. I'm gonna have to either consider switching away from Netlify or setting up something like Cloudflare on front.
2. Separately, start a conversation to upgrade us to a minimal (paid) Enterprise plan. I'd attempted this earlier, but both the support team and sales teams weren't helpful since we were on the OSS Plan. We are happy to pay for the things we care about (which is just bandwidth, and Log Drains), while not having to worry about any of the remaining Enterprise features, which aren't really helpful for as an open-source project.

[captn3m0]

Netlify has forgiven such usage bumps for other projects, so I'm hopeful. But I don't see our usage going down (we typically see lower traffic in December) long-term.

For now, I've updated the Credit Card on Netlify, and if the payment goes through, I'll update it on Open Collective. Might be a good time to add more years to our domain renewal.

[captn3m0]

Got a reply from Netlify:

I want to examine your traffic first to make sure everything looks as expected. These are your top 5 User Agents in the past 30 days to the sites in your account:

User Agent	Bytes	GiB
Mozilla/5.0 (Windows NT; Windows NT 10.0; fr-CA) WindowsPowerShell/5.1.19041.5007	416765664017	388.14
Mozilla/5.0 (Windows NT; Windows NT 10.0; fr-CA) WindowsPowerShell/5.1.22621.4249	150215646154	139.9
Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.19041.5007	114158970694	106.32
Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.22621.4249	46595596161	43.4
-	40798240139	38

I'm assuming you expect the WindowsPowerShell users, but the null user agent has consumed 38GB in just one month! These are typically crawlers, so we recommend an Edge Function to block their access. I'll put an example here of how to create that:

---

Edge Function for a null user agent:

export default async (request) => {
  const userAgent = request.headers.get('user-agent') || '';

if (!userAgent.length) {
    return new Response('Access Denied: User-Agent is required', {
      status: 403,
      headers: {
        'Content-Type': 'text/plain'
      }
    });
  }

  return;
};

and you want to specify it on all paths:
export const config = {path: '/*'}

---

To address your questions regarding custom plans, we unfortunately do not have a custom plan for adding on bandwidth, as that is only handled from bandwidth packages. I am happy to forward your contact info onto our Enterprise sales team, but I am afraid your budget might not be high enough to get a custom contract. Please let me know if you'd like me to set up a conversation with them in order to see what the minimum would be for a contract.

[captn3m0]

I'll look at setting up the Edge Rule, but with our current bandwidth usage, the edge function itself might bump our costs. Can you give an estimate - since we don't know our number of requests either.

The null-user-agent being 38GB actually makes sense, since we provide an API I'd expect 3% of users to not provide a user-agent while making a call.

What I'm curious about is why the powershell usage is so high. Since all of our content is static, it should be well-cached by Netlify, and not result in a bandwidth charge.

Maybe we should be setting cache-headers manually since the default headers that Netlify sets seem to be optimized for browser cache via e-tags, which non-browser clients like powershell will just ignore.

[BiNZGi]

Got another mail today from Netlify that we reached 50% of the current allowance in billing cycle from November 12 to December 12.

[captn3m0]

I enabled Netlify Analytics in hopes it might be helpful in debugging the issue, but it is utterly useless:

In particular:

• it doesn't give us user-agent level metrics
• The "top-pages" seems to only include HTML resources, ignoring our API calls.

The 404s for API calls are somewhat helpful, but not worth paying.

[captn3m0]

We've been upgraded to the Netlify Unlimited Open Source Plan!

We've gotten approval to upgrade your account endoflife.date to our Open Source Unlimited plan!

Some folks here at Netlify really appreciate your project, and even use it personally, so when I reached out about this, they advocated for endoflife.date to become fully sponsored. This means you will not have billing for bandwidth on this project. I've performed the upgrade for you, and there is no action to complete on your end.

I've removed the current bill you had for bandwidth, but I see there's a bandwidth package added for this month. Will you please let us know if our system tries to release this as a bill, so we can remove it?

Thanks so much for hosting on Netlify and for sharing your creation with the community! We'll be here if you need anything.

Thank you @netlify 🙌🏼

I'll keep this issue open for a bit (also remove the Analytics Plan).

[BiNZGi]

Wow, thank you @netlify 🙌🏼

[usta]

@captn3m0 do we have any kind of api limitter ? Even this might look solved the bandwith problem we still need to fix underlaying problem. I mean we might have some unwanted guests which are trying to make lots of api calls which are not for getting information ( Sometimes the middleware coders made a mistake that will continuesly try to make same api call instead of using some kind of caching )
So I suggest we at least limit the number of api calls for ip addresses per minute based
What do you think @BiNZGi @marcwrobel @captn3m0 ??

[marcwrobel]

Should we close this issue now that endoflife.date is on the Netlify Unlimited Open Source Plan ?

Note that I also think that some kind of rate-limiting would be nice, if that's easy to set up.