Add support to HEAD requests and Last-Modified response header

[watery]

• Operating System: Docker with node:10.15.3-alpine image
• Node Version: 10.15.3
• NPM Version:
• webpack Version: from react-scripts @ 2.1.8
• webpack-dev-middleware Version: from react-scripts @ 2.1.8

Feature Proposal

Make the dev-server reply to HEAD requests on resources where it already replies to GET requests; and add the Last-Modified response header if it's not already supported.

Feature Use Case

I'm doing some experiments in a custom project that rely on HEAD requests and the Last-Modified response header to detect a (React) app updated deploy. I'm already obtaining some results in a production build served by a full-featured webserver (nginx), and it would be nice to have here aswell, expecially in cases where the browser or page is closed and reopened between source code updates - which if I'm not mistaken, detaches the client code from the dev-server making the former lose the recompiled event that triggers a complete page reload.

The latter is the case with create-react-app applications, that do not provide a debug build for deployment to staging servers (see Support for staging builds #790); thus running a dev-server instance on a staging server is the only option to let testers / non-developers to access the application for review.

I actually noticed that a HEAD request returns 404 whereas a GET on the same resource returns 200, so maybe this is only a matter of supporting HEAD requests too.

I've been directed here from this issue of mine from the dev-server subproject. I think I copied over all the relevant information, but I'm linking it just in case.

[alexander-akait]

I'm already obtaining some results in a production build served by a full-featured webserver (nginx), and it would be nice to have here aswell, expecially in cases where the browser or page is closed and reopened between source code updates - which if I'm not mistaken, detaches the client code from the dev-server making the former lose the recompiled event that triggers a complete page reload.

Last-Modified is hot help here, webpack-dev-server always reload the page when new compilation happens, I think you do not quite understand how it works. What is the real use case for this feature?

[alexander-akait]

Please provide more information

[alexander-akait]

And we already have HEAD support

[watery]

I need some time to elaborate more on my inital explanation; but I'd like to know more about this:

And we already have HEAD support

Good!
So, which version has that supported? In my node_modules I see webpack-dev-server vers. 3.1.14 and webpack-dev-middleware at verss 3.4.0.

[alexander-akait]

You should update webpack-dev-server to the latest version

[watery]

I'm using create-react-app, which in turn depends on react-scripts, I'll check all their versions and see how far I can get.

I'll report back ASAP. Thank you.

[watery]

Ok, I updated the dependencies, ending up with:

• webpack-dev-server @ 3.11.0
• webpack-dev-middleware @ 3.7.2

For simplicity I started the project locally on my computer and with npm start (so without Docker); Node is v10.21.0.

I still get 404 on an existing page:

>curl http://localhost:3000/parameters -X HEAD --head
HTTP/1.1 404 Not Found
X-Powered-By: Express
Content-Security-Policy: default-src 'none'
X-Content-Type-Options: nosniff
Content-Type: text/html; charset=utf-8
Content-Length: 150
Vary: Accept-Encoding
Date: Wed, 28 Oct 2020 21:12:50 GMT
Connection: keep-alive

but GET gives 200:

>curl http://localhost:3000/parameters -X GET --head
HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
Content-Type: text/html; charset=UTF-8
Content-Length: 1808
ETag: W/"710-5eM8S1rSfpKR8my6++WVUxSyIu8"
Vary: Accept-Encoding
Date: Wed, 28 Oct 2020 21:13:00 GMT
Connection: keep-alive

Do I have to enable some option or similar?

[watery]

I tried to look around the whole dependencies, initializations and configurations from react-scripts to webpack-* components.
I didn't see any explicit configuration for the middleware, hence I'd expect the default configuration for the request methods to be active.

I however tried to explicitly add the methods config as stated in the README, I'm sure my changes are being picked up because I had errors when they weren't correctly written.

But I'm still getting the above results, 200 on GET, 404 on HEAD on the same path.

Just as a counter-test, I tried sending PUT and BREAK requests, expecting a 405 Method Not Allowed for both, but instead:

• PUT returns 404
• BREAK gives 400 Bad Request

[watery]

I just noticed that I obtain 200 even when I issue a GET request on non-existing paths;

>curl http://localhost:3000/aaeabfaefa -X GET --head
HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
Content-Type: text/html; charset=UTF-8
Content-Length: 1808
ETag: W/"710-5eM8S1rSfpKR8my6++WVUxSyIu8"
Vary: Accept-Encoding
Date: Thu, 29 Oct 2020 10:09:15 GMT
Connection: keep-alive

[watery]

Thinking twice about the results in my latest post, they look correct to me: the / path is handled by the server, which sends back the javascript files / bundle; but subpaths are effectively handled in-app (by react-router or by a similar routing solution), which is the only one who can tell wheter a subpath really exists or not.

In fact, GET and HEAD to / both do work:

>curl http://localhost:3000/ -X GET --head
HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
Content-Type: text/html; charset=UTF-8
Content-Length: 1808
ETag: W/"710-5eM8S1rSfpKR8my6++WVUxSyIu8"
Vary: Accept-Encoding
Date: Thu, 29 Oct 2020 10:38:33 GMT
Connection: keep-alive

>curl http://localhost:3000/ -X HEAD --head
HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
Content-Type: text/html; charset=UTF-8
Content-Length: 1808
ETag: W/"710-5eM8S1rSfpKR8my6++WVUxSyIu8"
Vary: Accept-Encoding
Date: Thu, 29 Oct 2020 10:38:37 GMT
Connection: keep-alive

So the issue involves subpaths only, it seems, with the server returning 200 to GET requests but 404 to HEAD ones, issued on the same subpath.

[alexander-akait]

Sorry, please open an issue in create-react-app, we support HEAD, please read code

[alexander-akait]

http://localhost:3000/aaeabfaefa is not our url, we can control only weback assets

[watery]

Sure, but isn't webpack who replies to any subpath here, always returning the whole bundle?

I mean, if I open my browser on http://localhost:3000/, http://localhost:3000/home or http://localhost:3000/categories/tools/ or any other address, for the first time ever, isn't webpack that returns all the assets?

That's the reasons I'm using cUrl and the command line to do these checks. From all these requests I always expect the whole js app / bundle to be returned, so no client side library is involved.

Correct me if this is wrong, of course.

[alexander-akait]

I mean, if I open my browser on http://localhost:3000/, http://localhost:3000/home or http://localhost:3000/categories/tools/ or any other address, for the first time ever, isn't webpack that returns all the assets?

No, webpack-dev-middleware in this case respect only for webpack assets

[watery]

No, webpack-dev-middleware in this case respect only for webpack assets

I looked around wepack-* javascript files, to try to track the paths where each request is being handled.
I've added the X-handled-by custom header to the response, with the filename and line number where that custom header is added to the request.

Here's what I found:

Issuing GET and HEAD to the root path /, both are handled by middleware.js and return 200:

>curl http://localhost:3000/ -X GET --head
HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
X-handled-by: middleware.js:106                           <-----
Content-Type: text/html; charset=UTF-8
Content-Length: 1808
ETag: W/"710-5eM8S1rSfpKR8my6++WVUxSyIu8"
Vary: Accept-Encoding
Date: Sat, 31 Oct 2020 12:25:59 GMT
Connection: keep-alive

>curl http://localhost:3000/ -X HEAD --head
HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
X-handled-by: middleware.js:106                           <-----
Content-Type: text/html; charset=UTF-8
Content-Length: 1808
ETag: W/"710-5eM8S1rSfpKR8my6++WVUxSyIu8"
Vary: Accept-Encoding
Date: Sat, 31 Oct 2020 12:26:21 GMT
Connection: keep-alive

The same holds true for javascript files - I guess these are the kind of requests you were referring to when you said:

Sorry, please open an issue in create-react-app, we support HEAD, please read code

>curl http://localhost:3000/static/js/main.chunk.js -X GET --head
HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
X-handled-by: middleware.js:106                           <-----
Content-Type: application/javascript; charset=UTF-8
Content-Length: 1044422
ETag: W/"fefc6-dgCJ/MwUVkDFkGk7AurVq/dvcPE"
Vary: Accept-Encoding
Date: Sat, 31 Oct 2020 12:27:24 GMT
Connection: keep-alive

>curl http://localhost:3000/static/js/main.chunk.js -X HEAD --head
HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
X-handled-by: middleware.js:106                           <-----
Content-Type: application/javascript; charset=UTF-8
Content-Length: 1044422
ETag: W/"fefc6-dgCJ/MwUVkDFkGk7AurVq/dvcPE"
Vary: Accept-Encoding
Date: Sat, 31 Oct 2020 12:27:28 GMT
Connection: keep-alive

Next, I've checked the subpaths, which I expect will return the HTML of the base/main page:

GET requests are still handled by middleware.js

>curl http://localhost:3000/login -X GET --head
HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
X-handled-by: middleware.js:106                           <-----
Content-Type: text/html; charset=UTF-8
Content-Length: 1808
ETag: W/"710-5eM8S1rSfpKR8my6++WVUxSyIu8"
Vary: Accept-Encoding
Date: Sat, 31 Oct 2020 12:26:35 GMT
Connection: keep-alive

And even for requests to non-existent paths - which totally makes sense, because apart for the root path, subpaths only exists within the application, but there's no real HTML file corresponding to them:

>curl http://localhost:3000/asdfao/aidfoasd -X GET --head
HTTP/1.1 200 OK
X-Powered-By: Express
Accept-Ranges: bytes
X-handled-by: middleware.js:106                           <-----
Content-Type: text/html; charset=UTF-8
Content-Length: 1808
ETag: W/"710-5eM8S1rSfpKR8my6++WVUxSyIu8"
Vary: Accept-Encoding
Date: Sat, 31 Oct 2020 12:42:55 GMT
Connection: keep-alive

And all of them return the same ETag and Content-Length as above, I can confirm that the body returned is the same - it's the base HTML page found in /public and returned by GET requests to /:

>curl http://localhost:3000/login -X GET
<!DOCTYPE html>
<html lang="en">
  [...]
  <script src="/static/js/bundle.js"></script><script src="/static/js/0.chunk.js"></script><script src="/static/js/main.chunk.js"></script></body>
</html>

>curl http://localhost:3000/ -X GET
<!DOCTYPE html>
<html lang="en">
  [...]
  <script src="/static/js/bundle.js"></script><script src="/static/js/0.chunk.js"></script><script src="/static/js/main.chunk.js"></script></body>
</html>

But if I issue HEAD requests on those subpaths, differently from HEAD requests to /, these are routed elsewhere, are handled in Server.js (from webpack-dev-server/lib) and they all end in 404:

>curl http://localhost:3000/login -X HEAD --head
HTTP/1.1 404 Not Found
X-Powered-By: Express
X-handled-by: Server.js:430                           <-----
Content-Security-Policy: default-src 'none'
X-Content-Type-Options: nosniff
Content-Type: text/html; charset=utf-8
Content-Length: 145
Vary: Accept-Encoding
Date: Sat, 31 Oct 2020 12:26:39 GMT
Connection: keep-alive

>curl http://localhost:3000/asdfao/aidfoasd -X HEAD --head
HTTP/1.1 404 Not Found
X-Powered-By: Express
X-handled-by: Server.js:430                           <-----
Content-Security-Policy: default-src 'none'
X-Content-Type-Options: nosniff
Content-Type: text/html; charset=utf-8
Content-Length: 155
Vary: Accept-Encoding
Date: Sat, 31 Oct 2020 12:27:03 GMT
Connection: keep-alive

I still have to find where GET and HEAD take different routes to end up in those different replies. Any hint? Should I open an issue on webpack-dev-server now?

[watery]

I went a little further. I added some console.log() here and there and found what follows.

The request:

>curl http://localhost:3000/jhuhiu/ujjj -X GET --head

Arrives to middleware.js, inside Promise() where filename is:

<project-directory>\dist/jhuhiu/ujjj

and then the flow falls to the line:

resolve(goNext());

Afterwards, the Promise() is re-entered, but this time filename is

<project-directory>\dist

and then I see the 200 status code and the other headers shown in the previous post.

If, though, the request is HEAD on the same path the Promise() is still entered twice, but the second time no filename change happens, the flow goes to the resolve(goNext()); line straight away and the response is 404.

[alexander-akait]

What do you want from webpack-dev-middleware, we have gone far from the original request

[watery]

I don't think so, I'm still trying to receive a reply to HEAD requests, which seem to be provided consistently when asking for javascript files, but not for the HTML of the base / index page.

I can't tell whether this belongs to this repository or another of course, I need your help here: you already directed my over here from the issue I opened on webpack-dev-server and maybe this need to be moved elsewhere again.

You then said in a prior post:

No, webpack-dev-middleware in this case respect only for webpack assets

So I wanted to have a closer look at those replies I'm getting.

To sum it up, it seems to me that HEAD requests for javascript files and for the base HTML page are all handled by middleware.js from this project; while HEAD requests for HTML in subpaths - which to me should be handled as requests on / - somehow do not pass through the same sequence of handler than their GET counterparts, and eventually end up in 404.

[alexander-akait]

Please provide small example, I can't understand your problem

[watery]

Sure, I'll do it.

Meanwhile, may I kindly request this issue be reviewed by an additional person? No offence of course, but I'm not a native English speaker and maybe we're just misunderstanding each other a little.

[alexander-akait]

We don't have many contributors

[watery]

Ok.

As a quick sample, this would replicate the behaviour:

npx create-react-app
npm start

Then

curl http://localhost:3000/a/b -X GET --head

and

curl http://localhost:3000/a/b -X HEAD --head

I made a small sample too, you can find it here: https://github.com/watery/head-req-webpack-test

Those sources have been made following the Getting started guide, so they should be very basic. I started the dev-server with node_modules\.bin\webpack-cli serve.

This was a good suggestion, as I discovered that in my small sample the issue doesn't show up:

curl http://localhost:8000/a/b -X GET --head

and

curl http://localhost:8000/a/b -X HEAD --head

Both return 404.

So I checked twice the dev-server configuration in the two samples and found that is the historyApiFallback option that makes the difference: If I enable it in my sample project, then I get 200 on GET's, while still getting 404 on HEAD's.

I saw that webpack-dev-server configures the connect-history-api-fallback module when that option is true. Checking its issues I found this one: HEAD requests are not accepted.

[alexander-akait]

So I checked twice the dev-server configuration in the two samples and found that is the historyApiFallback option that makes the difference: If I enable it in my sample project, then I get 200 on GET's, while still getting 404 on HEAD's.

I see... Unfortunately, this problem does not seem to be solved.

[watery]

Well, the good news is that we finally found where the root of this issue is.

I'm adding to the issue in that other repo to see if it can be sorted out; please leave this open for a little more, I'll update it when I get news.

[alexander-akait]

I'll keep this open until we solve the problem or mark it as won't fix

[alexander-akait]

Close in favor webpack/webpack-dev-server#2716