Add support for "listing" HTTP directories (HTTP version)

[bbockelm]

Currently, "listing" HTTP directories is unsupported by the HTTPDirectory class (the implementation just returns ENOSYS for all API calls).

This ticket is the companion to the WebDAV version. While WebDAV directory listing is properly defined by RFC, this ticket is to do a heuristic-based directory listing, assuming the remote server generates a human-readable HTML listing page (a-la classic Apache when you try to access a page corresponding to a directory on disk).

Duplicate the logic found in the HTTP fsspec implementation (see the _ls_real code and the corresponding regexes; it also links to this StackOverflow page).

• Add new code for listing directories based on HTTP responses. Buffer the entire response from the remote server in memory; if needed, we will later add support for streaming the response (for reducing memory consumption on large directories).
• Add a unit test using XRootD as a backend. Note that xrootd doesn't have a / as a suffix in the directory URLs, breaking the fsspec code. So, we shouldn't (currently) test that behaves correctly in the unit tests.

[rw2]

@bbockelm how do I test this? I don't seem to be able to get the calls to httpdirectory interface to be invoked.