spec: support exists() on directories pre-cache

[isidentical]

import secrets
from adlfs import AzureBlobFileSystem

BUCKET = 'bucket/'
FOLDER = f'{BUCKET}{secrets.token_hex(12)}/'

fs = AzureBlobFileSystem(**opts)
fs.touch(FOLDER + 'foo')
fs.touch(FOLDER + 'bar')
fs.invalidate_cache()

print(fs.exists(FOLDER))
fs.ls(fs._parent(FOLDER))
print(fs.exists(FOLDER))

While the first one is returning False, after the cache the second one is returning true. The reason is that, the exists() implementation first tries to check whether a such a directory exists by traversing the cache and when it fails it tries to check if a blob named path exists though if the directory is not cached, it doesn't check for that. This patch makes it fallback to the default implementation which is basically checking the result of info().

[isidentical]

The test earliest passes but the other failures are caused by /issues/240. CC: @hayesgb @TomAugspurger (it would be really cool to have this, since a lot of codepaths that we have depend on this exist() calls to result with True for directories on all filesystems, thanks!)

[TomAugspurger]

Looks good. One small question.

[isidentical]

One small question.

I can't see the question.

[TomAugspurger]

Strange, my review was lost. The gist was: should the .rstrip call be .rstrip("/", 1) to avoid cases where the path is /path/to/file// (if that's even allowed)?

[isidentical]

Strange, my review was lost. The gist was: should the .rstrip call be .rstrip("/", 1) to avoid cases where the path is /path/to/file// (if that's even allowed)?

There are multiple places where a regular rstrip() is used so I don't think that would help. E.g https://github.com/dask/adlfs/blob/96958331854c5a67917d4c9a802a62720d091374/adlfs/spec.py#L936

[TomAugspurger]

CI is passing now. Planning to merge in a few hours.

[TomAugspurger]

Thanks @isidentical!