Fetch doesn't work for recently updated package

[MartinNowak]

I have updated a package of mine on the registry around 30 minutes ago. When I try using it as a dependency, I get:
Root package psql-native contains reference to invalid package backtrace-d

If I try to dub fetch backtrace-d --version=0.1.0, I get:
No package backtrace-d was found matching the dependency 0.1.0

If I try the ~master version, it works.
If I try a different package, it works.

If I am doing a mistake, a better error message would be nice. Otherwise, there is a bug somewhere.

[yazd]

I had to dub clean-caches and then everything worked nicely.
Did I have to do that or were the caches corrupted/invalid for some reason?

[s-ludwig]

Hm, looks like the caches should be circumvented when an explicit request for an uncached Version happens. Currently, AFAIR, the local cache is always used and if it is out of date, this error will happen.

[MartinNowak]

Hm, looks like the caches should be circumvented when an explicit request for an uncached Version happens.

Yes, you're right. I might fix it later today.

[rcorre]

Just ran into this with a package that was updated 16hrs ago -- dub clean-caches fixed it for me.

Hm, looks like the caches should be circumvented when an explicit request for an uncached Version happens

What is an explicit request in this case? Would this fix only apply to dub fetch? Or should a dub build automatically check if there is a newer version matching my version spec without my having to clean the cache?

[MartinNowak]

Or should a dub build automatically check if there is a newer version matching my version spec without my having to clean the cache?

That greatly limits the usefulness of the cache to an etag comparison.

What is an explicit request in this case?

I'd say dub fetch foo --version=0.1.0 and maybe also dub upgrade, in general any request for a specific package version.

[s-ludwig]

I'd say dub fetch foo --version=0.1.0 and maybe also dub upgrade.

Yeah, let's do both. I guess that there will be a lot less confusion that way and speed is only really important for the build process anyway.

[MartinNowak]

We only ask the package suppliers for metadata when upgrading or
searching a specific package version. In particular building an already
fetched package does not ask the package suppliers (only the package
manager responsible for locally installed packages).

In all cases where we get data from a package supplier we want to get
the latest available information. Therefor it doesn't make sense to
cache any metadata from the package suppliers on disk.

Not sure whether we still need the in memory metadata cache (e.g. for
multiple queries during dependency resolution).

The other need to query package suppliers (regular upgrade check) is already covered by the project local upgrade cache in ./dub/dub.json.

[s-ludwig]

Sounds reasonable. Wonder how the current situation came to be.

[MartinNowak]

Sounds reasonable. Wonder how the current situation came to be.

I added it to not hit the website on a rebuild. Not sure why it did back then, maybe the upgrade cache wasn't yet working?
Pretty sure I tested it like described here.
When I added the cache I didn't know much about the distinction between PackageManager and PackageSupplier and why the metadata was actually queried.

[s-ludwig]

Yeah, I was mostly wondering, because there were hard numbers in #388. Anyway, not really important now either - I didn't notice any degrade in performance.

[MartinNowak]

s-ludwig commented 3 days ago
Yeah, I was mostly wondering, because there were hard numbers in #388. Anyway, not really important now either - I didn't notice any degrade in performance.

Maybe sth. w/ dub.selections.json or the upgrade cache wasn't working back then?
I remember that the metadata cache got rid of a 150ms round trip for every build.
I did test this PR with disabled network, just to be sure.