Missing package.json files for some packages?

[snoyberg]

When I turn on "require package hashes" in Stack on the Hackage Security branch, I get the following error message:

Package index Hackage is configured to require package hashes, but no hash is available for amazonka-codedeploy-1.4.0

Sure enough, if I look in the 01-index.tar file, there is no package.json file for that release of amazonka-codedeploy:

Is there some reason for this file to be missing? From a security standpoint, it would be nice to be able to depend on the existence of a hash for every package.

Also, I'm not sure if this is the appropriate repo for this question. I can move it to the Hackage Server tracker if that's better.

[hvr]

We have known about this for quite some time already (c.f. haskell/hackage-server#488) but this glitch is very hard to reproduce and hasn't re-occured ever since then. This is also the reason why the hackage-mirror-tool needs this temporary hack. At some point we're gonna supply those missing package.json files in the index tarball, we just didn't get to it yet.

[snoyberg]

Is there no method available to:

1. Manually add the missing package.json files, and
2. Add a sanity check (like each .cabal file has a package.json file) before the 01-index.tar.gz file is generated?

This does seem to significantly impede the ability of tools to provide security guarantees.

[hvr]

This does seem to significantly impede the ability of tools to provide security guarantees.

That's not really true though because as a kind of happy accident this unintentional situation currently exercises cabal's code-paths for when the package hashes are missing; and in fact cabal refuses to download those couple of packages whose cryptographic checksums are missing. This just forces us to be able to cope with this exceptional case which we would have to anyway.

[tfausak]

Is there any way for me to help with this issue? It has been a known problem for a year and a half. It continues to happen as recently as a week ago. It does not seem like something that tools should have to deal with; if a .cabal file is in the index, a package.json file should be there too.

• hermes-1.3.4.3 data is incomplete commercialhaskell/all-cabal-hashes#9
• gogol-admin-reports-0.2.0 data is incomplete commercialhaskell/all-cabal-hashes#11
• intro-0.1.0.9 data is incomplete commercialhaskell/all-cabal-hashes#12
• wai-middleware-prometheus-0.2.0 is incomplete commercialhaskell/all-cabal-hashes#15
• cabal-install cannot download gnuidn-0.2.2 cabal#4828
• 01-index entries missing package.json hackage-server#488
• Missing JSON meta data for gogol-admin-reports-0.2.0  hackage-server#570
• Hackage doesn't show latest uploads hackage-server#573
• Missing package.json files in 01-index.tar hackage-server#623
• CRITICAL exception: fixupIdx "gogol-admin-reports-0.2.0.tar.gz" haskell-hvr/hackage-mirror-tool#2

[tfausak]

It's hard to tell how often this happens because the original report (haskell/hackage-server#488) has been edited, but these are the packages that are currently affected by this problem:

• amazonka-codedeploy-1.4.0
• amazonka-config-1.3.5
• amazonka-importexport-1.3.7
• amazonka-importexport-1.4.0
• amazonka-s3-1.3.6
• amazonka-sdb-1.4.0
• eventloop-0.5.0.0
• gi-gdkpixbuf-0.2.31.12
• gi-gio-0.2.46.13
• gi-girepository-0.1.44.12
• glue-ekg-0.4.4
• gnuidn-0.2.2
• gogol-admin-reports-0.2.0
• gogol-autoscaler-0.0.1
• hops-0.4.0
• hprotoc-2.1.11
• hzaif-0.1.0.0
• repa-io-3.4.0.2
• wai-middleware-prometheus-0.2.0

Generated with https://gist.github.com/tfausak/ed463c32638b8bd876df1a0712fc88bd.

[snoyberg]

@tfausak That looks like to lines up with the list from hackage-mirror-tool: https://github.com/snoyberg/hackage-mirror-tool/blob/213064e4dd5983d01b6c7daa1c7eb37bed5255fd/src/IndexShaSum.hs#L165

[snoyberg]

Looks like aivika-transformers-5.3.1.tar.gz needs to be added to the list.

[tfausak]

As @snoyberg mentioned, aivika-transformers-5.3.1 is missing its package.json. It was uploaded at 2017-10-27T04:35:54Z.

Also this just happened again with llvm-hs-5.1.1 (haskell/hackage-server#643). That one was uploaded at 2017-12-16T14:00:07Z.

[gbaz]

This is resolved with haskell/hackage-server#488

[snoyberg]

Can you clarify how this resolved the issue? It seems like this is a one time fix-up, but doesn't solve the underlying cause, meaning this may recur in the future. I'd be happy to hear that the underlying cause was fixed too (and frankly would be really interested in learning what the problem is).

…

On Mar 19, 2018 4:30 AM, "gbaz" ***@***.***> wrote: This is resolved with haskell/hackage-server#488 <haskell/hackage-server#488> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#183 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADBB7n70lyzteie4789-hsYCIDWLQ6bks5tfxhBgaJpZM4LthXA> .

[gbaz]

The underlying cause was fixed in haskell/hackage-server#644

Before, insertion of the package into the packagedb occurred and then hackage security data was added to the db, in two steps. So it was possible a thread could die or the server could fall-over between the steps. That change made sure that either both changes happened or neither change happened.

[snoyberg]

Awesome, thanks!

…

On Mon, Mar 19, 2018 at 8:12 AM, gbaz ***@***.***> wrote: The underlying cause was fixed in haskell/hackage-server#644 <haskell/hackage-server#644> Before, insertion of the package into the packagedb occurred and *then* hackage security data was added to the db, in two steps. So it was possible a thread could die or the server could fall-over between the steps. That change made sure that either both changes happened or neither change happened. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#183 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADBB94KD5QWTfREIHbj4NjpygFHEHELks5tf0w5gaJpZM4LthXA> .