-
Notifications
You must be signed in to change notification settings - Fork 197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hackage doesn't show latest uploads #573
Comments
|
At a guess, CDN. Newer results than that is a little confusing though. |
sadly, not cdn. munging the url doesn't change the results. There may have been a brief window while drives were being swapped where uploads were missed, although the procedure for the swap was developed to avoid this eventuality. Checking the docbuilder, I see that between slack-web and mediabus, packages uploaded and which don't appear in the above listing were intro-0.1.0.9 We should probably contact the authors of those packages and warn them that they will need to upload again. |
cc @minad @thoughtpolice @cgorski @liskin and @joeyh |
I don't understand how |
Same way it made it into the docbuilder so I could see it in the logs... it was uploaded properly to the "old" hackage and processed, and somehow the snapshot that the "new" hackage was set up with wasn't fully up-to-date. :-/ I hope there are no ramifications with hackage security for this, in terms of the induced rollback and rebranch of the signing. cc: @edsko and @dcoutts ... |
The package made it into all-cabal-hashes, which is how it's included in Stackage right now: https://github.com/commercialhaskell/all-cabal-hashes/tree/hackage/tasty-stats/0.2.0.2. The next version of Stack (and the Stackage build tool) will be using Hackage Security for getting the package list instead of all-cabal-hashes. That should mean better compatibility with any upstream issues going on with Hackage in the future. I'm not sure if "bug for bug compatibility" is a good thing or a bad thing here, simply stating the facts :) |
Haha. They say you can't delete packages uploaded to Hackage and here I am, not uploading tasty-jenkins-xml again, pretending it never happened. :-) (the full story is that in between writing the code and publishing on Hackage, a better solution appeared and my package is not necessary/useful any more) |
Thx for looking into this! I re-uploaded the package versions again. |
I just discovered that our all-cabal-hashes mirror is no longer updating, due to mismatched hashes:
This is becoming more worrying for me, as Hackage has reported two different versions of the same packages to the Hackage mirror tool. (Pinging @hvr, who may be affected as well.) I'm not really sure what the right thing to do here is. It seems like the simplest is to delete the tarball from the S3 mirror and hope everything rights itself. |
@snoyberg Ups, sorry! Maybe I should have waited a little longer or just have skipped this version. |
@minad Were there actually differences in the two tarballs you uploaded? |
I don't think so. I used "stack upload ." both times. Will this create different hashes each time? Maybe because of timestamps? |
Yeah, it's probably timestamps. I'll confirm shortly... |
Cool, thank you! |
Yup, that's the case:
Note the different timestamp on Given this, I'd recommend that anyone else planning on reuploading use a new package version. It would be nice if Hackage could somehow block reuploads of these name/version combos too. |
I think we have a serious problem on Hackage right now. I just ran this script:
The output was:
Notice how the checksums differ between the package.json file and what I've calculated from the actual download. In fact, it appears that the original checksums that all-cabal-tool was complaining about (which are also the ones original present on the S3 mirror) are still present in package.json, and in conflict with what Hackage is now serving. Looks like there was an incomplete sync. |
So I don't yet know why we have different copies of certain packages, though almost certainly related to the server move, but the current inconsistency between the content of the intro-0.1.0.9.tar.gz and its expected content from the index is due to CDN caching. Compare
At the time of writing, the 1st has the content that @snoyberg says above, while the 2nd has the expected content. As an experiment we're purging the CDN cache for that file to check, so the above will likely no longer be true for other observers. We should identify all the inconsistencies and check what is going on with them, to check if there's anything suspicious or othrerwise try to identify the cause of the problem during the server move. One thing to note is that the new server does have the content stored (in it's blob store) for both the versions of the intro-0.1.0.9.tar.gz file we're talking about here. |
Thanks @dcoutts, I can confirm that using hackage-origin solves the mismatched hash info (I must have gotten "original" and "new" confused in my previous comments). I've manually uploaded the correct version to our S3 mirror, hopefully that will suffice for the all-cabal-tool job to be satisfied. |
The hackage move happened in the middle of the S3 outage, maybe that is where the CDN corruption came in |
@alanz My understanding of things is that the CDN is caching the original version of the intro package, which was uploaded to the old server and not synced to the new server. When the new server came online, it had no record of that intro release, so a new release was accepted. That new release had a different checksum. As @dcoutts now demonstrated, Hackage is serving a matching 01-index.tar file and intro-0.1.0.9.tar.gz file (contrary to my claim above). My demonstration of mismatched checksums are due to the CDN holding onto the original intro-0.1.0.9.tar.gz file. tl;dr: I don't think the S3 outage has anything to do with this. The CDN isn't corrupt, it's just serving older content. |
Re-uploaded git-annex under a new version number.
…--
see shy jo
|
Re-uploaded general-games under new minor version number 1.0.1 |
Ok, so:
On the |
Would you mind writing a post-mortem once the investigation is complete? I'm mostly interested in precisely how this went wrong and how you'll try to prevent it from happening again. |
FYI - the latest version of general-games did not go into tonight's build,
despite a version bump earlier in the day.
…-Chris
On Thu, Mar 2, 2017 at 1:28 AM Simon Jakobi ***@***.***> wrote:
Would you mind writing a post-mortem once the investigation is complete?
I'm mostly interested in precisely how this went wrong and how you'll try
to prevent it from happening again.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#573 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AA8gknD8vCz9XhyhUejIMR6Bb2_pihuKks5rhmGBgaJpZM4MPMgO>
.
|
From hvr's more complete list, also missing from that window was |
I'm reuploading machines-amazonka now, and I've got a GH account but rarely use it. Thanks! |
What are referring to here? |
He had logs from matrix.hackage that included a single further package, and forwarded them to me offthread. |
Ah, nice! Good to hear that hvr is active, I'd been wondering where he'd been. |
The incident with this particular transition I think was resolved by the end of this thread. Going to close this ticket as part of cleaning things up. |
Reposting my comment from Reddit in case you haven't seen it yet:
https://www.reddit.com/r/haskell/comments/5wpkhh/heads_up_short_hackage_downtime_today_30_minutes/decpxvo/
The text was updated successfully, but these errors were encountered: