Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start using the pantry package #4254

Merged
merged 249 commits into from
Aug 26, 2018
Merged

Start using the pantry package #4254

merged 249 commits into from
Aug 26, 2018

Conversation

snoyberg
Copy link
Contributor

This is a massive diff. Any reviews people have are most welcome. The basic idea is:

  • All code related to downloading packages, unpacking packages, parsing cabal files, and interacting with Hackage have been moved into a new package, pantry
  • This new package currently resides inside the Stack repo, but will be moved out in the future. (Same with the curator package present as well.)
  • The rest of the codebase has to adapt :)

For those looking to review, I'd recommend:

  • Don't bother looking at individual commits, too much noise
  • Check out the changes inside the doc directory and ChangeLog.md first
  • Review the code in subs/pantry next
  • As much as you'd like, review the changes in src

snoyberg added 30 commits July 16, 2018 16:17
This will be a new library for storing package information. This first
bit overhauls the Hackage index update code, and stores information in a
SQLite database instead of the old caches. This turns out to be
significantly faster for `stack update` calls.

Fixes #3586

Note that it would be nicer to just resume the caching from where we'd
last left off, or to parse the revision numbers from the cabal files
themselves. See the discussion in haskell/hackage-server#779 to see why
that isn't possible.
Thanks to @phadej for the inspiration for this in his comment:
haskell/hackage-server#779 (comment)
snoyberg and others added 12 commits August 23, 2018 06:50
Ideally, fixes #4247. Instead of immediately failing on the database
being busy (via usage by another process), Stack will now have a 2
second pause and then try again. The vast majority of interactions with
the database are much faster than 2 seconds. The only exception is the
population of the Hackage index cache, but that's hopefully rare enough,
and touching few enough tables, to not be a problem.
Use a hacked version of store which hopefully requires much less memory.
Pantry doesn't support archives and repos without cabal files. I'm
generally opposed to checking in generated files, but it may be the best
choice here. The alternative would end up with some crazy hacks to
support the non-deterministic nature of generating cabal files from
package.yaml files.
@dbaynard
Copy link
Contributor

Thanks, the changes look good. I'm grateful for the explanation of the cache-breaking behaviour.

@snoyberg
Copy link
Contributor Author

Thanks, the changes look good. I'm grateful for the explanation of the cache-breaking behaviour.

I'm very grateful for the great review, thanks!

Copy link
Contributor

@mihaimaruseac mihaimaruseac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skimmed over much of src/ changes, they look good to me but I'm still not 100% up to date with the entire codebase


Since 1.6.0
This flag was introduced in Stack 1.6, and removed in Stack 1.11 with
hte move to Pantry. You will receive a warning if this configuration
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the instead of hte

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks!

@mihaimaruseac
Copy link
Contributor

There's an argument to wait on merging this into master until after I've implemented #3922 to avoid breaking caches twice. However, my aversion to this is because:..

I'd say we should merge this first, it's already quite large.

@snoyberg
Copy link
Contributor Author

Integration tests are passing, so this is ready to merge. There are some merge conflicts with master, which I'm resolving. Once that's done and CI is green, anyone should feel free to merge this.

@snoyberg snoyberg merged commit 99ecb78 into master Aug 26, 2018
@snoyberg snoyberg deleted the pantry branch August 26, 2018 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants