addToStore to S3 and local binary cache #37

puffnfresh · 2019-03-10T19:52:07Z

No description provided.

puffnfresh · 2019-03-11T02:07:51Z

Just verified that this is now producing the exact same .nar and .narinfo files! 🎆

store = localBinaryStoreEffects "/tmp/xxx/hnix"
addToStore "/nix/store" "shell.nix" "shell.nix" True store

$ nix copy --to file:///tmp/xxx/nix $(nix add-to-store shell.nix)
$ diff -r /tmp/xxx/hnix /tmp/xxx/nix
Only in /tmp/xxx/nix: nix-cache-info

shlevy

Same concern as #36 , haven't reviewed the rest yet.

Same as using file://$DIRECTORY

sorki · 2020-07-27T13:27:01Z

hnix-store-core/src/System/Nix/StorePathMetadata.hs

+  , -- | The size of the nar the path, in bytes.
+    fileBytes :: !(Maybe Word64)
+  }
+


I've completely missed this and recently wrote/extracted from Cachix a lib - https://github.com/sorki/nix-narinfo which is hopefully polymorphic enough to accommodate our types.

sorki · 2020-07-27T13:28:52Z

hnix-store-binary/src/System/Nix/Store/Binary.hs

+  :: StorePath storeDir
+  -> Text
+narInfoFileFor p =
+  encodeBase32 (storePathHash p) <> ".narinfo"


This is now part of -cores System.Nix.StorePath called storePathToNarInfo.

sorki · 2020-07-27T13:29:30Z

hnix-store-core/src/System/Nix/Internal/Hash.hs

+-- | Encode a 'Digest' in algorithm:hash format.
+encodeSomeDigest :: SomeNamedDigest -> Text
+encodeSomeDigest (SomeDigest d) =
+  encodeDigest d


This is pending via #64.

sorki · 2020-07-27T13:30:17Z

hnix-store-core/src/System/Nix/ReadonlyStore.hs

+      case narHashMode of
+        Recursive -> ("source", h)
+        RegularFile -> ("output:out", hash ("fixed:out:" <> encodeUtf8 (encodeBase16 h) <> ":"))
+


This landed as part of #36.

sorki · 2020-07-27T13:38:35Z

hnix-store-binary/src/System/Nix/Store/Binary.hs

+  encodeBase32 (storePathHash p) <> ".narinfo"
+
+addToStore
+  :: forall storeDir m. (MonadIO m, KnownStoreDir storeDir)


As of #59 (1f86211) StorePath is no longer parametrized by KnownStoreDir storeDir, store root FilePath is part of storePathRoot of StorePath.

Suggested change

:: forall storeDir m. (MonadIO m, KnownStoreDir storeDir)

:: MonadIO m

sorki · 2020-07-27T13:39:06Z

hnix-store-binary/src/System/Nix/Store/Binary.hs

+  -> FilePath
+  -> NarHashMode
+  -> BinaryStoreEffects m
+  -> m (StorePath storeDir)


Suggested change

-> m (StorePath storeDir)

-> m StorePath

sorki · 2020-07-27T13:40:38Z

Looking good! Few things need to land and we can merge this. Your thoughts on effect systems and having GADT for ops like addToStore?

Ericson2314 · 2020-07-27T14:02:53Z

I would not bother with effect systems. Big waste of time.

sorki · 2020-07-27T14:12:06Z

I would not bother with effect systems. Big waste of time.

Care to elaborate on that?

I like how the effects compose compared to stack of transformers we use currently (mostly in hnix and hnix-store-remote) and came up with an hnix-store-polysemy experiment recently https://gist.github.com/sorki/10a035089e601045789357f37e7ab89c#file-a-hs-L32 that could be our high level effect, while parts of the store could be implemented in terms of effects again allowing for flexibility (e.g. running a store over TCP instead of socket, purely, local vs remote, s3..). We already have a home-brew effect system in both hnix and hnix-store so why not full blown effect system? (disregarding immaturity, which is gonna change when eff and related GHC proposal lands).

sorki · 2023-11-19T12:12:09Z

I would like to salvage this, it bothers me it's sitting here for so long. Lots of pieces added here landed in core in some form.

Some random remarks, I won't add anymore inline comments as it makes the code hard to read:

NarHashMode is no more and was replaced by FileIngestionMode.
It should use nix-narinfo, it has more fields but these are mostly Maybe or Set which could be left empty until required https://github.com/sorki/nix-narinfo/blob/master/src/Nix/NarInfo/Types.hs
- encodeContentAddressableAddress used by NarInfo part is no more and was replaced by System.Nix.ContentAddress.buildContentAddress
- encodeSomeDigest was replaced by System.Nix.Hash.algoDigestBuilder (SomeDigest and SomeNamedDigest are no more and they were replaced by DSum HashAlgo Digest or just Digest)
I think the better name for -binary is -local since it seems local-store related (and -binary suggests that is has something to do with binary package / instances). I have a stub -local in one of my worktrees which is just this

import qualified System.Linux.Mount
import qualified System.Linux.Namespaces

makeStoreWriteable :: IO ()
makeStoreWriteable = do
  System.Linux.Namespaces.unshare [System.Linux.Namespaces.Mount]
  System.Linux.Mount.rebind "/nix/store" []

It actually doesn't seem that hard, I might try cleaning up the PR after I'm done with #233 #235 #238

puffnfresh changed the title ~~Initial work on S3 addToStore~~ addToStore to S3 and local binary cache Mar 11, 2019

shlevy requested changes Mar 15, 2019

View reviewed changes

puffnfresh added 8 commits March 28, 2019 02:13

Add NarInfo

3052f1f

Add makeFixedOutputPath

690949a

Initial work on S3 addToStore

08f6be3

Add makeFixedOutputCA

9df278c

Separate S3 logic from addToStore into new hnix-store-binary lib

c4df329

Clean up dependencies of hnix-store-s3

59d5c28

Add a local binary store

b74a691

Same as using file://$DIRECTORY

Clean up s3BinaryStoreEffects slightly

5a5b860

puffnfresh force-pushed the feature/s3 branch from 7db72ef to 5a5b860 Compare April 3, 2019 02:18

puffnfresh requested a review from shlevy April 3, 2019 02:18

sorki reviewed Jul 27, 2020

View reviewed changes

shlevy removed their request for review October 13, 2022 12:02

sorki mentioned this pull request Nov 19, 2023

Move Recursive newtype to core #238

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

addToStore to S3 and local binary cache #37

addToStore to S3 and local binary cache #37

puffnfresh commented Mar 10, 2019

puffnfresh commented Mar 11, 2019 •

edited

Loading

shlevy left a comment

sorki Jul 27, 2020

sorki Jul 27, 2020

sorki Jul 27, 2020

sorki Jul 27, 2020

sorki Jul 27, 2020

sorki Jul 27, 2020

sorki commented Jul 27, 2020

Ericson2314 commented Jul 27, 2020

sorki commented Jul 27, 2020

sorki commented Nov 19, 2023 •

edited

Loading

	:: forall storeDir m. (MonadIO m, KnownStoreDir storeDir)
	:: MonadIO m

addToStore to S3 and local binary cache #37

Are you sure you want to change the base?

addToStore to S3 and local binary cache #37

Conversation

puffnfresh commented Mar 10, 2019

puffnfresh commented Mar 11, 2019 • edited Loading

shlevy left a comment

Choose a reason for hiding this comment

sorki Jul 27, 2020

Choose a reason for hiding this comment

sorki Jul 27, 2020

Choose a reason for hiding this comment

sorki Jul 27, 2020

Choose a reason for hiding this comment

sorki Jul 27, 2020

Choose a reason for hiding this comment

sorki Jul 27, 2020

Choose a reason for hiding this comment

sorki Jul 27, 2020

Choose a reason for hiding this comment

sorki commented Jul 27, 2020

Ericson2314 commented Jul 27, 2020

sorki commented Jul 27, 2020

sorki commented Nov 19, 2023 • edited Loading

puffnfresh commented Mar 11, 2019 •

edited

Loading

sorki commented Nov 19, 2023 •

edited

Loading