Chase improvements to input hashing in gps #97

sdboyer · 2017-01-16T00:20:23Z

The only real change here is improvements to input hashing. That's both algorithm changes:

I tightened down the data included in the hash inputs considerably to avoid superfluous hash changes
Overrides are now much more, and hopefully completely, correct in how they show up
Constraints aren't incorporated if they're ineffectual - if the project neither imports nor requires anything under that constraint
I resolved a longstanding issue where the string outputs of constraints had no type information. This meant that a hash could match if, given everything else being the same, a constraint was on a branch foo and a constraint was on a tag/version foo. (This is visible in the output, where you'll see things like svc-^1.0.0 when the input constraint was ^1.0.0.)

And some output changes, via the HashingInputsAsString() func:

Each discrete segment of input to the hasher gets a newline injected after it - far better for diffing and reading
Headers demarcate the different sections of output, just so it's easier to scan and see which parts of the input data you're glancing at. (Happily, these being constant inputs, they have no variable effect on the hash output after being introduced)

Because this fixes (let's hope) issues with memoization, I also removed wipeMemo(), so this fixes #59.

sdboyer · 2017-01-16T00:32:02Z

Ugh...failure seems to probably be this:

Which is probably caused by this change I merged: sdboyer/gps#141 . Guess that has unintended consequences, and I need to roll it back...

sdboyer · 2017-01-16T00:40:21Z

Adds a hash-inputs command that dumps the hash inputs. Also adds the capability to hide commands from help output, and makes hash-inputs hidden.

Obviously this is hacky, I'm happy to refactor if folks find it too objectionable.

sdboyer · 2017-01-16T21:07:27Z

hm, problem isn't sdboyer/gps#141. gotta be something else.

sdboyer · 2017-01-17T02:23:17Z

ah, i just needed to update the expected memo field. derp :)

jessfraz · 2017-01-17T18:33:54Z

If i run hoard ensure on this branch I get:

$ ./hoard ensure
ouchie, solve error: No versions of github.com/sdboyer/gps met constraints:
        master: Unable to update checked out version: fatal: reference is not a tree: 69fdac2db538237e27bbf7a68a305e78e0044591

        v0.13.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.13.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.12.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.11.2: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.11.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.11.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.10.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.10.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.9.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.8.2: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.8.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.8.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.7.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.6.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.5.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.5.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.4.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.3.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.2.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.1.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.0.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        master: Unable to update checked out version: fatal: reference is not a tree: 69fdac2db538237e27bbf7a68a305e78e0044591

        fix-race: Could not introduce github.com/sdboyer/gps@fix-race, as it is not allowed by constraint master from project github.com/golang/hoard.
refactor-hashing: Could not introduce github.com/sdboyer/gps@refactor-hashing, as it is not allowed by constraint master from project github.com/golang/hoard.
        sm-refactor: Could not introduce github.com/sdboyer/gps@sm-refactor, as it is not allowed by constraint master from project github.com/golang/hoard.ensure Solve(): No versions of github.com/sdboyer/gps met constraints:
        master: Unable to update checked out version: fatal: reference is not a tree: 69fdac2db538237e27bbf7a68a305e78e0044591

        v0.13.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.13.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.12.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.11.2: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.11.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.11.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.10.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.10.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.9.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.8.2: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.8.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.8.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.7.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.6.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.5.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.5.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.4.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.3.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.2.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.1.0: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.
        v0.0.1: Could not introduce github.com/sdboyer/[email protected], as it is not allowed by constraint master from project github.com/golang/hoard.

sdboyer · 2017-01-17T19:53:56Z

ugh. probably a stale repo on disk. i thought i had these cases covered, but dear god so much code.

could you cd $GOPATH/pkg/hoard/sources/https---github.com-sdboyer-gps and dump the results of git remote -av here?

once you do that, go ahead and blow away $GOPATH/pkg/hoard - that should fix it, right?

(not that that's an acceptable solution, just trying to work out why it's happening)

jessfraz · 2017-01-17T22:35:35Z

i think its because this PR updated vendor and not the lock & manifest tho right

jessfraz · 2017-01-17T22:39:16Z

oh nevermind this does change the lock... ok weird

jessfraz · 2017-01-17T22:41:37Z

jessfraz at jessfraz0 in ~/.go/pkg/hoard/sources/https---github.com-sdboyer-gps on 669a62f
$ git remote -v
origin  https://github.com/sdboyer/gps (fetch)
origin  [email protected]:sdboyer/gps (push)

jessfraz · 2017-01-17T22:42:44Z

ok after i blew it away I got no change to hash \0/

sdboyer · 2017-01-18T00:44:22Z

OK i'll rebase and fix this sometime tonight.

i'm gonna need a serious regression test for this in gps, too. i thought i had this covered, but it's annoying enough to test for that i never actually set one up. now that i see i clearly don't have it covered, there's no more delaying it.

jessfraz · 2017-01-18T22:58:26Z

hash_in.go

@@ -0,0 +1,53 @@
+package main


this file is missing the license header

Signed-off-by: Jess Frazelle <[email protected]>

jessfraz · 2017-01-18T23:06:02Z

hash_in.go

+	"github.com/sdboyer/gps"
+)
+
+func (cmd *hashinCommand) Name() string      { return "hash-inputs" }


wait why do we need this command?

exclusively for debugging. i'm totally fine to pull it out if it seems like overkill to add it

jessfraz · 2017-01-18T23:06:08Z

analyzer.go

@@ -37,5 +37,5 @@ func (a analyzer) DeriveManifestAndLock(path string, n gps.ProjectRoot) (gps.Man

 func (a analyzer) Info() (name string, version *semver.Version) {
 	v, _ := semver.NewVersion("v0.0.1")
-	return "example-analyzer", v
+	return "hoard", v


jessfraz · 2017-01-18T23:09:48Z

hash_in.go

+func (cmd *hashinCommand) Args() string      { return "" }
+func (cmd *hashinCommand) ShortHelp() string { return "" }
+func (cmd *hashinCommand) LongHelp() string  { return "" }
+func (cmd *hashinCommand) Hidden() bool      { return false }


if we do need it, it should be hidden right?

LOL literally the reason why i added the Hidden func and, in my copy/paste fervor, i forgot to use it correctly here

googlebot added the cla: yes label Jan 16, 2017

sdboyer mentioned this pull request Jan 16, 2017

This is the new hash on my machine #86

Closed

jessfraz mentioned this pull request Jan 17, 2017

add script to check if changes to vendor are valid #102

Merged

jessfraz mentioned this pull request Jan 17, 2017

tests: stop wiping memo #59

Closed

sdboyer added 5 commits January 18, 2017 16:16

Chase improvements to input hashing in gps

0f5f78a

Add hidden hash-inputs command to dump hash inputs

b108df4

Name the hoard analyzer appropriately

8e836c0

remove wipeMemo() in hopes they're stable now

7f6452c

Update lock.json

f10f4a1

jessfraz reviewed Jan 18, 2017

View reviewed changes

add license

1fbd8df

Signed-off-by: Jess Frazelle <[email protected]>

jessfraz reviewed Jan 18, 2017

View reviewed changes

Hide the hash inputs command

72df2c8

jessfraz merged commit e49a10c into golang:master Jan 19, 2017

sdboyer mentioned this pull request Jan 26, 2017

dep init fails, cause unknown, entire state rolls back so cannot adjust anything #158

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Chase improvements to input hashing in gps #97

Chase improvements to input hashing in gps #97

sdboyer commented Jan 16, 2017 •

edited

Loading

sdboyer commented Jan 16, 2017

sdboyer commented Jan 16, 2017

sdboyer commented Jan 16, 2017

sdboyer commented Jan 17, 2017

jessfraz commented Jan 17, 2017 •

edited

Loading

sdboyer commented Jan 17, 2017

jessfraz commented Jan 17, 2017 •

edited

Loading

jessfraz commented Jan 17, 2017

jessfraz commented Jan 17, 2017

jessfraz commented Jan 17, 2017 •

edited

Loading

sdboyer commented Jan 18, 2017

jessfraz Jan 18, 2017

jessfraz Jan 18, 2017

jessfraz Jan 18, 2017

sdboyer Jan 19, 2017

jessfraz Jan 18, 2017

jessfraz Jan 18, 2017

sdboyer Jan 18, 2017

Chase improvements to input hashing in gps #97

Chase improvements to input hashing in gps #97

Conversation

sdboyer commented Jan 16, 2017 • edited Loading

sdboyer commented Jan 16, 2017

sdboyer commented Jan 16, 2017

sdboyer commented Jan 16, 2017

sdboyer commented Jan 17, 2017

jessfraz commented Jan 17, 2017 • edited Loading

sdboyer commented Jan 17, 2017

jessfraz commented Jan 17, 2017 • edited Loading

jessfraz commented Jan 17, 2017

jessfraz commented Jan 17, 2017

jessfraz commented Jan 17, 2017 • edited Loading

sdboyer commented Jan 18, 2017

jessfraz Jan 18, 2017

Choose a reason for hiding this comment

jessfraz Jan 18, 2017

Choose a reason for hiding this comment

jessfraz Jan 18, 2017

Choose a reason for hiding this comment

sdboyer Jan 19, 2017

Choose a reason for hiding this comment

jessfraz Jan 18, 2017

Choose a reason for hiding this comment

jessfraz Jan 18, 2017

Choose a reason for hiding this comment

sdboyer Jan 18, 2017

Choose a reason for hiding this comment

sdboyer commented Jan 16, 2017 •

edited

Loading

jessfraz commented Jan 17, 2017 •

edited

Loading

jessfraz commented Jan 17, 2017 •

edited

Loading

jessfraz commented Jan 17, 2017 •

edited

Loading