Skip to content

Commit

Permalink
gj (#1)
Browse files Browse the repository at this point in the history
* plumbing: format: pktline, Accept oversized pkt-lines up to 65524 bytes

The canonical Git client successfully decodes sideband packets up to
65524 bytes in length (4-byte header + 65520-byte payload). The Git
protocol documentation was updated in August 2016 to reduce the maximum
payload size to 65516 bytes, however old implementations still exist in
the wild emitting 65520-byte payloads.

As there is no technical difficulty with accepting (not emitting) larger
payload sizes, this change adjusts the limit check to allow successful
decoding of packets up to 65524 bytes. This change increases
compatibility with the current canonical Git implementation.

Doc changes from August 2016:
  git/git@7841c48#diff-52695c8fe91b78b70cea44562ae28297L67

Current packet buffer size is still LARGE_PACKET_MAX (+1 null):
  https://github.com/git/git/blob/468165c1d8a442994a825f3684528361727cd8c0/sideband.c#L24
  https://github.com/git/git/blob/468165c1d8a442994a825f3684528361727cd8c0/sideband.c#L36

LARGE_PACKET_MAX definition:
  https://github.com/git/git/blob/468165c1d8a442994a825f3684528361727cd8c0/pkt-line.h#L100

Signed-off-by: Joseph Vusich <[email protected]>

* add PlainOpen variant to find .git in parent dirs

This is the git tool's behavior that people are used to; if one runs a
git command in a repository's subdirectory, git still works.

Fixes src-d#765.

Signed-off-by: Daniel Martí <[email protected]>

* use bsd superset for conditional compilation

Signed-off-by: wardn <[email protected]>

* config: adds branches to config for tracking branches against remotes, updates clone to track when cloning a branch. Fixes src-d#313

Signed-off-by: Jeremy Chambers <[email protected]>

* dotgit: ignore filenames that don't match a hash

For both packfiles and object files.

Issue: keybase/client#11366
Signed-off-by: Jeremy Stribling <[email protected]>

* storage: dotgit, init fixtures in benchmark. Fixes src-d#770

fixtures is not initialized in BenchmarkRefMultipleTimes and caused
panic.

Signed-off-by: Javi Fontan <[email protected]>

* git: remote, Add shallow commits instead of substituting. Fixes src-d#412

updateShallow substituted the previous shallow list with the one
returned by the UploadPackResponse. If the repository had previous
shallow commits these are deleted from the list.

This change adds the new shallow hashes to the old ones.

Signed-off-by: Javi Fontan <[email protected]>

* dotgit: add test for bad file in pack directory

Suggested by mcuadros.

Issue: src-d#807
Signed-off-by: Jeremy Stribling <[email protected]>

* Resolve full commit sha to plumbing hash

Signed-off-by: antham <[email protected]>

* storage: filesystem, close shallow file when read

Signed-off-by: Máximo Cuadros <[email protected]>

* git: worktree, Skip special git directory. Fixes src-d#814

Signed-off-by: kuba-- <[email protected]>

* travis: dropping 1.8.x support due to golang.org/x/crypto/ssh requirement

* Use remote name in fetch while clone

Fixes src-d#827

Signed-off-by: Dustin Frisch <[email protected]>

* Worktree: Provide ability to add excludes  (src-d#825)

Worktree: Provide ability to add excludes

* Teach ResolveRevision how to look up annotated tags

Signed-off-by: Mike Lundy <[email protected]>

* git: remote, Do not iterate all references on update.

The current code iterates all the references in the remote to check if
they match the refspec. This is OK when the refspec is a wildcard but
is a waste of time when they are not.

A hash with references is generated for fast access before starting the
update and used only when the refspec is not a wildcard.

In a repository with 7800 references this meant 7800 * 7800 checks. With
the current code it took 8m30s to update the references. With the new
code it takes less than 0.5s.

References are already extensively tested in remote_test.go.

Signed-off-by: Javi Fontan <[email protected]>

* idxfile: optimise allocations in readObjectNames

This makes all the required Entry allocations in one go,
instead of huge amounts of small individual allocations.

Signed-off-by: David Symonds <[email protected]>

* packfile: improve Index memory representation to be more compact

Instead of using a map for offset indexing, use a sorted slice.
Binary searching is fast, and a slice is much more compact.
This has a negligible hit on speed, but has a significant impact on
memory usage, especially for larger repos.

benchmark                         old ns/op     new ns/op     delta
BenchmarkIndexConstruction-12     15506506      14056098      -9.35%

benchmark                         old allocs     new allocs     delta
BenchmarkIndexConstruction-12     60764          60385          -0.62%

benchmark                         old bytes     new bytes     delta
BenchmarkIndexConstruction-12     4318145       3913169       -9.38%

Signed-off-by: David Symonds <[email protected]>

* config: modules, Ignore submodules with dotdot '..' path components. Fixes CVE-2018-11235

References:
 * https://blogs.msdn.microsoft.com/devops/2018/05/29/announcing-the-may-2018-git-security-vulnerability/
 * https://security-tracker.debian.org/tracker/CVE-2018-11235
 * git/git@0383bbb

Signed-off-by: Joseph Vusich <[email protected]>

* worktree: Don't allow .gitmodules to be a symlink. Fixes CVE-2018-11235

References:
 * https://blogs.msdn.microsoft.com/devops/2018/05/29/announcing-the-may-2018-git-security-vulnerability/
 * https://security-tracker.debian.org/tracker/CVE-2018-11235
 * git/git@10ecfa7

Signed-off-by: Joseph Vusich <[email protected]>

* dotgit: Move package outside internal.

Signed-off-by: Antonio Jesus Navarro Perez <[email protected]>

* Remove println

Signed-off-by: Antonio Jesus Navarro Perez <[email protected]>

* plumbing: object, adds tree path cache to trees. Fixes src-d#793

The cache is used in Tree.FindEntry for faster path search.

Signed-off-by: Javi Fontan <[email protected]>

* plumbing: packfile, Don't push empty objects. Fixes src-d#840

Signed-off-by: kuba-- <[email protected]>

* storage: filesystem, make ObjectStorage constructor public

Signed-off-by: Miguel Molina <[email protected]>

* plumbing/transport: http, Adds token authentication support [Fixes src-d#858]

Signed-off-by: Eric Billingsley <[email protected]>

* Fix documentation for Notes

It previously said that it returned all references that are branches, but that's not true.

Signed-off-by: Morgan Bazalgette <[email protected]>

* packfile: optimise NewIndexFromIdxFile for a very common case

Loading from an on-disk idxfile will usually already have the idxfile
entries in order, so check that before wasting time on sorting.

Signed-off-by: David Symonds <[email protected]>

* Remote.Fetch: error on missing remote reference

Signed-off-by: Máximo Cuadros <[email protected]>

* storage/filesystem: avoid norwfs build flag

norwfs build flag was used to work on filesystems that do not support neither opening a file in read/write mode or renaming a file (e.f. sivafs).

This had two problems:

- go-git could not be compiled to work properly both with regular filesystems and limited filesystems at the same time.
- the norwfs trick was not available on Windows.

This PR removes the norwfs build flag, as well as the windows conditional flag on the dotgit package.

For the file open mode, we use the new billy capabilities, to check at runtime if the filesystem supports opening a file in read/write mode or not.

For the renaming, we just try and fallback to alternative methods if billy.ErrNotSupported is returned.

Signed-off-by: Santiago M. Mola <[email protected]>

* utils: diff, skip useless rune->string conversion

According to library documentation :
https://github.com/sergi/go-diff/blob/master/diffmatchpatch/diff.go#L391

Signed-off-by: Marc Barussaud <[email protected]>

* plumbing: add context to allow cancel on diff/patch computing

Signed-off-by: Marc Barussaud <[email protected]>

* worktree: add test for correct tree sorting (issue src-d#881)

Signed-off-by: Mark Bartel <[email protected]>

* worktree: sort the tree object.  Fixes src-d#881

Signed-off-by: Mark Bartel <[email protected]>

* worktree: address PR comments: sort imports appropriately

Signed-off-by: Mark Bartel <[email protected]>

* plumbing: object, expose ErrEntryNotFound in FindEntry. Fixes src-d#883

FindEntry will return ErrDirNotFound if the directory doesn't exist. But
it doesn't return a public error if the entry itself is missing.  This
exposes the internal error ErrEntryNotFound, so users can
programmatically check for this condition.

Signed-off-by: James Ravn <[email protected]>

* plumbing/transport/internal: common, add support of Gogs for ErrRepositoryNotFound, avoiding to get an 'unknown error: '. Add some tests for existing supported services (github, gitlab, etc...) too.

Signed-off-by: Jerome Doucet <[email protected]>

* plumbing/object: fix pgp signature encoder/decoder

The way of reading pgp signatures was searching for pgp begin line in
the header. This caused problems when this string appeared and was not
part of the signature. For example if it appears in the message as an
example or is part of the author name the decoder starts treating it as
a signature. In this state the code was not able to notice then the
header ended so it entered in an infinite loop searching for pgp end
string.

Now it uses the same method as original git. Searches for gpgsig section
in header and starts getting all lines until the next part.

In encoder the string used to add signatures was incorrect. It is now
changed to the proper "gpgsig" string instead of "pgpsig".

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/format/idxfile: add new Index and MemoryIndex

Signed-off-by: Miguel Molina <[email protected]>

* plumbing/packfile: add new packfile parser

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/packfile: disable lookup by offset

In one case it disables the cache and the other disables lookup when
the scanner is not seekable. Could be added back later.

Signed-off-by: Javi Fontan <[email protected]>

* plumbing: idxfile, add idxfile.Writer with Observer interface

It's still not complete:

* 64 bit offsets
* IdxChecksum

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/idxfile: use Entry to hold object data

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/idxfile: support offset64 generating indexes

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/packfile: preallocate memory in PatchDelta

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/idxfile: fix bug searching in MemoryIndex

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/idxfile: add offset/hash mapping to index

This functionality may be moved elsewhere in the future but is needed
now to fit filesystem.ObjectStorage and the new index.

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/idxfile: index is created only once and retrieved with Index

Index is also automatically generated when OnFooter is called.

Signed-off-by: Javi Fontan <[email protected]>

* plumbing, storage: integrate new index

Now dotgit.PackWriter uses the new packfile.Parser and index.

Signed-off-by: Javi Fontan <[email protected]>

* plumbing: packfile, new Packfile representation

Signed-off-by: Miguel Molina <[email protected]>

* plumbing, packfile: delete index_test as is no longer used

Signed-off-by: Javi Fontan <[email protected]>

* plumbing: fix two errors in idxfile and packfile decoder

Signed-off-by: Javi Fontan <[email protected]>

* storage/filesystem: add back IndexStorage

Signed-off-by: Javi Fontan <[email protected]>

* plumbing: packfile, lazy object reads with DiskObjects

Signed-off-by: Miguel Molina <[email protected]>

* plumbing/idxfile: test FindHash and writer with 64 bit offsets

Signed-off-by: Javi Fontan <[email protected]>

* storage/filesystem: remove duplicated IndexStorage

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/packfile: add index generation to decoder

Signed-off-by: Javi Fontan <[email protected]>

* Fix wrong godoc on Tags() method.

Reword Tags() method documentation. Point to TagObjects() method to get all the tags on a repository.

Signed-off-by: Antonio Jesus Navarro Perez <[email protected]>

* plumbing: packfile, fix package tests

Signed-off-by: Miguel Molina <[email protected]>

* *: use parser to populate non writable storages and bug fixes

Signed-off-by: Miguel Molina <[email protected]>

* Fixed cloning of a single tag

Relates to src-d#870

Signed-off-by: Fedor Korotkov <[email protected]>

* plumbing: packfile, allow non-seekable sources on Parser

Signed-off-by: Miguel Molina <[email protected]>

* plumbing: packfile, add Parse benchmark

Signed-off-by: Miguel Molina <[email protected]>

* plumbing: packfile, read object content only once

Signed-off-by: Miguel Molina <[email protected]>

* storage: filesystem, benchmark PackfileIter

Signed-off-by: Miguel Molina <[email protected]>

* plumbing: packfile, rename DiskObject to FSObject

Signed-off-by: Miguel Molina <[email protected]>

* storage: filesystem, close Packfile after iterating objects

Signed-off-by: Miguel Molina <[email protected]>

* storage: filesystem, add PackfileIter benchmark reading object content

Signed-off-by: Miguel Molina <[email protected]>

* plumbing: packfile, open and close packfile on FSObject reads

Signed-off-by: Miguel Molina <[email protected]>

* git: add benchmark for iterating repository objects

Signed-off-by: Miguel Molina <[email protected]>

* plumbing: idxfile, Crc32 to CRC32 and return ok from findHashIndex

Signed-off-by: Miguel Molina <[email protected]>

* plumbing: add buffer cache and use it in packfile parser

It uses less memory and is faster as slices don't have to be converted
from/to MemoryObject and they are indexed by offset.

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/pacfile: tidy up objectInfo struct

* a new hasher is created when needed
* delete unused fields
* base content is no longer kept in memory

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/packfile: do not compute sha1 for already undeltified objects

Signed-off-by: Javi Fontan <[email protected]>

* added hook support

Signed-off-by: noxora <[email protected]>

trying a possible fix to the delete test

Signed-off-by: noxora <[email protected]>

still trying to fix this test

Signed-off-by: noxora <[email protected]>

fixes did not work, seems to be a windows env problem

Signed-off-by: noxora <[email protected]>

* plumbing: object, Don't add new line at end of commit signature

The way that commit signatures were being written out was causing an
extra newline to be written at the end of the commit when the message
encoding was already taking care of this. Ultimately, this results in a
corrupt object, rendering the object unverifiable with the signature in
the commit.

Signed-off-by: Chris Marchesi <[email protected]>

* git: Add ability to PGP sign commits

This adds the ability to sign commits by adding the SignKey field to
CommitOptions. If present, the commit will be signed during the
WorkTree.Commit call.

The supplied SignKey must already be decrypted by the caller.

Signed-off-by: Chris Marchesi <[email protected]>

* git: Remove use of strings.Builder

This was added in Go 1.10 and is not supported on Go 1.9. Switched to
bytes.Buffer to ensure compatibility.

Signed-off-by: Chris Marchesi <[email protected]>

* git: Remove old hash validation code

This will not work for a signed commit as with the GPG signature being a
part of the commit, the hash is now non-deterministic.

Verification of the commit is done through the validation of the
signature.

Signed-off-by: Chris Marchesi <[email protected]>

* git: Add extra test for testing bad key error case

I'm hoping this helps get codecov to a tolerable delta. :)

Signed-off-by: Chris Marchesi <[email protected]>

* dotgit: fix object delete test

Signed-off-by: Santiago M. Mola <[email protected]>

* object: fix panic when reading object header

When the first line of the pgp signature is an empty line or some header
is malformed it crashes as there's no data for the header element. For
example, if author name is "\n".

Signed-off-by: Javi Fontan <[email protected]>

* Fixed an edge case for .gitignore

Fixes src-d#923

Signed-off-by: Fedor Korotkov <[email protected]>

* plumbing/idxfile: object iterators returns entries in offset order

In the latest change the order was changed from offset order in
packfiles to hash order. This makes reading all the objects not as
efficient as before. It also created problems when the previous order
was expected.

Also added EntriesByOffset to indexes.

Signed-off-by: Javi Fontan <[email protected]>

* git: Add tagging support

This adds a few methods:

* CreateTag, which can be used to create both lightweight and annotated
tags with a supplied TagObjectOptions struct. PGP signing is possible as
well.
* Tag, to fetch a single tag ref. As opposed to Tags or TagObjects, this
will also fetch the tag object if it exists and return it along with the
output. Lightweight tags just return the object as nil.
* DeleteTag, to delete a tag. This simply deletes the ref. The object is
left orphaned to be GCed later.

I'm not 100% sure if DeleteTag is the correct behavior - looking for
details on exactly *what* happens to a tag object if you delete the ref
and not the tag were sparse, and groking the Git source did not really
produce much insight to the untrained eye. This may be something that
comes up in review. If deletion of the object is necessary, the
in-memory storer may require some updates to allow DeleteLooseObject to
be supported.

Signed-off-by: Chris Marchesi <[email protected]>

* plumbing: object, correct tag PGP encoding

As with the update in ec3d2a8, tag encoding needed to be corrected to
ensure extra newlines were not being added in during tag object
encoding, so that it did not corrupt the object for verification.

Signed-off-by: Chris Marchesi <[email protected]>

* plumbing: object, don't add extra newline on PGP signature

Tag encoding/decoding seems to be a lot more sensitive to requiring the
exact expected format in the object, which generally includes messages
canonicalized so that they have a newline on the end (even if they
didn't before).

As such, the message should be written with the newline (no need for an
extra), and the PGP signature right after that, which will be newline
split already, so there's no need to split it again.

All of this means it's very important for the caller to send the message
in the correct format - which I'm correcting in the next commit.

Signed-off-by: Chris Marchesi <[email protected]>

* git: Canonicalize incoming annotated tag messages

Tag messages are highly sensitive to being in the expected format,
especially when encoding/decoding for PGP verification.

As such, we do a simple trimming of whitespace on the incoming message
and add a newline on the end, to ensure there are no surprises here.

Signed-off-by: Chris Marchesi <[email protected]>

* git: Replace test signing key with one with longer expiry

The old one was created with defaults, which would have caused CI
failures in 2 years.

The new one is valid for 10 years:

> gpg --list-secret-keys
/root/.gnupg/pubring.kbx
------------------------
sec   rsa4096 2018-08-22 [SC] [expires: 2028-08-19]
      93A17FF01E54328546087C8E029395402EFCCD53
uid           [ unknown] foo bar <[email protected]>

Signed-off-by: Chris Marchesi <[email protected]>

* plumbing, storage: add bases to the common cache

After clone only resolved deltas were added to the cache. This caused
slowdowns in small repositories where most objects can be held in cache.

It also makes packfiles reuse delta cache from the store. Previously it
created a new delta cache each time a packfile object was created. This
also slowed down a bit accessing objects and had an impact on memory
consumption when bases are added to the cache.

Signed-off-by: Javi Fontan <[email protected]>

* git: Discern tag target type from supplied hash

I figured there was a way to do this without having to have
TagObjectOptions supply this in - there is.

Added support for this in and removed the object type from
TagObjectOptions.

Signed-off-by: Chris Marchesi <[email protected]>

* git: Don't return tag object with Tag, adjust docs for Tag and Tags

I've mainly noticed that in using the current Tag function, that there
were lots of times that I was ignoring the ref or the object, depending
on what I needed. This was evident in the tests as well. As such, I
think it just makes more sense for a singular tag fetcher to return just
a ref, through which the caller may grab the annotation if they need it,
and if it exists.

Also, contrary to the docs on Tags, all tags have a ref, even if they
are annotated. The difference between a lightweight tag and an annotated
tag is the presence of the tag object, which the ref will point to if
the tag is annotated. As such I've adjusted the docs with an example as
to how one can get the annotation for a tag ref through the iterator.

Source: https://git-scm.com/book/en/v2/Git-Internals-Git-References,
tags section.

Signed-off-by: Chris Marchesi <[email protected]>

* storage/dotgit: search for incoming dir only once

Search for incoming object directory was done once each time objects
were accessed. This means a ReadDir of the objects path that is
expensive. Now incoming directory is searched the first time an object
is accessed and its name kept in DotGit to be reused.

Signed-off-by: Javi Fontan <[email protected]>

* storage/dotgit: use HasPrefix instead of Split

Also reformatted function comment and fixed some typos.

Signed-off-by: Javi Fontan <[email protected]>

* Remove empty dirs when cleaning with Dir opt.

Signed-off-by: kuba-- <[email protected]>

* Add Status.IsUntracked function

Signed-off-by: kuba-- <[email protected]>

* plumbing: object: Clamp object timestamps before unix epoch to unix epoch

Signed-off-by: Taru Karttunen <[email protected]>

* config: add commentChar to core config struct

Signed-off-by: Zaq? Wiedmann <[email protected]>

* git: add Static option to PlainOpen

Also adds Static configuration to Storage and DotGit. This option means
that the git repository is not expected to be modified while open and
enables some optimizations.

Each time a file is accessed the storer tries to open an object file for
the requested hash. When this is done for a lot of objects it is
expensive. With Static option a list of object files is generated the
first time an object is accessed and used to check if exists instead of
using system calls.

A similar optimization is done for packfiles.

Signed-off-by: Javi Fontan <[email protected]>

* git, storer: use a common storer.Options for storer and PlainOpen

Signed-off-by: Javi Fontan <[email protected]>

* dotgit: fix typo in comment

Signed-off-by: Javi Fontan <[email protected]>

* git: do not expose storage options in PlainOpen

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/storer: rename Static option to ExclusiveAccess

Signed-off-by: Javi Fontan <[email protected]>

* storage/filesystem: make Storage options private

Signed-off-by: Javi Fontan <[email protected]>

* storage/filesystem: move Options to filesytem and dotgit

Signed-off-by: Javi Fontan <[email protected]>

* storage/dotgit: add ExclusiveAccess tests in dotgit

This functionality was already tested in storage/filesystem.
The coverage tool only takes into account files from the same
package of the test.

Signed-off-by: Javi Fontan <[email protected]>

* storage/dotgit: add KeepDescriptors option

This option maintains packfile file descriptors opened after reading
objects from them. It improves performance as it does not have to be
opening packfiles each time an object is needed.

Also adds Close to EncodedObjectStorer to close all the files manualy.

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/storer: do not expose Close in EncodedObjectStorer interface

Signed-off-by: Javi Fontan <[email protected]>

* storage/filesystem: add KeepDescriptors test

Also delete Close from MockObjectStorage and memory storer.

Signed-off-by: Javi Fontan <[email protected]>

* storage/filesystem: compare files using offset in test

Using equals to compare files it uses diff to do so. This can
potentially consume lots of ram. Changed the comparison to use file
offsets. If the descriptor is reused the offset is maintained.

Signed-off-by: Javi Fontan <[email protected]>

* plumbing/transport: ssh check if list of known_hosts files is empty

Signed-off-by: kuba-- <[email protected]>

* Fix fatal corrupt patch in unified diff format

Signed-off-by: Antonio Jesus Navarro Perez <[email protected]>

* Expose Storage cache.

Signed-off-by: kuba-- <[email protected]>

* git: s/fetch/returns/ on Tag function doc

This is to avoid any ambiguity with the act of "fetching" in git in
general.

Signed-off-by: Chris Marchesi <[email protected]>

* git: Add Tag objects to the list of supported objects for walking

This is necessary to support pruning on Tag objects.

Signed-off-by: Chris Marchesi <[email protected]>

* git: Don't touch tag objects orphaned by tag deletion

Deleting a tag ref for an annotated tag in normal git behavior does not
delete the tag object right away. This is handled by the normal GC
process.

Signed-off-by: Chris Marchesi <[email protected]>

* git: Add some tests for annotated tag deletion

Added a couple of tests for annotated tag deletion:

* The first one is a general test and should work regardless of the
fixture used - the tag object could possibly be packed, so we do a prune
*and* a repack operation before testing to see if the object was GCed
correctly.

* The second one actually creates the tag to be deleted, so that the tag
object gets created as a loose, unpacked object. This is so we can
effectively test that purning unpacked objects is now working 100%
correctly (this was failing before because tag objects were not
supported for walking).

Signed-off-by: Chris Marchesi <[email protected]>

* git: s/TagObjectOptions/CreateTagOptions/

Just renaming the TagObjectOptions type to CreateTagOptions so that it's
consistent with the other option types.

Signed-off-by: Chris Marchesi <[email protected]>

* repository: fix test for new Storage constructor

Signed-off-by: Máximo Cuadros <[email protected]>

* *: go modules support

Signed-off-by: Máximo Cuadros <[email protected]>

* travis: drop go1.9 add go1.11

* Fix potential LRU cache size issue.

Signed-off-by: kuba-- <[email protected]>

* Remove empty space to trigger windows build.

Signed-off-by: kuba-- <[email protected]>

* storage/filesystem: keep packs open in PackfileIter

PackfileIter was not taking into account the option KeepDescriptors
and was always closing the file. This caused "file already closed"
errors when iterating packfiles in with KeepDescriptors active.

Signed-off-by: Javi Fontan <[email protected]>

* storage/filesystem: add more doc to NewPackfileIter

Signed-off-by: Javi Fontan <[email protected]>

* all: remove extra 's' in "mismatch"

Signed-off-by: Jongmin Kim <[email protected]>

* test: improve test for urlencoded user:pass

Signed-off-by: Santiago M. Mola <[email protected]>

* use time.IsZero in Prune

Signed-off-by: u5surf <[email protected]>

* Add test for Windows local paths.

Signed-off-by: Filip Navara <[email protected]>

* git: Fix Status.IsClean() documentation

The documentation of the IsClean Method contained a negation, so it was
describing the opposite of its actual behavior.

Fixes src-d#838

Signed-off-by: David Url <[email protected]>

* Plumbing: object, Add support for Log with filenames. Fixes src-d#826 (src-d#979)

plumbing: object, Add support for Log with filenames. Fixes src-d#826

* object: get object size without reading whole object

Signed-off-by: Jeremy Stribling <[email protected]>

* tree: add a Size() method for getting plaintext size

Without reading the entire object into memory.

Signed-off-by: Jeremy Stribling <[email protected]>

* filesystem: add a new test for EncodedObjectSize

Suggested by taruti.

Signed-off-by: Jeremy Stribling <[email protected]>

* repository: allow open non-bare repositories as bare

Signed-off-by: Máximo Cuadros <[email protected]>

* use remote name in fetch while clone, test

Signed-off-by: Máximo Cuadros <[email protected]>

* references: sort: compare author timestamps when commit timestamps are equal, test

Signed-off-by: Máximo Cuadros <[email protected]>

* teach ResolveRevision how to look up annotated tags, test

Signed-off-by: Máximo Cuadros <[email protected]>

* teach ResolveRevision how to look up annotated tags, test

Signed-off-by: Máximo Cuadros <[email protected]>

* packfile: add comment on GetSizeByOffset

Suggested by mcuadros.

Issue: src-d#982
Signed-off-by: Jeremy Stribling <[email protected]>

* blame: fix edge case with missing \n in content length causing mismatched length error

Signed-off-by: Máximo Cuadros <[email protected]>

* repository: improve CheckoutOption.Hash doc

Signed-off-by: Máximo Cuadros <[email protected]>

* remote: use reference deltas on push when the remote server does not
support offset deltas

Signed-off-by: Benjamin Ash <[email protected]>

* Fixed a typo. (src-d#989)

README: Fixed a typo.

* Enables building on openbsd, dragonfly bsd and solaris

Signed-off-by: Yuce Tekol <[email protected]>

* plumbing/format/packfile: Fix broken "thin" packfile support. Fixes src-d#991

Signed-off-by: Javier Peletier <[email protected]>

* plumbing: ReferenceName constructors

Signed-off-by: Máximo Cuadros <[email protected]>

* examples  & documentation: PlainClone with Basic Authentication (Password & Access Token) (src-d#990)

examples: PlainClone with Basic Authentication (Password & Access Token)

* add StackOverflow to support channels

Since we are not redirecting users to StackOverflow for support
questions, it makes sense to add it to the official support channels.

Signed-off-by: Santiago M. Mola <[email protected]>

* plumbing: transport/http, Add missing host/port on redirect. Fixes src-d#820

Signed-off-by: Dave Henderson <[email protected]>

* Fix spelling and grammar in docs and example

Signed-off-by: Lukasz Kokot <[email protected]>

* update gcfg dependency to v1.4.0

Signed-off-by: Dave Henderson <[email protected]>

* repository: added cleanup for the PlainCloneContext()

Signed-off-by: Bartek Jaroszewski <[email protected]>

* improve cleanup implementation, add more tests

Signed-off-by: Santiago M. Mola <[email protected]>

* Update LICENSE

Signed-off-by: Máximo Cuadros <[email protected]>

* http: improve TokenAuth documentation

Users are often confused with TokenAuth, since it might look that it
should be used with GitHub's OAuth tokens. But that is not the case.

TokenAuth implements HTTP bearer authentication. Most git servers will
use HTTP basic authentication (user+passwords) even for OAuth tokens.

Signed-off-by: Santiago M. Mola <[email protected]>

* plumbing: ssh, Fix flaky test TestAdvertisedReferencesNotExists. Fixes src-d#969

Signed-off-by: Colton McCurdy <[email protected]>

* repository: Fix RefSpec for a single tag. Fixes src-d#960

Signed-off-by: Fedor Korotkov <[email protected]>

* storage/filesystem: Added reindex method to  reindex packfiles

Signed-off-by: Javier Peletier <[email protected]>

* plumbing/format/packfile: Added thin pack test

Signed-off-by: Javier Peletier <[email protected]>

* Remove unused method (src-d#1022)

Signed-off-by: Antonio Jesus Navarro Perez <[email protected]>

* plumbing: format/index: support for EOIE extension, by default on git v2.2.0

Signed-off-by: Máximo Cuadros <[email protected]>

* repository: fix plain clone error handling regression

PR src-d#1008 introduced a regression by changing the errors returned by
PlainClone when a repository did not exist.

This change goes back to returned errors as they were in v4.7.0.

Fixes src-d#1027

Signed-off-by: Santiago M. Mola <[email protected]>

* plumbing: format/packfile, performance optimizations for reading large commit histories (src-d#963)

Signed-off-by: Filip Navara <[email protected]>
  • Loading branch information
Snehal1112 authored Dec 1, 2018
1 parent 01c428c commit 3a5416f
Show file tree
Hide file tree
Showing 170 changed files with 10,167 additions and 2,209 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
coverage.out
*~
coverage.txt
profile.out
4 changes: 2 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
language: go

go:
- 1.8.x
- 1.9.x
- "1.10"
- "1.11"

go_import_path: gopkg.in/src-d/go-git.v4

Expand Down
3 changes: 2 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ This can be done easily using the [`-s`](https://github.com/git/git/blob/b2c150d

The official support channels, for both users and contributors, are:

- GitHub [issues](https://github.com/src-d/go-git/issues)*
- [StackOverflow go-git tag](https://stackoverflow.com/questions/tagged/go-git) for user questions.
- GitHub [Issues](https://github.com/src-d/go-git/issues)* for bug reports and feature requests.
- Slack: #go-git room in the [source{d} Slack](https://join.slack.com/t/sourced-community/shared_invite/enQtMjc4Njk5MzEyNzM2LTFjNzY4NjEwZGEwMzRiNTM4MzRlMzQ4MmIzZjkwZmZlM2NjODUxZmJjNDI1OTcxNDAyMmZlNmFjODZlNTg0YWM)

*Before opening a new issue or submitting a new pull request, it's helpful to
Expand Down
4 changes: 2 additions & 2 deletions LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright 2017 Sourced Technologies, S.L.
Copyright 2018 Sourced Technologies, S.L.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand All @@ -198,4 +198,4 @@
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
limitations under the License.
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,16 @@

*go-git* is a highly extensible git implementation library written in **pure Go**.

It can be used to manipulate git repositories at low level *(plumbing)* or high level *(porcelain)*, through an idiomatic Go API. It also supports several type of storage, such as in-memory filesystems, or custom implementations thanks to the [`Storer`](https://godoc.org/gopkg.in/src-d/go-git.v4/plumbing/storer) interface.
It can be used to manipulate git repositories at low level *(plumbing)* or high level *(porcelain)*, through an idiomatic Go API. It also supports several types of storage, such as in-memory filesystems, or custom implementations thanks to the [`Storer`](https://godoc.org/gopkg.in/src-d/go-git.v4/plumbing/storer) interface.

It's being actively develop since 2015 and is being use extensively by [source{d}](https://sourced.tech/) and [Keybase](https://keybase.io/blog/encrypted-git-for-everyone), and by many other libraries and tools.
It's being actively developed since 2015 and is being used extensively by [source{d}](https://sourced.tech/) and [Keybase](https://keybase.io/blog/encrypted-git-for-everyone), and by many other libraries and tools.

Comparison with git
-------------------

*go-git* aims to be fully compatible with [git](https://github.com/git/git), all the *porcelain* operations are implemented to work exactly as *git* does.

*git* is a humongous project with years of development by thousands of contributors, making it challenging for *go-git* implement all the features. You can find a comparison of *go-git* vs *git* in the [compatibility documentation](COMPATIBILITY.md).
*git* is a humongous project with years of development by thousands of contributors, making it challenging for *go-git* to implement all the features. You can find a comparison of *go-git* vs *git* in the [compatibility documentation](COMPATIBILITY.md).


Installation
Expand All @@ -24,12 +24,12 @@ The recommended way to install *go-git* is:
go get -u gopkg.in/src-d/go-git.v4/...
```

> We use [gopkg.in](http://labix.org/gopkg.in) for having a versioned API, this means that when `go get` clones the package, is the latest tag matching `v4.*` cloned and not the master branch.
> We use [gopkg.in](http://labix.org/gopkg.in) to version the API, this means that when `go get` clones the package, it's the latest tag matching `v4.*` that is cloned and not the master branch.
Examples
--------

> Please note that the functions `CheckIfError` and `Info` used in the examples are from the [examples package](https://github.com/src-d/go-git/blob/master/_examples/common.go#L17) just to be used in the examples.
> Please note that the `CheckIfError` and `Info` functions used in the examples are from the [examples package](https://github.com/src-d/go-git/blob/master/_examples/common.go#L17) just to be used in the examples.

### Basic example
Expand Down Expand Up @@ -71,7 +71,7 @@ r, err := git.Clone(memory.NewStorage(), nil, &git.CloneOptions{

CheckIfError(err)

// Gets the HEAD history from HEAD, just like does:
// Gets the HEAD history from HEAD, just like this command:
Info("git log")

// ... retrieves the branch pointed by HEAD
Expand Down Expand Up @@ -110,7 +110,7 @@ Date: Fri Nov 11 13:23:22 2016 +0100
...
```

You can find this [example](_examples/log/main.go) and many others at the [examples](_examples) folder
You can find this [example](_examples/log/main.go) and many others in the [examples](_examples) folder.

Contribute
----------
Expand Down
4 changes: 4 additions & 0 deletions _examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ Here you can find a list of annotated _go-git_ examples:
- [showcase](showcase/main.go) - A small showcase of the capabilities of _go-git_
- [open](open/main.go) - Opening a existing repository cloned by _git_
- [clone](clone/main.go) - Cloning a repository
- [username and password](clone/auth/basic/username_password/main.go) - Cloning a repository
using a username and password
- [personal access token](clone/auth/basic/access_token/main.go) - Cloning
a repository using a GitHub personal access token
- [commit](commit/main.go) - Commit changes to the current branch to an existent repository
- [push](push/main.go) - Push repository to default remote (origin)
- [pull](pull/main.go) - Pull changes from a remote repository
Expand Down
2 changes: 1 addition & 1 deletion _examples/branch/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ func main() {

// Create a new plumbing.HashReference object with the name of the branch
// and the hash from the HEAD. The reference name should be a full reference
// name and now a abbreviated one, as is used on the git cli.
// name and not an abbreviated one, as is used on the git cli.
//
// For tags we should use `refs/tags/%s` instead of `refs/heads/%s` used
// for branches.
Expand Down
4 changes: 2 additions & 2 deletions _examples/checkout/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ func main() {
})
CheckIfError(err)

// ... retrieving the commit being pointed by HEAD, it's shows that the
// repository is poiting to the giving commit in detached mode
// ... retrieving the commit being pointed by HEAD, it shows that the
// repository is pointing to the giving commit in detached mode
Info("git show-ref --head HEAD")
ref, err = r.Head()
CheckIfError(err)
Expand Down
40 changes: 40 additions & 0 deletions _examples/clone/auth/basic/access_token/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
package main

import (
"fmt"
"os"

git "gopkg.in/src-d/go-git.v4"
. "gopkg.in/src-d/go-git.v4/_examples"
"gopkg.in/src-d/go-git.v4/plumbing/transport/http"
)

func main() {
CheckArgs("<url>", "<directory>", "<github_access_token>")
url, directory, token := os.Args[1], os.Args[2], os.Args[3]

// Clone the given repository to the given directory
Info("git clone %s %s", url, directory)

r, err := git.PlainClone(directory, false, &git.CloneOptions{
// The intended use of a GitHub personal access token is in replace of your password
// because access tokens can easily be revoked.
// https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/
Auth: &http.BasicAuth{
Username: "abc123", // yes, this can be anything except an empty string
Password: token,
},
URL: url,
Progress: os.Stdout,
})
CheckIfError(err)

// ... retrieving the branch being pointed by HEAD
ref, err := r.Head()
CheckIfError(err)
// ... retrieving the commit object
commit, err := r.CommitObject(ref.Hash())
CheckIfError(err)

fmt.Println(commit)
}
37 changes: 37 additions & 0 deletions _examples/clone/auth/basic/username_password/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
package main

import (
"fmt"
"os"

git "gopkg.in/src-d/go-git.v4"
. "gopkg.in/src-d/go-git.v4/_examples"
"gopkg.in/src-d/go-git.v4/plumbing/transport/http"
)

func main() {
CheckArgs("<url>", "<directory>", "<github_username>", "<github_password>")
url, directory, username, password := os.Args[1], os.Args[2], os.Args[3], os.Args[4]

// Clone the given repository to the given directory
Info("git clone %s %s", url, directory)

r, err := git.PlainClone(directory, false, &git.CloneOptions{
Auth: &http.BasicAuth{
Username: username,
Password: password,
},
URL: url,
Progress: os.Stdout,
})
CheckIfError(err)

// ... retrieving the branch being pointed by HEAD
ref, err := r.Head()
CheckIfError(err)
// ... retrieving the commit object
commit, err := r.CommitObject(ref.Hash())
CheckIfError(err)

fmt.Println(commit)
}
6 changes: 3 additions & 3 deletions _examples/commit/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@ import (
"gopkg.in/src-d/go-git.v4/plumbing/object"
)

// Basic example of how to commit changes to the current branch to an existent
// Basic example of how to commit changes to the current branch to an existing
// repository.
func main() {
CheckArgs("<directory>")
directory := os.Args[1]

// Opens an already existent repository.
// Opens an already existing repository.
r, err := git.PlainOpen(directory)
CheckIfError(err)

Expand All @@ -44,7 +44,7 @@ func main() {

fmt.Println(status)

// Commits the current staging are to the repository, with the new file
// Commits the current staging area to the repository, with the new file
// just created. We should provide the object.Signature of Author of the
// commit.
Info("git commit -m \"example go-git commit\"")
Expand Down
2 changes: 1 addition & 1 deletion _examples/log/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ func main() {
})
CheckIfError(err)

// Gets the HEAD history from HEAD, just like does:
// Gets the HEAD history from HEAD, just like this command:
Info("git log")

// ... retrieves the branch pointed by HEAD
Expand Down
2 changes: 1 addition & 1 deletion _examples/open/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ func main() {
CheckArgs("<path>")
path := os.Args[1]

// We instance a new repository targeting the given path (the .git folder)
// We instanciate a new repository targeting the given path (the .git folder)
r, err := git.PlainOpen(path)
CheckIfError(err)

Expand Down
2 changes: 1 addition & 1 deletion _examples/pull/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ func main() {
CheckArgs("<path>")
path := os.Args[1]

// We instance a new repository targeting the given path (the .git folder)
// We instance\iate a new repository targeting the given path (the .git folder)
r, err := git.PlainOpen(path)
CheckIfError(err)

Expand Down
2 changes: 1 addition & 1 deletion _examples/showcase/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import (
// - Get the HEAD reference
// - Using the HEAD reference, obtain the commit this reference is pointing to
// - Print the commit content
// - Using the commit, iterate all its files and print them
// - Using the commit, iterate over all its files and print them
// - Print all the commit history with commit messages, short hash and the
// first line of the commit message
func main() {
Expand Down
2 changes: 1 addition & 1 deletion _examples/storage/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@


### and what this means ...
*git* has as very well defined storage system, the `.git` directory, present on any repository. This is the place where `git` stores al the [`objects`](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects), [`references`](https://git-scm.com/book/es/v2/Git-Internals-Git-References) and [`configuration`](https://git-scm.com/docs/git-config#_configuration_file). This information is stored in plain files.
*git* has a very well defined storage system, the `.git` directory, present on any repository. This is the place where `git` stores all the [`objects`](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects), [`references`](https://git-scm.com/book/es/v2/Git-Internals-Git-References) and [`configuration`](https://git-scm.com/docs/git-config#_configuration_file). This information is stored in plain files.

Our original **go-git** version was designed to work in memory, some time after we added support to read the `.git`, and now we have added support for fully customized [storages](https://godoc.org/gopkg.in/src-d/go-git.v4/storage#Storer).

Expand Down
2 changes: 1 addition & 1 deletion _examples/tag/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ func main() {
CheckArgs("<path>")
path := os.Args[1]

// We instance a new repository targeting the given path (the .git folder)
// We instanciate a new repository targeting the given path (the .git folder)
r, err := git.PlainOpen(path)
CheckIfError(err)

Expand Down
30 changes: 24 additions & 6 deletions blame.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"fmt"
"strconv"
"strings"
"time"
"unicode/utf8"

"gopkg.in/src-d/go-git.v4/plumbing"
Expand Down Expand Up @@ -106,24 +107,41 @@ type Line struct {
Author string
// Text is the original text of the line.
Text string
// Date is when the original text of the line was introduced
Date time.Time
// Hash is the commit hash that introduced the original line
Hash plumbing.Hash
}

func newLine(author, text string) *Line {
func newLine(author, text string, date time.Time, hash plumbing.Hash) *Line {
return &Line{
Author: author,
Text: text,
Hash: hash,
Date: date,
}
}

func newLines(contents []string, commits []*object.Commit) ([]*Line, error) {
if len(contents) != len(commits) {
return nil, errors.New("contents and commits have different length")
lcontents := len(contents)
lcommits := len(commits)

if lcontents != lcommits {
if lcontents == lcommits-1 && contents[lcontents-1] != "\n" {
contents = append(contents, "\n")
} else {
return nil, errors.New("contents and commits have different length")
}
}
result := make([]*Line, 0, len(contents))

result := make([]*Line, 0, lcontents)
for i := range contents {
l := newLine(commits[i].Author.Email, contents[i])
result = append(result, l)
result = append(result, newLine(
commits[i].Author.Email, contents[i],
commits[i].Author.When, commits[i].Hash,
))
}

return result, nil
}

Expand Down
32 changes: 32 additions & 0 deletions blame_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package git

import (
"gopkg.in/src-d/go-git.v4/plumbing"
"gopkg.in/src-d/go-git.v4/plumbing/object"

. "gopkg.in/check.v1"
"gopkg.in/src-d/go-git-fixtures.v3"
Expand All @@ -13,6 +14,31 @@ type BlameSuite struct {

var _ = Suite(&BlameSuite{})

func (s *BlameSuite) TestNewLines(c *C) {
h := plumbing.NewHash("ce9f123d790717599aaeb76bc62510de437761be")
lines, err := newLines([]string{"foo"}, []*object.Commit{{
Hash: h,
Message: "foo",
}})

c.Assert(err, IsNil)
c.Assert(lines, HasLen, 1)
c.Assert(lines[0].Text, Equals, "foo")
c.Assert(lines[0].Hash, Equals, h)
}

func (s *BlameSuite) TestNewLinesWithNewLine(c *C) {
lines, err := newLines([]string{"foo"}, []*object.Commit{
{Message: "foo"},
{Message: "bar"},
})

c.Assert(err, IsNil)
c.Assert(lines, HasLen, 2)
c.Assert(lines[0].Text, Equals, "foo")
c.Assert(lines[1].Text, Equals, "\n")
}

type blameTest struct {
repo string
rev string
Expand All @@ -32,6 +58,10 @@ func (s *BlameSuite) TestBlame(c *C) {
obt, err := Blame(commit, t.path)
c.Assert(err, IsNil)
c.Assert(obt, DeepEquals, exp)

for i, l := range obt.Lines {
c.Assert(l.Hash.String(), Equals, t.blames[i])
}
}
}

Expand All @@ -53,6 +83,8 @@ func (s *BlameSuite) mockBlame(c *C, t blameTest, r *Repository) (blame *BlameRe
l := &Line{
Author: commit.Author.Email,
Text: lines[i],
Date: commit.Author.When,
Hash: commit.Hash,
}
blamedLines = append(blamedLines, l)
}
Expand Down
Loading

0 comments on commit 3a5416f

Please sign in to comment.