3.0.2: failing tests during packaging (tests should be isolated and independent)

[dvzrv]

Hi!

I'm currently packaging gitpython for Arch Linux and am about to update to version 3.0.2.
However, during packaging, I tried to run the tests and many of them fail (see attachment).

We're currently on

• git 2.23.0
• python-gitdb 2.0.5
• python-smmap 2.0.5
• python-ddt 1.2.1
• python-nose 1.3.7

I'm running the tests (after build, before install) with nosetests -vd.
I'm using the pypi provided sources: https://files.pythonhosted.org/packages/51/46/993beca52f3b609d148071e129235b866626eeb6056f2faffb41d9d727a7/GitPython-3.0.2.tar.gz

I would be very grateful for any hint!
python-gitpython-3.0.2-check.log

[Byron]

Unfortunately, the GitPython tests indeed have very specific requirements. They expect to be run in their own git repository, with certain additional requirements being met.
Fortunately there is a script as run by travis which sets up these requirements.

Something I am not sure of is if it's enough to simply initialise a new git repository from the sources in the tarball.

I hope that helps, and please let me know if it worked or if there are additional issues.

[dvzrv]

I have tried to run the tests for the 3.0.3 build, but still get a lot of failing and erroring tests.

As I'm building a package, I can not rely on downloading from several different sources (am I supposed to git clone all of gitpython recursively just for testing purposes?).

That being said, there are plenty of tests, that hardcode specific commits, author names, git tags, files(!), etc. to test against.
This is really bad for reproducibility and makes all of these tests basically useless for downstreams (although they would be potentially be the ones reporting integration problems, etc.)!

This is how I have to prepare the test suite, when using a tarball of gitpython (while still having 34 erroring and 9 failing tests):

  local TEST_TMPDIR=$(mktemp -d -t gitpython.XXX)
  local GITDB_TMPDIR=$(mktemp -d -t gitdb.XXX)
  local SMMAP_TMPDIR=$(mktemp -d -t smmap.XXX)
  git config --global user.name "Test User"
  git config --global user.email "[email protected]"
  (                                
    cd "$SMMAP_TMPDIR"
    mkdir -vp bare           
    (
      cd bare                
      git init --bare
    )              
    git clone bare clone
    cd clone                                                      
    for commit in {1..50}; do                                                                                                                                                         
      touch "file${commit}"
      git add "file${commit}"
      git commit -m "file${commit}"
    done                   
    git push                            
  )                                     
  (                                      
    cd "$GITDB_TMPDIR"                      
    mkdir -vp bare                                             
    (                                      
      cd bare                                                           
      git init --bare
     )
    git clone bare clone                           
    cd clone                                                                                                                                                                          
                                                                                                                                                                                      
    mkdir -vp gitdb/ext
    git submodule add --name 'smmap' "${SMMAP_TMPDIR}/clone" gitdb/ext/smmap
    git submodule init gitdb/ext/smmap                                                                                                                                                                                                                                                                                                                                      
    git add gitdb/ext/smmap                                                                                                                                                           
    git commit -m "Adding smmap submodule"                                                                                                                                            
                                                                                                                                                                                      
    for commit in {1..50}; do                                                                                                                                                         
      touch "file${commit}"                                                                                                                                                           
      git add "file${commit}"                                                              
      git commit -m "file${commit}"                                                        
    done                                                              
    git push                                     
  )                                                                                        
  (                                           
    cd "$TEST_TMPDIR"                      
    mkdir -vp bare                              
    (                                                                   
      cd bare                                                                                                                                                                         
      git init --bare                                                             
    )                                                                                      
    git clone bare clone                                  
    cd clone                                                                               
                                                                                           
    mkdir -vp git/ext                                      
    git submodule add --name 'gitdb' "${GITDB_TMPDIR}/clone" git/ext/gitdb
    git submodule update --init --recursive git/ext/gitdb                                  
    git add git/ext/gitdb  
    git commit -m "Adding gitdb submodule"                       
                                                                                           
    # test hardcodes this file to test against
    touch AUTHORS
    git add AUTHORS                                    
    git commit -m "Adding AUTHORS."
    # test hardcodes this file to test against
    touch MANIFEST.in         
    git add MANIFEST.in          
    git commit -m "Adding MANIFEST.in."  
                                                                                                                                                                                      
    for commit in {1..50}; do                                                                                                                                                         
      touch "file${commit}"                                                                                                                                                           
      git add "file${commit}"
      git commit -m "file${commit}"                                                                                                                                                   
    done                                                                                   
    git tag -am '0.1.4' 0.1.4                                               
    for commit in {51..100}; do                    
      touch "file${commit}"
      git add "file${commit}"                                                              
      git commit -m "file${commit}"                                                        
    done                     
    git tag -am '0.1.5' 0.1.5
    for commit in {101..150}; do
      touch "file${commit}"                                                                
      git add "file${commit}"
      git commit -m "file${commit}"
    done                                         
    git tag -am '0.1.6' 0.1.6                 
    for commit in {151..160}; do              
      touch "file${commit}"                
      git add "file${commit}"                   
      git commit -m "file${commit}"
    done              
    git tag -am '0.3.5' 0.3.5
                                                                                           
    git tag __testing_point__
    git push                                                                               
    git push --tags  
  )                                                                       
  export GIT_PYTHON_TEST_GIT_REPO_BASE="${TEST_TMPDIR}/clone/.git"
  nosetests -vd

python-gitpython-3.0.3-build.log

I guess at this point it becomes more than evident, that this should be part of the test suite, creating assets for itself and removing all the specific hardcoding in the tests (while not relying on this repository and all of its specific components).

[Byron]

I absolutely agree that the way GitPython tests are done violates a lot of best practices: they are usually too long and convoluted, and downright assume they have access to a recursive clone of their own repository.

Back in the days, it seemed 'easier', not taking into consideration what this would mean further down the road stream. From your comment I can get a hint of the nightmare this seems to be, and I am sorry for the trouble it may cause.

Looking at the script above I would hope that eventually it manages to become a PR which helps make tests independent of their own repository. Within the test framework, everything boils down to two (?) base classes that initialise a repository for reading and/or writing. These should be one of the few places which may need adjustments on python-side.

[dvzrv]

On 2019-10-21 10:37:06 (-0700), Sebastian Thiel wrote: Back in the days, it seemed 'easier', not taking into consideration what this would mean further down the ~~road~~ stream. From your comment I can get a hint of the nightmare this seems to be, and I am sorry for the trouble it may cause.

It is a little nightmarish indeed, but I've also seen worse (!) tbh! :)

Looking at the script above I would hope that eventually it manages to become a PR which helps make tests independent of their own repository. Within the test framework, everything boils down to two (_?_) base classes that initialise a repository for reading and/or writing. These should be one of the few places which may need adjustments on python-side.

I'll try to look into it, when I find the time. Got my hands pretty full with other things (mainly packages) though. Additionally, I haven't worked with nose, but mainly pytest in the past.

[mcepl]

We used to have working tests for openSUSE/Factory with some tests skipped (mainly those which require network access), but with 3.1.31 (checkout of the tag from GitHub) it failed again (93 tests failed, 3 errors).

complete build log

[Byron]

I don't know what changed exactly or why the tests would fail now, but judging by the log the failures seem to be due to some foundational issue.

I recommend following the instructions on how to run tests if not done - a standard clone is assumed with the HEAD not detached.