Git clone fails when building from Gatsby cache

[jidicula]

There's a fair chance I'm taking the wrong approach here, so let me know if I'm doing something wrong somewhere...

I have 2 GitHub repos, one is for a Gatsby site at http://forcepush.tech/ and the other is for my resume. I have my Gatsby site deployed on Netlify, and I've set up build optimizations to speed up my build, so my build command in package.json is currently GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages and my Netlify build command is npm run build. In my gatsby-config.js, I have the gatsby-source-git plugin configured as follows:

    {
      resolve: `gatsby-source-git`,
      options: {
        name: `jidicula-resume`,
        remote: `https://github.com/jidicula/jidicula-resume.git`,
        branch: `master`,
        local: "./public/jidicula-resume",
        // Only import the compiled PDF.
        patterns: `**.pdf`,
      },

This clones the resume repo into the site's build directory (public/). However, when the build is triggered (by a hook in my resume repo for example) and is building from cache, I see the following error from the gatsby-source-filesystem plugin, and the repository doesn't get cloned again:

1:38:45 PM:   Error: fatal: No remote configured to list refs from.
1:38:45 PM:   
1:38:45 PM:   - promise.js:90 toError
1:38:45 PM:     [repo]/[simple-git]/promise.js:90:14
1:38:45 PM:   
1:38:45 PM:   - promise.js:61 
1:38:45 PM:     [repo]/[simple-git]/promise.js:61:36
1:38:45 PM:   
1:38:45 PM:   - git.js:725 Git.<anonymous>
1:38:45 PM:     [repo]/[simple-git]/src/git.js:725:18
1:38:45 PM:   
1:38:45 PM:   - git.js:1475 Function.Git.fail
1:38:45 PM:     [repo]/[simple-git]/src/git.js:1475:18
1:38:45 PM:   
1:38:45 PM:   - git.js:1433 fail
1:38:45 PM:     [repo]/[simple-git]/src/git.js:1433:20
1:38:45 PM:   
1:38:45 PM:   - git.js:1442 
1:38:45 PM:     [repo]/[simple-git]/src/git.js:1442:16
1:38:45 PM:   
1:38:45 PM:   - task_queues.js:97 processTicksAndRejections
1:38:45 PM:     internal/process/task_queues.js:97:5
1:38:45 PM:

You can see the entire build log here: https://app.netlify.com/sites/forcepush/deploys/5f357a83404af32690b43c31

I've since set up a workaround build command to remove the repo first before attempting a build: rm -rf public/jidicula-resume && npm run build

[jgaehring]

I experienced the same issue, as I mentioned in farmOS/farmOS.org#16, and the same workaround fixed it for me as well. Glad you documented that, @jidicula, much appreciated!

Seems like it shouldn't be too hard to catch this kind of error internally, no? Maybe somewhere around this line...?

gatsby-source-git/gatsby-node.js

Line 29 in a7fee1a

await Git().clone(remote, path, opts);

@stevetweeddale, are you taking PR's for this kind of thing?

[stevetweeddale]

Sorry @jidicula I never looked at this. It looks like you don't actually need to ingest anything from that repo into the gatsby graph, you just want a remote (public on github) file served up. In that case, I'd consider hotlinking directly – jsdelivr will happily serve up the latest version at a static URL. If I wanted a prettier URL on my own domain, I'd first try a netlify proxy or else just a redirect.

As for the specific issue raised here, that things get borked if you clone the repo to the public folder – I can't say I was envisioning this use-case! Without spending some time reproducing this it's not clear to me exactly what's going on to produce that error. I spotted @jidicula's blog where he suggests:

The cause of this was one of the key features of Gatsby and Netlify: caches of previous builds. The public/ directory and the .cache remains, with only the changed components updating. The Git plugin essentially tried to clone the same repo into the same location, where a repo of that name already existed, so naturally it would fail.

But there already is code intended to handle the repo already existing in the configured location – which works fine with the default directory within the cache folder. So I would guess this is something to do with the fact that it's in the public folder and/or that the GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES build option is used, neither of which I've tried this plugin with before.

@jgaehring So this error is actually already caught by the parent function:

gatsby-source-git/gatsby-node.js

Lines 64 to 68 in a7fee1a

	try {
	repo = await getRepo(localPath, remote, branch);
	} catch (e) {
	return reporter.error(e);
	}

The question is how to handle it in a better way than just reporting it! And that of course relies on understanding what the actual problem is! I can see that trashing the repo before running a build is a workaround, but I'd want to understand why before trying to bake a solution into this plugin I think.

I guess it might also be worth just hammering out what the use-case is for cloning into the public directory, and if it's one this plugin should even try to support. If it's just that you want some static assets inside your public folder, I would imagine you could pretty easily just write an npm command to run the git clone after the gastby build? Or are there features of this plugin that people are relying on when cloning into the public dir?

[jgaehring]

But there already is code intended to handle the repo already existing in the configured location – which works fine with the default directory within the cache folder. So I would guess this is something to do with the fact that it's in the public folder and/or that the GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES build option is used, neither of which I've tried this plugin with before.

I should note that I am using the default .cache/gatsby-source-git/, so this may be a more general issue.

The question is how to handle it in a better way than just reporting it! And that of course relies on understanding what the actual problem is!

Agreed! Happy to provide more background or debugging info. My plugin options are a little convoluted, but this is essentially what they look like:

const pluginConfigs = [
  {
    resolve: "gatsby-source-git",
    options: {
      name: 'farmOS',
      remote: 'https://github.com/farmOS/farmOS.git',
      branch: '2.x',
      patterns: 'docs/**',
    },
  },
  {
    resolve: "gatsby-source-git",
    options: {
      name: 'farmOS.js',
      remote: 'https://github.com/farmOS/farmOS.js.git',
      branch: 'main',
      patterns: 'docs/**',
    },
  },
];

I'm using the plugin for two repositories currently, and plan to add more. So far, they both fail if they fail at all.

Looking at that error message a little more carefully (No remote configured to list refs from), I would have to guess the point of failure is the call to the listRemotes method of the repo instance in isAlreadyCloned, not the call to Git.clone() as I originally though. The message itself seems to be propagating from git's ls-remote command, which I assume is being run by listRemotes. I still don't know why ls-remotes doesn't work in the Netlify environment, except that it seems like it's losing track of the remotes somehow between builds.

I still wonder if it might just be safer to wrap that call to listRemotes in a try/catch, and just return false from isAlreadyCloned if it fails. That would also require a little rearranging of the control flow in getRepo. It's still not a very elegant solution, but it seems better than blowing away the cache every time w/o even trying it. Happy to whip up a quick PR of what that might look like, although I'm not sure I know the best way to test it.

[stevetweeddale]

Thanks for the additional info! 2 more quick questions:

Looking at your repo, it looks like maybe you're not using the GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true build option either – is that right?

Secondly, you said,

I still don't know why ls-remotes doesn't work in the Netlify environment, except that it seems like it's losing track of the remotes somehow between builds.

Is that to say that it's not something you're seeing locally? Is it pretty consistent in the Netlify environment?

[jgaehring]

Looking at your repo, it looks like maybe you're not using the GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true build option either – is that right?

Not that I'm aware of? My build command is just gatsby build, no flags, although for the moment, I'm prefixing that with rm -rf .cache/gatsby-source-git/ &&.

Is that to say that it's not something you're seeing locally?

That's correct.

Is it pretty consistent in the Netlify environment?

Fairly. It doesn't seem to happen when my Netlify builds are triggered by changes I've pushed to the main Gatsby repository (maybe b/c a lot of those involve changes that would force Gatsby to clear the cache?), but I have a GitHub Action in my source repositories that triggers a Netlify build when there are changes to their documentation. Those are the builds that usually fail if I don't clear the cache first. So yea, all kinds of things could be going on in the Netlify environment that I'm not even aware of.

By the way, thanks for the attention you're giving this! ☺️