Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git server doesn't work with Win32-OpenSSH. Enclosed, please find three defects which I isolated and worked around.. #752

Open
razrjk opened this issue Jun 4, 2017 · 52 comments

Comments

@razrjk
Copy link

razrjk commented Jun 4, 2017

I pulled version: 0.0.14.0 on May 19. I just fetched version: 0.0.15.0 (June 4) today to compare. I don't believe the below issues have been corrected.

Git using Win32-OpenSSH ssh server runs as a nopty service. There are two defects in the nopty code:

  1. shell-host.c passes through to cmd.exe, single quotes as literals. The literal single quotes become part of the file path name, resulting in an incorrect file path.
  2. The service loop in shell-host.c, which passes its stdin to the child stdin is coded with incorrect file handles. Part of the input is currently passed back to sshd.

A temporary workaround for issue 1 is to replace all single quotes with double quotes just before the call to invoke cmd /c:

@@ -1136,7 +1136,21 @@ start_withno_pty(wchar_t *command)
	if (command) {
		GOTO_CLEANUP_ON_ERR(wcscat_s(cmd, MAX_CMD_LEN, L" /c"));
		GOTO_CLEANUP_ON_ERR(wcscat_s(cmd, MAX_CMD_LEN, L" "));

	+	int i = 0;
	+	while (command[i] != L'\0')
	+	{
	+		if (command[i] == L'\'' )
	+			command[i] = L'\"';
	+		i++;
	+	}
		GOTO_CLEANUP_ON_ERR(wcscat_s(cmd, MAX_CMD_LEN, command));

A fix for issue 2 is to use the child process file handle instead of the sshd file handle:

@@ -1184,7 +1198,7 @@ start_withno_pty(wchar_t *command)
			/* for backspace, we need to send space and another backspace for visual erase */
			if (buf[i] == '\b') {
				if (in_cmd_len > 0) {
-					GOTO_CLEANUP_ON_FALSE(WriteFile(pipe_out, "\b \b", 3, &wr, NULL));
+					GOTO_CLEANUP_ON_FALSE(WriteFile(child_pipe_write, "\b \b", 3, &wr, NULL));
					in_cmd_len--;
				}
				i++;
@@ -1194,10 +1208,10 @@ start_withno_pty(wchar_t *command)
			/* For CR and LF */
			if ((buf[i] == '\r') || (buf[i] == '\n')) {
				/* TODO - do a much accurate mapping */
-				GOTO_CLEANUP_ON_FALSE(WriteFile(pipe_out, buf + i, 1, &wr, NULL));
+				GOTO_CLEANUP_ON_FALSE(WriteFile(child_pipe_write, buf + i, 1, &wr, NULL));
				if ((buf[i] == '\r') && ((i == rd - 1) || (buf[i + 1] != '\n'))) {
					buf[i] = '\n';
-					GOTO_CLEANUP_ON_FALSE(WriteFile(pipe_out, buf + i, 1, &wr, NULL));
+					GOTO_CLEANUP_ON_FALSE(WriteFile(child_pipe_write, buf + i, 1, &wr, NULL));
				}
				in_cmd[in_cmd_len] = buf[i];
				in_cmd_len++;
@@ -1207,7 +1221,7 @@ start_withno_pty(wchar_t *command)
				continue;
			}
 
-			GOTO_CLEANUP_ON_FALSE(WriteFile(pipe_out, buf + i, 1, &wr, NULL));
+			GOTO_CLEANUP_ON_FALSE(WriteFile(child_pipe_write, buf + i, 1, &wr, NULL));
			in_cmd[in_cmd_len] = buf[i];
			in_cmd_len++;
			if (in_cmd_len == MAX_CMD_LEN - 1) {

Finally, there is defect and debug issue in signal_sigchld.c. The zombied children count can become too large, which causes a seg fault. When debugging sshd.exe /ddd, debug3() is writing during signal processing which corrupts the stack. Debug3() should not write while running on an exception frame. A check for incorrect zombie count should be added:

@@ -104,14 +104,20 @@ sw_child_to_zombie(DWORD index)
	DWORD last_non_zombie, zombie_pid;
	HANDLE zombie_handle;
 
-	debug3("zombie'ing child at index %d, %d zombies of %d", index,
-		children.num_zombies, children.num_children);
+	//debug3("zombie'ing child at index %d, %d zombies of %d", index,
+		//children.num_zombies, children.num_children);
 
	if (index >= children.num_children) {
		errno = EINVAL;
		return -1;
	}
-
+	//debug3(" %x zombies,  %x children",children.num_zombies, children.num_children);
+	if((children.num_children - children.num_zombies) == 0)
+	{
+		//debug2("children == zombies, can't make more zombies");
+		errno = EINVAL;
+		return -1;
+	}
	last_non_zombie = children.num_children - children.num_zombies - 1;
	if (last_non_zombie != index) {
		/* swap */

I'm running on Windows 10 and Windows 7 systems.

Extreme care must be taken with search paths when debugging a git server using ssh protocol on Windows, especially if cygwin and mingw, etc., are also installed on the system This is true for the client and server side. Multiple executables for ssh, sshd, and git commands can cause problems if close attention is not paid to search paths.

@manojampalam
Copy link
Contributor

Thanks for the detailed report and proposed changes. Found some details here on how to setup Git server using over SSH - https://www.systutorials.com/366/set-up-git-server-through-ssh-connection/

@razrjk
Copy link
Author

razrjk commented Jun 5, 2017

Re: shell-host.c
Git push was still failing.

It appears that the whole nopty implementation assumes a pty control stream. Yet nopty may be a binary data stream.
While I'm uncertain about the defaults and all the possible options of ssh (for instance, assuming pty control for a nopty connection), tty control should not be processed on a binary data stream.

After removing the pty processing from:
start_withno_pty(wchar_t *command)
Git push works.

I can now clone repos over ssh and push changes to them.
I will continue to test.

The whole send to child inner loop simplifies to:

		while(rd){
			GOTO_CLEANUP_ON_FALSE(WriteFile(child_pipe_write, buf + i, rd, &wr, NULL));
			rd = rd - wr;
			i = i + wr;		
		}

@manojampalam
Copy link
Contributor

manojampalam commented Jun 5, 2017

What repo are you working on? There were changes around this logic as part of fixing #658
Take a look at the PR, binary streaming over stdin should work with the latest changes included in v0.0.15.0.

@razrjk
Copy link
Author

razrjk commented Jun 6, 2017

The repo is [email protected]:PowerShell/Win32-OpenSSH.git.
I cloned, built, and tried to test v.0.0.15.0.

Issue 1) from my initial post is still a problem. The file path sent by git contains literal single quote characters which become part of the file path, so the path is invalid.

The remote repo in the following case is: localhost:D:\work\git_repo.git. Note the single quotes below. They may look like a double quote, but they are single quotes around single quotes. The inside single quotes are literals in the file path.

$ git ls-remote
fatal: ''d:/work/git_repo.git'' does not appear to be a git repository
fatal: Could not read from remote repository.

I also originally tried powershell.exe and bash.exe. None of the command line interpreters liked the literal single quotes when passed with the "/c" or "-c" option. cmd.exe works okay with double quotes, so I stopped looking for another solution to the problem.

Regarding issue 2) of my original post: I do see quite a few changes, but the writes to pipe_out are still going back to the sshd process, not the child process child_pipe_write handle. Next, unless ssh options are mapped directly to cmd.exe options, I don't see how you can possibly detect any sub-mode of nopty, When the pipe is nopty, it's binary. A shell with tty capabilities would need to look at the binary stream and implement any tty escapes, not shell-host.c.
After starting the process (cmd.exe), shell-host.c must simply get out of the way and let the binary data flow, in both directions.

Also, sshd.exe /ddd is still broken.

@razrjk razrjk closed this as completed Jun 6, 2017
@razrjk razrjk reopened this Jun 6, 2017
@razrjk
Copy link
Author

razrjk commented Jun 7, 2017

I finally resolved another problem with large binary transfers, typical of a big git push.

shell-host.c is currently using cmd.exe. cmd.exe was dropping the end of large binary streams over slow network connection. I tried bash.exe and powershell.exe as command interpreters and they do not drop any of the stream. It took quite a long time to discover that cmd.exe was the problem.

I settled on using powershell.exe because it acts like cmd.exe for gitExtension file path resolution and resolves the common forms of path for Windows:

ssh://<host1>://<host2>/<repo-path>
<host2>:<drive>:/<repo-path>

@joeyaiello
Copy link

@razrjk as an FYI, we do code work out of https://github.com/PowerShell/openssh-portable, and only take drops back to this repo when we do releases.

@manojampalam
Copy link
Contributor

manojampalam commented Jun 14, 2017

@razrjk please submit a PR on https://github.com/PowerShell/openssh-portable for the proposed changes. Also open a new issue for sshd.exe -ddd

Also add details on where you installed Git for Windows from.

@diogocp
Copy link

diogocp commented Jul 12, 2017

To work around this issue, you can run the following commands in your local repository:

git config --local remote.origin.receivepack "powershell git receive-pack"
git config --local remote.origin.uploadpack "powershell git upload-pack"

Replace origin with the name of the Windows-hosted remote.

After this you should be able to pull and push normally (works for me, anyway).

Make sure you don't select "Use Git from Git Bash only" when installing Git on the Windows server, otherwise Git won't be in the PowerShell path.

@diogocp
Copy link

diogocp commented Jul 13, 2017

After further testing, I still get an error (early EOF) when pushing binary files. This is with v0.0.17.0.

@razrjk
Copy link
Author

razrjk commented Jul 13, 2017 via email

@bingbing8
Copy link
Contributor

bingbing8 commented Sep 16, 2017

@razrjk, I start looking this. Could you please share more information so I can try it from end to end?

  1. Can you share the steps you setup the git client? Are you using the git client for window from this link? or a unix client?
  2. Can you share the steps you set up git server over win32_openssh?(steps to setup git server and steps to config sshd so it talk to the git server)
  3. The git commands you repro the problems ( connect from your client to the git server over sshd)

@razrjk
Copy link
Author

razrjk commented Sep 16, 2017

  1. Yes. Git for Windows with the mingw unix tools also installed. The ssh client doesn't matter (except to know which one is being used for debugging purposes). I've used several different combinations: OpenSSH, cygwin, Git ssh, Git putty.
  2. It has been a while since I got everything working. Attached are my raw notes, there is some random information in these notes.
    notes.txt
  3. With the fixes I described in this thread, my systems have been working, so I have no problems. I'm responding to let you know that the fixes worked, and similar fixes need to be added to the OpenSSH repo. We use OpenSSH sshd with Git now, all the time in our production development.

Above, I described three problems with the sshd nopty path (as of the date above - some of these problems may have been resolved. I have not pulled and rebuilt for a long time):

  1. File/path escapes did not work for Windows files/paths. The problem was corrected as described above.
  2. Data was being echoed back to sshd from cmd.exe because some file handles were incorrect.
  3. Finally, cmd.exe does not work for large streamed data. The pipe breaks under heavy traffic load. This is a difficult problem to isolate. I finally decided to use Powershell instead of cmd.exe to work around this problem (for nopty), and it is working great.

My Git repo paths take the form:

> ssh://<opensshd-host>:<port>//<git-host>/<Git_Repo>
ssh://<opensshd-host>:<port>//<git-host>/<Git_Repo>

This supports efficient transfer between sites using Git compression over ssh, and yet the sshd server can use all the different types of Windows file path syntax. The double "//" in the path looks unusual, but it works.

My systems are using Windows Active Directory and all accounts are domain accounts, validated by the Domain Controller.

Interactive ssh/sshd work as well.

@bingbing8
Copy link
Contributor

@razrjk, my remote repo is setup at d:\testgit\myrepo.git on hostmachine. How do you specify the url for the path to your repo. I always see a '/' prefix at my URL. How do you remove it?
I use this command:

git clone ssh://username@domain@hostmachine/D:/testgit/myrepo.git
Cloning into 'myrepo'...
fatal: ''/D:/testgit/myrepo.git'' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Here is the ssh log:
debug1: Sending command: git-upload-pack '/D:\testgit\myrepo.git'
debug2: channel 0: request exec confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: exec request accepted on channel 0
debug2: channel 0: rcvd ext data 73
fatal: ''/D:\testgit\myrepo.git'' does not appear to be a git repository

@diogocp
Copy link

diogocp commented Sep 19, 2017

Try git clone username@domain@hostmachine:D:/testgit/myrepo.git.

You will probably continue to get the "does not appear to be a git repository" error after that, because of the way that cmd.exe handles single quotes (notice that there are two sets of single quotes around ''/D:/testgit/myrepo.git'' in the error message). That is the first problem that @razrjk identified.

If you want to try my workaround, you can do it this way:

git init myrepo
cd myrepo
git remote add origin username@domain@hostmachine:D:/testgit/myrepo.git
git config --local remote.origin.uploadpack "powershell git upload-pack"
git fetch origin

@bingbing8
Copy link
Contributor

@diogocp Thanks.
yes, git clone username@domain@hostmachine:D:/testgit/myrepo.git works. your workaround works too. I am looking at the first issue @razrjk reported.

@bingbing8
Copy link
Contributor

@diogocp how do you specify the port if you are not using the default 22?
git clone username@domain@hostmachine:47003:D:/testgit/myrepo.git does not work.

@diogocp
Copy link

diogocp commented Sep 20, 2017

@bingbing8 I would set it in ~/.ssh/config. Another alternative would be to set the environment variable GIT_SSH_COMMAND="ssh -p 47003". I didn't test that, but it should work.

@bingbing8
Copy link
Contributor

@diogocp, I don't repro the issues when pushing binary files (I use an executable file). I add the path to git-upload-pack to the system environment variable %Path% on host machine. Did you try on 0.0.20.0?

@diogocp
Copy link

diogocp commented Sep 21, 2017

Yes, I tried on 0.0.20.0. However, this does not happen with all binary files.

@razrjk seems to think it is related to the size of the data ("cmd.exe does not work for large streamed data. The pipe breaks under heavy traffic load."). I have some doubts about this theory because:

  1. I have seen it succeed when pushing tens of megabytes and fail on pushes of less than 1 MB.
  2. When it fails, it seems to fail consistently, no matter how many times you retry. It also doesn't seem to be affected by network speeds.

You should be able to consistently reproduce it this way (at least I can):

git clone https://github.com/PowerShell/Win32-OpenSSH.git
git init newrepo
cd Win32-OpenSSH
git remote add newrepo localhost:path/to/newrepo
git config --local remote.newrepo.receivepack "powershell git receive-pack"
git push newrepo

@diogocp
Copy link

diogocp commented Sep 21, 2017

Here is a simpler alternative. Try pushing this 28K file: https://repo.gradle.org/gradle/libs-releases-local/org/gradle/gradle-wrapper/4.1/gradle-wrapper-4.1.jar

@razrjk
Copy link
Author

razrjk commented Sep 21, 2017 via email

@bingbing8
Copy link
Contributor

bingbing8 commented Sep 28, 2017

@razrjk, thanks for the details issues reported and steps to help to identify the problems.

  1. For the single quotes issue, we are still thinking different approach to fix on server. We recommend people to apply the temporary workaround before the fix on server.
  2. I reviewed the handle you mentioned in file shell-host, some lines pass its stdin to the pipe_out, they are on purpose. This is for echo the input to client. especially when interop with putty, which does not echo the input in client side. If you review the code, all the stdin eventually pass to child write handle too.
  3. sshd.exe /ddd issue got fixed and will be available in 0.0.21.0.
  4. we need the control stream in no pty. for the case where run "cmd /c XXXX", we need to echo the input back and process those input before pass to child write handle.

@diogocp I made some changes in shell-host.c. Please let me know if you still repro the big binary push issue after the fix.

I posted a Setting up a Git server on Windows using Git for Windows and Win32_OpenSSH. please let me know if you see any other issue.

@HrochL
Copy link

HrochL commented Sep 29, 2017

Hi All,
about "single quotes issue" - recomended workaround must be applied on each client. This is not good thing for a common use. I would argue for some solution on the server ... ideally for the option to choosing the shell.

@bingbing8
Copy link
Contributor

bingbing8 commented Oct 2, 2017

@HrochL , yes, it is a temporary workaround until we have a good fix on the server.

@bingbing8
Copy link
Contributor

bingbing8 commented Oct 2, 2017

@benvindo which step? this?
To check out a repository, go to where you want to put your local repo,

 cd c:\mygitrepros
 # initialize a local repo folder
 git init mylocalrepo
 cd mylocalrepo
 # add the remote repro
 git remote add origin user@domain@servermachine:C:/test/myrepo.git
 # work around the known issue by launching powershell to run the git commands
 git config --local remote.origin.uploadpack "powershell git-upload-pack"
 git config --local remote.origin.receivepack "powershell git-receive-pack"
 git fetch origin

@bingbing8
Copy link
Contributor

Move this issue to Beta until we have a proper fix on server

@benvindo
Copy link

benvindo commented Oct 2, 2017

@bingbing8
I don't understand first step in server.
I tried following some steps for configuration my remote git but unsuccessful.
when add remote like this and run git push:
git remote set-url --add--push origin ssh://[email protected]@R2D2:/C:/web/multiprodutos/repo/homol.git

show me this error:

git-receive-pack : The term 'git-receive-pack' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
again.

At line:1 char:1
+ git-receive-pack '/C:/web/multiprodutos/repo/homol.git'
+ ~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (git-receive-pack:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

fatal: Could not read from remote repository.

I tried add remote like this too: git remote set-url --add --push origin ssh://[email protected]/C:/web/multiprodutos/repo/homol.git
but show me same error above.

**When I put remote without slash in the start
as recommended git remote set-url --add--push origin ssh://[email protected]@R2D2:C:/web/multiprodutos/repo/homol.git I can't even log in

Do you have any idea what's causing this error?

@benvindo
Copy link

benvindo commented Oct 2, 2017

The cause of error above was I don't put environment 'git\mingw64\bin' in my variables env in server, however I got other error:
fatal: '/C:/web/multiprodutos/repo/homol.git' does not appear to be a git repository fatal: Could not read from remote repository.

@diogocp
Copy link

diogocp commented Oct 2, 2017

**When I put remote without slash in the start
as recommended git remote set-url --add--push origin ssh://[email protected]@R2D2:C:/web/multiprodutos/repo/homol.git I can't even log in

There is no ssh:// in the instructions.

@benvindo
Copy link

benvindo commented Oct 3, 2017

Hi, sorry for my lack of attention, I was focused on connecting the apps and forgetting the minimally details, Now its work perfectly. Thanks!!

@bingbing8
Copy link
Contributor

@diogocp, can you try the [latest release] (https://github.com/PowerShell/Win32-OpenSSH/releases/tag/v0.0.21.0) if you can repro early EOF error.

@diogocp
Copy link

diogocp commented Oct 6, 2017

@bingbing8 yes, same error with 0.0.21.0.

@bingbing8
Copy link
Contributor

@diogocp, I really can't repro it. what do you get in the client log?

@bingbing8 bingbing8 modified the milestones: Oct-Mid, Beta Oct 17, 2017
@benvindo
Copy link

benvindo commented Oct 19, 2017

I have a similar error to @diogocp, at some point when a member of my team sends commits to a TFS repository and after some time we try to send all the commits to the production happens the error:

`git push serve_prod homol:prod
Multiprodutos@[email protected]'s password:
Counting objects: 97, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (47/47), done.
Writing objects: 100% (97/97), 53.53 KiB | 0 bytes/s, done.
Total 97 (delta 73), reused 69 (delta 48)
remote: fatal: early EOF
error: remote unpack failed: unpack-objects abnormal exit
To [email protected]:C:/multiprodutos/web/repo/ui-multiprodutos-web.git
 ! [remote rejected] homol -> prod (unpacker error)
error: failed to push some refs to 'Multiprodutos@[email protected]:C:/multiprodutos/web/repo/ui-multiprodutos-web.git'
`

@larry191994
Copy link

@benvindo which step? this?
To check out a repository, go to where you want to put your local repo,

 cd c:\mygitrepros
 # initialize a local repo folder
 git init mylocalrepo
 cd mylocalrepo
 # add the remote repro
 git remote add origin user@domain@servermachine:C:/test/myrepo.git
 # work around the known issue by launching powershell to run the git commands
 git config --local remote.origin.uploadpack "powershell git-upload-pack"
 git config --local remote.origin.receivepack "powershell git-receive-pack"
 git fetch origin

I follow your approach but I still get the fatal: ''/C:/git/A.git'' does not appear to be a git repository when I try to git clone the remote repository from another server. I am really confused about this ....

@diogocp
Copy link

diogocp commented Dec 5, 2018

@larry191994 you cannot use git clone directly. You have to follow those instructions every time you want to fetch the repository.

@bingbing8
Copy link
Contributor

bingbing8 commented Dec 6, 2018

@larry191994 , @diogocp , after supporting default shell, you don't need the above workaround if you can set up Default Shell. Please refer to the link.

@yuliu
Copy link

yuliu commented Aug 8, 2019

TL;DR:
Alter the shell for Windows 10's built-in openssh (Sorry, I only installed it from Microsoft, not this repo), either cmd or powershell, to bash that comes with Git for Windows by changing a register key's value (change the path to your Git installation):

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Program Files\Git\bin\bash.exe" -PropertyType String -Force

I'm sorry that I'm not commenting for this repo, but having the same problem hosting a Git repo's files on Windows 10 and want to work with them in another system through SSH. The problem looks the same as @bingbing8's

git clone ssh://username@domain@hostmachine/D:/testgit/myrepo.git
Cloning into 'myrepo'...
fatal: ''/D:/testgit/myrepo.git'' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

and @benvindo's

+ git-receive-pack '/........'
+ ~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (git-receive-pack:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

fatal: Could not read from remote repository.

The URL is a trick to make the path to the repo valid. And we also have to alter/add configurations of remote.origin.uploadpack and remote.origin.receivepack to have a workaround for pull/push. But we have to initialize a bare repo first and then do this and do that, which complicates our work.

Since we have Git installed on Windows 10 (or server...), we might have git-bash installed as well. And we can use Git's bash as the default bash for ssh, so that we can use git normally, without tweaking settings. The way to do so is to change the default shell for Windows 10's built-in openssh to git-bash, read TL;DR in the beginning. Then we should git clone without any issue.

I don't really know how ssh works internally, but I just guess all these problems in this issue are so related to the host system's shell, openssh should not be blamed for and responsible for them. Probably, Git and Powershell can work together to get git working under powershell and/or even cmd.

@essadek
Copy link

essadek commented Mar 2, 2021

I come cross the above workaround and it solved the below issue in 2021.

does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests