Support authentication via .pem file within host configuration?

[lucasrla]

Thank you so much for creating SoS. It is such a great tool!

Quick question: is there an easy way to make the host setup/configuration work with a host that requires keypair (.pem) file? I couldn't find anything about it here in the repository nor in the host configuration docs.

Let's say I want to run a job in an EC2 instance.

Right now, I am using a workaround that is clearly suboptimal because it does not take any advantage of the great features SoS has with task statements.

Here is a toy example of what I am currently doing:

sh: expand=True
    ssh {username}@{hostname} -v -i {pem_path} -o ServerAliveInterval=180 -o ServerAliveCountMax=3 -t '{command}'

I am aware that I could edit the server SSH config in order to use another method of authentication, but I would do that only if there is no other solution.

Thanks in advance!

[BoPeng]

This is something that should and can be done. Let me look into it.

[BoPeng]

On the pem branch I have allowed the definition of pem_file under hosts, and an -i option will be added to the ssh command if it is specified. Something like

localhost: desktop
hosts:
    desktop:
        home: /Users/{user_name}
    mendel:
      address: [email protected]
      pem_file: /Users/{user_name}/.ssh/mendal.pem
      home: /home/{user_name}

pem_file is supposed to be different for each remote host, so I put it under definition of remote hosts. However, the remote host section should only define things that describe the remote host, not files defined locally, so a proper syntax should perhaps be

localhost: desktop
hosts:
    desktop:
        home: /Users/{user_name}
        pem_files:
            mendel: /Users/{user_name}/.ssh/mendal.pem
    mendel:
      address: [email protected]
      home: /home/{user_name}

[lucasrla]

Wow, that was fast! I will give it a try soon and let you know. Thanks a lot!

Regarding where in the yaml file the pem_file key should go, your suggestion looks good to me.

Another option would be something like auth > pem_files > mendel, instead of your hosts > desktop > pem_files > mendel.

[BoPeng]

I suggested hosts > desktop > pem_files > mendel because in theory different hosts can have different sets of pem files to other hosts. It however makes more sense to keep the configuration outside of hosts.yml (e.g. in ~/.sos/config.yml) for each host then your proposal makes sense.

Let us first see if adding the key to remote host definition works.

[lucasrla]

Unfortunately it did not work.

Here is my hosts.yml:

# ~/.sos/hosts.yml
localhost: mbp
hosts:
  mbp:
    home: /Users/{USER_NAME}/
  ec2-instance:
    address: ec2-user@{IP_ADDRESS}
    pem_file: /Users/{USER_NAME}/.ssh/{KEY_NAME}.pem
    paths:
      home: /home/ec2-user/

Here is my "new" workflow:

# remote_test.sos
[new]
input: "new_input.txt"
output: "new_output.txt"

task: to_host="{step_input}", from_host="{step_output}", queue='ec2-instance'

sh: expand=True, stdout=step_output, stderr=step_output
    ls; mv {step_input} {step_output}; ls

Running sos run -v4 remote_test.sos new results in the following error:

sos.executor_utils.ExecuteError: [new]: [b72b27a180272ad5]: Failed to connect to ec2-instance: ec2-user@{IP_ADDRESS}: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Inspecting the sshd logs inside the EC2 instance with sudo tail /var/log/secure -n 100 | grep ec2-user I see:

error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys ec2-user SHA256:XXXXXXXX failed, status 22

There is some kind of key mismatch going on – but I am far from an expert on ssh...

I did find an open issue with an AWS package that could be related to this error: aws/aws-ec2-instance-connect-config#20 So it is sensible to test this code with other servers to exclude this possibility... (I could only do that by the weekend)

---

If I try to log in manually from my terminal, it connects just fine:

# just in case, I added the `ControlMaster` flags you are using and it all went fine
ssh -o "ControlMaster=auto" -o "ControlPath=~/.ssh/controlmasters/%r@%h:%p" -o "ControlPersist=10m" -i /Users/{USER_NAME}/.ssh/{KEY_NAME}.pem ec2-user@{IP_ADDRESS}

A workflow based on my "old" approach works fine too. For instance, I get the correct output by running sos run -v4 remote_test.sos old:

# remote_test.sos
[old]
input: "old_input.txt"
output: "old_output.txt"

ec2_host = CONFIG['hosts']['ec2-instance']
local_host = CONFIG['hosts']['mbp']

sh: expand=True
    scp -i {ec2_host['pem_file']} {local_host['home']}/{step_input} {ec2_host['address']}:~/
    ssh {ec2_host['address']} -i {ec2_host['pem_file']} -t 'ls; mv {step_input} {step_output}; ls'
    scp -i {ec2_host['pem_file']} {ec2_host['address']}:~/{step_output} {local_host['home']}

---

Finally, just for the record, I am using code from the pem branch. I upgraded my pip package manually via pip install --upgrade git+https://github.com/vatlab/sos.git@pem.

[BoPeng]

Thanks. I do have an abandoned EC2 instance. Let me try to reproduce your problem there.

[BoPeng]

OK, here is what I did

1. Put the perm file somewhere with permission 600.
2. Login to the ec2 instance, install miniconda and conda install sos -c conda-forge.
3. set up host as

    aws:
        address: [email protected]
        pem_file: /path/to/my.pem
        paths:
            home:  /home/ec2-user/

4. test connection

 ✗ sos remote test aws
Alias Address                Queue Type ssh scp sos paths shared
----- -------                ---------- --- --- --- ----- ------
aws   [email protected] process    OK  OK  OK  OK    OK

5. Execute a simple workflow

output: 'aws_output.txt'
task:
sh: expand=True
  echo I am on $HOSTNAME > {_output}

and get

> sos run test -q aws
INFO: Running default:
INFO: 7a18d48b90137c36 started
INFO: 7a18d48b90137c36 completed
INFO: 7a18d48b90137c36 received 'aws_output.txt' from aws
INFO: default output:   aws_output.txt
INFO: Workflow default (ID=34c1c641f9ef6387) is executed successfully with 1 completed step and 1 completed task.

and

> cat aws_output.txt
I am on ip-xxx-xx-xx-xx.us-west-1.compute.internal

Note that:

1. I already merged pem branch to master, so you can try master instead.
2. Identify file can be specified as pem_file key in either localhost (for all remote host) or remote host (for all incoming host), or as a dictionary in localhost (for each remote host).

[BoPeng]

pem_file is documented at https://vatlab.github.io/sos-docs/doc/user_guide/host_setup.html#Use-of-identity-files

[lucasrla]

I updated SoS via pip to HEAD @ master and then I was able to reproduce what you did flawlessly. My own remote test also worked fine.

I will close this issue now. Thank you so much!

[BoPeng]

Thanks for your confirmation. I have released sos 0.21.12 with the new feature.