In a previous post we talked about SSH key management best practices which involved using of ssh-agent to store decrypted private keys to automate authentication of an SSH client.In this post we would do a deep dive into how SSH agents work including some edge cases in its use.

Starting An Agent

SSH agent has some mind puzzling behavior that could surprise even the most seasoned of system administrators. There isn’t just one command syntax for running an agent and there are caveats associated with each way.

There are two ways to start an agent:

• Single-shell method; uses your current login shell on a single terminal
• Subshell method; forks a subshell to use some inherited environment variables

In the first approach the agent is run in your current login shell. This is done by running the command:

$ ssh-agent

The variables printed to the shell after the command above is run, need to be set as environment variables. The command forks the ssh-agent in the background, detaching from the terminal and returning the prompt to you. Beware that after using after each SSH session terminates the agent doesn’t get killed even if you logout, unless you explicitly terminate the agent. So if you start another terminal session and run ssh-agent and ssh-add again another process gets forked in the background. Soon you have a series of agents running in the background doing nothing. To avoid this use the second method.

The second approach spawns a subshell from your current shell. All you need do is provide a path to a shell or shell script like so:

$ ssh-agent  $SHELL

This time instead of forking a background process, the ssh-agent runs in the foreground, spawning a subshell and setting the necessary environment variables. The rest of your login session runs in the subshell and when you logout the agent process gets terminated. The agent can also be terminated by running the command:

$ ssh-agent -k  #this won’t work if you started ssh-agent with the first approach

Once the agent is up and running we add keys to it, this is done by ssh-add command giving the path to the identity file (private key) as a parameter.

$ssh-add  /home/you/.ssh/id_rsa

Your passphrase is requested once by ssh-add to load up the private key. There are numerous command line options including listing and deleting keys in the ssh-agent using ssh-add command. Look up the man page for ssh-add on your favorite distribution to see the full list of command line options.

Agent Forwarding

A scenario were agents are of particular use is, suppose I want to copy a file from remote server shell.isp.com to another remote server other.host.net and SSH is installed on both of them and they both use public key authentication how do I achieve that? I could copy from remote server shell.isp.com to my local system then copy from my system to remote server other.host.net, but that’s a waste of time. Instead what we could do since I have an account on both remote servers is to directly copy the file from server shell.isp.com to server other.host.net with a command like this:

$ scp pat@shell.isp.com:print-me psmith@other.host.net:imprime-moi 

But if you run the above command it fails because there is no tty to enter the passphrase to decrypt the private key for other.host.net on the shell.isp.com host when it tries to run scp to do the actual copy of the file other.host.net. All this assumes you have your private key for other.host.net on shell.isp.net. It’s a very bad idea to store your private key on a server and generally the fewer places with Internet network access your private key is located the safer it is. If you must back up your private key keep it on a portable disk kept save from the everyone else. Another use case concerning developers is pulling code from a repository on github to a remote server. If the repository is private and you have ssh access set up for your github account then using an agent would make life a lot easier, instead of copying your github private keys to the remote server.

Fortunately ssh-agent easily solves all the above issues. The remote scp simply contacts the ssh-agent on your local machine for authentication, your private keys never leave your local machine, assuming you have loaded the private keys for other.host.net into ssh-agent. Similarly if you have loaded your private keys for your github account on your ssh-agent locally then authentication happens on your machine.

To use agent forwarding both the client and the server must permit it. The socket opened by the ssh-agent is responsible for the communication between the server and the client. The environment variable SSH_AUTH_SOCK points to the socket, the variable is created when the agent is initialized, the socket is usually stored in the /tmp directory. To enable agent forwarding on the server the variable “AllowAgentForwarding” must be set to yes in your SSH server config file, likewise the variable “ForwardAgent” must be set to yes on your SSH client config file.

Conclusion

SSH agents are a very useful feature. They allow automation and improve security of your systems. Its possible with agent forwarding to hop from machine to machine as agent-forwarding relationship is transitive in nature. As long as the each intermediate server has agent forwarding enabled then, the authentication will always take place on the client. This feature is especially useful in corporate firewalled networks where there is a bastion server facing the Internet preventing access to a private network of servers. With SSH agent forwarding it’s possible to reach behind the firewall and access the servers if SSH is installed on the servers and forwarding is enabled.

SSH agent consumes a significant amount of computing resources so it’s always good to make sure that you don’t have idle processes running in the background.

When using an SSH agent to hop from machine to machine it's important that each intermediate host isn't compromised as that would expose your keys to intruders.