One recently-completed project I mentioned in January’s “Now” post was locking down SSH in my personal computing infrastructure using Yubikeys. In this post, I’ll outline my goals, the strategy I took, and the problems and solutions I ran into along the way.
Goals & Strategy
Historically, I’ve used a pretty basic SSH setup for my personal projects: my user account on every laptop/desktop/server had its own key in
~/.ssh, and I’d try to keep the
authorized_keys lists on all my servers more-or-less up-to-date. This presents a number of obvious security problems.
I wanted to ensure that, should an attacker gain access to one of my servers, they can’t use that access to move onto any other computer I control. To do this, I moved to using a few Yubikeys to store my SSH keys; there’s no longer key material stored on any server for a hypothetical attacker to steal. The Yubikeys require a PIN, so this is an example of two-factor authentication: something I physically have, and something I know.
SSH agent forwarding is used to allow me to SSH from one server to another or fetch code from GitHub on a remote server. With
yubikey-agent, my preferred agent software, every single SSH operation — yes, even those performed via agent forwarding — requires a physical touch to confirm.
I use a private Git repository to synchronize SSH configuration (including
authorized_keys, the list of public keys corresponding to my Yubikeys) between machines, with a modular local configuration system allowing me to quickly enable commonly-used SSH configuration blocks which only apply to a subset of my machines.
I found it easiest by far to use
yubikey-agent for this project. It’s pretty straightforward to set this up; the real work was figuring out how to smooth out the various difficulties I encountered later.
(The commands and configuration changes under this heading apply to client machines with attached Yubikeys.)
yubikey-agent with Homebrew:
brew tap filippo.io/yubikey-agent https://filippo.io/yubikey-agent brew install yubikey-agent brew services start yubikey-agent
yubikey-agent -setup to generate a new SSH key on your Yubikey.
SSH_AUTH_SOCK environment variable (do this in your
.zshrc). In my dotfiles, I first check that
yubikey-agent is installed, then proceed:
command -v yubikey-agent >/dev/null 2>&1 && export SSH_AUTH_SOCK="/usr/local/var/run/yubikey-agent.sock"
Enable SSH agent forwarding for a specific, trusted host (don’t enable it for all hosts; that’s a potential security issue) by adding
ForwardAgent yes to that host’s block in your
~/.ssh/config. A complete example might look like:
Host dzombak.com User chris HostName dzombak.com ForwardAgent yes
This is a bit of a spoiler for a topic later in this blog post, but we’ll also add this to our
Host * IdentityAgent /usr/local/var/run/yubikey-agent.sock
This allows apps started from outside your terminal — like the GUI Git client, Fork.app — to find and use
A note: Secretive.app
I’d like to use the new macOS app Secretive, which stores SSH keys in the Secure Enclave on newer MacBooks and requires Touch ID to authenticate. Unfortunately, for Reasons™ I’m still using macOS Mojave, and Secretive requires Catalina or Big Sur. I plan to move to Big Sur soon enough, since I want to get an M1 MacBook Pro when the 16” models are released, so I’ll be able to try Secretive soon enough. (Worth noting, this changes the security model somewhat, as the second factor is biometric rather than a PIN, but it’s still two factors.)
Of course, moving to Yubikeys doesn’t solve much if your servers still allow password logins. On every server, in
/etc/ssh/sshd_config, I set the following. Make this change only after you’ve set up a Yubikey and added it to
authorized_keys for your user account on the server!
ChallengeResponseAuthentication no PasswordAuthentication no PermitRootLogin no
(That last line —
PermitRootLogin no — ensures that logins as
root via SSH are never allowed, which is a good SSH best practice unrelated to Yubikeys.)
Restart the SSH service, and immediately — before logging out — open a new terminal window and test that you can still login to the server with your Yubikey.
macOS tends to lose changes to
sshd_config during OS upgrades, so after installing macOS updates I make sure to check that my SSH server configuration is intact.
Git repo for SSH configuration
I keep my
~/.ssh directory, with a few important exceptions (keys are never committed!), in a private Git repo. This allows me to sync configuration and
authorized_keys changes between systems easily. I’ve created a stripped-down version at cdzombak/ssh-example which you can use as a basis for your own setup.
I’ll walk through the highlights here:
README.mdcovers initial installation & setup.
authorized_keysis where you’ll add the public keys associated with the new, private SSH keys on your Yubikeys.
configis where you’ll add
Hostblocks for your servers. At the top, it sets some SSH best practices that I’ve accumulated over the years.
~/.ssh/rchave the correct permissions. I run this after pulling from the repository; if it results in any changes the files’ permissions in the repo should be corrected.
rcruns after I log into a machine via SSH. In this case, it updates a symlink in
~/.ssh/sockto point to the new SSH agent socket. See the “Long-lived
screensessions” section, below, for an explanation on why this is necessary.
config.templates/contains SSH configuration blocks which can be included on a given machine, but shouldn’t be included everywhere. The most important of these are
yubikey-agent, which when enabled sets
yubikey-agentsocket as described above; and
homedir-ssh-auth-sock, which sets
IdentityAgentto the symlink
rccreates after an SSH login. On any given machine, I enable one and only one of these two configurations, depending whether it’s a client machine with Yubikey attached or a server which will rely on agent forwarding for any outgoing SSH connections.
config.local/is included by
configand ignored by Git. I can add symlinks from here to
config.templatesto enable a specific SSH configuration block on the machine.
Those last two —
config.templates — are important, because that’s how I achieve variations in SSH configuration between different machines. The README covers how to enable config templates on a given computer.
Challenges & Solutions
I use GNU
screen (yes, still; I haven’t bothered to learn
tmux) as a terminal multiplexer and to provide persistence between SSH sessions. This was a problem for SSH agent forwarding: when I first SSH in and start a
screen session, the
SSH_AUTH_SOCK environment variable would be set. But when I logged in from somewhere else and reattached to the
screen session, the
SSH_AUTH_SOCK environment variable wouldn’t get updated, so SSH agent forwarding was broken until I started a new
This Gist helped me fix this situation. There are a few parts to this solution. (The commands and configuration changes under this heading apply to servers you’ll SSH into.)
First, we have to have a location for our SSH agent socket that doesn’t change between logins. These few lines in
~/.ssh/rc achieve this:
if test "$SSH_AUTH_SOCK" ; then ln -sf "$SSH_AUTH_SOCK" ~/.ssh/sock/ssh_auth_sock fi
Great! Then we just need clients to use this new, always-updated socket. To do this, we configure
~/.screenrc to set the environment variable
setenv SSH_AUTH_SOCK $HOME/.ssh/sock/ssh_auth_sock
Finally, we’ll also want to include
IdentityAgent ~/.ssh/sock/ssh_auth_sock in our SSH configuration. We can do this by including the
homedir-ssh-auth-sock configuration block within my modular SSH configuration setup:
cd ~/.ssh/config.local ln -s ../config.templates/homedir-ssh-auth-sock .
SSH agent forwarding when running commands under sudo
When running a command with
sudo, you’re working in a new environment; your user’s environment variables are not preserved. This will, of course, break SSH agent forwarding.
To solve this, we want to preserve the
SSH_AUTH_SOCK environment variable when using
sudo. (The commands and configuration changes under this heading apply to servers you’ll SSH into.)
root) to edit your
sudoers file, and add:
This means that when using
sudo to run a command as
root (not as any other user), your
SSH_AUTH_SOCK variable remains intact, and agent forwarding works as expected.
SSH agent forwarding via
SSH_AUTH_SOCK doesn’t work with GUI macOS apps
This issue in the Fork app’s issue tracker was really helpful here.
.zshrc don’t apply to GUI apps (unless you launch them from the terminal), so setting
SSH_AUTH_SOCK only in shell configuration files won’t work.
This is why we need
IdentityAgent /usr/local/var/run/yubikey-agent.sock in our SSH configuration. To enable this within my modular SSH configuration setup:
cd ~/.ssh/config.local ln -s ../config.templates/yubikey-agent .
(This change applies to macOS client machines with attached Yubikeys.)
Avoiding repeated mystery Yubikey prompts (using HTTPS for GitHub and Bitbucket repositories)
After I set this up, my Yubikey would periodically blink as if I were trying to SSH into something, but I hadn’t tried to do anything with SSH! That was worrying, until I realized it was just Fork trying to update repository info in the background.
I decided to configure Git on my laptops to use HTTPS instead of SSH when communicating with GitHub and Bitbucket, so Fork can work in the background as it desires.
To do this, we’ll add the following to
~/.gitconfig. (This change applies to any Mac where you’d like Git to use HTTPS instead of SSH.)
[url "https://"] insteadOf = git:// [url "https://github.com/"] insteadOf = email@example.com: [url "https://bitbucket.org/"] insteadOf = firstname.lastname@example.org: [credential] helper = osxkeychain
Now, when you try to perform a Git operation for a GitHub or Bitbucket repo for the first time, Git will prompt for credentials:
Username for 'https://bitbucket.org': Password for 'https://email@example.com':
Git will cache these credentials on macOS’s Keychain for future use.
Servers that need to communicate autonomously
I do have a use case where one server needs to sync data to another periodically using rsync over SSH, meaning a Yubikey that requires physical, real-time interaction is out of the question.
To achieve this, the receiving server has a restricted user account which only allows access to the necessary data. The sending server, who initiates the sync, has an SSH key which is used only for this task; and which can only log into the restricted user account on the receiving server.
For an extra layer of moderate security (it’s still not a real second factor), the receiving server restricts the IP address from which that SSH key can login, using the
from directive in its
authorized_keys file. See Restricting SSH logins to particular IP addresses and Configuring Authorized Keys for OpenSSH for more details on that.
iPhone SSH clients (Prompt & Secure ShellFish)
I use Prompt 2 on my iPhone occasionally when a laptop isn’t handy. The app generates its own SSH key, which is stored on the iPhone (not in Panic Sync), and I add the corresponding public key to
authorized_keys in my SSH configuration repository.
I figure this is fairly secure, because the key stays on my phone and is secured behind Face ID & the iPhone’s PIN, meaning two factors are required: something I have and one of (biometrics or something I know).
It’s a similar story with Secure ShellFish. I use this to remotely access files on my home NAS. In this case, Secure ShellFish’s SSH key actually allows logging in only to a user account on the NAS with limited permissions; it isn’t added to my “core” SSH configuration.
An additional Yubikey lives in a fireproof safe to aid in recovery in case all other Yubikeys are lost or destroyed.
Windows clients (unsolved)
I’m just ignoring this for now. There’s only one Windows machine I use even somewhat regularly, and I never need to SSH into anything from it. It’d be nice to learn about an SSH agent solution that supports Yubikeys or the Google Titan security key, but I have no motivation to work on this myself.