SSH · Software Engineering Wiki

Cheatsheet #

Task	Command
Connect as another user	`ssh user@host`
Why did it fail	`ssh -vvv user@host`
Which key was offered	`ssh -v host 2>&1 \| grep -i 'offering\|accepted'`
Run one command	`ssh host 'uptime'`
Copy a key to a host	`ssh-copy-id -i ~/.ssh/id_ed25519.pub host`
New key	`ssh-keygen -t ed25519 -C 'you@example.com'`
Load key into the agent	`ssh-add ~/.ssh/id_ed25519`
Keys the agent holds	`ssh-add -l`
Through a bastion	`ssh -J bastion.example.com host`
Local forward	`ssh -L 5432:db.internal:5432 bastion`
Remote forward	`ssh -R 8080:localhost:3000 host`
SOCKS proxy	`ssh -D 1080 bastion`
Host key fingerprint	`ssh-keyscan host \| ssh-keygen -lf -`
Drop a stale host key	`ssh-keygen -R host`
Effective config for a host	`ssh -G host`
Reuse a connection	`ControlMaster auto` in config

Client configuration #

~/.ssh/config is read top to bottom and the first value for each option wins, so put specific hosts above wildcards. Everything you would otherwise retype belongs here.

Host bastion
  HostName bastion.example.com
  User jodis
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes

Host db-*
  ProxyJump bastion
  User postgres
  ForwardAgent no

Host *
  AddKeysToAgent yes
  ServerAliveInterval 30
  ServerAliveCountMax 3
  ControlMaster auto
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlPersist 10m
  HashKnownHosts yes

Option	Why
`IdentitiesOnly yes`	Offer only the listed key; without it the agent offers every key and the server may hit `MaxAuthTries` first
`ControlMaster`/`ControlPersist`	Reuse one TCP connection for subsequent sessions: near-instant repeat logins
`ServerAliveInterval`	Keeps NAT and idle timeouts from silently dropping the session
`ProxyJump`	Connect through a bastion without forwarding your agent to it
`ForwardAgent`	Dangerous on a shared host: root there can use your agent
`StrictHostKeyChecking accept-new`	Accept a first-time key but still fail on a change

ssh -G host            # every option that will apply, after merging
ssh -o 'ProxyJump=none' host   # override config from the command line

Keys and the agent #

Use Ed25519: shorter, faster, no parameter choices to get wrong. RSA is only needed for systems that have not caught up, and then at 4096 bits.

ssh-keygen -t ed25519 -C 'jodis@laptop'
ssh-keygen -t ed25519 -f ~/.ssh/id_deploy -N ''        # unencrypted, for automation only
ssh-copy-id -i ~/.ssh/id_ed25519.pub host
ssh-keygen -lf ~/.ssh/id_ed25519.pub                   # fingerprint
ssh-keygen -y -f ~/.ssh/id_ed25519 > id_ed25519.pub    # regenerate a lost public key
ssh-keygen -p -f ~/.ssh/id_ed25519                     # change the passphrase

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
ssh-add -l                                             # loaded keys
ssh-add -D                                             # forget everything
ssh-add -t 3600 ~/.ssh/id_ed25519                      # expire after an hour

Permissions are enforced: 700 on ~/.ssh, 600 on private keys, 644 on public keys and authorized_keys. A group-writable home directory also makes sshd refuse the key, which produces a confusing “permission denied (publickey)” with a perfectly good key.

Restrict what a key may do on the server side, in authorized_keys:

restrict,pty,command="/usr/local/bin/backup" ssh-ed25519 AAAA... backup@ci
from="10.0.0.0/8",restrict ssh-ed25519 AAAA... deploy@ci

Port forwarding #

ssh -L 5432:db.internal:5432 bastion     # local 5432 -> db.internal:5432, via bastion
ssh -R 8080:localhost:3000 host          # host's 8080 -> my local 3000
ssh -D 1080 bastion                      # SOCKS5 proxy on 1080, browse as if inside
ssh -L 5432:db.internal:5432 -N -f bastion   # no shell, background
ssh -J bastion db-01                     # jump, no agent forwarding, no key on bastion

-L binds locally and reaches out from the remote; -R binds on the remote and reaches back to you. -R only listens on the server’s loopback unless GatewayPorts yes is set there.

# Kill a backgrounded forward
pkill -f 'ssh -L 5432'
# Or manage it through the control socket
ssh -O check bastion; ssh -O exit bastion

Server configuration #

/etc/ssh/sshd_config, applied after systemctl reload sshd. Validate before reloading, and keep a second session open until you have confirmed the new config works.

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
AllowGroups ssh-users
MaxAuthTries 3
LoginGraceTime 20
X11Forwarding no
AllowAgentForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2

sshd -t                       # syntax check
sshd -T | grep -i passwordauth  # effective configuration
systemctl reload sshd

Do not lock yourself out

Keep the current session open, reload in another, and verify a fresh login before closing anything. On a cloud instance, confirm console access exists first.

Later Match blocks apply conditionally and override the global values:

Match Group sftp-only
  ChrootDirectory /srv/sftp/%u
  ForceCommand internal-sftp
  AllowTcpForwarding no

Troubleshooting #

ssh -vvv host 2>&1 | tail -40         # client side: key offers, kex, auth methods
sudo journalctl -u sshd -f            # server side: the actual rejection reason
ssh -o PreferredAuthentications=publickey -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 host
ssh-keyscan -t ed25519 host | ssh-keygen -lf -    # compare against the expected fingerprint

Message	Cause
`Permission denied (publickey)`	Key not in `authorized_keys`, wrong user, or home/`.ssh` permissions too open
`Too many authentication failures`	Agent offered several keys first — set `IdentitiesOnly yes`
`Host key verification failed`	Host rebuilt or MITM; verify out of band, then `ssh-keygen -R host`
`Connection closed by remote host`	Rejected by `AllowUsers`/`AllowGroups`, or fail2ban
Hangs after banner	MTU or firewall dropping large packets; try `-o IPQoS=none`
Slow login, fast once connected	Reverse DNS timeout on the server — `UseDNS no`

File transfer #

scp file host:/tmp/                        # simple, fine for one file
rsync -avzP --delete ./dir/ host:/srv/dir/ # resumable, incremental, the default choice
sftp host                                  # interactive, or batch with -b
ssh host 'tar czf - /var/log' > logs.tgz   # stream without staging a file
tar czf - ./dir | ssh host 'tar xzf - -C /srv'

rsync transfers only the differences and can resume; prefer it for anything larger than a config file. See scp for the copy syntax itself.

Oneliners #

# Run a command on many hosts, 8 at a time
printf '%s\n' host{1..20} | xargs -P8 -I{} ssh -o ConnectTimeout=5 {} 'uptime'

# Which key does this host accept
ssh -v host 2>&1 | grep -E 'Offering|Server accepts|Authenticated'

# Copy a public key without ssh-copy-id
cat ~/.ssh/id_ed25519.pub | ssh host 'mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys'

# Persistent tunnel that reconnects
autossh -M 0 -o ServerAliveInterval=30 -o ExitOnForwardFailure=yes -N -L 5432:db.internal:5432 bastion

# Reachability test from inside, through a bastion
ssh -J bastion host 'nc -zv db.internal 5432'

# Fingerprints of every key in authorized_keys
ssh-keygen -lf ~/.ssh/authorized_keys

# Show, then remove, a host's known_hosts entry after a rebuild
ssh-keygen -F host; ssh-keygen -R host

# Compare local and remote file without copying
ssh host 'sha256sum /etc/app.conf' | awk '{print $1}' | diff - <(sha256sum /etc/app.conf | awk '{print $1}')

# Mount a remote directory locally
sshfs host:/srv/data /mnt/data -o reconnect,ServerAliveInterval=15

# Tail a remote log through a bastion
ssh -J bastion app-01 'journalctl -u myapp -f'

# Measure login time with and without connection reuse
time ssh -o ControlPath=none host true; time ssh host true