How to Build OpenSSH with Custom Options Step-by-Step Guide

Building OpenSSH from source with custom options allows you to tailor the SSH server and client to your exact needs. Whether you want to enable or disable specific features, optimize for performance, or apply patches, compiling OpenSSH yourself gives you full control over its capabilities and behavior. This guide walks you through the entire process, from preparing your system to configuring and compiling OpenSSH with your chosen options.

What You’ll Need

A Unix-like operating system (Linux, BSD, macOS, or similar)
Basic command line knowledge and sudo or root access
Development tools installed: gcc, make, autoconf, and openssl development libraries
Internet connection to download source code and dependencies
Optional: familiarity with SSH configuration and security best practices

Step-by-Step

Update your system and install build dependencies. Before beginning, ensure your package lists and installed software are up to date. On Debian/Ubuntu, run:
```
sudo apt update && sudo apt upgrade
sudo apt install build-essential zlib1g-dev libssl-dev libpam0g-dev libselinux1-dev libedit-dev
```
Adjust package names for your distribution accordingly.
Download the latest OpenSSH source code. Visit the official OpenSSH website (https://www.openssh.com/portable.html) and download the latest portable release tarball or use wget:
```
wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.4p1.tar.gz
```
Replace the version number with the latest available.
Extract the source code archive.
```
tar -xzf openssh-9.4p1.tar.gz
cd openssh-9.4p1
```
This creates a directory with the source files.
Configure your build options. OpenSSH uses a configure script to customize the build. Run ./configure --help to see all available options. Common options include:
- --prefix=/usr/local to specify installation directory
- --with-ssl-dir=/path/to/openssl if OpenSSL is in a non-standard location
- --with-pam to enable PAM authentication
- --without-openssl-header-check to skip OpenSSL version checks
- --with-md5-passwords to enable MD5 password support (not recommended)
Example:
```
./configure --prefix=/usr/local --with-pam --with-ssl-dir=/usr/local/ssl
```
Adjust options based on your requirements.
Compile the source code. Once configured, build OpenSSH by running:
```
make
```
This step may take a few minutes depending on your system.
Run tests (optional but recommended). To verify the build integrity, run:
```
make tests
```
This runs a suite of automated tests. Review any failures carefully.
Install OpenSSH. Use:
```
sudo make install
```
This installs the binaries and configuration files to the specified prefix.
Configure your SSH server and client. Edit the main SSH server config file, usually at /usr/local/etc/sshd_config or /etc/ssh/sshd_config depending on your install prefix. Customize options such as port, authentication methods, and allowed users.
Restart your SSH daemon to apply changes, for example:
```
sudo systemctl restart sshd
```
or manually start it if your system does not use systemd.
Verify your custom OpenSSH build. Check the version and compile options with:
```
/usr/local/bin/ssh -V
/usr/local/sbin/sshd -T
```
This confirms you are running your custom build with your chosen options.

Common Mistakes

Missing dependencies: Forgetting to install development libraries like libssl-dev causes configure failures.
Not cleaning old builds: Running make without make clean after reconfiguring can cause build errors.
Incorrect prefix paths: Installing to a non-standard location without updating system paths leads to running the wrong SSH version.
Ignoring test failures: Skipping tests or ignoring errors can result in unstable or insecure binaries.
Overlooking config file locations: Installing OpenSSH in a custom directory but not updating or creating config files causes the daemon to fail or run with defaults.
Running SSHD without root privileges: SSH daemon typically needs root or appropriate capabilities to bind to low-numbered ports and access keys.

Tips

Use a dedicated build directory: Keep source code and build artifacts separate for easier maintenance.
Backup existing SSH configs: Before installing your custom build, back up /etc/ssh/sshd_config and related files.
Enable verbose logging: Increase SSHD log level during testing to catch configuration or runtime issues (LogLevel DEBUG in sshd_config).
Cross-compile if needed: You can build OpenSSH for different architectures by setting environment variables and cross-compilers.
Stay updated: Regularly check for OpenSSH security patches and update your custom build accordingly.
Use feature flags wisely: Disable unused features to reduce attack surface, such as X11 forwarding or password authentication if not needed.

Troubleshooting

If your build or installation does not go as planned, consider these steps:

Build errors: Re-run ./configure and check output for missing dependencies. Install any missing -dev packages and try again.
Failed tests: Review test logs in tests/ directory. Some tests may fail due to environment constraints but critical failures indicate problems.
SSHD won’t start: Check system logs (journalctl -xe or /var/log/auth.log) for error messages. Verify config syntax with sshd -t.
Wrong SSH version running: Confirm which ssh or sshd binary is executing with which ssh and which sshd. Update your PATH or specify full path.
Port binding errors: If SSHD cannot bind to port 22, ensure no other SSH server is running and you have root privileges.
Authentication issues: Check PAM modules and permissions on key files. Adjust sshd_config authentication settings carefully.

Frequently Asked Questions

Can I build OpenSSH on Windows?: OpenSSH can be built on Windows using environments like Cygwin or Windows Subsystem for Linux (WSL), but it is more complex than on Unix-like systems.
How do I enable FIDO/U2F support?: Ensure libfido2 development libraries are installed and configure with --with-fido2. Then rebuild OpenSSH.
Is it safe to disable password authentication?: Yes, disabling password authentication in favor of public key authentication increases security, but ensure you have key access before disabling passwords.
How do I update my custom OpenSSH build?: Download the latest source, reapply your configure options, rebuild, and reinstall. Backup configs before upgrading.
Can I build OpenSSH with a different crypto library?: OpenSSH primarily depends on OpenSSL but supports LibreSSL and BoringSSL. You can specify paths during configuration if using alternatives.
What permissions should SSH host keys have?: Host key files should be owned by root and have permissions set to 600 to prevent unauthorized access.

Building OpenSSH with custom options empowers you to optimize your SSH setup for security, performance, and features. By following this guide, you avoid common pitfalls and ensure your custom build runs reliably. Regularly updating and testing your build further protects your systems from vulnerabilities.

For additional customization, consider exploring OpenSSH’s extensive configuration options and patch sets available from the community to tailor your SSH environment even more precisely.

How to Build OpenSSH with Custom Options Step-by-Step Guide

How to Build OpenSSH with Custom Options Step-by-Step Guide

What You’ll Need

Step-by-Step

Common Mistakes

Tips

Troubleshooting

Frequently Asked Questions

Related