Skip to main content

A quick SFTP server in Elixir

Utilizing the Erlang ecosystem to set up a quick SFTP server in Elixir.

· 10 min read
Max Mulatz
Andreas Knöpfle

SFTP - which stands for “Secure File Transfer Protocol” - is probably not the freshest and hippest kid in town any more. Many modern day developers may never had any interaction with it or only know it vaguely as a term from the internet’s past. As its name suggests, it’s a protocol used to securely transfer files to and from remote servers. And as it happens with old and robust tools and systems, it’s obviously still around and kicking! We recently had to integrate an Elixir application with a third party service that communicates via SFTP.

While this initially sounded like a task out of the comfort zone of our regular tooling, we were really surprised how well the Erlang ecosystem equipped us for the task. After some reading here and there we were, with surprisingly low effort, able to spin up our own SFTP server in Elixir and connect to it. While SFTP clients in Elixir are out and well documented, SFTP servers seem to be a rarer, less documented use-case. So we hope to shine some more light onto the topic here 🔦.

Why an SFTP server?​

Even in HTTP/2 times, there are enough established systems out there on the internet which still require communication via SFTP. It is often big-L legacy systems, old and forever running APIs where any change, outage or modernization would deprecate millions of integrations. At the core, communicating with these systems usually evolves around writing or reading files with very strictly formatted content and names on remote servers. It seems more involving than communicating with a Rest API, but for a lot of cases, it “just works™” and there is no need to change a running system.

In our Elixir applications, we usually build mocks for all external systems we connect with. Doing so, we - to a certain degree - have guarantees that our contracts with the external system are set up correctly, but don’t have to interact with them all the time, for instance during local development or testing. This has proven to be a good middle-ground between full autonomy from and full dependency on external systems.

For the majority of cases, the external third party is an HTTP service and we provide our own small and simplified implementation for it based on cowboy. With the integration of the service using SFTP for communication, we obviously needed something different: an SFTP server or at least something that acts like one and allows us to control the core behavior of things like authentication, etc. to make it appear close enough to the real world system.

Building our SFTP server​

Being mostly familiar with HTTP server mocks, our first thought was to write a GenServer in Elixir and have it implement the FTP protocol. We knew from our previous adventures into Erlang with sshkit.ex, that there is a great :ssh application for handling all the work around SSH. But for a mock server primarily used for local development, implementing the FTP protocol would have been an enormous effort and would have required to dive way deeper into the details of the protocol than actually necessary for the integration of the third party service alone. Searching the internet on the topic of SFTP and Elixir, the majority of content was about SFTP clients, not servers. But digging a bit, we found that our beloved Erlang ssh module (see Introducing SSHKit Erlang for another :ssh lovestory) could again prove useful here!

TL;DR we also have a working example of the code over at GitHub ➡️.

Erlang docs to the rescue​

The “using SSH” section of Erlang’s documentation of the :ssh picked us up just right! It even includes an example of an SFTP server which looked promising for us to utilize as a basis for the mock:

1> ssh:start().
ok
2> ssh:daemon(8989, [{system_dir, "/tmp/ssh_daemon"},
                     {user_dir, "/tmp/otptest_user/.ssh"},
                     {subsystems, [ssh_sftpd:subsystem_spec(
                                            [{cwd, "/tmp/sftp/example"}])
                                  ]}]).
{ok,<0.54.0>}
3>

Erlang code and the docs take a bit of time to get used to reading. But once one got there, they provided super detailed infos on how to get an SFTP server running and what is additionally required for it left and right. While it blew our mind to see just how much OTP ships with, the level of detail in the docs is at times a bit intimidating too: sometimes it’s too much detail and too many options. So let‘s pick our way through this together 👭.

SSH and SSH daemon​

The guide states:

Start the Erlang ssh daemon with the SFTP subsystem

The SFTP server is going to run as a subsystem within Erlangʼs :ssh application. So we first need to make sure, the :ssh app is started as part of the application callback in our mix.exs file, by adding it to the list of OTP apps our app depends on (extra_applications):

# in mix.exs
def application do
[
  extra_applications: [:ssh, :logger]
]
end

Next, we need to start the SSH daemon with an SFTP subsystem. Within the supervision tree, the daemon and respectively the SFTP server are going to be leaves of the :ssh application. To not just dump everything into the application callback and have a simple, but intuitive abstraction, we’ll add an SFTPServer module with a single start/0 function to our project:

defmodule SFTPServer do
  @moduledoc "A small SFTP server"

  @doc """
  Starting the server.
  """
  def start do
    # TODO: start the SSH daemon with the SFTP subsystem
  end
end

Starting the daemon takes the following arguments which are relevant for our use-case:

  • system_dir: path to a directory which contains a host key file (defaults to /etc/ssh)
  • user_dir: path to the SSH configuration directory of the connecting user (defaults to ~/.ssh)
  • subsystems: the list of subsystems we want to start

System Dir​

Our mock server should be small and autonomous within our Elixir application and does not need to have anything to do with the /etc/ssh directory of the machine we’re running our code on. So weʼll provide a system_dir path somewhere in the priv directory of our application and place a host key there. Erlang’s docs give a hint on how to generate that key:

ssh-keygen -t rsa -f priv/sftp_daemon/ssh_host_rsa_key

Authentication​

Erlang’s SSH daemon defaults to SSH-key authentication. For our SFTP mock server, authentication via password was actually closer to how the real server behaved though. Instead of the user_dir argument, Erlang also accepts a user_passwords list. Thatʼs what we’re going to use and just hardcode the most obvious combination: user -> password

{:user_passwords, [{'user', 'password'}]},

Note that we’re using charlists instead of strings here, as we’re dealing with an Erlang API ℹ️.

SFTP subsystem​

For the SFTP subsystem, we provide a directory which serves as the actual “remote server directory” where files are uploaded to and downloaded from. For our mock implementation we point this to a temporary directory. That’s enough for local development and testing. The important detail is that we want to specify this directory as root. Citing Erlang’s docs here:

Sets the SFTP root directory. Then the user cannot see any files above this root. If, for example, the root directory is set to /tmp, then the user sees this directory as /. If the user then writes cd /etc, the user moves to /tmp/etc.

Without restriction, we’d run the risk of our app messing with other files on the developers’ machines. A user connected to the SFTP server could happily navigate around everywhere else on the system ⚠️.

Starting the daemon​

Our start/0 function then looks like this:

def start do
  port = 8989
  system_dir = Path.join(:code.priv_dir(:sftp_server_example), "sftp_daemon")

  sftp_dir = Path.join(System.tmp_dir!(), "sftp_dir")
  File.mkdir(sftp_dir)

  opts = [
    {:system_dir, system_dir |> to_charlist},
    {:user_passwords, [{'user', 'password'}]},
    {:subsystems, [:ssh_sftpd.subsystem_spec([{:root, sftp_dir |> to_charlist}])]}
  ]

  :ssh.daemon(port, opts)
end

Port 8989 is from an Erlang docs example, so that’s also what we’re going with 🤷.

With sftp_dir = Path.join(System.tmp_dir!(), "sftp_dir"), we generate a path to a new temporary directory with the name sftp_dir somewhere down in the temporary directory of the host machine (most likely somewhere within /var/folders/<something-cryptic>/) and with File.mkdir(sftp_dir), we ensure that this directory actually exists. Otherwise, :ssh would not be able to start an SFTP server serving this directory.

Finally :ssh.daemon(port, opts) is the actual call to start Erlangʼs SSH daemon.

Start the SFTP server​

We can then wire up our small SFTP server and start it as part of our application in application.ex:

# in application.ex
@impl true
def start(_type, _args) do
  SFTPServer.start()

  children = []
  # …
  Supervisor.start_link(children, opts)
end

Note that the SFTPServer is not a child of our applicationʼs supervisor. It is managed in the supervision tree of Erlangʼs :ssh application instead.

With that, we can start our example app by running mix run --no-halt to keep it running and connect to it via SFTP:

sftp -P 8989 user@localhost

Weʼll be prompted for the password ("password"):

SSH server
Enter password for "user"
(user@localhost) password:
Connected to localhost.

Once there, we can perform some basic SFTP operations to verify weʼre on an actual SFTP server:

# See what's on the server:
sftp> ls
# nothing there yet…

# Let's upload a file:
sftp> put /Users/bitcrowd/Desktop/file.txt
Uploading /Users/bitcrowd/Desktop/file.txt to /file.txt
file.txt        100%  254   108.1KB/s   00:00

# Check if the file is there:
sftp> ls
file.txt

# Verify we cannot navigate up directories:
sftp> cd ../..
sftp> ls
file.txt

Thatʼs it, we have our own SFTP server 🎉.

Further topics​

For us, a small mock server was all we needed. But there are more topics which may become relevant when dealing with SFTP servers in Elixir:

SFTP client​

To communicate with an SFTP server from within our Elixir application, the community already got us covered with solutions like sftp_client. The same can of course also be accomplished in plain Erlang (see the docs for an example).

Authentication​

While our mock server uses password authentication, Erlangʼs default would have been SSH key authentication. For password-less key authentication of known users, we need to slightly adjust our options for the SSH daemon and provide a user_dir path:

user_dir = Path.join(:code.priv_dir(:sftp_server_example), "user_dir")

opts = [
  {:system_dir, system_dir |> to_charlist},
  {:user_dir, user_dir |> to_charlist},
  {:subsystems, [:ssh_sftpd.subsystem_spec([{:root, sftp_dir |> to_charlist}])]}
]

In this directory, :ssh expects an authorized_keys file with the SSH key of the connecting user. For illustration purpose we can place our own public key there:

cp ~/.ssh/id_rsa.pub priv/user_dir/authorized_keys

Our user should then be able to connect without providing a password:

sftp bitcrowd@localhost

When using this authentication mechanism for the mock server, you probably want to generate a key in Erlang.

Additional resources​

ps

If you enjoyed reading this, you might be interested in working with Elixir at bitcrowd. Check our job offerings!

Max Mulatz

Max Mulatz

Whitespace wrestler on a functional fieldtrip

Andreas Knöpfle

Andreas Knöpfle

Assistant code archeology officer

We're hiring

Work with our great team, apply for one of the open positions at bitcrowd