SFTP: Secure File Transfer Protocol - A Comprehensive Guide

SFTP (also known as Secure File Transfer Protocol or SSH File Transfer Protocol) is a secure network protocol used for transferring files between systems over an encrypted SSH connection. SFTP was designed to be universal, secure, and fast, while solving a lot of the limitations around the older and widely supported FTP protocol. 

What is SFTP used for?

SFTP is widely supported as a protocol and method of transferring files to and from servers or systems. Commonly used as a sysadmin and developer tool to quickly upload and download data from production and/or test servers, SFTP has become a common tool in the toolbox for moving data. 

But limiting SFTP to just that would be a bit insulting.

Today, SFTP has all but replaced FTP as the standardized method for automating the exchange and transfer of files to and from services and servers. SFTP exists in tech, healthcare, logistics, and finance to exchange files in a secure and robust way.

Much like FTP was a pillar in the 90s, SFTP has taken on that role today.

The SFTP Protocol

SFTP operates as a binary protocol that operates over a secure SSH channel. You can think of it as a pipe between two systems where each system is trying to send files to the other.

As a protocol, it's incredibly simple and has been built with extensibility and security in mind. SFTP works by leveraging the SSH protocol for authentication, encryption, and data integrity. SFTP over SSH uses asymmetric SSL encryption that is to date considered leading the way in terms of secure communication channels. SSH is fully supported by firewalls, NAT, and proxy servers, which means a lot of the limitations around FTP and modern internet infrastructure are solved intrinsically.  

Protocol Versions

The current version of the SFTP protocol is version 6, which was introduced with SSH-2. Earlier versions (1-5) were used with SSH-1 and are now considered obsolete due to security vulnerabilities in the SSH-1 protocol.

SFTP Authentication and Encryption

Since SFTP uses SSH for authentication and encryption, authentication is standardized and built around the robust backbone of SSH. Connections are established before the SFTP session begins, providing a secure and authenticated channel for file transfers.

SFTP supports various methods, including:

• Password-based
• Public key using SSH key pairs
• GSSAPI (Generic Security Services API)
• Two-factor authentication

Once authenticated, all data transmitted over the SFTP session is encrypted using strong encryption algorithms like AES, ensuring data confidentiality and integrity - unlike FTP, which is an insecure protocol.

SFTP Protocol Structure

Keeping things simple, the mavericks behind the SFTP protocol advocated for a very simple async messaging protocol, with options for extensibility baked in. Messages are sent back and forth between the client and server over TCP.

SFTP does not use external data channels like FTP, which makes NAT issues a thing of the past.

If you are curious; The SFTP messages look like this:

• Length: A 32-bit integer representing the packet length, including the header.
• Type: An 8-bit integer denoting the packet type (e.g., init, version, open, close, read, write).
• Payload: The packet-specific data, which varies based on the packet type.

The client initiates the SFTP session by sending an "init" packet, which includes the client's supported protocol version. The server responds with a "version" packet, specifying the agreed-upon protocol version for the session.

SFTP Ports and Firewall Considerations

SFTP typically operates over SSH port 22, which is the default port for the SSH protocol. This single-port design makes SFTP more firewall-friendly compared to traditional FTP, which requires separate ports for control and data connections (ports 21 and 20, respectively).

In some cases, admins may configure their SSH servers to listen on a non-standard port for added security or to avoid conflicts with other services. In such scenarios, the SFTP client must be configured to connect to the appropriate port.

When setting up firewalls, it's important to ensure that the SSH port (either the default 22 or the custom port) is open for inbound and outbound traffic to allow SFTP connections.

SFTP vs Other Protocols

Here's a comparison table of SFTP with other popular file transfer protocols and solutions:

Read more about protocols at What is SFTP and how are FTP, SFTP, SCP, and Rsync different? 

How To Transfer Files with SFTP

Using the SFTP protocol to transfer files is one of those things that can be done a few different ways and how you get it done will depend a lot on what problem you are trying to solve. There are of course a few common methods. In general, you would use an SFTP client or an SFTP server.

An SFTP Client is the piece of software that you are using to upload or download files. This client connects to the SFTP server with a hostname, username, password, and port. If you don't have these details from your "server provider", grab them quick, real quick.

When transferring files, you normally need a client. You have three main options when it comes to clients.  You can use the built-in client from the terminal, use an external GUI-based client, or use a proprietary connector built into the software that you want to upload or download files to or from.

The SFTP Server is the part on the other end. If you think you need a server, you can configure it manually or jump over to Couchdrop.io and set one up in minutes, with Couchdrop doing the difficult parts for you automatically.

Using SFTP from the terminal to transfer files

All modern operating systems support SFTP out of the box. This is part of what makes SFTP so useful as a protocol. However, using the SFTP client built into your operating system will require some familiarity with the terminal and command line tools. If this does not sound like your cup of tea, then jump down to the next section on using SFTP from the GUI to transfer files.

Uploading a file using the SFTP from the command line:

1. Open a terminal;
2. In the terminal, type 'sftp <username>@<hostname>:/'
3. Enter your password if prompted
4. Upload the file by typing 'put <filename and path>'

Downloading a file using the SFTP from the command line:

1. Open a terminal;
2. In the terminal, type 'sftp <username>@<hostname>:/'
3. Enter your password if prompted
4. Download the file by typing 'get <filename and path>'

Other commands:

You can navigate the SFTP server using the following commands as well:

• ls [<path>]: List the contents of a remote directory.

• cd <path>: Change the current directory on the remote server

• mkdir <path>: Create a new directory on the remote server.

• rmdir <path>: Remove a directory from the remote server.

• rm <path>: Remove a file from the remote server.

• rename <oldpath> <newpath>: Rename a remote file or directory.

Using SFTP from the GUI to transfer files 

If using the command line or terminal is not what you are familiar with, then a GUI SFTP client is going to make sense and be the fastest way to get up and running with SFTP. Luckily there are a lot of great GUI SFTP clients available, both for free and for a small license.

At Couchdrop, we are big fans of Transmit and Filezilla, but most SFTP clients work roughly the same. 

Common Clients Available

The beauty of a GUI client is that it's point-and-click. For most clients, you can simply:

1. Download the SFTP client and install it
2. Create a new connection
3. Provide the hostname, port, username, and password
4. Drag and drop files between the SFTP server and your local machine

Using SFTP to transfer files inside an application (Salesforce/Zapier/Etc.)

SFTP is so well supported as a standardized method for transferring files, that support for it is baked into many applications. This is a major advantage for SFTP over proprietary methods of data transfer and means you can import and export files into various different software applications automatically from any location. 

The configuration steps all vary slightly, but generally, you need to follow the steps below to use SFTP inside an application.

1. Configure a new connection
2. Provide the connection details, hostname, port, username and password
3. Provide a folder name

Troubleshooting and available options are normally well documented on the software provider's knowledge base. 

Automating File Transfers with SFTP

It's important to remember that SFTP is not a collaboration tool. Where it really shines is in backend file transfers and automated file transfers. With this in mind, it's remarkably easy to automate, allowing you to schedule and automate recurring file transfer tasks. Automating file transfers can save time, reduce manual effort, and ensure consistent and reliable data exchange between systems.

Automating file transfers is normally done in one of three ways.

1. By creating a bash or PowerShell script 
2. By creating a small application using a SFTP SDK (Like Python or Golang)
3. By using an automation platform like Zapier or Couchdrop

For more details about automating file transfers with SFTP, see our article Streamline Your File Transfer Process with Automated SFTP.

How-to: Uploading a file from a Bash Script using SFTP

You can write scripts using shell scripting languages like Bash or PowerShell to automate SFTP file transfers. These scripts can include SFTP commands to connect to the server, navigate directories, and transfer files. Here's an example of a simple Bash script that automates file upload:

#!/bin/bash

SFTP_SERVER="sftp.example.com"
SFTP_USER="username"
SFTP_PASSWORD="password"
LOCAL_FILE="/path/to/local/file.txt"
REMOTE_DIR="/path/to/remote/directory"

sshpass -p "$SFTP_PASSWORD" sftp "$SFTP_USER@$SFTP_SERVER" << EOF
put "$LOCAL_FILE" "$REMOTE_DIR"
EOF

How-to: Uploading a file with Python using SFTP

For more advanced automation scenarios, you can use programming languages and SFTP libraries or SDKs to develop custom SFTP automation scripts or applications. Many programming languages have SFTP client libraries that allow you to programmatically interact with SFTP servers. Some popular SFTP libraries include:

• Python: Paramiko, pysftp
• Java: JSch, Apache Commons VFS
• .NET: SSH.NET, Renci.SshNet
• Node.js: ssh2, node-ssh

These libraries provide APIs to establish SFTP connections, authenticate, navigate directories, and perform file transfer operations programmatically.

At Couchdrop, we are Python Fan Boys, so here's an example of using the Python Paramiko library to automate file downloads:

import paramiko

# Create an SFTP client
sftp = paramiko.SFTPClient.from_transport(paramiko.Transport(('sftp.example.com', 22)))

# Authenticate with username and password
sftp.login('username', 'password')

# Download a file
sftp.get('/path/to/remote/file.txt', '/path/to/local/file.txt')

# Close the SFTP connection
sftp.close()

By automating SFTP file transfers, you can streamline data exchange processes, reduce manual intervention, and ensure consistent and timely transfer of files between systems. Automation also helps minimize errors and provides a more reliable and efficient way to handle recurring file transfer tasks.

Using a file transfer automation platform

If you'd rather avoid writing code, there are plenty of platforms that make automating the transfer of files to and from an SFTP server really easy. These platforms offer reliability and simplicity and provide a myriad of scheduling and advanced options. 

Some of the services in the market at the moment:

• FileZilla offers the FileZilla Server, which supports event-based actions and file transfer automation.
• WinSCP provides a scripting and automation interface that allows you to create and execute SFTP automation scripts.
• Couchdrop provides file transfer automation and an SFTP server as a SaaS platform
• Zapier provides a very flexible automation platform that can upload and download files to SFTP

These automation platforms enable you to define file transfer tasks, schedule them, and set up event-triggered actions.

In some circles, they are also called Managed File Transfer Solutions (MFT). 

SFTP and Managed File Transfer (MFT) Solutions

Managed File Transfer (MFT) solutions often include SFTP as one of the supported protocols. MFT solutions provide a centralized platform for automating, managing, and monitoring file transfers across an organization. They offer features like:

• Workflow automation. Define complex file transfer workflows with conditional logic, error handling, and data transformations.
• Scheduling. Schedule file transfers to run at specific times or intervals.
• Event-driven transfers. Trigger file transfers based on specific events, such as the arrival of a new file or a change in a database.
• Auditing and reporting. Maintain detailed audit trails and generate reports on file transfer activities.

How to set up an SFTP server

When it comes to setting up an SFTP server, you have two main options:

1. Set up an SFTP server manually by installing a SFTP server on your own infrastructure
2. Use a SaaS SFTP solution like Couchdrop

Setting up an SFTP Server Manually

To set up an SFTP server manually, you'll need to follow these steps:

1. Choose an SFTP Server Software: Select an SFTP server software that suits your needs. One popular option is OpenSSH, which comes pre-installed on most Linux distributions.

2. Install the SFTP Server Software: Install the chosen SFTP server software on your server machine. The installation process may vary depending on the operating system and the specific software you are using.

3. Configure SSH Server Settings: Modify the server configuration file (usually located at /etc/ssh/sshd_config on Linux) to enable SFTP and set the desired settings. Some common settings to configure include:

   • Port: Specify the port number on which the server should listen (default is 22).

   • PermitRootLogin: Determines whether root login is allowed (recommended to set to "no" for security reasons).

   • PubkeyAuthentication: Enables public key authentication (set to "yes" for key-based authentication).

   • PasswordAuthentication: Enables password-based authentication (set to "no" if only key-based authentication is desired).

   • Subsystem sftp: Specifies the path to the SFTP server subsystem (e.g., /usr/lib/openssh/sftp-server).

4. Create SFTP User Accounts: Set up user accounts for SFTP access. You can use existing system user accounts or create new ones specifically for SFTP.

5. Configure User Permissions: Set appropriate file and directory permissions for SFTP users. Use tools like chmod and chown to control access rights.

6. Configure Firewall: Open the necessary ports in your firewall to allow SFTP connections. By default, SFTP uses port 22. If you configured a different port, make sure to open that port as well.

7. Restart the SSH Server: Restart the server to apply the configuration changes. On Linux, you can typically use the command sudo systemctl restart ssh or sudo service ssh restart.

8. Test SFTP Connection: Use an SFTP client to test the connection to your newly set up SFTP server. Verify that you can successfully authenticate and perform file transfer operations.

By following these steps, you can set up a basic SFTP server. However, depending on your specific requirements, you may need to perform additional configuration, such as setting up chrooted directories, enabling logging, or configuring security features like IP whitelisting or rate limiting.

It's important to regularly update your SFTP server software to ensure you have the latest security patches and features. Additionally, monitoring SFTP server logs can help you detect and investigate any suspicious activities or potential security breaches.

Use a hosted, managed SaaS platform like Couchdrop

Couchdrop is a managed SaaS SFTP solution that simplifies the process of setting up and managing an SFTP server and integrates with your Cloud Storage (Dropbox/Sharepoint/S3). Couchdrop takes the pain out of server management away and offers a simple, integrated, and scalable SFTP platform.

Setting up and configuring a cloud SFTP server is incredibly simple with Couchdrop:

1. Create an SFTP Server. Sign up for a Couchdrop account and create a new SFTP server with just a few clicks. Choose a server location, specify the storage capacity, and set up user accounts.

2. Configure your SFTP Server. Setting up Couchdrop is done via an intuitive web interface. 

3. Customer Support. All customers benefit from dedicated customer support provided by Couchdrop. Get assistance with setup, configuration, and troubleshooting if you ever need help.

By using Couchdrop, you can offload the complexities of setting up and managing an SFTP server. Couchdrop takes care of the underlying infrastructure, security, and maintenance, allowing you to focus on your core business activities.

Couchdrop is particularly suitable for businesses that require a reliable, scalable, and easy-to-use SFTP solution without the overhead of managing the server themselves.

Case Study: Inzata's SFTP Solution with Couchdrop

Inzata, a data analytics platform, faced challenges in receiving data files from their customers in various formats like Excel files and CSVs. Their existing solution using FileZilla was cumbersome and required significant manual effort.

Alex Durante, the Product Director at Inzata, explains, "Our developer would have ended up keeping track of 50 SFTPs himself, which wasn't feasible."

Inzata discovered Couchdrop, an SFTP solution that streamlined their file transfer process. Instead of managing separate SFTPs for each customer, Couchdrop provided a single SFTP with individual folders for each customer, making it easy for Inzata to manage.

Alex highlights the benefits of Couchdrop:

• Simplified SFTP setup and management for customer support team
• Reduced turnaround time for onboarding new customers
• User-friendly drag-and-drop interface for less technical customers
• Enhanced privacy and security by acting as a transfer portal without storing data
• Improved customer experience and conversion rates

Alex also praised Couchdrop's exceptional customer service, stating, "Any time we had an issue with anything, they went above and beyond to help us and even put in place custom items specifically for us. There's nothing about Couchdrop I dislike or would change."

Inzata's success story demonstrates how SFTP solutions like Couchdrop can simplify and automate file transfer processes, enhance security, and improve the overall customer experience for data-driven businesses.

SFTP: a secure and reliable file transfer protocol

SFTP is a secure and reliable protocol for transferring files between systems, offering strong encryption, authentication, and data integrity. Its single-port design and firewall-friendly nature make it an ideal choice for automated file transfers in various business scenarios.

If you would like to learn more about SFTP or Couchdrop, visit our website.