.svg){kind=logo}
If you’re looking for a way to move files between two locations, you may have come across a few options that seem like they’re doing the same thing. After all, MFT, ETL, and ELT all seem like they automate file transfers and make changes to the files in the process. So how are they different? And which option is right for you?
What is Managed File Transfer?
A Managed File Transfer (MFT) platform assists in transferring files between different locations. It acts as a mid-point between the source and destination, commonly used when the two endpoints can’t directly communicate or transfer files between each other. As the middleman, the file transfer is initiated using the MFT platform–which supports multiple transfer protocols–over to the destination, which may support a whole different set of protocols.
What the MFT platform does is bridge that communication gap, allowing users to initiate a transfer between a source and destination that normally wouldn’t be possible, or by using a more secure transfer protocol like SFTP. Often, Managed File Transfer platforms temporarily store data on a storage layer using something like S3, then transfer the file from that storage layer to the final destination.
However, that temporary storage can be a problem when it comes to sensitive data like healthcare information and financial data. To avoid storing these sensitive files, some more recent MFT platforms like Couchdrop are able to “stream” the data directly between the source and destination without ever storing the files at any point. Instead, the platform provides an encrypted path between the source and destination platforms.
Often, this process can be completely automated. This can shave time off transfer workflows by removing all of the manual components like grabbing the file, renaming it, and uploading it to the destination. Instead, all of these processes can be set up in the MFT platform and forgotten about, letting the transfer happen automatically in the backend instead.
For more information on MFT, see What is Managed File Transfer?
What is ETL?
ETL stands for Extract, Transform, Load, which are the main stages of transferring and altering files when using one of these platforms. Unlike Managed File Transfer, ETL not only takes the file and puts it in the destination, but it transforms the file contents itself along the way.
In an ETL platform, the Transformation stage happens in a staging environment. Here is where the file contents are changed before being Loaded into the next stage. Transforming data in this environment is ideal when the next stage has a predefined scheme that the data needs to match.
One important difference between ETL and transferring files via SFTP or another method is that the data itself needs to be changed (transformed) as part of the process. This makes the emphasis on altering data over efficiently transferring from one place to another like with MFT.
What is ELT?
ELT has the same three stages of ETL, just in a different order: Extract, Load, Transform. The difference is which spot the Load and Transform stages are at.
As it's become possible to quickly move larger data sets thanks to advances in data transfer speeds and storage options, ELT has become a more viable option. Data lakes and warehouses can hold massive quantities of data, which can move quickly thanks to increased bandwidth speed.
With ELT, the target system is almost always a data warehouse or data lake. Data sets can stay unchanged until transformations need to be performed, instead of having to be transformed before being moved like with ETL. From there, files can be moved into other applications.
ETL vs ELT
At first glance, ETL and ELT seem like they’re doing the exact same thing but in a different order, so they should achieve the same result. But where the data is transformed can make a significant difference in the outcome.
ETL and ELT platforms are good when the data isn’t in a state that’s ready to use. For example, suppose a piece of software exports a report with useful data, and then that data needs to be imported into another piece of software to be analyzed. However, that software has a specific template that’s different from the original file. One option is to manually change each file as it’s created, but this is impractical at scale, so these platforms can rename rows, split data, and normalize templates from multiple sources so they can be ready to go once they’re loaded.
In ETL, transforming the data involves a staging area. Why this matters is because the data has to be loaded into the system before it’s changed, so it has to be in a compatible format already. And, because the data is loaded first, that means the heavy compute operations are done on the target system instead of a separate staging area.
ELT does the transforming from within the system, usually a data lake or data warehouse. Doing it here allows for more flexibility but requires more compute power and tends to be a slower process.
Depending on your specific systems and workflow, one method or the other might be better. Some IPaaS (Integration Platform as a Service) will even do both, allowing you to choose either method or ETLT, Extract, Transform, Load & Transform (again).
In the end, both ETL and ELT accomplish similar things, but go about it in a different way.
ETL & ELT vs MFT
ETL, ELT, and MFT all grab files from a source and move them somewhere else, but the method and reason for doing it are much different. Because of this, the way that files are transformed, workflows are set up and used, and file transfers are handled is quite different.
Transforming files
Both ETL/ELT and MFT “make changes” to files, but they act on different parts. While ETL/ELT goes into the file, making changes to rows and contents, MFT only changes the file metadata such as the file name and adding file encryption. The contents aren’t altered or touched at any point, and the file moves as is. These will usually be files like standard office format, photos, and videos.
The goal of ELT and ETL isn’t to share files or collaborate, but instead to use data for integration, warehousing, and analytics. Data can be pulled in from multiple sources, consolidated, and transformed for one of these purposes. Data can be unstructured, like documents, videos, or images, or structured, like spreadsheets and data tables.
Automated workflows
Since MFT is all about transferring files, setting up workflows is quick with an emphasis on the transfer portion, and has actions like renaming and making copies.
ELT and ETL automations, on the other hand, can take a significant investment to get up and running. This is because it’s not only the connection between the sources and destinations that need to be configured, but also how the data will be transformed and changed. Setting up and testing these rules can be time-consuming and involve several iterations, but once they’re done, the processes can be automated at scale with minimal intervention.
Because ETL and ELT platforms move files between systems that are within an organization, the emphasis is on having high-quality data for internal use. Workflows load and transform data for processing or other applications. MFT, on the other hand, is usually for exchanging files with partners, vendors, and other external parties, so automations are designed around that purpose.
Transferring files
Because ELT/ETL and MFT have a different purpose, they transfer files in different ways. ETL and ELT platforms are usually integrated into a highly-secure environment that is tightly managed and controlled. The systems are usually all internal and managed by the organization.
MFT usually involves exchanging files between organizations and can involve a number of different external parties. Because of this exposure, it isn't possible to control the entire environment, so using secure transfer protocols is very important for data integrity. Protocols like SFTP and AS2 are important for different kinds of file exchange, as well as being able to encrypt/decrypt sensitive files.
While ETL/ELT focus on flexibility and transforming data to fit numerous different systems, MFT focuses on secure and reliable transfers between different parties.
When should you choose ELT or ETL?
When literally millions of data points are being passed along to different applications, using an ELT or ETL platform for significant transformations makes sense.
Instead of transferring files between platforms, ETL/ELT “load” the data for further immediate action. The loaded data is then used for other applications like Tableau and Amazon Redshift to analyze data at scale. For business intelligence, detailed analytics, forceasting, and big data analysis, ELT/ETL is a solid choice, especially when you have huge amounts of data volumes that need to be transformed like when working with a data lake or warehouse.
For these purposes, a tool like Skyvia would be a good option to look into.
When should you choose MFT?
For transferring files from A to B, what matters is how the transfer process itself is handled. If the file contents themselves don’t need to be changed (or shouldn’t be due to compliance), using a Managed File Transfer platform makes more sense than ETL or ELT. You can automate transferring between normally incompatible platforms in a secure way and change file metadata in the process.
MFT is usually much easier and faster to get set up as well. Since the focus is on the transfer portion itself, the performance of transfers is quick, secure, and easy to manage. When the goal is to exchange files between other organizations with a focus on security and no need to alter file contents, look into Managed File Transfer platforms like Couchdrop.
Try Couchdrop for a simpler MFT platform
Making sure files transfer quickly and reliably between organizations is crucial, but it doesn’t have to be complicated. Couchdrop simplifies setup and management of secure automated file transfers with no infrastructure to worry about and no-code automation setup with pre-built connectors. Connect your cloud or on-prem storage in a few clicks, or use Couchdrop’s hosted storage and start automating transfers in minutes.
You can try Couchdrop for free for 14 days with no credit card required. Connect platforms, set up automations, and test that Couchdrop works for your use case. Register for an account now to get started.