Pull from FTP
Pull in files from an FTP, SFTP, or FTPS server. Target exact files, or find files based on rules within a folder. Supports csv, tsv, excel, xml, and json file parsing. Can parse edi files as csv.
Connecting to your FTP server
The first thing that you need to do is connect to your server in order to pull in any files.
When you first add an FTP step to a flow, you can open it and will see an Authorize button.
Click Authorize, and you will see this form:
You will need to fill in each field in order to connect.
The Port can be manually set, or it will default to a port depending on which choice you have selected for the transfer protocol.
Using FTP (instead of SFTP or FTPS) is not recommended. Most FTP servers offer one of the other options.
If you are connecting via SFTP and are using a private key to connect, you will need to check the "Use public key authentication" box to see the option to upload that key and connect.
Editing your FTP connection settings or adding another connection
If you need to edit or add another connection, open your FTP step, click on "Select accounts", and then either click to add a new account, or edit the existing one.
After editing your connection settings, click the refresh button to have the step re-connect with the new settings.
Pulling in a specific file
The main option at the top of the step allows you to switch between pulling in a specific file and a file based on rules.
When pulling in a specific file, enter the path to that file. All paths should start with / and then progress through any folders until ending in the name of the file and its extension.
Click the 3-dot more menu to override how to parse a file. By default, this step will parse the file based on the extension of the file. But you can change that. For example, if you have a .txt file that is really a csv file inside, you can choose to parse that txt file as if it were a csv.
Pulling in a file based on rules
The main option at the top of the step allows you to switch between pulling in a specific file and a file based on rules.
When pulling a file based on rules, a new file will be pulled in every time the flow is run, or the step is refreshed.
A file can be selected based on:
- The last_modified date of the file
- The folder the file is in
- Matching part of the name of the file
First, choose between pulling the newest file or the oldest file, based on its last modified date.
Second, choose a file name pattern. If you select is anything, no filtering based on file name will be applied. You can select to filter for files that start with, end with, or contain a certain set of characters. This can also be used to match the file extension (.csv for example).
Third, choose a folder to find the file within. If you use / then it will search the root folder. Other folders that are inside of the folder that you have indicated will not be searched and will be ignored.
Finally, select a parsing option if you want to override the default.
Every time a file is pulled in from a rule, the name will be displayed in the step settings.
Moving files after processing (archiving)
Enable the Archive file once processed setting to automatically move files from the target folder to a different folder.
Files will be moved immediately after the data from the file is fetched by the Pull from FTP step. If the step fails for some reason with an error, the file will not be moved.
If the file is pulled in successfully, but another step causes the Flow to fail, then the file will still be archived, even if the overall Flow failed to complete.
In the run history of the Flow, the names of any files pulled in from FTP will be listed to show what file was moved during successful or failed runs.
Use of this setting is best combined with the “Pull in a file based on rules” setting. With this combination, a Pull from FTP step can continuously cycle through a specific FTP folder and process any files available within it.
Parsing XML Files
Sometimes XML files will not successfully pull into this step. In that case, it may be due to how the step is parsing the file by default. Use the Top Level Key field to indicate which key to expand into the table. This can help if there is a list of data, but there are other keys surrounding it, and you just need to get to that interior list. You can indicate a deeper key by placing dots between each key level. For example, if you have an object called Cars, and inside it is a list called Colors, which you want to expand, you would put Cars.Colors in the Top Level Keys field.
Limits
This FTP step can be used to pull in files up to 600MB. Contact us if you need larger files to be pulled in.
Global limits my stop your file before its size does, however. Steps can only run for 1 hour, and can only pull in 5 million rows and 5000 columns.
Related Recipes
Send to FTP
Create or overwrite files in an FTP, SFTP, or FTPS server. Supports CSV, TSV, Excel, and JSON file creation and overwriting.
Connecting to your FTP server
The first thing that you need to do is connect to your server in order to send any files.
When you first add an FTP step to a flow, you can open it and will see an Authorize button.
Click Authorize, and you will see this form:
You will need to fill in each field in order to connect.
The Port can be manually set, or it will default to a port depending on which choice you have selected for the transfer protocol.
Using FTP (instead of SFTP or FTPS) is not recommended. Most FTP servers offer one of the other options.
Editing your FTP connection settings or adding another connection
If you need to edit or add another connection, open your FTP step, click on "Select accounts", and then either click to add a new account, or edit the existing one.
After editing your connection settings, click the refresh button to have the step re-connect with the new settings.
Creating a new file each time
The main option at the top of the step allows you to switch between creating a new file, and overwriting a file.
When creating a new file, you have a few settings to fill out:
- The format of the file
- The name of the file
- What folder to place the file within.
JSON Files
JSON files generated have their array as the top level element. Each row will be converted into an object, and then each row-object will be comma separated in the top level array.
Given data in Parabola that looks like this:
You can expect JSON that looks like this:
Excel Files
Excel files that are created by this step are in the .xlsx format. They will have no additional formatting applied.
Naming a file
In the field for the name of your file, you can type anything that you'd like to name your file. Do not include the extension, as one will be automatically added by the step, according to the format you have chosen.
If you put "my file.csv" in the file name field, and then have the step create a CSV file, it will ultimately be named "my file.csv.csv" in your FTP server.
Most servers will not be happy if you try to name a file, and that name already exists in that folder. To get around this, you can use merge tags to add dates and time to your file name. Anywhere you place that tag in the name field, the date of the run will be inserted in the following formats:
- {date} YYYY-MM-DD
- {dateNoDash} YYYYMMDD
- {dateTime} YYYY-MM-DD HH:MM:SS
All dates and times are in UTC timezone.
Selecting a folder to put the file in
The final setting is used to indicate where the file should go.
The root of your server will be at / and any other folder will start with / as well. If you have a folder named "reports" that is located in the Root folder, then you would use /reports in the folder field.
Overwriting a file
The main option at the top of the step allows you to switch between creating a new file or overwriting a file.
Overwriting a file is simple - enter the path to the file to overwrite each time, and the format for the new data inside that file.
Selecting a format
It is best to select the format of the file that it's extension indicates. Because the data is fully replaced within the file, the format that Parabola sends does not strictly need to match the format that the name of the file indicates.
For example, you could send CSV data to a file named jobs.txt and it would work fine. But having an extension on a file that does not represent how it should be used or read can cause issues down the line.
Selecting a file to overwrite
The final setting is used to indicate the path to the file to overwrite.
Paths should always start with a / which is the root folder. From there, you can add more folders (or not), and end with the file name and its extension.
In the image above, we are targeting a file named customers.csv which is in the root folder. If that file was in a sub folder named crm, then the path would look like this:
