Create a transfer

After you have created a share with a source, you are able to begin to create transfers. You do not need to have a destination set in the share to create a transfer.

When is the data sent?

• In the “options” object, set the transferNow parameter to “true” to create and send the transfer immediately. Set this value to “false” to only create the transfer, but not start it yet.
  • If you set the transferNow to true, you must have a destination set in the share.
  • You can use the “start/resume” transfer endpoint to start the data transfer.

What is the frequency of transfers?

Automated transfers

When the source is stored in cloud object storage, providers may select an entire bucket or specific folders (prefix), to be transferred to a specified cloud object storage technology and a region.

Once the transfer is started, all of the files under the selected folders are transferred to the destination under the prefix format /latest/.

Every 5 minutes Bobsled will check to see if there are changes to the selected folder paths and synchronize any new, updated, or deleted files to the destination.

Sending data as tables (Data warehouse destination)

• If you share data as tables (create a transfer entity with the type “table”), you must be sending data to a data warehouse destination.
• Each table transfer contains a loading pattern to represent how you want the tables to load in the destination. For more information, please visit: https://docs.bobsled.co/docs/table-loading-reference
• When sending data as tables, transferEntity is an array. This is because each transferEntity represents a single table.
  • To send multiple tables in this transfer, create multiple transferEntities. Each table/transferEnitity can have its own loading pattern and fileType configuration.

Sending data as files (File storage destination)

• If you share data as files (create a transfer entity with the type “files”), you must be sending data to a file storage destination.
• When sending data as files, transferEntity is an object. This is because only one transferEntity is created for a file transfer.
  • You are able to input multiple files/folders into the pathNames array to send many items in the transfer.
  • It is encouraged that you add folder paths to the pathNames array rather than individual file paths to not exceed the API’s soft limits. See Limits to learn more.

Important callouts

• When adding files/folders to the pathNames array, please make sure that you include the full path (including the file type). For folders, make sure that the folder name ends in a slash (‘/’).
  • For example:
    s3://databits-source-us-east-2/folder/file.csv
    s3://databits-source-us-east-2/folder/

Get a transfer and access the data

• To get information to access the transfer, the status of a transfer, transfer errors, and additional transfer metadata use this endpoint.
• To access the transfer, use this endpoint to retrieve the accessDetails.access string. Only the access identifiers configured in the share will have access to the transfers. To learn more about how to access the data transferred, please see our Destination documentation.
  • For file storage destinations: The URL to the Bobsled managed bucket/container to consume the data transfer using the access URL.
  • For warehouse destinations: The warehouse share identifier to consume the data transfer using the warehouse share identifier.

Update a transfer

• After you have created a transfer, you are able to update the contents of the transfer before you have sent it using this endpoint. Please keep in mind that the same rules apply to update a transfer as they do for the create a transfer endpoint.

Start/Resume a transfer

• When a transfer was created, but not yet started, use this endpoint to start the data transfer.

Pause an automated transfer

• This endpoint enables you to pause an automated transfer. When an automated transfer is paused, no data will be transferred to the destination until it is started again.

Editing an automated transfer

To edit an automated transfer:

1. Use “Pause an automated transfer” to pause the automation
2. Use “Update a transfer” to update the contents of the transfer
   1. Get the new transfer ID in response
3. Input the new transfer ID into the “Start/Resume a transfer” to resume the automation.

List all transfers

Use this endpoint to list all the transfers in a share.