Skip to content

Bulk File Operations

Single-file upload and download are covered in Core Concepts: File Upload and Download. This page covers operating on many files at once: uploading a set with progress reporting, downloading whole containers as a single archive, and iterating large file sets without exhausting memory.

For bulk import and export, prefer the CLI or web tools

The SDK is the right tool for programmatic file work — uploading derived results, pulling files for analysis, or wiring file operations into a script. For moving large volumes of data into or out of Flywheel, the purpose-built tools are faster and more robust: use the CLI or the web uploader to ingest data, and the web project exporter to export it. They handle batching, retries, and resumption for you. Reach for the SDK patterns below when you need that work to happen inside your own code.

Multi-file upload with progress

There is no batch-upload method — uploading many files is a loop over upload_file_to_*. The SDK upload methods do not expose a progress callback, so "progress" for a bulk upload means tracking your own position in the loop and printing it.

To upload (or download) many files concurrently rather than one at a time, see Performance Optimization: Parallel Processing.

Upload a set of files with progress reporting
--8 < --"test_docs_advanced_bulk.py:bulk_upload_with_progress"

To skip files that already exist at the destination when re-running an interrupted upload, check whether the file is already present on the container by name (for example, container.get_file(name)) before uploading.

Do not rely on file hashes to detect duplicates

File entries expose a hash field, but it is not always populated, so it is not a dependable basis for "have I already uploaded this?" logic. A file hash is computed only when the object is ingested directly from bucket / cloud storage — it is not generated for files uploaded through the SDK (nor through the CLI or UI upload paths). A file you uploaded with upload_file() will typically have hash = None, so a hash comparison silently treats every such file as new. Only compare hashes when you know your files originated from a bucket ingest; otherwise match on the file name.

Archive downloads (tar / zip)

To retrieve every file under one or more containers in a single request, use fw.download_tar() or fw.download_zip() instead of downloading files one at a time. Both accept a single container or a list of containers and support Projects, Sessions, Acquisitions, and Analyses.

Download a container as a tar archive
--8 < --"test_docs_advanced_bulk.py:download_tar"

download_zip() takes the same arguments and writes a zip instead:

Download a container as a zip, filtered by file type
--8 < --"test_docs_advanced_bulk.py:download_zip"

Both methods accept include_types and exclude_types to filter by file type, and return a summary describing the archive:

Parameter Type Description
containers container or list One container or a list of them. Projects, Sessions, Acquisitions, and Analyses are supported.
dest_file str Path on disk to write the archive to.
include_types list[str] Optional. Only include these file types (e.g. ["nifti"]).
exclude_types list[str] Optional. Exclude these file types (e.g. ["dicom"]).

include_types and exclude_types are mutually exclusive in practice — supply one or the other, not both.

A container object also exposes container.download_tar(dest_file, ...) directly, which is equivalent to passing that single container to the client method:

Download a single container via the container method
session = fw.get_session("000123456789abcdefABCDEF")
session.download_tar("session.tar", include_types=["nifti"])

Prefer an archive download over a per-file loop whenever you need most or all of a container's files — it is one request instead of one per file.

Processing large file sets

When a file set is too large to hold in memory, two rules keep memory flat:

  1. Iterate containers with iter() / iter_find() rather than find(), so only one page of containers is resolved at a time.
  2. Process each file as you reach it rather than collecting file objects into a list.
Iterate every file in a project one at a time
--8 < --"test_docs_advanced_bulk.py:iterate_large_file_set"

For files that are individually large, you do not have to download the whole thing to work with part of it: