Performance Optimization

Flywheel SDK operations are network-bound — every find(), upload_file(), and update() call is at least one HTTP request. This page covers three strategies for improving script throughput: choosing the right iteration method to avoid loading more data than necessary, using the SDK's built-in async support to dispatch multiple requests concurrently, and applying thread- or process-based parallelism for larger-scale workloads. For strategies around minimizing redundant fetches and managing stale data, see Reloading and Caching.

Efficient Iteration

The Finder interface offers several methods for retrieving results. Choosing the right one has a direct impact on both memory usage and request count.

When to use iter() vs find()

find() with no limit collects all matching results into a Python list before returning. For collections with thousands of items this means holding the entire result set in memory at once.

iter() and iter_find() yield items one page at a time — the next page is not fetched until the current page is exhausted. Use them whenever the result set might be large or when you want to process items as they arrive.

# Small, bounded query — fine as a list
recent_sessions = project.sessions.find("created>=2024-01-01", limit=50)

# Large export — stream to avoid loading all into memory
for session in project.sessions.iter_find("label!=excluded"):
    process(session)

# Single lookup — returns as soon as the first match is found
session = project.sessions.find_first("label=baseline")

See Core Concepts: Finders for the full method reference and filter syntax.

Lazy loading with iterators

iter() and iter_find() use cursor-based pagination by default (after_id). The SDK fetches each page only when iteration reaches the end of the previous one. This means memory usage stays proportional to one page — not the full result set — and network requests are spread over the lifetime of the loop rather than made all at once.

count = 0
for acquisition in fw.acquisitions.iter_find("label=~.*T1.*"):
    count += 1
print(f"Found {count} acquisitions")

When you pass a sort argument to iter_find(), it falls back to page-number pagination because cursor pagination is incompatible with sorting. This is still lazy — pages are fetched on demand — but adds a small overhead from computing sort order on the server.

Batch size considerations

The default page size for iter() and iter_find() is 250 items per request. Adjust the limit parameter to tune throughput for your use case:

• Small pages (50–100): Lower per-request latency, more round trips. Useful when processing time per item is high (for example, a download per item) and you want fetching to interleave with processing.
• Large pages (400–800): Fewer round trips, higher per-request latency. Better for pure enumeration or lightweight transforms.
• Above 1 000: Risk of server-side timeouts and oversized responses. Avoid.

for session in project.sessions.iter(limit=500):
    ids.append(session.id)

Memory management for large datasets

When processing large hierarchies, avoid accumulating full container objects in memory. Extract only the fields you need as you iterate:

--8 < --"test_docs_advanced_usage.py:iter_extract_fields"

For very large datasets, write results out incrementally rather than building a single in-memory list:

import csv

output_path = "sessions.csv"  # path to output file
--8 < --"test_docs_advanced_usage.py:iter_write_csv"

The same stream-and-write approach works for any external store — a relational database, a data warehouse, or a flat file. Streaming into a local SQLite database, for example:

import sqlite3

db_path = "export.db"  # path to output database
--8 < --"test_docs_advanced_usage.py:export_sqlite"

Because each iteration processes one item, this scales to result sets far larger than available memory. To export across the whole instance rather than one container, pair it with advanced search for discovery and a finder or fw.get() for the live, authoritative record of each hit.

Fetching full subtrees with fetch_tree

When you need data from multiple hierarchy levels together — sessions alongside their acquisitions and files, for example — iterating level-by-level issues one HTTP request per container. For a project with hundreds of sessions and thousands of acquisitions, that can mean thousands of API calls.

fw.fetch_tree() replaces that pattern with a single API call. You describe the shape of the response as a nested dict, specifying which fields to include at each level:

--8 < --"test_docs_advanced_usage.py:fetch_tree_project"

Set limit: 0 at any level to retrieve all items at that level. Results are plain dicts — fields not listed in fields are absent from the response, which keeps payloads compact even for large hierarchies.

Each level also accepts a filter key (using the same Finder filter syntax) and a sort key ("label:asc", "timestamp:desc", etc.). Pass include_all_info=True to fw.fetch_tree() to include all custom info fields at every level:

--8 < --"test_docs_advanced_usage.py:fetch_tree_filter_sort"

Use fetch_tree when:

• You need data from two or more hierarchy levels at once.
• The subtree is bounded and fits comfortably in memory.
• Minimizing total API call count or latency is a priority.

Keep per-level iteration when:

• The result set is too large to hold in memory — fetch_tree returns the full response before any processing can begin.
• You need streaming or incremental processing.
• Finder-specific capabilities (find_first, find_one, cursor pagination) are required.

Async Calls

The SDK's own async_=True mechanism is thread-based and does not use async/await — it uses multiprocessing.pool.ThreadPool internally. For async/await usage with the SDK, see asyncio later in this section.

Pass async_=True to an underlying API method (for example, fw.get_all_sessions()). The call returns a ThreadPool.AsyncResult immediately, without blocking. Call .get() on the result to block and retrieve the value. If the background thread raised an exception, .get() re-raises it in the calling thread.

The primary use case is firing several independent queries concurrently and collecting results after all requests have been dispatched.

--8 < --"test_docs_advanced_usage.py:async_parallel_queries"

This pattern is more useful when you have many queries — fan out first, then collect:

project_ids = [
    "000123456789abcdefABCDEF",
    "000123456789abcdefABCDE0",
    "000123456789abcdefABCDE1",
    "000123456789abcdefABCDE2",
]

pending = [fw.get_all_sessions(filter=f"project.id={pid}", async_=True) for pid in project_ids]

all_sessions = []
for result in pending:
    all_sessions.extend(result.get())

Flywheel exposes a get_all_<container> method for every container type — get_all_subjects, get_all_sessions, get_all_acquisitions, get_all_projects, and get_all_files — each running a finder-style filter across the whole instance and each accepting async_=True.

get_all_files is the odd one out: it returns a PageGenericFileOutput page object rather than a plain list. Read .results for the files, .count for the size of this page, and .total for the full match count.

page = fw.get_all_files(filter="type=dicom", limit=500)
print(f"{page.count} of {page.total} matches in this page")
for file in page.results:
    print(file.name, file.file_id)

# Same query, async
result = fw.get_all_files(filter="type=dicom", limit=500, async_=True)
page = result.get()

For bulk metadata updates, the same pattern applies to write operations:

from flywheel.models import Info

--8 < --"test_docs_advanced_usage.py:async_metadata_update"

Concurrency limit: The SDK's internal thread pool and HTTP connection pool are both size 10 by default. Dispatching more concurrent requests than the connection pool can hold produces Connection pool is full, discarding connection warnings in the logs as excess connections are dropped and recreated — the requests still complete, but with added overhead. Keep concurrent dispatch to 4–16 requests for most Flywheel instances.

Async-capable container queries

Finder.find() cannot run async, but the SDK's get_<parent>_<children> client methods can. Each one fetches a container's children directly and accepts the same finder-style filter as the equivalent .find(), plus sort, limit, skip, page, after_id, and async_=True. They return a plain list of containers, so an async_=True call hands back an AsyncResult whose .get() resolves to that list.

For queries that span the whole instance rather than one parent, see the instance-wide get_all_* family above.

Pass include_all_info=False to drop custom-info payloads from the response when you don't need them — it noticeably shrinks each page on info-heavy containers.

Paginating an unknown number of containers

These calls page by limit/page, and the list they return is the page itself. You usually don't know the page count up front, so page until a page comes back shorter than limit — that short (or empty) page marks the end of the collection. Fire each batch of pages concurrently and stop as soon as any page in the batch is short:

def get_all_project_acquisitions(fw, project_id, limit=500, pool_size=10):
    """Fetch every acquisition in a project, paging concurrently until exhausted."""
    acquisitions = []
    page = 0
    while True:
        batch = [
            fw.get_project_acquisitions(project_id, limit=limit, page=page + offset, async_=True)
            for offset in range(pool_size)
        ]
        pages = [result.get() for result in batch]
        for page_results in pages:
            acquisitions.extend(page_results)
        if any(len(page_results) < limit for page_results in pages):
            break
        page += pool_size
    return acquisitions

The any(len(page) < limit ...) check is the entire termination condition: pages are sequential, so the first short page means every later page is empty. Overshooting the end just collects a few empty pages in the final batch, which extend ignores.

A limit of 400–800 is the sweet spot here — large enough to keep the round-trip count down, small enough to stay clear of the server-side timeouts that hit above ~1 000 (the same range recommended for iter() page sizes earlier on this page).

asyncio

The Flywheel SDK is not async-native — its methods are not coroutines and cannot be called with await directly. If you are already working in an async/await codebase, use asyncio.to_thread to run SDK calls in a thread without blocking the event loop:

import asyncio
from flywheel.models import Info

--8 < --"test_docs_advanced_usage.py:asyncio_to_thread"

Because asyncio.to_thread creates a new OS thread per call, dispatching many tasks at once can exceed the SDK client's connection pool size (10 by default) and produce Connection pool is full warnings. Use a semaphore to cap concurrency:

--8 < --"test_docs_advanced_usage.py:asyncio_semaphore"

For a fully async-native experience, the fw-client Python package provides async variants of HTTP methods (aget, apatch, etc.). Unlike the Flywheel SDK, FWClient uses a single async connection and does not produce connection pool warnings:

from fw_client import FWClient
import asyncio

fw_async = FWClient(api_key=api_key)


async def update_subject(sub_id):
    await fw_async.apatch(f"/api/subjects/{sub_id}/info", json={"set": {"reviewed": True}})


async def main():
    tasks = [update_subject(s["_id"]) for s in subjects]
    await asyncio.gather(*tasks)


asyncio.run(main())

Limitations

async_=True is best suited to a small set of independent, known API calls where the fan-out pattern is straightforward. Its main constraints:

• Fixed pool size. The SDK's internal thread pool cannot be configured — there is no max_workers equivalent. You cannot tune concurrency beyond the SDK default.
• Deferred errors. Exceptions are held until you call .get(). If you dispatch many calls and do not call .get() on each result, errors from those calls are silently lost.
• Finder methods are not supported. find(), iter_find(), and related paginated methods do not accept async_=True. To get finder-style filtering with async, use the project-level client methods instead — see Async-capable container queries above.

When you need configurable concurrency, per-request error handling, or need to parallelize Finder-based queries, use ThreadPoolExecutor instead — see Parallel Processing below. For codebases that already use async/await, see asyncio above for the asyncio.to_thread and FWClient approaches.

Parallel Processing

ThreadPoolExecutor (from Python's concurrent.futures standard library module) picks up where async_=True leaves off. It works with any callable — including Finder methods and file operations — and gives you direct control over pool size, task submission, and per-request error handling. HTTP I/O releases Python's Global Interpreter Lock (GIL), so threads issue requests simultaneously without being serialized by the interpreter.

Using concurrent.futures

Use ThreadPoolExecutor for I/O-bound tasks such as uploading files, downloading files, or reading metadata for many containers in parallel.

import concurrent.futures


def upload_report(acquisition_id):
    fw.upload_file_to_acquisition(
        acquisition_id,
        flywheel.FileSpec("report.csv"),
    )


project_id = "000123456789abcdefABCDEF"
acquisition_ids = [a.id for a in fw.acquisitions.iter_find(f"parents.project={project_id}")]

with concurrent.futures.ThreadPoolExecutor(max_workers=8) as executor:
    futures = [executor.submit(upload_report, acq_id) for acq_id in acquisition_ids]
    for future in concurrent.futures.as_completed(futures):
        future.result()  # re-raises any exception from the worker

When you need results in the original order and do not need to handle exceptions per item, executor.map() is more concise:

import concurrent.futures

--8 < --"test_docs_advanced_usage.py:threadpool_fetch_sessions"

Thread safety considerations

The Flywheel Client object is safe to use from multiple threads — each method call issues its own independent HTTP request. However, container objects are not thread-safe for writes. Do not call update(), update_info(), or upload_file() on the same object instance from multiple threads simultaneously, and do not mutate a container object's attributes from more than one thread.

The safe pattern is to pass IDs between threads and let each thread fetch its own copy of the container:

--8 < --"test_docs_advanced_usage.py:threadpool_pass_ids"

Threads vs. processes

ThreadPoolExecutor (threads) and ProcessPoolExecutor / multiprocessing (separate OS processes) are both valid for parallelizing Flywheel SDK work, but they have different trade-offs.

Use threads when your workers spend most of their time waiting on HTTP responses. SDK API calls release the GIL, so threads give real concurrency for this workload with minimal overhead.

Use processes when workers do substantial CPU work per item (parsing DICOM headers, image transforms, running inference), or when you want full isolation between workers so that one failure does not corrupt shared state. The key requirement is that each worker process must create its own flywheel.Client instance — clients cannot be forked or pickled across process boundaries.

Any data passed as arguments to worker functions or shared across processes via multiprocessing.Manager must be pickleable. Standard Python types (dicts, lists, strings, numbers) work. File handles, lambda functions, and Client objects do not.

For large-scale hierarchy curation — where a walker process distributes containers to a queue and independent worker processes pull from it — the fw-curation library implements this pattern on top of the Flywheel SDK. Each worker creates its own client, processes its slice of the hierarchy, and reports results back through a managed queue. See the fw-curation documentation for details on the workers configuration option and the HierarchyCurator interface.

Rate limiting and backoff

Sending a large number of parallel requests can trigger rate-limit errors (429 Too Many Requests) or intermittent timeouts. Two complementary strategies help:

Limit concurrency. Keep max parallel requests between 4 and 16 for most workflows. Higher values rarely improve throughput and increase the chance of rate limiting.

Retry with exponential backoff. When a request fails with a transient error, wait and retry rather than failing immediately:

import time


def upload_file_with_retry(acquisition_id, file, retries=4):
    for attempt in range(retries):
        try:
            fw.upload_file_to_acquisition(acquisition_id, file)
            return
        except Exception:
            if attempt == retries - 1:
                raise
            time.sleep(2**attempt)

Related Topics

• Advanced: Reloading and Caching — when to call reload(), avoiding unnecessary fetches, and caching container lookups
• Core Concepts: Finders — iter(), iter_find(), pagination, and filter syntax
• fw-client - A Flywheel-maintained Python package for making Flywheel API calls
• fw-curation — a Flywheel-maintained Python library that provides utilities for data curation on Flywheel, including traversing the hierarchy and implementing multiprocessing