Running Gears in Batch

A batch schedules a single gear across many containers in one proposal, then runs, monitors, and cancels the resulting jobs as a group. Use it when you want to run the same gear over dozens or thousands of acquisitions, sessions, or subjects without launching each job by hand.

This is the gear-execution counterpart to the data-oriented patterns in the rest of this section: instead of looping a metadata or file operation, you hand the server a list of targets and let it create one job per target.

The two paradigms

Flywheel has two distinct ways to build a batch. They use different endpoints, different input models, and behave differently. Choosing the right one is the first decision.

To run a gear across a set of containers and let Flywheel resolve which file feeds each input, use classic. To pin the same file (for example, a shared configuration profile) on every job, use premade-jobs.

Lifecycle at a glance

Both paradigms follow the same propose → run → monitor → cancel arc, with one important state rule:

1. Propose — build the batch. It is created in the pending state. No jobs run yet.
2. Run — proposal.run() (classic) or fw.start_batch(id) (premade) materializes and launches the jobs. The batch moves to running.
3. Monitor — poll the individual jobs for their state.
4. Cancel — stop the running jobs.

Classic batch

gear.propose_batch() takes the containers to run on and returns a proposal — a preview of what the server matched. No jobs exist until you call run().

gear = fw.lookup("gears/file-classifier")

# The acquisitions (or sessions, subjects) you want to run the gear on
project = fw.get_project("000123456789abcdefABCDEF")
targets = list(project.acquisitions.iter())

proposal = gear.propose_batch(targets, config={}, tags=["my-batch"])

Inspecting the proposal

The proposal sorts every target into one of three lists:

• matched — targets the server could fully resolve. Each entry's .inputs shows the file chosen for each gear input.
• ambiguous — targets where the matcher could not decide which file fills which input. These carry no resolved inputs and will not run.
• not_matched — targets that could not satisfy the gear's required inputs.

# A proposal that matches zero targets comes back with id=None and is not
# persisted server-side. Always guard before running.
if proposal.id is None:
    raise RuntimeError("No targets matched — nothing to run")

print(f"matched={len(proposal.matched)} ambiguous={len(proposal.ambiguous)} not_matched={len(proposal.not_matched)}")

# Each matched entry echoes the resolved per-input file
print(proposal.matched[0].inputs)
# {'file-input': {'id': '<acq_id>', 'name': 'scan-0.txt', 'type': 'acquisition'}}

Running the batch

proposal.run() launches the jobs and returns a list of JobOutput objects — full job documents, not id strings. Read .id and .state from each.

jobs = proposal.run()  # -> list[JobOutput], batch is now "running"
print(f"started {len(jobs)} jobs; first job {jobs[0].id} is {jobs[0].state}")

Optional inputs: optional_input_policy

When a gear has optional inputs, optional_input_policy controls how the classic matcher treats them. It accepts "ignored" (the default), "flexible", or "required". The policy governs only the gear's optional inputs; required inputs behave the same under every policy. The outcome depends entirely on how many candidate files the optional input has in a given container.

proposal = gear.propose_batch(
    targets,
    config={},
    optional_input_policy="required",
    tags=["my-batch"],
)

The behavior, verified against a gear whose optional profile input accepts a specific file type:

The single behavioral difference between ignored and flexible is the middle row: given exactly one candidate, ignored leaves the optional input empty while flexible fills it. required promotes the optional input to mandatory — it is then treated exactly like a required input (0 candidates means not_matched, 2+ means ambiguous).

Premade-jobs batch

When every job must use the same pinned input — the classic example is a shared configuration profile file applied to the whole batch — build the jobs explicitly. You construct one InputJob per target and supply its inputs, destination, and config yourself. Nothing is matched.

Use the real SDK models rather than raw dicts:

• FileReference(type, id, name) — type is the parent container type (project / subject / session / acquisition / analysis), id is that container's id, name is the file name.
• ContainerReference(type, id) — the job's destination container.

from flywheel.models import (
    ContainerReference,
    FileReference,
    InputJob,
    PremadeJobsBatchProposalInput,
)

gear = fw.lookup("gears/file-classifier")

project = fw.get_project("000123456789abcdefABCDEF")
acquisitions = list(project.acquisitions.iter())

# The constant file pinned on every job. type is the parent container type.
profile_ref = FileReference(type="project", id=project.id, name="shared_PROFILE.txt")

jobs = []
for acq in acquisitions:
    acq = acq.reload()
    jobs.append(
        InputJob(
            gear_id=gear.id,
            destination=ContainerReference(type="acquisition", id=acq.id),
            inputs={
                "file-input": FileReference(type="acquisition", id=acq.id, name=acq.files[0].name),
                "profile": profile_ref,  # same constant ref on every job
            },
            config=gear.get_default_config(),
            tags=["my-premade-batch"],
        )
    )

created = fw.create_batch_job_from_jobs(PremadeJobsBatchProposalInput(jobs=jobs))
# created.state == "pending"; created.jobs is None — jobs are not materialized yet.
# Inspect them before launch via the proposal:
print(created.proposal.preconstructed_jobs[0].inputs)

started = fw.start_batch(created.id)  # materializes + launches; batch -> "running"
print(f"started {len(started)} jobs")

Monitoring batch jobs

fw.get_batch(id) returns the batch record. Its .jobs attribute is a list of job-id strings, not inflated job objects — fetch each one with fw.get_job() to read its live state.

batch_id = proposal.id  # from the proposal you ran above

batch = fw.get_batch(batch_id)
print(f"batch state: {batch.state}")

# batch.jobs is a list of job-id STRINGS, not inflated job objects.
# Fetch each job to read its live state.
for job_id in batch.jobs:
    job = fw.get_job(job_id)
    print(f"  {job.id}: {job.state}")

Because every job carries the tags you set at propose time, a robust way to track a batch is a finder query on its tag:

# Track every job in the batch by the tag you assigned at propose time.
for job in fw.jobs.iter_find("tags=my-batch"):
    print(f"{job.id}: {job.state}")

To wait for a batch to finish, poll the jobs until none remain in a non-terminal state:

import time

terminal_states = {"complete", "failed", "cancelled"}
while True:
    jobs = [fw.get_job(job_id) for job_id in fw.get_batch(batch_id).jobs]
    remaining = [j for j in jobs if j.state not in terminal_states]
    if not remaining:
        break
    print(f"{len(remaining)} jobs still running…")
    time.sleep(10)

succeeded = sum(1 for j in jobs if j.state == "complete")
print(f"batch finished: {succeeded}/{len(jobs)} complete")

Cancelling a batch

fw.cancel_batch(id) cancels a running batch's jobs and returns a bare int — the number of jobs cancelled.

# cancel_batch returns a bare int — the number of jobs cancelled.
cancelled = fw.cancel_batch(batch_id)
print(f"cancelled {cancelled} jobs")

Analysis-gear batches

For an analysis gear, pass analysis_label to propose_batch(). At run() the server creates one analysis per matched target instead of a plain job, and each returned JobOutput has destination.type == "analysis".

session = fw.get_session("000123456789abcdefABCDEF")

analysis_gear = fw.lookup("gears/my-analysis-gear")
proposal = analysis_gear.propose_batch(
    [session.reload()],
    config={},
    analysis_label="batch-analysis-2026",
    tags=["my-analysis-batch"],
)
jobs = proposal.run()

Two behaviors are worth knowing before you run one:

• The analysis attaches to the target's session, not the target itself. Targeting acquisitions produces analyses on their parent session. Analyses are created at run(), not at propose time.
• proposal.analysis on the returned proposal is None — the proposal does not echo back the analysis input you sent.

Caveats reference

A condensed list of the non-obvious behaviors verified during research:

Related

• Core Concepts: Finders — select the containers to batch over, and track batch jobs by tag with iter_find
• Core Concepts: Containers — the targets a batch runs against
• Bulk Metadata Updates — apply metadata across the same container sets you batch over