Advanced Search (fw.search())

fw.search() runs FlyQL, a SQL-like query language, against a separate Elasticsearch index rather than MongoDB. That single fact explains both its power and its surprises: it can query across the entire hierarchy and the whole instance with full-text matching, but the index is maintained asynchronously, so newly created data may not be searchable yet and deleted data may linger. It is the right tool for discovery across many projects; finders remain the right tool for precise, live, hierarchy-scoped work.

Not every field is searchable. Only fields that the crawler maps into the index can be queried, and some metadata is deliberately excluded from indexing. If a field you expect to filter on returns nothing, confirm it is actually indexed — see Discovering searchable fields.

The examples below assume an initialized client named fw (see Getting Started).

Finders or advanced search?

Beyond finders, fw.search() is a second, more powerful way to locate data. The two look similar but read from different stores and behave differently. Choosing the wrong one is the most common source of confusion, so settle it before writing a query.

Use a finder when you need:

• Precise, literal matches on known fields (label="baseline").
• Data that is current to the millisecond, including containers you just created.
• Work scoped to one container and its descendants.

Use advanced search when you need:

• To search across many projects or the whole instance in one query.
• Wildcard, regular-expression, or case-insensitive substring matching.
• To filter on fields at one level of the hierarchy while returning containers at another (filter on session.label, return subject).

The correct call pattern

A search is a SearchQuery plus a call to fw.search(). This pattern is the foundation for everything else on the page.

project_id = "000123456789abcdefABCDEF"

query = flywheel.SearchQuery(
    structured_query="session.label CONTAINS abdomen",   # your FlyQL
    return_type="session",                                # what to return
    filters=[{"term": {"project._id": project_id}}],      # scope to a project
    all_data=False,                                        # membership scope
)

results = fw.search(query, size=1000)                      # flat list by default
for r in results:
    print(r.session.label)

Four rules are non-obvious and each one causes silent, hard-to-debug failures. State them to yourself before writing a query.

1. Put FlyQL in structured_query, not search_string

SearchQuery accepts both structured_query and search_string. Only structured_query evaluates FlyQL operators correctly. search_string silently mis-evaluates them — it does not raise an error, it just returns the wrong results.

Use search_string only for a plain, free-text match — it runs an Elasticsearch match against the document's combined all_fields, so it hits any indexed field without you naming one. For anything with an operator (=, CONTAINS, EXISTS, <, …), use structured_query.

query = flywheel.SearchQuery(
    search_string="abdomen",                              # matches any field
    return_type="session",
    filters=[{"term": {"project._id": project_id}}],
)

2. Scope to a project with an Elasticsearch term filter

To restrict a search to one project, pass an Elasticsearch term filter — do not write a FlyQL project._id = ... clause, which is unreliable and can return nothing.

query = flywheel.SearchQuery(
    structured_query="session.label EXISTS",
    return_type="session",
    filters=[{"term": {"project._id": project_id}}],
)

3. all_data is a membership gate, not a feature toggle

all_data controls which projects the search can see:

• all_data=False (default): only projects you are a member of.
• all_data=True: the entire instance, regardless of membership. Site admins only.

4. return_type selects the container level returned

return_type is one of file, acquisition, session, subject, project, or analysis. The query may reference fields at any level of the hierarchy while returning a different level — for example, filter on session.label but return subject.

Result shape

Results are structured documents, not live containers. Each result carries the matched container and its ancestors, with the populated levels depending on return_type.

{
    "return_type": "session",
    "project": {"id": "...", "label": "C4KC-KiTS"},
    "group":   {"id": "...", "label": "..."},
    "subject": {"id": "...", "code": "KiTS-00191"},
    "session": {"id": "...", "label": "...", "timestamp": "...", "created": "..."},
    "acquisition": None, "file": None, "analysis": None,
}

Access fields by attribute (r.session.label, r.subject.code, r.project.id). To get a full, mutable container, re-fetch it by id — and handle 404, because the index may still list a container that has been deleted.

for r in results:
    try:
        session = fw.get(r.session.id)
    except flywheel.rest.ApiException as exc:
        if exc.status == 404:
            continue  # stale index entry — already deleted in MongoDB
        raise
    session.update(label=session.label + "-reviewed")

Complex queries

Operators

FlyQL supports the following operators.

What each operator does at runtime

FlyQL compiles to an Elasticsearch query. Two consequences are worth internalizing: matching against the raw value (=, !=, IN, LIKE, =~) is case-sensitive and anchored to the whole value, while CONTAINS runs against the analyzed field and is case-insensitive.

Boolean operators and nesting

Combine conditions with AND, OR, and a prefix NOT (or !), and group them with parentheses.

query = flywheel.SearchQuery(
    structured_query=(
        "(session.label CONTAINS abdomen OR session.label CONTAINS chest) "
        "AND NOT session.label CONTAINS localizer"
    ),
    return_type="session",
    filters=[{"term": {"project._id": project_id}}],
)

Two syntax rules govern NOT:

• NOT is prefix-only. Write NOT session.label EXISTS or NOT file.info.key CONTAINS Fail. The forms session.label NOT EXISTS and file.info.key NOT CONTAINS Fail are invalid syntax.
• AND binds tighter than OR. Without parentheses, a OR b AND c is evaluated as a OR (b AND c). Parenthesize mixed AND/OR anyway — it makes intent obvious to the next reader.

Other syntax rules

• Keyword operators are case-rigid: use ALL-CAPS or all-lowercase (CONTAINS or contains, never Contains). The same applies to AND / OR / NOT / EXISTS / LIKE / IN.
• String values use double quotes only: session.label = "two words". Single quotes break parsing.
• Field paths are container.field: session.label, subject.label, file.info.cohort. Unquoted paths may contain dashes, leading digits, and deep a.b.c nesting.
• Quote a whole path that contains spaces: "session.info.odd key" EXISTS, not session.info."odd key".

Regular expressions

=~ matches a field against a regular expression, and !~ is its negation. Both compile to an Elasticsearch regexp query against the raw value.

query = flywheel.SearchQuery(
    structured_query=r'subject.code =~ ".*KiTS-0019.*"',
    return_type="subject",
    filters=[{"term": {"project._id": project_id}}],
)

EXISTS matches non-null fields

EXISTS returns documents where the field is present and not null or empty, and compiles to an Elasticsearch exists query. Combine it with prefix NOT to find documents where a field is missing.

query = flywheel.SearchQuery(
    structured_query="NOT subject.species EXISTS",
    return_type="subject",
    filters=[{"term": {"project._id": project_id}}],
)

Validate a query before running it

fw.parse_search_query() checks FlyQL syntax without executing the search, returning the parse status and any errors with their position. Use it to validate user-supplied queries or to debug a query that misbehaves.

parsed = fw.parse_search_query(
    flywheel.StructuredQuery(structured_query="session.label = baseline")
)
if not parsed.valid:
    for error in parsed.errors:
        print(f"line {error.line}, offset {error.offset}: {error.message}")

Worked examples

# 1. Everything in a project (scope only, no query)
flywheel.SearchQuery(
    filters=[{"term": {"project._id": project_id}}],
    return_type="session",
)

# 2. Substring match on a label
flywheel.SearchQuery(
    structured_query="session.label CONTAINS abdomen",
    filters=[{"term": {"project._id": project_id}}],
    return_type="session",
)

# 3. Date range — sessions before 2002
flywheel.SearchQuery(
    structured_query="session.timestamp < 2002-01-01",
    filters=[{"term": {"project._id": project_id}}],
    return_type="session",
)

# 4. Cross-hierarchy — filter on sessions, return the subjects
flywheel.SearchQuery(
    structured_query="session.label CONTAINS chest",
    filters=[{"term": {"project._id": project_id}}],
    return_type="subject",
)

Saved searches

A saved search stores a query server-side under a parent container so it can be reused and shared, rather than rebuilt in code each time. This is Flywheel's mechanism for a reusable, named query.

project_id = "000123456789abcdefABCDEF"

# Save a search under a project
search_id = fw.save_search({
    "label": "Abdomen sessions",
    "search": {"return_type": "session", "structured_query": "session.label CONTAINS abdomen"},
    "parent": {"type": "project", "id": project_id},
})

# Retrieve one, or list all
saved = fw.get_saved_search(search_id)
all_saved = fw.get_all_saved_searches()

# Replace fields on an existing saved search
fw.replace_search(search_id, {"label": "Abdomen sessions (reviewed)"})

# Delete it
fw.delete_save_search(search_id)

Index freshness and the async connector

Because the search index is populated asynchronously, check its sync state when result freshness matters. fw.get_search_status() reports whether the index has caught up with MongoDB.

status = fw.get_search_status()
if status.status != flywheel.StatusType.UP_TO_DATE:
    print("Search index is still catching up; results may be stale.")

This is the concrete expression of the freshness caveat from Finders or advanced search?: if you just created data and need to act on it immediately, use a finder, not search.

Discovering searchable fields

fw.get_mapped_fields() returns the Elasticsearch field mapping for a level of the hierarchy — the authoritative list of fields you can query.

mapping = fw.get_mapped_fields(path="session")

Working with large search result sets

fw.search() does not paginate. It issues a single request and returns the whole result set at once, capped by size:

results = fw.search(query, size=500)

There is no offset, page, or cursor parameter — you cannot fetch "the next page." The size limit is a hard server-side ceiling of 10,000; a request that would return more raises a validation error asking you to add filters. Passing size="all" returns every match up to that same ceiling. The default size is 100.

Two flags shape the response:

• simple=True (the default) unwraps results into a flat list of structured documents.
• simple=False returns the full response object, including the results key and search aggregations.

For result sets larger than 10,000, or too large to hold in memory, operate with finders instead — they page lazily — see Core Concepts: Finders and Performance Optimization. A common pattern is to use search for instance-wide discovery, then a finder or fw.get() to act on each hit with live, authoritative data.

Verify these on your instance

The operator semantics above match the server's FlyQL implementation. A few behaviors still depend on your data or how fields are indexed — test these against your own instance rather than assuming:

• The behavior of custom info.* fields under each operator (depends on how each field is mapped and analyzed).
• filters keys other than project._id.