Python asyncio TaskGroups: Structured Concurrency That Survives Cancellation

Async Python services rarely fail because a single coroutine is difficult to write. The harder failures appear when one request starts several pieces of work, one child fails, the client disconnects, a timeout fires, or the server begins shutting down. If those tasks are detached from the request that created them, they can outlive their owner, keep sockets and database connections busy, or fail without being observed.

asyncio.TaskGroup, added in Python 3.11, brings structured concurrency to the standard library. It gives concurrent work an explicit lifetime: child tasks belong to a lexical scope, the scope waits for them, and a non-cancellation failure triggers sibling cancellation before the error is reported. The benefit is not simply cleaner syntax. It is a more predictable failure and cleanup model for production services.

This guide explains the lifecycle guarantees, cancellation semantics, exception groups, timeout composition, and practical patterns for request-scoped fan-out. Examples target Python 3.11+ and avoid relying on newer, version-specific conveniences.

1. The problem with detached tasks

asyncio.create_task() is useful, but it creates an independently scheduled task. The caller must retain a reference when necessary, decide who awaits it, collect exceptions, and define what happens when the caller is cancelled. A common mistake is to start work and then return without retaining or supervising it:

async def load_dashboard(user_id: str):
    asyncio.create_task(write_audit_event(user_id))
    return await build_dashboard(user_id)

This may be intentional for a truly independent background job, but it is not a safe default. The audit task can fail after the response has completed; request cancellation does not automatically express ownership of that task; and application shutdown now needs a separate mechanism to drain it. Adding a callback that logs exceptions solves only one part of the lifecycle problem.

Structured concurrency applies a simple rule: concurrent child work should not silently escape the scope responsible for it. When the scope exits, its children have either completed or been cancelled and awaited. This makes resource ownership and failure propagation visible in the code.

2. TaskGroup gives child tasks one lifetime

Use an asynchronous context manager to define the task group's lifetime. The context manager waits for every child before it exits, and task results remain available from the returned Task objects:

import asyncio

async def fetch_profile(user_id: str) -> dict:
    await asyncio.sleep(0.05)
    return {"id": user_id, "name": "Ada"}

async def fetch_permissions(user_id: str) -> list[str]:
    await asyncio.sleep(0.03)
    return ["read", "write"]

async def build_dashboard(user_id: str) -> dict:
    async with asyncio.TaskGroup() as group:
        profile_task = group.create_task(fetch_profile(user_id))
        permissions_task = group.create_task(fetch_permissions(user_id))

    return {
        "profile": profile_task.result(),
        "permissions": permissions_task.result(),
    }

The important guarantee is at the boundary of the async with block: after it exits normally, both tasks are finished. You do not need separate await statements just to ensure completion. Calling result() after the block is safe because the group has already waited for its children.

This pattern is particularly useful for independent reads that contribute to one response: profile plus permissions, product plus inventory, or an account summary plus recent activity. It does not make the operations transactional. If one read succeeds and another fails, the successful read is not rolled back; the group controls task lifetime, not business-level atomicity.

3. A child failure cancels its siblings

The key difference from asyncio.gather() is failure handling. With the default gather() behavior, the first exception is propagated to the awaiter, but other submitted awaitables are not automatically cancelled simply because one failed. A task group instead treats an ordinary child failure as a failure of the concurrent operation: it cancels the remaining children, waits for their cleanup, and then raises the collected non-cancellation failures.

import asyncio

async def read_catalog():
    await asyncio.sleep(0.05)
    raise RuntimeError("catalog database unavailable")

async def read_recommendations():
    try:
        await asyncio.sleep(10)
        return ["item-1", "item-2"]
    finally:
        print("recommendation task cleaned up")

async def handler():
    async with asyncio.TaskGroup() as group:
        group.create_task(read_catalog())
        group.create_task(read_recommendations())

asyncio.run(handler())

When read_catalog() raises, the task group requests cancellation of read_recommendations(). The finally block runs as cancellation is delivered, and the group waits for that cleanup before propagating the failure. If multiple children fail before shutdown completes, their errors can be included together rather than hiding every failure behind whichever exception happened to surface first.

That fail-fast policy is appropriate when the results form one logical response and partial output is not useful. It is not always appropriate for independent best-effort work. If one optional enrichment failing should not cancel a required core result, catch and convert that expected failure inside the relevant child, or model the work as separate supervision scopes with explicit policy.

4. ExceptionGroup and except*

Task groups report failures using ExceptionGroup or BaseExceptionGroup, as appropriate. This follows Python's exception-group model introduced in Python 3.11. A group can contain more than one failure, so ordinary single-exception handling is not always sufficient when you want to handle particular exception types while allowing unrelated failures to continue propagating.

try:
    async with asyncio.TaskGroup() as group:
        group.create_task(update_search_index())
        group.create_task(refresh_cache())
        group.create_task(publish_metric())
except* TimeoutError as errors:
    for error in errors.exceptions:
        log_warning("A dependency timed out", error=repr(error))
except* OSError as errors:
    for error in errors.exceptions:
        log_error("An operating-system operation failed", error=repr(error))

except* handles matching members of an exception group. Unmatched exceptions are re-raised after the matching handlers finish. Avoid flattening every group into a generic log message: preserve the individual exception types and tracebacks so the failure remains diagnosable. Also avoid catching broad groups merely to suppress them; that can turn a failed request into an apparently successful response.

Do not assume that every task failure will be grouped. KeyboardInterrupt and SystemExit receive special treatment and are re-raised after the group has shut down its children. Cancellation is also not an ordinary application error; it has its own control-flow meaning.

5. Cancellation is a request, not a kill switch

Calling Task.cancel() arranges for asyncio.CancelledError to be raised inside the coroutine at its next opportunity to run. It does not forcibly stop Python code in the middle of a CPU-bound operation. A coroutine that does not reach an await point cannot respond promptly to cancellation. Long CPU-bound work should be moved out of the event-loop thread, for example to a process pool, or broken into cooperative chunks.

Use try/finally to release resources and normally let cancellation propagate:

async def consume_messages(client):
    connection = await client.connect()
    try:
        while True:
            message = await connection.receive()
            await process(message)
    finally:
        await connection.close()

If you explicitly catch asyncio.CancelledError, perform the cleanup you need and re-raise it:

async def perform_operation():
    resource = await acquire_resource()
    try:
        await use_resource(resource)
    except asyncio.CancelledError:
        await record_interruption()
        raise
    finally:
        await resource.close()

Swallowing CancelledError is especially dangerous inside structured-concurrency code. TaskGroup and asyncio.timeout() use cancellation internally to coordinate their behavior. If application code suppresses cancellation and returns normally, the surrounding scope may no longer be able to enforce its intended shutdown semantics. The standard guidance is to propagate cancellation after cleanup. Calling uncancel() is a specialized operation, not a routine way to make cancellation disappear.

6. Compose TaskGroup with timeouts

A request deadline should bound the whole unit of work, not just one child. asyncio.timeout() provides a scope-level deadline and composes naturally with a task group:

import asyncio

async def load_summary(account_id: str):
    async with asyncio.timeout(1.5):
        async with asyncio.TaskGroup() as group:
            balance = group.create_task(fetch_balance(account_id))
            invoices = group.create_task(fetch_invoices(account_id))
            alerts = group.create_task(fetch_alerts(account_id))

    return {
        "balance": balance.result(),
        "invoices": invoices.result(),
        "alerts": alerts.result(),
    }

If the deadline expires, the timeout scope cancels the current task. The cancellation propagates through the task-group scope, which cancels and waits for its children. The timeout context then translates its own cancellation into TimeoutError as it exits. Catch that timeout at a boundary where you can map it to an HTTP response or retry policy; do not catch cancellation deep inside every helper.

Be precise about timeout ownership. A per-dependency timeout can be useful when one optional downstream service is allowed less time than the overall request. An overall request deadline is still needed so nested calls cannot collectively consume unbounded time. Set connection, read, and total deadlines deliberately in the HTTP/database clients too; an outer coroutine timeout is not a substitute for correct client-level timeout configuration or server-side query cancellation.

7. A production fan-out pattern with partial failures

Sometimes the business requirement is not fail-fast. For example, a product page may require product data but can render without recommendations. In that case, make the optional failure an explicit result rather than allowing it to escape and cancel required work:

from dataclasses import dataclass
import asyncio

@dataclass
class OptionalResult:
    value: object | None = None
    error: str | None = None

async def safely_load_recommendations(product_id: str) -> OptionalResult:
    try:
        value = await fetch_recommendations(product_id)
        return OptionalResult(value=value)
    except TimeoutError as exc:
        return OptionalResult(error=f"recommendations timed out: {exc}")

async def product_page(product_id: str):
    async with asyncio.timeout(2.0):
        async with asyncio.TaskGroup() as group:
            product_task = group.create_task(fetch_product(product_id))
            stock_task = group.create_task(fetch_stock(product_id))
            recommendations_task = group.create_task(
                safely_load_recommendations(product_id)
            )

    return {
        "product": product_task.result(),
        "stock": stock_task.result(),
        "recommendations": recommendations_task.result(),
    }

This design makes the policy visible: product and stock failures fail the operation; a recommendation timeout becomes a typed, inspectable partial result. Notice that the helper catches only the expected dependency timeout. It does not catch BaseException or suppress cancellation. If the request itself is cancelled, the cancellation still propagates and the group still shuts down its children.

8. Common mistakes and operational trade-offs

Creating tasks outside the owning scope

Calling asyncio.create_task() inside a task-group child does not automatically add the new task to the parent group. That detached task can outlive the group. Pass the group to code that needs to create sibling tasks, or have the child create its own nested TaskGroup so the nested work remains owned.

Using TaskGroup for fire-and-forget jobs

A request-scoped group is designed to wait for its children. If work must continue after the response, it needs an application-level job system or a supervised background-worker lifecycle. Use a durable queue for work that must survive process restarts; an in-memory asyncio task is not a durable job.

Assuming cancellation guarantees immediate cleanup

Cancellation is cooperative. A child can take time to finish cleanup, and a child that blocks the event loop or suppresses cancellation can delay group exit. Keep cleanup bounded, avoid blocking calls in async code, and instrument slow shutdown paths.

Ignoring concurrency limits

TaskGroup does not impose a concurrency limit. Creating a task for every row in a million-row result can exhaust memory, sockets, or downstream capacity. Use a bounded worker pool, asyncio.Semaphore, or a queue with a fixed number of consumers. Structured lifetime and bounded concurrency solve different problems; production systems usually need both.

Confusing task cancellation with transaction rollback

Cancelling a coroutine does not guarantee that a remote service did not complete its operation. A database query may already have committed, an HTTP request may have reached the server, or a message may have been accepted before the client observed cancellation. Use idempotency keys, transaction boundaries, and reconciliation where retries could duplicate side effects.

9. A practical review checklist

• Does every child task have a clear owner and a bounded lifetime?
• Does the scope wait for all children before returning?
• Are expected optional failures converted into explicit result types?
• Are unexpected failures allowed to propagate with their tracebacks?
• Does cleanup use finally, and does cancellation normally propagate?
• Are request deadlines and dependency-level timeouts configured?
• Is concurrency bounded independently of task lifetime?
• Are side effects safe under retries and ambiguous cancellation outcomes?
• Are detached background jobs moved to a supervised worker or durable queue?

10. The design principle

asyncio.TaskGroup is most valuable when several operations jointly form one logical unit of work. It makes the ownership boundary explicit, waits for child completion, coordinates sibling cancellation, and reports failures without leaving cleanup to chance. It does not replace deadlines, backpressure, durable job processing, transaction design, or dependency-specific error policy.

For new concurrent code on Python 3.11 and later, prefer a task group when child operations belong to the same request or operation. Use asyncio.gather() when its result-collection semantics are specifically what you need, and use detached tasks only when you have an explicit supervisor that owns their lifecycle. The goal is not to use one API everywhere; it is to make task ownership and failure behavior deliberate.

Further reading

• Python 3.14 documentation: Task Groups
• Python 3.14 documentation: Task Cancellation
• Python 3.14 documentation: Timeouts
• PEP 654: Exception Groups and except*
• Python documentation: Developing with asyncio