diff --git a/docs/available-components/brokers.md b/docs/available-components/brokers.md index 5ba91dbb..22f9cb4d 100644 --- a/docs/available-components/brokers.md +++ b/docs/available-components/brokers.md @@ -80,6 +80,158 @@ await my_task.kicker().with_broker(broker).kiq() ::: +## MultiBroker + +This broker combines multiple real brokers into one, so a single worker +can consume messages from all of them at the same time. +Every inner broker keeps its own connection and queue. The `MultiBroker` +only merges the listening and routes kicks. + +This is useful when queues live in different clusters or even different +technologies (two RabbitMQ clusters, RabbitMQ and Kafka, etc.), but you +want a single worker deployment processing all of them. + +### Usage + +Tasks are registered on the `MultiBroker` itself and routed to inner +brokers with a `routing` mapping and the `queue_name` label: + +```python +# broker.py +from taskiq import MultiBroker +from taskiq_aio_pika import AioPikaBroker, Queue + +fast_broker = AioPikaBroker( + "amqp://guest:guest@localhost:5672/", + task_queues=[Queue(name="fast_tasks")], +) +slow_broker = AioPikaBroker( + "amqp://guest:guest@localhost:5672/", + task_queues=[Queue(name="slow_tasks")], +) +broker = MultiBroker( + routing={"fast": fast_broker, "slow": slow_broker}, + default_routing_label="fast", +) +``` + +```python +# tasks.py +from my_project.broker import broker + +@broker.task(labels={"queue_name": "fast"}) +async def fast_task() -> None: + ... + +@broker.task(labels={"queue_name": "slow"}) +async def slow_task() -> None: + ... +``` + +With this setup a single worker command consumes messages from both queues: + +```bash +taskiq worker my_project.broker:broker my_project.tasks +``` + +Messages without the `queue_name` label are kicked using the broker mapped +to `default_routing_label`. If no default label is configured, unlabeled +messages are rejected with an error. + +Tasks cannot be registered on inner brokers directly: the `MultiBroker` +disables the `task` decorator of every broker it manages, so such mistakes +fail loudly instead of silently breaking task lookup in the worker. + +### How it works + +- Every inner broker is a full broker with its own connection, queue + declarations and settings (for example, its own `qos` when using + taskiq-aio-pika). The composite opens one connection per inner broker + in every worker process. +- `listen()` of all inner brokers run concurrently and are merged into + a single stream for the worker. There are no ordering or fairness + guarantees between queues: messages are processed as they arrive. +- Kicks are routed by the `queue_name` label of the message: the composite + looks the label value up in the `routing` mapping and kicks the message + with the mapped broker. +- Startup and shutdown are delegated to all inner brokers. A failing + shutdown of one broker is logged and doesn't prevent the others + from stopping. The `is_worker_process` and `is_scheduler_process` + flags propagate, so inner brokers fire their WORKER_* events. +- All tasks live in the registry of the `MultiBroker`, since only it + accepts task registration. The worker finds every task there. +- On shutdown the merged listening and all inner generators are closed + deterministically: every inner broker runs its cleanup logic, and + running tasks are awaited (bounded by `--wait-tasks-timeout`). + +### Failure handling + +The MultiBroker fails soft: an error of one broker doesn't stop the others. + +- **Connection flaps are handled by brokers themselves.** For example, + taskiq-aio-pika uses aio-pika's RobustConnection and reconnects + automatically. Such errors never reach the MultiBroker and change + nothing in the worker. +- **If an inner broker's listening fails anyway** (an unrecoverable + error, or a dead cluster for brokers without built-in reconnection), + the error is logged with a traceback, the failed broker is excluded + from listening, and the worker keeps consuming the remaining brokers. +- **Exclusion is permanent for the lifetime of the worker process.** + The MultiBroker doesn't re-subscribe failed brokers in place. Restart + the worker (or let the supervisor do it) to re-enable all brokers; + messages accumulated in the recovered queues are picked up after that. +- **If every inner broker is down**, the listening ends and the worker + process stops. The supervisor restarts it: with the default + `--max-fails -1` it restarts indefinitely until the brokers come back. +- **In-flight messages keep processing.** Messages already handed over + to the worker before the failure complete normally. Messages that + stayed in the failed broker follow that broker's delivery semantics + (see the guarantees table below). +- **Kicks are not affected by worker-side degradation.** Exclusion only + stops consuming. If the target cluster is down, a kick fails on the + caller side with the inner broker's own error. +- **Task-level errors don't degrade anything.** As with any taskiq + broker, an exception inside a task is logged and saved to the result + backend, the message is acknowledged, and the worker keeps running. + +### Guarantees + +The composite only merges brokers. All delivery guarantees come from +the inner brokers: + +| Aspect | MultiBroker | Inner brokers | +| --- | --- | --- | +| One worker consuming all queues | yes | | +| Kick routing by `queue_name` label | yes | | +| Partial degradation on broker failure | yes | | +| Graceful shutdown of all brokers | yes | | +| Reconnection of a failed broker | no | broker's own (aio-pika reconnects; Redis list queues don't) | +| Delivery guarantee (at-least/at-most once) | no | broker's own: RabbitMQ requeues unacked messages; Redis list queues lose popped messages on crash | +| Message redelivery, dead-lettering | no | broker's own | +| Per-queue concurrency limits | no | configured per inner broker (e.g. `qos` of taskiq-aio-pika) | +| Cross-queue ordering or fairness | no (messages arrive interleaved) | per-queue ordering per broker | + +::: caution Be careful! +The worker side uses formatters and serializers of the `MultiBroker`, +while kicks go through inner brokers. Default settings are compatible, +but if you customize them, make sure they match on both sides. + +Also, middlewares of inner brokers work only on the kick side, +and middlewares of the `MultiBroker` — only on the execution side. + +Use `MultiBroker.with_result_backend()` to set a result backend: +it propagates the backend to all inner brokers, keeping client-side +polling and worker-side saving in sync. +::: + +If all of your queues live in a single RabbitMQ broker, prefer the native +multi-queue support of [taskiq-aio-pika](https://github.com/taskiq-python/taskiq-aio-pika) +(it listens to all `task_queues` over a single connection). +The `MultiBroker` is useful when queues are served by different brokers, +or by brokers without native multi-queue support. Note that it opens +one connection per inner broker in every worker process. + + ## Custom brokers These brokers are not part of the core Taskiq lib. But they are maintained by Taskiq developers. diff --git a/taskiq/__init__.py b/taskiq/__init__.py index 1a764bc4..0c2d1244 100644 --- a/taskiq/__init__.py +++ b/taskiq/__init__.py @@ -11,6 +11,7 @@ from taskiq.abc.schedule_source import ScheduleSource from taskiq.acks import AckableMessage, AcknowledgeType from taskiq.brokers.inmemory_broker import InMemoryBroker +from taskiq.brokers.multi_broker import MultiBroker from taskiq.brokers.shared_broker import async_shared_broker from taskiq.brokers.zmq_broker import ZeroMQBroker from taskiq.context import Context @@ -49,6 +50,7 @@ "BrokerMessage", "Context", "InMemoryBroker", + "MultiBroker", "NoResultError", "PrometheusMiddleware", "ResultGetError", diff --git a/taskiq/brokers/multi_broker.py b/taskiq/brokers/multi_broker.py new file mode 100644 index 00000000..766d6ddb --- /dev/null +++ b/taskiq/brokers/multi_broker.py @@ -0,0 +1,265 @@ +import contextlib +import sys +from collections.abc import AsyncGenerator +from logging import getLogger +from typing import TYPE_CHECKING, Any, NoReturn + +from taskiq.abc.broker import AckableMessage, AsyncBroker +from taskiq.message import BrokerMessage +from taskiq.utils import merge_async_iterables + +if sys.version_info >= (3, 11): + from typing import Self +else: + from typing_extensions import Self + +if TYPE_CHECKING: # pragma: no cover + from taskiq.abc.result_backend import AsyncResultBackend + +logger = getLogger(__name__) + + +def _task_registration_disabled(*args: Any, **kwargs: Any) -> NoReturn: + """Replace the `task` method of brokers managed by a MultiBroker.""" + raise ValueError( + "Cannot register tasks on brokers managed by a MultiBroker. " + "Register tasks on the MultiBroker itself.", + ) + + +class MultiBroker(AsyncBroker): + """ + Broker that combines multiple brokers into one. + + It wraps several real brokers and allows a single worker + to consume messages from all of them at the same time. + Every inner broker keeps its own connection and queue, + this broker only merges the listening and routes kicks. + + If an inner broker fails, it's excluded from listening + and the worker keeps consuming the remaining brokers. + The exclusion lasts until the worker process restarts. + When all inner brokers are down, the worker stops + and is restarted by the supervisor. + + Tasks are registered on the MultiBroker itself and routed + to inner brokers with the routing label: + + >>> fast_broker = AioPikaBroker(url, queue_name="fast") + >>> slow_broker = AioPikaBroker(url, queue_name="slow") + >>> broker = MultiBroker( + ... routing={"fast": fast_broker, "slow": slow_broker}, + ... default_routing_label="fast", + ... ) + + >>> @broker.task(labels={"queue_name": "slow"}) + ... async def my_task() -> None: ... + + A single `taskiq worker` command consumes messages + from all inner brokers this way. + + :param routing: mapping of routing label values to brokers, + at least one entry is required. Brokers are deduplicated, + every unique broker is listened to once. + :param default_routing_label: label used to kick messages + without the routing label, must be a key of `routing`. + If it's not set, unlabeled messages are rejected. + :param routing_label: message label used for routing, + defaults to `queue_name`. + :raises ValueError: if `routing` is empty or has values + of a wrong type, or if `default_routing_label` + has no mapped broker. + """ + + def __init__( + self, + routing: "dict[str, AsyncBroker] | None" = None, + default_routing_label: str | None = None, + routing_label: str = "queue_name", + ) -> None: + if not routing: + raise ValueError( + "MultiBroker requires a non-empty routing mapping " + "of queue names to brokers.", + ) + for queue_name, inner_broker in routing.items(): + if not isinstance(inner_broker, AsyncBroker): + raise ValueError( + "All routed brokers must be AsyncBroker instances, " + f"got {type(inner_broker).__name__} for queue '{queue_name}'.", + ) + if default_routing_label is not None and default_routing_label not in routing: + raise ValueError( + "There is no broker mapped for the default queue " + f"'{default_routing_label}'. Known queues: {sorted(routing)}.", + ) + # Inner brokers must be saved before calling super().__init__(), + # since it sets the is_worker_process flag, + # which propagates to all inner brokers. + # Several labels can point to the same broker, + # so brokers are deduplicated to listen to each of them once. + self.brokers: list[AsyncBroker] = list(dict.fromkeys(routing.values())) + self.routing: dict[str, AsyncBroker] = dict(routing) + self.default_routing_label = default_routing_label + self.routing_label = routing_label + for inner_broker in self.brokers: + # Tasks must live in the registry of this broker, + # so accidental registration on inner brokers is loud. + inner_broker.task = _task_registration_disabled # type: ignore[method-assign] + super().__init__() + + @property + def is_worker_process(self) -> bool: + """True only if broker runs in worker process.""" + return self._is_worker_process + + @is_worker_process.setter + def is_worker_process(self, value: bool) -> None: + self._is_worker_process = value + for inner_broker in self.brokers: + inner_broker.is_worker_process = value + + @property + def is_scheduler_process(self) -> bool: + """True only if broker runs in scheduler process.""" + return self._is_scheduler_process + + @is_scheduler_process.setter + def is_scheduler_process(self, value: bool) -> None: + self._is_scheduler_process = value + for inner_broker in self.brokers: + inner_broker.is_scheduler_process = value + + async def kick(self, message: BrokerMessage) -> None: + """ + Kick a message using one of the inner brokers. + + The broker is selected by the routing label of the message. + Messages without the label go to the broker mapped + to the default routing label. + + :param message: message to send. + :raises ValueError: if the message has no routing label + and no default is configured, or the label value + has no mapped broker. + """ + label_value = message.labels.get(self.routing_label) + if label_value is None: + if self.default_routing_label is None: + raise ValueError( + f"Message has no '{self.routing_label}' label and the " + "MultiBroker has no default_routing_label, cannot route it.", + ) + label_value = self.default_routing_label + target_broker = self.routing.get(label_value) + if target_broker is None: + raise ValueError( + f"There is no broker mapped for queue '{label_value}'. " + f"Known queues: {sorted(self.routing)}.", + ) + await target_broker.kick(message) + + async def listen(self) -> AsyncGenerator["bytes | AckableMessage", None]: + """ + Listen to all inner brokers simultaneously. + + This method merges listen generators of all inner brokers + into a single generator, so the worker receives messages + from every broker at the same time. + + If one of the inner brokers fails, it's excluded from + listening, but all remaining brokers keep working. Only when + every inner broker is down, this generator ends and the worker + restarts. + + :yield: incoming messages from all inner brokers. + :return: nothing. + """ + merged = merge_async_iterables( + *(self._listen_one(inner_broker) for inner_broker in self.brokers), + ) + async for message in merged: + yield message + + async def _listen_one( + self, + inner_broker: AsyncBroker, + ) -> AsyncGenerator["bytes | AckableMessage", None]: + """ + Listen to a single inner broker, degrading on its failure. + + If the inner broker's listen fails, the error is logged + and this generator ends, so the failed broker is excluded + from the merged listening, but doesn't stop the whole worker. + + :param inner_broker: broker to listen to. + :yield: incoming messages from the inner broker. + :return: nothing. + """ + iterator = inner_broker.listen() + try: + async for message in iterator: + yield message + except Exception as exc: + logger.exception( + "Inner broker %s failed: %r. " + "It's excluded from listening, " + "continuing with remaining brokers.", + inner_broker, + exc, + ) + finally: + # Catches both the failure case and the outer shutdown: + # a suspended generator must run its cleanup logic. + with contextlib.suppress(RuntimeError): + await iterator.aclose() + + async def startup(self) -> None: + """ + Startup the composite broker and all inner brokers. + + Event handlers and middlewares of this broker are called + first, then every inner broker is started. + """ + await super().startup() + for inner_broker in self.brokers: + await inner_broker.startup() + + async def shutdown(self) -> None: + """ + Shut down all inner brokers and the composite broker. + + Exceptions raised by inner brokers are logged + and don't prevent other brokers from shutting down. + """ + for inner_broker in self.brokers: + try: + await inner_broker.shutdown() + except Exception as exc: + logger.warning( + "Cannot shutdown broker %s: %s", + inner_broker, + exc, + exc_info=True, + ) + await super().shutdown() + + def with_result_backend( + self, + result_backend: "AsyncResultBackend[Any]", + ) -> Self: + """ + Set a result backend for the composite and all inner brokers. + + Inner brokers use their result backend to serve + `wait_result` calls on the client side, while the worker + saves results using the result backend of this broker. + Setting the same backend everywhere keeps them in sync. + + :param result_backend: new result backend. + :return: self + """ + self.result_backend = result_backend + for inner_broker in self.brokers: + inner_broker.result_backend = result_backend + return self diff --git a/taskiq/utils.py b/taskiq/utils.py index 725adc1f..5754d668 100644 --- a/taskiq/utils.py +++ b/taskiq/utils.py @@ -1,8 +1,19 @@ +import asyncio +import contextlib import inspect -from collections.abc import Awaitable, Coroutine +import sys +from collections.abc import AsyncGenerator, Awaitable, Coroutine from types import CoroutineType from typing import Any, TypeVar, Union +import anyio +from anyio.streams.memory import MemoryObjectSendStream + +if sys.version_info >= (3, 11): + from builtins import BaseExceptionGroup +else: + from exceptiongroup import BaseExceptionGroup + _T = TypeVar("_T") @@ -29,3 +40,75 @@ async def maybe_awaitable( if inspect.isawaitable(possible_coroutine): return await possible_coroutine return possible_coroutine # type: ignore + + +async def _start_producer( + stream: AsyncGenerator[_T, None], + send_stream: MemoryObjectSendStream[_T], +) -> None: + """ + Forward all values of one stream to the shared send stream. + + The source generator is closed even if the producer + was cancelled while waiting in `send()` and not inside + the generator itself. + """ + try: + async with send_stream: + async for val in stream: + await send_stream.send(val) + finally: + await stream.aclose() + + +async def _run_producers( + iterables: "tuple[AsyncGenerator[_T, None], ...]", + send_stream: MemoryObjectSendStream[_T], +) -> None: + """ + Pump every iterable into the send stream in a task group. + + This coroutine runs as a dedicated task, so its task group + is always entered and exited in the same task, no matter + which task iterates over or closes the merged generator. + """ + async with send_stream, anyio.create_task_group() as tg: + for iterable in iterables: + tg.start_soon(_start_producer, iterable, send_stream.clone()) + # The original send stream is closed, so only clones remain. + # When all of them are closed, the receive stream ends. + send_stream.close() + + +async def merge_async_iterables( + *iterables: AsyncGenerator[_T, None], +) -> AsyncGenerator[_T, None]: + """ + Merge multiple async generators into a single one. + + All values from every generator are yielded + as soon as they arrive, in no particular order. + + If any of the inner generators raises an exception, + all other generators are cancelled and the error is + re-raised for the consumer wrapped in an exception group. + + When the merged generator is closed or cancelled, + all inner generators are closed as well. + + :param iterables: async generators to merge. + :return: single generator with values from all generators. + """ + send_stream, receive_stream = anyio.create_memory_object_stream[_T]() + producers = asyncio.create_task(_run_producers(iterables, send_stream)) + try: + async for message in receive_stream: + yield message + # The receive stream ended, so every producer finished. + # If any of them failed, its error surfaces now. + await producers + finally: + producers.cancel() + with contextlib.suppress(asyncio.CancelledError, BaseExceptionGroup): + await producers + receive_stream.close() diff --git a/tests/brokers/test_multi_broker.py b/tests/brokers/test_multi_broker.py new file mode 100644 index 00000000..cc6249c3 --- /dev/null +++ b/tests/brokers/test_multi_broker.py @@ -0,0 +1,369 @@ +import asyncio +from collections.abc import AsyncGenerator + +import pytest + +from taskiq import MultiBroker +from taskiq.acks import AckableMessage +from taskiq.brokers.inmemory_broker import InmemoryResultBackend +from taskiq.events import TaskiqEvents +from taskiq.message import BrokerMessage +from taskiq.receiver import Receiver +from tests.utils import AsyncQueueBroker + + +def build_message(labels: dict[str, str] | None = None) -> BrokerMessage: + """Build a minimal broker message for kick tests.""" + return BrokerMessage( + task_id="task-id", + task_name="tests:test_task", + message=b"payload", + labels=labels or {}, + ) + + +def test_multi_broker_requires_non_empty_routing() -> None: + with pytest.raises(ValueError, match="non-empty routing"): + MultiBroker() + + with pytest.raises(ValueError, match="non-empty routing"): + MultiBroker(routing={}) + + +def test_multi_broker_validates_routed_broker_types() -> None: + with pytest.raises(ValueError, match="AsyncBroker"): + MultiBroker(routing={"fast": 42}) # type: ignore[dict-item] + + +def test_multi_broker_validates_default_routing_label() -> None: + with pytest.raises(ValueError, match="no broker mapped"): + MultiBroker( + routing={"fast": AsyncQueueBroker()}, + default_routing_label="slow", + ) + + +def test_task_registration_on_inner_broker_raises() -> None: + first = AsyncQueueBroker() + MultiBroker(routing={"first": first}) + + with pytest.raises(ValueError, match="MultiBroker itself"): + + @first.task + async def inner_task() -> None: ... + + with pytest.raises(ValueError, match="MultiBroker itself"): + first.task() + + +async def test_kick_routes_by_queue_name_label() -> None: + first, second = AsyncQueueBroker(), AsyncQueueBroker() + broker = MultiBroker(routing={"first": first, "second": second}) + + await broker.kick(build_message(labels={"queue_name": "second"})) + + assert first.queue.qsize() == 0 + assert second.queue.qsize() == 1 + + +async def test_kick_without_label_uses_default_routing_label() -> None: + first, second = AsyncQueueBroker(), AsyncQueueBroker() + broker = MultiBroker( + routing={"first": first, "second": second}, + default_routing_label="second", + ) + + await broker.kick(build_message()) + + assert first.queue.qsize() == 0 + assert second.queue.qsize() == 1 + + +async def test_kick_without_label_and_default_raises() -> None: + broker = MultiBroker(routing={"first": AsyncQueueBroker()}) + + with pytest.raises(ValueError, match="default_routing_label"): + await broker.kick(build_message()) + + +async def test_kick_with_unknown_queue_name_raises() -> None: + first, second = AsyncQueueBroker(), AsyncQueueBroker() + broker = MultiBroker(routing={"first": first, "second": second}) + + with pytest.raises(ValueError, match="no broker mapped for queue 'fast'"): + await broker.kick(build_message(labels={"queue_name": "fast"})) + + +async def test_kick_uses_custom_routing_label() -> None: + first, second = AsyncQueueBroker(), AsyncQueueBroker() + broker = MultiBroker( + routing={"first": first, "emails": second}, + default_routing_label="first", + routing_label="topic", + ) + + await broker.kick(build_message(labels={"topic": "emails"})) + + assert first.queue.qsize() == 0 + assert second.queue.qsize() == 1 + + +async def test_same_broker_under_two_labels_is_listened_once() -> None: + class ListenCountingBroker(AsyncQueueBroker): + def __init__(self) -> None: + super().__init__() + self.listen_calls = 0 + + async def listen(self) -> AsyncGenerator[AckableMessage, None]: + self.listen_calls += 1 + async for message in super().listen(): + yield message + + shared, other = ListenCountingBroker(), AsyncQueueBroker() + broker = MultiBroker( + routing={"a": shared, "b": shared, "other": other}, + default_routing_label="a", + ) + await other.kick(build_message()) + + iterator = broker.listen() + await asyncio.wait_for(iterator.__anext__(), timeout=1) + await iterator.aclose() + + assert shared.listen_calls == 1 + + +async def test_listen_yields_messages_from_all_brokers() -> None: + first, second = AsyncQueueBroker(), AsyncQueueBroker() + broker = MultiBroker(routing={"first": first, "second": second}) + await first.kick(build_message()) + await second.kick(build_message()) + await second.kick(build_message()) + + received = 0 + iterator = broker.listen() + for _ in range(3): + await asyncio.wait_for(iterator.__anext__(), timeout=1) + received += 1 + await iterator.aclose() + + assert received == 3 + assert first.queue.qsize() == 0 + assert second.queue.qsize() == 0 + + +async def test_listen_with_single_broker() -> None: + first = AsyncQueueBroker() + broker = MultiBroker(routing={"first": first}) + await first.kick(build_message()) + + iterator = broker.listen() + await asyncio.wait_for(iterator.__anext__(), timeout=1) + await iterator.aclose() + + assert first.queue.qsize() == 0 + + +def test_process_flags_propagate_to_inner_brokers() -> None: + first, second = AsyncQueueBroker(), AsyncQueueBroker() + broker = MultiBroker(routing={"first": first, "second": second}) + + broker.is_worker_process = True + assert first.is_worker_process is True + assert second.is_worker_process is True + + broker.is_worker_process = False + assert first.is_worker_process is False + + broker.is_scheduler_process = True + assert first.is_scheduler_process is True + assert second.is_scheduler_process is True + + +async def test_startup_delegates_to_inner_brokers() -> None: + started: list[str] = [] + first, second = AsyncQueueBroker(), AsyncQueueBroker() + first.add_event_handler( + TaskiqEvents.WORKER_STARTUP, + lambda _: started.append("first"), + ) + second.add_event_handler( + TaskiqEvents.WORKER_STARTUP, + lambda _: started.append("second"), + ) + broker = MultiBroker(routing={"first": first, "second": second}) + broker.add_event_handler( + TaskiqEvents.WORKER_STARTUP, + lambda _: started.append("composite"), + ) + broker.is_worker_process = True + + await broker.startup() + + assert sorted(started) == ["composite", "first", "second"] + + +async def test_shutdown_delegates_to_inner_brokers() -> None: + shut_down: list[str] = [] + first, second = AsyncQueueBroker(), AsyncQueueBroker() + first.add_event_handler( + TaskiqEvents.WORKER_SHUTDOWN, + lambda _: shut_down.append("first"), + ) + second.add_event_handler( + TaskiqEvents.WORKER_SHUTDOWN, + lambda _: shut_down.append("second"), + ) + broker = MultiBroker(routing={"first": first, "second": second}) + broker.add_event_handler( + TaskiqEvents.WORKER_SHUTDOWN, + lambda _: shut_down.append("composite"), + ) + broker.is_worker_process = True + + await broker.shutdown() + + assert sorted(shut_down) == ["composite", "first", "second"] + + +async def test_shutdown_tolerates_failing_inner_brokers() -> None: + shut_down: list[str] = [] + + class FailingBroker(AsyncQueueBroker): + async def shutdown(self) -> None: + raise RuntimeError("shutdown failed") + + failing = FailingBroker() + ok_broker = AsyncQueueBroker() + ok_broker.add_event_handler( + TaskiqEvents.CLIENT_SHUTDOWN, + lambda _: shut_down.append("ok"), + ) + broker = MultiBroker(routing={"bad": failing, "ok": ok_broker}) + broker.add_event_handler( + TaskiqEvents.CLIENT_SHUTDOWN, + lambda _: shut_down.append("composite"), + ) + + await broker.shutdown() + + assert "ok" in shut_down + assert "composite" in shut_down + + +async def test_find_task_finds_tasks_registered_on_composite() -> None: + broker = MultiBroker(routing={"first": AsyncQueueBroker()}) + + @broker.task + async def composite_task() -> None: ... + + assert broker.find_task(composite_task.task_name) is composite_task + + +async def test_get_all_tasks_returns_tasks_registered_on_composite() -> None: + broker = MultiBroker(routing={"first": AsyncQueueBroker()}) + + @broker.task + async def composite_task() -> None: ... + + assert broker.get_all_tasks()[composite_task.task_name] is composite_task + + +async def test_with_result_backend_propagates_to_inner_brokers() -> None: + first, second = AsyncQueueBroker(), AsyncQueueBroker() + broker = MultiBroker(routing={"first": first, "second": second}) + backend = InmemoryResultBackend[None]() + + broker.with_result_backend(backend) + + assert broker.result_backend is backend + assert first.result_backend is backend + assert second.result_backend is backend + + +async def test_receiver_processes_tasks_from_all_brokers() -> None: + first, second = AsyncQueueBroker(), AsyncQueueBroker() + broker = MultiBroker( + routing={"first": first, "second": second}, + default_routing_label="first", + ) + broker.with_result_backend(InmemoryResultBackend()) + executed: list[str] = [] + + @broker.task(labels={"queue_name": "first"}) + async def task_one() -> str: + executed.append("one") + return "one" + + @broker.task(labels={"queue_name": "second"}) + async def task_two() -> str: + executed.append("two") + return "two" + + first_handle = await task_one.kiq() + second_handle = await task_two.kiq() + + receiver = Receiver(broker=broker) + finish_event = asyncio.Event() + listener = asyncio.create_task(receiver.listen(finish_event)) + + first_result = await first_handle.wait_result(timeout=5) + second_result = await second_handle.wait_result(timeout=5) + + assert first_result.return_value == "one" + assert second_result.return_value == "two" + assert sorted(executed) == ["one", "two"] + + finish_event.set() + await asyncio.wait_for(listener, timeout=5) + await broker.shutdown() + + +class DyingBroker(AsyncQueueBroker): + """Broker that yields N messages and then fails like a dead cluster.""" + + def __init__(self, messages_before_death: int) -> None: + super().__init__() + self.messages_before_death = messages_before_death + + async def listen(self) -> AsyncGenerator[AckableMessage, None]: + while self.messages_before_death > 0: + self.messages_before_death -= 1 + data = await self.queue.get() + yield AckableMessage(data=data, ack=self.queue.task_done) + raise RuntimeError("cluster is down") + + +async def test_failed_broker_does_not_stop_healthy_brokers( + caplog: pytest.LogCaptureFixture, +) -> None: + dying = DyingBroker(messages_before_death=2) + healthy = AsyncQueueBroker() + broker = MultiBroker(routing={"dying": dying, "healthy": healthy}) + + for _ in range(2): + await dying.kick(build_message()) + for _ in range(3): + await healthy.kick(build_message()) + + received = 0 + iterator = broker.listen() + for _ in range(5): + await asyncio.wait_for(iterator.__anext__(), timeout=1) + received += 1 + await iterator.aclose() + + assert received == 5 + assert "cluster is down" in caplog.text + + +async def test_listen_ends_when_all_inner_brokers_fail() -> None: + broker = MultiBroker( + routing={ + "first": DyingBroker(messages_before_death=0), + "second": DyingBroker(messages_before_death=0), + }, + ) + + with pytest.raises(StopAsyncIteration): + await asyncio.wait_for(broker.listen().__anext__(), timeout=1) diff --git a/tests/test_utils.py b/tests/test_utils.py index 485ab319..48860a80 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -1,4 +1,16 @@ -from taskiq.utils import maybe_awaitable +import asyncio +import sys +from collections.abc import AsyncGenerator +from typing import cast + +import pytest + +from taskiq.utils import maybe_awaitable, merge_async_iterables + +if sys.version_info >= (3, 11): + from builtins import BaseExceptionGroup +else: + from exceptiongroup import BaseExceptionGroup async def test_maybe_awaitable_coroutine() -> None: @@ -14,3 +26,63 @@ def meme() -> int: return 1 assert await maybe_awaitable(meme()) == 1 + + +async def test_merge_async_iterables_merges_all_iterables() -> None: + async def gen(*items: int) -> AsyncGenerator[int, None]: + for item in items: + await asyncio.sleep(0) + yield item + + merged = [ + item + async for item in merge_async_iterables( + gen(1, 2, 3), + gen(4, 5, 6), + ) + ] + + assert sorted(merged) == [1, 2, 3, 4, 5, 6] + + +async def test_merge_async_iterables_without_iterables() -> None: + # The TypeVar cannot be inferred from an empty argument list, + # so the return type is provided explicitly. + merged = cast("AsyncGenerator[int, None]", merge_async_iterables()) + assert [item async for item in merged] == [] + + +async def test_merge_async_iterables_propagates_inner_errors() -> None: + async def failing() -> AsyncGenerator[int, None]: + yield 1 + raise ValueError("inner generator failed") + + with pytest.raises(BaseExceptionGroup) as exc_info: + async for _ in merge_async_iterables(failing()): + pass + + assert isinstance(exc_info.value.exceptions[0], ValueError) + + +async def test_merge_async_iterables_closes_inner_iterables() -> None: + closed: list[str] = [] + + def tracking(name: str) -> AsyncGenerator[int, None]: + async def gen() -> AsyncGenerator[int, None]: + try: + while True: + yield 1 + await asyncio.sleep(0) + finally: + closed.append(name) + + return gen() + + merged = merge_async_iterables(tracking("first"), tracking("second")) + await merged.__anext__() + # At this point one producer is blocked in send() and the other + # waits for its next value. Closing must be clean and run the + # cleanup of every inner generator. + await merged.aclose() + + assert sorted(closed) == ["first", "second"]