Open source — August 2026

Four concurrency bugs on Haystack's async path

All four are the same mistake wearing different clothes: an async method written by copying the synchronous one, inheriting assumptions that only hold while nothing else is running. Three of them were invisible to a test suite that did test the async path.

haystack#12364 haystack#12358 haystack#12359 haystack-core-integrations#3790

Haystack is deepset's framework for production RAG and agent pipelines — 26k stars, and the thing a lot of retrieval systems are actually built on. Like most Python libraries that grew an async story after the fact, it has components that expose both run and run_async, and the second was usually written by working through the first and putting await in front of the calls that had one.

That translation is where the bugs live. The synchronous version of a component gets to assume three things for free: that only one call is in flight at a time, that blocking the caller is the caller's problem, and that state stored on selfbelongs to whoever is currently using it. Every one of those assumptions is false the moment the method is awaited concurrently — and none of them announce themselves, because the code still reads correctly.

Below are four I found and fixed, none of them from an issue. I found them by reading run_async against run and asking what the sync version was getting away with.

1. State on the component, shared by everything in flight

LinkContentFetcherhaystack#12364→merged

LinkContentFetcher downloads a list of URLs. Sites block scrapers by User-Agent, so it takes a list of user agents and rotates to the next one after a failed attempt. The rotation cursor lived on the component:

self.current_user_agent_idx: int = 0

And run fetches the URLs concurrently— that is the whole point of the component. So the cursor is not one fetch's rotation state, it is everybody's. A single 403 on one URL advanced the user agent for every other request in flight, including the ones that were doing fine. Worse, each fetch reset it on the way out:

finally:
    self.current_user_agent_idx = 0

The first fetch to finish rewound the cursor underneath everything still running. The practical result is that most retries went out on the sameuser agent that had just been rejected — the one behaviour the feature exists to provide, quietly not happening. Nothing raises. You just get a lower success rate than you think you have, and no way to tell it from the sites being hostile.

The fix is to stop storing per-call state on the object. The cursor becomes a local, and the retry callback closes over it:

def _get_response(self, url: str) -> httpx.Response:
    user_agent_idx = 0

    def rotate_user_agent(retry_state: RetryCallState) -> None:
        nonlocal user_agent_idx
        user_agent_idx = (user_agent_idx + 1) % len(self.user_agents)

    @retry(
        reraise=True,
        stop=stop_after_attempt(self.retry_attempts),
        wait=wait_exponential(multiplier=1, min=2, max=10),
        retry=retry_if_exception_type((httpx.HTTPStatusError, httpx.RequestError)),
        after=rotate_user_agent,
    )
    def get_response(url: str) -> httpx.Response:
        response = self._client.get(url, headers=self._get_headers(self.user_agents[user_agent_idx]))
        response.raise_for_status()
        return response

    return get_response(url)

Each fetch now rotates through the list on its own, and the finally block that reset shared state is gone because there is no shared state left to reset.

2 & 3. Blocking work, on the event loop, in the method that promised not to

EmbeddingBasedDocumentSplitterhaystack#12358→merged

EmbeddingBasedDocumentSplitter splits a document by embedding its sentences and cutting where meaning shifts. Chunks that come out over max_length get recursively re-split. run_async awaited the first pass properly and then did this:

final_splits = self._split_large_splits(splits=merged_splits)

No await, because there was nothing to await: _split_large_splits is the sync recursion, and it calls the blockingembedder. So the component was async for the cheap part and synchronous for the expensive one. Every long document in the batch froze the event loop while it embedded — and long documents are precisely the ones that reach that branch.

This is the failure mode that makes async code slower than the sync code it replaced. The pipeline awaits, the loop yields, and then one component sits on the thread doing network-bound work through a blocking client while every other coroutine waits its turn. The fix was an _split_large_splits_async mirroring the recursion through _split_text_async, with a keep in sync note on both, since the duplication is the honest cost of supporting two paths.

LLMDocumentContentExtractorhaystack#12359→merged

The same shape, one layer up. LLMDocumentContentExtractor.run_async called DocumentToImageContent.rundirectly — a component that reads every file from disk, renders the requested page of every PDF, and base64-encodes the result. It has no run_async. So the entire batch was converted on the event loop before the first LLM call was even scheduled: the part that could have overlapped with rendering was serialised behind it.

The module already imported the right tool and was using it for the chat generator two lines away:

# before
image_contents = DocumentToImageContent.run(...)

# after — prefers run_async when a component has one, else runs run in a thread
image_contents = await _execute_component_async(DocumentToImageContent, ...)

A one-line fix, which is the interesting part. Nobody wrote this deliberately; the async method was assembled from the sync method, and one call in the middle never got looked at again.

4. The lock that belongs to a loop, not to an object

OAuthRefreshTokenSourcehaystack-core-integrations#3790→merged

This one is my favourite, because the code looks not just correct but thoughtful— it has a comment explaining why it is written the way it is, and the comment is right about everything except the conclusion.

# Create the asyncio.Lock lazily (not in __init__): it must bind to the running
# event loop, but __init__ is sync and may run without one.
if self._async_lock is None:
    self._async_lock = asyncio.Lock()
async with self._async_lock:
    ...

The author knew an asyncio.Lock has a relationship with an event loop. They built it lazily to avoid creating it without one. What they missed is that lazily is not the same as per loop: the lock is created once and then kept for the lifetime of the source. An asyncio.Lock binds to the loop that first awaits it under contention, and raises from any other:

RuntimeError: <asyncio.locks.Lock object> is bound to a different event loop

One asyncio.runper request is an ordinary deployment shape. A token source that outlives a single loop — which is what a long-lived credential object is for — hits a new loop on the next request. And contention is not an edge case here: collapsing a burst of concurrent callers into a single network refresh is the only reason the lock exists.

The fix rebuilds the lock whenever the running loop changes:

def _get_async_lock(self) -> asyncio.Lock:
    loop = asyncio.get_running_loop()
    with self._async_lock_guard:
        if self._async_lock is None or self._async_lock_loop is not loop:
            self._async_lock = asyncio.Lock()
            self._async_lock_loop = loop
        return self._async_lock

Two details that are not decoration. The guard is a threading Lock held across the check and the assignment and never across an await— otherwise two threads driving separate loops could each install a lock and silently lose mutual exclusion with each other, which is a worse bug than the one being fixed. And it is a separate lock from the existing _sync_lock: that one is held across a blocking network call, so an async caller waiting on it would stall its own event loop — bug #2 again, introduced by the fix for bug #4.

Why the tests did not catch three of these

Haystack has a real test suite, and it tests the async paths. It missed these anyway, for one reason that is worth stating plainly:

An async test that never runs two things at the same time is a synchronous test with extra syntax.

Every one of these tests passed because it awaited a single call:

—One URL means one fetch, so the shared cursor is never shared — there is nothing to race with.
—One asyncio.run per test, and an uncontended acquire never binds the lock to a loop. The bug needs two concurrent callers and a second loop; the existing async test had neither.
—Blocking the event loop is invisible when your coroutine is the only thing on it. It produces a correct result, slightly later, and the assertion is on the result.

So the regression tests had to manufacture the condition rather than the input. The OAuth one runs two callers through asyncio.gather, in two successive asyncio.run calls, with expires_in=0so the cache cannot serve the second round and both rounds genuinely refresh. That is three separate things all of which must be true or the bug does not reproduce, and the test carries a comment saying so — because the natural simplification of any of them turns it back into a test that passes either way.

I checked each regression test fails with the fix reverted. That step is not ceremony: on this kind of bug, a test that passes both ways is the default outcome, not the unlucky one.

The pattern, if you want to go find your own

Open any Python library that grew async support after the fact and read the *_async methods against their synchronous twins. Ask three questions:

01Does it write to self? If the async version can be entered twice, that attribute is shared mutable state between concurrent callers.
02Does every call inside it have an await? A bare call to something that does I/O is blocking the loop, and it will be the expensive branch, because cheap branches get converted first.
03Does it hold an asyncio primitive built somewhere else? Locks, events and queues belong to a loop, and objects routinely outlive loops.

Four of my five merged Haystack fixes came out of exactly that reading. The maintainers reviewed and merged all of them, which I mention for a specific reason: none of these needed deep familiarity with the framework. They needed someone to read the async path as its own code instead of as a translation.

See the rest of my work All contributions