I ship LLM features
you can trust in production.

26k stars — deepset's framework for production RAG and agent pipelines

Three merged fixes, all concurrency bugs on the async path, all found by reading the components rather than from an issue. LinkContentFetcher rotated its User-Agent on a cursor kept on the component, but run() fetches the URLs concurrently: a retry triggered by one URL advanced the user agent for all the others, and each finished fetch reset the cursor underneath the requests still in flight, so most retries went out un-rotated — the one thing the feature exists to do. And EmbeddingBasedDocumentSplitter.run_async was only async for its first pass: the recursive re-split of over-long chunks called the blocking embedder, so the most expensive part of the work ran on the event loop. The third was the same shape one layer up: LLMDocumentContentExtractor.run_async read every file from disk, rendered its PDF pages and base64-encoded them inline, so the whole conversion ran on the event loop before the first LLM call was even scheduled. Each ships a regression test, and I checked each one fails without the fix rather than passing either way.

the 100 provider integrations behind Haystack, each its own package

Two merged fixes, found with scanners I wrote for the two contracts every integration has to honour. The first: OAuthRefreshTokenSource built one asyncio.Lock and kept it for the life of the source, but that lock binds to whichever event loop first contends for it and raises on any other — so a source reused across loops, which is what one asyncio.run per request gives you, failed on the second loop's first contended refresh. The second: three components took an init parameter, used it at run time and left it out of to_dict, so the setting silently reverted to its default once a pipeline was saved and reloaded, with nothing in the saved file to show it had ever been set. TransformersZeroShotTextRouter lost multi_label, which decides whether label scores are normalised across labels or scored independently — and the router picks its output branch from those scores, so the same text can route somewhere else after a round trip. A TEI ranker stopped asking its endpoint for raw scores, and a Ragas evaluator dropped from 16 concurrent LLM judgements back to 4. I opened them as three PRs and the maintainers reviewed them as one, which is how they merged.

PR #3873 — the same defect class as my merged fix, found again by script

After fixing three components that dropped an init parameter from to_dict, I wrote the check as a script: a small AST pass comparing every @component's __init__ parameters against the keys that actually reach default_to_dict. It flagged two more, each verified by hand. S3Downloader dropped boto3_config, which carries the timeouts, retries and proxy settings of the AWS client, while five sibling components in the same integration serialize it; TransformersExtractiveReader dropped overlap_threshold, which decides which overlapping answers get deduplicated away. In each case a sibling already serialized the parameter, and the existing tests showed the omission was an oversight rather than a decision: one was parametrized over a value that could not change its own assertion, so the parametrization could never fail. The audit also flagged google_vertex, and a maintainer pointed out on a separate issue of mine that the integration is archived — it says so in the README status table, which I had not opened. I pulled that commit; the PR covers the two active integrations. A script that reads code sees only code, and whether anyone still ships it is written somewhere else.

PR #12518 — the same audit, run against the framework itself

Merged. OpenAIImageGenerator stored timeout and max_retries, used them to build its OpenAI client, and left both out of to_dict, so a saved pipeline came back with the 30-second and 5-retry fallbacks instead of the values it was configured with. Its siblings — the chat generator and both embedders — already serialize them, and the embedders because of a fix the maintainers made for exactly this in 2025. The clincher was their own test: it passed timeout=60 and max_retries=10, then asserted a dictionary containing neither.

PR #3925 — the same audit, one more integration

Merged. AzureAISearchDocumentStore stored include_search_metadata and read it when converting search results — it decides whether Azure's @search.* fields ride along on every retrieved document — and left it out of to_dict, so the setting reverted to its default on reload. A sibling already serialized it; the round-trip test fails without the fix.

PR #3926 — sync/async parity, same class as the concurrency fixes

CohereDocumentEmbedder's sync path loops over texts in slices of batch_size before calling the embed endpoint; run_async sent them all in one call. Cohere caps texts per request, so on a real document set the async path fails where the sync path works. run_async now batches the same way, with a test asserting the batch count.

one of the two standard Python frameworks for RAG and agents

Five fixes in llama-index-core, none of them from an issue — I found them reading the retrieval and evaluation code. The retrieval metrics returned scores outside their own range when a ranking repeated a node id, which is what fusion retrievers produce: hit rate and average precision both came back as 2.0, NDCG as 1.63, so any mean over an eval set stopped being comparable. MMR discounted each candidate only against the result picked immediately before it, so a near-duplicate stopped looking redundant as soon as anything unrelated was picked in between and returned to the ranking — the one thing MMR exists to prevent. The multi-modal evaluator scored image nodes as text results, because ImageNode subclasses TextNode and the two type checks were written independently. default_parser, the parser behind CorrectnessEvaluator's judge output, unpacked a two-line response and raised ValueError — aborting the whole eval run — whenever the judge answered with a score and no reasoning line. And CohereRerankRelevancyMetric guarded a missing API-key variable with except IndexError, but a missing env var raises KeyError, so the helpful "pass in an API key" message never replaced the bare traceback. Each fix ships with a test that fails without it.

Python framework for ML workflows and LLM agents

Added .docx support to the RAG document loader, so the framework ingests Word documents alongside PDFs and text.

Python framework for ML workflows and LLM agents

Corrected the RAG optional-dependency install instructions, which pointed at a package name that does not exist.

open — robustness fixes in the config and remote-client code

Parser.load_configuration ran yaml.safe_load then indexed the result, so an empty or non-mapping fenn.yaml raised a bare TypeError into framework internals instead of a message naming the file. And RemoteClient wrapped request timeouts as the library's typed NetworkError but re-raised connection and TLS errors raw — which the CLI does not catch — so an unreachable host produced a traceback rather than a clean error. Each ships tests; the second added the first tests RemoteClient had.

the standard RSpec linter

Merged. RSpec/LeadingSubject crashed on Ruby 3.4's implicit `it` block parameter: the cop walked `:block` AST ancestors only, so an example group written as an itblock or numblock was never found and the lookup returned nil. Widened to `:any_block`, with a regression spec pinned to Ruby 3.4. The spot check the maintainer asked for turned up seven more cops with the same blind spot — I opened that as a separate PR and closed it when they said they had already decided not to chase every block-syntax case, which is their call to make.

performance cops for Ruby

Fixed Performance/ConstantRegexp emitting invalid code when autocorrecting a regexp used as a pattern in case/in pattern matching.

Rails inventory app for nonprofit essentials banks

The participant drop-downs displayed one column and were ordered by another, so the list was sorted on a value the user could not see — and where those collided the order fell through to the database cluster's collation, which is not the same in CI as in production. They now order on the name actually rendered, with runs of digits compared by value so Store 9 comes before Store 10. Brakeman flagged my first attempt as SQL injection, because the ordering expression interpolated into Arel.sql; it is now a literal constant with nothing interpolated. The maintainer also asked for a survey of every other drop-down in the app, so I traced each rendered select back to the query that builds it and posted the results on the issue.

API-powered email delivery for Ruby apps

Four merged, all shipped in the gem's 1.1.0 release: MailerSend, Mailtrap and SMTP.com provider integrations, which closed the gem's standing request for more providers, plus a NameError that broke Mailgun and Mailjet on Ruby 3.4. Base64 left Ruby's default gems in 3.4 and those two providers were the only ones calling it without requiring it, so the gem installed cleanly and raised on send — the kind of break that only shows up on the version you are not testing on. The fifth, the cc/bcc fix below, shipped in the same release.

PR #62 — the fix for a bug I reported, at the maintainer's request; merged in courrier 1.1.0

The gem accepts cc: and bcc: on every email and six of its providers never read them, so the copies were dropped with no warning: a working cc became a silent no-op the moment you switched provider. I reported it as #58 with the per-provider table; the maintainer asked for the PR. Each provider now takes the fields in the shape its own API wants — a comma-separated line for Mailgun, MailPace and Postmark, address objects for Mailjet and SendGrid. SparkPost has no cc or bcc field at all: every copy is a recipient there, and what separates a cc from a bcc is whether the address is repeated in the CC header, with header_to holding the visible To line so a bcc does not see itself addressed directly. Reading the lists through one helper also fixes Mailjet, SendGrid and SparkPost sending several to: addresses as a single malformed one — filed as #59, closed as done but still reproducible on main, which the PR fixed too. Merged and shipped in 1.1.0.

the agent and eval framework from the Pydantic team

pydantic-evals reads expected_output=None as "no expectation", so a Case written to assert that a task returns None is skipped instead of checked: EqualsExpected records no assertion at all and the case averages 1.00 whether the task returns None or the wrong answer outright. The sentinel and the legitimate value are the same object, which is why no amount of care at the call site can tell them apart. I filed it as #7934 with a reproduction through the public API; the maintainers hold the skip as intended design, and they are right that changing it would move documented, serialized semantics. So the fix is the other one available: the trap is now stated where the behaviour is defined, with Equals(value=None) named as the evaluator that does assert it. An eval that cannot fail is worse than no eval, and a user who cannot see that from the docs will not find it from the report either.

the LLM framework for Ruby

Six merged, all edge cases that crashed a caller instead of degrading. #tool_calls raised NoMethodError on a response with no usable choices — an error payload, a content-filtered completion — and on an explicit "tool_calls": null, which is the shape that closed the year-old issue #1011. The JSONL loader raised on a blank line; the directory loader returned an exception object into the results array where the caller then called Data methods on it; AnthropicResponse#tool_calls returned only the first tool_use block, silently dropping Claude's parallel tool calls. Each ships a regression test, and one PR was a follow-up adding the CHANGELOG lines the maintainer asked for after merging the batch.

PR #911 — one framework for every AI provider, in Ruby

Two sibling tool-call parsers disagreed on the same input. ChatCompletions::Tools guards empty arguments and returns {}; Interactions::Tools ran JSON.parse on any string, so a function_call step with "arguments": "" — a valid zero-argument call — raised ToolCallParseError and failed the whole run. Mirrored the guard, with a spec covering empty string, missing key, hash passthrough and malformed JSON.

issue #7927 — accepted for implementation by the maintainers' triage

pydantic-evals builds the judge's prompt by iterating anything that is a Sequence and is not a str. bytes, bytearray and memoryview are all Sequences, so an eval task returning binary content had it rendered as one decimal byte value per line: asked whether the output mentions a fox, the judge received 84 104 101 32 113 and graded it. No exception and no warning — a plausible score for content the judge never saw, which is the one failure an eval framework must not have. Reproduced through the public API with no provider key, using a stub model to capture the prompt. Their triage reproduced it, corrected one detail of my mechanics, and accepted it for implementation; I confirmed the correction on the issue. A second report from the same reading (#7928) is with the maintainers: a run's averages show no denominator, so an evaluator that scored 1 case of 4 reads exactly like one that scored all 4 — and a judge that runs out of quota mid-run therefore reports a higher score than the run earned.

issue #12519 — main was red for every contributor

My own pull request came back with a red test job I had not caused, so I checked before explaining it away: the same two tests fail on a clean main, on all three operating systems. An openai release had added three fields to its usage models, and those tests assert an exact usage dictionary. Bisected to the version — 3.5.0 passes, 3.6.0 fails, nothing else changed — and filed with the reproduction and the three field names. A maintainer merged the fix the same day, deriving the expected usage dict from the model instead of hard-coding it, so the next additive field does not turn main red again.

issue #3789 — triaged P3 by the maintainers, closed by my fix

Filed separately from the fix, with a standalone reproduction: an asyncio.Lock cached for the life of an OAuth token source binds to the first event loop that contends for it, so reusing the source in a new loop raises. Writing the report before the patch is the part that makes it reviewable — the maintainers can confirm the defect without reading my diff first.

issues #58 and #59

cc and bcc are accepted by the gem's public API and silently dropped by 8 of its 14 email providers, so those recipients never reach the message that goes out. Found by diffing what each provider adapter does with the fields the shared interface promises, and filed with the per-provider table and a runnable reproduction rather than a single example. The maintainer asked for a PR, which merged as #62 above and shipped in courrier 1.1.0; he had already shipped the address-list helper the report suggested. #59 is narrower and from the same pass: Mailjet sends multiple recipients as a single malformed address.

18k stars — LLM evaluation framework

Four fixes, all found by reading the metric and benchmark code rather than from an issue. A custom judge model returning (text, cost) worked under a_measure() and raised AttributeError under measure() — the async helper unwrapped that tuple, the sync one never did, a gap left over from a fix to only one side after an internal test hit it (#3270). TruthfulQA's MC2 scorer counted every repeated correct index in a model's structured output, so a repeated answer pushed the reported percentage past 100 (#3271) — the same class of defect as my llama-index fix above, in a different framework. HumanEval's reported accuracy treated each task's pass@k estimate as a pass/fail flag instead of averaging it: two tasks scoring 0.3 and 0.4 pass@1 printed "Overall HumanEval Accuracy: 1.0" (#3272). And IFEval's lowercase/uppercase checker failed a response with no letters at all, like "42", because str.islower() requires a cased character to return True — a stricter condition than the instruction it was checking (#3273). Each ships a regression test verified to fail without the fix.

22k stars — LLM evaluation and observability platform

SpearmanRanking checked that two rankings had the same length and the same set of items, but not that each was actually a permutation. A duplicate that kept both sets equal slipped through, and the rank lookup — a dict comprehension — silently kept only the last occurrence's index, returning a numeric correlation for an input whose ranks were never well defined instead of raising. Filed as issue #8275 with the reproduction first, per the repo's own contribution process, then opened as a draft PR with both fixes and tests.