Open source — August 2026

The pipeline you load is not the pipeline you saved

A component accepts a parameter, stores it, uses it — and leaves it out of to_dict. Save the pipeline, load it back, and the setting is gone, replaced by its default, with nothing in the saved file to show it was ever set. I fixed three of those, then wrote the check as a script and ran it over 254 components.

Haystack pipelines are meant to round-trip. You build one in Python, call to_dictor dump it to YAML, and something else — a worker, a container, a colleague — calls from_dict and gets the pipeline back. That contract is what makes a pipeline a configuration artefact rather than a script, and every component has to hold up its end: whatever __init__ accepted has to come back out of to_dict.

Nothing enforces it. to_dict is written by hand, listing the parameters one at a time, and __init__ is written by hand somewhere above it. They are two lists that must agree, and the only thing keeping them in agreement is that whoever adds a parameter remembers to add it in both places.

When they disagree, nothing raises. The value simply reverts to its default on the way back in, and the saved file looks complete because the key was never written.

The one that started it

TransformersZeroShotTextRouterhaystack-core-integrations#3808→merged

A zero-shot router classifies a piece of text against a set of labels and sends it out of the matching branch. Its multi_label parameter decides how the label scores are normalised: across the labels as a distribution, or each one independently. The branch is chosen from those scores.

to_dict did not include it. So a pipeline saved with multi_label=True came back with False, the scores were normalised differently, and the same text could leave through a different branch than it did before the round trip. Not an error — a different answer. Two sibling fixes shipped with it: a ranker that stopped asking its endpoint for raw scores, and an evaluator that fell from 16 concurrent LLM judgements back to 4.

Three components in one afternoon, found by reading. That is the part worth being suspicious of. If reading three files turns up three instances, reading is not the bottleneck — the class of defect is common, and the only reason it is not reported more often is that nobody is looking.

Writing the reading down

The check I had been doing by eye is mechanical: take a class decorated with @component, list the parameters of its __init__, list the keyword arguments its to_dict hands to default_to_dict, and subtract. It needs no imports, no API keys and no environment — the answer is in the syntax tree.

def serialized_keys(node: ast.FunctionDef) -> tuple[set[str], bool]:
    """Keys to_dict passes on, and whether we understood the shape of the function."""
    keys, understood = set(), False
    for sub in ast.walk(node):
        if isinstance(sub, ast.Call):
            name = getattr(sub.func, "id", None) or getattr(sub.func, "attr", None)
            if name == "default_to_dict":
                understood = True
                keys.update(kw.arg for kw in sub.keywords if kw.arg)
                if any(kw.arg is None for kw in sub.keywords):
                    understood = False   # **kwargs: shape not statically knowable
        elif isinstance(sub, ast.Dict):
            for k in sub.keys:
                if isinstance(k, ast.Constant) and isinstance(k.value, str):
                    keys.add(k.value)
    return keys, understood

Eighty-two lines in total, standard library only. The understood flag is the important half: when to_dict forwards **something, the set of keys is not knowable from the source, so the class is skipped rather than reported. A tool that guesses in the ambiguous case produces a list nobody finishes reading.

What it found

Across haystack and haystack-core-integrations:

• 329components decorated with @component
• 254of them define both their own __init__ and their own to_dict
• 13flagged as dropping at least one parameter
• 5real, after reading all thirteen by hand — eight parameters between them

Five out of thirteen is a precision of 38%, which sounds bad and is not the point. The point is 254 down to 13: the script does not decide anything, it decides what to read. Thirteen files is an evening; 254 is a project nobody starts.

The one with teeth

VertexAITextEmbedderhaystack-core-integrations#3873→withdrawn

Google's text embedding models take a task_type — RETRIEVAL_QUERY, RETRIEVAL_DOCUMENT, CODE_RETRIEVAL_QUERY and so on. It is not metadata. It goes into every TextEmbeddingInput and changes the vector that comes back; embedding a query as a document is a well-known way to lose retrieval quality for no visible reason.

VertexAITextEmbedder.to_dict serialized the model name and the two GCP secrets, and dropped task_type, progress_bar and truncate_dim. Its sibling, VertexAIDocumentEmbedder, in the same integration, serializes all three.

Save a pipeline as CODE_RETRIEVAL_QUERY, load it back, and it embeds as RETRIEVAL_QUERY. Same corpus, same code, different vectors, no error.

And it is not going in. A maintainer pointed out, on a separate issue I had filed about the same integration, that google_vertexis archived — it says so in the status table of the top-level README, which I never opened. I pulled the commit; the pull request now covers the two active integrations only. The finding holds and the fix is right, and neither matters if the package is not maintained.

The part worth keeping is the shape of the mistake. I had already left NvidiaGenerator out of the same pull request because the component is deprecated, and then failed to apply that exact test one level up, at the integration. A script that reads code sees only code. Whether anyone still ships it is written somewhere else, and the audit had no step that went and looked.

The rest, in three more pull requests: S3Downloader dropped boto3_config, which carries the timeouts, retries and proxy settings of the AWS client — while five sibling components in the same integration serialize it. TransformersExtractiveReader dropped overlap_threshold, which decides which overlapping answers get deduplicated away. AzureAISearchDocumentStore dropped include_search_metadata, the flag that decides whether Azure's @search.* fields ride along on every retrieved document. And in Haystack itself, OpenAIImageGenerator dropped timeout and max_retries, so a reloaded pipeline quietly went back to a 30-second timeout and 5 retries — that one and the Azure fix have since merged.

original = OpenAIImageGenerator(timeout=120.0, max_retries=10)
restored = OpenAIImageGenerator.from_dict(original.to_dict())

original._client_kwargs()   # {'timeout': 120.0, 'max_retries': 10}
restored._client_kwargs()   # {'timeout': 30.0,  'max_retries': 5}

The tests already knew

The strongest evidence that these were oversights rather than decisions came from the test suites, which had quietly written the omission down three different ways.

• 1VertexAITextEmbedder's test_to_dict carried the three missing fields commented out, under the note “The following are missing because they are not explicitly included in to_dict”. Someone saw it and recorded it instead of fixing it.
• 2S3Downloader's test_to_dict is parametrized over boto3_config=[None, {"read_timeout": 10}] and asserts the same dictionary for both. The parameter could not change the assertion, so the parametrization could never fail.
• 3OpenAIImageGenerator's test_to_dict_with_params passes timeout=60 and max_retries=10, then asserts a dictionary containing neither. Nobody writes that on purpose; it is what you get when you copy a sibling's test and adjust it until it goes green.

A test written to match the current behaviour will always pass, and it converts a bug into a documented feature. All three now assert the values, and all three fail with the one-line fix reverted — which I checked, because a regression test that passes either way is not a regression test.

The eight it got wrong

The false positives matter more than the hits, because they are what a reviewer will ask about. Every one of the eight came from a legitimate pattern the script cannot see, and they fall into three shapes:

• aSerialized outside the call. DocumentSplitter and ChineseDocumentSplitter take a splitting_function, which is a callable and cannot go through default_to_dict. They add it afterwards, by assigning into the dictionary the call returned. The parameter is serialized; it just is not a keyword argument.
• bFolded into another parameter. Every transformers component resolves model, task, device and token into one huggingface_pipeline_kwargs dict in __init__, and serializes that. HuggingFaceAPIChatGenerator does the same with stop_words, merging it into generation_kwargs["stop"]. The value survives the round trip under a different name.
• cExcluded on purpose. LangfuseConnector drops httpx_client and TransformersChatGenerator drops async_executor. Both are live objects that cannot be written to a dictionary at all — one of them says so in a comment on the line above, the other does not, which is the difference between a decision recorded and a decision remembered.

Shape (c) is the one worth noticing. The script and the author disagree, and the author is right — but the same code shape also produces a real defect when the exclusion is an accident rather than a decision. Only reading the file tells you which, which is why the output of a script like this is a reading list and never a patch.

The one I left alone, then didn't

NvidiaGenerator drops timeout exactly like the others. It also raises a FutureWarning on construction saying it is deprecated and will be removed in favour of NvidiaChatGenerator. Fixing serialization on a component scheduled for deletion is churn in someone else's review queue, so at first it was named in the pull request and left out of the diff, for the maintainers to call.

I changed my mind and opened it as its own one-line pull request (#3923). Deprecated is not removed: the component still shipped, still serialized, and its four sibling Nvidia components all serialize timeout. A maintainer closed it a week later — they are deleting the component outright in another PR, so the fix has nowhere to land. The first instinct was the right one; opening it cost a reviewer the thirty seconds to say so.

What this generalises to

The specific bug is Haystack's, but the shape is not. Any framework that serializes objects by listing their fields by hand has two lists that must agree and nothing checking that they do: LangChain's serializable classes, anything with a to_dict/from_dict pair, any dataclass written out field by field. The check costs 82 lines and runs in a second over a whole repository.

And the honest limit: it finds parameters that are missing, not parameters that are serialized wrongly. A value written under the wrong key, or run through a converter that loses precision, passes this audit and still breaks the round trip. The stronger check is a property test — construct with non-default values, round-trip, assert equality — which needs the class to be constructible, which is exactly what a static pass avoids needing. They catch different things, and the cheap one is the one you can run today.