Meet Collaudino, our new QA agent

Tundr builds a platform that companies use to give benefits to their employees.

When an engineer here changes our software, the change goes out as a pull request. A pull request is a proposal for a change, and colleagues review it before they accept it. In the twelve months to August 2025, we opened 703 of them, about sixty a month. In the twelve months after that, we opened 1151, nearly a hundred a month, and 201 in August 2026 alone.

We think that part of the increase comes from people who write code faster with AI assistants. Another part is Claudito, our coding agent (an AI program that writes code). In May, Claudito started to open pull requests of its own, and it now opens about thirty a month. A person reviews each one before it goes anywhere. We have no separate team for testing. The engineer who owns a pull request must also test it, on top of writing code and reviewing the work of colleagues.

Reviewing a pull request has two parts. The first part is reading the code, and that part kept up. The second part is trying the change out for real. For every pull request that adds a feature or a fix, we build a private, throwaway copy of our whole platform with the change inside it. We call it a feature environment.

Our platform has many small services that talk to each other with messages. The feature environment contains all of these services at the version of the pull request. It also contains the channel that carries their messages, the databases, and a set of made-up test data. None of the data is real, and no real users are on it. Each feature environment has its own web address. Anyone on the team can log in there as a test user and try the change as if it were live.

Our build system creates the feature environment automatically when the author first uploads the code of the pull request. It updates the feature environment each time that the author uploads more code, and it deletes it when the pull request closes. To save money, we stop the feature environments that nobody uses.

Trying a change out on the feature environment was manual work. A person logged in as a test user, pressed the buttons, and sent the messages that our services send each other. Then the person read the result in the logs and in the database. The logs are the running record that each service keeps of what it did. The database is where each service stores its data. This work took time, and at two hundred pull requests a month, there were not enough hours to do it for every pull request.

There is also one kind of problem that our automatic tests cannot see. Each service runs its own tests. When one service changes the content of a message, the services that receive the message keep passing their tests. Their tests use a frozen copy of the old message, so the new message never reaches the code that must read it.

Collaudino is the agent that we built for the second part of review: trying each change out on its feature environment. Claudito writes code, and Collaudino tests it. People still review both and decide what goes in. Collaudino decides what to try, and it writes a report of what it proved.

In its first four weeks, from 24 August to 18 September, Collaudino wrote 97 reports on 58 pull requests. One of them reported a bug across two services, before the change reached customers.

We also built some limits into Collaudino from the start. It cannot run commands of its own choice on our systems. It has no way to approve, edit or merge a pull request (merging is the step that puts an accepted change into our code).

This post is a high-level overview. Two more posts will go deeper into the technical side: the internal architecture of Collaudino, and how it works with our systems.

What it does

You ask for Collaudino with a comment on the pull request: @collaudino e2e. (E2e is short for end to end: run the change, not only read it.) Collaudino reads the change and then runs it on the feature environment. If the feature environment is stopped, Collaudino first starts the services and waits until they are up. A run usually takes fifteen to twenty-five minutes.

It uses the platform as real users do, as five different kinds of user, for example an HR manager or an employee in the app. It also sends the messages that the services send each other. Then it reads the results in the logs, in the database (which it can only read), and in the log of undelivered messages.

Each thing that Collaudino tries is a scenario, and each scenario ends with one of five results: passed, failed, blocked, skipped or flaky. Blocked means that something on our side got in the way, usually the feature environment itself, so the scenario tells us nothing about the change. Skipped means that Collaudino set the scenario aside and gave the reason, and a skipped scenario does not count as a failure. Most often, the scenario cannot exist in a feature environment, for example anything that needs a real bank. Flaky is for a scenario that passed on one try and failed on another.

To decide what to try, Collaudino starts from what the change says it does. It reads the description of the pull request, its task and design document, and the scenarios that we already wrote for the same parts of the product. While the services start, a second agent reads the code and lists what could go wrong. Its rules ask for one normal case and at least two wrong ones for each behavior that the change promises, for example bad input, a user without permission, or the same message delivered twice.

When the run ends, the report is on the pull request. It has one headline and one line for each scenario. Each line has a reference code, so that the author can find the matching entries in our logs. The findings (the problems that Collaudino saw) appear next to the lines of code that they concern. A status mark on the pull request shows the outcome. If the author uploads new code while a run is in progress, the run stops.

The run that convinced us came in early September. Companies that use Tundr register their offices with us, and a partner of ours keeps its own copy of those records. A change made it possible to clear an optional address on a company office. Nine scenarios passed, and one failed.

Collaudino followed the cleared address into a second service, which keeps the records of our partner in sync. It found that the message that announced the change dropped the address. So the update that this sync service sent to the partner said nothing about the address, and the partner never learned that it was gone.

In the report, a blocking finding is a problem that Collaudino thinks the author must fix before the change goes in. A suggestion is a finding that the author can take or leave. This is the headline and the finding, with identifying details removed:

E2E failed: 9 passed, 1 failed, 1 blocking finding, 1 suggestion.

Blocking: the cleared address never reaches the partner. Clearing the field is saved in the first service, but the message it sends out drops the address field, so the service that syncs offices to the partner sends the partner an update with no address in it.

The author uploaded a fix 35 minutes later, and the bug itself is an ordinary one. What we talked about afterward was where Collaudino looked for it. When a service changes what it sends, the rules of Collaudino tell it to test the services that depend on that service, for real.

Who does what

• The author asks for the run, reads the report, and fixes what is real or explains why it is not.
• The reviewers still read the code and decide whether the change goes in, for Claudito’s pull requests as much as for ours. Collaudino has no say in that decision.
• Each team is meant to write the test scenarios for the parts of the product that it knows.
• The people who run the feature environments are responsible for a broken one.
• The team that built Collaudino maintains it and its written methods, the step-by-step guide that Collaudino uses to test a change.

A scenario file is plain text in our code. It says which services take part, what a good run looks like and how to prove it, and which wrong inputs to try. The team that built Collaudino wrote the five files that exist today. They cover cases such as the creation of a company or an update to a wallet. An automatic test fails when their facts go out of date, and anyone can add a file with a normal pull request.

How it works

Two programs share the work. The runner is a small, fixed program that does the parts that never change. It reads the comment, makes sure that the feature environment is ready, and publishes the report at the end. Between these steps, an AI model does the thinking. It chooses the scenarios, runs them, and works out what the evidence means.

The model can act only through a short list of actions that we wrote for it. Examples are “ask this service to do something, as the HR person of a company”, “send this one message”, and “read these logs”. The model cannot do anything that is not on that list, and most of the rest of the design follows from this limit.

What the model can do

The model has 13 actions on our systems, and seven of them only read. Every action that changes something on our platform is locked to the feature environment of that pull request. An action aimed anywhere else is refused, and the model gets the refusal as a plain answer, so that it can change its plan. Collaudino writes only one thing on its own: a ticket in our task tracker, for a confirmed bug that the pull request did not cause. Only the runner can write to the pull request. The model has no action for it, and an automatic test makes sure that this stays true.

Its instructions say this in one line:

You read the code; the feature environment runs it.

How a run gets its result

If the feature environment is out of date or a service is still starting, the scenario ends as blocked, and sometimes the whole run does. Then the status mark on the pull request is neutral, and it never counts against the change.

The model does not decide the result of the whole run. The runner calculates it from the results of the scenarios, so a run cannot claim success if one of its own scenarios failed. Whatever happens, the runner publishes a report. If a run ends without a report, nobody knows that they must go and look. In plain words, its rules say this:

Do not claim anything that you cannot back up. If you cannot prove a result, say so. Do not guess.

Where the instructions come from

Collaudino’s instructions, both its fixed rules and its written methods, come from the main version of our code. The runner opens the pull request in a separate place, where Collaudino can read it but cannot change it. The pull request has no effect on those instructions, so no pull request can rewrite the rules that judge it. One of its first reviews was of the pull request that added those written methods, so that pull request contained its own copy of them. Collaudino loaded the copy from the pull request instead of its own copy, found this mistake itself, and reported it as a blocking finding. We fixed it by the next morning.

We enforce the main limits in code: either the code refuses the action, or the action does not exist. The instructions help the model to plan, so a change to their words gives the model no new abilities.

What broke, and what we changed

In the first four weeks, 38 of the 97 reports came from full runs, in which Collaudino tried the change on a feature environment. For the rest, people asked it only to read the code and comment. Of the 38 full runs, 9 failed and 8 passed every scenario. The other runs did not give a full answer: 13 ended blocked, and 8 skipped at least one scenario.

Asking the log too early

When a message cannot be delivered, our platform sets it aside and records this in a log. One run asked that log about a message before the log existed. It took “not found” as a permanent fact, marked the scenario as blocked, and wrote a ticket in our task tracker. At the end of the same run, the final check of that log found that same message.

Collaudino keeps short notes from one run to the next, so that it does not discover the same odd behavior again every time. The same run made a second mistake, and this mistake went into those notes. It searched the log for a period of time and found nothing. From this, it decided that the clock of the feature environment was a year ahead. It wrote that down as if it were a fact.

Since that run, the action that reads the log does not answer with only “not found”. It also says at what time the message was still missing, and it repeats the period that it searched. The notes also refuse this kind of claim, when Collaudino writes a note and when it reads one. Roughly, the rule that we added says this:

A note that says that undelivered messages never arrive, or that the log never ran, is only one old observation. Read the log yourself, after the period that you searched.

What it does not do

Nothing in our process depends on the result of Collaudino, so we can switch it off at any time. This design limits the damage that Collaudino can do. It does not test how the platform behaves when many people use it at the same time. A person still decides whether the change does what its task asked for. A task is the item in our task tracker that describes the work.

What is next

Answering replies to its findings

Each finding of Collaudino starts a short conversation on the pull request. Today, when an author replies to a finding, Collaudino does not answer. We measured this over the three weeks before we planned the work: people wrote 165 replies to its findings, and Collaudino wrote none. Most replies were short acknowledgments. About a dozen were questions or objections that deserved an answer.

The next run on a pull request will read the replies that nobody answered yet, and it will answer them. It can agree with a fix, accept a correction, answer a question, explain why it still disagrees, or leave a decision to the people. Each reply will say which of these it is. It can disagree once, with evidence, and after that it will let the people decide. It will give at most three answers in each conversation, so it cannot argue forever.

It will not close a conversation (mark it as settled), and everything that it writes to the pull request will still go through the runner. Later, it will also look for replies at fixed times, even when nobody starts a new run.

What a run costs

Each run uses the AI model, and that costs money. One line at the top of each report shows how much that run used the model. Each run also records this usage in a small table, with one line for each model that did work. Claudito records its usage in the same table, so that one table covers both Claudito and Collaudino. Next, we will use this table to show the cost of testing for each run and each week. It will also help us decide when to change how we pay for the model.

Learning from past runs

Every run already stores its transcript (the full record of what the model said and did), its report, and its evidence. Today, we read them and improve the written methods by hand. We want to automate most of this work in three steps:

1. A program records what later happened to each finding: fixed, accepted, dismissed with a reason, or ignored.
2. Once a week, a job reads these records and writes up the patterns that it sees.
3. Another job turns each write-up into one proposed change to the written methods. A test replays past pull requests with known answers and refuses any change that makes the results worse.

A person still decides whether to merge each change. Collaudino never reads those write-ups during a run, and this process cannot change its rules or its actions. The idea comes from a recent paper, WikiSkill.

Two smaller changes are already in use. First, we wrote down the known behavior of the platform in one place, so that Collaudino stops reporting it as a finding. Second, each run now starts with the comments on its task and with the related pull requests. One day, we want it to run on every pull request without anyone asking.

If you want to work on problems like these at Tundr, complete our open application form.