The Devin Issue Pilot

In October I wanted Devin to own an issue the way Claude Code already could on my laptop: read what our sales lead filed, ask what it needs to ask, fix it, prove it, and come to me only for the exceptions. Devin had worked on our product briefly in late August and early September, then gone quiet. Almost every fix still went through Claude Code in a terminal on my MacBook, so every fix waited for me.

This is what happened from Friday 2 to Tuesday 6 October 2026, when I tried again.

In short

• What happened. Devin merged its own fixes, a review found something wrong in every one, and the rule for who merges took five positions in four days.
• What works now. Every new issue starts a Devin session. A second Devin reviews the change before any PR exists, and a human merges.
• Why a reviewer. Devin caught every defect we planted in code it had not written, but its own work kept passing its own checks and failing review.
• The cost. We used 86% of a weekly quota on a mode we had never checked. It now runs on SWE-2.
• Still open. Whether a Devin reviewer is as good as Claude Code on Opus 5.5. We compared them once, and Devin found about half.

Terms used here

• Claude - Claude Code on Opus 5.5, in a terminal on my MacBook. My usual coding agent, and the reviewer in this story.
• The playbook - the agent fix playbook in our company brain: rules every fix must follow, such as listing every place that reads or writes the data, and proving each new test fails without the fix. Scores such as "4 of 8" count the rules a change met, out of those that applied to it; a half point means partly met.
• Guards and the bypass - checks that run before every push and block it on a known bad pattern. A bypass skips them for one push.
• Reviewer child - a second Devin session that the working session starts to review its branch with fresh context, before any PR exists.
• Verdicts - the reviewer's answer: PASS, PASS WITH GAPS (fine, with the gaps listed) or FIX FIRST. NEEDS JENSEN means the issue is waiting for me.
• SMALL and STANDARD - size tiers. A STANDARD change gets the full test suites re-run.
• Fusion, Normal and SWE-2 - Devin's agent modes. Fusion pairs a frontier lead model with a cheaper sidekick, Normal balances cost and intelligence across frontier models, and SWE-2 High is where we ended up.
• ACU - Devin's unit of compute. Budgets are set in ACUs.

Disclosure. I am a Devin ambassador. This account comes from my Claude Code transcripts, Devin's session records, our company brain and the Git history. Repository names, usernames, issue and PR numbers, customer data and internal identifiers are left out, and other people are described by role. My quotes are copied as I typed them; square brackets mark a replaced name. Times are Singapore time (UTC+8).

Friday 2 October

• 21:31 - "ok i'm planning to use devin cloud." Claude turned every past Claude session into a readiness checklist in the company brain, where Devin can read it.
• 21:49 - I let Devin open PRs and merge them into the development branches: merge rule 1. Two minutes later Claude warned that nothing else would stop a bad change.

Saturday 3 October

• 01:07 - Devin merged six of its seven overnight PRs within 30 seconds. CI was green.
• 07:42 - Claude's review found something wrong in every fix, including tests that still passed with the fix removed. Six repairs were merged by 11:42, and Claude wrote the playbook.
• 11:41 to 15:00 - Merge rules 2 and 3: first a separate reviewer or a human approves, then Devin never merges. GitHub could only enforce either on a paid plan.
• 17:35 - Asked to review branches in which we had planted five defects, Devin caught all five with no false alarm.

Sunday 4 October

• 08:36 - Merge rule 4: agents may merge into the development branches again, with a backup and a reverse script for any data change.
• 08:46 - I merged two of Devin's PRs unreviewed. Send could never open its dialog. Score: 4 of 8.
• 12:03 - "how can we make sure devin when assigned a task can perform up to your standards without our supervision?" Four minutes later: "Ideally we start an independent devin session".
• 19:33 - A blind second Devin reviewed one fix alongside Claude and found about half of what Claude found.

Monday 5 October

• 00:51 - I approved the automation design with merge rule 5: the sales lead or I merge. The form said "Org default (Normal)". Neither of us noticed.
• By 07:51 - The automation was live.
• 13:52 to 14:07 - The sales lead merged four of Devin's PRs within minutes. A review afterwards found false claims in the PRs and a data change nobody had disclosed.
• 17:10 to 17:35 - "None of your suggestions will solve the problem. I'm NOT buying github teams". Review moved before the PR, done by a second Devin (prompt v4).
• 23:46 - "my devin is at 86% usage. so let's switch to SWE 2 instead?" Claude found the automation had run on Normal all day, not Fusion. I set SWE-2 High as the default; Claude confirmed it at 23:51.

Tuesday 6 October

• 00:27 - Prompt v5.1 saved: a softer verdict, PASS WITH GAPS, and sessions that archive themselves when they finish.
• 00:31 - I let Devin reach the deployed development site, which its network policy had blocked all day.

My Sunday question set the bar: "up to your standards". "Your" meant Claude Code, and every Claude Code session and reviewer in this story ran Opus 5.5. The bar was not the model alone. It was a fresh reviewer for each change, briefed to find what green CI misses and to break the code on purpose, and a lead session that re-checked its evidence. Against that bar, Devin's own checks kept passing work the reviews then failed:

Yet reviewing code it had not written, Devin caught every planted defect. Claude's diagnosis on Sunday: "Devin reviews others' code well, but it does not apply that adversarial review to its own work." When I said they were "actually the same model", Claude answered: "It's the setup, not ability". Devin started each session without the lessons, on a half-built machine with no browser test tools, and nothing made it back up its claims. More instructions would not fix that; the playbook already said it all. A second session briefed to find fault would. (Whether they really were the same model I could not tell: Devin does not show which model runs a step.)

Opus 5.5 was not flawless either; Claude's mistakes are in the table further down. What made the Claude Code side trustworthy was a separate reviewer, not a model that never errs. The reviewer child copies that.

How we got to the design

Three things ruled out the obvious answers. The sales lead merged within minutes, so a human merge step was not a review. GitHub's branch protection needed a paid plan I would not buy. And I did not want Claude in the loop. Devin's own PR review only starts after a PR opens, but Managed Devins let one session start another. So the review moved before the PR: the working session starts a reviewer child, and only a pass opens a PR. The automation prompt went through seven versions in under a day:

Is it on par with Claude Code on Opus 5.5?

Not measured, beyond one blind review. What we have:

• Devin reviewing planted defects: five of five, no false alarm. The defects were modelled on what the Opus 5.5 review had caught.
• Devin reviewing real work, blind: about half of the Opus 5.5 findings, one real problem Claude missed, and a more lenient grade. One fix, so not like for like: Claude's reviewers tried 18 to 35 deliberate breakages per round.
• Devin's own work: the scores above.

So the reviewer was necessary: working alone, Devin fell short of the bar, while reviewing others it met much of it. Whether builder plus reviewer closes the gap is not measured yet. I will lighten review once about five PRs in a row pass with no blocker found.

1. A green build is not a reviewed change

What went wrong. Devin merged six PRs in 30 seconds with CI green. Every one had a flaw CI could not see.

What we did. Repaired them, and wrote the playbook: list every reader, writer and render path; prove each test fails without the fix; script every data change; say what you could not verify.

What we learnt. CI and the author's tests check what the author thought of. The defects were in what it did not.

2. Devin reviews others well, not itself

What went wrong. Devin caught every planted defect in code it had not written, then shipped a blocker in its own, on a machine without browser test tools.

What we did. Fixed the machine, then made review someone else's job: a reviewer child with fresh context and a brief to find fault.

What we learnt. Instructions do not create scepticism about your own work; a separate session does. Check the environment before blaming the model.

3. Stay a grader

What went wrong. On Sunday morning Claude began lining up Devin's remaining tasks for itself.

What we did. I stopped it: "don’t forget you are grading Devin’s work. So.. Devin should be the one doing the tasks not u". A coordination board in the company brain replaced my hand-relaying, with one rule: a message from Devin is a request, never an approval.

What we learnt. If the grader finishes the job, you never learn what the agent cannot do.

4. Decide who merges, and how fast

What went wrong. Mine. The merge rule took five positions in four days. I merged two PRs unreviewed, and they carried a blocker. I made the sales lead a merger against Claude's advice, without thinking about speed; they merged four PRs within minutes, which broke no rule.

What we did. Moved review before the PR exists. No paid plan, no Claude in the loop. "this is all part of the learning process", I wrote at 17:05.

What we learnt. A human merge step is only a review if that human reviews.

5. Fix the cause of a recurring bypass

What went wrong. For weeks I approved the guard bypass push by push. Old failures had reached the development branch through merges on GitHub, where the guards never run, so every push was blocked. Only 3 of the 47 blocking checks also ran in CI.

What we did. Cleared the old failures, made the guards block only new problems, and required a written reason for any bypass.

What we learnt. Every approved bypass teaches every agent that bypassing is normal.

6. Read the live setting, not the one you remember

What went wrong. I assumed Devin ran on Fusion, because almost all my own Devin sessions did. The automation form said "Org default (Normal)" at 00:57 on Monday, and Claude read it twice more that day without flagging it. Claude also reported prompt v3 saved when only part of it was.

What we did. SWE-2 is not on the automation's own menu, so I set it as the organisation default and the API default, which reviewer children use. Every prompt save is now reloaded and compared with the file.

What we learnt. Before reasoning about cost or behaviour, read the value on the page that sets it. "this is a learnable mistake.", I wrote at 23:53.

7. Every restart costs a full read

What went wrong. Each reply starts a fresh session that re-reads everything and re-runs the baseline tests. One issue took four sessions. Claude twice posted new instructions on issues already in progress, restarting each one.

What we did. v5 added size tiers, a resume block and a leaner reviewer, and we stopped posting instructions mid-flight.

What we learnt. When every event starts a session, a human comment is the expensive unit. Batch them.

8. A strict reviewer creates a second queue

What went wrong. Two failed review rounds sent an issue to me. By midnight almost everything was waiting for me, mostly over test strength. One change failed only on its PR wording.

What we did. In v5.1, defects, process gaps and test gaps on a STANDARD change still mean FIX FIRST. Wording problems, and test gaps on a SMALL change, mean PASS WITH GAPS: fix what is cheap, list the rest.

What we learnt. Block on what changes users or stored data; list the rest.

9. The work was better than the report

What went wrong. PR descriptions claimed results a re-run did not reproduce. A handover said "browser-tested" about a local build.

What we did. The reviewer marks every claim TRUE, FALSE or UNVERIFIED with evidence, and a false claim is fixed before the PR opens.

What we learnt. Read the claims before the diff.

10. The issue is the record; sessions are disposable

What went wrong. By Monday night 68 sessions sat sleeping, and an old session wakes on a PR comment, outside the issue flow. My question at midnight: "but how do we know when it's DONE?"

What we did. GitHub already knows: a closed issue with a DONE comment. In v5.1 every session archives its reviewers and then itself. Archiving never touches a PR.

What we learnt. Keep state where people already look.

Who made which mistake

The flow

1. The sales lead opens an issue: screen, document number, screenshot. Nothing to tag.
2. A Devin session claims it, or ends quietly if the issue is not for it.
3. It asks, if it must: one headed comment, then it ends. A reply starts a fresh session.
4. It fixes, sized SMALL or STANDARD, and breaks its own fix on purpose to test the tests.
5. A reviewer child checks it before any PR. PASS or PASS WITH GAPS opens the PR. FIX FIRST means another round; after two, NEEDS JENSEN.
6. A human merges and comments merged. Devin checks the change on the development site and posts test steps.
7. The sales lead tests. works closes the issue with DONE, and Devin records the fix in the company brain. Each session ends by archiving its reviewers and itself.

What the sales lead does

What Devin posts on the issue

The reviewer

• It gets only the issue, the branches, the draft PR description and the handover page, never the builder's conclusions.
• It breaks the code in its own small ways and records which breakages the tests catch.
• It marks every claim TRUE, FALSE or UNVERIFIED, and checks tenant isolation and every place that reads, writes or prints the data.
• It never pushes, comments, merges or writes to a database.

Guards and settings

• Triggers: any new issue in the parent repository, and comments from the sales lead or me. Devin's and the support agent's comments do not wake it, so bots cannot loop.
• One Devin per issue, held by a working label, and at most eight sessions at once.
• Budgets: about 15 ACUs for a SMALL change and 30 for a STANDARD one.
• Mode: SWE-2 High, the organisation default since 5 October.
• Limits: development branches and the development database only, with backups and reverse scripts; never UAT or production. devin stop keeps Devin off one issue; disabling the automation stops everything.

• Whether SWE-2 stops the quota climbing from 86%, and whether it changes how often reviews pass.
• Which model Devin's Normal mode actually ran.
• Whether a Devin reviewer matches Claude Code on Opus 5.5. One blind review is all we have.
• The first v5.1 runs: self-archiving, PASS WITH GAPS, and the check on the development site.
• The stop command, a reply arriving mid-session, and an issue filed by the support agent: none tested end to end.
• Cost per issue. We only measured the weekly percentage.

Early on Saturday Devin merged six PRs in 30 seconds. Now it takes an issue from a sales lead, asks in plain English, fixes, has a second Devin review the change, and waits for a human. It still sends too much back to me, and it spent most of a week's quota showing me why. Both are fixable, and both are mine to fix. If you are building something similar, I would love to compare notes.