Before I built this, “what’s going on?” took me forty minutes to answer. Two projects, two Jira boards, two forges (GitHub for one, Bitbucket Cloud for the other), a dev machine to SSH into, and Slack. I had no single view of it, no alerting, and no record of what the AI tooling had done overnight.
If you work across more than one project you have some version of this. Here’s the shape I built to make it one message, and the eight decisions you’ll hit building your own.
Now one Slack message shows up every weekday morning. A header per project reads FREIGHT · 1 to review · 3 tickets, then the sprint tickets, then one card per pull request with a verdict from 0 to 10, a one-line take and, unless the build is red, an Approve button. I tap it from my phone and six seconds later the card repaints to approved freight-ws#412 · by @matthew at 8:04. The review landed on GitHub from the dev machine that was already logged in. That’s the one interruption I built in on purpose.
It’s less than it sounds. All it is is a Java server on a cheap VPS that SSHes into a dev machine and runs bash scripts. Data pipelines and old-fashioned ETL, with Claude filling in the wishy-washy parts between deterministic steps. Basically cron with a human attached. One Spring Boot JVM on a EUR 4 VPS: 5.7k lines of Java, 13 bash scripts and 160 tests, written in the week after the product requirements doc (PRD). No free-running agent loop anywhere. The personas get a bounded tool loop, eight iterations and a fixed tool list, and that is the only one. The tests are all on the JVM. The box scripts have none. I didn’t really care. They’re so bespoke it will only ever be me running them.
The rest is eight decisions you’d hit too, each with what it cost me. The code is private tooling, so the images are mock-ups with two made-up projects instead of screenshots: Freight, a search migration on GitHub, and Ledger, a legacy banking migration on Bitbucket Cloud.
Decision 1: Where Do the Credentials Live?
The orchestrator has to read Jira, list and approve PRs on two forges, and launch Claude Code runs. The default answer is a vault on the VPS with a token for each service. That means minting or registering a new credential per service and putting all of them on one internet-facing machine. There’s a simpler answer if you already have a dev machine that’s logged in to everything: a Mac on your desk, or a dev VM. Run the scripts there.
So the scripts run where the tokens already are. Every Jira, GitHub, Bitbucket Cloud and Claude Code call runs on the dev machine (the box, from here on), mostly through ~/tasks/*.sh scripts and otherwise as one-line gh or claude -p commands, reading tokens from the box’s own login environment or local files. The VPS reaches the box over a Tailscale tailnet by SSH and holds only its own secrets: Slack tokens, the Anthropic API key, SSH private keys. Nothing had to be duplicated or re-issued.
The PRD added one constraint I’d keep in any version of this: no orchestrator service on the box, just a ~/tasks/ directory of bash scripts. That bounds what the model can reach. A persona can only call the scripts it was given, and a script can only do so much. It can’t go nuts.
The SSH key is a different matter. The VPS holds a private key that opens a shell on the box, so whoever owns the VPS owns the box. Slack is not that path. A Slack message can only stage a row, and a tap from my user id runs one fixed, validated command. The VPS is the thing to guard. Mine sits on a private tailnet with no public port. The sensible next steps are a dedicated SSH user on the box, a command= wrapper in authorized_keys that only dispatches into ~/tasks/, and tailnet ACLs so only the VPS can reach the box’s port 22. Mine has the private network and none of the other three yet. That is the consequence I accepted for not minting a token per service.
Each project is one directory in git:
# projects/freight/project.yml
boxHost: 203.0.113.10 # the box's tailnet address (example)
sshUser: matthew
jiraProjectKey: FRT
githubOrg: freight-platform # or bitbucketWorkspace: ledger-platform
slackChannel: C0EXAMPLE
watchedRepos: [freight-dispatch, freight-billing]What it cost. The box’s login shell is fish, so every remote command gets wrapped in bash -lc, and anything arbitrary (Claude prompts, PR titles) crosses as base64. A Mac also ships bash 3.2 with no GNU timeout, so long commands run under perl -e 'alarm shift; exec @ARGV' and a timeout comes back as exit 142.
Decision 2: What State Do You Keep, and Where?
The default answer is managed Postgres, JPA and a migration tool. One person uses this and the database will never pass a few thousand rows, so that would be three more things to run and patch for no gain. I went the other way: embedded SQLite through Spring’s JdbcClient, no JPA, no migration tool. schema.sql runs on every boot with CREATE TABLE IF NOT EXISTS, and Hikari is pinned to maximum-pool-size: 1. One pooled writer means your own code never sees SQLITE_BUSY, and the nightly backup goes through SQLite’s online backup API, so it doesn’t compete either.
Configuration lives in git, runtime state lives in the DB, and the box only stores files you can regenerate: run logs, exit and result files, the per-session resume markers, the day’s PR verdicts and Claude Code’s own session files.
Concurrency between the poller, the approval executor and the stale reaper is one idiom. Every state transition is a guarded UPDATE ... WHERE state = ?, and the caller posts to Slack only if the rowcount came back as 1. One lock lives in the schema itself:
CREATE UNIQUE INDEX IF NOT EXISTS idx_runs_one_active_per_thread
ON runs(slack_thread_ts)
WHERE slack_thread_ts IS NOT NULL
AND state IN ('QUEUED','RUNNING','WAITING_BOX','WAITING_PROVIDER');The same store is the audit trail the PRD asked for. An aspect around every @ExecutionLogged and @Scheduled method writes trigger, project, args, duration and outcome into execution_log. A nightly job prunes it and takes a WAL-safe .backup.
What it cost. The unique index and the stranded-run pass weren’t in the spec. An adversarial review before the first deploy found the check-then-insert race on thread follow-ups and the restart-during-launch window.
Decision 3: When Do You Actually Call an LLM?
Deterministic code owns the control flow here. The model runs at fixed transform points inside the sweeps, and the scheduled pipeline has exactly two of them.
When the brief runs, MorningBriefJob sweeps each project’s sprint, PRs, watched repos and stored verdicts into typed records. That fact list goes once to LlmService.briefTakes, which returns one JSON object: a headline of at most 20 words and a take of at most 14 words per id. Block Kit is then rendered from script data with no model in the loop.
The second call runs on the box, against a Claude Max subscription instead of the API key. Half an hour before the brief, pr-verdicts.sh fetches each PR’s diff and runs one claude -p --max-turns 1 --output-format text per PR:
You are reviewing pull request $REF for Matthew, a senior engineer who is a requested reviewer. Decide whether he should approve it AS-IS.
Score 0-10 (10 = trivially safe to approve).
verdict: APPROVE if score>=7, LOOK if 4-6 (needs a real read), HOLD if <=3.
reason: max 22 words, concrete.freight-ws#412 from priya.n comes back 7/10 approve, “Pipeline-only change, self-validating: the modified pipeline passed”. freight-ws#409 from marco.r gets 4/10 look first, “Title says comment cleanup but the diff adds a DB migration and new status logic”. The metric is a plain question: would I approve this PR? The answer is text on the card. It never triggers anything on its own.
The third place the model shows up is the chief channel, the one Slack channel the orchestrator listens to, and there it can only stage a change. Prefix a message with a project id (freight: what's blocking FRT-2114?) and that project’s persona answers: an Anthropic API chat with read tools and a few write tools inside a bounded tool loop. It reads the ticket and the project’s open PRs and answers in the thread. Anything without a prefix goes to the cross-project chief persona, which has read tools only.
When the model is down the brief still goes out with the same facts, just without the takes. Past a circuit breaker and three attempts, briefTakes returns null and titles stand in. An unparseable verdict writes no line instead of a fake hold.
What it cost. Takes once fell back to raw PR descriptions because the model’s JSON got truncated, hence a 6000-token budget and a parser that salvages every "key": "value" pair from a partial reply.
Decision 4: How Does a Mutation Get Approved?
Four ways to let a chat assistant change things: read-only, execute and audit-log, let the model ask “shall I?” and parse the reply, or split staging from execution so any job or persona can only stage a row and a human tap runs it. I took the fourth.
The staged row lives in prepared_actions: a UUID, project_id, an action_type such as transition_issue or approve_pr, a payload JSON and a status of PENDING, APPROVED, EXECUTED, CANCELLED or EXPIRED. Every transition is the compare-and-set idiom from Decision 2, UPDATE ... WHERE id = :id AND status = 'PENDING', so a double-click or a Socket Mode redelivery produces exactly one winner.
The button value carries only the UUID. approveAndExecute re-reads the row, re-validates the payload, executes, then marks EXECUTED. Validation runs at prepare time and again at execute time: issue keys must match [A-Z][A-Z0-9]+-\d+, branches are normalized to a fixpoint so refs/heads/refs/heads/main can’t slip past, and main, master and develop are refused outright. Persona-prepared actions show the exact command that will run, so you approve what actually executes. Ask the Ledger persona to move LGR-88 to Done because Lee Park confirmed the fix, and what you get is a card with the transition command and a button. The brief cards skip the preview because the command is always the same one-line approve. Only the configured operator’s clicks count. Anyone else gets an ephemeral refusal.
None of this depends on the prompt behaving. Write tools only create PENDING rows, every tool description starts “PREPARES (does not execute)”, and the definitions only exist for project personas inside a thread. Nothing changes on the box unless I tap.
What it cost. Two things. The default TTL is 60 minutes, but approve_pr got 24 hours from day one, because the brief posts in the morning and you might not tap Approve until the evening. And a crash mid-execute leaves a row stuck in APPROVED with no automatic retry. You have to ask for a fresh action.
Decision 5: How Do You Survive Slack’s Limits?
Slack turned out to be three decisions.
Events endpoint or Socket Mode? The PRD rules out public internet exposure, so Socket Mode.
Post directly or through an outbox? Outbox. SlackNotifier.send only inserts a slack_outbox row. A scheduler flushes up to 10 rows every 10 seconds and parks a row as FAILED after 20 attempts.
Prose or Block Kit? Block Kit, because Slack’s limits are tight enough that you have to budget for them. Every handler has to ack within 3 seconds, so the work runs on a virtual thread. A message holds 50 blocks. I budget 48. If the full layout would overflow, every card collapses to one section (context folded in, dividers dropped) plus its button, and anything still past the budget is dropped with a footer note.
The first brief was a wall of text, and one card per PR won out of four candidates. Then the approve click. Slack’s button spinner dies at the ack, and the box-side approve takes seconds, so the message itself has to be the progress indicator.
Before each chat.update the current blocks are re-read under a per-message lock, so two quick approvals compose instead of overwriting each other. A failed update often arrives as HTTP 200 with ok:false. That falls back to an outbox thread reply.
What it cost. Three versions of the approve click in one day.
Decision 6: How Do You Run a Two-Hour Job on Another Machine?
A headless Claude Code run can take two hours on a box you only reach over the tailnet. Hold the SSH session open and one dropped link kills the run. You could use nohup and a pid file instead, but pids get recycled. So: a detached tmux session plus an exit file the server polls, with one more split. The server picks the prompt. The script on the box decides whether to start fresh or resume.
The launch is one SSH command that removes any stale .exit file and starts a detached tmux session running ~/tasks/claude-run.sh. The script uses set -uo pipefail and deliberately not -e, because it has to keep running after claude fails to capture the log tail. The EXIT trap records the real exit code atomically:
finish() { c=$?; printf '%d\n' "$c" > "$EXITF.tmp" && mv -f "$EXITF.tmp" "$EXITF"; } # atomic: readers never see a partial file
trap finish EXIT; trap 'exit 143' TERM; trap 'exit 130' INTThe script also owns the resume decision. If ~/runs/.session-<sessionId>.started exists it runs claude -p --resume, otherwise it touches the marker before --session-id. Partial progress survives a provider-limit retry.
The poller probes every 60 seconds with one SSH command:
exit file present -> EXIT:<code> + log tail
tmux has-session -> ALIVE
neither -> sleep 1, re-check the exit file
(the run can finish between the two checks)
still nothing -> GONETriage of a non-zero exit is first-match-wins:
| Signal | Category |
|---|---|
| exit 142 | TIMEOUT |
| exit 143, 137 or 130 | KILLED |
| auth marker in the log tail | AUTH |
| provider marker in the log tail | PROVIDER |
| anything else | TASK_FAILURE |
PROVIDER parks with backoff min(30, 2^retryCount) minutes. AUTH fails and alerts the chief channel.
Every cross-machine step has to be idempotent, and decisions that need box-local facts, like fresh versus resume or the real exit code, get made on the box.
What it cost. Two idempotency rules. The spec already treated a duplicate session reply from tmux as success, since an SSH reply can get lost after the launch has landed. Review then pointed out that a restart during the launch’s serial SSH window would strand a QUEUED row forever, so there’s a relaunch pass for QUEUED rows older than five minutes.
Decision 7: What Happens When Something Is Down?
Four things can be down: the box, a forge or Jira API, the model, and Slack. The rule is the same for all four. When one of them is down, the work parks until it comes back. Nothing fails loudly, and nobody gets paged for a problem that will fix itself.
The box. Before each launch the server runs the pre-flight ~/tasks/env-check.sh on the box (tools installed, gh auth status green, the Jira token present). If the box is unreachable or the check fails, the run parks as WAITING_BOX with backoff. The poller leaves RUNNING runs alone while the box is unreachable, because the detached tmux session keeps working without the server. A probe reaches every box over SSH every 5 minutes and writes the result to a dependency status table, one row per box. On DOWN to UP it requeues every parked run. On UP to DOWN, if runs are waiting, it prepares one approval whose button reads “Retry queued runs”.
A circuit breaker, and deliberately no retry. One resilience4j breaker per box counts only BoxUnreachableException. A command that ran and failed does not trip it. There is no retry layer, because every caller already re-runs on its own schedule (the poller every 60 seconds, the status probe every 5 minutes) and the idempotency from Decision 6 makes those re-runs safe.
The forge and Jira. In the brief each per-project sweep is caught separately and becomes a one-line :warning: in that project’s section. A GitHub outage never blanks the Ledger section. The approve call itself is not retried. If the forge returns an error the card says so, and you ask for a fresh action.
The model. Decision 3 already covered it: a breaker, three attempts, then titles stand in for takes and a missing verdict writes no line.
Slack. The outbox from Decision 5 covers this. During a Slack outage rows pile up and flush when it returns, and Socket Mode reconnects on its own.
Silence is the default. Probes post only on a transition to DOWN, and a long-parked run gets a single :zzz: note. The only :rotating_light: alerts are a box going DOWN, a failed pre-flight env-check, a hard timeout and a box auth error. After each brief the job GETs a deadman URL so an external check notices when the pipeline itself stops.
What it cost. Nothing here recovers a box that needs a human. When gh auth status fails on the machine, the run parks as AUTH and the fix is you at the keyboard. That was the PRD’s own target: zero run failures caused by a dependency being down, parked instead.
Decision 8: What Goes on the Card?
Version one shipped repo hygiene metrics. I cut those. The brief now only shows what needs a decision, ordered by what happens next: the PRs and sprint work you have to act on.
Lead lines come from rules, and the model doesn’t write them. Your own PR leads with its review state or mine · N comments to address. A PR waiting on you leads with CI failing, the verdict badge, or no verdict yet. Only PRs that need your review and pass the not-red CI gate get the one-tap Approve, capped at 10 per project. Ledger’s statement-batch#60 from jonas.k gets “Approve statement-batch#60”. Sam Okafor’s ledger-ui#1602 with a red build gets none.
Every card is a pipe-delimited row from a box script:
REVIEW|freight-platform/freight-ws|412|Split compile and test stages in the pipeline|priya.n|REVIEW_REQUIRED|green|...GitHub and Bitbucket Cloud need different handling for both PR discovery and the approve call:
| Step | GitHub | Bitbucket Cloud |
|---|---|---|
| PR discovery | one gh search prs per role | no workspace-level PR list, so: sweep the repos pushed to in the last 21 days, plus a cache of every repo that ever had a PR for you |
| Approve call | gh pr review --approve | REST call; 200 is approved, 409 means already approved |
Watched repos from project.yml and comment counts on your own PRs finish the card.
What it cost. A 1000-repo Bitbucket workspace with no workspace-level PR list gets a bounded sweep instead of a query. So a PR in a repo that’s been idle three weeks and never showed up in an earlier sweep is invisible until someone touches the repo.
Build It Yourself
You can build this without my code. The eight decisions above and an agent that asks you the right questions before it writes anything are enough. This is the prompt I’d hand Claude Code to start from zero. It interviews you first, one question at a time, then builds in the same order I did. Paste it into a fresh repo and answer honestly. “I don’t know yet” is a fine answer; the prompt tells the agent to pick the boring default and move on.
You are building me a "chief of staff" orchestrator: one always-on service that
posts a morning brief to Slack (sprint tickets, open PRs with a 0-10 approve
verdict, watched repos, comments I still owe), lets me approve a PR from a
button, and runs long headless coding jobs on a dev machine I already own.
Ground rules, non-negotiable:
- Scripted first. Deterministic code owns control flow. The model is called only
at fixed transform points: one-line "takes" as JSON, and a per-PR verdict from
the diff. If the model fails, titles stand in and nothing else changes.
- Nothing mutates without my tap. Every write (approve, transition, push) is a
PENDING row that a button in Slack claims with a compare-and-set update. The
button value carries only that row's id. Only my Slack user id may approve.
- Tokens stay where they already are. The orchestrator holds its own Slack and
model API keys and nothing else. Every git, Jira and Claude Code call runs as
a small script on the dev machine over SSH, and only text comes back.
- One Slack channel. Use Block Kit for every message. Respect the limits: ack a button
within 3 seconds and repaint the card in place, 50 blocks per message, 3000
characters per section, 75 per button label.
- When something is down the brief loses detail but never posts wrong data: an outbox table for Slack, circuit
breakers per dependency, alerts only on a transition, a deadman ping after
each brief.
- Log every command and result with duration and outcome. Tests on the server
side for every state transition.
Before writing any code, interview me. Ask ONE question at a time, wait for the
answer, and keep a running summary. Questions:
1. Which machine already has working `gh`, Bitbucket or Jira credentials and
Claude Code logged in? How do you reach it (Tailscale, LAN, SSH key)? What
is its login shell?
2. For each project: the forge (GitHub org or Bitbucket workspace), the Jira
project key, and which repos matter. One project is fine.
3. Slack: workspace, whether you can create an app with Socket Mode, and the
channel it should own. Your Slack user id.
4. When should the brief land, in which timezone, on which days? Should PR
scoring run before it?
5. What counts as "needs my review": requested reviews only, or every open PR
in the watched repos?
6. What must never happen without a tap? List the actions. Anything you'd let
run unattended?
7. Do you need long-running coding jobs on that machine (yes/no)? If yes, how
long may one run before it is killed?
8. Stack preference for the server. Default if none: Java 21, Spring Boot,
SQLite in WAL mode, one small VPS.
9. Which of my answers above are you unsure about? For those, take the boring
default and note it in a DECISIONS.md.
Then produce, in this order, stopping for my review after each:
M1 Skeleton: config per project, SSH executor that wraps every command in a
login shell, SQLite schema, execution log, Slack outbox and poster, health
endpoint. A first brief with raw text.
M2 Sweeps and the brief: sprint tickets with a one-line take, PRs with author,
review state and CI, watched repos, unresolved comments on my PRs. Block Kit
cards sorted by what I do next. Compact fallback when 50 blocks would
overflow.
M3 Approvals: prepared_actions table, Approve button per card, two-step
repaint (button becomes "approving" at once, then the outcome), 24 hour TTL
for PR approvals, operator gating, ephemeral refusal for anyone else.
M4 Verdicts: a box script that scores each PR diff 0-10 with a one-line
reason using the local coding agent, results stored on the box as JSON
lines, dropped after 24 hours, shown on the card as advisory text only.
M5 Runs (only if I said yes in 7): detached tmux, exit file written atomically,
resume decided on the box by a marker file, poller with a hard timeout,
exit triage into timeout, killed, auth, provider and task failure.
For every milestone list what you built, what you skipped, and how I test it
from Slack on my phone. Never invent a repo, ticket, person or credential. If
a step needs access I don't have, say so and stop.Two things to expect. The agent will want to skip the interview and start coding; don’t let it, the answers to questions 1, 3 and 6 decide the architecture. And it will propose a cloud agent product at some point. Say no, because the tokens and the work have to stay on a machine you already control.
When to Get Help
If you have one project, one forge, and a dev machine you already control, build this yourself. The parts worth copying port to any language: the outbox table, prepared actions with compare-and-set claims, scripts that run where the tokens already are, tmux plus an exit file with box-decided resume, and LLM calls only at fixed transform points with deterministic fallbacks. I used Java 21 and Spring Boot because that’s what we use at Katyella.
The hard case is more than one forge (GitHub and Bitbucket Cloud already needed two sweep scripts), a box that’s often unreachable mid-run, a compliance requirement that every mutating action trace back to an explicit approval event, or a team instead of one person, where one Slack user id gate has to become per-person isolation.
We do this work regularly at Katyella. If you want agentic tooling in front of real systems without giving it your credentials or letting it change anything on its own, reach out and we can walk through your situation.
Related Articles
- AI Code Review for Java, the review-first habit the verdict cards automate
- Multi-Agent AI Development Workflows, where a bounded agent loop does earn its place, and the free-running style this build avoided
- AI-Assisted Java Development with Claude Code, the headless
--max-turnsdiscipline the box scripts rely on - Spring Boot Consulting, if you want this shape of system built around your own stack
Java Modernization Readiness Assessment
15 questions your team should answer before starting a migration. Takes 10 minutes. Could save you months.