Skip to content

Issue acceptance check

Last updated

When a pull request links an issue, Argus asks an LLM to judge each of the issue's acceptance criteria against the diff. No manual setup: if you use GitHub's native issue-linking, Argus picks it up.

How to trigger

You have three options, any of which work:

  1. PR body text — write Closes #123, Fixes #123, or Resolves #123. Cross-repo: Closes owner/repo#123.
  2. GitHub UI — use the "Development" panel on the right side of the PR page to link an issue. No PR description change needed.
  3. Non-closing mention — Related to #123 or refs #123 also triggers the check (catches PRs that partially address issues).

Closing keywords and the Development panel come from GitHub's closingIssuesReferences GraphQL field. Non-closing mentions come from a regex over the PR body, which needs the keyword, a space, then #N or owner/repo#N. Refs: #123 or see #123 is not picked up.

How to structure issues

Argus extracts acceptance criteria from structured sections. Use one of these headings in your issue body:

  • ## Acceptance Criteria
  • ## Definition of Done
  • ## Expected Behavior
  • ## Steps to Reproduce

Under the heading, write specific criteria as a bulleted list or checklist. Be specific. Instead of "login works", write "POST /login returns 401 on invalid credentials". A specific criterion can be checked against the diff; a vague one tends to come back ambiguous.

If no structured section is found, Argus treats the issue body, cut to 500 characters, as one free-form criterion. Up to 15 criteria are read per issue.

Verdicts

Each criterion gets one of four labels:

  • addressed — the LLM judge found the diff clearly satisfies it (it is told to use this label only when it can cite file:line evidence; the citation is not shown in the comment)
  • partial — the diff addresses some aspects but not all
  • unaddressed — the diff doesn't touch it
  • ambiguous — the criterion is too vague or the diff's intent isn't clear

Cross-repo issues

If the issue lives in a different repo and Argus can't read it (usually because Argus isn't installed there), the issue is skipped and does not appear in the Issue Coverage section. The primary review still completes normally.

Disable

Go to Settings → Org defaults → Verification features to toggle the check off. It's on by default and costs one LLM call per linked issue, for up to 5 issues per review.