Manual issue triage is a black hole for engineering time. Someone opens a vague issue titled “it’s broken,” and now a human has to read it, figure out if it’s a bug or a feature request, slap on the right labels, decide if it’s urgent, and assign it to someone. Multiply that by 30 issues a week, and you’ve got a part-time job nobody signed up for.
An LLM agent can do this in seconds. It reads the issue title and body, classifies it into a type (bug, feature, question, docs), assigns a priority level, suggests labels, and routes it to the right person – all through the GitHub API. Here’s how to build one.
Setting Up the GitHub API Client#
You need two packages: PyGithub for the GitHub API and openai for the LLM.
1
| pip install PyGithub openai pydantic
|
First, set up the GitHub client and fetch issues that haven’t been labeled yet. This is your triage queue – any issue with zero labels is a candidate.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
| import os
from github import Github
GITHUB_TOKEN = os.environ["GITHUB_TOKEN"]
REPO_NAME = "your-org/your-repo" # e.g., "acme/backend"
g = Github(GITHUB_TOKEN)
repo = g.get_repo(REPO_NAME)
def get_unlabeled_issues(limit: int = 20):
"""Fetch open issues that have no labels assigned."""
unlabeled = []
for issue in repo.get_issues(state="open", sort="created", direction="desc"):
if issue.pull_request is not None:
continue # skip PRs, they show up in the issues endpoint
if len(list(issue.labels)) == 0:
unlabeled.append(issue)
if len(unlabeled) >= limit:
break
return unlabeled
issues = get_unlabeled_issues()
print(f"Found {len(issues)} unlabeled issues to triage")
|
The get_issues endpoint returns pull requests too, so the pull_request check filters those out. We limit to 20 per run to stay within rate limits.
Building the Classification Pipeline#
This is where the LLM does the heavy lifting. Define a Pydantic model for the structured output – the LLM must return an issue type, priority, suggested labels, and an optional team assignment.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
| from pydantic import BaseModel, Field
from openai import OpenAI
client = OpenAI()
class TriageResult(BaseModel):
issue_type: str = Field(
description="One of: bug, feature, question, docs"
)
priority: str = Field(
description="One of: P0 (critical), P1 (high), P2 (medium), P3 (low)"
)
labels: list[str] = Field(
description="Suggested labels, e.g. ['bug', 'P1', 'auth', 'backend']"
)
assigned_team: str = Field(
description="One of: backend, frontend, infra, docs, security"
)
reasoning: str = Field(
description="One sentence explaining the classification"
)
SYSTEM_PROMPT = """You are a senior engineering lead triaging GitHub issues.
Given an issue title and body, classify it and return structured JSON.
Rules:
- issue_type must be one of: bug, feature, question, docs
- priority: P0 = production outage/data loss, P1 = broken feature, P2 = degraded but workarounds exist, P3 = nice to have
- labels: include the issue_type, priority, and 1-3 topic labels (e.g., "auth", "api", "ui", "database", "performance")
- assigned_team: pick the most relevant team based on the issue content
- Be conservative with P0/P1 -- most issues are P2 or P3"""
def classify_issue(title: str, body: str) -> TriageResult:
"""Send issue to LLM for classification using structured outputs."""
issue_text = f"Title: {title}\n\nBody:\n{body or '(no body provided)'}"
completion = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": issue_text},
],
response_format=TriageResult,
temperature=0,
)
return completion.choices[0].message.parsed
|
The client.beta.chat.completions.parse() method handles structured outputs natively – it constrains the LLM to return valid JSON matching your Pydantic model. No regex parsing, no retry loops. The temperature=0 keeps classifications deterministic so the same issue always gets the same triage result.
Applying Labels and Assignments#
Now wire the classification back into GitHub. The agent creates missing labels, applies them to the issue, assigns a team member, and leaves a triage comment so humans can see what happened.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
| # Map teams to GitHub usernames for assignment
TEAM_ROSTER = {
"backend": ["alice-dev", "bob-eng"],
"frontend": ["charlie-ui", "diana-fe"],
"infra": ["eve-ops"],
"docs": ["frank-docs"],
"security": ["grace-sec"],
}
# Track round-robin index per team
assignment_index: dict[str, int] = {}
def ensure_label_exists(repo, label_name: str):
"""Create a label if it doesn't already exist."""
try:
repo.get_label(label_name)
except Exception:
# Label colors by category
colors = {
"bug": "d73a4a", "feature": "0075ca", "question": "d876e3",
"docs": "0075ca", "P0": "b60205", "P1": "d93f0b",
"P2": "fbca04", "P3": "0e8a16",
}
color = colors.get(label_name, "ededed")
repo.create_label(name=label_name, color=color)
def assign_from_team(team_name: str) -> str:
"""Round-robin assignment within a team."""
members = TEAM_ROSTER.get(team_name, TEAM_ROSTER["backend"])
idx = assignment_index.get(team_name, 0)
assignee = members[idx % len(members)]
assignment_index[team_name] = idx + 1
return assignee
def apply_triage(issue, result: TriageResult):
"""Apply labels, assignment, and comment to a GitHub issue."""
# Ensure all labels exist before applying
for label_name in result.labels:
ensure_label_exists(repo, label_name)
issue.set_labels(*result.labels)
# Assign to a team member
assignee = assign_from_team(result.assigned_team)
issue.add_to_assignees(assignee)
# Post a triage comment
comment_body = (
f"**Auto-triage result**\n\n"
f"- **Type:** {result.issue_type}\n"
f"- **Priority:** {result.priority}\n"
f"- **Team:** {result.assigned_team}\n"
f"- **Assignee:** @{assignee}\n\n"
f"**Reasoning:** {result.reasoning}\n\n"
f"_If this classification is wrong, update the labels manually._"
)
issue.create_comment(comment_body)
print(f" Triaged #{issue.number}: {result.issue_type} / {result.priority} -> @{assignee}")
|
The ensure_label_exists function handles the cold-start problem where your repo might not have P0, P1, etc. as labels yet. It creates them with sensible colors on first use. Round-robin assignment distributes work evenly across team members instead of dumping everything on one person.
Running as a Scheduled Job#
Wrap the whole thing in a main function and run it on a cron schedule. Every 15 minutes is a reasonable interval – frequent enough that issues get triaged quickly, but not so aggressive that you burn through API rate limits.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
| import time
def triage_all():
"""Main loop: fetch unlabeled issues, classify, and apply triage."""
issues = get_unlabeled_issues(limit=20)
print(f"[{time.strftime('%H:%M:%S')}] Processing {len(issues)} unlabeled issues")
for issue in issues:
try:
result = classify_issue(issue.title, issue.body)
apply_triage(issue, result)
except Exception as e:
print(f" Error triaging #{issue.number}: {e}")
continue
print(f"Done. Triaged {len(issues)} issues.")
if __name__ == "__main__":
triage_all()
|
Schedule it with cron:
1
2
| # Run every 15 minutes
*/15 * * * * cd /opt/triage-agent && GITHUB_TOKEN=ghp_xxx OPENAI_API_KEY=sk-xxx python3 triage.py >> /var/log/triage.log 2>&1
|
For production use, a GitHub Actions workflow on a schedule trigger is cleaner than a raw cron job – it handles secrets properly and gives you logs in the Actions tab. But for a quick proof of concept, cron works fine.
Common Errors and Fixes#
github.GithubException: 403 - Resource not accessible by integration
Your token doesn’t have the right scopes. You need repo scope for private repos, or public_repo for public ones. If you’re using a GitHub App, make sure it has Issues read/write permissions.
github.GithubException: 403 - API rate limit exceeded
The GitHub API allows 5,000 requests per hour for authenticated users. Each issue triage takes roughly 3-4 API calls (get issue, set labels, assign, comment). If you’re processing more than ~1,200 issues per hour, you’ll need to add a rate limiter:
1
2
3
4
5
6
7
8
9
10
11
12
| import time
def triage_with_rate_limit(issues, delay: float = 1.0):
"""Process issues with a delay between each to avoid rate limits."""
for issue in issues:
try:
result = classify_issue(issue.title, issue.body)
apply_triage(issue, result)
time.sleep(delay)
except Exception as e:
print(f" Error triaging #{issue.number}: {e}")
continue
|
github.UnknownObjectException: 404 - Not Found when setting labels
This happens when you try to apply a label that doesn’t exist yet. The ensure_label_exists function above handles this, but if you skip it, you’ll get a 404. PyGithub doesn’t auto-create labels.
openai.BadRequestError: Invalid response_format
Make sure you’re using client.beta.chat.completions.parse() (not client.chat.completions.create()) when passing a Pydantic model as response_format. The parse method is the one that supports structured outputs with Pydantic directly.
Labels with special characters fail
GitHub label names can’t contain certain characters. Stick to alphanumeric characters, hyphens, and spaces. If your LLM generates labels with slashes or colons, sanitize them before calling set_labels.