🍃 Tellia / 🥷🏿 Engineering

Appearance

Alert Triage (#alerts → Linear) ​

How we turn production alerts into tracked work. The review-issue Claude Code skill reads the #alerts Slack channel (Sentry + Grafana alerts), pulls the real error from the logs, dedups against Linear, and — on explicit approval — files or links an issue and closes the loop in Slack with a reaction + threaded link.

It runs inside Claude Code via MCP. This page is the setup; the full procedure lives in the skill itself (apps/docs/skills/review-issue/SKILL.md).

Prerequisites — connect 4 MCP servers ​

The process needs grafana, slack, linear, and sentry. The simplest path: ask Claude directly — "connect the grafana, slack, linear and sentry MCP servers" — and follow the auth flow.

ServerWhat to provide
grafanaURL https://tellia.grafana.net + a Viewer service-account token. Both are in 1Password ("Grafana MCP" / "Grafana service account").
slackAuthorize the connector. The account must be a member of #alerts (C0AQ0UZL0RM) and #dev (C09CL4Q7Y3D).
linearAuthorize the connector (workspace tell-ia).
sentryAuthorize the connector for org tellia-dl (tellia-dl.sentry.io).

Don't hardcode tokens in a file — let Claude wire the connections; only Grafana needs values pasted (from 1Password). Confirm all four with /mcp before triaging.

Install the skill ​

The skill is not auto-loaded — .claude/ is gitignored, so a fresh checkout has no review-issue skill. Its tracked source of truth lives at apps/docs/skills/review-issue/SKILL.md. Install it into your .claude/ — project-local (.claude/skills/review-issue/) or global (~/.claude/skills/review-issue/):

  • Ask Claude — "add the review-issue skill" — it copies the tracked source into .claude/.
  • Or copy it yourself: cp -R apps/docs/skills/review-issue .claude/skills/review-issue

Restart Claude Code and confirm /review-issue appears in the skill list. The full procedure lives in that SKILL.md; see the shared skills README for the complete list and install options.

How triage works ​

Run it by saying "check #alerts". For every alert the skill:

  1. Log-first (hard gate). Reads the real error before any disposition — never infers from the alert title or source code. This is non-negotiable; skipping it has filed real bugs late.
    • Backend (Grafana) alerts → Loki error-level sweep for the exact window: {service_name="tellia-agri-backend", level="error", deployment_environment="production"}, then Tempo for provider/worker/latency spans. (A loose |~ "<id>" filter that returns info lines is not the check.)
    • Frontend (Sentry) alerts → Sentry is the frontend log. Read occurrences, users-impacted, status (escalating?), first/last-seen. A high-volume / multi-user / still-firing FE error is actionable even when the backend is clean — client↔socket/transport and render bugs never touch the backend. "transient / browser / Failed-to-fetch" is a verdict you earn from the counts, never a default.
  2. Match alert → error by exact timestamp (alert fires ~1–3 min after the error; times are CEST).
  3. Assess ownership — our code · provider · browser/device · mislogged noise · rule misconfig.
  4. Dedup — search Linear and read the candidate's full content (get_issue) before calling it a duplicate.
  5. Present, then wait. Nothing is written until you give an explicit action word (do it / create / dup / skip). Flagging a missed alert is not approval.

Assignee is derived from .github/CODEOWNERS (last match wins) + TEAM.md, never guessed. Estimates use the Linear estimate field in points per PLANNING.md (1 pt = ¼ day).

Slack reaction key ​

ReactionMeaning
✅ white_check_markNew issue created (thread links it)
⛔ no_entryDuplicate of / covered by an existing linked issue
⏭ fast_forwardNot-actionable, no linked issue (browser/3rd-party/transient/rule noise)
(none)On hold — log not found yet; come back to it

The Slack MCP can't edit messages or remove reactions — get it right the first time.

Fixed facts ​

  • #alerts C0AQ0UZL0RM · #dev C09CL4Q7Y3D
  • Loki datasource UID grafanacloud-logs · Tempo grafanacloud-traces
  • Backend log label: service_name="tellia-agri-backend" · env label deployment_environment
  • #alerts carries several Grafana rules (error-log, OpenAI-error, BullMQ-failure, HTTP-5xx, HTTP-latency) — the Slack block is empty, so one failing job can trip several at once.

Weekly recap ​

When asked for the weekly report, the skill renders it in chat first (wait for approval), then saves to ~/Documents/WeeklyReport/triage-report-<start>_to_<end>.md and posts to #dev. It includes a per-issue fix-status (Linear state + merged-PR check) and a Completed-this-week section (dev-team velocity).

See also ​