AI Analyze Failed Pipelines and Notify

About 760 wordsAbout 3 min

When a pipeline fails, manually digging through logs is costly. This example demonstrates an automated chain of Fail → AI Summary → Group Notification:

When the normal stages fail, failStages is triggered
Within failStages: gather failure context into structured Markdown → call npc:go to let AI produce root cause + solution → read the Markdown and send to a WeCom group

Why Use AI Inside failStages

Compared to launching a separate fix pipeline or debugging manually, failStages + AI offers two natural advantages:

Preserves the failure scene: failStages shares the same workspace as stages. The AI has direct access to the actual failure scene — build artifacts, intermediate files, leftover dependencies, environment variables — all untouched and ready for inspection. No need to painstakingly reproduce the failure in a fix pipeline.
Zero wait, zero interaction: Failure → AI analysis → solution notification is fully automated. No re-triggering builds, no waiting for containers to start, no SSH login to debug. By the time the WeCom notification arrives, the report already contains the root cause and actionable steps — just follow and fix.

Full Example

.cnb.yml:

.cnb.yml
main:
  push:
    - stages:
        - name: test
          # Intentionally use echo1 (typo) to trigger failStages for demonstration
          script: echo1 1
      failStages:
        # 1) Gather failure context into structured Markdown for AI input
        - name: prepare-failure-context
          script: |
            {
              echo "## Failure Overview"
              echo ""
              echo "| Item | Detail |"
              echo "| --- | --- |"
              echo "| Failed Stage | $CNB_BUILD_FAILED_STAGE_NAME |"
              echo "| Failure Reason | \`$CNB_BUILD_FAILED_MSG\` |"
              echo "| Trigger Branch | $CNB_BRANCH |"
              echo "| Commit | $CNB_COMMIT_SHORT — $CNB_COMMIT_MESSAGE_TITLE |"
              echo "| Triggered By | $CNB_BUILD_USER |"
              echo "| Build ID | $CNB_BUILD_ID |"
              echo "| Build Logs | $CNB_BUILD_WEB_URL |"
              echo ""
              echo "## Raw Error Message"
              echo '```'
              echo "$CNB_BUILD_FAILED_MSG"
              echo '```'
            } > _fail_ci_.md
        # 2) Let AI read the context, produce root cause + solution, append to the same file
        - name: ai-analyze-failure
          type: npc:go
          options:
            systemPrompt: |
              You are a senior DevOps engineer, skilled at analyzing CI/CD pipeline failures.
              Read the failure context in _fail_ci_.md (failed stage, error message,
              build log link, commit info), then identify the root cause by considering
              common failure patterns (typos, missing dependencies, permission issues,
              path errors, network timeouts, disk full, configuration errors, etc.),
              and provide an actionable solution.
            userPrompt: |
              Read _fail_ci_.md and append the following sections as Markdown:
              ## Root Cause Analysis
              ## Solution
              ## Actionable Steps (commands or yaml diff)
              ## Impact Scope
        # 3) Send the AI-written Markdown file to a WeCom group
        - name: notify-wecom-group
          image: tencentcom/wecom-message
          imports: https://cnb.cool/<your-repo-slug>/-/blob/main/xxx/wework.yml
          settings:
            robot: ${WECOM_ROBOT}
            msgType: markdown_v2
            fromFile: _fail_ci_.md

WECOM_ROBOT is the WeCom group bot Webhook URL. The example pulls it via imports from a secret repo's wework.yml, injecting it through env without exposing the raw URL in .cnb.yml. Example secret repo file:

secret repo wework.yml
env:
  WECOM_ROBOT: https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxx

Real-world Effect

When a build fails, the WeCom group receives an AI-generated report containing:

Failure Overview + Raw Error — written by the prepare-failure-context stage, including the failed stage name, error message, build log link, commit info, etc.
Root Cause Analysis — the AI reasons from multiple dimensions (direct cause, root cause, trigger condition, common patterns) with supporting evidence
Solution + Actionable Steps — executable commands or yaml diffs you can follow immediately upon receiving the notification

The report combines the manual overview from the script stage with the AI-appended analysis. The notify-wecom-group stage reads it all at once via fromFile, sending a complete post-mortem in a single message.

Key Design Decisions

Split "gather context" into its own stage: Embedding variables directly into userPrompt makes shell escaping error-prone (quotes, newlines, backticks may truncate or inject the prompt). Writing context to a file first, then having npc:go read it, keeps the shell/prompt boundary clean
Append AI output to the same file: The three failStages run sequentially and share a workspace. The final notification stage reads the complete report via fromFile: _fail_ci_.md — overview + analysis combined
Use msgType: markdown_v2: The older markdown type struggles with nested tables and code blocks, often degrading to plain text. npc:go requires a Docker environment; it reuses the pipeline's docker.image if configured

Notes

If the notification stage itself errors, the build is still marked failed but you may not receive the message — dry-run first to verify connectivity
npc:go consumes LLM quota; for frequently-failing projects consider rate-limiting (e.g., only notify once per commit)
Build-scope variables like $CNB_BUILD_FAILED_MSG are only accessible within failStages