IBM Bob kit · Theme 1: Explore, Fix, and Build

Know what's broken beforeyou fix it.

OnRamp is an IBM Bob kit for joining an unfamiliar codebase. It records which tests already fail, reproduces a bug with a failing test first, and verifies the fix without ever mistaking an old failure for a new one.

Run the classifier Read the kit on GitHub Pitch deck (PDF)

Built for AngelHack × IBM, “Building with IBM Bob”, Theme 1: explore an unfamiliar app, fix a real bug, add a feature. What did not work is listed in corrections and limitations.

What the evidence shows
Baseline1,761Healthchecks application at upstream 0968fa2; the kit recorded it at e10db36. 29 already fail on Windows (AF_UNIX sockets, a Linux-only strftime format, a DST switch)
After Bob's fix and feature0regressions in an independent full run of 1,767 tests on the same Windows machine, against the pinned baseline
Blind fix1line changed in hc/api/transports.py, the same line the maintainer changed later
Generated through IBM Bob23commits, each ending Generated-by: IBM Bob. The upstream test suite passed in recorded Linux CI run 38460743005; the 29 baseline failures are Windows-only.

Inherited failure or regression?

Pick a later test run of Healthchecks and OnRamp classifies it against the baseline, right here in your browser.

Baseline: Healthchecks at the pinned upstream commit, 1,761 tests, 29 failing on Windows. With focus in the demo, keys 1–4 pick a run.

Later run

Fig. 1, suite map

Live visual index of outcomes, not a dependency graph: every dot is one Healthchecks test, placed in test-id order and coloured by the selected run. Position carries no test meaning; the counts and test details remain authoritative.

Drag to rotate through 360°. Use the arrow controls or keyboard arrows for the same view changes. Pick a dot to inspect its test.

The 3D suite map needs WebGL. The counts on the left carry the same information.
  • unchanged
  • inherited
  • regression / new failing
  • fixed / new passing
  • not run
How this demo works

The runs are real Healthchecks test logs, except the two marked synthetic: those are classifier safety tests built from the real full run (one test flipped to failing; one log cut off), not production evidence, normalized to versioned JSON by the Python implementation (onramp_normalize.py). This page only classifies that JSON, and CI checks on every push that the browser and Python agree exactly. It is not a live IBM Bob session: the full kit runs inside Bob IDE.

The browser classifier and the Python implementation (py/onramp_normalize.py) give identical results on every golden case, checked by CI on each push: full run (29 inherited, 0 fixed, 0 regressions), targeted run (1,761 not run, 0 fixed), synthetic regression (1), crashed run and unsupported schema (both inconclusive). IBM Bob's own corrected classifier in the kit (gen_report.py, 2a527f8) agrees on the full, regression and crash cases; it reports the docstring-bearing targeted run as inconclusive, a known limitation listed in the corrections file.

Raw logs: baseline, full run, targeted run, normalized JSON, classifier.js. Local paths in the logs are replaced with <repo> and <home>.

29 inherited Windows failures, zero regressions

Before Bob changed a line, OnRamp recorded which tests already fail on this Windows machine, so red tests are never blamed on the fix. On Linux CI the same suite passes; the 29 are Windows-only.

Environment
Windows, Python 3.13.2
Tests in the baseline
1,761
Recorded at
kit commit e10db36; app code equals upstream 0968fa2
Scope of the 29
Windows only; the same suite passes in Linux CI
Error typeTestsWhy it fails here
AttributeError26Signal integration tests need socket.AF_UNIX, which Windows lacks
ValueError2A Linux-only strftime format (e.g. %-d)
AssertionError1DST test needs time-zone switching, unavailable on Windows

Fingerprint recorded at kit commit e10db36, whose application code (hc/, templates/, static/) is identical to upstream Healthchecks 0968fa2; Windows, Python 3.13.2. All 29 are Windows-only; on Linux CI the same suite passes.

The first baseline was captured before the kit's parser fix (8165552) and stored error types as "Unknown"; Bob re-classified a fresh run in baseline_reclassified.json. Its error types match Claude's independent run exactly.

Bob vs the maintainer, blind

Healthchecks was pinned one commit before its maintainer fixed a real webhook bug. The issue-specific brief was sealed before the fix work; after Bob's fix was committed, we revealed the maintainer's.

Same affected behaviourSame source lineDifferent accepted-status policy

Fig. 2, IBM Bob, commit 5f16187

@@ -158,7 +158,7 @@ class HttpTransport(Transport):
auth=auth,
timeout=30,
)
- if r.status_code not in (200, 201, 202, 204):
+ if r.status_code not in range(200, 300):
cls.raise_for_response(r)
except curl.CurlError as e:
raise TransportError(e.message)

Fig. 3, maintainer, upstream 0a9bff9, revealed afterwards

@@ -158,7 +158,7 @@ class HttpTransport(Transport):
auth=auth,
timeout=30,
)
- if r.status_code not in (200, 201, 202, 204):
+ if r.status_code not in (200, 201, 202, 204, 205):
cls.raise_for_response(r)
except curl.CurlError as e:
raise TransportError(e.message)
  1. Step 1 of 4: Brief sealed

    The symptom report was hashed before the issue-specific work began, so it could not be adjusted after the fix.

    21:05:38 UTC, SHA-256 12c482dc2578402a…
    benchmark_webhook-205.json"sealed_at": "2026-10-11T21:05:38.561200+00:00""brief_sha256": "12c482dc2578402a5c77c5ae60bca4f5…"
  2. Step 2 of 4: Failing test first

    Bob wrote a test that reproduces the bug and fails on the unfixed code.

    AssertionError: 3 != 1
    pre_fix_webhook-205.txtFAIL: test_webhooks_accept_http_205AssertionError: 3 != 1FAILED (failures=1)
  3. Step 3 of 4: One-line fix

    One line changed in hc/api/transports.py. The new test passes, and the later full run shows no regressions.

    committed 21:10 UTC
    git show 5f16187 -- hc/api/transports.py, post_fix_webhook-205.txt- if r.status_code not in (200, 201, 202, 204):+ if r.status_code not in range(200, 300):Ran 1 test in 0.007sOK
  4. Step 4 of 4: Reveal & compare

    Only then was the maintainer's own fix revealed: the same line, with a narrower accepted-status policy.

    Corrected comparison: the upstream patch adds only status 205.
    reference_upstream_0a9bff9.patch+ if r.status_code not in (200, 201, 202, 204, 205):# annotation: same line as Bob's fix; upstream adds only 205

Terminal panels are excerpts copied from the evidence files named in each panel (whitespace collapsed; the line starting with # is our annotation). They are not a live session.

Our brief said "a 2xx response such as 205" should succeed, and Bob accepted every 2xx; the maintainer added only 205, because HttpTransport is shared by about 25 integrations. Corrected evidence report, historical report (superseded: it counted 29 never-run tests as fixed).

Read the sealed brief and what the seal proves
While testing Healthchecks with a personal project, I set up a webhook integration whose endpoint answers with HTTP 205 (Reset Content). When a check changes state, my endpoint receives three identical requests, and the integration is marked as failed with "Received status code 205". I reproduced it locally: the integration's "Test" button also reports an error for the same endpoint, although it only sends one request. Expected: a 2xx response such as 205 means the notification was delivered; it should not be retried or reported as a failure.
The seal proves the brief's exact text and time. It was sealed before the issue-specific work began; Bob had mapped the codebase earlier. Isolation was enforced separately: a single-commit snapshot with no remote or upstream history, and a guard in Claude Code that blocked mixing the upstream clone into Bob's workspace. The seal's "allowed / withheld" fields were left empty at the time and are kept as recorded.

Readable JSON webhook bodies (#1090)

Feature #1090: JSON request bodies are shown indented, while Download Original still serves the exact bytes. The evidence below demonstrates the formatting and non-ASCII preservation behavior.

Before: the request body is one long line of JSON that has to be scrolled sideways
Before, upstream Healthchecks: one long line
After: the same request body pretty-printed over several indented lines, with café and a check mark intact
After, Bob's branch: indented, non-ASCII intact

Captured from local development servers with a throwaway test check. An independent check found json.dumps escaping "café"; Bob added a test and fixed it (e4032c2).

View the code change (786c034 and the follow-up e4032c2)
@@ -1,6 +1,7 @@
from __future__ import annotations
import email
+import json
import logging
import os
import re
@@ -758,10 +759,18 @@ def ping_details(
except Ping.GetBodyError:
body_bytes = None
+ body_str = body_bytes.decode(errors="replace") if body_bytes else None
+ if body_str and ping.scheme != "email":
+ try:
+ parsed_json = json.loads(body_str)
+ body_str = json.dumps(parsed_json, indent=2)
+ except ValueError:
+ pass
+
ctx = {
"check": check,
"ping": ping,
- "body": body_bytes.decode(errors="replace") if body_bytes else None,
+ "body": body_str,
"plain": None,
"html": None,
"active": None,
@@ -763,7 +763,7 @@ def ping_details(
if body_str and ping.scheme != "email":
try:
parsed_json = json.loads(body_str)
- body_str = json.dumps(parsed_json, indent=2)
+ body_str = json.dumps(parsed_json, indent=2, ensure_ascii=False)
except ValueError:
pass

Not hardcoded for Healthchecks

Profile-driven portability was tested across three TaskFlow runs, a Flask app using plain unittest. Each run's gaps went back to Bob, with public fixes in 4facc4b and 6be3235 (profile-driven kit), plus 8165552 (unittest parser).

  1. Run 1kit a3acbc3

    7 kinds of Django assumptions found

    Loaded unchanged; exposed 7 kinds of Django assumptions (hardcoded hc/, Django-only output parsing, baseline.txt vs baseline.json, Reproducer edit scope, Django-specific skill steps).

    4 tests · failure type recorded: Unknown

  2. Run 2kit e4032c2

    1 parser gap found

    After Bob made the kit profile-driven: worked, but recorded the unittest failure type as "Unknown".

    4 tests · failure type recorded: Unknown (the bug this run exposed)

  3. Run 3kit 8165552

    worked unchanged

    After Bob's parser fix: profile, baseline and map produced, failure classified as AssertionError.

    4 tests · failure type recorded: AssertionError

Who did what

IBM Bob

The original kit (modes, skills, commands, hook and gen_report.py), plus the Healthchecks fix and feature. Every commit ends with Generated-by: IBM Bob.

Claude Code

Operated Bob at the participant's direction and verified its work independently. After Bob's 180-Bobcoin allowance ran out, it wrote a separately attributed normalizer/bootstrap module, this page and the corrections.

  1. 8d595cc6ST-1: add explorer, reproducer, verifier custom modes
  2. e10db366ST-4: enhance hook — profile-driven source roots, baseline classification
  3. c3ae2e34ST-5: add gen_report.py, test suite, profile, and baseline
  4. 2c7a8636onramp: save bug report and seal benchmark for webhook-205
  5. 5f16187efix: accept all 2xx HTTP status codes in HttpTransport
  6. 6be32359onramp: make kit profile-driven for Flask / plain-unittest portability
  7. 786c0347Pretty-print JSON request bodies in the ping details view
  8. 81655522fix(onramp): capture error type for plain unittest -v FAIL headers; make map-codebase steps conditional on profile
  9. 703037e2docs: add reclassified baseline and fix evidence.md baseline.txt refs
  10. 5c0b878atest: add failing tests for correct classify_run behaviour
  11. 2a527f8efix: correct classify_run classifier and add robust file reading
All 23 Bob commits (UTC)
  1. 8d595cc6ST-1: add explorer, reproducer, verifier custom modes
  2. 49414247ST-2: add four OnRamp skills
  3. 1a9e8dd0ST-3: add /onramp and /evidence slash commands
  4. b8f526ebST-4: add PostToolUse hook to run app tests on hc/ Python edits
  5. e10db366ST-4: enhance hook — profile-driven source roots, baseline classification
  6. c3ae2e34ST-5: add gen_report.py, test suite, profile, and baseline
  7. 38c4ea29ST-6: generate CODEBASE_MAP.md using map-codebase skill
  8. 2f618b2afix: hook summary extraction picks up OK/FAILED before teardown line
  9. a3acbc38cleanup: consolidate generator tests, add PLAN.md, remove duplicate files
  10. 2c7a8636onramp: save bug report and seal benchmark for webhook-205
  11. 5f16187efix: accept all 2xx HTTP status codes in HttpTransport
  12. a0340d22onramp: add evidence artefacts and report for webhook-205
  13. fa7aae56onramp: attach upstream patch and add comparison to webhook-205 report
  14. 4facc4beonramp: support plain unittest output in gen_report; add tests
  15. 6be32359onramp: make kit profile-driven for Flask / plain-unittest portability
  16. 786c0347Pretty-print JSON request bodies in the ping details view
  17. e4032c29fix: preserve non-ASCII chars in pretty-printed JSON ping body
  18. 81655522fix(onramp): capture error type for plain unittest -v FAIL headers; make map-codebase steps conditional on profile
  19. c582176ddocs: add .github/README.md landing page for judges
  20. cda917addocs: correct README baseline counts, feature #1090 description, portability section, and quick-start instructions
  21. 703037e2docs: add reclassified baseline and fix evidence.md baseline.txt refs
  22. 5c0b878atest: add failing tests for correct classify_run behaviour
  23. 2a527f8efix: correct classify_run classifier and add robust file reading

Check it yourself

Every number on this page comes from a file you can open. Start from any of these.

  1. The kitOnRamp applied to Healthchecks, with every Bob commit
  2. Linux CI run 38460743005The upstream suite passing after the fix
  3. Corrections and limitationsWhat did not work, stated plainly
  4. This page's sourceClassifier, fixtures and the parity check against Bob's Python