ReviewPilot AI

Context

Code review tools often produce unstructured commentary that is difficult to scan, verify or compare. This project explores a more explicit contract between a diff, an analysis provider and the review interface.

Problem

Turn a noisy unified diff into findings that remain tied to files and lines, while keeping provider behavior replaceable and outputs safe to render.

Constraints

• Webhook payloads and signatures must be verified before processing.
• AI-shaped output cannot be trusted without runtime validation.
• The public demo must remain useful without requiring paid model credentials.

My contribution

Product design, frontend architecture, GitHub integration and evaluation workflow.

Important technical decisions

• Zod validates provider output at the application boundary.
• A provider abstraction separates workflow logic from model-specific transport.
• Line-aware findings preserve a traceable relationship to the source diff.
• The deterministic provider is an explainable fixture—not a real language model—and supports repeatable demos and tests.

Testing and quality

• Vitest covers parsing and workflow behavior.
• Golden cases evaluate whether structured outputs remain stable across representative diffs.
• CI runs automated checks on changes.

Results

• Structured findings with confidence, severity and source locations
• File-level risk, test suggestions and a merge recommendation
• A deterministic demo path plus an optional OpenAI-compatible integration

Trade-offs and limitations

• Structured output improves consistency but constrains free-form provider responses.
• The deterministic provider demonstrates the workflow, not production model quality.
• Automated feedback remains advisory and requires human review.

Alternatives considered

• A model-specific implementation was rejected because it would couple product logic to one provider.
• Free-form Markdown was considered but offers weaker validation and UI guarantees.

Current status

Working public demo; continued experimentation with evaluation cases and provider behavior.