Dataset Forge

The problem

Turning raw documents into dependable training data requires more than an unconstrained prompt: source claims, schemas, metadata, formatting, and duplicate handling all need verifiable controls.

Why I built it

Dataset Forge explores how a simple dataset building interface can sit on top of a rigorous asynchronous workflow without hiding the reliability work behind it.

Intended users

Builders who need structured JSON, JSONL, or CSV datasets derived from reviewed source documents.

The solution

The documented pipeline separates extraction, planning, bounded generation, deterministic validation, second pass critique, and packaging, with progress polling and observability across the workflow.

How it works

• Upload a PDF, DOCX, or TXT source and define the dataset contract.
• Generate canonical records in bounded batches and validate them with deterministic Python rules.
• Apply the critic layer, package accepted records, and retain evaluation evidence for release decisions.

My contribution

Justin designed the product workflow and engineering controls, trained the Dataset Forge critic with QLoRA on a Gemma backbone, and built the evaluation and monitoring approach documented in the presentation.

Engineering decisions

• Use explicit output contracts instead of open ended prompts.
• Keep deterministic software in control of schema validation and release gates.
• Treat failed certification as actionable engineering evidence rather than silently repairing invalid model output.

Validation

The deck records a reproducible 105 case held out comparison with frozen corpus evidence. Both the base model and the fine tuned critic failed the strict structured output certification, so production integration was correctly blocked pending remediation.

Outcomes

The implemented workflow, critic training run, and evaluation harness produced a concrete remediation plan; the deck does not claim the critic is ready for production.