VantageAI: Fine-Tuning an LLM to Grade DECA Cases

DECA is a business and entrepreneurship competition. In the case study events, students role-play as consultants being evaluated by a judge. The judge scores them against a rubric — typically 10–20 criteria, each rated on a scale. Multiply that by thousands of students across Ontario, and you have a grading problem: slow, inconsistent, and expensive in judge time.

The rubric structure is what made me think this was tractable. DECA cases aren't open-ended essays — they have structured evaluation criteria. Structure is something you can teach a model.

The Fine-Tuning Process

Dataset construction is the part nobody talks about when they write about fine-tuning. The model is only as good as what you train it on. I needed examples of DECA case performances with high-quality rubric scores — not crowdsourced ratings, but scores from experienced DECA judges who understand what the rubric criteria actually mean.

Getting that data took longer than the technical work. Once I had it, I formatted the examples as (performance transcript, rubric) → (score, justification) pairs. The justification component is key — it forces the model to ground its score in specific rubric criteria rather than producing a number without reasoning.

The training process itself was straightforward by comparison. Evaluate on held-out examples, iterate on the training data and prompt format, measure inter-rater reliability against human judges.

What Nobody Tells You About Fine-Tuning

The base model matters more than the fine-tuning. If you're starting with a model that has weak reasoning capabilities, fine-tuning won't fix that — you're just adding a thin layer of domain-specific behavior on top of a weak foundation. Start with a capable base.

More data is not always better. Low-quality examples dilute the training signal. I removed examples from the dataset that showed inconsistent scoring, even though that made the dataset smaller. The model trained on the smaller, cleaner set outperformed the one trained on everything.

The evaluation methodology is the hardest part to get right. What does 'good' mean for a grader? Agreement with human judges is the obvious metric, but human judges disagree with each other. I ended up measuring consistency (same input → same output across runs) and calibration (scores distributed similarly to the human distribution) separately from pure accuracy.

Getting into an Actual Pilot

There's a gap between 'I built a thing that works in my testing' and 'a real organization is running it on real data.' That gap is mostly trust and bureaucracy, not technical difficulty.

Getting VantageAI into an active pilot with DECA Ontario required showing that the model's outputs were interpretable (not just a number, but a justified score), that it performed consistently on edge cases, and that there was a clear process for human review when the model was uncertain.

The hardest part wasn't the model. It was convincing DECA to try it.