Hi, I am Mr Lark. In Chinese, I go by 拉克先生.
When I choose a model now, I first ask whether it can handle my real tasks. Leaderboards come second.
Leaderboards are not enough
Model leaderboards change every day.
One model reaches the top today. Another claims to beat it tomorrow. A score tells me how a model performs on public tasks, but not whether it can match my tone, fix my code, or understand my context.
I have seen many strong models fail in daily use. They miss constraints in writing, hallucinate in research, and change the wrong boundary in coding. A model can be good at benchmarks and still fail on a vague two-sentence request.
Public leaderboards rank average ability. Your own eval ranks which model can do your work.
Your eval is also few-shot material
Building an eval set can feel heavy.
You need to collect cases, write expected outputs, and define rubrics. That can sound like unnecessary process.
But those 20 cases have a second use: they become your few-shot library.
Each case contains an input, an expected output, and a scoring standard. You are recording how you want AI to handle this type of task.
The next time you face a similar task, you can paste 2-3 cases into the prompt. Even a weaker model can do better when it sees your pattern.
Start with 20 real failures
The first version should not chase scale.
Start with 20 high-quality cases. Once you can see clear differences between models, expand the set to 100+.
The best cases come from your AI usage over the past 1-2 months:
- Failed conversations
- Tasks with the wrong tone
- Summaries that missed key details
- Code tasks with the wrong change boundary
- Research answers with hallucinations
- Tasks that only worked after repeated follow-up
Real failures are more valuable than invented questions.
You are not testing whether a model can answer a polished prompt. You are testing whether it fails in the situations you actually face.
Each case needs three parts
Each case needs three parts:
Input: the original prompt. Keep it messy if that is how the real request looked.
Expected Output: the answer you wanted. Write it yourself or edit a model draft.
Rubric: the scoring standard. Avoid vague comments like "pretty good." Split the judgment into concrete checks:
| Dimension | Weight |
|---|---|
| Correctness | 40% |
| Completeness | 20% |
| Tone and style | 20% |
| No hallucination | 20% |
JSONL is an easy storage format. Score each answer from 1 to 10, or use Pass/Fail with a concrete reason.
Prioritize edge cases
The set needs diversity, but it should not spread effort evenly.
Include simple, intermediate, and hard tasks. Include writing, coding, summarization, search, rewriting, and analysis.
The cases that separate models are usually edge cases: wrong tone, missed constraints, misunderstood context, invented facts, or answered directly when the model should have asked a follow-up question.
These cases expose model weaknesses quickly.
Compare manually first
The lightest workflow is to feed the same case to several models and compare the answers yourself.
That already solves most model-selection problems.
If you want a more convincing result, turn the workflow into a table:
- One row per case
- One column per model output
- One column for the ground truth
- One column for score and reason
Later, you can use a judge model. Give it the input, expected output, rubric, and candidate answer. Ask it to score the answer and explain why.
Judge models can also be biased, so sample-check part of the scores.
Build your own standard
Public leaderboards are still useful, but only as a first filter.
What matters in daily work is whether a model understands your tone, codebase, business context, and follow-up habits.
Only your own eval set can answer those questions.
Start with 20 real cases. Expand to 100+ after you see clear differences. Add rubrics and a judge model when you need a stable process.
Those cases can also become few-shot examples.
That is more reliable than chasing the newest leaderboard.