How to scope a first dataset

Most first datasets are scoped by working backwards from a number — a budget, a row count someone quoted, a figure from a paper. That is understandable and it is the wrong starting point, because it fixes the one variable you should be discovering.

Start from the failure, not the volume

If a model already exists, its errors are the specification. Sample them, group them, and describe each group precisely enough that someone could go and collect examples of it. That list is your dataset scope.

If no model exists yet, the equivalent exercise is to describe the situations the system must handle and be honest about which of them you have no data for. The gaps are the scope.

Test the specification before the volume

A few hundred examples taken through the entire path — collection, annotation, review, delivery, and a real training run — will surface almost every problem that a large batch would, at a fraction of the cost.

• Whether the label set can be applied consistently by two people
• Whether the delivered structure loads without transformation
• Whether the annotation effort per example matches the estimate
• Whether the model moves in the expected direction

Any one of those coming back negative changes the plan. Discovering it at three hundred examples is a scheduling adjustment. Discovering it at thirty thousand is a write-off.

Decide the output format first

This sounds administrative and it removes an entire category of rework. Agree the schema, the identifiers, the timestamp granularity and the file structure before annotation begins, and delivery becomes a handover rather than a translation exercise.

Budget for rejection

Some proportion of collected data will not meet the specification. That is normal and it should be planned for rather than treated as a failure. A project that has not rejected anything has usually not been checking.

Plan the second batch into the first

The first batch's real output is not just data. It is a corrected guideline, a measured cost per example, and a clearer picture of which cases matter. Scope the engagement so that learning has somewhere to go, rather than treating the specification as fixed once signed.

Agree what rejected means

Rejection only functions as a quality mechanism if both sides agreed in advance what would trigger it. Otherwise it becomes a negotiation at delivery, where the party who wants to be paid and the party who wants the data argue about a standard neither wrote down.

The acceptance definition should be specific enough to check: what makes an example unusable, what proportion of a batch may fail before the batch is returned, and who arbitrates. Written before work starts, this is a paragraph. Written afterwards, it is a dispute.

Common ways a first dataset goes wrong

• Scoped by budget rather than by the failures it is meant to fix
• Volume committed before two annotators have agreed on a hundred examples
• Output schema settled after annotation, producing a transformation step nobody costed
• Difficult examples excluded because they slowed the pipeline down
• No provenance recorded, so the second batch cannot be compared to the first

None of these are exotic mistakes. They are all the result of treating the first dataset as a procurement exercise rather than an experiment whose main output is a better specification.