Sostingut Lab: measure before you train
Sostingut Lab is our in-house tool for evaluating and training AI models. With it we decide, with numbers, whether a company is fine with an open model and its documents or whether training one for its task is worth it.
Does your business need a model like these, or will a well-placed rule do? We look at it in the free assessment.
Book an assessmentWhy a lab of our own
When a company asks whether it should train a model, the honest answer is almost always “it depends”. Sostingut Lab exists to turn that “it depends” into a measurement: with which model, which data and which result.
We built it for ourselves, and it is the basis of our local models service: the method we use to decide is the one we hand to the client, with their numbers.
The method, in seven steps
We define the task and the success criterion. We measure a baseline: a simple rule or the current system. We build a question bank with the source of every answer, validated by a person.
We calibrate the judge: if a model scores the answers, we first check on a blind sample that it agrees with a person. And we compare options: open models running locally and commercial ones, with and without documents.
We only train if the comparison shows a gap that the documents don’t close. At the end, a report and a technical sheet in the style of the EU AI Act, with a new evaluation every month.
No measured baseline, no training.
The first case: questions in Catalan about public regulations
To test the method without anyone’s data, we work with the public bylaws and regulations of Tortosa, which anyone can download: 75 documents, 528 pages and 1.5 million characters, stored with their source and fingerprint.
The questions can only be answered by citing a specific passage, and one in nine has no answer in the documents. That way we also measure whether the model can say “I don’t know”.
The comparison pits open models that run locally, such as Qwen and Salamandra, against a commercial model, each with and without document retrieval. The winner is the one that gets it right and stays silent when it has no answer, not the one that sounds most convincing.
Who judges the judge
Scoring hundreds of answers by hand does not scale, so a model acts as judge. But an uncalibrated judge only adds noise that looks like rigour.
That is why the judge never competes, and its agreement with a person is measured on a blind, double-labelled sample before we trust its scores.
Where the data travels
Public data can go through any model. Internal data, preferably through local models. Client or personal data, only through local models or providers with servers in the EU and a processing agreement.
Every reference answer keeps its source, who validated it, when and how confidently. Whatever a model drafts is flagged as synthetic and never enters a training set.
When we train, and how
The ladder is always the same: first a rule, then document retrieval, then a language model and, only if the numbers justify it, a trained model.
To train we use QLoRA on open models such as Salamandra on our own GPU, and a rented GPU for larger models when the comparison shows a gap to close, always within a spending cap. For classification with little data, SetFit; for tabular data, gradient-boosted trees.
What comes out of the lab
Every comparison produces a report that is never edited by hand: success rate with bootstrap confidence intervals, paired contrasts with McNemar’s test, and the latency and cost of each run.
With the report comes a technical sheet with the data used, how it was evaluated and where the model fails, meant for the client’s inventory of AI systems.
The reports also say what cannot be concluded: with few questions the intervals are wide, and we say so.
Tech stack
- Python and FastAPI for the lab’s API
- PostgreSQL with pgvector
- React for the dashboard
- Ollama and llama.cpp for the local models
- Unsloth for QLoRA
- Bootstrap and McNemar for the statistics