AI/ML Model Development
A model that scores well in a notebook has changed nothing yet. We build the ones that run in production, get watched for drift, and change a decision somebody is accountable for.
Most prediction projects stall in the same place. The notebook works, the offline numbers look good, and then the questions start: where does it run, who computes the features at request time, what happens when the inputs shift after a pricing change. Answering those is a different job from fitting the model, and it is usually the job nobody was funded to do. That is where we start.
Scoring well offline is not the hard part
Offline accuracy is measured on a tidy dataset, with all the time in the world to compute a feature. Production has a latency budget, a feature store that may not exist, and inputs that arrive late or not at all. The model that wins the offline comparison is sometimes the one you cannot serve.
- Training features computed in pandas, inference features recomputed by hand in the application
- No owner named for the model once the data science engagement ends
- Accuracy reported on a frozen extract that predates the last product change
- Nothing watching input drift, so the first sign of decay is a complaint
What separates an asset from an experiment
Most of that difference is bookkeeping, which is why it gets skipped. Which data produced this model, which code, which parameters. Whether the training run repeats next quarter for somebody who was not there, and whether a feature called `days_since_last_order` means the same thing in the training set as in the service that calls the model overnight.
So the work includes the unglamorous parts: one definition of each feature used by both paths, versioned datasets and model artefacts, a monitor that compares live inputs and predictions against what training saw, and a written route to retrain or roll back. Decide the rollback before you need it, because the day you need it nobody will be thinking clearly.
How the work runs
No modelling starts until we can name the decision that changes and the number that would tell us it worked. After that the order is conventional, because the conventional order is the one that catches a bad idea early.
- 01
Frame the decision
We start from the action the output is meant to trigger: who sees a score, and what they do differently. What being wrong costs in each direction matters as much as accuracy. You get a problem statement naming the metric and both error costs.
- 02
Data and baseline
Before any model, we check what the data can support and build the dumbest thing that works: a rule, a moving average, last year's value. Plenty of projects end here, honestly. The baseline and its score become the bar everything later has to clear.
- 03
Modelling and evaluation
Candidates get compared on a split that respects time, not a random shuffle, and reported in terms the business uses: how many alerts an analyst works through to find one real case, rather than an F1 score. You review the evaluation against the baseline.
- 04
Serving
The model goes behind an interface the system can call, packaged with the same feature code used in training. Batch scoring into a table is often the right answer and we will say so. You get the deployed endpoint or job and its runbook.
- 05
Monitoring
Once it is live, something has to notice when reality drifts away from the training data, and somebody has to be named to act on that. We set up the dashboards and the retraining path before handover, and walk your team through both.
What we typically build with
Modelling choices follow the problem and the team that will keep the thing running. If your organisation is already on one cloud with a platform team who knows it, that usually decides the serving side before we arrive.
- Python
- scikit-learn
- XGBoost
- PyTorch
- pandas
- MLflow
- DVC
- Docker
- FastAPI
- Kubernetes
- Azure Machine Learning
- Amazon SageMaker
- Vertex AI
Questions worth asking first
Before modelling, not after. We pick the metric from the decision: if a false positive costs a customer and a false negative costs a fraud loss, the threshold follows from those two costs rather than from a default. The number gets agreed with whoever owns the process, written into the problem statement, and used unchanged at review.
Not sure which one you need?
Describe the problem in a paragraph and we will tell you which service applies.