My baby cry classifier was learning to recognise babies, not cries
I wanted the feature every baby app advertises — hold up your phone, find out why the baby is crying. This is what happened when I measured it honestly: a model that trained at chance, a data leak that flattered every number, a dataset that was 473 clips but only eight babies in one class, and the much smaller thing I shipped instead.
Counted by infant rather than by clip, the public dataset I started from had only 8 babies in its smallest class, and a speaker leak between train and validation was inflating my best score by about a third. No architecture, augmentation or transfer-learning trick made four classes beat a constant predictor. What shipped is a two-class model — hungry vs discomfort — on frozen YAMNet embeddings with a 2,050-parameter linear head, scoring macro-F1 0.50 against a 0.43 baseline under speaker-disjoint cross-validation, and shown to parents as a probability, not an answer.
Every baby app with a microphone permission sells the same dream: a crying baby, a phone held up for five seconds, and a label — hungry, tired, needs burping. I'm building a baby tracker called Araia, and I wanted that feature too.
This is the honest log of how it went. The short version is that almost every number I produced along the way was either at chance or quietly inflated, and the version that shipped promises much less than I set out to build. I think the reasons are more useful than a success story would have been.
The setup
The very first version of the classifier was trained on synthesised tones, as plumbing: it proved that audio could go from a phone, through ffmpeg, into a TensorFlow model and back as a label. Useless as a model, which was the point.
The real starting corpus was the Baby Cry Sense Dataset on Kaggle, plus a feedback loop in the app: after each analysis a parent can give a thumbs-up, or a thumbs-down with the correct reason, and either way the clip joins the training set. I mapped the dataset to four classes — hungry, discomfort, tiredness, burping — and got 473 usable clips.
The metric throughout is macro-F1, not accuracy. The corpus is heavily skewed toward "hungry", so a model that says hungry to everything scores high accuracy while telling a parent nothing. Macro-F1 averages per-class F1, so ignoring a class costs you.
Model one: a CNN that couldn't fit anything
The first real model was a small CNN — two conv blocks, 16 and 32 channels — over log-mel spectrograms, scored over 3-second windows. Training accuracy settled around 0.22. With four classes, that's chance. It couldn't even overfit.
Standardising each window's log-mel input (per-window mean and variance) and moving to a deeper VGG-style network, 32/64/128 channels with BatchNorm, fixed that: training accuracy climbed to 0.71–0.93 depending on the seed. Validation did not follow.
| Four classes | Validation macro-F1 | Accuracy |
|---|---|---|
| Always predict "hungry" | 0.221 | 0.794 |
| Deeper CNN, range across seeds | 0.13 – 0.36 | lower than the constant |
| Deeper CNN, mean across seeds | 0.23 |
Averaged over seeds, the trained network was indistinguishable from a constant predictor, and worse than it on accuracy. Any single seed could be quoted as a win or a disaster. I quoted a single seed as a result once, early on, and had to take it back. Lesson one: put the constant predictor in the same table as your model, and report a range across seeds.
The leak
While I was trying to work out why validation was so noisy, I looked at the filenames. Clips in the dataset carry the recording device's UUID — which, for a dataset of crying infants, effectively means which baby. My train/validation split was by clip. The same infant appeared on both sides.
That matters more here than in most audio tasks. Infants have distinct voices, and with this few of them a network can score well by recognising the baby rather than the reason. In production every baby is one the model has never heard, so a validation clip from a familiar infant measures nothing useful.
I added a source_id to every clip, matched the UUID anywhere in the filename (the dataset author's own augmentations, named generated1_<uuid>, belong to the same infant), and grouped the split by infant. The best macro-F1 I had measured dropped from 0.368 to 0.256.
That gap was the leak. A third of my best result had been the model recognising voices.
Counting the right thing
Once clips had infants attached, I counted again:
| Class | Distinct infants |
|---|---|
| Hungry | 205 |
| Discomfort / colic | 22 |
| Tiredness | 17 |
| Burping | 8 |
"473 clips" had sounded like a small but workable dataset. "Eight babies" is not a dataset. The model has to learn what makes a burping cry a burping cry from eight voices, and generalise it to a ninth. A 296,000-parameter network trained from scratch on that will learn the eight voices.
Lesson two, and the one I'd most like someone else to skip learning the hard way: for anything people-shaped, the size of your dataset is the number of people in it.
Things that didn't help
I tried the standard remedies, measured each across several seeds, and deleted the code afterwards rather than leave it lying around unused.
- Augmentation made it worse. Speed perturbation, gain, added noise at varying SNR, and SpecAugment, balancing minority classes up to eight variants per source clip. Mean macro-F1: 0.354 with no augmentation, 0.340 with waveform augmentation, 0.302 adding SpecAugment. Training accuracy rose to 0.99. The network memorised the variants along with the originals. The missing axis is speaker identity, and you cannot synthesise new infants by resampling thirteen clips.
- More epochs overfit harder. Sixty epochs did worse than thirty.
- Transfer learning didn't rescue four classes. Frozen YAMNet embeddings with a small head, under speaker-disjoint 5-fold cross-validation, scored 0.237–0.274 against a 0.223 baseline — margins smaller than the spread between folds. Worse, the best-looking configurations got there with zero recall on two of the four classes. They were the constant predictor, slightly hedged.
One useful thing did come out of the YAMNet experiment: results were dramatically steadier across seeds than the CNN's. That stability is what later made an automatic promotion gate trustworthy.
What shipped: two classes
The honest conclusion was that the data could not support four reasons for crying. So I changed the product promise instead of the model.
Everything that isn't hunger now folds into a single discomfort class. That does two things at once: the minority class goes from 8 infants to 51, and categories I'd been discarding as too small — cold or hot, lonely, scared — start contributing instead of being thrown away. The corpus grew to 505 clips without collecting a single new recording.
The model is deliberately tiny:
- YAMNet embeddings (1,024-dimensional), mean-pooled to one row per clip, so a long recording can't outvote a short one;
- one
Denselayer with a softmax — 2,050 parameters; - class weights in the loss. Without them it collapses toward "hungry" (discomfort recall 0.04).
| Two classes, speaker-disjoint | Macro-F1 |
|---|---|
| Always predict "hungry" | 0.431 |
| MLP(256) head on YAMNet | 0.486 |
| Linear head on YAMNet, 5-fold CV | 0.499 ± 0.039 |
| Linear head, held-out set (baseline 0.457) | 0.503 |
Every fold cleared the baseline, and the held-out number agrees with cross-validation, so it isn't a lucky draw. The MLP, with 128 times the parameters, did slightly worse. When the limit is the number of distinct infants, capacity buys overfitting.
I want to be clear about how modest this is. Accuracy is 0.62, below the 0.76 you'd get by always saying "hungry". Discomfort recall is 0.33. The model trades raw hit rate for actually distinguishing the two classes. That's a real signal, and it is a small one.
How it runs in production
- The app shows a probability, not a verdict — "sounds more like hunger" next to the time of the last feed and nap. In practice the log usually tells you more than the sound does, and I've said so to parents.
- It retrains daily from the growing corpus, when there are new clips since the last run.
- A new model goes live only if it clears two bars: it must not score worse than the live model, and it must beat the constant predictor on the same held-out infants. Early on there's no live model to beat, and "better than nothing" isn't good enough to put in front of a tired parent. Most days nothing promotes, which is the correct outcome.
- Every candidate is versioned and kept, promoted or not, with its metrics, so a rejection is auditable and a rollback is one call. The architecture number is stored with each version, so incompatible weights are skipped rather than failing at request time.
- YAMNet is baked into the API image and loaded with
tf.saved_model.load— no TensorFlow Hub at runtime.
What I'd tell anyone doing this with small, human data
- Count people, not samples. Do it before you choose a model.
- Split by person. Look for identity hiding in filenames, device IDs and timestamps. If fixing the split doesn't drop your score, check again.
- Always show the constant predictor beside your result, on a metric that punishes ignoring a class.
- Report ranges across seeds. On small data a single seed is an anecdote.
- Augmentation can't create identities the data doesn't have. If training accuracy rises and validation doesn't, you've just given the network more to memorise.
- Make the model smaller before you make it bigger. A linear head on good frozen features beat everything with more capacity.
- When the data can't carry the promise, change the promise. Fewer classes, honestly presented, beat four classes that are secretly a constant.
What's next
The only thing that will meaningfully move these numbers is more infants. The in-app feedback loop adds a new baby with each family that uses it, and I'm looking for other open infant-cry datasets that record which infant each clip came from. If you know of one, I'd genuinely like to hear about it — support@araia.me reaches me directly.
Cry analysis is one small part of Araia, a baby tracker for feeds, sleep, nappies, vaccines and growth. Most of the app's predictions — like when the baby will next be hungry or tired — come from the family's own logs and plain arithmetic, partly because of everything above.
This guide is general information for parents, written from published paediatric guidance (WHO, AAP, CDC, NHS, IAP). It is not medical advice and cannot account for your baby specifically. Anything that worries you — feeding, breathing, fever, weight, a rash — is a call to your paediatrician, not a search. In an emergency, go to a hospital.
Stop keeping it all in your head
Araia logs feeds, nappies, sleep and weight in one tap, tracks your vaccination schedule (CDC, NHS, WHO or IAP) for you, and answers questions using your own baby's history.