Same pipelines, different answers: how evaluation design shapes EEG seizure prediction
How to prepare an accurate paper press kit? Illustrated guide
z. song, M. Pedersen, D. Henare · DOI: 10.64898/2026.10.28.26366763 · Summary page
Why cover this story this week?
The seizure predictor depends on the test
A seizure-prediction model can look promising or nearly uninformative without changing its architecture. In a preprint using scalp EEG, researchers kept five published analysis pipelines fixed and changed how training and test data were separated. When entire patients were held out, all pipelines scored close to the 0.5 reference for area under the ROC curve. Within patients, a spectral baseline reached 0.815 when windows were randomly allocated, but only 0.572 when preictal blocks stayed intact and later seizures supplied the test data.
Tracing windows back to recordings exposed raw EEG samples crossing the training–test boundary in all ten eligible cases of an index-based split. Thresholds calibrated on validation data also failed to maintain their false-positive rate in new patients, detecting only 2-6% of preictal windows. The lesson reaches beyond neurology: a test must reproduce the separation a tool will face in use. These results concern retrospective window classification, not a continuously operating seizure alarm; the released manifests, partitions and predictions should make comparisons more accountable.
Editorial context written by AI from abstracts, not independent scientific validation. The kit and its verified quotes rely on the source paper, not on this letter.
Talk about a paper in public without overstating it: yours, one you are asked to comment on, or one you write about as a journalist.
- Your paper: a draft release for your press office, the questions journalists will ask with what the paper lets you answer, the likely misreadings, a lay summary and a short thread.
- Asked to comment: what the paper shows and does not, its strengths and weaknesses with their passages, and what would settle its main claim.
- Writing about it as a journalist: reporting angles, the wording to check before publication, and the questions to ask the authors.
- For everyone: findings with their passages checked word for word, the novelty set against the closest earlier papers, a hype check, figures with their credit, and researchers for an outside view who have not signed with the authors.