Same pipelines, different answers: how evaluation design shapes EEG seizure prediction

z. song; M. Pedersen; D. Henare

doi:10.64898/2026.10.28.26366763

AI summary · DOI: 10.64898/2026.10.28.26366763 · medRxiv: Neurology

Same pipelines, different answers: how evaluation design shapes EEG seizure prediction

z. song, M. Pedersen, D. Henare

Original paper Published on 29 Oct. 2026 medRxiv preprint License: CC BY-NC-ND 4.0 Also in: fr

How to read this summary? Illustrated guide

The essentials

Fixed EEG seizure-prediction pipelines performed near the 0.5 AUC reference on unseen patients, while window allocation and threshold transfer substantially changed their apparent performance.

Question

How much do seizure-prediction results change when evaluation design changes but model pipelines remain fixed? The study examines unseen-patient testing, chronological separation of seizure episodes, and transfer of validation-selected thresholds.

Method

The retrospective study evaluated five pipelines on 17,424 CHB-MIT EEG windows from 22 pediatric subject groups, using 20 partitions with 14 training, four validation and four test groups. A PSD+LDA control compared chronological seizure-block allocation with class-count-matched random allocation in 11 cases, while a separate analysis transferred thresholds permitting at most 1/60 validation interictal windows to be positive. An external control tested CHB-MIT-trained PSD+LDA models without refitting on 5,657 windows from 13 Siena participants.

Findings

Result On unseen subjects, mean pooled AUC ranged from 0.527 to 0.553, and equally weighting subject groups reduced the means to 0.502–0.523.
See the passage in the paper
Across the 20 partitions, mean pooled AUC ran from 0.527 (PSD+LDA and EEG-Conformer) to 0.553 (TCN), a spread of 0.026 between models.
Result In the 11-case temporal control, PSD+LDA achieved mean AUC 0.572 with chronological seizure blocks versus 0.815 with matched random allocation.
See the passage in the paper
Under this design PSD+LDA reached a mean AUC of 0.572, against 0.815 when the same windows were allocated at random with identical class counts.
Result Validation-selected false-positive constraints did not transfer reliably: mean test false-positive proportions were 1.81–3.13%, while mean preictal-window sensitivity was only 2.33–5.56%.
See the passage in the paper
Thresholds that let through 1 in 60 validation interictal windows gave mean test false-positive proportions of 1.81–3.13%, and every pipeline exceeded the validation target in 6–10 of the 20 partitions (Figure 4A; Supplementary Table S6).
Result Without refitting, the PSD+LDA baseline achieved mean pooled AUC 0.480 on the external Siena dataset.
See the passage in the paper
Applied without any refitting to the 13 Siena participants, the 20 CHB-MIT-trained PSD+LDA fits gave a mean pooled AUC of 0.480 ± 0.022 (participant-macro AUC 0.477 ± 0.024).
Authors’ interpretation The authors interpret evaluation design as more consequential than architecture choice here, but do not attribute the random-versus-chronological gap solely to overlapping samples.
See the passage in the paper
Randomization also changes which periods of a recording end up in training and in testing, so the difference cannot be attributed to overlapping samples alone.

Limitations

The primary dataset was small and pediatric, with usable preictal windows covering only 69 of 182 annotated seizures.
See the passage in the paper
The primary dataset is small and pediatric, and under the window construction and exclusion rules usable preictal windows represented only 69 of 182 annotated seizures.
The sampled-window evaluation did not reproduce continuous surveillance, and preprocessing used noncausal filtering.
See the passage in the paper
Class-dependent sampling changes the distribution of windows and does not reproduce continuous surveillance.
External testing covered only one baseline, and the temporal and threshold controls were exploratory additions after the original analyses.
See the passage in the paper
The Siena check covers one baseline only. The temporal and threshold controls were added after the original analyses and are exploratory.

Why it matters

The authors argue that seizure-prediction results require explicit reporting of subject separation, seizure membership, sample overlap and threshold selection, rather than architecture alone. Released partitions, window manifests and predictions provide a common reference for evaluating new models under matched conditions.

Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.

AI-generated keywords: EEG seizure prediction; Evaluation design; Cross-patient generalization; Temporal data leakage; Threshold transfer

Read further

Comprehensive Summary
Layman's Summary
Blog article
Original abstract

This medRxiv preprint examines how EEG seizure-prediction performance changes when models remain fixed but evaluation design varies. It distinguishes three questions: whether a model generalizes to unseen patients, whether it predicts later seizure-related periods within a known patient, and whether a threshold selected on validation data maintains its false-positive constraint in new patients. Its central argument is that these are distinct tasks: high accuracy on randomly allocated EEG windows does not necessarily demonstrate useful seizure warning. The primary dataset, CHB-MIT, contained 182 annotated seizures from 22 pediatric participants across 23 case folders. The researchers constructed 17,424 normalized, 20-second EEG windows using 18 bipolar channels. Preictal windows fell entirely within the period 5–30 minutes before seizure onset. Ictal periods, the final five minutes before onset, and the four hours after seizure termination were excluded. Usable preictal windows represented 69 seizures. Five pipelines were evaluated: a power spectral density–based linear discriminant classifier (PSD+LDA) and four neural networks—EEGNet, a temporal convolutional network, a one-dimensional convolutional network, and EEG-Conformer. The primary evaluation used 20 partitions, each assigning 14 subject groups to training, four to validation, and four to testing, without access to test-patient data. On unseen subjects, all five pipelines performed close to the nominal AUC reference of 0.5. Mean pooled AUC ranged from 0.527 to 0.553, a between-model spread of just 0.026, while individual models’ standard deviations across partitions were approximately 0.06–0.08. Calculating AUC within each subject and weighting subjects equally reduced mean AUCs to 0.502–0.523. Performance varied substantially between individuals, and model rankings changed with the summary measure, offering little support for architectural superiority. The researchers also reconstructed each window’s source recording, timing, sample interval, and seizure membership. This audit found that the original within-patient, index-based split allowed seizure episodes and overlapping raw samples to cross partition boundaries in all ten eligible cases. A subsequent chronological control kept complete preictal blocks together and compared PSD+LDA performance with random allocations using identical class counts. Across 11 eligible cases containing 11,175 windows, mean chronological AUC was 0.572, compared with 0.815 under random allocation. Random allocation performed better in every case, with a mean paired difference of 0.243. However, this gap cannot be attributed solely to sample overlap, because allocation also changes temporal separation and the recording periods available for training. A further analysis selected thresholds using validation interictal scores alone, allowing at most approximately 1.67% positive decisions. When applied unchanged to unseen subjects, these thresholds produced mean false-positive proportions of 1.81–3.13%, exceeding the target in 6–10 of 20 partitions per pipeline. Mean preictal-window sensitivity was only 2.33–5.56%. An external evaluation, limited to PSD+LDA on 13 Siena participants, yielded a pooled AUC of 0.480 without refitting. The authors conclude that evaluation boundaries and threshold transfer can matter more than architecture choice. They recommend documenting subject, episode, and sample separation and testing validation-selected thresholds without adjustment. Important limitations include the small pediatric cohort, selective window sampling, noncausal filtering, overlapping repeated partitions, and exploratory temporal and threshold analyses added after inspection of the original results. Only one baseline underwent chronological and external testing. Crucially, the study evaluated retrospective windows rather than continuous warnings. Its false-positive proportions cannot be interpreted as alarms per hour, and it did not establish event-level sensitivity, clinical utility, or prospective performance.

Seizure Prediction: Why the Test Can Matter More Than the Model

Can artificial intelligence warn someone that a seizure is approaching? EEG-based prediction research offers reasons for hope, but a high performance score does not necessarily mean a system can provide useful warnings. Sometimes, the way researchers divide and evaluate their data changes the answer more than the model itself.

That is the central message of the medRxiv preprint “Same pipelines, different answers: how evaluation design shapes EEG seizure prediction,” by z. song, M. Pedersen, and D. Henare. By keeping prediction pipelines fixed while changing evaluation conditions, the researchers show why seemingly similar seizure-prediction experiments can measure very different capabilities.

Three Questions That Should Not Be Confused

The study separates three practical questions:

Can a model generalize to patients it has never encountered?
Can it identify later seizure-related periods in a patient whose earlier recordings were available during training?
Will a decision threshold chosen on validation data maintain its false-positive constraint in new patients?

These are not interchangeable achievements. Recognizing patterns in randomly distributed EEG windows is different from predicting future periods, and neither automatically demonstrates that a warning threshold will behave reliably for a new person.

What the Researchers Tested

The primary dataset, CHB-MIT, contained 182 annotated seizures from 22 pediatric participants across 23 case folders. The researchers constructed 17,424 normalized, 20-second EEG windows using 18 bipolar channels. Usable preictal windows—those entirely within 5–30 minutes before seizure onset—represented 69 seizures.

The analysis excluded seizure periods, the final five minutes before onset, and the four hours after seizure termination. Those choices defined a specific classification task, rather than a test of continuous monitoring throughout everyday life.

Five pipelines were compared: a power spectral density–based linear discriminant classifier, or PSD+LDA, alongside EEGNet, a temporal convolutional network, a one-dimensional convolutional network, and EEG-Conformer.

Unseen Patients Proved Difficult

The main evaluation used 20 partitions, each assigning 14 subject groups to training, four to validation, and four to testing. Test-patient data were unavailable during model development within each partition.

Under this stricter evaluation, all five pipelines performed close to the nominal AUC reference of 0.5. AUC measures how well scores distinguish the two classes across possible thresholds. Mean pooled AUC ranged from 0.527 to 0.553—a difference of just 0.026 between models.

Variation across partitions was larger: individual models had standard deviations of roughly 0.06–0.08. When AUC was calculated separately for each subject and subjects received equal weight, mean values fell to 0.502–0.523. Rankings also changed with the summary measure, leaving little evidence that one architecture was consistently superior.

Random Splits Produced a Brighter Picture

To investigate within-patient evaluation, the researchers reconstructed each window’s source recording, timing, sample interval, and seizure membership. This audit found that the original index-based split allowed seizure episodes and overlapping raw samples to cross partition boundaries in all ten eligible cases.

A subsequent chronological control kept complete preictal blocks together. Using PSD+LDA, the researchers compared chronological splits with random allocations containing identical class counts. Across 11 eligible cases and 11,175 windows, mean chronological AUC was 0.572, versus 0.815 under random allocation.

Random allocation performed better in every case, with a mean paired difference of 0.243. That is a substantial change without switching model architecture. However, it cannot be blamed solely on overlapping samples: allocation also changed temporal separation and which recording periods were available for training.

A Threshold That Works Here May Fail Elsewhere

The researchers also selected decision thresholds using validation interictal scores alone, allowing at most approximately 1.67% positive decisions. They then applied those thresholds unchanged to unseen subjects.

Mean false-positive proportions rose to 1.81–3.13%, exceeding the target in 6–10 of the 20 partitions per pipeline. Meanwhile, mean preictal-window sensitivity was only 2.33–5.56%. These results highlight a separate challenge: even a carefully selected validation threshold may not transfer reliably to new patients.

An external evaluation offered another caution. Without refitting, PSD+LDA achieved a pooled AUC of 0.480 on 13 Siena participants. Only this baseline received external and chronological testing, so those findings should not be generalized to every architecture examined.

What the Findings Do—and Do Not—Establish

The study has important limitations: a small pediatric cohort, selective window sampling, noncausal filtering, and overlapping repeated partitions. The temporal and threshold analyses were exploratory additions made after inspection of the original results.

Most importantly, this was retrospective window classification, not continuous seizure warning. Its false-positive proportions are not alarms per hour. The work did not establish event-level sensitivity, clinical usefulness, or prospective performance.

The takeaway is methodological, not a verdict that seizure prediction is impossible. Researchers should document separation between subjects, seizure episodes, and raw samples, and test validation-selected thresholds without adjustment. Before asking which model wins, it is essential to ask what the evaluation actually proves.

Created on 02 Nov. 2026

Was this synthesis useful?

Score: 0

Ask questions about this paper to our AI assistant

How to chat with this paper? Illustrated guide

You can also chat with multiple papers at once here.

AI assistant instructions?

Tools for this paper

Explain it differently 3 free rewrites left today

Save to a collection Group papers in collections to compare them, chat with them and export them. Create a free account Log in

Export BibTeX · RIS (Zotero, Mendeley, EndNote) · Obsidian (Markdown) · Notion (Markdown)

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero

Citation alerts Citation alerts guide

Who works on it? The people and institutions who publish on its topic

Weekly letter Presented in "Brain and behaviour this week" of 2 November 2026. Brain and behaviour this week: What wiring explains—and what context changes

Journal club Premium+ Slides to present it Propose for the journal club

Talk about it in public Premium+ Press kit: release, questions, hype check Press kit guide

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

Close Broader context
Window leakage in clinical EEG machine learning: matched subject-independent evaluation across ADHD and Parkinson cohorts
Burak Aggul · 2026 · Frontiers in Neuroinformatics · Not summarized yet

Window leakage in clinical EEG machine learning measures cross-participant contamination in ADHD and Parkinson classification, rather than seizure prediction.

Abstract
Windowed electroencephalography (EEG) pipelines can produce misleadingly high performance when segments from the same participant occur in both training and testing. We define this contamination as window leakage and test it in two clinically and technically distinct public cohorts: 121 children (61 attention-deficit/hyperactivity disorder [ADHD], 60 controls) and 149 adults (100 Parkinson's disease, 49 controls). Signals were harmonized to 18 common channels and 128 Hz. All primary models used stratified 8-fold participant-disjoint evaluation, 50%-overlapping windows, participant-level probability aggregation, and validation participants drawn exclusively from each outer-training fold. A de…
Close Broader context
Leakage-Controlled Recording-Level Evaluation of Machine Learning, Deep Learning, and Convolutional-Kernel Models for EEG-Based Epileptic Seizure Detection
S. Sunitha, Umadevi Ramamoorthy · 2026 · International Journal of Engineering and Manufacturing · Not summarized yet

Leakage-Controlled Recording-Level Evaluation compares seizure-detection models with recording-grouped validation, not seizure-prediction pipelines.

Abstract
Electroencephalogram (EEG)-based seizure detection is a challenging time-series classification problem because EEG signals are nonlinear, non-stationary, and temporally heterogeneous. This study presents a unified comparison of classical machine-learning, neural and deep-learning, convolutional-kernel time-series, and probability-level ensemble models using the Epileptic Seizure Recognition benchmark dataset. The 11,500 EEG segments were reconstructed into 500 recording groups, and recording-grouped nested cross-validation was applied to prevent segments from the same recording from crossing training and test partitions. Depending on the modelling paradigm, segments were represented either b…
Close Broader context
Seeing Seizures: A Large-Scale Empirical Study of Video and EEG Deep Learning for Automated Rodent Seizure Detection and Severity Grading
Alloy Das, Shreyan Ganguly, N. Massey et al. · 2026 · Not summarized yet

Seeing Seizures evaluates video and EEG models for rodent seizure detection and severity grading, rather than forecasting seizures in people.

Abstract
Manual annotation of synchronized video and EEG recordings is a bottleneck in preclinical seizure research. We present an empirical study of seizure detection and behavioral severity grading using 24,131 paired clips from 601 sessions across 20 rats in a soman-induced epilepsy model. Ten video architectures and seven single-channel EEG models are evaluated independently on binary detection and three- and five-class severity tasks. In the session-exclusive reference comparison, R(2+1)D-18 and GRU achieve detection macro-F1 scores of 0.968 and 0.974, respectively; for five-class grading, the corresponding scores are 0.657 and 0.560. Subject-exclusive evaluation reduces five-class macro-F1 to 0…
Close Broader context
Selection of Low‐Density EEG Montages for Optimal Seizure Detection Performance
S. Abdullateef, Iris L. Soare-Nguyen, B. Jordan et al. · 2026 · Healthcare technology letters · Not summarized yet

Selection of Low-Density EEG Montages optimizes channel configurations for seizure detection, not evaluation design for prediction.

Abstract
ABSTRACT Multi‐channel electroencephalography (EEG) is widely recognised for its ability to non‐invasively capture dynamic brain activity with high temporal resolution. However, conventional EEG systems require meticulous setup, expert interpretation and specialised infrastructure, which limits their applicability in real‐time, resource‐constrained environments such as critical care and emergency settings. As wearable and point‐of‐care EEG technologies gain traction, there is increasing demand for low‐density, automated seizure detection methods that are clinically reliable and scalable. This study presents a data‐driven pipeline for an EEG‐based task, seizure detection, that identifies comp…
Close Broader context
Predicting ICU Seizures Using 1 hour of cEEG.
S. Phadikar, Santiago Philibert-Rosas, Steven E. Haworth et al. · 2026 · Journal of clinical neurophysiology · Not summarized yet

Predicting ICU Seizures estimates near-term seizure risk from one hour of adult ICU EEG, a different prediction setting.

Abstract
PURPOSE Continuous electroencephalography (cEEG) is the gold standard for detecting electrographic seizures in the intensive care unit (ICU), but limited resources limit prolonged monitoring for all patients. We developed an automated, interpretable pipeline that uses a single hour of cEEG to estimate near-term seizure risk across multiple prediction windows. METHODS We retrospectively studied 50 adults who underwent ICU cEEG monitoring at a single tertiary center. From a 1-hour segment, we computed automated interictal-ictal continuum (IIC) burden probabilities for lateralized periodic discharges (LPDs), generalized periodic discharges (GPDs), and lateralized rhythmic delta activity (LRDA),…
Close Broader context
Measuring Evaluation Leakage in EEG Motor Imagery Decoding, with an Application to Stroke Patients
Aria Chadha · 2026 · Proceedings of the OJSS Virtual Research Conference 2026 · Not summarized yet

Measuring Evaluation Leakage studies how CSP leakage affects EEG motor-imagery decoding, not seizure prediction.

Abstract
Brain-computer interfaces (BCIs) that decode imagined movement from EEG are often evaluated using Common Spatial Patterns (CSP), a supervised filtering method. If CSP is fitted to all data before splitting into training and test sets, information from the test labels leaks into the model and inflates reported accuracy. This study measures the size of that effect. In a null experiment where true accuracy should be at chance (50%), fitting CSP before the split produced 91% within-subject accuracy, a 41-percentage-point inflation, compared with about 5 points in cross-subject evaluation. On real data from 87 healthy participants and 46 post-stroke patients, leakage added 14.3 and 10.6 points re…
Close Broader context
How much does the reduced electroencephalographic montage matter for seizure detection? A large-cohort simulation study.
J. Kojima, Hao-Er Shi, Svanik Jaikumar et al. · 2026 · Epilepsia · Not summarized yet

How much does the reduced electroencephalographic montage matter compares full and reduced montages for automated seizure detection.

Abstract
OBJECTIVE Subscalp electroencephalographic (EEG) systems with few channels have emerged as promising solutions for ultra-long-term seizure monitoring, but the impact of montage configuration on automated seizure detection is unclear. We compared automated detection performance between full-scalp and simulated reduced montages approximating published subscalp devices, and assessed variation by epilepsy type, lateralization, and localization. METHODS We conducted a retrospective cross-sectional study of consecutive epilepsy monitoring unit (EMU) admissions from January 2017 to December 2024 at the Hospital of the University of Pennsylvania. Admissions with at least one clinician-annotated seiz…
Close Broader context
Patient-independent EEG seizure detection using channel attention and dual-branch time–frequency features
Hrishikesh Tiwary, Arnav V. Bhavsar, M. H · 2026 · Neural computing & applications (Print) · Not summarized yet

Patient-independent EEG seizure detection develops a channel-attention model under subject-disjoint protocols, addressing detection rather than prediction.

Abstract
We developed a patient-independent EEG-based seizure detection model that generalizes across subjects despite inter-subject variability in spatial seizure expression, electrode characteristics, and artifacts. Scalp EEG recordings from the CHB–MIT (18 channels) and the TUSZ v2.0.3 (20 channels) datasets were evaluated under strict subject-disjoint protocols, employing leave-one-subject-out (LOSO) cross-validation for CHB–MIT and the official subject-disjoint train/dev/eval split for TUSZ. A channel-wise attention front end adaptively reweights EEG channels prior to feature extraction. The reweighted signals are processed through dual-branch time–frequency representations based on empirical mo…

Go further

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 2 November 2026.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.

Same pipelines, different answers: how evaluation design shapes EEG seizure prediction

The essentials

Question

Method

Findings

Limitations

Why it matters

Read further

Definitions

Seizure Prediction: Why the Test Can Matter More Than the Model

Three Questions That Should Not Be Confused

What the Researchers Tested

Unseen Patients Proved Difficult

Random Splits Produced a Brighter Picture

A Threshold That Works Here May Fail Elsewhere

What the Findings Do—and Do Not—Establish

Ask questions about this paper to our AI assistant

Tools for this paper

Advanced tools Premium

Papers around this one

Closest papers

Go further

Cite this paper (BibTeX)