â¢In line with recent models (Alexander and Brown 2011; 2014), the two experiments here presented found fERN-like components associated with outcomes that ...
ERP signals of Timing Prediction Error in aversive reinforcement learning Sara Garofalo1-2, Martin E. Maier1-3, Christopher Timmermann1 & Giuseppe di Pellegrino1 Department of Psychology, CNC, University of Bologna, Italy 2Department of Psychiatry and Deparment of Psychology, BCNI, Cambridge University, England 3 University of Eichstätt, Germany 1
Introduction The importance of medial prefrontal cortex (mPFC) and its interactions with the dopaminergic system in learning and predicting events is clearly established in current literature. Mediofrontal event-related potential (ERP) components originating from mPFC, such as feedback-related negativity (fERN), have been found to code for Feedback Prediction Error during conflict detection and performance monitoring (Holroyd et al., 2004; Gehring & Willoughby, 2002). Such fERN-like components are usually observed after unexpected feedback or unexpected omission of feedback (Garofalo et al., 2014; Alexander & Brown, 2011; 2014). Electrophysiological studies with animals show that dopamine activity modulations follow precisely timed patterns when upcoming events are either delayed or anticipated in time (Hollerman & Schultz, 1998; Fiorillo et al., 2003).
Recent fMRI studies also reported mPFC activity not only following violations of feedback expectancy, but also following violations of the expected timing of feedback (i.e., for expected feedbacks occurring at unexpected timing), thus also coding for a Timing Prediction Error (TPE) (Forster & Brown, 2011; Klein-Flügge et al., 2011). Nevertheless, to date, ERP components of TPE have never been directly investigated, and it is still unclear if this process relates to fERN-like components. The present studies aimed at testing whether fERN-like components are associated with outcomes that occur at unexpected timing, even if the outcomes themselves are predicted. Two unexpected time shifts conditions were tested: the first using an anticipated timing of feedback and the second using a delayed timing of feedback.
Methods and Procedure
ANTICIPATION CONDITION
Forty-one healthy volunteers participated in the study and were randomly assigned to the “Anticipation” condition (9 females, 11 males; mean age= 23.55, sd = 2.32, 4 left-handed) or to the “Delay” condition (12 females, 9 males; mean age= 24.04, sd = 2.37, 1 left-handed). A Pavlovian aversive conditioning task was used. Participants were presented with two visual stimuli (japanese kanji), paired either with an aversive (shock-associated) (CS+ trials) or with a neutral (CS- trials) visual feedback, consisting in a colored frame appearing around the image for 850 ms. The aversive visual feedback was followed by a shock delivery during the last 200 ms, while during the neutral visual feedback presentation nothing happened until its termination. In the Anticipation Condition, the feedback was presented 2.5 sec. after stimulus presentation on 80% of trials (expected timing of feedback), while it was unexpectedly anticipated on 20% of trials and presented 1 sec. after stimulus presentation (unexpected timing of feedback) (Fig. 1 - Anticipation condition). In the Delay Condition, the feedback was presented 1 sec. after stimulus presentation on 80% of trials (expected timing of feedback), while it was unexpectedly delayed on 20% of trials and presented 2.5 sec. after stimulus presentation (unexpected timing of feedback) (Fig. 1 - Delay Condition). Electroencephalogram (EEG) signal was recorded along the task and stored for offline analysis. The ERP component of interest (fERN) was calculated as the peak-to-peak amplitude recorded on FCz electrode in the 250-350 milliseconds after feedback onset.
Results A 2x2x2 repeated measure ANOVA was performed, using Feedback (Shock/Neutral) and Timing (Expected/Unexpected) as within-subjects independent variables, Condition (Anticipation/Delay) as between-subjects independent variable and peakto-peak fERN amplitude as dependent variable. Results showed a significant main effect of Feedback (F(1, 39) = 5.73; p=.02; η2=.13) a significant main effect of Timing (F(1, 39) = 12.63; p=.001; η2=.24) and a significant Feedback X Timing interaction (F(1, 39) = 5.99; p=.02; η2=.13). Bonferroni-corrected post-hoc analysis on the Feedback X Timing interaction revealed a significant difference between shock-expected and shock-unexpected (p=.0003) and between shock-unexpected and neutral-unexpected (p=.007) conditions (Fig. 2 - 3). No other comparisons were statistically significant (ps >.2). Scalp topography of the peak-to-peak amplitude in the 250-350 ms time-window showed a frontocentral activation coherent with fERN scalp distribution (Fig. 4)
SHOCK TRIALS
NEUTRAL TRIALS
DELAY CONDITION SHOCK TRIALS
NEUTRAL TRIALS
Fig. 1 – Schematic representation of the task structure in the Anticipation and Delay conditions.
Potential (μV)
SHOCK EXPECTED SHOCK UNEXPECTED NEUTRAL EXPECTED NEUTRAL UNEXPECTED
TIMING
Time (ms)
Fig. 3 - Peak-to-peak amplitude in the 250-350 ms time-window on FCz. Bars represent standard error.
Fig. 4 - Scalp topography of the peak-to-peak amplitude in the 250-350 ms time-window.
Fig. 2 - Grandaverage ERP waveforms on FCz electrode. 0 represents feedback onset.
Discussion •In line with recent models (Alexander and Brown 2011; 2014), the two experiments here presented found fERN-like components associated with outcomes that occur at both unexpectedly anticipated or delayed timings, even if the outcomes themselves are expected. The presence of mediofrontal ERPs coding for Timing Prediction Error (TPE) seem to indicate that fERN-like components not only signal unexpected feedback, as reported by the vast majority of studies, but also code when the feedback itself is likely to be received. These evidences represent a first step towards a possible link between mPFC and dopaminergic activity following violations of the expected timing of References Alexander, W. H., & Brown, J. W. (2011). Medial prefrontal cortex as an action-outcome predictor. Nature neuroscience, 14(10), 1338-1344. Alexander, W. H., & Brown, J. W. (2014). A general role for medial prefrontal cortex in event prediction. Frontiers in Computational Neuroscience, 8, 69. Falkenstein, M., Hohnsbein, J., Hoormann, J., & Blanke, L. (1991). Effects of crossmodal divided attention on late ERP components. II. Error processing in choice reaction tasks. Electroencephalography and clinical neurophysiology, 78(6), 447-455. Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299(5614), 1898-1902. Forster, S. E., & Brown, J. W. (2011). Medial prefrontal cortex predicts and evaluates the timing of action outcomes. Neuroimage, 55(1), 253-265. Garofalo, S., Maier, M. E., & di Pellegrino, G. (2014). Mediofrontal negativity signals unexpected omission of aversive events. Scientific reports, 4,
feedback, reported by previous studies (Forster & Brown, 2011; Klein-Flügge et al., 2011; Hollerman & Schultz, 1998; Fiorillo et al., 2003), and mediofrontal ERP signals of Prediction Error, such as fERN. Furthermore a saliency effect was highlighted by the present experiment, since shock-associated unexpected feedback generated a stronger fERN-like component as compared with shock-associated expected feedback as well as with neutral unexpected feedback. This result seems to be in line with previous work which reported fERN-like components coding for salience prediction errors (Talmi et al., 2013). Gehring, W. J., & Willoughby, A. R. (2002). The medial frontal cortex and the rapid processing of monetary gains and losses. Science, 295(5563), 2279-2282. Hollerman, J. R., & Schultz, W. (1998). Dopamine neurons report an error in the temporal prediction of reward during learning. Nature neuroscience, 1(4), 304-309. Klein-Flügge, M. C., Hunt, L. T., Bach, D. R., Dolan, R. J., & Behrens, T. E. (2011). Dissociable reward and timing signals in human midbrain and ventral striatum. Neuron, 72(4), 654-664. Talmi, D., Atkinson, R., & El-Deredy, W. (2013). The feedback-related negativity signals salience prediction errors, not reward prediction errors. The Journal of Neuroscience, 33(19), 8264-8269.
Acknowledgmens The authors thank Giulia Petrillo and Simone Battaglia for helping with data acquisition.
