New ERP Decoding Paper: Reactivation of Previous Experiences in a Working Memory Task

Bae, G.-Y., & Luck, S. J. (in press). Reactivation of Previous Experiences in a Working Memory Task. Psychological Sciencehttps://doi.org/10.1177/0956797619830398

Gi-Yeul Bae and I have previously shown that the ERP scalp distribution can be used to decode which of 16 orientations is currently being stored in visual working memory (VWM). In this new paper, we reanalyze those data and show that we can also decode the orientation of the stimulus from the previous trial. It’s amazing that this much information is present in the pattern of voltage on the surface of the scalp!

Here’s the scientific background: There are many ways in which previously presented information can automatically impact our current cognitive processing and behavior (e.g., semantic priming, perceptual priming, negative priming, proactive interference). An example of this that has received considerable attention recently is the serial dependence effect in visual perception (see, e.g., Fischer & Whitney, 2014). When observers perform a perceptual task on a series of trials, the reported target value on one trial is biased by the target value from the preceding trial. 

We also find this trial-to-trial dependency in visual working memory experiments: The reported orientation on one trial is biased away from the stimulus orientation on the previous trial. On each trial (see figure below), subjects see an oriented teardrop and, after a brief delay, report the remembered orientation by adjusting a new teardrop to match the original teardrop’s orientation. Each trial is independent, and yet the reported orientation on one trial (indicated by the blue circle in the figure) is biased away from the orientation on the previous trial (indicated by the red circle in the figure; note that the circles were not actually colored in the actual experiment). 

N-1-Decoding--Stimuli.jpg

These effects imply that a memory is stored of the previous-trial target, and this memory impacts the processing of the target on the current trial. But what is the nature of this memory?

We considered three possibilities: 1) An active representation from the previous trial is still present on the current trial; 2) The representation from the previous trial is stored in some kind of “activity-silent” synaptic form that influences the flow of information on the current trial; and 3) An activity-silent representation of the previous trial is reactivated when the current trial begins. We found evidence in favor of this third possibility by decoding the previous-trial orientation from the current-trial scalp ERP. That is, we used the ERP scalp distribution at each time point on the current trial to “predict” the orientation on the previous trial.

This previous-trial decoding is shown for two separate experiments in the figure below. Time zero represents the onset of the sample stimulus on the current trial. In both experiments, we could decode the orientation from the previous trial in the period following the onset of the current-trial sample stimulus (gray regions are statistically significant after controlling for multiple comparisons; chance = 1/16). 

N-1-Decoding--Results.jpg

These results indicate that a representation of the previous-trial orientation was activated (and therefore decodable) by the onset of the current-trial stimulus. We can’t prove that this reactivation was actually responsible for the behavioral priming effect, but this at least establishes the plausibility of reactivation as a mechanism of priming (as hypothesized many years ago by Gordon Logan).

This study also demonstrates the power of applying decoding methods to ERP data. These methods allow us to track the information that is currently being represented by the brain, and they have amazing sensitivity to quite subtle effects. Frankly, I was quite surprised when Gi-Yeul first showed me that he could decode the orientation of the previous-trial target. And I wouldn’t have believed it if he hadn’t shown that he replicated the result in an independent set of data.

Gi-Yeul has made the data and code available at https://osf.io/dbgh6/. Please take his code and apply it to your own data!

New Paper: fMRI study of working memory capacity in schizophrenia

Hahn, B., Robinson, B. M., Leonard, C. J., Luck, S. J., & Gold, J. M. (2018). Posterior parietal cortex dysfunction is central to working memory storage and broad cognitive deficits in schizophrenia. The Journal of Neuroscience37, 8378–8387. https://doi.org/DOI: https://doi.org/10.1523/JNEUROSCI.0913-18.2018 https://doi.org/10.1523/JNEUROSCI.0913-18.2018.

In several behavioral studies using change detection/localization tasks, we have previously shown that people with schizophrenia (PSZ) exhibit large reductions in visual working memory storage capacity (Kmax). In one large study with 99 PSZ and 77 healthy control subjects (HCS), we found an effect size (Cohen's d) of 1.11, and the degree of Kmax reduction statistically accounted for approximately 40% of the reduction in overall cognitive ability exhibited by PSZ (as measured with the MATRICS Battery). Change detection tasks are much simpler than most working memory tasks, focus on storage rather than manipulation, and can be used across species. Thus, Kmax gives us a measure that is both neurobiologically tractable and strongly related to broad cognitive dysfunction.

In our most recent work, led by Dr. Britta Hahn at the Maryland Psychiatric Research Center, we used fMRI to examine the neuroanatomical substrates of reduced Kmax in PSZ. We took advantage of an approach pioneered by Todd and Marois (2004, Nature), in which a whole-brain analysis is used to find clusters of voxels where the BOLD signal is related to the amount of information actually stored in working memory (K). As shown in the figure below, we found the same areas of posterior parietal cortex (PPC) that were observed by Todd and Marois.

In the left PPC, however, the K-dependent modulation of activity was reduced in PSZ relative to HCS. As shown in the scatterplots, the BOLD signal in this region was strongly related to the number of items being held in working memory (K) in HCS, but the function was essentially flat in PSZ. However, the overall level of activation was just as great in PSZ as in HCS (the Y intercept). The reduced slope was driven mainly by an overactivation in PSZ relative to HCS when relatively little information was being stored in memory. Moreover, the slope was strongly correlated with overall cognitive ability (again measured using the MATRICS Battery), and the degree of slope reduction statistically accounted for over 40% of the reduction in broad cognitive ability in PSZ.

One particularly interesting aspect of these results is that they point to posterior parietal cortex as a potential source of cognitive dysfunction in schizophrenia, whereas most research and theory has focused on prefrontal cortex. Studies with healthy young adults have consistently identified PPC as a major player in working memory capacity and in the ability to divide attention, both of which are strongly impaired in PSZ. We hope that our study motivates more research to examine the potential contribution of the PPC to cognitive dysfunction in schizophrenia.

Hahn fMRI Change Detection.jpg

New paper: What happens to an individual visual working memory representation when it is interrupted?

Bae, G.-Y., & Luck, S. J. (2018). What happens to an individual visual working memory representation when it is interrupted? British Journal of Psychology. https://onlinelibrary.wiley.com/doi/full/10.1111/bjop.12339

Working memory is often conceived as a buffer that holds information currently being operated upon. However, many studies have shown that it is possible to perform fairly complex tasks (e.g., visual search) that are interposed during the retention interval of a change detection task with minimal interference (especially load-dependent interference). One possible explanation is that the information from the change detection task can be held in some other form (e.g., activity-silent memory) while the interposed task is being performed.  If so, this might be expected to have subtle effects on the memory for the stimulus.

To test this, we had subjects perform a delayed estimation task, in which a single teardrop-shaped stimulus was held in memory and was reproduced at the end of the trial (see figure below). A single letter stimulus was presented during the delay period on some trials. We asked whether performing a very simple task with this interposed stimulus would cause a subtle disruption in the memory for the teardrop's orientation.  In some trial blocks, subjects simply ignored the interposed letter, and we found that it produced no disruption of the memory for the teardrop. In other trial blocks, subjects had to make a speeded response to the interposed letter, indicating whether it was a C or a D. Although this was a simple task, and only a single object was being maintained in working memory, the interposed stimulus caused the memory of the teardrop to become less precise and more categorical.

Thus, performing even a simple task on an interposed stimulus can disrupt a previously encoding working memory representation. The representation is not destroyed, but becomes less precise and more categorical, perhaps indicating that it had been offloaded into a different form of storage while the interposed task was being performed. Interestingly, we did not find this effect when an auditory interposed task was used, consistent with modality-specific representations.

Interruption_Paradigm.jpg

Decoding the contents of working memory from scalp EEG/ERP signals

Bae, G. Y., & Luck, S. J. (2018). Dissociable Decoding of Working Memory and Spatial Attention from EEG Oscillations and Sustained Potentials. The Journal of Neuroscience, 38, 409-422.

In this recent paper, we show that it is possible to decode the exact orientation of a stimulus as it is being held in working memory from sustained (CDA-like) ERPs.  A key finding is that we could decode both the orientation and the location of the attended stimulus with these sustained ERPs, whereas alpha-band EEG signals contained information only about the location.  

Our decoding accuracy is only about 50% above the chance level, but it's still pretty amazing that such precise information can be decoded from brain activity that we're recording from electrodes on the scalp!

Stay tuned for more cool EEG/ERP decoding results — we will be submitting a couple more studies in the near future.