New Paper: Using ERPs and RSA to examine saliency maps and meaning maps for natural scenes

Kiat, J.E., Hayes, T.R., Henderson, J.M., Luck, S.J. (in press). Rapid extraction of the spatial distribution of physical saliency and semantic informativeness from natural scenes in the human brain. The Journal of Neuroscience. https://doi.org/10.1523/JNEUROSCI.0602-21.2021 [preprint]

The influence of physical salience on visual attention in real-world scenes has been extensively studied over the past few decades. Intriguingly, however, recent research has shown that semantically informative scene features often trump physical salience in predicting even the fastest eye movements in natural scene viewing. These results suggest that the brain extracts visual information that is, at the very least, predictive of the spatial distribution of potentially meaningful scene regions very rapidly.

In this new paper, Steve Luck, Taylor Hayes, John Henderson, and I sought to assess the evidence for a neural representation of the spatial distribution of meaningful features and (assuming we found such a link!) contrast the onset of its emergence relative to the onset of physical saliency. To do so, we recorded 64-channel EEG data from subjects viewing a series of real-world scene photographs while performing a modified 1-back task in which subjects were probed on 10% of trials to identify which of four scene quadrants was part of the most recently presented image (see Figure 1).

Figure 1. Stimuli and task. Subjects viewed a sequence of natural scenes. After 10% of scenes, they were probed for their memory of the immediately preceding scene.

With this dataset in hand, we next obtained spatial maps of meaning and saliency for each of the scenes. To measure the spatial distribution of meaningful features, we leveraged the “meaning maps” that had previously been obtained by the Henderson group. These maps are obtained by crowd-sourced human judgments of the meaningfulness of each patch of a given scene. The scene is first decomposed into a series of partially overlapping and tiled circular patches, and subjects rate each circular patch for informativeness (see Figure 2 and Henderson & Hayes, 2017). Then, these ratings are averaged and smoothed to produce a “meaning map,” which reflect the extent to which each location in a scene contains meaningful information. Note that these maps do not indicate the specific meanings, but simply indicate the extent to which any kind of meaningful information is present at each location.

Figure 2. Top: Example scene with corresponding saliency map and meaning map. Two areas are highlighted in blue to make it easier to see how saliency, meaningfulness, and the image correspond in these areas. Bottom: Examples of patches that were used to create the meaning maps. Observers saw individual patches, without any scene context, and rated the meaningfulness of that patch. The ratings across multiple observers for each patch were combined to create the meaning map for a given scene.

The spatial distribution of physical saliency was estimated algorithmically using the Graph-Based Visual Saliency approach (Harel et al., 2006). This algorithm extracts low-level color, orientation, and contrast feature vectors from an image using biologically inspired filters. These features are then used to compute activation maps for each feature type. Finally, these maps are normalized, additively combined, and smoothened to produce an overall “saliency map”. A few examples of meaning and saliency maps for specific scenes are shown in Figure 3. We chose this algorithm in particular because of its combination of biological plausibility and performance at matching human eye movement data.

Figure 3. Examples of images used in the study and the corresponding saliency and meaning maps. The blue regions are intended to make it easier to see correspondences between the maps and the images.

We then used the meaning maps and saliency maps to predict our ERP signals using Representational Similarity Analysis. For an overview of Representational Similarity Analysis in the context of ERPs, check out this video and this blog post.

The results are summarized in Figure 4. Not surprisingly, we found that a link between physical saliency and the ERPs emerged rapidly (ca. 78 ms after stimulus onset). The main question was how long it would take for a link to the meaning maps to be present. Would the spatial distribution of semantic informativeness take hundreds of milliseconds to develop, or would the brain rapidly determine which locations likely contained meaningful information? We found that the link between the meaning maps and the ERPs occurred extremely rapidly, less than 10 ms after the link to the saliency maps (ca. 87 ms after stimulus onset). You can see the timecourse of changes in the strength of the representational link for saliency and meaning in panel A (colored horizontal lines demark FDR corrected p < .05 timepoints) and the jackknifed mean onset latencies for the representational link of saliency and meaning in Panel B (error bars denote standard errors).

Figure 4. Primary results. A) Representational similarity between the ERP data and the saliency and meaning maps at each time point, averaged over participants. Each waveform shows the unique variance explained by each map type. B) Onset latencies from the representational similarity waveforms for saliency and meaning. The onset was only slightly later for the meaning maps than for the saliency maps.

Note that the waveforms show semipartial correlations (i.e., the unique contribution of one type of map when variance due to the other type is factored out). These findings therefore show that meaning maps have a unique neurophysiological basis from saliency.

The rapid time course of the meaning map waveform also indicates that information related to the locations containing potentially meaningful information is computed rapidly, early enough to influence even the earliest eye movements. This is a correlation-based approach, so these results do not indicate that meaning per se is calculated by 87 ms. However, the results indicate that information that predicts the locations of meaningful scene elements is computed by 87 ms. Presumably, this information would be useful for directing shifts of covert and/or overt attention that would in turn allow the actual meanings to be computed.

The data and code are available at https://osf.io/zg7ue/. Please feel free to use this code and dataset (high-density ERP averages for 50 real-world scenes from 32 subjects) to explore research questions that interest you!

New papers on the hyperfocusing hypothesis of cognitive dysfunction in schizophrenia

Luck, S. J., Hahn, B., Leonard, C. J., & Gold, J. M. (2019). The hyperfocusing hypothesis: A new account of cognitive dysfunction in schizophrenia. Schizophrenia Bulletin, 45, 991–1000. https://doi.org/10.1093/schbul/sbz063

Luck, S. J., Leonard, C. J., Hahn, B., & Gold, J. M. (2019). Is selective attention impaired in schizophrenia? Schizophrenia Bulletin, 45, 1001–1011. https://doi.org/10.1093/schbul/sbz045

SCHIZOPHRENIA BY ANNDEEF, http://fav.me/d4ggpqk, (Creative Commons License)

The most distinctive symptoms of schizophrenia are hallucinations, delusions, and disordered thought/behavior. However, people with schizophrenia also typically have impairments in basic cognitive processes, such as attention and working memory, and the degree of cognitive dysfunction is a better predictor of long-term outcome than is the severity of the psychotic symptoms.

Researchers have tried to identify the nature of cognitive dysfunction in schizophrenia since the 1960s, and our collaborative research group has spent almost 20 years on this problem. We now have a well-supported theory, which we call the hyperfocusing hypothesis, and we recently published a pair of papers that review this theory. The first paper describes the hyperfocusing hypothesis in detail and reviews the evidence for it, and the second paper contrasts it with the traditional idea that schizophrenia involves impaired filtering.

The hyperfocusing hypothesis proposes that schizophrenia involves an abnormally narrow but intense focusing of processing resources. That is, people with schizophrenia are not impaired at focusing their attention; on the contrary, they tend to focus their attention more intensely and more narrowly compared to healthy control subjects. This hypothesis can explain findings from several different cognitive domains, including reductions in working memory capacity (because people with schizophrenia have difficulty dividing resources among multiple memory representations), deficits in experimental paradigms that involve spreading attention broadly (such as the Useful Field of View task), and abnormal capture of attention by irrelevant stimuli that share features with active representations. In addition to explaining many previous findings, the hyperfocusing hypothesis has also led to many new predictions that have been tested and verified. We also find that the degree of hyperfocusing is often correlated with the degree of impairment in measures of broad cognitive function, which are known to be related to long-term outcome.

When a psychiatric group exhibits impaired performance relative to a control group, there are usually many possible explanations (e.g., reduced motivation, impaired task comprehension). However, the hyperfocusing hypothesis proposes that people with schizophrenia focus more strongly than control subjects, which leads to the counterintuitive prediction that people with schizophrenia will exhibit supranormal focusing of processing resources under some conditions. And this is exactly what we have found in several experiments. For example, in both ERP and fMRI studies, we have found that delay-period activity is enhanced in people with schizophrenia relative to control subjects when only a single object is being maintained. This is an example of what we mean by a “more intense” focusing of processing resources. You might be concerned that people with schizophrenia exert greater effort to achieve the same memory performance, and this leads to greater delay-period activity. However, when we examine subgroups that are matched on behavioral measures of working memory capacity, we still find that people with schizophrenia exhibit enhanced activity relative to control subjects when a single item is being remembered.

Classically, schizophrenia has been thought to involve an impairment in selective attention, a “broken filter.” For example, one individual wrote the following in an online forum: “Ever since I started having problems due to schizophrenia, my senses have been thrown out of whack... I remember one day when I got caught in the rain. Each drop felt like an electric shock and I found it hard to move because of how intense and painful the feeling was.” How can we reconcile this evidence for increased distraction with the idea that schizophrenia involves hyperfocusing? The most likely rapprochement between the hyperfocusing hypothesis and the broken filter hypothesis is that schizophrenia also involves impaired executive control, so people with schizophrenia often point their “spotlight” of attention in the wrong direction. As a result, they may focus narrowly and intensely on inputs that would ordinarily be ignored (e.g., drops of rain), producing greater distractibility even though the filtering mechanism itself is operating very intensely.

New paper: N2pc versus TELAS (target-elicited lateralized alpha suppression)

Bacigalupo, F., & Luck, S. J. (in press). Lateralized suppression of alpha-band EEG activity as a mechanism of target processing. The Journal of Neuroscience. https://doi.org/10.1523/JNEUROSCI.0183-18.2018

Since the classic study of Worden et al. (2000), we have known directing attention to the location of an upcoming target leads to a suppression of alpha-band EEG activity over the contralateral hemisphere. This is usually thought to reflect a preparatory process that increases cortical excitability in the hemisphere that will eventually process the upcoming target (or decreases excitability in the opposite hemisphere). This can be contrasted with the N2pc component, which reflects the focusing of attention onto a currently visible target (reviewed by Luck, 2012). But do these different neural signals actually reflect similar underlying attentional mechanisms? The answer in a new study by Felix Bacigalupo (now on the faculty at Pontificia Universidad Catolica de Chile) appears to be both “yes” (the N2pc component and lateralized alpha suppression can both be triggered by a target, and they are both influenced by some of the same experimental manipulations) and “no” (they have different time courses and are influenced differently by other manipulations).

The study involved two experiments that we were designed to determine whether (a) lateralized alpha suppression would be triggered by a target in a visual search array, and (b) whether this effect could be experimentally dissociated from the N2pc component. The first experiment (shown in the figure below) used a fairly typical N2pc design. Subjects searched for an item of a specific color for a given block of trials. The target color appeared (unpredictably) at one of four locations. Previous research has shown that the N2pc component is primarily present for targets in the lower visual field, and we replicated this result (see ERP waveforms below). We also found that, although alpha-band activity was suppressed over both hemispheres following target presentation, this suppression was greater over the hemisphere contralateral to the target. Remarkably, like the N2pc component, the target-elicited lateralized alpha suppression (TELAS) occurred primarily for targets in the lower visual field. However, the time course of the TELAS was quite different from that of the N2pc. The scalp distribution of the TELAS also appeared to be more posterior than that of the N2pc component (although this was not formally compared).

The second experiment included a crowding manipulation, following up on a previous study in which the N2pc component was found to be largest when flanked by distractors that are at the edge of the crowding range, with a smaller N2pc when the distractors are so close that they prevent perception of the target shape (Bacigalupo & Luck, 2015). We replicated the previous result, but we saw a different pattern with the lateralized alpha suppression: The TELAS effect tended to increase progressively as the flanker distance decreased, with the largest magnitude for the most crowded displays. Thus, the TELAS effect appears to be related to difficulty or effort, whereas the N2pc component appears to be related to whether or not the target is successfully selected.

The bottom line is that visual search targets trigger both an N2pc component and a contralateral suppression of alpha-band EEG oscillations, especially when the targets are in the lower visual field, but the N2pc component and the TELAS effect can also be dissociated, reflecting different mechanisms of attention.

These results are also relevant for the question of whether lateralized alpha effects reflect an increase in alpha in the nontarget hemisphere to suppress information that would otherwise be processed by that hemisphere or, instead, a decrease in alpha in the target hemisphere to enhance the processing of target information. If the TELAS effect reflected processes related to distractors in the hemifield opposite to the target, then we would not expect it to be related to whether the target was in the upper or lower field or whether flankers were near the target item. Thus, the present results are consistent with a role of alpha suppression in increasing the processing of information from the target itself (see also a recent review paper by Josh Foster and Ed Awh).

One interesting side finding: The contralateral positivity that often follows the N2pc component (similar to a Pd component) was clearly present for the upper-field targets. It was difficult to know the amplitude of this component for the lower-field targets given the overlapping N2pc and SPCN components, but the upper-field targets clearly elicited a strong contralateral positivity with little or no N2pc. This provides an interesting dissociation between the post-N2pc contralateral positivity and the N2pc component.

New paper: Using ERPs and alpha oscillations to decode the direction of motion

Bae, G. Y., & Luck, S. J. (2018). Decoding motion direction using the topography of sustained ERPs and alpha oscillations. NeuroImage, 18: 242-255. https://doi.org/10.1016/j.neuroimage.2018.09.029

This is our second paper applying decoding methods to sustained ERPs and alpha-band EEG oscillations. The first one decoded which of 16 orientations was being maintained in working memory. In the new paper, we decoded which of 16 directions of motion was present in random dot kinematograms.

The paradigm is shown in the figure below. During a 1500-ms motion period, 25.6% or 51.2% of the dots moved coherently in one of 16 directions and the remainder moved randomly. After the motion ended, the subject adjusted a green line to match the direction of motion (which they could do quite precisely).

We asked whether we could decode (using machine learning) the precise direction of motion from the scalp distribution of the sustained voltage or alpha-band signal at each moment in time. Decoding the exact direction of motion is very challenging, and chance performance would be only 6.25% correct. During the motion period for the 51.2% coherence level, we were able to decode the direction of motion well above chance on the basis of the sustained ERP voltage (see the bottom right panel of the figure). However, as shown in the bottom left panel, we couldn’t decode the direction of motion on the basis of the alpha-band activity until the report period (during which time attention was presumably focused on the location of the green line).

When the coherence level was only 25.6% (and perception of coherent motion was much more difficult), we could not decode the actual direction of motion above chance. However, we were able to decode the direction of perceived motion (i.e., the direction that the subject reported at the end of the trial).

This study shows that (a) ERPs can be used to decode very subtle stimulus properties, and (b) sustained ERPs and alpha-band oscillations contain different information. In general, alpha-band activity appears to reflect the direction of spatial attention, whereas sustained ERPs contain information about both the direction of attention and the specific feature value being represented.

New Paper: fMRI study of working memory capacity in schizophrenia

Hahn, B., Robinson, B. M., Leonard, C. J., Luck, S. J., & Gold, J. M. (2018). Posterior parietal cortex dysfunction is central to working memory storage and broad cognitive deficits in schizophrenia. The Journal of Neuroscience, 37, 8378–8387. https://doi.org/DOI: https://doi.org/10.1523/JNEUROSCI.0913-18.2018 https://doi.org/10.1523/JNEUROSCI.0913-18.2018.

In several behavioral studies using change detection/localization tasks, we have previously shown that people with schizophrenia (PSZ) exhibit large reductions in visual working memory storage capacity (Kmax). In one large study with 99 PSZ and 77 healthy control subjects (HCS), we found an effect size (Cohen's d) of 1.11, and the degree of Kmax reduction statistically accounted for approximately 40% of the reduction in overall cognitive ability exhibited by PSZ (as measured with the MATRICS Battery). Change detection tasks are much simpler than most working memory tasks, focus on storage rather than manipulation, and can be used across species. Thus, Kmax gives us a measure that is both neurobiologically tractable and strongly related to broad cognitive dysfunction.

In our most recent work, led by Dr. Britta Hahn at the Maryland Psychiatric Research Center, we used fMRI to examine the neuroanatomical substrates of reduced Kmax in PSZ. We took advantage of an approach pioneered by Todd and Marois (2004, Nature), in which a whole-brain analysis is used to find clusters of voxels where the BOLD signal is related to the amount of information actually stored in working memory (K). As shown in the figure below, we found the same areas of posterior parietal cortex (PPC) that were observed by Todd and Marois.

In the left PPC, however, the K-dependent modulation of activity was reduced in PSZ relative to HCS. As shown in the scatterplots, the BOLD signal in this region was strongly related to the number of items being held in working memory (K) in HCS, but the function was essentially flat in PSZ. However, the overall level of activation was just as great in PSZ as in HCS (the Y intercept). The reduced slope was driven mainly by an overactivation in PSZ relative to HCS when relatively little information was being stored in memory. Moreover, the slope was strongly correlated with overall cognitive ability (again measured using the MATRICS Battery), and the degree of slope reduction statistically accounted for over 40% of the reduction in broad cognitive ability in PSZ.

One particularly interesting aspect of these results is that they point to posterior parietal cortex as a potential source of cognitive dysfunction in schizophrenia, whereas most research and theory has focused on prefrontal cortex. Studies with healthy young adults have consistently identified PPC as a major player in working memory capacity and in the ability to divide attention, both of which are strongly impaired in PSZ. We hope that our study motivates more research to examine the potential contribution of the PPC to cognitive dysfunction in schizophrenia.

New Paper: Combined Electrophysiological and Behavioral Evidence for the Suppression of Salient Distractors

Gaspelin, N., & Luck, S. J. (in press). Combined Electrophysiological and Behavioral Evidence for the Suppression of Salient Distractors. Journal of Cognitive Neuroscience.

Evidence that people can suppress salient-but-irrelevant color singletons has come from ERP studies and from behavioral studies.  The ERP studies find that, under appropriate conditions, singleton distractors will elicit a Pd component, a putative electrophysiological signature of suppression (discovered by Hickey, Di Lollo, and McDonald, 2009). The behavioral studies show that processing at the location of the singleton is suppressed below the level of nonsingleton distractors (reviewed by Gaspelin & Luck, 2018).  Are these electrophysiological and behavioral signatures of suppression actually related?

In the present study, Nick Gaspelin and I used an experimental paradigm in which it was possible to assess both the ERP and behavioral measures of suppression.  First, we were able to demonstrate that suppression of the salient singleton distractors was present according to both measures.  Second, we found that these two measures were correlated: participants who should a larger Pd also showed greater behavioral suppression.  

Correlations like these can be difficult to find (and believe).  First, both the ERP and behavioral measures can be noisy, which attenuates the strength of the correlation and reduces power.  Second, spurious correlations are easy to find when there are a lot of possible variables to correlate and relatively small Ns.  A typical ERP session is about 3 hours, so it's difficult to have the kinds of Ns that one might like in a correlational study.  To address these problems, we conducted two experiments.  The first was not well powered to detect a correlation (in part because we had no idea how large the correlation would be, making it difficult to assess the power). We did find a correlation, but we were skeptical because of the small N.  We then used the results of the first experiment to design a second experiment that was optimized and powered to detect the correlation, using an a priori analysis approach developed from the first experiment.  This gave us much more confidence that the correlation was real.

We also included a third experiment that was suggested by the alway-thoughtful John McDonald. As you can see from the image above, the Pd component was quite early in Experiments 1 and 2. Some authors have argued that an early contralateral positivity of this nature is not actually the suppression-related Pd component but instead reflects an automatic salience detection process.  To address this possibility, we simply made the salient singleton the target.  If the early positivity reflects an automatic salience detection process, then it should be present whether the singleton is a distractor or a target.  However, if it reflects a task-dependent suppression mechanism, then it should be eliminated when subjects are trying to focus attention onto the singleton. We found that most of this early positivity was eliminated when the singleton was the target. The very earliest part (before 150 ms) was still present when the singleton was the target, but most of the effect was present only when the singleton was a to-be-ignored distractor. In other words, the positivity was not driven by salience per se, but occurred primarily when the task required suppressing the singleton.  This demonstrates very clearly that the suppression-related Pd component can appear as early as 150 ms when elicited by a highly salient (but irrelevant) singleton.

Electrophysiological Evidence for Spatial Hyperfocusing in Schizophrenia

Kreither, J., Lopez-Calderon, J., Leonard, C. J., Robinson, B. M., Ruffle, A., Hahn, B., Gold, J. M., & Luck, S. J. (2017). Electrophysiological Evidence for Spatial Hyperfocusing in Schizophrenia. The Journal of Neuroscience, 37, 3813-3823.

This paper from last spring describes new evidence for our hyperfocusing theory of cognitive dysfunction in schizophrenia.  Remarkably, we found that people with schizophrenia were actually better able to focus centrally and filter peripheral distractors than were control subjects. Under the right conditions, we even observed a (slightly) larger P3 wave in patients than in controls.
 

New Paper: Visual short-term memory guides infants’ visual attention

Mitsven, S. G., Cantrell, L. M., Luck, S. J., & Oakes, L. M. (in press). Visual short-term memory guides infants’ visual attention. Cognition. https://doi.org/10.1016/j.cognition.2018.04.016 (Freely available until June 14, 2018 at https://authors.elsevier.com/a/1Wxvg2Hx2bbMQ)

This new papers shows that visual short-term memory guides attention in infants. Whereas adults orient toward items matching the contents of VSTM, infants orient toward non-matching items.