General Discussion

In this thesis I investigated inference about absence in visual perception, and its relation with self-modeling and default-mode reasoning. In chapters 1 and 2 I focused on visual search, and asked what people know about their visual search behaviour, and how this knowledge related to their ability to efficiently terminate a search in the absence of a target. In Chapter 3 I used reverse correlation to ask what information is incorporated into confidence judgments in decisions about the presence and absence of a stimulus. Then, in chapter 4 I used functional imaging to compare the neural processes governing metacognitive evaluation of decisions about stimulus type and stimulus presence or absence. Finally, in chapter 5 I borrowed ideas from the visual search literature to ask at what cognitive level does the metacognitive asymmetry between judgments of presence and absence emerge.

In what follows I evaluate my original proposal, that inference about absence critically relies on self-knowledge, in light of my findings. Specifically, I list observations that don’t fit with this idea. Before concluding, I critically review two approaches to inference about absence that obviate the need in self-modelling, and briefly describe two directions for future research that build on and extend my work here.

What I didn’t find

The theoretical proposal put forward in this thesis is that inference about absence is unique in that it requires relying on a self-model. In previous chapters I tried to make sense of my data in light of this proposal. However, some patterns that I expected to find were missing in the data, and some patterns that I did find were difficult to reconcile with this overarching idea. In the following, and following Charles Darwin’s advice to make an effort to remember observations that don’t fit with one’s theory (Darwin & Darwin, 1958, p. 123), I list some things that I thought I should find, but didn’t.

Chapter 1: no correlation with explicit metacognition

In Chapter 1, I show that participants can immediately recognize the absence of a salient target in a display, and that they can do so even before having experience with the task. My interpretation of the results abscribes this ‘absence pop-out’ to pre-existing metacognitive knowledge of the parallel nature of feature search. I further show that even participants whose explicit metacognitive reports show no insight into the parallel nature of feature search exhibit the same pattern of search time in target-absent trials - a finding that we interpret as indicating a dissociation between explicit and implicit metacognitive knowledge.

A much stronger support for the proposal I put forward here would be a correlation between explicit metacognitive knowledge of search efficiency and search slopes in target-absent trials, such that participants who rate feature searches as easier also quit them earlier in the absence of a target. Instead, I found no correlation between explicit metacognition and behaviour on target-absent trials. A full dissociation between implicit and explicit metacognition is one possible interpretation for my results. An alternative interpretation is that the immediate recognition of target absence in these first trials is not at all dependent on metacognitive knowledge - explicit or implicit. In the discussion of Chapter 1 I list some alternative accounts, such as immediate recognition of absence via ensemble perception, and explain how they depend on implicit metacognitive knowledge, for example in the form of a firing threshold on neurons in the visual pathway. One concern is that a notion of self-modelling that encompasses the firing thresholds of visual neurons is too permissive to be scientifically useful. The visual system translates incoming sensory signals into beliefs about the external world. To do that, neurons in associative visual areas implicitly represent a likelihood function going from world states into firing patterns, and invert this function to approximate a representation of the world state, given observed firing patterns in primary visual areas. This likelihood function is a form of self-knowledge, in the sense that it is knowledge about how the brain responds to incoming signals. But its importance is not specific to the representation of absence (although representations of absence may be more sensitive to these internal models compared with other decisions).

In the pre-registration documents for Experiments 1 and 2 I focused on a contrast between blocks 1 and 3 (before and after experience with target-present trials), with the hypothesis that task experience will affect the search slope for target-absent trials. A learning effect between blocks 1 and 3 would have allowed me to further ask how generalizable this new knowledge is, and for how long is it retained in the system, giving us a better hold of this mental self-model. In a later section I expand on how focusing on model failures (such as mismatches between target-present and target-absent search efficiency) can be useful for understanding the structure of the mental self model.

Chapter 3: no effect of confidence in signal presence

In Chapter 3, I asked what drives confidence in decisions about target absence. Since decisions about target absence are based on the absence of perceptual evidence, I hypothesized that subjective confidence in such decisions may rely on other factors. For example, if participants are using a counterfactual heuristic, their confidence in previous ‘target present’ trials may inform their confidence in ‘target absent’ decisions (“When a target was present it was highly visible, so I would have seen the target if it were present”). This effect should be stronger than the effect of previous confidence in target-absence decisions on confidence in presence, because confidence in presence can be based on perceptual evidence. To test this, I looked at the correlation between confidence in absence and confidence in the last target-presence decision. This correlation was not significantly different from 0 in Experiment 1 (\(t(9) = 0.86\), \(p = .412\)). In Experiment 2, this correlation was significantly higher than 0 at the group level (\(M = 0.17\), 95% CI \([0.10\), \(0.23]\), \(t(100) = 5.37\), \(p < .001\)), but a similarly high correlation between confidence in ‘yes’ responses and in the last decision about target absence suggests that this was not specific to decisions about absence (\(\Delta M = 0.01\), 95% CI \([-0.07\), \(0.09]\), \(t(197.32) = 0.25\), \(p = .803\)).

Similarly, a counterfactual heuristic predicts lower confidence in absence when local target prevalence (for example, the number of targets presented in the last 5 trials) is low (Hsu, Horng, Griffiths, & Chater, 2017). To see this, compare two participants that are presented with random noise: one that hasn’t seen a target for 4 trials in a row, and one that just saw a target in the previous trial. The first participant may doubt their perceptual sensitivity and give a low confidence rating, but the second can be more confident that they would not have missed a target. Contrary to this prediction, local target prevalence had no effect on confidence in ‘no’ responses in Experiment 2 (quantified as the distance in number of trials from the last encounter with a target; \(t(100) = -1.22\), \(p = .224\)). In Experiment 1, a significant correlation in the opposite direction was observed, such that participants were more confident in their ‘no’ responses when a target hasn’t been observed for a longer series of trials (\(t(9) = 3.11\), \(p = .013\)). Overall, we found no evidence for higher susceptibility of confidence in absence to confidence and stimulus prevalence in previous trials.

Chapter 4: only minor differences in brain activity between inference about absence and presence

In Chapter 4, I compared brain activity in discrimination and detection. Within detection, we compared decisions about signal presence and absence. Our pre-registration document largely focused on the behavioural differences between ‘yes’ and ‘no’ responses, and their possible neural underpinning, with a focus on the lateral prefrontal cortex and regions that have been associated with counterfactual reasoning [“I would have seen it if it were there”; Boorman, Behrens, Woolrich, & Rushworth (2009)]. To my surprise, I found no significant difference in overall activity between ‘yes’ and ‘no’ responses (for an uncorrected map, see here). A significant difference in the parametric modulation of confidence was found not in our pre-defined regions of interests, but in the right temporo-parietal junction (rTPJ).

Given the reliable behavioural differences in reaction time, overall confidence, and metacognitive sensitivity, a similar profile of BOLD activation for ‘yes’ and ‘no’ responses was unexpected. In language comprehension, for example, the processing of negation shows distinct neurobiological markers that are overlapping with those of response inhibition (Papeo & Vega, 2020). In visual search, the right lateral prefrontal cortex was more engaged in target-absent trials (Vallesi, 2014), and the right temporo-parietal junction showed differential activation in visual search hit and miss trials (Shulman, Astafiev, McAvoy, d’Avossa, & Corbetta, 2007). Surprisingly, however, despite the robust behavioural differences, BOLD activations for ‘yes’ and ‘no’ responses gave rise to indistinguishable baseline activation, differing only in the modulation of confidence. When focusing on the frontopolar cortex, confidence modulation was different between detection and discrimination, but remarkably similar for detection ‘yes’ and ‘no’ responses. Again, this is in contrast to my original hypothesis, that counterfactual reasoning should play a major role in decisions about target absence, more so than in decisions about target presence.

Chapter 5: no metacognitive asymmetry between default-complying and default-violating signals

In Chapter 5, I focused on three behavioural asymmetries between detection ‘yes’ (stimulus present) and ‘no’ (stimulus absent) responses: in reaction time, global confidence, and metacognitive sensitivity. Using stimulus pairs that generate asymmetries in visual search, I asked whether detection-like asymmetries would emerge in discrimination tasks that can be described as the detection of sub-stimulus presences: local stimulus features (such as the line that distinguishes a Q from an O), global features (such as the presence or absence of curvature), and expectation violations (such as the presence or absence of letter inversion). My reasoning was the following: if presence/absence asymmetries emerge because participants assume absence as default unless they have evidence for presence, similar asymmetries should emerge for other things that we take as default (e.g., letters are not mirrored, objects are not floating in space). This default-reasoning framework also provided a conceptual link to metacognitive asymmetries in recognition memory: there also, participants assume as default that an item is new (i.e., hasn’t been presented in the study phase), unless they have evidence for that it is old.

I found no evidence for a metacognitive asymmetry between default-complying and default-violating signals. Furthermore, in Experiment 6 participants were slower and gave lower confidence ratings when they reported seeing a flipped letter: a significant finding that stood in direct contrast to the default-reasoning proposal. We interpreted our findings as placing the metacognitive asymmetry for detection judgments at lower levels of the cognitive hierarchy, potentially in early visual processing.

A similar behavioural profile for the identification of default-complying and default violating stimuli does not stand in contrast to the proposal that inference about absence does involve a default-reasoning component, and that as a result it requires reliance on a mental self model (see Section 0.2.1). Nevertheless, finding a metacognitive asymmetry for expectation violations would have provided strong support for this framework, which we did not get from our findings.

Inference about absence without self-modelling

In this thesis, I focused on the role of self-modelling in inference about absence. My investigation was guided by a conceptual analysis, based on default-reasoning (Section 0.2). In Chapter 1 I also considered alternative accounts of inference about absence in visual search, where decisions about the absence of a target object are guided not by counterfactual reasoning based on a self-model, but by a model-free heuristic based on success in previous trials, or by an immediate perception of ensemble statistics of a visual scene. In the following, I describe two approaches to inference about absence that do not involve self-modelling: patch-leaving heuristics in foraging, and philosophical accounts of absence perception. I unpack some of their strengths and limitations.

Patch-leaving in foraging

Chapter 4 opens with a foraging example: an agent deciding whether a bush bears ripe fruit or not. In this example, detecting berries is an instance of inference about presence, and deciding that a bush bears no fruit is an instance of inference about absence. Since evidence can only be available for the presence but not for the absence of berries, decisions about the absence of berries must rely on some form of counterfactual reasoning (“I would be seeing the berries if they were present”), that in turn relies on a self-model. However, when considering the behaviour of foragers, explicitly deciding that a bush bears no ripe fruit is mostly unnecessary. Instead, a decision to move to the next bush can be motivated by an explicit or implicit belief that leaving the current bush will be more rewarding than staying.

Heuristics for approximating when is the optimal time to leave a patch in search for other sources of food have been formalised and tested against animal behaviour. For example, in Charnov’s Marginal Value Theorem [MVT; Charnov (1976)], a decision to leave the current patch (a spatially defined source of food, like a bush of berries) is optimal when the instantanous rate of return (e.g., berries found per minute) falls below the mean rate for the environment as a whole. Thus, MVS predicts that agents would exploit patches for longer under conditions in which patches yield less returns on average, or in which patches are physically farther away from each other. Foraging behaviour that is consistent with these qualitative predictions has been observed in birds (R. J. Cowie, 1977; Krebs, Ryan, & Charnov, 1974), armadillos and guinea pigs (Cassini, Kacelnik, & Segura, 1990). Similarly, when searching for an unspecified number of visual targets in an array (e.g., gas stations in satellite images), online participants set their giving up times in accordance with MVT (Ehinger & Wolfe, 2016).

As MVS shows, a decision to move on to the next bush can be made without a self-model (or any model other than knowledge of basic properties of the environment). Importantly, however, it does not show that absence can be inferred without a self-model, but that patch-leaving is not necessarily an instance of inference about absence. This is because search in natural foraging tasks is not exhaustive: the task is rarely to find all berries on a bush, but to find as many berries as possible, considering the cost of search itself in energy and exposure to threats.

In ecological settings outside of controlled experiments, instances of exhaustive search are usually ones where the cost of missing a target are considerable, such as when scanning a lake for predators before approaching to drink, or checking a memograpm for potential indicators of a tumor. In these cases, basing decisions on inference about absence is crucial, rendering MVS-like approaches dangerous. Still, patch-leaving algorithms reveal that for many behavioural functions, including foraging for food, strict inference about absence is not necessary.

Direct perception

According to some contemporary philosophers absence need not be inferred because it is directly perceived. For example, philosopher Anna Farennikova explains the perception of absence as a perception of a mismatch between sensory input and expectations of presence: “The phenomenology of absence is the experience of incongruity” (Farennikova, 2013, 2015). Farennikova presents the following example of absence perception:

“You’ve been working on your laptop in the cafe for a few hours and have decided to take a break. You step outside, leaving your laptop temporarily unattended on the table. After a few minutes, you walk back inside. Your eyes fall upon the table. The laptop is gone! This experience has striking phenomenology. You do not infer that the laptop is missing through reasoning; you have an immediate impression of its absence.”

According to this account, the absence of a laptop is directly perceived, instantaneously and without any conscious effort, as a mismatch of sensory input relative to a perceptual template of a laptop on a table. This seems to contrast with the account presented here in several ways.

First, according to this account, absence is perceived, whereas in the account I defend it is inferred. On closer inspection, this is not in fact a point of disagreement. Perception is widely held to involve, and depend on, inference from noisy sensory data about unknown world states (Friston, 2010; Gershman, Vul, & Tenenbaum, 2012; Helmholtz, 1948). Therefore, that absence is inferred does not mean that is cannot also be perceived. Indeed, Gow (2021) proposes that absence is perceived via “intellectual seeming”: a form of inference that results not in beliefs or judgments, but in perceptual states.

The next point of potential disagreement concerns what knowledge is necessary to infer absence. According to the template-mismatch account, any sensory mismatch relative to an expected template immediately results in a perception of, or inference about, absence. In the account defended here, absence can only be inferred when one believes that they would have perceived the missing object if it were present. Consider, for example, returning from a break and finding a waiter occluding some of the table. As in Farennikova’s example, the sensory input is not consistent with your expectation to find your laptop on the table, but this time you are not inferring that it is absent, because you know that the waiter might be occluding it. Similarly, if you believe the laptop would be difficult to see (for example, if your forgot your glasses inside), you will not infer absence until you check the table more closely. In both cases, inference about absence depends on much more than a comparison to a sensory template: it depends on sophisticated inference based on sensory and metacognitive cues. In support of this more elaborate account of inference about absence, in Chapter 1 I show that participants take longer to infer absence in displays that make finding the target more difficult.

In defense of a template-mismatch account, one may argue that the difference between seeing the absence of a laptop in Farennikova’s example and not seeing it in my occluding-waiter or missing-glasses variants is not in post-perceptual inferences, but in the sensory templates against which the sensory input is compared. For example, my sensory template of a laptop on a table may itself become less clear when I know the lighting has changes. Critically, this flexible updating of sensory templates based on changing environmental and internal conditions is a model-based process, one that involves not only modelling of objects and other agents, but of my own perception and attention too.

Finally, in support of the template-mismatch account, Farennikova mentions that many experiences of absence feel instantaneous and lacking in conscious effort, indicating some automaticity of absence processing. However, introspection can be misleading. Using different tasks and stimuli, in Chapters 1, 4, 3 and 5 I show that inference about absence is significantly slower than inference about presence or stimulus type, even when controlling for response requirements (Chapter 3), and when presenting the decision as a discrimination task between two stimuli (Chapter 5, Exp. 7). The difference in response times between inferences about presence and absence ranged from 46 ms in Chapter 4 to 124 ms in Chapter 5. These are neurally and psychologically significant differences, that are comparable in size to congruency effects in Stroop Flanker tasks, and to perceptual priming effects (Semmelmann & Weigelt, 2017). This strongly suggest that, at least in the context of a detection task, inference about absence is slower than inference about presence.

To conclude, a template-mismatch account of inference about absence as the one put forward by Farennikova (2013) either includes implicit self- and world-modelling in the generation of context-sensitive templates, or fails to account for the flexibility with which subjects infer absence in dynamic environments and internal conditions.

Future directions

As the list in the section “what I didn’t find” makes clear, the investigation of a link between inference about absence and self-modeling is far from complete. The data are telling us that there is more to the story than default reasoning and reliance on self knowledge, or that my particular formulation of these concepts is lacking. More work is needed to further investigate the mechanisms that allow humans to form explicit representation of absence based on the absence of evidence, and how this relates to their generative models of their own perception and cognition. In the following, I list two avenues for future studies: leveraging failures of a self-model, and using inference about absence in more naturalistic settings.

Failures of a self-model

The structure of models is best revealed when they fail to faithfully represent their object. For example, a failure of the body-schema to correctly identify the position of one’s arm following synchronous touch (Botvinick & Cohen, 1998) has advanced our understanding of how humans represent their own bodies, and how they update these representations based on sensory evidence (D. Cowie, Makin, & Bremner, 2013; Kammers, Vignemont, Verhagen, & Dijkerman, 2009; Tsakiris & Haggard, 2005). Systematic errors in participants’ predictions of the physical effects of collisions informed theories of people’s intuitive understanding of physics (A. N. Sanborn, Mansinghka, & Griffiths, 2013). Lastly, children’s failure to ascribe a false-belief to an agent provided cognitive scientists with an experimental handle on the development of a Theory of Mind between the ages of 2.5 and 4 (Gopnik & Wellman, 1992).

Similarly, a scientific investigation of the mental self model can make use of cases in which this model fails to accurately represent the mental self. In the introduction, I provided two examples for misrepresentations that were revealed by suboptimal inference about absence: in near-threshold detection, participants overestimate the effect of eccentricity on perceptual sensitivity (Odegaard, Chang, Lau, & Cheung, 2018; Solovey, Graney, & Lau, 2015), and in visual search participants fail to fully represent the search advantage for unfamiliar targets [Wang, Cavanagh, & Green (1994); Zhang & Onyper (2020); but see Chapter 2 for evidence that intuitive theories of visual search are sensitive to this advantage, at least to some extent]. Focusing on these mismatches between the mental self (including perception, attention, and higher cognition) and the mental self-model has the potential to uncover the boundaries of the self model, the simplifications it makes, and its computational building blocks.

For example, large-scale online data collection with adaptive (Cavagnaro, Pitt, & Myung, 2011; He, Chen, & Li, 2020) and sequential designs (Hsu, Martin, Sanborn, & Griffiths, 2019; Langlois, Jacoby, Suchow, & Griffiths, 2021; A. Sanborn & Griffiths, 2008) now affords to identify stimulus features that affect visual search in target-present trials more than in target-absent trials or vice versa, indicating a mismatch between visual attention and participants’ model of their own attention. Beside directly contributing to our knowledge of the contents and structure of the mental self model (e.g., the mental self model is better calibrated for basic visual features than for experience-based ones), this systematic mapping of model misrepresentations can then be used to ask how is the mental self model expanded and adjusted based on experience, by measuring how these biases change in light of task experience. Collecting data from multiple subjects on multiple test items has been successful in investigating factors that contribute to image memorability, to subjective memorability scores, and to mismatches between the two (Isola, Xiao, Parikh, Torralba, & Oliva, 2013; Rust & Mehrpour, 2020).

Inference about asbence in multi-dimensional and hierarchical representational spaces

in order to achieve high levels of experimental control, experiments in this thesis have mostly used simple stimuli: random dot kinematograms, visual gratings, flickering patches, and simple geometrical shapes. Using low-level visual stimuli has made is possible to precisely control the input to participants’ perceptual system (e.g., in the form of signal to noise ratio). However, restricting our focus to the representation of presence and absence in low-dimensional representational spaces has potentially masked some crucial properties of inference about absence that are revealed in high-dimensional, hierarchically structured representational spaces (see Section 0.2.1 in the introduction).

Consider, for example, the results of the imaging experiment in Chapter 4, where to our surprise we found no univariate difference in activation between detection ‘yes’ and ‘no’ responses. One explanation is that our limited stimulus set (noise and right- and left-tilted gratings embedded in noise) has allowed subjects to form active representations of stimulus absence (in this case, noise), bypassing the need for counterfactual reasoning for inferring absence. In Chapter 3, Experiment 2, reverse correlation analysis revealed an active accumulation of evidence for absence (in the form of overall darkness).

In future studies, using high-dimensional stimuli such as photographs of animals (Kellij, Fahrenfort, Lau, Peters, & Odegaard, 2018), faces, or words (Kay & Yeatman, 2017) may render it impossible to accumulate evidence for absence, instead pushing participants to adopt a counterfacutal reasoning heuristic. Combinatorically, the number of all possible stimuli is exponential in the number of dimensions, making an exhaustive search impossible in high-dimensional stimulus spaces. In such highly asymmetric spaces, a counterfactual heuristic (would I have seen a target stimulus if it were present?) may be more advantageous.

Conclusion

In five studies I investigated inference about absence, self-modeling, and the relation between the two. Using visual search and perceptual detection and discrimination, I asked what separates decisions about the presence of a signal from decisions about signal absence, and to what extent is the difference between these two types of decisions related to the reliance of the latter on counterfactual reasoning on the basis of a self-model. Overall, I observed mixed results for and against this proposal. In visual search, participants’ rich and accurate explicit theory of their own visual search behaviour played no role in deciding that a target was absent from a display. In near-threshold detection, a parametric modulation of confidence on brain activation was similar for decisions about the presence and the absence of a stimulus, except for in the right temporoparietal junction - a brain region that has been associated with monitoring one’s own attention, as well as the attention of other agents. Finally, discrimination tasks with stimuli that varied in the presence or absence of a local feature, a global feature, or a default violation, allowed a dissociation of the factors that independently contribute to behavioural differences between decisions about the presence and absence of a stimulus. Overall, my findings suggest that decisions about presence and absence differ in more than one way. Self-modeling and counterfactual thinking may account for some of these differences, but not for all of them.