Selective attention operates in visual cortex by facilitating processing of selected stimuli and by filtering out unwanted information from nearby distracters over circumscribed regions of visual space. The neural representation of unattended stimuli outside this focus of attention is less well understood. We studied the neural fate of unattended stimuli using functional magnetic resonance imaging by dissociating the activity evoked by attended (target) stimuli presented to the periphery of a visual hemifield and unattended (distracter) stimuli presented simultaneously to a corresponding location of the contralateral hemifield. Subjects covertly directed attention to a series of target stimuli and performed either a low or a high attentional-load search task on a stream of otherwise identical stimuli. With this task, target-search-related activity increased with increasing attentional load, whereas distracter-related activity decreased with increasing load in areas V4 and TEO but not in early areas V1 and V2. This finding presents evidence for a load-dependent push-pull mechanism of selective attention that operates over large portions of the visual field at intermediate processing stages. This mechanism appeared to be controlled by a distributed frontoparietal network of brain areas that reflected processes related to target selection during spatially directed attention.