Selecting visual information from cluttered real-world scenes involves the matching of visual input to the observer's attentional set--an internal representation of objects that are relevant for current behavioral goals. When goals change, a new attentional set needs to be instantiated, requiring the suppression of the previous set to prevent distraction by objects that are no longer relevant. In the present fMRI study, we investigated how such suppression is implemented at the neural level. We measured human brain activity in response to natural scene photographs that could contain objects from (1) a currently relevant (target) category, (2) a previously but not presently relevant (distracter) category, and/or (3) a never relevant (neutral) category. Across conditions, multivoxel response patterns in object-selective cortex carried information about objects present in the scenes. However, this information strongly depended on the task relevance of the objects. As expected, information about the target category was significantly increased relative to the neutral category, indicating top-down enhancement of task-relevant information. Importantly, information about the distracter category was significantly reduced relative to the neutral category, indicating that the processing of previously relevant objects was suppressed. Such active suppression at the level of high-order visual cortex may serve to prevent the erroneous selection of, or interference from, objects that are no longer relevant to ongoing behavior. We conclude that the enhancement of relevant information and the suppression of distracting information both contribute to the efficient selection of visual information from cluttered real-world scenes.