The composite face paradigm (Young, Hellawell, & Hay, 1987) is widely used to demonstrate holistic perception of faces (Rossion, 2013). In the paradigm, parts from different faces (usually the top and bottom halves) are recombined. The principal criterion for holistic perception is that responses involving the component parts of composites in which the parts are aligned into a face-like configuration are slower and less accurate than responses to the same parts in a misaligned (not face-like) format. This is often taken as evidence that seeing a whole face in the aligned condition interferes with perceiving its separate parts, but it remains unclear to what extent the composite face effect also reflects contributions from other potential sources of interference. We present a new variant of the paradigm involving composites created from top and bottom parts of familiar faces drawn from orthogonal social categories of gender and occupation. This allows us to examine the contributions of differences in relatively visual properties (gender) or relatively semantic properties (occupation) to composite interference and to measure whether variation in a task-irrelevant category (e.g., differences in gender across the parts of the composite when the task is to categorize the occupation of one of the parts) will influence the size of the composite effect. Our findings show that the composite face effect can be modulated by task-irrelevant social categories and that this interference is primarily visual in nature because the influence of face gender is more direct and more consistent than the influence of occupation.