Recent research on individual differences in object cognition (OC) focused on determining how objects group together, and what type of processing lies behind the clusters--a single domain-general or multiple domain-specific processes. The expertise hypothesis suggests that all object categories are processed by the same mechanism that is responsible for differentiating visually similar objects. This ability is expected to be more specifically expressed in processing objects for which people have higher expertise. The domain-specificity hypothesis postulates that different object categories, for example, living versus nonliving objects, are processed employing different mechanisms. In the present study we aimed to study (a) how multiple objects group together in terms of individual differences; (b) the expertise hypothesis based on up-to-date statistical methods of modeling individual differences; (c) whether task difficulty influences the structure of individual differences in OC. We applied a memory task to N = 186 participants, by using multiple, theoretically chosen object categories divided into five groups--living/mobile (fish, butterflies), living/immobile (flowers, leaves), nonliving/mobile (cars, motorbikes), nonliving/immobile (houses, chairs) and faces--and report three main findings. (a) In terms of individual differences, the factor space of OC can be accounted by three factors--general OC factor along with a specific Vehicle and Face factor; (b) there is no clear evidence for the expertise hypothesis; and (c) task difficulty does not influence the OC structure. We conclude that domain-specific mechanisms in object processing cannot be dismissed.