We explore the following two cognitive questions regarding crosslinguistic variation in lexical semantic systems: Why are some linguistic categories--that is, the associations between a term and a portion of the semantic space--harder to learn than others? How does learning a language-specific set of lexical categories affect processing in that semantic domain? Using a computational word-learner, and the domain of color as a testbed, we investigate these questions by modeling both child acquisition of color terms and adult behavior on a non-verbal color discrimination task. A further goal is to test an approach to lexical semantic representation based on the principle that the more languages label any two situations with the same word, the more conceptually similar those two situations are. We compare such a crosslinguistically based semantic space to one based on perceptual similarity. Our computational model suggests a mechanistic explanation for the interplay between term frequency and the semantic closeness of learned categories in developmental error patterns for color terms. Our model also indicates how linguistic relativity effects could arise from an acquisition mechanism that yields language-specific topologies for the same semantic domain. Moreover, we find that the crosslinguistically inspired semantic space supports these results at least as well as--and in some aspects better than--the purely perceptual one, thus confirming our approach as a practical and principled method for lexical semantic representation in cognitive modeling.