Portals have become indispensable for organizations of all types trying to establish themselves on the Web. Unfortunately, there have only been a few evaluative studies of portal software and even fewer of open source portal software. This study aims to add to the available literature in this important area by proposing and testing a checklist for open source portal software evaluation. The study describes the experience of evaluating open source portal software for developing a portal for cultural information exchange. Using the checklist, four candidate open source portal software packages, namely Jetspeed, uPortal, Liferay Portal and eXo Platform were evaluated. Results indicated that uPortal was the best performer followed by eXo platform, Liferay and Jetspeed. The present work represents a first step in the development of an exhaustive evaluation tool for portals. While the focus of this paper is entirely on open source software, the checklist can also be readily applied to commercial products. (Contains 1 table.)