Since about 1962, researchers have explored the effectiveness of educational software in the schools under various conditions, and many have concluded that computer assisted instruction (CAI) is effective. Many of the studies reviewed to support this conclusion, however, had design flaws. Reviews of educational software effectiveness studies have noted four major research design weaknesses: (1) small sample sizes; (2) lack of identified criteria for determining "quality" software; (3) inappropriately used statistics; and (4) inadequate time allocated to conducting the study. Because the actual studies have design flaws, these research design weaknesses will be reflected in the results of the studies and even compounded should meta-analyses using these studies be performed. These major design weaknesses are discussed and selected studies are reviewed to illustrate them. Practical solutions to the design flaws are proposed, and the effects of the design flaws on meta-analyses are outlined. An appended table displays the 16 randomly-selected studies used to illustrate the design flaws together with data for each one on sample size, duration of the study, the software used, and the statistical techniques used to analyze the data. (45 references) (GL)