This paper presents the results of a study on Calculus students' use of web-based homework. We collected data from students' web-based homework usage, such as their grades, time spent, number of attempts used to solve problems, and problem solutions. We also collected grades on in-class quizzes, which were given the day after homework had been submitted, and included problems identical or similar to those on the homework. Our results showed that student performance in web-based assignment does not reflect their actual knowledge of the course content. Most of the students received very high grades on the web-based homework, but half of them scored either a D or F on the quizzes. Overwhelming majority of the students used at most two attempts to find the correct answers to the homework problems. Students who scored D or F on the quizzes spent the least amount of time on the homework, but their homework grades were no different from their high performing peers. The results indicated that many students might complete their homework with a great deal of external assistance, but without understanding the concepts being assessed. The paper will also discuss how a minor intervention on students' usage of web-based homework can make sizeable impact on their performances on in-class quizzes and exams. Students in the treatment group outperformed their control group peers on the quizzes and exams, despite scoring significantly lower in web-based assignment.