The present study investigated students' perceptions of the new criterion-referenced assessment on the English section of the College Scholastic Ability Test (CSAT). A total of 285 college freshmen, who took the 2018 CSAT, completed a questionnaire comprised of eighteen close-ended questions and four open-ended questions. Both quantitative and qualitative data were analysed to answer the research questions. The results of the study showed that, overall, students tended to think favorably of the new policy since it has alleviated the excessive pressure to achieve high scores in English, which in turn has motivated them to set their own learning goals and made self-directed learning possible. However, students with negative attitudes remarked that the new policy has rendered English worthless on the CSAT, causing them to put little emphasis on studying English. Moreover, the new policy was negatively perceived by students who are good at English because they could no longer leverage their superior English skills to excel compared to others at the CSAT. Students who were neutral expressed their skepticism about the much-expected change this new policy was supposed to bring to the current educational setting, mainly because it was the scoring system that was modified, not the content of evaluation. The results further indicated that high school English classrooms are still dominated by rote learning of grammar rules and reading practice, with a heavy emphasis on CSAT preparation. We argue that the new policy needs careful readjustment in order for it to be successfully implemented, thus achieving the goal of normalizing public education and creating communicative classrooms in secondary schools in Korea.