This study explores two English classes taught in English at a four-year university to investigate the effectiveness of using the Automated Essay Scoring (AES) system on the improvement of essay writing abilities of both high level and low level EFL students. The instruments involved in this study include students` essays, Criterion, an automated essay scoring system, and a survey. In order to examine the influence of the feedback provided by the Criterion system on the participants` essays, their first and second draft were each evaluated and given holistic scores by the teacher-researcher. Following the evaluation, the scores on the first draft and the revised draft were compared. According to the statistical results, both high and low level students all showed overall improvement in their writing after using Criterion. Following an examination of their writings, a questionnaire was used to measure the strengths and weaknesses of using Criterion. Additionally, the students` perception towards face-to-face human feedback, which was provided through an individual writing conference, was investigated. The findings of this study can offer pedagogical suggestions for incorporating AES in L2 writing class and implications for future research in EFL context.