English rhythm is related to contrast between the stressed and the unstressed in duration structure. In native English speech, in general, an intra-speaker average duration of stressed syllable is longer than that of syllable as a whole. On the contrary, that of unstressed syllable is shorter than that of syllable as a whole. In the previous paper by the present author, it was reported that stressed syllable durations of speech of learners tend not to be lengthened enough, and an unstressed ones tend not to be shortened enough as compared with that of native speakers (Nakamura, 2010). For these reason, learner speech does not have such a high ratio of intra-speaker average durations of stressed to unstressed syllables as native speech. Since this lower ratio affected subjective evaluation, a correlation coefficient between the ratios and subjective evaluation scores given by English language teachers was observed to be 0.48. In this paper, an indicator, which demonstrates more adequately a duration contrast between the stressed and the unstressed, was investigated, for the purpose of increasing the correlation with subjective evaluation. A rhythm unit was defined here as a stressed syllable connected to the preceding and succeeding unstressed syllables. Then, a value based on a ratio of stressed to unstressed syllable durations in the rhythm unit was treated as an indicator to represent learner characteristics. As a result, a correlation coefficient of the indicators with subjective evaluation scores was increased to 0.65. A substantial part of mechanism in subjective evaluation of rhythm in English speech was revealed and became possible to simulate reasonably by objective evaluation.