Scoring Overview
Comprehensive overview of scoring methodologies and metrics used in the Speech Assessment API
The API evaluates a speaker across Pronunciation, Fluency, Grammar, and Vocabulary, with scores for each area contributing to an overall proficiency assessment.
Key Areas of Assessment
1. Pronunciation
Calculation: The scoring model is built around intelligibility, not imitation of a specific native accent. It assesses both segmental (individual sounds/phonemes) and it provides scores down to the phoneme level, generated in partnership with Oxford Languages and incorporating IPA (International Phonetic Alphabet) output. The data collected to train this model is comprised of ~80% native speaker data and ~20% ESL or accented data, giving the model a robustness to handle speakers from all origins.
2. Fluency
Calculation: Fluency is evaluated by measuring attributes related to the naturalness and continuity of speech. This includes the speaker's rate of speech and rate of speech variation, as well as the number and location of filler words and disfluent pauses.
3. Grammar
Calculation: The assessment involves analyzing the grammatical accuracy and complexity of the language used. It can detect specific grammar mistakes and provide suggested corrections. In the Unscripted API, it assesses the diversity of sentence structures (complexity) and lexical accuracy.
4. Vocabulary
Calculation: Vocabulary is scored based on the effective usage of words, their appropriateness within the given context, and the overall lexical complexity. The system can also detect the use of idiomatic expressions.
IELTS and CEFR Alignment
To ensure the API's scores are globally recognized and highly accurate for high-stakes testing, Language Confidence used a rigorous process involving professional examiners.
We utilized professional IELTS assessors to conduct a double-blind test. In this process, the assessors independently scored a vast number of spoken English samples. These human-rated scores were then used to train the AI models to accurately predict the scores an examiner would give. This method allowed us to map the AI's technical scores to the official 9-band IELTS scoring scale. Following this, the resulting IELTS scores were aligned to the Common European Framework of Reference for Languages (CEFR), which classifies proficiency into six levels (A1 to C2), providing a clear and globally accepted benchmark of the learner's language ability.