Applications

In our prior research on multimedia learning, we collected a dataset of recordings (mp3 files) for 30 targeted Standard Chinese words, each with at least 100 recordings from different learners. Mandarin Chinese is a tonal language, which means that the meaning of a word can change depending on the tone in which it is spoken. There are four tones in Mandarin Chinese: (1) high level, (2) high rising crescendo, (3) low falling diminuendo with glottal friction (with an extra rise from low to high when final), and (4) falling diminuendo. Each recording in our dataset has two binary labels: one for sound correctness and the other for tone correctness (the tone label is irrelevant if the sound is incorrect). The purpose of this project is to develop a speech recognition model that provides feedback on both sound and tone correctness in Mandarin Chinese for our vocabulary dataset. If the developed model can achieve similar results to human teachers in evaluating speech performance, it can potentially be integrated into online courses to save teachers' time and provide instant feedback.