Applying machine learning in recognizing vietnamese speech
APPLYING MACHINE LEARNING
IN RECOGNIZING VIETNAMESE SPEECH
SVTH: Nguyễn Duy Thái Sơn
GVHD: ThS Đặng Đình Quân
Tó m tắt- Nhận diện giọng nó i là một lĩnh vực mang lại nhiều ứng dụng hữu í ch và thực tiễn.
Hơn nữa, những ứng dụng của chúng đang ngày một trở nên được ưa chuộng và dễ tiếp cận với cô ng
chúng hơn. Tuy nhiên, các thành quả của những mô hì nh nhận diện giọng nó i tiếng Việt vẫn chưa
được nhâ n rộng như các ngôn ngữ lớn khá c. Dẫu vậy, trong thập kỉ qua, nhiều học giả và nhà nghiên
cứu Việt Nam đã dành thời gian phá t triển và cải thiện cá c giải phá p nhận diện giọng nó i cho tiếng
Việt. Nhận thấy rằng có nhiều tiến bộ về nhận diện giọng nó i tiếng Việt, nghiên cứu này muốn xem xé t
những bước cần thiết để tạo ra một giải phá p gọn nhẹ và dễ tiếp cận đối với người Việt để giọng nó i
của họ có thể được hiểu bởi cá c hệ thống di động một cá ch thuận tiện, sau đó thử phá t triển một giải
phá p nhận diện giọng nói di động.
Abstract- Speech recognition is a field with many realistic and helpful applications.
Furthermore, they are becoming more and more popular, portable and accessible to the wide public.
However, applications of speech recognition for the Vietnamese language are yet to be as accessible
as more popular languages. Nevertheless, in the past decade, many Vietnamese researchers devoted
their time into developing and improving speech recognition solutions for their mother’s tongue.
Acknowledging that advancements in Vietnamese speech recognition have been made, this paper
aims to take a look at the steps required create a solution that is portable, accessible for many
Vietnamese people to conveniently make their speech utterances understood by mobile systems, then
attempts to develop a portable speech recognition solution.
Keywords- Speech recognition, machine learning, multipart form-data, Android
I. INTRODUCTION
Speech recognition, the ability for computers to understand and turn human utterances
into data, is a useful subfield of computer science with popular, thriving applications: Smart
home systems such as Amazon Echo and Google Home are capable to process and execute
owner’s imperatives at a moment’s notice, operating systems allowing speech as an
alternative method of interaction and input for vision and motion-impaired people, doctors
using speech recognition-enabled software to take notes while performing surgical operations,
etc… The possibilities that this technology brings is promising and seemingly limitless.
II. BACKGROUND
To help Vietnamese people to take advantage of the benefits that automatic speech
recognition (ASR) systems bring, many scholars and researchers have devoted their time to
developing an ASR system for the Vietnamese language. A large vocabulary continuous ASR
83
system with a Word Error Rate (WER) of 11.7% on digital newspaper speech was constructed
using a statistical language model [1]. Other scholars turned to neural-network based models
and obtained similar results [2]. Efforts have been made to construct and find the most
suitable acoustic models for the Vietnamese language using different methods such as
“knowledge-based and data-driven phone mapping” [3] or using “diphthongs and triphthongs
as phonemes” rather than just vowels and monophthongs [4] [5]. Many large tech firms such
as Viettel and FPT made investments into the field and currently provide speech-to-text
solutions to their customers commercially. In order to research and develop Vietnamese
speech solutions, there is a great demand of Vietnamese speech data. While many institutions
decide to preserve the data for internal use or commercial purposes, some others have made
their datasets available to the public or upon request for researching purposes for those in
need of the data to work on speech recognition projects. These datasets will be discussed
more in the next section.
III. COLLECTING SPEECH DATA
A pivotal part in any speech recognition research is to collect speech utterances from
actual humans. For this purpose, we utilize corpuses that have been previously collected for
research purposes and is made available to the public [6] or is granted upon request [7].
Furthermore, we invited ten speakers, including 8 males and 2 females, from various parts of
the country, with different accents to record around two hours of speech. We gave each of
them an Android apk. file to install our speech collection tool as well as a script comprised of
99 phrases that mostly contains the most frequently used words in the Vietnamese language:
và, của, có, các, là, không, cho, etc… [8]. The speech collection tool works by utilizing the
MediaRecorder API to record speech directly from the device’s microphone and save it under
the .3gp format for data efficiency (7KB file size in average). The app then creates a HTTP
POST request to a PHP form, which uses the multipart/form-data encoding type to allow the
content of the recorded file to be sent to the server. The server then tries to process the file
and return the status accordingly. If successful, the file is persisted to the server, which can be
later accessed using an FTP client. The files are then manually transcribed all renamed for
later use.
84
Figure 1. Example multipart/form-data POST request
Figure 2. Multipart/form-data POST request code
Figure 3. PHP code to process data from POST request
85
Figure 4. The SpeechCollect app
IV. ADAPTING THE MODEL
Unfortunately, since training an acoustic model from scratch requires a lot of time and
experience, we have chosen to instead adapt the default Vietnamese model from the Vosk
API [9]. For this purpose, we adapted the model using the speech we collected via our speech
collector tool, as we assumed the VIVOS dataset may already be used in the process of
training the model, and the VLSP dataset is largely comprised of Vietnamese audiobooks
(most of which are generated artificially using text-to-speech technology). Then we fine-tuned
some metrics of the model such at the frequency range, number of Mel-frequency bins, etc…
Figure 5. Configuration for MFCC features
The original model was created using the Kaldi toolkit, which heavily uses the OpenFst
library, a library used for “constructing, combining, optimizing and searching weighted finite-
state transducers (FSTs)” [10]. In FSTs, each transition has a weight, an input and output
labels. When the system receives an input, the state may transit to another accordingly, in this
case, to another string in a related set. FSTs are probabilistic and they are used to represent
the language model. The acoustic modeling process of Kaldi supports Gaussian Mixture
86
Models (GMMs) and other kinds of models. Kaldi also extract Mel-frequency cepstral
coefficients (MFCCs) and Perceptual Linear Prediction (PLP) from audio waveforms.
V. TESTING & RESULTS
After having the original model adapted, we ran a script to evaluate the Word Error
Rate (WER) of the adapted model with audio from the aforementioned VLSP 2019 dataset,
and then compare it to the WER of the original model.
The WER is calculated by the following formula:
(
)
푆 + 퐷 + 퐼
푁
푊퐸푅 =
In which:
+S signifies substitution occurrences: a word is replaced/ misidentified, for example,
“chuyện” is recognized as “truyện”
+D signifies deletion occurrences: a word is omitted from the result, for example “máy
vi tính” is recognized as “máy tính”
+I signifies insertion occurrences: a word that was not uttered is added to the transcript,
for example, “khoán” is recognized as “khóa án"
+N signifies the total count of words in the test.
The WER comparison between the original and adapted model is as follows:
Original model
Adapted model
24.2%
26.7%
Initially, the research team did not expect the accuracy of the adapted model to be lower
than that of the original model.
However, there could be a few contributing factors that led to the reduced accuracy of
the adapted model:
+Overall, the speech data we collected from the invited speakers may have caused
“saturation” in the model’s data processing. It provided the model alternate readings of the
same words or phrases and with another accent, which leads to alteration in the system’s
weights. In other words, the system perceives words differently, and depending on the reader,
it may or may not recognize the word for which it was trained.
+In the speaking prompt, although we stated beforehand that the phrases use many
common words, we also do introduce new words to the model to enlarge its vocabulary and
87
use somewhat complex phrases (“Không ai không thể không có”), which intentionally repeats
common words many times and the different form of said words (“ba” as in number three can
also be a noun meaning “father” or may even be a proper noun).
+We also asked the speakers to record imperative, exclamation and interrogative
phrases. Accuracy of the model against those kinds of phrases are considerably low,
potentially attributed to the nature of these phrases, which requires tonal variation from the
beginning to the end of the phrase. A word may be recognized differently if the speaker raises
or lower their tone.
+The speakers record directly from their mobile phones. Some speakers possess high
quality devices whose microphone does not pick up noise, whereas other speakers have noisy
and lower-quality microphones or recorded in a noisy environment, which have impacted the
fidelity of recognition.
+The data from the VLSP 2019 set used to calculate the WER is mostly read by text-to-
speech programs, which have almost no tonal variation in different parts of the phrase and
kinds of phrases, no regional accents/ dialects/, and reads at a consistent, steady rate, unlike
the other datasets. It is a high quality data set with no noise, however that does not match the
noisy recording that came from the data that we collected. Finally, data from the VLSP 2019
are from audiobooks, and most of them are stories/ fictional works, which introduce many
terms and slangs that simply was not included in the model’s vocabulary. Even proper nouns
such as character names (sometimes left unlocalized to Vietnamese) proves to be nearly
impossible for the system to recognize in case it does not have matching pronunciation with
any other word in the vocabulary (foreign names).
VI. BUILDING THE SPEECH RECOGNITION APPLICATION
To build our final speech recognition application, we choose to deploy it on Android,
and as our model is comparatively lightweight in size (40 MB), it is viable to be used for
offline speech recognition on mobile.
This application will use the Vosk Android API, a speech recognition toolkit that works
offline on lightweight devices, supports loading models of different languages (as long as
they follow a predesigned format) and provides real-time speech recognition. This application
also takes the kaldi-android module as dependency. It is built on API level 29/ Android 10, a
recent build of Android, and still has support for lower Android versions.
After creating the project, importing the libraries, writing/adapting the code based on
the Vosk Android tutorial [11], we need to unpack our model into models/src/main/assets,
rewrite the code to update the path to our model, and rebuild the UUID file in Gradle. This is
the final product with a simple interface that turns the recognizer on and off at the user’s will:
88
Figure 6. Sample code for the app
Figure 7. The VNVosk app interface
VII. CONCLUSION
In conclusion, there were many aspects of the project that could have been improved. For
example, with more resource and time as well as experience, we could try to build our own
acoustic model. We could also try to collect more speech data and in better, controller
environments. Although this project only yielded limited results, it is our first-hand
89
experience with speech recognition, from which we were able learn the steps to build a speech
recognition system, to enlarge our knowledge about the field and prepare ourselves better for
the future works related to this project. I would like to express my gratitude to my instructor,
Mr. Dang Dinh Quan, who helped me throughout the project and supported me with the PHP
form code to handle speech data sent from the speech collector tool as well as providing a
server to store the files. We highly appreciate comments and suggestions of readers on our
paper.
REFERENCES
[1] Vu, N. T., & Schultz, T. (2009, November). Vietnamese large vocabulary
continuous speech recognition. In 2009 IEEE Workshop on Automatic Speech Recognition &
Understanding (pp. 333-338). IEEE.
[2] Hung, P. D., Giang, T. M., Nam, L. H., & Duong, P. M. (2019). Vietnamese speech
command recognition using recurrent neural networks. J. Adv. Comput. Sci. Appl.(IJACSA),
10(7), 194-200.
[3] Le, V. B., & Besacier, L. (2005, March). First steps in fast acoustic modeling for a
new target language: application to Vietnamese. In Proceedings. (ICASSP'05). IEEE
International Conference on Acoustics, Speech, and Signal Processing, 2005. (Vol. 1, pp. I-
821). IEEE.
[4] Nguyen, T. & Vu, Q. (2009). Advances in Acoustic Modeling for Vietnamese
LVCSR. 2009 International Conference on Asian Language Processing, doi:
10.1109/IALP.2009.66.
[5] Luong, H. T. & Vu, H. Q. (2016). A non-expert Kaldi recipe for Vietnamese speech
recognition system. In Proceedings of the Third International Workshop on Worldwide
Language Service Infrastructure and Second Workshop on Open Infrastructures and Analysis
Frameworks for Human Language Technologies (WLSI/OIAF4HLT2016).
[6] Vu, H. Q (2009). VIVOS Corpus. Available at ailab.hcmus.edu.vn/vivos. Vietnam
National University- Ho Chi Minh City- University of Science.
for Vietnamese Language and Speech Processing.
[8] Dinh, D. & Ho, X. V. (2016). Ứng dụng Kho Ngữ liệu trong việc day tiếng Việt cho
người nước ngoài. Hội thảo quốc tế Giảng dạy, Nghiên cứu Việt Nam học và tiếng Việt,
1/2016, pp. 172-180.
90
Cephei.
[10]
Micheal,
R.
(2007).
OpenFst
Library.
Available
at
91
9 trang
24160
Free
Bạn đang xem tài liệu "Applying machine learning in recognizing vietnamese speech", để tải tài liệu gốc về máy hãy click vào nút Download ở trên
File đính kèm:
- applying_machine_learning_in_recognizing_vietnamese_speech.pdf
