Vinagpt-2: Generative pretrained transformer for vietnamese
VINAGPT-2: GENERATIVE PRETRAINED TRANSFORMER
FOR VIETNAMESE
SVTH: Nguyễn Hoàng Long
GVHD: ThS. Bùi Quốc Khá nh
Tó m tắt - Bài nghiên cứu về việc pre-train một model NLP dựa vào kiến trúc được phá t triển gần
đây có kết quả tốt qua nhiều tá c vụ. Trong bài nghiên cứu khoa học này, chúng tô i sẽ sử dụng mạng
thần kinh nhâ n tạo (mạng Transformer chỉ-decoder) để làm nên một model có khả năng viết lách đoạn
văn ngắn ở mức gần giống với một con người. Chúng tôi cũng tiến hành so sá nh với cá c model khá c
có sẵn. Tuy nhiên bài làm này cũng vẫn chứa một số giả định nên được đánh giá lại; cũng như nhiều
sức mạnh của cạc đồ họa hơn để khả thi hóa điều đó. Model có thể truy cập tại
Abstract - When thinking about Artificial Intelligence, people envision a machine that has quite
an ability to perform human-exclusive tasks. And programs that can communicate and/or converse
with many similarities to a human being are really convincing. Fortunately, such a machine is not too
far-fetched for our current technological advancement: GPT-2 - a model that excels at generating
comprehensive passages down-streamed for a wide range of writing purposes - is currently close to
that machine. But given how humongous a model is in English, we will need ways to turn the model to
Vietnamese if we want to build creative things with it. This paper will tell how has VinaGPT-2alpha
been pre-trained by us.
1. Introduction
Before the attention mechanism was proposed in 2015 [1], sequential data processing
was mainly handled with encoder-decoder RNNs/LSTMs. Although an LSTM can better
capture long-term dependency than regular RNN [2], it has its drawbacks and can be forgetful
in specific cases [3]. Another issue is that, when processing the sequence, there is no way to
focus on some input tokens over others. Not only does attention consider all of the input
tokens in the context vector, but it can also assign relative importance to each of them.
The Transformer [4] is a newer architecture that uses only the attention mechanism
instead of RNN to encode position, and the design of the Transformer paved the way for
parallel training on high-performance devices such as GPUs. Most applications under
Transformer neural networks are in the area of natural language processing, but it can also
process other sequential data such as genome sequences, sound signals, or time-series data. In
this paper, we will focus on a variant that is unidirectional (decoder-only) architecture. Since
it is unidirectional it tends to perform better at generating data as opposed to bi-directional
(with encoder) models, which are better at analyzing [5].
Generative Pretrained Transformer 2 (GPT-2) [6] was developed in an AI research and
266
deployment company named OpenAI and, the model the company is best known for. It could
do reasonably well and all it's trying to do is to predict the next word, or token, given what it
has seen so far. There was an interesting finding that, if you took this architecture and scale it
up in terms of model size and training dataset size, there was no indication would we end up
having the performance plateaued. GPT-2 is the second release of the GPT architectural
design, with the company’s investment in more computing power and 40GB of Internet text
data. The fact that GPT-2 was trained on such a massive dataset is the enabler for its good
results.
For usage in Vietnamese, one could retrain a language-specific model using the
Transformer architecture or find existing pre-trained multilingual GPT-based models. But so
far we haven’t found a publicly available model that is readily available and produces
acceptable results. This is the main motivation for us to create VinaGPT-2alpha, which can
simply be seen as Vietnamese GPT-2 that commensurate with other languages’ GPT-2. And
with the availability of different models ready to use and deploy from HuggingFace [7], we
have the means to compare the confidence of the text-generation tasks with other models.
2. Model1
The Transformer neural network was born from the discovery that the attention
mechanism alone could deliver improved accuracy, and the RNN design was no longer
necessary. The neural network receives an input sequence and converts it into two vectors:
input steps embeddings, and positional embeddings. An embedding layer transforms and
compresses the input feature space into a vector.
Scaled dot-product attention is an attention mechanism where the dot products are
scaled down by √dk. Formally we have a query, a key, and a value and calculate the attention
as:
Where Q is a vector of queries of dimension dk; K is a vector of keys of dimension dk; V
is a vector of values of dimension dk, and d is the size of the attention keys (this is a
hyperparameter chosen at design time).
According to the paper, “multi-head attention allows the model to jointly attend to
information from different representation subspaces at different positions. With a single
1
https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html, this article could be a good read
267
attention head, averaging inhibits this”. This allows computation to run in parallel and be
very fast on a GPU. The formula is described as:
GPT-2: This model can be classified into the “auto-regression” category of models
because each predicted token then will be chained into the next input sequence for a
continually stochastic generation.
Figure 1. Stacked decoder-only Transformer (Image source: Fig 1 & 2 in [4])
GPT-2 consists singly of Transformer decoders, the encoder part is stripped entirely and
the self-attention can attend only to the left token. You cannot allow the decoder to attend to
the right of the current token, hence the term unidirectional. Figure 1 visually represents the
inner workings of it.
268
Representing text as number: The word vector embeddings are a numeric
representation of the text. Assume we have a dictionary of words corresponding to their
number counterparts. The word embedding layer gives us a way to use an efficient, dense
representation and similar words are expected to have similar encodings. This layer is
trainable in a similar way a dense layer learns. The number of dimensions this layer has is a
hyperparameter and the larger it is and the training data is, the finer relationships between
words it can capture [8].
To convert words into tokens automatically, a tokenizer is often preferred. The inputs
for a model are prepared by a tokenizer. This component will break strings into sub-word
token strings, convert token strings to ids and back, and encode/decode (i.e., tokenizing and
converting to integers). It can also introduce new tokens to the vocabulary that is independent
of the underlying structure using a predefined algorithm; and managing special tokens (like
masking, beginning-of-sentence, etc.) [9]
The original GPT-2 was trained on a massive 40GB dataset called WebText that the
OpenAI researchers crawled from the internet. The smallest variant of the trained GPT-2, the
117M parameters, takes up 500MB of storage to store all of its weights. Followed by the
345M version at 1.5GB, the 762M version at 3.25GB, and the largest variant is 1542M so it
could take up 6.5GB of storage space. They are conveniently named “small”, “medium”,
“large”, and “XL” respectively.
3. VinaGPT-2alpha
Architecture: Our VinaGPT-2alpha is based on the “small” version of the released GPT-
2 models. “Vina” refers to Vietnamese and “alpha” refers to the fact that this model is still at
a development stage. Hyperparameters consist vocab size of 50257, input length of 1024,
embeddings dimension of 768, 12 decoder layers, 12 attention heads for each attention layer.
Pre-training data: VinaGPT-2alpha training data collecting approach is inspired by
PhoBERT [10] which optimizes some pre-processing of the text dataset procedure for more
robust performance. We used a concatenation of two Vietnamese corpora: Binhvq News
Corpus2 and the ViWikiText3; which add up to a grand total of close to 20GB of
uncompressed text. It is worth noting that Wikipedia data is relatively small and not
2
3
269
representative of the general language use. Therefore, the News Corpus is required to achieve
good results.
Preprocessing the text needs to be done the way the model can learn. For Wikitext, we
use a parsing library4 so 1GB of articles and be extracted from a 6GB XML file that includes
withal metadata, changes history, and discussions. We analyzed the dataset and found out that
the corpora consists of over 100 million lines of raw text (each separated by a newline
character). We also see that a new line doesn’t necessarily mean that it does not connect to the
previous line. And by shuffling the lines the model will never see word context dependencies
to learn to create coherent passages. One of the problems in Vietnamese is that unlike English
and other languages, spaces in Vietnamese don’t necessarily distinguish words from each
other but they only can separate syllables5. Inherently, almost all Vietnamese words consist of
at least two syllables and this might make it less efficient for the model. We can effectively
lower the ambiguity in sequences of words at the same space complexity by employing a
segmenter. RDRSegmenter [11] was chosen to use. But we quickly realized the drawback of
this is the heavy performance hit in both pre-processing and deployment runtime. To solve the
first problem, we ran the RDRSegmenter in parallel and reduced the amount of time it takes
to segment all the corpus.
Without Segmentation
Toàn thể nhân dân chung tay Toàn_thể
chống đại_dịch Covid
RDR Segmented (word-level)
nhân_dân
chung
tay
chống đại dịch Covid
Table 1. Before-after segmentation
After this process, we need to vectorize the text in order to serve it to the model. To do
this, we employ a popular and effective algorithm that is Byte-Pair Encoding [12]. First, we
need to train the tokenizer to get all the word coupling using the segmented text. This process
took around 2 hours and we have the output files: vocab.json and merges.txt. Next, we
decided to pre-tokenize the dataset simply because it was too time-consuming not to so we
think that’d be a good idea.
For text that is heavily contextual and depends a lot on previous sentences, we tokenize
the text using the sliding window strategy. In which it takes a defined size of the 1-
dimensional window, tokenizes the chunk, slides to the right by a defined stride, then
4
5
tu for more info on word segmentation problem.
270
tokenizes the new chunk. This process happens recursively until all the blocks of long text
have been tokenized. Considering that the model has a maximum input length of 1024, we set
the window size to 300, stride length to 100, and apply to chunks with lengths of 301 and up.
The number counts how many substrings are separated by a space in a line of text.
Optimization: We ran into the infamous CUDA OOM error many times simply
because of how colossal the model appears to be. We even ran into RAM OOM but it turned
out to be that some libraries weren’t optimized for the sheer dataset size. We tried to figure
out the most efficient combination settings to test the model. We wouldn’t reduce the input
length below 1024. Training batch size is 2 and employ the AdamW [13] optimizer. Trial and
errors also involved tinkering with gradient_checkpointing, gradient_accumulation_steps,
and optimizers like Adafactor [14]; in the end wasn’t necessary. We used GELU activation
function, initial learning rate of 5e-5, dropout probability of 0.1, and trained for 440,000
steps.
4. Experimental Set-up and Results
We save the training checkpoint every finished epoch to evaluate the training progress.
A model takes up a large chunk of storage so we made use of Google Drive’s unlimited
storage to store them.
Perplexity (PPL): We evaluate the text-generation performance of VinaGPT-2alpha on
the perplexity score. This metric measures the log-likelihood of the predicted token compared
to the ground truth of the test dataset. Although there are cases where multiple tokens are
acceptable, on average a better model is expected to be more confident toward the next token.
And the PPL of fixed-length models should be evaluated with a sliding-window strategy. We
set the stride length to 512 throughout our evaluation process6. For a generative language
model, overfitting indicates a continually lowered loss value while reversal or increments of
PPL value.
Main results: We picked the last checkpoint to compare with different GPT-2 based
models provided by HuggingFace. The table below shows the score inferred from the
corresponding dataset:
6
https://huggingface.co/transformers/perplexity.html HuggingFace documentation was nice enough to provide
us a working piece of code that implements this.
271
Model
No. of params Test Dataset
PPL
AWD-LSTM
DistilGPT2
GPT-2small
GPT-2XL
33M
82M
117M
1542M
WikiText-2
44.3 [15]
39.3
25.2
18.3
VinaGPT-2alpha
117M
ViWikiText
Binhvq News Corpus
31.2
19.2
Table 2. PPL between previous models. The lower the score the better.
Table 2 compares scores among different models pre-trained on text generation task for
different languages. As well as previous attempts on the task of text generation that did not
make use of the Transformer. We can see that GPT-based models outperform previous SOTA
thanks to fast training time and VinaGPT-2alpha is trying to be as good as other 114M GPT-2
based models.
Another performance assessment we carried out involves humans reading the output and
seeing for ourselves. After all, this model was designed to communicate to be as “human” as
possible. Table 3 shows end-user output, given the input is “Có một”, compared throughout
each training epoch, after we have set up a text generating pipeline.
Steps
PPL
on Hand-picked generated output
NewsCorpus
27.1978
Có một số tín hiệu như lãi suất huy động
được bao nhiêu VND, VND hàng nhất có lãi tại
các doanh nghiệp đang gặp khó khăn ở doanh
nghiệp Việt Nam mà có rất nhiều doanh nghiệp
đang tìm cách tăng thêm tiền để bán vốn.
100000
21.0411
29.1174
Có một thực tế để thấy, từ khi đưa con đến
các tỉnh khu vực miền núi phía Bắc Từ những
năm 1990 đến nay, các nước vùng núi phía Nam
Bắc Trung bộ luôn thiếu công cụ để xây dựng
và tạo ra các nhóm sinh đôi nhằm tăng khả
năng vận động các nguồn lực trong phát triển
kinh tế - xã hội ở các địa phương của Tây
Nguyên.
300000
390000
Có một số học giả đặt câu hỏi về những thành
phần và nguyên tắc căn bản của "chiến
272
tranh", trong các tiểu luận của Homer là sự
tồn tại chính trị. Một số học giả quan tâm
đến chủ nghĩa phi chiến tranh của Homer và
những luận cứ về các nhà quân sự của ông về
chủ đề của họ, mặc dù chúng không hề có liên
quan gì về các chiến thuật
Table 3. Generation with output length of 100, input is “Có một”
Discussion: We use RDRSegmenter followed by Byte-Pair Encoding based on
observed practices, not experimentation. Although this might improve the main model’s
performance, the current segmenter might be a bit slow and inaccurate at times: an edge case
example is “con ngựa đá con ngựa đá”. This is the reason for an evaluation and comparison
between ways of pre-processing Vietnamese text in future work. Another point we would like
to make is that we don’t have access to greater computing power so we trained the 117M
parameter variant. However, we believe this version offers the best trade-off between size
versus performance, consequently, better to deploy. Model variants with larger parameter
count trained temporally longer can have better performance.
5. Conclusion
In this paper, we presented VinaGPT-2alpha, and by publicly releasing the model, it is
easier for people interested in the said ability of an AI to experiment and play around with it.
Multiple hours of uninterrupted high-memory, high-performance GPU workload might be too
demanding. Furthermore, the API contributorily lowers the effort needed to get the model up
and running. Uses of the model when fine-tuned include (but not limited to):
●Text Generation: Writing aid for writers alike. Text-based adventure games that
instead of a pre-made storyline like what was popular in the 90s, will generate imaginary
events on the fly.
●Code Generation: Code a simple HTML file from prompt.
●Chatbots: With the right special tokens, we can make the model respond to an input!
●Machine Translation and Text Summarization: Although bidirectional models tend to
do these tasks better, we can see the versatility of a pre trained NLP model.
Acknowledgments
The author acknowledges computational support externally and would like to thank
HANU for providing a student Gmail account associated with Google Drive’s unlimited
cloud storage. We also made good use of free cloud GPU services otherwise would be very
expensive to train.
273
References
[1] Dzmitry Bahdanau, KyungHyun Cho, Yoshua Bengio. 2015. Neural Machine
[2] Nhâm Gia Hoàng Anh, Nguyễn Tiến Huy, Nguyễn Khắc Phúc, Bùi Đình Quân, Bui
Quoc Khanh, Sentiment Analysis Using Long Short-term Memory Recurrent Neural Network,
Hanoi University
[4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan
[6] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya
[8]
Word
embeddings.
Available:
Available:
[9] Tokenizer.
[10] Dat Quoc Nguyen, Anh Tuan Nguyen. 2020. PhoBERT: Pre-trained language
[11] Dat Quoc Nguyen, Dai Quoc Nguyen, Thanh Vu, Mark Dras, Mark Johnson. 2018.
[13] Diederik P. Kingma, Jimmy Lei Ba. 2017. Adam: A Method For Stochastic
[14] Noam Shazeer, Mitchell Stern. 2018. Adafactor: Adaptive Learning Rates with
274
9 trang
33000
Free
Bạn đang xem tài liệu "Vinagpt-2: Generative pretrained transformer for vietnamese", để tải tài liệu gốc về máy hãy click vào nút Download ở trên
File đính kèm:
- vinagpt_2_generative_pretrained_transformer_for_vietnamese.pdf
