👉 Benchmark datasets here

VLSP 2023 VLLMs: Vietnamese Large Language Models

VLSP 2023 challenge on Vietnamese Large Language Models

To contact us, mail to: leanhcuong@gmail.com (Lê Anh Cường)

1. Important dates

2. Task Description

In recent years, Large Language Models (LLMs) have gained widespread recognition and popularity worldwide, with models such as GPT-X, BARD and LLaMa making significant strides in natural language processing tasks. In Vietnam, there is also a growing interest in developing LLMs specifically tailored for the Vietnamese language. However, unlike LLMs developed for other languages, the availability of publicly accessible evaluation data for Vietnamese LLMs is significantly limited. The limited availability of evaluation data for Vietnamese LLMs presents a substantial obstacle for organizations seeking to establish uniform evaluation standards. The goal of VLSP2023-VLLMs is to promote the development of large language models for Vietnamese by constructing an evaluation dataset for VLLMs. This dataset will be different from conventional datasets for downstream NLP tasks, as it will focus on 4 primary abilities, divided into 8 skills and divided into 9 domains.

Abilities

Domains

The teams participating in this challenge will build their own LLMs for Vietnamese, and these models will be provided with a public test dataset and instructions on how to evaluate them. The models participating in this competition remain the copyright of the respective development groups and are not required to be open-source. We will provide the following instructions to the participating groups:

3. Evaluation

Results would be evaluated by model-based evaluation and human-based evaluation.

4. Registration

👉 Shared Task Registration Form

5. Resources

We will provide the following instructions to the participating groups:

Note that the participating teams can use any resources to train their models.

Organizers

Sponsors

INT2

Intelligent Integration Co., Ltd. (INT2)

Vietnam

www.int2.vn

 HPC-systems

HPC SYSTEMS Inc.

Japan

www.hpc.co.jp

References

[1] Long Ouyang and Jeff Wu and Xu Jiang and Diogo Almeida and Carroll L. Wainwright and Pamela Mishkin and Chong Zhang and Sandhini Agarwal and Katarina Slama and Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022.

[2] Seonghyeon Ye and Doyoung Kim and Sungdong Kim and Hyeonbin Hwang and Seungone Kim and Yongrae Jo and James Thorne and Juho Kim and Minjoon Seo. FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets. arXiv preprint arXiv:2307.10928, 2023.