Publications | LAGoM NLP

Latest preprints

arXiv
A Principled Framework for Evaluating on Typologically Diverse Languages

Esther Ploeger, Wessel Poelman, Andreas Holck Høeg-Petersen, and 3 more authors

Jul 2024

Abs Bib URL

Beyond individual languages, multilingual natural language processing (NLP) research increasingly aims to develop models that perform well across languages generally. However, evaluating these systems on all the world’s languages is practically infeasible. To attain generalizability, representative language sampling is essential. Previous work argues that generalizable multilingual evaluation sets should contain languages with diverse typological properties. However, ’typologically diverse’ language samples have been found to vary considerably in this regard, and popular sampling methods are flawed and inconsistent. We present a language sampling framework for selecting highly typologically diverse languages given a sampling frame, informed by language typology. We compare sampling methods with a range of metrics and find that our systematic methods consistently retrieve more typologically diverse language selections than previous methods in NLP. Moreover, we provide evidence that this affects generalizability in multilingual model evaluation, emphasizing the importance of diverse language sampling in NLP evaluation.
@misc{ploeger2024principled, title = {A {{Principled Framework}} for {{Evaluating}} on {{Typologically Diverse Languages}}}, author = {Ploeger, Esther and Poelman, Wessel and {H{\o}eg-Petersen}, Andreas Holck and Schlichtkrull, Anders and {de Lhoneux}, Miryam and Bjerva, Johannes}, year = {2024}, month = jul, number = {arXiv:2407.05022}, eprint = {2407.05022}, urldate = {2024-07-22}, keywords = {Computer Science - Computation and Language}, primaryclass = {cs}, publisher = {arXiv}, }
arXiv
How Good Is Your Wikipedia?

Kushal Tatariya*, Artur Kulmizev*, Wessel Poelman, and 6 more authors

Nov 2024

Abs Bib URL

Wikipedia’s perceived high quality and broad language coverage have established it as a fundamental resource in multilingual NLP. In the context of low-resource languages, however, these quality assumptions are increasingly being scrutinised. This paper critically examines the data quality of Wikipedia in a non-English setting by subjecting it to various quality filtering techniques, revealing widespread issues such as a high percentage of one-line articles and duplicate articles. We evaluate the downstream impact of quality filtering on Wikipedia and find that data quality pruning is an effective means for resource-efficient training without hurting performance, especially for low-resource languages. Moreover, we advocate for a shift in perspective from seeking a general definition of data quality towards a more language- and task-specific one. Ultimately, we aim for this study to serve as a guide to using Wikipedia for pretraining in a multilingual setting.
@misc{tatariya2024how, title = {How {{Good}} Is {{Your Wikipedia}}?}, author = {Tatariya*, Kushal and Kulmizev*, Artur and Poelman, Wessel and Ploeger, Esther and Bollmann, Marcel and Bjerva, Johannes and Luo, Jiaming and Lent, Heather and de Lhoneux, Miryam}, year = {2024}, month = nov, number = {arXiv:2411.05527}, eprint = {2411.05527}, publisher = {arXiv}, urldate = {2024-11-13}, keywords = {Computer Science - Computation and Language}, primaryclass = {cs}, }

Conference papers

NoDaLiDa
The Roles of English in Evaluating Multilingual Language Models

Wessel Poelman, and Miryam de Lhoneux

In The Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies Mar 2025

Abs Bib URL

Multilingual natural language processing is getting increased attention, with numerous models, benchmarks, and methods being released for many languages. English is often used in multilingual evaluation to prompt language models (LMs), mainly to overcome the lack of instruction tuning data in other languages. In this position paper, we lay out two roles of English in multilingual LM evaluations: as an interface and as a natural language. We argue that these roles have different goals: task performance versus language understanding. This discrepancy is highlighted with examples from datasets and evaluation setups. Numerous works explicitly use English as an interface to boost task performance. We recommend to move away from this imprecise method and instead focus on furthering language understanding.
@inproceedings{poelman2024roles, title = {The {{Roles}} of {{English}} in {{Evaluating Multilingual Language Models}}}, author = {Poelman, Wessel and {de Lhoneux}, Miryam}, booktitle = {The {{Joint}} 25th {{Nordic Conference}} on {{Computational Linguistics}} and 11th {{Baltic Conference}} on {{Human Language Technologies}}}, year = {2025}, month = mar, }

NAACL
BPE-knockout: Pruning Pre-existing BPE Tokenisers with Backwards-compatible Morphological Semi-supervision

Thomas Bauwens, and Pieter Delobelle

In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) Jun 2024

Abs Bib URL Poster Slides Video

Byte-pair encoding (BPE) has become the default subword tokeniser in language models (LMs), allowing the representation of an infinite space of text with a finite set of units. Yet, BPE training is unsupervised, receiving no explicit information about a language’s morphology. This results in a subword vocabulary wherein many units are a concatenation of partial morphemes, preventing their formation as tokens. This, in turn, causes consistent intra-word patterns to be displayed inconsistently to downstream models, and bloats the vocabulary, hence requiring unnecessary embedding storage. In this paper, we address this issue by identifying blameworthy BPE merges and removing the resulting subwords from the BPE vocabulary, without impeding further use of merges that relied on them. We find that our method, BPE-knockout, is effective at making BPE’s segmentation positions adhere better to derivational and compound boundaries in English, Dutch and German, and improves token-based tasks in Dutch RoBERTa models, indicating that a tokeniser’s adherence to morphology impacts downstream models. We demonstrate the latter not only by training LMs from scratch, but also by continuing the pre-training of existing LMs. This proves promising, showing that suboptimal tokenisers can be remedied whilst salvaging training cost of downstream LMs.
@inproceedings{bauwens-delobelle-2024-bpe, title = {{BPE}-knockout: Pruning Pre-existing {BPE} Tokenisers with Backwards-compatible Morphological Semi-supervision}, author = {Bauwens, Thomas and Delobelle, Pieter}, booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)}, month = jun, year = {2024}, address = {Mexico City, Mexico}, publisher = {Association for Computational Linguistics}, url = {https://aclanthology.org/2024.naacl-long.324}, pages = {5810--5832}, }
EMNLP
What Is “Typological Diversity” in NLP?

Esther Ploeger*, Wessel Poelman*, Miryam de Lhoneux, and 1 more author

In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing Nov 2024

Abs Bib URL

The NLP research community has devoted increased attention to languages beyond English, resulting in considerable improvements for multilingual NLP. However, these improvements only apply to a small subset of the world’s languages. Aiming to extend this, an increasing number of papers aspires to enhance generalizable multilingual performance across languages. To this end, linguistic typology is commonly used to motivate language selection, on the basis that a broad typological sample ought to imply generalization across a broad range of languages. These selections are often described as being ’typologically diverse’. In this work, we systematically investigate NLP research that includes claims regarding ’typological diversity’. We find there are no set definitions or criteria for such claims. We introduce metrics to approximate the diversity of language selection along several axes and find that the results vary considerably across papers. Furthermore, we show that skewed language selection can lead to overestimated multilingual performance. We recommend future work to include an operationalization of ’typological diversity’ that empirically justifies the diversity of language samples.
@inproceedings{ploeger2024what, title = {What Is {{``Typological Diversity''}} in {{NLP}}?}, author = {Ploeger*, Esther and Poelman*, Wessel and {de Lhoneux}, Miryam and Bjerva, Johannes}, booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, year = {2024}, month = nov, publisher = {Association for Computational Linguistics}, }

EMNLP

Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models

Kushal Tatariya, Vladimir Araujo, Thomas Bauwens, and 1 more author

In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing Nov 2024

Bib URL

@inproceedings{tatariya2024pixology,
  title = {Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models},
  author = {Tatariya, Kushal and Araujo, Vladimir and Bauwens, Thomas and {de Lhoneux}, Miryam},
  year = {2024},
  publisher = {Association for Computational Linguistics},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
}

COLM

Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP

François Remy, Pieter Delobelle, Hayastan Avetisyan, and 3 more authors

In First Conference on Language Modeling Nov 2024

Bib URL

@inproceedings{remy-delobelle2024transtokenization,
  title = {Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of {LLM}s for Low-Resource {NLP}},
  author = {Remy, Fran{\c{c}}ois and Delobelle, Pieter and Avetisyan, Hayastan and Khabibullina, Alfiya and {de Lhoneux}, Miryam and Demeester, Thomas},
  booktitle = {First Conference on Language Modeling},
  year = {2024},
  url = {https://openreview.net/forum?id=sBxvoDhvao},
}

Journal papers

TACL

CreoleVal: Multilingual Multitask Benchmarks for Creoles

Heather Lent, Kushal Tatariya, Raj Dabre, and 18 more authors

Transactions of the Association for Computational Linguistics Sep 2024

Abs URL

Creoles represent an under-explored and marginalized group of languages, with few available resources for NLP research. While the genealogical ties between Creoles and a number of highly resourced languages imply a significant potential for transfer learning, this potential is hampered due to this lack of annotated data. In this work we present CreoleVal, a collection of benchmark datasets spanning 8 different NLP tasks, covering up to 28 Creole languages; it is an aggregate of novel development datasets for reading comprehension relation classification, and machine translation for Creoles, in addition to a practical gateway to a handful of preexisting benchmarks. For each benchmark, we conduct baseline experiments in a zero-shot setting in order to further ascertain the capabilities and limitations of transfer learning for Creoles. Ultimately, we see CreoleVal as an opportunity to empower research on Creoles in NLP and computational linguistics, and in general, a step towards more equitable language technology around the globe.

Workshop papers

SIGTYP
Sociolinguistically Informed Interpretability: A Case Study on Hinglish Emotion Classification

Kushal Tatariya, Heather Lent, Johannes Bjerva, and 1 more author

In Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP Mar 2024

Abs Bib URL Poster

Emotion classification is a challenging task in NLP due to the inherent idiosyncratic and subjective nature of linguistic expression,especially with code-mixed data. Pre-trained language models (PLMs) have achieved high performance for many tasks and languages, but it remains to be seen whether these models learn and are robust to the differences in emotional expression across languages. Sociolinguistic studies have shown that Hinglish speakers switch to Hindi when expressing negative emotions and to English when expressing positive emotions. To understand if language models can learn these associations, we study the effect of language on emotion prediction across 3 PLMs on a Hinglish emotion classification dataset. Using LIME and token level language ID, we find that models do learn these associations between language choice and emotional expression. Moreover, having code-mixed data present in the pre-training can augment that learning when task-specific data is scarce. We also conclude from the misclassifications that the models may overgeneralise this heuristic to other infrequent examples where this sociolinguistic phenomenon does not apply.
@inproceedings{tatariya_sociolinguistically_2024, address = {St. Julian's, Malta}, title = {Sociolinguistically {Informed} {Interpretability}: {A} {Case} {Study} on {Hinglish} {Emotion} {Classification}}, shorttitle = {Sociolinguistically {Informed} {Interpretability}}, url = {https://aclanthology.org/2024.sigtyp-1.9}, urldate = {2024-03-22}, booktitle = {Proceedings of the 6th {Workshop} on {Research} in {Computational} {Linguistic} {Typology} and {Multilingual} {NLP}}, publisher = {Association for Computational Linguistics}, author = {Tatariya, Kushal and Lent, Heather and Bjerva, Johannes and {de Lhoneux}, Miryam}, editor = {Hahn, Michael and Sorokin, Alexey and Kumar, Ritesh and Shcherbakov, Andreas and Otmakhova, Yulia and Yang, Jinrui and Serikov, Oleg and Rani, Priya and Ponti, Edoardo M. and Muradoğlu, Saliha and Gao, Rena and Cotterell, Ryan and Vylomova, Ekaterina}, month = mar, year = {2024}, pages = {66--74}, }
SIGTYP
A Call for Consistency in Reporting Typological Diversity

Wessel Poelman*, Esther Ploeger*, Miryam de Lhoneux, and 1 more author

In Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP Mar 2024

Abs Bib URL

In order to draw generalizable conclusions about the performance of multilingual models across languages, it is important to evaluate on a set of languages that captures linguistic diversity.Linguistic typology is increasingly used to justify language selection, inspired by language sampling in linguistics.However, justifications for ‘typological diversity’ exhibit great variation, as there seems to be no set definition, methodology or consistent link to linguistic typology.In this work, we provide a systematic insight into how previous work in the ACL Anthology uses the term ‘typological diversity’.Our two main findings are: 1) what is meant by typologically diverse language selection is not consistent and 2) the actual typological diversity of the language sets in these papers varies greatly.We argue that, when making claims about ‘typological diversity’, an operationalization of this should be included.A systematic approach that quantifies this claim, also with respect to the number of languages used, would be even better.
@inproceedings{poelman2024call, title = {A {{Call}} for {{Consistency}} in {{Reporting Typological Diversity}}}, booktitle = {Proceedings of the 6th {{Workshop}} on {{Research}} in {{Computational Linguistic Typology}} and {{Multilingual NLP}}}, author = {Poelman*, Wessel and Ploeger*, Esther and {de Lhoneux}, Miryam and Bjerva, Johannes}, editor = {Hahn, Michael and Sorokin, Alexey and Kumar, Ritesh and Shcherbakov, Andreas and Otmakhova, Yulia and Yang, Jinrui and Serikov, Oleg and Rani, Priya and Ponti, Edoardo M. and Murado{\u g}lu, Saliha and Gao, Rena and Cotterell, Ryan and Vylomova, Ekaterina}, year = {2024}, month = mar, pages = {75--77}, publisher = {Association for Computational Linguistics}, address = {St. Julian's, Malta}, url = {https://aclanthology.org/2024.sigtyp-1.10}, urldate = {2024-04-22}, }

MRL

Recipe for Zero-shot POS Tagging: Is It Useful in Realistic Scenarios?

Zeno Vandenbulcke, Lukas Vermeire, and Miryam de Lhoneux

In 4th Multilingual Representation Learning (MRL) workshop Mar 2024

Bib URL

@inproceedings{vandenbulcke2024recipe,
  title = {Recipe for Zero-shot POS Tagging: Is It Useful in Realistic Scenarios?},
  author = {Vandenbulcke, Zeno and Vermeire, Lukas and {de Lhoneux}, Miryam},
  year = {2024},
  booktitle = {4th Multilingual Representation Learning (MRL) workshop},
}

HumEval

Exploratory Study on the Impact of English Bias of Generative Large Language Models in Dutch and French

Ayla Rigouts Terryn, and Miryam de Lhoneux

In Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024 May 2024

Bib URL

@inproceedings{rigouts-terryn-de-lhoneux-2024-exploratory,
  title = {Exploratory Study on the Impact of {E}nglish Bias of Generative Large Language Models in {D}utch and {F}rench},
  author = {Rigouts Terryn, Ayla and {de Lhoneux}, Miryam},
  editor = {Balloccu, Simone and Belz, Anya and Huidrom, Rudali and Reiter, Ehud and Sedoc, Joao and Thomson, Craig},
  booktitle = {Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024},
  month = may,
  year = {2024},
  address = {Torino, Italia},
  publisher = {ELRA and ICCL},
  url = {https://aclanthology.org/2024.humeval-1.2},
  pages = {12--27},
}

WASSA
Transfer Learning for Code-Mixed Data: Do Pretraining Languages Matter?

Kushal Tatariya, Heather Lent, and Miryam de Lhoneux

In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis Jul 2023

Abs Bib URL Poster

Monolinguals make up a minority of the world’s speakers, and yet most language technologies lag behind in handling linguistic behaviours produced by bilingual and multilingual speakers. A commonly observed phenomenon in such communities is code-mixing, which is prevalent on social media, and thus requires attention in NLP research. In this work, we look into the ability of pretrained language models to handle code-mixed data, with a focus on the impact of languages present in pretraining on the downstream performance of the model as measured on the task of sentiment analysis. Ultimately, we find that the pretraining language has little effect on performance when the model sees code-mixed data during downstream finetuning. We also evaluate the models on code-mixed data in a zero-shot setting, after task-specific finetuning on a monolingual dataset. We find that this brings out differences in model performance that can be attributed to the pretraining languages. We present a thorough analysis of these findings that also looks at model performance based on the composition of participating languages in the code-mixed datasets.
@inproceedings{tatariya_transfer_2023, address = {Toronto, Canada}, title = {Transfer {Learning} for {Code}-{Mixed} {Data}: {Do} {Pretraining} {Languages} {Matter}?}, shorttitle = {Transfer {Learning} for {Code}-{Mixed} {Data}}, url = {https://aclanthology.org/2023.wassa-1.32}, doi = {10.18653/v1/2023.wassa-1.32}, urldate = {2023-12-27}, booktitle = {Proceedings of the 13th {Workshop} on {Computational} {Approaches} to {Subjectivity}, {Sentiment}, \& {Social} {Media} {Analysis}}, publisher = {Association for Computational Linguistics}, author = {Tatariya, Kushal and Lent, Heather and {de Lhoneux}, Miryam}, editor = {Barnes, Jeremy and De Clercq, Orphée and Klinger, Roman}, month = jul, year = {2023}, pages = {365--378}, }