Post
26
Master-Thesis: Syntax Probing of Instruction Models
Abstract:
This study aims to explore fundamental
linguistic abilities by systematically evaluating the impact of different datasets and language model architectures. Surprisingly, the tested instruction models are not very good at applying general grammar rules, especially for nonsensical but well-formed
sentences.
Download the full study as PDF-Version: https://gitlab.com/Bachstelze/syntax_transformer/-/blob/main/Abgabe-Syntax_probing_of_instruction_models.pdf
Datasets:
Bachstelze/GEC_CoT_explanation
Bachstelze/PAWS_CoT_explanation
Future research:
One might think about training the recent deepseek architecture as causal encoder-decoder and mitigate the wrong training objective of t5-models and readd the bi-directional potential to generative models.
Abstract:
This study aims to explore fundamental
linguistic abilities by systematically evaluating the impact of different datasets and language model architectures. Surprisingly, the tested instruction models are not very good at applying general grammar rules, especially for nonsensical but well-formed
sentences.
Download the full study as PDF-Version: https://gitlab.com/Bachstelze/syntax_transformer/-/blob/main/Abgabe-Syntax_probing_of_instruction_models.pdf
Datasets:
Bachstelze/GEC_CoT_explanation
Bachstelze/PAWS_CoT_explanation
Future research:
One might think about training the recent deepseek architecture as causal encoder-decoder and mitigate the wrong training objective of t5-models and readd the bi-directional potential to generative models.