Configuration and Evaluation of a Custom GPT for Abstract Semantic Annotation of European Portuguese According to the LMR Framework
DOI:
https://doi.org/10.26334/2183-9077/rapln15ano2026a9Keywords:
Abstract semantic annotation, large language models, custom GPT, Lexicalized Meaning Representation (LMR), reported speechAbstract
This article presents the configuration and evaluation of a custom GPT for the abstract semantic annotation of European Portuguese texts according to the Lexicalized Meaning Representation (LMR) framework. LMR is an adaptation of Abstract Meaning Representation (AMR), designed to accommodate properties of Portuguese grammar while preserving a closer connection between the source text and its semantic representation. The GPT was configured through a dedicated instruction set and a closed knowledge base comprising the LMR guidelines and a collection of previously annotated sentences. The study also introduces relations and operators for representing selected configurations of reported speech and quotation. For evaluation purposes, a corpus of 42 open-access news articles on artificial intelligence was compiled, from which five sentences were selected. These sentences were independently annotated by the two authors and subsequently adjudicated by consensus to form the reference set. The experiments compare the performance of GPT-4o and GPT-5-Thinking under four conditions: without examples explicitly included in the prompt, few-shot out-of-domain, many-shot out-of-domain, and few-shot in-domain. Evaluation using the Smatch metric, complemented by a qualitative analysis of the observed divergences, shows that GPT-5-Thinking produces annotations that are, on average, closer to the reference representations, achieving a mean F-score of 0.62, compared with 0.51 for GPT-4o. The effect of the examples was neither uniform nor cumulative, varying according to the model and experimental condition. The results confirm the potential of these systems as tools for supporting semantic annotation, although human supervision remains necessary.Downloads
References
Anchiêta, Rafael (2020). Abstract meaning representation parsing for the Brazilian Portuguese language. Tese de doutoramento, Universidade de São Paulo, Brasil.
Anchiêta, R., Cabezudo, M. & Pardo, T. (2019) SEMA: An extended semantic evaluation metric for AMR. arXiv e-prints, pp. arXiv–1905.
Anchiêta, R. & Pardo, T. (2018). Towards AMR-BR: A SemBank for Brazilian Portuguese Language. In Proceedings of the 11th edition of the Language Resources and Evaluation Conference (LREC), pp. 974–979. May 7–12. Miyazaki/Japan. pdf.
Antunes, D., Baptista, J. & Mamede, N. (2025) Processamento automático de expressões idiomáticas do português europeu. Linguamática, 17(1):55-73. URL: https://linguamatica.com/index.php/linguamatica/ article/ view/464
Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., Knight, K., Koehn, P., Palmer, M., & Schneider, N. (2013). Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pp. 178–186.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2019. Abstract Meaning Representation (AMR) 1.2.6 specification. https://github.com/ amrisi/amr-guidelines/blob/master/amr.md
Baptista, J. (2010). Verba dicendi: A structure looking for verbs. Les Tables. La grammaire du français par le menu. Mélanges en hommage à Christian Leclère. Cahiers du CENTAL, 6, pp. 11–20.
Baptista, J. (2024). Lexical Meaning Representation – Guidelines (Technical report). University of Algarve/INESC-ID Lisboa. https://gitlab.hlt.inesc-id.pt/u000803/lmr4pt/
Baptista, J. & Mamede, N. (2020a) Dicionário gramatical de verbos do Português Europeu. Universidade do Algarve Editora.
Baptista, J. & Mamede, N. (2020b) Syntactic transformations in rule-based parsing of support verb constructions: examples from European Portuguese. In Alberto Simões, Pedro Rangel Henriques, & Ricardo Queirós (eds.), 9th Symposium on Languages, Applications and Technologies (SLATE 2020). Schloss Dagstuhl–Leibniz-Zentrum für Informatik, pp. 11:1–11:14. https://doi.org/10.4230/OASIcs.SLATE.2020.11
Baptista, J., Müller, I., & Reis, S. (2025). Semantic Representation of Adverbs in the Lexicalized Meaning Representation (LMR) Framework. In 14th Symposium on Languages, Applications and Technologies (SLATE 2025), OpenAccess Series in Informatics (OASIcs), Volume 135, pp. 9:1-9:18, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.SLATE.2025.9
Baptista, J., Reis, S., Dias, J., & Santos, P. (2024a). Lexicalized Meaning Representation (LMR). In Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024, pp. 101–111, Torino, Italia. ELRA and ICCL.
Baptista, J., Reis, S., Santos, P., & Dias, J. (2024b). Comparação de anotações AMR entre línguas: problemas e soluções. Revista Da Associação Portuguesa de Linguística, (11), 27–54. https://doi.org/10.26334/2183-9077/rapln11ano2024a2.
Bevilacqua, M., Blloshmi, R., & Navigli, R. (2021). One SPRING to Rule Them Both: Symmetric AMR Semantic Parsing and Generation without a Complex Pipeline. In Proceedings of the AAAI Conference on Artificial Intelligence, 35(14), 12564-12573. https://doi.org/10.1609/aaai.v35i14.17489
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877–1901.
Cai, S. & Knight, K. (2013). Smatch: an Evaluation Metric for Semantic Feature Structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics.
Damonte, M., Cohen, S. B., & Satta, G. (2017). An Incremental Parser for Abstract Meaning Representation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 536–546, Valencia, Spain. Association for Computational Linguistics.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), Volume 1: Long and Short Papers (pp. 4171–4186). Association for Computational Linguistics. [https://doi.org/10.18653/v1/N19-1423] (https://doi.org/10.18653/v1/N19-1423)
Fahim, A., Azad, S., Rashid, T., Albaidaei, E., & Ighzali, H. (2024). Harnessing Large Language Models for Transformative Applications. In Natural Language Processing. European Journal of Theoretical and Applied Sciences, 2(6), pp. 428–438. https://doi.org/10.59324/ejtas.2024.2(6).37
He, X., Lin, Z., Gong, Y., Jin, A. L., Zhang, H., Lin, C., ... & Chen, W. (2024, June). Annollm: Making large language models to be better crowdsourced annotators. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track) (pp. 165–190).
Kostina, A., Dikaiakos, M. D., Stefanidis, D., & Pallis, G. (2025). Large Language Models For Text Classification: Case Study and Comprehensive Review. https://doi.org/10.48550/arxiv.2501.08457
OpenAI. (2022). ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/research/chatgpt
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
Ren, M. (2024). Advancements and Applications of Large Language Models in Natural Language Processing: A Comprehensive Review. Applied and Computational Engineering, 97(1), pp. 55–63. https://doi.org/10.54254/2755-2721/97/20241406
Savelka, J., & Ashley, K. D. (2023). The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts. Frontiers in Artificial Intelligence, 6:1279794. doi: 10.3389/frai.2023.1279794
Tao, C., Shen, T., Gao, S., Zhang, J. L., Li, Z., Tao, Z., & Ma, S. (2024). LLMs are Also Effective Embedding Models: An In-depth Overview. https://doi.org/10.48550/arxiv.2412.12591
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 30). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Sónia Reis, Jorge Baptista
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Authors retain copyright and concede to the journal the right of first publication. The articles are simultaneously licensed under the Creative Commons Attribution License, which allows sharing of the work with an acknowledgement of authorship and initial publication in this journal.
The authors have permission to make the version of the text published in RAPL available in institutional repositories or other platforms for the distribution of academic papers (e.g., ResearchGate).
