کنفرانس مهندسی برق ایران

صفحه اصلی / سی و چهارمین کنفرانس بین المللی مهندسی برق

Improving Persian Legal Question Answering: Reasoning Prompts, Retrieval, and Failure Analysis

نویسندگان :

Omid Moradi¹ Saeed Masoudnia² Alireza Hadi³

1- دانشگاه تهران، دانشکده علوم و فنون نوین 2- دانشگاه تهران، دانشکده علوم و فنون نوین 3- دانشگاه تهران، دانشکده علوم و فنون نوین

کلمات کلیدی :

Persian legal question answering،retrieval-augmented generation (RAG)،prompt engineering،large language model evaluation،Persian legal NLP

چکیده :

Large language models (LLMs) are increasingly used for legal question answering, yet their reliability remains limited in non-English and jurisdiction-specific settings. We present a model-agnostic retrieval-augmented generation (RAG) system for Persian legal multiple-choice question answering that improves accuracy across both open-source and closed-source LLMs. We curate an expert-validated benchmark of 120 questions from the 1403 Iran Bar Exam and evaluate 14 LLMs under five prompting strategies, including zero-shot, few-shot, Chain-of-Thought (Explain-to-Predict), and self-consistency. Our results show that reasoning-oriented prompting (Few-shot CoT E2P) provides the best non-retrieval baseline, achieving 45.3% average accuracy. When augmented with retrieved evidence from statutes and judicial decisions, our proposed RAG system increases the average accuracy to 55.52%, corresponding to a gain of 10.22% under the same prompting setup across 14 LLMs. A token-level analysis further indicates a verbosity-related failure mode, where incorrect predictions are often associated with longer explanations. Overall, our benchmark, structured evaluation, and RAG system provide a foundation for advancing Persian legal QA and other low-resource legal NLP settings.