About BREV-RAG@SIGIR-AP 2025

BREV-RAG (Beyond Relevance-based EValuation of RAG systems) is an in-person workshop
that will take place on Day 4 of SIGIR-AP 2025 (Xi’an, China).

---

CALL FOR PAPERS

BREV-RAG (Beyond Relevance-based EValuation of RAG systems) is a new half-day workshop.

Part I (90 minutes) of BREV-RAG will feature refereed paper presentations that address evaluation issues based on axes other than just relevance, correctness, or groundedness – for example, group fairness [1], modesty (Can the RAG systems attach appropriate confidence scores to their responses that align with response accuracy? [2]), etc. More examples of evaluation axes can be found in [3] (Table 1). The organisers will also provide a short opening talk.

Part II (90 minutes) of BREV-RAG will be breakout sessions, where each group will be asked to discuss how to evaluate RAG systems using a specific evaluation axis. The axes will be chosen based on the accepted papers, the organisers’ talk, or new suggestions from the workshop participants. Each breakout group will submit a one page summary of their discussion to the organisers by March 1. The organisers will merge the summaries and submit a workshop report to SIGIR Forum by May 1 (for the June 2026 issue).

This CFP solicits paper contributions for Part I mentioned above.
Each relevant submission must clearly state what evaluation axis/axes are being addressed. Reports on ongoing research and late breaking results are encouraged. Each submitted paper will be reviewed by our program committee. Our current plan is to publish the accepted papers as a CEUR workshop proceedings.

At least one of the authors of each accepted paper must register for and attend the workshop in person.

---

Submission format

Please use this Overleaf template. Here’s a sample paper pdf.
In the Introduction section, please clearly state what evaluation axis/axes (other than relevance, correctness, or groundedness) you are addressing.

Anonymity

No need to anonymise your paper – we adopt single-blind reviewing.

Page limit

[At submission time] 5-10 pages (including everything except references) + unlimited #pages for references.
Upon acceptance, we will ask the authors to extend the paper by taking the reviewers’ feedback into account, with a page limit of 12 pages + unlimited #pages for references (as CEUR requires each paper to be at least 12,500 characters or so).

Submission site

https://easychair.org/conferences?conf=brevrag2025

---

IMPORTANT DATES (AoE except where indicated)

September 30, 2025 Workshop paper submissions due
October 14, 2025 Workshop paper notifications
November 3, 2025 (tentative) Camera-ready papers due
December 10, 2025 (China Time) BREV-RAG@SIGIR2025 in Xi’an, China
March 1, 2026 Deadline for breakout group summaries from workshop attendees

---

Program Committee

Tetsuya Sakai, Waseda University (workshop organiser)
Sijie Tao, Waseda University (workshop organiser)
Zhicheng Dou, Renmin University of China (workshop organiser)
Junjie Wang, Tsinghua University (workshop organiser)
Haoxiang Shi, Inner Mongolia University of Technology (workshop organiser)
Nuo Chen, The Hong Kong Polytechnic University (workshop organiser)
Atsushi Keyaki, Hitotsubashi University (workshop organiser)
Maria Maistro, University of Copenhagen
Marwah Alaofi, RMIT University
Haitao Yu, University of Tsukuba
Yuto Nakachi, University of Tsukuba
Negar Arabzadeh, University of California
Fabrizio Silvestri, University of Rome
Giovanni Trappolini, University of Rome
Federico Siciliano, University of Rome
Mohammad Aliannejadi, University of Amsterdam
Nandan Thakur, University of Waterloo

---

Contact

brev-rag-org at list.waseda.jp

---

References

[1] Tao et al.: Overview of the NTCIR-18 FairWeb-2 Task
[2] Sakai: Evaluating System Responses Based On Overconfidence and Underconfidence
[3] Sakai: A Generic Framework for Auditing Textual Conversational Systems