Comparative evaluation of LLMs with RAG to support information on Childhood Vaccination

The complexity of the vaccination schedule of the National Immunization Program (PNI), with rules that vary by age, dose, interval, history, and special situations, can make quick consultation difficult for Community Health Workers (ACS). This work evaluates a Retrieval-Augmented Generation (RAG) architecture as an informational support tool based on official documents, comparing five Large Language Models (LLMs): DeepSeek Chat, Gemini 2.5 Flash, GPT-4o, LLaMA 3.3-70b, and Mistral Large. Thirteen questions were evaluated, repeated 10 times per model, at temperatures 0.1, 0.2, 0.4, and 0.6, with BLEU and BERTScore used as evaluation metrics. Overall F1 differences were small; however, queries requiring the combination of multiple rules distinguished better among the models. DeepSeek Chat combined competitive performance with low cost.