Metaheuristic Hyperparameter Optimization for a Transformer Encoder: A Case Study on Detecting Depressive Signals in Tweets

Depression-related signals in social media have motivated the use of Natural Language Processing models for automatic text classification. This study investigates hyperparameter optimization of a Transformer encoder for detecting depressive signals in tweets. Differential Evolution (DE), Particle Swarm Optimization (PSO), Simulated Annealing (SA), and Random Search are compared with a predefined baseline under an equal evaluation budget. The protocol comprises 20 matched runs per method and 60 effective evaluations per search. All strategies significantly improved the baseline in accuracy and F1-score. DE achieved the highest mean accuracy (0.8768) and precision (0.8479), whereas PSO achieved the highest mean F1-score (0.8473) and recall (0.8610). Pairwise tests indicated comparable performance among search strategies. Winner-enrichment analysis further revealed significant preferences for higher learning rates, more attention heads, and fewer encoder layers, highlighting recurring patterns in competitive Transformer configurations.