Comparison of LLM Prompting Strategies for Identifying Indonesian-Language Hoax Narratives

Abstract

Large language models (LLMs) can classify text through instructions alone, but their output may depend on the prompting strategy, and the effect of adding labelled examples on the identification of Indonesian-language hoax narratives has rarely been tested empirically. This study aimed to (1) compare the initial results of zero-shot and few-shot prompting with 2, 4, and 8 examples, (2) examine the relationship between the number of examples and latency, and (3) formulate a validation protocol for trustworthy HOAX/NON-HOAX evaluation. A comparative quantitative experiment applied the four configurations to a single LLM accessed through an API, using a dataset of 4,000 narratives (1,000 HOAX and 3,000 NON-HOAX). Recorded accuracy, macro-F1, and mean latency were audited analytically through single-class metric identities, 95% Wilson intervals, and a least-squares latency trend. The recorded test set contained only 400 HOAX items, and every recorded macro-F1 equalled A/(1 + A); the recorded accuracies (86.75%, 65.75%, 70.50%, and 52.00%) therefore represent hoax recall rather than binary performance. Hoax recall was highest for zero-shot, and its gap from every few-shot configuration exceeded sampling uncertainty. Mean latency rose almost linearly from 0.418 to 0.923 seconds, about 0.065 seconds per added example, reaching 2.21 times the zero-shot value. Adding examples increased computational cost without improving hoax recall, whereas binary classification performance remains undetermined. Because the model identity and few-shot examples were not preserved, the proposed protocol stratified splitting, leakage-free example selection, per-class metrics, and complete model documentation must be satisfied before any configuration is declared superior.