Limitations & Further Study
One limitation of this program was the compressed timeline. To align with CISLM’s schedule and reflect the pace of a small newsroom environment, the outreach, development, and testing phases took place over just three weeks before the chatbots were launched. This rapid pace limited opportunities for extended debugging or refinement. The 45-day running period was also relatively short. With a longer timeline, we could have updated the knowledge bases more consistently and evaluated the chatbot’s performance with more meaningful user engagement over time.
Another constraint is the absence of direct user feedback beyond the chatbot interaction logs. While conversation data provided insights into user interests and behavior, it could not capture user satisfaction, clarity of intent, or usability experience. We relied on newsrooms to share any unsolicited feedback they received from readers, but only one newsroom reported hearing from average users, and even those reflections were limited and anecdotal.Future iterations of this research would benefit from structured user studies, such as interviews, focus groups, or embedded feedback tools. These approaches could yield a more comprehensive understanding of how readers experience the chatbot, including ease of use, trust in responses, and overall value.
A potential area for further study is the collection and analysis of evaluable metrics data from newsrooms to better understand the chatbot’s impact. Although we invited participating newsrooms to anonymously share quantitative indicators such as website analytics, customer service tickets, and user engagement data through a post-run survey, only two provided usable responses. A future iteration of this program could incorporate structured data reporting requirements or built-in analytics tools to support a more consistent and representative evaluation.
Additionally, this pilot focused exclusively on text-based interfaces and content. The chatbot did not incorporate or respond to multimedia inputs such as images, audio, or video. As local newsrooms continue to experiment with diverse formats, future AI tools might be designed to support richer forms of content interaction and delivery.
We also did not evaluate the chatbot’s accuracy using systematic verification. While measures were taken to reduce hallucinations, including curated knowledge bases and prompt constraints, no formal assessment was conducted to measure factual correctness at scale. Establishing a framework for evaluating response accuracy, especially for time-sensitive or civic information, will be critical for responsible deployment.
Finally, the scope of this pilot was limited to information retrieval and content search. We did not explore applications of AI chatbots for task automation, such as submitting customer service requests or event listings. Integrating such features could further support newsroom workflows and expand the utility of AI tools for both internal operations and public engagement.