How News Consumers Interacted with Chatbots: Findings from Analysis
Between May 1 and June 15, the chatbots received a total of 185 inquiries. For getting a validated amount of exposure and usage for analysis, we asked each newsroom to promote the chatbot to their audiences once on social media. Besides the post, Chapelboro briefly mentioned the chatbot in their radio podcast, too.
A portion of conversations may include occasional testing conducted by newsroom staff during the period. While we were not able to reliably identify test conversations, we estimate they represent a minor share of the total activity based on interviews with newsrooms.
All data collected is anonymous and does not contain any personal information. The raw records include only the user input, chatbot responses, and timestamps.
1) Conversations with chatbots tended to end fast, but content-focused bots received more follow-up questions compared to customer service bots on average.
Among all conversations, only about 41% included more than one user question. The remaining contains a single question and a response from the chatbot.
Content-related bots showed significantly higher engagement than customer service bots. About 43% of conversations with content-based bots included more than one question, compared to no follow up question for customer service bots.
The number of questions per inquiry varied slightly across chatbots. For “Chappy,” the average was two questions per conversation, with a maximum of ten in a single session. While questions within the same session were not always direct follow-ups, they tended to be thematically related. For instance, one user asked about college sports at UNC, including baseball and football, and then shifted to questions about community events.
2) Users asked about reporting, but sometimes got creative, too
During the pilot, users of content-focused chatbots at Chapelboro and ACC often engaged with the bots to learn more about the newsrooms’ reporting. Many of these inquiries reflected interest in local issues the newsrooms regularly cover.
At Chapelboro, users frequently asked about community events(16 mentions), followed by facts about the newsroom(15), local government (15), and sports (13). Other questions touched on higher education, particularly UNC-related topics, and some even asked about the chatbot’s capabilities. A smaller portion of queries diverged from the news altogether, including topics such as math, poetry, cooking, and archival access.
ACC users focused on civic issues, the beats that this chatbot was built upon. The most common topics of interest included city officials (29), elections (19), and local government (6). There were also some outlier inquiries that strayed from the core editorial scope, with users asking about DOGE, finance, and school board activities.
These patterns suggest that while users largely treated the chatbots as tools for exploring reported content, they occasionally used them as open-ended search assistants. Some tested the chatbot’s limits or playfully interacted with it, which is consistent with typical early-stage chatbot usage in public-facing platforms.
Although “Henry,” the chatbot for Henrico Citizen, was also designed to answer questions about the newsroom’s reporting, users primarily treated it as a customer service tool. The most common inquiry was about the chatbot itself (“Who is Henry?”), which appeared in 11 of the 23 recorded conversations. Seven sessions included questions about how to advertise with Henrico Citizen. Only three questions across all conversations focused on reporting related to local government.
A common pattern evolved from previous usages is asking for news feeds directly to the chatbot. For example, 7 conversations with Chapelboro’s chatbots included “the latest reporting”, “the most recent articles”, and similar requests.
For content-related chatbots (ACC, Chapelboro, and Henrico Citizen), “what” is how most questions start with — users are trying to make sense of something, including getting a definition (“What does a sheriff do?”), asking about facts (“What is happening this week downtown?”), and sometimes opinions (“What is the best restaurant?”).
We also found that for ACC’s politics explainer chatbot, 31% of questions started with “How”, meaning that more actionable information was requested by users regarding local politics. Examples included “How do I run?” as well as questions aimed at understanding political processes, such as “How are my property taxes decided?” or “How does the candidate stand on this issue?” Note that these question wordings have been slightly modified to protect user privacy.
For customer service chatbots (the News Reporter), users interacted with the chatbot with more focused purpose. More than 90% of the questions were about general inquiries such as subscription management, tip, or obituary submission. The chatbot gave actionable answers or pointed users to related references.
3) Prompt questions influenced user engagement, with usage rates varying widely across newsrooms depending on the number and relevance of the questions provided.
Approximately 34% of users clicked on the prompt questions rather than asked their own. Each chatbot is pre-registered with 3-5 prompt questions to provide some examples of what questions it can answer. These questions are designed by newsrooms based on their understanding of what readers may need or be interested in.
Here are each news organization’s prompt questions:
Henrico Citizen
The News Reporter
Atlanta Civic Circle
Chapelboro
An interesting finding is that there is a huge disparity between prompt question usages of different newsrooms. Chapelboro had relatively lower percentages of prompt question usage. Users of these chatbots are more likely to come up with their own questions. For the News Reporter’s and ACC’s chatbot, about 1 in 3 conversations included a prompt question. Last, 83% of Henrico Citizen’s chatbots received prompt questions, making up the highest percentage of usage rate.
One potential reason behind this giant difference is that Henrico Citizen’s chatbot had more prompt questions listed than any other chatbots. And these questions range from customer service questions to content-based questions, which might be representative for the majority of users’ inquiries.
Measuring the effectiveness of chatbot: did these AI answer users’ questions?
Handling Unknown Questions
To reduce the risk of hallucination or inaccurate responses, we’ve configured the chatbot to rely solely on its backend knowledge built by us with newsrooms’ materials. The chatbots did not access or use external resources on the internet.
However, this setting limits the information available to the chatbot and might cause the chatbot to answer “i don’t know” to some questions that seem easy and common. When users asked questions not covered in the chatbot’s knowledge base, the chatbot answered with the preset response such as “I don’t know” or encouraged users to follow up with the newsroom directly.
For example, when a user asked the Atlanta Civic Circle’s chatbot “who is the current president of the United States”, the chatbot was not able to give the correct answer since there is no relevant reporting involving President Donald Trump in the chatbot’s knowledge base. On the other hand, when asked who the mayor of Atlanta City is, the chatbot gave the correct and detailed answers based on previous reporting.
About 32% of total conversations included at least one unanswered question that triggered a fallback response like “I don’t know.” In many cases, these were general knowledge questions that an average person could answer but were not addressed in the available content. Other questions were either too vague or unclear for the chatbot to process effectively.
Challenges with Time-sensitive Information
A recurring challenge for content-based chatbots was identifying and responding appropriately to time-sensitive information. When users asked for “latest stories,” the chatbot often failed to retrieve the most recent content in the database. Some newsrooms noted that stories gathered before the testing period quickly became outdated, and the lack of ongoing updates to the knowledge base reduced the accuracy and usefulness of responses.
Timeliness is a core value in news reporting, but it also presents a moving target for content-based chatbots. As new, time-bound stories are continuously published, bots relying on static knowledge bases can quickly become outdated. Without mechanisms for real-time updates or contextual understanding of time references, chatbots risk returning responses that are incomplete, outdated, or inaccurate.
These observations suggest that while the chatbots performed reliably within their scope, maintaining relevance and accuracy in fast-changing news environments remains an ongoing challenge.