Reflections from newsrooms
Atlanta Civic Circle: People used a tool when you have something for them
The Atlanta Civic Circle created a policy explainer chatbot that helped readers understand the local politics. They said that ACC was the very first local newsroom in Metro Atlanta that implemented an AI-powered tool like the chatbot, a new novel concept for the market.
Saba Long, the Executive Director at ACC, said that she was glad that people were willing to test this tool for consuming information and a new approach for reporting. She said that readers were looking for easy, convenient ways to get accurate, factual information. Compared to browsing through social media or using popular AI models like ChatGPT, this chatbot by the ACC is one of the best options available. “They know that if they go to the ACC site or any ACC platform, they (readers)’re going to get factual information,” said Long.
To introduce this tool to readers, they promoted this as a pilot program in their newsletter and a social media post. In the post, ACC briefly disclosed how they built this chatbot with CISLM and how it can help readers by answering questions based on previous reporting.
“I encouraged people to test it out and to tell us what they liked about it, and what they would like us to consider in a future iteration.”
Besides individual users, ACC heard from local nonprofits about the chatbot. Some organizations said that this is an interesting tool and might be adaptable for service oriented nonprofits.
A limitation of this chatbot, according to Long, was that there were inquiries that it wasn’t able to answer. In response to unsolvable questions, Long would like the chatbot to bring up relevant reporting to users when the questions cannot be answered directly. And having reporters follow up with readers when unanswered questions occurred could be another way to improve user experience.
For example, a reader asked a question about their public service commission district. As Atlanta’s primaries were going on, ACC mentioned that it would be great to follow up with readers to help them figure these out, or at least point them to the direction.
Looking further at the future iterations of this product, ACC would love to update the knowledge base and prepare it for Atlanta’s upcoming 2025 election. They said that they would like to add their latest reporting promptly so that the chatbot can provide the most up-to-date information.
Chapelboro: A tool that was not effective enough
Chapelboro built a chatbot that answered readers’ inquiries about their reporting and the newsroom. It was a combination of information about the radio station and the coverage in the past year.
The structure of Chapelboro’s homepage presented a bit of a challenge for initial implementation of their chatbot, which they named Chappy. The Chapelboro team eventually created a graphic and attached a link to a full-screen page of the chatbot.
For promoting Chappy, Chapelboro shared the chatbot on their social media and radio podcast once, framing as a tool to “better help (users) navigate our website and the content,” said Brighton McConnell, the News Director. They also mentioned that this is a partnership project that helped with CISLM’s research.
However, the chatbot was not as effective as Chapelboro hoped for. McConnell said that even with a knowledge base with content from the newsroom, the chatbot still occasionally failed to give the correct answers. “You still had to tailor questions very specifically to get the response.” For questions phrased differently, the chatbot often answered “I don’t know.”
He said that the chatbot provided incorrect answers from older stories, which happened both in readers’ questions and the team’s testing of the chatbot.
“Some of it was a flaw in how we approached it,” said McConnell. Chapelboro would like the chatbot to be a “super search” tool which people can interact with to get answers about content on their website, including previous stories. But people “came in thinking, oh, we can just ask this anything about Chapel Hill, or anything about the University, or who Bill Belichick is, and you’re not going to get an accurate response from that.”
Usage of the chatbot itself was small compared to the website’s usual traffic or their normal audience numbers, but the team at Chapelboro expected low traffic for the chatbot. “The project from our perspective was to just see if a tool like this makes sense, and how it would be received by our audience.”
Based on direct feedback from readers, many weren’t big fans of the chatbot. This might be in part because of its function. Others did not like how the chatbot graphic was positioned on the website, complaining about accidentally clicking on it. A small proportion of readers also messaged the team about taking the chatbot down.
During the running period, journalists at Chapelboro didn’t feel the chatbot was helpful to their audience or the team. Most of the time, their interaction with the chatbot was testing it by trying it out themselves and making sure that users got correct responses.
The Chapelboro team were hesitant about adapting to a product like this after the research period. Even if, in the long run, they would like to implement it again, “we’d probably want a little bit more to make sure it’s exactly what we were looking for,” said McConnell, “and putting it out for our audience as fleshed out as possible.”
Despite not being satisfied with the chatbot’s performance, Chapelboro said that they were interested in the option of using chatbot as their website’s super search function, if it one day could be mistake-free.
Knowing the limitations of the chatbot also informed Chapelboro how to better approach similar tools and work on what they can do. For example, the team said that using it for more static information queries, such as questions only about the station, could lower its chance of making mistakes and efforts to maintain it.
Henrico Citizen: it helped providing basic information
Henrico Citizen is a small digital newspaper covering Henrico County in Virginia since 2001. In this program, they created a chatbot called “Henry” to manage general inquiries to the newsroom and answer questions about stories in the past year.
Tom Lappas, the founder of Henrico Citizen, said that the idea of having an automated way of answering readers’ questions that didn’t involve journalists’ time was appealing to him. As a team with 3 full-time and 3 part-timer journalists, they would like this to be an efficient tool that answers readers’ questions.
“Henry” received 23 inquiries in the 45 days it was installed, which is a small amount compared to the average monthly viewers of Henricon Citizen’s website (about 60,000 to 85,000). Lappas said that he didn’t want to promote “Henry” knowing the limitations it had. He wanted to “let people discover it” on the website with the chatbot’s minimalistic but still discoverable display.
35% of the inquiries were about advertising with Henrico Citizen, one of the most frequent topics users asked the chatbot about. This might be because “How to advertise with Henrico Citizen” was one of the four prompt questions of “Henry.” However, the team did get a few more advertising interest email requests throughout the running period than normal, which might have come as a result of the chatbot, according to Lappas.
He said that the chatbot was helpful in providing basic information outlined on the website that was less visible to readers, such as how to advertise. “It’s not as obvious to typical readers,” Lappas said, “so for them to be able to ask somebody (questions) like where do I find something could be pretty valuable.”
There wasn’t direct feedback from readers, either positive or negative.
Lappas said that he wished the chatbot could update itself after the initial build and spontaneously update as new stories were published on the website. With that, he hoped the chatbot could work as an alternative to the website’s internal search bars which “are not even that great.” For example, if a reader is interested in an article written four years ago, the chatbot could “hunt it down for me, or give me some idea about where it might be.”
He mentioned adding unique interactions that took advantage of their website’s existing infrastructures such as story tags. “It’d be nice to be able to, say like, we want you (the chabot) to learn every article that we tagged with this tag”.
Maintenance and updating the chatbot was costly for Henrico Citizen in the long run due to the limited amount of time they could put on this tool. For the team’s tight schedule and heavy workload, they weren’t able to spend much time building or testing the chatbot during the program. If Lappas was to do something differently, he said that he would have put more information together to the chatbot’s knowledge base so that it’d be a more robust tool.
The News Reporter: Find the best way to integrate the chatbot
The News Reporter is a local newsroom based in North Carolina. They set up a customer service type of chatbot during the program to answer general inquiries from readers. Because they constantly receive customer service questions, they would like a chatbot to aggregate their guidance and provide prompt, precise responses to readers even outside regular worktime.
The team put together a spreadsheet for commonly asked questions such as subscription management, advertising, and tips. With that, we created a chatbot called “the News Reporter Help Desk” and put it on the “help” and “about us” pages of the website. On top of this, we implemented a workflow of asking for users’ name and contact information when the chatbot failed to answer a question so that staff from the team would be able to follow up.
After launching the chatbot, the team promoted it in their newsletter that goes out to about 8,000 subscribers with a brief intro and the link to it. Although that generated some uses, the total numbers were still low: it received a total of 11 inquiries between May 1 and June 15.
Rachel Smith, the Director of Operations of the News Reporter, said that this might be because most of their traffic usually came from their mobile app rather than website. “I think it (the chatbot) has a lot of good potential,” said Smith, “I just don’t know that we have it placed correctly.”
Smith said that there was a trade-off between visibility and distraction. The team gave away the pop-up function so that readers can focus on the content of the webpage instead of getting intervened by the chatbot. In retrospect, Smith said that “it almost would have to be a pop up or something a little bit more visible to grab people.”
During the pre-launch testing period, the chatbot showed hallucinations, specifically “making up an office location, days of the week that the paper comes up, and the price of a single-copy paper.” These errors were traced to the absence of explicit information in the structured knowledge base. While a link to the newsroom’s website was included in the dataset, details on the website were outdated.
To fix this, we added the relevant information directly to the knowledge base in structured text form and revised the prompt instructions to ensure the chatbot referenced only validated entries. These changes significantly reduced hallucination risk. Following this debugging, no further hallucinations were observed during subsequent testing or post-launch operation.
For potential use in the future, Smith said that the chatbot can be built off the current model to support a variety of needs from readers, such as locating content on the website or in the archives.