New York City’s business chatbot is reportedly giving out “dangerously inaccurate” information

An AI chatbot launched by the New York government to help business owners access relevant information is spewing falsehoods, sometimes even misinforming users about actions that violate the law, according to a report from Marking. The report, which was published in conjunction with local nonprofit newsrooms Documented and The cityincludes numerous examples of inaccuracies in the chatbot’s responses to questions related to housing policies, workers’ rights, and other topics.

Mayor Adams’ administration introduce the chatbot in October as a complement to the MyCity portal which launched in March 2023 as a ‘one stop shop for city services and benefits’. The chatbot, powered by Microsoft’s Azure AI, is aimed at current and aspiring business owners and is billed as a source of “actual and reliable information” that comes directly from city government sites. But it’s a pilot program, and the website’s disclaimer notes that it “may from time to time create incorrect, harmful or biased content.”

in Marking, the chatbot repeatedly provided incorrect information. In response to the question “Can I make my store cashless?”, for example, he replied, “Yes, you can make your store cashless in New York” — despite the fact that New York City banned cashless stores in 2020. The report shows that the chatbot also answered incorrectly about whether employers can accept their workers’ tips, whether landlords must accept Section 8 vouchers or renters for rental assistance, and whether businesses must inform staff of schedule changes. A housing policy expert who talks to Marking called the chatbot “dangerously inaccurate” at its worst.

The city indicated that the chatbot is still a work in progress. In a statement to Engadget, Leslie Brown, a spokeswoman for the New York City Office of Technology and Innovation, said, “In keeping with the city’s key principles of trustworthiness and transparency around AI, the site informs users that the clearly labeled beta pilot product should only be used for business-related content, tells users that potential risks exist and encourages them through a disclaimer to both double-check his answers with the links provided and not to use them as a substitute for professional advice.”

“The site has already provided thousands of people with timely and accurate answers and offers a feedback option to help drive continuous improvements to the beta tool,” Brown said. “We will continue to focus on building on this tool so we can better support small businesses across the city.”

Update, March 31, 2024, 9:23 a.m. ET: This story has been updated to include a statement from the New York City Office of Technology and Innovation.