A chatbot designed for routine customer support went beyond its assigned role and began acting as an authorized representative of the company. It independently accepted business meeting times, confirmed my attendance and invented reasons for why I failed to appear. I only learned about the correspondence, which had continued for days, when a potential business partner called me.
The phone call began with a question that caught me completely off guard: Why was I failing to show up for scheduled meetings?
I had no idea what meetings he was referring to. I had not accepted the proposed times, and I had not confirmed that I would attend. The person on the other end, however, had been receiving exactly those kinds of messages for days from my company’s business email account. 
Only after that phone call did I learn that the correspondence existed at all. I then reviewed its contents and determined what had happened.
The Chatbot Was Supposed to Answer Only Basic Questions
The chatbot was integrated into my company’s website, business email, Facebook, Instagram and WhatsApp. It was created to automate a simple and routine task: answering basic questions about the company and the services we provide based on documentation prepared in advance.
If the documentation did not contain an answer to a question, or if a user wanted to reach a specific agreement, the bot was supposed to direct that person to contact me by email or phone. It had a predefined response for such situations.
The system prompt had been carefully written, the documentation prepared, and the boundaries of the chatbot’s role clearly defined. The bot was explicitly prohibited from presenting itself as a real person, employee or, in this case, company director, from speaking on my behalf or from assuming obligations without my approval.
I had already developed similar customer-support systems for several companies, and they had operated without any problems. The bot itself was well built and successfully performed the tasks for which it had been designed—until this incident.
The Bot Scheduled a Meeting on My Behalf
The problem began when a potential business partner sent an email proposing an online meeting to discuss possible cooperation in greater detail. Instead of directing him to me, as instructed, the chatbot took over the communication and began arranging the meeting on its own.
When the other party sent a Google Meet link and said he was waiting for me to join, the bot responded in the first person as though it were me and said I would join immediately.
The chatbot obviously could not join the video call, nor did it inform me that it had accepted the meeting time on my behalf. Nevertheless, its messages gave the potential business partner an unequivocal confirmation that I would attend.
When I failed to appear, the other party sent another email saying that he was waiting for me and had tried several times to reach me by phone. Even then, the chatbot failed to use the predefined fallback response and hand the communication over to me. Instead, it invented a claim that technical problems had prevented me from attending, apologized for the missed meeting and offered to reschedule.
When the other party suggested holding the meeting two hours later, the bot accepted the new time. After receiving a new invitation and access link, it again confirmed that I would join the conversation.
Most troubling of all, the messages were signed “Milena SMARTGORITHM.” Although the messages indicated that the content had been generated by artificial intelligence, the way the responses were worded created the impression that the bot was communicating my decisions and acting with my approval.
The Bot Violated Its Instructions for Days
This case cannot be reduced to an ordinary hallucination in which a model gets a fact, name or date wrong. The bot independently made decisions in my name, provided false information about my availability and invented a reason for my absence from a meeting, even though it had no authority to do so.
The correspondence continued for days. Every new message was an opportunity for the system to recognize that the matter was outside its authority and direct the other party to a human—namely, me. Instead, it remained in the role it had assumed and maintained the appearance that it had information about my intentions and obligations.
It is particularly important that the other party was not attempting to manipulate the chatbot or induce it to violate its restrictions. There was no provocation, hidden instruction or attempt to circumvent the rules. The person was conducting ordinary business communication, while the system itself stepped outside its assigned role.
What Happened Inside the System
The chatbot’s behavior can be reconstructed precisely from the preserved correspondence, but the cause of its initial departure from the instructions cannot be determined with confidence.
There is no basis for attributing the problem to poorly prepared documentation, an unclear prompt or the other party’s behavior. The system had all the information and restrictions necessary to respond correctly, and the situation it encountered was routine. It only needed to state that it could not confirm the meeting time and direct the person to the appropriate contact.
What can be established with certainty is that the system experienced a prolonged failure to follow its instructions. After accepting the meeting once, the bot continued to treat its own false statement as a real business obligation. When the promised attendance did not materialize, it created another falsehood to explain the first.
Why the bot abandoned its assigned role remains unknown. The preserved correspondence clearly shows how it behaved, but the available information is insufficient to determine reliably what triggered that behavior. Any more specific explanation at this stage would therefore be speculation.
Similar Problems Have Not Occurred With Much More Complex Chatbots
Incidents like this have not occurred with other chatbots I have created, including chatbots representing historical figures, memorial chatbots or systems used in educational institutions.
The paradox is that the serious problem did not occur with the most complex chatbots, but with a system responsible for completely routine customer support.
Although those chatbots are significantly more complex, rely on much more extensive documentation and are subject to numerous content, historical and ethical restrictions, they have not made errors, hallucinated or stepped outside their roles. Users have deliberately tested them many times and tried to induce them to say things they were not permitted to say, but the systems did not give in to those attempts.
When a Language Error Becomes a Business Action
The difference between an inaccurate answer and exceeding authority is not merely terminological. Incorrect information may mislead a user, but the consequences become far more serious when a bot assumes an obligation on behalf of a person who does not even know that the communication is taking place.
In this case, the consequences were missed meetings and damaged business communication. In another situation, a bot could similarly promise a service, accept terms of cooperation, approve a discount or communicate a decision to a client that the company had never actually made.
I still have not determined the cause of this incident. So far, I have not encountered publicly reported experiences from users who faced a similar problem, but the absence of such reports does not mean that these incidents have never occurred. System owners may not yet have discovered them, may not have fully recognized their seriousness or may choose not to discuss them publicly because of the potential damage to client trust.
Incidents like these, however, need to be discussed openly. Every documented case should be analyzed to determine its causes and prevent a recurrence. Ignoring the problem only creates a false impression of the reliability of artificial intelligence and delays the search for solutions.
You can view the full conversation in the photo gallery below the article.
Photo: Private Archive
This article is protected by copyright. It may not be copied, reproduced, adapted, or republished, in whole or in part, without prior written permission from the editors of Chicago na dlanu.
