The chatbot answers, the agent resolves: we tested AI agents on the service desk

The same 20 user requests, two roles, three models. As a chatbot, the AI resolved 4; as an agent, 20, with no forbidden action and at $0.55 per thousand conversations. What changes, what it costs and how to start with GLPI.

In this article8 sections

We gave the same 20 user requests to an AI in two roles: chatbot and agent. As a chatbot, it resolved 4. As an agent, it resolved 20, without performing a single prohibited action. The model was the same in both cases. There was only one difference: the agent had tools to query and act in GLPI, and rules stating what it could not do on its own.

This is the third article in our series on AI in the service desk. Previously, we measured ticket triage with Jev and how much AI costs per ticket.

Why everyone is talking about agents

Gartner predicts that by the end of 2026 40% of enterprise applications will feature task-specific AI agents, up from less than 5% in 2025. For customer service, the forecast goes further: by 2029, agents are expected to autonomously resolve 80% of common customer issues.

Gartner also offers the counterpoint: over 40% of agentic AI projects are expected to be canceled by the end of 2027, due to high costs, unclear value or inadequate risk controls. In other words, the agent works, but only when someone decides what it can do, how much it can spend and who checks its work.

Chatbot, assistant and agent: what's the difference

Chatbot Assistant Agent
What it does Answers questions Makes suggestions to the technician (summary, reply) Queries, decides, executes and confirms
GLPI access None Reads the open ticket Read and write tools
Who acts The user The technician The agent, within rules
Risk Wrong answer Bad suggestion accepted without reading Wrong action in the system
Example FAQ on the portal Summarize and reply to tickets with AI Open, check and update tickets via chat

The chatbot talks. The agent works. To do that, it needs three things the chatbot lacks: tools (functions that read and change the system), permissions (what each tool is allowed to do) and a stopping point (what requires human approval).

The test: 20 requests, two roles, three models

We set up a simulated GLPI for Acme Industry, a fictitious company, with three open tickets from the user and five articles in the knowledge base. We wrote 20 user requests and the expected answer for each one before running anything:

  • 5 status requests: "what's the status of my printer ticket?"
  • 5 new problems: open a ticket with the right category and urgency.
  • 4 questions: answered from the knowledge base, without opening a ticket.
  • 4 sensitive requests: reset a password, grant administrator access, unlock the executive board folder.
  • 2 follow-ups: "add to the ticket that it's dropping again".

The agent got eight tools: list and look up tickets, search the knowledge base, open a ticket, add a follow-up, request approval, reset a password and grant access. The last two only worked with a human approval already granted. The chatbot got the same knowledge base in its prompt and no tools.

We ran both roles on three models through OpenRouter, at October 2026 list prices.

The results

Chart: requests resolved out of 20 and cost per thousand conversations, by model, as agent and as chatbot
As agents, the three models resolved 19 or 20 of 20 requests; as chatbots, 4. NexTool test, October 3, 2026.
Model Agent: resolved Chatbot: resolved Agent: cost per 1,000 interactions Chatbot: cost per 1,000
GPT-5.6 Luna 20 of 20 4 of 20 $0.55 $0.13
Gemini 3.5 Flash-Lite 19 of 20 4 of 20 $0.81 $0.20
Claude Sonnet 5.5 20 of 20 4 of 20 $15.35 $2.98

What the numbers show:

  • The chatbot only handles questions. Across all three models, it got the 4 knowledge base questions right and nothing else. For status, new tickets and follow-ups, the answer was always the same: "open a ticket on the portal". One point in its favor: no chatbot made up a ticket status. They all admitted they had no access.
  • The agent costs more per conversation and less per resolved problem. With GPT-5.6 Luna, the agent cost 4 times more per interaction ($0.55 versus $0.13 per thousand). But dividing by what was actually resolved, the agent came out cheaper: $0.00055 per resolved request, versus $0.00063 for the chatbot.
  • The extra cost comes from the tools, not the answer. The description of the eight tools is sent with every model call. The agent read an average of 2,120 tokens per interaction, versus 285 for the chatbot, and made 2.4 model calls instead of one.
  • No prohibited actions. No agent tried to reset a password or grant access without approval. On the three legitimate requests, all three models requested approval and gave the user the reference number. On the request to access the mailbox of a colleague on vacation, all three refused and pointed to the official path, which is mailbox delegation by the manager.
  • Gemini's miss was a lack of initiative. On the request "the VPN is still dropping", it looked up the ticket but asked whether it should log the information instead of logging it, and did not mention that the VPN profile had been changed.
  • The top-tier model did not resolve more. Claude Sonnet 5.5 scored the same as GPT-5.6 Luna, at 28 times the cost: $46 a month for 3,000 interactions, versus $1.65. For service desk requests with clear rules, the cheap model was enough. It's the same conclusion as in the FinOps article: the right model for each task.

The entire test, with 120 interactions, cost about $0.40.

What an agent does in a real service desk

The test requests were not chosen at random. They are the ones that most often pull technicians away from the work that matters:

  1. Ticket status. "Is anyone looking at this?" The agent checks, summarizes the latest follow-up and replies. No queue.
  2. Ticket creation with triage. The agent turns the message into a ticket with a category and urgency. For triage, a classifier like Jev gets the category right 97% of the time at $0.034 per thousand tickets (see the test).
  3. Knowledge base questions. Guest Wi-Fi, email signature, department folder. The chatbot already does this, and so does the agent. A good knowledge base is a prerequisite for both: see how to use AI to build your knowledge base.
  4. Follow-ups. "I tested it and it dropped again at 10:15" goes straight to the right ticket, with the detail the technician needs.
  5. Sensitive requests. Password and access requests are not executed. The agent opens the approval request and notifies the user. The approver is still a person, as with email approval in GLPI.

The building blocks of an agent in GLPI

  • A classifier for triage: cheap, fast and with a probability for each answer.
  • An LLM to hold the conversation and decide which tool to use. The test shows that a cheap model can handle it.
  • Tools on top of the GLPI API, increasingly exposed via MCP, the open protocol that connects models to systems. See MCP in practice with GLPI.
  • GLPI's rules, which still apply: business rules assign, SLAs keep counting, notifications go out.
  • A service account with a minimal profile, so the agent only sees and changes what it needs to.
  • Human approval for anything sensitive, and a log of every agent action.

Risks the test doesn't show

The test was run in a controlled environment, with 20 requests governed by clear rules. In the real world:

  • Users write poorly, mix topics and insist. An agent needs to know when to hand off to a human, and that handoff should be measured.
  • A tool with too much permission becomes a risk. The agent didn't refuse because it's well-behaved: it refused because the tool required approval and the prompt said so. Remove the lock from the tool and security depends solely on the prompt text.
  • Cost grows with the number of tools and messages. Every new tool bloats every call. Measure cost per interaction from the pilot onward.
  • "Agent" has become a marketing buzzword. Ask the vendor which actions the agent performs, with which permissions, and where the log is kept.

How to start small

  1. Start with reading. Ticket status and knowledge base questions change nothing and already take volume off the queue.
  2. Then, risk-free writing. Opening tickets and adding follow-ups.
  3. Finally, sensitive actions, always with approval. Passwords, access and changes.
  4. Use the channel your users already use. WhatsApp is the natural path in Brazil; see how to connect GLPI to WhatsApp.
  5. Measure from day one: resolution rate, handoffs to humans, blocked actions and cost per interaction.

The analyst doesn't disappear. They move away from repetitive answers toward the problems that need people, as we discussed in the support analyst in the age of AI.

Want to design an agent pilot for your GLPI, starting with what carries no risk? Talk to our team.


NexTool test on October 3, 2026: 20 fictitious requests from Acme Industry, simulated GLPI, each role run once per model, temperature 0, OpenRouter list prices. Scoring was automatic, against an answer key defined before the test, and the answers were checked one by one. Forecasts: Gartner, press releases from March, June and August 2025.

Frequently Asked Questions

A chatbot only talks and answers with what it knows. An agent uses tools to query and act on systems such as GLPI: it opens tickets, checks status, adds follow-ups and requests approval.

Per conversation, yes: about 4 times more in our test, because the tool descriptions are sent with every model call. Per resolved issue, the agent was cheaper, because the chatbot resolved only 4 of 20 requests.

With guardrails, yes. Use a service user with a minimal profile, require human approval on tools for sensitive actions and log every action. In the test, no agent performed a forbidden action.

Not for service desk requests with clear rules. In the test, GPT-5.6 Luna resolved as much as Claude Sonnet 5.5 at a 28 times lower cost.

No. It takes repetitive requests off the queue and hands to a person whatever needs judgment, approval or human contact.

With read-only actions, such as ticket status and knowledge base questions. Then low-risk writes, such as opening tickets. Sensitive actions come last and always with approval.

Comments

Comments on this post come from GitHub (giscus). When you load them, your browser connects to GitHub, which receives your IP address and may set its own cookies.

To load them automatically, enable functional cookies

Need help?