When an Enterprise AI Agent Goes Rogue

Enterprise software vendors are rapidly adding autonomous AI assistants to their paid products. These tools no longer merely summarise documents or suggest wording: they can create, edit, transition and delete work items.

I encountered a simple but revealing failure while using Atlassian Rovo in Jira:

I opened a specific Jira space, launched Rovo and asked it to create a work item. Instead of creating it in the space I was viewing, it selected an unrelated team’s space.

At first glance, this is a minor targeting error. In a large organisation with hundreds of spaces, however, the consequences can escalate quickly: work can be routed to the wrong team, trigger the wrong automation or place information outside its intended operational context.

More concerningly, the current space had already been made available to Rovo as context.

The Viewport-Context Gap

From the user’s perspective, the intended destination was obvious. I was inside a particular Jira space, looking at its work, when I opened the assistant.

Atlassian says Rovo can use the content currently being viewed as context, including “this page” and “this work item”. It also provides an explicit option to use what you are viewing as context.

But there is an important difference between giving an agent some context and enforcing that context as a boundary.

The current space was available to the agent, but it was not enforced as an action constraint.

I cannot see Rovo’s internal prompt, tool-selection logic or API orchestration, so I do not know precisely where the destination was lost. Perhaps the agent inferred another space from the wider conversation or from information elsewhere in Jira. Perhaps the active viewport was passed as supporting context but not as a fixed parameter.

Whatever the internal cause, the resulting behaviour was wrong. The destination should either have been inherited from the interface or confirmed with me before the work item was created.

A user-interface location is not automatically an authority boundary. For an agent with write access, it needs to become one.

Increasing the model’s reasoning budget might improve how often it infers the correct destination, but it is no silver bullet. More inference cannot replace an explicit constraint at the point where the action is executed.

More Than a Hallucination

It is tempting to dismiss this as another example of an AI model “getting confused”. That understates the engineering problem.

Rovo supports actions including creating, editing, assigning, transitioning and deleting Jira work items. Atlassian requires additional confirmation for deletion, but its documented action set shows that the assistant can modify persistent enterprise data.

Once a model can call tools that change an external system, its output must be treated as an untrusted proposal—not as an instruction that should automatically be executed.

The failure resembles the risk OWASP describes as LLM06: Excessive Agency: an AI system has enough functionality, permissions or autonomy to cause unintended changes when its interpretation is wrong, ambiguous or manipulated.

In this case, I may have had legitimate permission to create work items in both spaces. Jira’s access-control system could therefore allow the action without detecting anything unusual.

But permission and intention are not the same thing.

Traditional access control asks:

Is this user permitted to write to Space B?

An agentic control layer must also ask:

Did the user intend this action to affect Space B?

The first question protects the system from unauthorised users. The second protects users from authorised agents making the wrong decision.

Why Ordinary Testing Misses the Problem

A conventional integration test can confirm that Jira successfully creates a work item when given a valid space key. It does not establish whether an AI agent will consistently choose the correct key after an ambiguous request, a long conversation or a change in viewport.

That is why these failures can survive apparently successful testing. The API works. The tool call is valid. The user has permission. The work item is created exactly as requested by the agent—just in the wrong place.

Happy-path demonstrations make the problem even easier to miss. An agent performs impressively when there is one user, one space and one obvious destination. Real enterprise environments contain hundreds of spaces, overlapping terminology, stale browser state and users with broad permissions.

Even high average accuracy can be misleading. An agent that selects the right destination 98 times out of 100 may look reliable in an evaluation dashboard. In production, the other two actions are still unintended mutations of business data.

Rovo is included across Jira’s paid Standard, Premium and Enterprise plans, with different monthly credit allowances. Once agentic functionality becomes part of a paid enterprise product, customers should expect predictable controls around every mutation. Jira pricing and Rovo availability

Context Must Be Enforced Below the Model

The answer is not simply a stronger system prompt telling the agent to pay more attention.

If an assistant is launched from Space A, the application should supply that space key as trusted runtime state. The model can propose the title, description and work-item type, but it should not be free to silently replace the destination with Space B.

If the user explicitly requests another destination, the runtime can allow it after checking permissions and, where appropriate, asking for confirmation. If no destination can be established confidently, the agent should stop and ask:

Which space should I create this in?

That additional interaction is not a failure of intelligence. It is evidence of appropriate restraint.

Validation also needs to go beyond syntax. JSON Schema can confirm that a tool call contains a correctly formatted spaceKey. It cannot determine whether that key represents the space the user intended.

That requires semantic checks:

• Does the target match the active space?
• Did the user explicitly request a different destination?
• Is that destination permitted for this workflow?
• Should changing the destination require confirmation?

The same limitation applies to the Model Context Protocol. MCP can standardise how agents discover and call tools, but it is not automatically a security boundary. The receiving tool must still enforce scope, permissions and confirmation rules.

Test the Conversation, Not Only the API

Agent tests need to cover the complete prompt-and-tool sequence together with the state surrounding it.

A basic regression test might state:

• The active viewport is Space A.
• The user asks to create a task for the work being discussed.
• The resulting mutation must target Space A.
• If the model supplies another space key, the runtime must reject it.

Further tests should cover missing context, similarly named spaces, switching spaces during a conversation, stale browser state and instructions found in retrieved content.

The objective is not to prove that the model will always make the right choice. It is to ensure that a wrong choice cannot silently become a production action.

Every mutation should also be logged, attributable and reversible.

A Reasonable Standard for Enterprise AI

An enterprise agent is not ready simply because it can translate natural language into a valid API call.

The surrounding system must assume that the model can misunderstand a request and prevent that misunderstanding from becoming an unintended change.

I do not know precisely why Rovo selected the wrong space in this case. I should not need to know.

If the destination is available from the interface, bind it to the action. If it is ambiguous, ask. If the model proposes a target outside that scope, reject it.

That is not an unreasonable standard for an enterprise agent with write access.