Generative AI is at an interesting crossroads. In the last couple of years, enterprise IT has changed significantly and AI is a major reason why. The technology is powerful, but it has taken time to understand how to manage it and make it perform reliably inside real organizations in service of real outcomes.
We’re now moving past the loudest part of the hype cycle. The conversation is shifting away from dystopian sci‑fi narratives and toward practical, grounded applications. That’s the point where AI stops being a demo and starts becoming a production-ready capability.
It is also the point where you need a strong business case. To build one, you need a clear view of expected benefits, costs, risks, and timelines.
If you are trying to enable AI across a business, the hard question isn’t whether you can use it. The hard question is whether you should. With more real-world experience, we can better separate signal from noise. Below are key considerations when evaluating GenAI use-cases on platforms such as ServiceNow and Salesforce.
1) Hallucination Risk, Catastrophic Errors, and Survivorship Bias
A major difference between humans and AI is how they handle risk.
Humans are error-prone, but most of the time we instinctively avoid decisions with catastrophic downside – even when the probability of failure is low. We do this almost unconsciously.
In the ServiceNow and Salesforce ecosystems, teams typically don’t deploy code to production without approvals, testing, and validation. We don’t create a new data table without a security review. Not because bypassing controls is impossible, but because the downside is unacceptable (taking a production application down can be a career-ending decision).
GenAI doesn’t have that built-in risk filter. It can fail on trivial and high-stakes tasks with the same confident tone. That’s why enterprise cautionary tales keep repeating: hallucinated reports that damage reputations, agentic AI that deletes or corrupts production data, and chatbots that confidently provide guidance that leads to consequential bad decisions.
Any use case with severe downside requires, at minimum, a human in the loop for review and accountability. Not because humans are smarter – but because accountable humans tend to avoid catastrophic outcomes. When risk is high, that behavior is a feature, not a bug.
2) The Token-Driven Technical Debt Spiral
LLM-driven agentic tools are changing how software is built. But with increased autonomy comes a real risk: technical debt growth and rising token costs as codebases expand.
This happens for a few reasons:
- Unchecked agentic coding tools can introduce unnecessary code that passes review but adds little value.
- As more code is introduced, it becomes harder for humans to understand, govern, and maintain.
- When maintainability drops, teams lean even harder on agentic tools to refactor and clean up the environment, which introduces even more churn and more code.
As systems grow, the amount of analysis, retrieval, and context required to work safely grows exponentially. This spirals into increasing code complexity, bigger code bases, and larger token consumption.
Mitigation requires treating AI-generated code as any other code base: keep it comprehensible, enforce standards, require tests, and constrain autonomy. In ServiceNow, AI can draft a full script or flow, but it should never reach production unless someone understands every line of code. In this paradigm, the developer’s job shifts from typing code to forcing clarity, correctness, and simplicity in the service of business outcomes.
3) Vendor Concentration and Institutional Dependency
Another under-discussed risk is dependency on a small set of AI vendors, and over time, dependency on automation for core business processes.
If organizations aren’t careful, they can end up locked into a model provider while also experiencing institutional knowledge atrophy: fewer people understand how workflows actually work because execution has been outsourced to AI-enabled automation.
When that happens, organizations are left negotiating from a weak position if vendors change pricing, contract terms, or product direction.
The ecosystem is maturing quickly – open-source models, locally run models, and enterprise platforms improve every year. The goal isn’t to avoid vendors. The goal is to make dependency a conscious design choice, not an accident.
4) Privacy and Security Assumptions
Depending on the vendor and configuration, GenAI prompts and outputs may be logged, retained, or used to improve models. Even when GenAI is embedded in an enterprise platform, a third-party model provider may still be involved.
Don’t assume – verify:
- What data leaves your environment
- What is retained (and for how long)
- Who can access logs
- What contractual terms allow
For sensitive use cases, consider dedicated deployments, stricter retention controls, or locally run models – especially when data classification requirements are strict.
5) Model Vulnerability and Exploitation
Large language models are software. Like any software, they are exploitable.
Research shows that a surprisingly small number of malicious documents can create backdoor vulnerabilities. Even if this feels unlikely, it is worth taking into account when designing solutions. For enterprise workflows, this means strict tool permissions, role-based guardrails, robust access controls, and active monitoring for suspicious behavior.
A simple rule: never let the model access what the user cannot see.
The worst-case scenario isn’t a wrong answer. It’s a weaponized assistant operating across enterprise systems.
Closing: Disciplined Leverage Over Hype
AI is changing how work happens. Rather than trying to put AI everywhere, the organizations that win will be the ones that pick the right use cases, constrain failure modes, measure outcomes, and build the governance needed to turn novelty into habit.
That is what AI enablement actually means. It is not hype. It is not fear. It is the disciplined leverage of a powerful tool, used exactly where it fits, and avoided where it does not.
