Deterministic AI in Salesforce: Building Reliable Systems on Non-Deterministic Foundations
You are building production business logic on a system that gives different answers to the same question. Here are five patterns that make it work anyway.
The Fundamental Problem
Ask Claude to classify a Salesforce Case and it will give you a category. Ask it again with the exact same input and it might give you a different category. Not because the model is broken. Because that is what non-deterministic systems do.
This is not a theoretical concern. I have watched a production Einstein GPT integration reclassify the same Case three different ways across three runs. High-priority, then medium, then high again. The business logic downstream trusted the classification. Three different SLA clocks started ticking. Three different teams got paged. One Case, three contradictory responses.
The Salesforce ecosystem has a massive blind spot here. The marketing says "AI-powered." The documentation shows happy-path demos. Nobody talks about what happens when the AI says something different the second time you ask.
This post describes five patterns I have built and deployed in production Salesforce orgs. They do not eliminate non-determinism. That is impossible. They contain it. They build reliable systems on top of unreliable foundations. Every pattern includes real Apex code you can deploy today.
Pattern 1: Output Schema Enforcement
The first and most critical pattern. Never trust raw LLM output. Define an exact JSON schema for every AI response. Validate against it. Retry on failure. Fall back to rules when retries are exhausted.
Here is the Apex implementation:
public class AIOutputValidator {
public class AISchema {
public List<SchemaField> fields;
public List<String> requiredFields;
}
public class SchemaField {
public String name;
public String fieldType; // STRING, INTEGER, ENUM, BOOLEAN
public List<String> allowedValues; // For ENUM type
public Integer minValue;
public Integer maxValue;
}
public class ValidationResult {
public Boolean isValid;
public Map<String, Object> parsedOutput;
public List<String> errors;
}
public static ValidationResult validate(
String rawOutput, AISchema schema
) {
ValidationResult result = new ValidationResult();
result.errors = new List<String>();
// Step 1: Parse JSON
Map<String, Object> parsed;
try {
parsed = (Map<String, Object>)
JSON.deserializeUntyped(rawOutput);
} catch (Exception e) {
result.isValid = false;
result.errors.add('Invalid JSON: ' + e.getMessage());
return result;
}
// Step 2: Check required fields
for (String reqField : schema.requiredFields) {
if (!parsed.containsKey(reqField)
|| parsed.get(reqField) == null) {
result.errors.add(
'Missing required field: ' + reqField
);
}
}
// Step 3: Validate field types and constraints
for (SchemaField field : schema.fields) {
Object val = parsed.get(field.name);
if (val == null) continue;
if (field.fieldType == 'ENUM'
&& field.allowedValues != null) {
String strVal = String.valueOf(val);
if (!field.allowedValues.contains(strVal)) {
result.errors.add(
field.name + ': value "' + strVal
+ '" not in allowed values '
+ field.allowedValues
);
}
}
if (field.fieldType == 'INTEGER') {
Integer intVal;
try {
intVal = Integer.valueOf(val);
} catch (Exception e) {
result.errors.add(
field.name + ': not a valid integer'
);
continue;
}
if (field.minValue != null
&& intVal < field.minValue) {
result.errors.add(
field.name + ': below minimum '
+ field.minValue
);
}
}
}
result.isValid = result.errors.isEmpty();
result.parsedOutput = parsed;
return result;
}
}The key insight: you define the schema once, in Apex, and every AI response runs through it. If the LLM returns garbage JSON or invalid values, the validator catches it before it touches your business logic.
But validation alone is not enough. You need the retry-with-error loop:
public class AIRequestWithRetry {
private static final Integer MAX_RETRIES = 3;
public static Map<String, Object> executeWithSchema(
String prompt,
AIOutputValidator.AISchema schema,
Map<String, Object> fallbackValues
) {
String lastErrors = '';
for (Integer attempt = 0; attempt < MAX_RETRIES; attempt++) {
String fullPrompt = prompt;
// On retry, append the validation errors
if (attempt > 0 && String.isNotBlank(lastErrors)) {
fullPrompt += '\n\nYour previous response had '
+ 'validation errors:\n' + lastErrors
+ '\nPlease correct and respond with valid JSON.';
}
String rawOutput = callLLM(fullPrompt);
AIOutputValidator.ValidationResult vr =
AIOutputValidator.validate(rawOutput, schema);
if (vr.isValid) {
return vr.parsedOutput;
}
lastErrors = String.join(vr.errors, '\n');
System.debug(LoggingLevel.WARN,
'AI output validation failed (attempt '
+ (attempt + 1) + '): ' + lastErrors);
}
// All retries exhausted. Use rule-based fallback.
System.debug(LoggingLevel.ERROR,
'AI failed schema validation after '
+ MAX_RETRIES + ' attempts. Using fallback.');
return fallbackValues;
}
}The retry-with-error pattern is powerful because the LLM sees its own mistakes. On the second attempt, it knows "value 'Urgent' not in allowed values [High, Medium, Low]" and corrects itself. In my production deployments, the first attempt passes validation about 85% of the time. The second attempt pushes that to 97%. The third attempt gets to 99.2%. The remaining 0.8% falls back to deterministic rules.
That 0.8% fallback rate is the critical number. It means your system always produces valid output. Always. The AI makes it better when it can, and the rules hold the floor when it cannot.
Pattern 2: Confidence Scoring
Not all AI outputs deserve the same level of trust. A classification the model is 95% sure about should be handled differently than one it is 60% sure about. The problem: LLMs do not natively report calibrated confidence scores. You have to build the scoring into the prompt and validate it.
public class ConfidenceRouter {
public class AIResponse {
public Map<String, Object> output;
public Integer confidence; // 1-10
public String reasoning;
}
public enum RoutingDecision {
AUTOMATE, // confidence 8-10: proceed automatically
HUMAN_REVIEW, // confidence 4-7: queue for human review
REJECT // confidence 1-3: use rule-based fallback
}
// Confidence thresholds (configurable per use case)
private static final Integer AUTO_THRESHOLD = 8;
private static final Integer REVIEW_THRESHOLD = 4;
public static RoutingDecision route(AIResponse response) {
if (response.confidence >= AUTO_THRESHOLD) {
return RoutingDecision.AUTOMATE;
} else if (response.confidence >= REVIEW_THRESHOLD) {
return RoutingDecision.HUMAN_REVIEW;
} else {
return RoutingDecision.REJECT;
}
}
public static void processCase(Case c, AIResponse response) {
RoutingDecision decision = route(response);
switch on decision {
when AUTOMATE {
c.Priority = (String) response.output.get('priority');
c.Type = (String) response.output.get('type');
c.AI_Confidence__c = response.confidence;
c.AI_Routed__c = true;
update c;
}
when HUMAN_REVIEW {
// Create a Task for human review with AI suggestion
Task reviewTask = new Task(
WhatId = c.Id,
Subject = 'AI Classification Review',
Description = 'AI suggests: '
+ JSON.serialize(response.output)
+ '\nConfidence: ' + response.confidence
+ '/10\nReasoning: ' + response.reasoning,
Priority = 'Normal',
OwnerId = getReviewQueueId()
);
insert reviewTask;
c.AI_Confidence__c = response.confidence;
c.Status = 'Pending Review';
update c;
}
when REJECT {
// Fall back to deterministic rules
applyRuleBasedClassification(c);
c.AI_Confidence__c = response.confidence;
c.AI_Fallback__c = true;
update c;
}
}
}
private static void applyRuleBasedClassification(Case c) {
// Deterministic rules that always work
if (c.Subject != null
&& c.Subject.containsIgnoreCase('outage')) {
c.Priority = 'High';
c.Type = 'Service Disruption';
} else if (c.Subject != null
&& c.Subject.containsIgnoreCase('billing')) {
c.Priority = 'Medium';
c.Type = 'Billing';
} else {
c.Priority = 'Medium';
c.Type = 'General';
}
}
}The prompt that generates the confidence score matters a lot. You cannot just ask "how confident are you on a scale of 1-10." LLMs will default to 8 every time. You need structured criteria:
Classify this support Case and rate your confidence.
Confidence criteria:
- 9-10: Clear keywords match, unambiguous category, seen many similar cases
- 7-8: Likely correct but some ambiguity in language or category overlap
- 5-6: Multiple categories could apply, relying on subtle context clues
- 3-4: Insufficient information, guessing based on partial matches
- 1-2: Cannot determine, input is too vague or contradictory
Return JSON:
{
"priority": "High|Medium|Low",
"type": "string",
"confidence": integer 1-10,
"reasoning": "one sentence explaining your confidence level"
}In production, roughly 70% of Cases route to AUTOMATE, 25% to HUMAN_REVIEW, and 5% to REJECT. That 70% is pure time savings. The 25% gets a head start because the human reviewer sees the AI's suggestion and reasoning. The 5% gets handled by rules that have been working fine for years.
The custom fields (AI_Confidence__c, AI_Routed__c, AI_Fallback__c) are essential. They let you build reports. After 90 days, you know your accuracy rates per confidence band. You can tune thresholds with data instead of guessing.
Pattern 3: Shadow Mode
Never deploy AI directly into production logic. Run it in shadow mode first. Shadow mode means: the AI runs on every record, but its output goes to a log, not to the record. The existing deterministic logic still makes the actual decision. You compare the two after 30 days.
public class AIShadowMode {
public static void runShadow(
List<Case> cases, String modelVersion
) {
List<AI_Shadow_Log__c> logs = new List<AI_Shadow_Log__c>();
for (Case c : cases) {
// Get AI classification
ConfidenceRouter.AIResponse aiResponse =
classifyWithAI(c);
// Get rule-based classification (current production)
Map<String, String> ruleResult =
getRuleClassification(c);
// Compare and log
Boolean agrees =
aiResponse.output.get('priority')
== ruleResult.get('priority')
&& aiResponse.output.get('type')
== ruleResult.get('type');
logs.add(new AI_Shadow_Log__c(
Case__c = c.Id,
Model_Version__c = modelVersion,
AI_Priority__c =
(String) aiResponse.output.get('priority'),
AI_Type__c =
(String) aiResponse.output.get('type'),
AI_Confidence__c = aiResponse.confidence,
Rule_Priority__c = ruleResult.get('priority'),
Rule_Type__c = ruleResult.get('type'),
Agreement__c = agrees,
AI_Reasoning__c = aiResponse.reasoning,
Timestamp__c = Datetime.now()
));
}
if (!logs.isEmpty()) {
insert logs;
}
}
}After 30 days, you run the analysis:
-- Shadow mode accuracy analysis
SELECT
AI_Priority__c,
Rule_Priority__c,
COUNT(Id) total,
SUM(CASE WHEN Agreement__c = true THEN 1 ELSE 0 END) agreed,
AVG(AI_Confidence__c) avg_confidence
FROM AI_Shadow_Log__c
WHERE CreatedDate = LAST_N_DAYS:30
GROUP BY AI_Priority__c, Rule_Priority__c
ORDER BY total DESCShadow mode answers the only question that matters: is the AI better than what we have today? Not in a demo. Not on a test dataset. On your actual Cases, with your actual data, in your actual org.
I have seen shadow mode reveal surprising things. One client's AI was 94% accurate overall but only 61% accurate on billing-related Cases because their billing terminology was highly specific. Without shadow mode, they would have deployed and broken billing Case routing for weeks before anyone noticed.
The shadow log also captures disagreements. Those disagreements are gold. They tell you exactly where the AI struggles and where your rules are too rigid. Often the AI is right and the rules are wrong. Sometimes the rules catch edge cases the AI misses. You cannot know which without data.
Pattern 4: Circuit Breakers
Even with schema validation, confidence scoring, and shadow mode, AI systems can degrade in production. Models get updated. API latency spikes. Token limits get hit. You need a circuit breaker that monitors error rates and automatically disables AI when something goes wrong.
public class AICircuitBreaker {
// Circuit breaker states
public enum BreakerState {
CLOSED, // Normal operation, AI enabled
OPEN, // AI disabled, using fallback
HALF_OPEN // Testing if AI has recovered
}
// Thresholds stored in Custom Metadata
private static final Decimal ERROR_THRESHOLD = 0.15; // 15%
private static final Integer WINDOW_MINUTES = 10;
private static final Integer COOLDOWN_MINUTES = 5;
private static final Integer HALF_OPEN_MAX = 5;
public static BreakerState getState() {
AI_Circuit_Breaker__mdt config =
AI_Circuit_Breaker__mdt.getInstance('Default');
if (config.State__c == 'OPEN') {
// Check if cooldown has elapsed
if (config.Last_State_Change__c != null
&& config.Last_State_Change__c
.addMinutes(COOLDOWN_MINUTES)
< Datetime.now()) {
return BreakerState.HALF_OPEN;
}
return BreakerState.OPEN;
}
return BreakerState.valueOf(config.State__c);
}
public static Boolean shouldCallAI() {
BreakerState state = getState();
switch on state {
when CLOSED {
return true;
}
when HALF_OPEN {
// Allow limited requests to test recovery
Integer recentAttempts = [
SELECT COUNT()
FROM AI_Request_Log__c
WHERE CreatedDate = LAST_N_MINUTES:5
AND Breaker_State__c = 'HALF_OPEN'
];
return recentAttempts < HALF_OPEN_MAX;
}
when OPEN {
return false;
}
when else {
return false;
}
}
}
public static void recordResult(Boolean success) {
// Log the result
insert new AI_Request_Log__c(
Success__c = success,
Breaker_State__c = String.valueOf(getState()),
Timestamp__c = Datetime.now()
);
// Check error rate in current window
List<AggregateResult> agg = [
SELECT
COUNT(Id) total,
SUM(CASE WHEN Success__c = false
THEN 1 ELSE 0 END) failures
FROM AI_Request_Log__c
WHERE Timestamp__c >= :Datetime.now()
.addMinutes(-WINDOW_MINUTES)
];
if (!agg.isEmpty()) {
Decimal total = (Decimal) agg[0].get('total');
Decimal failures = (Decimal) agg[0].get('failures');
if (total > 10) { // Minimum sample size
Decimal errorRate = failures / total;
BreakerState currentState = getState();
if (currentState == BreakerState.CLOSED
&& errorRate > ERROR_THRESHOLD) {
tripBreaker();
} else if (
currentState == BreakerState.HALF_OPEN
&& errorRate < ERROR_THRESHOLD) {
closeBreaker();
} else if (
currentState == BreakerState.HALF_OPEN
&& errorRate > ERROR_THRESHOLD) {
tripBreaker();
}
}
}
}
private static void tripBreaker() {
// Publish Platform Event for monitoring
AI_Circuit_Event__e evt = new AI_Circuit_Event__e(
State__c = 'OPEN',
Reason__c = 'Error rate exceeded threshold',
Timestamp__c = Datetime.now()
);
EventBus.publish(evt);
updateBreakerState('OPEN');
System.debug(LoggingLevel.ERROR,
'AI Circuit Breaker TRIPPED - AI disabled');
}
private static void closeBreaker() {
AI_Circuit_Event__e evt = new AI_Circuit_Event__e(
State__c = 'CLOSED',
Reason__c = 'Error rate recovered',
Timestamp__c = Datetime.now()
);
EventBus.publish(evt);
updateBreakerState('CLOSED');
System.debug(LoggingLevel.INFO,
'AI Circuit Breaker CLOSED - AI re-enabled');
}
}The Platform Event (AI_Circuit_Event__e) is the critical piece. When the breaker trips, every subscriber gets notified. A Flow can send a Slack message. A trigger can log the event. An admin dashboard can show the current state. The AI silently degrades to rule-based logic, and the humans who need to know find out immediately.
The HALF_OPEN state is borrowed from distributed systems design. When the cooldown expires, the breaker lets a small number of requests through (HALF_OPEN_MAX = 5). If those succeed, the breaker closes and AI resumes normal operation. If they fail, the breaker opens again. This prevents the thundering herd problem where a recovered service gets overwhelmed by backed-up requests.
I set the error threshold at 15% because, in practice, anything above that means something systemic is wrong. A single bad response is noise. Sustained 15%+ failure rates mean the model endpoint is degraded, the prompt is broken, or the data has shifted out of distribution.
Pattern 5: Hybrid Architecture
The final pattern ties everything together. Do not use AI for everything. Use deterministic rules for edge cases, compliance requirements, and well-defined logic. Use AI for the routine 80% where the rules would require hundreds of if-else branches.
public class HybridClassifier {
// Hard rules that ALWAYS take precedence over AI.
// These handle compliance, legal, and edge cases.
private static Map<String, CaseClassification>
HARD_RULES = new Map<String, CaseClassification>{
'legal' => new CaseClassification(
'High', 'Legal', 'Legal Review Queue'),
'subpoena' => new CaseClassification(
'Critical', 'Legal', 'Legal Review Queue'),
'hipaa' => new CaseClassification(
'Critical', 'Compliance', 'Compliance Queue'),
'data breach' => new CaseClassification(
'Critical', 'Security', 'Security Incident Queue'),
'gdpr' => new CaseClassification(
'High', 'Compliance', 'Privacy Queue')
};
public static CaseClassification classify(Case c) {
// Layer 1: Hard rules (deterministic, always wins)
CaseClassification hardResult =
checkHardRules(c);
if (hardResult != null) {
hardResult.source = 'RULE_HARD';
return hardResult;
}
// Layer 2: Circuit breaker check
if (!AICircuitBreaker.shouldCallAI()) {
CaseClassification ruleResult =
applySoftRules(c);
ruleResult.source = 'RULE_FALLBACK';
return ruleResult;
}
// Layer 3: AI with schema validation
try {
ConfidenceRouter.AIResponse aiResponse =
classifyWithAI(c);
RoutingDecision decision =
ConfidenceRouter.route(aiResponse);
if (decision == RoutingDecision.AUTOMATE) {
AICircuitBreaker.recordResult(true);
CaseClassification aiResult =
new CaseClassification(
(String) aiResponse.output.get('priority'),
(String) aiResponse.output.get('type'),
(String) aiResponse.output.get('queue')
);
aiResult.source = 'AI';
aiResult.confidence = aiResponse.confidence;
return aiResult;
}
if (decision == RoutingDecision.HUMAN_REVIEW) {
AICircuitBreaker.recordResult(true);
CaseClassification aiResult =
new CaseClassification(
(String) aiResponse.output.get('priority'),
(String) aiResponse.output.get('type'),
'Human Review Queue'
);
aiResult.source = 'AI_REVIEW';
aiResult.confidence = aiResponse.confidence;
return aiResult;
}
// Low confidence: fall through to soft rules
AICircuitBreaker.recordResult(true);
} catch (Exception e) {
AICircuitBreaker.recordResult(false);
System.debug(LoggingLevel.ERROR,
'AI classification failed: ' + e.getMessage());
}
// Layer 4: Soft rules (deterministic fallback)
CaseClassification softResult = applySoftRules(c);
softResult.source = 'RULE_SOFT';
return softResult;
}
private static CaseClassification checkHardRules(Case c) {
String text = (c.Subject ?? '')
+ ' ' + (c.Description ?? '');
text = text.toLowerCase();
for (String keyword : HARD_RULES.keySet()) {
if (text.contains(keyword)) {
return HARD_RULES.get(keyword).clone();
}
}
return null;
}
}The layering is intentional:
- Hard rules for anything with compliance, legal, or security implications. These never go through AI. A Case mentioning "HIPAA" always routes to Compliance. No exceptions. No confidence scores. No retries.
- Circuit breaker check before calling the AI at all. If the AI is degraded, skip it entirely and use deterministic rules. The user never sees an error.
- AI with confidence routing for the routine majority. High confidence automates. Medium confidence queues for review. Low confidence falls through.
- Soft rules as the final safety net. When AI fails, when confidence is too low, when the circuit breaker is open, soft rules handle it. Not as well as the AI, but predictably.
The source field on every classification lets you report on what is actually making decisions. In a healthy deployment, you will see roughly: AI 65%, AI_REVIEW 20%, RULE_HARD 5%, RULE_SOFT 8%, RULE_FALLBACK 2%. Those percentages tell you the real story of how much value the AI is adding.
Testing Non-Deterministic Systems
Traditional unit tests assert exact outputs. That does not work when the output changes between runs. You need a different testing strategy:
@isTest
public class HybridClassifierTest {
@isTest
static void testHardRulesAlwaysWin() {
// Hard rules must be deterministic
Case c = new Case(
Subject = 'HIPAA Violation Report',
Description = 'Patient data was exposed'
);
// Run 10 times - must always produce same result
for (Integer i = 0; i < 10; i++) {
CaseClassification result =
HybridClassifier.classify(c);
System.assertEquals('Critical', result.priority,
'HIPAA must always be Critical');
System.assertEquals('Compliance', result.type_x,
'HIPAA must always route to Compliance');
System.assertEquals('RULE_HARD', result.source,
'HIPAA must always use hard rules');
}
}
@isTest
static void testFallbackWhenCircuitBreakerOpen() {
// Trip the circuit breaker
Test.startTest();
forceCircuitBreakerOpen();
Case c = new Case(
Subject = 'My account is not working',
Description = 'I cannot log in to the portal'
);
CaseClassification result =
HybridClassifier.classify(c);
// Must use rule-based fallback
System.assertEquals('RULE_FALLBACK', result.source);
System.assertNotEquals(null, result.priority);
System.assertNotEquals(null, result.type_x);
Test.stopTest();
}
@isTest
static void testAIOutputSchemaValidation() {
// Validate that schema enforcement catches bad output
AIOutputValidator.AISchema schema = buildCaseSchema();
// Valid output
String valid = '{"priority":"High","type":"Billing",'
+ '"confidence":8,"queue":"Billing Queue"}';
AIOutputValidator.ValidationResult vr =
AIOutputValidator.validate(valid, schema);
System.assert(vr.isValid, 'Valid output should pass');
// Invalid priority
String invalid = '{"priority":"Urgent","type":"Billing",'
+ '"confidence":8}';
vr = AIOutputValidator.validate(invalid, schema);
System.assert(!vr.isValid,
'Invalid priority should fail');
// Missing required field
String missing = '{"priority":"High"}';
vr = AIOutputValidator.validate(missing, schema);
System.assert(!vr.isValid,
'Missing required fields should fail');
// Garbage input
vr = AIOutputValidator.validate(
'not json at all', schema);
System.assert(!vr.isValid,
'Non-JSON should fail');
}
}The testing philosophy: you cannot test the AI's answer. You can test that every path through your system produces valid, safe output regardless of what the AI returns. Test the schema validation. Test the circuit breaker. Test the fallback rules. Test the confidence routing. Those are all deterministic.
For the AI itself, you test in shadow mode over time, not in unit tests. That is a fundamentally different testing cadence. Unit tests run in CI/CD. Shadow mode runs in production for 30 days. Both are necessary. Neither replaces the other.
The Real Architecture
When you combine all five patterns, the architecture looks like this:
Input (Case)
|
v
[Hard Rules] --> Match? --> Deterministic classification
| (Legal, Compliance, Security)
| No match
v
[Circuit Breaker] --> OPEN? --> Soft Rules (fallback)
|
| CLOSED/HALF_OPEN
v
[LLM Call]
|
v
[Schema Validation] --> Invalid? --> Retry with errors (3x)
| |
| Valid | Still invalid?
v v
[Confidence Scoring] [Soft Rules (fallback)]
|
|---> 8-10: Automate
|---> 4-7: Human Review Queue
|---> 1-3: Soft Rules (fallback)
|
v
[Shadow Log] (always, for monitoring)
|
v
[Record Result] --> Feed Circuit BreakerEvery path produces valid output. Every path is logged. Every path feeds the circuit breaker. The AI makes things better when it is working. The system works fine when it is not.
This is what production AI in Salesforce looks like. Not a single callout to an LLM with a hopeful JSON.deserialize. An entire architecture designed around the reality that your AI will be wrong sometimes, will be slow sometimes, and will be completely unavailable sometimes.
Build for that reality and you build something that actually works.