Key takeaways
  • In one banking customer-intent classification evaluation, TypeSafe AI's Jev scored 98.5% accuracy and Gemini scored 98.7%, a gap of 0.2 percentage points.
  • Jev finished the evaluation in 1 minute 22 seconds against Gemini's 9 minutes 27 seconds, about 6.9 times faster.
  • Jev's reported evaluation cost was $0.11 against $9.65 for Gemini, about 88 times lower.
  • Jev answers typed questions (choice, scoring, yes/no) and returns a selected option with probabilities and a confidence score, instead of free-form text.
  • The figures come from one run on one task. Rerun the comparison on your own data before choosing a model.

When better AI doesn't have to cost more

Financial institutions are increasingly adopting AI-powered agents to handle customer queries, automate routine workflows, and improve customer service. From identifying account-opening requests to routing transaction disputes, these systems are becoming an important part of modern banking operations.

Deploying AI in banking comes with a distinct set of challenges. Models must produce accurate results, operate efficiently, deliver consistent decisions, and provide evidence that supports evaluation and risk review.

Traditional large language models (LLMs) are powerful general-purpose reasoners, but many classification workflows still follow a familiar process: generate text, parse the response, then convert it into a structured decision. When a task requires one answer from a predefined set, that approach can introduce unnecessary latency, processing overhead, and cost.

TypeSafe AI's Jev, now integrated with GenGuardX by Corridor Platforms, takes a more direct approach. In one banking customer-intent classification evaluation, Jev achieved 98.5% accuracy in 1 minute and 22 seconds at a reported evaluation cost of $0.11, compared with Gemini's 98.7% accuracy in 9 minutes and 27 seconds at $9.65.

These are task-specific evaluation results, not estimates of end-to-end production cost or latency. They suggest that, for narrowly defined decision tasks, how a model is designed to respond can materially affect evaluation speed and cost without a meaningful observed loss of accuracy.

The challenge: evaluating AI for real-world banking workflows

Banking customer interactions are varied, conversational, and often ambiguous. A single message may contain a straightforward request, a complaint, or a request that requires human intervention.

  • “I'd like to open a new savings account.” — Open an account.
  • “I don't recognize this transaction on my statement.” — Dispute a charge.
  • “Can I speak with someone from your support team?” — Talk to a person.

An AI-powered banking agent needs to identify the customer's intent and route the request to the appropriate workflow. Although that classification task may appear simple, evaluating it at scale requires more than measuring accuracy.

Accuracy is only one part of the equation

A model can classify customer messages accurately but still be too expensive or slow to run across a large evaluation dataset. For institutions processing substantial volumes of customer interactions, those costs accumulate quickly.

Structured decisions should produce structured outputs

When a system needs to select one intent from a predefined list, a lengthy natural-language response is often unnecessary. The downstream application needs a reliable classification, not an explanation that must be parsed to extract the answer.

Evaluation must support review and accountability

In regulated environments, teams need to understand how models perform, compare alternatives, and maintain records of evaluation results. A strong accuracy score is useful, but production decisions require a broader view of performance, efficiency, and evidence.

Meet Jev: a structured approach to AI decision-making

Jev is designed to answer typed questions rather than generate free-form text for every task. Its question types, listed in TypeSafe's model documentation, include:

  • Choice: select the most appropriate option from a predefined set.
  • Scoring: assign a score on a specified scale.
  • Yes/no: evaluate whether a particular condition is true.

For a classification task, the system provides the selected option alongside probabilities for the alternatives and a confidence score, as shown in the Jev model reference on Cloudflare Workers AI. That makes the result directly useful to applications that need a decision rather than a paragraph of generated text.

How typed questions change the workflow

Consider the message: “I noticed a transaction I don't recognize. Can you help me understand what happened?”

A conventional LLM workflow may ask the model to explain the customer's intent in natural language, then require the application to interpret that answer and map it to an internal category. With a structured choice-based approach, the task can instead be framed explicitly:

Question: What is the customer's primary intent?
Available options: Open an account; Dispute a charge; Talk to a person.

The model returns a structured decision with associated probabilities and confidence information. The application can use the result directly, subject to appropriate validation and business rules. The key distinction is that the task and output are both structured around the decision the application needs to make.

The benchmark: Jev vs. Gemini

Corridor Platforms tested Jev against Gemini inside GenGuardX on the same banking customer-intent classification task.

MetricJevGemini
Accuracy98.5%98.7%
Evaluation time1 min 22 sec9 min 27 sec
Evaluation cost$0.11$9.65

Reported results from the banking customer-intent classification evaluation.

Methodology and scope

Both models were assessed on the same banking customer-intent classification task inside GenGuardX. Accuracy, elapsed evaluation time, and reported evaluation cost are presented together so that performance and efficiency can be considered in the same context.

These figures come from one evaluation run on one task. They are not a general claim about every workload or production deployment. Before you choose a model, rerun the comparison on your own data and record the dataset, model versions, prompt, pricing and run date with the result.

Watch the Jev and Gemini banking customer-intent evaluation.

Near-identical classification accuracy

Jev achieved 98.5% accuracy compared with Gemini's 98.7%: a difference of just 0.2 percentage points in this evaluation. Both models delivered high accuracy on the tested task, while Jev completed it with substantial advantages in evaluation time and cost.

A general-purpose model may be capable of a wide range of reasoning and generation tasks, but not every workflow requires open-ended text generation. The appropriate model should be evaluated against the actual requirements of the application.

Approximately 7× faster evaluation

Jev completed the evaluation in 1 minute and 22 seconds; Gemini took 9 minutes and 27 seconds. That is approximately 6.9× faster, an elapsed-time reduction of around 85.5%.

Faster execution can make experimentation more practical for teams testing multiple models, iterating on prompts, or evaluating changes to AI agents. Actual production latency depends on the deployment environment, workload, and system configuration; evaluation speed is not a guarantee of equivalent end-to-end production performance.

Approximately 88× lower evaluation cost

Jev's reported evaluation cost was $0.11, compared with $9.65 for Gemini: approximately 88 times lower, or a reduction of about 98.9%.

For organizations conducting frequent evaluations or processing high volumes of structured classification tasks, cost efficiency can be an important factor. Actual costs vary with dataset size, model pricing, usage patterns, and deployment requirements.

Why this matters for banking and financial services

Financial institutions need AI systems that work within existing business processes while meeting expectations around reliability, efficiency, and oversight. Structured decision-making is relevant to several banking workflows:

  • Customer-intent classification: categorize messages into account opening, transaction disputes, card issues, or requests for human support.
  • Customer-service automation: provide an input to agents that determine the next action, retrieve information, or transfer a conversation to a specialist.
  • Human escalation and exception handling: use confidence and alternative-class probabilities to route uncertain cases for review.
  • High-volume classification: evaluate cost and speed when many messages have a limited set of possible outcomes.

These are potential application areas, not a claim that this benchmark establishes production readiness for every banking workflow. Confidence thresholds should be calibrated and validated on representative data before they are used to make operational decisions.

From benchmark results to trustworthy AI

Fast, accurate, and cost-efficient models are valuable, but selecting an enterprise model on benchmark numbers alone is rarely sufficient. Teams need a repeatable way to compare alternatives and preserve evidence of the result.

With Jev integrated into GenGuardX, teams can evaluate it alongside existing models using the same report and retain the resulting records for review. A strong evaluation process should examine:

  • Class-level performance and error severity: which intents are missed, and what is the impact?
  • Confidence calibration: do reported probabilities correspond to observed correctness?
  • Operational efficiency: what are the measured time and cost under documented conditions?
  • Reproducibility and change management: can the team identify the configuration and detect regressions after a change?

GenGuardX logs evaluation runs and results to support model-risk review. Those records contribute to governance workflows; they do not, by themselves, establish regulatory compliance or remove the need for independent validation and appropriate controls.

How to decide whether Jev is right for your workflow

Jev's results suggest that structured decision-making models deserve consideration when a task has a clearly defined output. Before adopting any model, teams should assess:

  • Task suitability: does the application select an option, assign a score, or make a yes/no decision—or require open-ended reasoning and generation?
  • Accuracy requirements: does the model meet the required performance level on representative, task-specific data?
  • Error severity: are some mistakes significantly more consequential than others?
  • Cost and latency: does the model meet operational requirements under realistic workloads?
  • Integration: can the output be used with appropriate validation, error handling, and fallback behavior?
  • Governance: can the team document the evaluation process and retain the evidence needed for review?

For structured classification, Jev may provide a compelling alternative to a general-purpose LLM. For richer reasoning, explanations, or flexible text generation, a general-purpose model may remain the better choice. The objective is to select the right model for each task and validate that choice with evidence.

Better model selection starts with better evaluation

This banking customer-intent evaluation illustrates that model performance is about more than accuracy alone. Jev achieved 98.5% accuracy compared with Gemini's 98.7%, while completing this evaluation approximately 6.9 times faster and at approximately 88 times lower reported evaluation cost.

That makes a case for evaluating structured decision-making models alongside general-purpose LLMs when the application calls for clearly defined outputs. It is not a claim that one model is best for every workload: benchmark performance is the starting point for task-specific validation, not a replacement for it.

Choose models based on evidence, optimize for the task at hand, and make AI evaluation a repeatable part of responsible deployment.

FAQ

Is Jev as accurate as Gemini for banking intent classification?

In this banking customer-intent classification evaluation, Jev reached 98.5% accuracy and Gemini reached 98.7%, a difference of 0.2 percentage points. Both results come from one evaluation run on one task inside GenGuardX.

How much cheaper was Jev than Gemini?

Jev's reported evaluation cost was $0.11, compared with $9.65 for Gemini: about 88 times lower, or a reduction of about 98.9%. This is evaluation cost, not an estimate of end-to-end production cost, which varies with dataset size, model pricing, usage patterns, and deployment requirements.

How much faster was Jev than Gemini?

Jev completed the evaluation in 1 minute and 22 seconds; Gemini took 9 minutes and 27 seconds. That is about 6.9 times faster. Production latency depends on the deployment environment, workload, and system configuration.

What is TypeSafe AI's Jev?

Jev is a model from TypeSafe AI that answers typed questions rather than generating free-form text. Its question types include choice, scoring, and yes/no. For a classification task it returns the selected option with probabilities for the alternatives and a confidence score. Jev is integrated with GenGuardX by Corridor Platforms.

When is a general-purpose LLM a better choice than Jev?

When the task needs richer reasoning, explanations, or flexible text generation. Jev suits tasks with a clearly defined output: selecting an option, assigning a score, or making a yes/no decision.

Do these benchmark results apply to my workload?

Not automatically. The figures come from one evaluation run on one task. Rerun the comparison on your own data and record the dataset, model versions, prompt, pricing, and run date with the result.

Resources

Internal resources

External resources