Tenacious-Bench-v0.1: Evaluating Sales Agents for Policy Reliability (Not Just Fluency)

I spent the last week building a benchmark for sales agents.

Not to measure whether they complete tasks — but whether they can be trusted while doing them.

Because in production sales systems, the biggest failures aren’t broken workflows.

They’re confident, polished, policy-violating outputs that look correct — and aren’t.

This post is about what we built, why we built it, and what we learned from trying to train models against it.

Generic agent benchmarks are useful, but they are usually optimized to answer a different question than the one sales teams need answered in production.

The generic question is:

Can the agent complete multi-step tasks with tools?

The production sales question is:

Will the agent make safe, honest, high-signal claims under constraints that directly affect customer trust and revenue risk?

Tenacious-Bench v0.1 was built to answer the second question.

Why this matters

If you’re building LLM systems for production — especially in customer-facing roles — this distinction matters:

• Fluency ≠ correctness

• Correctness ≠ safety

• Safety ≠ policy compliance

Most benchmarks optimize for the first.

Production systems fail on the last.

---

1) The Gap: Why Generic Benchmarks Under-Specify Sales Failures

Benchmarks such as tau²-Bench are strong for evaluating broad agent workflows, tool use, and task progression. They are not incorrect—they are measuring a wider target.

---

Failure types that matter in sales but are weakly represented in generic evals

1. Confidence calibration failures - Claiming high certainty from weak or partial signals.

2. Capacity commitment failures - Implying bench or delivery capacity that is not authorized.

3. Pricing-scope failures - Quoting contract values, pricing ranges, or discount language outside policy.

4. Tone under constraints - Maintaining non-condescending, professional tone within 60–120 words while still being specific

---

Example mismatch

A generic benchmark may reward:

1. Correct structure

2. Correct tool call

3. A “plausible” final answer

A sales-specific benchmark should penalize the same output if it states:

1. “We can support all your urgent hires this quarter” (unauthorized capacity claim)

2. “Your annual contract should be around $X” (policy violation on pricing specificity)

3. Promotional language that conflicts with current policy

This mismatch motivated a narrower benchmark:

---

2) Audit Method: From Agent Failures to Measurable Rubrics

We started with the agent probes and traces, then converted observed failure patterns into machine-checkable requirements.

1. trace_respond_874662476a68: policy boundary handling issues

2. trace_advance_2ef64021c4f8: invalid state-transition behavior

3. trace_schedule_book_2dc2d85ac0fc and trace_slots_fail: scheduling reliability breakdown

4. trace_mem_get_03bdfa202017 and trace_outreach_ae9e643c953b: context/memory coupling failures

---

Design rules for the benchmark

1. Machine-verifiable first -Deterministic rubric markers take precedence over subjective scoring.

2. Sealed held-out split - Strict separation with contamination checks.

3. Evidence-linked reporting - Every headline claim maps to a concrete artifact.

4. Publish negative results - If training underperforms a strong prompt baseline, report it directly.

---

3) Dataset Construction: 250 Tasks, Four Generation Modes, One Sealed Held-Out

Tenacious-Bench v0.1 contains:

1. Train: 125

2. Dev: 75

3. Held-out: 50

---

Authoring mix

1. Trace-derived (~30%)

2. Programmatic sweeps (~30%)

3. Multi-LLM synthesis (~25%)

4. Hand-authored adversarial (~15%)

---

Why multiple authoring modes

1. Trace-derived tasks preserve real failure signatures from prior runs.

2. Programmatic sweeps stress known variables (signal strength, segment, ask type) at scale.

3. Multi-LLM synthesis increases lexical diversity and reduces overfitting to a single style.

4. Hand-authored adversarial tasks directly target known policy breakpoints.

---

Quality and leakage controls

1. Model-family separation between generation and judge paths

2. Pairwise filtering for near-duplicate synthetic candidates

3. Contamination checks across train vs held-out

4. Held-out split excluded from preference construction

The result is not a “perfect dataset.” It is an

---

4) Why We Chose (DPO + LoRA)

This was a method choice grounded in observed failure types.

---

The three options

1. SFT generator - Strong for phrasing quality, weaker for enforcing “block this output” behavior.

2. preference tuning / critic / intervention - Directly optimizes chosen vs rejected behavior for safety and consistency.

3. process reward model - owerful but significantly heavier in both data preparation and runtime complexity.

---

Why DPO + LoRA fit this project

After analyzing the agent traces, we have found that the dominant issues were guardrail and reliability failures, not just phrasing quality.

DPO (preference tuning / critic / intervention) provided the best alignment with:

• observed failure modes

• dataset size constraints

• iteration speed requirements

---

Practical constraints

1. 125 train preference pairs and 75 dev pairs

2. Colab-class hardware

3. Need to encode:

“Looks fluent but should be rejected”

DPO (preference tuning / critic / intervention) satisfied all three.

---

Why DPO?

We selected DPO first, with SimPO and ORPO as alternatives.

1. DPO (Rafailov et al., 2023) - Stable baseline for pairwise preference optimization

2. SimPO (Meng et al., 2024) - Promising, simpler objective; reserved for follow-up

3. ORPO (Hong et al., 2024) - Fallback if DPO behavior regressed

---

LoRA choice

LoRA was selected for:

1. Lower memory cost

2. Faster iteration cycles

3. Adapter-only publishing

---

Core setup

1. Qwen2.5 Instruct (3B class) via Unsloth

2. Max sequence length: 1024

3. Seed: 42

4. Effective batching via gradient accumulation

---

5) Paper Grounding Behind the Method Stack

The methodology is informed by:

1. DPO (Rafailov et al., 2023)

2. SimPO (Meng et al., 2024)

3. ORPO (Hong et al., 2024)

4. Prometheus 2 (Kim et al., 2024)

5. Preference leakage analysis (Li et al., 2025)

This project does not attempt a full method comparison—it selects a practical first path while preserving room for ablations.

---

6) Experiment Adjustments That Materially Changed Outcomes

A key lesson: evaluation wiring can mask real capability.

1. Introduced trained_intervene mode → trained model acts as intervention over baseline drafts

2. Added strict postprocessing → subject prefix, CTA constraints, word limits, cleanup, safety normalization

3. Tightened inference prompt and decoding controls

---

These changes removed formatting/directness artifacts and exposed actual behavioral improvements.

---

7) Results: Positive Delta A, Negative Delta B

Held-out results (n = 50):

1. Baseline mean score: 93.44 (pass rate 0.86)

2. Prompt-only mean score: 100.0 (pass rate 1.0)

3. Trained mean score: 97.92 (pass rate 0.82)

---

Delta A (trained vs baseline)

1. Mean diff: +4.48

2. 95% CI: [3.68, 5.44]

3. One-sided p-value: 0.0002

4. Significance: true

---

Delta B (trained vs prompt-only)

1. Mean diff: -2.08

2. 95% CI: [-3.36, -0.96]

3. One-sided p-value: 1.0

4. “Training beats prompt-only”: false

---

Interpretation

1. Training produced statistically supported lift over baseline

2. Prompt-only intervention remained stronger

This is not contradictory—it defines the boundary of improvement for this setup.

---

8) What Did Not Work (and Why We Kept It)

After fixing evaluation issues, failures concentrated in:

1. Capacity over-commitment

2. TCV quoting

3. Discount/promo language

---

These matter because they:

• pass superficial fluency checks

• introduce real operational risk

We kept them visible because:

1. They define the next benchmark expansion target

2. They prevent over-claiming progress

---

9) What Changes in v0.2

Planned updates:

1. Expand adversarial slices for capacity/pricing/promo constraints

2. Add stricter intervention-time policy enforcement

3. Run cost-aware ablations

4. Maintain explicit non-win reporting

---

10) Reproducibility and Artifacts

Public artifacts:

1. Dataset: https://huggingface.co/datasets/gemechisw/tenacious_bench_v0.1

2. Model adapter: https://huggingface.co/gemechisw/Tenacious-DPO-LoRA-v0.1

3. Repository: https://github.com/gemechisworku/tenacious_bench_v01

---

Key evidence files

The below evidence files are provided on this project's github repo added above.

1. ablation_results.json

2. held_out_traces.jsonl

3. training/config.yaml

4. training/metrics.json

5. training/training_run.log

6. methodology_rationale.md

---

References

1. Rafailov et al., 2023. Direct Preference Optimization (DPO). https://arxiv.org/abs/2305.18290

2. Meng et al., 2024. SimPO. https://arxiv.org/abs/2405.14734

3. Hong et al., 2024. ORPO. https://arxiv.org/abs/2403.07691

4. Kim et al., 2024. Prometheus 2. https://arxiv.org/abs/2405.01535

5. Li et al., 2025. Preference leakage analysis. https://arxiv.org/abs/2502.01534