KDASO: Knowledge-Data Dual-Driven Automated Skill Optimization for Liability Adjudication

Overview of the KDASO framework: neighborhood-conditioned skill induction, execution-aligned skill refinement and context-budgeted skill routing

KDASO evolves a skill library from adjudication data. Neighborhood-conditioned induction turns recently adjudicated neighbours into new skills, execution-aligned refinement rewrites them until the LLM's reasoning trajectory matches expert logic, and context-budgeted routing feeds each case only the skills that can fire.

Abstract

Liability adjudication tasks (e.g., legal charge determination) require analyzing event descriptions against domain-specific skills to determine whether a party should bear liability. While large language models (LLMs) have shown promise in this area by leveraging executable skills for interpretable reasoning, existing approaches face three critical limitations: skill libraries become stale as real-world scenarios dynamically evolve, skill descriptions often suffer from a semantic execution gap where LLMs deviate from intended expert logic, and growing skill collections exceed context budgets, causing critical skill omission.

To address these challenges, we propose Knowledge-Data Dual-Driven Automated Skill Optimization (KDASO), which continuously evolves a skill library through neighborhood-conditioned induction from recent adjudication data, execution-aligned refinement based on LLM trajectory analysis, and context-budgeted routing for scalable skill selection. Experiments across ride-hailing adjudication, legal liability determination and logical reasoning judgment demonstrate that KDASO significantly outperforms current general-purpose and domain-specific baselines. KDASO further achieves an 8.96% improvement when transferring skills to concept-similar scenarios and maintains consistent performance gains through continuous optimization, underscoring the high transferability and stability of the generated skills.

1. Three Failures of Skill-Based Adjudication

Giving an LLM a library of domain skills is attractive because the reasoning stays explicit and auditable. Yet the paradigm fails in three systematic ways: skills cannot be acquired at the speed of the data, cannot be executed as the expert intended, and cannot be deployed at scale inside a finite context window. These are not engineering defects. They are tensions intrinsic to skill-based adjudication.

Coverage under distribution shift

Skill libraries are curated from historical cases and lag behind emerging behaviours such as GPS-spoofing “virtual displacement” fraud. Inducing from a single case captures idiosyncratic noise; inducing from the whole dataset exceeds the context window. A local neighborhood is the only tractable middle ground.

Semantic execution gap

A skill that reads as logic to a human may be executed inconsistently by an LLM: divergent priors and ambiguous operator precedence let the reasoning path drift from the expert's. As Figure 1 of the paper shows, description quality dominates adjudication precision far more than the model does.

Context-budget scalability

The skill omission ratio grows almost linearly with the number of skills in the prompt (R² = 0.82), and precision decays with it. Attention is selectively allocated, so indiscriminate skill presentation silently discards decision-critical evidence.

2. The KDASO Framework

A skill is a triple s = (C_s, A_s, T_s) — preconditions that trigger it, adjudication logic that produces the label, and the execution template that structures the reasoning. KDASO treats these three components as the object of optimization rather than the environment of optimization: no parameter of the adjudicator is ever updated, and every improvement is written back into human-readable text.

Neighborhood-Conditioned Skill Induction

Cases are embedded with BGE and scored by cosine similarity. The top-k (k = 4) neighbours of a case form a local chart on the data manifold that supplies sufficient yet non-redundant evidence. The LLM induces a skill from that neighbourhood, which is then validated on real data, an empty support set means a hallucination and the skill is dropped. Conflicting skills are resolved by empirical accuracy or merged by the LLM.

Execution-Aligned Skill Refinement

For each skill, G stochastic trajectories are sampled and compared with the human reference. The misalignment set collects trajectories below threshold τ, and the instability rate ρ(s) flags the skill for rewriting. The LLM diagnoses the lowest-scoring trajectories and rebuilds the skill as an explicit, step-wise execution scaffold, iterating until all trajectories align or the budget is exhausted.

Context-Budgeted Skill Routing

One lightweight XGBoost classifier per skill predicts whether that skill's preconditions hold for a test case. Classifier hits are unioned with direct structural matches, then truncated to the largest subset that fits the context budget, ordered by classifier confidence and match strength, a minimal yet sufficient skill set for every case.

Over a temporal horizon, KDASO maximizes cumulative trajectory alignment while penalizing library growth, so the library stays accurate, compact and adaptable. The loop is stable in practice: with monthly rolling updates the number of newly induced skills and of merges both decline, the average skill length grows mildly, and routing keeps reusing roughly the same number of skills per case as the library expands.

3. Experimental Results

KDASO is evaluated on three real-world domains — five ride-hailing tasks from DiDi, five legal liability tasks from LawBench and LegalBench, and two logical reasoning benchmarks (ReClor, AR-LSAT) against proprietary LLMs, liable-adjudicated general-purpose algorithms, domain-specific models and open-source general LLMs. All numbers are averaged over three seeds.

Table 1: Ride-hailing adjudication (%). Ordinary liability (O) is the metric DiDi treats as decisive for platform fairness. Bad-faith (B) metrics are in the paper.
Model	CA O-Prec.	CA O-Rec.	CR O-Prec.	CR O-Rec.	FD O-Prec.	FD O-Rec.
NL2FOL	68.21±0.54	62.13±1.87	62.46±0.73	50.66±0.68	77.17±0.64	60.85±0.77
NeoBERT	57.98±1.62	11.02±0.21	63.95±0.71	24.13±0.33	71.13±0.83	9.61±0.18
ModernBERT	63.44±0.75	18.85±0.28	59.02±0.66	16.73±0.04	70.30±0.79	9.89±0.20
SIM-RAG	71.31±0.88	59.16±0.74	58.43±0.92	15.48±0.42	74.27±0.85	37.46±0.51
NodeRAG	72.24±2.89	61.70±0.76	61.66±0.90	31.21±0.53	74.91±0.86	45.23±1.89
Qwen3-8B	68.08±1.82	74.28±0.90	28.56±0.40	85.44±0.95	78.06±2.92	60.44±1.83
QwQ-32B	72.49±0.89	62.30±0.97	38.78±0.52	85.32±0.94	76.55±0.87	63.65±0.78
KDASO (Qwen3-8B)	69.56±0.83	92.01±0.98	74.41±0.85	79.53±0.93	77.99±0.89	63.65±0.84
KDASO (QwQ-32B)	73.42±0.90	87.38±0.97	80.39±0.94	79.09±0.92	79.07±0.91	66.29±0.81

+6% / +20% / +5% / +15%

Average gains in O-Precision, O-Recall, B-Precision and B-Recall over the strongest liable-adjudicated general-purpose algorithm, including a 16-point absolute O-Precision gain on the reservation (CR) task.

+3.5 / +3.9 on logic benchmarks

KDASO tops the public ReClor leaderboard, gaining 3.5 over the previous best logic-specific model and lifting AR-LSAT by 3.9 over its own QwQ-32B backbone.

Table 2: Legal liability determination. LA-ITI and LA-CP come from LawBench. LE-OPP, LE-UN and LE-LH are built on LegalBench.
Model	LA-ITI	LA-CP	LE-OPP	LE-UN	LE-LH
Gemini 3 Pro	39.8	62.9	82.9	85.8	83.2
GPT-4o	44.8	52.9	77.0	87.2	76.6
NL2FOL	34.0	53.8	69.8	83.5	74.3
NeoBERT	38.9	51.7	71.7	83.0	75.0
ModernBERT	33.4	53.7	68.8	83.7	74.5
SIM-RAG	34.2	53.0	72.8	82.1	74.8
NodeRAG	36.3	55.6	72.6	84.8	74.4
Lawyer-LLAMA	4.4	31.3	52.4	83.2	75.3
LexiLaw-6B	22.8	40.0	57.2	82.7	74.6
ChatLaw-13B	27.6	27.9	53.8	82.7	73.2
ChatLaw-33B	19.8	12.7	56.1	84.3	71.7
DeepSeek-R1	44.2	61.0	74.2	84.3	79.0
Qwen3-8B	33.6	53.6	74.2	83.4	73.4
QwQ-32B	39.8	60.9	74.2	85.4	80.6
KDASO (Qwen3-8B)	39.6	56.4	73.8	83.4	76.1
KDASO (QwQ-32B)	44.0	63.5	74.4	86.8	80.4

Table 3: Logical reasoning judgment. Test-H is the hardest AR-LSAT split.
Model	ReClor	AR-LSAT Test	AR-LSAT Test-E	AR-LSAT Test-H
Gemini 3 Pro	96.6	96.1	96.9	94.2
GPT-4o	90.0	89.5	90.3	28.3
NL2FOL	88.1	89.5	87.0	77.0
NeoBERT	91.5	92.4	90.8	80.1
ModernBERT	85.5	88.2	83.4	78.9
SIM-RAG	87.6	88.9	86.6	76.1
NodeRAG	86.7	89.4	84.6	77.2
MERIt	79.3	85.2	74.6	73.0
LReasoner	76.1	87.0	67.5	73.9
IDOL	80.6	87.7	75.0	76.1
AMR-LDA	90.2	91.6	89.1	77.0
DeepSeek-R1	96.0	96.1	95.9	95.2
Qwen3-8B	87.3	87.9	86.8	77.0
QwQ-32B	92.2	94.1	90.7	83.9
KDASO (Qwen3-8B)	87.7	89.3	86.4	78.3
KDASO (QwQ-32B)	93.7	93.9	93.6	87.8

4. Transferability, Stability and Deployment

Skills travel to concept-similar tasks

Skills induced on Cancel–Appeal are dropped into Cancel–Passenger-cancel and Cancel–Driver-cancel with no adaptation, and improve all metrics except a comparable O-Precision, a +8.96% average gain, concentrated in bad-faith precision and recall.

Rolling optimization keeps gaining

With monthly rolling updates from July to December 2025, each month's skills are generated from that month's data plus the previous library and evaluated on the next month, giving a steady month-over-month improvement under a drifting distribution.

From research loop to production review

One hour of expert work yields roughly 100 reviewed cases and 5 valid skills. KDASO processes 3,000 cases and proposes 30 candidate skills in the same hour. No skill is deployed autonomously, every candidate passes mandatory manual review before entering the base.

Failure modes are inspectable

Because each decision is traced to a numbered skill, both directions of error can be audited: false exemptions caused by over-generalised evidence clauses, and missed liability caused by over-strict conjunctions. The paper's case study follows one case in each direction.

BibTeX

@inproceedings{cheng2026kdaso,
  title     = {KDASO: Knowledge-Data Dual-Driven Automated Skill Optimization
               for Liability Adjudication Tasks},
  author    = {Cheng, Zi-Jian and Wu, Weiming and Meng, Jie and Peng, Zhen
               and Huang, Shan and Li, Qun and Wu, Guobing and Guo, Lan-Zhe},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

KDASO: Knowledge-Data Dual-Driven Automated Skill Optimization for Liability Adjudication Tasks