Agent Trace to Dataset Studio

1. Sampling Strategy 8 / 24 Traces

Random Baseline Control

Unbiased pull of everyday production flows (33% sample).

Stratified Balance Tier / Reason

Equal quota across VIP tiers & refund edge cases.

Failure-Weighted High Signal

Over-samples errors, rule breaches & dispute fallouts.

Sampled Trace Queue

2. Span Tree & Rubric Evaluation Trace TR-104

Automated Evaluation Rubric

Refund within Terms Deterministic

Checks order delivery timestamp ≤ 30 days and refund amount ≤ total value.

No Invented Terms LLM-as-Judge

Flags hallucinated restocking fees or non-existent company policies.

Escalate Chargebacks Safety

Requires handoff to human supervisor if active dispute/chargeback exists.

Execution Spans (OpenTelemetry OTLP GenAI) 4 spans • 1,280 ms • 642 tokens

3. Training Dataset Compiler 5 Golden Runs

Dataset Pairs

3,410

Est. Tokens

Violations Flagged

Compiled Output (sft.jsonl)

{"messages": [{"role": "system", "content": "You are a customer support agent..."}, ...]}

* Ingested traces follow OpenTelemetry GenAI semantic conventions. Compiles verified behavioural training rows ready for Axolotl, Unsloth, or Llama-Factory.

Sampling and rubric filters determine which fixture traces enter JSONL

Your browser does not support video.

Read the explanation

The saved studio contains twenty four bundled trace fixtures. The default failure weighted strategy selects the first eight and leaves sixteen outside the sample. At ten pixels per trace pool, sample and remainder measure two hundred forty, eighty and one hundred sixty. The strategy called random actually uses eight fixed spaced indexes. Stratification takes three VIP, three Standard and two Enterprise traces. These are local fixtures, not independently proven real agent runs, and no sampling service or model executes. Choosing a strategy changes which examples the local checks inspect, not the complete population. With all three default rubrics active the eight sampled traces divide into five passing and three flagged. At thirty pixels per trace the bars measure two hundred forty, one hundred fifty and ninety. Refunds after thirty days, refunding an active dispute and the stored invented policy flag trigger rejection. The comment calls one check a language model judge, but the implementation only reads that boolean. Turning a rule off can admit a previously flagged example without changing its response. Passing therefore means these enabled local predicates passed, not verified policy quality or safety. SFT exports one line for each passing trace, with system, user and assistant messages. Five rows contain fifteen message objects. At twenty pixels per count rows measure one hundred, three messages per row sixty and total messages three hundred. DPO instead writes prompt, chosen, rejected and a fixed score of one. Its rejected text is prewritten for each passing trace; the three flagged traces are excluded rather than automatically paired. Token totals divide character lengths by three point eight and round, not a tokenizer. Actual JSONL download and format switching can be tested locally, but there is no training, provider upload or model evaluation.