RAG vs. fine-tuning: how to prepare support data for each

“Clean the support data for AI” sounds like one task. In practice, retrieval-augmented generation (RAG), fine-tuning, and evaluations need different records, quality rules, and outputs. Choosing the task after cleaning can lead to a polished dataset that is structurally wrong for the system using it.

The most durable approach is to build one lossless normalized conversation layer, then derive task-specific datasets from it.

The short decision rule

Use RAG when the system needs access to changing facts, policies, product details, or prior resolutions at response time. Use fine-tuning when you need to shape consistent behavior, output format, classification, or response style. Use evals in both cases to measure whether the system works on representative support scenarios.

These techniques are not mutually exclusive. A fine-tuned model can use retrieval, and both should be evaluated against a versioned test set.

What RAG needs from support data

RAG retrieves relevant source material and supplies it to a model when answering. The dataset therefore needs to be searchable, attributable, and segmented around useful knowledge.

Raw ticket threads are rarely ideal retrieval documents. They contain greetings, repeated quotes, status updates, and dead ends. A RAG-oriented transformation might produce a resolved case containing:

• the customer problem stated without personal details;
• product, feature, channel, and issue metadata;
• the validated resolution steps;
• source date and policy version where relevant;
• authorization metadata required for access filtering;
• a stable non-sensitive source reference.

Chunk boundaries should preserve a complete troubleshooting idea. Splitting a numbered procedure halfway through can retrieve the symptom without the resolution. Conversely, storing a fifty-message thread as one document can dilute the matching signal.

Before committing to a chunking rule, test retrieval with real questions and inspect what comes back. Retrieval quality is an empirical property of the corpus, query set, embedding model, and ranking configuration.

What fine-tuning needs from support data

Fine-tuning learns patterns from examples. It does not provide a reliable way to keep changing facts current. The dataset should demonstrate the behavior you want the model to repeat.

For support conversations, that usually means selecting agent responses that are correct, complete, appropriately toned, and based on enough context. Exclude examples where the agent guessed, followed an obsolete policy, or relied on private notes that the deployed model will not receive.

A conversation example might include:

{
  "messages": [
    { "role": "system", "content": "Follow the approved support policy." },
    { "role": "user", "content": "I was charged twice for [ORDER_ID]." },
    { "role": "assistant", "content": "I can help verify the duplicate charge..." }
  ]
}

Consistency matters more than volume. Contradictory examples teach contradictory behavior. Normalize role labels, remove hidden information, redact PII, and ensure the prompt contains the context required to justify the target response.

Maintain training and validation splits that prevent near-duplicate conversations from appearing on both sides. Otherwise, validation results can overstate how well the behavior generalizes.

What an eval dataset needs

Evals turn product expectations into repeatable tests. Build them before or alongside the first RAG or fine-tuning experiment, not only after launch.

Each case should contain:

• a realistic user input and available context;
• the relevant issue and risk categories;
• an expected answer, set of required facts, or scoring rubric;
• explicit failure conditions such as fabricating policy or exposing PII;
• segmentation fields for channel, language, product, and difficulty.

Do not reuse the fine-tuning training set as the primary eval set. Select separate cases that represent production traffic, important edge cases, and known failure modes. Version the eval set so score changes can be traced to either the system or the test data.

Build one normalized source layer

The shared cleaning work happens before the paths diverge:

1. Parse exports into a consistent conversation schema.
2. Map custom field IDs to documented meanings.
3. Reconstruct turns and distinguish public replies from internal notes.
4. Remove exact and near duplicates.
5. Detect and redact PII using generic and domain-specific rules.
6. Add quality flags rather than silently deleting uncertain records.
7. Preserve private lineage from derived records to source records.

From that layer, generate separate RAG documents, fine-tuning examples, and eval cases. This keeps privacy and normalization policies consistent while allowing each output to evolve.

The Zendesk export cleaning guide walks through this shared pipeline in platform-specific detail.

Avoid common mismatches

Using fine-tuning as a knowledge base

If policies, prices, and product behavior change often, encoding them in training examples makes updates slow and difficult to verify. Retrieve current approved material instead, and fine-tune only when behavior needs additional control.

Indexing every raw ticket

More documents do not automatically produce better retrieval. Low-quality and contradictory resolutions compete with good ones. Filter, label, and test the corpus against realistic queries.

Evaluating only average cases

A high overall score can hide failure on refunds, account access, security incidents, or another small but costly segment. Report results by risk and issue category.

Redacting after the dataset spreads

Apply the required privacy transformations before chunking, embedding, training-file creation, or vendor transfer. See the full PII redaction guide for detection and audit practices.

Choose the output by evidence

Start with a small representative slice and compare approaches against the same eval set. Measure retrieval relevance, answer correctness, policy compliance, privacy failures, latency, and operating cost. The decision should come from the task and measured behavior, not from which technique is currently more fashionable.

A practical first delivery may contain:

• normalized conversations in Parquet for analysis;
• curated resolution documents in JSONL for RAG;
• a smaller set of high-quality examples for fine-tuning experiments;
• a held-out eval set with rubrics and segmentation;
• a redaction audit, field mapping, and quality report.

ChatClean prepares support exports for all three paths without forcing the downstream decision before the data is understood. Review the cleaning workflow and pricing when you want a task-ready dataset and its audit artifacts delivered together.

Sources

• RAG vs. fine-tuning — AWS Prescriptive Guidance
• Fine-tuning best practices — OpenAI
• Microsoft Presidio documentation