Do you need a custom LLM? Fine-tuning vs RAG vs a private ChatGPT

A custom LLM means a model trained or retrained on your data. A private ChatGPT means a business workspace where staff upload files and ask questions. RAG sits between them: a system that searches your documents for each question and hands the matching passages to a standard model.

• Documents in the prompt. The system sends the whole document set with every question. Anthropic says this works for knowledge bases under 200,000 tokens, about 500 pages.
• RAG. The system finds the passages that match each question and sends only those to the model, so it can cite the passage it used.
• Fine-tuning. Extra training on example inputs and ideal outputs. It changes how a model answers more than what it knows.

Softomate's 4 to 8 weeks covers simple chatbots with under 50 documents (4 to 6 weeks) and standard ones with one CRM connection (6 to 8 weeks).

Fine-tuning compute is cheap. The expensive parts are preparing the examples and hosting the tuned model.

OpenAI's fine-tuning wind-down

OpenAI's fine-tuning guide, checked 7 November 2026, says "OpenAI is winding down the fine-tuning platform" and that it "is no longer accessible to new users". Existing users can create training jobs "for the coming months", and fine-tuned models stay available until their base models are deprecated. An agency that pitches OpenAI fine-tuning to a new client in late 2026 cannot deliver it on OpenAI's platform.

Open-weight fine-tuning prices

Together AI lists LoRA supervised fine-tuning per 1 million training tokens as of November 2026:

A worked example

Take 2,000 example pairs of 500 tokens each, which is 1 million tokens. A training run that reads the set three times uses 3 million tokens. That costs $1.02 on Llama 3.1 8B, so the $4 minimum applies, $6.09 on Llama 3.3 70B and $21 on DeepSeek-V3.1.

Hosting costs more. If the tuned model needs its own dedicated H100, Together AI charges $5.49 an hour. Running it all month (720 hours) costs $3,952.80, before anyone has written the 2,000 examples.

Training from scratch

Training a new model means renting GPU clusters for long runs, and Together AI lists on-demand H100 clusters at $3.99 per GPU-hour. A small business has no reason to do this, because RAG on a current model already answers from its documents.

Business plans and paid APIs from OpenAI and Anthropic do not train on your data by default. Consumer plans differ, and retention rules differ by provider.

• ChatGPT Business lists "No training on your business data by default". The same page shows that on the Free, Go, Plus and Pro plans, content is used to train models unless the user opts out.
• Claude Team lists "No model training on your content by default".
• OpenAI says API data is not used to train its models unless the customer opts in, and keeps abuse-monitoring logs for up to 30 days. Its UK endpoint stores data in the UK but does not process it there. Its Europe endpoint does both.
• Anthropic says API prompts and outputs are not retained by default. The exception is its Covered Models (Fable and Mythos), which require 30-day retention.

Under UK GDPR the business stays the controller of any personal data it puts into these tools. The ICO's guidance on AI and data protection, last updated 15 March 2023 and under review after the Data (Use and Access) Act, covers data protection impact assessments, lawfulness, transparency and statistical accuracy. Run an impact assessment before you load customer or staff records, and sign the provider's data processing terms.

RAG is easier to keep compliant than fine-tuning. A retrieval index is a store of documents you can edit or delete. A fine-tuned model holds what it learned in its weights, so removing one person's data means training again.

Work down this list and stop at the first option that does the job.

1. Staff only need answers from internal files. Buy ChatGPT Business or Claude Team seats for a month and test them on 20 real questions.
2. The document set is under about 500 pages and rarely changes. Put it in the prompt of a simple build. No retrieval step is needed.
3. Customers will ask the questions, or the set is large or changes weekly. Build RAG with source links in every answer and a test set before launch.
4. Answers are right but come back in the wrong shape, such as a fixed report layout. Try better instructions and examples in the prompt first, then fine-tune an open-weight model.
5. Someone proposes training a model from scratch. Ask what a current model plus RAG fails to do. For a small business the answer is almost always nothing.

For systems that act rather than answer, see AI agent or automation.