The real downsides of ChatGPT, and the fix

Every downside on this page is a mechanism, not a mood. “AI slop” is a complaint. “The model regresses to the most common phrasing, and the first draft is the most common phrasing” is a mechanism you can do something about. The risk register below is the difference.

If you typed chatpt looking for a verdict, this is the honest version: ChatGPT is useful and costly at the same time, and the costs are unevenly distributed. Some are worth paying. Some are not.

The risk register

The rest of the page expands the four that cause the most expensive mistakes.

1. Confident wrongness

OpenAI’s own explainer is direct: language models hallucinate because training and evaluation reward guessing over admitting uncertainty. The model is optimised to produce a plausible answer, not to produce a true one. Fluency is the default; accuracy is the exception that has to be checked.

That has a practical consequence: a wrong answer and a right answer arrive in the same tone. You cannot use confidence as a signal. You need provenance.

A claim ledger is the cheapest fix:

For the answer you just gave, build a table with one row per factual claim:
the claim, whether it came from the material I supplied, from your general knowledge, or
from neither, and the sentence or source that supports it.
If a claim has no support, write UNSUPPORTED. Do not remove it; I want to see it.

Then check the UNSUPPORTED rows. Do not ask the model to “be more accurate”; that is a request to hide uncertainty, not to reduce it.

The same check applies to citations. A citation that looks real can be fabricated: author, title, year, journal, and page number can all be plausible and wrong. If the citation matters, find it in the library or the publisher, not in the chat.

2. You stop noticing

The second risk is not the model’s error. It is yours.

Fluency is persuasive. A polished paragraph reads as competence, and under deadline pressure the review step is the first thing to go. The failure is quiet: nobody catches the wrong number because nobody looks. The model becomes the only reader, and the person who signs off becomes a forwarding layer.

The fix is procedural, not motivational:

• Name the sign-off. “I reviewed this” is not a rule. “The account lead checks every number against the dashboard before send” is.
• Sample, don’t skim. Pick three claims at random and trace them. If one is wrong, stop and check the rest.
• Change the reader. Ask a colleague, or start a clean chat with no history and ask it to read as a stranger. A model inside the thread defends its earlier output.
• Keep a failure log. One line per catch: what was wrong, how it was caught, what check would have caught it earlier. That log is more valuable than another prompt.

3. Homogenised voice

The first draft tends toward the most probable phrasing. That means it drifts toward the most common phrasing in the training data: the same transitions, the same balanced clauses, the same polite summary. If you publish it unchanged, your voice becomes the average of the internet.

This is a mechanism, not a moral failing. The counterweight is a voice spec and an edit pass:

1. Collect three samples of your own writing you are happy with.
2. Extract rules, not adjectives: sentence length range, how you open, how you close, words you never use, how formal you get when the topic is technical.
3. Write the first draft with the model, then edit it against those rules.
4. Keep one section handwritten — the opening, the conclusion, or the paragraph that carries your actual opinion.

A prompt that helps:

Here are three pieces I wrote: [paste].
Write a one-page voice spec with five sections: sentence length and rhythm (with the range
I actually use), how I open, how I close, words and constructions I never use, and how
formal I get on technical topics.
Give each rule one example from the samples and one counter-example.
Then rewrite this draft [paste] to follow the rules, and mark every sentence where you had
to choose between my voice and the generic phrasing.

The last instruction matters. It shows you where the model wanted to sand off the edges.

4. Data you should not have sent

The model is not the main privacy risk. The text you paste is.

A prompt box is a third party. If you would not email the text to an outside contractor, do not paste it. That rule covers customer records, employee data, student records, credentials, unreleased financials, privileged legal material, and anything your employer’s policy forbids.

Temporary Chat is the right mode for a sensitive one-off, but it is not a guarantee of anonymity, and it does not make an inappropriate paste appropriate. De-identify first; use the minimum context; keep the secret out.

The full list and the recovery steps are in What not to paste into ChatGPT. The short version: rotate the credential first, then delete the chat. A deleted message is not a revoked key.

5. Skill atrophy

This one is hard to measure and easy to dismiss, so treat it as a risk rather than a fact. The mechanism is simple: if you delegate the reps, you do not get the reps. A junior writer who never writes an opening learns to edit openings, not to write them. A junior analyst who never builds the spreadsheet learns to check a spreadsheet.

The fix is a draft-only zone: pick one task you are still learning, and use the model only after you have done a first pass yourself. Or use it to critique your attempt, not to replace it. The point is to keep the part of the work that builds the skill.

This is a personal and management decision, not a product setting. No prompt can make it for you.

6. Invisible labour

The demo shows the first output. The job includes the prompt, the reading, the correction, the re-prompt, and the final check. If you time only the first output, the tool looks faster than it is.

Time the whole loop for one week:

• minutes spent writing and revising prompts;
• minutes spent reading and checking;
• minutes spent redoing the parts that were wrong;
• minutes spent on the task you would have done instead.

If the loop is shorter, keep it. If it is longer, the task is not a good fit yet. That is a more useful answer than a generic “AI saves time” claim.

7. Fixed cost for variable need

A subscription is usually monthly. The need may be seasonal: a job hunt, a launch, a dissertation, a tax season. Paying every month for a tool you use intensely for three weeks is a shape mismatch, not a moral one.

Do the one-week test: use the free tier for the tasks you actually have, write down where you hit a wall, and only pay for the plan that removes that wall. We do not publish prices; they change, and the official pricing page is the current source.

8. Institutional risk

If you work in a school, a hospital, a law firm, a bank, or any organisation with a data policy, the risk is not only yours. It is the organisation’s.

• Policy: does the organisation allow this tool, this data, this use?
• Disclosure: does the client, student, patient, or reader need to know AI was used?
• Audit: can you show what you pasted, where it went, and who checked the output?
• Contract: does a data-processing agreement cover this tool, or does it forbid it?

When the answer is unclear, the safe default is the strictest policy in the room. Ask before you paste. A one-line answer from the person who owns the policy is worth more than a hundred prompt tips.

When the downside is worth it

This is not an argument for abstaining. Some trade is obviously worth it:

• low-stakes drafting where a human review is cheap;
• summarising material you already own and can check;
• generating options you will choose between, not publish directly;
• explaining a concept at a different level, with a source you can verify;
• turning a structured input into a structured output, where the shape is the check.

The question is not “is ChatGPT good?” It is “which task, with which data, checked by whom, at what cost?” If you can answer those four, you can make a decision. If you cannot, the demo is doing the thinking for you.

A 15-minute risk review

Before you add ChatGPT to a recurring workflow, answer these on one page:

• ☐ Which task, exactly, and what does “done” look like?
• ☐ What data goes in, and who owns it?
• ☐ Which claims will a human check, and who signs off?
• ☐ What happens when the answer is wrong — who catches it, and how?
• ☐ How long does the whole loop take, including checking?
• ☐ What is the fallback if the tool is unavailable or the policy changes?
• ☐ Would I be comfortable explaining this workflow to the person whose data it is?

If any box is empty, the workflow is not ready. That is not a reason to avoid the tool forever. It is a reason to start smaller.

The short version

The real downsides are not “it is sometimes wrong”. They are the systems that grow around a fluent wrong answer: the review that gets skipped, the voice that gets flattened, the data that gets pasted, the skill that stops being practised, the hour that gets spent fixing what looked free. Name the mechanism, put a check next to it, and the trade becomes a decision instead of a vibe.