LLM Returns Invalid JSON — Diagnosis & Fix Guide

You asked the model for JSON. It gave you JSON for two weeks. Now JSON.parse is throwing in production, on maybe one request in fifty, and the logs show something like this:

SyntaxError: Unexpected token ` in JSON at position 0

This guide covers the four distinct causes behind that error and the order to address them in.

🔍 First: find out which failure you have

Before changing anything, log the raw response body on parse failure. The four causes look identical from the exception and completely different from the payload.

try {
  return JSON.parse(raw);
} catch (err) {
  // Log the payload, bounded. The exception alone tells you nothing useful.
  console.error('LLM JSON parse failed', { raw: raw.slice(0, 500) });
  throw err;
}

1. 🛠️ Fencing and preambles — use constrained decoding

Chat-tuned models are trained to present code nicely to humans. Asking for "only JSON, no markdown" lowers the rate; it does not eliminate it, and the rate changes every time the provider updates the model behind the same name.

❌ Prompting and hoping

const prompt = 'Return ONLY valid JSON. No markdown. No explanation.';
const raw = await model.generate(prompt + input);
return JSON.parse(raw); // throws on the polite responses

✅ Constraining the decoder

Every major provider now supports forcing the output to a schema. The decoder is physically prevented from emitting a token that would make the output invalid, so fences and preambles cannot occur:

const result = await ai.run(MODEL, {
  messages,
  response_format: {
    type: 'json_schema',
    json_schema: {
      type: 'object',
      properties: {
        category: { type: 'string', enum: ['billing', 'bug', 'feature', 'other'] },
        estimatedHours: { type: 'number' },
        confidence: { type: 'string', enum: ['low', 'medium', 'high'] },
      },
      required: ['category', 'estimatedHours', 'confidence'],
    },
  },
});

If your provider genuinely has no structured-output mode, then — and only then — strip defensively, and treat it as a temporary measure:

// Fallback only. This handles the failure you have seen, not the next one.
const cleaned = raw.replace(/^[^{[]*/, '').replace(/[^}\]]*$/, '');

2. 🛠️ Truncation — check the finish reason

If the payload stops mid-object, the JSON is not malformed. It is incomplete, because generation hit the output token ceiling.

if (response.finishReason === 'length') {
  // Not a parse problem. The model was cut off.
}

Three fixes, best first:

1. Ask for less. Return ids instead of full records; return the top five, not everything. Most truncation is a schema asking the model to restate data you already have.
2. Raise the output limit. Cheap, but it moves the ceiling rather than removing it.
3. Paginate the work. For genuinely large outputs, loop over chunks rather than requesting one enormous object.

3. 🛠️ Schema drift — validate, do not trust

Structurally valid JSON with the wrong shape is the most dangerous case, because nothing throws. The field is simply undefined three functions away.

import { z } from 'zod';
 
const ClassificationSchema = z.object({
  category: z.enum(['billing', 'bug', 'feature', 'other']),
  estimatedHours: z.number().positive().max(200),
  confidence: z.enum(['low', 'medium', 'high']),
});
 
// Constrained decoding shapes the output; this proves it.
const parsed = ClassificationSchema.safeParse(result.response);
if (!parsed.success) {
  // A caught, logged, handleable failure — not a silent undefined downstream.
  return fallbackClassification();
}

Note the .max(200) on hours. Constrained decoding guarantees a number; it does not guarantee a sensible number. Range checks on numeric fields catch the responses that are valid, parseable, and absurd.

4. 🛠️ Decide the fallback before you need it

Every LLM call needs an answer to "what does the user see when this fails?" — and for a parse failure specifically, retrying the identical request is rarely it. If the model produced unusable output once, the same prompt at the same temperature will often do it again.

Better options, in order:

• Serve a degraded result. The AI-assisted part is absent; the feature still works. Correct for suggestions, summaries, and classifications with a sane default.
• Retry once, with a lower temperature. Worth one attempt, not three.
• Queue for human review. Correct when the output feeds something consequential.

Whatever you pick, put the failure on a counter. A parse failure rate that quietly rises from 1% to 20% after a provider-side model update is invisible unless something is watching it.

✅ Checklist

[ ] Raw payload logged (bounded) on every parse failure
[ ] response_format json_schema used where the provider supports it
[ ] enum constraints on every fixed-set string field
[ ] Finish reason checked for `length` before blaming the parser
[ ] Runtime validation with range checks, not just a cast
[ ] A defined, tested fallback path
[ ] Failure rate on a counter with an alert

Work through it in that order. The first two items remove most of the volume; the rest is what stops the remainder becoming an incident.