How I Fine Tuned Qwen2.5-Coder-7B to Output Production Ready LaTeX Documents

Overview

• The LaTeX Reliability Gap in Small Open Models
• So How Does an LLM Even Work
• What Fine-Tuning Actually Changes
• Why QLoRA Beats Full Fine-Tuning on Cost
• Where the LaTeX Documents Came From
• How the Dataset Became Training Data
• The Training Run
• What the Loss Curve Told Me
• Evaluation: Did It Actually Compile?
• Reflection

I use LaTeX PDFs to learn new topics in depth. I kept hitting Claude rate limits during long study sessions. That friction pulled me down this rabbit hole.

The LaTeX Reliability Gap in Small Open Models

If you have ever prompted Claude Opus 5 or GPT-5.6 for a technical whitepaper, an academic paper template, or a complex system architecture diagram in TikZ, the output is often remarkably competent. Modern frontier models understand macro expansions, handle nested math environments reasonably well, and structure multi-page documents with consistent styling.

The moment you attempt the same task with open-weights or edge-sized models (the 7B to 8B parameter class), the illusion shatters.

Smaller base code models reliably stumble into the same structural failure modes:

• Unclosed environments: An \begin{align*} or \begin{figure} opened 3,000 tokens prior simply never terminates.
• Macro & package hallucinations: Importing packages that don't exist or invoking arguments not supported by standard TeX Live distributions.
• Syntax drift over long horizons: Over 5,000 to 15,000 tokens, formatting discipline degrades into markdown hybridizations or unescaped special characters (%, _, &).
• Brittle compilation: Unlike Markdown or Python, where an extra whitespace or missing tag often renders gracefully or fails with a localized trace, LaTeX operates under a zero-tolerance compiler gate. A single missing closing brace in a 900-line document causes pdflatex or xelatex to throw a fatal error with exit code 1, leaving you with zero output.
• Flat default styling: Small models fall back to plain article defaults with inconsistent fonts, weak spacing, and no visual hierarchy. The PDF compiles but looks amateur next to frontier output.

Live Comparison: Base Model vs Fine-Tuned

Rather than taking my word for it, test both models side by side below. The base Qwen2.5-Coder-7B and the merged fine-tuned checkpoint use vLLM on L40S GPUs. Full documents usually take 6–20 minutes, and longer ones can take over 25. Save the comparison link to leave and check the results later.

Pick one of the pre-configured prompts (such as Raft Consensus or Transformer Attention) to see how the base model struggles with structure while the fine-tuned model emits verified environments. Click Open in Overleaf on either panel to run the code through a live TeX Live compiler.

Browse the visitor gallery to see prompts and documents generated with this demo. Source downloads appear immediately; PDFs appear after compilation. Prompts containing contact details, links, or blocked words are omitted from the public listing.

---

The High Cost of Relying on Frontier APIs

Frontier models can write LaTeX because their parameter scale and pretraining budgets absorb the vast, messy corpus of arXiv source files and web documents. But relying on closed commercial APIs for document generation carries obvious trade-offs:

1. Inference cost at scale: A single 15-page document with complex TikZ diagrams can easily span 10,000 to 25,000 tokens across input prompts and generated output. Running batch document synthesis through commercial APIs gets expensive quickly.
2. Privacy and sovereignty: Internal system architectures, proprietary whitepapers, financial ledgers, and institutional engineering reports frequently cannot be piped through external APIs.
3. No offline or self-hosted deployment: If you want to embed a document generation engine inside a local workflow, an on-prem microservice, or an automated CI pipeline, proprietary APIs are a non-starter.

Frontier-level open-weights models do exist today in the 70B+ class, but they demand datacenter GPUs with large VRAM. They cannot run on edge devices, laptops, or single-GPU microservices.

This framed the central engineering challenge: can we distill frontier-grade LaTeX generation into a compact, open-weights model, specifically Qwen2.5-Coder-7B-Instruct, so that it consistently produces publication-ready, 100% compilable documents from a single English prompt, all while running within modest hardware budgets?

To make this work, the model couldn't just produce "convincing-looking" TeX markup. It needed to survive an unforgiving compiler pipeline: multi-page documents, clean mathematical proofs, complex tabular data, and fully rendered vector diagrams in TikZ, without human intervention or post-hoc syntax patching.

Getting there required curating a strictly verified 51M-token dataset, engineering a QLoRA fine-tuning pipeline capable of handling 32k context lengths on cloud GPUs, and surviving a gauntlet of memory, checkpointing, and library failures.

So How Does an LLM Even Work

Before we dive into fine-tuning, it is worth stepping back and asking a more basic question: how does an LLM even work, and what actually happens when we train one?

An LLM does only one thing: it predicts the next token. Give it some text, and it scores every possible next piece, picks one, appends it, and repeats. Do that loop thousands of times and you get an essay, some code, or in our case, a full LaTeX document.

\documentclass{article} \begin{document} Press play to start.

Step 0 of 8

Each step predicts and appends one token.

So what guides that choice at every step? The learned weights. Training is what sets those weights, and that is what makes the loop smart.

The model reads billions of pages, guesses the next token at each step, measures how wrong it was, and nudges its weights a little. Repeat that at massive scale and the model absorbs grammar, facts, code patterns, and even LaTeX structure.

At each step the model makes a guess and then checks itself against reality. It predicts the next token, looks at what actually came next in the training text, and computes a single number called loss, low means a good guess, high means a bad one. That loss signal then flows backward through the network and nudges every weight by a tiny amount, so the next guess gets slightly better. Do this over billions of tokens, in batches, over multiple epochs, and those tiny nudges compound into grammar, reasoning, code style, and LaTeX structure. If you want a deeper refresher, I have covered epochs, learning rate, and inference in separate posts.

Pretraining builds the broad base. Fine-tuning steers it.

What Fine-Tuning Actually Changes

A pretrained model knows a little about everything, but it has no reason to prefer one style over another. It can write a poem, a tweet, or a half-broken LaTeX snippet with equal confidence. Fine-tuning changes the odds. You show it thousands of pairs of instructions and ideal answers, and you keep training it with the same next-token loss, but now only on your data.

In my case, that data was 3,000+ verified LaTeX documents, every single one compiled cleanly to PDF. The prompt says "write a paper on X," the answer is a full publication-ready document, and after enough repeats the model stops treating LaTeX as plain text and starts treating it as a strict compiler target. Same engine, new habits.

The chart shows what changes in the model’s predictions. The training objective explains how those changes happen: it scores each correct answer token and adjusts the model to make those tokens more likely.

Why QLoRA Beats Full Fine-Tuning on Cost

You can train all 7 billion weights. You need big GPUs for it. In BF16 the weights alone need around 14GB. Gradients add 14GB more. The two Adam states add 56GB more. The total crosses 84GB before activations load. Long LaTeX documents with 32k context need even more. You need multiple datacenter GPUs for this path.

QLoRA lowers this cost. You freeze the base model in 4-bit quantization, so the base uses about 3.5GB. You train only a pair of small low-rank adapters at each layer. You update less than 1% of the numbers, but the model still learns the new style. This is how a 7B coder model fits on a single A100 with space left for long context.

Where the LaTeX Documents Came From

I didn’t want a dataset full of short LaTeX snippets. I wanted complete documents I’d actually want to read and study from. One dataset I found was DaTikZ-V4, which pairs figure images with TikZ code. That is useful for generating individual diagrams, but I was looking for something different: complete 20- to 30-page documents with the structure, visual polish, and good taste I wanted.

I already had a skill I used in Claude.ai to generate the kind of high-quality PDFs I wanted. I also wanted the generated documents to have some of my own taste: a clear visual hierarchy, readable typography, and diagrams that helped teach rather than just decorate. I turned the skill’s instructions into a prompt for generating training data, then added planning and deduplication logic to produce a more varied set of documents. Variety mattered because I didn’t want the model to learn the same structure over and over. I used the OpenAI API with GPT-5.6 Luna for the bulk of it, about 3,000 documents. About 93% of prompts yielded a compilable document; only those were kept, so every kept document compiles cleanly, averaging about 23 pages and 18k tokens per document, and the total API cost was only about $3. The rest were Claude-extracted examples I had created earlier.

To check the generated documents, I built a LaTeX environment that closely mimicked Overleaf. It included the required fonts and packages, handled some issues automatically, and could still produce a PDF when it encountered recoverable errors. That let me compile every document instead of trusting the source code alone.

How the Dataset Became Training Data

Raw tex files are not trainable. The trainer needs prompt answer pairs in chat format with clean masking. I merged all tracks into one set, shuffled with seed 42, and split into 3,200 train and 177 val for deterministic reruns. Each example became ChatML messages with a fixed system prompt, then user prompt, then assistant LaTeX.

That fixed prompt matters because it is the contract the model learns to obey at inference.

I trained on responses only. Labels for system and user tokens are set to minus 100 so loss ignores them, so the model does not waste capacity memorizing prompts, only generating LaTeX. One formatting trap lived here, and it was about a speed trick called packing. Normally each training example sits in its own slot with empty padding around it, which wastes space. Packing glues several short examples into one slot so almost no space is wasted, which makes training faster. The problem is that packing also attaches a tracking note saying where each example starts and ends, and the part of the trainer that hides prompt tokens from grading cannot read that note, so training crashes. I turned packing off and kept correct grading, accepting slower training for correct output.

The Training Run

With the dataset formatted and masking verified, the next question was infrastructure. I ran the training on Unsloth, which patches HuggingFace's training stack with fused CUDA kernels for roughly 2x speed at half the memory cost, deployed through Modal, a serverless GPU platform where you define the environment in Python, request an A100-80GB, and pay only for the seconds it runs. The training script doubled as the container definition: one file that declared dependencies, mounted a persistent volume for checkpoints and data, and ran training. No Dockerfiles, no orchestration.

The hyperparameters: LoRA rank 16 with alpha 32, targeting all linear layers (not just attention projections, because LaTeX structure needs feed-forward adjustments too). Learning rate 2e-4 with a cosine schedule and 30-step warmup. Max sequence length 32,768 to avoid truncating the longest documents. One epoch over 3,200 examples, 388 optimizer steps, because a single pass over high-quality verified data was enough to absorb the distribution without risking memorization.

The batch size came from a crash. My first attempt used batch size 2 and died with an out-of-memory error. Batch size 2 means the GPU holds activations for two 32k-token sequences simultaneously, and that pushed past 80 GB. I dropped to batch size 1 with gradient accumulation set to 8: same effective batch size, identical training math, but peak memory drops by roughly half because only one sequence lives on the GPU at any moment.

Where 80 GB Goes

The A100-80GB is a spacious GPU, but 32k context fills it faster than you might expect.

The base model in 4-bit quantization takes about 3.5 GB. Adapters, gradients, and optimizer states add roughly 0.5 GB more. The real cost is activations: even with gradient checkpointing, a single 32k-token forward pass still produces roughly 20 GB of activation memory. The remaining headroom absorbs longer documents and the memory spikes during gradient computation. Two sequences at 32k would need roughly 40 GB of activations alone, confirming why batch size 1 was non-negotiable.

Total cost: $13.25 for 2 hours and 25 minutes of A100 time on the final run.

What the Loss Curve Told Me

The trainer logged loss every 10 steps. The early drop was steep, from 1.4765 at step 10 to 1.2079 at step 50, as the adapter picked up basic LaTeX formatting that Qwen already half-knew from general code pretraining. After that, progress slowed. The easy wins were gone and the remaining improvements came from learning subtler things like matching nested environments across long documents. The raw measurements jump around more in this range because short and long documents naturally produce different losses, but the trend line still falls.

Recorded loss5-point averageStep 380 · loss 1.0671

Fast improvement early, smaller gains later. Lowest recorded loss: 1.0481 at step 300.

After step 200 the trend line barely moves. The model hit the floor of what one epoch over 3,200 documents can teach. A second epoch would push the loss lower, but with a dataset this small the risk is memorization, not generalization. One clean pass, then stop. And a falling loss curve only proves the model got better at predicting training tokens. Whether the output actually compiles and looks right needs a different kind of test.

Evaluation: Did It Actually Compile?

Of the 176 documents generated from the held-out validation prompts, 156 compiled to PDF, an 88.6% compile rate.

The successful documents had a median of about 32 pages and 16.6k output tokens. You can inspect all 156 PDFs in the LaTeX evaluation gallery.

For serving, I merged the trained LoRA adapters into the BF16 base model, folding the learned updates into its weights. That gave me one model to load, without separate adapter calculations or 4-bit weight dequantization during generation. I served it with vLLM on an L40S, which managed the attention cache and batched 16 concurrent document requests.

I explicitly configured YaRN with a 1.25 scaling factor and set the serving context limit to 40,960 tokens, up from 32,768. This allowed longer sequences at inference, but the adapters were still trained at 32k. I did not separately measure output quality beyond that training length.

The model followed the document structure I wanted, which was the main thing I wanted to test. Diagrams still need work: some are incomplete, have overlapping elements, or extend beyond the page. A document can compile successfully and still need visual fixes.

Reflection

This started with a small frustration: I wanted to keep learning without waiting for a rate limit to reset. It turned into a project that taught me far more than I expected. There are still diagrams to fix, but opening a readable document from a model I trained myself feels special. I now have something I can keep improving for the study sessions that started all of this.

The LoRA adapter and tokenizer are available on Hugging Face. The training dataset remains private.

Thanks for reading. If you're experimenting with fine-tuning, building something with LaTeX, or have questions about this project, shoot me a DM on LinkedIn. I'd love to hear what you're working on.

— Arnab