Eugene.Devhello@eugene.dev
All articles

LLM, Fine-tuning, LoRA, Healthcare, Evaluation

Adapting a 2B Language Model to Health on a Free GPU

How I fine-tuned Qwen3.5-2B-Base on open health text and medical Q&A with LoRA on a Colab T4, and what the held-out numbers actually say.

Eugene Sewor

18 November 2026, 7 min read

General-purpose language models are generalists by design. For a course project in my MSc NLP class, I wanted to see how far I could push a small open model towards one domain, general-practice-style health consultation, using only free hardware and data I could legally defend.

The setting was Ghana, so the training mix includes Ghana's national treatment guidelines alongside international research and patient-information text. The constraint was a single free Google Colab T4 GPU.

This was a learning exercise in adapting language models. The model is not a medical device, has not been validated by clinicians, and must never replace professional care.

The plan in one picture

Pipeline diagram: collect open health text, split held-out sets first, then Stage 1 continued pre-training, Stage 2 instruction tuning, and evaluation of three models.
The pipeline. Test data is set aside in step 2, before any model sees the training text.

I compared three versions of the same model on identical tests:

  • Model A: the untouched base model, Qwen3.5-2B-Base
  • Model B: A plus Stage 1, continued pre-training on health text ("read widely")
  • Model C: B plus Stage 2, instruction tuning on medical Q&A ("practise answering")

Starting from a base model rather than an instruct model was deliberate. It means every change in behaviour can be attributed to my two stages and not to someone else's post-training.

Data I could defend

The most time-consuming part of the project was not training. It was deciding what I was allowed to train on.

Stage 1 text (about 14 million words):

Source What it is Used Licence
PubMed Central Open Access Full-text research articles via the AWS open-data bucket ~2,575 clinical articles Recorded per article (CC BY, CC BY-NC, CC BY-NC-SA, CC0); no-derivatives licences excluded
MedlinePlus Plain-language health topics from the US National Library of Medicine 1,014 topics Public domain, with credit
Ghana Standard Treatment Guidelines and Essential Medicines List (2017) National clinical guidance ~176k words, repeated 3× Ghana Ministry of Health

Stage 2 instruction data (9,000 training, 300 validation examples) came from medical-o1-reasoning-SFT, MedText, MedQuAD, and the training splits of MedMCQA and MedQA, each with its licence checked.

Just as important is what I left out: scraped patient–doctor chat datasets (unclear rights and possible personal data), StatPearls (its no-derivatives licence conflicts with fine-tuning), copyrighted USMLE textbook corpora, and GPL-licensed term lists. Leaving them out cost a little data and saved a lot of risk.

Two rules from this stage I now apply to every ML project:

  1. Keep the licence with every item. Each document carries its source URL and licence, so "we used some data" becomes "here is exactly what we used and why we were allowed to."
  2. Split before you train. About 5% of PubMed articles, 10% of MedlinePlus topics and every tenth window of the Ghana guidelines were set aside as held-out text before anything else happened, along with WikiText-2 passages as a general-English control.

Fine-tuning cheaply with LoRA

Retraining all two billion parameters on a T4 is not realistic. LoRA freezes the original weights and learns a small low-rank update for each targeted layer instead:

W' = W + (α / r) · B: A
Diagram of LoRA: a frozen weight matrix W with two small trainable matrices B and A added alongside it.
LoRA in one picture: the original weights stay frozen, and only two small matrices are trained.

I used rank r = 32 and α = 64 on the attention and feed-forward projections (q, k, v, o, gate, up, down). That is 21.8 million trainable parameters out of 2.2 billion, under 1%, and the base model stays intact, so Model A is always exactly recoverable.

Setting Value
Hardware Google Colab T4 (16 GB)
Stack Unsloth, Hugging Face Transformers v5, PyTorch
Model Qwen3.5-2B-Base in 16-bit (Unsloth advises against 4-bit training for Qwen3.5; the 4B model did not fit)
Optimiser 8-bit AdamW, cosine schedule, 10 warm-up steps, learning rate 1e-4
Stage 1 Documents packed into 2,048-token blocks, 16-step gradient accumulation
Stage 2 ChatML format, loss computed on answer tokens only

That last detail matters. In Stage 2 the prompt tokens are masked out of the loss, so the model is graded only on its answer and doesn't waste capacity learning to predict the question.

Was it learning?

The first evidence comes from the loss curves.

Stage 1 loss chart: noisy training loss between 1.5 and 1.65, and a smooth validation loss falling from about 1.34 to 1.31.
Stage 1, continued pre-training. Training loss is noisy (each step sees different documents), but validation loss on held-out blocks falls steadily and never turns upward.
Stage 2 loss chart: training and validation loss both falling, validation from about 1.48 to 1.16.
Stage 2, instruction tuning. Validation loss on unseen Q&A drops from about 1.48 to 1.16 nats per answer token.

Falling validation loss on text the model never trained on is a good sign, but it's not the end of the story. The real test is held-out perplexity and benchmark accuracy.

Results

Two bar charts. Left: held-out perplexity for MedlinePlus, Ghana guidelines, PubMed Central and WikiText-2 across models A, B and C. Right: zero-shot accuracy on MedQA, MedMCQA and PubMedQA.
Held-out perplexity (lower is better) and zero-shot multiple-choice accuracy (higher is better) for all three models. The dashed line is chance on four-option questions.

Held-out perplexity (lower is better)

Model PubMed Central MedlinePlus Ghana STG/EML WikiText-2 (control)
A: Base 5.15 4.12 6.59 9.66
B: + Stage 1 4.54 4.06 5.99 9.50
C: + Stages 1 & 2 4.58 4.07 6.20 9.51

Zero-shot multiple-choice accuracy (N = 300 per task)

Model MedQA MedMCQA PubMedQA
A: Base 59.3% 58.3% 70.0%
B: + Stage 1 60.3% 58.0% 72.7%
C: + Stages 1 & 2 58.0% 59.0% 76.0%

What the numbers actually say

Stage 1 clearly worked as domain adaptation. Perplexity on unseen research articles fell by about 12% and on held-out Ghana guideline text by about 9%. The model became measurably less surprised by health writing it had never seen.

There was no catastrophic forgetting. The general-English control (WikiText-2) didn't get worse; it actually improved slightly, from 9.66 to 9.50. I added this control specifically to catch the case where the model trades general language skill for medical vocabulary, and that didn't happen.

Exam accuracy barely moved, and I'm not going to pretend otherwise. With 300 questions per task, the margin of error is roughly ±5.7 percentage points. MedQA and MedMCQA changes are well inside that, so the honest reading is no reliable difference. PubMedQA rose six points from A to C, which sits right at the edge of significance: encouraging, not conclusive.

This is a legitimate finding, not a failure. A modern 2B base model has already read a lot of biomedical text during pre-training. A few million tokens of extra reading sharpens its language model of the domain (hence the perplexity gains) without adding much new exam-style knowledge.

Instruction tuning nudged perplexity back up slightly. Model C is a little worse than B on raw text perplexity, which is expected: Stage 2 optimises for answering in a fixed format, not for predicting article text.

Lessons I'm keeping

  • "Public on the internet" does not mean "free to train on." Before this health project, our team started on a Ghanaian legal assistant and found that two legal sites explicitly forbade scraping and AI training. We checked before collecting and used official sources instead.
  • Check that your base model is current. My first pick was two generations old. A quick search found Qwen3.5, which came with new rules of its own (Transformers v5, no 4-bit training advised).
  • Use controls. Without WikiText-2 I could not have claimed "no forgetting."
  • Beware small differences. Confidence intervals stopped me from over-claiming a one-point MCQ gain as progress.
  • Compute sets the scope. Roughly 19 million training tokens were available; the T4 budget only allowed training on a fraction. That's a clear future-work item, not a hidden flaw.

What I would do next

The natural next steps are retrieval over the Ghana guidelines so answers can cite their source, a clinician-graded evaluation of the GP vignettes, safety alignment for refusals and red-flag symptoms, and support for local languages, which connects directly to my work on Ewe.

Code, data-collection scripts, licence notes and the full technical journal are in the repository.

Code for this article