General-purpose language models are generalists by design. For a course project in my MSc NLP class, I wanted to see how far I could push a small open model towards one domain, general-practice-style health consultation, using only free hardware and data I could legally defend.
The setting was Ghana, so the training mix includes Ghana's national treatment guidelines alongside international research and patient-information text. The constraint was a single free Google Colab T4 GPU.
This was a learning exercise in adapting language models. The model is not a medical device, has not been validated by clinicians, and must never replace professional care.
The plan in one picture

I compared three versions of the same model on identical tests:
- Model A: the untouched base model,
Qwen3.5-2B-Base - Model B: A plus Stage 1, continued pre-training on health text ("read widely")
- Model C: B plus Stage 2, instruction tuning on medical Q&A ("practise answering")
Starting from a base model rather than an instruct model was deliberate. It means every change in behaviour can be attributed to my two stages and not to someone else's post-training.
Data I could defend
The most time-consuming part of the project was not training. It was deciding what I was allowed to train on.
Stage 1 text (about 14 million words):
| Source | What it is | Used | Licence |
|---|---|---|---|
| PubMed Central Open Access | Full-text research articles via the AWS open-data bucket | ~2,575 clinical articles | Recorded per article (CC BY, CC BY-NC, CC BY-NC-SA, CC0); no-derivatives licences excluded |
| MedlinePlus | Plain-language health topics from the US National Library of Medicine | 1,014 topics | Public domain, with credit |
| Ghana Standard Treatment Guidelines and Essential Medicines List (2017) | National clinical guidance | ~176k words, repeated 3× | Ghana Ministry of Health |
Stage 2 instruction data (9,000 training, 300 validation examples) came from medical-o1-reasoning-SFT, MedText, MedQuAD, and the training splits of MedMCQA and MedQA, each with its licence checked.
Just as important is what I left out: scraped patient–doctor chat datasets (unclear rights and possible personal data), StatPearls (its no-derivatives licence conflicts with fine-tuning), copyrighted USMLE textbook corpora, and GPL-licensed term lists. Leaving them out cost a little data and saved a lot of risk.
Two rules from this stage I now apply to every ML project:
- Keep the licence with every item. Each document carries its source URL and licence, so "we used some data" becomes "here is exactly what we used and why we were allowed to."
- Split before you train. About 5% of PubMed articles, 10% of MedlinePlus topics and every tenth window of the Ghana guidelines were set aside as held-out text before anything else happened, along with WikiText-2 passages as a general-English control.
Fine-tuning cheaply with LoRA
Retraining all two billion parameters on a T4 is not realistic. LoRA freezes the original weights and learns a small low-rank update for each targeted layer instead:
W' = W + (α / r) · B: A

I used rank r = 32 and α = 64 on the attention and feed-forward projections (q, k, v, o, gate, up, down). That is 21.8 million trainable parameters out of 2.2 billion, under 1%, and the base model stays intact, so Model A is always exactly recoverable.
| Setting | Value |
|---|---|
| Hardware | Google Colab T4 (16 GB) |
| Stack | Unsloth, Hugging Face Transformers v5, PyTorch |
| Model | Qwen3.5-2B-Base in 16-bit (Unsloth advises against 4-bit training for Qwen3.5; the 4B model did not fit) |
| Optimiser | 8-bit AdamW, cosine schedule, 10 warm-up steps, learning rate 1e-4 |
| Stage 1 | Documents packed into 2,048-token blocks, 16-step gradient accumulation |
| Stage 2 | ChatML format, loss computed on answer tokens only |
That last detail matters. In Stage 2 the prompt tokens are masked out of the loss, so the model is graded only on its answer and doesn't waste capacity learning to predict the question.
Was it learning?
The first evidence comes from the loss curves.


Falling validation loss on text the model never trained on is a good sign, but it's not the end of the story. The real test is held-out perplexity and benchmark accuracy.
Results

Held-out perplexity (lower is better)
| Model | PubMed Central | MedlinePlus | Ghana STG/EML | WikiText-2 (control) |
|---|---|---|---|---|
| A: Base | 5.15 | 4.12 | 6.59 | 9.66 |
| B: + Stage 1 | 4.54 | 4.06 | 5.99 | 9.50 |
| C: + Stages 1 & 2 | 4.58 | 4.07 | 6.20 | 9.51 |
Zero-shot multiple-choice accuracy (N = 300 per task)
| Model | MedQA | MedMCQA | PubMedQA |
|---|---|---|---|
| A: Base | 59.3% | 58.3% | 70.0% |
| B: + Stage 1 | 60.3% | 58.0% | 72.7% |
| C: + Stages 1 & 2 | 58.0% | 59.0% | 76.0% |
What the numbers actually say
Stage 1 clearly worked as domain adaptation. Perplexity on unseen research articles fell by about 12% and on held-out Ghana guideline text by about 9%. The model became measurably less surprised by health writing it had never seen.
There was no catastrophic forgetting. The general-English control (WikiText-2) didn't get worse; it actually improved slightly, from 9.66 to 9.50. I added this control specifically to catch the case where the model trades general language skill for medical vocabulary, and that didn't happen.
Exam accuracy barely moved, and I'm not going to pretend otherwise. With 300 questions per task, the margin of error is roughly ±5.7 percentage points. MedQA and MedMCQA changes are well inside that, so the honest reading is no reliable difference. PubMedQA rose six points from A to C, which sits right at the edge of significance: encouraging, not conclusive.
This is a legitimate finding, not a failure. A modern 2B base model has already read a lot of biomedical text during pre-training. A few million tokens of extra reading sharpens its language model of the domain (hence the perplexity gains) without adding much new exam-style knowledge.
Instruction tuning nudged perplexity back up slightly. Model C is a little worse than B on raw text perplexity, which is expected: Stage 2 optimises for answering in a fixed format, not for predicting article text.
Lessons I'm keeping
- "Public on the internet" does not mean "free to train on." Before this health project, our team started on a Ghanaian legal assistant and found that two legal sites explicitly forbade scraping and AI training. We checked before collecting and used official sources instead.
- Check that your base model is current. My first pick was two generations old. A quick search found Qwen3.5, which came with new rules of its own (Transformers v5, no 4-bit training advised).
- Use controls. Without WikiText-2 I could not have claimed "no forgetting."
- Beware small differences. Confidence intervals stopped me from over-claiming a one-point MCQ gain as progress.
- Compute sets the scope. Roughly 19 million training tokens were available; the T4 budget only allowed training on a fraction. That's a clear future-work item, not a hidden flaw.
What I would do next
The natural next steps are retrieval over the Ghana guidelines so answers can cite their source, a clinician-graded evaluation of the GP vignettes, safety alignment for refusals and red-flag symptoms, and support for local languages, which connects directly to my work on Ewe.
Code, data-collection scripts, licence notes and the full technical journal are in the repository.



