  |
Evaluation and interpretation of written material |
| Home > For doctors > GP training > MRCGP exam |
|
|
Evaluation and interpretation of written material |
We will now look at preparation for the questions on evaluation and interpretation of written material.
The material may vary from the usual methods and results examples to meta-analyses, editorial pieces, abstracts or even letters.
Reading time is provided.
These questions test
Although these questions cause most of the worry try and get them into perspective. No one expects a statistician or referee's response to the material provided. You have the material to use, so look at it and practise reading abstracts, summaries, methods and results sections from papers n a calm and controlled 'Nay (easier said than done). Also adopt a systematic approach to the written material, stating the obvious is often all that is required to get a pass but you need to practice this technique as once you get flustered in exam conditions, you will lose easy marks. Questions are likely on study design and the methods and results section of papers.
Marks in this section are awarded for criticism or endorsement of aspects of the material presented (design, method and results). Do not try and comment in detail on the statistics unless you are comfortable in this area. Simple appreciation of statistical significance is expected and the attached notes are designed to help with this.
Critical appraisal of a study of therapy is perhaps the likeliest to arise. When appraising such a study the criteria to address are:
To answer this consider the following questions:
Was patient assignment randomised and was randomisation concealed?
Randomisation is the best mechanism for ensuring that the groups, at the outset, are identical in risk of the events you are hoping to prevent with the treatment. Randomisation methods should be described, and ideally not enacted by clinicians who might exert some conscious or unconscious bias. Usually this leads to a false positive result as patients with a more favourable prognosis are given the experimental treatment.
Were all patients entering the trial accounted for at the end? Were they analysed in the groups to which randomised?
To be sure of a trial's conclusion it should be possible to assign all lost patients from each group the worst case outcomes and still be possible to support the original conclusion. With more than a 20% drop out This is unlikely.
Since anything that happens to subjects after randomisation can affect the measured outcomes, it is important that all are analysed in the groups to which they were randomised (intention to treat analysis)
Were patients and clinicians blind?
Where it is possible this is desirable as it prevents clinicians adding co-interventions or being influenced in their assessment of patients.
Aside from the experimental intervention were the groups treated equally?
Were the groups similar at the start of the trial?
Relative risk reduction - RRR
This is a measure of clinical significance and is calculated from the formula:
(CER - EER) / CER.
Where CER = control event rate (those on placebo), EER = experimental event rate (those on the active treatment).
Absolute risk reduction - ARR and Number needed to treat - NNT
RRR is a percentage, thus although it provides an idea of the proportional benefit from using a treatment as opposed to a placebo or other intervention, it fails to inform about the absolute degree of benefit (90% of hardly anything is still not much!).To get around this problem the ARR is used. This is the absolute risk reduction, and is simply the difference between event rates in the experimental and control groups:
ARR= CER - EER
The problem with this is that it is usually a number less than 1 and non-mathematicians are allergic to these in the main.
Happily a mathematician realised that an ARR of, say .46 means that a hundred people would need to have the experimental treatment to avoid 46 unwanted events: therefore 100/46 people would need to be treated to prevent one such event. This is the NNT (number needed to treat).
Thus NNT = 1 / ARR
With some euphoria it is evident that NNTs are whole numbers (or at least numbers greater than one. Perhaps an example will help.
The question is whether aspirin treatment was equally beneficial to heparin therapy in unstable angina arose in the practice. A search revealed an article which seemed to cover the issue:
Article: Theroux. P et al. Circulation 1994, Aug:90(2): 1107 Double blind RCT Aspirin 325 mgs bd vs. heparin perfusion, 484 patients.
Outcome:
- MI occurred in 2 (O.8%) of heparin group
- MI occurred in 9 (3.7%) of aspirin group
Treatment lasted over 5.7 (+/- 3.3) days
CER EER RRR ARR NNT 3.7% 0.8% 78.3% 2.9% 34
Common sense applies to much of the above as to anything else e.g., studies based on someone s memory of event five years ago is subject to "recall bias" so state it.
Most GP based studies are observational where the natural course of events is observed in matched crowds, as opposed to interventional studies where people who already have a risk factor are followed and matched to a treatment or non-treatment group. The gold standard for trials is one that is randomised and double blind with well-matched controls but this is particularly difficult to achieve in GP. Usually, the intervention is obvious and the matching, if it involves a clinical condition, may involve soft entry and end criteria (better/not better: more/less angina).
Many GP studies use questionnaires and these are always worthy of detailed appraisal. Disparity of numbers questioned in two groups needs comment as does differences in the setting in which the questionnaire is completed. This is particularly important with follow-up (post intervention) questionnaires where the setting is often at home (postal questionnaire) with the original being in the surgery. In addition, questionnaire items may be susceptible to different interpretations. This could be a source of unreliability, and piloting may help this. The questionnaire is not usually shown with written material and if it might have helped, comment accordingly.
Non responders to questionnaires (and to follow-up) or to invitations to interventions (usually at a surgery) are a major worry in studies as they may indicate bias in the "response sets' i.e. the study may just be recruiting and follow-up a self selecting/motivated population whose behaviour is not generalisable even within the study setting. Look at numbers of non-responders and drop outs (they should be stated (there are always some) and see what was done to chase up non-responders to ensure like was being compared with like. More than 20% drop out from a study group would be worrying as regards validity. Questionnaires to groups whose main language is not English can cause obvious difficulties over understanding but also more subtle misinterpretation of the questions.