Method 1: Standardised Mean Differences, weights based on adjusted standard error
This method is commonly used, including in the SOCIAL systematic review8 and other reviews14-17. This method is simple to use, it utilises information that is generally reported, and it can be performed using standard statistical software. All reported measures of intervention effect are converted into an approximation of the standardised mean difference (SMD) using the formulae in Table 21819.
The formula for a standardised mean difference for a continuous outcome (Table 2) refers to Cohen’s d for ease of calculation. As an alternative Hedges g may be used 19which allows a correction for small sample size.
We applied these methods to the SOCIAL meta-analysis using the inverse of the squared adjusted standard error as weights (inverse variance method 13).
Where a trial reported both a continuous and binary outcome measure with appropriately adjusted standard errors, we utilised the continuous measure but also calculated the SMDs and standard errors using the binary measure to check for anomalies. Note that rules such as this should be pre-specified to avoid post-hoc decisions that could introduce bias.
Method 2: Separate analyses for binary and continuous outcomes, weights based on adjusted standard errors
In this method two separate analyses are produced for the same outcome; one for those that were reported as binary measurements and one for those that were reported as continuous measurements. This method has been used in a number of systematic reviews of health behaviour change4,5,16. This method requires very little data manipulation and adopts a conservative approach to heterogeneity by keeping the two types of outcome separate.
For illustration we performed meta-analysis on the SOCIAL data using odds ratios for binary data and standardised mean differences for continuous data, using the inverse variance method with weights based on adjusted standard errors.
Outcomes were reported as continuous measures on a variety of different scales; therefore they were converted to standardised mean differences as above 20 . If all continuous measures had been measured on the same scale, e.g. mean percentage on a scale of 0 to 100, they could be meta-analysed using means and standard deviations. For meta-analysis of binary data, all summaries need to be converted to the same format (odds ratio, risk ratio or risk difference); it is recommended that this be chosen in advance at the protocol stage to avoid selective reporting. We chose odds ratios 18 here as they have certain desirable mathematical properties; their symmetrical nature would mean that an analysis where the outcome measure is ‘compliance’ or an analysis where the outcome measure is ‘non-compliance’ would lead to identical conclusions.
Sometimes trials report the same outcome measure in both binary and continuous formats – for example in a trial where the desired behaviour is ‘test ordering’; summary data could be reported both in terms of the overall proportion of patients who had a test ordered, and the mean proportion of tests ordered by health care professional. Where a trial has reported an outcome in both binary and continuous formats, we included both measures in the two separate meta-analyses. Note that when using this method, continuous and binary results may not be later combined together as this would lead to double counting of the same participants.