In Harmoni, you can calculate statistical significant differences between the results for one group and a reference group, often the total sample **(SIG)**, or you can perform multiple reference significance testing **(M-SIG)**.

- Significant differences
**(SIG)**are represented using green and red arrows. Learn more about Significance Test (SIG DIFF)

- Multiple reference significant differences
**(M-SIG)**are represented using the 26 letters in the English alphabet. Learn more about multiple reference significance test (M-SIG)

The multiple reference significance test **(M-SIG)** makes several simultaneous pairwise comparisons and therefore needs to adjust for the inflation in false-positive rates (known as the family-wise error rate). Harmoni automatically corrects the family-wise error rate using a technique known as the Holm method.

Standard **SIG** testing, are simple comparisons between two proportions and do not apply any multiple comparisons correction.

Therefore you may notice some discrepancies when comparing the two techniques.

**1. Multiple comparison significance testing statistics**

When doing multiple comparison significance testing on **proportions (percentages), ** Harmoni performs a **pairwise proportions test**.

A pairwise proportions test is the equivalent of a **chi-square test** between the proportions in each pair of cells in the row (or column).

The process of conducting multiple statistical tests introduces the **multiple testing problem**, which must be corrected. Essentially, we must correct for the fact that as we make more and more comparisons, we increase the chances that we’ll find false positives, i.e., detect a significant difference when one does not exist.

- When the confidence level of 95% is set, we are saying that, if in reality, there is no difference in the values, we will accept a probability of 1 in 20 (or 0.05) of getting a false positive, i.e., concluding a difference exists when it does not.
- When we do two tests, the chance of getting a false positive from either test (if there are no actual differences) increases to (1 - 0.95*0.95) = 0.0975.
- As we perform more and more tests, this chance increases. This is known as the
**family-wise error rate**.

Many techniques have been developed to prevent the inflation of false-positive rates that occur with multiple statistical tests. By default, when Harmoni performs multiple comparison significance testing on proportions, it applies the **Holm method** to correct the family-wise error rate.

The Holm method is a commonly applied technique that is uniformly more powerful than some others such as the Bonferroni correction. (Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.)

If a new variable is added to the analysis, for example, a new time period is added with a new update, multi-ref significant differences may change in previously saved analyses (stories). Because a new variable has been added to the calculation, different results may occur when all the comparisons between the variables are recalculated.

**Multiple reference significant differences on Measures**

For multiple comparison significance testing on **measures** (real-valued numeric variables), Harmoni applies **Tukey's HSD test**. Tukey's HSD tests all pairwise comparisons among values and is based on the studentized range distribution.

Tukey's test is based on a formula very similar to that of the **t-test**. Tukey's test is essentially a t-test, except that it corrects for family-wise error rate.

As with proportions, when multiple comparisons are made, the probability of making a type I error (false positive) increases. Tukey's test corrects for that and is thus more suitable for multiple comparisons than performing several t-tests.

As with comparisons made in significance testing on proportions, multi-ref significant differences may change when new measures are added to the analysis.

**Where to from here?**

Learn more about statistics.