In Harmoni, you can calculate statistical significant differences against a single reference group using SIG, or you can perform multiple reference significance testing using MRSD.
- Significant differences (SIG) are represented using green
and red 
arrows. Learn more about Significance Test (SIG DIFF)
- Multiple reference significant differences (MRSD) are represented using the 26 letters in the English alphabet.
In this article
- Significant difference - SIG and MRSD
- Multi-Reference Significance Testing Pop-Up
- Reference letter colors and cases
- Multi-Reference Significance Testing Options
- Multi-Reference Significance export to Excel and PPT
1. Significant Difference - SIG and MRSD
In Harmoni, options for significance testing (SIG and MRSD) are available under tests in the modify menu. The modify menu is available to Creators and Explorers. Viewers can also interact with these features when they have access to dashboards, or stories where the analysis can be zoomed.
SIG and MRSD can be viewed alongside or you can choose one at a time.
Harmoni typically applies a default confidence level of 95% (this can be changed using a personalization key) for determining significant differences. As a user, you can override this via the modify menu by expanding the SIG icon under tests, which allows you to change the confidence level.
The confidence level set applies to both SIG and MRSD simultaneously.
a. Significant differences between one group and a reference group (SIG)
In Harmoni, significant differences (SIG) are represented using green and red arrows.
- Green - significantly higher compared to the reference.
- Red - significantly lower compared to the reference.
- The confidence level is displayed on the info bar.
When you see a red arrow pointing down in a cell, it means that the number in the cell is significantly lower than the reference number it is being compared to. And of course, the reference number is, therefore, significantly higher.
Learn more about Significance Test (SIG DIFF)
b. Significant differences between one group and multiple groups (MRSD)
Multiple testing refers to any instance that involves the simultaneous testing of several
hypotheses.
In Harmoni, multiple reference significant differences (MRSD) are represented using the 26 letters in the English alphabet.
- The first column or row in the dimension chosen as the reference for MRSD is assigned the letter “A”. It excludes the total row/column.
- If the order of rows/columns change (due to ranking or manual reordering, for example), letters are reassigned so they are always in alphabetical order.
- Items hidden from the analysis can be included or excluded from the reference set.
- Only the items dragged into the analysis can be included in the reference set. This is useful when you want to test against a subset of brands by only including that subset in the analysis.
- Each element in the SigDiff dimension is assigned a letter up to a maximum of 26. Elements in the 27th position and beyond are still tested, but the significance of their differences is not displayed in the table, only in the pop-up.
2. Multi-Reference Significance Testing Pop-Up
When calculating multiple reference significant differences, you can select any cell in the analysis to display a hover pop-up menu in which you will find the following information:
- The axis/element name of the reference cell and the value of the cell you have selected.
- Items that are significantly higher than or lower than the value of the cell you have selected.
- Confidence level at which the tests have been performed.
To display the pop-up, click on the dots on the right-hand side within the cell.
2. Multi-Reference letter color and cases
There are four different settings that can be set using personalization keys. Contact support@infotools.com for more information.
Let's look at the behavior of the different keys, with this example where:
- Cell A (Australia) - 20% of visitors from Australia are aged 45-54 years old.
- Cell B (United Kingdom) - 13% of visitors from UK are aged 45-54 years old.
- Cell E (China) - 27% of visitors from China are aged 45-54 years old.
a. Options when cell is higher - "Color Green when I am higher"
These options use a green font when the reference cell is significantly higher than other cells.
|
Option |
B<A B lower than A |
E>A E higher than A |
|
a. Color green when I am higher (A) |
||
|
Option 1- Default setting
|
b |
E |
|
Option 2
|
B |
e |
b. Options when cell is lower - "Color Green when they are higher"
These options use a green font when the reference cell is significantly lower than other cells.
|
Option |
B<A B lower than A |
E>A E higher than A |
|
b. Color green when they are higher |
||
|
Option 3
|
b |
E |
|
Option 4
|
B |
e |
c. No font color applied
|
Option |
B<A B lower than A |
E>A E higher than A |
|
c. No color applied |
||
|
Option 5
|
b |
E |
|
Option 6
|
B |
e |
3. Multi-Reference Sig Testing Options
a. Multi Reference Across or Down
Multiple comparison significance testing can compare cells on the across (columns) or on the down (rows).
You can select the direction by expanding the multi-ref dropbox (MRSD) and selecting the relevant option (i.e..across or down).
b. Show High/Low Significant Differences
When doing multiple comparison significance testing, you may choose to only show which cells are significantly higher or to only show those that are lower.
You can select by expanding the multi-ref dropbox (MRSD) and selecting the relevant options:
- Show SigDiff High
- Show SigDiff Low
c. Include hidden items in sig test calculations
When doing multiple comparison significance testing, items hidden from the analysis are included in the reference set. You may choose to exclude by unticking the option "Include Hidden Items" from the multi-ref dropbox.
b. Show Color
When doing multiple comparison significance testing, you may choose to exclude the red and green coloring. Unticking the option "Show SigDiff Color" from the multi-ref dropbox removes the coloring.
The upper and lower case still serves as a good indicator of significant differences and also the information pop-up can give more context on the results.
5. Multi-Reference Significance export to Excel and PPT
When you have an analysis with MRSD applied, you can export your table to PowerPoint and Excel. The intention of these export functions is to provide slides and tables that are close to the Harmoni view being exported, but not identical.
