In Harmoni, you can calculate statistical significant differences against a single reference group using SIG, or you can perform multiple reference significance testing using M-SIG.
- Significant differences (SIG) are represented using green
and red
arrows. Learn more about Significance Test (SIG DIFF)
- Multiple reference significant differences (M-SIG) are represented using the 26 letters in the English alphabet.
In this article
- Significant difference - SIG and M-SIG
- Multi-Reference Significance Testing Pop-Up
- Reference letter colors and cases
- Multi-Reference Significance Testing Options
- Multi-Reference Significance on charts
- Multi-Reference Significance export to Excel and PPT
1. Significant Difference - SIG and M-SIG
In Harmoni, options for significance testing (SIG and M-SIG) are available under tests in the modify menu. The modify menu is available to Creators and Explorers. Viewers can also interact with these features when they have access to dashboards, or stories where the analysis can be zoomed.
SIG and M-SIG can be viewed alongside or you can choose one at a time.
Harmoni typically applies a default confidence level of 95% (this can be changed using a personalization key) for determining significant differences. As a user, you can override this via the modify menu.
The confidence level applies to both SIG and M-SIG simultaneously.
a. Significant differences between one group and a reference group (SIG)
In Harmoni, significant differences (SIG) are represented using green and red arrows.
- Green - significantly higher compared to the reference.
- Red - significantly lower compared to the reference.
- The confidence level is displayed on the info bar.
When you see a red arrow pointing down in a cell, it means that the number in the cell is significantly lower than the reference number it is being compared to. The reference number is, therefore, significantly higher.
Learn more about Significance Test (SIG DIFF)
b. Significant differences between one group and multiple groups (M-SIG)
Multiple testing refers to any instance that involves the simultaneous testing of several
hypotheses.
In Harmoni, multiple reference significant differences (M-SIG) are represented using the 26 letters in the English alphabet.
- The first column or row in the dimension chosen as the reference for M-SIG is assigned the letter “A”.
- If the order of rows/columns change (due to ranking or manual reordering, for example), letters are reassigned so they are always in alphabetical order.
- Items hidden from the analysis can be included or excluded from the reference set.
- Total row/column can be included or excluded from the reference set.
- Only the items dragged into the analysis can be included in the reference set. This is useful when you want to test against a subset of brands by only including that subset in the analysis.
- Any new variables added to the analysis may change the significance results. For example, additional brands added into a previously selected subset mean that more parameters are passed into the algorithm, which can create different results. Another example is adding a new time period to an analysis when a new data update is completed.
- Each element in the SigDiff dimension is assigned a letter up to a maximum of 26. Elements in the 27th position and beyond are still tested, but the significance of their differences is not displayed in the table, only in the pop-up.
2. Multi-Reference Significance Testing Pop-Up
When calculating multiple reference significant differences, you can select any relevant cell in the analysis to display a hover pop-up menu in which you will find the following information:
- The axis/element name of the reference cell and the value of the cell you have selected.
- Items that are significantly higher than or lower than the value of the cell you have selected.
- Confidence level at which the tests have been performed.
To display the pop-up, hover over the cell and click the three dots on the left side.
3. Multi-Reference letter color and cases
There are four different settings that can be set using personalization keys. Contact support@infotools.com for more information.
Let's look at the behavior of the different keys, with this example where:
- Cell A (Australia) - 20% of visitors from Australia are aged 45-54 years old.
- Cell B (United Kingdom) - 13% of visitors from UK are aged 45-54 years old.
- Cell E (China) - 27% of visitors from China are aged 45-54 years old.
a. Options when reference cell is higher - "Color Green when I am higher than others"
These options use a green font when the reference cell is significantly higher than other cells.
|
Option |
B<A B lower than A |
E>A E higher than A |
|
Color green when I (cell A) am higher.
|
||
|
Option 1 - Default
|
B |
e |
|
Option 2
|
b |
E |
b. Options when reference cell is lower - "Color Green when others are higher"
These options use a green font when the reference cell is significantly lower than other cells.
|
Option |
B<A B lower than A |
E>A E higher than A |
|
Color green when they are higher than me (Cell A)
|
||
|
Option 3
|
B |
e |
|
Option 4
|
b |
E |
c. No font color applied
|
Option |
B<A B lower than A |
E>A E higher than A |
|
c. No color applied |
||
|
Option 5
|
b |
E |
|
Option 6
|
B |
e |
4. Multi-Reference Sig Testing Options
a. Multi Reference Across or Down
Multiple comparison significance testing can compare cells on the across (columns) or on the down (rows).
You can select the direction by expanding the multi-ref drop box (M-SIG) and selecting the relevant option (i.e., across or down).
b. Multi Reference on Filter
Multiple comparison significance testing can compare cells based on a filter item.
In an analysis where a filter variable is included, you can select the multi-reference on filter option by expanding the multi-ref dropbox (M-SIG).
- Elements in the filter axis are assigned the ABC reference letters.
- When more than one filter is included in the analysis, the first axis in the filter is assigned the reference letters.
- You can choose a different filter axis using the select menu and choosing the M-SIG option next to the axis name.
c. Show High/Low Significant Differences
When doing multiple comparison significance testing, you can choose to show only the cells that are significantly higher or only those that are lower.
You can select by expanding the multi-ref drop box (M-SIG) and selecting the relevant options:
- Show When This Cell Higher
- Show When This Cell Lower
d. Include hidden items in sig test calculations
When doing multiple comparison significance testing, items hidden from the analysis are included in the reference set. You may choose to exclude by unticking the option "Include Hidden Items" from the multi-ref drop box.
e. Include Total in sig test calculations
When doing multiple comparison significance testing, you may choose to include Totals by ticking the option Include Total from the multi-ref drop box.
This makes the Total column or row in the dimension chosen the first reference for M-SIG and it is assigned the letter “A”.
Total rows with Measures/Averages
When creating an analysis with measures, Harmoni will not automatically include M-SIG testing for the averages in the total row. You need to tick the Include Total option to add M-SIG to the total row.
You can select a single measure or a measure group and with Include Total selected, the significant difference letters are added. Note that single ref statistical differences are automatically applied to the total row.
f. Reference Within a Group
In an analysis where you have more than one axis in a group, you have the option to apply multiple comparison significance testing for each axis independently.
This option makes the first column or row for each axis in the group the first reference for M-SIG and it is assigned the letter “A”.
g. Show Color
When doing multiple comparison significance testing, you may choose to exclude the red and green coloring. Unticking the option "Show SigDiff Color" from the multi-ref dropbox removes the coloring.
The upper and lower case still serves as a good indicator of significant differences and also the information pop-up can give more context on the results.
5. Multi-Reference Significance on charts
You can choose to visualize an analysis where M-SIG is applied. M-SIG is supported in all charts except single number and space graph.
In the charts, the letters are not displayed; however, you can display the hover pop-up menu in which you will find the following information:
- The axis/element name of the reference cell and the value of the cell you have selected.
- Items that are significantly higher than or lower than the value of the cell you have selected.
- Confidence level at which the tests have been performed.
To display the pop-up, click on the three dots next to the relevant data point.
6. Multi-Reference Significance export to Excel and PPT
You can export your analysis to PowerPoint and Excel and M-SIG is included in the chart or table. The
M-SIG letters are included in the export. The intention of these export functions is to provide slides and tables that are close to the Harmoni view being exported, but not identical.
Where to from here?
Learn more about significance testing.