Statistical tests are used in hypothesis testing. They can be used to:
- Determine whether a predictor variable has a statistically significant relationship with an outcome variable.
- Estimate the difference between two or more groups.
In Harmoni, there are options in the modify menu that allow for statistical testing. The modify menu is available to Creators and Explorers. Viewers can also interact with these features when they have access to dashboards, or stories where the analysis can be zoomed.
Significance testing is available for record/respondent level data and is not available in aggregate data.
In this article
- Significance Testing (SIG)
- Reference for Testing
- Adjusting the Confidence Level
- Low Sample Size
- Effective Base
1. Significance Testing (SIG)
Statistical significant differences between the results for one group and a reference group (often the total sample).
The option SIG enables significant testing.
The SIG option becomes available when nothing is selected.
- SIG testing only switches on when you create an analysis with a cross tab.
- SIG testing remains off in an analysis with a single axis; the only exception with a single axis is when you use a time axis as a filter.
SIG testing on as a default.
The personalization keyKeys that can be applied at the user, site (all users), or company (all users) level. for significance difference testing can be set to ON (True) as a default. Contact support@infotools.com for more information.Project Owners can also turn significance difference testing on at a project level using the Settings menu. Learn more here.
When significance difference testing is on by default, SIG testing will only switch on when you create an analysis with a cross tab.
SIG testing remains off in an analysis with a single axis. There are a couple of exceptions:
- With a single axis and a time axis as a filter.
- When you have intentionally switched SIG on, it will remain on for your session or until you reset.
Green and Red Arrows
In Harmoni, significant differences are represented using green and red arrows.
- Green - significantly higher compared to the reference.
- Red - significantly lower compared to the reference.
- The confidence level is displayed on the info bar.
When you see a red arrow pointing down in a cell, it means that the percentages in the cell is significantly lower than the reference it is being compared to. And of course, the reference is therefore significantly higher.
Significance differences use the red arrows to indicate a significantly lower result as it is often the case where a lower percentages means a 'worse' result, for example when comparing brand usage.
Statistical Tests for Significance
The test applied depends on the variables included in your analysis.
| Statistical Test | Variable Type | Description |
Chi-squared test |
Discrete variables |
Discrete variables contain labeled responses that have a yes, no, or missing response for each record. The statistical significance of the difference between two discrete variables is calculated using both counts and bases. The statistical test used is the chi-squared test. |
t-test |
Continuous variables | Continuous variables are called 'measures' or 'values'. Measures in an analysis may either be sums (i.e. totals) or averages. For comparisons of continuous variables (measures or averages), the statistical test used to determine significant differences is a t-test. The test is applied using numerical values. |
SIG testing for discrete variables is performed on percentages. Counts and unweighted counts inherit the significance results from percentages.
Important:
If Harmoni's default statistical testing has been changed to consider overlapping groups, t-tests are used for both discrete and continuous variables.
Learn more about Significance Testing with Overlapping Groups.
2. Reference for Testing (REF)
You can select a specific item (i.e. column or row) as the reference for testing. Once selected, the reference is fixed to that item; if you flip or reorder your analysis, the reference will remain fixed.
The REF option allows you to select the reference for testing. For the REF option to become available, you need to select a column or row.
- The selected row or column becomes the reference for significant difference calculations.
- When a time-based axis is a part of your analysis you have the ability to set significance testing to be based on a rolling period. Learn more.
- The comparison reference is displayed on the info bar.
Previous Period Referencing
When a time-based axis is a part of your analysis (generally columns/across) you have the ability to set significance testing to be based on a rolling period. This could be the previous period or even the previous 4th period.
The offset determines how far away from the test cell the cell to be compared with is, for example, an offset of 1 is the cell next on the left of the cell being compared.
Use the drop-down menu under the REF icon, to adjust the reference period. The comparison reference is displayed on the info bar.
Here we can see the REF option in an analysis with 3MMT in the across dimension. The offset is the 3rd previous 3MMT, so Dec 21 3MMT is compared to Oct 21 3MMT.
Time Axis in the Across Dimension
In the example below, the time axis, Year, is in the across dimension. The default offset is 1, the Previous Year. By accessing the REF drop-down menu, this is changed to 2 so that the reference is 2nd Previous Year. At the bottom of the analysis, open the i-bar to display the reference information.
Time Axis as a Filter
In the example below, Year is dragged to the filter drop zone and the reference changes to Year. When a time axis is dragged to the filter, Harmoni defaults to using the time axis as the reference for significant differences. The offset defaults to 1 and can be changed as demonstrated above.
In this example, the offset is 1 and each cell displayed for the most recent Year (2020) is compared to the previous Year (2019) and so the REF label in the i-bar is Year: 2019. When you change the filter to display 2019 in the analysis, each cell is then compared to 2018 and so on.
Include or Exclude Hidden Items
When a time-based axis is a part of your analysis (generally columns/across) hidden items are included in the reference set. You may choose to exclude by unticking the option "Include Hidden Items" from the REF drop box.
This is a useful option when, for example, there are missing time periods in your survey, but you still want to compare vs. the most recent period.
Analysis with a Single-Time Axis
When you only have a time axis in your analysis, the default reference for significance testing Total. As soon as you create a cross-tab analysis or add a time axis as a filter you get the option of previous period time referencing.
3. Adjusting the Confidence Level
Harmoni typically applies a default confidence level of 95% (this can be changed using a personalization key) for determining significant differences. As a user, you can override this in the modify menu. Type the statistical confidence level you want and press enter.
The confidence level used for the significant difference calculation can be set independently for each saved view.
During your session, your setting preferences are retained. Site settings can be controlled using personalization keysKeys that can be applied at the user, site (all users), or company (all users) level.. Contact support@infotools.com for more information.
4. Low Sample Size
When the results in the analysis are based on small sub-samples the low sample size feature hides or fades values in the table when the unweighted value of the denominator (base) used for calculations is equal or lower than a specified level. The unweighted value relates to respondents, not responses.
The LOW option, allows you to easily identify when the denominator (base) goes below certain thresholds. The LOW option becomes available when nothing is selected.
| Low Sample Option | Default Threshold | Description |
Low sample
|
200 or under | Transparency is applied to the results to match the warning threshold. |
Too low sample
|
50 or under | An asterisk (*) is shown instead of the result. |
Expanding the LOW icon allows you to adjust the thresholds for low sample size calculation. During your session, your setting preferences are retained. Low sample values are included in the information panel at the base of the screen.
Low Sample Size Calculation Rules
For most calculation types, the unweighted base (uB), is used for low sample size calculation. Harmoni considers the share number of people in the cell and the total/base in the relevant column.
For AVG and ∑ , Harmoni considers the summation of all values and therefore uses the u123.
Low sample size considers respondents, not responses.
| Calculation Type | Determine LOW using |
Compare both self cell and reference cell |
| AVG | u123 | |
| 123 | uB | |
| IND | uB | Yes |
| IND Σ | uB | Yes |
| u123 | uB | |
| Σ u123 | uB | |
| uB | NA - Always show | |
| uB Σ | uB | |
| Σ | u123 | |
| % | uB | |
| % Σ | uB | |
| uΣ | u123 | |
| B | NA - Always show | |
| B Σ | uB | |
| AVG Diff | u123 | Yes |
| 123 Diff | uB | Yes |
| IND Diff | uB | Yes |
| IND Σ Diff | uB | Yes |
| u123 Diff | uB | Yes |
| Σ u123 Diff | uB | Yes |
| uB Diff | NA - Always show | |
| uB Σ Diff | uB | Yes |
| Σ Diff | u123 | Yes |
| % Diff | uB | Yes |
| % Σ Diff | uB | Yes |
| uΣ Diff | u123 | Yes |
| B Diff | uB | Yes |
| B Σ Diff | uB | Yes |
5. Effective Base
In statistics, effective base can be used as a safeguard against making statistical conclusions from a sample that experiences extreme weighting. Using the effective base is considered a more conservative approach, but it provides accurate statistical results for weighted data.
Learn more about Effective Base.
Where to from here?
Learn more about statistical tests in Harmoni