Statistical tests are used in hypothesis testing. They can be used to:
- Determine whether a predictor variable has a statistically significant relationship with an outcome variable.
- Estimate the difference between two or more groups.
In Harmoni, the options under tests, in the modify menu, allow for statistical testing. The modify menu is available to Creators and Explorers. Viewers can also interact with these features when they have access to dashboards, or stories where the analysis can be zoomed.
Significance testing is available for record/respondent level data and is not available in aggregate data.
In this article
- Significance test (Sig Diff)
- Reference for Testing
- Adjusting the Confidence Level
- Low Sample Size
- Effective Base
1. Significance Test (Sig Diff)
Statistical significant differences between the results for one group and a reference group (often the total sample).
The option SIG under test, enables significant testing.
The SIG option becomes available when nothing is selected.
- SIG testing only switches on when you create an analysis with a cross tab.
- SIG testing remains off in an analysis with a single axis; the only exception with a single axis is when you use a time axis as a filter.
SIG testing on as a default.
The personalization key Keys that can be applied at the user, site (all users), or company (all users) level. Contact support@infotools.com for more information. for significance difference testing can be set to ON (True) as a default.
Project Owners can also turn significance difference testing on at a project level using the Settings menu. Learn more here.
When significance difference testing is on by default, SIG testing will only switch on when you create an analysis with a cross tab.
SIG testing remains off in an analysis with a single axis. There are a couple of exceptions:
- With a single axis and a time axis as a filter.
- When you have intentionally switched SIG on, it will remain on for your session or until you reset.
Green and Red Arrows
In Harmoni, significant differences are represented using green and red arrows.
- Green - significantly higher compared to the reference.
- Red - significantly lower compared to the reference.
- The confidence level is displayed on the info bar.
When you see a red arrow pointing down in a cell, it means that the percentages in the cell is significantly lower than the reference it is being compared to. And of course, the reference is therefore significantly higher.
Significance differences use the red arrows to indicate a significantly lower result as it is often the case where a lower percentages means a 'worse' result, for example when comparing brand usage.
Statistical Tests for Significance
The test applied depends on the variables included in your analysis.
| Statistical Test | Variable Type | Description |
Chi-squared test |
Discrete variables |
Discrete variables contain labeled responses that have a yes, no, or missing response for each record. The statistical significance of the difference between two discrete variables is calculated using both counts and bases.
The statistical test used is the chi-squared test. |
t-test |
Continuous variables |
Continuous variables are called 'measures' or 'values'. Measures in an analysis may either be sums (i.e. totals) or averages. For comparisons of continuous variables (measures or averages), the statistical test used to determine significant differences is a t-test. The test is applied using numerical values. |
SIG testing for discrete variables is performed on percentages. Counts and unweighted counts inherit the significance results from percentages.
2. Reference for Testing
You can select a specific item (i.e. column or row) as the reference for testing. Once selected, the reference is fixed to that item; if you flip or reorder your analysis, the reference will remain fixed.
The option REF under tests, allows you to select the reference for testing. For the REF option to become available, you need to select a column or row.
- The selected row or column becomes the reference for significant difference calculations.
- When a time-based axis is a part of your analysis you have the ability to set significance testing to be based on a rolling period. Learn more.
- The comparison reference is displayed on the info bar.
Previous Period Referencing
When a time-based axis is a part of your analysis (generally columns/across) you have the ability to set significance testing to be based on a rolling period. This could be the previous period or even the previous 4th period.
The "offset" determines how far away from the test cell the cell to be compared with is, for example, an offset of 1 is the cell next on the left of the cell being compared.
Expanding the REF icon under tests, allows you to adjust the reference period. The comparison reference is displayed on the info bar.
Include or Exclude Hidden Items
When a time-based axis is a part of your analysis (generally columns/across) hidden items are included in the reference set. You may choose to exclude by unticking the option "Include Hidden Items" from the REF drop box.
This is a useful option when, for example, there are missing time periods in your survey, but you still want to compare vs. the most recent period.
Analysis with a Single-Time Axis
When you only have a time axis in your analysis, Harmoni defaults to the Total for significance testing. As soon as you create a cross-tab analysis or add a time axis as a filter you get the option of previous period time referencing.
3. Adjusting the Confidence Level
Harmoni typically applies a default confidence level of 95% (this can be changed using a personalization key) for determining significant differences. As a user, you can override this in the modify menu. Type the statistical confidence level you want and press enter.
The confidence level used for the significant difference calculation can be set independently for each saved view.
During your session, your setting preferences are retained. Site settings can be controlled using personalization keys.Keys that can be applied at the user, site (all users), or company (all users) level. Contact support@infotools.com for more information.
4. Low Sample Size
When the results in the analysis are based on small sub-samples the low sample size feature hides or fades values in the table when the unweighted value of the denominator (base) used for calculations is equal or lower than a specified level.
The option LOW under test, allows you to easily identify when the denominator (base) goes below certain thresholds. The LOW option becomes available when nothing is selected.
| Low Sample Option | Default Threshold | Description |
Low sample
|
200 or under | Transparency is applied to the results to match the warning threshold. |
Too low sample
|
50 or under |
An asterisk (*) is shown instead of the result. |
Expanding the LOW icon under tests, allows you to adjust the thresholds for low sample size calculation. During your session, your setting preferences are retained. Low sample values are included in the information panel at the base of the screen.
Site settings can be controlled using personalization keys.Keys that can be applied at the user, site (all users), or company (all users) level. Contact support@infotools.com for more information.
Low Sample Size Calculation Rules
For most calculation types, the unweighted base (uB), is used for low sample size calculation. Harmoni considers the share number of people in the cell and the total/base in the relevant column.
For AVG and ∑ , Harmoni considers the summation of all values and therefore uses the u123.
| Calculation Type | Determine LOW using | Compare both self cell and reference cell |
| AVG | u123 | |
| 123 | uB | |
| IND | uB | Yes |
| IND Σ | uB | Yes |
| u123 | uB | |
| Σ u123 | uB | |
| uB | NA - Always show | |
| uB Σ | uB | |
| Σ | u123 | |
| % | uB | |
| % Σ | uB | |
| uΣ | u123 | |
| B | NA - Always show | |
| B Σ | uB | |
| AVG Diff | u123 | Yes |
| 123 Diff | uB | Yes |
| IND Diff | uB | Yes |
| IND Σ Diff | uB | Yes |
| u123 Diff | uB | Yes |
| Σ u123 Diff | uB | Yes |
| uB Diff | NA - Always show | |
| uB Σ Diff | uB | Yes |
| Σ Diff | u123 | Yes |
| % Diff | uB | Yes |
| % Σ Diff | uB | Yes |
| uΣ Diff | u123 | Yes |
| B Diff | uB | Yes |
| B Σ Diff | uB | Yes |
5. Effective Base
Please click here to access information about the Effective Base
Where to from here?
Learn more about statistical tests in Harmoni