Statistical tests are used in hypothesis testing. They can be used to:
- Determine whether a predictor variable has a statistically significant relationship with an outcome variable.
- Estimate the difference between two or more groups.
Statistical tests assume a null hypothesis of no relationship or no difference between groups. Then they determine whether the observed data fall outside of the range of values predicted by the null hypothesis.
In Harmoni, the options under tests, in the modify menu, allow for statistical testing. The modify menu is available to Creators and Explorers. Viewers can also interact with these features when they have access to dashboards, or stories where the analysis can be zoomed.
In this article
- Significance test
- Reference for testing
- Adjusting the confidence level
- Low sample size
- Effective base
1. Significance Test
Statistical significant differences between the results for one group and a reference group (often the total sample). The significance level is the likelihood that the difference is not due purely to chance. If the confidence level is set to 95%, the change is said to be statistically significant, as there’s only a 5% chance that the difference is due purely to chance.
The option SIG under test, enables significant testing. The SIG option becomes available when nothing is selected. However, SIG testing will only switch on when you create an analysis with a cross tab. SIG testing remains off in an analysis with a single axis; the only exception with a single axis is when you use a time axis as a filter.
The personalization key Keys that can be applied at the user, site (all users), or company (all users) level. Contact email@example.com for more information. for significance difference testing can be set to ON (True) as a default.
When significance difference testing is on by default, SIG testing will only switch on when you create an analysis with a cross tab.
SIG testing remains off in an analysis with a single axis with the following exceptions:
- With a single axis and a time axis as a filter.
- When you have intentionally switched SIG on, it will remain on for your session or until you reset.
An analysis can contain discrete or continuous variables.
|Statistical Test||Variable Type||Description|
|Discrete variables||Discrete variables contain labeled responses that have a yes, no, or missing response for each record. The statistical significance of the difference between two discrete variables is calculated using both counts, and bases.
The statistical test used is the chi-squared test.
Continuous variables are called 'measures' or 'values'. Measures in an analysis may either be sums (i.e. totals) or averages. For comparisons of continuous variables (measures or averages), the statistical test used to determine significant differences is a t-test. The test is applied using numerical values.
In Harmoni, significant differences are represented using green and red arrows.
- Green - significantly higher compared to the reference.
- Red - significantly lower compared to the reference.
- The confidence level is displayed on the info bar.
2. Reference for testing
You can select a specific item (i.e. column or row) as the reference for testing. Once selected, the reference is fixed to that item; if you flip or reorder your analysis, the reference will email fixed.
The option REF under tests, allows you to select the reference for testing. For the REF option to become available, you need to select a column or row.
- The selected row or column becomes the reference for significant difference calculations.
- When a time-based axis is a part of your analysis you have the ability to set significance testing to be based on a rolling period. Learn more.
- The comparison reference is displayed on the info bar.
Previous period referencing
When a time-based axis is a part of your analysis (generally columns/across) you have the ability to set significance testing to be based on a rolling period. This could be the previous period or even the previous 4th period.
The "offset" determines how far away from the test cell the cell to be compared with is, for example, an offset of 1 is the cell next on the left of the cell being compared.
Expanding the REF icon under tests, allows you to adjust the reference period. The comparison reference is displayed on the info bar.
3. Adjusting the confidence level
Harmoni applies an automatic (and generally industry default) confidence level of 95% for significant difference calculation. As a user, you can override this via the modify menu.
Expanding the SIG icon under tests, allows you to change the confidence level.
The confidence level used for the significant difference calculation can be set independently for each saved view.
During your session, your setting preferences are retained. Site settings can be controlled using personalization keys.Keys that can be applied at the user, site (all users), or company (all users) level. Contact firstname.lastname@example.org for more information.
4. Low sample size
When the percentages shown in an analysis are based on small sub-samples, the results may be unreliable. The Low sample size feature hides or fades values in the table when the unweighted value of the denominator (base) used for percentage calculations is equal or lower than a specified level.
The option LOW under test, allows you to easily identify when the unweighted base goes below certain thresholds. The LOW option becomes available when nothing is selected.
|Low Sample Option||Default Threshold||Description|
|Unweighted base of 200 respondents or under||Transparency is applied to the results to match the warning threshold.|
Too low sample
|Unweighted base of 50 respondents or under||
An asterisk (*) is shown instead of the result.
Expanding the LOW icon under tests, allows you to adjust the thresholds for low sample size calculation. During your session, your setting preferences are retained.
Site settings can be controlled using personalization keys.Keys that can be applied at the user, site (all users), or company (all users) level. Contact email@example.com for more information.
5. Effective base
Traditionally the test for significance uses the unweighted base in the calculation. This can lead to an overestimate of the significant difference between the two data cells when extreme weighting has been applied. By using the effective base instead of the unweighted base in the calculation we can adjust for this overestimate.
The effective base can be set through personalization keys.Keys that can be applied at the user, site (all users), or company (all users) level. Contact firstname.lastname@example.org for more information.
Where to from here?
Learn more about statistical tests in Harmoni