Harmoni can calculate significance testing with overlapping groups. This option is available via a personalization key and overrides the standard significance testing which is set by default.
In this article
1. Overlap Significance Testing
Harmoni can add an overlap correction factor to the standard significance testing. It is available for both multi-reference and single-reference significance testing.
This method does not assume independent sampling between the reference cell and the test cell, instead it considers respondents who can be in both the test cell and the reference cell and adjusts the result accordingly.
In some cases, the overlap respondent count between the reference cell and the test cell may be 100%. An example of this is if the reference cell is the total cell. In other cases, the overlap respondent count may be 0%. An example of this is if the reference cell and the test cell are both in one single response variable. Otherwise, the overlap respondent count is calculated and considered.
T-tests are used for both discrete and continuous variables with the overlap method.
2. Verifying Results
Verifying Overlap significance difference results is not possible in Harmoni directly. If you need assistance, please contact Support@infotools.com
3. Effective Base
In statistics, effective base is used as a safeguard against making statistical conclusions from a sample that has been drastically adjusted using weights to match target values. Using the effective base is considered a more conservative approach, but it provides accurate statistical results for weighted data.
The effective base indicates how much statistical power is lost by weighting. The closer the effective base is to the unweighted base, the better the weighting is.
Effective base = (sum of weighted base) squared divided by the sum of the squared weighted base.
If requested, the effective base is the default base for calculating significant differences.
Learn more about effective base.
Where to from here?
Learn more about statistical significance in Harmoni