Skip to main content

Significance Testing Against Norms

  • Updated
    Tags:
  • Creator
  • Explorer
  • Viewer

 

 

In this article

  1. Norm Flags
  2. Norms In Analysis
  3. Stat Testing using Norms

 

As a Creator (Project Owner), you can flag a measure or measure group as a target (Norm), against which significant differences can be calculated. The option is available in the design menu under the flags section.

 

Norms_Analysis.jpg

 

1. Norm Flags

There are three Norm flags available in Harmoni. The flag to use depends on the nature of your target. 

  • Proportional Norm - use this flag if your target is between the values of 0 and 1. This can also be referred to as Ratio. For example, Brand Link Ratio.
  • Percentage Norm - use this flag for percentage-related values 0-100%, for example, Campaign Recall.
  • Value Norm - use this flag for exact values, for example, sales and profits.

 

Flags.jpg

 

2. Norms In Analysis

In an analysis with a Norm, Harmoni will use the Norm Significance Test rather than the standard chi-squared test/t-test calculations for single reference difference calculations.

Significant differences are represented using green Green.PNG and red Red.PNG arrows. 

 

Norms_Analysis_2.jpg

 

Harmoni defaults to the standard significance test calculations (SIG) between regular cells if a standard column/row is set as a reference. The Norm (SIG) calculation is ONLY used between the reference cells and the test Norm cells.

 

Norms_Analysis_3.jpg

 

In an analysis with Multiple reference significant differences (M-SIG), the Norm significance test will not apply and will be excluded from calculations at this stage.

 

3. Statistical Testing using Norms

When a measure or measure group is flagged as a Norm item, Harmoni performs statistical testing against this target. Harmoni uses the flag to determine the significance test to use.

 

Proportions/Percentages against a Norm

Harmoni uses one sample proportion test.

One-sample proportion test is used to test whether a sample proportion is statistically different from a hypothesized proportion. The hypothesized proportion is often called a norm or a target and is a given constant value.

For proportional norm, we add Yate’s Continuity Correction. In short, Yate's Continuity Correction adjusts the number passed to the NORM significance test a little bit, to improve the accuracy of the test. However, it makes the calculation more conservative and leads to slightly reduced statistical power

 

Value against a Norm

Harmoni uses a one-sample t-test.

The one-sample t-test assesses whether the mean of a sample is statistically different from a hypothesized value, for example, a specified norm or target. You would use this test for measures or averages.

 

Bases Used for Testing

  • If the Effective Base is enabled, the effective base from the Total row/column is used as the sample size for all Norm sig calculations (Ratio, Percentage, Value).
  • If the Effective Base is OFF, the Unweighted Count (U123) from the Total row/column is used as the sample size for all Norm sig calculations (Ratio, Percentage, Value).

Learn more about effective base.

 

Where to from here?

Learn more about statistical testing:

Up Arrow.png

Related articles