Tags:
• Project Owner
• Creator

A data source is the starting point for all you do in Harmoni. A source is a collection of data that is either in an imported file or a connected data store.

Harmoni can use sources in any language, including languages requiring multi-byte storage (e.g., traditional Chinese).

Harmoni automatically maps source variables into Harmoni types when data sources contain an inherent dictionary. A dictionary contains meta-data to guide interpretation. Variables in sources without an inherent dictionary need to be defined. 

In this article

1. Source types
   • Imported data sources
   • Direct Connections (APIs)
2. Supported source structures
   • Record Level/Respondent Level
   • Summary Level/Aggregate Level Data

Data Sources - Project Size and Performance

1. Source Types

a. Imported into Harmoni (Upload)

Imported or uploaded files capture data for a moment in time. 

b. Direct Connections to Harmoni (Connect)

Direct connections allow updated data to flow into Harmoni in real-time.

• Achieved using APIs (Application Programming Interface).
• If an API is available in a data collection system, we can potentially develop a direct connection to Harmoni.

Harmoni currently supports direct connections to:

• Tables and Views in SQL Server.
• Projects in Decipher.
• Projects in Voxco.
• Projects in Qualtrics.

2. Source Structures

Harmoni supports both record level/respondent level data and summary or aggregate level data.

a. Record Level/Respondent Level Data

• Each record represents a respondent, a log entry, a sales transaction, etc.
• Analyzing the count of records is relevant - i.e., How many respondents?

b. Summary Level/Aggregate Level Data

• Summed and/or categorized data that can answer research questions about populations or groups of organizations.
• The data has been compiled from record-level data. 
• Analyzing the count of records is irrelevant.
• The analysis only has meaning when a measure is applied, e.g., scores, GRPs, Spend, etc.

The process of importing aggregate data into Harmoni is just the same as for respondent level.  If the data respondent and aggregated data are in the same project you need to ensure that all analyses use appropriate bases, e.g. ensure the base for the survey data doesn’t now include the aggregate record counts. 

Dashboards can include analyses from different projects, so if the data is feeding a separate analysis on the same page, it doesn’t have to be in the same project.  Please note that if you want to apply page level filters, they will need to be available in all analyses on the page, so you would need to create ‘dummy’ filters (i.e. have the same variable and element labels, but include all records in them) to apply the filters to the aggregate data.

Another thing to keep in mind though is that the data drives the possibilities, so if you want to have the filters change the data, you just need to have the results for each of the filters within the aggregate source.

3. Data Sources - Project Size and Performance

In Harmoni the size of the project has no technical limitations. However, the number of records and the number of sources influence the time required for analysis.

We recommend taking into account the below considerations.

Project

The number of variables in the dictionary, and particularly the number of user-created variables, is the biggest influencer on response time. 

We recommend keeping the dictionary to less than 500,000 items. The following are considered to be items in the dictionary:

• Each continuous variable = 1 item, e.g. Number of Shoes owned
• Each category in categorical variables = 1 item, e.g. Gender.Male
• Each cell in a yes/no matrix variable = 1 item, e.g. Brand.Attribute
• Each scale-point in a scale matrix variable = 1 item, e.g. Brand.Attribute.ScalePt
• Each verbatim = 1 item

Analysis

The more items included in the analysis (cube), the longer it will take to run and render it to the screen. For ease of interpretation, we recommend keeping the number of cells in the analysis under 10,000 cells (e.g., 100 rows x 100 columns). However, performance degradation will not occur until well beyond that.

Where to from here?

• Learn more about adding data to your project.
• Best practice - preparing your data before importing or connecting to Harmoni.