Append allows you to add new variables to respondents or cases within a project when information of common respondents is captured in separate data sources.
For example, one source contains the main survey information, and another source includes the weighting variables and you need these to link together.
- Based on the number and order of records, or
- Based on unique identifiers.
When using append Harmoni has two options available.
- You can append a file with the same respondents as the primary data file, or
- You can append data that may contain, for example, information about the same respondents' drinking occasions, and this is referred to as multi-level data.
Harmoni's default setting is to append files with the same respondents to the primary file, but if the setting has been turned on for multi-level data, you also need to take into account what each record in each of the data files represents (Unit of Count).What a record in the data represents, whether it be a respondent (person), or an occasion, etc. If you need to append multi-level data, where one of the files contains information about respondents and a separate file contains information about their consumption occasions, please refer to the Multi-Level Data section to learn more about this.
Use append to add new variables into a project when respondents are common across data sources.
In this article
- Append Data Sources
- Append Data Wizard
- Append Data - Multi-Level (Coming Soon)
- Replacing or removing appended data sources
1. Append Data
You can append data sources when you create a project for the first time or append to a project that already contains the primary source. Regardless, the first step is to upload or connect to the data sources that are required in your project. Currently, there is no known limit on the number of secondary sources that you can append.
You need to identify the primary or parent source and then select the link option. Selecting the link option opens the append data wizard.
In the example below, there is an existing project which contains the primary source with the main survey information, and we want to append a secondary source that includes the weights.
- Select view/add sources.
- Select Add/Remove.
- This will open the sources area and display the data sources linked to your project.
- Choose Upload or Connect.
- Locate the secondary data source(s) and click Open.
- Ensure your primary source is selected and then click on the three-dot menu and select Link, then work through the wizard.
2. Append Data Wizard
1. Link - Select sources to append to the primary source
The first step is to identify the secondary sources you want to append to your primary source, the column on the left displays the available secondary sources.
Once selected, the secondary source will move to the right column, under the primary data source.
If the multi-level data setting in Harmoni is turned on, you will also see a Unit of Count text box next to the data source. If the Unit of Count of the secondary source is:
- The same as the primary source, usually respondents, you don't need to enter any information and can proceed to the next step.
- Different from the primary source (e.g., occasions), you need to define the unit of count for each of the sources before proceeding to the next step. Learn more.
In the example below, each record in both the primary source and secondary source represents a respondent; therefore the same unit of count. The Harmoni setting is off so you cannot enter a unit of count.
- Select the secondary source.
- If the multi-level setting is turned on, leave the unit of count blank when appending data that is the same as the primary source.
- If it is not turned on, you will not see the unit of count text box.
- Proceed to the Next step.
2. Define - data types for delimited sources
Harmoni automatically maps variables There are six variable types in Harmoni: Headings, Axes, Grids, Measures, Weights and Verbatims. Learn more about Harmoni Variable Types. when data sources contain an inherent dictionary.Meta-data to guide interpretation Learn more about source dictionaries.
When this is the case, the append wizard displays a message indicating there are no delimited sources and that you can carry on to the next step.
Delimited sources (i.e. Excel-XLXS, comma-delimited-CSV or tab delimited-TXT) do not contain an inherent dictionary. Harmoni determines the best possible match to convert the source variable into Harmoni types, but you have the option to override the automated mapping. You must define the source before you link it to the primary data source. Learn more about defining sources.
3. Match - select how to match records
When appending variables to existing respondents with the same unit of count (e.g. records in all sources represent respondents), you have a couple of options to match the records.
a) Based on the number and order of records
Harmoni will check if the number of records in all selected sources is the same. If they are equal, the sources you've selected will merge with the primary source.
- Respondent number and order must be the same across the primary and secondary sources.
- If the secondary file contains a different number of records, a warning message will appear: "The selected data sources do not have the same number of records." In this case, it is best to append based on unique identifiers.
If you select this option, your sources will append and once the append process is complete you will be taken back to the upload/connect area. Click OK to create the project.
b) Based on unique identifiers
Harmoni will ask you to select the identifier in the primary source as well as the secondary source. Harmoni will then match the records based on this identifier.
- A unique common identifier must exist across the sources.
- The unique identifier:
- Can be either a measure (numeric) or verbatim (text), but it must be the same type across the sources you want to append.
- Must not have blanks, else Harmoni will flag them as duplicates if more than one is found.
- If the identifier is unique and can be matched, the sources will append.
- If the identifier is not unique (i.e., repeated across multiple records), the warning: "We have found duplicated records identifiers in your sources." In this case, you will need to insert unique identifiers in both your primary and secondary sources.
- Orphaned records are ignored.
If you have multiple units of count (i.e. multi-level) you only have the option to match based on unique identifiers. Learn more.
If you select to append using unique identifiers, you need to complete one more step using the wizard!
4. Append - Select a unique identifier
If you select to append based on a unique identifier, you will need to select the identifier in each of the sources. If the identifier is unique and can be matched, the sources will append and once the append process is complete you will be taken back to the upload/connect area. Click OK to create the project.
3. Append Data - Multi-Level
Data may be collected at multiple levels, for example, one of the files contains information about respondents and a separate file contains information about their consumption occasions. If this is the case, you can append data across multiple levels by specifying the unit of count.
Learn more about appending Multi-Level Data.
4. Replacing or removing appended data sources
In Harmoni, you can replace or remove existing data sources from your project. When working with linked data sources, Harmoni will ask you to validate the linking information. Even when the sources have been appended previously, the append process needs to be performed again to ensure that the matching information is still valid.
Overwrite appended sources
You can overwrite primary and secondary sources. after uploading or connecting to your new data source, the append process will initiate. This gives the opportunity to make sure the linking information is still valid.
Unlink and remove appended sources
To remove appended data sources from your project, you first need to unlink them. You can then unselect the source(s) that need to be removed from the project. In between, you need to load the project to action the changes.
A source with an 'orange tinge' indicates the source needs to be defined or that project needs to load for action to take place.
- Select view/add sources.
- Select Add/Remove.
- This opens the sources area and displays a list of the sources in your project.
- First, you need to unlink the data sources.
- Before you can unselect the sources you no longer need, you need to load the project to action the unlinking.
- Click OK to create the project and after it loads, select again view/add sources followed by Add/Remove.
- You can now unselect the sources you no longer want to use. Remove them by unticking directly on the source.
- Make sure that all sources you wish to include in your project are selected.
- When ready, click the OK button.
- Prior to creating and loading your project, Harmon may display a warning when there are any original project items that are no longer present in any of the sources and will, therefore, be removed completely from the project. (Preview removed items). If you choose to Continue, the project will load.
Where to from here?