Online Help

Help Table of Contents

Duplicate Management

Important: If your installation includes both Convio Luminate Online and Convio Luminate CRM, the Luminate Online duplicate management tool is disabled. Use the Luminate CRM duplicate management tool instead.

The Duplicate Management feature allows you to locate and reconcile duplicate records in both Constituent360 or your external donor database. Common criteria such as name, address, email and phone number can be applied to the bulk merge of constituent records.

Duplicate Management, consists of four features:

Note: Automatic Duplicate Management is disabled by default; to enable it, contact Blackbaud Support. You may want the utility disabled so that it does not run against data imports that you want to check manually. You can import datasets that do not contain duplicates of any online records.

[Back to Top]

Automated Duplicate Management

Automated duplicate matching has an improved duplicate detection algorithm, with added matching logic flexibility that takes into account the most common constituent data entry errors. More duplicate constituent records can correctly be detected and made available for resolution. The intended result is fewer new duplicate records. Two phases underlie this feature:

  • Pre-Duplicate Matching: the high-level determination of existing potential duplicates between Convio Online Marketing and your offline database, such as Convio Common Ground.
  • Duplicate Scoring: a determination of the degree of similarity (with a numeric score of 0 to 100) between the new constituent record and each the potential duplicate identified in the matching.

The above matching and scoring together constitute a two-pass process that is automatically run on new constituent data.

This system is controlled by a site option (PREDUP_CONS_ROUGH_MATCH_ENABLE in Duplicate Management) which Convio Support can enable. By default, this feature is disabled, and exact matching of constituents by last name is instead used.

[Back to Top]

Pre-Duplicate Matching - Constituent Rough Match (Phase 1)

In Phase 1, when a constituent record is initially created, the duplicate checking process evaluates potential duplicates within your Constituent360 database, as well as potential duplicates between Convio and the external system and scores those candidates. Any records that have a match will be recorded and displayed to the Administrator. This phase must be enabled by Convio and is a prerequisite for Phase 2.

When a constituent is added to Convio Online Marketing, their last name is reformatted and divided into component parts based on the following rules:

  • Last names are converted to lowercase characters only
  • Accented characters are converted to non-accented (à becomes a)
  • Any words after a comma are removed
  • Hyphens are replaced with spaces
  • All characters other than lowercase alphabetic characters and spaces are removed
  • Names are split on spaces ('mcguire jones' becomes 'mcguire' and 'jones')
  • Prefixes are merged into the name that follows (for example, di, O', Mc, (von|van)der?)
  • Any remaining name under three characters is removed

The words are then indexed against the constituent's ID for later searching. When that search is performed, the same process is repeated for the last name to be searched on, and a set of constituent IDs is returned by that match, possibly constrained by address if the number of results is large.

Householding is also applied, with the algorithm looking at first and last name, email address, and title and suffix to predetermine whether two similar records are indeed two members of the same household.

[Back to Top]

Duplicate Scoring - Match Rule Fuzziness (Phase 2)

In Phase 2, you can view each set of matches and determine whether they are indeed duplicates or not. If they are duplicates, you can merge the two records. In the case where one of the records is in the external database, you can simply match the Convio record with the external record or go through the process of merging them.

Automated Duplicate Management enhances this process by automatically merging two records that have a score of 100 (which canbe adjusted to suit your needs). This score is enabled by a Site Administrator.

Match rules for first, middle and last name, and address have been extended to allow for a greater degree of flexibility in identifying duplicate records. Before this new algorithm, matches had to be exact. The matching algorithm will now not only indicate exact matches, but will also allow matches that have the following common user input errors:

  • Character transpositions (such as, ’tyop’ for ‘typo’)
  • Missing characters (such as, ‘typ’ for ‘typo’)
  • Incorrect characters (such as, ‘typi’ for ‘typo’)

The scores generated as a result of this process are displayed in the interface to help you determine how to reconcile potential data matches.

[Back to Top]

Duplicate Management Activity Page

To perform duplicate management of your records, including records within Constituent360 and records in Convio and your external database:

  1. Click Data Management on the navigation menu, and then click Duplicate Management. The Duplicate Management Dashboard displays. On this page, you can quickly assess duplicate merge activity:
    • How many potential duplicates the system identifies in your records
    • How many duplicate records were merged in the preceding two weeks
    • When the most recent activity occurred
    • How many records were identified as highly likely duplicates
    • How many of those likely duplicates were processed and merged
    • How many were not merged
    • How many were not processed
    • When the next processing of duplicate records is scheduled
    • Duplicate processing over (selectable) timeframes spanning one week to one year
    • How many duplicate records in the Cons360 database were merged across the selected timeframe
      • How many manual merges
      • How many automatic merges
      • How many merges were undone
    • How many duplicate records in the Constituent360 database were matched with records in the external database across the selected timeframe
      • How many manual matches
      • How many automatic matches
    • How many profiles in the Cons360 database were reconciled with records in the external database across the selected timeframe
      • How many reconciled manually
      • How many reconciled automatically
  2. From this page you can perform two actions:
    • Resolve New Registrations - Loads a page with contact records that have been flagged as possible duplicates in your external (offline) database. You can merge the records or mark them as Not a Match.
    • Undo Merges (Duplicate Merge Undo Management) - Loads a page with a list of merged records that have run nightly over the previous two weeks. You can browse the merges on any of the dates, locate one or more records to unmerge and then undo them.

[Back to Top]

Resolve New Contacts

When your site receives new online constituent registrations and the new records appear to match existing online or offline records, it is important to resolve any conflicts promptly. Resolution is particularly important before running data sync operations because these operations will skip some records when potential duplicates exist.

You can resolve potential duplicates in your Convio database and between Constituent360 and external databases by marking the records as a match, marking them as not a match, or by merging two records into an online master. You can also unmerge a record previously created by merging.

The Search field above the New Online Record column can be used to search the list of potential duplicates by name.

This section contains the following procedures:

To resolve duplicates:

  • Click either the Resolve New Registrations tab or the link in the Snapshot box. The Resolve New Registrations page displays. The New Registration Conflicts table displays Constituent360 and external database records suspected of being duplicates. The table contains the following columns:
    • A checkbox column to let you resolve multiple records as not matching
    • New Online Record - new registration records
    • Actions - contains two or three actions:
      • Not a Match
      • Merge (the new and the matching records)
      • Resolve Duplicates (displays only when two or more duplicate records are identified by the system)
    • Database records identified by the matching algorithm as possible duplicates
    • A match score (the threshold for which is set with a site option)

To resolve two records as a match:

  1. Compare a record in the New Online Record column with the corresponding online or offline record (each in the same row) in the Best Matching Record column.
  2. If you determine that a row of two records are matches, click the Match action to remove them from the table. The page will update without displaying that row.

[Back to Top]

To resolve two records as "not a match":

  1. Compare a record in the New Online Record column with the corresponding online or offline record (each in the same row) in the Best Matching Record column.
  2. If you determine that a row of two records are not matches, click the Not a Match action to remove them from the table. The page will update without displaying that row.

To resolve multiple records as "not a match":

  1. You can also resolve multiple rows of records as not matches. Check the box in the first column for each row to select multiple registrations for resolution as not matching the suspected (online or offline) record duplicate.
  2. After you have finished selecting the registrations to resolve as not matching, click the Selected Do Not Match button. The page will update without displaying those rows.

    Note: you cannot select multiple rows across more than a single display page at a time.

To merge records:

When potential duplicates do appear to match, you can merge the two records into one online master. When you do so, the duplicate record is removed from the Constituent360 database (if both potential duplicates were online records) or from the offline database otherwise.

  1. Merge new online registration records in Constituent360 (and if appropriate also in an external database) one at a time by clicking the Merge action in the same record row. The Merge Record page will display.
  2. On the Merge Record page, action will open a page where you can select the data for the master (merged) record. The new registration record is shown at the top of the page, while the Master Record fields and the Duplicate Record fields are shown beneath it.
  3. Prepare the final Master (merged) Record, taking advantage of any of the following:
    • To access the actual contact record in either column (available for online records only):
      • Click View at the top of the column. This action opens the record in a new window.
      • Close the record when finished.
    • To display the potential Duplicate Record entries as the Master Record entries (and vice versa), click the Show Duplicate Record as the Master Record link above the columns. Again, this action is only available if both records are online records. Here are some tips for deciding which
      record to display as the Master Record:
      • The record with the most recent activity is often the best choice as the initial Master Record.
      • If the recent activity is similar for two records, a record that represents a Self-Registration may be preferable to show initially as the Master Record (rather than an Administrator Registration) because the registrant would have knowledge of the username and password associated with that record and may have a "cookie" on his computer that allows him to access your site easily.
      • If the recent activity is similar for two records, you may also want to consider the number of transactions associated with each record in determining which record to display initially as the Master Record.
    • To copy individual field entries from the Duplicate Record into the Master Record, check the box for that field in the Duplicate Record column.
    • To copy all of the fields of a certain type from the Duplicate Record for example, all the Name fields or all the Address fields check the appropriate box in the Duplicate Record column.
    • Some information about the records is shown even though it cannot be changed so that you are better informed for the merging process. To hide this information, click the Hide read-only fields link above the Duplicate Record column. The link will change to Show read-only fields so you can display it again if needed.

      Tip: This action serves to shorten the page.

  4. Scroll down through the fields to be certain you have handled all available data.

    Important: If for any reason you have additional data available or note an obvious error, you can edit the fields in the Master Record.

  5. When you are ready to accept the Master Record, click the Finish button at the bottom of the page. You will return to the list of potential duplicates where the records that you just merged will no longer be listed. The master record, however, will be available in your Constituent360 database. The duplicate, if it was an online record, will be removed.

[Back to Top]

Resolving Duplicates With More than One Match

When multiple potential duplicates are present, each potential duplicate must be handled individually. Consequently, the process flow differs slightly, although the options for each duplicate maintaining separate records or merging records -- remain the same.

To resolve duplicates when multiple matches are present:

  1. Access the Resolve New Registrations list page.
  2. Click the Resolve Duplicates link in the Actions column between the New Online Record and the Best Matching Record, as shown.
  3. On the Resolve Duplicates page that opens, observe the new record at the top, with each potential duplicate listed individually in the table beneath it.
  4. Handle each potential duplicate by clicking Not a Match, Match, or Merge in the Actions column. These actions behave just as they do when single duplicates are present except that the Finish button on the Merge page will bring you back to this page. You can also place a check beside records in individual rows and use the Selected Do Not Match button to resolve several records simultaneously.
    When the list of potential conflicts for this new registration is empty, this page will automatically display the next record from the list page even if it has only one potential match so that you can continue working through the list.
  5. Continue resolving potential matches as they are displayed on this page, keeping in mind the following:
    • If you cannot resolve duplicates for a record at this time, and want to be able to deal with them later.
    • Click the Next Record button to move on to the next new registration record with potential conflicts.
    • If none of the potential matches appear to match or have data to merge and you want the record removed from the list of potential duplicates:
      1. Check the box that reads Yes, I have resolved all conflicts for this record.

        Note: You are effectively saying Not a Match for all potential duplicates.

      2. Click the Next Record button to move on to the next new registration record with potential conflicts. The record will no longer appear in the list of potential duplicates, even though you have not specifically resolved it by marking it as not a match or by merging it.
  6. When you have finished resolving potential duplicates for the present time, click the Finish button to return to the remaining list of potential duplicates on the Resolve New Registrations page. Note that you will return to that page automatically when all records have been resolved.

    Note: Normally, when the list of potential duplicates on the Resolve Duplicates page is empty, the page will automatically display the next record for resolution. In the following situations, however, the page will display an empty list, rather than moving on:

    • If you have checked all of the potential duplicate records for a new registration and clicked the Selected Do Not Match button, the page will not display the next record, even though the list of potential duplicates is empty. In this specific case, you will need to click the Next Record button to move on.
    • If your list of potential duplicates has been filtered using the Search feature, the list may appear to be empty, when actually it is not. In this situation, clicking the Show All button will reveal any hidden duplicates.

[Back to Top]

Undo Merges

You can undo a merge for up to two weeks afterwards. However, if the record is changed within the two weeks (for example, a new transaction or interaction is recorded), you cannot undo the merge even if it is within the two weeks.

To undo a merge:

  1. Click the Duplicate Merge Undo Management tab. The View All Merges page displays.
  2. To undo just one merge, click the Undo merge icon in the last column of the row containing the Master and Purged records. The Undo Selected Merges page displays.
  3. To undo more than one merge, click the checkbox in the first column of each row containing records to unmerge. The Undo Selected Merges page displays.
  4. Click the Yes, Undo these Merges button to complete the action. The page displays a confirmation that the merge was reversed.
  5. To cancel the action, click the Return to Undo Management button. You will return to the previous page.

[Back to Top]

Duplicates in Common Ground

If you use both Convio (COM) and Common Ground (Common Ground) you may want to share your COM duplicates with Common Ground in a nearly seamless interface. If so, several points should be kept in mind:

  • The duplicate sharing and management is not real-time but depends on the nightly data sync
  • The back and forth sharing is two way
  • All duplicates are sent to Common Ground
  • Scoring information is sent to Common Ground
  • The data maintains sync between COM Constituent360 and Common Ground

Enabling this feature requires some intervention on the part of Convio Support. See below under Configure COM to Send Duplicates to Common Ground for details.

Sending Duplicate Match Scores to Common Ground

You can handle all your duplicate resolutions in Common Ground. Once the synchronization of Luminate Online to Common Ground is configured, any constituent data in match state will display in Common Ground. After duplicates are resolved in Common Ground, they will [automatically] be updated in Luminate Online. You need to make sure three preconditions are met:

  • Duplicate management is enabled on your site
  • The site option, PREDUP_MATCH_ENABLED, is set to true
  • You have latest version of Luminate Online and Common Ground

You can allow duplicate management in Common Ground. After you finish the setup, for any offline record, or any online record that has a member id,

  • A matched pair and the duplicate score,will display on the Common GroundPotential Duplicates page (under the Potential Duplicates tab). The same information, less the match score, will show in Common GroundDuplicate Management (click Utilities to find that utility). Any matches found by the predupe process will flow to Common Groundwith the sync and will be removed from the predupe tabs in Luminate Online.
  • And the match score for the pair

Configure Common Ground to Import Duplicates

Note: Contact Convio Support to configure Luminate Online to Send Duplicates to Common Ground.

  1. Select Setup from the dropdown menu (down arrowhead to the right of the administrator name at top of splash page).
  2. Under Application Setup (in the navigation on the left side of the page), click Develop.
  3. Click Custom Settings. The Custom Settings page displays.
  4. Click Manage for Common Ground Settings. The Common Ground Settings page displays.
  5. Click the Edit button. The Common Ground Settings page displays.
  6. Check the box labeled Track Duplicate Merges and click Save.
  7. Return to Luminate Online and go to Site Setup > DataSync. Click the Data Sync Operations tab.
  8. Click the Run action for the Convio Connector for Common Ground Data Sync.

    Note: If a duplicate originates in Common Ground, it must be resolved in Common Ground.

[Back to Top]