Unifying your product integrations data with entity mappings

One of the biggest challenges companies face when building product integrations is unifying highly variable data across different SaaS providers and customers into a normalized format their applications can consume.

No two SaaS platforms are the same, even when they’re in the same category (e.g. CRM). Just ask anyone who’s had to integrate with both HubSpot and Salesforce, and you’ll quickly learn about the differences between their data models and APIs (hint: it’s not fun).

And of course, no two companies configure their CRMs exactly the same way. For example, Salesforce is notorious for its customizability through custom fields, custom objects, and overrides. You’ll need to reconcile these customizations across customers in your integration.

The data variability problem

To make this more tangible, let’s look at a common scenario for companies integrating with CRMs:

You want to ingest your customer’s Account data from their CRM and join it against other customer data or enrich it with 3rd party data. To do so, you’ll need to sync and normalize certain account fields across different customers and CRM providers into your own data model. But what is the Account object?

In Salesforce, there’s an “Account” object, but the corresponding standard objects in HubSpot and Pipedrive are called “Company” and “Organization”, respectively. At the same time, different customers who use the same CRM can use different objects to store the same information. For example, a Salesforce customer may have configured their CRM to use “Person Accounts” instead of “Accounts”, or even a custom object based on geography or product line (e.g. "Company_Europe").

Information will also be stored in different fields, either because the field doesn’t exist for certain providers (e.g. Pipedrive doesn’t have a native domain field on their Organization object), or because a specific customer has chosen to rename the field or use a custom field like "domain_custom" or "domain__c" instead.

You can keep track of these object and field mappings manually at first, but the data complexity only grows with more providers and more customers. It’s easy to end up with a bunch of band-aid application logic, which quickly becomes an expensive maintenance headache for resource-strapped startups trying to move fast.

Introducing Entity Mappings in Supaglue

We built Entity Mappings to be the flexible normalization layer that companies need to make sense of their customers’ data. It eliminates the need to write custom business logic for each customer, run manual schema migrations, or maintain an ever-growing set of data integration pipelines.

In short, entity mappings make it much easier to build and maintain deep product integrations by abstracting away provider and customer-specific idiosyncrasies from application developers.

The core insight behind Entity Mappings is simple:

1. Let the application developer define the business concepts (entities) they care about.
2. Expose an interface for their customers to map the objects in their SaaS providers to those entities (entity mappings).
3. Automatically apply these mappings when you read and write to those entities.

In the above example, a developer would define an “Account” entity with a name and domain field. Each customer then maps an object (standard or custom) to the Account entity, followed by the fields on that object to the two entity fields. Once the entity mapping is saved, Supaglue applies this mapping when the developer reads or writes to the “Account” entity.

Entity mappings work seamlessly with Supaglue’s read and write access patterns, and for any SaaS provider and database destination supported by our library of 25+ connectors (and growing).

Comparison against other approaches

We’re not the first integrations platform to tackle this problem, but we are the first to solve the problem in a first-class way. Specifically, we designed entity mappings to address the flaws of other approaches.

For example, Unified APIs attempt to solve the data variability problem by providing a common model across SaaS providers within a single category. However, they suffer from a “lowest common denominator” problem: unified APIs are limited to the 8-10 objects in the common model so you’re out of luck if you’re not covered. By contrast, Entity Mappings offer a “bring-your-own common model” option and the ability to map to any object supported by the SaaS provider, including custom objects. Unified APIs also require developers to write additional code to fetch data, whereas Supaglue directly syncs the mapped data directly into your application database.

Another approach some products take is normalizing data through field mappings and transformations that live inside workflows. Unfortunately, these field mappings tend to be error-prone, difficult to maintain, and limited to operating on single records or small batches. On the other hand, entities in Supaglue are first-class concepts that can be tested, synced, and written to in minutes and at scale.

The third approach companies sometimes adopt is the ELT model, where a traditional data integration product lands all the raw data, and a separate, metadata-driven transformation pipeline that handles the data mapping afterward. One benefit of this approach is that you can re-run transformations without resyncing data from the SaaS provider. However, it’s super costly for companies to take on all the extra responsibility of building and maintaining the transformation pipeline. Most companies in this group don’t know there’s a better way.

Other benefits of entity mappings

Entity mappings are particularly useful for integrating with systems of record like CRMs and ERP systems, where high degrees of data customizability is both a core feature offered by the platforms themselves and standard practice for their end users.

But it’s worth mentioning entity mappings offer some other important benefits, even when custom objects and custom fields aren’t involved:

1. Cleanliness - all entities are associated with first-class API endpoints and destination tables, so you have a clean interface to reason about your data and build on top of.
2. Efficiency - you’re only reading/syncing and writing the data you actually need, which improves performance when dealing with large volumes of data.
3. Future-proofing - when you’re only building one or two integrations, data variability may not seem important. But starting with entity mappings lets you seamlessly add more integrations without needing to run any data migrations or redo your data model.

Conclusion

Data variability in product integrations is a real problem, especially when integrating with systems of record like CRMs. We’ve talked to dozens of teams that have built native product integrations over the last year, and the most successful implementations are the ones that have thoughtfully solved it. Unfortunately, this often required months or more of engineering investment in an internal entity mappings framework.

The truth is that building a flexible normalization framework for product integrations is simply the wrong tradeoff for the vast majority of startups, and even some later-stage companies. There are too many core product demands that should take priority. But not investing in it creates all kinds of scaling and integration maintenance issues.

For companies in this camp, we hope to make your lives easier: entity mappings will help you build deeper, more advanced product integrations for your customers while saving you countless hours of development and ongoing maintenance.

If you’re interested in learning more about Entities and Entity Mappings, check out our docs or book a demo.