Skip to main content
Solved

Difference between Full Load and Complex Delta Load

  • 19 July 2024
  • 1 reply
  • 22 views

I need to understand the difference between Full and Complex Delta Load in MDM.
As per this document on Complex Delta Load,  Delta Dependency Mode  needs to be setup for each entity. I see there are two options for it -Autonomous and Central. Can someone please explain the difference between these and in what scenario these should be used?
I also see that The engine expects an additional column in the input interface denoting the type of record change (change_type: I - insert, U - update, D - delete). Where shall we define these and what how can we identify which records needs which change_type?

TIA 😊

1 reply

A
Rank
Userlevel 5
Badge +2

Hi @Kundan,

This document describes autonomous, central, and dependent entities.

The difference between full loads and delta loads is as follows:

Full Loads

  • Expect the entire dataset as input for a given source.
  • Missing records are considered deleted.
  • Remaining records are classified as new, updated, or unchanged.
  • Easier to configure.
  • Requires transferring the entire dataset, which can be problematic for larger implementations or micro-batching.

Delta Loads

  • Expect only a subset of the data as input, with a change type (insert/update/delete) for each record.
  • Suitable if your source provides change data capture (CDC) or you have ETL infrastructure (e.g., data lake or ODS) with CDC capabilities.

 

Reply


Community Terms of Use & ConditionsCookie settings