Skip to main content

We had a internal discussion, but ended up not being 100% sure. So here's the question.

When we run DQE on a CI, where are the rules evaluated? Does the DPE get the full data (possibly with filters on the MP?) and the run the rules evaluation IN the DPE. Or does the DPE create a SQL-like statement and does the source DB return the results?

As we also run on Hadoop with DPE on Hadoop. Is this the same.

Hello @Marnix Wisselaar , with Hadoop, if you use Spark processing, the DQE is done in Spark to have the best performance. The same if you would have Snowflake pushdown.

In the rest of the cases, the DPE will download the data from the source into temporary files stored on the DPE server, and do the DQE there. Once the job is finished, all temporary files are deleted.

I hope this answers your question.

Kind regards,

Anna

Hi @Marnix Wisselaar - I’m marking this thread as solved but please do let us know if you have any further questions. Happy holidays!

Reply


Community Terms of Use & ConditionsCookie settings