Solved

Where are DQE rules applied when executing (DPE or data source)

  • 18 December 2023
  • 2 replies
  • 60 views

Userlevel 4
Badge +3

We had a internal discussion, but ended up not being 100% sure. So here's the question.

When we run DQE on a CI, where are the rules evaluated? Does the DPE get the full data (possibly with filters on the MP?) and the run the rules evaluation IN the DPE. Or does the DPE create a SQL-like statement and does the source DB return the results?

As we also run on Hadoop with DPE on Hadoop. Is this the same.

icon

Best answer by anna.spakova 18 December 2023, 14:26

View original

2 replies

Userlevel 5
Badge +8

Hello @Marnix Wisselaar , with Hadoop, if you use Spark processing, the DQE is done in Spark to have the best performance. The same if you would have Snowflake pushdown.

In the rest of the cases, the DPE will download the data from the source into temporary files stored on the DPE server, and do the DQE there. Once the job is finished, all temporary files are deleted.

I hope this answers your question.

Kind regards,

Anna

Userlevel 6
Badge +7

Hi @Marnix Wisselaar - I’m marking this thread as solved but please do let us know if you have any further questions. Happy holidays!

Reply