Filter data for Evaluation

Hi ​@karangalepranav ,
Could you please clarify, will the latest timestamp be always consistent for newer records that have to be evaluated for dq issues?
By that i mean having a dataset with historical data with different timestamps and for the sake of dq evaluation only a subset of data with specific timestamp has to be filtered.

Depending on how data looks like you can either trying to build some data transformation logic in VCI (on top of the original catalog item), if you can group certain rows of data you should be able to identify the latest timestamp for each of the groups and then you can filter records based on timestamo, for example, condition being Timestamp (for specific record) = Latest_Timestamp (for specific group). This will basically filter out the records from a subgroup where timestamp value is the latest one found for that group.

Alternatively, if the latest timestamp is consistent and predictable, you can try using data slicing functionality (which i believe is available from v15.3) and this way you can limit which data is taken during each dq evaluation for specific CI.
https://docs.ataccama.com/one/15.4.0/catalog-items/create-data-slice.html

I hope this helps.
Ivan

Its like I have data since 2023 till date and every month data is inserted once. Only on the newly inserted data we need to do dq evaluation through monitoring project. Second requirement is can we create a variable or parameter which will ask user for which date they want to do dq evaluation. Basically making it more dynamic.

​@karangalepranav If you’d like to define the parameters for data filtering from the UI - you can use the Slicing functionality. You will define the slicing criteria during slice creation and can, for example, provide a data range:

If you’d like the filtering be applied automatically based on some criteria - then VCI might be a better option​ as you can build a logic which will identify the latest timestamp from all the available records and then filter out all the records that do not much that date.​​​​​​

Thanks ​@ivan.kozlov , I tried this. I do not have specified range for date. It should be UI to ask us everytime for what date you want to run DQ evaluation. I think Data slice wont help so I am thinking to create SQL catalog Item and run DQ checks on filtered data.

​@karangalepranav , if you add the parent (unfiltered) catalog item to monitoring project and then go to configuration tab for specific CI you can configure data slice to be used directly in the project, slice can be created on top of SQL CI as well. So before the run you can either modify existing slice configuration (change time window) or create a new one with a new one with updated configuration and then update project configuration to use a different data slice.
In the test example i created original SQL CI had 326 records, but only 2 had timestamp within a range specified for the data slice. As a result, once i ran the project using the slice i created it evaluated only those 2 records that matched the filtering criteria. I believe this might be exactly what you need in this case.