Hi All,
When we insert relations via Collibra writer we have defined source/target fields in the writer step. There is only one way to send relations to Collibra. We got into a case where if 1 source can have many targets eg( 1 schema with 1000 tables) and you give schema in source and table in target and you define the Import File Size as 100, the writer does not breaks the calls in 100 records instead it will send 1 call with all 1000 records since 1 schema has 1000 tables. This makes sense as sending at once makes sure that all the tables get associated to 1 schema. If it will break it into 100 then only the last call with 100 records will get related to that schema.
Now to solve this problem we can send the relations to Collibra in reverse direction as well (1 table has only 1 schema). In this case the writer will correctly batch the records in 100. But inorder to do that the collibra writer should be updating the json call it sends internally. Which will be a product enhancement. I have attached the DQRule1.txt which is how the writer sends call and DQRule1.reverse.txt where the same relation can be send in reverse way having same impact in Collibra. I am working with Ataccama to get this enhancement done where we can define which way to send the relation.
By the time this enhancement comes, I need to implement solution to send the relation in reverse way. So I need help in finding how can i implement batching in component. IE if my file has 1000 records how can I process 100 records at a time. I am planning to batch, generate a json file of 100 records, make collibra import json-job api call, get job id, track the response. Any guidance achieving this will help.