RDM REST API - using count and offset to get all records

Also, if total number or rows is reasonably low, you can just increase the ataccama.one.rdm.api.response.max-count parameter (RDM Application Properties :: Ataccama ONE). Although, that might lead to increased memory consumption

Thanks ​@AKislyakov !

This is what I ended up with:

In the first Json call, I pass in my query parameters, and queried my table:

I create a data stream of “root” to get my data count:

For each input row, I calculate how many calls are actually required based on the data count returned:

The seq_list text file is my very quick and dirty way to get a sequence in 1 line. I know my source data and it’s highly unlikely to get to 30000 rows per combo:

Join this list together with my required calls:

Split the list comma separated:

Calculate the offset per “call”:

And then make essentially the same json call again, but with the offset param:

Not hundred percent happy with the dirty file generating the list (e.g. for 3, make list 1,2,3 ) but it worked. So it’ll do for now.

What magic do you do in the Alter Format to come up with the list so you could split it?

Thanks,

May

Sorry, should’ve shared that earlier

We had this code

toString(page_size) + 
iif(total_count < page_size, '',
	replicate(' ' + toString(page_size), (total_count div page_size) - 1) + iif(total_count % page_size > 0, ' ' + toString(total_count % page_size), ''))

For inputs e.g. total_count = 34 and page_size = 10 it will generate value of 10 10 10 4
which we than split by space, and re-number by descriptor builder and finally use it to calculate page_num and offset
 

Thanks ​@AKislyakov  this worked beautifully. I got rid of the text file. This replicate expression is gold!

For those interested, here’s my final plan looking something like this:

1. Set page size: just hard code an integer column to 1000
2. Get counts per page: the big expression that ​@AKislyakov shared above, that uses the replicate expression. I modified slightly to make it comma separated.
3. Then I use Regex splitter recognising comma as a pattern to split
4. Record descriptor builder to make sure my multiple input query parameters are considered as a group
5. Get page num is this: tointeger(right(rd, length(rd) - lastIndexOf(rd, ':') -1)) -1
6. Get offset is this: tostring(page_size * page_num)

Thank you for sharing the plan and steps ​@may_kwok 🙌