Solved

How can we ingest data from RDM tables in ataccama to either to Snowflake or Amazon S3.

  • 14 June 2023
  • 5 replies
  • 107 views
A
Badge +1
  • Akash
  • Universe Traveller
  • 3 replies

My requirement is to get all the RDM tables in Ataccama to Snowflake or to amazon S3.

Could anyone please let me know how to do that.

icon

Best answer by AKislyakov 14 June 2023, 07:32

View original

5 replies

A
Rank
Userlevel 4
Badge +4

Hi Akash,

Depending on the number of tables, there are two options to consider:

  • For a low to moderate number of tables, you can create batch export plans that read data from the RDM and push it to Snowflake or S3 bucket. This approach is straightforward and doesn't require advanced knowledge, but it does require manual configuration and maintenance for every table. You can learn more about this option by visiting the batch export plans page.
  • For a high number of tables, or if you plan to have many tables, you can leverage the new RDM REST API released in version 14.3. With a third-party ETL tool, you can develop a generic process that reads data through the API and pushes it to a desired target. You can learn more about this option by visiting the RDM REST API release notes.
A
Badge +1

Hi AKislyakov/ Team,

Thanks for your valuable input. I have gone with link provided.

Could you please elaborate on the same to just directly export the data of a table from RDM web app to Snowflake.

Can it be done in a single plan file with two components (RDM extended reader and JDBC writer)?

Also, could you please help me to get correct JDBC driver for Snowflake to connect with Ataccama 12.6.2 version.

@Cansu  Could you also look into this and help. Thanks

 

 

A
Rank
Userlevel 4
Badge +4

Hi Akash,

Can it be done in a single plan file with two components (RDM extended reader and JDBC writer)?

 

To export data, you will need at least three steps: a Random Record generator to set a timestamp, and RDM Extended reader, and a JDBC writer. It is important to note that this process should be done per table. This means that you will either need a separate plan for each table or combine several flows into a single plan.

In addition, you will need a workflow and a scheduler to automate the execution of these plans.

 

Also, could you please help me to get correct JDBC driver for Snowflake to connect with Ataccama 12.6.2 version.

According to DQC Supported Platforms and Databases - Ataccama ONE Platform 12.6.2, the version of snowflake-jdbc-3.6.28.jar was tested. However, as a rule of thumb, any later version should work just fine.

A
Badge +1

Hi AKislyakov @Cansu , could you please let me know how I can schedule the job and also if it can be triggered on event based. For example if someone modify the record in RDM web app then workflow should automatic run and we should get the updated data in Snowflake. Could you please suggest how I can do this event based trigger. 

A
Rank
Userlevel 4
Badge +4

Hi @Akash 

To Schedule a job you need a Schedule object.

The easiest way to create one for RDM Synchronization is to follow Configuring RDM Synchronization with External Databases - Ataccama ONE Platform 12.6.3 there is Step 5 to create a Schedule.

If you want to have event-based synchronization, you can configure the On publish action that will emit a message for every record once it’s published. 

Reply


Community Terms of Use & ConditionsCookie settings