My requirement is to get all the RDM tables in Ataccama to Snowflake or to amazon S3.
Could anyone please let me know how to do that.
Solved
How can we ingest data from RDM tables in ataccama to either to Snowflake or Amazon S3.
+2- Universe Traveller
- 3 replies
Best answer by AKislyakov
Hi Akash,
Depending on the number of tables, there are two options to consider:
- For a low to moderate number of tables, you can create batch export plans that read data from the RDM and push it to Snowflake or S3 bucket. This approach is straightforward and doesn't require advanced knowledge, but it does require manual configuration and maintenance for every table. You can learn more about this option by visiting the batch export plans page.
- For a high number of tables, or if you plan to have many tables, you can leverage the new RDM REST API released in version 14.3. With a third-party ETL tool, you can develop a generic process that reads data through the API and pushes it to a desired target. You can learn more about this option by visiting the RDM REST API release notes.
- Ataccamer
- 146 replies
- Answer
Hi Akash,
Depending on the number of tables, there are two options to consider:
- For a low to moderate number of tables, you can create batch export plans that read data from the RDM and push it to Snowflake or S3 bucket. This approach is straightforward and doesn't require advanced knowledge, but it does require manual configuration and maintenance for every table. You can learn more about this option by visiting the batch export plans page.
- For a high number of tables, or if you plan to have many tables, you can leverage the new RDM REST API released in version 14.3. With a third-party ETL tool, you can develop a generic process that reads data through the API and pushes it to a desired target. You can learn more about this option by visiting the RDM REST API release notes.
+2- Universe Traveller
- 3 replies
Hi AKislyakov/ Team,
Thanks for your valuable input. I have gone with link provided.
Could you please elaborate on the same to just directly export the data of a table from RDM web app to Snowflake.
Can it be done in a single plan file with two components (RDM extended reader and JDBC writer)?
Also, could you please help me to get correct JDBC driver for Snowflake to connect with Ataccama 12.6.2 version.
- Ataccamer
- 146 replies
Hi Akash,
Can it be done in a single plan file with two components (RDM extended reader and JDBC writer)?
To export data, you will need at least three steps: a Random Record generator to set a timestamp, and RDM Extended reader, and a JDBC writer. It is important to note that this process should be done per table. This means that you will either need a separate plan for each table or combine several flows into a single plan.
In addition, you will need a workflow and a scheduler to automate the execution of these plans.
Also, could you please help me to get correct JDBC driver for Snowflake to connect with Ataccama 12.6.2 version.
According to DQC Supported Platforms and Databases - Ataccama ONE Platform 12.6.2, the version of snowflake-jdbc-3.6.28.jar was tested. However, as a rule of thumb, any later version should work just fine.
+2- Universe Traveller
- 3 replies
Hi AKislyakov
- Ataccamer
- 146 replies
Hi
To Schedule a job you need a Schedule object.
The easiest way to create one for RDM Synchronization is to follow Configuring RDM Synchronization with External Databases - Ataccama ONE Platform 12.6.3 there is Step 5 to create a Schedule.
If you want to have event-based synchronization, you can configure the On publish action that will emit a message for every record once it’s published.
1 person likes this
Reply
Most Liked this week
Login to the Ataccama Community
No account yet? Create an account
For Ataccama Customers and Partners
or
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.