Hi everyone, 

We are here with Part II of our VCIs best practice series and today we will delve deeper into two significant aspects of working with VCIs - connecting to natively supported data sources and non-natively supported data sources.

In the first post, we explored the creation and transformation of Virtual Catalog Items (VCIs) within Ataccama ONE. You can check it out here

Connecting to Natively Supported Sources

Ataccama ONE makes it simple to connect with data sources that are natively supported by following these steps:

1. Create a VCI, as explained in the previous section.

2. To read data from a natively supported source, add the "JDBC Reader" step to your plan.

   

   • Depending on your transformation requirements, you can use the JDBC Reader step alone or combine it with existing catalog items.

3. Configure the "JDBC Reader" step as follows:

   • On the "General" tab, in "Data Source Name," provide the source identifier of the connected database. You can find the source identifier in ONE under "Sources."

   • In "Query String," provide the SQL query for your transformation. To write the SQL query, check the data source structure in ONE under "Sources > cdata source name] > mconnection name] > Browse."

   • Specify "Columns" as needed and select "OK."
      

     

      

4. Publish the item following the steps outlined in Part 1.

5. In ONE, open the VCI you are working with and select "Edit."

6. Specify the connection to the source in "Connection" and save your changes.
    

   

    

7. Publish the changes. Your catalog item now has the "Origin" specified.
    

   

    

Please note that data sources that are not set up as combinable in the Data Processing Engine cannot be combined in ONE. Make sure to check the Data Processing Engine's support for the connection before attempting to use non-combinable data sources.

Connecting to Non-Natively Supported Sources

When dealing with data sources not natively supported by Ataccama ONE, follow these steps:

1. Create a VCI, as explained in Part 1.

2. Add the necessary reader step to retrieve data from the source, which could be a "JSON Reader" step or "API calls," depending on the source.

   • Ensure compatibility with different sources by checking how the source was configured in Data Processing Engine (DPE). Ensure that DPE supports the connection.

3. Configure the selected reader step.

4. In ONE, create a data source connection as described in the Ataccama documentation.

5. Open the VCI in ONE, select "Edit," and specify the newly created data source connection in "Connection." Save your changes.

6. Publish the changes. Your catalog item now has the "Origin" specified.

Using Global Runtime Configuration

You can also set up connections through the Global Runtime Configuration, which can be used for both natively and non-natively supported sources. However, note that this method cannot be used with the JDBC Reader step configured for Spark due to limitations regarding driver paths.

To set up a connection using the Global Runtime Configuration:

1. Connect to the source you want to read data from, and obtain the connection details.

   

    

2. Export the runtime configuration from ONE Desktop.

   

    

3. Specify the container and name for the runtime configuration export.

   

    

4. Open the exported runtime configuration file in ONE Desktop.

   

    

5. Delete lines that do not refer to the necessary connection.

   

    

6. In the DPM Admin Console, copy and save the modified runtime configuration.
7. Publish the component in ONE Desktop with the connection to the source.

Notes and Limitations

While working with VCIs, keep these important notes and limitations in mind:

1. Viewing data from virtual catalog items is not supported at this time.
2. You can assign terms to virtual catalog items the same way you would with other catalog items.
3. Virtual catalog items can only be processed using full profiling (not sample or custom).
4. Structure checks in monitoring projects cannot be set on virtual catalog items.

   

    

5. Policies can be applied to virtual catalog items in the same way as with other catalog items.
6. Permissions for virtual catalog items apply to the entire workspace, so changes should be made for the entire workspace.
7. Lineage and relationships tabs are not added automatically, but you can set relationships manually using the "Relationships" widget.
8. You can use lookup files in components. Simply upload the file to ONE and access it through ONE Desktop.

   

    

And that’s all for the VCIs (at least for now )! How do you use VCIs in your day-to-day role? Share it with the community in the comments below