How to schedule profiling of specific schema in the database Or subset of that schema if schema is too big.

Hi ​@Ojaswini -- while you can’t schedule profiling of a selected schema from the web app, you could use the following GraphQL query to trigger profiling of selected catalog items programmatically. 

query profiling (
  $gid: GID!
  $configId: GID!
) {catalogItem(gid: $gid) {
    gid
    profile(defaultCredential: true, configId: $configId)
     {gid}
     }
    }

Using a ONE Desktop plan, you can export the list of locations (schemas) and the list of catalog items, then filter down to the list of schemas you’re interested in, and then pass the resulting list of catalog items to the JSON call step with the above query. You can further automate this with a scheduled orchestration workflow. Have you created similar plans and workflows before? Let me know if you have questions, I’ll be happy to point you to some more info!

Hi,

We have the same requirement: we would like to schedule profiling only for Catalog Items belonging to a specific schema, while completely excluding Catalog Items from other schemas in the same source.

We understand the GraphQL workaround described above. However, we have also discussed this with Ataccama Support (SR-36020), and they mentioned that it might be possible to use Documentation Flow configuration and a metadata field common to all Catalog Items from the selected schema.

As we understand it, this would allow matching Catalog Items to be fully profiled, but Catalog Items from other schemas would still be processed using sample profiling.

Is there a configuration option in the Documentation Flow to completely exclude Catalog Items from other schemas from the scheduled profiling run, rather than profiling them in sample mode?

​@onufry hi -- there’s no way to set the documentation flow to completely skip profiling on certain assets.
To add to the option covered in this post, you migth consider these:

1. Trigger the documentation flow from outside with paths. The GraphQL operation extendedDocumentationFlow(configurationGid, paths, excludedPaths, …) on the connection or source limits the run to the given schema(s). Run it on a timer from an orchestration workflow or an external cron. This is a variant of the approach above, which calls catalogItem(gid){ profile(...) } for each table instead.
2. Separate Connection for the wanted schema(s), then schedule that Connection. Two ways to do that:

• DPE driver config: each JDBC driver type in DPE supports schema-include-pattern / schema-exclude-pattern (and table-include-pattern / table-exclude-pattern) as a regex, e.g. plugin.jdbcdatasource.ataccama.one.driver.postgresql.schema-include-pattern=^sales$. This setting covers every connection using that driver type on that DPE, and it affects browsing as well as import. To affect only one connection, define a second driver entry (copy of the original with its own key and the include pattern) and point the dedicated Connection at it.
• JDBC URL parameter: some JDBC drivers accept a URL parameter that limits metadata calls to one database/schema. For example, Snowflake with db=…&schema=…&CLIENT_METADATA_REQUEST_USE_CONNECTION_CTX=true. This depends on the driver: PostgreSQL's currentSchema, for instance, only sets the default schema and does not hide the others. Which technology are you using? To evaluate this option, check the corresponding driver's documentation.

3. Data Observability on the Source.

• The DO schedule runs on catalog items that match its filter. The filter can be an AQL expression (e.g. items under a given schema), with a blocklist for exceptions. Anomaly detection and domain detection then profile only those items.
• Caveat: it works on catalog items that already exist and doesn't discover new tables. New tables still need a Source or Connection import.

Hope this is useful! If you have additional questions, just let me know. 

Lisa