Solved

How to get the metadata for a post-processing integration input?

  • 22 September 2022
  • 5 replies
  • 124 views
M
Userlevel 4
Badge +3

Hi Experts,

Maybe I’m missing something here. I’d like to get the name and id of the monitoring project, catalog item in a post process, but the integration input doesn’t provide those, right? (Why nut? Or am I missing some system pars I could use here?) So how can I get these quickly?

Thank you!

 

icon

Best answer by anna.spakova 23 September 2022, 11:57

View original

5 replies

anna.spakova
Rank
Userlevel 5
Badge +8

Hello @Marnix Wisselaar , maybe someone else will correct me, but I don’t think there is an automated way how to do that. I see two ways how you can achieve it, although they require manual configuration in the post-processing component:

  • Add the name of the project and CI manually (you have to create the component for each CI manually anyway). Using e.g. Alter Format step.
  • Add the names using Metadata Reader steps and IDs of the project and the CI in the project (you can obtain those from the URL when you open the project and then the CI in the project). This is better in case you expect the name of the project and the CI will change over time so that you don’t have to update it every time. The Metadata Reader will always download the current information and using the ID you can filter for your project/CI. Esp. for the name of the project this may be useful as you would have to update it in every post-processing component for every CI.

Anna

M
Userlevel 4
Badge +3

Hi Anna,

I did exactly that, but thought there might be a faster way. 

 

<?xml version='1.0' encoding='UTF-8'?>
<purity-config workingDir="workspace://Stater/bin" version="13.7.1.220422-1645-3b4937d6">

<!-- (Add LastRun DateTime) --><step mode="NORMAL" className="com.ataccama.dqc.tasks.flow.AlterFormat" disabled="false" id="Add LastRun DateTime">
        <properties>
            <addedColumns>
                <addedColumn name="LastRun" type="DATETIME"/>
            </addedColumns>
            <removedColumns>
                <columnRef name="processingName"/>
            </removedColumns>
        </properties>
        <visual-constraints layout="vertical" bounds="480,264,-1,-1"/>
    </step>

<!-- (Filter row with max starttime) --><step mode="NORMAL" className="com.ataccama.dqc.tasks.conditions.Filter" disabled="false" id="Filter row with max starttime">
        <properties condition="startedAt == LastRun"/>
        <visual-constraints layout="vertical" bounds="528,456,-1,-1"/>
    </step>

<!-- (Get Max startdatetime) --><step mode="NORMAL" className="com.ataccama.dqc.tasks.analysis.aggregator.AggregatingColumnAssigner" disabled="false" id="Get Max startdatetime">
        <properties>
            <aggregations>
                <aggregation>
                    <assignments>
                        <columnAssign expression="maximum(startedAt)" name="LastRun"/>
                    </assignments>
                    <groupBy>
                        <keyComponent expression="processingMPID"/>
                    </groupBy>
                </aggregation>
            </aggregations>
        </properties>
        <visual-constraints layout="vertical" bounds="528,360,-1,-1"/>
    </step>

<!-- (IN: MonitoringProjectID) --><step mode="NORMAL" className="com.ataccama.dqc.tasks.common.usersteps.io.InputStep" disabled="false" id="IN: MonitoringProjectID">
        <properties>
            <columns>
                <columnDef name="MonitoringProjectID" type="STRING"/>
            </columns>
            <shadowColumns/>
        </properties>
        <visual-constraints layout="vertical" bounds="528,72,-1,-1"/>
    </step>

<!-- (monitoringProject) --><step mode="NORMAL" className="com.ataccama.dqc.tasks.io.mmm.reader.MMMReader" disabled="false" id="monitoringProject">
        <properties templateEndMark="#" workflowState="published" entityType="monitoringProject" parentIdColumnName="" serverName="ONEPlatform" templateBeginMark="#" entityColumnName="mpName" inputTemplate="$id = &#39;#MonitoringProjectID#&#39;" idColumnName="MPID">
            <childEntities/>
            <columns>
                <mMMColumnDef src="monitoringProject.name" name="name" type="STRING"/>
                <mMMColumnDef src="monitoringProject.description" name="description" type="STRING"/>
            </columns>
            <dataFormatParameters falseValue="false" dateTimeFormat="yyyy-MM-dd HH:mm:ss" decimalSeparator="." dayFormat="yyyy-MM-dd" trueValue="true" dateFormatLocale="en_US" thousandsSeparator=""/>
            <embeddedEntityStreams>
                <mMMReaderRootStreamConfig name="MPprocessings" parentIdColumnName="processingMPID" attribute="processings" entityColumnName="processingName" idColumnName="processingID">
                    <childEntities/>
                    <childStreams/>
                    <columns>
                        <mMMEmbeddedEntityColumnDef src="monitoringProjectProcessing.state" name="state" type="STRING"/>
                        <mMMEmbeddedEntityColumnDef src="monitoringProjectProcessing.startedAt" name="startedAt" type="TIMESTAMP"/>
                    </columns>
                    <shadowColumns/>
                </mMMReaderRootStreamConfig>
            </embeddedEntityStreams>
            <shadowColumns/>
        </properties>
        <visual-constraints layout="vertical" bounds="528,168,-1,-1"/>
    </step>

<!-- (out) --><step mode="NORMAL" className="com.ataccama.dqc.tasks.common.usersteps.io.OutputStep" disabled="false" id="out">
        <properties enforceFormat="false">
            <requiredColumns/>
        </properties>
        <visual-constraints layout="vertical" bounds="528,552,-1,-1"/>
    </step>

<!-- (Trash) --><step mode="NORMAL" className="com.ataccama.dqc.tasks.io.Trash" disabled="false" id="Trash">
        <properties/>
        <visual-constraints layout="vertical" bounds="600,264,-1,-1"/>
    </step>

<connection className="com.ataccama.dqc.model.elements.connections.StandardFlowConnection" disabled="false">
        <source endpoint="out" step="Add LastRun DateTime"/>
        <target endpoint="in" step="Get Max startdatetime"/>
        <visual-constraints>
            <bendpoints/>
        </visual-constraints>
    </connection>
    <connection className="com.ataccama.dqc.model.elements.connections.StandardFlowConnection" disabled="false">
        <source endpoint="out" step="Filter row with max starttime"/>
        <target endpoint="in" step="out"/>
        <visual-constraints>
            <bendpoints/>
        </visual-constraints>
    </connection>
    <connection className="com.ataccama.dqc.model.elements.connections.StandardFlowConnection" disabled="false">
        <source endpoint="out" step="Get Max startdatetime"/>
        <target endpoint="in" step="Filter row with max starttime"/>
        <visual-constraints>
            <bendpoints/>
        </visual-constraints>
    </connection>
    <connection className="com.ataccama.dqc.model.elements.connections.StandardFlowConnection" disabled="false">
        <source endpoint="out" step="IN: MonitoringProjectID"/>
        <target endpoint="in" step="monitoringProject"/>
        <visual-constraints>
            <bendpoints/>
        </visual-constraints>
    </connection>
    <connection className="com.ataccama.dqc.model.elements.connections.StandardFlowConnection" disabled="false">
        <source endpoint="MPprocessings" step="monitoringProject"/>
        <target endpoint="in" step="Add LastRun DateTime"/>
        <visual-constraints>
            <bendpoints/>
        </visual-constraints>
    </connection>
    <connection className="com.ataccama.dqc.model.elements.connections.StandardFlowConnection" disabled="false">
        <source endpoint="out" step="monitoringProject"/>
        <target endpoint="in" step="Trash"/>
        <visual-constraints>
            <bendpoints/>
        </visual-constraints>
    </connection>
</purity-config>

M
Userlevel 4
Badge +3

It would be nice if the PP Integration input would give it's processingID!

S
Userlevel 2
Badge +3

Agree. There is already a pain in creating a PP for each of the catalog items which is not a great solution if you have 100’s of table. 

secondly, without Catalog name, ID. this feature is half baked. The developers have created without thinking how the end user will implement it.

M
Userlevel 4
Badge +3

Hi Siva,

In the Desktop IDE the Integration Input could give a second output: The ID of the instance. 

Reply


Community Terms of Use & ConditionsCookie settings