Configuring Lineage via Lineage DB

Hi there,

Lineage DB is automatically installed at the same DB server as the mmm database (if not configured otherwise), which is PostgreSQL. By default, all necessary properties are included in the installation and configured during it. All tables are created automatically and migrated to the latest version of the lineage DB schema for a specific version of ONE platform.

Kind regards,

Max

There are 3 tables in lineage database with information about:

1. catalogitems
2. attributes
3. relations between attributes

To build our own lineage (without using Manta) we supplied (inserted into those tables) data using One Desktop (but it essentially doesn’t matter, you can use DBeaver or `psqli`).
Once you put the data into lineage tables it is immediately available in web app, provided you put correct information there.

Note that:

• Server name should be the same in your catalogitems table and data source name in web app. (you can edit it in web app, usually it is empty)
• CatalogItems should have the same path as you can see in One Web App. Note that all the names are case sensitive too.

@Maxim Kim @Max 

Thank you for your responce!

You mean that we should put some data in existing tables? What data is correct and where to get it?

By the way, we didn’t have database lineage and 3 tables also. Should we create it? or just make connection to mmm database(no PostgreSQL in our sources), which is mentioned by @Max ?

@FCB developer It depends on whether we correctly understood what exactly is your goal.

For each installation of One Gen2, there is a postgres database. In that DB you should have a schema named external_lineage where those 3 tables should be located.

Now if your goal is to upload your own lineage to be visualized in web app -- you can do it by inserting corresponding information to these tables. Structure should be self-explanatory (and to be frank I don’t have it at the moment, but I remember it was quite simple)

@Maxim Kim 

Thanks!

We found the main PostgreSQL database and looked, made sure that the lineage schema is present, but it is empty, there is not a single table

We are now trying to connect to a postgresql database from One Web App (some credential problems occured)

Question: Should we create this 3 tables? because says that it should automatically be there already

@Maxim Kim thank you!

Yes. we fixed it! And see all tables from your screenshot and also made connection with One Web.

I have a little lack of representation in filling in the tables, it turns out that this way of setting up a lineage is to add each catalog item’s information separately into 3 tables of external_lineage?

Indeed, usually Manta calculates the lineage and One gets the information out of it and puts into those tables. But we had a manual lineage that was governed by special ppl in excel/csv file, so we just transformed it and put there ourselves.

Here are some hints for you:

Lineage of the customername attribute

And how it looks in DB:

@Maxim Kim 

Thank you! I am very grateful for your answers. 

I will try it and let you know.

Good day! @Maxim Kim 

I have some questions

1)Should we profile 3 tables after inserting data? I launched full profiling, but lineage does not visualized in web app.

2)Or is it mean that inserted information are wrong ? By the way, I inserted id as (1, 2)

3)Furthemore I noticed that our external_linege in mmm, but not in linege as yours

1. No, profile/re-profile is not needed, you only have to have tables imported(or profiled or documented) at least once
2. It might be that info is not sufficient to have a lineage
3. here I am not sure, it might depend on version of Ataccama One.

For nr.2 make sure that you have server name matching your connection:
 

According to your screenshots, it should be mmm. Also make sure that database_name and schema_name are exactly the same as in catalog.

@Maxim Kim Good day!

I managed to do it. Thank you!

Good news! I am glad it worked out for you!