Skip to main content
Solved

updating LKP via a web service for other web services to consume

  • July 26, 2022
  • 2 replies
  • 130 views
A
Data Voyager
Forum|alt.badge.img+2

Is there a best approach to have LKPs updated via a webservice so that other web services can consume the updated lkp without restarting DQS?

Ā 

I have Web Service ā€˜Aā€™ which builds a dictionary file.Ā 

I also have Web Service ā€˜Bā€™ which uses the dictionary built by Web Service ā€˜Aā€™.

I would like to run Web Service ā€˜Aā€™ so that Web Service ā€˜Bā€™ is able to immediately use the updated dictionary without restarting the DQS service.Ā 

Ā 

Ā 

Currently when i run Web Service ā€˜ Aā€™ IĀ get a locked file issue:

26.07.2022 12:43:56 [FATAL] Ā  [OnlineServicesComponent][buidTestDictionary.online][buidTestDictionary][buidTestDictionary.comp] [40] Internal error occurred during run of the plan: java.nio.channels.OverlappingFileLockException[STEP Lookup Builder[Lookup Builder]]
java.nio.channels.OverlappingFileLockException
Ā Ā  Ā at sun.nio.ch.SharedFileLockTable.checkList(Unknown Source)
Ā Ā  Ā at sun.nio.ch.SharedFileLockTable.add(Unknown Source)
Ā Ā  Ā at sun.nio.ch.FileChannelImpl.tryLock(Unknown Source)
Ā Ā  Ā at com.ataccama.dqc.io.storage.RafROStorage.open(RafROStorage.java:49)
Ā Ā  Ā at com.ataccama.dqc.tasks.common.data.table.StandaloneTableBuilder.<init>(StandaloneTableBuilder.java:103)
Ā Ā  Ā at com.ataccama.dqc.tasks.builders.LookupBuilder$TheInstance.runStandard(LookupBuilder.java:328)
Ā Ā  Ā at com.ataccama.dqc.tasks.common.hadoop.MRStepInstanceBase.run(MRStepInstanceBase.java:32)
Ā Ā  Ā at com.ataccama.dqc.processor.internal.runner.ComplexStepNode.runNode(ComplexStepNode.java:69)
Ā Ā  Ā at com.ataccama.dqc.processor.internal.runner.RunnableNode.run(RunnableNode.java:28)
Ā Ā  Ā at com.ataccama.dqc.commons.threads.AsyncExecutor$RunningTask.run(AsyncExecutor.java:131)
Ā Ā  Ā at java.lang.Thread.run(Unknown Source)

Ā 

Ā 

any ideas or best approaches to do this will be appreciated. I have added the location of the LKP file to theĀ 

ā€˜versionedFileā€™ segement in the ServerConfig xml file and this get the same issue.Ā 

Thanks

Best answer by DannyRyan

Hi Asif,
Ā 

Does the .lkp file being used by web service B exist in the same location as the .lkp file being updated by web service A? If so, this might be the root cause of the overlapping file lock error message you are receiving.

There is the best approach method to refreshing .lkp files in productionĀ that are being used by other services. The approach involves having two separate locations for your .lkps files.

The first location ext_new/lkp/ is where the build plans would create the new lkp files. Whilst the second location ext/lkp/ is where other services would be pointed to.Ā (web service X, please use the .lkp files in this location).

With a technical workflow (.ewf) the new .lkp files would be created, moved to the location of the existing .lkps and the virtual file system refreshed so all the services are aware of the .lkp files have changed on the file system.

There is an example of this process in the ONE Desktop application, under the Workflow Tutorials / 02_Run_Plans / 02_00_Build_Lookups.ewf which might be helpful for you.

I hope this helps, please let us know how to get on.
Have a great day.

Danny

    2 replies

    DannyRyan
    Ataccamer
    Forum|alt.badge.img+2
    • DannyRyanAtaccamer
    • Head of Technical Training
    • Answer
    • July 26, 2022

    Hi Asif,
    Ā 

    Does the .lkp file being used by web service B exist in the same location as the .lkp file being updated by web service A? If so, this might be the root cause of the overlapping file lock error message you are receiving.

    There is the best approach method to refreshing .lkp files in productionĀ that are being used by other services. The approach involves having two separate locations for your .lkps files.

    The first location ext_new/lkp/ is where the build plans would create the new lkp files. Whilst the second location ext/lkp/ is where other services would be pointed to.Ā (web service X, please use the .lkp files in this location).

    With a technical workflow (.ewf) the new .lkp files would be created, moved to the location of the existing .lkps and the virtual file system refreshed so all the services are aware of the .lkp files have changed on the file system.

    There is an example of this process in the ONE Desktop application, under the Workflow Tutorials / 02_Run_Plans / 02_00_Build_Lookups.ewf which might be helpful for you.

    I hope this helps, please let us know how to get on.
    Have a great day.

    Danny

    Danny | Did it answer your question? Mark it as the correct answer!
      A
      Data Voyager
      Forum|alt.badge.img+2
      • asif shaikhData Voyager
      • Author
      • Data Voyager
      • September 21, 2022

      Hi Danny,

      Many thanks for your reply. Sorry for my delayed response to your update.

      You suggestion of moving the file would work but the problem we do not have the workflow capability. We simply have DQS only. We do not have any other similar tools we can use on the machine where DQS resides.

      I noticed that the file lock is enforceĀ by the JVM. So i was able to configure the FileVersion in theĀ serverConfig file and then create 2 DQS window services. One which updates the LKP and another which uses it. Its not ideal but it gets around the problem.Ā 

      Thank you.

      Asif

      Community Terms of Use & ConditionsCookie settingsAccessibility statement