Zaloni Ideas
Created by Chawjyoti Buragohain
Created on Nov 27, 2019

Watermark action in Bedrock 4.4.1 does not create a partitioned entity

Hi We are using watermark action of Bedrock and it seems like it does not create a partitioned entity. Everytime watermark action runs it updates the entity with the new path - based on the output path specified in the watermark action.This reduces the usability of the watermark entity. If parallel workflow runs for the same entity and subsequent steps port watermark actions are dependent on the output of the watermark action we cannot do a select * from the table. Instead we have to use the output path and load the data into a temporary table and then use the data. This is an additional overhead for data pipeline workflows.We need this fixed in the product so that it becomes more user friendly

Support Ticket PS-18869
  • Attach files
  • Chawjyoti Buragohain
    Aug 27, 2020

    Hi Team,

    I see that it is mentioned as "Will not implement". Can you please share more details on this decision.