Problem Statement: [Workflow Trigger Dependency]
Customer has multiple use cases where a transformation needs to be triggered based on the onboarding of its dependent datasets onto Data Lake. Here is an example-
- Transformation X needs to be triggered daily and it depends upon Dataset D1, D2, and D3 which gets ingested and loaded into Normalized zone daily as well.
- D1 comes during business hours (approx 10:00 am)
- D2 and D3 comes post business hours (approx 7:30 to 8:30)