Problem Statement: [Ingestion SLA]
Currently, there are approx 60+ sources (File Patterns) configured for ingestion via Bedrock in customer Prod env. Each of these data files has a different frequency/schedule (Daily/Weekly/Monthly) along with a time window (e.g. File X should get delivered between 7:00pm to 7:30pm each day) of arrival onto the Landing Zone. Each of these data files is configured with Post Ingestion workflow for ingesting and processing from Raw -> Normalized zone for consumption by the downstream applications. There are instances where some of these file(s) doesn't get delivered to Landing Zone and usually business figures out gaps in data while querying.