Ingestion history captured by File Move Action does not capture all attributes needed by customer. The necessary attributes are source file name, file size, file type & associated entity. We should revisit and verify the completeness of teleme...
almost 3 years ago
in Data Ingestion
On the Backlog
1. As a data miner that isresponsible for bringing data in the the ZDP from a variety of sources, but especially from files, 2. I want the ZDP system to assist me in assuring that the Destination Path specified in the File Pattern matches the Loca...
Today if the customer has to sync the metadata from Hcatalog/Hive, they have run hive inventory action (which is sub-optimal in performance) per schema. Request is to be able to automatically scan hcatalog and catalog the changes/new tables.
With reference to 9154: Teradata char /varchar changed to string by default in hive using db import
DB Import action is being used for importing tables from Teradata. Sqoop, which is the internal implementation of DB Import action, converts the datatype VARCHAR to the corresponding Hive type STRING and hence the corresponding ZDP entity field is...
Customer wants to remove the middle man (eg. ingestion warden) in way that BDCA agents can be spawned directly on landing zone server and joins the ingestion cluster. File patterns and other ingestion definition etc. can be further defined by the ...
about 3 years ago
in Data Ingestion
Should be able to run Data Profiling as next step directly after ADI
customer is trying to run DP after ADI which they are unable to do in a single workflow. We currently do not have any provision to read the content of the "[STEP-NAME]_DI_FILE_WITH_REGISTERED_ENTITIES" file generated via ADI directly in the Data p...