Calculate uniqueness inter and intra cluster records to measure the correctness of DME output
As with the Spark v 2.3.0 or later version, has a API which can calculate measure of Clustering Prediction Score - It will nice to integrate this in our DME plugin as a confidence scoring model out of box.More details about the Spark API :https://...
Ability to parse multi level complex json file in ZDP
Current json serde unable to parse multi level json. can we add openx serde which is available as open source. details: open source link : https://github.com/rcongiu/Hive-JSON-Serdejar files link: http://www.congiu.net/hive-json-serde/1.3.8/cdh5/
Need ability to have automated schema mapping of data being ingested
PS has implemented a mechanism of schema mapping of the data being ingested. This allows columns to be not fixed within a data-file and it gets assigned to the right position within the Hive table at run-time. It uses column-headers to determine t...
about 2 years ago
in Data Ingestion
Provide ability to comment and crowdsource business information on ZDP entities/fields.
Business user/Data Stewards would like to collaborate and share their comments and findings on the data sets. They would like review these comments before underlying data can be provisioned to downstream systems. Provide ability to capture user fe...
FDQ Reporting should have more feature to generated selective data
We have observed that when we run FDQ for multiple rows with multiple rules, the report generated is huge.We have been reported by a client that for a file of size about 1.5GB and 2.5 crore records, the fdq report generated was of about 350GB whic...
Display ingestion history for db wizard, db import created entities in entity view ingestion history tab when Display is 'Ingested File Size Per Day'
Display ingestion history for db wizard, db import created entities in entity view ingestion history tab when Display is 'Ingested File Size Per Day'Current behavior:The 'Ingested File Size Per Day' is shown only for entities associated with file ...
Ability to auto tag entities based on the vocabulary/business glossary
Adding labels to the entity is tedious and time consuming effort. provide capability to auto tag based on the business glossary. or by referring business vocabulary. some of the competitors are leveraging modified Maui - Multi-purpose automatic to...
Hi,In Arena we can input Good, bad and report entity data path. If a user clicks Entity name text box after entering a custom value in entity path text box then the value in entity path text box is reset to /target schema/entity nameThis behavior ...
Enable data profiling on Hive Views from profile of underlying Tables
Enable data profiling on views in Zaloni via linking with table instead of physically creating data profilingFrom customer: Views should not have their own profiling but profile information should come from the original table profiled from where t...
about 2 years ago
On the Backlog
Hi,As a user I need an ability to change the connection associated to a Landing zone.Currently we only allow view and delete of the source directory. https://internal.docs.zaloni.com/6.2.0/ingestion/file_view/adding_a_source_directory.htm