Zaloni Ideas
Add a new idea Subscribe
Status Will not implement
Categories Data Ingestion
Created by Ajinkya Rasam
Created on Apr 3, 2020

RELATED IDEAS

Improve DB Import design

Hi,


In DB import action we have an option to import all tables in DB. When this option is selected ZDP creates SQOOP command for every single table and executes a MR job for each table. Meaning, if there are 1000 tables in db then ZDP will fire 1000 sqoop commands. This is bad design, Sqoop provides an option to import all tables in database at once. Maybe we can utilize this:

sqoop-import-all-tables

https://sqoop.apache.org/docs/1.4.1-incubating/SqoopUserGuide.html

  • PRODUCT MANAGEMENT RESPONSE
    Apr 8, 2020

    We are considering Spark as the engine for RDBMS Ingestion going forward with controls for performance optimization.

  • Attach files