Ideas
Status On the Backlog
Categories Transformations
Created by Gary Asmus
Created on Oct 23, 2019

Convert fixed-width data to columnar format

1. As a part of the big data team I am responsible for ingesting file data into the lake. Many of the files that we receive are fixed length files. It is difficult to process these files within the lake so I generally create a delimited version of these entities.

2. I want a much easier way to perform this conversion. Basically, I perform a transformation on these files even though the fixed length files are not really supported by the transformation. There doesn't appear to be any way to perform this conversion via workflows. Basically, as long as I select the fixed length file when I first add the Load object to the transformation surface, I can then basically just add an Output object where I can create a new entity that has the same fields as the input entity.

3. So that I don't have to delete the load step when creating this conversion because when I go back to the Load object to select a different entity to be loaded it show me a list of entities that are ONLY delimited. I can delete the Load object and drag a new one which will then show me a list of ALL entities including Fixed Length entities. This appears to be the only way that I can do anything useful with data that starts from a fixed width file. Ideally, I'd like a conversion object that I could use to just grab a Fixed Length entity and easily output a "copy" of the entity expect that it is in a delimited format.

4. Acceptance Criteria:
One step conversion process from fixed length entity to a delimited (or other) entity.

5. Business Justification:
I have many fixed length files. We are in healthcare and this field has been exchanging data for many years and many of these exchanges involved the older fixed length file formats.

Support Ticket PS-18852
Customer Impact Nice to have
  • Attach files
  • kikeyye cici
    Jan 22, 2021

    Censtry Electronics is a Global Electronics Components Distributor, our website is an online one-stop sourcing platform. We supply Electronic Components with best price, new original in stock & PDF Datasheet.