Skip to Main Content
IBM Data and AI Ideas Portal for Customers


Shape the future of IBM!

We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:

Post your ideas

Post ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,

  1. Post an idea

  2. Upvote ideas that matter most to you

  3. Get feedback from the IBM team to refine your idea

Help IBM prioritize your ideas and requests

The IBM team may need your help to refine the ideas so they may ask for more information or feedback. The product management team will then decide if they can begin working on your idea. If they can start during the next development cycle, they will put the idea on the priority list. Each team at IBM works on a different schedule, where some ideas can be implemented right away, others may be placed on a different schedule.

Receive notification on the decision

Some ideas can be implemented at IBM, while others may not fit within the development plans for the product. In either case, the team will let you know as soon as possible. In some cases, we may be able to find alternatives for ideas which cannot be implemented in a reasonable time.

Additional Information

To view our roadmaps: http://ibm.biz/Data-and-AI-Roadmaps

Reminder: This is not the place to submit defects or support needs, please use normal support channel for these cases

IBM Employees:

The correct URL for entering your ideas is: https://hybridcloudunit-internal.ideas.aha.io


Status Delivered
Workspace Connectivity
Created by Guest
Created on Aug 6, 2019

Greenplum connector in DataStage: CSV ipv TEXT bij external tables

When in a field (varchar/text) of the source query (postgres) a carriage return exists then we can not use the Greenplum connector of Datastage.

Needed by Date Sep 1, 2019
  • Admin
    Virginie Grandhaye
    Nov 30, 2020

    This enhancement development is now complete. It is now available as a patch (https://www.ibm.com/support/pages/apar/JR62978)

    You can open a support ticket from there https://www.ibm.com/support/home/ to get access to the patch download.

  • Guest
    Nov 18, 2020

    Dear Virginie,

    Thanks a lot for the great news ! We're looking forward to it.

    Any idea about timing ?

    Kind regards,

    Philippe

  • Admin
    Virginie Grandhaye
    Nov 18, 2020

    Dear Philippe,

    We are currently working on this enhancement request. It should thus be part of the next release of the product.

    Have a great day.

  • Guest
    Nov 17, 2020

    Hi Virginie,

    Any news about the implementation of this idea ?

    Kind regards,

    Philippe

  • Guest
    Aug 4, 2020

    Hi Virginie,

    For the Vlaamse Milieumaatschappij it remains a very important improvement of the Greenplum Connector in DataStage.

    The problem can be described as follow:

    When a newline (Linefeed of carriage return+linefeed) is present in a text/varchar column on the source query, the load procedure using the Greenplum Connector aborts.

    This is caused by the command used to create the external table: the connector uses by default the format TEXT and we cannot specify the QUOTE character. Therefore the <newline> character in the varchar column is considered as the end-of-line delimiter, and not as part of the varchar column.

    CREATE [READABLE] EXTERNAL TABLE table_name     
    ( column_name data_type [, ...] | LIKE other_table )
         LOCATION ('file://seghost[:port]/path/file' [, ...])
           | ('gpfdist://filehost[:port]/file_pattern[#transform=trans_name]'
    [, ...]
           | ('gpfdists://filehost[:port]/file_pattern[#transform=trans_name]'
               [, ...])
           | ('gphdfs://hdfs_host[:port]/path/file')
           | ('pxf://path-to-data?PROFILE[&custom-option=value[...]]'))
           | ('s3://S3_endpoint[:port]/bucket_name/[S3_prefix]
    [region=S3-region]
    [config=config_file]')
         [ON MASTER]
         FORMAT 'TEXT'
               [( [HEADER]
                  [DELIMITER [AS] 'delimiter' | 'OFF']
                  [NULL [AS] 'null string']
                  [ESCAPE [AS] 'escape' | 'OFF']
                  [NEWLINE [ AS ] 'LF' | 'CR' | 'CRLF']
                  [FILL MISSING FIELDS] )]
              | 'CSV'
               [( [HEADER]
                  [QUOTE [AS] 'quote']
                  [DELIMITER [AS] 'delimiter']
                  [NULL [AS] 'null string']
                  [FORCE NOT NULL column [, ...]]
                  [ESCAPE [AS] 'escape']
                  [NEWLINE [ AS ] 'LF' | 'CR' | 'CRLF']
                  [FILL MISSING FIELDS] )]
              | 'AVRO'
    | 'PARQUET'
              | 'CUSTOM' (Formatter=<formatter_specifications>)
        [ ENCODING 'encoding' ]
        [ [LOG ERRORS] SEGMENT REJECT LIMIT count
          [ROWS | PERCENT] ]

    Workaround:

    As a workaround we replaced the <newline> character in the source query by a string we hope will not be used elsewhere, and substitute it back to the <newline> after the load.

    But it takes a lot of time and effort to implement, and is error prone.

    Expected solution:

    The best solution would be to incorporate new parameters in the Greenplum connector to specify some parameters/options of the CREATE EXTERNAL TABLE statement or the "gpfdist" command, to be able to cover each and every case we could encounter.

    For example we could specify the FORMAT (TEXT/CSV) and with CSV be able to specify the QUOTE parameter to encapsulate the <newline>. (see the parameter in green)

    Don't hesitate to contact me if something is not clear or if you need more information.

  • Guest
    Aug 4, 2020

    We still consider this a high priority.

  • Admin
    Virginie Grandhaye
    Aug 4, 2020

    Thanks for submitting this idea. This was submitted last year, and tagged as urgent, so I'd need to know if you could finally proceed with the ODBC option mentioned as a workaround, or if you still consider this one as a high priority on your side.

    Thanks for any details you can provide.