IBM Data Platform Ideas Portal for Customers
This portal is to open public enhancement requests against products and services offered by the IBM Data Platform organization. To view all of your ideas submitted to IBM, create and manage groups of Ideas, or create an idea explicitly set to be either visible by all (public) or visible only to you and IBM (private), use the IBM Unified Ideas Portal (https://ideas.ibm.com).
Shape the future of IBM!
We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:
Search existing ideas
Start by searching and reviewing ideas and requests to enhance a product or service. Take a look at ideas others have posted, and add a comment, vote, or subscribe to updates on them if they matter to you. If you can't find what you are looking for,
Post your ideas
Post ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,
Post an idea
Upvote ideas that matter most to you
Get feedback from the IBM team to refine your idea
Specific links you will want to bookmark for future use
Welcome to the IBM Ideas Portal (https://www.ibm.com/ideas) - Use this site to find out additional information and details about the IBM Ideas process and statuses.
IBM Unified Ideas Portal (https://ideas.ibm.com) - Use this site to view all of your ideas, create new ideas for any IBM product, or search for ideas across all of IBM.
ideasibm@us.ibm.com - Use this email to suggest enhancements to the Ideas process or request help from IBM for submitting your Ideas.
IBM Employees should enter Ideas at https://ideas.ibm.com
Hi @Guest , I am happy to announce that that Avro Schema problem was already fixed in the 5.11 version. Our recommendation is to check the updates on these engines.
Hi @Guest , could you tell us what version of engines is using the client? because in Documentation, we can see that the Origins Schema can be taken and we have checked that it works from Origin Directory. Maybe they are using and older version and the fix is in a newer one.
https://docs.streamsets.com/portal/platform-datacollector/latest/datacollector/UserGuide/Origins/Directory.html?hl=directory%2Corigin
My thought for this is that if there were a schema that was nested with different types, writing it into parquetSchema would bypass the need to have to infer from the records and generate it.
Hi @Guest @Guest Could you please give us the information that Fran is requesting? it would really help us to resolve this for Simon. Thanks in advance
Hi. We really appreciate your comments and we'll take a careful look to it so you can successfully run your pipeline.
As you say, the parquet schema should be extracted right away from the parquet file's metadata and most probably is the case. Anyways seems to be an error there.
I'd like to take a deeper look but I'd need some more information.
I understand that you have one pipeline to generate parquet files and then a second pipeline to ingest them into some data lake. Am I right?
If this is the case, in the second pipeline which exact origin you are using for reading parquet files.
Looking forward to hearing from you.
Best regards.