SFTP Client Origin: Additional Offset Management Configuration Options

The current offset behavior of the SFTP Client origin is based on two checks after a file is processed:

The stage stores the filename and modification timestamp of the last processed file in the offset.

When new files arrive, the stage evaluates each file by:

Checking whether the file has a newer modification timestamp than the one stored in the offset.
Checking whether the filename matches the one stored in the offset.

A file will not be queued for processing if:

Its modification timestamp is older or equal to the one in the offset, or
Its filename matches the name stored in the offset.

As a result, if the same filename appears again with a newer timestamp, the stage still skips it because the filename matches the offset entry. This is expected behavior but prevents scenarios where customers intentionally reprocess the same file at regular intervals.

A customer has requested a configurable option that would allow the stage to prioritize modification timestamp over filename. This would effectively let the origin treat a file as new when its timestamp changes, even if the filename remains the same.

Needed By

Not sure -- Just thought it was cool

Post comment

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.

Please enter your email address

RELATED IDEAS

SFTP Client Origin: Additional Offset Management Configuration Options