File inclusion and file exclusion filtering in crawler

The Dutch Ministry of Defense is using Watson Discovery for text document archiving purposes as well as text document analysis from these archives. They have experience with Watson Explorer Content Analytics and are migrating to Watson Discovery. They are facing serious issues with file system crawling and the inflexibility of the Watson Discovery file system crawler. The Ministry of Defense's analytics pipeline heavily relies on the filtering options available in WEX-AC (the include and exclude filtering). What they want in Watson Discovery is the ability to provide a file extension inclusion list (see 2_Crawler_inclusion_filter), just like they know from the Watson Explorer Content Analytics file system crawler. In essence, the inclusion list directs the crawler only to crawler files with a pre-defined set of extensions, while leaving the rest of the data it may encounter untouched. That would enable them to point the crawler to their data sources without the need of cleaning them prior to indexing. The reality with WD is different. What they face with WD is that they have to do tedious, time-consuming and labour-intensive data source cleaning and/or operations before they can ingest data from the sources. We've tried to point the WD file system crawler to these data sources, but we end up in 'black holes' of directories with filetypes that a) we don't want to index, b)time consuming operations to go through folders and zip-files containing non-supported file types. And actually, with the amounts of data the Defense Ministry wants to analyze, the systems becomes very unstable and unpredictable. The bad message is that because of the file system crawling issues experienced, the Ministry of Defense is contemplating to migrate backwards to WEX-AC.