Toolkits > com.ibm.streamsx.nlp 1.9.2 > com.ibm.streamsx.nlp > DictionaryFilter
This operator checks the words of a text against a dictionary. Words are removed from the text if they are found in the dictionary and the filterMode remove is selected. The text will contain words of the dictionary only if filterMode keep is selected.
Required: dictionaryFile, textAttribute
Optional: commandAttribute, filterMode, ignoreCase, wordAttribute
Port that ingests tuples to be manipulated
The second input port is the command input port to clear and update the dictionary.
If the command input port exists, the commandAttribute parameter is mandatory. The command input port must have an rstring attribute with a value, also known as the command string, (see the commandAttribute parameter) and a rstring attribute that specifies the word (see the wordAttribute parameter).
Supported commands are
Port that outputs the modified tuples.
Required: dictionaryFile, textAttribute
Optional: commandAttribute, filterMode, ignoreCase, wordAttribute
Specifies the rstring attribute of the optional command input port that holds the command that is described in “Input Ports.”
If you specify a value for this parameter, the optional command input and status output ports are mandatory. If the optional command input port exists, this parameter is mandatory.
Specifies the dictionary file loaded on startup. If relative path is used, then root is application directory. It is recommended to store the file in etc directory.
Words in the dictionary file can be separated by space or new line.
Mode that determines which parts of text is forwarded. If filterMode remove is selected, then all words of the input stream textAttribute are removed in case they exist in the dictionary. This is the default value if filterMode parameter is not specified. In case the filterMode keep is selected, then those words of the textAttribute are forwarded that are stored in the dictionary only.
Specifies whether to perform a case sensitive dictionary match or not. The default value is true, meaning that a not case sensitive match is performed.
Specifies the text attribute of the input stream.
Specifies the rstring attribute of the optional command input port that holds the word as described in "Input Ports.”
If this parameter is specified, the optional command input and status output ports are mandatory. If this parameter is omitted and the command input port exists, the read and write commands are not supported.
stream<${schema}> ${outputStream} = DictionaryFilter(${inputStream}) {
param
filterMode : DictionaryFilter.remove;
textAttribute : text;
dictionaryFile : "etc/stopwords.csv";
output
${outputStream} : ${outputAttribute} = ${value};
}