tf.keras.preprocessing.text.text_to_word_sequence
Converts a text to a sequence of words (or tokens).
tf.keras.preprocessing.text.text_to_word_sequence(
input_text, filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n', lower=True, split='
'
)
This function transforms a string of text into a list of words while ignoring filters which include punctuations by default.
sample_text = 'This is a sample sentence.'
tf.keras.preprocessing.text.text_to_word_sequence(sample_text)
['this', 'is', 'a', 'sample', 'sentence']
| Arguments | ||
|---|---|---|
input_text |
Input text (string). | |
filters |
list (or concatenation) of characters to filter out, such as punctuation. Default: '!"#$%&()*+,-./:;<=>?@[\]^_{|}~\t\n', includes basic punctuation, tabs, and newlines. </td> </tr><tr> <td>lower</td> <td> boolean. Whether to convert the input to lowercase. </td> </tr><tr> <td>split` |
str. Separator for word splitting. |
| Returns | |
|---|---|
| A list of words (or tokens). |
© 2020 The TensorFlow Authors. All rights reserved.
Licensed under the Creative Commons Attribution License 3.0.
Code samples licensed under the Apache 2.0 License.
https://www.tensorflow.org/versions/r2.3/api_docs/python/tf/keras/preprocessing/text/text_to_word_sequence