tokenizer

Classes

tokenizer

String tokenization and transformation

class tokenizer.tokenizer

String tokenization and transformation

transform_text(input_string, batches=None, read_file_delimiter='\n', LOCALE_UTF='', to_lower=False, to_upper=False, language='english', REMOVE_characters='', remove_punctuation_string=False, remove_numbers=False, trim_token=False, split_string=False, separator=' \r\n\t., ;:()?!//', remove_punctuation_vector=False, remove_stopwords=False, min_num_char=1, max_num_char=9223372036854775807, stemmer=None, min_n_gram=1, max_n_gram=1, n_gram_delimiter=' ', skip_n_gram=1, skip_distance=0, stemmer_ngram=4, stemmer_gamma=0.0, stemmer_truncate=3, stemmer_batches=1, vocabulary_path=None, concat_delimiter=None, path_2folder='', threads=1, verbose=False)

Parameters:

• input_string – either a character string of length 1 or a character-string-path to a file (for big .txt files it’s recommended to use a path to a file)
• batches – a numeric value. If the batches parameter is not None then the object parameter should be a valid path to a file and the path_2folder parameter should be a valid path to a folder. The batches parameter should be used in case of small to medium data sets (for zero memory consumption). For big data sets the big_tokenize_transform class and especially the big_text_tokenizer method should be used.
• read_file_delimiter – the delimiter to use when the input file will be red (for instance a tab-delimiter or a new-line delimiter).
• LOCALE_UTF – the language specific locale to use in case that either the to_lower or the to_upper parameter is TRUE and the text file language is other than english. For instance if the language of a text file is greek then the utf_locale parameter should be ‘el_GR.UTF-8’ ( language_country.encoding ). A wrong utf-locale does not raise an error, however the runtime of the method increases.
• to_lower – either True or False. If True the character string will be converted to lower case
• to_upper – either True or False. If True the character string will be converted to upper case
• language –

  a character string which defaults to english. If the remove_stopwords parameter is True then the corresponding stop words vector will be uploaded. Available languages ‘afrikaans’,

  ’arabic’, ‘armenian’, ‘basque’, ‘bengali’, ‘breton’, ‘bulgarian’, ‘catalan’, ‘croatian’, ‘czech’,’danish’, ‘dutch’, ‘english’, ‘estonian’, ‘finnish’, ‘french’, ‘galician’, ‘german’, ‘greek’, ‘hausa’, ‘hebrew’, ‘hindi’, ‘hungarian’, ‘indonesian’, ‘irish’, ‘italian’, ‘latvian’, ‘marathi’, ‘norwegian’, ‘persian’, ‘polish’, ‘portuguese’, ‘romanian’, ‘russian’, ‘slovak’, ‘slovenian’, ‘somalia’, ‘spanish’, ‘swahili’, ‘swedish’, ‘turkish’, ‘yoruba’, ‘zulu’

• REMOVE_characters – a character string with specific characters that should be removed from the text file. If the remove_char is “” then no removal of characters take place
• remove_punctuation_string – either True or False. If True then the punctuation of the character string will be removed (applies before the split method)
• remove_numbers – either True or False. If True then any numbers in the character string will be removed
• trim_token – either True or False. If True then the string will be trimmed (left and/or right)
• split_string – either True or False. If True then the character string will be split using the separator as delimiter. The user can also specify multiple delimiters.
• separator – a character string specifying the character delimiter(s)
• remove_punctuation_vector – either True or False. If True then the punctuation of the vector of the character strings will be removed (after the string split has taken place)
• remove_stopwords – either True, False or a character vector of user defined stop words. If True then by using the language parameter the corresponding stop words vector will be uploaded.
• min_num_char – an integer specifying the minimum number of characters to keep. If the min_num_char is greater than 1 then character strings with more than 1 characters will be returned
• max_num_char – an integer specifying the maximum number of characters to keep. The max_num_char should be less than or equal to Inf (in this method the Inf value translates to a word-length of 1000000000)
• stemmer – a character string specifying the stemming method. One of the following porter2_stemmer, ngram_sequential, ngram_overlap.
• min_n_gram – an integer specifying the minimum number of n-grams. The minimum number of min_n_gram is 1.
• max_n_gram – an integer specifying the maximum number of n-grams. The minimum number of max_n_gram is 1.
• n_gram_delimiter – a character string specifying the n-gram delimiter (applies to both n-gram and skip-n-gram cases)
• skip_n_gram – an integer specifying the number of skip-n-grams. The minimum number of skip_n_gram is 1.
• skip_distance – an integer specifying the skip distance between the words. The minimum value for the skip distance is 0, in which case simple n-grams will be returned.
• stemmer_ngram – a numeric value greater than 1. Applies to both ngram_sequential and ngram_overlap methods. In case of ngram_sequential the first n characters will be picked, whereas in the case of ngram_overlap the overlapping stemmer_ngram characters will be build.
• stemmer_gamma – a float number greater or equal to 0.0. Applies only to ngram_sequential. Is a threshold value, which defines how much frequency deviation of two N-grams is acceptable. It is kept either zero or to a minimum value.
• stemmer_truncate – a numeric value greater than 0. Applies only to ngram_sequential. The ngram_sequential is modified to use relative frequencies (float numbers between 0.0 and 1.0 for the ngrams of a specific word in the corpus) and the stemmer_truncate parameter controls the number of rounding digits for the ngrams of the word. The main purpose was to give the same relative frequency to words appearing approximately the same on the corpus.
• stemmer_batches – a numeric value greater than 0. Applies only to ngram_sequential. Splits the corpus into batches with the option to run the batches in multiple threads.
• vocabulary_path_file – either None or a character string specifying the output path to a file where the vocabulary should be saved once the text is tokenized
• concat_delimiter – either None or a character string specifying the delimiter to use in order to concatenate the end-vector of character strings to a single character string (recommended in case that the end-vector should be saved to a file)
• path_2folder – a character string specifying the path to the folder where the file(s) will be saved
• threads – an integer specifying the number of cores to run in parallel
• verbose –

  either True or False. If True then information will be printed out

  Example:

  tok = tokenizer()
  
  res_tok = tok.transform_text(input_string = '/myfolder/file_text.txt', to_lower = True, trim_token = True, split_string = True)
  

transform_vec_docs(input_list, as_token=False, LOCALE_UTF='', to_lower=False, to_upper=False, language='english', REMOVE_characters='', remove_punctuation_string=False, remove_numbers=False, trim_token=False, split_string=False, separator=' \r\n\t., ;:()?!//', remove_punctuation_vector=False, remove_stopwords=False, min_num_char=1, max_num_char=9223372036854775807, stemmer=None, min_n_gram=1, max_n_gram=1, n_gram_delimiter=' ', skip_n_gram=1, skip_distance=0, vocabulary_path=None, concat_delimiter=None, path_2folder='', threads=1, verbose=False)

Parameters:

• input_list – a character string list of documents
• as_token – if True then the method will return a list of (split) token. Otherwise it will return a list of character strings (sentences)
• LOCALE_UTF – the language specific locale to use in case that either the to_lower or the to_upper parameter is TRUE and the text file language is other than english. For instance if the language of a text file is greek then the utf_locale parameter should be ‘el_GR.UTF-8’ ( language_country.encoding ). A wrong utf-locale does not raise an error, however the runtime of the method increases.
• to_lower – either True or False. If True the character string will be converted to lower case
• to_upper – either True or False. If True the character string will be converted to upper case
• language –

  a character string which defaults to english. If the remove_stopwords parameter is True then the corresponding stop words vector will be uploaded. Available languages ‘afrikaans’,

  ’arabic’, ‘armenian’, ‘basque’, ‘bengali’, ‘breton’, ‘bulgarian’, ‘catalan’, ‘croatian’, ‘czech’,’danish’, ‘dutch’, ‘english’, ‘estonian’, ‘finnish’, ‘french’, ‘galician’, ‘german’, ‘greek’, ‘hausa’, ‘hebrew’, ‘hindi’, ‘hungarian’, ‘indonesian’, ‘irish’, ‘italian’, ‘latvian’, ‘marathi’, ‘norwegian’, ‘persian’, ‘polish’, ‘portuguese’, ‘romanian’, ‘russian’, ‘slovak’, ‘slovenian’, ‘somalia’, ‘spanish’, ‘swahili’, ‘swedish’, ‘turkish’, ‘yoruba’, ‘zulu’

• REMOVE_characters – a character string with specific characters that should be removed from the text file. If the remove_char is “” then no removal of characters take place
• remove_punctuation_string – either True or False. If True then the punctuation of the character string will be removed (applies before the split method)
• remove_numbers – either True or False. If True then any numbers in the character string will be removed
• trim_token – either True or False. If True then the string will be trimmed (left and/or right)
• split_string – either True or False. If True then the character string will be split using the separator as delimiter. The user can also specify multiple delimiters.
• separator – a character string specifying the character delimiter(s)
• remove_punctuation_vector – either True or False. If True then the punctuation of the vector of the character strings will be removed (after the string split has taken place)
• remove_stopwords – either True, False or a character vector of user defined stop words. If True then by using the language parameter the corresponding stop words vector will be uploaded.
• min_num_char – an integer specifying the minimum number of characters to keep. If the min_num_char is greater than 1 then character strings with more than 1 characters will be returned
• max_num_char – an integer specifying the maximum number of characters to keep. The max_num_char should be less than or equal to Inf (in this method the Inf value translates to a word-length of 1000000000)
• stemmer – a character string specifying the stemming method. Available method is porter2_stemmer.
• min_n_gram – an integer specifying the minimum number of n-grams. The minimum number of min_n_gram is 1.
• max_n_gram – an integer specifying the maximum number of n-grams. The minimum number of max_n_gram is 1.
• n_gram_delimiter – a character string specifying the n-gram delimiter (applies to both n-gram and skip-n-gram cases)
• skip_n_gram – an integer specifying the number of skip-n-grams. The minimum number of skip_n_gram is 1.
• skip_distance – an integer specifying the skip distance between the words. The minimum value for the skip distance is 0, in which case simple n-grams will be returned.
• vocabulary_path_file – either None or a character string specifying the output path to a file where the vocabulary should be saved once the text is tokenized
• concat_delimiter – either None or a character string specifying the delimiter to use in order to concatenate the end-vector of character strings to a single character string (recommended in case that the end-vector should be saved to a file)
• path_2folder – a character string specifying the path to the folder where the file(s) will be saved
• threads – an integer specifying the number of cores to run in parallel
• verbose –

  either True or False. If True then information will be printed out

  Example:

  tok = tokenizer()
  
  res_tok = tok.transform_vec_docs(input_list = ['first word sentence', 'second word sentence'], as_token = True, to_lower = True, trim_token = True, split_string = True)
  

Note

It is memory efficient to read the data using a path file in case of a big file, rather than importing the data and then calling the tokenize method.

The utf_locale and split_string functionality is based on the boost library ( http://www.boost.org ).

It is memory efficient to specify a path_2folder in case that a big file should be saved, rather than return the vector of all character strings.

The skip-grams are a generalization of n-grams in which the components (typically words) need not to be consecutive in the text under consideration, but may leave gaps

that are skipped over. They provide one way of overcoming the data sparsity problem found with conventional n-gram analysis.

Stemming of the english language is done using the porter2-stemmer, for details see https://github.com/smassung/porter2_stemmer. N-gram stemming is language independent

and supported by the following two functions:

ngram_overlap : The ngram_overlap stemming method is based on N-Gram Morphemes for Retrieval, Paul McNamee and James Mayfield ( http://clef.isti.cnr.it/2007/working_notes/mcnameeCLEF2007.pdf )

ngram_sequential : The ngram_sequential stemming method is a modified version based on Generation, Implementation and Appraisal of an N-gram based Stemming Algorithm, B. P. Pande, Pawan Tamta, H. S. Dhami ( https://arxiv.org/pdf/1312.4824.pdf )

The list of stop-words in all available languages was downloaded from the following link https://github.com/6/stopwords-json