diff options
Diffstat (limited to 'textproc/py-tokenizer/pkg-descr')
| -rw-r--r-- | textproc/py-tokenizer/pkg-descr | 14 |
1 files changed, 4 insertions, 10 deletions
diff --git a/textproc/py-tokenizer/pkg-descr b/textproc/py-tokenizer/pkg-descr index 665fa0186f94..c1f700edffe5 100644 --- a/textproc/py-tokenizer/pkg-descr +++ b/textproc/py-tokenizer/pkg-descr @@ -1,11 +1,5 @@ -This python utility package helps to create lazy modules. A lazy module defers -loading (some of) its attributes until these attributes are first accessed. The -module's lazy attributes in turn are attributes of other modules. These other -modules will be imported/loaded only when (and if) associated attributes are -used. A lazy import strategy can drastically reduce runtime and memory -consumption. +Tokenizer: A tokenizer for Icelandic text -Additionally, this package provides a utility for optional imports with which -one can import a module globally while triggering associated import errors only -at use-sites (when and if a dependency is actually required, for example in the -context of a specific functionality). +Tokenization is a necessary first step in many natural language processing +tasks, such as word counting, parsing, spell checking, corpus generation, and +statistical analysis of text. |