diff options
Diffstat (limited to 'textproc/py-tokenizer')
| -rw-r--r-- | textproc/py-tokenizer/Makefile | 1 | ||||
| -rw-r--r-- | textproc/py-tokenizer/pkg-descr | 14 |
2 files changed, 5 insertions, 10 deletions
diff --git a/textproc/py-tokenizer/Makefile b/textproc/py-tokenizer/Makefile index 4f8afff7b8be..b4ad88c9c8d9 100644 --- a/textproc/py-tokenizer/Makefile +++ b/textproc/py-tokenizer/Makefile @@ -1,5 +1,6 @@ PORTNAME= tokenizer PORTVERSION= 3.5.0 +PORTREVISION= 1 CATEGORIES= textproc python MASTER_SITES= PYPI PKGNAMEPREFIX= ${PYTHON_PKGNAMEPREFIX} diff --git a/textproc/py-tokenizer/pkg-descr b/textproc/py-tokenizer/pkg-descr index 665fa0186f94..c1f700edffe5 100644 --- a/textproc/py-tokenizer/pkg-descr +++ b/textproc/py-tokenizer/pkg-descr @@ -1,11 +1,5 @@ -This python utility package helps to create lazy modules. A lazy module defers -loading (some of) its attributes until these attributes are first accessed. The -module's lazy attributes in turn are attributes of other modules. These other -modules will be imported/loaded only when (and if) associated attributes are -used. A lazy import strategy can drastically reduce runtime and memory -consumption. +Tokenizer: A tokenizer for Icelandic text -Additionally, this package provides a utility for optional imports with which -one can import a module globally while triggering associated import errors only -at use-sites (when and if a dependency is actually required, for example in the -context of a specific functionality). +Tokenization is a necessary first step in many natural language processing +tasks, such as word counting, parsing, spell checking, corpus generation, and +statistical analysis of text. |