Skip to content

fasttext quantization documentation is confusing/misleading #108

Description

@yrro

https://docs.rspamd.com/modules/neural/ says

# Download and extract a full model
wget https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz
gunzip cc.en.300.bin.gz

# Quantize to reduce size and dimensions
./fasttext quantize -input cc.en.300 -output cc.en.300.q

but:

$ fasttext quantize -input cc.en.300 -output cc.en.300.q
terminate called after throwing an instance of 'std::invalid_argument'
  what():  cc.en.300.q.bin cannot be opened for loading!
Aborted (core dumped)

This is with

ii  fasttext       0.9.2+ds-7build1 arm64        Efficient learning of word representations and sentence classification library

My assumption is that the CLI changed since the docs were written. But then this version of fasttext is 6 years old. So maybe the docs are just totally wrong?

https://www.facebook.com/groups/1174547215919768/posts/1668216449886173/ lead me to try:

$ fasttext quantize -input cc.en.300 -output cc.en.300
terminate called after throwing an instance of 'std::invalid_argument'
  what():  For now we only support quantization of supervised models
Aborted (core dumped)

... research this leads to Gemini saying stuff like "Because cc.en.300.bin is a set of raw word vectors (unsupervised), it lacks the classification headers required by the quantize utility." and at this point I have to throw my hands up and admit I have no earthly idea what any of this means!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions