Flops regularizer looks odd

[dangkhoasdc]

This implement does not look similar to the formula mentioned in SPLADE paper. Also, to minimize this, the 2nd operand need to be equal to threshold, which is not the goal of FLOPS.

neural-cherche/neural_cherche/losses/flops.py

Line 94 in 2df0214

input=torch.mean(input=torch.abs(input=activations), dim=0) ** 2, dim=0

There is another implementation more akin to the formula:

https://github.com/thongnt99/learned-sparse-retrieval/blob/d702026aacf1ab7c47011f55edcb2646a6bb646d/lsr/losses/regularizer.py#L56

[raphaelsty]

Hi @dangkhoasdc, I wrote this so it act as a margin-based flop loss, the model is asked to achieve a certain amount of flops.

Feel free to make a MR which will provide by default the correct flop loss :)