Describe the bug
It looks like xxHash is installed with lighteval through the dependency on the datasets library, however the latest release of xxHash (4.0) is not compatible with the latest release of lighteval (0.13).
Although xxhash is not listed as direct requirement of lighteval, it is directly imported by lighteval.
Lighteval relies on a now-unsupported feature, which is passing string directly for hashing.
On xxHash side, this raises:
TypeError: Strings must be encoded before hashing
I would recommend fixing this issue as well as listing xxhash as direct dependency.
To Reproduce
from lighteval.logging.info_loggers import DetailsLogger
from lighteval.models.model_output import ModelResponse
from lighteval.tasks.requests import Doc
details_logger = DetailsLogger()
details_logger.log(
task_name="reproducer|0",
doc=Doc(query="xxhash compatibility reproducer", choices=[], gold_index=0),
model_response=ModelResponse(input_tokens=[1], output_tokens=[[2]]),
metrics={})
print("LightEval detail logging completed successfully.")If it could help with the fix, this is the traceback of the error when running our test suite:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/brevitas_examples/test_llm.py:146: in wrapper_main
results, model = quantize_llm(args, extra_args=extra_args)
src/brevitas_examples/llm/main.py:747: in quantize_llm
few_shot_eval_results = run_lighteval(
src/brevitas_examples/llm/eval_lighteval.py:277: in run_lighteval
pipeline.evaluate()
.nox/tests_brevitas_examples_llm_lighteval-3-10-jit_disabled-pytorch_2-7-1/lib/python3.10/site-packages/lighteval/pipeline.py:291: in evaluate
self._compute_metrics(outputs)
.nox/tests_brevitas_examples_llm_lighteval-3-10-jit_disabled-pytorch_2-7-1/lib/python3.10/site-packages/lighteval/pipeline.py:399: in _compute_metrics
self.evaluation_tracker.details_logger.log(task_name, doc, response, output)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = DetailsLogger(hashes=defaultdict(<class 'list'>, {}), compiled_hashes=defaultdict(<class 'lighteval.logging.info_logge...er_all_tasks=DetailsLogger.CompiledDetailOverAllTasks(hashes={}, truncated=0, non_truncated=0, padded=0, non_padded=0))
task_name = 'winogrande|0'
doc = {"query": "People think ", "choices": ["Samantha is embarassed, because Samantha made snide comments about the shirt R...ll, "generation_size": -1, "stop_sequences": ["\n"], "use_logits": false, "num_samples": 1, "generation_grammar": null}
model_response = ModelResponse(input=None, input_tokens=tensor([ 1, 11647, 1348]), text=[], output_tokens=tensor([[29871, 3685, 9...gits_eq_gold=[False, False], logits=None, unconditioned_logprobs=None, truncated_tokens_count=0, padded_tokens_count=0)
metrics = {'acc': 1}
def log(
self,
task_name: str,
doc: Doc,
model_response: ModelResponse,
metrics: dict,
) -> None:
"""Stores the relevant information for one sample of one task to the total list of samples stored in the DetailsLogger.
Args:
task_name (str): Name of the current task of interest.
doc (Doc): Current sample that we want to store.
model_response (ModelResponse): Model outputs for the current sample
metrics (dict): Model scores for said sample on the current task's metrics.
"""
detail = self.Detail(doc, model_response, metrics)
self.details[task_name].append(detail)
hash = self.Hash()
> hash.example = xxhash.xxh64(doc.query).hexdigest()
E TypeError: Strings must be encoded before hashing
Expected behavior
Compatibility between lighteval and xxhash or alternative hashing solution.
Version info
lighteval==0.13
Describe the bug
It looks like xxHash is installed with lighteval through the dependency on the
datasetslibrary, however the latest release of xxHash (4.0) is not compatible with the latest release of lighteval (0.13).Although xxhash is not listed as direct requirement of lighteval, it is directly imported by lighteval.
Lighteval relies on a now-unsupported feature, which is passing string directly for hashing.
On xxHash side, this raises:
TypeError: Strings must be encoded before hashingI would recommend fixing this issue as well as listing xxhash as direct dependency.
To Reproduce
If it could help with the fix, this is the traceback of the error when running our test suite:
Expected behavior
Compatibility between lighteval and xxhash or alternative hashing solution.
Version info
lighteval==0.13