The Challenge of Reproducible LLM Inference: Why Even Greedy Sampling Isn't Deterministic

Reproducibility is a bedrock of scientific progress. However, it’s remarkably difficult to get reproducible results out of large language models. For example, you might observe that asking ChatGPT the same question multiple times provides different result