๐ŸŽฐ Deterministic flash-attn

June 17, 2024

[NOTE]: For additional details, refer to the W&B Report.

Simple tests to confirm the loss is exactly reproducible across independent runs (when launched with the same seed).

Figure 1: Plot of the loss curve for 3 independent runs with deterministic=True
No matching items
Back to top

Citation

BibTeX citation:
@online{foreman2024,
  author = {Foreman, Sam},
  title = {Personal {Website}},
  date = {2024-06-17},
  url = {https://samforeman.me},
  langid = {en}
}
For attribution, please cite this work as:
Foreman, Sam. 2024. โ€œPersonal Website.โ€ June 17, 2024. https://samforeman.me.