# Summation bfloat16

**URL:** <https://dev-discuss.pytorch.org/t/summation-bfloat16/390>\
**Category:** documentation\
**Created:** [October 29, 2021, 3:52pm UTC](https://dev-discuss.pytorch.org/t/summation-bfloat16/390 "2021-10-29T15:52:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dwidemann-lm](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/dwidemann-lm/32/297_2.png) [@dwidemann-lm](https://dev-discuss.pytorch.org/u/dwidemann-lm)\
**Post date:** [October 29, 2021, 3:52pm UTC](https://dev-discuss.pytorch.org/t/summation-bfloat16/390/1 "2021-10-29T15:52:44Z")

</div>

Hi,

I’m seeing that torch’s summation of a bfloat16 tensor is more accurate than my naive summation, e.g. x.sum() is better than x[0] +…+ x[n-1]. Could somebody please point me towards where in the repo is the code for torch’s summation? Thanks!

---

<div class="post-metadata">

**Author:** ![tom](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/tom/32/26_2.png) [@tom](https://dev-discuss.pytorch.org/u/tom)\
**Post date:** [October 30, 2021, 6:18pm UTC](https://dev-discuss.pytorch.org/t/summation-bfloat16/390/2 "2021-10-30T18:18:47Z")

</div>

sum is executed through TensorIterators, so it is a bit of a complex setup, this [bit of code in ReduceOps.cpp](https://github.com/pytorch/pytorch/blob/b24c34426f47d6c311ad80ebba1d2575e6c7a6aa/aten/src/ATen/native/ReduceOps.cpp#L978) plays a role.

The cause of your accuracy observation is likely that operations on [16 bit floats including bfloat16 typically use 32 bit floats as the internal computation (“accumulation”) scalar type](https://github.com/pytorch/pytorch/blob/b24c34426f47d6c311ad80ebba1d2575e6c7a6aa/aten/src/ATen/AccumulateType.h#L58-L70).

Best regards

Thomas

---

<div class="post-metadata">

**Author:** ![dwidemann-lm](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/dwidemann-lm/32/297_2.png) [@dwidemann-lm](https://dev-discuss.pytorch.org/u/dwidemann-lm)\
**Post date:** [November 1, 2021, 1:10am UTC](https://dev-discuss.pytorch.org/t/summation-bfloat16/390/3 "2021-11-01T01:10:16Z")

</div>

Thank you for showing me that. Yes, it’s the FP32 accumulator that improves the accuracy.

---

<div class="post-metadata">

**Author:** ![Lezcano](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/lezcano/32/300_2.png) [@Lezcano](https://dev-discuss.pytorch.org/u/Lezcano)\
**Post date:** [November 4, 2021, 12:34pm UTC](https://dev-discuss.pytorch.org/t/summation-bfloat16/390/4 "2021-11-04T12:34:10Z")

</div>

That is one part of the story. The other part is that adding lots of numbers in a stable way is not a trivial task! PyTorch implements a fairly involved algorithm based on reducing parts of the tensor and then adding the results of those reductions.

The code is here: [pytorch/SumKernel.cpp at bceb1db885cafa87fe8d037d8f22ae9649a1bba0 · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/bceb1db885cafa87fe8d037d8f22ae9649a1bba0/aten/src/ATen/native/cpu/SumKernel.cpp)

A fantastic and up to date talk on the topic with an algorithm similar (actually better) than the one that PyTorch currently implements is here: [Talk by Nicholas J. Higham (University of Manchester) - YouTube](https://www.youtube.com/watch?v=B1eFGn5nN84)

---

<div class="post-metadata">

**Author:** ![dwidemann-lm](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/dwidemann-lm/32/297_2.png) [@dwidemann-lm](https://dev-discuss.pytorch.org/u/dwidemann-lm)\
**Post date:** [November 4, 2021, 6:07pm UTC](https://dev-discuss.pytorch.org/t/summation-bfloat16/390/5 "2021-11-04T18:07:41Z")

</div>

@Lezcano Thank you for sending this. I’ll check it out.