# Supporting new dtypes in PyTorch
**URL:** https://dev-discuss.pytorch.org/t/supporting-new-dtypes-in-pytorch/1833
**Category:** Uncategorized
**Created:** [January 24, 2024, 7:03pm UTC](https://dev-discuss.pytorch.org/t/supporting-new-dtypes-in-pytorch/1833 "2024-01-24T19:03:16Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@jerryzh168](https://dev-discuss.pytorch.org/u/jerryzh168)
#### Post date: [January 24, 2024, 7:03pm UTC](https://dev-discuss.pytorch.org/t/supporting-new-dtypes-in-pytorch/1833/1 "2024-01-24T19:03:16Z")
tldr; This post explains what adding a new dtype to PyTorch core means, the criteria of adding a new dtype to PyTorch core and the official recommendation of how to support new “secondary dtypes” use cases like (group) quantized uint4 etc. that will be backed up by native dtypes like uint4.
# Context
We had some recent discussions for adding int4 dtypes in PyTorch with our customers, main sentiment from our users is that it’s becoming pretty popular and multiple people/teams has requested this both internally at meta and in OSS. However, all existing cases can be served and it’s not clear the exact benefit of adding a new dtype in PyTorch, and we do not have a criteria for features that need to be supported for a native dtype in PyTorch. This post plans to answer some of the common questions around how we think about adding new dtypes to PyTorch (the same reasoning applies to other core constructs like device as well).
# 1. Current native dtypes in PyTorch
See: [pytorch/c10/core/ScalarType.h at main · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/main/c10/core/ScalarType.h) and [torch.Tensor — PyTorch 2.1 documentation](https://pytorch.org/docs/stable/tensors.html)
Here are the existing dtypes for torch.Tensor
- float32, float64, float16, bfloat16, float8\_e5m2, float8\_e4m3fn
- complex32, complex64, complex128 (historical reasons)
- uint1 to uint7, uint8, int8, int16, int32, int64, uint16, uint32, uint64
- bool
- quint8, qint8, qint32, quint4x2, quint2x4
- Uninterpreted bits types (bits8, bits4x2, bits2x4, bits1x8)
Some notes
- We have some requests for adding uint16 from quantization customers, but the previous recommendation has been please use int32 and quant\_min/quant\_max to [simulate](https://github.com/pytorch/pytorch/blob/main/test/quantization/fx/test_quantize_fx.py#L3225) it, but now we have [uint16](https://github.com/pytorch/pytorch/pull/116594) in PyTorch core with some barebone support, it should be enough for quantization use case
- Not all dtypes have the same support for all operators/features, (e.g. complex dtypes) but it’s not clear how well each dtype is supported
# 2. What are the features that need to be supported for a native PyTorch dtype?
- Basic Tensor support
- Feature/Composability (might have some overlap with others)
- Other Costs
# 3. What is the criteria for adding a new dtype to PyTorch core?
- We can predict future wide usage of a dtype, if the dtype will be supported in Silicon on major accelerator hardware we support (e.g., if the next generation of NVIDIA GPUs is going to natively support a format, this is highly predictive of it being widely used.)
- The dtype must be meaningful without any extra metadata. For example, fp8 has a well defined interpretation without a scaling factor, and so can be a float. any4 is only defined with a 2^4 lookup table mapping int values to float values, and so cannot be a dtype in the traditional sense. Another way to think about it: torch.tensor([0], dtype=your\_dtype) must be meaningful; if it is not meaningful, you don’t have a dtype.
# 4. What is enabled by having a native PyTorch dtype
- Reduced friction for user
- Simplified packaging and dependency
- Better native integration with hardwares
# 5. Should we define an official “secondary dtype” path that doesn’t meet criteria in 3.?
There are use cases like group quantized uint4, any4, mx that need extra metadata to make sense and we might see more of these coming as quantization becomes increasingly popular among LLM, but these did not meet the criteria in 3. The official extension point we recommend for these use cases will be Tensor subclass, or look into adding extension points to the existing systems, for example, if we want to build a quantized uint4 Tensor based on the native torch.uint4 dtype, we could do the following:
```auto
class QuantizedUInt4Tensor(torch.Tensor):
...
```
As long as you are relying on supported operators/features on the underlying dtype and implemented the Tensor subclass correctly, we expect it to work automatically with all existing systems like dynamo, torch.compile etc. We’ll have a separate post about official support for the new dtypes like uint4 after we validate them with some use cases. For now please wait for these docs about feature support for each dtype (e.g. uint4) before trying to implement “secondary dtypes” backed up by these native dtypes.
A more complete example can be found [here](https://github.com/pytorch-labs/ao/pull/13)
# Appendix: Some case studies
- qint8 - despite being in ScalarType, this is NOT a good dtype, because it requires extra metadata to be interpreted. Stuff like torch.ones(N, dtype=torch.quint8) doesn’t work! We should not have added it.
- float8\_e5m2 and other variants - these are OK to add because they have a meaningful interpretation without extra metadata, and they have Silicon support in NVIDIA H100. Unlike classic dtypes, they have limited operator support.
- uint16, uint32, uint64 - these are extremely well known (e.g., Numpy supports them) and we have added support for them in PyTorch. However, they have limited operator support for binary size reasons (but we expect PT2 to be able to deal with the coverage gap.
- uint1,2,3,4,5,6,7 - these are OK to add because sub-byte dtypes are reasonably popular and it is useful to have native support for sub-byte size in PyTorch core to ease arithmetic. These are quite difficult to implement in C++ so we will only have Python support in the mid term. You’ll use these as the basis for other sub-byte formats. Sub-byte is a bit difficult to implement.
- [mx4](https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf) this is NOT a good dtype as the microexponents need to be stored in some way so that you can interpret the numbers. OK to have f8\_e8m0 as a type to represent the exponents though
---
### Author:  [@tringwald](https://dev-discuss.pytorch.org/u/tringwald)
#### Post date: [January 24, 2024, 8:41pm UTC](https://dev-discuss.pytorch.org/t/supporting-new-dtypes-in-pytorch/1833/2 "2024-01-24T20:41:18Z")
Thanks for the great summary. What are the current thoughts on adding more complex dtypes? cuFFT 12.3, for example, now offers a [`CUDA_C_16BF` dtype](https://docs.nvidia.com/cuda/cufft/index.html), which would be equivalent to our current complex32, but with bfloat16s instead of float16s.
Has there been any discussion about how we could make it easier to add more complex types? Right now, it seems to me that every new dtype (bfloat16, maybe float8 in the future?) requires their own complex equivalent. Maybe we should make the complex dtype more composable for future subtypes.
---
### Author:  [@ezyang](https://dev-discuss.pytorch.org/u/ezyang)
#### Post date: [January 25, 2024, 1:22am UTC](https://dev-discuss.pytorch.org/t/supporting-new-dtypes-in-pytorch/1833/3 "2024-01-25T01:22:38Z")
This is not so much a plan as it is an intuition, since no one is signed up to implement it, but complex numbers should get turned into a wrapper subclass tensor on top of a base dtype like float32/bfloat16, which would solve the compositionality problem. There are some downsides to doing it this way, but a lot of upside in the PT2 world too