This would be useful (and we are leveraging that) on a further stage, when we are actually lowering a FX node to ONNX.
However, the problem above arises before the actual lowering. In order to implement the ONNX operator registry, we need to know what are all possible dtype variations for each aten/prim operator so that we can implement and register them.
We could do this manually or creating yet another yaml file to annotate all variations, but a single yaml as source of true would be easier to maintain in the long run.