# Different points at which fusion occurs? **URL:** \ **Category:** FX\ **Created:** [May 16, 2024, 9:46pm UTC](https://dev-discuss.pytorch.org/t/different-points-at-which-fusion-occurs/2099 "2024-05-16T21:46:41Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Amogh](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@Amogh](https://dev-discuss.pytorch.org/u/Amogh)\ **Post date:** [May 16, 2024, 9:46pm UTC](https://dev-discuss.pytorch.org/t/different-points-at-which-fusion-occurs/2099/1 "2024-05-16T21:46:41Z")
I understand that `can_fuse` and `scores_fusion` are responsible for fusion of buffers at the Inductor IR level. But I also see some fusions in the `fx_passes` before the lowering to Inductor IR goes through. Can someone clarify the entire flow from a fusion perspective? Another question would be is it possible to access the Dict on `scores_fusion` or output of `self.get_possible_fusions_with_highest_priority`, basically a list of all possible fusions with their scores ---
**Author:** ![Amogh](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@Amogh](https://dev-discuss.pytorch.org/u/Amogh)\ **Post date:** [May 20, 2024, 8:50pm UTC](https://dev-discuss.pytorch.org/t/different-points-at-which-fusion-occurs/2099/2 "2024-05-20T20:50:01Z")
Any idea @jansel regarding how I can do this? ---
**Author:** ![jansel](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/jansel/32/74_2.png) [@jansel](https://dev-discuss.pytorch.org/u/jansel)\ **Post date:** [May 20, 2024, 9:41pm UTC](https://dev-discuss.pytorch.org/t/different-points-at-which-fusion-occurs/2099/3 "2024-05-20T21:41:48Z")
The pattern matching in `fx_passes` is different from fusions and matters much less for performance. A lot of the stuff in pattern matching are graph rewrites (such as removing redundant ops), not fusing things together. Inlining happens during lowering, which is more similar to fusions. This is controlled by `buffer.realize()`, where any access to a buffer that hasn’t been realized will be inlined into the consumer (possibly recomputing it in multiple consumers). An unrealized buffer is never allocated or computed on its own. The list of all possible fusions (ordered by `score_fusion`) is computed in this function: > You could call that function from a debug environment or print it out. ---
**Author:** ![Amogh](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@Amogh](https://dev-discuss.pytorch.org/u/Amogh)\ **Post date:** [May 20, 2024, 9:55pm UTC](https://dev-discuss.pytorch.org/t/different-points-at-which-fusion-occurs/2099/4 "2024-05-20T21:55:44Z")
Thanks I was confused because of the existence of a fuse\_fx() function in here [pytorch/torch/\_inductor/fx\_passes/pre\_grad.py at main · pytorch/pytorch · GitHub](https://github.com/pytorch/pytorch/blob/main/torch/_inductor/fx_passes/pre_grad.py#L250) which includes a bunch of other fusion calls, but I guess like you said these are primarily just graph replacements and have no performance related impacts ---
**Author:** ![jansel](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/jansel/32/74_2.png) [@jansel](https://dev-discuss.pytorch.org/u/jansel)\ **Post date:** [May 20, 2024, 11:52pm UTC](https://dev-discuss.pytorch.org/t/different-points-at-which-fusion-occurs/2099/5 "2024-05-20T23:52:37Z")
The fusions in `fuse_fx()` are disabled by default. It only does something if you set `permute_fusion=True` (for GPU-only, three patterns) or `freezing=True` (for CPU-only, only one pattern). The `permute_fusion` stuff actually isn’t even fusions, it is rewrite rules to simplify cases of matmul().permute(). The freezing one is more like constant folding. Likely a distraction to focus on that, it is inconsequential for perf unless your model has some pretty uncommon patterns in it. ---
**Author:** ![Amogh](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@Amogh](https://dev-discuss.pytorch.org/u/Amogh)\ **Post date:** [May 21, 2024, 12:48pm UTC](https://dev-discuss.pytorch.org/t/different-points-at-which-fusion-occurs/2099/6 "2024-05-21T12:48:41Z")
Thanks Do the pointwise ops get fused even before inductor lowering happens? Because I noticed in the buffers generated by Triton codegen that my manual RMSNorm implementation was already a single buffer even in the Inductor IRs that were generated pre\_fusion ---
**Author:** ![jansel](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/jansel/32/74_2.png) [@jansel](https://dev-discuss.pytorch.org/u/jansel)\ **Post date:** [May 21, 2024, 4:15pm UTC](https://dev-discuss.pytorch.org/t/different-points-at-which-fusion-occurs/2099/7 "2024-05-21T16:15:17Z")
No, the opposite, before inductor we apply decompositions to break larger ops up into more minimal pieces to simplify lowering. ---
**Author:** ![Amogh](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@Amogh](https://dev-discuss.pytorch.org/u/Amogh)\ **Post date:** [May 21, 2024, 7:05pm UTC](https://dev-discuss.pytorch.org/t/different-points-at-which-fusion-occurs/2099/8 "2024-05-21T19:05:04Z")
Sure the decompositions part makes sense, but then what about these `ComputedBuffers` which aren’t just one pointwise op or reduction op, but a combination of multiple nodes ![image](https://canada1.discourse-cdn.com/flex036/uploads/pytorch1/original/2X/e/e8aa3846a7b9b09c2dc1b998cdb213c4624bd153.png) Is this the inlining that you mentioned before? ---
**Author:** ![jansel](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/jansel/32/74_2.png) [@jansel](https://dev-discuss.pytorch.org/u/jansel)\ **Post date:** [May 21, 2024, 10:26pm UTC](https://dev-discuss.pytorch.org/t/different-points-at-which-fusion-occurs/2099/9 "2024-05-21T22:26:04Z")
Yes, that is the inlining / buffer.realize() thing I talked about before.