I don’t want to speak for Will here, so just some personal thoughts.
The “recompilation challenge” is significant, and is arguably the most fundamental challenge in using any kind of trace-based solution (including LazyTensor) in practice. If you don’t recompile, then you error out when encountering behavior that can’t be traced (a la FX or Jax).
So, addressing recompilation is not easy . But, what can we do to improve the experience? I think to talk about that, it’s useful to think about what causes recompilations, in order from easiest to hardest. The linked doc from Ailing also talks about these challenges.
Of those challenges, I think 1/2.1 (i.e. dynamic shapes without control flow/materialization of sizes) are the easiest to fix, and the one that I think LTC already fixes (simply by virtue of not compiling to XLA lol).
In my opinion, 2.2 is hard to fix, but probably doable with a lot of effort . There’s 2 big problems that need to be solved there. The first is actually capturing what happens, which can be done either through tracing through things in Python (like FX), or returning Tensors for sizes. But even when that’s done, we still need to lower this shape computation into a compiler, which is… non-trivial.
Problem 3 (control flow) is the most … difficult to address (or the easiest, depending on how you think about it). I have a lot of ideas for how to address this, but it seems to difficult to balance UX decisions with actually tracing it in a manner conducive to optimizations.
Regardless, resolving any of these is likely to significantly address the recompilation problem. The other option, to be honest, is simply to rewrite your code in a static manner (which we could also provide the facilities to do). In the 2 examples you provided, I think it makes sense to allow for 1. nonzero to return static shapes (also useful for vmap!), and 2. to provide control flow ops (also useful for vmap!).
TL;DR: It’s a hard problem, we’re thinking about it, and there’s a lot of possible avenues. We welcome any suggestions/opinions