# Working With \`c10::IValue\` Efficiently **URL:** \ **Category:** performance\ **Created:** [April 18, 2022, 10:17pm UTC](https://dev-discuss.pytorch.org/t/working-with-c10-ivalue-efficiently/585 "2022-04-18T22:17:37Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![swolchok](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/swolchok/32/58_2.png) [@swolchok](https://dev-discuss.pytorch.org/u/swolchok)\ **Post date:** [April 18, 2022, 10:17pm UTC](https://dev-discuss.pytorch.org/t/working-with-c10-ivalue-efficiently/585/1 "2022-04-18T22:17:37Z")
[`c10::IValue`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue.h) (I think the “I” stands for “interpreter”) is a [tagged union](https://en.wikipedia.org/wiki/Tagged_union) that allows you to hold a wide variety of PyTorch C++ types, including `Tensor`, primitive integer and floating-point types, `Tuple`, `String`, lists, dictionaries, [and more](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue.h#L140). It functions similarly to `PyObject` in CPython: code that needs to work generically with PyTorch values (e.g., for handling operator arguments in the dispatcher) tends to hold them as `IValue`s. `IValue` has two main storage modes: it holds primitive types by value, and it holds non-primitive types via [`c10::intrusive_ptr`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/c10/util/intrusive_ptr.h), which is our alternative to `std::shared_ptr` that does intrusive reference counting. Using it to contain primitive types is fairly efficient; it is 16 bytes in size (so 1 extra pointer of overhead) and accesses require a tag check (though we could add unsafe accessors if needed), but that’s about it. Reference counting is where the problems start. First, we get all the inefficiencies of reference counting as well as the same overheads as for primitive `IValue`s. In addition, the `const` (non-moving) accessors for reference-counted types (e.g., [`IValue::toString()`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L114)) return [a new `intrusive_ptr`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L67), which means that each call to one of these accessors necessarily entails an atomic refereounce count increment/decrement pair. ## Ref Accessors To alleviate the problems with accessing reference counted types, `IValue` has a smattering of `toFooRef` accessors that return direct references instead (e.g., [`toStringRef`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L2095), [`toObjectRef`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L1525), and [`toTupleRef`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L1916)). These should be used instead of the non-ref accessors whenever possible. (Thankfully, `IValue::toTensor` also returns a direct `Tensor` reference.) ## Tuples `IValue` [`Tuple`s](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L595) are heap-allocated `(`[`TupleElements`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L256), `std::shared_ptr)` pairs. `TupleElements` is a custom “small array”: it is a union of either a 3-element `IValue` array or a `std::vector`; this saves a separate heap allocation for Tuples whose size is 3 or less. (In the future, we could potentially import [`llvm::TrailingObjects`](https://llvm.org/doxygen/TrailingObjects_8h_source.html) to c10 and use it to improve on this by consolidating the space for the tuple elements right into the `Tuple` itself – this would bring efficiency closer in line with Python tuples by avoiding wasted space for 0/1/2-element tuples and saving 2 or 3 pointers of space for 4-or-more-element tuples as well.) The most efficient way to create small tuples is to use [the `Tuple` constructor overloads that take 1, 2, or 3 `IValue`s](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L743) directly; these do a placement `new` directly into the inline storage. You will get a similar result if you pass a `std::initializer_list` with 1, 2, or 3 elements, but either the compiler or the CPU at runtime will have to do more work and you’ll have to type two extra characters as well, so don’t do that! Unless you are working with code that really, truly wants a `std::vector`, you should not need to use [`TupleElements::vec()`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L557), because `TupleElements` supports `front`, `back`, `operator[]`, `begin`, `end`, `empty`, and `size`. Unfortunately, we also have to have [`TupleElements::operator std::vector()`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L569) because `Tuple::elements()` used to return `std::vector elems = someTuple.elements();`. **A linter that detects and flags uses of these operator overloads would probably help with efficiency!** ## Strings `IValue` string storage is straightforward: [`c10::ivalue::ConstantString`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L228) is a reference-counted wrapper for a `std::string` that exists so that it can inherit from `c10::intrusive_ptr_target`. In addition to [`IValue::toString`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L114) (slow, avoid!) and [`IValue::toStringRef`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L2095), [`IValue::toStringView`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L2118) returns a `c10::string_view` for code that needs it. ## Lists and Dicts `IValue`’s native storage for lists and dicts is [`c10::GenericList`](https://github.com/pytorch/pytorch/blob/e70fea8e767e723767a4090d1eb59744b296acb3/aten/src/ATen/core/List.h#L470) (an alias for `c10::List`) and [`c10::GenericDict`](https://github.com/pytorch/pytorch/blob/e70fea8e767e723767a4090d1eb59744b296acb3/aten/src/ATen/core/Dict.h#L381) (an alias for `c10::Dict`). The template parameters for `c10::List` and `c10::Dict` control type checking and conversions, _not_ the underlying representation, which is always a container of `IValue`s. Some important consequences: - Accessing any kind of `std::vector` or `std::unordered_map` from an `IValue` (e.g., with [`IValue::toIntVector`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L1812) involves [a copy](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L1681). - Putting any kind of `std::vector` or `std::unordered_map` into an `IValue` involves [a copy](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L1978). - Because they use `IValue`s, boxed calls to operators that take `std::vector` or `std::unordered_map` as arguments or return them involve the same copies as outlined in the previous two points for their arguments. - If `sizeof(T) < sizeof(IValue)`, `List` uses more memory than `std::vector` because the underlying storage is always `std::vector`. While `c10::List` is not directly part of `c10::IValue`, there are also some efficiency pitfalls to be aware of with it. [`c10::List`](https://github.com/pytorch/pytorch/blob/e70fea8e767e723767a4090d1eb59744b296acb3/aten/src/ATen/core/List.h#L233) is an `intrusive_ptr` to a `(std::vector, std::shared_ptr)` pair. Because there is an [invariant](https://github.com/pytorch/pytorch/blob/e70fea8e767e723767a4090d1eb59744b296acb3/aten/src/ATen/core/List.h#L233) that this pointer is never null, there is no way to efficiently move construct `c10::List` (the move constructor [was deleted](https://github.com/pytorch/pytorch/pull/69370) because it was slower than the copy constructor). `List` operations also use `IValue::to` internally, and [`IValue::to`](https://github.com/pytorch/pytorch/blob/381e725911c1cfb85734df7b98067cfffe5aa284/aten/src/ATen/core/ivalue_inl.h#L1535) will often (but not always! see [`ivalue_to.h`](https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/core/ivalue_to.h) use `IValue`’s slow `intrusive_ptr`-creating `IValue` accessors. Similar concerns apply to `c10::Dict`. ## MaybeOwned: Borrowed IValues! While it is not used directly anywhere yet, [`c10::MaybeOwned`](https://github.com/pytorch/pytorch/blob/master/c10/util/MaybeOwned.h) (see also [MaybeOwned\ — PyTorch main documentation](https://pytorch.org/cppdocs/notes/maybe_owned.html)) supports efficiently borrowing `IValue`s. It may be useful if you really need to minimize reference count overhead. ## Tagged pointers I have a prototype-quality implementation of a [tagged pointer](https://en.wikipedia.org/wiki/Tagged_pointer) representation of `IValue`, but I haven’t committed it because I wasn’t able to measure a significant speedup on any particular workloads and it added extra costs to `IValue` destruction. If there are workloads where reducing the size of `IValue` to 8 bytes would be useful, we could try it again.