# PyTorch Faster/Mask RCNN resize images badly

**URL:** <https://dev-discuss.pytorch.org/t/pytorch-faster-mask-rcnn-resize-images-badly/244>\
**Category:** Uncategorized\
**Created:** [May 27, 2021, 5:38am UTC](https://dev-discuss.pytorch.org/t/pytorch-faster-mask-rcnn-resize-images-badly/244 "2021-05-27T05:38:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bernd1969](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/bernd1969/32/184_2.png) [@Bernd1969](https://dev-discuss.pytorch.org/u/Bernd1969)\
**Post date:** [May 27, 2021, 5:38am UTC](https://dev-discuss.pytorch.org/t/pytorch-faster-mask-rcnn-resize-images-badly/244/1 "2021-05-27T05:38:17Z")

</div>

Hello  
I am new here. I work since 21 years as software dev and I think I found an issue during PyTorch Faster/Mask RCNN usage.  
Deep down in GeneralizedRCNNTransform (transform.py@39-43) PyTorch makes the decidion if an image needs to be resized.

def \_resize\_image\_and\_masks(image, self\_min\_size, self\_max\_size, target):  
\_\_\_\_# type: (Tensor, float, float, Optional[Dict[str, Tensor]]) → Tuple[Tensor, Optional[Dict[str, Tensor]]]  
\_\_\_\_im\_shape = torch.tensor(image.shape[-2:])  
\_\_\_\_min\_size = float(torch.min(im\_shape))  
\_\_\_\_max\_size = float(torch.max(im\_shape))  
\_\_\_\_scale\_factor = self\_min\_size / min\_size  
\_\_\_\_if max\_size \* scale\_factor \> self\_max\_size:  
\_\_\_\_\_\_\_\_scale\_factor = self\_max\_size / max\_size  
\_\_\_\_image = torch.nn.functional.interpolate(  
\_\_\_\_\_\_\_\_image[None], scale\_factor=scale\_factor, mode=‘bilinear’, recompute\_scale\_factor=True,  
\_\_\_\_\_\_\_\_align\_corners=False)[0]  
\_\_\_\_…

There are 4 parameters used here to decide about the scale\_factor:  
min\_size and max\_size which are the min and max dimensions of the image  
self\_min\_size and self\_max\_size which are defined during initialization of the backbone and are by default set to 800 and 1333.

If I now come in with an image that fits perfectly well, like for example 832 x 1333, this algorithm will set scale\_factor=800/832=0.96153… So the image gets resized even though it fits perfectly well into the backbone. I debugged this and set the scale\_factor to 1.0 in line 44 and it worked perfectly like a charm without resizing anything.

I believe the algorithm should be changed to:

def \_resize\_image\_and\_masks(image, self\_min\_size, self\_max\_size, target):  
\_\_\_\_# type: (Tensor, float, float, Optional[Dict[str, Tensor]]) → Tuple[Tensor, Optional[Dict[str, Tensor]]]  
\_\_\_\_im\_shape = torch.tensor(image.shape[-2:])  
\_\_\_\_min\_size = float(torch.min(im\_shape))  
\_\_\_\_max\_size = float(torch.max(im\_shape))  
\_\_\_\_if min\_size \< self\_min\_size or max\_size \> self\_max\_size: # avoid rescaling if not required  
\_\_\_\_\_\_\_\_scale\_factor = self\_min\_size / min\_size  
\_\_\_\_\_\_\_\_if max\_size \* scale\_factor \> self\_max\_size:  
\_\_\_\_\_\_\_\_\_\_\_\_scale\_factor = self\_max\_size / max\_size  
\_\_\_\_\_\_\_\_image = torch.nn.functional.interpolate(  
\_\_\_\_\_\_\_\_\_\_\_\_image[None], scale\_factor=scale\_factor, mode=‘bilinear’, recompute\_scale\_factor=True,  
\_\_\_\_\_\_\_\_\_\_\_\_align\_corners=False)[0]  
\_\_\_\_…

(sorry for bad formatting, the post interface removed all indentations, so I replaced that spaces with \_)

What do the pro’s in this formum think about this?

P.S.: this call to interpolate with a float scale\_factor creates a very annoying warning that should be addressed :

UserWarning: The default behavior for interpolate/upsample with float scale\_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale\_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute\_scale\_factor=True. See the documentation of nn.Upsample for details.  
warnings.warn("The default behavior for interpolate/upsample with float scale\_factor changed "

---

<div class="post-metadata">

**Author:** ![Bernd1969](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/bernd1969/32/184_2.png) [@Bernd1969](https://dev-discuss.pytorch.org/u/Bernd1969)\
**Post date:** [May 27, 2021, 7:22am UTC](https://dev-discuss.pytorch.org/t/pytorch-faster-mask-rcnn-resize-images-badly/244/2 "2021-05-27T07:22:46Z")

</div>

To illustrate my change here all test cases:

what : \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ dimensions : current : \_\_ proposed (if changed)  
both too big : \_\_\_\_\_\_\_\_\_\_\_\_\_ 1500 x 4000 : 500 x 1333  
one too big, one too small : 4000 x 500 : 1333 x 166  
one too big, one ok: \_\_\_\_\_\_ : 4000 x 900 : 1333 x 333  
perfect min, one ok \_\_\_\_\_\_ : 800 x 1000 : 800 x 1000  
perfect min, perfect max\_ : 800 x 1333 : 800 x 1333  
both ok \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ : 900 x 1000 : 800 x 888 : CHANGED: 900 x 1000  
one ok, perfect max\_\_\_\_\_\_ : 900 x 1333 : 800 x 1184 : CHANGED: 900 x 1333  
one too small, one ok \_\_\_\_\_: 500 x 1000 : 666 x 1333  
both too small \_\_\_\_\_\_\_\_\_\_\_ : 500 x 200 \_\_: 1333 x 533

---

<div class="post-metadata">

**Author:** ![smth](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/smth/32/3_2.png) [@smth](https://dev-discuss.pytorch.org/u/smth)\
**Post date:** [June 2, 2021, 6:13am UTC](https://dev-discuss.pytorch.org/t/pytorch-faster-mask-rcnn-resize-images-badly/244/3 "2021-06-02T06:13:19Z")

</div>

hey it’s really hard to read your post.

Could you enclose your code in 3 back-ticks (i.e. the symbol ` three times ).  
That preserves formatting / indentation.  
Then, you can remove the \_ prefixes that you’ve added to the code and replace them back with spaces.

---

<div class="post-metadata">

**Author:** ![Bernd1969](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/bernd1969/32/184_2.png) [@Bernd1969](https://dev-discuss.pytorch.org/u/Bernd1969)\
**Post date:** [June 15, 2021, 4:20pm UTC](https://dev-discuss.pytorch.org/t/pytorch-faster-mask-rcnn-resize-images-badly/244/4 "2021-06-15T16:20:13Z")

</div>

Ok, I try again. Sorry for late reply…

* * *

Hello  
I am new here. I work since 21 years as software dev and I think I found an issue during PyTorch Faster/Mask RCNN usage.  
Deep down in GeneralizedRCNNTransform (transform.py@39-43) PyTorch makes the decidion if an image needs to be resized.

```auto
def _resize_image_and_masks(image, self_min_size, self_max_size, target):
    # type: (Tensor, float, float, Optional[Dict[str, Tensor]]) → Tuple[Tensor, Optional[Dict[str, Tensor]]]
    im_shape = torch.tensor(image.shape[-2:])
    min_size = float(torch.min(im_shape))
    max_size = float(torch.max(im_shape))
    scale_factor = self_min_size / min_size
    if max_size * scale_factor > self_max_size:
        scale_factor = self_max_size / max_size
        image = torch.nn.functional.interpolate(
            image[None], scale_factor=scale_factor, mode=‘bilinear’, recompute_scale_factor=True,
                align_corners=False)[0]
    …

```

There are 4 parameters used here to decide about the scale\_factor:  
min\_size and max\_size which are the min and max dimensions of the image  
self\_min\_size and self\_max\_size which are defined during initialization of the backbone and are by default set to 800 and 1333.

If I now come in with an image that fits perfectly well, like for example 832 x 1333, this algorithm will set scale\_factor=800/832=0.96153… So the image gets resized even though it fits perfectly well into the backbone. I debugged this and set the scale\_factor to 1.0 in line 44 and it worked perfectly like a charm without resizing anything.

I believe the algorithm should be changed to:

```auto
def _resize_image_and_masks(image, self_min_size, self_max_size, target):
    # type: (Tensor, float, float, Optional[Dict[str, Tensor]]) → Tuple[Tensor, Optional[Dict[str, Tensor]]]
    im_shape = torch.tensor(image.shape[-2:])
    min_size = float(torch.min(im_shape))
    max_size = float(torch.max(im_shape))
    if min_size < self_min_size or max_size > self_max_size: # avoid rescaling if not required
        scale_factor = self_min_size / min_size
        if max_size * scale_factor > self_max_size:
            scale_factor = self_max_size / max_size
        image = torch.nn.functional.interpolate(
             image[None], scale_factor=scale_factor, mode=‘bilinear’, recompute_scale_factor=True,
                   align_corners=False)[0]
…

```

What do the pro’s in this formum think about this?

P.S.: this call to interpolate with a float scale\_factor creates a very annoying warning that should be addressed :

```auto
UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
warnings.warn("The default behavior for interpolate/upsample with float scale_factor changed "

```

---

<div class="post-metadata">

**Author:** ![benkoger](https://yyz2.discourse-cdn.com/flex036/user_avatar/dev-discuss.pytorch.org/benkoger/32/1024_2.png) [@benkoger](https://dev-discuss.pytorch.org/u/benkoger)\
**Post date:** [March 24, 2023, 11:01pm UTC](https://dev-discuss.pytorch.org/t/pytorch-faster-mask-rcnn-resize-images-badly/244/5 "2023-03-24T23:01:54Z")

</div>

Can someone follow up on this? Even though the code for \_resize\_image\_and\_masks ([vision/transform.py at db3ead1656a0ec9d523ca6b07c68394edf87b45a · pytorch/vision · GitHub](https://github.com/pytorch/vision/blob/db3ead1656a0ec9d523ca6b07c68394edf87b45a/torchvision/models/detection/transform.py#L25)) has changed since the original post it still seems to always rescale the image so one side is either length min\_val or max\_val even if both of the original image’s sides are between min\_val and max\_val. Is this the intended outcome? If it is, that’s not how most people interpret it online and the documentation is unclear for a key part of the image loading pipeline. Also, it’s not clear to me why this is useful default functionality.