Video shows much smaller than expected. Video rotation and scaling issue

Shotstack API
, , ,
1

Hey - I’m trying to render videos, but some of them do not fill the full screen and I cannot figure out why. This video has a 1080x1920 resolution, and is a vertical video. When I render it with Scale=1, position=topLeft, offset=0, and fit = contain I would expect it to fill up the full screen. Instead it is way smaller and centered at the top of the canvas for some reason.

Any idea why this is happening? Thanks!

Rendered video:

Input JSON:

{
   "timeline":{
      "background":"#000000",
      "tracks":[
         {
            "clips":[
               {
                  "asset":{
                     "type":"video",
                     "src":"https://pu-gshenar-public-assets.s3.amazonaws.com/403a178b-1dcd-4ec7-a07c-987916c97694.mov",
                     "trim":5.15,
                     "volume":1.0
                  },
                  "start":0.0,
                  "length":2.015,
                  "fit":"contain",
                  "scale":1.0,
                  "position":"topLeft",
                  "offset":{
                     "x":0.0,
                     "y":0.0
                  },
                  "opacity":1.0
               }
            ]
         }
      ]
   },
   "output":{
      "format":"mp4",
      "size":{
         "width":1080,
         "height":1920
      }
   }
}
2

Hi @gshenar,

When using the media inspector endpoint it looks like the video has a width of 1920 and a height of 1080. It also has a 90 degree rotation.

{
    "success": true,
    "message": "ok",
    "response": {
        "metadata": {
            "streams": [
                {
                    ...
                    "width": 1920,
                    "height": 1080,
                    ...,
                    "tags": {
                        "rotate": "90",
                        ...
                    },
                    ...
                }
            ],
            "chapters": [],
            "format": { ... }
        }
    }
}

This is an issue typical to user generated content (UGC) where devices use different ways to rotate video. While this normally shows up fine on any player it becomes an issue when trying to edit the footage as the default crop value for the fit property uses the width and height to calculate the scale to make the video fit inside of the viewport.

We have a ticket to handle the different ways rotation is handled, but until that time you can scale the clip based on the height / width ratio as follows:

{
    "timeline": {
        "tracks": [{
            "clips": [{
                "asset": {
                    "type": "video",
                    "src": "https://pu-gshenar-public-assets.s3.amazonaws.com/403a178b-1dcd-4ec7-a07c-987916c97694.mov",
                    "trim": 5
                },
                "start": 0.0,
                "length": 2,
                "scale": 0.5625
            }]
        }]
    },
    "output": {
        "format": "mp4",
        "size": {
            "width": 1080,
            "height": 1920
        }
    }
}
3

@dazzatron Thanks for the quick response.

Can you elaborate a little bit on how the scale property behaves, it isn’t fully clear from the documentation. Does it just match the percent of the width to the width of the video and then scale the height according to the aspect ratio?

Also - you mentioned you have a ticket to handle this rotation on the backend - do you have a rough estimate of when this is going live? We’d like to avoid having to probe every uploaded video file that we get if it will only be a temporary solution as we aren’t live with customers yet.

Thanks!

4

Hey @gshenar,

Firstly, to answer your initial question:

The scale property takes the width and height of the asset and scales it based on the ratio you set in your JSON. It works slightly differently depending on what fit property you use.

The following examples use an image with dimensions of 400px by 102px and an output of 1280px by 720px. By default a clip always uses the crop property.

Scaling with fit property set to crop (default)

When you don’t set the fit property and set no scale your image will scale to fit the viewport.

No scale

As the example video is wider than it is tall this means it gets scaled automatically to fill the height of the viewport. In this case the implied scale equals 720 / 102 = 7.0589. This means its width will be 400px * 7.0589 = 2823.6px, and the height will be 720px.

image
image1280×720 40.2 KB

{
    "timeline": {
        "tracks": [
            {
                "clips": [
                    {
                        "asset": {
                            "type": "image",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/branding/logo-reverse.png"
                        },
                        "start": 0,
                        "length": 1
                    }
                ]
            }
        ]
    },
    "output": {
        "format": "png",
        "resolution": "hd"
    }
}

“Scale”: 0.5

When you set the scale property on a clip with the default crop property the clip will scale based on the size of your output resolution. In the particular case the height of the clip will be 1280px * 0.5 = 640px.

image
image1280×720 81.1 KB

{
    "timeline": {
        "tracks": [
            {
                "clips": [
                    {
                        "asset": {
                            "type": "image",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/branding/logo-reverse.png"
                        },
                        "start": 0,
                        "length": 1,
                        "scale": 0.5
                    }
                ]
            }
        ]
    },
    "output": {
        "format": "png",
        "resolution": "hd"
    }
}

Scaling with fit property set to cover

When you set the fit property to cover you stretch the asset to fill the viewport without maintaining the aspect ratio.

No scale

The clip gets scaled to fill the viewport completely. In case of its width it gets scaled by 1280px / 400px = 3.2 and its height gets scaled by 720px / 102px = 7.0589.

image
image1280×720 62.6 KB

{
    "timeline": {
        "tracks": [
            {
                "clips": [
                    {
                        "asset": {
                            "type": "image",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/branding/logo-reverse.png"
                        },
                        "start": 0,
                        "length": 1,
                        "fit": "cover"
                    }
                ]
            }
        ]
    },
    "output": {
        "format": "png",
        "resolution": "hd"
    }
}

“Scale”: 0.5

When you scale a clip with the fit property set to cover and set the scale property to 0.5 your image gets stretched to a width of 1280px * 0.5 = 640x and a height of 720px * 0.5 = 360px.

image
image1280×720 62.8 KB

{
    "timeline": {
        "tracks": [
            {
                "clips": [
                    {
                        "asset": {
                            "type": "image",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/branding/logo-reverse.png"
                        },
                        "start": 0,
                        "length": 1,
                        "fit": "cover",
                        "scale": 0.5
                    }
                ]
            }
        ]
    },
    "output": {
        "format": "png",
        "resolution": "hd"
    }
}

Scaling with fit property set to contain

When the fit property is set to contain the entire asset is scaled to fit within the viewport while maintaining the original aspect ratio.

No scale

In the case of this example, where the image is wider than it is tall, it will have a width of 1280px as the width of the viewport is 1280px. This effectively scales the clip by 1280px / 400px = 3.2

image
image1280×720 80 KB

{
    "timeline": {
        "tracks": [
            {
                "clips": [
                    {
                        "asset": {
                            "type": "image",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/branding/logo-reverse.png"
                        },
                        "start": 0,
                        "length": 1,
                        "fit": "contain"
                    }
                ]
            }
        ]
    },
    "output": {
        "format": "png",
        "resolution": "hd"
    }
}

“Scale”: 0.5

When you scale a clip with the fit property set to contain and set the scale property to 0.5 your image gets scaled to a width of 1280px * 0.5 = 640x and a height of 102px * 3.2 * 0.5 = 163.2px.

image
image1280×720 37.7 KB

{
    "timeline": {
        "tracks": [
            {
                "clips": [
                    {
                        "asset": {
                            "type": "image",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/branding/logo-reverse.png"
                        },
                        "start": 0,
                        "length": 1,
                        "fit": "contain",
                        "scale": 0.5
                    }
                ]
            }
        ]
    },
    "output": {
        "format": "png",
        "resolution": "hd"
    }
}

Scaling with fit property set to none

When the fit property is set to contain the asset preserves the original asset dimensions and does not apply any scaling.

No scale

When no scaling is applied to the clip the dimensions will stay the same as the asset used. In this case setting the clip dimensions at 400px by 102px.

image
image1280×720 5.23 KB

{
    "timeline": {
        "tracks": [
            {
                "clips": [
                    {
                        "asset": {
                            "type": "image",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/branding/logo-reverse.png"
                        },
                        "start": 0,
                        "length": 1,
                        "fit": "none"
                    }
                ]
            }
        ]
    },
    "output": {
        "format": "png",
        "resolution": "hd"
    }
}

“Scale”: 0.5

When you scale a clip with the fit property set to none and set the scale property to 0.5 your image gets scaled by the dimensions of the asset used. In this case the image will be 400px * 0.5 = 200px wide and 102px * 0.5 = 51px high.

image
image1280×720 4.46 KB

{
    "timeline": {
        "tracks": [
            {
                "clips": [
                    {
                        "asset": {
                            "type": "image",
                            "src": "https://shotstack-assets.s3-ap-southeast-2.amazonaws.com/branding/logo-reverse.png"
                        },
                        "start": 0,
                        "length": 1,
                        "fit": "none",
                        "scale": 0.5
                    }
                ]
            }
        ]
    },
    "output": {
        "format": "png",
        "resolution": "hd"
    }
}
5

And to answer your second question:

As changing the current logic may be a breaking change for some users it may take a while until we understand the best way to proceed with this. Regardless of this particular oddity we will launch an ingestion service that will allow you to standardise certain elements such as rotation. We expect to launch this service later this year.

In your case I would suggest doing one of the following which will ensure the same method will continue working regardless of what we decide to do on our end:

Apply the scale property

If you know that your videos will always be encoded this way you can safely use the scale property to make these fit within the viewport.

Fix the video to display the correct width and height

Similar to what we’ll be doing using the ingestion service (and explained in more detail in this article) you can use a command like this:

ffmpeg -i source.mov -metadata:s:v:0 rotate=0 -c:a copy output.mov | ffmpeg -vf "transpose=1" -c:a copy output.mov

This removes the rotation tag and re-encodes the video to display the correct width and height. Then you don’t need to use any scale property:

{
    "timeline": {
        "cache": false,
        "tracks": [{
            "clips": [{
                "asset": {
                    "type": "video",
                    "src": "https://shotstack-customer.s3.amazonaws.com/gshenar.mov",
                    "trim": 5
                },
                "start": 0.0,
                "length": 2
            }]
        }]
    },
    "output": {
        "format": "mp4",
        "size": {
            "width": 1080,
            "height": 1920
        }
    }
}
6

@dazzatron Thanks for the detailed response! This makes alot more sense to me now.

I assume the reencoding the whole video may be quite slow for us since we are an end-user application with generated content and would like to render videos as quickly as possible. We don’t expect our videos to always be generated with this same metadata, so I think our path forward will probably just be to FFProbe this metadata on the client and be able to generate the correct scale values that we need.

Thanks again for your support.

1 Like