I think the issue is because of how Google Drive handles redirects, the URL is not actually the URL of the image but a 302 redirect to another file. You can verify that using this tool: Redirect Checker | Check your Statuscode 301 vs 302
It shows a 302 redirect to a new file. The API is probably downloading HTML instead of jpg/png/gif which is why the file is invalid.
As workarounds I would investigate if there is an alternative URL you can use or work out what the redirected URL is. There might also be a way to get the real image URL using one of Googles APIs.