Looks like the Wordpress Activitypub plugin doesn't handle video cover images in media attachments. Also, I had forgotten that it is not compatible with page caching in W3 Total Cache (another plugin) so I had turned that on. (Either it doesn't send the Vary header or W3TC doesn't respect it)