KAWO sits in the middle of various APIs and due to a high volume of requests, we’re required to push and pull images hundreds of times every day. For simplicity and in order to keep our frontend app servers stateless, we decided to put all of these images into Amazon’s marvelous S3. It seemed easy enough, and we thought the job was done knowing that our images were being served out of S3.
At first, the problems began slowly but rapidly increased in quantity and magnitude. The failure rate of API requests when publishing to Chinese SNS jumped massively as the servers in mainland China were unable to reach our images hosted in S3. Then some frontend app users reported that none of the images were loading. A quick subdomain mapping onto the disagreeable S3 urls appeared to solve both of these problems, but before long they were back. Constant testing revealed that some Chinese social networks actually preferred being sent the S3 urls while others liked our subdomain. We also tried using NGINX on our Mainland China servers to reverse proxy the images to yet another subdomain, which also showed promise for a little while.
After exhausting most of the proxying options we increased the image compression and experimented with sending images with smaller dimensions. To this day we still don’t have a perfect solution and we still aggressively monitor and test on a weekly basis to ensure all posts published through KAWO go out on time with all of their images.