The Nginx configurations that I had created to serve only robots.txt and sitemap-index.xml over http
I had initially configured a set of Nginx rules that will redirect every single request made over http to https. However, that breaks my doing so breaks my reference on how to send HTTP GET request with Java without using any external libraries.
Since the robots.txt is not sensitive data, I had decided to allow it to be retrieved via http as well. And since my robots.txt points to sitemap-index.xml via http, I also allowed sitemap-index.xml to be retrieved via http as well.
This post documents the changes that I had made to my Nginx configurations to serve only robots.txt and sitemap-index.xml over the http protocol.
Follow us