The Post Office Hates Java

I am updating the section of Core Java that explains how to make a POST request. The service that I used in the previous edition was discontinued, so I tried using the USPS ZIP code lookup. The POST redirects to a GET, but that's ok—HttpURLConnection can handle redirects. But when the user agent contains the string Java, the post office redirects to a dead end!

The United States Post Office has a form at https://www.usps.com/zip4/ for looking up ZIP codes by address, or cities by ZIP code. Let's try the latter because it is simpler. Looking at the form source, you can see that it expects a POST to https://tools.usps.com/go/ZipLookupAction.action with form parameters tZip set to the ZIP code and mode set to 2. Let's try it out with HTTPie:

http -v --form POST https://tools.usps.com/go/ZipLookupAction.action tZip=95014 mode=2

Ok, it redirects:

HTTP/1.1 302 Moved Temporarily
Location: https://tools.usps.com/go/ZipLookupResultsAction!input.action?resultMode=2&companyName=&address1=&address2=&city=&state=&urbanCode=&postalCode=95014&zip=
. . .

And it sets a cookie:

. . .
Set-Cookie: JSESSIONID=0000JH-YT_X_gMV2Z7rhWq5i4Vv:15ubolsqa; HTTPOnly; Path=/; Secure
Set-Cookie: reg-ns-sessionid=390000355-nscip-50.0.191.2-Wed, 20 Apr 2016 17:43:19 GMT; path=/;  domain=.usps.com
Set-Cookie: NSC_uppmt.vtqt.dpn_443=ffffffff3b2236ae45525d5f4f58455e445a4a4212d3;Version=1;path=/;secure;httponly

Not a problem. The HttpURLConnection class can deal with that.

But it doesn't work in Java:

Exception in thread "main" java.net.ProtocolException: Server redirected too many  times (20)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1848)
  . . .

I wanted to see the redirects. First, I needed the source for that sun.net.www.protocol.http.HttpURLConnection class. (Btw, people, don't use the same name for implementation classes as for API classes!) The nifty grepcode service gave it to me.

Can I log them? The source code shows that various events are logged to the sun.net.www.protocol.http.HttpURLConnection logger. Except, that's a sun.util.logging.PlatformLogger. I had never run into those, but this StackOverflow post told me that the magic lines

Logger rootLogger = Logger.getLogger("");
rootLogger.setLevel(Level.ALL);

sufficed to have the messages show up in the standard logging framework.

Then, these lines make it so that the messages show up on the console without messing with a log configuration file:

Logger logger = Logger.getLogger("sun.net.www.protocol.http.HttpURLConnection");
logger.setLevel(Level.ALL);
Handler handler = new ConsoleHandler();
handler.setLevel(Level.ALL);
logger.addHandler(handler);

Weird—you can see that the initial request is redirected to a different URL:

FINE: Redirected from https://tools.usps.com/go/ZipLookupAction.action to https://www.usps.com/root/global/server_responses/webtools-msg.htm
avr. 20, 2016 10:57:18 AM sun.net.www.protocol.http.HttpURLConnection plainConnect0

The logs also show the request headers. All of them look the same, except for

User-Agent: Java/1.8.0_65

So, no big deal—you can change the user agent:

connection.setRequestProperty("User-Agent", "HTTPie/0.9.2");

or globally by setting the http.agent property.

But they both don't work, for different reasons. Let's first call connection.setRequestProperty. Then the call gets redirected to the correct URL, but the redirect still fails, because it doesn't use the changed user agent! It uses Java/1.8.0_65 and then gets redirected to the URL that causes a redirection loop.

When you set http.agent, then the connection immediately gets misdirected. The logs show why: the user agent isn't what you set, but it is

User-Agent: HTTPie/0.9.2 Java/1.8.0_65

The HttpURLConnection class clearly goes through a lot of trouble to make sure that the server knows it's being called from Java. Here is what it does:

public static final String userAgent;
static {
   . . .
   version = java.security.AccessController.doPrivileged(
      new sun.security.action.GetPropertyAction("java.version"));
   String agent = java.security.AccessController.doPrivileged(
      new sun.security.action.GetPropertyAction("http.agent"));
   if (agent == null) {
      agent = "Java/"+version;
   } else {
      agent = agent + " Java/"+version;
   }
   userAgent = agent;
   . . .
}

Just to make sure that I am not making this up, I used reflection to change the string:

  
Field f = sun.net.www.protocol.http.HttpURLConnection.class.getDeclaredField("userAgent");
f.setAccessible(true);
Field modifiersField = Field.class.getDeclaredField("modifiers");
modifiersField.setAccessible(true);
modifiersField.setInt(f, f.getModifiers() & ~Modifier.FINAL);            
f.set(null, "HTTPie/0.9.2");

Note how setting the modifiers enables us to change a static field—see this StackOverflow post.

Sure enough, everything worked ok, and I got the result.

So there you have it. Someone at USPS hates Java. Whenever the user agent contains the four-letter string java (case insensitive), then the request gets misdirected.

Of course, that seems pretty pointless. Apart from the reprehensible hack, one can manually carry out the redirect. And if more web services hate Java, then perhaps Oracle will fix HttpURLConnection to apply the user agent setting to redirects.