I am updating the section of Core Java that explains how to make a POST request. The service that I used in the previous edition was discontinued, so I tried using the USPS ZIP code lookup. The POST redirects to a GET, but that's ok—HttpURLConnection can handle redirects. But when the user agent contains the string Java, the post office redirects to a dead end!
The United States Post Office has a form at https://www.usps.com/zip4/ for looking up ZIP codes by address, or cities by ZIP code. Let's try the latter because it is simpler. Looking at the form source, you can see that it expects a POST
to https://tools.usps.com/go/ZipLookupAction.action
with form parameters tZip
set to the ZIP code and mode
set to 2. Let's try it out with HTTPie:
http -v --form POST https://tools.usps.com/go/ZipLookupAction.action tZip=95014 mode=2
Ok, it redirects:
HTTP/1.1 302 Moved Temporarily Location: https://tools.usps.com/go/ZipLookupResultsAction!input.action?resultMode=2&companyName=&address1=&address2=&city=&state=&urbanCode=&postalCode=95014&zip= . . .
And it sets a cookie:
. . . Set-Cookie: JSESSIONID=0000JH-YT_X_gMV2Z7rhWq5i4Vv:15ubolsqa; HTTPOnly; Path=/; Secure Set-Cookie: reg-ns-sessionid=390000355-nscip-50.0.191.2-Wed, 20 Apr 2016 17:43:19 GMT; path=/; domain=.usps.com Set-Cookie: NSC_uppmt.vtqt.dpn_443=ffffffff3b2236ae45525d5f4f58455e445a4a4212d3;Version=1;path=/;secure;httponly
Not a problem. The HttpURLConnection
class can deal with that.
But it doesn't work in Java:
Exception in thread "main" java.net.ProtocolException: Server redirected too many times (20) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1848) . . .
I wanted to see the redirects. First, I needed the source for that sun.net.www.protocol.http.HttpURLConnection
class. (Btw, people, don't use the same name for implementation classes as for API classes!) The nifty grepcode service gave it to me.
Can I log them? The source code shows that various events are logged to the sun.net.www.protocol.http.HttpURLConnection
logger. Except, that's a sun.util.logging.PlatformLogger
. I had never run into those, but this StackOverflow post told me that the magic lines
Logger rootLogger = Logger.getLogger(""); rootLogger.setLevel(Level.ALL);
sufficed to have the messages show up in the standard logging framework.
Then, these lines make it so that the messages show up on the console without messing with a log configuration file:
Logger logger = Logger.getLogger("sun.net.www.protocol.http.HttpURLConnection"); logger.setLevel(Level.ALL); Handler handler = new ConsoleHandler(); handler.setLevel(Level.ALL); logger.addHandler(handler);
Weird—you can see that the initial request is redirected to a different URL:
FINE: Redirected from https://tools.usps.com/go/ZipLookupAction.action to https://www.usps.com/root/global/server_responses/webtools-msg.htm avr. 20, 2016 10:57:18 AM sun.net.www.protocol.http.HttpURLConnection plainConnect0
The logs also show the request headers. All of them look the same, except for
User-Agent: Java/1.8.0_65
So, no big deal—you can change the user agent:
connection.setRequestProperty("User-Agent", "HTTPie/0.9.2");
or globally by setting the http.agent
property.
But they both don't work, for different reasons. Let's first call connection.setRequestProperty
. Then the call gets redirected to the correct URL, but the redirect still fails, because it doesn't use the changed user agent! It uses Java/1.8.0_65
and then gets redirected to the URL that causes a redirection loop.
When you set http.agent
, then the connection immediately gets misdirected. The logs show why: the user agent isn't what you set, but it is
User-Agent: HTTPie/0.9.2 Java/1.8.0_65
The HttpURLConnection
class clearly goes through a lot of trouble to make sure that the server knows it's being called from Java. Here is what it does:
public static final String userAgent; static { . . . version = java.security.AccessController.doPrivileged( new sun.security.action.GetPropertyAction("java.version")); String agent = java.security.AccessController.doPrivileged( new sun.security.action.GetPropertyAction("http.agent")); if (agent == null) { agent = "Java/"+version; } else { agent = agent + " Java/"+version; } userAgent = agent; . . . }
Just to make sure that I am not making this up, I used reflection to change the string:
Field f = sun.net.www.protocol.http.HttpURLConnection.class.getDeclaredField("userAgent"); f.setAccessible(true); Field modifiersField = Field.class.getDeclaredField("modifiers"); modifiersField.setAccessible(true); modifiersField.setInt(f, f.getModifiers() & ~Modifier.FINAL); f.set(null, "HTTPie/0.9.2");
Note how setting the modifiers enables us to change a static field—see this StackOverflow post.
Sure enough, everything worked ok, and I got the result.
So there you have it. Someone at USPS hates Java. Whenever the user agent contains the four-letter string java
(case insensitive), then the request gets misdirected.
Of course, that seems pretty pointless. Apart from the reprehensible hack, one can manually carry out the redirect. And if more web services hate Java, then perhaps Oracle will fix HttpURLConnection
to apply the user agent setting to redirects.