A Gotcha Using Node.js + Request In a Daemon
13 April 2012I have a Node.js program running as a daemon on a Linux VPS. Periodically, it polls a list of URLs using request. When it first starts, everything runs smoothly. But after running for a while, it starts getting 400 errors, and the longer it runs, the more URLs return 400 errors.
I could not understand what was going on. My code was basically structured like this:
Given that code, we know the req object is initialized with each function call. So, how could this script degrade over time?
Well, I finally tracked it down: COOKIES!
Yup, request has cookies enabled by default. So, I think what was happening was that cookies were being set (presumably, top domain-level cookies having the same name at different URLs or subdomains on the same domain) but the values in request's cookie jar were not being returned properly. That means the remote host was getting invalid cookies -- hence the 400 response for a "Bad Request."
I haven't yet spent the time to figure out if this is a bug in request. It's on my TODO list.
In the meantime, I've disabled cookies in the req object:
var req = { url: url, timeout: options.timeout, jar: false };
It's now working as expected.
