Hi,
In my http.log, I am seeing some lines being written without response code etc. What could be the reason for this? One reason I could think of was, what if the server or some entity between bro and the server that dropped the request/response thus preventing the response from reaching bro or the connection is closed on receiving the request by a downstream security device. How does bro react in such cases? could one of these scenarios explain why the response fields are missing from the log?
You seem to have a pretty good handle on what could be causing the problem. One additional thing you didn't list is if you have load balancing happening incorrectly. That could cause the same problem because the request could have gone to a different process than the reply.
What would help most at this point is if you could send a conn.log entry for a connection where you saw the http.log missing the response code (feel free to redact IP addresses, they don't matter).