Firewall gone crazy!

Today I saw the solution to one of my weirdest problems ever. We have been searching for about a whole week why our JEE application would suddenly stop. The problem was that it took at least an hour to reproduce it (if at all), and even then we could find next to nothing in the logs. We only started to see the light when we noticed the difference between environments where the error was reproducable, and where is was not reproducable: the firewall.

Turns out the firewall between the application server and the database server would stop all trafic on the JDBC connection after an hour if idling, without actually killing the connection. Weblogic, and all other applications we tried, totally freak out when this happens. What on earth did the creators of that firewall think? When you kill a connection, then kill it! Do not just stop the data flow!

Well anyway, the solution? We now let Weblogic 'test' the connection every 10 minutes. Apparently, the firewall is happier when there is some traffic now and then.