Irregular error 404 when java EE application deployed

I made a small webapp using java EE and tomcat 6. Everything works #1 when tested on localhost. I packaged my webapp and deployed it on our test server. Sometimes it works as intended and sometimes I get 404 error while accessing the url.

Some infos:

I'm using tomcat 7 on the server (could the change in version be the problem?). Server uses https. My index page is in WEB-INF and is mapped to a servlet. The webapp is actually deployed on 2 servers which toggles (I show the hostname at the top of my app and I can confirm both works).

Other note : When I get the 404 error, refreshing doesn't work right away, it can take a few minutes before it works again. While I'm getting 404 error when I call the URL mapped to my servlet, I can still call test.html which is outside of WEB-INF and it will display properly. It feels as if the mapping isn't done properly sometimes. Could it be a problem with web.xml ?