We made some progres finally: It turns out to be the MTU size which triggers bad performance:
The host that performed good, dit have a MTU size of 9000 on the switch, but this was not effectuated. After we rebooted this host the performance went down as well. Changeing the MTU size back to 1500 and a reboot of both hosts gave good performance on both hosts.
Switching to the on board broadcom netwerk cards gave good performance on both 1500 MTU en 9000 MTU sizes.
So the issue seems to be:
4 intel nics -> in ip hash load balancing with a MTU size of 9000. Give poor performance when read AND write to the same datastore in some circumstances.
We tried to increase the HEAP memory.
We tried to set a fixed Interupt.
Both has not worked so far.
We decided we go into production with this cluster with a mtu size of 1500.
Still if anybody runs into the same issue and finds a solution we would appreciate it being shared.