Intermittent Slow Performance in all areas except Administration in v7 and higher (Syracuse/MongoDB)
Description
  • Note: This issue only affects v7 or higher.
  • Symptoms: Intermittent slow performance in all areas except the Administration menu.
    • Any menu item in Administration comes up within seconds.
    • Any menu item in any other menu block (other than Administration) may take 1-10 minutes to come up. The next time you try again, it may take only a few seconds to come up.
      • If inside a classic screen, selecting a value from the left list may take 1-10 minutes to display results, intermittently as well.
  • When properly working, it should only take a few seconds to open up these screens or display results when selecting different items from the left list.
Cause

Setup which causes this issue (VMs on different host machines): The exact mechanism is unknown, however, this issue is reported in scenarios where there are two or three virtual machines hosted on two different host (physical, non VM) servers and the SQL Server is hosted on a different host machine than the X3 App/Syracuse server.

Example: You have two physical servers, both running Hyper-V (or VMware), and they are each running their own virtual machines. One virtual machine has the X3 application and/or Syracuse server and other components while the second virtual machine ha the SQL Server

Observed VM technologies affected (but not limited to): Microsoft Hyper-V and VMware vCenter

Observed hardware affected (but not limited to): Dell and HP servers

Resolution
  1. If these symptoms are present, the only known solution to consistently increase performance is to have both virtual machines under the same host server.
    • Note: With vCenter, a customer was able to instantly and seamlessly switch the VMs unto the same host using a VMware feature (if available) called vFlow. The resolution was instantaneous. The customer also noted he could put in a place a setting so the VMs do not get automatically load balanced unto different host machines.
  2. Another possible solution is to review knowledgebase article 83823 below. This article has best practices information from non-Sage documents obtain from the Internet on virtual machines from VMware's standpoint. One user reported performance issue resolved after switching power to "High performance" mode.

VMware further states on page 14 of the sql-server-on-vmware-best-practices-guide.pdf (Related KB: Virtualization best practice documentation obtained) saying:

"VMware defines latency-sensitive applications as workloads that require optimizing for a few microseconds to a few tens of microseconds end-to-end latencies. This does not apply to applications or workloads in the hundreds of microseconds to tens of milliseconds end-to-end -latencies. SQL Server is not usually considered a “latency sensitive” application. However, given the adverse impact of incorrect power settings in a Windows operating system, customers should pay special attention to power management."

 

And on page 16:

 

"The default power policy option in Windows Server 2012 is “Balanced”. This configuration allows Windows OS to save power consumption by periodically throttling power to the CPU and turning off devices such as the network cards in the guest when Windows determines that they are idle or unused. This capability is inefficient for critical SQL Server workloads due to the latency and disruption introduced by the act of powering-off and powering-on CPUs and devices. Allowing Windows to throttle CPUs can result in what Microsoft describes as core parking and should be avoided."

 

Additional information:

Many tests have been done to confirm the issue isn't with the VMs themselves or with the X3 setup, but instead the VMs being on different hosts. In some cases, ping results to the VMs may increase drastically when the symptoms are present. Ping results return to normal when symptoms are not present. Various network utility tests simply show there is a long pause in between VMs communicating between each other, but does not reveal the cause. The most likely scenario is the physical host machines are not efficiently allowing the VMs to communicate with each other. The VMs would not be aware of this being they are on a lower level (i.e. VMs are not aware that a physical host machine is running them). The only known solution is to move the VMs unto the same physical host machine.

Steps to duplicate
Related Solutions

Virtualization best practice documentation obtained