Loading ...

UPS'es in Redundant configuration not rebooted at the same time | UPS Management Devices & PowerChute Software

Home » Spaces » UPS Management Devices & PowerChute Software » discussion » General » UPS'es in Redundant configuration not rebooted at the same time

UPS'es in Redundant configuration not rebooted at the same time

Discussion in UPS Management Devices & PowerChute Software started by Wojciech , 4/11/2021 10:11 AM
Login to follow, share, and participate in this space.
Not a member?Join now
Announcement 

Please review Schneider Electric Knowledge Base articles for assistance with most technical support questions.  http://www.apc.com/us/en/faqs

Posted in: General

UPS'es in Redundant configuration not rebooted at the same time

Subscribe to RSS
  • I have 2 ESXi hosts (Host1 and Host2) managed by remote vCenter Server and powered by 2 Smart-UPS RT 2000 RM XL (UPS1 and UPS2) running in Redundant configuration as shown in picture as below. RT1 is router connecting Country_1 and Country_2 networks

    During power-outage all VM's and both hosts were shutdown ad powered off successfully. But when external power returned these ESXi hosts did not start automatically as expected. They stayed in power OFF mode even their BIOS is set to be powered ON when external power returns. I checked logs of NMC cards in UPS1 and UPS2. See these logs below. Seems that are differences between some parts of these logs. Both UPS'es were commanded at 11:37:11 to start graceful shutdown process before the UPS turns off. During process UPS1  turned power off at 11:57:39 and then turn it on at 11:57:44. While UPS2 turned power off at 12:04:33 and then turn it on at 12:04:38. It seems that UPS2 was powering both ESXi hosts while UPS1 has been rebooting and opposite, UPS1 was powering both hosts while UPS2 has been rebooting. Because of that external power for ESXi hosts has been always ON and hosts stayed OFF at the end of process, for them power was not lost

    Below you can also find PCNS log entries refering to process

    Question: why it happened that both UPS1 and UPS2 were not rebooted at the same time even their NMC cards configuration is the same and they are controlled by the same PCNS causing EXSi hosts to stay OFF?

    PCNS log

    04/07/2021 11:11:50 UPS1 is running on battery power.
    04/07/2021 11:11:50 UPS's have switched to battery power.
    04/07/2021 11:19:37 Available runtime has been exceeded.
    04/07/2021 11:36:50 UPS critical event: On Battery.
    04/07/2021 11:37:04 Shutdown sequence started on Hosts Host1, Host2 in response to UPS critical event: On Battery.
    04/07/2021 11:37:09 PowerChute is turning off UPS1
    04/07/2021 11:37:09 PowerChute is turning off UPS2
    04/07/2021 11:37:14 Multiple UPS's have been commanded to turn off.
    04/07/2021 11:37:14 UPS critical event: UPS turn off initiated.
    04/07/2021 11:37:19 Starting Maintenance Mode Task on Host2
    04/07/2021 11:37:19 Shutting down VMs on Host2
    04/07/2021 11:37:21 Starting Maintenance Mode Task on Host1
    04/07/2021 11:37:21 Shutting down VMs on Host1
    04/07/2021 11:37:22 Starting vApp shutdown process.
    04/07/2021 11:37:22 No vApp to shutdown on Host2
    04/07/2021 11:41:28 Maintenance Mode task canceled on Host1 as there are still powered on VMs. Please verify that sufficient time has been configured for VM/vApp/vCSA VM shutdown duration.
    04/07/2021 11:43:07 Host2 has successfully entered Maintenance Mode.
    04/07/2021 11:44:54 Shutting down Host2
    04/07/2021 11:45:53 Shutting down Host1
    04/07/2021 11:46:09 Multiple Critical Events occurred: Low Battery, On Battery.
    04/07/2021 11:46:19 UPS critical event: Multiple Critical Event.
    04/07/2021 11:46:35 Multiple Critical Events resolved.
    04/07/2021 11:46:35 UPS critical event: Multiple Critical Event resolved.
    04/07/2021 11:46:46 UPS critical event: Multiple Critical Event resolved on Host2, Host1.

    NMC log of UPS1

    07.04.2021,13:00:35,System,"Web user 'admin' logged out from qqq.vvv.xxx.245.",0x001F
    07.04.2021,11:57:47,Device,"UPS: Returned from bypass in response to the UPS front-panel, a user-initiated software command, or transition to critical bypass.",0x01B4
    07.04.2021,11:57:45,Device,"UPS: In bypass in response to the UPS front-panel or a user-initiated software command, typically for maintenance.",0x011C
    07.04.2021,11:57:44,Device,"UPS: The output power is now turned on.",0x0113
    07.04.2021,11:57:40,Device,"UPS: The graceful shutdown period has ended.",0x014F
    07.04.2021,11:57:39,Device,"UPS: The output power is turned off.",0x0114
    07.04.2021,11:57:20,Device,"UPS: Started a reboot process.",0x0117
    07.04.2021,11:57:09,Device,"UPS: A discharged battery condition no longer exists.",0x0108
    07.04.2021,11:56:18,Device,"UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately.",0x0107
    07.04.2021,11:55:58,Device,"UPS: No longer on battery power.",0x010A
    07.04.2021,11:37:11,Device,"UPS: A graceful shutdown process is being used to shut down the load equipment before the UPS turns off. Initiated by PowerChute ns from xxx.yyy.zzz.15.",0x0125
    07.04.2021,11:11:48,Device,"UPS: On battery power in response to an input power problem.",0x0109

    NMC log of UPS2

    07.04.2021,13:00:29,System,"Web user 'admin' logged out from qqq.vvv.xxx.245.",0x001F
    07.04.2021,12:04:41,Device,"UPS: Returned from bypass in response to the UPS front-panel, a user-initiated software command, or transition to critical bypass.",0x01B4
    07.04.2021,12:04:38,Device,"UPS: In bypass in response to the UPS front-panel or a user-initiated software command, typically for maintenance.",0x011C
    07.04.2021,12:04:38,Device,"UPS: The output power is now turned on.",0x0113
    07.04.2021,12:04:34,Device,"UPS: The graceful shutdown period has ended.",0x014F
    07.04.2021,12:04:33,Device,"UPS: The output power is turned off.",0x0114
    07.04.2021,12:04:14,Device,"UPS: Started a reboot process.",0x0117
    07.04.2021,11:58:17,Device,"UPS: A discharged battery condition no longer exists.",0x0108
    07.04.2021,11:57:52,Device,"UPS: The battery power is too low to support the load; if power fails, the UPS will be shut down immediately.",0x0107
    07.04.2021,11:55:58,Device,"UPS: A low battery condition no longer exists.",0x0110
    07.04.2021,11:55:58,Device,"UPS: No longer on battery power.",0x010A
    07.04.2021,11:46:06,Device,"UPS: The battery power is too low to continue to support the load; the UPS will shut down if input power does not return to normal soon.",0x010F
    07.04.2021,11:37:11,Device,"UPS: A graceful shutdown process is being used to shut down the load equipment before the UPS turns off. Initiated by PowerChute ns from xxx.yyy.zzz.15.",0x0125
    07.04.2021,11:11:48,Device,"UPS: On battery power in response to an input power problem.",0x0109

  • Beeing honest, I was hoping that Bill would look at this thread ...

    I still have hope :-)

  • Hi,

    The logs show that both UPS received the power off command simultaneously, so we know both UPS received the command. I suggest you check the outlet group power off delay.

    Open the NMC web interface and go to Configuration, Outlet Groups

    select the outlet group the PCNS client has been assigned to and verify the power off delay is set the same on each UPS.

    When PCNS sends the power off signal to the NMC, it counts down the power off delay and then cuts power to the outlet group. If the delays are different, that would explain why one cut output later than the other. 

  • On 04/15/2021 15:53, Bill said:

    The logs show that both UPS received the power off command simultaneously, so we know both UPS received the command. I suggest you check the outlet group power off delay.

    Open the NMC web interface and go to Configuration, Outlet Groups

    Hi Bill,

    Thanks for your replay :-)

    I have no Outlet Group in my NMC2 Configuration menu. See picture as below

    My both UPS'es are: Smart-UPS RT 2000 RM XL. Do they have Outlet Group feature?

    What else can be reason for different rebooting time?

    BR, Wojtek

     

     

  • Hi,

    If there are no outlet groups then you should check the low battery duration, maximum required delay, and the shutdown delay. The settings are found under Configuration - Shutdown. 

    The UPS will wait the amount of time indicated by one of the following, whichever is greater: Low Battery Duration or Maximum Required Delay After this delay, a further non-configurable two minute delay is counted down. The UPS will then turn off after the user-configurable Shutdown Delay time has elapsed.

    Maximum Required Delay
    • It is the longest shutdown delay needed by any server listed as a PowerChute Network Shutdown client.
    • It is calculated whenever the management interface of the UPS turns on or is reset, or when the Force Negotiation option is selected and you click Apply.
  • On 04/15/2021 22:02, Bill said:

    If there are no outlet groups then you should check the low battery duration, maximum required delay, and the shutdown delay. The settings are found under Configuration - Shutdown. 

    Hi Bill,

    I am pretty sure that these settings were the same for both NMC cards

    BR, Wojtek

Page 1 of 1 (6 items)
Choose your language:  
powered by Communifire
Version 8.0.7757.16597