Loading ...
Home » Spaces » UPS Management Devices & PowerChute Software » discussion » General » Maintenance, Shutdown - execution order.

Maintenance, Shutdown - execution order.

Discussion in UPS Management Devices & PowerChute Software started by Tom , 9/30/2015 9:17 AM
Login to follow, share, and participate in this space.
Not a member?Join now
  • memphiz
    Tom
    Novice Novice
    Tom 9/30/2015 9:17 AM

    Hello

    Our enviroment is: 4 Esxi 6.0 hosts in vCenter cluster (with DRS enabled)  + fresh install of PowerChute 4.1 Vappliance + 1 APC SMART-UPS VT 15 kVA
    PCNS is configured to shut down VMs on "on battery event"  but we do not want VM to be migrated (migration on PCNS is disabled)

    Everything would work just fine if not one exeption.
    1. If "on battery event" occurs PCNS trying to put all host in maintenance mode.
    2. Because of maintanace mode  DRS is trying to migrate VMs as quick as possible to any avaliable host.
    3. Because hosts are being put into maitanace sequentially (not all at the same time) - VM's are starting to migrate crazy over whole enviroment.

    So there are lots of migration going on - but not shuting down.  Eventually all VM's will be shut, but is this "migration confusion" really necessary ?

    Am'I missing an idea of network shutdown or do we have some inproper settings in PCNS ?Should't PCNS shutdown VM at a first place and then try to put host in maintenance ?

    How we should set our envirometn (VMware + PCNS) to prevent from migrating during shutdown sequence ?

  • wpasquil
    Bill
    =S= Representative
    Bill 9/30/2015 12:58 PM (in response to Tom)

    Tom,

    if you do not want the VMs migrated you need to disable DRS or create a DRS rule that would cause the VMs to be migrated to 1 of the host that are being put into maintained mode which should prevent the migration. 

    VM migration with DRS is discussed starting on page 29 of the user’s guide.

  • memphiz
    Tom
    Novice Novice
    Tom 10/1/2015 7:32 AM (in response to Bill)

    Bill, thank you for your quick reply

    I think creating a DRS rule is't solution to my problem. 
    At a normal conditions (no critical ups event) i would like to operate on a full automated DRS. Creating a rule (as mentioned above) will disturb a normal DRS operations of cluster.

    and another question... :

    After our last shutdown tests most of VM' were shutdown but none of our hosts....
    Hosts were Powered off when UPS ran out of battery power.
    in PCNS log we get:
    Shutdown Host failed for Host 192.168.......
    Luckily all hosts auto-starts after power back, but all of them stuck in maintanace mode.
    Could you please tell whats the reason of "Shutdown Host failed" and "stuck in maintanance mode after restart" ?

  • wpasquil
    Bill
    =S= Representative
    Bill 10/1/2015 12:55 PM (in response to Tom)

    Hi,

    Below is the sequence of shutdown events when vCenter Server is in use.

    1. PowerChute reports that the UPS is on battery.
    2. Shutdown delay for the On Battery event elapses. PowerChute starts a maintenance mode task on each Host. At the same time it sends a command to turn off the UPS or Outlet Group.
    3. PowerChute starts VM shutdown followed by vApp shutdown.
    4. VM/vApp shutdown durations elapse and PowerChute gracefully shuts down the vCenter Server VM. 5. vCenter VM shutdown duration elapses.
    5. PowerChute starts executing the shutdown command file. - VM & vApp Shutdown enabled with a 120 second delay (i.e. 120 seconds allocated for each action to complete). - The option to turn off the UPS or Outlet Group is enabled. - A shutdown command file has been configured with a 120 second duration. When a critical UPS event, such as UPS on Battery occurs, the following sequence is triggered:
    6. Shutdown command file duration elapses. PowerChute shuts down the VMware hosts using the order on the VMware Host Protection page. (The host running the vCenter VM will be shut down last in this scenario).
    7. OS shutdown sequence starts on the PowerChute physical machine.
    8. After a 70 second delay the OS starts to shut down.
    9. UPS waits for greater of Low Battery Duration/Maximum Required Delay (NonOutlet Aware UPS‟s) or the Outlet Group Power Off Delay (initiated during step 2).
    10. UPS turns off after the user-configurable Shutdown Delay time has elapsed or the Outlet Group turns off after the power off Delay elapses.

    In your case most likely the issue is permissions. Once step 5 happens vCenter is powered down PCNS can no longer communicate with the hosts. The hosts get stuck in maintenance mode and since they are not powered down properly do not restart properly. If you have not done so you should create a local account on vCenter and each host that PCNS will use. Application Note 180 page 19 and the User’s Guide page 17 discuss this.

  • memphiz
    Tom
    Novice Novice
    Tom 1/18/2017 1:04 PM (in response to Bill)

    Thanks for the answer but I still cant figure out what's wrong with our config.
    At point 2 you mentioned that PowerChute starts a maintenance mode (and that is a real problem for us)

    What's happening in our environment is that VM's on all 4 hosts starts to migrate from one host to another in search of host that is not on maintenance. You could imagine what happens in the VMware environment when all of the sudden, all VM's trying to migrate on another machine. This take long period of time what eventually causes VM's aren't shut gracefully.

  • wpasquil
    Bill
    =S= Representative
    Bill 1/19/2017 2:55 PM (in response to Tom)

    Hi,

    If you have not done so you should upgrade to PCNS 4.2 and configure for migration. See page 28 of the PowerChute Network Shutdown 4.2 VMware User's Guide.  

    Below is from the troubleshooting section in the PowerChute Network Shutdown 4.2 VMware User's Guide.

    DRS is set to fully automated for the cluster. Enable VM Migration and set the duration.

    If DRS is enabled and set to fully automated for the cluster, VM Migration is enabled by default. If you disable VM Migration while DRS remains fully automated and enabled, when a maintenance mode task begins on the host, DRS will start migrating VMs to other available hosts. If PowerChute begins VM shutdown on the host at the same time as the DRS migration occurs, VMs that are in migration will not successfully shut down. If DRS is enabled and set to fully automated, VM Migration must be enabled in PowerChute with a VM migration duration set, in order to allow Virtual Machines to migrate successfully.

  • wpasquil
    Bill
    =S= Representative
    Bill 1/24/2017 6:24 PM (in response to Bill)

    Hi,

    Here is more information concerning maintenance mode and PowerChute

    PCNS starts a maintenance mode task at the start of the shutdown sequence to prevent VMs from being migrated to a host that is in the process of being shut down - that is the reasoning behind it i.e. to avoid the possibility that a VM might get migrated to the host during the shutdown sequence and consequently skipped for shutdown.

    Also if you are leveraging DRS for VM migration instead of using PCNS to do VM migration then it is necessary to start the maintenance mode task to trigger VM migration. 

  • CGPhelpdesk
    Scott
    Novice Novice
    Scott 2/14/2017 5:47 PM (in response to Bill)

    Hi

    I believe I am experiencing the very same issues and wondered if you could help me?  The issue I have is that I do not want to perform any type of migration under a power outage scenario.  I have DRS fully automated under normal circumstances.

    I have three hosts running on two SURTD3000XLI which I want to configure in redundant mode.

    I simply want to shut down the guests, in a specific order when a critical events occur on BOTH ups'

    The Advice I was given by APC was to switch on VM prioritization on PCNS 4.2 and configure the machines into priority groups.  I cannot however disable migration because DRS is fully automated. 

    At present, I am contemplating using a Powershell script but would rather not.

    The part that confuses me is the mandatory migration? If you can point out the part that I am missing I would be really happy! :)

    Thanks

  • wpasquil
    Bill
    =S= Representative
    Bill 2/14/2017 8:08 PM (in response to Scott)

    Scott,

    Why don't you want to use PCNS prioritization? It will allow you to shutdown in the order you required.

  • CGPhelpdesk
    Scott
    Novice Novice
    Scott 2/15/2017 9:12 AM (in response to Bill)

    Hi Bill

    Thanks for coming back to me.  I really do want to use the prioritization feature as it is really useful for startup and shutdown.  

    The part I am unsure about it that I think that will 'enforce' migration and I do not want to migrate any VM's during a power event.  

    I have just had a reply from APC that basically says if I dont have a spare HA cluster / set of hosts then I should disable DRS :(

    How else can I shut down and start up the machines in order, in a HA cluster? Its a really simple and effective setup in standalone mode.  I was really hopeful that I would be able to do something similar in redundant mode for my cluster.


    Thanks

  • CGPhelpdesk
    Scott
    Novice Novice
    Scott 2/16/2017 4:09 PM (in response to Bill)

    Hi Tom

    Did you every get to the bottom of this one? I am currently in the situation where APC are saying that I need to disable DRS because I do not have a spare set of hosts on a redundant power supply.  It just feels wrong.  I want to be able to shut down my virtual machines, in a specific order when there is a power cut (in the same way you do).  And, during normal operations I want to use DRS.

    This doesnt seem to be an option??? It feels completely wrong or that I am really missing something fundamental.

    Thanks

  • memphiz
    Tom
    Novice Novice
    Tom 3/3/2017 10:55 AM (in response to Scott)

    Hi Scott

    I feel exactly the same.  Am I missing some fundamentals or is there lack of understanding by APC engineers.  It seems that “Do not migrate on shutdown” should be an easy option to implement but the feedback you received from APC is totally unacceptable.  How could they suggest to give up on DRS functionality in order to get proper shutdown… ?
    Maybe they assume that there’s always free cluster with no power issues where you can migrate to, but in small or medium organizations it’s not an option.
    We are using small VM cluster (4 hosts) located in the same (small) datacenter with one power line. If power failure occurs there’s an impact on whole organization. If I had another VM cluster under another power line I wouldn’t even bother to gracefully shutdown the affected one.
    Scott please let me know if you have any solution on this.

    Thanks in advice and sorry for my bad English ;)

  • wpasquil
    Bill
    =S= Representative
    Bill 3/3/2017 4:46 PM (in response to Tom)

    Hi,

    You can enable DRS as fully automatic but if you do so you also need to enable VM migration in PCNS. This will allow the VMs to be migrated prior to PCNS starting the shutdown. 

  • wopper
    Rob
    Novice Novice
    Rob 5/26/2017 10:53 AM (in response to Bill)

    On 3/3/2017 6:46 PM, Bill said:

    Hi,

    You can enable DRS as fully automatic but if you do so you also need to enable VM migration in PCNS. This will allow the VMs to be migrated prior to PCNS starting the shutdown. 

    Hi Bill

    Is there any progression made regarding the unnecessary Maintenance Mode? I have a 2 HOST ESXi cluster both powered via one SMT UPS with a AP9631 card.

    When i pull the power from the UPS PCNS starts with a maintenance mode action in vCenter? Which does not make sense because with two host moving the VM's to each other while they are both on the same UPS? The effect is that the VM's start "dancing" between the hosts and it needs to timeout first (waste of battery runtime) before the graceful shutdown job starts.

    Both of the hosts are powered by the same UPS. In the end i only want that PCNS gracefully shutdown all the VM's and the Hosts. The maintenance mode should become user selectable within PCNS for those who have more hosts on other UPS power feeds available. Or Maintenance mode actions should be disabled by default on "Single mode" PCNS installations?

    Could i get in contact with developers about this, or submit a feature request?

    Thanks Rob

    PS: like any other VMware vSphere user, disabling DRS is not a solution i need it for other benefits.

  • wpasquil
    Bill
    =S= Representative
    Bill 5/26/2017 3:56 PM (in response to Rob)

    Hi,

    A request has already been entered based on end user feedback. The change will be reviewed for a future release of PowerChute. 

  • wopper
    Rob
    Novice Novice
    Rob 5/29/2017 11:08 AM (in response to Bill)

    Hi thanks Bill

  • Ben3
    Ben
    New Member New Member
    Ben 10/22/2017 11:03 AM (in response to Bill)

    Hi Bill - has there been any movement on this change request?  I have a number of circumstances that need this change, would love an update!

  • wpasquil
    Bill
    =S= Representative
    Bill 10/22/2017 3:53 PM (in response to Ben)

    Hi,

    There is no update at this time.

  • paolonesi
    paolo
    New Member New Member
    paolo 1/12/2018 11:27 AM (in response to Bill)

    on this line, we have the same problem with 4 APC of 5000 supporting a datacenter with 20 hosts of which 8 are in DRS,

    as advanced configuration.

    One solution to shutdown would be to avoid putting in maintenance the Hosts which are in DRS, and shutting down the VM on them before, once the VMs on DRSed hosts are down, all the hosts can be put in maintenance safely

    Another solution, would be to command from the Powershute to VCENTER as DRS OFF and then put in maintenance the HOST

    that would in turn shutdown all the VM in them...

    UNFORTUNALY, none of the two solutions can be implemented so far by the Powershute configuration and its conbination with Vcenter6.5... CAN YOU HELP ME ??

    paolo

  • wpasquil
    Bill
    =S= Representative
    Bill 1/15/2018 2:51 PM (in response to paolo )

    Hi,

    An enhancement request has been entered and the ability to change when maintenance mode is entered should be implemented in the next release of PowerChute Network Shutdown.

    I do not have a date for when the next revision will be release but I will announce the release when I occurs. 

  • edtarento
    Ed
    Novice Novice
    Ed 9 days ago (in response to Bill)

    Good news Bill.  Please keep us all posted.  I too have a client who has a DRS Fully Automated Cluster and no other cluster to migrate VMs to.  They really want to keep their DRS Fully Automated (they see an auto vMotion about once every 2 weeks) and have PCNS 4.2 shutdown VMs (not vMotion)

    We'll try the timer option you've specified in several posts.

    BTW their VMware administrator asks if PCNS could issue the "Enter Maintenance Mode" command to all the ESXi hosts at once, as he tells me he can do in vCenter, by highlighting all the hosts, right clicking and selecting "Enter Maint Mode".  He tells me this would also solve the problem as VMs would not be migrated.  

    Cheers

  • Page 1 of 1 (21 items)
Choose your language:  
powered by Communifire
Version 5.2.6420.11692