Loading ...
Home » Spaces » Smart-UPS & Symmetra LX / RM » discussion » General » Symmetra LX 16000 RM - UPS: Turned off - during power outage - No on battery event.

Symmetra LX 16000 RM - UPS: Turned off - during power outage - No on battery event.

Discussion in Smart-UPS & Symmetra LX / RM started by Alex , 13 days ago
Login to follow, share, and participate in this space.
Not a member?Join now
  • aphillips
    Alex
    Novice Novice
    Alex 13 days ago

    As the title states I have a Symmetra LX 16000 RM.  The last self test was may 8th and came back as fine.   The unit says it has 15 minutes of run time.

    We had a Momentary power failure and the unit just shut off, there was no on battery notification.


    08/02/2017 17:32:00 UPS: Turned off.

    The UPS turned back on 7 seconds later and it was stated that it had 100% battery until  the following about an hour later this was in the logs:


    08/02/2017 19:03:57 UPS: Runtime below alarm threshold cleared.
    08/02/2017 19:02:38 UPS: Battery discharge condition cleared.
    08/02/2017 18:56:57 UPS: Batteries discharged.
    08/02/2017 18:56:57 UPS: Runtime below alarm threshold.

    The batteries still say they are 100% and are only about 18 months old.  I am trying to figure out what might have happened.

    Thanks,

    Alex

  • ipicKedawinna
    Angela
    =S= Representative
    Angela 13 days ago (in response to Alex)

    Hi Alex,

    It will be extremely helpful if you can provide the full event.txt and data.txt files for review from the onboard management card which you've provided snippets from. Or, depending on your Network Mgmt Card model/firmware, the .tar file would be good. More explanation and help for downloading log files here -> http://www.apc.com/us/en/faqs/FA156131

    If you don't feel comfortable with posting them here, you can also ask for help from esupport@apcc.com or this link (http://www.apc.com/us/en/support/contact/ask_apc.cfm) to have support review them.

  • aphillips
    Alex
    Novice Novice
    Alex 13 days ago (in response to Angela)

    I added files, but the data file does not go back far enough.

  • ipicKedawinna
    Angela
    =S= Representative
    Angela 13 days ago (in response to Alex)

    Thanks Alex. First thing I noticed is that I think you have really old firmware on the Network Mgmt Card (NMC).

    Can you confirm the interface is blue and white with navigation on the side? If there is a Help->About system option, can you tell me what you see in there for AOS and APP versions? Also the model number? AP9619 maybe?

    I would like to make sure you get upgraded to the latest rev for many years worth of bug fixes including logging capability, etc moving forward.

    Looking at the log, to confirm, the power outage began on Aug 2nd sometime? Do you know what time it started by chance, maybe based on another device's log or your own knowledge? 

    Secondly, you said the power came on 7 seconds later - that was automatically or by human intervention? If by human intervention, what specific steps? Did anyone catch the LEDs on the front panel on the off chance someone was physically there?

    I also noted the following messages in your log which may be key here:

    08/02/2017	17:32:38	UPS: Overload condition cleared.	0x0104
    08/02/2017	17:32:36	UPS: Sensed a load greater than 100 percent of rated capacity.	0x0103

    I looked at the data log (knowing it did not go back to the 2nd as you said) and I see your phase 2 output wattage % hovers around ~68-75%. My guess is that it is possible you had a temporary overload condition that the UPS was not able to sustain. There is a setting I think on the UPS/management card relating to if the UPS should go to bypass during an overload or if it should drop the load. You can check that setting too. Read more here on that -> http://www.apc.com/us/en/faqs/FA157500

    Is there anything by chance downstream you know of that is connected on phase 2 somewhere, possible 120V load, that could have had in-rush current or a high load (air conditioner, vacuum, etc) that may have led to this temporary overload condition that was logged?

    The temporary overload/bypass/drop load may have happened all so quickly that the NMC log didn't catch it. Upgrading the NMC firmware will be good to make sure we have all of the latest enhancements/robustness on logging as possible. Once I confirm your version info, model number, etc I'll help you get the latest version and the card updated.

  • aphillips
    Alex
    Novice Novice
    Alex 13 days ago (in response to Angela)

    Thank you for your response, to expand:

    AOS:  v2.5.3

    APP:  v2.5.3

    Model: AP9619

    Hardware Rev: A10

    The power was out for a half a second, a flicker.   The only items attached the UPS are servers.  When the UPS Shut down so did all the servers, when it came back on 7 seconds later all the servers did to which cause the load issue.   I am getting 7 seconds from the APC logs,  which i was wrong about, it was 27 seconds.   The events as I understand it  Power fluctuated (Storm) --> UPS Immediately failed -->UPS Comes right back on.   Power issue was only momentary. 

    The loads on the two phases are not balanced,  my belief is it happens because all the servers are 2 PSU servers, so whichever rail comes back first, gets all the load.

    I do not have an exact time for the power failure. I was not onsite at the the time.  But I was told the lights flickered and all the servers went off at the same time.

  • ipicKedawinna
    Angela
    =S= Representative
    Angela 13 days ago (in response to Alex)

    Thanks.

    As far as getting the NMC updated, you can use the firmware attached to this post and then use the following procedure to upgrade (refer to NMC1 instructions): http://www.apc.com/us/en/faqs/FA156047

    Please note, I had to .zip the .exe file attachment of firmware for the forum here to accept it. So, make sure when you try to apply the firmware, you have apc_hw02_aos373_sy372.exe.

    Thanks for your explanation on the other questions I had. I'd like to ask my colleague if he could check further into this from the Symmetra side. My expertise is more the network management side beyond what I already observed embarassed

  • ipicKedawinna
    Angela
    =S= Representative
    Angela 13 days ago (in response to Angela)

    One other thing actually I just thought of - you can compare what is in the NMC log to what is in the PowerView menu log that is on the front of the UPS. It usually starts at event #64 which is the most recent and you can walk backwards through it to see what it logged.

  • SymmetraGuru
    David
    =S= Representative
    David 13 days ago (in response to Alex)

    Hi Alex,

    On top of what Angela asked for (powerview logs)could you also provide the serial numbers on the batteries as well as the raw status on one of them? You can get the raw status by going to "Diags" -> "Batts". If you hit enter on the second line of this screen (Status) it should bring up the first battery. If you hit enter 3 more times you will notice the upper right hand corner says page 4/5. The data on this page and on page 5/5 is what we are looking for. 

  • aphillips
    Alex
    Novice Novice
    Alex 12 days ago (in response to Angela)

    On the unit itself, it does say on battery and then shutdown due to low battery.   I did not know the display on the unit was different from the web interface logs.

    I am working on the update now.

  • aphillips
    Alex
    Novice Novice
    Alex 12 days ago (in response to David)

    Battery Info:

  • SymmetraGuru
    David
    =S= Representative
    David 12 days ago (in response to Alex)

    Thanks Alex,

    Did you replace the entire SYBT5 battery module, or did you replace the smaller individual batteries inside each module? 

  • aphillips
    Alex
    Novice Novice
    Alex 12 days ago (in response to David)

    The battery modules were shipped to us with the unit 18 months ago.

  • SymmetraGuru
    David
    =S= Representative
    David 12 days ago (in response to Alex)

    Would you be able to provide page 5/5 for the battery information as well as the serial numbers on the batteries and the LX itself? You can send it here or via a private chat. 

    Thanks. 

  • aphillips
    Alex
    Novice Novice
    Alex 12 days ago (in response to David)

    Thanks David, here is the information.

    .

  • aphillips
    Alex
    Novice Novice
    Alex 11 days ago (in response to David)

    If I attempt to do a Battery Calibration;

    it starts saying 15 minutes.

    30 seconds later it stops the calibration with 28% battery left and 9 minutes run time.

    10 seconds later it is at 99% battery

    10 minutes later it gets to 100% battery.

    Is that a battery problem or a calibration issue?

  • SymmetraGuru
    David
    =S= Representative
    David 9 days ago (in response to Alex)

    Based on the data you provided it would appear that the batteries are in rough shape. There doesn't appear to be much power left in them and some of the data is not what we would typically see. Was this LX purchased from a reseller or secondhand? If this is a new unit I would call tech support so they can determine the cause of the batteries discharging and issue replacement batteries. 

  • aphillips
    Alex
    Novice Novice
    Alex 9 days ago (in response to David)

    I have one final question:

    Is possible for the UPS to state 15 minutes, but only have 1 minute?

  • SymmetraGuru
    David
    =S= Representative
    David 9 days ago (in response to Alex)

    The Intelligence modules estimate the runtime  based off of what a new, fresh battery would provide. From this point the runtime  will be updated based on the performance of the batteries. If a unit never goes on battery then the Intelligence module would not be able to accurately provide the best runtime. This is why we recommend doing a "runtime  calibration" yearly or bi-yearly for units that never get to experience power failures and run off battery. This would be the only time you could have some discrepancy between the runtime  shown and actual runtime  you get. It is usually not as dramatic as you are experiencing.

    The issue with your batteries is the data is not reading correctly. It appears as if the batteries inside the modules have been replaced, as opposed to the entire SYBT5 module being replaced. The batteries inside each module are programmed to a board inside that module which the Intelligence module uses to access them. Swapping out the batteries inside the module would result in invalid data as the new batteries are not programmed/calibrated to the board.  

  • Page 1 of 1 (18 items)
Choose your language:  
powered by Communifire
Version 5.2.6420.11692