Loading ...

[SOLVED] AP7723 ATS front LEDs cycling left to right | Racks, Rack Accessories, & Cooling

Home » Spaces » Racks, Rack Accessories, & Cooling » discussion » General » [SOLVED] AP7723 ATS front LEDs cycling left to right

[SOLVED] AP7723 ATS front LEDs cycling left to right

Discussion in Racks, Rack Accessories, & Cooling started by Adam , 8/15/2018 7:42 AM
Login to follow, share, and participate in this space.
Not a member?Join now
Posted in: General

[SOLVED] AP7723 ATS front LEDs cycling left to right

Subscribe to RSS
  • Malvineous

    Hi all,

    I recently picked up an AP7723 automatic transfer switch in unknown condition and I'd like to see if I can get it back into service.  I've opened it up and everything seems fine inside (no visible damage) however when powering it up the front panel LEDs cycle from left to right, with the Input A/B, arrows, preference A, then preference B LEDs turning on in order, then going back off again, left to right, in the same order.  The 'output' LED stays unlit.  The speed of the cycling varies somewhat unpredictably.

    The NIC gets connected and I am able to ping it (luckily the static IP it is set to is written on a label on the front) however there appear to be no services running as there are no open TCP ports.  Even running a port scan with nmap shows all common TCP ports closed.

    I made up a serial cable to go into the front-panel RJ12 serial port, first testing it on a different AP7723 to confirm the serial cable works, however I cannot get any response from the unit over the serial port.

    If I press the reset button on the front panel then I do get some characters appearing in the terminal, but they are generally nonprintable and seem to be the same ones over and over, maybe 10-15 chars on each (I assume) boot.  The characters appear at irregular intervals and in bursts, so it looks to me like I am listening at the wrong baud rate, so I have tried 1200, 2400, 4800, 9600, 19200, 38400, 57600, and 115200, all 8-N-1, however all this succeeds in doing is changing the number of nonprintable characters I get.

    I read some very helpful messages on this forum from Angela about accessing the bootloader in a SmartUPS management card, however the suggestion there of using 57600 8-N-1 doesn't seem to get any life out of this unit.  Does the bootloader on these ATSG2 units use different serial settings?

    Is there anything else I can try to see if I can get this unit to come back to life?  Is there any documentation that might explain what has failed in this unit?  Thanks in advance for any hints!

  • Malvineous

    Ok, so I got distracted and suddenly the unit clicked and the cycling LEDs went away.  I was able to get into the management card and all.

    After some experimentation, it looks like the network management card was trying to update the controller firmware, but it was failing for some reason (the logs are full of firmware download failure messages).  Although the management card was at the latest firmware (3.9.2) I reflashed it again (via FTP) just to be sure, in case the existing flash data was degrading for some reason.

    Through the serial console (which now works normally at 9600) I was able to go into the ATS control menu and choose 'Download ATS controller firmware' which puts the unit back into the cycling front-panel LEDs (although they now cycle at a regular rate, not spuriously as before) and displays what looks like Xmodem negotiation.  I would like to reflash this controller firmware just to make sure there's no corruption happening however I can't see where I can download it.  Is it possible to get a copy of this from somewhere?  It doesn't seem to be part of the normal firmware package for the management card.

    Hopefully some dodgy flash is all that caused this unit to be discarded and reflashing it will be enough to bring it back to life!

  • ipicKedawinna

    Hi,

    This firmware download mode can also be forced if you press and hold the "preference" button on the front panel. Is it possible someone did that purposely or accidentally? 

    The controller firmware is updated internally by the ATS's Network Management Card (NMC) only. A user can't upgrade it directly. When you upgrade the NMC AOS (APC OS) and ATS application file, it upgrades itself and then pushes the controller firmware to the ATS afterwards. The controller update is embedded in the g2ats 3.9.2 file so it seems to have already been up to date when you started. To reflash it, you would just push the management card firmware download again. This procedure is covered and documented here -> http://www.apc.com/us/en/faqs/FA156178

    I think it is a little misleading that you can hold the preference button down to force a controller upgrade but that may have been an early design decision vs just having the NMC perform the upgrade automatically. Or, I guess it allows the ATS to re-try asking for the controller firmware download from the NMC.

    Hope this helps.

  • Malvineous

    Hi Angela,

    Many thanks for the info!  Not sure on the history of the unit so can't say whether the prefs button was pressed, however the LED display appears nonfunctional as well so looks like something else has happened.  The logs say:

    ---- Event Log ----------------------------------------------------------------
    
         Date: 08/17/2018      Time: 00:39:38
         ------------------------------------
    
         Automatic Transfer Switch : Lost communication with the ATS
    
    
         Date        Time      Event
         --------------------------------------------------------------------------
         08/17/2018  00:39:28  Automatic Transfer Switch: ATS Controller firmware
                               download failed.
         08/17/2018  00:37:56  Automatic Transfer Switch: ATS Controller firmware
                               download failed.
         08/17/2018  00:36:39  System: Console user 'apc' logged in from serial
                               port.
         08/17/2018  00:35:39  Automatic Transfer Switch: ATS Controller firmware
                               download failed.
         08/17/2018  00:34:47  Automatic Transfer Switch: ATS Controller firmware
                               download failed.
         08/17/2018  00:34:17  Automatic Transfer Switch: 7-Segment display hardware
                               failure.
    
       - Exit, - Refresh, - Next, - Delete
    > 
    

    The message "*** DOWNLOAD BEGUN ***" also regularly appears on the serial console.

    I tried holding down the prefs button.  Seems holding it for ~10 seconds reboots the controller, and if you keep holding it during the reboot then it goes into flash mode.  At this point the front LEDs cycle regularly but then become erratic.  When this happens I see a "download begun" log message, so I'm guessing this erratic cycling of the LEDs means the controller is being reflashed.  This seems to be the mode the unit was in when I first got it.  Unfortunately now the logs consist of this:

    ---- Event Log ----------------------------------------------------------------
    
         Date: 08/17/2018      Time: 00:45:32
         ------------------------------------
    
         Automatic Transfer Switch : Lost communication with the ATS
    
    
         Date        Time      Event
         --------------------------------------------------------------------------
         08/17/2018  00:44:39  Automatic Transfer Switch: ATS Controller firmware
                               download failed.
         08/17/2018  00:44:39  Automatic Transfer Switch: ATS Controller firmware
                               download begun.
         08/17/2018  00:44:38  Automatic Transfer Switch: ATS Controller firmware
                               download failed.
         08/17/2018  00:44:38  Automatic Transfer Switch: ATS Controller firmware
                               download begun.
         08/17/2018  00:44:37  Automatic Transfer Switch: ATS Controller firmware
                               download failed.
         08/17/2018  00:44:37  Automatic Transfer Switch: ATS Controller firmware
                               download begun.
    
    <ESC>- Exit, <ENTER>- Refresh, <SPACE>- Next, <B>- Back, <D>- Delete
    > 08/17/2018 00:44:36 Automatic Transfer Switch: ATS Controller firmware download failed. 08/17/2018 00:44:36 Automatic Transfer Switch: ATS Controller firmware download begun. 08/17/2018 00:44:35 Automatic Transfer Switch: ATS Controller firmware download failed. 08/17/2018 00:44:35 Automatic Transfer Switch: ATS Controller firmware download begun. 08/17/2018 00:44:33 Automatic Transfer Switch: ATS Controller firmware download failed. 08/17/2018 00:44:33 Automatic Transfer Switch: ATS Controller firmware download begun. 08/17/2018 00:44:19 Automatic Transfer Switch: ATS Controller firmware download failed. 08/17/2018 00:44:19 Automatic Transfer Switch: ATS Controller firmware download begun. 08/17/2018 00:44:18 Automatic Transfer Switch: ATS Controller firmware download failed. 08/17/2018 00:44:18 Automatic Transfer Switch: ATS Controller firmware download begun. 08/17/2018 00:44:17 Automatic Transfer Switch: ATS Controller firmware download failed.

    <ESC>- Exit, <ENTER>- Refresh, <SPACE>- Next, <B>- Back, <D>- Delete
    >

    So I am guessing if I leave it long enough the flash will (hopefully) succeed, as the unit does seem to work again once that happens.  It just seems rather weird that it keeps trying and failing to flash the controller!  Especially after less than one second...

  • ipicKedawinna

    Hmm, yes, not sure exactly why that is happening as you're saying and showing with that event log. There doesn't seem to be an obvious problem with that preference button sticking, right? I know you don't know the history so we are at a loss there if something that happened caused this odd behavior.

    Not sure if you can do this (if you have attached load being powered) but what if you unplug the ATS from both sources and let it sit a minute (AKA a coldstart) and boot it back up?

    What was in the event log from the when the unit appeared to recover after sitting the first time?

    Another thing you can try at your own risk after removing all AC power from the ATS (and I can't recommend it officially since this product isn't user servicable but I know you opened it already anyway smile) but you can open the lid again and reseating the ATS controller card in its slot. It is the piece that looks like a stick of RAM. I'd pull that out and verify you reseat it properly and securely. Can check the behavior after that as well. 

    I agree the erratic cycling of the LEDs from left to right is indicating the firmware download is starting/failing.

  • Malvineous

    The preference button seems fine - I can use it to change preferences without any trouble and the preference LEDs don't change on their own once set, so doesn't look like there's anything amiss with the button itself.

    You're right, I have opened the unit tongue-out  I did try reseating the "RAM stick" ATC controller originally (didn't realise it was the whole controller, just assumed it was the firmware flash chip) but it didn't change anything.  There's no load as these units are out of service and just on my desk for testing.  They've had the power off overnight for a few nights running so they have been cold started every morning.  For some reason once the controller gets into firmware flash mode, it doesn't seem to exit it (not even after a coldstart) and the only way to get out of that mode seems to be to provide the data to flash.

    So because of this, the unit did not come back after this latest test of holding down the prefs button, having gotten stuck in flash-mode.  I actually have two of these units, the other one appearing to work fine, so as a test I swapped the network controllers (NMC) so that this suspect unit has a known good NMC.  Since this gave me a working LED display, I could see it briefly show d' as it started to download the firmware but almost instantly it flicked back to 0 as the download failed.  I left it a while but it did not successfully update the firmware, nor did it really seem to try again, even though the controller was stuck in firmware flash mode.  I then realised the NMC firmware was older than the original NMC (3.7.2 vs 3.9.2), so I reflashed it via FTP to the latest version.  After the second g2ats firmware file was uploaded and the NMC rebooted, it did try to reflash the controller firmware and for some reason this time it succeeded on the first attempt, and the unit returned to a working state.  This time I saw the LED display show a "d" and an animation during reflashing, and the other LEDs cycled quickly left-to-right but at a uniform speed.

    I'm a bit hesitant to try the prefs button again (to see whether this was a bug that's been fixed in the latest firmware version) seeing as the unit appears to work fine now (apart from the dead 7-seg display).  But if there's anything you'd like me to try to satisfy your own curiosity, let me know.  At any rate, the solution for a G2ATS unit stuck in "cycling LEDs" mode seems to be to just reflash the NMC firmware and then leave it for a few minutes afterwards.

    As far as what was in the logs after the first time it recovered, it's difficult to say because I'm not sure exactly what the time was, and the older log messages are lost because of all the download begun/fail/begun/fail messages.  I think this is the relevant section, taken from the longer log file you can get via FTP:

    08/15/2018      17:46:51        System: Console user 'apc' logged in from serial port.  0x0014
    08/15/2018      17:46:24        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:46:23        Automatic Transfer Switch: ATS Controller firmware download completed.  0x0C1D
    08/15/2018      17:45:20        Automatic Transfer Switch: 7-Segment display hardware failure.  0x0C20
    08/15/2018      17:45:20        Automatic Transfer Switch: Phase Leds/Push-button hardware failure.     0x0C20
    08/15/2018      17:44:59        System: Warmstart.      0x0002
    08/15/2018      17:44:23        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:44:23        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:44:22        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:44:22        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:44:21        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:44:21        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:44:20        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:44:19        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:44:18        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:44:18        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:44:17        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:44:17        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:44:16        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:44:16        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:44:15        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:44:15        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:44:14        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:44:14        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:42:48        Automatic Transfer Switch: 7-Segment display hardware failure.  0x0C20
    08/15/2018      17:42:48        Automatic Transfer Switch: Phase Leds/Push-button hardware failure.     0x0C20
    08/15/2018      17:42:27        System: Warmstart.      0x0002
    08/15/2018      17:41:09        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:41:08        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:41:08        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:41:07        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    08/15/2018      17:41:06        Automatic Transfer Switch: ATS Controller firmware download begun.      0x0C1C
    08/15/2018      17:41:05        Automatic Transfer Switch: ATS Controller firmware download failed.     0x0C1E
    

    What I don't really understand is why it completed when there was no 'download begun' message that came first.  And why did it keep trying to download the firmware and keep failing, once the unit was working again and comms had been established?  Looking further at the logs it seems that the unit kept trying to download the firmware, even after it was successful, until it was next powercycled.  When it had a coldstart event with working firmware there were no further download attempts until I held the prefs button as mentioned before.  Looking now at the latest logs after the unit was off overnight after a successful flash, there are no further messages at all referring to firmware downloads.

    Thanks again Angela for your help with this.  I wish we had access to all the internal documentation you do because it would be fascinating to read!  I did ask APC Australia and they said they have no problem sharing the stuff (service manuals etc.) for all their discontinued products, but they don't have copies of it as only the repair places do, and sadly I wasn't able to contact anyone via e-mail who had access to it.  If you do, I'd love to hear from you!

    Here's a quick photo for your amusement.  Yes that's not the real APC serial cable, it's a Cisco one that I put an RJ12 connector on, real professional stuff smile

    Swapped NMC controllers

  • ipicKedawinna

    Hmm, I am not quite sure what is going on there after reading what you explained. 

    These products are "replace only" worldwide as far as I am aware so there is no secret technical service manual that I or anyone has unfortunately undecided . Anything that may be out there wouldn't be authorized or provided by us for the reason I just mentioned as we don't provide schematics or anything else.

    On 8/16/2018 10:42 PM, Adam said:

    For some reason once the controller gets into firmware flash mode, it doesn't seem to exit it (not even after a coldstart) and the only way to get out of that mode seems to be to provide the data to flash.

    That is interesting. I've never seen a unit get stuck in a mode like this. I wonder what happened to this unit prior to cause this type of behavior. I have to assume something has been damaged electrically here?

  • Malvineous

    It's hard to say.  Given the LED display doesn't work it does look like the unit suffered some damage.  I have two AP8881 three-phase PDUs as well that are faulty, and I was told an electrician got confused with the three phase colour coding (the APC units are the new brown/grey/black and the DC is the old red/blue/white) so he accidentally put 415V between neutral and the phase that powers the NMC.  Since it was only expecting 240V, it doesn't work anymore.  I have to investigate fully but the fuse has blown, and replacing the fuse just blows again so something else has happened too.

    Each rack in this DC is typically configured with two AP8881s and an ATS for equipment that doesn't have redundant PSUs, so the fact that I ended up with these three pieces of equipment, all of which are faulty, suggests that the ATS was also exposed to 415V.

    I separated the NMC in the ATS (it comes apart into two circuit boards) and by testing with the known-good parts from the working ATS I was able to establish that the half with the LED display on it still works fine.  It's only the back half that no longer works.  There's not really a whole lot on the back board so probably not much further I can go with it.

    Re the service manuals and schematics, do you know who within APC/Schneider who would have access to these?  The Australian APC folks weren't sure who to go through, so I'm thinking if I can point them in the right direction then maybe that would help.  Of course all this stuff would be unofficial and not supported, but it sure helps people like me if we can get access to schematics and things, especially when we're working with long discontinued products like the old Smart UPS units.

    Thanks again for your help with this, I really appreciate it!

  • ipicKedawinna

    Hi Adam,

    On 8/25/2018 5:53 AM, Adam said:

    Thanks again for your help with this, I really appreciate it!

    I don't know how helpful I am but thanks embarassed

    On 8/25/2018 5:53 AM, Adam said:

    Re the service manuals and schematics, do you know who within APC/Schneider who would have access to these?  The Australian APC folks weren't sure who to go through, so I'm thinking if I can point them in the right direction then maybe that would help.  Of course all this stuff would be unofficial and not supported, but it sure helps people like me if we can get access to schematics and things, especially when we're working with long discontinued products like the old Smart UPS units.

    While I definitely hear and understand what you're saying and the reasons why you're looking for this type of info, we don't and can't provide schematics for anything as they contain proprietary information. There is also a safety concern with releasing this information as the PDU and ATS products just aren't designed to be user serviceable. Just our engineering teams have access to the level of information you're requesting. There are no secret/internal service guides that anyone in tech support or any other dept has access to for these offers since they don't exist. Whatever exists is just what engineering/design teams develop and document which is not shared outside of their group per what I just explained. 

    We have an FAQ on this here and while it focuses on UPS, the general statement applies to PDU/ATS and most of our products. (I am not super familiar with 3ph and cooling solutions but I imagine it may be the same.) http://www.apc.com/us/en/faqs/FA158607

Page 1 of 1 (9 items)
Choose your language:  
powered by Communifire
Version 6.0.7207.29305