Loading ...
Home » Spaces » UPS Management Devices & PowerChute Software » discussion » General » SMTP alerts no longer working

SMTP alerts no longer working

Discussion in UPS Management Devices & PowerChute Software started by Jacob , 1/3/2019 8:09 PM
Login to follow, share, and participate in this space.
Not a member?Join now
Announcement 

Please review Schneider Electric Knowledge Base articles for assistance with most technical support questions.  http://www.apc.com/us/en/faqs

  • Cobb
    Jacob
    Novice Novice
    Jacob 1/3/2019 8:09 PM

    Long post incoming....

    I have a few Smart-UPS RT 6000 RM XL systems on my network that I have configured to send email alerts via a user/mailbox hosted in Office 365. This was working great, however after installing another unit today, my settings no longer work (according to the APC E-mail Test, and the email not reaching my address). I also checked my previously installed units, and they are also receiving the same error now. I'm currently receiving this error: 

    Test e-mail failed; SMTP configuration error.
    Last Server Response
    220 2.0.0 SMTP server ready
    My previous settings that were working correctly in the past, were:
    SMTP Server: outlook.office365.com
    Port: 587
    Authentication Enabled (with a user/email with a real mailbox & password)
    Use SSL/TLS: If Supported

    These settings worked in the past, sending the alert from the specified mailbox to my address hosted within the same o365 org/domain.
    Troubleshooting today, I've updated the NMC FW from v6.5.0 to v6.6.4. Also tried changing SMTP servers to smtp.office365.com, different ports (25,465,587), and Use SSL/TLS settings with no luck. Tested DNS, used an internal server and then Google DNS to verify name resolution. I've also referenced the APC documentation hosted here: https://www.apc.com/us/en/faqs/FA164769/ , still no luck.
    I've confirmed that my firewall/web filter are not blocking this traffic. I also verified I can sign into this mailbox in o365 with the credentials provided within the Authentication settings on the UPS. 
    I've ran out of ideas moving forward to troubleshoot, and why I'm posting, hoping someone can point something out that I may be missing.
    Thank you in advance.
  • Cobb
    Jacob
    Novice Novice
    Jacob 1/3/2019 8:14 PM (in response to Jacob)

    Oops. Forgot to mention that I have other devices on the same network (copier/scanners), using the same SMTP settings, that are currently functioning properly. These devices use a different o365 user. I even tried using that different account on the UPS, same error.

  • ipicKedawinna
    Angela
    =S= Representative
    Angela 1/7/2019 5:07 PM (in response to Jacob)

    Hi Jacob,

    I am seeing the same issue and will be looking into this and debugging.

    I am not quite sure what we'll find yet but just wondering if you're willing to help test or debug if we need it? If so, we can work offline together on this.

    P.S. I am wondering if it has to do with this and any changes they made: https://docs.microsoft.com/en-us/office365/securitycompliance/technical-reference-details-about-encryption?redirectSourcePath=%252fen-ie%252farticle%252ftechnical-reference-details-about-encryption-in-office-365-862cbe93-4268-4ef9-ba79-277545ecf221 

  • Cobb
    Jacob
    Novice Novice
    Jacob 1/7/2019 6:35 PM (in response to Angela)

    Angela,

    Thank you for the response and confirmation of the issue. The o365 TLS change definitely came to mind when troubleshooting this, as we did have to make changes to our o365 setup/connectors to fully support TLS 1.2. We went through that process a few months ago, prior to the cut off date stated by Microsoft. I do know, that these alerts were working AFTER we made those changes for TLS 1.2, and this appears to be something that "broke" about 2-3 weeks ago. I initially contacted APC support about the NMC and if it supported TLS 1.2, and from the documentation provided, the AP9631 NMC card supports this encryption/protocol starting with FW v6.5.0. This is why I updated the FW to the latest release(I think) to v6.6.4 during my troubleshooting steps. Unfortunately, the FW update did not solve the issue, and outputs the same error message.

  • ipicKedawinna
    Angela
    =S= Representative
    Angela 1/7/2019 6:58 PM (in response to Jacob)

    Hi Jacob,

    You by chance don't have access to look at connection logs or do a packet capture on this do you? With the cloud setup, I am not sure if that is really a thing anymore depending on if your O365 is cloud or on-premise and what it allows you to see or do.

    I don't in my environment either so wasn't sure if you did.

  • JJordan
    James
    Novice Novice
    James 1/9/2019 11:20 PM (in response to Jacob)

    Figured I would post this in hopes of saving someone's troubleshooting efforts. 

    I have the exact same problem you described as well as the behavior it working previously and then suddenly stopped working.  I have been working on this for 4+ days and today a Wireshark captured revealed a "Bad Certificate" on the SMTP transmission.

    Here is the packet capture.  Bad certificate (In my case) on host 40.97.165.210  (smtp.office365.com).  I currently have an open Microsoft O365 case for this as well.  

    Angela I have a existing case open on this.  #  54986873.  I haven't updated this ticket with these findings yet because I am waiting to hear Microsoft's response on this.

  • Cobb
    Jacob
    Novice Novice
    Jacob 1/9/2019 11:24 PM (in response to James)

    James, Thank you! I have been swamped the last few days and haven't been able to set up a packet capture to troubleshoot this further. Please post any updates/information that you receive from MS or any steps required to solve this issue(once found).

    Thanks again!

  • ipicKedawinna
    Angela
    =S= Representative
    Angela 1/9/2019 11:44 PM (in response to James)

    Hi James (and Jacob),

    James - thanks so much for sharing. This is a big help. I will add this to our internal issue logged on the topic as well. We are eager to hear what MS says too - if they recently imposed a requirement to use CA certificate verification or something - which we can do on the NMC as we have a section for email certs. I will also make a note in the case so we are all on the same page.

  • ipicKedawinna
    Angela
    =S= Representative
    Angela 1/9/2019 11:46 PM (in response to Angela)

    Also curious why it seemingly works on other devices? Like Jacob, you said it is working on printers and I had it working on another device as well (though I get an authentication error which I am working on with our administrator).

  • JJordan
    James
    Novice Novice
    James 1/10/2019 12:27 AM (in response to Angela)

    Interesting you mention that....  I have dozens of MFP units and applications using authenticated SMTP in my environment.  On the surface, those "things" appears to be working as expected.  

    The only reason I noticed this problem was finding a NMC in my environment using SMTP without authentication.  I went through my normal motions of configuring it to point to smtp.office365.com.  When I performed the mail test, it failed.  Thinking I did something wrong on the config, I checked another NMC that I knew was working (At least I thought I did).  I compared the settings to the original card I was working with and they were identical.  On a whim i tested SMTP on the known working NMC and it surprisingly failed too.  I looked at three other NMC with various firmware's and they all failed the SMTP tests (If you check the ticket, you will see all of the NMCs I have with the various firmware's).  As it stands right now, I have six NMCs failing the SMTP test.  The packet capture I ran was for only one of those NMCs and I presume the packet capture results would reveal the same error for the other NMCs.

    Let's see what Microsoft has to say.

  • JJordan
    James
    Novice Novice
    James 1/10/2019 9:55 PM (in response to James)

    Angela I have heard nothing from MS on this.  If you check the ticket, you should see a set of O365 credentials.  Can you test those on an NMC available to you?

  • ipicKedawinna
    Angela
    =S= Representative
    Angela 1/11/2019 2:15 PM (in response to James)

    Hi James - yes, I just tried your credentials and get stuck at 220 2.0.0 SMTP server ready like with my account too.

  • JJordan
    James
    Novice Novice
    James 1/11/2019 4:02 PM (in response to Angela)

    Here is the reply from MS so far

    "Unfortunately, we are not certificate providers, if that is something with the certificate being bad, you may have to check with the certificate providing authority."

  • JJordan
    James
    Novice Novice
    James 1/15/2019 4:54 PM (in response to Angela)

    Angela, Can I message/email you privately?  

  • ipicKedawinna
    Angela
    =S= Representative
    Angela 1/15/2019 5:55 PM (in response to James)

    Hi James - yes, I'll send you an email using the address in your profile.

  • awallace
    Aaron
    New Member New Member
    Aaron 1/16/2019 2:47 PM (in response to Angela)

    Can we keep updates to this issue public?  Just had an outage at a site and was investigating why no alerts were received.  Went through identical steps to James and some additional searching led me to this thread.  Curious to find a solution and happy to help troubleshoot as needed.

  • ipicKedawinna
    Angela
    =S= Representative
    Angela 1/16/2019 6:39 PM (in response to Aaron)

    Hi Aaron,

    Yes, I'll do that. I have been working with Jim (James) and he has been helping me debug some clients that do and don't work with packet captures and other observations. Will be sharing my analysis I'll be doing today on Jim's data with our development team so we can figure out what is going on.

  • awallace
    Aaron
    New Member New Member
    Aaron 1/17/2019 2:55 PM (in response to Angela)

    Thanks.  Let me know if you need another set of data points and I can open an official ticket.

  • JJordan
    James
    Novice Novice
    James 1/22/2019 4:19 PM (in response to Angela)

    I think I found the issue.  We use Meraki firewalls and I discovered the NMC device being flagged with "SMTP_RESPONSE_OVERFLOW" in the threat category.  Meraki uses SNOT for Intrusion/Threat Protection.  Subsequently, I am also finding that some of our MFP device traffic is also being as well (Not continuously though).

    Here is the specific SNOT SID
    https://snort.org/rule_docs/124-3

    CVE 
    http://cve.mitre.org/cgi-bin/cvename.cgi?name=2002-1090

    I will be placing a call with Meraki to figure out why this this traffic is being flagged and potentially working towards whitelisting rules.

    More to follow....

    Jim

  • exonix
    Alesk
    Novice Novice
    Alesk 1/23/2019 9:57 AM (in response to James)

    Hi James,

    we use Zyxel router, but we have the same problem: we can't connect to O365. Also, you said that it worked before - did you do any changes on Meraki?

  • rrubiojr
    Robert
    New Member New Member
    Robert 2/7/2019 12:29 AM (in response to Alesk)

    We too had been experiencing the same SMTP error using O365, confirmed by doing a wireshark capture as one of the previous posters (also using Meraki firewall appliances). However, noticed since Sunday evening that alerts have resumed. Appears Microsoft has resolved the issue with the certificate being rejected

  • Cobb
    Jacob
    Novice Novice
    Jacob 2/7/2019 10:16 PM (in response to Robert)

    Can confirm, the problem has been resolved on my systems. SMTP email alerts are now functioning as they were before.

  • JJordan
    James
    Novice Novice
    James 2/8/2019 1:40 PM (in response to Jacob)

    All six of my NMCs are now working as well.  The consensus is that this was a Microsoft issue?

  • exonix
    Alesk
    Novice Novice
    Alesk 2/8/2019 2:20 PM (in response to James)

    Could anyone tell me, whether UPS should be connected to the Internet directly via Ethernet to be able to send Emails?

    We have Smart-UPS C 1500I with COM-port only. I configured PowerChute Business Edition to send all types of the notification including any configuration changes, but I still don't receive any emails after any changes.

    Thank you!

  • Page 1 of 1 (24 items)
Choose your language:  
powered by Communifire
Version 5.2.6420.11692