Helpppp! Entire company not receiving external emails!

paintb4707

---

Long version

---

Well, I'm the brand spanking new Network Admin for a vitamin manufacturer of about 100 users.

Let me explain a little about the job first. When I interviewed, they were specifically looking for an entry level admin that was still in school. I assume this is so that they can pay fairly lower and reimburse you with on-the-job learning. However, now that I'm hired, I'm the ONLY IT guy there. The guy that interviewed me only knew enough to get by to support the end users. Keep in mind this isn't his primary job, he's in charge of the packaging and shipping of the vitamins. So.... here I am jumping into the admin world fairly early into the game with only 7 months of HD experience.

Now its only my second day on the job and this morning everyone stopped receiving external emails. Sending/receiving internally works fine, sending externally works fine, but not receiving externally. This looks terrible for me because I got two things going against me:

1) It's my second day
2) I'm young (19 years old), so I can just imagine what's going through everyone's mind

5 hours later, I'm here at home still trying to send myself an external email and its a no-go. I honestly don't recall changing anything on the exchange server, but if I did, what could possibly be preventing external email from coming in?

We're behind a Firebox Fireguard NAT router. DNS apparently seems to be fine according to the dnsstuff.com, the MX record is being directed to the router WAN IP. Domain name is naturesvalue.com

If it matters, I've spent just about 7 out of my 8 hour work day researching this issue and the only clues lead me to believe its an issue with our ISP's DNS servers. Also considering that my interviewer claims that we have "network issues in bad weather" and that this has happened before and it was supposedly on Lightpath's end.

I'm actually scared to go into work tomorrow because I don't want to face this all day.

---

Short Version

---

Just this morning the entire company stopped receiving external emails. Sending/receiving internally works fine, sending externally works fine, receiving external does not.

Only a single NDR recieved out of the hundreds of test emails I sent.

Reason: Remote SMTP server has rejected address
Diagnostic code: smtp;550 relay not permitted
Remote system: dns;nature.naturesvalue.com

Another funny thing I noticed is, I recieved 2 of my test emails after a 40 minute delay but nothing after that.

I don't recall changing anything on the Exchange server but if I did, what could be preventing external emails from coming in?

We're behind a Firebox Fireguard NAT router and according to dnsstuff.com, the MX record for naturesvalue.com is pointing to our router WAN IP (as it should, correct?)

What confusing me about the whole process is, our actual domain name is nature.naturesvalue.com, however we use Network Solutions to direct traffic from www.naturesvalue.com to our webhost and webmail/mail.naturesvalue.com to our router WAN IP.

sprkymrk

Did you mess with the firewall? DNS? Did the IP Address on the Exchange Server get changed so now the firewall can't forward mail to it? Best thing to do is retrace your steps. If it wasn't you then get your ISP involved, ask for tier 2 right away. If it was something you did, they should be able to pin-point it.

Sorry, but we really don't have enough information to make an educated guess. Good luck, I feel for you.

paintb4707

sprkymrk wrote:

Did you mess with the firewall? DNS? Did the IP Address on the Exchange Server get changed so now the firewall can't forward mail to it?

From what I can tell, I don't see anything wrong with the firebox. SMTP incoming and outgoing is allowed and being forwarded to the exchange server.

DNS I didn't touch and the Exchange server has a static IP, so it hasn't changed.

Even if I enter my email address into dnsstuff it finds the MX record, it can't possibly be our end of the DNS could it?

sprkymrk

paintb4707 wrote:

Even if I enter my email address into dnsstuff it finds the MX record, it can't possibly be our end of the DNS could it?

If DNS stuff reports the correct IP of your firebox, then it's not your ISP DNS. I am assuming here that you are using NAT and private IP's internally?

I would definately go over your Exchange server. Also consider any Anti Virus for SMTP you might be running - Symantec or whatever. A configuration change there could be a problem. However, if it were me, I would be on the phone with the ISP right now (not tomorrow morning when they are busy and you are getting interrupted by everyone asking you when you're going to fix it). As I mentioned, if it is something you did, they should at least be able to point you in the right direction or narrow down the problem. If it happens to be something on thier end at least they can be made aware of it sooner rather than later.

mwgood

Go into the Exchange System Manager.

Right click the Default SMTP Virtual Server and select properties.

Go to the Access tab and click the connection button under the connection control section.

Verify that you are not denying access to the external world.

(I see that port 25 connections from the external world fail for mail.naturesvalue.com.) It seems as if your Exchange server is rejecting connections on port 25, or your firewall isn't working like you say it is. You can see this by running the command:

telnet mail.naturesvalue.com 25


If all else fails - try rebooting the Exchange server - who knows?

paintb4707

mwgood wrote:

Go into the Exchange System Manager.

Right click the Default SMTP Virtual Server and select properties.

Go to the Access tab and click the connection button under the connection control section.

Verify that you are not denying access to the external world.

(I see that port 25 connections from the external world fail for mail.naturesvalue.com.) It seems as if your Exchange server is rejecting connections on port 25, or your firewall isn't working like you say it is. You can see this by running the command:

telnet mail.naturesvalue.com 25


If all else fails - try rebooting the Exchange server - who knows?

You know... I had a feeling it was the firewall too. But Incoming SMTP is allowed to the internal exchange server IP. Outgoing and optional network is also allowed.

mwgood

Did you check the setting I suggested on the Exchange server?

paintb4707

mwgood wrote:

Did you check the setting I suggested on the Exchange server?

I'm not at work right now so I don't have access to the Exchange server. I'm pulling my hair out trying to do last minute research before going in tomorrow morning just so I have some sense of direction.

mwgood

OK - well, for what it's worth...

My connection fails to your Exchange server...


C:\Users\mgood>telnet mail.naturesvalue.com 25
Connecting To mail.naturesvalue.com...Could not open connection to the host, on
port 25: Connect failed

paintb4707

mwgood wrote:

OK - well, for what it's worth...

My connection fails to your Exchange server...


C:\Users\mgood>telnet mail.naturesvalue.com 25
Connecting To mail.naturesvalue.com...Could not open connection to the host, on
port 25: Connect failed

Would NAT have anything to do with that?

mwgood

Possibly.

Here's how I would rate the possibilites based upon the current information you've provided -

1) Exchange server blocking connections
2) NAT or Firewall not functioning properly or misconfigured
3) External DNS misconfiguration issue

You should be able to see on your firewall whether external smtp connections are being allowed or blocked, and what address they are being translated to. So - you should be able to rule that out. Once that is ruled out - if you are getting the right IP address for your mail server, then the Exchange server is probably blocking connections.

Finally - you'll want to remember that your value as an employee is not purely in what you know or don't know - it is whether or not you are able to get the job done.

Since it is your 2nd or 3rd day - I would suggest not spending too much time sweating it out - hire an expert if you need to. The bottom line is to get the job done as quickly as possible.

Netstudent

We have had these kinds of issues and nine times out of ten, it's external DNS. Our data provider hosts our external DNS, but more specifically the problem lied in Reverse DNS on the provider's side.

When nothing changes and I haven't touched our firewall or exchange server, and I come in one morning and it just isn't working, I don;t even hesitate. I phone our provider to check all DNS possibilities and have them do a push, to make sure our domain name is getting replicated out to the internet.

dtlokee

I can telnet to your exchange server now, I don't know if you changed anything but iit seems so be ok at this point. If you're still having issues see what domains the SMTP connector is configured to forward and the status of the "require users to authenticate". External users don't authenticate to the server before they relay to your domain (but whatever you do don't add an "*" domain)

Turgon

CheckDNS.NET is asking root servers about authoritative NS for domain
Got DNS list for 'naturesvalue.com' from e.gtld-servers.net
Found NS record: ns35.worldnic.com[205.178.190.18], was resolved to IP address by e.gtld-servers.net
Found NS record: ns36.worldnic.com[205.178.189.18], was resolved to IP address by e.gtld-servers.net
Domain has 2 DNS server(s)

CheckDNS.NET is verifying if NS are alive
DNS server ns35.worldnic.com[205.178.190.18] is alive and authoritative for domain naturesvalue.com
DNS server ns36.worldnic.com[205.178.189.18] is alive and authoritative for domain naturesvalue.com
2 server(s) are alive

CheckDNS.NET checks if all NS have the same version
All 2 your servers have the same zone version 2007111902

CheckDNS.NET verifies www servers
Checking HTTP server www.naturesvalue.com [64.13.237.35]
HTTP server www.naturesvalue.com[64.13.237.35] answers on port 80
Received: HTTP/1.1 200 OK (Server: Apache/2.0.54) 2d .ba . . .f23 .Nature's Value | A Leading Manufacturer of Vitamins and Nutraceuticals. ); . . 65 .2 .COPYRIGHT NATURES VALUE INC.. 2 .A TVI DESIGN. 47 .2 .0 .

CheckDNS.NET tests mail-servers
Domain naturesvalue.com has only one mail-server
Checking mail server (PRI=10) MAIL.naturesvalue.com [167.206.192.10]
Mail server MAIL.naturesvalue.com[167.206.192.10] answers on port 25
<<< 220 exchange.Nature.Naturesvalue.com Microsoft ESMTP MAIL Service, Version: 6.0.3790.3959 ready at Wed, 21 Nov 2007 10:49:47 -0500
>>> HELO www.checkdns.net
<<< 250 exchange.Nature.Naturesvalue.com Hello [195.60.98.252]
>>> MAIL FROM: <dnscheck@uniplace.com>
<<< 250 2.1.0 [email]dnscheck@uniplace.com....Send[/email]er OK
>>> RCPT TO: <postmaster@naturesvalue.com>
<<< 250 2.1.5 postmaster@naturesvalue.com
>>> QUIT
Mail server MAIL.naturesvalue.com [167.206.192.10] accepts mail for naturesvalue.com
All MX are configured properly


You will need to divide and conquer to get to the bottom of your problem. The name servers for your domain are alive and your mail server is answering on port 25. Website is fine too.

A few tips.

Have your boss release you to attempt to manage the situation, he should be prepared to cover for you while you make some calls and investigate the emergency. Keep him updated of your findings.

Get some help..

Talk to your ISP. From the looks of things they are hosting your external DNS. Get a ticket raised there and have them check everything out. It would be useful to arrange a conference call for this sort of thing so they can check things with you in real time while you look out for your own infrastructure.

It looks like an application issue to me. What about mail relay? Have you been blacklisted? You ought to be checking that out anyway. Mention this to your ISP.

Do you have a mailer? This would be an external facing host that receives email on behalf of your domain for content filtering BEFORE handing it on to your Exchange Server. Many organisations use such a device these days. Exim is one example, and eSafe and Mimesweeper are others. If one is being used check it out.

Mostly, don't panic. Obtain whatever network diagram they have and trace things through hop by hop, check configurations and logs and ensure services are running and the appropriate ports are open. Do NOT change anything until you have cleared it with whoever is your boss.
Don't make things worse.


If the situation persists, try a controlled shutdown of your mailserver services and server.

If you have any support contracts for your Exchange server and firewall get some tickets raised asap with whoever provides external support for these devices so you can get extra help.

Last time I checked you could raise a support call with MS for a couple of hundred dollars. That might be money well spent not least as you can get the Exchange Server checked out anyway which will be useful as you are new.

Keep us informed on your progress. Once you are on top of things be aware that this may very well happen again so you will want to put a case forward for improving matters. This may very well involve bringing a designer in.

paintb4707

Ahhhh I feel like an idiot. Apparently it was the firewall. I use a KVM switch to switch between our 2k3 server and the exchange server. Apparently the configuration on the firewall rolled back to a previous state and incoming SMTP traffic was being sent to the 2k3 server. I guess I got the IPs mixed up and thought it was being sent correctly.

Thanks for all your help guys. What's that old saying, error exists between user and keyboard? Yeah... thats me

dps

It happens, man.
Don't feel bad. Good thing you got it all working and right before Thanksgiving!
It sucks when you have to think of work related problems during holidays.

Maybe your boss screwed everything up so he can test your troubleshooting skills.

snadam

Glad its resolved. As embarrassing as it might be, look at all this VERY handy info that came out of it though! I can relate, I started working for a company as local support when I was 19. Sometimes learning the "hard" way works best. To that affect, I always try to take knowledge from these kind of posts.

As a PART of the troubleshooting process, I would have suggested to run a packet sniffer on exchange and see where exactly the packets are being dropped; if any. This way, you can better determine any trans. issues from the end user to the server. From there, its the FW and on. Came very handy when we were troubleshooting transmission issues with MS ActiveSync and our exchange server a while back.

Our sever quantity doubled in the past year, as did our KVM switch size. Even with everything labeled properly, I still get mixed up once in a blue moon (Hell, BGinfo is even enabled!!!).

Turgon

paintb4707 wrote:

Ahhhh I feel like an idiot. Apparently it was the firewall. I use a KVM switch to switch between our 2k3 server and the exchange server. Apparently the configuration on the firewall rolled back to a previous state and incoming SMTP traffic was being sent to the 2k3 server. I guess I got the IPs mixed up and thought it was being sent correctly.

Thanks for all your help guys. What's that old saying, error exists between user and keyboard? Yeah... thats me

Im glad it's resolved! KVMs can be confusing if you get flustered. From the sounds of things you thought you were looking at a different box and drew the wrong conclusion during your troubleshooting?

Even so that is an alarming fault with your firewall. I would try and get to the bottom of that issue if I was you. I don't know what firewall you are using but that really shouldn't happen. You need something solid there.

Netstudent

I've gotten mixed up on a KVM before too. Had a couple Integrated Voice Response systems that looked identical and loaded a new script in the wrong one. It happens man. Live and learn.