Incident Response tips

What a day. I had my first taste of incident response today and it didn't go so well. I received a call that an internal user was receiving emails from another internal user, only the user wasn't sending those emails. At first, they thought the email address was being spoofed. I ran some traces on O365 and looking at the header, I could tell the email was definitely not spoofed. The account was compromised. I was able to tell from the Originating IP, that the email was coming from a remote location in the US. I reset the user's password immediately, and logged off all remote sessions for this user ID. We don't allow mail forwarding, so I quickly looked at mailbox rules.The attacker created some mailbox rules to hide the responses from the user they were sending emails to, in a different folder. At this point, it's clear what happened. The password was changed, so I can continue with my investigation. I had the pc off the network and looked at for keyloggers,malware, etc. All clean. I asked the user if they've given out their password to anyone or logged into webmail or any other computer. Their answer was no. I asked if the password was the same or close to any pw that is being used for other services, answer was no.

I checked for failed login attempts and 1 day a few weeks ago, there were about 30 failed attempts from a single IP in China, but there was never a successful attempt from that IP.

I then reviewed o365 logs for this user to see if there are any abnormal logins. Other than the ones from this incident, I could not find a single one. I went back months, and the user had changed their password at the end of last month. I ran a message trace to see if any emails contained hyperlinks that could be phishing links. I couldn't find any.

I feel like I might be missing something very basic, but can't put my finger on it. I have no answer for this, other than the answer I've been saying for a long time, we need multi-factor auth to help combat this. However, that doesn't give me the answer I want, which is how did they get the password in the first place???

My assumption is if it was brute forced or some other kind of password attack, I should see a lot of failed attempts and then a successful attempt.

I'm taking the L on this one.

Find more posts tagged with

Comments

iBrokeIT

Do you require 2FA for all user accounts accessible through Office 365?

There have been numerous credential harvesting attacks targeting o365 and other externally exposed email portals, sometimes originating from other infected legitimate business contacts. Those emails can sometimes appear as an o365 document share request that prompts you to sign in first but the user enters their credential into the phishing site. Minutes later the attacker attempts to sign in their o365 account and succeeds if you don't have 2FA enabled.

We've actually had to block entire domains of two of our largest business partners due to outbreaks described above until they implemented 2FA because keeping up with the infected accounts sending us phishing links was an exercise in futility.

paul78

The victim probably was phished with a powershell one-liner that injected the redirect rule. Did you check the mail logs for the victim? See if any there were any outbound emails that look suspicious. I saw this attack earlier this year and I am told it's common in some industries.

MitM

@iBrokeIT - Not currently, we don't require 2FA for any user. I have it on my account for testing but it was never approved for all. I'm in a new position now, so I have been pushing for it for a few weeks now. I've heard about those harvesting attacks, especially the sharepoint ones. I use Safe Links, so I'd think (and hope) I'd be able to see in my URL trace?

@paul78 I did check the mail logs. Nothing outbound looks suspicious. The rules that were created just moved the emails into a subfolder where the user wouldn't check. The attacker was logged into OWA from a remote computer, responding to their victim, while the actual user was working. The initial login happened in the early morning hours before the employee comes to work. I also believe they were phished, just cannot prove it.

iBrokeIT

Yes, check your web proxy logs, dns logs and/or user browser history for the few day(s) right before the attack. Most likely they deleted the phishing email when they logged in so it may or may not be gone depending on your retention and time of attack.

Lack of 2FA on email portals, VPN and SSO is a huge and unacceptable problem right now that could have prevented this issue. At least you have some ammo now to show a critical need in this area.

Best of luck to you sir!

MitM

yup, its a huge problem. I def have the ammo now!

Thanks for the info

Danielm7

I'm with the others on phishing. My team deals with users like this often enough right after phishing alerts and they almost always claim they didn't see anything suspicious, didn't click the link, etc, when I'm staring at 3 different log sources that said they did.

the_Grinch

I'd figure the user is not being completely honest. The likely scenario is they either have a very easy password or are using something they use everywhere else. With all of the recent break ins (such as LinkedIn) have exposed lots of passwords. The other option I could think of is a piece of malware that was completely run from memory. Once the computer has been restarted poof goes your evidence. I'd agree very possible that some sort of Powershell script is being run on the system. Did you check the scheduled tasks to see if anything was in there?

Did you check the system logs? If they weren't a sophisticated group they probably just flat out cleared the log. If otherwise, there are ways to remove specific logs from a specific day/time. Can you narrow down your time window anymore?

EnderWiggin

I can help out with that. Just send me your password and I'll take a look at the logs.

paul78

Just my 2 cents - 2fa with O365 should hopefully not be a hurdle. Especially since it is provided with O365 without any additional costs. You may want to use this as ammo to get an EDR solution or some sort in place.

gespenstern

Yeah, as others have said, the user is unlikely to be honest. Most likely they were phished.

2FA, unless it is fully seamless (i.e. machine/user certs), is typically a pain for users as it slows things down and the work performance drops down a bit -- business hates it. But that's the price one has to pay for "ubiquity of access" that comes with the cloud territory. I hate cloud if you ask me...

And yeah, as others have said, you need this moron's browser history. Use a Nirsofer's Browser History View tool, for example, a free tool that does miracles. Also, check their browsing history on your firewall, which may not reveal all the URLs (unless you do MitM on it, like your name suggests). If they deleted their browser history view, which is typical for users when they feel threatened by a subsequent investigation that may expose their wrongdoings, try to collect it from shadow copies. But cleaning up the history is a tell by itself: if they did it -- it increases the chance that they did the offense.

EDR won't help against phishing. Phishing is a huge problem and not many things help against it, the most successful one I've seen is the anti-phishing module of Checkpoint's Sandblast endpoint security software. Educating users through security awareness program should be there, but it's barely helpful. Users are just being dumb and in a large enough enterprise someone will click, someone will enter credentials, etc.

What I've found to be helpful on user education part is making sure everything works via SSO and you communicate to the users like every month that under no circumstances any company software asks for user's credentials besides the initial windows logon screen. This makes them suspicious if something asks for their creds.

TechGromit

the_Grinch wrote: »

I'd figure the user is not being completely honest. The likely scenario is they either have a very easy password or are using something they use everywhere else. With all of the recent break ins (such as LinkedIn) have exposed lots of passwords?

I think this is the most common situation, Passwords used on multiple systems. Just have to obtain one and apply to all other accounts users have. Face it the average user is a complete moron, hackers don't have to be rocket scientists, just a tad smarter than the average user.

paul78

gespenstern wrote: »

EDR won't help against phishing.

Agreed. That's not my point. Having an EDR solutions helps a lot with forensics and IR.

Sylabicuma

Have you checked INBOUND emails for this user in an attempt to detect any phishing emails that might have made it past your email security stack? If you find an email that looks suspicious, attempt to correlate web browsing activity for that user within the same time the email was received. Then you have indicators of compromise (domain browsed for possible credential harvesting website, email address that sent the original phishing email) to search through other logs to determine if any other account is compromised. You then can block these indicators to prevent further damage. Have you required every user to reset their passwords? If one account has been compromised, you can't rule out the possibility that more have been.

Have you investigated created accounts in the past 30, 60, 90 days? An attacker will, more than likely, attempt to create an account for persistence (to avoid losing access when they are discovered, and the admin changes the password to the account they compromised).

Did the user have admin access to any resources? If so, you should investigate those resources for any abnormal logins from that user's accounts, or any other account that doesn't seem right.

Long term solution, and you're on the right track, is 2FA. Good luck!

Tekn0logy

MitM wrote: »

My assumption is if it was brute forced or some other kind of password attack, I should see a lot of failed attempts and then a successful attempt.

I think that 2FA can give a false sense of security, convincing end users to use a weak/dictionary password.

Until you can get management to agree on 2FA, what you can implement in the short term is increased password strength rules, reuse/change policy.
It will take time for management to agree, set policy, select vendor, set up 2FA architecture, educate users and distribute tokens. (unless already in place, but waiting for teams to throw the switch) There are a lot of resources at the hackers disposal that make cracking simple passwords a trivial task. Increased password length+strength will make the passwords "resistant" to brute forcing, but not eliminate the threat. I don't know the O365 setup, but willing to gamble that there is a hash of the password somewhere on the system.

Problem is, if you have users on the real internet with untrusted devices, you cant control their level of access on their own PC and if they have functional anti-malware solution. No malware can lead to viruses like a keylogger or proxy installed which would lead to zero failed login attempts. Users on untrusted devices do things like creating a spreadsheet called passwords.xls making easy work for midnight script kiddies.

Lastly, don't always blame the user! Once upon a time, I used IE11 (I accept all responsibilities for my past decisions) and used the default "blank" Bing startup page. One day I get a popup message telling me to dial an 866 number to fix my PC as soon as I opened IE and I'm like holy cr@p, I'm getting a drive-by from King Bill. I'm not running as admin and I killed the process without clicking anywhere on the box. Lo-and-behold, 3 days later at about 3:30am I get a text message from Microsoft that somebody has logged into live.com with my legacy Skype credentials and was trying to change contact info. However I am partly to blame as my Skype account had an incredibly weak password, however when MS took control of Skype, the accounts were merged and I never looked back to change the old credentials since I was already using the new.

the_Grinch

TechGromit wrote: »

I think this is the most common situation, Passwords used on multiple systems. Just have to obtain one and apply to all other accounts users have. Face it the average user is a complete moron, hackers don't have to be rocket scientists, just a tad smarter than the average user.

Yup. I wish I had a dollar for every user that lied to me when something happened on their machine. I get it, no one wants to believe they caused an issue and in IT we should probably start the conversation with something like "this happens all the time so don't worry" or "you aren't in any trouble and the issue is contained, but we need to ascertain how this occurred in order to combat it in the future...you're assistance will greatly enhance our security for future issues". Really no different than questioning a suspect, they'll deny for hours of any wrong doing, but eventually the truth will come out.

cyberguypr

We see a LOT of opportunistic O365 phishing campaigns. Oh man, I could talk about this for days. We do a lot of internal phishing testing exactly because it's such a prevalent issue. Not a month goes by where someone says "no, I didn't click, I swear" and when we show all logs and artifacts that prove the clicking they go silent. Lesson learned: people have no morals and lie. I rather have you accept that you screwed up and we can have a nice talk to make sure it doesn't happen again.

Then there's the lady who argues she didn't click. When knew she clicked but didn't see any proxy hits or artifacts on the machine. She had no mobile device so we were a bit puzzled. Upon investigating it turned out that "well, I thought it looked suspicious so I forwarded it to my home PC and opened it there". (insert head bang here). Lesson learned: we now monitor for phishing test emails being forwarded.

the_Grinch

I steer away from saying most user have no morals and just lie. The environment the past ten years has been one where a mistake might cost you your job later on. Fact is people don't like to admit mistakes, which is often the case of the matter. Who would want to admit that you thought something looked suspicious, but decided to open it anyway? Just about no one would admit that. Factor in that people don't understand technology and they figure they'll be alright to tell the white lie.

It really is the same as when people fall for a scam like the IRS ones, fake car sales or the Nigerian Prince. Lots of things like that will go unreported because they don't want to appear to be stupid. Einstein put it best, "Everybody is a genius. But if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid". Having been on both sides of the house I can attest to how IT people come off when trying to figure out what occurred. End users are dumb, we come off in a holier than though or down right hostile manner and then wonder why we aren't given the truth.

The biggest point to make is that we learn from these things and then turn it into education for the user (along with ourselves). Users see being honest gets the whole thing tied up faster, we learn about tactics to combat it in the future (such as checking for forwarding like one commenter pointed out) and the company becomes safer as a whole.

MitM

@theGrinch, I did review the logs, did not see any abnormal, like powershell but still checking

@gespenstern. Thanks for the tip! I'm definitely going to check out Nirsofer's Browser History View, in case I missed something. I did check the firewall logs and yes, we do TLS Decrypt. I didn't see much but I found some URL Categories not being logged, which I'm annoyed with. Plan to deal with that tomorrow.

@Sylabicuma, I did check all incoming emails since the last password change. I found emails that were for sure phishing attempts but they were quarantined and not released by the user. I also found a few questionable ones that were delivered to the mailbox. I need to look into those more. I have not required all users to reset their passwords, but I've been monitoring logins closely, looking for risky logins. As for domain accounts being created, we have that logged, so no issues there. We don't allow local admin rights

I'm looking into a credential theft feature on Palo Alto, that will check if your users are using their AD password on any external resources. Requires a read-only domain controller but seems like a good feature.

What's interesting about this user is they NEVER use Outlook Web Access, which I was able to confirm. They only check their email on the company device. The person did tell me a portion of what their old password was, and I can tell you, it was NOT very secure. I could easily say that's it, your pw wasn't secure, but I doubt anyone could guess the right password on the first try. I would think I would have saw some bad login attempts followed by a successful one, but that wasn't the case. During my email checks, I did see the user received an email about changing her LinkedIn password, so that could possibly be interesting.

As far as EDR, we use Sophos w Intercept X. They will be adding some EDR functionality soon