Large upload/download at 2 am; but can't figure out which machine
Good Day
Have some large upload/download bandwith occurring from 2:10 am until around 2:50 am.
About 3GB down, and 4 Up
If I check the Bandwidth by users, I don't see any thing which matches the usage
For the life of me, I can't figure out what machine (or machines) generated the traffic (I think user may of been tricked into running something as we have just been barraged with directed attack spam email over the last week or so - pretty scary when you can't tell if the correspondence is legit or not, and the Antivirus says its safe; but find obfuscated vb scripting embedded in documents)
It shows in both the report server and dimensions in the external bandwidth reports; but I cant correlate what machine, and the external IP address. Been digging through the firewall logs filtering out anything I can identify as good; but unsure if I can determine bandwidth
Anyone have any insight or be able to help me figure this out?
Unit is a M270, running Version : 12.5.3.B616762
Paul
External Bandwidth
TIME UPLOAD (MB) UPLOAD RATE (MBPS) DOWNLOAD (MB) DOWNLOAD RATE (MBPS)
2020-08-18 02:10:00 0.76 0.01 1.97 0.03
2020-08-18 02:20:00 3,168.18 42.24 4,097.65 54.64
2020-08-18 02:50:00 0.27 0 0.91 0.01
Running a DSL 25Mbit down, 10Mbit up so am not even sure that dsl modem throughput can handle
Comments
Hi @PWGS
Where are you looking? Dimension, Windows Log/Report server, or the WebUI?
If you're able to log into the firewall when the issue is happening, the Firewatch page on the WebUI will give you a real time bandwidth page you can click through. That'd likely be the easiest way you can track it down.
If you're logging to dimension, you can also look at firewatch historically in the Dimension page(s).
-James Carson
WatchGuard Customer Support
You can learn more about firewatch here:
(FireWatch)
https://www.watchguard.com/help/docs/help-center/en-US/Content/en-US/Fireware/system_status/firewatch_web.html
-James Carson
WatchGuard Customer Support
If you have managed switches, you can use SNMP monitoring tools which can provide graphs of all of the swicht posts.
I have used MRTG running on Windows in the past, which is free.
https://oss.oetiker.ch/mrtg/
Many use PRTG and other SNMP tools to do things like this.
With your DSL Internet connection it would take about 18 minutes to download the 3 gigs, and almost an hour to upload the 4 gigs, which drastically surpasses the time frame given of about 40 minutes.
Sure it isn't something internal, maybe a WSUS Server pushing out updates?
It's usually something simple.
Hi All
I tried to post the message (think on the 20th), but it was stuck in a pending que awaiting authorization (looks like the 23rd). Was like this for a few days, and so opened a ticket directly with Watchguard; but it was slow to solve (so also posted something spice works - Hi Bruce!)
Just came back in today, and saw the ticket had been posted so following up.
Solution:
Got bumped up to 2nd level Watchguard support after more than a week.............. Issue was with the DSL connection being in a degraded state which was causing some sort issue with the byte counter, or log info being sent from/to the M270 firewall Dimensions/Report. Maybe a ... DSL... You up? No.. You Up, No.. You up, No... for 40 mins.
I should of caught this but was honestly in a panic as I thought we'd been compromised and was under the worse case assumption that data had been copied out and I needed to figure out who, why, and how bad was it.
We had emails sent to us (from one customer) which contain emails we had sent in the past. Then a few days later, a second email source with same m.o. came in, all tying back to this one user, then the crazy upload.
The emails had office vba hacks which a few people opened as the included history was valid.
Ant Virus, Non Execute in local profile, locked down standard windows users, restrictive firewall proxcies with secruity enabled, and office Macro set to high was hopefully enough to stopped attacks
..but then this crazy bandwidth was noted on my bandwidith check right around the time (I check the logs every 1-2 days).
OH NO.. Something must of got through. What no firewall logs... OH NO... Was thinking that someone's in deep and has gotten into the watchguard logging..
Was CLINICALLY insane for a few days.
But after some digging, am sure the email leak wasn't us directly... The external sources confirmed they had been compromised. Phew Lucky it was the external people who'd been compromised (well not for them)
But there was still the large up/down to be accounted for...
And then second line support got back to me with some extracted log lines
From watchguard 2nd line support
2020-08-17 23:29:09 FWStatus, device=wan, unix_time=1597732149.387987, in_octets=6158090406, out_octets=973326405
2020-08-17 23:30:09 FWStatus, device=wan, unix_time=1597732209.518964, in_octets=4294967295, out_octets=4294967295
Looking at event log, it shows that PPPoE link was down and it appears byte count for interface wasn't updated correctly.
2020-08-17 23:29:44 pppoe 6 [eth0 (External_PPOE)]PPPoE session[-1] is disconnected.
2020-08-17 23:29:44 networkd 4 [eth0 (External_PPOE)] Interface is deactivated due to link-monitor failure.
2020-08-17 23:29:44 networkd 6 [eth0 (External_PPOE)] Deactivating external interface
It is false reporting on this case. It wasn't used up all the bandwidth.
#
Phew... So I canceled my plane ticket to Peru, and put my fake ID back in the box under my bed.
James_Carson: It was post-review. Both Rpt/Dimension showed the same
Bruce: The main backbone is managed (don't much use the functionality). Thats something I want to look into. I want more granular reporting of bandwidth in/out, and maybe some active alerts when excessive use is detected (I'm trying to be less reactive and more proactive)
shaazaminator: exactly; but the DSL modem speed is software capped and I have seen it hickup and get better up/down some times... Gotta remember that I was insane around this time, and all bets were off
Thanks for all the Replies.. Sorry for the delay it getting back to you
Paul
Hi All
I tried to post the message (think on the 20th), but it was stuck in a pending que awaiting authorization (looks like the 23rd). Was like this for a few days, and so opened a ticket directly with Watchguard; but it was slow to solve (so also posted something spice works - Hi Bruce!)
Just came back in today, and saw the ticket had been posted so following up.
Solution:
Got bumped up to 2nd level Watchguard support after more than a week.............. Issue was with the DSL connection being in a degraded state which was causing some sort issue with the byte counter, or log info being sent from/to the M270 firewall Dimensions/Report. Maybe a ... DSL... You up? No.. You Up, No.. You up, No... for 40 mins.
I should of caught this but was honestly in a panic as I thought we'd been compromised and was under the worse case assumption that data had been copied out and I needed to figure out who, why, and how bad was it.
We had emails sent to us (from one customer) which contain emails we had sent in the past. Then a few days later, a second email source with same m.o. came in, all tying back to this one user, then the crazy upload.
The emails had office vba hacks which a few people opened as the included history was valid.
Ant Virus, Non Execute in local profile, locked down standard windows users, restrictive firewall proxcies with secruity enabled, and office Macro set to high was hopefully enough to stopped attacks
..but then this crazy bandwidth was noted on my bandwidith check right around the time (I check the logs every 1-2 days).
OH NO.. Something must of got through. What no firewall logs... OH NO... Was thinking that someone's in deep and has gotten into the watchguard logging..
Was CLINICALLY insane for a few days.
But after some digging, am sure the email leak wasn't us directly... The external sources confirmed they had been compromised. Phew Lucky it was the external people who'd been compromised (well not for them)
But there was still the large up/down to be accounted for...
And then second line support got back to me with some extracted log lines
From watchguard 2nd line support
2020-08-17 23:29:09 FWStatus, device=wan, unix_time=1597732149.387987, in_octets=6158090406, out_octets=973326405
2020-08-17 23:30:09 FWStatus, device=wan, unix_time=1597732209.518964, in_octets=4294967295, out_octets=4294967295
Looking at event log, it shows that PPPoE link was down and it appears byte count for interface wasn't updated correctly.
2020-08-17 23:29:44 pppoe 6 [eth0 (External_PPOE)]PPPoE session[-1] is disconnected.
2020-08-17 23:29:44 networkd 4 [eth0 (External_PPOE)] Interface is deactivated due to link-monitor failure.
2020-08-17 23:29:44 networkd 6 [eth0 (External_PPOE)] Deactivating external interface
It is false reporting on this case. It wasn't used up all the bandwidth.
#
Phew... So I canceled my plane ticket to Peru, and put my fake ID back in the box under my bed.
James_Carson: It was post-review. Both Rpt/Dimension showed the same
Bruce: The main backbone is managed (don't much use the functionality). Thats something I want to look into. I want more granular reporting of bandwidth in/out, and maybe some active alerts when excessive use is detected (I'm trying to be less reactive and more proactive)
shaazaminator: exactly; but the DSL modem speed is a software cap and I have seen it hickup and get better up/down some times... Gotta remember that I was insane around this time, and all bets were off
Thanks for all the Replies.. Sorry for the delay it getting back to you
Paul
Aside: Please note, that I still seem to require approval for any message I post -> "Your comment will appear after it is approved."
Hopefully this wont go on for too long, as I have been active in watchguard old forums for 7-10 years, and the only thing thats different is the new forum.
Its kinda a kick in the pants when you can't use the support forum to get quick a response
Bruce - I'm going to message you on spiceworks as I have another question concerning my proxy SSL certificate no longer working after a recent forced windows 2016 server update
P.
Hi All
I tried to post the message (think on the 20th), but it was stuck in a pending que awaiting authorization (looks like the 23rd). Was like this for a few days, and so opened a ticket directly with Watchguard; but it was slow to solve (so also posted something spice works - Hi Bruce!)
Just came back in today, and saw the ticket had been posted so following up.
Note: this reply has been sitting in the Pending Approval for a few days. Today is Sept8/2020 and watchguard support thinks they may have fixed it so I can post without delay.. Hopefully this goes through...
Solution:
Got bumped up to 2nd level Watchguard support after more than a week.............. Issue was with the DSL connection being in a degraded state which was causing some sort issue with the byte counter, or log info being sent from/to the M270 firewall Dimensions/Report. Maybe a ... DSL... You up? No.. You Up, No.. You up, No... for 40 mins.
I should of caught this but was honestly in a panic as I thought we'd been compromised and was under the worse case assumption that data had been copied out and I needed to figure out who, why, and how bad was it.
We had emails sent to us (from one customer) which contain emails we had sent in the past. Then a few days later, a second email source with same m.o. came in, all tying back to this one user, then the crazy upload.
The emails had office vba hacks which a few people opened as the included history was valid.
Ant Virus, Non Execute in local profile, locked down standard windows users, restrictive firewall proxcies with secruity enabled, and office Macro set to high was hopefully enough to stopped attacks
..but then this crazy bandwidth was noted on my bandwidith check right around the time (I check the logs every 1-2 days).
OH NO.. Something must of got through. What no firewall logs... OH NO... Was thinking that someone's in deep and has gotten into the watchguard logging..
Was CLINICALLY insane for a few days.
But after some digging, am sure the email leak wasn't us directly... The external sources confirmed they had been compromised. Phew Lucky it was the external people who'd been compromised (well not for them)
But there was still the large up/down to be accounted for...
And then second line support got back to me with some extracted log lines
From watchguard 2nd line support
2020-08-17 23:29:09 FWStatus, device=wan, unix_time=1597732149.387987, in_octets=6158090406, out_octets=973326405
2020-08-17 23:30:09 FWStatus, device=wan, unix_time=1597732209.518964, in_octets=4294967295, out_octets=4294967295
Looking at event log, it shows that PPPoE link was down and it appears byte count for interface wasn't updated correctly.
2020-08-17 23:29:44 pppoe 6 [eth0 (External_PPOE)]PPPoE session[-1] is disconnected.
2020-08-17 23:29:44 networkd 4 [eth0 (External_PPOE)] Interface is deactivated due to link-monitor failure.
2020-08-17 23:29:44 networkd 6 [eth0 (External_PPOE)] Deactivating external interface
It is false reporting on this case and there was no real bandwidth usage.
#
Phew... So I canceled my plane ticket to Peru, and put my fake ID back in the box under my bed.
James_Carson: It was post-review. Both Rpt/Dimension showed the same
Bruce: The main backbone is managed (don't much use the functionality). Thats something I want to look into. I want more granular reporting of bandwidth in/out, and maybe some active alerts when excessive use is detected (I'm trying to be less reactive and more proactive)
shaazaminator: exactly; but the DSL modem speed is a software cap and I have seen it hickup and get better up/down some times... Gotta remember that I was insane around this time, and all bets were off
Thanks for all the Replies.. Sorry for the delay it getting back to you
Paul
Oh no.. Please ignore the double posts as I thought they hadn't posted as they were still sitting in the drafts section waiting to be approved
Hopefully not double posting going forward.
Paul