Internal Infrastructure Crash
So on Wednesday, we had a little issue where our entire internal infrastructure went down from 16:00 to Thursday at 10:30.
I was not sure what caused this issue initially but I went to work on the issue, it suddenly came to light what the issue was.
My first response of this was people coming up to me saying they could not connect to the internet and their phones were not working. My first troubleshooting was to try and connect to the Domain Controller, to which I was unable to. This made me promptly make my way over to the server room to investigate.
At first, I thought there was an issue with the DC as it was not responding and I have also had it before where people cannot connect to the internet because of this, so obviously I restart the DC and wait for the internet to come back on... It didn't...
I then took my eyes to the router and firewall to which I restarted as well... There was no luck there either.
At this point, I was getting extremely frustrated as nothing was working and I did not know why.
I was still in the office at 18:30 trying to figure this out when I had to give up and call it a night.
On the way home I was on the phone to BT Business trying to see if it's their end that was the issue and they advised that it was all good with them.
It came to Thursday morning and I got in early at 07:30 to get a good start on this issue.
I called our internal IT support team to work through the issues and see if they can see where the issue lies. It took a few hours for us to figure out that the issue lies with the TP-Link switch that we have.
One of the ports on this was DOS attacking its self with commands, therefore stopping anything else to travel across the network. I had to go one by one and take out each cable to see if traffic would be able to pass through, it took about 30 cable to get there but eventually, I got the correct cable and traffic started working again.
After getting it all up and running again, I decided to investigate where this port connects to the wall sockets. So I finally found out where this port was and what devices were causing the issue.
I spoke to the girl that had plugged a phone into that port and she said "I plugged my phone into the wall and then my internet stopped working, so I just left it for a bit as I was leaving soon", she didn't think to mention that she plugged her desk phone into a port on the wall that said "do not use" on it...
People are stupid!
I was not sure what caused this issue initially but I went to work on the issue, it suddenly came to light what the issue was.
My first response of this was people coming up to me saying they could not connect to the internet and their phones were not working. My first troubleshooting was to try and connect to the Domain Controller, to which I was unable to. This made me promptly make my way over to the server room to investigate.
At first, I thought there was an issue with the DC as it was not responding and I have also had it before where people cannot connect to the internet because of this, so obviously I restart the DC and wait for the internet to come back on... It didn't...
I then took my eyes to the router and firewall to which I restarted as well... There was no luck there either.
At this point, I was getting extremely frustrated as nothing was working and I did not know why.
I was still in the office at 18:30 trying to figure this out when I had to give up and call it a night.
On the way home I was on the phone to BT Business trying to see if it's their end that was the issue and they advised that it was all good with them.
It came to Thursday morning and I got in early at 07:30 to get a good start on this issue.
I called our internal IT support team to work through the issues and see if they can see where the issue lies. It took a few hours for us to figure out that the issue lies with the TP-Link switch that we have.
One of the ports on this was DOS attacking its self with commands, therefore stopping anything else to travel across the network. I had to go one by one and take out each cable to see if traffic would be able to pass through, it took about 30 cable to get there but eventually, I got the correct cable and traffic started working again.
After getting it all up and running again, I decided to investigate where this port connects to the wall sockets. So I finally found out where this port was and what devices were causing the issue.
I spoke to the girl that had plugged a phone into that port and she said "I plugged my phone into the wall and then my internet stopped working, so I just left it for a bit as I was leaving soon", she didn't think to mention that she plugged her desk phone into a port on the wall that said "do not use" on it...
People are stupid!
Oops! I bet that went down well! Great tenacity Ziggy :-)
ReplyDeleteYeah it went down great... At first I thought it was Matt sabotaging me as he had the week off slacking, but then realised he had no idea either :D
DeleteDefinitely wasn't me!! :D
DeleteYeah sure bro!!
DeleteYou can't create a fool-proof system for system-proof fools. Nice one David.
ReplyDeleteBut I can shout at the fools that break our potential fool-proof system.
Delete