network bogging challenge!

Print Thread

#24163 11/15/12 03:19 PM

Joined: Aug 2009

Iowa

Virtual1

Virtual1

Joined: Aug 2009

Iowa

I am working with a customer right now that has a VERY peculiar networking problem. They contacted us recently for slow network speeds, but when we tested, they were good. The issue seems to come and go.

They're using a DSL line with 10mbit downstream. Sometimes they get 9 or 10, and sometimes it's under 1. Centurylink tells them it's not on their end. (but don't they always?)

I use a speed sampling script in a few places, and installed it on their network to record network downstream, by sampling a 10MB file hosted at a speed testing site (which should have good bandwidth) every minute. (normally I'd do this every 5 or 10 minutes, but we were looking for more granular results, and at this time network performance is already an issue so the occasional big downloads wouldn't really be noticed)

The script has been running for several days now and has produced very surprising results. In the evening, from around 6 pm to 9 am, speeds remain solid, between 7-10mbit. During the day however, speeds are either good or bad. After reviewing the graph, it appears that a gremlin is rolling a die every 10 minutes of the hour, (0, 10, 20, 30, 40, 50) and deciding if the next 10 minutes will be good or bad. And that's how it goes. Also, at the time the die gets rolled, for that one minute, speeds will also be good. Here is a sample:

Code:


2012/11/14 14:20 -    7 - ########|##################
2012/11/14 14:21 -   10 - ########|#############################
2012/11/14 14:22 -   10 - ########|#############################
2012/11/14 14:23 -   10 - ########|##############################
2012/11/14 14:24 -   10 - ########|###############################
2012/11/14 14:25 -    9 - ########|###########################
2012/11/14 14:26 -    9 - ########|############################
2012/11/14 14:27 -    9 - ########|###########################
2012/11/14 14:28 -   10 - ########|############################
2012/11/14 14:29 -    8 - ########|########################
2012/11/14 14:30 -    2 - ########|#
2012/11/14 14:31 -    1 - #####
2012/11/14 14:32 -    1 - #####
2012/11/14 14:33 -    2 - ########
2012/11/14 14:34 -    0 - ###
2012/11/14 14:35 -    6 - ########|################
2012/11/14 14:36 -    1 - ######
2012/11/14 14:37 -    2 - ########
2012/11/14 14:38 -    2 - ########|#
2012/11/14 14:39 -    4 - ########|#######
2012/11/14 14:40 -    8 - ########|########################
2012/11/14 14:41 -   10 - ########|##############################
2012/11/14 14:42 -    9 - ########|##########################
2012/11/14 14:43 -    9 - ########|###########################
2012/11/14 14:44 -   10 - ########|##############################
2012/11/14 14:45 -   10 - ########|#############################
2012/11/14 14:46 -    9 - ########|#########################
2012/11/14 14:47 -   10 - ########|###############################
2012/11/14 14:48 -    9 - ########|##########################
2012/11/14 14:49 -   10 - ########|##############################
2012/11/14 14:50 -    1 - #####
2012/11/14 14:51 -    1 - #####
2012/11/14 14:52 -    2 - ########
2012/11/14 14:53 -    1 - ####
2012/11/14 14:54 -    1 - ####
2012/11/14 14:55 -    1 - ######
2012/11/14 14:56 -    1 - #####
2012/11/14 14:57 -    0 - ###
2012/11/14 14:58 -    1 - #####
2012/11/14 14:59 -   10 - ########|##############################
2012/11/14 15:00 -    1 - ####
2012/11/14 15:01 -    1 - ######
2012/11/14 15:02 -    1 - ######
2012/11/14 15:03 -    1 - ####
2012/11/14 15:04 -    1 - #####
2012/11/14 15:05 -    1 - ####
2012/11/14 15:06 -    1 - #######
2012/11/14 15:07 -    0 - ###
2012/11/14 15:08 -    1 - #####
2012/11/14 15:09 -    7 - ########|##################
2012/11/14 15:10 -    1 - #####
2012/11/14 15:11 -    1 - ####
2012/11/14 15:12 -    1 - #####
2012/11/14 15:13 -    1 - #####
2012/11/14 15:14 -    1 - ####
2012/11/14 15:15 -    1 - ###
2012/11/14 15:16 -    1 - ####
2012/11/14 15:17 -    1 - ###
2012/11/14 15:18 -    1 - #####
2012/11/14 15:19 -   10 - ########|###############################
2012/11/14 15:20 -    1 - ####
2012/11/14 15:21 -    0 - ###
2012/11/14 15:22 -    0 - ###
2012/11/14 15:23 -    1 - ###
2012/11/14 15:24 -    0 - ###
2012/11/14 15:25 -    1 - ######
2012/11/14 15:26 -    0 - ###
2012/11/14 15:27 -    1 - #####
2012/11/14 15:28 -    1 - ####
2012/11/14 15:29 -   10 - ########|###############################

So.... what do we have here? I've never seen anything do this before. The problem never occurs in the evening after about 7pm, and never before 8am, but will continue all day. The speeds shown correspond with when they see slow network performance, so I'm fairly certain the problem is not on the end or route to where I am downloading the test file, the graph is an accurate representation of their internet connection performance. I have some ideas but I'd like to get some unbiased input before I throw that hat into the ring. Let me know what you think.

The full current report is available here

I work for the Department of Redundancy Department

Re: network bogging challenge! Virtual1 #24164 11/15/12 03:48 PM
Joined: Aug 2009 Likes: 16 Fort Worth, Texas joemikeb Moderator
joemikeb Moderator Joined: Aug 2009 Likes: 16 Fort Worth, Texas	intriguing situation. I have some questions about the network configuration. Is the LAN wired, wireless, or both? Is your script running on a machine that has a wired connection to the router? If it is a WiFi LAN have you run something like AirRadar to see what the noise levels are during the slowdowns? Is there a file server or some sort of backup server on the LAN? Are you sure everyone is holding their mouth right? If we knew what it was we were doing, it wouldn't be called research, would it? — Albert Einstein

Re: network bogging challenge! joemikeb #24165 11/15/12 04:11 PM
Joined: Aug 2009 Iowa Virtual1 OP
OP Virtual1 Joined: Aug 2009 Iowa	1. their dsl modem broadcasts their public wifi, which doesn't have enough range to get outside their property tho it has a password. ethernet feeds into a firewall router and then to a wired and several waps on the private lan. Both the public wifi and the private wifi/ethernet are affected. 2. script is running on their server which is on the wired private lan. 3. not tested, but probably not relevant 4. no. they are backing up locally to a drobo connected via firewire800. 5. not really now that you mention #1 though I think I will see if someone is messing with their public wifi, although that's not a likely cause. I work for the Department of Redundancy Department

Re: network bogging challenge! Virtual1 #24176 11/16/12 04:46 PM
Joined: Aug 2009 Iowa Virtual1 OP
OP Virtual1 Joined: Aug 2009 Iowa	I have generated a new full report along with a 24 hr average, those reports were moved here I work for the Department of Redundancy Department

Re: network bogging challenge! Virtual1 #24177 11/16/12 08:33 PM
Joined: Aug 2009 Likes: 16 Fort Worth, Texas joemikeb Moderator
joemikeb Moderator Joined: Aug 2009 Likes: 16 Fort Worth, Texas	Many years ago, when I was working at a Microsoft facility, network problems such as this happened more often than anyone wanted to admit. The cause was almost always traced to an ethernet problem. The causes ranged from intermittently flakey NICs to bad fluorescent ballasts too close to an unshielded twisted pair. Isolating the problem often required the use of signal analyzers, oscilloscopes, and selectively isolating network nodes. Ie. a lot of detective work. (It was that kind of problem that made me glad I was a software engineer and not a hardware tech.) The regular ten minute intervals would seem to rule out noisy NICs, but makes me suspect interference from some electrical device/motor that is electrically noisy, operates cyclically, and has an unshielded or improperly grounded shielded twisted pair running under/over/behind, or anywhere in close proximity. Since this is apparently a new situation a recently relocated machine or computer might be a logical place to start. If we knew what it was we were doing, it wouldn't be called research, would it? — Albert Einstein

Re: network bogging challenge! joemikeb #24178 11/16/12 08:40 PM
Joined: Aug 2009 Iowa Virtual1 OP
OP Virtual1 Joined: Aug 2009 Iowa	it's not an approximate 10 minute interval though, it's clock-bound. so I think that rules out things like thermostats and other mechanical and non-timebased timers. I work for the Department of Redundancy Department

Moderated by alternaut, dianne, MacManiac

Link Copied to Clipboard

Home Forums Networks & Cross-Platform Networking network bogging challenge!