I am working with a customer right now that has a VERY peculiar networking problem. They contacted us recently for slow network speeds, but when we tested, they were good. The issue seems to come and go.
They're using a DSL line with 10mbit downstream. Sometimes they get 9 or 10, and sometimes it's under 1. Centurylink tells them it's not on their end. (but don't they always?)
I use a speed sampling script in a few places, and installed it on their network to record network downstream, by sampling a 10MB file hosted at a speed testing site (which should have good bandwidth) every minute. (normally I'd do this every 5 or 10 minutes, but we were looking for more granular results, and at this time network performance is already an issue so the occasional big downloads wouldn't really be noticed)
The script has been running for several days now and has produced very surprising results. In the evening, from around 6 pm to 9 am, speeds remain solid, between 7-10mbit. During the day however, speeds are either good or bad. After reviewing the graph, it appears that a gremlin is rolling a die every 10 minutes of the hour, (0, 10, 20, 30, 40, 50) and deciding if the next 10 minutes will be good or bad. And that's how it goes. Also, at the time the die gets rolled, for that one minute, speeds will also be good. Here is a sample:
2012/11/14 14:20 - 7 - ########|##################
2012/11/14 14:21 - 10 - ########|#############################
2012/11/14 14:22 - 10 - ########|#############################
2012/11/14 14:23 - 10 - ########|##############################
2012/11/14 14:24 - 10 - ########|###############################
2012/11/14 14:25 - 9 - ########|###########################
2012/11/14 14:26 - 9 - ########|############################
2012/11/14 14:27 - 9 - ########|###########################
2012/11/14 14:28 - 10 - ########|############################
2012/11/14 14:29 - 8 - ########|########################
2012/11/14 14:30 - 2 - ########|#
2012/11/14 14:31 - 1 - #####
2012/11/14 14:32 - 1 - #####
2012/11/14 14:33 - 2 - ########
2012/11/14 14:34 - 0 - ###
2012/11/14 14:35 - 6 - ########|################
2012/11/14 14:36 - 1 - ######
2012/11/14 14:37 - 2 - ########
2012/11/14 14:38 - 2 - ########|#
2012/11/14 14:39 - 4 - ########|#######
2012/11/14 14:40 - 8 - ########|########################
2012/11/14 14:41 - 10 - ########|##############################
2012/11/14 14:42 - 9 - ########|##########################
2012/11/14 14:43 - 9 - ########|###########################
2012/11/14 14:44 - 10 - ########|##############################
2012/11/14 14:45 - 10 - ########|#############################
2012/11/14 14:46 - 9 - ########|#########################
2012/11/14 14:47 - 10 - ########|###############################
2012/11/14 14:48 - 9 - ########|##########################
2012/11/14 14:49 - 10 - ########|##############################
2012/11/14 14:50 - 1 - #####
2012/11/14 14:51 - 1 - #####
2012/11/14 14:52 - 2 - ########
2012/11/14 14:53 - 1 - ####
2012/11/14 14:54 - 1 - ####
2012/11/14 14:55 - 1 - ######
2012/11/14 14:56 - 1 - #####
2012/11/14 14:57 - 0 - ###
2012/11/14 14:58 - 1 - #####
2012/11/14 14:59 - 10 - ########|##############################
2012/11/14 15:00 - 1 - ####
2012/11/14 15:01 - 1 - ######
2012/11/14 15:02 - 1 - ######
2012/11/14 15:03 - 1 - ####
2012/11/14 15:04 - 1 - #####
2012/11/14 15:05 - 1 - ####
2012/11/14 15:06 - 1 - #######
2012/11/14 15:07 - 0 - ###
2012/11/14 15:08 - 1 - #####
2012/11/14 15:09 - 7 - ########|##################
2012/11/14 15:10 - 1 - #####
2012/11/14 15:11 - 1 - ####
2012/11/14 15:12 - 1 - #####
2012/11/14 15:13 - 1 - #####
2012/11/14 15:14 - 1 - ####
2012/11/14 15:15 - 1 - ###
2012/11/14 15:16 - 1 - ####
2012/11/14 15:17 - 1 - ###
2012/11/14 15:18 - 1 - #####
2012/11/14 15:19 - 10 - ########|###############################
2012/11/14 15:20 - 1 - ####
2012/11/14 15:21 - 0 - ###
2012/11/14 15:22 - 0 - ###
2012/11/14 15:23 - 1 - ###
2012/11/14 15:24 - 0 - ###
2012/11/14 15:25 - 1 - ######
2012/11/14 15:26 - 0 - ###
2012/11/14 15:27 - 1 - #####
2012/11/14 15:28 - 1 - ####
2012/11/14 15:29 - 10 - ########|###############################
So.... what do we have here? I've never seen anything do this before. The problem never occurs in the evening after about 7pm, and never before 8am, but will continue all day. The speeds shown correspond with when they see slow network performance, so I'm fairly certain the problem is not on the end or route to where I am downloading the test file, the graph is an accurate representation of their internet connection performance. I have some ideas but I'd like to get some unbiased input before I throw that hat into the ring. Let me know what you think.
The full current report is available
here