Logging internet bandwidth with Airport Extreme

[getrealbro]

I’m trying to log the internet bandwidth I use each month. My Airport Extreme is an older model that supports SNMP. These two terminal commands extract the correct counters for downloads and uploads (in bytes).

snmpget -v 2c -c public Hostname ifInOctets.2
snmpget -v 2c -c public Hostname ifOutOctets.2

The results look like this

IF-MIB::ifInOctets.2 = Counter32: 3994037720
IF-MIB::ifOutOctets.2 = Counter32: 2161993538

I would appreciate help automating a log of the extracted data with a shell script that would periodically run both commands and append the results to a text file with the current time.

It would be nice for the resulting text file to be in the following format: dd/mm/yy DownloadCounter UploadCounter. This would make the data easier to load into a spreadsheet for analysis/graphing. But I don't mind cleaning up the log manually if necessary.

FWIW I already own NetUse Traffic Monitor which tells me the downloads and uploads in real time and will report a cumulative total. But it does not maintain a record of these data that can be analyzed/graphed.

Thanks in advance.

—GetRealBro

 

[getrealbro]

Here's my first stab at a Terminal command that produces a log of the current value of my Airport Extreme's download counter every minute.

while true; do snmpget -v 2c -c public SH-Extreme.local ifInOctets.2 | while read line ; do echo -e "$(date +%H:%M:%S)\t $line" ; done ; sleep 60; done >> APlog2.txt

The resulting log requires a little manual cleanup in Text Wangler before importing to Excel. But it works

22:45:22 IF-MIB::ifInOctets.2 = Counter32: 3026571806
22:46:23 IF-MIB::ifInOctets.2 = Counter32: 3026986142
22:47:23 IF-MIB::ifInOctets.2 = Counter32: 3027811370
22:48:23 IF-MIB::ifInOctets.2 = Counter32: 3028436918
22:49:23 IF-MIB::ifInOctets.2 = Counter32: 3028863908
22:50:23 IF-MIB::ifInOctets.2 = Counter32: 3029363584

I could still use a little scripting help to clean up the output and include the upload counter on the same line.

---GetRealBro

 

[getrealbro]

Here’s v2 of the terminal command which retrieves the download and upload counters (in bytes) from my AirPort Extreme every 10 minutes. Note: this command creates an infinite loop that you stop with cntrl-C.

while true; do echo -e $(snmpget -v 2c -c public SH-Extreme.local ifInOctets.2)'\t' $(snmpget -v 2c -c public SH-Extreme.local ifOutOctets.2) | while read line ; do echo -e "$(date "+%Y-%m-%d \t%H:%M:%S")\t $line" | sed 's/IF-MIB::ifInOctets.2 = Counter32://' | sed 's/IF-MIB::ifOutOctets.2 = Counter32://'; done; sleep 600; done >> AirPortLog.txt
​

The resulting AirPortLog.txt looks like this: Date, Time, Download Counter, Upload Counter (in bytes).

2015-12-23 21:40:02 2351170846 1778234690
2015-12-23 21:50:02 2357373810 1846660420
2015-12-23 22:00:02 2365024870 1916433456
2015-12-23 22:10:02 2371788150 1982068200
2015-12-23 22:20:02 2379356720 2040425804​

It’s fairly easy to import these tab separated values into the spreadsheet of your choice and graph the download/upload volume during any given period e.g. sample interval, day, week, month, etc..

Note: These raw octet data must be processed a little to compensate for how the AirPort handles the raw 32 bit counters. And this terminal command only works on older AirPorts, because Apple dropped the SNMP capability from its current crop of tower shaped AirPorts.

If you are interested in more details, post a reply.

—GetRealBro

 

[SailorMoon5]

I'm not sure who to ask, or where to go...Is this normal keychain storage logs?

Apple Persistent State Encryption
Call History User Data Key
com.apple.ids: DeviceHBI
ids: identity-rsa-private-key
iTunes iAd password
Safari Extensions List
Safari WebCrypto Master Key

I didn't create any of these.
normal?

 

[getrealbro]

Here is a graph of the 1st 24 hrs of AirPort monitoring beginning yesterday at 8AM. The red bars are the average download throughput in KB/sec during each sample period (10 minutes). The blue line is the cumulative downloads in MB since the log began.

I’ve already learned quite a bit about our internet bandwidth/usage.
* The 1st two red blips (approx 64 KB/s) are when I was updating a few iOS apps via iTunes.
* The larger group is when we were streaming Netflix video in the evening.
* The last pair were while we were on the web reading news/comics etc. the next morning.

This last pair (approx 192KB/Sec) is much too big for just a little morning web browsing. You can see that traffic on either side of the two larger red bars. After a little investigation I realized that these two bars also include when Apple automatically downloaded the iOS 9.2 update to our iPad. This confirms my suspicion that Apple downloads these automatic iOS updates at their convenience, not ours

—GetRealBro

 

[adam9c1]

What tool did you use to graph?

FYI I used two snmp apps to monitor my AE (from the app store). They were both running at the same time and they were giving me different results during my test.

I needed to test how much data IN and OUT when I was updating an iPad.

I ended up pulling this info instead from my wireless access point (I have wifi turned off on the Airport)

 

[getrealbro]

adam9c1:

The graphing is done in Excel. This can be semi automated if you build the graphs using named ranges that cover a few more rows than the current data. I just insert a bunch of new rows within the graph data range, then copy and paste the new data from the log. Then I drag down a few columns of formulas that convert the raw data to cumulative MB and KB/sec in the graph.

I only posted an example of the download graph. The upload graph is often more interesting because it captures when our iOS devices are doing automatic iCloud backups. I previously assumed that iOS devices would do their iCloud backups at night. But that isn’t the case. The iOS devices will kick off their backups at any time. Even when our slow internet connection is struggling to stream Netflix without buffering No wonder iCloud backups frequently fail.

The Airport Extreme I’m monitoring is our wireless access point. So the raw data is our total traffic. I checked the values by downloading a large file and comparing to the log. BTW the Airport Extreme reports twice the actual traffic and the 32 bit counters wrap back to zero every 4GB. Some network monitoring apps may not correct for these quirks.

---GetRealBro

 

[getrealbro]

Since my last post I converted the bash command into a bash script. This let me set up a cron job that collects the incoming and outgoing data on the AirPort Extreme. The cron job runs every 10 minutes (*/10 * * * * ~/AirPortLog.sh).

I also monitor the incoming and outgoing data on our Macs using the same SNMP technique. Since I run the scripts using the same cron timing I get logs that are synchronized. On our home network, with only a few Macs and iDevices, I can usually deduce which device was up/down loading data via the AirPort Extreme.

These logs are how I recognized that every night my iPhone was desperately trying to backup to the iCloud but not being successful.

This chart of the recent downloads (red bars, left scale) and uploads (blue bars, right scale) has a sample interval of 10 minutes. Those clusters of blue bars at 35MB/10min, translate into over GB of data being uploaded by my iPhone each night as it tried, unsuccessfully, to do an incremental iCloud backup of roughly 45MB. Ironically the full backup for this iPhone is only around 500MB.

—GetRealBro