# wget vs wget



## Dylan (Feb 13, 2014)

I'm sure this is a dumb question, but my scripting knowledge is limited at best. Why does this (snipped from the FreeVPS bench):

wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}'

give a different (lower) speed result than a plain wget like this?

wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip


----------



## sv01 (Feb 13, 2014)

both has similliar result 

1st script :

download test file, hide output and print speed

2n script :

just run standart wget command


----------



## CraigA (Feb 13, 2014)

Dylan,

  I think the speed difference that you are seeing might just be a matter of variance. As sv01 stated, both produce the same results and any post-processing in the first script is done after the download, which wouldn't affect the actual download speed.


----------



## Dylan (Feb 13, 2014)

That's what I thought too, but I can run both commands over and over and consistently get results like these:

# wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}'

24.8MB/s

# wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip

2014-02-13 02:57:07 (51.6 MB/s) - `/dev/null' saved [104874307/104874307]

# wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}' 

24.1MB/s

# wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip

2014-02-13 02:57:19 (57.6 MB/s) - `/dev/null' saved [104874307/104874307]

# wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}' 

25.0MB/s

# wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip

2014-02-13 02:57:28 (61.0 MB/s) - `/dev/null' saved [104874307/104874307]

# wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}' 

24.8MB/s

# wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip

2014-02-13 02:57:38 (60.6 MB/s) - `/dev/null' saved [104874307/104874307]

 

I can even use a different file for the test and mix up the order, yet the discrepancy is still consistent:

 


# wget -O /dev/null http://speed.atl.coloat.com/100mb.test

2014-02-13 03:04:01 (68.9 MB/s) - `/dev/null' saved [104857600/104857600]

# wget -O /dev/null http://speed.atl.coloat.com/100mb.test 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}' 

25.6MB/s


# wget -O /dev/null http://speed.atl.coloat.com/100mb.test 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}'

25.2MB/s

# wget -O /dev/null http://speed.atl.coloat.com/100mb.test 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}'

24.5MB/s

# wget -O /dev/null http://speed.atl.coloat.com/100mb.test

2014-02-13 03:07:30 (64.5 MB/s) - `/dev/null' saved [104857600/104857600]

# wget -O /dev/null http://speed.atl.coloat.com/100mb.test

2014-02-13 03:07:33 (70.8 MB/s) - `/dev/null' saved [104857600/104857600]


# wget -O /dev/null http://speed.atl.coloat.com/100mb.test

2014-02-13 03:08:25 (64.3 MB/s) - `/dev/null' saved [104857600/104857600]

# wget -O /dev/null http://speed.atl.coloat.com/100mb.test 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}' 

25.9MB/s

 

 


I feel like a crazy person here, but there's no way all those can just be a coincidence? I've even tried at completely different times of the day -- same thing.


----------



## NickM (Feb 13, 2014)

Edit: Nevermind, read the results wrong.


----------



## eva2000 (Feb 13, 2014)

yeah looks like variance maybe ?

tried on one of my VPS ... look pretty much the same


```
wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}'
79.3MB/s

 wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip
--2014-02-13 07:18:15--  http://speedtest.wdc01.softlayer.com/downloads/test100.zip
Resolving speedtest.wdc01.softlayer.com... 208.43.102.250
Connecting to speedtest.wdc01.softlayer.com|208.43.102.250|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104874307 (100M) [application/zip]
Saving to: /dev/null

100%[======================================================================================================================>] 104,874,307 74.6M/s   in 1.3s    

2014-02-13 07:18:17 (74.6 MB/s) - /dev/null saved [104874307/104874307]


 wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}'
83.1MB/s

 wget -O /dev/null http://speedtest.wdc01.softlayer.com/downloads/test100.zip
--2014-02-13 07:18:33--  http://speedtest.wdc01.softlayer.com/downloads/test100.zip
Resolving speedtest.wdc01.softlayer.com... 208.43.102.250
Connecting to speedtest.wdc01.softlayer.com|208.43.102.250|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104874307 (100M) [application/zip]
Saving to: /dev/null

100%[======================================================================================================================>] 104,874,307 86.3M/s   in 1.2s    

2014-02-13 07:18:34 (86.3 MB/s) - /dev/null saved [104874307/104874307]
```


----------



## Dylan (Feb 13, 2014)

Out of curiosity, what virtualization is that VPS running on? This is 100% reproducible on both of my Xen VPSes (one with CentOS, one with Ubuntu), but it doesn't seem to happen at all on OpenVZ.


----------



## perennate (Feb 13, 2014)

wget will print out two speeds when you run it. The first speed is part of the process bar, and it is the instantaneous speed; after the command terminates (and when awk processes it) this will be the instantaneous speed at the end of the file download (or rather, the average speed over the last second or so of the download). The second speed is the average speed over the entire download. The discrepancy seems to be due to boosting at your ISP, where they boost the initial download and then since download is large it slows down (this increases speed of loading smaller websites, but not downloading very large files).

The instantaneous speed at the end has the advantage that it is not affected by boosting. The average speed over entire download has the advantage of incorporating more statistics.

Edit: I could be wrong, because I don't know awk well, but this would explain your thing

Edit2: actually probably this is wrong, hm


----------



## perennate (Feb 13, 2014)

Well, pretty sure previous post was wrong, but you can try by:

wget http://example.com/download.dat &> blue.txt

cat blue.txt | awk ...


----------



## BuyCPanel-Kevin (Feb 13, 2014)

What you could do is snip off little portions of the command until it is exactly the same as the second command or it's speed it similar to the second command and pin point where exactly the speed is coming from


----------



## sv01 (Feb 13, 2014)

that's maybe problem with your provider. send them ticket 

you alway run


wget -O /dev/null xxxxxx 2>&1 | awk '/\/dev\/null/ {speed=$3 $4} END {gsub(/\(|\)/,"",speed); print speed}' 
first. Run wget without additional script/command first then run above.
Run both script in 5/10 or 30 minute interval


----------

