Raid array verification

[Virtovo]

Granted, I've never run an 8 disk array before (generally four or twelve).  If anyone has an experience of an 8 disk raid 10 array, what kind of speeds would you expect to see from the misguided but widely accepted DD test?

Spec of the array:

8 x WD RE4 2TB

LSI 9621-8i

I would have expected to be able to plot an 8 disk array on a line between the four and twelve disk array.

 

[TruvisT]

What kind of results are you currently getting?

 

[Virtovo]

What kind of results are you currently getting?

Right around 300 MB/s

 

[serverian]

Right around 300 MB/s

Write back?

 

[Virtovo]

Write back?

Yes write-back and cache enabled. 

 

[serverian]

Yes write-back and cache enabled. 

That card doesn't have a cache. It's a pretty low-end card so you are probably hitting its limits. For better results use a real RAID card like 9260.

 

[Virtovo]

That card doesn't have a cache. It's a pretty low-end card so you are probably hitting its limits. For better results use a real RAID card like 9260.

I was referring to Disk cache enabled.  Card has 512mb cache.  9261 is AFAIK pretty much the same as 9260 except a different board layout.

 

[ndelaespada]

I have a server with a p400 (which is old and only has 256mb cache) with 4x1tb and I get 250mb/s.. so if you're getting only 300mb/s out of 8 disks in raid 10 you really need to consider another raid controller, better hds may also help improve i/o speed

 

[Virtovo]

I have a server with a p400 (which is old and only has 256mb cache) with 4x1tb and I get 250mb/s.. so if you're getting only 300mb/s out of 8 disks in raid 10 you really need to consider another raid controller, better hds may also help improve i/o speed

Which is exactly why the results were confusing as I can push just under 300mb/s on a four disk H/W raid 10 array or just under 250mb/s on a four disk S/W raid 10 array.

I know the 9261 is not up there with the likes of a 9271; however I certainly wouldn't consider a 9261 a low end raid card.  Also WD RE4s aren't the latest a greatest now, but again no slouches either.  

 

[serverian]

I was referring to Disk cache enabled.  Card has 512mb cache.  9261 is AFAIK pretty much the same as 9260 except a different board layout.

Oh, you have written 9621-8i instead of 9261-8i on your post. That's why I got confused. Run this command:

dd if=/dev/zero of=test bs=64k count=7k conv=fdatasync; unlink test

 

[Virtovo]

Oh, you have written 9621-8i instead of 9261-8i on your post. That's why I got confused. Run this command:

dd if=/dev/zero of=test bs=64k count=7k conv=fdatasync; unlink test

Ahh, total typo there, thanks for pointing it out.  It is indeed a 9261.  

dd if=/dev/zero of=test bs=64k count=7k conv=fdatasync; unlink test
7168+0 records in
7168+0 records out
469762048 bytes (470 MB) copied, 1.34486 s, 349 MB/s

 

[serverian]

Ahh, total typo there, thanks for pointing it out.  It is indeed a 9261.  


dd if=/dev/zero of=test bs=64k count=7k conv=fdatasync; unlink test
7168+0 records in
7168+0 records out
469762048 bytes (470 MB) copied, 1.34486 s, 349 MB/s

So, even it's hitting the 100% cache, it's still slow. I don't have experience with that specific card. I'd ask the DC is that's what they usually get with that setup.

 

[Virtovo]

So, even it's hitting the 100% cache, it's still slow. I don't have experience with that specific card. I'd ask the DC is that's what they usually get with that setup.

They are saying that sub 300 is normal; however based on my experience with other cards and different configs it can't be the case.  I can push around the 300 mark on a four disk array on a lesser card.  This should show a linear increase in performance (doubled), which is why I was puzzled.  I just wanted to see if anyone had any experience with an 8 disk array as I don't directly so can't quote actual numbers to the DC.

 

[serverian]

They are saying that sub 300 is normal; however based on my experience with other cards and different configs it can't be the case.  I can push around the 300 mark on a four disk array on a lesser card.  This should show a linear increase in performance (doubled), which is why I was puzzled.  I just wanted to see if anyone had any experience with an 8 disk array as I don't directly so can't quote actual numbers to the DC.

8 x Seagate ES.3 RAID10 on LSI 9271-8i:

1073741824 bytes (1.1 GB) copied, 1.52601 s, 704 MB/s

 

[Virtovo]

8 x Seagate ES.3 RAID10 on LSI 9271-8i:

1073741824 bytes (1.1 GB) copied, 1.52601 s, 704 MB/s

Well I was expecting around 600-650 from this setup.  The ES.3 are superior drives and obviously the 9271 is a superior card so that makes sense with your result.  Did you do anything special to the filesystem beyond general alignment and read-ahead?

 

[serverian]

Well I was expecting around 600-650 from this setup.  The ES.3 are superior drives and obviously the 9271 is a superior card so that makes sense with your result.  Did you do anything special to the filesystem beyond general alignment and read-ahead?

All default settings.

 

[Virtovo]

All default settings.

Thanks.  At least I've got something to go back to the provider with.  

 

[Virtovo]

Well even with trying some things I've seen in the past to cause this (such as switching the GEN3 PCIE slots to GEN2 and adjusting PCIE timings) I'm still unable to get anything better.  I've let the company know and they will be looking into it Monday.  I'll update this thread with any solution as it may server as a reference for others in the future. 

 

[Patrick]

You should be getting double, with 4x1TB RE4s we were getting 300 mb/s with LSI 9260-4i. 

I can take a look for you if you want, PM me over your Skype or msg me on IRC

 

[Virtovo]

You should be getting double, with 4x1TB RE4s we were getting 300 mb/s with LSI 9260-4i. 

I can take a look for you if you want, PM me over your Skype or msg me on IRC

It was my thinking too.  I appreciate the offer Patrick.  I've checked everything I can possibly think of and have now handed it back over to the server provider who are giving it the once over, re-configuring RAID and doing an OS install themselves.

If this heads nowhere, I'll shoot you a PM. Again, I appreciate the interest!