<HTML dir=ltr><HEAD><TITLE>RE: [DRBD-user] Dell Server Question</TITLE>
<META content="text/html; charset=unicode" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 9.00.8112.16437">
<STYLE><!--
font-face
{font-family:"Cambria Math";}
font-face
{font-family:Calibri;}
font-face
{font-family:Tahoma;}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{
color:purple;
text-decoration:underline;}
p
{
margin-right:0in;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle18
{
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{
font-size:10.0pt;}
div.WordSection1
{page:WordSection1;}
--></STYLE>
</HEAD>
<BODY lang=EN-US vLink=purple link=blue>
<DIV dir=ltr id=idOWAReplyText647>
<DIV dir=ltr><FONT color=#000000 size=2 face=Arial>Well I might be able to give you such a sent of benchmarks. I have 2 new servers that are using 3.5" 15k drives so I can pull some random IO patterns and later post them. Plus I have data from some 2.5" drive sets that can be compared against.</FONT></DIV>
<DIV dir=ltr><FONT size=2 face=Arial></FONT> </DIV>
<DIV dir=ltr><FONT size=2 face=Arial>-Morey</FONT></DIV></DIV>
<DIV dir=ltr><BR>
<HR tabIndex=-1>
<FONT size=2 face=Tahoma><B>From:</B> John Lauro [mailto:john.lauro@covenanteyes.com]<BR><B>Sent:</B> Fri 12/9/2011 8:01 PM<BR><B>To:</B> Roof, Morey R.; drbd-user@lists.linbit.com<BR><B>Subject:</B> RE: [DRBD-user] Dell Server Question<BR></FONT><BR></DIV>
<DIV>
<DIV class=WordSection1>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt">You are correct, it’s only 12+2 drives on the R515. I was confusing it with another server (non Dell).</SPAN></P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt"></SPAN> </P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt">You would think that 2.5” drives would be faster. Maybe it’s just the particular drives I have tested with. Have you done any benchmarks, especially random I/O? It was only 10K on the 2.5 drives vs. 15K on the 3.5 with much higher capacities. For my needs, I am mostly interested in random I/O than sequential.</SPAN></P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt"></SPAN> </P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt"></SPAN> </P>
<DIV style="BORDER-BOTTOM: medium none; BORDER-LEFT: blue 1.5pt solid; PADDING-BOTTOM: 0in; PADDING-LEFT: 4pt; PADDING-RIGHT: 0in; BORDER-TOP: medium none; BORDER-RIGHT: medium none; PADDING-TOP: 0in">
<DIV>
<DIV style="BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<P class=MsoNormal><B><SPAN style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt">From:</SPAN></B><SPAN style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt"> Roof, Morey R. [mailto:MRoof@admin.nmt.edu] <BR><B>Sent:</B> Friday, December 09, 2011 9:34 PM<BR><B>To:</B> John Lauro; drbd-user@lists.linbit.com<BR><B>Subject:</B> RE: [DRBD-user] Dell Server Question</SPAN></P></DIV></DIV>
<P class=MsoNormal> </P>
<DIV id=idOWAReplyText44697>
<DIV>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Arial','sans-serif'; COLOR: black; FONT-SIZE: 10pt">The R515 only does 12 drives, not 16 drives and the two internal mean a server shutdown for replacement which is slight annoying.</SPAN></P></DIV>
<DIV>
<P class=MsoNormal> </P></DIV>
<DIV>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Arial','sans-serif'; FONT-SIZE: 10pt">One of the IBM servers (which also has an LSI card) holds 28 2.5" drives. It is the model x3630 M3 and it is another I have been looking at.</SPAN></P></DIV>
<DIV>
<P class=MsoNormal> </P></DIV>
<DIV>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Arial','sans-serif'; FONT-SIZE: 10pt">Very strange that you would say that 3.5 are faster than 2.5. With the higher densities of the 2.5 drives in the 10k range the larger bit densities per cm^2 means a much higher data transfer speed. Also, if you look at the seek times they are pretty much identical and a full drive sweep is a bit faster on the 2.5 than the 3.5 since the drive is smaller.</SPAN></P></DIV>
<DIV>
<P class=MsoNormal> </P></DIV>
<DIV>
<P class=MsoNormal> </P></DIV></DIV>
<DIV>
<P class=MsoNormal> </P>
<DIV style="TEXT-ALIGN: center" class=MsoNormal align=center>
<HR align=center SIZE=2 width="100%">
</DIV>
<P style="MARGIN-BOTTOM: 12pt" class=MsoNormal><B><SPAN style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt">From:</SPAN></B><SPAN style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt"> John Lauro [<A href="mailto:john.lauro@covenanteyes.com">mailto:john.lauro@covenanteyes.com</A>]<BR><B>Sent:</B> Fri 12/9/2011 5:21 PM<BR><B>To:</B> Roof, Morey R.; 'Kushnir, Michael (NIH/NLM/LHC) [C]'; <A href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</A><BR><B>Subject:</B> RE: [DRBD-user] Dell Server Question</SPAN></P></DIV>
<DIV>
<P><SPAN style="FONT-SIZE: 10pt">Supermicro might be your best option, but for comparisons a more standard<BR>model for a Dell is the R515 (or 510) instead of the C-series, with 16<BR>3.5" drives and 2 2.5" drives all in 2U. Unless you count SSD, despite<BR>the higher density (24 vs 16) I haven't found 2.5" drives that really<BR>match the performance of 3.5" drives, although they are getting closer.<BR>Much easier to find trays/drives for the R515 than the C-series.<BR><BR>PS: Often you can get Dell to match prices on systems if you give them a<BR>quote to compete with. Given the shortage of hard drives, it might be<BR>harder than normal right now...<BR><BR><BR>My biggest annoyance with Dell (and LSI as they make the controllers), is<BR>they don't properly do Raid 1/10 despite their configuration info implying<BR>they do. They really only read from 1 drive, so performance-wise you are<BR>better off not utilizing the hardware RAID and making a md raid 1/10 on<BR>top of the individual drives. That said, their RAID 5 performance is<BR>excellent.<BR><BR><BR><BR>> -----Original Message-----<BR>> From: <A href="mailto:drbd-user-bounces@lists.linbit.com">drbd-user-bounces@lists.linbit.com</A> [<A href="mailto:drbd-user-">mailto:drbd-user-</A><BR>> <A href="mailto:bounces@lists.linbit.com">bounces@lists.linbit.com</A>] On Behalf Of Roof, Morey R.<BR>> Sent: Friday, December 09, 2011 2:54 PM<BR>> To: Kushnir, Michael (NIH/NLM/LHC) [C]; <A href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</A><BR>> Subject: Re: [DRBD-user] Dell Server Question<BR>><BR>> Hi Michael,<BR>><BR>> Thanks a bunch for the information. I was looking at the Supermicro<BR>> boxes (used quite a few of them) and was a bit curious about the Dell<BR>> offering. I'm glad to know I shouldn't use them and I believe I will<BR>> just go with the Supermicro setup as I was planning.<BR>><BR>> One big issue I wanted to know was the backplane for the HDDs. Having<BR>> it split like that it is really stupid. With the supermicro chassis<BR>> they can be ordered with dual expander chips so cards like the Megaraid<BR>> can load balance the IO to both ports on the HDDs.<BR>><BR>> Pretty sad that Dell would act like that but it is their loss.<BR>><BR>> Thanks again,<BR>> Morey<BR>><BR>> -----Original Message-----<BR>> From: Kushnir, Michael (NIH/NLM/LHC) [C]<BR>> [<A href="mailto:michael.kushnir@nih.gov">mailto:michael.kushnir@nih.gov</A>]<BR>> Sent: Friday, December 09, 2011 11:53 AM<BR>> To: Roof, Morey R.; <A href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</A><BR>> Subject: RE: Dell Server Question<BR>><BR>> Hi Morey,<BR>><BR>> I am using two C2100 servers with 2.5" inch chassis with DRBD. To save<BR>> you the long read, my quick and short recommendation is: AVOID THEM LIKE<BR>> THE PLAGUE!<BR>><BR>> For the money, get<BR>> <A href="http://www.supermicro.com/products/chassis/4U/417/SC417E16-R1400U.cfm">http://www.supermicro.com/products/chassis/4U/417/SC417E16-R1400U.cfm</A><BR>> (72-bay - 4U) or the<BR>> <A href="http://www.supermicro.com/products/chassis/2U/216/SC216E16-R1200U.cfm">http://www.supermicro.com/products/chassis/2U/216/SC216E16-R1200U.cfm</A><BR>> (24 bay - 2U) instead.<BR>><BR>> If you want more detailed info, read on! :)<BR>><BR>> My servers came with the LSI 9260-8i RAID cards, single E5620 CPU, 12GB<BR>> RAM, and the Intel based "mezzanine" dual-port 10GbE card. The backplane<BR>> is split with 12 slots connected to one port on the RAID card, and 12<BR>> slots on the other.<BR>><BR>> CPU and RAM:<BR>> I've found the E5620 is more than capable of handling the load from DRBD<BR>> and IET (software iscsi target). 12GB RAM is also ample for my needs.<BR>><BR>> Network:<BR>> I've found the Intel 10GbE card to be a good performer. I am using it<BR>> with Finisar SFP+ modules and have it connected to a Cisco 4900 switch<BR>> with Cisco optics on the switch end. I have the two ports in an LACP<BR>> (mode 4) bond. I am getting ping latencies of about 0.23-0.3ms. I'm<BR>> happy with that performance.<BR>><BR>> RAID:<BR>> I have my OS (RHEL 5) and swap on a pair of 500GB SATA2 disks in RAID1,<BR>> and my DRBD on a RAID10 made up of 10 Constellation.2 1TB disks.<BR>> Metadata is set to internal. The max replication rate (for the initial<BR>> sync) was ~3.2Gbits/s (measured by iftop).<BR>><BR>><BR>> Issues:<BR>><BR>> 1.<BR>><BR>> This server is not actually made by Dell. It is completely outsourced to<BR>> another manufacturer. I was not able to get any useful support from Dell<BR>> with any issues that I had. Their support people simply don't know crap<BR>> about this server, how to use it, or how to manage it. Considering that<BR>> I'm with the Federal government, and go through a dedicated support<BR>> team, that should say something. Dell was not even able to tell me what<BR>> software to use to manage the RAID array and failure notifications.<BR>><BR>> Dell kept telling me that the LSI command line utility is the only way<BR>> to go, and that the server, "is intended for hyper scale environments<BR>> where the expectation is that a failed server would just fail and be<BR>> replaced by another, and where administrators should know how to manage<BR>> such an environment."<BR>><BR>> Thankfully, LSI support people pointed me to their GUI which unlocks<BR>> great features like SNMP and SMTP notifications, SSD caching, and SSD<BR>> protector for SSD RAID.<BR>><BR>> 2.<BR>><BR>> Because the hardware is not actually "made" by Dell, this server is not<BR>> compatible with Open Manage and you can't update your drivers firmware<BR>> from Dell's yum repositories firmware.<BR>><BR>> 3.<BR>><BR>> Drive trays--- wow was this a surprise for me!!! Dell has a new policy<BR>> where they do not sell drive trays without hard drives in them. As you<BR>> might know, Dell's drives are highly overpriced. They will also not sell<BR>> you a server fully populated with trays unless you get it fully<BR>> populated with drives. We bought over $100k of Dell equipment this year,<BR>> and they would not make an exception and give us the drive trays.<BR>><BR>> So, finally, we relented, and we bought 16 (8 for each server) Crucial<BR>> M4 512GB SSDs from Dell for about $13k. Once we bought the drives, we<BR>> asked for trays. Dell told us to go... The policy is that trays are only<BR>> supplied with Dell BRAND drives, not any drives purchased from Dell.<BR>><BR>> We relented again, and asked Dell to sell us their cheapest hard drives<BR>> just to get trays. Dell told us that they cannot, because they are out<BR>> of hard drives due to flooding in Thailand. And they cannot just charge<BR>> us for drives and ship only the trays. GRR!!! Bastards!!!<BR>><BR>> I've tried finding the trays online. It took me months to find some at<BR>> codemicro.com. Then they suddenly sold out of all 400+ units. So, the<BR>> last batch, I had to buy on eBay. I thank God that Ivan from New Jersey<BR>> was able to find some to put them on eBay.<BR>><BR>><BR>> So, my recommendation is: STAY AWAY! Get the SuperMicro chassis (it<BR>> comes with all 72 drive trays and plenty of expansion capacity. Get ASUS<BR>> server motherboards (I use them in my private lab), Myri 10GbE NICs (I<BR>> use them in other production servers), and LSI 9260-8i (or better) RAID<BR>> cards. You will be in a good place in terms of hardware and money spent.<BR>><BR>><BR>><BR>> Best,<BR>> Michael Kushnir<BR>><BR>> Lead VMware and UNIX Systems Administrator<BR>> Communications Engineering Branch,<BR>> Lister Hill National Center for Biomedical Communications,<BR>> National Library of Medicine,<BR>> National Institutes of Health<BR>><BR>> -----Original Message-----<BR>> From: Roof, Morey R. [<A href="mailto:MRoof@admin.nmt.edu">mailto:MRoof@admin.nmt.edu</A>]<BR>> Sent: Thursday, December 08, 2011 1:30 PM<BR>> To: <A href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</A><BR>> Subject: [DRBD-user] Dell Server Question<BR>><BR>> Hi Everyone,<BR>><BR>> Anyone out there using a Dell Poweredge C2100 with the 24 2.5" HDD<BR>> backplane? I'm curious to know a little about the backplane, mainly if<BR>> it is a dual port backplane where the RAID controller can use both ports<BR>> on the SAS drives to load balance the IO operations. Dell's website<BR>> isn't very specific on this point. Also, some information about how the<BR>> server seems work with DRBD would be very helpful to me.<BR>><BR>> Thanks,<BR>> Morey<BR>> _______________________________________________<BR>> drbd-user mailing list<BR>> <A href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</A><BR>> <A href="http://lists.linbit.com/mailman/listinfo/drbd-user">http://lists.linbit.com/mailman/listinfo/drbd-user</A><BR>> _______________________________________________<BR>> drbd-user mailing list<BR>> <A href="mailto:drbd-user@lists.linbit.com">drbd-user@lists.linbit.com</A><BR>> <A href="http://lists.linbit.com/mailman/listinfo/drbd-user">http://lists.linbit.com/mailman/listinfo/drbd-user</A></SPAN></P></DIV></DIV></DIV></DIV></BODY></HTML>