//
you're reading...
Uncategorized

Dragging up ancient history – How NetApp fooled everyone (FAS3040 v. CX3-40)

Foreword:

Back in January 2009, NetApp published a Storage Performance Council SPC-1 comparison between the NetApp FAS3040 and the EMC CX3-40, claiming that NetApp had the superior array with the FAS charming out 30,985.60 IOPS and CX3 trailing with a woeful 24,997.49 IOPS.

NetApp shopped this score to anyone and everyone who would listen, the media, customers, heck even EMC engineers copped an earful of it whenever NetApp had the chance to deliver it.

But, whilst many fell for NetApps shenanigans, there were a few who questioned it, some even fought back – realising NetApp Long stroked the EMC array, yet short stroked the FAS; but most missed the fact that hidden in plain sight was NetApps deceit – they doctored the test.

Now many suspected that, but NetApp would often come prepared (at least they did with me), and would bring printed copies of the EXECUTIVE SUMMARY version of the document to any meetings where they were presenting these “facts”.

Now, we someone presents this kind of data to you in a meeting, you’re normally respectable enough to give them the meeting without you jumping for the laptop to check if this is true, spending the entirety of the meeting with you face buried in the screen looking for any little piece of evidence which would be a saving grace.

They knew that –they knew most (if not all) would not check the facts then and there, and probably never. And even if they did check, most people wouldn’t know the differences.

But a few went out and decided that, given our experiences of EMC Clariion’s that these numbers could not simply be right; we had to look at the report for ourselves. – And sure enough, people appeared from the woodwork crying foul, they’d spotted that NetApp long stroked the EMC from a mile away.

However, so far as I know, nobody really knew the levels of deceit that NetApp did go to.

The following has been a long time in the making and lucking or un-lucky as it may be, I’ve had a recent spout of being jet-lagged and getting up a 4am when my first meeting is at 10am, stuck in hotels far away from family and bored, with just that little chip on my shoulder about what NetApp did.

I had long since ignored it, but recently, one of my regular reads: http://storagewithoutborders.com/ , a known NetApp devotee decided to post a couple of NetApp responses to EMC’s announcements.

Don’t get me wrong, there’s nothing wrong with being an evangelist for your company, but at the very least, respect your competition.

Now I don’t know about you, but I really hate it when someone pisses on somebody else’s parade!

I hate it even more when they use the competitor’s announcement to coat-tail their own announcement, I think it’s crass and offensive.

I think it’s even worse when it’s been commented on by someone who has NO experience of the product their comparing – only being another minaret for their company.

I posted some comments on his blog about this behaviour, but he seemed to think it was ok, acceptable even, I didn’t.

When he responded, his comments were varied, but polite (John is a really nice guy), but he really does drink the NetApp Kool-Aid.

It got my goat, John had commented in his blog about (even 4 years latter) how much better NetApp’s now e-series and FAS was than the EMC CX3-40 –he just had to bring that old chestnut up.

http://storagewithoutborders.com/2011/11/03/breaking-records-revisited/

See below:

SWB> “1. You need to be careful about the way benchmarks are used and interpreted
2. You should present the top line number honestly without resorting to tricks like unrealistic configurations or aggregating performance numbers without a valid point of aggregation.”

Thank you John, I’ll take that into consideration.

So I just wanted to set the scene for what I’m about to point out:

SWB > “The benchmark at the time was with an equivalent array (the 3040) which is now also three generations old, the benchmark and what it was proving at the time remains valid in my opinion. I’d be interested to see a similar side by side submission for a VNX5300 with FAST Cache and a FAS3240 with Flashcache, but this time maybe we should wait until EMC submits their own configuration first.

SWB > “I’m pretty happy with the way NetApp does their bench-marking, and it seems to me that many others abuse the process, which annoys me, so I write about it.”

I understand John, we all want to believe the things closest to us are infallible – we don’t want to believe our kid is the one who bit another kid; our mother is the better cook and the company and/or products we associate ourselves with are the superior products and our companies practices unblemished – but this can cloud our judgment and cause us to look past the flaw’s.

Now, I want to make something perfectly clear, I’m not Pro/Anti NetApp or EMC in any regard; I have work with both for many years.

I will be taking any other vendors to task as and when I see fit, for now however, your blog is one of my regular reads and I know your distain for FUD – You just happened to be the first, and your post contradicted your supposed dislike for FUD and engineering a benchmark.

I want to go back to the FAS 3040/Clariion CX3-40 example as a demonstration of how NetApp DOES NOT (highlight not shouty) play by the rules.

Executive Summary:

Way back in March 2009, NetApp published a comparison of the two products in an attempt to show NetApps superiority as a performance array, the NetApp FAS3040 achieved SPC-1 result of 30,985 IOPS, whilst the EMC CX3-40 a seemingly meager 24,997 SPC-1 IOPS – the EMC left wanting with a horrific 5,988 IOPS behind the NetApp.

On the face of it; it would appear NetApp had a just and fair lead, but this is simply not true – NetApp Engineered the EMC to be pig-slow, and whilst I wasn’t there at the time and can only speculate the intentions drawn in this post – I cannot, in any sense of the word, believe it was not intentional.

When committing a benchmark, it’s important to ensure a Like-for-like configuration – NetApp simply did not do this!

NetApp used:

· Different Hardware for the Workload Generator,

· Different methods for the ASU presentation,

· Short Stroked the NetApp and Long Stroked the EMC,

· Engineered in higher latency equipment and additional hardware and services into the EMC BoM and;

· Falsified the displayed configuration.

My goal here is to show why I place no faith in NetApp’s or any other vendor’s competitive benchmark.

End Executive Summary.

Now for the Nuts and Bolts:

Now John, I know you have a passion for Benchmarks, and to reiterate the first quote in this reply, I will add that you need to be careful to ensure you are not causing undue and unfair differences in the equipment, tools, software and pricing to give an undue competitive advantage.

I know you’ll probably be crying foul by now and stating it was fair and just, but I can prove that it was not – without a doubt – and for the life of me, I cannot believe how any of this was missed and that NetApp got away with it.

I must warn you, this level of detail is normally reserved for the kind of person who wears anoraks and strokes their beard.

I’ll give a breakdown of how NetApp did this by breaking it into sections and their differences:

1. LUN to volume presentations

2. Workload Generator (WG) Hosts

3. HBAs

4. Array Configurations and BoM

5. RAID Group and LUN Configuration

6. Workload Differences

7. Other Differences and issues

So let’s look at these differences in detail (John, when benchmarking, the devil IS in the detail):

1. LUN to volume presentations:

When NetApps Steve Daniels configured the WG’s (Workload Generators) volumes, he stripped 36 LUNs from the Clariion at the host level, and the NetApp:

NetApp FAS3040:Page 63: http://www.storageperformance.org/results/a00057_NetApp_FAS3040_full-disclosure-r1.pdf
The Volumes as striped LUNs by the WG:
The LUNs presented to the WG:
EMC CX3-40:Page 64: http://www.storageperformance.org/results/a00059_NetApp_EMC-CX3-M40_full-disclosure-r1.pdf
The Volumes as striped LUNs by the WG:
The LUNs presented to the WG:~
Note: because the procedure was not as well documented in the NetApp configuration, it makes reading somewhat difficult.

Despite the fact that he must have known full well that the EMC Clariion had the capability to stripe the volumes in-array or to just simply create 1 larger LUN, it was presented in an atrocious layout.

This wouldn’t seem like a big deal, but it’s a huge difference and creates many host and array performance issues – it’s certainly known to anyone with a strong knowledge of storage networking not to do this unless you have no other choice (which he did):

The phenomenon of striping performance loss at the host is well observed here:

http://sqlblog.com/blogs/linchi_shea/archive/2007/03/12/should-i-use-a-windows-striped-volume.aspx

It would seem that NetApp created a greater depth of striping for the EMC array and utilised for no possible technical reason (other than to make the workload as high as possible) small stripes for the EMC broken over the SP’s, thereby negating any possible use of the cache.

Now, I want to make it clear, there is NO technical reason to create so many LUNs from within each RAID Group, only to create performance problems and in my years of working with EMC Clariion and the many Clariion’s I have worked with, I have never seen a Clariion laid out in such a manor.

2. Workload Generator (WG) hosts

At first glance, the WG hosts seem identical IBM X3650 servers; however, NetApp chose to give themselves a competitive advantage by using a Workload Generator which is considerably better spec’d, mostly around the bus.

For ease of viewing, I’ve circled the offending areas in red:

NetApp FAS3040:
The IBM X3650 use for the NetApp WG is:PCIe based.PAGE 16: http://www.storageperformance.org/results/a00057_NetApp_FAS3040_full-disclosure-r1.pdf
PCIe· is full-duplex – It CAN transmit and receive at the same time· is a Serial Interface and is point to point· has a line speed of 2GB/s per 4 lanes (32 PCIe lanes)· is direct to the north-bridge and CPU· two PCIe HBA’s have 2GB/s each for a total of 4GB/s (8 PCIe lanes, 4 lanes each of 32)
EMC:
The IBM X3650 used for the EMC WG is:PCI-X 133MHz basedPAGE 15: http://www.storageperformance.org/results/a00059_NetApp_EMC-CX3-M40_full-disclosure-r1.pdf
PCI-X· is half-duplex – It CANNOT transmit and receive at the same time· is a Parallel bus and relies on arbitration, scheduling and shares the bus bandwidth· has a Line Speed of 1GB/s· is channeled through multiple chipsets and bridges to before interaction at the north-bridge and CPU· two PCI-X HBA’s still have only 1GB/s bus to share

As you can see, the two IBM x3650 servers are different, even though the EMC WG server had more memory and a faster CPU (I can’t answer for the CPU architectures as it’s not listed).

The WG host bus given to the EMC WG was:

· Slower

· Parallel

· Bandwidth Limited

· Higher Latency

· And half-duplex

Anyone with knowledge of networking will understand the implications or full and half duplex and serial vs. parallel is much the same jump in performance as SATA/PATA.

NetApp speak regularly on the benefits of PCIe over PCI-X:

http://partners.netapp.com/go/techontap/fas6070.html

And I quote:

“We have also changed the system interface on NVRAM to PCIe (PCI Express). This eliminates potential bottlenecks that the older PCI-X based slots might introduce.”Howard: PCIe was designed to overcome bandwidth limitation issues with earlier PCI and PCI-X expansion slots.”Naresh: 100 MHz PCI-X slots are 0.8GB/s peak, and x8 PCIe slots are 4GB/s. PCI-X slots at 100 MHz could be shared between two slots, so a couple of fast HBAs could become limited by the PCI-X bandwidth.”Tom: In addition to increased bandwidth, PCIe provides improved RAS features. For example, instead of a shared bus, each link is point to point.”

It seems NetApp used the inferior Workload Generator (WG) for the EMC and the superior WG for NetApp.

Why did they not use the same host? I can only imagine to increase the Total Service Time when measuring the EMC, possibly doubling the response time!

3. HBAs

Again, at first glance, it would seem NetApp use the same Qlogic HBA’s for both tests – but as highlighted before, the two hosts used were different, one PCIe, the other PCI-X.

The same is applied to the HBA’s, NetApp used the faster and unrestricted HBA’s for their configuration and used the slower and restricted HBA’s for the EMC configuration:

NetApp FAS3040:
The HBA given to the NetApp is the QLE2462, which is:· PCIe· Superiority highlighted beforePAGE 16: http://www.storageperformance.org/results/a00057_NetApp_FAS3040_full-disclosure-r1.pdf
PCIe· is full-duplex – It CAN transmit and receive at the same time· is a Serial Interface and is point to point· has a line speed of 2GB/s per 4 lanes (32 PCIe lanes)· is direct to the north-bridge and CPU· two PCIe HBA’s have 2GB/s each for a total of 4GB/s (8 PCIe lanes, 4 lanes each of 32)
EMC CX3-40:
The HBA given to the EMC is the QLA2462, which is:· PCI-X· The HBA is 266 MHz but limited to 133Mhz because of the hostPAGE 15: http://www.storageperformance.org/results/a00059_NetApp_EMC-CX3-M40_full-disclosure-r1.pdf
PCI-X· is half-duplex – It CANNOT transmit and receive at the same time· is a Parallel bus and relies on arbitration, scheduling and shares the bus bandwidth· has a Line Speed of 1GB/s· is channeled through multiple chipsets and bridges to before interaction at the north-bridge and CPU· two PCI-X HBA’s still have only 1GB/s bus to share

It’s important to note that 1GB/s is PCI-X total bus peek speed which is easily drowned with a 2 port 4Gb/s HBA, let alone 2 of them (as per the BoM and config) – Totalling 2GB/s for both cards, yet only 1GB/s being available.

Whereas PCIe has a maximum throughput of 8GB/s, meaning 2 x PCIe x4 HBA’s would only be using 2GB/s only a quarter of the available host bus bandwidth.

To give half/full duplex and Serial/Parallel some context, imagine two office buildings of the same number of floors (10):· Building Ahas 1 elevator (half-duplex / Parallel – PCIx)o If a person on the ground floor wants to go to level 10, he has to wait until the lift arrives at ground before he can travelo If another person on the ground floor wants to travel to level 5, he has to wait until the lift has completed its travel to the 10th floor and return· Building Bhas 32 elevators (full-duplex / Serial – PCIe)o If a person on the ground floor wants to go to level 10, the elevator is already at the ground floor ready to go.

o If another person on the ground floor wants to travel to level 5, the elevator is already at the ground floor ready to go or has another 30 ready or will be there shortly.

Clearly again, NetApp has engineered a superior WG host for themselves and inferior for EMC.

4. Array Configurations and BoM

Here are the two BoM’s from NetApp and EMC arrays:

NetApp FAS3040:NetApp Page 14: http://www.storageperformance.org/results/a00057_NetApp_FAS3040_full-disclosure-r1.pdf :
EMC CX3-40:EMC – Page 13: http://www.storageperformance.org/results/a00059_NetApp_EMC-CX3-M40_full-disclosure-r1.pdf :

Now here are the interesting titbits, let’s take a look at what I’ve highlighted and why:

Firstly, costs:

– EMC: NetApp included the HBA’s and Switches and multi-pathing as part of the EMC array costs:

1 PP-WN-WG – PPATH WINDOWS WGR EA $1,440 0% $1,440 see attached third party quotation

2 QLA2462-E-SP – 2 PORT 4GB PCI-X EA $1,700 0% $3,400 see attached third party quotation

2 Brocade 16-Port 200e FC Full Fab Switch,-C,R5 EA $8,700 0% $17,400 Network Appliance, Inc.

2 BSWITCH-16PORT-R5 HW Support,Premium,4hr,y mths:36 EA $1,697 0% $3,393 Network Appliance, Inc.

2 BSWITCH-16PORT-R5 SW Subs,Premium,4hr,y mths:36 EA $0 0% $0 Network Appliance, Inc.”

– NetApp: NetApp added the HBA’s and Switches and multi-pathing as the add-on costs

Host Attach Hardware and Software

SW-DSM-MPIO-WINDOWS 1 $0.00 0 $0.00 $0.00

X6518A-R6 Cable,Optical,LC/LC,5M,R6 4 $150.00 0 $150.00 $600.00

X1089A-R6 HBA,QLogic QLE2462,2-Port,4Gb,PCI-e,R6 2 $2,615.00 0 $2,615.00 $5,230.00

SW-DSM-MPIO-WIN Software,Data ONTAP DSM for Windows MPIO 1 $1,000.00 0 $1,000.00 $1,000.00”

Not a big deal there, as the TSC is incorporating the entirety.

– EMC: NetApp included Professional in the EMC costs

1 PS-BAS-PP1 – POWERPATH 1HOST QS EA $1,330 0% $1,330 see attached third party quotation

1 PS-BAS-PMBLK – POWERPATH 1HOST QS EA $1,970 0% $1,970 see attached third party quotation

My side note: (Who the hell needs PS to install PowerPath? And For that matter, who needs a Project management block for 1 host?) (I hope they got their money’s worth!)

– NetApp: There were no included Professional services costs

No wonder the EMC came out more expensive, they put in services which no-body needs, bundled the HBA’s, switching and multi-pathing into the array costs, but didn’t do the same for the NetApp!!!! – Sneaky!

Secondly, Cabling:

– EMC: NetApp included 4 x 8 meter HSSDC2 cables for connection from the array to the first Disk Shelf of each bus with 1m cables from then on:

4 FC2-HSSDC-8M – 8M HSSDC2 to HSSDC2 bus cbl EA $600 0% $2,400 see attached third party quotation

(Added costs in using 8m cables? Yup.)

– NetApp: NetApp included 16 x 0.5 meter HSSDC2 cables for connection from the array to the first disk shelf of each bus and 0.5 meter from then on:

X6530-R6-C Cable,Patch,FC SFP to SFP,0.5M,-C,R6 16 $0.00 0 $0.00 $0.00

Now, this might not seem like a big deal, but 8m cables are the reserve of only very difficult scenarios such as having to stretch many racks to join shelves to the array, it is never used in latency sensitive scenarios and here’s why:

Fibre and copper have similar latencies of 5ns per meter.

For an 8m cable, that translates to 80ns round-trip (the EMC config),

Whereas;

For a 0.5m cable, its 5ns round-trip (.25ns per 0.5 meter) (the NetApp config)

Extend that to a mirrored system with 2 busses that’s 160ns round-trip then add every meter and enclosure after that (up to 0.005ms port-to-port).

Now I want to state again, EMC never use 8m cables except in extreme circumstances and never when low latency is needed!

It’s clear NetApp Engineered the EMC to have a slow a bus as possible when compared to the NetApp!

Thirdly, Bus Layout:

I’m make no representation for the correctness of the NetApp Bus layout, although a little over the time and the fact that NetApp configured it with far more point to point connections for the disk-shelves that was needed (which they would have to boost their performance), it is a configuration which I have seen more than once.

EMC: In the diagram Page 14: http://www.storageperformance.org/results/a00059_NetApp_EMC-CX3-M40_full-disclosure-r1.pdf:

You can clearly see that NetApp show that they use only 11 DAE’s for the Configuration:

Left (bus 0):

1 x Vault Pack with 5 disks

5 x DAE with 75 disks

Right (bus 1):

5 x DAE with 74 disks

But when I look at the RAID Group configuration I see that they use all 12 DAE’s from the BoM, different from the stated configuration:

1 V-CX4014615K – VAULT PACK CX3-40 146GB 15K 4GB DRIVES QTY 5 EA $8,225 0% $8,225 see attached third party quotation

+

11 CX-4PDAE-FD – 4G DAE FIELD INSTALL EA $5,900 0% $64,900 see attached third party quotation

That makes 12 DAE’s – Who cares? You’ll see!

If we look at the configuration scripts used on Page 60: http://www.storageperformance.org/results/a00059_NetApp_EMC-CX3-M40_full-disclosure-r1.pdf

Create Raid Groups:

We see that the first raid group (RG0) Mirror Primary starts at 0_1_0 and the Mirror Secondary starts at 1_3_0

(x_x_x is Bus_Enclosure_Device/Disk):

naviseccli -User admin -Password password -Scope 0 -h 10.61.162.55 createrg 0 0_1_0 1_3_0 0_1_1 1_3_1 0_1_2 1_3_2 0_1_3 1_3_3 0_1_4 1_3_4 0_1_5 1_3_5

And as we go further down, we see the last raid group (RG11) Mirror Primary starts at 0_4_12 and the Mirror Secondary starts at 1_6_12

Then extends into Bus 1 Enclosure 7

(x_x_x is bus_Enclosure_Device/Disk):

naviseccli -User admin -Password password -Scope 0 -h 10.61.162.55 createrg 11 0_4_12 1_6_12 0_4_13 1_6_13 0_5_12 1_7_12 0_5_13 1_7_13 0_1_14 1_3_14 0_2_14 1_4_14

Now I first read that as a typo, why would 0_1_0 be mirrored to 1_3_0 and not 1_0_0 or 1_1_0?

This is what the layout of the Clariion that NetApp setup looked like:

The black boarders represent where the configuration should be and the coloured cells represent where the RAIDGroups are configured with colours matching the raid pair according to the full disclosure.

Hang on a minute, what happened to the 3 x shelves beforehand on bus 1?

(Bus number starts at 0 and continues to 7)

Well, with 12 DAE’s (15 slots per DAE) there are a total of 180 drive slots.

155 disks in total were purchased (150 + 5 in Vault pack)

5 of which are taken by Flare/Vault

There are 12 x 12 disks RAID 1/0 RAIDGroups, so 144 disks used to present as capacity

No mention of how many hot spares were used (only a total of OE and spare capacity), best practice is generally 1:30.

This is what it should look like (forgetting for a minute the fact that NetApp laid it poorly), if enclosures were not jumped:

Because by placing the mirror pair further down the chain, you increase the latency to get to the pair disk, increasing the service time drastically.

There is no reason to do so other than to engineer slowness! No one in their right mind would do so!

NetApp engineered the EMC to have a slow Backend!

5. RAID Group and LUN Configuration

When it came to the RAIDGroup and LUN Layout, this is where it got even worse:

NetApp:NetApp Short-Stroked and created one very large aggregate:Page 62/63: http://www.storageperformance.org/results/a00057_NetApp_FAS3040_full-disclosure-r1.pdf
The following is a diagram showing how the Aggregate for each controller of the NetApp FAS 3040 was configured, to achieve the total capacity presented and unallocated/unused multiply this config x2:Please note: Some of the numbers are estimated, NetApp uses a mix of Base2 and Base10 when presenting and not all numbers were disclosed.I did however; to the best of my ability calculated it as accurately as possible within <5%
Representing the 2 Controllers:
Controller 1 Controller 2
create aggregate with the following configuration:- aggr0 settings, 4 rgs, rg sizes (1×18 + 3×17), 1 spare- aggr0 options:- nosnap=on- set snap reserve = 0 on aggregate aggr0- set snap sched to 0 0 0 on the aggregate aggr0

· spc1 data flexible volume (vol1):

§ create vol1 of size 8493820 MB

· set volume options on vol1:

o nosnap=on

o nosnapdir=off

· set snap reserve = 0 on vol1

· set snap sched to 0 0 0 on vol1

· set space reservation (guarantee) to “none”

Create zeroed luns with no space reservation on each NetApp controller with the following sizes and then map them to the windows igroup created earlier assigning each lun a unique lun id.

o 6 lun files for ASU1 of size 450100 MB each

o 6 lun files for ASU2 of size 450100 MB each

o 6 lun files for ASu3 of size 100022 MB each

Essentially it shows that of the ~19TB available from the raidgroups (after DP capacity loss), the vol1 had ~2,4TB unused/unallocated and that the aggregate aggr0 (or the collective of disks) had a total of ~3.9TB unallocated/unused. Or almost ~35% of the disk capacity per controller (after DP losses) as whitespace.
EMC:NetApp deliberately Long-Stroked the EMC Disks: Page 60: http://www.storageperformance.org/results/a00059_NetApp_EMC-CX3-M40_full-disclosure-r1.pdf
EMC: NetApp it seems, also limited the performance of each EMC RAIDGroup by using only 12 disks per RAID group offering, basically 6 disks mirrored of performance. Eg:naviseccli -User admin -Password password -Scope 0 -h 10.61.162.55 createrg 11 0_4_12 1_6_12 0_4_13 1_6_13 0_5_12 1_7_12 0_5_13 1_7_13 0_1_14 1_3_14 0_2_14 1_4_14
They then Long Stroked each RAIDGroup by having a slice of each broken up into 3 LUNS per RG for a total of 36 LUNS EG:.naviseccli -User admin -Password password -Scope 0 -h 10.61.162.55 bind r1_0 0 -rg 0 -cap 296 -sp anaviseccli -User admin -Password password -Scope 0 -h 10.61.162.55 bind r1_0 1 -rg 0 -cap 296 -sp anaviseccli -User admin -Password password -Scope 0 -h 10.61.162.55 bind r1_0 2 -rg 0 -cap 65 -sp a

Now Typically, we (the storage industry) quote a 15k spindle as being ~180 IOPs per disk avg, but we know that it’s more like ~250 IOPS at the outside and ~100 IOPS at the inside of a disk so the average is about 180 end to end.

NetApp engineered the EMC to utilize all but 23GB of disk when presented to the host>Volume>ASU, essentially using almost all of the capacity of the RAID1/0 RAIDGroups; The only reason is to make sure the EMC utilized almost the full length of the disks or LONG STROKE.

There is no conceivable reason for the EMC to have so many small RaidGroups with little LUNS in them, why not just have a LUN presented form each RAID 1/0 RAIDGroup, heck even stripe inside the array.

NetApp created an aggregate many times larger yet only provisioned ~65% of the capacity of the Aggregates and RAID Groups under them, meaning that NetApp SHORT STROKED their array!

NetApp Per Disk Distribution: EMC Per Disk Distribution:
~65.0% Disk Utilization (Short/Mid Stroke)Free Capacity: 46.53 GB ~82.3% Disk Utilization (High-Long Stroke)Free Capacity: 23.24 GB
Note: To even out the diagram and aid simplicity, I have standardized the two at 133GB which is a balanced breakdown of the two configurations, RAIDGroup striping and other layout methods will layout data slightly differently, but the result is the same.(NetApp place usable with no reserve at 133.2 and EMC at 133.1GB for a 146/144GB drive)Please also note: QD1 IOPS can vary from various disk manufacturers and densities / platter / spindle diameter etc.

When examining the SPC-1 specifications it reveals the following:

http://www.storageperformance.org/specs/SPC-1_v1.11.pdf

2.6.8 SPC-1 defines three ASUs:

– The Data Store (ASU-1) holds raw incoming data for the application system. As the application system processes the data it may temporarily remain in the data store, be transferred to the user store, or be deleted. The workload profile for the Data Store is defined in Clause 3.5.1. ASU-1 will hold 45.0% (+-0.5%) of the total ASU Capacity.

– The User Store (ASU-2) holds information processed by the application system and is stored in a self-consistent, secure, and organized state. The information is principally obtained from the data store, but may also consist of information created by the application or its users in the course of processing. Its workload profile for the User Store is defined in Clause 3.5.2. ASU-2 will hold 45.0% (+-0.5%) of the total ASU Capacity.

– The Log (ASU-3) contains files written by the application system for the purpose of protecting the integrity of data and information the application system maintains in the Data and User stores. The workload profile for the Log is sequential and is defined in Clause 3.5.3. ASU-3 will hold 10.0% (+-0.5%) of the total ASU Capacity.

So, that’s:

o 45.0% for ASU1

o 45.0% for ASU2

o 10.0% for ASU3

By Spreading the benchmark over almost the entire length of the disk for the EMC CX3-40 yet just over half-way for the NetApp is of course give NetApp the advantage and disadvantage the EMC in both IOPs and Latency.

6. Workload differences:

Another interesting dynamic was that NetApp ran different workloads for the FAS3040 vs. the CX3-40.

We already know that by now the test is completely invalid, but what would have been the result if NetApp had have made the test the same?

NetApp:

Page 70 – http://www.storageperformance.org/results/a00057_NetApp_FAS3040_full-disclosure-r1.pdf :

APPENDIX D: SPC-1 WORKLOAD GENERATOR STORAGE COMMANDS

AND PARAMETERS

The content of SPC-1 Workload Generator command and parameter file, used in this

benchmark, is listed below.

javaparms=”-Xmx1024m -Xms512m -Xss256k”

sd=asu1_1,lun=\.E:

sd=asu2_1,lun=\.F:

sd=asu3_1,lun=\.G:

EMC:

APPENDIX D: SPC-1 WORKLOAD GENERATOR STORAGE COMMANDS

AND PARAMETERS

The content of SPC-1 Workload Generator command and parameter file, used in this

benchmark, is listed below.

javaparms=”-Xmx512m”

sd=asu1_1,lun=\.F:

sd=asu2_1,lun=\.H:

sd=asu3_1,lun=\.J:

Page 29 –SPC1 Spec.:

The storage for the SPC-1 workload consists of three Application Storage Units:

􀁸 ASU 1 – Data Store

SPC Benchmark-1􀂥 (SPC-1) Version 1.11 Page 30 of 119

Official Specification Effective – 19 July 2009

􀁸 ASU 2 – User Store

􀁸 ASU 3 – Log/Sequential Write

7. Other differences and issues:

The following are a selection of other issues and differences I found in the full disclosure documents and other locations that I found most interesting, but did not feel they needed to be addressed in as much detail as the rest. Maybe I will spend more time on them later.

NetApp:

On the NetApp system, we found we could improve performance by changing the memory management policy to reflect the fact that most SPC-1 data is not referenced repeatedly. This policy change can be implemented with the following priority settings with Data ONTAP® 7.3:

priority on

priority set enabled_components=cache

priority set volume <volume-name> cache=reuse

The net effect of these commands is to tell the memory system to reuse memory for newer items more aggressively than it would normally. (The enabled_components subcommand is new in Data ONTAP 7.3. If you are using Data ONTAP 7.2 you can skip that command.)

A couple of the things we tuned are still being refined, so they are enabled by the setflag command. In future versions of Data ONTAP either these flags will become options or they will disappear as the system becomes self-tuning for these features.

priv set diag
setflag wafl_downgrade_target 0
setflag wafl_optimize_write_once 0

The “downgrade_target” command changes the priority of a process within Data ONTAP that handles incoming SCSI requests. This process is used by both FC SAN and iSCSI. If your system is not also running NAS workloads, then this priority shift improves response time.

We’re explicitly calling out these settings because, based on our testing, we think they will yield performance benefits for online business application workloads. If you are interested, you can read more about them in a recent NetApp technical report.

System Flags:

o wafl_optimize_write_once: change default value of 1 to 0. This flag affects the initial layout of data within a newly created aggregate. The default data layout favors applications which do not overwrite their data.

o wafl_downgrade_target: change default value of 1 to 0. This flag changes the runtime priority of the process that handles the SCSI protocol for incoming Fibre-Channel requests. For storage systems that are not also servicing NAS requests this change to the process priority is recommended.

Let me read those again:

  •  The default data layout favors applications which do not overwrite their data.
  •  So, that means, that this flag will not optimise the array layout for normal workloads with limited changes like file shares, exchange and sql (without archive of course) but closer to that of HPC data sets. – this seems to be compounded by next flag:
  •  For storage systems that are not also servicing NAS requests this change to the process priority is recommended
  •  Now a bit of rewording and it says – if you are going to use this as a normal SAN only array (no file), then setting this flag will ensure you get decent SAN performance as you won’t be juggling CPU and memory with those functions.

· EMC don’t have this issue, because they build a platform with each component (file, block) being optimised for each function rather than building a General Purpose array that needs to be tuned to specific tasks.

Interestingly, Stephen Daniel was also the author of the NetApp TR whitepaper “Configuring and Tuning NetApp Storage Systems for High-Performance Random-Access Workloads

http://media.netapp.com/documents/tr-3647.pdf

Where he wrote:

“4. Final Remarks This paper provides a number of tips and techniques for configuring NetApp systems for high performance. Most of these techniques are straightforward and well known. Using special flags to tune performance represents a benchmark-oriented compromise on our part. These flags can be used to deliver performance improvements to customers whose understanding of their workload ensures that they will use them appropriately during both the testing and deployment phases of NetApp FAS arrays. Future versions of Data ONTAP will be more self-tuning, so the flags will no longer be required.

Does NetApp consider it normal to be asked to set a parameter that elicits the following response?

“Warning: These diagnostic commands are for use by NetWork Appliance personnel only”.

Clearly not.

Is this not in direct contravention of Storage Performance Council, SPC-1 specification terms and conditions?

Page 13 – http://www.storageperformance.org/specs/SPC-1_v1.11.pdf :

0.2 General Guidelines

The purpose of SPC benchmarks is to provide objective, relevant, and verifiable data to purchasers of I/O subsystems. To that end, SPC specifications require that benchmark tests be implemented with system platforms and products that:

1. Are generally available to users.

2. A significant percentage of the users in the target market segment (server class systems) would implement.

3. Are relevant to the market segment that SPC-1 benchmark represents.

In addition, all SPC benchmark results are required to be sponsored by a distinctly identifiable entity, which is referred to as the Test Sponsor. The Test Sponsor is responsible for the submission of all required SPC benchmark results and materials. The Test Sponsor is responsible for the completeness, accuracy, and authenticity of those submitted results and materials as attested to in the required Letter of Good Faith (see Appendix D). A Test Sponsor is not required to be a SPC member and may be an individual, company, or organization.

The use of new systems, products, technologies (hardware or software) and pricing is encouraged so long as they meet the requirements above. Specifically prohibited are benchmark systems, products, pricing (hereafter referred to as “implementations”) whose primary purpose is performance optimization of SPC benchmark results without any corresponding applicability to real-world applications and environments. In other words, all “benchmark specials,” implementations that improve benchmark results but not general, realworld performance are prohibited.

EMC:

CX3-40 Storage System

The following changes must be made on the CX3-40 storage system:

o Disable write caching on all underlying LUNs used for ASU1 and ASU2. Do not change the default setting of read/write caching for the ASU3 LUNs.

o Set the read policy: low water mark is 30%, high water mark is 50%.

o Set the read caches to 1716 and the write cache to 1300 MB.

Why was the Low and High water marks set so low?

Why not present the cache testing? – I have no doubt if performed so poorly, but this is as a consequence rather than a result.

Closing notes:

Personally, I find what NetApp did here beyond reprehensible, discusting and absurd.

Since March 2009, I have no regarded whatsoever for NetApp’s claims of performance and indignation when questioned or challenged.

Whilst I very much like NetApps products, I have absolutely no faith in NetApp as a company.

I have no trust in NetApp’s claims of their own performance or function and will not accept it until I see it for myself; because NetApp have constantly tried to clamber their way upwards though deceit.

Additionally, due to NetApp, I have no belief that the Storage Performance Council has any merit.

How can the SPC have any credibility when they allow an array vendor to directly manipulate results? Why are there no standards for the testing hardware and software, why is there no scrutiny of sponsored competitive tests by the SPC and for that matter, why is there no scrutiny of sponsors own tests and hardware?

To give some analogies here to what NetApp have done, it’s like:

o Golf bat maker “A” claiming their Iron is better than Golf bat maker “B”’s, but in ‘proving’ so, use theirs to hit a new ball from the fairway, on a tee with a tailwind and using “B”’s with an old ball, from the rough in a headwind.

o Have motorcycle maker “A” testing a similar spec’d “B” bike, putting a 60kg/165cm rider on bike “A” and a 120kg/190cm rider on bike “B”.

You get the idea; not only did NetApp change the variables to suit themselves, but they also modified their own array to run well, whilst configuring the EMC array to perform poorly.

So, how would I have laid it out differently? – I’ll address that at another time.

Rest assured, it would have been very different – one of the many right ways.

Now NetApp may claim that this is a normal environment, yet every best practice guide advises against it, I have never – and nor have any of my colleagues over a collectively very long level of experience – seen a Clariion laid out and configured in such a manour.

If yours even closely resembles 10% of this atrocity, then it is time for a new reseller/integrator or to send your admin on a training course.

Other than that, NetApp should be completely and utterly ashamed of themselves!

That’s it for now, I hope you made it too the end and didn’t succumb to boredom related mortality.

Aus Storage Guy

“Keeping the bastards honest”

Configuring and Tuning NetApp Storage Systems for High-Performance Random-Access Workloads

About ausstorageguy

I am a IT Enterprise Storage Specialist based in Australia, with multi-vendor training and a knack for getting to the truth behind storage. I specialise in and have previously worked for EMC, however; I am also highly skilled and trained in: -NetApp FAS -NetApp E-Series (Formally LSI Enginio) -Dell Compellent and Equalogic -Hitachi USP, VSP and AMS -HP EVA, Lefthand, 3Par (and HDS OEMs) -LSI Engenio OEMs (IBM DS and Sun StorageTek) -TMS RamSAN -IBM XIV As you can imagine, that's alot of training... Thankfully; as a speciallist in storage, I don't have to think about much else. I try (Very hard) to leave any personal/professional attachment to any given product at the door and I have a zero tollerance for FUD. (Fear Uncertainty Doubt) So I beg all vendors commentators to leave the FUD, check the facts and let's just be real about storage. There may be some competetive analysis done on this blog, but I assure you, I will have check, re-check and checked again the information I present. However, should I get it wrong - which, above all else is a much greater tallent - I will correct it as quickly as possible.

Discussion

15 thoughts on “Dragging up ancient history – How NetApp fooled everyone (FAS3040 v. CX3-40)

  1. Austorageguy,

    You missed the entire point of NetApp’s exercise.

    The goal was to show the vast performance gap between EMC’s snapshot technology versus NetApp’s snapshot technology. You managed to go through all this trouble and never once addressed this.

    There are at least six other fallacies in your analysis, but they don’t matter. The whole point was to demonstrate the problem with EMC’s snapshot software, which I can tell you from pesonal experience, is still very poor compared to other vendors.

    Posted by Clock$peedy | February 26, 2013, 06:50
    • Another key point to this comparison… NetApp’s system wasn’t running a Reallocation scan. Thus in the writes to disk blocks they are scattered to the first available block on their disks. This caused the system to have a fragmented disk sectors that can cause a huge performance impact at a later date (more or less in terms of months). If they were really stupid and never had this run, the system could eventually hit a time period where the system’s becomes practically unusable while they have to run the reallocation to clean up the disks.

      What NetApp does well is that their snapshots cause a nearly instant backup replication and this a great point they have on their system. But for a true side by side comparison, they’d have to have the Reallocation turned on during the test as a CX3 array has a always on reallocation during writes (which causes the data to remain consistent across all of the disks).

      In comparison for this factor, the EMC’s system will always have a consistent maximum performance. While a NetApp system can have a temporary boost to performance (because you can disable then run reallocation scans later in the day).

      Overall you can’t have a true side by side comparison on both systems as they’re programmed differently with how they’ll handle the data. It is deceitful of NetApp to skew results but all companies do this. (I.E. Microsoft having comparisons with web browsers and bragging about how Internet Exploder is more Energy Efficient….)

      Posted by Tech Guy - EMC VNX certified & NCDA certified | November 30, 2013, 03:29
  2. You know, I can stipulate to your analysis of Netapp’s flawed SPC-1 result for the EMC CX3-40 as far as the actual IOPS and $$/IOPS outcome goes, but one thing Netapp definitely showed – what they seemed to be trying to show with this whole thing – was that enabling snapshots does not impact Netapp performance, but does, quite severely, impact EMC/CX/Flare-code performance and IOPS.

    The greater picture was that Netapp published TWO SPC-1 benchmarks for the FAS 3040, and TWO SPC-1 benchmarks for the CX3-40 EMC array. For each array, they published one benchmark with snapshots enabled, and one with snapshots disabled.

    The total IOPS may be in doubt, your thorough and insightful (inciteful?) analysis definitely calls the max IOPS figures into question, but the IOPS degradation due to the use of snapshots on the EMC array and the non-impact of snapshot use on the Netapp array is as definite, and seemed to me to be the main point of the four benchmarks that Netapp conducted and had published all at the same time.

    I heard that this was all due to an EMC blog that denied that Netapp snapshot technology was in any way superior to EMC’s snapshot technogy, or something along those lines. Netapp felt compelled to prove the EMC blogger wrong, and went to the extreme of publishing these four benchmarks as proof that EMC snapshot technology was inferior, and that the EMC blog post was completely off base. Whether or not this is the case, Netapp definitely proved the snapshot part, at least to my satisfaction.

    Since you know how both arrays work, and presumably how each of them handle snapshots, you would probably have guessed that the outcome of the four benchmarks would be roughly what was shown – Netapp with no impact from snapshot use, and EMC with very detectable impact to IOPS from snapshot use. I too have used both arrays, and have long wished that EMC would revise how snapshots are handled. Other vendors than Netapp have come up with zero impact snapshots (LeftHand, 3Par, Compellent, Pillar, Isilon, and more), why have EMC, HDS, LSI, HP/EVA, and others not fixed their snapshots to also exhibit no performance impact from snapshot use?

    Obviously it would take a lot of work at a very low-level in the array code to make such a change, but with snapshots in such common use these days across the spectrum, one would think it would pay dividends not to be so demonstrably behind the times in this important area of array and data center operations.

    Anyway, I enjoyed your blog entry, very in-depth and very well done analysis. I don’t really agree that this invalidates the entire concept of the SPC or the SPC-1 benchmark, however, since you’ve proved that it IS possible to uncover skulduggery and chicanery through analysis and careful reading. Also, since most vendors are only publishing THEIR products and results, most or all of the items you decry and call foul on from Netapp vis-a-vis their EMC published results would simply not exist – nobody would do that to themselves, and everyone should put their own best foot forward.

    It is interesting that EMC participates in a number of benchmarks, the ones they do reasonably well on. Why single out the SPC-1 for dismissal unless they felt they could not make a superior showing? Regardless, they seem to do well enough without participating in this one…

    Take Care, and thanks for all your work on this.

    Posted by Brian Rawlings | December 18, 2012, 06:53
  3. Hi

    I can answer some of the “why didn’t EMC …?” questions.

    First, to interact with the SPC, you have to be a member of the SPC. We weren’t at the time.

    That means, in addition to any resource commitment, you essentially are validating who they are, what they do, etc. EMC is a member of many testing and benchmarking organizations, but it’s fair to say that we’ve had issues with how the SPC does its business for a very long time.

    Second, to rerun the test properly takes some time and effort in addition to the aforementioned joining of the SPC. We’ve got a lot on our plate here already, and rock-fetches aren’t a favorite.

    Many of us felt that doing so would simply add more fuel to an already pointless discussion — at least from most customers’ perspective. Besides, we don’t like to encourage bad behavior when we can avoid it. I think it’s safe to say that smaller vendors usually enjoy picking a fight with larger vendors as it makes them look like equals, which — in this case — is not the case.

    I think from my perspective, I would weigh in on “since when did it become our job to correct stupid and misleading results from another vendor?” Especially when considering a test (and a supporting organization) of dubious credentials to begin with?

    Claiming it’s a “standard” brings absolutely nothing to the discussion in this regard, does it?

    At the end of the day, though, we try to focus as much as possible on the needs of our customers and partners, just like all good vendors should.

    Taken from that point of view, we saw little merit in the SPC test in this regard.

    And we saw even less merit in arguing back-and-forth regarding the specifics of childish benchmarketing stunts with vendors that really ought to know better.

    Thanks again

    — Chuck

    Posted by Chuck Hollis | February 4, 2012, 04:13
    • Hi Chuck,

      Thanks for that.

      I’ve got to say, I agree.

      It’s like validating kids screaming in a shopping centre by giving them candy to shut them up.
      Yeah, it’ll only work for the short term, but only serves to validate the same screaming the next time.

      It won’t silence them for long.

      No one should accept a standard which is below average.

      Aus storage guy

      Posted by ausstorageguy | February 4, 2012, 06:49
  4. Hello all, D from NetApp here (www.recoverymonkey.org)

    I got dizzy going through the article… 🙂

    I somehow doubt any back end was maxed out, this benchmark doesn’t generate that much throughput.

    Maybe I can boil this down a bit – since I had some of the same concerns when I first saw the benchmarks before I even joined NetApp:

    1. Netapp tried multiple configurations including MetaLUNs, write caching, etc.SPC oversaw and audited the results and only the fastest result on the best config was published.

    2. EMC was given a chance to have the results not published before publication. They did not respond.

    3. After publication, EMC was given a minimum of 30 days to have the publication pulled if they didn’t believe this was correctly configured. They did not respond.

    So, if this was so grossly misconfigured, why did EMC not stop it? They were allowed to. Given multiple chances, in fact.

    They will of course complain every chance they get.

    They are also a recent member of SPC. It is a simple matter to submit a system (even an older IDENTICAL CX3) and show how, “properly” configured, it will perform.

    They are also free to submit large VMAX and VNX all-flash configs.

    Whether we all like it or not, SPC-1 is a standard test, like SPEC SFS.

    And, simply, because SPC-1 is about 60% writes, it is brutal on most arrays, whereas NetApp enjoys some write optimization through WAFL.

    http://recoverymonkey.org/2010/11/09/176/

    Thx

    D

    Posted by Dimitris Krekoukias | February 3, 2012, 02:33
    • Hi Dimitris,

      Thank you for you comment.

      In no way am I suggesting that the backend was maxed out on either array, you’re right in saying it doesn’t generate throughput. (Especially as it’s driven from only one workload generator.)

      But in response, I’ll answer your comments one by one:
      1. Netapp tried multiple configurations including MetaLUNs, write caching, etc.SPC oversaw and audited the results and only the fastest result on the best config was published.

      I’m sure they did, however, that doesn’t discount the fact that NetApp used a much more restrictive workload generator, bound by PCI-X, where as for the FAS, a PCIe WG was used, NetApp themselves speak often about the benifits of the move to a full PCIe architecture in more recent FAS generations.

      No matter what was tested, oversaw or audited, it was a WG with a much slower bus and if the person is auditing it, he/she’s going to say “yup, 2x PCI-X cards, just as you say”

      Additionally, 8m cables were used for the second bus (which was the mirrored pair), which even NetApp knows will kill response time.
      No disclosure is given as to the type of the Meta-LUNs, so I will hazard a guess that concat LUNs were used.

      And to add insult to injury, NetApp even used bechmark specific parameters to their own FAS, which contraviened the SPC-1 rules. This from the company who foams at the mouth over “Benchmark Queens”.

      I would bet that the NetApp would have also got a better score if the daft windows striping wasn’t used.

      2. EMC was given a chance to have the results not published before publication. They did not respond.

      Belive me, I have no idea why they didn’t, I can only assume people much smarter that I had their reasons.
      But I do wonder, did they have the right to respond? Would it have been a worse PR nightmare to have “EMC pulls benchmark, cries foul” or were they obliged to become members to SPC to ask for a withdrawl?

      3. After publication, EMC was given a minimum of 30 days to have the publication pulled if they didn’t believe this was correctly configured. They did not respond.

      Again, with no stretch of the imagination, I can guess smarter people than I, made that decision for a good reason that is beyond my fathoming.

      So, if this was so grossly misconfigured, why did EMC not stop it? They were allowed to. Given multiple chances, in fact.

      I’m probably grasping at straws here, but I would guess that they handed it to one of the many beard strokers in engineering who would have duplicated the config and come to the same conclusion without properly configuring it. I.e what is there to contest.

      They will of course complain every chance they get.

      As does NetApp, I remember an terrible amount of moaning comming from the NetApp camp when EMC published “NetApp performance in SAN Environments” which I equally conceed was poorly configured.

      They are also a recent member of SPC. It is a simple matter to submit a system (even an older IDENTICAL CX3) and show how, “properly” configured, it will perform.

      You know, that’s not such a bad idea.

      They are also free to submit large VMAX and VNX all-flash configs.

      Whether we all like it or not, SPC-1 is a standard test, like SPEC SFS.

      Yeah, it’s just a bad test really though isn’t it? If a vendor is allowed to use benchmark specific parameters, then what’s the point?

      And, simply, because SPC-1 is about 60% writes, it is brutal on most arrays, whereas NetApp enjoys some write optimization through WAFL.

      Sure, but I would bet a pint with you that it would have handled it, given a decent chance.

      Really, it’s like Toyota has better fuel efficieny than Honda, because Toyota tested their car with 4 slim girls and tested Honda’s with 4 shoulder to shoulder sumo wrestlers.

      I just can’t get around the PCIe/PCI-X bus issue.

      Anyway D. Thanks again for the comments.

      Aus Storage Guy (Of no vendor affiliation)

      Posted by ausstorageguy | February 3, 2012, 03:30
  5. Great job on documenting what Netapp did. This why no one will ever really trust SPC benchmarks. Maybe someone should start a company like consumer reports who is a trusted third party to test our systems.

    Posted by richswain | February 2, 2012, 23:23
    • Thanks Richard,

      I think many people know there was somthing fishy going on, but could never pick exactly what.

      Great idea, now if only I could find a VC who won’t take over once it starts to make money :).
      I think that’s how most of the market research firms started out, but ended up discovering that more money was to be made in “bought” papers; that’s just the way of the world.

      As for the trust placed in SPC benchmarks, unfortunately, I’m confronted with it all the time, even when working with a vendor who does have a published SPC-1 benchmark. It just muddies up the waters, they want their array configured with the maximum usable capacity available, but is supprised when their array doesn’t meet the SPC-1 benchmark; because the customer ordered half the disks tested in the benchmark, got the baby version of the array or isn’t using R1/0. Add that to the fact that they often use a different benchmarking tool to come to the conclusion.

      Anyway, thanks again for the comment.

      Posted by ausstorageguy | February 2, 2012, 23:36
  6. Thank you.

    You’ve done a great service to infrastructure people everywhere by showing them how to dig into an “industry standard benchmark” and look closely. Unfortunately, few people have the time and skills you have — but your work is an excellent example of good critical thinking.

    I won’t comment on NetApp’s gross behavior here — as I work for EMC — but even if we spot such nonsense, our protestations are usually ignored since we’re a competing vendor, don’t you know?

    The real bad guy here (in my mind) is the Storage Performance Council (?!?) who lets this sort of stuff routinely go on, much to users’ disadvantage. For this particular event, Walter Baker, the part-time “administrator” of the SPC was trotted out to “validate” the results.

    A very sorry episode for all involved.

    To this day, neither NetApp nor the SPC have acknowledged any shortcomings in either their methodology or results. Which means there’s a good possibility we’ll see this sort of thing again.

    The vendor fanbois are an annoying lot; it’s like talking sports to a rabid fan, or arguing religion with an Evangelical — there’s no room for alternative perspectives whatsoever. Maybe they’ll grow out of it.

    — Chuck Hollis

    Posted by Chuck Hollis | February 2, 2012, 22:08
    • Hi Chuck,

      Thanks for your comment.

      Having the time is the easy one… be on the otherside of the planet, away from loved ones and a head full of coffee and jetlag.
      Having the skill? I don’t think I’ve stepped outside of the skillbase of any decent storage engineer, just put it down on paper so to speak, but thanks for the compliment.

      All vendors do this – it must be said – right or wrong; however, I do have to agree with you.
      Not so much as to single out an individual, but the SPC benchmark as a whole; the trouble is, Walter may never have seen, nor configured a NetApp or EMC CX3-40 in his life and therefore identify items which shouldn’t be in the BoM nor know the differences between PCI busses.

      But the trouble I do have is with the SPC-1 benchmark. To this day, I am yet to meet a customer who; given the life time of an array, will knowingly only ever use 45% of the arrays provisioned capacity, or even worse, forsake further capacity in the array (which allows vendors to short stroke) which is no provisioned.

      I don’t know about you, but in this climate, most can barely convince buyers to add an extra 10% for short-term growth, let alone sell an array with more than 55% headroom. But what staggers me the most, is even if they did, how long would it remain at 45%?

      I’ll have to dedicate a post to the SPC-1 benchmark in the future.

      For me, so long as I never have to sit through someone dragging out that sorry excuse for a doctored benchmark, I’ll be happy.

      As for the fanbois, well, aren’t we all just a little bit sometimes as you point out – It’s been going on since before the dawn of time.
      Bone vs. stick club
      iron vs. stone axe
      Ford vs. GM
      Canon vs. Nikon
      Punk vs. New Romantic

      But I tell you what, some of the comments and emails I’ve had were downright venomous.

      Maybe the SPC Council will get a glimpse of this blog and consider if investigation is required.

      Belive me when I say this, but I do like NetApp, I’m just so tired of this blasted benchmark being dragged out by them, when I know – and they know – it’s factually way of kilter.

      Waiting with baited breath for the 6th….

      With regards,

      Aus Storage Guy.

      Posted by ausstorageguy | February 2, 2012, 22:44
  7. Well, well, you so passionate, my dear. 🙂 It’s not a sign of pro.

    So, now you ready to show us a proper, proven, ‘short stroked’ configuration for EMC CX _without_ 3x performance penalty during using of snapshots, right? It will be great to see it here in your next post. 😀

    About PCI-X and PCIe.

    Well, I saw it’s a _EMC’s_ fault when they still using a PCI-X in their mainline storage in 2009, not a NetApp’s fault isnt’ it?
    So, your blaming now hitting to wrong target 🙂

    Posted by romx | January 2, 2012, 17:53
    • Hi romx,

      Thanks for your comment.

      Yes it was a passionate response, maybe a little over the top even, but one should be passionate about what one does.

      I wrote this whilst jet lagged in another hotel and saw another abuse of this example which angered me.
      I’ve been just as angered with other vendors when they speak ill of NetApp and will post on this in the future.

      As for the snapshot issue, oracle storage guy wrote an excellent piece on this some time ago:http://oraclestorageguy.typepad.com/oraclestorageguy/2007/08/oracle-backup-1.html

      I might write a piece on this also at some stage.

      Now about the pcie/pci-x bit, the cx3 is totally pcie, however, for the spc1 benchmark which NetApp posted, NetApp choose to use pci-x hba’s for the host computer used to test the cx3, but used pcie for the fas.

      In other words NetApp gave themselves several advantages.

      I would think it unfair if another vendor did the same to NetApp, wouldn’t you?

      So it was right on target, sad to say.

      My main wish is that NetApp and anyone else using this manufactured and cheated benchmark can now cease. It’s just plain lying to customers.

      Aus storage guy.

      Posted by ausstorageguy | January 4, 2012, 11:50

Trackbacks/Pingbacks

  1. Pingback: Storage Administrator Primer | Matt Davis - November 1, 2015

  2. Pingback: Why I belive you should stop worrying about SPC-1 benchmarks. « ausstorageguy - November 15, 2012

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: