Tongfamily (Richtong.com Sync)

Well, we got a pretty amazing server two years ago and it is interesting to see how little has changed, most of the parts are still good (Haswell-E), but the main thing that has changed are:
SSDs in Server

• NVMe. Wow boot drives and cache drives are much faster. With a Xeon platform there are plenty of PCIe lanes. So getting an adapter with 4x NVMe on one side and 2 SATA to m.2 is really a nice high density solution. 
• There are many flavors of SSDs now particularly in the m.2 (tiny) form factor so in order of performance. The pricing is such that it doesn’t make sense to get anything but the top end NVMe 4x Pcie
  • SATA b key. These are limited to the 6GBps of SATA and typically top out at 400MB-500MBps. You can get some great cheap SSDs this way like the performance oriented SanDisk Extreme Pro (480 and 960GB) at $200-400. Or if you need lots of consumer grade read-only store, the Mushkin Reactor 1TB ($225) is hard to beat. 
  • m.2 2x PCIe b-key. Some SSDs like the Plextor M6e were in the middle and can use up to 2x PCIexpress to give you 1.2Gbps. These were an intermediate step. Check your motherboard to see what kind of slot you have as some are 2x.
  • m.2 4x PCIe. Another transitional product, they can deliver 2.2Gbps (!!) of performance but need 4 channels. You can normally get one on a motherboard particularly the latest Z170 flavors. If you want more you need a PCIexpress adapter and 4 free channels. Remember your graphics card typically uses 16x. The older Samsung 850 is an example. I’m not sure if this is m- or b-key.
  • Ngff m key m.2 4x PCIe NVMe. Right now pricing wise these are the ones to get. With NVMe short writes and reads are 10-20x faster and of course you are running at 2.2Gbps. The XP941 and Samsung 950s are the go here.
• Note that read and write are very different and also large vs random. Also there is something call overprovisioning so more expensive really have way more flash than you think. But in general flash really dislikes writes and can be read all day long. So consumer drives are good for lots of reads and passive storage. Enterprise writes. 

SSD Drive Adapters to PCI Express and into SAS Racks

• Amazon has a very cool adapters. While there are dedicated PCIe NVMe, having an M.2 SSD is way more flexible and it is only $25 for a simple 4x PCIexpress adapter. SMAKN makes one for $18 which is highly riate
• There is another dual adapter made by XCG that $16 and has space for a b key Sata and the newer m key ngff m.2. So it is a nice way to mount the standard m.2 that then goes to a SATA connector. Sedna also sells one that looks identical for $23
• Finally if you want a standard ssd in a hot swap chassis you can get at puts all  The ICY DOCK MB882SP-1S-3B is the same as the -1B but is an open system and a little cheaper at $13 vs $16 and supports any drive size. The -2B only works for 9mm drives while the -2B works for 7mm as well (the new Samsung 850 are thinner).
• Then on the other end of a SAS system you would get a 4x SATA to mini-SAS reverse breakout to plug it into your SATA ports controller. HIghpoint makes one for about $30.

SAS Hard Disk

Then there is what is up with hard drives. This is mainly remembered but the big things are picking the right SAS drive

• SAS vs SATA. You want SAS as they are 10 more reliable. 10^15 vs 10^14 for undetectable errors. They  have a premium but not as much as you think. A 6TB SAS drive from Seagate is called nearline SAS and is about $260. You want the advanced format 512E drives for maximum compatibility. These are native 4K underneath. If there isn’t much of a cost difference, the SED drive is really nice. it is AES-256 encrypted so you don’t have to wipe when throwing it away. 
• Seagate vs. Hitachi. These seem to be the two big names. Seagate has more performance for their enterprise series although Hitachi is quite reliable. The models are really confusing and poorly documented. but Tweaktown has a good decoder ring. In short models ending. Note that the ST6000NM0004 is the SATA drive:
  • ST6000NM0034 are the standard 512e and the cheapest usually (right now $270 at Amazon)
  • ST6000NM0054 are the SED models so most desirable if the price is right (right now $459)
  • ST6000NM0014 are native 4K (only get these if cheaper and you know you are going to use modern Linux or Windows 7 or so or higher right now $320)
  • ST6000NM0104 are the government models FIPS-SED and are usually very expensive. (right now $454)

• SAS 6Gbps vs 12Gbps. There are two version. No true hard disk can go blow out 6Gbps. As an aside, SATA is also 6Gbps but SAS is about 10% more efficient since it has less overhead. So if you know you are going to have HDs then you can get away with 6GBps controllers as they are compatible

SAS PCI Express Adapters

Finally you have to connect these SAS drives. They use something called a miniSAS connector. This has four drives worth of connection and you have to get a pcie SAS controller. Now a 12Gbps SAS needs a full pci express lane. So the most common controller supports four drives. You can get motherboard like the Supermicro X10 with built in SAS adapters. But if you say 24 drives then you will need at least two more SAS adapters:

• 3x mini-SAS-8087
• Hardware RAID or not. The hardware RAID are more expensive and tradeoff against the software raid used by systems like ZFS. 
• 6Gbps or 12Gbps. The 6Gbps can use PCI Express 2.0 (half the speed of 3.0) and require a lane per SAS drive so a SAS-8087 x 4 would need 4x PCI Express 2.0 lanes while a x 8 would need for PCI Express 2.0 lanes.

So some sample cards are. Most are 8x SAS and require 8x PCI Express 2.0 lanes running at 6Gbps. The main issue is making sure it is well supported on Ubuntu:

• Supermicro SAS2LP-MV8. For $100 this is a completely stupid JBOD only card. Not a bad match if you plan to use ZFS or software RAID anyway. And it is directly supported in Ubuntu. So a winner if you are using an advanced file system on top.
• LSI Logic 9260-8i. This is actually Avago and has a good rating on Newegg. 8x SAS, 8x PCIe. 512MB cache. Good reviews. This is well supported.
• Adaptec 8885. This is a top of the line card with 8 internal ports and 8 external. It has 12GBps so ready for SSD based system and uses PCI Express 3.0 as well. A good future proofed choice and is the techie’s dream and decently priced at $540. 1GB cache too.
• LSI Logic 9361-8i. The card to really love. 12GBps. 1GB cache from the market leader. 8 internal ports. $600. main issue is that it runs very hot and needs a cooling fan next to it. For it to really do hardware RAID, you need $200 more components in the BBU and Cachevault, but that’s the enterprise business for you.

Some others I looked at but rejectd

• Highpoint RocketRaid 2720 GL. This is a very cheap $200 card which doesn’t have an onboard CPU, it just has software like ZFS that does the software RAID. This is great if you are going to use systems like ZFS or btrfs that have their own RAID flavors. The main issue is reliability of the card.
• Adaptec 6805E. Another example of a low cost $220 card with just a little cache and is quite popular. 128MB cache.
• Intel RS2BL080. This is a good example of a smart controller that costs $500 with it’s own onboard hardware raid. The really fancy controllers go up 

Fast Disk Copies with SSD Adapters
If the data sets are mainly read-only like machine learning training sets, the a sneaker net thing to do is to copy the data onto an SSD and use an SSD to USB 3.1 or Thunderbolt adapter to get the information across. This is not as slow as you think. 400MBps is possible acorss Thunderbolt 2 or USB 3.1. You just need an inexpensive adapter:

• For $200 you can get a Thunderbolt 2 to SATA adapter and this will transfer 400MBps easily 
• Startech SATA to USB. For $22 you can get a SATA to USB 3.0, to USB 3.1 (this is 10Gbps so in theory gives you a little more with SATA at 6Gbps)  in either USB A or USB C format factor. Just plug your SATA drive into it and go.

10GB Ethernet
Getting terabytes across is pretty slow on a 1Gb Ethernet. Even with transfer speeds of 100MBps, it takes a long time to get a terabyte across. The answer is an expensive internal ring of servers with 10Gbps ethernet ports. This is not as expensive as you would think you need. But the idea of getting 1GBps across makes a terabyte transfer possible in a decent amount of time (like an hour).

• 10GBps Ethernet switch. These are $100 a port so 8 ports are $800
• 10GBps Ethernet host adapters in the file server and the gpu farms or a few power workstations. $220 a NIC for the Startech.
• Cat 6a cables. This allows 100 meters between 10GBps systems so you need to wire to specific machines to make this all work but that isn’t too expensive about $20 for 50 feet of Cat 6A cable

SAS Expanders
You can also hang a completely different enclosure off of an external SAS connector. Then all the drives in that chassis share the single 6Gbps bandwidth which seems terrible in theory but in practice apparently does not affect performance that much. These can be truly huge systems with tons of drives against a single controller. Not terribly necessary for most folks.