Mailing List CGatePro@mail.stalker.com Message #92666
From: Jon R. Doyle <jdoyle@communigate.com>
Subject: Re: Filesystem layout and CGP performance
Date: Wed, 19 Sep 2007 20:12:23 -0700
To: <cgatepro@communigate.com>
X-Mailer: CommuniGate Pro Pronto 1.2.1
Hello;

well, there are alot of things that _could_ be wrong. Mail servers tend to stress filesystems in ways the manufacturers never expect, because they tend to not test with them, but rather DBs and other business apps. Over the last 15 years, am I that old? I have seen a lot of large systems, and I think been involved with just about every storage/server/OS combo out there.

First, 27k users and your usage for 3 years does not raise eyebrows, rather it points to some new fact, your capacity ceiling.

I have seen many times Linux go postal under high load, but for the above I could put that aside for the moment. Same with EXT3, it has been a while from doing native Linux FS testing, because normally, at least in CGatePro terms, we move to a shared FS in an ISP test or deployment. I can say ReiserFS used to perform _very_ well, and there are docs out there about those tests, in particular Mailserver tests. But, over time I think we can say with the deployments of EXT3, it should be stable and not freaking out.

So then, that leaves your array, first thing, the more spindles, the better. Put all the money you can in more of them vs less with more capacity. We had some metrics for that at Compaq for Oracle deployments, you might find those papers on a google search. But the jist is, spending the same for more qty of smaller discs to reach the same capacity size will yield a far higher performance metric. Do not split up the array into paks, it does little good, and causes the controller to do more work.

Lastly, I would point to your use of RAID 5 at 85% usage, and running multiple paks. RAID 5 is not the best choice, and most controllers do have issues at near full capacity. Remember, in RAID 5 they are dealing with a lot of parity calculations. There are better controllers out there, more cache and tuning these days, but it is better to sink the money in spindles.

I would say the same for your investments in SAN, servers etc, put the money in spindles, and remove the complexities. With SAN you will introduce a CFS, unless you are putting a NAS head there?, and if you are using NFS, the same applies, more spindles the better, and use RAID 0+1 stripe the whole pak and use different mount points.

Regards,

Jon
Subscribe (FEED) Subscribe (DIGEST) Subscribe (INDEX) Unsubscribe Mail to Listmaster