From ajt at rri.sari.ac.uk Wed Oct 1 03:49:11 2008 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Re: MOSIX2 In-Reply-To: <05B21F91-7E08-4015-B23E-AED651AE457A@cesr.fr> References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E1EDE3.2080205@rri.sari.ac.uk> <05B21F91-7E08-4015-B23E-AED651AE457A@cesr.fr> Message-ID: <48E355A7.6070008@rri.sari.ac.uk> J?rgen Kn?dlseder wrote: > Hi Tony, > > I'm in the same situation as your are: I'm running an openMosix cluster, > but since it's more and more > difficult to integrate new hardware with an old 2.4.26 kernel I think > that I have to move to MOSIX2. > I just got the latest version of MOSIX2 sent from Amnon Barak (I'm using > the cluster for academic > work), and started to play around with installations ... yet I had some > kernel crash problems that > I could not yet resolve on one of the machines. So I guess without the > $1,000 per year support > fee it'll also be difficult to run a MOSIX2 cluster ... Hello, J?rgen. I ported 2.4.26-om1 to compile it using the 'new' GNU tool-chain and run it under Ubuntu 6.06.1. I also patched the kernel.org 2.4.32 sources for openMosix (to get SATA drivers). However, I couldn't get openMosix process migration working properly between 2.4.32-om1 kernels... Ironically, process migration worked between a 2.4.26-om1 and 2.4.32.om1 kernel! Anyway, I gave up the attempt at 2.4.32-om1, and just continued to use my version of 2.4.26-om1 in production with SCSI controllers. If you're interested, my Ubuntu 6.06.1 openMosix deb's are at: http://bioinformatics.rri.sari.ac.uk/openmosix Are you going to continue using MOSIX2? Tony. -- Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk mailto:ajt@rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt From ajt at rri.sari.ac.uk Wed Oct 1 04:10:13 2008 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Re: MOSIX2 In-Reply-To: References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E1EDE3.2080205@rri.sari.ac.uk> Message-ID: <48E35A95.7020908@rri.sari.ac.uk> Vincent Diepeveen wrote: > I agree tony that paying for such crap is not very good idea. Hello, Vincent. I don't think MOSIX2 is crap! However, I don't like the idea of having to pay for 'updates'. > You might want to move to open-ssi in this case; the project is alive > and there is in theory work getting > performed on support for cards over infiniband as well. I have looked at OpenSSI, which uses the openMosix load-balancer, but process migration is more coarsely grained than in openMosix. Only the active pages of the user context of openMosix processes are migrated. I've been looking at alternatives and I think Kerrighed looks very promising but, in our hands, Kerrighed is very fragile: I've mentioned on this list before that if one Kerrighed node goes down you lose the entire cluster. We've been talking to Christine Morin's group at INRIA and they tell us that the next release of Kerrighed with be more robust: http://www.kerrighed.org > Most importantly is that you are gonna get more replies. Yes, thanks for yours :-) > Additionally the manner open-ssi implements shared memory is very > transparant; in principle on each write it migrates a page > to the node writing. > > Maybe the only big lack of open-ssi is its limited support so far for > highend network cards. What bothers me about OpenSSI is that it's based on an open-sourced version HP's (now Compaq) discontinued commercial product "non-stop clusters for Unix". The OpenSSI project also came in for a lot of criticism from the openMosix community for stealing ideas, so my concerns about it might not be all that well founded ;-) The main reason I didn't use OpenSSI, previously, was that many features had not been implemented fully and, like Kerrighed, it wasn't really a viable option for a 'production' cluster even though it was interesting as a research project. What makes me take MOSIX2 seriously now is that it is a commercially supported 'product' with all the same virtues (and most of the vices) of openMosix. Tony. -- Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk mailto:ajt@rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt From ajt at rri.sari.ac.uk Wed Oct 1 04:13:00 2008 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Re: MOSIX2 In-Reply-To: <200810010212.00791.mm@yuhu.biz> References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E1EDE3.2080205@rri.sari.ac.uk> <200810010212.00791.mm@yuhu.biz> Message-ID: <48E35B3C.20401@rri.sari.ac.uk> Marian Marinov wrote: > There are a few developers that continue to work on openMosix and to port it > to 2.6 kernels. > > They forked a project called LinuxPMI - Linux Process Migration Infrastructure > > http://linuxpmi.org > > Currently the site is unavailable but there was one Russion guy who commented > big parts of the openMosix code he started workin on that in the begining of > February this year and in May he and a few other developers forked LinuxPMI > from openMosix. > > I had a working 2.6 kernel with openMosix functionality in Jul but I never had > the chance to test its process migration capabilities as the nodes from my > home cluster died :( Hello, Marian. Great! I didn't know about LinuxMPI: I've not been following openMosix developments recently. I can't access their web site - Is it mirrored anywhere? Thanks, Tony. -- Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk mailto:ajt@rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt From ajt at rri.sari.ac.uk Wed Oct 1 04:33:38 2008 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Re: MOSIX2 In-Reply-To: <48E35B3C.20401@rri.sari.ac.uk> References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E1EDE3.2080205@rri.sari.ac.uk> <200810010212.00791.mm@yuhu.biz> <48E35B3C.20401@rri.sari.ac.uk> Message-ID: <48E36012.1070406@rri.sari.ac.uk> Tony Travis wrote: > [...] > Hello, Marian. > > Great! I didn't know about LinuxMPI: I've not been following openMosix > developments recently. I can't access their web site - Is it mirrored > anywhere? Following up my own message, I meant LinuxPMI of course ;-) Not much evidence that the LinuxPMI project is still active, though. Google reports a lot of hits for the pending DNS de-registration of "linuxmpi.com'... Anyone else know what's happening with LinuxPMI? Thanks, Tony. -- Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk mailto:ajt@rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt From ajt at rri.sari.ac.uk Wed Oct 1 05:16:29 2008 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Re: MOSIX2 In-Reply-To: References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E1EDE3.2080205@rri.sari.ac.uk> <05B21F91-7E08-4015-B23E-AED651AE457A@cesr.fr> <48E355A7.6070008@rri.sari.ac.uk> Message-ID: <48E36A1D.2020901@rri.sari.ac.uk> J?rgen Kn?dlseder wrote: > Hi Tony, > > in fact, I also patched SATA drivers in the 2.4.26-om kernel. Yet I > could so far not manage to > get the kernel working for my latest DELL PE1950 with a Perc 6i driver > ... maybe the megaraid_sas > driver I use is outdated ... Hello, Jurgen. I did have 'some' SATA support using 2.4.27-om1 from the ClusterKnoppix sources, but the drivers were very old. The 2.4 kernel *is* still supported at Kernel.org and the more recent SATA drivers worked a lot better. However, I don't want to invest a lot of time and effort keeping 2.4.xx-om1 alive without a critical mass of other openMosix users and developers. Sadly, whatever the virtues of openMosix, I think it is now becoming unsupportable for use on a 'production' Beowulf cluster. > I also search for LinuxPMI project information, but without any success. > So I share your feeling that > this projects is probably dead ... It's probably just as well, or I would be tempted to join in ;-) Tony. -- Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk mailto:ajt@rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt From Bogdan.Costescu at iwr.uni-heidelberg.de Wed Oct 1 05:52:29 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: References: Message-ID: On Tue, 30 Sep 2008, Donald Becker wrote: > Ahhh, your first flawed assumption. > > You believe that the OS needs to be statically provisioned to the nodes. > That is incorrect. Well, you also make the flawed assumption that the best technical solutions are always preferred. From my position I have seen many cases where political or administrative reasons have very much restricted the choice of technical solutions that could be used. Other reasons are related to the lack of flexibility from ISVs which provide applications in binary form only and make certain assumptions about the way the target cluster works. Yet another reason is the fact that a solution like Scyld's limits the whole cluster to running one distribution (please correct me if I'm wrong), while a solution with node "images" allows mixing Linux distributions at will. > The only times that it is asked to do something new (boot, accept a > new process) it's communicating with a fully installed, up-to-date > master node. It has, at least temporarily, complete access to a > reference install. I think that this is another assumption that holds true for the Scyld system, but there are situations where this is not true. Some years ago I have developed a rudimentary batch system for which the master node only contacted the first node allocated/desired for the job; this node was then responsible to contact the other nodes allocated/desired and start the rest of the job. This was very much modelled after the way the naive rsh/ssh based launchers for MPI jobs work: once mpirun is running, there is no connection to the master node, only between the node where mpirun is running and the rest of the nodes specified in the hosts file. I think that Torque also has a similar design (Mother Superior being in control of the job), but I haven't look closely at the details so I might be wrong. > If you design a cluster system that installs on a local disk, it's > very difficult to adapt it to diskless blades. If you design a > system that is as efficient without disks, it's trivial to > optionally mount disks for caching, temporary files or application > I/O. If you design a system that is flexible enough to allow you to use either diskless or diskfull installs, what do you have to loose ? The same node "image" can be used in several ways: - copied to the local disk and booted from there (where the copying could be done as a separate operation followed by a reboot or it can be done from initrd) - used over NFS-root - used as a ramdisk, provided that the node "image" is small enough Note: I have used "image" in this and previous e-mails to signify the collection of files that the node needs for booting; most likely this is not a FS image (like an ISO one), but it could also be one. Various documents call this a "virtual node FS", "chroot-ed FS", etc. -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From rgb at phy.duke.edu Wed Oct 1 07:18:20 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] precise synchronization of system clocks In-Reply-To: <20081001020421.GA8126@compegg.wr.niftyegg.com> References: <20081001020421.GA8126@compegg.wr.niftyegg.com> Message-ID: On Tue, 30 Sep 2008, Nifty niftyompi Mitch wrote: > Also while focusing on network/transport in this discussion none of us > made a comment on rotational latency as a source of uncertainty for the > kernel state. If we had the ability to synchronize the systems exactly > starting a process would lag for want of rotational/seek disk latency > in beowulf'y. Sure, but starting processes is presumed to occur once and then (an efficient computation) proceeds for a long time. All the parallelized tasks effectively arrive at the same point at the computation at the first barrier AFTER all this is finished. If the communications involve giving the kernel a slice at (say) the end when the barrier is released for all nodes at "the same time", then the distributed kernels CAN have roughly the same state IF one suppresses enough of the "random" sources of state-noise -- asynchronous demands for the kernel's attention. To the extent that the kernel can accumulate all of its regular housekeeping and do it an elective time, if one gets the system clocks together within a usec and the kernel does its elective work on clock ticks, that work will end up being done (mostly) synchronously across the nodes. Truthfully, all one is trying to do is to generalize your parallel process to have a double synchronous barrier, with one phase of the computation being "kernel housekeeping". Compute (in parallel) -> barrier (IPCs) -> Kernel (in parallel) -> barrier -> Compute -> barrier -> Kernel -> barrier ... ad inifinitum. If the work done by the kernel is fairly tightly bounded -- predictably completes in (say) 100 usec (which is a LOT of time, far more than one almost ever sees one STILL has 900 usec per tick to work on your compute task. If the kernel (more reasonably) completes in 1-10 usec your cluster should have a 99+% duty cycle but avoid the "noise" that desynchronizes everything. > > Shared memory machines and transports will behave differently. > > The very high accuracy and high precision clock synchronization is a very real > problem for some data gathering systems. Once the data is gathered the > computation should be less sensitive. These are different problems and > might be addressed by the data sampling devices. > > Synchronization brings problems.... for example a well synchronized campus > can hammer yp server and file servers when cron triggers the same actions on > 5000+ systems... I try never to fetchmail at the hour, half hour... > > I suspect that some system cron tasks should no longer run from cron. Common > housekeeping tasks necessary for system health should be run via the batch system > in a way that is fashionably late enough to not hammer site services. Absolutely. In fact, you'd want the nodes to be isolated and not running ANY of this stuff, I'd guess. You'd want the nodes to have quiescent, non-demanding hardware (except for devices doing the bidding of the running parallel process) so that nothing "random" needed to be done that couldn't be saved for the kernel slices. > One site service of interest is AC power. A modern processor sitting > in an idle state that then starts a well optimized loop will jump from > a couple of watts to 100 watts in as many clocks as the set of pipelines > is deep behind the instruction decode and instruction cache fill. A 1000 > processor (4000 cores) might jump from 4000 watts to 100000 watts in the > blink of an eye (err did the lights blink). Buffer that dI/dT through > the PS and it is less but still interesting on the mains which are synchronized. Interesting. I never have seen the lights blink although I don't run synchronous computations. One wonders if the power supply capacitors (which should be quite large, I would think) don't soak up the transient, though, even on very large clusters. Also, I think that the power differential is smaller than you are allowing for -- I don't think most idle processors draw "no" power... rgb > > > > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From prentice at ias.edu Wed Oct 1 08:30:10 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? Message-ID: <48E39782.4020006@ias.edu> In the ongoing saga that is my new cluster, we were just told today that Cisco is no longer manufacturing DDR IB cables, which we, uhh, need. Has DDR IB gone the way of the dodo bird and been supplanted by QDR? If so, why would anyone spec a brand new cluster with DDR? -- Prentice From prentice at ias.edu Wed Oct 1 08:44:01 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Advanced Clustering's Breakin References: 20670.89.180.225.196.1217678257.squirrel@www.di.fct.unl.pt Message-ID: <48E39AC1.7080001@ias.edu> > We have a tool on our website called "breakin" that is Linux 2.6.25.9 > patched with K8 and K10f Opteron EDAC reporting facilities. It can > usually find and identify failed RAM in fifteen minutes (two hours at > most). The EDAC patches to the kernel aren't that great about naming > the correct memory rank, though. > > Make sure you have multibit (sometimes says 4-bit) ECC enabled in your BIOS. > > http://www.advancedclustering.com/software/breakin.html I've been using breakin for the past week or two on my new cluster. I get some results that seem to be inconsistent. For example on a node I'll get this: Test | Pass | Fail | Last Message ------------------------------------------ hdhealth | 315 | 0 | No disk devices found Then in the log section: 00h 57m 40s: Disabling burnin test 'hdhealth' If I reboot and restart the testing, it will see a hard disk. Why is breaking not always seeing the disk? I've tried to dump logs to a USB drive, but breakin refuses to mount the correct partition on my usb drive (/dev/sdb vs. /dev/sdb1, or vice versa). I sent e-mail to Advanced Clustering regarding these issues, but didn't get any response, so I"m hoping I have better luck here. -- Prentice From jurgen.knodlseder at cesr.fr Wed Oct 1 04:54:44 2008 From: jurgen.knodlseder at cesr.fr (=?ISO-8859-1?Q?J=FCrgen_Kn=F6dlseder?=) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Re: MOSIX2 In-Reply-To: <48E355A7.6070008@rri.sari.ac.uk> References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E1EDE3.2080205@rri.sari.ac.uk> <05B21F91-7E08-4015-B23E-AED651AE457A@cesr.fr> <48E355A7.6070008@rri.sari.ac.uk> Message-ID: Hi Tony, in fact, I also patched SATA drivers in the 2.4.26-om kernel. Yet I could so far not manage to get the kernel working for my latest DELL PE1950 with a Perc 6i driver ... maybe the megaraid_sas driver I use is outdated ... I also search for LinuxPMI project information, but without any success. So I share your feeling that this projects is probably dead ... J?rgen Le 1 oct. 08 ? 12:49, Tony Travis a ?crit : > J?rgen Kn?dlseder wrote: >> Hi Tony, >> I'm in the same situation as your are: I'm running an openMosix >> cluster, but since it's more and more >> difficult to integrate new hardware with an old 2.4.26 kernel I >> think that I have to move to MOSIX2. >> I just got the latest version of MOSIX2 sent from Amnon Barak (I'm >> using the cluster for academic >> work), and started to play around with installations ... yet I had >> some kernel crash problems that >> I could not yet resolve on one of the machines. So I guess without >> the $1,000 per year support >> fee it'll also be difficult to run a MOSIX2 cluster ... > > Hello, J?rgen. > > I ported 2.4.26-om1 to compile it using the 'new' GNU tool-chain > and run it under Ubuntu 6.06.1. I also patched the kernel.org > 2.4.32 sources for openMosix (to get SATA drivers). However, I > couldn't get openMosix process migration working properly between > 2.4.32-om1 kernels... > > Ironically, process migration worked between a 2.4.26-om1 and > 2.4.32.om1 kernel! Anyway, I gave up the attempt at 2.4.32-om1, and > just continued to use my version of 2.4.26-om1 in production with > SCSI controllers. If you're interested, my Ubuntu 6.06.1 openMosix > deb's are at: > > http://bioinformatics.rri.sari.ac.uk/openmosix > > Are you going to continue using MOSIX2? > > Tony. > -- > Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition > and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK > tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk > mailto:ajt@rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From mm at yuhu.biz Wed Oct 1 06:03:10 2008 From: mm at yuhu.biz (Marian Marinov) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Re: MOSIX2 In-Reply-To: <48E36012.1070406@rri.sari.ac.uk> References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E35B3C.20401@rri.sari.ac.uk> <48E36012.1070406@rri.sari.ac.uk> Message-ID: <200810011603.10435.mm@yuhu.biz> I lost touch with Juri (the guy who started the documentation project of openMosix) 2-3 months ago. More info about LinuxPMI you can find on Freenode #linuxpmi or #openmosix As far as I can see the domain is registered until next year and there is some routing problem New York before the server. I'm sorry but I do not know of any mirrors as Juri's machine was the central storage of the documented patches. Hi actually split the openMosix into series of patches so that it can be ported to 2.6 easier. Regards Marian On Wednesday 01 October 2008 14:33:38 Tony Travis wrote: > Tony Travis wrote: > > [...] > > Hello, Marian. > > > > Great! I didn't know about LinuxMPI: I've not been following openMosix > > developments recently. I can't access their web site - Is it mirrored > > anywhere? > > Following up my own message, I meant LinuxPMI of course ;-) > > Not much evidence that the LinuxPMI project is still active, though. > Google reports a lot of hits for the pending DNS de-registration of > "linuxmpi.com'... > > Anyone else know what's happening with LinuxPMI? > > Thanks, > > Tony. From becker at scyld.com Wed Oct 1 09:35:37 2008 From: becker at scyld.com (Donald Becker) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: Message-ID: On Wed, 1 Oct 2008, Bogdan Costescu wrote: > On Tue, 30 Sep 2008, Donald Becker wrote: > > Ahhh, your first flawed assumption. > > You believe that the OS needs to be statically provisioned to the nodes. > > That is incorrect. > Well, you also make the flawed assumption that the best technical > solutions are always preferred. From my position I have seen many ... > a solution like Scyld's limits the whole cluster to running one > distribution (please correct me if I'm wrong), while a solution with > node "images" allows mixing Linux distributions at will. That's correct. Our model is that a "cluster" is a single system -- and a single install. That's for a good reason: To keep the simplicity and consistency of managing a single installation, you pretty much can have... only a single installation. There is quite a bit of flexibility. The system automatically detects the hardware and loads the correct kernel modules. Nodes can be specialized, including mounting different file systems and running different start-up scripts. But the bottom line is that to make the assertion that remote processes will run the same as local processes, they have to be running pretty much the same system. If you are running different distributions on nodes, you discard many of the opportunities of running a cluster. More importantly, it's much more knowledge- and labor-intensive to maintain the cluster while guaranteeing consistency. > > The only times that it is asked to do something new (boot, accept a > > new process) it's communicating with a fully installed, up-to-date > > master node. It has, at least temporarily, complete access to a > > reference install. > > I think that this is another assumption that holds true for the Scyld > system, but there are situations where this is not true. Yes, there are scenarios where you want a different model. But "connected during important events" is true for most clusters. We discard the ability for a node to boot and run independently in order to get the advantages of zero-install, zero-config consistent compute nodes. > > If you design a cluster system that installs on a local disk, it's > > very difficult to adapt it to diskless blades. If you design a > > system that is as efficient without disks, it's trivial to > > optionally mount disks for caching, temporary files or application > > I/O. > > If you design a system that is flexible enough to allow you to use > either diskless or diskfull installs, what do you have to loose ? In theory that sounds good. But historically changing disk-based installations to work on diskless machines has been very difficult, and the results unsatisfactory. Disk-based installations want to do selective installation based on the hardware present, and write/modify many links and configuration files on installation -- many more than they "need" to. > The same node "image" can be used in several ways: > - copied to the local disk and booted from there (where the copying > could be done as a separate operation followed by a reboot or it can > be done from initrd) > - used over NFS-root > - used as a ramdisk, provided that the node "image" is small enough While memory follows the price-down capacity-up curve, we aren't quite to the point where holding a full OS distribution in memory is negligible. Most distributions (all the commercially interesting ones) are workstation-oriented, and the trade-off is "disk is under $1/GB, so we will install everything". It's foreseeable that holding an 8GB install image in memory will be trivial, but that will be a few years in the future, not today. And we will need better VM and PTE management to make it efficient. -- Donald Becker becker@scyld.com Penguin Computing / Scyld Software www.penguincomputing.com www.scyld.com Annapolis MD and San Francisco CA From landman at scalableinformatics.com Wed Oct 1 10:23:00 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <48E39782.4020006@ias.edu> References: <48E39782.4020006@ias.edu> Message-ID: <48E3B1F4.3000702@scalableinformatics.com> Prentice Bisbal wrote: > In the ongoing saga that is my new cluster, we were just told today that > Cisco is no longer manufacturing DDR IB cables, which we, uhh, need. > > Has DDR IB gone the way of the dodo bird and been supplanted by QDR? > > If so, why would anyone spec a brand new cluster with DDR? Hmmm.... Cisco isn't the only provider of IB cables. Last I understood, they buy theirs from others. Cisco does appear to be transitioning to *other* technologies. Again, they aren't the only IB provider out there (seems such a shame, gobbling up TopSpin and then effectively discarding them). Qlogic, Voltaire, Mellanox are happy to sell you stuff. You can even call some of the tier2+ players and they will help. Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From Bogdan.Costescu at iwr.uni-heidelberg.de Wed Oct 1 10:39:37 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: <48E2C6C1.60108@neuralbs.com> References: <48E2C6C1.60108@neuralbs.com> Message-ID: On Tue, 30 Sep 2008, Eric Thibodeau wrote: > This has given me much flexibility and a very fast path to upgrade > the nodes (LIVE!) since they would only need to be rebooted if I > changed the kernel. I can install/upgrade the node's environment by > simply chrooting into it and using the node's package manager and > utilities as if it were a regular system). Only the first is an advantage of using NFS-root; the second is shared by most methods that use a node "image". However random installations or modifications of configuration file within the chroot become very difficult to reproduce when you build the next node "image" - either scripting everything or using cfengine/puppet/etc. can save a lot of time in the long run, despite the initial effort to set up. > But I am in a special case where, if I break the cluster, I can fix > it quickly and I always have a backup copy of the boot "root" image > ready to switch to if my fiddling goes wrong. Why not keeping several "images" around and only point the nodes to mount the one considered current or "good" ? -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From prentice at ias.edu Wed Oct 1 10:49:43 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Advanced Clustering's Breakin In-Reply-To: References: <48E39AC1.7080001@ias.edu> Message-ID: <48E3B837.4020306@ias.edu> billycrook@gmail.com wrote: > On 2008-10-01, Prentice Bisbal wrote: > ... >> I sent e-mail to Advanced Clustering regarding these issues, but didn't >> get any response, so I"m hoping I have better luck here. > > Prentice, > > I'm the primary support contact at Advanced Clustering. Our Support > email address is Support@AdvancedClustering.com. I looked through our > ticket system, but didn't see a message from you there. We test > breakin against clusters we sell, but would like it to work on as much > hardware as possible. When hdhealth doesn't see hard drives, is > usually because of some raid hardware sheilding SMART data, or the > hard drive controller driver not being in the kernel yet. When its > intermittent, it could be cabeling, heat related, or the drives > actually failing. > > What hardware specifically are you testing? The motherboard model, > and any storage adapters involved would help. > > I will open a ticket momentarily from our helpdesk to prentice@ias.edu > to look at this issue off list and where it is accessible to all of > our support personnel until the issue is resolved. > > Thanks, > Billy Crook > > Advanced Clustering Customer Support > Support: 866.802.8222 x2 > Support: 913.643.0300 x2 > Fax: 913.378.9117 > Billy, I got your support e-mail off list, and replied already. We can continue this disucssion off-list. Thanks for the help. I sent my previous e-mail to some address on the web page for breakin. -- Prentice From prentice at ias.edu Wed Oct 1 10:59:05 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <48E3B1F4.3000702@scalableinformatics.com> References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com> Message-ID: <48E3BA69.2000405@ias.edu> Joe Landman wrote: > Prentice Bisbal wrote: >> In the ongoing saga that is my new cluster, we were just told today that >> Cisco is no longer manufacturing DDR IB cables, which we, uhh, need. >> >> Has DDR IB gone the way of the dodo bird and been supplanted by QDR? >> >> If so, why would anyone spec a brand new cluster with DDR? > > Hmmm.... Cisco isn't the only provider of IB cables. Last I understood, > they buy theirs from others. I'm sure you're right. It's my understanding that IB is a well-documented standard, so all IB cables should be created equally, but you know vendors, they'll say that their cables are more equal than others and refuse support if we're not using all Cisco kit. And what is the status of DDR? Are people still using it, or has it already been replaced by QDR in the marketplace? > > Cisco does appear to be transitioning to *other* technologies. Again, > they aren't the only IB provider out there (seems such a shame, gobbling > up TopSpin and then effectively discarding them). What *other* technolgies are you talking about QDR IB, or something other than IB altogether, like 10 Gb Ethernet, or something all new and proprietary? > > Qlogic, Voltaire, Mellanox are happy to sell you stuff. You can even > call some of the tier2+ players and they will help. > > Joe > > -- Prentice From jan.heichler at gmx.net Wed Oct 1 11:19:11 2008 From: jan.heichler at gmx.net (Jan Heichler) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <48E3BA69.2000405@ias.edu> References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com> <48E3BA69.2000405@ias.edu> Message-ID: <1848483940.20081001201911@gmx.net> Hallo Prentice, Mittwoch, 1. Oktober 2008, meintest Du: PB> And what is the status of DDR? Are people still using it, or has it PB> already been replaced by QDR in the marketplace? DDR is still the most important Infiniband in the market. DDR provides enough bandwidth for most applications at the moment because most applications suffer from latency - not limited bandwidth. QDR is more interesting for building spine networks. And QDR might get more important if we see more cores in typical compute nodes. >> Cisco does appear to be transitioning to *other* technologies. Again, >> they aren't the only IB provider out there (seems such a shame, gobbling >> up TopSpin and then effectively discarding them). PB> What *other* technolgies are you talking about QDR IB, or something PB> other than IB altogether, like 10 Gb Ethernet, or something all new and PB> proprietary? Cisco is dropping IB - probably to go for 10GE. It was a short time for Cisco offering IB solutions - bur probably they weren't very successful. Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081001/5967224f/attachment.html From diep at xs4all.nl Wed Oct 1 11:22:57 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Re: MOSIX2 In-Reply-To: <48E35A95.7020908@rri.sari.ac.uk> References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E1EDE3.2080205@rri.sari.ac.uk> <48E35A95.7020908@rri.sari.ac.uk> Message-ID: <61D42F6C-2115-40E3-A394-A531AD2B12DA@xs4all.nl> Well Tony, Things are pretty simple for using SSI at your beowulf cluster. The short summary: a) openmosix is dead b) open-ssi is still alive Simple tip: go for open-ssi. A bit longer explanation: a) openmosix is dead. I can confirm that even wikipedia has that. Openmosix was an Israeli project and therefore died when It's Israeli main developer left, for whatever reason. He has left a very clear statement that he no longer works on openmosix. What usually happens is that 1 or 2 enthusiasts then do an effort to support it. If that doesn't work for you, then consider it dead. Using kernels 2.4.x is not realistic for todays quadcores. That was the status in 2005, that is in 2008 still the status. In 2005 if i remember well i used kernel 2.6.7 for a quad opteron dual core. Using kernel 2.4.x versus 2.6.x numa gave at a dual opteron dual core a speedloss of 50% for my chessproggie. 50% is *a lot* to lose in speed. Supporting a thing like openmosix requires a lot more than 1 guy who in order to modify 3 bytes needs 1000 dollar. What you see a lot is that open source projects get hijacked by people who want to make cash out of it. In the world i come from, computerchess, i remember already since 1988 that this happens every year several times. The developers usually get really demotivated when they work for hundreds of hours at their 'money project', and then just make under a 1000 dollar; at the macdonalds you make more money. In short such projects usually die soon also, as there is no money for them into it. Most nerds are social seen total robots. b) open-ssi is there fore several distributions and actively supported by several developers. It is there, it works, it improves and it works for latest kernels also (yes also 2.6.x). In itself it would be GREAT if there is 1 open source project there, as that joins forces more. My hope is that open-ssi will work great for highend nic's also and slowly get to a phase that all features work great. OpenMosix nor openssi could migrate processes that work with shared memory, a nerd feature i like personally a lot. Just claiming that they use 'stolen' features like page migration is like claiming that linux stole multithreading from unix; it is a bad attempt to smear dirt just to earn a 1000 dollar. It is not a reason to not use it. It is a bad attempt to spit at these guys who donate time and their money without asking for payment. Vincent On Oct 1, 2008, at 1:10 PM, Tony Travis wrote: > Vincent Diepeveen wrote: >> I agree tony that paying for such crap is not very good idea. > > Hello, Vincent. > > I don't think MOSIX2 is crap! > > However, I don't like the idea of having to pay for 'updates'. > >> You might want to move to open-ssi in this case; the project is >> alive and there is in theory work getting >> performed on support for cards over infiniband as well. > > I have looked at OpenSSI, which uses the openMosix load-balancer, > but process migration is more coarsely grained than in openMosix. > Only the active pages of the user context of openMosix processes > are migrated. > > I've been looking at alternatives and I think Kerrighed looks very > promising but, in our hands, Kerrighed is very fragile: I've > mentioned on this list before that if one Kerrighed node goes down > you lose the entire cluster. We've been talking to Christine > Morin's group at INRIA and they tell us that the next release of > Kerrighed with be more robust: > > http://www.kerrighed.org > >> Most importantly is that you are gonna get more replies. > > Yes, thanks for yours :-) > >> Additionally the manner open-ssi implements shared memory is very >> transparant; in principle on each write it migrates a page >> to the node writing. >> Maybe the only big lack of open-ssi is its limited support so far >> for highend network cards. > > What bothers me about OpenSSI is that it's based on an open-sourced > version HP's (now Compaq) discontinued commercial product "non-stop > clusters for Unix". The OpenSSI project also came in for a lot of > criticism from the openMosix community for stealing ideas, so my > concerns about it might not be all that well founded ;-) > > The main reason I didn't use OpenSSI, previously, was that many > features had not been implemented fully and, like Kerrighed, it > wasn't really a viable option for a 'production' cluster even > though it was interesting as a research project. What makes me take > MOSIX2 seriously now is that it is a commercially supported > 'product' with all the same virtues (and most of the vices) of > openMosix. > > Tony. > -- > Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition > and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK > tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk > mailto:ajt@rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From tegner at nada.kth.se Wed Oct 1 12:09:39 2008 From: tegner at nada.kth.se (Jon Tegner) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: References: Message-ID: <48E3CAF3.3070800@nada.kth.se> There seem to be significant advantages using Scyld ClusterWare, I did try it (Scyld?) many years ago (when it was free?) and I was impressed then. However, when looking at penguincomputing.com I don't find any price quotes. It seems - unless I miss something - one has to fill in a rather lengthy form in order to get that information? In order to consider the "Scyld solution" I think it would be good to have at least an estimate of the price? Regards, /jon Donald Becker wrote: > On Wed, 1 Oct 2008, Bogdan Costescu wrote: > > >> On Tue, 30 Sep 2008, Donald Becker wrote: >> >>> Ahhh, your first flawed assumption. >>> You believe that the OS needs to be statically provisioned to the nodes. >>> That is incorrect. >>> >> Well, you also make the flawed assumption that the best technical >> solutions are always preferred. From my position I have seen many >> > ... > >> a solution like Scyld's limits the whole cluster to running one >> distribution (please correct me if I'm wrong), while a solution with >> node "images" allows mixing Linux distributions at will. >> > > That's correct. Our model is that a "cluster" is a single system -- and a > single install. > > That's for a good reason: To keep the simplicity and consistency of > managing a single installation, you pretty much can have... only a single > installation. > > There is quite a bit of flexibility. The system automatically detects the > hardware and loads the correct kernel modules. Nodes can be specialized, > including mounting different file systems and running different start-up > scripts. But the bottom line is that to make the assertion that remote > processes will run the same as local processes, they have to be running > pretty much the same system. > > If you are running different distributions on nodes, you discard many of > the opportunities of running a cluster. More importantly, it's much > more knowledge- and labor-intensive to maintain the cluster while > guaranteeing consistency. > > >>> The only times that it is asked to do something new (boot, accept a >>> new process) it's communicating with a fully installed, up-to-date >>> master node. It has, at least temporarily, complete access to a >>> reference install. >>> >> I think that this is another assumption that holds true for the Scyld >> system, but there are situations where this is not true. >> > > Yes, there are scenarios where you want a different model. But "connected > during important events" is true for most clusters. We discard the > ability for a node to boot and run independently in order to get the > advantages of zero-install, zero-config consistent compute nodes. > > >>> If you design a cluster system that installs on a local disk, it's >>> very difficult to adapt it to diskless blades. If you design a >>> system that is as efficient without disks, it's trivial to >>> optionally mount disks for caching, temporary files or application >>> I/O. >>> >> If you design a system that is flexible enough to allow you to use >> either diskless or diskfull installs, what do you have to loose ? >> > > In theory that sounds good. But historically changing disk-based > installations to work on diskless machines has been very difficult, and > the results unsatisfactory. Disk-based installations want to do selective > installation based on the hardware present, and write/modify many links > and configuration files on installation -- many more than they "need" to. > > >> The same node "image" can be used in several ways: >> - copied to the local disk and booted from there (where the copying >> could be done as a separate operation followed by a reboot or it can >> be done from initrd) >> - used over NFS-root >> - used as a ramdisk, provided that the node "image" is small enough >> > > While memory follows the price-down capacity-up curve, we aren't quite to > the point where holding a full OS distribution in memory is negligible. > Most distributions (all the commercially interesting ones) are > workstation-oriented, and the trade-off is "disk is under $1/GB, so we > will install everything". It's foreseeable that holding an 8GB install > image in memory will be trivial, but that will be a few years in the > future, not today. And we will need better VM and PTE management to make > it efficient. > > > From lindahl at pbm.com Wed Oct 1 13:03:38 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <48E3BA69.2000405@ias.edu> References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com> <48E3BA69.2000405@ias.edu> Message-ID: <20081001200338.GB32180@bx9.net> On Wed, Oct 01, 2008 at 01:59:05PM -0400, Prentice Bisbal wrote: > I'm sure you're right. It's my understanding that IB is a > well-documented standard, so all IB cables should be created equally, > but you know vendors, they'll say that their cables are more equal than > others and refuse support if we're not using all Cisco kit. Then buy from an IB vendor that isn't silly like that. QLogic and Voltaire are choices. Is Cisco still re-selling QLogic switches? Joe wrote: > > Cisco does appear to be transitioning to *other* technologies. Again, > > they aren't the only IB provider out there (seems such a shame, gobbling > > up TopSpin and then effectively discarding them). Apparently the main attraction of TopSpin was their virtualization software. Cisco, of course, supports all the major technologies, so it can be hard to tell if they really are transitioning to something else. At a minimum Cisco will want to sell gateways for all non-Ethernet technologies. After TopSpin was bought, Cisco continued to sell gateways developed by TopSpin, but sold rebranded QLogic DDR switches. -- greg From niftyompi at niftyegg.com Wed Oct 1 13:18:04 2008 From: niftyompi at niftyegg.com (NiftyOMPI Mitch) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <1848483940.20081001201911@gmx.net> References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com> <48E3BA69.2000405@ias.edu> <1848483940.20081001201911@gmx.net> Message-ID: <88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com> On Wed, Oct 1, 2008 at 11:19 AM, Jan Heichler wrote: > Hallo Prentice, > > Mittwoch, 1. Oktober 2008, meintest Du: > > PB> And what is the status of DDR? Are people still using it, or has it > > PB> already been replaced by QDR in the marketplace? > > > DDR is still the most important Infiniband in the market. DDR provides > enough bandwidth for most applications at the moment because most > applications suffer from latency - not limited bandwidth. QDR is more > interesting for building spine networks. And QDR might get more important if > we see more cores in typical compute nodes. > > > >> Cisco does appear to be transitioning to *other* technologies. Again, > > >> they aren't the only IB provider out there (seems such a shame, gobbling > > >> up TopSpin and then effectively discarding them). > > > PB> What *other* technolgies are you talking about QDR IB, or something > > PB> other than IB altogether, like 10 Gb Ethernet, or something all new and > > PB> proprietary? > > > Cisco is dropping IB - probably to go for 10GE. It was a short time for > Cisco offering IB solutions - bur probably they weren't very successful. > Yes folks tell me that Cisco is backing away from the IB business -- all the parts on the Cisco price book that I know of were rebranded products and I suspect that some of their providers were too happy to sell directly to customers at a discount sometimes below the OEM price. Perhaps more importantly IB is not a router and management rich layer. i.e. It does not facilitate all the routing and management value add that Cisco focuses on for their bread and butter. They are however well involved and commited in Open MPI which is agnostic to the transport layer beyond the want to go fast part. I know that QLogic has well specified and tested cables for sale. Call the Qlogic "King of Prussia, Penn" office and tell them what you want. Gore and Leoni make excellent cables as do others. QDR is interesting... in all likelyhood the QDR game will be optical for any link further away than a single rack. Once IB goes optical there will be a lot of reason to install IB in machine rooms and campus sites that are just out of reach today. I should say that copper QDR is hard but not impossible. IMO optical has advantages and once it becomes easy for the end user (site) to install optical it should change the game. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081001/3613d7c6/attachment.html From alscheinine at tuffmail.us Wed Oct 1 14:07:39 2008 From: alscheinine at tuffmail.us (Alan Louis Scheinine) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com> References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com> <48E3BA69.2000405@ias.edu> <1848483940.20081001201911@gmx.net> <88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com> Message-ID: <48E3E69B.4030202@tuffmail.us> NiftyOMPI Mitch wrote > QDR is interesting... in all likelyhood the > QDR game will be optical for any link further away than a single rack. > Once IB goes optical there will be a lot of reason to install IB in > machine rooms and campus sites that are just out of reach today. When will IB go optical at a reasonable price? Perhaps you are not an expert but if you happen to have any pointers where we can learn more about the time frame it would be useful. Just a few days ago heard colleagues trying to figure-out the solution to connecting file system hardware just a bit too far from a cluster in another building. Ethernet and NFS would work but latency is a problem. Best regards, Alan -- Alan Scheinine 5010 Mancuso Lane, Apt. 621 Baton Rouge, LA 70809 Email: alscheinine@tuffmail.us Office phone: 225 578 0294 Mobile phone USA: 225 288 4176 [+1 225 288 4176] From kyron at neuralbs.com Wed Oct 1 14:27:11 2008 From: kyron at neuralbs.com (Eric Thibodeau) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: References: <48E2C6C1.60108@neuralbs.com> Message-ID: <48E3EB2F.2070102@neuralbs.com> Bogdan Costescu wrote: > On Tue, 30 Sep 2008, Eric Thibodeau wrote: > >> This has given me much flexibility and a very fast path to upgrade >> the nodes (LIVE!) since they would only need to be rebooted if I >> changed the kernel. I can install/upgrade the node's environment by >> simply chrooting into it and using the node's package manager and >> utilities as if it were a regular system). > > Only the first is an advantage of using NFS-root; the second is shared > by most methods that use a node "image". More or less, NFS-root changes are "propagated" instantly, most other approaches require a re-sync. Another way to see this, the NFS root approach only does changes on the head node and changed files don't need to be propagated and are accessed on a as-needed basis, this might have significant impacts on large deployments....not that I suggest that they use this approach ;) > However random installations or modifications of configuration file > within the chroot become very difficult to reproduce when you build > the next node "image" Document document document...which no one does...but document. > - either scripting everything or using cfengine/puppet/etc. can save a > lot of time in the long run, despite the initial effort to set up. I'll take your word for it that they have a version tracking mechanism. >> But I am in a special case where, if I break the cluster, I can fix >> it quickly and I always have a backup copy of the boot "root" image >> ready to switch to if my fiddling goes wrong. > Why not keeping several "images" around and only point the nodes to > mount the one considered current or "good" ? Well, that's what I meant... probably not clearly. ;) Eric From niftyompi at niftyegg.com Wed Oct 1 14:33:42 2008 From: niftyompi at niftyegg.com (NiftyOMPI Mitch) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] precise synchronization of system clocks In-Reply-To: References: <20081001020421.GA8126@compegg.wr.niftyegg.com> Message-ID: <88815dc10810011433m1fa59641g417643b15726a770@mail.gmail.com> On Wed, Oct 1, 2008 at 7:18 AM, Robert G. Brown wrote: > On Tue, 30 Sep 2008, Nifty niftyompi Mitch wrote: > ..... > One site service of interest is AC power. A modern processor sitting > in an idle state that then starts a well optimized loop will jump from > a couple of watts to 100 watts in as many clocks as the set of pipelines > is deep behind the instruction decode and instruction cache fill. A 1000 > processor (4000 cores) might jump from 4000 watts to 100000 watts in the > blink of an eye (err did the lights blink). Buffer that dI/dT through > the PS and it is less but still interesting on the mains which are > synchronized. > Interesting. I never have seen the lights blink although I don't run > synchronous computations. One wonders if the power supply capacitors > (which should be quite large, I would think) don't soak up the > transient, though, even on very large clusters. Also, I think that the > power differential is smaller than you are allowing for -- I don't think > most idle processors draw "no" power... > The dI/dT for processors can be quite high. AMD Phenom? X4 Quad-Core is listed as a 140 watt part (thermal) it is unlikely that all 450 million transistors are active in an idle loop. Tom's Hardware list the idle power at 21 watts. The speed at which a modern processor can go from idle to full power is astonishing. The local on board power supply regulation must respond very quickly. The delta from 21 to 140 fit inside one half cycle of a 50/60 Hz AC mains service. So inside of one AC cycle the part can move from 21 to 140.... which is large when multiplied by a 1000 node dual socket cluster. I do know of clusters and labs of workstations that power on hosts and disks in sequence to limit the startup power surge. Lots of us have been at it long enough to know that induction motors like elevators, refrigeration compressors and even vacuums can hit the mains hard enough to trigger errors. My home vacuum does dim the lights a little bit. In normal practice I doubt that this is an issue but synchronization in the extreme is interesting in its details and side effects. -- NiftyOMPI T o m M i t c h e l l -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081001/e9e6f732/attachment.html From andrew at moonet.co.uk Wed Oct 1 15:22:08 2008 From: andrew at moonet.co.uk (andrew holway) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <48E39782.4020006@ias.edu> References: <48E39782.4020006@ias.edu> Message-ID: > Has DDR IB gone the way of the dodo bird and been supplanted by QDR? I don't think front side busses are fast enough for QDR yet. From rpatienc at cisco.com Wed Oct 1 16:58:51 2008 From: rpatienc at cisco.com (Rob Patience) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <20081001200338.GB32180@bx9.net> References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com> <48E3BA69.2000405@ias.edu> <20081001200338.GB32180@bx9.net> Message-ID: <1F19606C-6310-40F9-B2F3-AC32671D620E@cisco.com> Only the larger switches were from QLogic (144 and 288 ports) all other switches were made by TopSpin/Cisco directly. Cisco is not leaving the HPC space only IB. Btw: you can still buy cables from Cisco. But as mentioned, all cables come from 3rd party vendors. ~rob On Oct 1, 2008, at 1:03 PM, Greg Lindahl wrote: > On Wed, Oct 01, 2008 at 01:59:05PM -0400, Prentice Bisbal wrote: > >> I'm sure you're right. It's my understanding that IB is a >> well-documented standard, so all IB cables should be created equally, >> but you know vendors, they'll say that their cables are more equal >> than >> others and refuse support if we're not using all Cisco kit. > > Then buy from an IB vendor that isn't silly like that. QLogic and > Voltaire are choices. Is Cisco still re-selling QLogic switches? > > Joe wrote: > >>> Cisco does appear to be transitioning to *other* technologies. >>> Again, >>> they aren't the only IB provider out there (seems such a shame, >>> gobbling >>> up TopSpin and then effectively discarding them). > > Apparently the main attraction of TopSpin was their virtualization > software. Cisco, of course, supports all the major technologies, so it > can be hard to tell if they really are transitioning to something > else. At a minimum Cisco will want to sell gateways for all > non-Ethernet technologies. After TopSpin was bought, Cisco continued > to sell gateways developed by TopSpin, but sold rebranded QLogic DDR > switches. > > -- greg > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Oct 1 21:14:06 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] precise synchronization of system clocks In-Reply-To: <88815dc10810011433m1fa59641g417643b15726a770@mail.gmail.com> References: <20081001020421.GA8126@compegg.wr.niftyegg.com> <88815dc10810011433m1fa59641g417643b15726a770@mail.gmail.com> Message-ID: On Wed, 1 Oct 2008, NiftyOMPI Mitch wrote: > The dI/dT for processors can be quite high. > AMD Phenom??? X4 Quad-Core is listed as a 140 watt part (thermal) > it is unlikely that all 450 million transistors are active in an idle > loop. Tom's Hardware list the idle power at 21 watts. The speed at > which a modern processor can go from idle to full power is astonishing. > The local on board power supply regulation must respond very quickly. The > delta > from 21 to 140 fit inside one half cycle of a 50/60 Hz AC mains service. So > inside > of one AC cycle the part can move from 21 to 140.... which is large when > multiplied by a 1000 node dual socket cluster. I do know of clusters and > labs of workstations that power on hosts and disks in sequence to limit the > startup power surge. Lots of us have been at it long enough to know that > induction motors like elevators, refrigeration compressors and even vacuums > can hit the mains > hard enough to trigger errors. My home vacuum does dim the lights a little > bit. I understand inductive surge when powering up, I understand in detail browning out a primary power transformer, but I think those are different issues and irrelevant here. So far, using my trusty Kill-a-Watt on real world nodes, I haven't seen more than a 30% differential draw loaded to unloaded. Large parts of the CPU require power at all times to function. Memory, for example, both on and offboard. Nearly everything inside a computer has a nontrivial idle draw, plus (sure) peak draw when it or one of its subsystems are in use. Exceptions are modern laptops -- with variable speed clocks, they draw much less idling than they do at speed, in part because power (idle or otherwise) IS very nearly proportional to CPU clock in at least parts of the system. And I don't really know how the latest designs do in this regard -- but there is a tendency to design the bleeding edge PERFORMANCE CPUs to work at "constant heat", as a major rate limiting factor in CPU design is getting rid of heat from the package. It's one reason they don't just crank the clock to infinity and beyond -- not the only one, but a major one. Multicores, of course, may function like hybrid cars, and somehow run more nearly idle when they are idle. But I'd have to hear from someone who slapped a KaW on an actual system and clocked it from idle (solidly post-boot, running the OS, at idle "equilibrium") to loaded (running flat out on e.g. a benchmark suite that loads all cores and/or the memory etc.). Has anyone actually done this and observed (say) a 2 or 3 to 1 increase in power draw loaded to idle? 50W idle to 200W loaded in 1 second? 150W idle to 200W loaded is more like what I've seen... > In normal practice I doubt that this is an issue but synchronization in the > extreme is interesting in its details and side effects. I completely agree with this, both parts. Although if one IS bumping from 50->200W "instantly" on not even an entire cluster but just all the nodes on a single circuit, that's popping over a KW on a 20A line -- ballpark where one MIGHT see something inductive (although as I said, probably nothing that the power supply capacitor(s) cannot buffer, although I'm too tired to compute the number of joules (watt-seconds) one can probably deliver and what RC probably is, etc). Popping multiple (as in 10+) KW in less than a 60 Hz cycle would very likely be hard on the primary, no doubt about it. rgb -- NiftyOMPI T o m M i t c h e l l -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From hearnsj at googlemail.com Thu Oct 2 02:05:30 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Linux Magazine - What He Said Message-ID: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com> I just read Douglas Eadline's article on Linux Magazine, entitled "What He Said" http://www.linux-mag.com/id/7087 Very thought provoking article, and took me back to thinking about genetic algorithms, a subject I flirted with 20 years ago. I didn't find it worthwhile on a Sparc1 system with a whopping 2 Mbytes of RAM. I guess I should encourage responses to be made on the Linux Mag site. John Hearns -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081002/bac57c28/attachment.html From ajt at rri.sari.ac.uk Thu Oct 2 02:25:33 2008 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Re: MOSIX2 In-Reply-To: <61D42F6C-2115-40E3-A394-A531AD2B12DA@xs4all.nl> References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E1EDE3.2080205@rri.sari.ac.uk> <48E35A95.7020908@rri.sari.ac.uk> <61D42F6C-2115-40E3-A394-A531AD2B12DA@xs4all.nl> Message-ID: <48E4938D.202@rri.sari.ac.uk> Vincent Diepeveen wrote: > [...] > Supporting a thing like openmosix requires a lot more than 1 guy who in > order to modify 3 bytes needs 1000 dollar. Hello, Vincent. I've been involved in the openMosix project from the start, and I still use it. My comment about the $1,000 for 'updates' concerned the cost an 'academic' update and support subscription to MOSIX2, which has nothing to do with openMosix. I think that is quite a reasonable cost, actually, and I dodn't criticise people who want to pay for support. It's not the cost that bothers me most, but the restrictions of the MOSIX2 licence. > [...] > In itself it would be GREAT if there is 1 open source project there, as > that joins forces more. My hope is that open-ssi > will work great for highend nic's also and slowly get to a phase that > all features work great. OK, maybe I should have another look at the most recent OpenSSI... > OpenMosix nor openssi could migrate processes that work with shared > memory, a nerd feature i like personally a lot. Actually there is a version of openMosix that can migrate programs using shared memory. Unfortunately, when I tried it, processes would migrate away from the 'home' node but wouldn't migrate back! This is a problem for openMosix, because processes have to migrate back to the 'home' node to make system calls. One of my (small) contributions to openMosix was to patch it to avoid migrations just to read the time. This is slightly relevant to another thread on the Beowulf list, because reading the time locally on a node to avoid the process migration overhead requires that the nodes all run NTP otherwise you get problems with clock skew... > Just claiming that they use 'stolen' features like page migration is > like claiming that linux stole multithreading from unix; > it is a bad attempt to smear dirt just to earn a 1000 dollar. > > It is not a reason to not use it. It is a bad attempt to spit at these > guys who donate time and their money without asking for payment. Sadly, there was a lot of bad feeling between the openMosix and OpenSSI communities and I should not have tried to make light of it. In fact it is a compliment to openMosix that the load-balancer algorithm was used in OpenSSI, not an insult, and BTW I'm one of those guys who donated their time without asking for payment. I did put a wry smile after my comment ;-) Tony. -- Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk mailto:ajt@rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt From Bogdan.Costescu at iwr.uni-heidelberg.de Thu Oct 2 05:03:30 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: References: Message-ID: On Wed, 1 Oct 2008, Donald Becker wrote: > That's correct. Our model is that a "cluster" is a single system -- > and a single install. That's the idea that I've also started with, almost 10 years ago ;-) Not using Beo*/bproc, but NFS-root which allowed a single install in the node "image" to be used on all nodes - although you'd probably call this 2 system installs (the master node itself and the node "image"). But over the course of the years I have changed my mind... > If you are running different distributions on nodes, you discard > many of the opportunities of running a cluster. More importantly, > it's much more knowledge- and labor-intensive to maintain the > cluster while guaranteeing consistency. It indeed requires more work, however in some cases it cannot be avoided. From my own experience: a quantum chemistry program was distributed some 5 years ago as a binary statically compiled on RH9 or RHEL3 (kernel 2.4 based) with MPICH included. This meant that when I wanted to switch to running a 2.6 kernel this program could not run anymore so some of the nodes had to be kept to an older distribution until a newer program version could be obtained (that took about a year); it also meant that whenever there were discussions about using higher performance interconnects than GigE, this software's users were insisting on buying more nodes rather than a faster interconnect. This situation has caused both technical and administrative issues and the possibility of running different distributions has solved all of them easily. Having the possibility to run several distributions side-by-side requires spending some effort in organizing the other installed software, normally shared through NFS or a parallel FS to the nodes. But once you make the jump from 1 to 2, you might as well make it from 1 to many. This leads me to observe that we have non-similar points of view: you are a maker of a cluster-oriented distribution, trying to promote it and its underlying ideas (which are fine ideas, no question about that :-)), and sure that it works because it was bought and used successfully. I, on the other hand, have to find solutions to keep the scientists productive (whatever productive means ;-)) and to keep them as far as possible from the system details so that they can concentrate on their work. So it's not surprising that we come to different conclusions - at least they sustain an interesting discussion :-) I would be interested to hear Mark Hahn's opinion on this, as from how he presented himself to this list it seemed to me that he is in a very similar position to mine: supporting a variety of users with a variety of needs. But others should not feel left out, write your opinions as well ;-) > Most distributions (all the commercially interesting ones) are > workstation-oriented I don't really agree with this statement (looking at RHEL and SLES), but anyone who installs a workstation-oriented distribution on a cluster node gets what (s)he pays for :-) I have seen very recently (identity hidden to protect the guilty ;-)) such a node "image" which contained OpenOffice - to be fair, it was used via NFS-root so it wasn't wasting node memory, only master disk space... -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From Bogdan.Costescu at iwr.uni-heidelberg.de Thu Oct 2 05:18:30 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: <48E3EB2F.2070102@neuralbs.com> References: <48E2C6C1.60108@neuralbs.com> <48E3EB2F.2070102@neuralbs.com> Message-ID: On Wed, 1 Oct 2008, Eric Thibodeau wrote: > the NFS root approach only does changes on the head node and changed > files don't need to be propagated and are accessed on a as-needed > basis, this might have significant impacts on large deployments NFS-root doesn't scale too well, the implementation of NFS in Linux is quite chatty. > I'll take your word for it that they have a version tracking mechanism. Take my word that if you're going into larger installations with the least amount of non-homogeneity you'll want to at least read about, if not use, such mechanisms :-) -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From hearnsj at googlemail.com Thu Oct 2 05:25:17 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: References: Message-ID: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> 2008/10/1 Donald Becker > > > It's foreseeable that holding an 8GB install > image in memory will be trivial, but that will be a few years in the > future, not today. And we will need better VM and PTE management to make > it efficient. > > > Hmmmm.... can I forsee Puppy Linux HPC Edition ???? http://www.puppylinux.org/ Being half serious here, is it worth trying to get one of these slimmed-down distros to the state where it will run an HPC job? Oh, and in addition to a barebones install for our contemplative cluster, they have a onebones install which cuts out the GUI (cue more dog puns). http://www.puppylinux.com/pfs/ looks interesting... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081002/7727b883/attachment.html From smulcahy at aplpi.com Thu Oct 2 05:56:41 2008 From: smulcahy at aplpi.com (stephen mulcahy) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> References: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> Message-ID: <48E4C509.3000407@aplpi.com> John Hearns wrote: > Hmmmm.... can I forsee Puppy Linux HPC Edition ???? > http://www.puppylinux.org/ > > > Being half serious here, is it worth trying to get one of these > slimmed-down distros to the state where it will run an HPC job? > Oh, and in addition to a barebones install for our contemplative > cluster, they have a onebones install which cuts out the GUI (cue more > dog puns). Is there that much of a difference between Puppy and a minimal Debian where you install only the "standard server" task (I understand Puppy is a Debian derivative, apologies if this is incorrect)? Or am I swinging my Debian swiss-army chainsaw indiscriminately here? -stephen -- Stephen Mulcahy Applepie Solutions Ltd. http://www.aplpi.com Registered in Ireland, no. 289353 (5 Woodlands Avenue, Renmore, Galway) From diep at xs4all.nl Thu Oct 2 06:28:38 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Linux Magazine - What He Said In-Reply-To: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com> References: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com> Message-ID: To comment a bit on the article: >What if a high level programing description language was developed. Note I did not say programming language. >This description language would allow you to ?describe? what you needed to do and not how to do it (as >discussed before). This draft description would then be presented to an AI based clarifier which would examine >the description, look for inconsistencies or missing information and work with the programmer to create a formal >description of the problem. At that point the description is turned over to a really smart compiler that could target >a particular hardware platform and produce the needed optimized binaries. >Perhaps a GA could be thrown in to help optimize everything. The typical average PHD way of approaching problems they need to solve: Suppose you need to solve a huge problem A at a major parallel machine. We throw the problem A into a blackbox B giving output A'. Now our fantastic new brilliant method/algorithm comes into place that works on A' what our paper is about, and we solve the problem with that. I see that scenario a lot in all kind of different variations. A great example of it is: We invent algorithm f( x ) Now in our program we have if( magicfunction(x) && f(x) ) then printf("problem solved. jippie i got my research done.\n"); They test it at 1 case C and shout victory. typical AI way of doing things. There is so little AI dudes who do more than fill 50 pages of paper a year with ideas that keep untested, and because of never testing their understanding of the problem never advances and the next guy still shows up with the same untested solution. So the above guy who actually is *doing* something and testing something, already gets cheered for (with some reasons). But of course magicfunction(x) only works for their case C. There is no statistic significance. The number of AI guys who test their algorithm/method at state of the art software, be it selfwritten or from someone else, AND test is statistical significant without using some flawed testset where magicfunction(x) is always true, those guys you can really count on 2 hands the past 30 years. In parallel AI it is even worse in fact. There is for example just 2 chessprograms that scale well (so without losing first factor 50 to the parallel search frame) at supercomputers. There is some fuzz now about go programs using UCT/Monte Carlo and a few other algorithms combined; but these random algorithms miss such simple tactics, which for their computing power should be easy to not miss, that they really should think out a better way to search there parallel. Each AI solution that is not embarrassingly parallel is so difficult to parallellize well, so not losing a factor 50, that it is just real hard to make generic frameworks that work real efficient. Such a framework would pose a solution for only 1 program; as soon as you improve 1 year later the program with a new enhancement the entire parallel framework might need to get rewritten from scratch again, to not again lose some factors in speed in scaling. The commercial way most AI guys look to the parallel problem is therefore real simple: "can i get faster than i am at a PC, as it is a government machine anyway, i didn't pay for efficient usage of it, i just want to be faster than my home PC without too much effort". I'm not gonna fight that lemma. However a generic framework that works like that is not gonna get used of course. It just speeds you up so much to make a custom parallel solution, that everyone is doing it. Besides, the hardware is that expensive, that it's worth doing it. >This process sounds like it would take a lot of computing resources. Guess what? We have that. >Why not throw a cluster at this problem. Maybe it would take a week to create a binary, >but it would be cluster time and not your time. There would be no edit/make/run cycle >because the description tells the compiler what the program has to do. The minutia >(or opportunities for bugs) of programming whether it be serial or parallel would be >handled by the compiler. Talk about a killer application. On Oct 2, 2008, at 11:05 AM, John Hearns wrote: > I just read Douglas Eadline's article on Linux Magazine, entitled > "What He Said" > http://www.linux-mag.com/id/7087 > > Very thought provoking article, and took me back to thinking about > genetic algorithms, > a subject I flirted with 20 years ago. I didn't find it worthwhile > on a Sparc1 system with a whopping 2 Mbytes of RAM. > > I guess I should encourage responses to be made on the Linux Mag site. > > John Hearns > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Oct 2 06:37:58 2008 From: james.p.lux at jpl.nasa.gov (Lux, James P) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] precise synchronization of system clocks In-Reply-To: Message-ID: Rgb wrote: > > I understand inductive surge when powering up, I understand in detail > browning out a primary power transformer, but I think those are > different issues and irrelevant here. Inductive surge -> magnetizing current in large iron core inductors (depends on where you are in line frequency cycle at "switch on") Sag from overload -> impedance (both R and L) in transformer and wires from transformer to load. 2% voltage drop is the NEC guideline for "in premises wires".. From panel to load. The voltage at the utility entrance could probably be +/- 5% at any given time. > > So far, using my trusty Kill-a-Watt on real world nodes, I haven't seen > more than a 30% differential draw loaded to unloaded. Large parts of > the CPU require power at all times to function. Memory, for example, > both on and offboard. Nearly everything inside a computer has a > nontrivial idle draw, plus (sure) peak draw when it or one of its > subsystems are in use. Very much true. DRAM needs refresh, for instance. > > Exceptions are modern laptops -- with variable speed clocks, they draw > much less idling than they do at speed, in part because power (idle or > otherwise) IS very nearly proportional to CPU clock in at least parts of > the system. And I don't really know how the latest designs do in this > > Multicores, of course, may function like hybrid cars, and somehow run > more nearly idle when they are idle. But I'd have to hear from someone > who slapped a KaW on an actual system and clocked it from idle (solidly > post-boot, running the OS, at idle "equilibrium") to loaded (running > flat out on e.g. a benchmark suite that loads all cores and/or the > memory etc.). Has anyone actually done this and observed (say) a 2 or 3 > to 1 increase in power draw loaded to idle? 50W idle to 200W loaded in > 1 second? 150W idle to 200W loaded is more like what I've seen... Don't forget that the power supply efficiency drops dramatically when DC load drops, too. They don't spend a penny more on sophisticated design than required to get that "energy star" rating, and that has more to do with having a good "low power hibernate" mode than good efficiency at 25% load. > >> In normal practice I doubt that this is an issue but synchronization in the >> extreme is interesting in its details and side effects. > > I completely agree with this, both parts. Although if one IS bumping > from 50->200W "instantly" on not even an entire cluster but just all the > nodes on a single circuit, that's popping over a KW on a 20A line -- > ballpark where one MIGHT see something inductive (although as I said, > probably nothing that the power supply capacitor(s) cannot buffer, > although I'm too tired to compute the number of joules (watt-seconds) > one can probably deliver and what RC probably is, etc). Popping > multiple (as in 10+) KW in less than a 60 Hz cycle would very likely be > hard on the primary, no doubt about it. If one considers that a single wire is about 1 uH/meter (typical electrical wiring will be much less, because it's a pair, with currents flowing opposite directions), the series L might be a few tens of uH. At, say, 20 A, there's just not much energy stored there. From rgb at phy.duke.edu Thu Oct 2 07:41:24 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] precise synchronization of system clocks In-Reply-To: References: Message-ID: On Thu, 2 Oct 2008, Lux, James P wrote: > If one considers that a single wire is about 1 uH/meter (typical electrical > wiring will be much less, because it's a pair, with currents flowing > opposite directions), the series L might be a few tens of uH. At, say, 20 > A, there's just not much energy stored there. I was thinking in terms of load on transformers, but yeah, I forgot (again) that switching power supplies ain't got no transformers. Voltage regulators and MAYBE UPS have transformers, but they also have bloody damn big capacitors. I'm just not used to a transformer-free world. Back in the very old days we had power problems in our server room (that I think might have been connected to people using really big physics apparatus in the building) and we bought a honker power conditioner to run a subset of our systems. If one set up a monitor within two meters of the sucker, the CRT fuzzed and distorted -- the rapidly varying field was strong enough to deflect electrons at two meters. We kept it far far away from our backup tapes...;-) rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From jamesb at loreland.org Thu Oct 2 08:37:50 2008 From: jamesb at loreland.org (James Braid) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: References: <48E2C6C1.60108@neuralbs.com> <48E3EB2F.2070102@neuralbs.com> Message-ID: <25446eb90810020837v6d8e91bdt6dc36a634dca18b5@mail.gmail.com> 2008/10/2 Bogdan Costescu : > On Wed, 1 Oct 2008, Eric Thibodeau wrote: > >> the NFS root approach only does changes on the head node and changed files >> don't need to be propagated and are accessed on a as-needed basis, this >> might have significant impacts on large deployments > > NFS-root doesn't scale too well, the implementation of NFS in Linux is quite > chatty. It's scaled great in our experience. We run 1000+ machines off NFS root, running lots of large third-party applications from there as well. The NFS servers are just a pair of Linux servers, nothing fancy. From peter.st.john at gmail.com Thu Oct 2 09:41:13 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:50 2009 Subject: [Beowulf] Linux Magazine - What He Said In-Reply-To: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com> References: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com> Message-ID: John, After I first thought up my nutty GA scheme, I was then astonished by the John Holland Scientific American artifcle. I was aghast that anyone could even imagine thinking along those lines with 1960's hardware, as he had done. I got my first working version done on a 386 (daughtercard on a 286 motherboard) with one and a half MB. Peter On 10/2/08, John Hearns wrote: > > I just read Douglas Eadline's article on Linux Magazine, entitled "What He > Said" > http://www.linux-mag.com/id/7087 > > Very thought provoking article, and took me back to thinking about genetic > algorithms, > a subject I flirted with 20 years ago. I didn't find it worthwhile on a > Sparc1 system with a whopping 2 Mbytes of RAM. > > I guess I should encourage responses to be made on the Linux Mag site. > > John Hearns > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081002/97a131d6/attachment.html From hearnsj at googlemail.com Thu Oct 2 09:54:13 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Linux Magazine - What He Said In-Reply-To: References: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com> Message-ID: <9f8092cc0810020954mca589feyd27bdea53d3b7b7f@mail.gmail.com> 2008/10/2 Peter St. John > John, > After I first thought up my nutty GA scheme, I was then astonished by the > John Holland Scientific American artifcle. I was aghast that anyone could > even imagine thinking along those lines with 1960's hardware, as he had > done. > > I believe I read the same article. In fact, the first algorithm I tried was simulated annealing (I know this does not equal a GA). It swapped so horrendously that I was discouraged, and the thing took all night to run even for a simple 2D case (I was working on radiation therapy planning). I guess I should have been smarter and worked out how to do it within the constraints of RAM that I had. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081002/c9edb986/attachment.html From peter.st.john at gmail.com Thu Oct 2 11:06:55 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Linux Magazine - What He Said In-Reply-To: <9f8092cc0810020954mca589feyd27bdea53d3b7b7f@mail.gmail.com> References: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com> <9f8092cc0810020954mca589feyd27bdea53d3b7b7f@mail.gmail.com> Message-ID: John, When I tried to ressurect my thing a couple years ago, I realized my original code was all wrong in trading time for space (plenty of time on the 386, then SunOS servers; not enough space, but new machine had plenty of unused RAM). I thought some about redesigning to reverse the trade-off, which would be helpful, but I'm sure it would not just be easier, but more effective, to run it on a cluster (many nodes, not much ram per node needed, and any nonzero amount of communication sufficient, but more can be usefull). Peter On 10/2/08, John Hearns wrote: > > > > 2008/10/2 Peter St. John > >> John, >> After I first thought up my nutty GA scheme, I was then astonished by the >> John Holland Scientific American artifcle. I was aghast that anyone could >> even imagine thinking along those lines with 1960's hardware, as he had >> done. >> >> I believe I read the same article. > In fact, the first algorithm I tried was simulated annealing (I know this > does not equal a GA). It swapped so horrendously that I was discouraged, and > the thing took all night to run even for a simple 2D case (I was working on > radiation therapy planning). > I guess I should have been smarter and worked out how to do it within the > constraints of RAM that I had. > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081002/9661c9b8/attachment.html From gdjacobs at gmail.com Thu Oct 2 11:42:58 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: <48E4C509.3000407@aplpi.com> References: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> <48E4C509.3000407@aplpi.com> Message-ID: <48E51632.8090001@gmail.com> stephen mulcahy wrote: > John Hearns wrote: >> Hmmmm.... can I forsee Puppy Linux HPC Edition ???? >> http://www.puppylinux.org/ >> >> >> Being half serious here, is it worth trying to get one of these >> slimmed-down distros to the state where it will run an HPC job? >> Oh, and in addition to a barebones install for our contemplative >> cluster, they have a onebones install which cuts out the GUI (cue more >> dog puns). > > Is there that much of a difference between Puppy and a minimal Debian > where you install only the "standard server" task (I understand Puppy is > a Debian derivative, apologies if this is incorrect)? > > Or am I swinging my Debian swiss-army chainsaw indiscriminately here? > > -stephen > Damn Small (DSL) is, Puppy is not. I believe Puppy is it's own beast. This is an interesting topic, though. How much difference does shrinking the size of a kernel build make in improving the performance of a tightly coupled lockstep algorithm scaled to hundreds, or thousands of nodes? Anything in the literature which is directly comparable, not just the ASCI Q paper? -- Geoffrey D. Jacobs From xclski at yahoo.com Thu Oct 2 12:10:40 2008 From: xclski at yahoo.com (Ellis Wilson) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Linux Magazine - What He Said Message-ID: <217759.53687.qm@web37906.mail.mud.yahoo.com> In the article: "What if a high level programing description language was developed. Note I did not say programming language. This description language would allow you to ?describe? what you needed to do and not how to do it (as discussed before)." I would ask then, how does one "describe what you need to do"? As a brief (and heinously simplistic) example, let us say that we wanted to replace a specific string in a long text file with another string. In assembler, this would be a heinous and explicit long process where we tell it exactly what we are doing. One could even argue at the C level we have a fair amount of control, but once we hit fairly high-level programming languages one merely can say text.replaceAll(oldstring,newstring); and you are done. You have told the program what you want done, not how to do it. Would I call Java, C#, etc. "Programming Description Languages"? No. Therefore I wouldn't call an even higher level HPC language a description language either. In the article: "This draft description would then be presented to an AI based clarifier which would examine the description, look for inconsistencies or missing information and work with the programmer to create a formal description of the problem." Sounds like regular programming in an intolerant IDE with fancy terminology. In the article: "At that point the description is turned over to a really smart compiler that could target a particular hardware platform and produce the needed optimized binaries. Perhaps a GA could be thrown in to help optimize everything." Later on it is also mentioned that "Maybe it would take a week to create a binary, but it would be cluster time and not your time", where in reality with those really troublesome (useful) problems there are truly terribly long running times. With a GA (which produces eons more bad solutions than good) we would not only have to ascertain the fitness of the really nice solution (for those useful problems it could take a week or more at fastest) but also the fitness of the really really poor solution that swaps out constantly and computes redundantly. That could take years... The basic premise of the GA for code is Genetic Programming or an Evolutionary Algorithm, and so with these the same problems exist - bad solutions that monopolize time on the cluster. Compilers will eventually be entirely AI (though I doubt I will see it) and when they are, singularity will have already happened and infinite resources will be available since designing hardware is naturally more space constrained than software. All I'm saying is for right now, we are making the most of what we have without involving AI that extensively in our programming. Just my opinions, and no hard feelings towards Doug. Typically I enjoy thoroughly his articles. Ellis From hearnsj at googlemail.com Thu Oct 2 12:15:31 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: <48E51632.8090001@gmail.com> References: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> <48E4C509.3000407@aplpi.com> <48E51632.8090001@gmail.com> Message-ID: <9f8092cc0810021215q71060486w5d7f0b6cb540acf5@mail.gmail.com> > > Damn Small (DSL) is, Puppy is not. I believe Puppy is it's own beast. > > The FAQ on the site says its a Slackware derivative, if I'm not wrong. What goes around comes around I guess :-) Maybe those kipper ties from the 70s will be back too. Actually, and here I toss in a handgrenade, if we are considering a Damn Small Puppy PXE bootable Linux in RAM, what difference does the distro make, so long as the GLIBC/maths libraries/MPI libraries are the appropriate ones. I guess that statement goes for any Linux install really. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081002/4144d932/attachment.html From dgs at slac.stanford.edu Thu Oct 2 11:47:48 2008 From: dgs at slac.stanford.edu (David Simas) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Linux Magazine - What He Said In-Reply-To: References: <9f8092cc0810020205t66a8a8d7wab9e034fe6979b05@mail.gmail.com> <9f8092cc0810020954mca589feyd27bdea53d3b7b7f@mail.gmail.com> Message-ID: <20081002184748.GA29815@horus.slac.stanford.edu> > When I tried to ressurect my thing a couple years ago, I realized my > original code was all wrong in trading time for space (plenty of time on the > 386, then SunOS servers; not enough space, but new machine had plenty of > unused RAM). I thought some about redesigning to reverse the trade-off, > which would be helpful, but I'm sure it would not just be easier, but more > effective, to run it on a cluster (many nodes, not much ram per node needed, > and any nonzero amount of communication sufficient, but more can be > usefull). In case you don't know about PGAPack: http://www-fp.mcs.anl.gov/CCST/research/reports_pre1998/comp_bio/stalk/pgapack.html It's a genetic algorithm with MPI support. I've used the serial version, and it works great. I made a half-effort at getting the MPI version working, without success. DGS From YXU11 at PARTNERS.ORG Thu Oct 2 13:09:36 2008 From: YXU11 at PARTNERS.ORG (Xu, Jerry) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <200810021900.m92J05RP012792@bluewest.scyld.com> References: <200810021900.m92J05RP012792@bluewest.scyld.com> Message-ID: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> Hello, Currently I generate nearly one TB data every few days and I need to pass it along enterprise network to the storage center attached to my HPC system, I am thinking about compressing it (most tiff format image data) as much as I can, as fast as I can before I send it crossing network ... So, I am wondering whether anyone is familiar with any hardware based accelerator, which can dramatically improve the compressing procedure.. suggestion for any file system architecture will be appreciated too.. I have couple of contacts from some vendors but not sure whether it works as I expected, so if anyone has experience about it and want to share, it will be really appreciated ! Thanks, Jerry Jerry Xu PhD HPC Scientific Computing Specialist Enterprise Research Infrastructure Systems (ERIS) Partners Healthcare, Harvard Medical School http://www.partners.org The information transmitted in this electronic communication is intended only for the person or entity to whom it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this information in error, please contact the Compliance HelpLine at 800-856-1983 and properly dispose of this information. From niftyompi at niftyegg.com Thu Oct 2 14:29:44 2008 From: niftyompi at niftyegg.com (NiftyOMPI Mitch) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <48E3E69B.4030202@tuffmail.us> References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com> <48E3BA69.2000405@ias.edu> <1848483940.20081001201911@gmx.net> <88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com> <48E3E69B.4030202@tuffmail.us> Message-ID: <20081002212944.GA6110@compegg.wr.niftyegg.com> On Wed, Oct 01, 2008 at 04:07:39PM -0500, Alan Louis Scheinine wrote: > NiftyOMPI Mitch wrote >> QDR is interesting... in all likelyhood the >> QDR game will be optical for any link further away than a single rack. >> Once IB goes optical there will be a lot of reason to install IB in > > machine rooms and campus sites that are just out of reach today. > > When will IB go optical at a reasonable price? Perhaps you are not > an expert but if you happen to have any pointers where we can learn more > about the time frame it would be useful. Just a few days ago heard > colleagues trying to figure-out the solution to connecting file system > hardware just a bit too far from a cluster in another building. Ethernet > and NFS would work but latency is a problem. I have no good information and I do not know what a reasonable price is to you. If you wish to connect to a cluster in another building with optical links and IB there are some optical solutions today. Without "a lot of links" (=money) bandwidth maps will be lumpy. Building to building links and beyond solutions will be expensive many have active boxes switch-Cu<-magicbox->---optical---Cu-switch. For the current solutions scan the vendor lists from recent Supercomputer Shows and IB interoperability events. Floor to floor and room to room solutions like the Intel Optical IB cables are close to par with copper when you multiply by the link length. If you want good information put out a sane RFP and see what the vendors can come up with. Most importantly describe what you want solved and see what solutions are offered. Building to building clustering just seems hard to me but not all clustering requirements are equal. -- T o m M i t c h e l l Found me a new hat, now what? From landman at scalableinformatics.com Thu Oct 2 14:40:31 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> Message-ID: <48E53FCF.3020008@scalableinformatics.com> Xu, Jerry wrote: > Hello, > > Currently I generate nearly one TB data every few days and I need to pass it > along enterprise network to the storage center attached to my HPC system, I am > thinking about compressing it (most tiff format image data) as much as I can, as > fast as I can before I send it crossing network ... So, I am wondering whether > anyone is familiar with any hardware based accelerator, which can dramatically > improve the compressing procedure.. suggestion for any file system architecture > will be appreciated too.. I have couple of contacts from some vendors but not > sure whether it works as I expected, so if anyone has experience about it and > want to share, it will be really appreciated ! Hi Jerry: Sounds like a bunch of sequencers or arrays going, and dumping data to a file system. I am not sure you can get deterministic compression run times from a compressor, or even deterministic compression ratios on random (binary) data. I have heard of some "xml accelerators" in the past (back when XML was considered a good buzzword) that did on-the-fly compression. I guess it boils down to if T(compression) + T(comppressed transfer) << T(uncompressed_transfer) And the cost benefit analysis would focus upon the cost of T(compression) as compared to faster networks. That is, if you spent $1000/node more to get a faster fabric, which dropped your transfer time to 20%, is this better/more cost effective than spending $10,000 or so on an accelerate that may get 70% file compression, and double the overall time? Obviously the above numbers are made up, but you get the idea. Will look around. If you want to talk to a group doing FPGA stuff in other markets, let me know and I can hook you up. Just be aware that this might not be cost/time effective. Joe > > > > Thanks, > > Jerry > > Jerry Xu PhD > HPC Scientific Computing Specialist > Enterprise Research Infrastructure Systems (ERIS) > Partners Healthcare, Harvard Medical School > http://www.partners.org > > The information transmitted in this electronic communication is intended only > for the person or entity to whom it is addressed and may contain confidential > and/or privileged material. Any review, retransmission, dissemination or other > use of or taking of any action in reliance upon this information by persons or > entities other than the intended recipient is prohibited. If you received this > information in error, please contact the Compliance HelpLine at 800-856-1983 and > properly dispose of this information. > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From lindahl at pbm.com Thu Oct 2 14:55:02 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E53FCF.3020008@scalableinformatics.com> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E53FCF.3020008@scalableinformatics.com> Message-ID: <20081002215502.GA6059@bx9.net> On Thu, Oct 02, 2008 at 05:40:31PM -0400, Joe Landman wrote: > I have heard of some "xml accelerators" in the > past (back when XML was considered a good buzzword) that did on-the-fly > compression. Well, given how wordy the tags are, simply compressing those is inexpensive and is a big win, if they're a large % of the data. To get back to the question that was asked, (1) no hardare compresser compresses smaller than a good software compresser (hardware compressors tend to be faster but can't compress as well), and (2) sounds like your data can be compressed in an embarrassingly parallel fashion, so on a quad-core box you might find that you can keep up with your link with software compression in parallel. -- greg From niftyompi at niftyegg.com Thu Oct 2 15:07:16 2008 From: niftyompi at niftyegg.com (Nifty niftyompi Mitch) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> Message-ID: <20081002220716.GB6110@compegg.wr.niftyegg.com> On Thu, Oct 02, 2008 at 04:09:36PM -0400, Xu, Jerry wrote: > > Currently I generate nearly one TB data every few days and I need to pass it > along enterprise network to the storage center attached to my HPC system, I am > thinking about compressing it (most tiff format image data) as much as I can, as > fast as I can before I send it crossing network ... So, I am wondering whether > anyone is familiar with any hardware based accelerator, which can dramatically > improve the compressing procedure.. suggestion for any file system architecture > will be appreciated too.. I have couple of contacts from some vendors but not > sure whether it works as I expected, so if anyone has experience about it and > want to share, it will be really appreciated ! If I recall correctly TIFF files are hard to compress any more than they already are. Linux has a handful of compression tools -- how much compression are you able to get on your data with each of these tools and each command line set of options. My guess is that the best and most cost effective hardware solution you will find is a hot Opteron or Intel box with not too many cores and a good chunk fast DRAM in it. You might find that contrary to the rest of linux a good optimizing compiler like PGI, Pathscale, Intel... will speed up the compression code enough to matter so consider rebuilding things like bzip2, gzip, p7zip and benchmarking the best compression tool for speed and correctness. And when you find the best compression program for your data you might look for some good DSP cards to run your compression on. My bet is that a new hot box will win. Since this is a cluster mailing list just bang the compression out to as a cluster job. -- T o m M i t c h e l l Found me a new hat, now what? From reuti at staff.uni-marburg.de Thu Oct 2 15:09:39 2008 From: reuti at staff.uni-marburg.de (Reuti) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> Message-ID: <84083434-B04C-4081-8BD0-3F288403F2F7@staff.uni-marburg.de> Hi, Am 02.10.2008 um 22:09 schrieb Xu, Jerry: > Currently I generate nearly one TB data every few days and I need > to pass it > along enterprise network to the storage center attached to my HPC > system, I am > thinking about compressing it (most tiff format image data) is it plain tiff or already using any compression like RLE or LZW inside? Do you want or must stay with tiff? -- Reuti > as much as I can, as > fast as I can before I send it crossing network ... So, I am > wondering whether > anyone is familiar with any hardware based accelerator, which can > dramatically > improve the compressing procedure.. suggestion for any file system > architecture > will be appreciated too.. I have couple of contacts from some > vendors but not > sure whether it works as I expected, so if anyone has experience > about it and > want to share, it will be really appreciated ! > > > > Thanks, > > Jerry > > Jerry Xu PhD > HPC Scientific Computing Specialist > Enterprise Research Infrastructure Systems (ERIS) > Partners Healthcare, Harvard Medical School > http://www.partners.org > > The information transmitted in this electronic communication is > intended only > for the person or entity to whom it is addressed and may contain > confidential > and/or privileged material. Any review, retransmission, > dissemination or other > use of or taking of any action in reliance upon this information by > persons or > entities other than the intended recipient is prohibited. If you > received this > information in error, please contact the Compliance HelpLine at > 800-856-1983 and > properly dispose of this information. > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From gdjacobs at gmail.com Thu Oct 2 15:26:39 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: <9f8092cc0810021215q71060486w5d7f0b6cb540acf5@mail.gmail.com> References: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> <48E4C509.3000407@aplpi.com> <48E51632.8090001@gmail.com> <9f8092cc0810021215q71060486w5d7f0b6cb540acf5@mail.gmail.com> Message-ID: <48E54A9F.6020002@gmail.com> John Hearns wrote: > Damn Small (DSL) is, Puppy is not. I believe Puppy is it's own beast. > > The FAQ on the site says its a Slackware derivative, if I'm not wrong. > What goes around comes around I guess :-) Maybe those kipper ties from > the 70s will be back too. First question on the page... http://puppylinux.org/wiki/archives/old-wikka-wikki/categorydocumentation/historypuppy > Actually, and here I toss in a handgrenade, if we are considering a Damn > Small Puppy PXE bootable Linux in RAM, what difference does the distro > make, so long as the GLIBC/maths libraries/MPI libraries are the > appropriate ones. > I guess that statement goes for any Linux install really. Well, you want to use something similar to the head in the compute nodes, and you want the head node install to have all the necessary infrastructure. Technically, you could use a different install on each the nodes, but it wouldn't be smart. Building a pill for the compute nodes a la Scyld or Perseus seems like the best compromise in terms of reducing the compute node operating environment while maintaining a common software base. -- Geoffrey D. Jacobs From grumiche at integrityit.com.br Thu Oct 2 15:37:19 2008 From: grumiche at integrityit.com.br (Rodrigo Grumiche Silva) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> Message-ID: Hi Jerry I think HDF5 can help you in some way... - http://www.hdfgroup.org/HDF5/ Rodrigo 2008/10/2 Xu, Jerry > Hello, > > Currently I generate nearly one TB data every few days and I need to pass > it > along enterprise network to the storage center attached to my HPC system, I > am > thinking about compressing it (most tiff format image data) as much as I > can, as > fast as I can before I send it crossing network ... So, I am wondering > whether > anyone is familiar with any hardware based accelerator, which can > dramatically > improve the compressing procedure.. suggestion for any file system > architecture > will be appreciated too.. I have couple of contacts from some vendors but > not > sure whether it works as I expected, so if anyone has experience about it > and > want to share, it will be really appreciated ! > > > > Thanks, > > Jerry > > Jerry Xu PhD > HPC Scientific Computing Specialist > Enterprise Research Infrastructure Systems (ERIS) > Partners Healthcare, Harvard Medical School > http://www.partners.org > > The information transmitted in this electronic communication is intended > only > for the person or entity to whom it is addressed and may contain > confidential > and/or privileged material. Any review, retransmission, dissemination or > other > use of or taking of any action in reliance upon this information by persons > or > entities other than the intended recipient is prohibited. If you received > this > information in error, please contact the Compliance HelpLine at > 800-856-1983 and > properly dispose of this information. > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081002/ca184f03/attachment.html From bill at cse.ucdavis.edu Thu Oct 2 18:11:10 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> Message-ID: <48E5712E.7070504@cse.ucdavis.edu> Xu, Jerry wrote: > Hello, > > Currently I generate nearly one TB data every few days and I need to pass it > along enterprise network to the storage center attached to my HPC system, I am > thinking about compressing it (most tiff format image data) tiff uncompressed, or tiff compressed files? If uncompressed I'd guess that bzip2 might do well with them. > as much as I can, as > fast as I can before I send it crossing network ... So, I am wondering whether > anyone is familiar with any hardware based accelerator, which can dramatically > improve the compressing procedure.. Improve? You mean compression ratio? Wall clock time? CPU utilization? Adding forward error correction? > suggestion for any file system architecture > will be appreciated too.. Er, hard to imagine a reasonable recommendation without much more information. Organization, databases (if needed), filenames and related metadata are rather specific to the circumstances. Access patterns, retention time, backups, and many other issues would need consideration. > I have couple of contacts from some vendors but not > sure whether it works as I expected, so if anyone has experience about it and > want to share, it will be really appreciated ! Why hardware? I have some python code that managed 10MB/sec per CPU (or 80MB on 8 CPUs if you prefer) that compresses with zlib, hashes with sha256, and encrypts with AES (256 bit key). Assuming the compression you want isn't substantially harder than doing zlib, sha256, and aes a single core from a dual or quad core chip sold in the last few years should do fine. 1TB every 2 days = 6MB/sec or approximately 15% of a quad core or 60% of a single core for my compress, hash and encrypt in python. Considering how cheap cores are (quad desktops are often under $1k) I'm not sure what would justify an accelerator card. Not to mention picking the particular algorithm could make a huge difference to the CPU and compression ratio achieved. I'd recommend taking a stack of real data and trying out different compression tools and settings. In any case 6MB/sec of compression isn't particularly hard these days.... even in python on a 1-2 year old mid range cpu. From hahn at mcmaster.ca Thu Oct 2 19:31:48 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> Message-ID: > Currently I generate nearly one TB data every few days and I need to pass it Bill's right - 6 MB/s is really not much to ask from even a complex WAN. I think the first thing you should do is find the bottleneck. to me it sounds like you have a sort of ropey path with a 100 Mbps hop somewhere. > thinking about compressing it (most tiff format image data) as much as I can tiff is a fairly generic container that can hold anything from a horrible uncompressed 4-byte-per-pixel to jpeg or rle. looking at the format you're really using would be wise. I'm guessing that if you transcode to png, you'll get better compression than gzip/etc. dictionary-based compression is fundamentally inappropriate for most non-text data - not images, not double-precision dumps of physical simulations, etc. png is quite a lot smarter about most kinds of images than older formats, and can be lossy or lossless. hardware compression would be a serious mistake unless you've already pursued these routes. specialized hardware is a very short-term and quite narrow value proposition. I would always prefer to improve the infrastructure. > The information transmitted in this electronic communication is intended only uh, email is publication. regards, mark hahn. From coutinho at dcc.ufmg.br Thu Oct 2 19:42:32 2008 From: coutinho at dcc.ufmg.br (Bruno Coutinho) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E5712E.7070504@cse.ucdavis.edu> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> Message-ID: 2008/10/2 Bill Broadley <...> Why hardware? I have some python code that managed 10MB/sec per CPU (or > 80MB > on 8 CPUs if you prefer) that compresses with zlib, hashes with sha256, and > encrypts with AES (256 bit key). Assuming the compression you want isn't > substantially harder than doing zlib, sha256, and aes a single core from a > dual or quad core chip sold in the last few years should do fine. > > 1TB every 2 days = 6MB/sec or approximately 15% of a quad core or 60% of a > single core for my compress, hash and encrypt in python. Considering how > cheap cores are (quad desktops are often under $1k) I'm not sure what would > justify an accelerator card. Not to mention picking the particular > algorithm > could make a huge difference to the CPU and compression ratio achieved. > I'd > recommend taking a stack of real data and trying out different compression > tools and settings. > > In any case 6MB/sec of compression isn't particularly hard these days.... > even > in python on a 1-2 year old mid range cpu. > > > In Information Retrieval, they compress almost everything and they have papers showing that using compression can result in a *faster* system. You process a little more, but get great gains in disk throughput. If you compress before even storing data, your system could store faster by using less disk/storage bandwidth per stored file. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081002/d60c8b87/attachment.html From diep at xs4all.nl Fri Oct 3 01:13:16 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E5712E.7070504@cse.ucdavis.edu> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> Message-ID: <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> Bzip2, gzip, Why do you guys keep quoting those total outdated compressors :) there is 7-zip for linux, it's open source and also part of LZMA. On average remnants are 2x smaller than what gzip/bzip2 is doing for you (so bzip2/gzip is factor 2 worse). 7-zip also works parallel, not sure whether it works in linux parallel. 7za is command line version. Linux distributions should include it default. Uses PPM, that's a new form of multidimensional compression that all that old junk like bzip2/gzip doesn't use. TIFF files compress real bad of course. Maybe convert them to some more inefficient format, which increases its size probably, which then compresses real great with PPM. When googling for the best compressors, don't try PAQ, that's a benchmark compressor. Was worse for my terabyte of data than even 7-zip (which is not by far best PPM compressor, but it's open source). Vincent On Oct 3, 2008, at 3:11 AM, Bill Broadley wrote: > Xu, Jerry wrote: >> Hello, Currently I generate nearly one TB data every few days and >> I need to pass it >> along enterprise network to the storage center attached to my HPC >> system, I am >> thinking about compressing it (most tiff format image data) > > tiff uncompressed, or tiff compressed files? If uncompressed I'd > guess that > bzip2 might do well with them. > >> as much as I can, as >> fast as I can before I send it crossing network ... So, I am >> wondering whether >> anyone is familiar with any hardware based accelerator, which can >> dramatically >> improve the compressing procedure.. > > Improve? You mean compression ratio? Wall clock time? CPU > utilization? > Adding forward error correction? > >> suggestion for any file system architecture >> will be appreciated too.. > > Er, hard to imagine a reasonable recommendation without much more > information. > Organization, databases (if needed), filenames and related metadata > are rather > specific to the circumstances. Access patterns, retention time, > backups, and many other issues would need consideration. > >> I have couple of contacts from some vendors but not >> sure whether it works as I expected, so if anyone has experience >> about it and >> want to share, it will be really appreciated ! > > Why hardware? I have some python code that managed 10MB/sec per > CPU (or 80MB > on 8 CPUs if you prefer) that compresses with zlib, hashes with > sha256, and > encrypts with AES (256 bit key). Assuming the compression you want > isn't > substantially harder than doing zlib, sha256, and aes a single core > from a > dual or quad core chip sold in the last few years should do fine. > > 1TB every 2 days = 6MB/sec or approximately 15% of a quad core or > 60% of a > single core for my compress, hash and encrypt in python. > Considering how > cheap cores are (quad desktops are often under $1k) I'm not sure > what would > justify an accelerator card. Not to mention picking the particular > algorithm > could make a huge difference to the CPU and compression ratio > achieved. I'd > recommend taking a stack of real data and trying out different > compression > tools and settings. > > In any case 6MB/sec of compression isn't particularly hard these > days.... even > in python on a 1-2 year old mid range cpu. > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From bill at cse.ucdavis.edu Fri Oct 3 02:17:52 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> Message-ID: <48E5E340.6080408@cse.ucdavis.edu> Vincent Diepeveen wrote: > Bzip2, gzip, > > Why do you guys keep quoting those total outdated compressors :) Path of least resistance, not to mention python bindings. > there is 7-zip for linux, it's open source and also part of LZMA. On > average remnants > are 2x smaller than what gzip/bzip2 is doing for you (so bzip2/gzip is > factor 2 worse). > 7-zip also works parallel, not sure whether it works in linux parallel. > 7za is command line > version. Seems like the question is related to CPU utilization as well as compression ratios. Assuming the TIFF files are not already compressed, how fast would you expect 7-zip to be relative to bzip2 and gzip's compression and decompression speeds? I was looking for decent bandwidth, and I did look around a bit and it seemed like things often would compress somewhat better, often the bandwidth achieved was 5-6x worse. So for squeezing the most out of a 28k modem... sure. For keeping up with a 100mbit or GigE connection on a local LAN, not so much. Google finds: http://blogs.reucon.com/srt/2008/02/18/compression_gzip_vs_bzip2_vs_7_zip.html Compressor Size Ratio Compression Decompression gzip 89 MB 54 % 0m 13s 0m 05s bzip2 81 MB 49 % 1m 30s 0m 20s 7-zip 61 MB 37 % 1m 48s 0m 11s So sure you save 28MB, at the cost of 95 seconds. Might make sense if you are transfering over a slow modem. Also considering the original file was 163MB it's nowhere near the 6MB/sec that seems to be the target. At 1.5MB/sec you'd need 4 CPUs running flat out for 2 days to manage 2TB, instead of 1 CPU running for just 24 hours. Definitely the kind of thing that sounds like it might make a big difference. Another example: http://bbs.archlinux.org/viewtopic.php?t=11670 7zip compress: 19:41 Bzip2 compress: 8:56 Gzip compress: 3:00 Again 7zip is a factor of 6 and change slower than gzip. > Linux distributions should include it default. > > Uses PPM, that's a new form of multidimensional compression that all > that old junk like > bzip2/gzip doesn't use. One man's junk and another man's gold. My use was backup related and I definitely didn't want to become CPU limited even on large systems with 10TB of disk and a healthy I/O system. From the sounds of it even with 8 fast cores that 7zip might easily be the bottleneck. > TIFF files compress real bad of course. Maybe convert them to some more > inefficient format, > which increases its size probably, which then compresses real great with > PPM. Er, that makes no sense to me. You aren't going to end up with a smaller file by encoding a file less efficiently.. under ideal circumstances you might get back to where you started with a substantial use of cycles. Seems pretty simple, if the TIFFs are compressed, just send them as is, significant additional compression is unlikely. If they are uncompressed there's a decent chance of significant lossless compression, the best thing to do would be to try it or at least a reference to some similar images. From carsten.aulbert at aei.mpg.de Fri Oct 3 02:27:36 2008 From: carsten.aulbert at aei.mpg.de (Carsten Aulbert) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E5E340.6080408@cse.ucdavis.edu> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> Message-ID: <48E5E588.1050508@aei.mpg.de> Hi all Bill Broadley wrote: > > Another example: > http://bbs.archlinux.org/viewtopic.php?t=11670 > > 7zip compress: 19:41 > Bzip2 compress: 8:56 > Gzip compress: 3:00 > > Again 7zip is a factor of 6 and change slower than gzip. Have you looked into threaded/parallel bzip2? freshmeat has a few of those, e.g. http://freshmeat.net/projects/bzip2smp/ http://freshmeat.net/projects/lbzip2/ (with the usual disclaimer that I haven't tested them myself). HTH carsten From bill at cse.ucdavis.edu Fri Oct 3 02:28:52 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E5E340.6080408@cse.ucdavis.edu> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> Message-ID: <48E5E5D4.5030102@cse.ucdavis.edu> For uncompressed TIFF files this might be of use: http://optipng.sourceforge.net/features.html It seems to mention the lossless compression of TIFF files. From diep at xs4all.nl Fri Oct 3 03:29:16 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E5E340.6080408@cse.ucdavis.edu> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> Message-ID: hi Bill, 7-zip is of course faster than bzip2 and a bit slower than gzip. Thing is that after the files have been compressed you need to do less i/o; the big bottleneck in most systems is the bandwidth from and to i/o. A single disk start of 90s was several megabytes per second up to 10 MB/s. Speed i see SATA2 drives get now is roughly 133MB/s. I remember a speech from NCSA director a few weeks ago where he showed a graph, and it's quite obvious that we are gonna get enough cpu power in future, what to do with it? Let's be realistic. Suppose you've got a bunch of cores. Soon hundreds. Then for each byte you get off disk, you have hundreds of cpu cycles to decompress. That bandwidth to and from i/o you can limit bigtime with a bigger compression, cpu power there is enough, provided it's not taking too long to decompress. Decompression speed of 7-zip is faster than bzip2 in my experience, simply because it can decompress in parallel and i don't see bzip2 ever getting parallellized. Compression is one of those areas where there gets put little money in making a real good realtime compression. In fact i'm busy making a kind of filesystem myself now where compression is allowed to take long and decompression must be realtime (simply at a given cpu speed a certain bandwidth to decompress) for any bucket. Best compressors today have just 1 basic problem for GIS type and in my case chess database systems; they decompress the entire file. With files of gigabytes, if you just want to access random byte number X in the file, you're not interested in bandwidth nor whether gzip is faster or slower than PPM type compressors in decompression. You just want that single byte that might be at the end of the file. I'm planning to build such a compressor which is doing a lot of mathematical function comparisions to get PPM type compression, yet taking care decompression can be done realtime. there is a few filesystems which allow this, but the compression they use for it, is just 70s/80s huffman based. Real ugly bad, to say polite. Note i'm speaking of lossless compression here, lossy compression is yet another subject. On the web if you ask me, too many photo's are of a quality that is just ugly. Lots to improve there as well. Lossy you can do way more of course than lossless. Yet lossless i do not see real great systems there yet. Time to build one myself :) My argument is that something like a GPU might be real interesting to take a look at for compression. You can dedicate a bunch of cheapo GPU's to just compression; for databases what really matters is the decompression time. They store so much, i'd argue a good compression, both lossless and lossy is what we need; Nowadays governments want to put in database every phoneconversation, no matter how hard they're gonna deny that to you; I'd forsee that the amount of storage needed is huge, tiny company i worked for was putting away in petabyte level, and majority will never get decompressed, other than for some AI type software programs that do some scanning later on. The real big difference between the 'testsets' that get used in compression and the status of compression, is that there is no money to make for those guys writing a superior compressor that works in terabytes. Companies do not soon invest in such technologies, they just invest in products they can sell. Having also worked with GIS data i see possibilities to compress that factors better and still realtime look it up from embedded cpu's. Vincent On Oct 3, 2008, at 11:17 AM, Bill Broadley wrote: > Vincent Diepeveen wrote: >> Bzip2, gzip, >> Why do you guys keep quoting those total outdated compressors :) > > Path of least resistance, not to mention python bindings. > >> there is 7-zip for linux, it's open source and also part of LZMA. >> On average remnants >> are 2x smaller than what gzip/bzip2 is doing for you (so bzip2/ >> gzip is factor 2 worse). >> 7-zip also works parallel, not sure whether it works in linux >> parallel. 7za is command line >> version. > > Seems like the question is related to CPU utilization as well as > compression ratios. Assuming the TIFF files are not already > compressed, how fast would you expect 7-zip to be relative to bzip2 > and gzip's compression and decompression speeds? I was looking for > decent bandwidth, and I did look around a bit and it seemed like > things often would compress somewhat better, often the bandwidth > achieved was 5-6x worse. So for squeezing the most out of a 28k > modem... sure. For keeping up with a 100mbit or GigE connection on > a local LAN, not so much. > > Google finds: > http://blogs.reucon.com/srt/2008/02/18/ > compression_gzip_vs_bzip2_vs_7_zip.html > > Compressor Size Ratio Compression Decompression > gzip 89 MB 54 % 0m 13s 0m 05s > bzip2 81 MB 49 % 1m 30s 0m 20s > 7-zip 61 MB 37 % 1m 48s 0m 11s > > So sure you save 28MB, at the cost of 95 seconds. Might make sense > if you are transfering over a slow modem. Also considering the > original file was 163MB it's nowhere near the 6MB/sec that seems to > be the target. At 1.5MB/sec you'd need 4 CPUs running flat out for > 2 days to manage 2TB, instead of 1 CPU running for just 24 hours. > Definitely the kind of thing that sounds like it might make a big > difference. > > Another example: > http://bbs.archlinux.org/viewtopic.php?t=11670 > > 7zip compress: 19:41 > Bzip2 compress: 8:56 > Gzip compress: 3:00 > > Again 7zip is a factor of 6 and change slower than gzip. > >> Linux distributions should include it default. >> Uses PPM, that's a new form of multidimensional compression that >> all that old junk like >> bzip2/gzip doesn't use. > > One man's junk and another man's gold. My use was backup related > and I definitely didn't want to become CPU limited even on large > systems with 10TB of disk and a healthy I/O system. From the > sounds of it even with 8 fast cores that 7zip might easily be the > bottleneck. > >> TIFF files compress real bad of course. Maybe convert them to some >> more inefficient format, >> which increases its size probably, which then compresses real >> great with PPM. > > Er, that makes no sense to me. You aren't going to end up with a > smaller file by encoding a file less efficiently.. under ideal > circumstances you might get back to where you started with a > substantial use of cycles. Seems pretty simple, if the TIFFs are > compressed, just send them as is, significant additional > compression is unlikely. If they are uncompressed there's a decent > chance of significant lossless compression, the best thing to do > would be to try it or at least a reference to some similar images. > From diep at xs4all.nl Fri Oct 3 03:53:49 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E5E588.1050508@aei.mpg.de> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de> Message-ID: <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl> hi Carsten, Ah you googled 2 seconds and found some oldie homepage. Try this homepage www.maximumcompression.com Far better testing over there. Note that it's the same testset there that gets compressed a lot. In real life, database type data is having all kind of patterns which PPM type compressors find. My experience is that at terabyte level the better compressors at maximumcompression.com, are a bit too slow (PAQ) and not so good like simple things like 7-zip. Look especially at compressed sizes and decompression times. The only thing you want to limit over your network is the amount of bandwidth over your network. A real good compression is very helpful then. How long compression time takes is nearly not relevant, as long as it doesn't take infinite amounts of time (i remember a new zealand compressor which took 24 hours to compress a 100MB data). Note that we are already at a phase that compression time hardly matters, you can buy a GPU for that to offload your servers for that. Query time (so decompression time) is important though. If we look to graphics there: 026 7-Zip 4.60b -m0=ppmd:o=4 764420 81.58 1.4738 .. 94 BZIP2 1.0.5 -9 890163 78.55 1.7162 .. 158 PKZIP 2.50 -exx 1250536 69.86 2.4110 159 HIT 2.10 -x 1250601 69.86 2.4111 160 GZIP 1.3.5 -9 1254351 69.77 2.4184 161 ZIP 2.2 -9 1254444 69.77 2.4185 162 WINZIP 8.0 (Max Compression) 1254444 69.77 2.4185 Note a real supercompressor is getting it even tinier: 003 WinRK 3.0.3 PWCM 912MB 568919 86.29 1.0969 Again all these tests are at microlevel. Just a few megabtes of data that gets compressed. You don't build a big infrastructure just for a few megabytes, it's not so relevant. The traffic over your network dominates there, plenty of idle server cores there is, in fact there is so many companies now that buy dual cores, as they do not know how to keep the cores in quad cores busy. This is all microlevel. Things really change when you have terabytes to compress and HUGE files. Bzip2 is ugly slow for files in gigabyte size, 7-zip is totally beating it there. Vincent On Oct 3, 2008, at 11:27 AM, Carsten Aulbert wrote: > Hi all > > Bill Broadley wrote: >> >> Another example: >> http://bbs.archlinux.org/viewtopic.php?t=11670 >> >> 7zip compress: 19:41 >> Bzip2 compress: 8:56 >> Gzip compress: 3:00 >> >> Again 7zip is a factor of 6 and change slower than gzip. > > Have you looked into threaded/parallel bzip2? > > freshmeat has a few of those, e.g. > > http://freshmeat.net/projects/bzip2smp/ > http://freshmeat.net/projects/lbzip2/ > > (with the usual disclaimer that I haven't tested them myself). > > HTH > > carsten > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From hahn at mcmaster.ca Fri Oct 3 07:00:05 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: References: <48E39782.4020006@ias.edu> Message-ID: >> Has DDR IB gone the way of the dodo bird and been supplanted by QDR? > > I don't think front side busses are fast enough for QDR yet. qdr would be 40 Gb (raw data rate, right? so 4 GB/s before any sort of packet overhead, etc.) I don't really see why that's a problem - even a memory-constrained current-gen Intel box has about twice that much memory bandwidth available. AMD or next-gen-Intel will be even less constrained. otoh, I rarely see workloads that are clearly net-bw-limited even on 4-core myrinet 2g machines - and that's only about 250 MB/s. regards, mark hahn. From carsten.aulbert at aei.mpg.de Fri Oct 3 04:49:04 2008 From: carsten.aulbert at aei.mpg.de (Carsten Aulbert) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl> Message-ID: <48E606B0.3050708@aei.mpg.de> Hi Vincent, Vincent Diepeveen wrote: > Ah you googled 2 seconds and found some oldie homepage. Actually no, I just looked at my freshmeat watchlist of items still to look at :) > > Look especially at compressed sizes and decompression times. Yeah, I'm currently looking at http://www.maximumcompression.com/data/summary_mf3.php We have a Gbit network, i.e. for us this test is a null test, since it takes 7-zip close to 5 minutes to compress the data set of 311 MB which we could blow over the network in less than 5 seconds, i.e. in this case tar would be our favorite ;) > > The only thing you want to limit over your network is the amount of > bandwidth over your network. > A real good compression is very helpful then. How long compression time > takes is nearly not relevant, > as long as it doesn't take infinite amounts of time (i remember a new > zealand compressor which took 24 > hours to compress a 100MB data). Note that we are already at a phase > that compression time hardly > matters, you can buy a GPU for that to offload your servers for that. > No, quite on the contrary. I would like to use a compressor within a pipe to increase the throughput over the network, i.e. to get around the ~ 120 MB/s limit. > Query time (so decompression time) is important though. > No for us that number is at least as important than the compression time. Imagine the following situation: We have a file server with close to 10 TB of data on it in nice chunks with a since of about 100 MB each. We buy a new server with new disks and the new one can now hold 20 TB and we would like to copy the data over. So for us the more important figure is the compression/decompression speed which should be >> 100 MB/s on our systems. If 7-zip can only compress data at a rate of less than say 5 MB/s (input data) I can much much faster copy the data over uncompressed regardless of how many unused cores I have in the system. Exactly for these cases I would like to use all cores available to compress the data fast in order to increase the throughput. Do I miss something vital? Cheers Carsten From diep at xs4all.nl Fri Oct 3 08:27:36 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E606B0.3050708@aei.mpg.de> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl> <48E606B0.3050708@aei.mpg.de> Message-ID: Hi Carsten, In your example the only thing that seems to matter to you is *collecting* data speed, in short the realtime compression speed that tapestreamers can get, to give one example. In your example you need to compress each time stuff. That's not being realistic however. I'll give you 2 arguments. In an ideal situation you compress things only a single time. So decompression speed matters if you want to realtime lookup data. It would be far more ideal in your situation to already have things very well compressed at your drive, doing some realtime compression/decompression is not very useful then, the hardware compression of those tape streamers already is doing some simplistico Runlength encoding usually. Now another lack is that you assume stuff that's on your 10TB array is never getting used by not a single user over the network. Vincent On Oct 3, 2008, at 1:49 PM, Carsten Aulbert wrote: > Hi Vincent, > > Vincent Diepeveen wrote: >> Ah you googled 2 seconds and found some oldie homepage. > > Actually no, I just looked at my freshmeat watchlist of items still to > look at :) > >> >> Look especially at compressed sizes and decompression times. > > Yeah, I'm currently looking at > http://www.maximumcompression.com/data/summary_mf3.php > > We have a Gbit network, i.e. for us this test is a null test, since it > takes 7-zip close to 5 minutes to compress the data set of 311 MB > which > we could blow over the network in less than 5 seconds, i.e. in this > case > tar would be our favorite ;) > >> >> The only thing you want to limit over your network is the amount of >> bandwidth over your network. >> A real good compression is very helpful then. How long compression >> time >> takes is nearly not relevant, >> as long as it doesn't take infinite amounts of time (i remember a new >> zealand compressor which took 24 >> hours to compress a 100MB data). Note that we are already at a phase >> that compression time hardly >> matters, you can buy a GPU for that to offload your servers for that. >> > > No, quite on the contrary. I would like to use a compressor within a > pipe to increase the throughput over the network, i.e. to get > around the > ~ 120 MB/s limit. > >> Query time (so decompression time) is important though. >> > > No for us that number is at least as important than the compression > time. > > Imagine the following situation: > > We have a file server with close to 10 TB of data on it in nice chunks > with a since of about 100 MB each. We buy a new server with new disks > and the new one can now hold 20 TB and we would like to copy the data > over. So for us the more important figure is the > compression/decompression speed which should be >> 100 MB/s on our > systems. > > If 7-zip can only compress data at a rate of less than say 5 MB/s > (input > data) I can much much faster copy the data over uncompressed > regardless > of how many unused cores I have in the system. Exactly for these > cases I > would like to use all cores available to compress the data fast in > order > to increase the throughput. > > Do I miss something vital? > > Cheers > > Carsten > From landman at scalableinformatics.com Fri Oct 3 08:45:43 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E606B0.3050708@aei.mpg.de> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl> <48E606B0.3050708@aei.mpg.de> Message-ID: <48E63E27.6060701@scalableinformatics.com> Carsten Aulbert wrote: > If 7-zip can only compress data at a rate of less than say 5 MB/s (input > data) I can much much faster copy the data over uncompressed regardless > of how many unused cores I have in the system. Exactly for these cases I > would like to use all cores available to compress the data fast in order > to increase the throughput. This is fundamentally the issue. If the compression time plus the tranmit time for the compressed data is greater than the transmit time for the uncompressed data, then the compression may not be worth it. Sure, if it is nothing but text files, you may get 60-80+% compression rates. But for the case of (non-pathological) binary data, it might be only a few percent. So in this case, even if you could get a few percent delta from the compression, is that worth all the extra time you spend to get it? At the end of the day the question is how much lossless compression can you do in a short enough time for it to be meaningful in terms of transmitting the data? > > Do I miss something vital? Nope. You got it nailed. Several months ago, I tried moving about 600 GB of data from an old server to a JackRabbit. The old server and the JackRabbit had a gigabit link between them. We regularly saw 45 MB scp rates (one of the chips in the older server was a Broadcom). I tried this with and without compression. With compression (simple gzip), the copy took something like 28 hours ( a little more than a day). Without compression, it was well under 10 hours. In this case, compression (gzip) was not worth it. The command I used for the test was uncompressed: cd /directory tar -cpf - ./ | ssh jackrabbit "cd /directory ; tar -xpvf - " compressed: cd /directory tar -czpf - ./ | ssh jackrabbit "cd /directory ; tar -xzpvf - " if you want to spend more time, use "j" rather than "z" in the options. YMMV, but I have been convinced that, apart from specific use cases with text only documents or documents known to compress quickly/well, that compression prior to transfer may waste more time than it saves. This said, if someone has a parallel hack of gzip or similar we can pipe through, by all means, I would be happy to try it. But it would have to be pretty darned efficient. 100MB/s means 1 byte transmitted,on average, in 10 nanoseconds. Which means for compression to be meaningful, you would need to compute for less time than that to increase the information density. Put another way, 1 MB takes about 10 ms to send over a gigabit link. For compression to be meaningful, you need to compress this 1MB in far less than 10 ms and still transmit it in that time. Assuming that any compression algorithm has to walk through data at least once, A 1 GB/s memory subsystem takes about 1 ms to walk through this data at least once, so you need as few passes as possible through the data set to construct the compressed representation, as you will still have on the order of 1E+5 bytes to send. I am not saying it is hopeless, just hard for complex compression schemes (bzip2, etc). When we get enough firepower in the CPU (or maybe GPU ... hmmmm) the situation may improve. GPU as a compression engine? Interesting ... Joe > > Cheers > > Carsten -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From diep at xs4all.nl Fri Oct 3 08:55:08 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E63E27.6060701@scalableinformatics.com> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl> <48E606B0.3050708@aei.mpg.de> <48E63E27.6060701@scalableinformatics.com> Message-ID: <0D74B691-5A4B-4D0D-98B5-DDCBE062F729@xs4all.nl> The question is Joe, Why are you storing it uncompressed? Vincent On Oct 3, 2008, at 5:45 PM, Joe Landman wrote: > Carsten Aulbert wrote: > >> If 7-zip can only compress data at a rate of less than say 5 MB/s >> (input >> data) I can much much faster copy the data over uncompressed >> regardless >> of how many unused cores I have in the system. Exactly for these >> cases I >> would like to use all cores available to compress the data fast in >> order >> to increase the throughput. > > This is fundamentally the issue. If the compression time plus the > tranmit time for the compressed data is greater than the transmit > time for the uncompressed data, then the compression may not be > worth it. Sure, if it is nothing but text files, you may get 60-80+ > % compression rates. But for the case of (non-pathological) binary > data, it might be only a few percent. So in this case, even if > you could get a few percent delta from the compression, is that > worth all the extra time you spend to get it? > > At the end of the day the question is how much lossless compression > can you do in a short enough time for it to be meaningful in terms > of transmitting the data? > >> Do I miss something vital? > > Nope. You got it nailed. > > Several months ago, I tried moving about 600 GB of data from an old > server to a JackRabbit. The old server and the JackRabbit had a > gigabit link between them. We regularly saw 45 MB scp rates (one > of the chips in the older server was a Broadcom). > > I tried this with and without compression. With compression > (simple gzip), the copy took something like 28 hours ( a little > more than a day). Without compression, it was well under 10 hours. > > In this case, compression (gzip) was not worth it. The command I > used for the test was > > uncompressed: > > cd /directory > tar -cpf - ./ | ssh jackrabbit "cd /directory ; tar -xpvf - " > > compressed: > > cd /directory > tar -czpf - ./ | ssh jackrabbit "cd /directory ; tar -xzpvf - " > > if you want to spend more time, use "j" rather than "z" in the > options. > > YMMV, but I have been convinced that, apart from specific use cases > with text only documents or documents known to compress quickly/ > well, that compression prior to transfer may waste more time than > it saves. > > This said, if someone has a parallel hack of gzip or similar we can > pipe through, by all means, I would be happy to try it. But it > would have to be pretty darned efficient. > > 100MB/s means 1 byte transmitted,on average, in 10 nanoseconds. > Which means for compression to be meaningful, you would need to > compute for less time than that to increase the information > density. Put another way, 1 MB takes about 10 ms to send over a > gigabit link. For compression to be meaningful, you need to > compress this 1MB in far less than 10 ms and still transmit it in > that time. Assuming that any compression algorithm has to walk > through data at least once, A 1 GB/s memory subsystem takes about > 1 ms to walk through this data at least once, so you need as few > passes as possible through the data set to construct the compressed > representation, as you will still have on the order of 1E+5 bytes > to send. > > I am not saying it is hopeless, just hard for complex compression > schemes (bzip2, etc). When we get enough firepower in the CPU (or > maybe GPU ... hmmmm) the situation may improve. > > GPU as a compression engine? Interesting ... > > Joe > >> Cheers >> Carsten > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics LLC, > email: landman@scalableinformatics.com > web : http://www.scalableinformatics.com > http://jackrabbit.scalableinformatics.com > phone: +1 734 786 8423 x121 > fax : +1 866 888 3112 > cell : +1 734 612 4615 > From landman at scalableinformatics.com Fri Oct 3 08:59:53 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <0D74B691-5A4B-4D0D-98B5-DDCBE062F729@xs4all.nl> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl> <48E606B0.3050708@aei.mpg.de> <48E63E27.6060701@scalableinformatics.com> <0D74B691-5A4B-4D0D-98B5-DDCBE062F729@xs4all.nl> Message-ID: <48E64179.8070808@scalableinformatics.com> Vincent Diepeveen wrote: > The question is Joe, > > Why are you storing it uncompressed? Lets see now... Riddle me this Vincent: What happens when you compress uncompressable binary data? Like RPMs, or .debs? Or pdf's with images, or images which have already been compressed? Any idea? -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From diep at xs4all.nl Fri Oct 3 09:00:16 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] GPU (was Accelerator) for data compressing In-Reply-To: <48E63E27.6060701@scalableinformatics.com> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl> <48E606B0.3050708@aei.mpg.de> <48E63E27.6060701@scalableinformatics.com> Message-ID: <6D3E24D4-714E-48A3-82D7-AFF1E98AC37F@xs4all.nl> On Oct 3, 2008, at 5:45 PM, Joe Landman wrote: > > GPU as a compression engine? Interesting ... > > Joe > For great compression, it's rather hard to get that to work. With a lot of RAM some clever guys manage. GPU has a lot of stream processors, yet little RAM a stream processor. Additionally they all have to execute the same code at a bunch of SP's at the same time. So there is a big need for some real clever new algorithm there, as the lack of RAM is gonna hurt really bigtime. Would be a mighty interesting study to get something to work there. It allows real complicated mathematical functions for PPM functionality. What's in that GPU soon will be in a CPU anyway, so it benefits the entire planet a thing like that. Where can i ask for funding? Vincent >> Cheers >> Carsten > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics LLC, > email: landman@scalableinformatics.com > web : http://www.scalableinformatics.com > http://jackrabbit.scalableinformatics.com > phone: +1 734 786 8423 x121 > fax : +1 866 888 3112 > cell : +1 734 612 4615 > From diep at xs4all.nl Fri Oct 3 09:13:46 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E64179.8070808@scalableinformatics.com> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl> <48E606B0.3050708@aei.mpg.de> <48E63E27.6060701@scalableinformatics.com> <0D74B691-5A4B-4D0D-98B5-DDCBE062F729@xs4all.nl> <48E64179.8070808@scalableinformatics.com> Message-ID: <83F10C80-F97B-48D2-A586-260DCD4BD9A0@xs4all.nl> Well that's obvious what happens there, that's not really new, already the old good pkzip was storing files it couldn't compress uncompressed. Note the type of files you mention you can compress quite well still with PPM, which really is nothing new anymore. All the old zippers are LZ77/Huffman/RLE based, so they miss the big picture always. JPG is a lot tougher to compress well, but well as i said, there is always the cheap option of a filesystem storing that uncompressesd :) What gets shipped however most over the networks is XML data from all those media networks. Now that compresses *real well*. Just compress it single time and keep it compressed :) Vincent On Oct 3, 2008, at 5:59 PM, Joe Landman wrote: > Vincent Diepeveen wrote: >> The question is Joe, >> Why are you storing it uncompressed? > > Lets see now... > > Riddle me this Vincent: What happens when you compress > uncompressable binary data? Like RPMs, or .debs? Or pdf's with > images, or images which have already been compressed? > > Any idea? > > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics LLC, > email: landman@scalableinformatics.com > web : http://www.scalableinformatics.com > http://jackrabbit.scalableinformatics.com > phone: +1 734 786 8423 x121 > fax : +1 866 888 3112 > cell : +1 734 612 4615 From mathog at caltech.edu Fri Oct 3 09:55:48 2008 From: mathog at caltech.edu (David Mathog) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Re: Accelerator for data compressing Message-ID: Carsten Aulbert wrote > We have a Gbit network, i.e. for us this test is a null test, since it > takes 7-zip close to 5 minutes to compress the data set of 311 MB which > we could blow over the network in less than 5 seconds, i.e. in this case > tar would be our favorite ;) Many compression programs have a parameter to adjust how hard it tries to squeeze things, offering a trade off between speed and compression. For 7za you want to look at the -m switch, see: http://www.bugaco.com/7zip/MANUAL/switches/method.htm That looks a little old, on my system this file is: /usr/share/doc/p7zip/MANUAL/switches/method.htm Depending on what you are sending, and it all depends on that, you might get some speed up with a simple run length encoding (for data that tends to be in long blocks), word or character encoding (for highly repetitive data, like DNA), or not be able to do any better no matter what compression method you use (random bits.) One of these is always rate limiting in data distribution: read, compression, transmit, receive, decompress, write. Ignoring compression, which of these hardware parameters is rate limiting in your case? If it is read or write you can speed things up by adopting a different storage configuration (details vary, but underneath it will come down to spooling data off of/onto N disks at once, instead of just 1, to multiply the disk IO by N.) Regards, David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From prentice at ias.edu Fri Oct 3 10:53:01 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E5E588.1050508@aei.mpg.de> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de> Message-ID: <48E65BFD.1050607@ias.edu> Carsten Aulbert wrote: > > Hi all > > > > Bill Broadley wrote: >> >> Another example: >> >> http://bbs.archlinux.org/viewtopic.php?t=11670 >> >> >> >> 7zip compress: 19:41 >> >> Bzip2 compress: 8:56 >> >> Gzip compress: 3:00 >> >> >> >> Again 7zip is a factor of 6 and change slower than gzip. > > > > Have you looked into threaded/parallel bzip2? > > If you can parallelize compression, has anyone done any work using a GPU to do this? Maybe there's a CUDA-fied gzip/bzip2/7sip out there. IF that was the case, a cheap video card could do the trick, assuming the machines in question have slots/space for such a video card. I'm googling now, but not finding anything. :( It was just a thought... -- Prentice From award at uda.ad Fri Oct 3 11:06:02 2008 From: award at uda.ad (Alan Ward) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Streamlined standard Linux installation Message-ID: Hi. Several days ago there was a thread on using a streamlined standard Linux distribution. These days I have been fiddling with the new Debian Lenny version (still beta, but it seems not for long). A "no frills, no X" install with vi, aptitude and a few other rescue tools fits comfortably into about 300 Mbytes on an old USB pendrive. Setting a maximum of 512 MBytes for a basic cluster node setup seems feasable for some applications. Boot time from GRUB to login prompt is 22s on a 6-year-old Celeron laptop. And no - no OpenOffice on this one! ;-) Cheers, -Alan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081003/77ab5b45/attachment.html From bill at cse.ucdavis.edu Fri Oct 3 11:24:50 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <48E3E69B.4030202@tuffmail.us> References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com> <48E3BA69.2000405@ias.edu> <1848483940.20081001201911@gmx.net> <88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com> <48E3E69B.4030202@tuffmail.us> Message-ID: <48E66372.8030707@cse.ucdavis.edu> Alan Louis Scheinine wrote: > NiftyOMPI Mitch wrote >> QDR is interesting... in all likelyhood the >> QDR game will be optical for any link further away than a single rack. >> Once IB goes optical there will be a lot of reason to install IB in >> machine rooms and campus sites that are just out of reach today. > > When will IB go optical at a reasonable price? Folks I know in the IB industry mentioned: * QDR price life cycle should be similar to the DDR price life cycle. * QDR Cable lengths over copper are rather poor * fiber transceivers are finally starting to catch up. QDR over fiber should be "reasonably priced", here's hoping that the days of Myrinet 250MB/sec optical cables will return. Corrections/comments welcome. From atchley at myri.com Fri Oct 3 11:47:05 2008 From: atchley at myri.com (Scott Atchley) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <48E66372.8030707@cse.ucdavis.edu> References: <48E39782.4020006@ias.edu> <48E3B1F4.3000702@scalableinformatics.com> <48E3BA69.2000405@ias.edu> <1848483940.20081001201911@gmx.net> <88815dc10810011318tf540ef2m51ce4692b7c042a9@mail.gmail.com> <48E3E69B.4030202@tuffmail.us> <48E66372.8030707@cse.ucdavis.edu> Message-ID: On Oct 3, 2008, at 2:24 PM, Bill Broadley wrote: > QDR over fiber should be "reasonably priced", here's hoping that the > days of > Myrinet 250MB/sec optical cables will return. > > Corrections/comments welcome. I am not in sales and I have no access to pricing besides our list prices, but I am told that optical QSFP is getting very close to CX-4 when you price NICs and cables (QSFP NICs cost less than CX-4 NICs, the cables are more, the total package is very close). Scott From coutinho at dcc.ufmg.br Fri Oct 3 12:06:13 2008 From: coutinho at dcc.ufmg.br (Bruno Coutinho) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Streamlined standard Linux installation In-Reply-To: References: Message-ID: 2008/10/3 Alan Ward > > Hi. > > Several days ago there was a thread on using a streamlined standard Linux > distribution. These days I have been fiddling with the new Debian Lenny > version (still beta, but it seems not for long). > > A "no frills, no X" install with vi, aptitude and a few other rescue tools > fits comfortably into about 300 Mbytes on an old USB pendrive. Setting a > maximum of 512 MBytes for a basic cluster node setup seems feasable for some > applications. > > Boot time from GRUB to login prompt is 22s on a 6-year-old Celeron laptop. > On ubuntu server you can get less. Thanks to upstart. The only problem is that login screen gets polluted with starting services output. > > And no - no OpenOffice on this one! ;-) > > Cheers, > -Alan > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081003/e71b9dfc/attachment.html From lindahl at pbm.com Fri Oct 3 12:58:16 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48E5E340.6080408@cse.ucdavis.edu> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> Message-ID: <20081003195816.GB16760@bx9.net> On Fri, Oct 03, 2008 at 02:17:52AM -0700, Bill Broadley wrote: > Er, that makes no sense to me. You aren't going to end up with a smaller > file by encoding a file less efficiently. I often find that floating-point data doesn't compress much, but that ASCII representations of the same data compresses to a file smaller than the binary one. This is with gzip/bzip2, so we're talking dictionary-based compression -- and it's easy to imagine why this might be true. I've never seen any float-point-oriented compressor like the ones specialized for photos (jpeg, etc.) -- greg From lindahl at pbm.com Fri Oct 3 13:05:48 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: References: <48E39782.4020006@ias.edu> Message-ID: <20081003200548.GC16760@bx9.net> On Fri, Oct 03, 2008 at 10:00:05AM -0400, Mark Hahn wrote: > qdr would be 40 Gb (raw data rate, right? so 4 GB/s before any sort > of packet overhead, etc.) I don't really see why that's a problem - > even a memory-constrained current-gen Intel box has about twice that much > memory bandwidth available. AMD or next-gen-Intel will > be even less constrained. Let's say that I'm sending data which is in a cache. So when the HCA does the DMA operation, all the bytes have to be flushed from cache to main memory, and then transferred from main memory to the HCA. And the system isn't idle while you do this, so these transfers are less efficient than you might think. Of course, your "latency and bandwidth" benchmark won't see this problem, because it only uses a single core, and it sends the same buffer over and over without touching it. -- greg From atp at piskorski.com Fri Oct 3 13:05:49 2008 From: atp at piskorski.com (Andrew Piskorski) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] MonetDB's lightweight compression; Re: Accelerator for data compressing In-Reply-To: <48E606B0.3050708@aei.mpg.de> References: <48E606B0.3050708@aei.mpg.de> Message-ID: <20081003200549.GB51913@piskorski.com> On Fri, Oct 03, 2008 at 01:49:04PM +0200, Carsten Aulbert wrote: > No, quite on the contrary. I would like to use a compressor within a > pipe to increase the throughput over the network, i.e. to get around the > ~ 120 MB/s limit. Carsten, it is probably not directly relevant to you, but you may want to check out MonetDB, particularly their newer "X100" bleeding edge R&D version. Among other things, they've published papers with lots of interesting detail on using super-lightweight software compression to greatly increase database disk IO bandwith. Their main software tricks for faster disk IO seemed to be: One, EXTREMELY lightweight compression schemes - basically table lookups designed to be as cpu friendly as posible. Two, keep the data compressed in RAM as well so that you can cache more of the data, and indeed keep it the compressed until as late in the CPU processing pipeline as possible. From what I remember, MonetDB/X100 actually does all decompression solely in the CPU cache, inline with query processing. Back c. July 2005, a Sandor Heman, one of the MonetDB guys, looked at zlib, bzlib2, lzrw, and lzo, to improve database disk IO bandwith, and claimed that: "... in general, it is very unlikely that we could achieve any bandwidth gains with these algorithms. LZRW and LZO might increase bandwidth on relatively slow disk systems, with bandwidths up to 100MB/s, but this would induce high processing overheads, which interferes with query execution. On a fast disk system, such as our 350MB/s 12 disk RAID, all the generic algorithms will fail to achieve any speedup." http://www.google.com/search?q=MonetDB+LZO+Heman&btnG=Search http://homepages.cwi.nl/~heman/downloads/msthesis.pdf "Super-Scalar Database Compression between RAM and CPU Cache" Back in 2006, the cool new X100 features were not released in MonetDB proper (which is Open Source), but by now that may have changed. Lots of links: http://www.bestechvideos.com/2008/02/21/monetdb-x100-a-very-fast-column-store ftp://ftp.research.microsoft.com/pub/debull/A05june/issue1.htm "MonetDB/X100 - A DBMS In The CPU Cache" http://www.monetdb.nl/ http://homepages.cwi.nl/~mk/MonetDB/ http://sourceforge.net/projects/monetdb/ http://homepages.cwi.nl/~boncz/x100.html http://www.jsequeira.com/blog/2006/01/17.html -- Andrew Piskorski http://www.piskorski.com/ From landman at scalableinformatics.com Fri Oct 3 13:30:21 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <20081003200548.GC16760@bx9.net> References: <48E39782.4020006@ias.edu> <20081003200548.GC16760@bx9.net> Message-ID: <48E680DD.3@scalableinformatics.com> Greg Lindahl wrote: > Of course, your "latency and bandwidth" benchmark won't see this > problem, because it only uses a single core, and it sends the same > buffer over and over without touching it. Yup. We have a storage test using MPI that has each node generate random numbers, and passes the buffer around the nodes for IO. The idea is of course to get a little closer to actual use cases ... -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From Shainer at mellanox.com Fri Oct 3 16:55:57 2008 From: Shainer at mellanox.com (Gilad Shainer) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? Message-ID: <9FA59C95FFCBB34EA5E42C1A8573784F0157FAF6@mtiexch01.mti.com> Prentice wrote: > In the ongoing saga that is my new cluster, we were just told > today that Cisco is no longer manufacturing DDR IB cables, > which we, uhh, need. > > Has DDR IB gone the way of the dodo bird and been supplanted by QDR? > > If so, why would anyone spec a brand new cluster with DDR? > DDR is still the most used IB solution. I can not speak on Cisco behalf, but there are many other vendors selling DDR cables - Voltaire, Mellanox, Gore, QLogic, Molex and many others, including integrators. Gilad From kyron at neuralbs.com Fri Oct 3 17:27:43 2008 From: kyron at neuralbs.com (Eric Thibodeau) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: References: <48E2C6C1.60108@neuralbs.com> <48E3EB2F.2070102@neuralbs.com> Message-ID: <48E6B87F.6060304@neuralbs.com> Bogdan Costescu wrote: > On Wed, 1 Oct 2008, Eric Thibodeau wrote: > >> the NFS root approach only does changes on the head node and changed >> files don't need to be propagated and are accessed on a as-needed >> basis, this might have significant impacts on large deployments > > NFS-root doesn't scale too well, the implementation of NFS in Linux is > quite chatty. Someone else responded to this, and expect even more scalability performance with NFSv4...lots more ;) > >> I'll take your word for it that they have a version tracking mechanism. > > Take my word that if you're going into larger installations with the > least amount of non-homogeneity you'll want to at least read about, if > not use, such mechanisms :-) > Well, "large" is relative, if I have a 32k core cluster with all identical HW, all I really need for this "large" installation is a single image.. It's the diversity that kills that then requires tight version/config tracking, even if all the images are managed from a single point..but that's trivial common sense ;) Eric From coutinho at dcc.ufmg.br Fri Oct 3 18:33:59 2008 From: coutinho at dcc.ufmg.br (Bruno Coutinho) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] GPU (was Accelerator) for data compressing In-Reply-To: <6D3E24D4-714E-48A3-82D7-AFF1E98AC37F@xs4all.nl> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <48E5E588.1050508@aei.mpg.de> <5FBA5A49-BEF5-4C59-86A1-39EEED4578CC@xs4all.nl> <48E606B0.3050708@aei.mpg.de> <48E63E27.6060701@scalableinformatics.com> <6D3E24D4-714E-48A3-82D7-AFF1E98AC37F@xs4all.nl> Message-ID: 2008/10/3 Vincent Diepeveen > > On Oct 3, 2008, at 5:45 PM, Joe Landman wrote: > >> >> GPU as a compression engine? Interesting ... >> >> Joe >> >> > For great compression, it's rather hard to get that to work. > With a lot of RAM some clever guys manage. > > GPU has a lot of stream processors, yet little RAM a stream processor. A tesla have 4GB ram for 240 stream processors. That gives ~16MB for processor. But SMID arrays (stream multiptocessors in CUDA) need a lot of threads to tolerate memory latencies and not get idle. Something of the order of 512 threads per multiprocessor (tesla have 30), so we have 4GB divided among 15360, that gives 273k per thread. > > Additionally they all have to execute the same code at a bunch of SP's at > the same time. > > So there is a big need for some real clever new algorithm there, > as the lack of RAM is gonna hurt really bigtime. You need to get data from machine main memory, compress and send results back several times. The bandwidth of pci express today is 8GB/s, so this is the maximum data rate a gpu can compress. You can use some tricks like computation and i/o (to main memory) parallelization, but will be constrained to 8GB/s anyway. > > > Would be a mighty interesting study to get something to work there. It > allows real > complicated mathematical functions for PPM functionality. > > What's in that GPU soon will be in a CPU anyway, so it benefits the entire > planet a thing like that. This will be interesting. Removing pci express from the path, the gpu will become really a parallel coprocessor. With the pressure for a more flexible programming model caused by Larabee, the coprocessor could become as programable as the main processor and we will have a processor with few big cores for serial workloads and many cores for parallel workloads. > > > Where can i ask for funding? > > Vincent > > > > Cheers >>> Carsten >>> >> >> -- >> Joseph Landman, Ph.D >> Founder and CEO >> Scalable Informatics LLC, >> email: landman@scalableinformatics.com >> web : http://www.scalableinformatics.com >> http://jackrabbit.scalableinformatics.com >> phone: +1 734 786 8423 x121 >> fax : +1 866 888 3112 >> cell : +1 734 612 4615 >> >> > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081003/9eee6422/attachment.html From xclski at yahoo.com Fri Oct 3 22:36:56 2008 From: xclski at yahoo.com (Ellis Wilson) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] GPU (was Accelerator) for data compressing Message-ID: <499045.95838.qm@web37907.mail.mud.yahoo.com> Bruno Coutinho wrote: > You need to get data from machine main memory, compress and send results > back several times. > The bandwidth of pci express today is 8GB/s, so this is the maximum data > rate a gpu can compress. > You can use some tricks like computation and i/o (to main memory) > parallelization, but will be constrained to 8GB/s anyway. Check out PCI Express 2 then, you'll get 16GB/s bandwidth but I'm not sure all the GPU's are moved that direction yet. Ellis From peter.st.john at gmail.com Sat Oct 4 13:32:57 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] OT: the ongoing computer chess championship in China Message-ID: The World Computer Chess Championship is onging in China (part of the World Mind Sports thing). The leaders, about half-way through , are a 40 core cluster from the US and a "8 x 4GHz" machine from the UK, with 4.5 out of 5. There are 10 participants, it's a nine-round round robin. Cluster Toga, also a cluster, is notable for drawing both of the leaders. >From http://www.chessbase.com/newsdetail.asp?newsid=4935 Rybka USA Cluster, 40 cores 3.5 4 5.0 4.50 Hiarcs GBR Intel Skulltrail, 8 x 4Ghz 3.5 4 3.0 2.50 Sjeng BEL Intel Core 2, 4 x 2.8Ghz 2.5 4 4.0 2.00 Junior ISR Intel Dunnington, 12 x 2.67Ghz 2.5 3 3.0 2.50 The Baron NLD AMD Opteron 270, 4 x 2Ghz 2.0 4 7.0 1.75 Jonny GER Cluster, 16 cores 1.0 4 12.0 2.50 Cluster Toga GER 24 cores 1.0 3 9.5 3.50 Shredder GER Intel Core 2, 8 x 3.16Ghz 1.0 3 8.0 2.25 Falcon ISR Intel Core 2, 2 x 2.1Ghz 1.0 3 6.0 0.00 Mobile Chess CHN Nokia 6120c 0.0 4 9.0 0.00 Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081004/29ba23ab/attachment.html From peter.st.john at gmail.com Sat Oct 4 14:38:43 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] OT: the ongoing computer chess championship in China In-Reply-To: References: Message-ID: That's the most remarkable crosstable I've ever seen (in Shahrokh's link). The players are essentially well-ordered; the top beat everyone, 2nd beat everyone except 1st, 3rd beat everyone except 1 and 2, etc, all the way down. It makes a upper-trianglular matrix. Remarkable. I used to give 13 stones to ManyFaces on my own PC. I suspect this beast, on that cluster, could give me several. I still give 6 to programs running on just PCs. (3 stones is about knight odds, in chess terms) Peter On 10/4/08, Shahrokh Mortazavi wrote: > > > > On a related note, at the same tournament, the cluster version of "Many > Face of Go" has won the Gold medal in both 9X9 and 19X19 categories with no > defeats. Go is notoriously harder than Chess for computers to play as you > might now. The results are here: > > http://www.grappa.univ-lille3.fr/icga/tournament.php?id=181 > > incidentally, the code was running on a Windows HPC Server 2008, 32 core > intel cluster (we helped the author port/tune his MPI code a little). > > For some info re challenges in computer Go see: > > http://en.wikipedia.org/wiki/Computer_go > > > Shahrokh > > > > From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On > Behalf Of Peter St. John > Sent: Saturday, October 04, 2008 1:33 PM > To: beowulf@beowulf.org > Subject: [Beowulf] OT: the ongoing computer chess championship in China > > > The World Computer Chess Championship is onging in China (part of the World > Mind Sports thing). The leaders, about half-way through , are a 40 core > cluster from the US and a "8 x 4GHz" machine from the UK, with 4.5 out of 5. > There are 10 participants, it's a nine-round round robin. Cluster Toga, also > a cluster, is notable for drawing both of the leaders. > > >From http://www.chessbase.com/newsdetail.asp?newsid=4935 > Rybka > USA > Cluster, 40 cores 3.5 4 5.0 4.50 > Hiarcs > GBR > Intel Skulltrail, 8 x 4Ghz 3.5 4 3.0 2.50 > Sjeng > BEL > Intel Core 2, 4 x 2.8Ghz 2.5 4 4.0 2.00 > Junior > ISR > Intel Dunnington, 12 x 2.67Ghz 2.5 3 3.0 2.50 > The Baron > NLD > AMD Opteron 270, 4 x 2Ghz 2.0 4 7.0 1.75 > Jonny > GER > Cluster, 16 cores 1.0 4 12.0 2.50 > Cluster Toga > GER > 24 cores 1.0 3 9.5 3.50 > Shredder > GER > Intel Core 2, 8 x 3.16Ghz 1.0 3 8.0 2.25 > Falcon > ISR > Intel Core 2, 2 x 2.1Ghz 1.0 3 6.0 0.00 > Mobile Chess > CHN > Nokia 6120c 0.0 4 9.0 0.00 > > > Peter > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081004/3d9c4eeb/attachment.html From coutinho at dcc.ufmg.br Sat Oct 4 15:26:48 2008 From: coutinho at dcc.ufmg.br (Bruno Coutinho) Date: Wed Nov 25 01:07:51 2009 Subject: [Beowulf] GPU (was Accelerator) for data compressing In-Reply-To: <499045.95838.qm@web37907.mail.mud.yahoo.com> References: <499045.95838.qm@web37907.mail.mud.yahoo.com> Message-ID: 2008/10/4 Ellis Wilson > Bruno Coutinho wrote: > > You need to get data from machine main memory, compress and send results > > back several times. > > The bandwidth of pci express today is 8GB/s, so this is the maximum data > > rate a gpu can compress. > > You can use some tricks like computation and i/o (to main memory) > > parallelization, but will be constrained to 8GB/s anyway. > > Check out PCI Express 2 then, you'll get 16GB/s bandwidth but I'm not > sure all the GPU's are moved that direction yet. > This is in a x32 slot (that I never saw) in a typical x16 slot pci-e 1.0 make 4GB/s and 2.0 makes 8GB/s. All nvidia cards from 9 series and ATI from 3xxx series have pci express 2.0. In chipsets only the newest ones like intel P45 have pci-e 2.0. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081004/a10b4ac4/attachment.html From hunting at ix.netcom.com Sat Oct 4 23:51:21 2008 From: hunting at ix.netcom.com (Michael Huntingdon) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Has DDR IB gone the way of the Dodo? In-Reply-To: <48E66372.8030707@cse.ucdavis.edu> Message-ID: <4504CA87400144E09DAD0D6F782D0554@lucky13> And once thought to be so the "C" word....(ummmm commercial) Michael Huntingdon Systems Performance Consultants Higher Education Account Manager (408) 294-6811 -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Bill Broadley Sent: Friday, October 03, 2008 11:25 AM To: alscheinine@tuffmail.us Cc: NiftyOMPI Mitch; Beowulf Mailing List Subject: Re: [Beowulf] Has DDR IB gone the way of the Dodo? Alan Louis Scheinine wrote: > NiftyOMPI Mitch wrote >> QDR is interesting... in all likelyhood the >> QDR game will be optical for any link further away than a single rack. >> Once IB goes optical there will be a lot of reason to install IB in >> machine rooms and campus sites that are just out of reach today. > > When will IB go optical at a reasonable price? Folks I know in the IB industry mentioned: * QDR price life cycle should be similar to the DDR price life cycle. * QDR Cable lengths over copper are rather poor * fiber transceivers are finally starting to catch up. QDR over fiber should be "reasonably priced", here's hoping that the days of Myrinet 250MB/sec optical cables will return. Corrections/comments welcome. _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From eugen at leitl.org Sun Oct 5 02:21:33 2008 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Connectionists: NIPS workshop CfP: Parallel Implementations of Learning Algorithms Message-ID: <20081005092133.GJ11968@leitl.org> ----- Forwarded message from Dave_Touretzky@cs.cmu.edu ----- From: Dave_Touretzky@cs.cmu.edu Date: Sun, 05 Oct 2008 00:34:08 -0400 To: connectionists@cs.cmu.edu Subject: Connectionists: NIPS workshop CfP: Parallel Implementations of Learning Algorithms X-Mailer: MH-E 7.4.2; nmh 1.2-20070115cvs; XEmacs 21.5 (beta27) NIPS08 Workshop Call for Posters: Parallel Implementations of Learning Algorithms: What Have You Done For Me Lately? Overview: Interest in parallel hardware concepts, including multicore, specialized hardware, and multimachine, has recently increased as researchers have looked to scale up their concepts to large, complex models and large datasets. In this workshop, a panel of invited speakers will present results of investigations into hardware concepts for accelerating a number of different learning and simulation algorithms. Additional contributions will be presented in poster spotlights and a poster session at the end of the one-day workshop. Our intent is to provide a broad survey of the space of hardware approaches in order to capture the current state of activity in this venerable domain of study. Approaches to be covered include silicon, FPGA, and supercomputer architectures, for applications such as Bayesian network models of large and complex domains, simulations of cortex and other brain structures, and large-scale probabilistic algorithms. Potential participants include researchers interested in accelerating their algorithms to handle large datasets, and systems designers providing such hardware solutions. The oral presentations will include plenty of time for questions and discussion, and the poster session at the end of the workshop will afford further opportunities for interaction among workshop participants. Confirmed Speakers: - David Andersen, Carnegie Mellon University Using a Fast Array of Wimpy Nodes. - Michael Arnold, Salk Institute Multi-Scale Modeling in Neuroscience - Dan Hammerstrom, Portland State University Nanoelectronics: The Original Positronic Matrix? - Kenneth Rice, Clemson University A Neocortex-Inspired Cognitive Model on the Cray XD1 - Robert Thibadeau, Seagate Research When (And Why) Storage Devices Become Computers - Additional speakers TBA. Important Dates: October 31, 2008 Poster abstract submission deadline November 7, 2008 Notification of acceptance December 12 or 13 Workshop at NIPS How To Submit: Algorithm developers, hardware developers, and researchers building large-scale applications are all encouraged to present their work at the workshop. Send a brief abstract (one page should suffice) in any format describing the approach and results you wish to present in poster form at the meeting to. David S. Touretzky, Carnegie Mellon University, dst@cs.cmu.edu. Submissions are due by October 31; decisions will be announced by November 7. Authors of accepted posters will be asked to give a 2 minute spotlight presentation before the start of the poster session. Workshop Organizing Committee: Robert Thibadeau, Seagate Research Dan Hammerstrom, Portland State University David Touretzky, Carnegie Mellon University Tom Mitchell, Carnegie Mellon University ----- End forwarded message ----- -- Eugen* Leitl leitl http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE From eugen at leitl.org Sun Oct 5 08:30:34 2008 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] FY;) the Helmer cluster Message-ID: <20081005153034.GU11968@leitl.org> A Linux cluster built in a Helmer IKEA cabinet: http://helmer.sfe.se/ Here's another version of it: http://obscuredclarity.blogspot.com/2008/09/24-core-linux-cluster-in-2999-case-from.html Cooling looks a bit suboptimal. From diep at xs4all.nl Sun Oct 5 12:33:46 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] IKEA priced supercomputer - the Helmer cluster In-Reply-To: <20081005153034.GU11968@leitl.org> References: <20081005153034.GU11968@leitl.org> Message-ID: <867A068C-7E3B-4D1B-992E-A6F6840B70A2@xs4all.nl> This is great stuff to read :) Those Swedes can do everything with wood it seems :) Especially the IKEA idea and the height comparision with the Michael Johnson shoe gave me some great laughs :) Maybe you can modify the drawers of the ikea box and then have also real easy maintenance with them. Without kidding, it's a great idea, and even better that it got documented. I should do that too in case i win the lottery and can make a cluster here. Do i see it correct that maintenance an entire mainboard can get shifted to outside? How can maintenance be done? What i kind of miss is how to get the iron backplates where the fans and psu's fit in and how a possible highend network, in my case 2nd hand can be mounted into it. More pictures from how it looks like would be fantastic for those who want to take a look at how to themselves make a bunch of nodes in this manner; simply how it looks like. It seems to me it can store a lot of ATX mainboards in real cheap manner. The way the cooling gets blown into the case doesn't seems very professional to me. Does it limit maintenance to the motherboards? Vincent On Oct 5, 2008, at 5:30 PM, Eugen Leitl wrote: > > A Linux cluster built in a Helmer IKEA cabinet: > > http://helmer.sfe.se/ > > Here's another version of it: > > http://obscuredclarity.blogspot.com/2008/09/24-core-linux-cluster- > in-2999-case-from.html > > Cooling looks a bit suboptimal. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From niftyompi at niftyegg.com Sun Oct 5 17:15:14 2008 From: niftyompi at niftyegg.com (Nifty niftyompi Mitch) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <20081003195816.GB16760@bx9.net> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <20081003195816.GB16760@bx9.net> Message-ID: <20081006001514.GA4502@compegg.wr.niftyegg.com> On Fri, Oct 03, 2008 at 12:58:16PM -0700, Greg Lindahl wrote: > On Fri, Oct 03, 2008 at 02:17:52AM -0700, Bill Broadley wrote: > > > Er, that makes no sense to me. You aren't going to end up with a smaller > > file by encoding a file less efficiently. > > I often find that floating-point data doesn't compress much, but that > ASCII representations of the same data compresses to a file smaller > than the binary one. This is with gzip/bzip2, so we're talking > dictionary-based compression -- and it's easy to imagine why this > might be true. I've never seen any float-point-oriented compressor > like the ones specialized for photos (jpeg, etc.) Floating point numbers not compressing makes sense sense to me. The noise of the LSB and layout would mask redunancy in the exponent and other fields. Since most compression tools are byte (character and string) oriented in their design the ASCII version could expose the right stuff. It does point out a gap in the compression tool suite that I think the hard data folks might look at..... interesting stuff. -- T o m M i t c h e l l Found me a new hat, now what? From gerry.creager at tamu.edu Mon Oct 6 06:11:02 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> References: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> Message-ID: <48EA0E66.6060403@tamu.edu> John Hearns wrote: > > > 2008/10/1 Donald Becker > > > > > It's foreseeable that holding an 8GB install > image in memory will be trivial, but that will be a few years in the > future, not today. And we will need better VM and PTE management to > make > it efficient. > > > > Hmmmm.... can I forsee Puppy Linux HPC Edition ???? > http://www.puppylinux.org/ > > > Being half serious here, is it worth trying to get one of these > slimmed-down distros to the state where it will run an HPC job? > Oh, and in addition to a barebones install for our contemplative > cluster, they have a onebones install which cuts out the GUI (cue more > dog puns). > > http://www.puppylinux.com/pfs/ looks interesting... Starts to sound like Rocks. Which I'll finish installing on a small cluster tomorrow and will form more opinions then. First impressions (really second: I've had another small cluster up with it for 3 years) is that it's seamless for the scientists once they figure out where their apps are. I've been pretty happy with it: It's been mostly "set and forget". Bogdan, we're in a similar position, save I also do some of the science. Our latest cluster serves users in Atmospheric Science, Chemistry, Math, Statistics, Agriculture & Life Sciences, Computer Science, Oceanography and Nuclear (or was that "nuculeer"?) engineering... Most are somewhat astute to computation but not one rises to the point of "computational scientist" in being able to combine extensive computer science/engineering knowledge with their much more important discipline knowledge. Hiding the details from them, but meeting their needs at the same time is a constant challenge. -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From smortaz at exchange.microsoft.com Sat Oct 4 14:08:36 2008 From: smortaz at exchange.microsoft.com (Shahrokh Mortazavi) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] OT: the ongoing computer chess championship in China In-Reply-To: References: Message-ID: On a related note, at the same tournament, the cluster version of "Many Face of Go" has won the Gold medal in both 9X9 and 19X19 categories with no defeats. Go is notoriously harder than Chess for computers to play as you might now. The results are here: http://www.grappa.univ-lille3.fr/icga/tournament.php?id=181 incidentally, the code was running on a Windows HPC Server 2008, 32 core intel cluster (we helped the author port/tune his MPI code a little). For some info re challenges in computer Go see: http://en.wikipedia.org/wiki/Computer_go Shahrokh From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Peter St. John Sent: Saturday, October 04, 2008 1:33 PM To: beowulf@beowulf.org Subject: [Beowulf] OT: the ongoing computer chess championship in China The World Computer Chess Championship is onging in China (part of the World Mind Sports thing). The leaders, about half-way through , are a 40 core cluster from the US and a "8 x 4GHz" machine from the UK, with 4.5 out of 5. There are 10 participants, it's a nine-round round robin. Cluster Toga, also a cluster, is notable for drawing both of the leaders. >From http://www.chessbase.com/newsdetail.asp?newsid=4935 Rybka USA Cluster, 40 cores 3.5 4 5.0 4.50 Hiarcs GBR Intel Skulltrail, 8 x 4Ghz 3.5 4 3.0 2.50 Sjeng BEL Intel Core 2, 4 x 2.8Ghz 2.5 4 4.0 2.00 Junior ISR Intel Dunnington, 12 x 2.67Ghz 2.5 3 3.0 2.50 The Baron NLD AMD Opteron 270, 4 x 2Ghz 2.0 4 7.0 1.75 Jonny GER Cluster, 16 cores 1.0 4 12.0 2.50 Cluster Toga GER 24 cores 1.0 3 9.5 3.50 Shredder GER Intel Core 2, 8 x 3.16Ghz 1.0 3 8.0 2.25 Falcon ISR Intel Core 2, 2 x 2.1Ghz 1.0 3 6.0 0.00 Mobile Chess CHN Nokia 6120c 0.0 4 9.0 0.00 Peter From i.n.kozin at googlemail.com Sun Oct 5 13:41:20 2008 From: i.n.kozin at googlemail.com (Igor Kozin) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] FY;) the Helmer cluster In-Reply-To: <20081005153034.GU11968@leitl.org> References: <20081005153034.GU11968@leitl.org> Message-ID: lol the guy surely is very ambitious - 4 PFLOPS for $0.9M http://helmer3.sfe.se/ 2008/10/5 Eugen Leitl > > A Linux cluster built in a Helmer IKEA cabinet: > > http://helmer.sfe.se/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081005/c000ae51/attachment.html From jason at acm.org Mon Oct 6 09:50:30 2008 From: jason at acm.org (Jason Riedy) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Re: Accelerator for data compressing In-Reply-To: <20081003195816.GB16760@bx9.net> (Greg Lindahl's message of "Fri, 3 Oct 2008 12:58:16 -0700") References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48E5712E.7070504@cse.ucdavis.edu> <692A035F-D1C2-4F49-9DF5-A04C29B30233@xs4all.nl> <48E5E340.6080408@cse.ucdavis.edu> <20081003195816.GB16760@bx9.net> Message-ID: <87y711wtux.fsf@sparse.dyndns.org> And Greg Lindahl writes: > I often find that floating-point data doesn't compress much, but that > ASCII representations of the same data compresses to a file smaller > than the binary one. However, some care is necessary if you're using decimal output rather than hex. I've run into problems with the identical decimal (in ASCII) number converting to different binary numbers. The difference is only in the last bit, but that ends up being important for my uses. Sometimes the cause is an actual bug in the conversion, but most times the decimal version provided *more* than the number of decimal digits expected for the binary format. For example, some people store 20 digits rather than 17 for decimal. Different systems handle the extra digits differently. sigh. And remember that 754-1985 did *not* require correct conversion for all numbers, so some people have not provided correct conversion. (The new 754-2008 *does* require correct conversion.) But there's C99-style hex output. Makes my life much easier. And conversions are relatively fast, unlike the conversions to/from decimal. Jason From kyron at neuralbs.com Mon Oct 6 12:55:42 2008 From: kyron at neuralbs.com (Eric Thibodeau) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] FY;) the Helmer cluster In-Reply-To: References: <20081005153034.GU11968@leitl.org> Message-ID: <48EA6D3E.8070804@neuralbs.com> Igor Kozin wrote: > lol the guy surely is very ambitious - 4 PFLOPS for $0.9M > http://helmer3.sfe.se/ LOL, I say we put it up at my cottage, electricity is cheap in Quebec and we have a massive lake besides it :) /me bypasses exit heat to warm up house. Now, doesn't anyone seem to recognize a Crayish look to this cluster... :P > > > 2008/10/5 Eugen Leitl > > > > A Linux cluster built in a Helmer IKEA cabinet: > > http://helmer.sfe.se/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081006/7583f027/attachment.html From diep at xs4all.nl Mon Oct 6 15:57:33 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] FY;) the Helmer cluster In-Reply-To: References: <20081005153034.GU11968@leitl.org> Message-ID: Nah if you just want to run 1 application at 4 pflop that should be no problem to design some special hardware for it for $0.9M Vincent On Oct 5, 2008, at 10:41 PM, Igor Kozin wrote: > lol the guy surely is very ambitious - 4 PFLOPS for $0.9M > http://helmer3.sfe.se/ > > > 2008/10/5 Eugen Leitl > > A Linux cluster built in a Helmer IKEA cabinet: > > http://helmer.sfe.se/ > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From Dan.Kidger at quadrics.com Tue Oct 7 00:34:22 2008 From: Dan.Kidger at quadrics.com (Dan.Kidger@quadrics.com) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> Message-ID: <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> Andrea, MPI is of course used by many applications running on commercial clusters. Two obvious examples are computational chemistry by the drug companies and computational fluid dynamics (CFD) for aerospace companies and F1 design teams. These are all long-term 'traditional' uses of MPI for scientific/engineering codes. Is this what you are asking? Or are you thinking of non-traditional uses in say computational finance or gaming sites? Daniel ------------------------------------------------------------- Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Mobile: +44 (0)779 209 1851 Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519 ----------------------- www.quadrics.com -------------------- -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Andrea Di Blas Sent: 13 August 2008 00:37 To: beowulf@beowulf.org Subject: [Beowulf] large MPI adopters hello, I am curious about what companies, besides the national labs of course, use any implementation of MPI to support large applications of any kind, whether only internally (like mapreduce for google, for example) or not. does anybody know of any cases? thank you and best regards, andrea -- Andrea Di Blas, UCSC School of Engineering _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From Bogdan.Costescu at iwr.uni-heidelberg.de Tue Oct 7 04:37:22 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: <48EA0E66.6060403@tamu.edu> References: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> <48EA0E66.6060403@tamu.edu> Message-ID: On Mon, 6 Oct 2008, Gerry Creager wrote: > Bogdan, we're in a similar position, save I also do some of the > science. I (try to) do some of the science too, so the similarity is even better :-) > Hiding the details from them, but meeting their needs at the same > time is a constant challenge. Thanks for sharing all these details, but I would be more interested to know about the technical solutions that you have applied. How do you deal with the various software packages that require specific conditions to run ? Do you have separate clusters for specific purposes/software and only direct users to them ? -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From hearnsj at googlemail.com Tue Oct 7 04:49:19 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: References: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> <48EA0E66.6060403@tamu.edu> Message-ID: <9f8092cc0810070449p24dc1d0bx1e555141a1f0aa62@mail.gmail.com> 2008/10/7 Bogdan Costescu > > Thanks for sharing all these details, but I would be more interested to > know about the technical solutions that you have applied. How do you deal > with the various software packages that require specific conditions to run ? > Do you have separate clusters for specific purposes/software and only direct > users to them ? > > > If we're talking about different interconnects to run specific jobs, then its quite easy to build a cluster with (say) a few racks of Myrinet equipped nodes. You just set up your job scheduler such that scientists needing to use Myrinet can request them. Or by "conditions" do you mean diferent libraries or compilers? John Hearns -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081007/b302f687/attachment.html From diep at xs4all.nl Tue Oct 7 05:58:43 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> Message-ID: <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> Hi Dan, I'd rather guess his question is more: "i want a big list of 10000 companies that are actively using clusters". That list is not there and if it was there it is utmost secretly compiled by some NSA type instance. I'm now speaking more for western Europe, not necessarily for USA: The research that companies perform is usually applied research: the minimum requirement to produce a new product, or when forced at gunpoint by government for safety reasons. For fundamental research usually most computational resources get thrown into battle, which is for nearly 100% sponsored by government be it directly or indirectly. In fact there is entire industries, especially clean energy research, that is 100% paid by government subsidies. Investigations after new products, in 99% of cases get funded by government; usually the name of a big company is behind it, but it's still government dudes who sit in that company doing the research. What matters of course is who pays it. The exception is persons who have their own company and are some sort of big expert at a certain area. Take me. I want to do some research after learning algorithms, at a very high level (so far they didn't even conclude the right value for material for chess, let alone that they were useful at anything other than debugging your code, which is a big important thing by the way). Without getting the help of someone working at an university to run such a thing overnight at a 160-200 cores, i would of course only be able to run it at a single pc. Now i don't get any money for all this from government, so the big question is whether i'm supported by government then or not. What i do know is that all public research here at government level is major shit. This is explainable by the fact that a field is nonstop on the move, only those who have been busy in a field for 10+ years have any clue what happens there and are the first to try something; the PHD's are simply busy with inferior software meanwhile having massive hardware that's idle. A big problem is the nonpublishing of accomplishments; First of all i'm not publishing *how* i'm doing the learning. When some university offers me a job, i might publish things, i'm not getting paid for publications. Why help "the competition"? What might get published is the guy who helps me; one tiny thing of all this he'll publish probably. That's maybe 1/100 of the conclusions drawn. Only the NSA type organisations know as they all spy on the internet, and you BET that all big countries know exactly what i'm doing, they all tap the internet like crazy there. If 1 researcher is real brilliant and CAN need some big computation, there is at least a 100 spies who do know something of that field, and have to interpret the tapped data. What i do not know, as i don't work for any of the N*SA type organisations, is in how far *those* verify things using massive hardware. Most researchers have no clue how big the spying and anti-spying is; most might get total scared if they'd realized how much they get protected. It is easier to steal it than to find an Einstein in your own nation figuring it out. That is the principle that every nation uses; CIA is a very tiny organisation compared to what other nations have, note that the weird thing is that CIA is notorious for violating agreements with other nations (not spying in friendly nations, to give an example; and also passing on information to their companies that they got from the information stream they got from friendly nations spying onto their own companies). The fact that i write this down already means i have not been employed in that field nor am; otherwise i would not even *mention* the word. Or to quote someone who works in that field when i said that there gotta be an awful lot of civil servants in Netherlands busy in that field, as there is 3+ million tourists a year in Netherlands: "we have tourism in Spain also". Yet nothing on the internet is safe, that's the basic problem here. It's too easy to hack internet lines. 1024 bits or 2048 RSA or something that gets used for SSH? 1024 bits is too easy to crack for 1 organisation of each country by simply throwing specialized hardware at it. 2048 bits RSA is a tad harder, but also not a problem. 128 bits AES? No manner to attack it is public known (would only deliver $20k, which is too little anyway, for the risk you take publishing a method). Yet even if there would be some sort of randomized GA-algorithm that needs some sort of Pollard-Rho order O( 2 ^ 0.25n ), then a simple dedicated cpu can already crack it handsdown. Obviously most companies are not busy spamming the net what they do with clusters or do not do. Keeping it secret for their competitors is just too important, yet if you ask me i feel big companies do real little research in areas that do not lead directly to products of them. There might be 1 or 2 exceptions, like oil companies, but the question is in how far researchers there can be seen as employees of that company as they get indirectly sponsored by government whose interest in everything that happens with oil is real big. If you ask me however, way too little fundamental research gets sponsored by government. If you see just in a tiny nation like netherlands; the number of 'researchers' that work for government is just a very small part of the number of professors. It's like 1 researcher for each 2 professors; PHD's not counted. Note this is also what a few studies recently showed. Just the comparision to intelligence agencies which can put into action each one of them quarter of a million to millions of people, and who hire real easy people be it direct or indirect, it is obvious that the saying: "Better well stolen than bad invented", gets written in just too big capitals. As we're speaking of a 1000 individuals in a nation having 16.5 million inhabitants, and only very few of those have time to devote a big part of their time to do research; maybe 5% is? Most are too busy with meetings and other organisational work and posting on the internet. Realize that same nation also has over a 1000 companies with more than 1000 employees and half a million charity organisations. That really lets fundamental research look like something that gets total neglected. Vincent On Oct 7, 2008, at 9:34 AM, wrote: > Andrea, > > MPI is of course used by many applications running on commercial > clusters. > Two obvious examples are computational chemistry by the drug companies > and computational fluid dynamics (CFD) for aerospace companies and > F1 design teams. > > These are all long-term 'traditional' uses of MPI for scientific/ > engineering codes. > > Is this what you are asking? Or are you thinking of non-traditional > uses in say computational finance or gaming sites? > > Daniel > > ------------------------------------------------------------- > Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger@quadrics.com > One Bridewell St., Mobile: +44 (0)779 209 1851 > Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519 > ----------------------- www.quadrics.com -------------------- > > > > > -----Original Message----- > From: beowulf-bounces@beowulf.org [mailto:beowulf- > bounces@beowulf.org] On Behalf Of Andrea Di Blas > Sent: 13 August 2008 00:37 > To: beowulf@beowulf.org > Subject: [Beowulf] large MPI adopters > > hello, > > > I am curious about what companies, besides the national labs of > course, > use any implementation of MPI to support large applications of any > kind, > whether only internally (like mapreduce for google, for example) or > not. > > does anybody know of any cases? > thank you and best regards, > > > andrea > > > > > -- > Andrea Di Blas, UCSC > School of Engineering > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From Bogdan.Costescu at iwr.uni-heidelberg.de Tue Oct 7 06:11:37 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Compute Node OS on Local Disk vs. Ram Disk In-Reply-To: <9f8092cc0810070449p24dc1d0bx1e555141a1f0aa62@mail.gmail.com> References: <9f8092cc0810020525u4fb55634obd9b0100e11e8ff8@mail.gmail.com> <48EA0E66.6060403@tamu.edu> <9f8092cc0810070449p24dc1d0bx1e555141a1f0aa62@mail.gmail.com> Message-ID: On Tue, 7 Oct 2008, John Hearns wrote: > If we're talking about different interconnects to run specific jobs Most of my previous messages in this thread were related to having to run multiple distributions to cater for the needs of specific application software which would not run or would not be supported if run on a distribution other than the one specified by the ISV. The question of interconnect which limits running of specific software is a similar, but different question. I also had to deal with software that uses f.e. GlobalArrays/ARMCI for which only GM but not MX is supported on Myrinet cards. > You just set up your job scheduler such that scientists needing to > use Myrinet can request them. Yes, this is already in use here for some time. > Or by "conditions" do you mean diferent libraries or compilers? Different libraries is partly covered already when talking about different distributions. Different compilers... hmm, I don't want to start on this subject, I've already written enough in this thread :-) But can f.e. someone shed some light as to why PathScale compilers are not supported on Debian or Ubuntu (LTS) ? -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From gerry.creager at tamu.edu Tue Oct 7 06:52:54 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> Message-ID: <48EB69B6.5080602@tamu.edu> I'm top posting, 'cause everyone else is. It's easier. Vincent, there are a lot of folks interested in computer and infrastructure security. They are interested in making it more secure and interested in exploiting its holes. And these are the "good guys". Stick to chess. Don't blather about security, an area you appear to be less aware of than the board games. I don't tell you how to be a chess grand master, and you don't tell me about security from a purely speculative view... OK? Andrea, a lot of scientific computation uses big MPI. Off the top of my head, and based on examples I'm personally aware of (but don't work in directly), seismic and reservoir flow models are big MPI operations. Think "oil company", e.g., Shell, Chevron, Exxon-Mobil, etc., I do know that, in Houston, all employ big clusters running MPI on a regular basis. Quantum chemistry is often run as an MPI job. Weather modeling is a big MPI user, as are ocean and coastal models run at universities, government facilities (not solely national labs) and corporate entities. NASA does a fair bit of MPI work... where did Beowulfery start (Don, that's a cue)? We have a group using MPI applications in genomics research, and others using it for physical chemistry modeling, LHC data operations/reduction/interpretation, and nuclear systems prognostication. The list isn't small: As Dan mentions, the drug companies are big users, although there's a fair bit of openMPI operation in their codes, too. CFD in aerospace, and auto manufacturers. The US military almost certainly does parallel processing involving both message passing and Monte Carlo simulations. I've written code that solves physical and satellite geodesy problems using PVM (but I was young and that was a long time ago; today I'd use MPI); a variant of that was apparently moved to the production world after I changed positions and couldn't support it anymore. I'm not sure to tell you who the biggest corporate users are. It likely depends on one's exposure to even have an opinion. However, for the biggest corporate users I'm aware of in my immediate area, Schlumberger, Exxon-Mobil, Shell Exploration, and Chevron come to mind. This is obviously a very incomplete list. Gerry Vincent Diepeveen wrote: > Hi Dan, > > I'd rather guess his question is more: "i want a big list of 10000 > companies that are actively using clusters". > That list is not there and if it was there it is utmost secretly > compiled by some NSA type instance. > > I'm now speaking more for western Europe, not necessarily for USA: > > The research that companies perform is usually applied research: the > minimum requirement to produce a new product, > or when forced at gunpoint by government for safety reasons. > > For fundamental research usually most computational resources get thrown > into battle, which is for nearly 100% sponsored > by government be it directly or indirectly. > > In fact there is entire industries, especially clean energy research, > that is 100% paid by government subsidies. > Investigations after new products, in 99% of cases get funded by > government; usually the name of a big company is behind it, > but it's still government dudes who sit in that company doing the > research. What matters of course is who pays it. > > The exception is persons who have their own company and are some sort of > big expert at a certain area. Take me. > I want to do some research after learning algorithms, at a very high > level (so far they didn't even conclude the right > value for material for chess, let alone that they were useful at > anything other than debugging your code, which is a big > important thing by the way). > > Without getting the help of someone working at an university to run such > a thing overnight at a 160-200 cores, > i would of course only be able to run it at a single pc. Now i don't get > any money for all this from government, > so the big question is whether i'm supported by government then or not. > > What i do know is that all public research here at government level is > major shit. This is explainable by the fact that a field is nonstop > on the move, only those who have been busy in a field for 10+ years have > any clue what happens there and are the first to try something; > the PHD's are simply busy with inferior software meanwhile having > massive hardware that's idle. A big problem is the nonpublishing of > accomplishments; First of all i'm not publishing *how* i'm doing the > learning. When some university offers me a job, i might publish things, > i'm not getting paid for publications. Why help "the competition"? > What might get published is the guy who helps me; one tiny thing of all > this he'll publish probably. That's maybe 1/100 of the conclusions > drawn. > Only the NSA type organisations know as they all spy on the internet, > and you BET that all big > countries know exactly what i'm doing, they all tap the internet like > crazy there. > > If 1 researcher is real brilliant and CAN need some big computation, > there is at least a 100 spies who do know something of that field, > and have to interpret the tapped data. What i do not know, as i don't > work for any of the N*SA type organisations, is in how far *those* > verify things using massive hardware. > > Most researchers have no clue how big the spying and anti-spying is; > most might get total scared if they'd realized how much they > get protected. > > It is easier to steal it than to find an Einstein in your own nation > figuring it out. > That is the principle that every nation uses; CIA is a very tiny > organisation compared to what other nations have, > note that the weird thing is that CIA is notorious for violating > agreements with other nations (not spying in friendly nations, > to give an example; and also passing on information to their companies > that they got from the information stream they > got from friendly nations spying onto their own companies). > > The fact that i write this down already means i have not been employed > in that field nor am; otherwise i would not even *mention* the word. > > Or to quote someone who works in that field when i said that there gotta > be an awful lot of civil servants in Netherlands busy in that field, > as there is 3+ million tourists a year in Netherlands: > "we have tourism in Spain also". > > Yet nothing on the internet is safe, that's the basic problem here. It's > too easy to hack internet lines. 1024 bits or 2048 RSA or something that > gets used for SSH? > > 1024 bits is too easy to crack for 1 organisation of each country by > simply throwing specialized hardware at it. > 2048 bits RSA is a tad harder, but also not a problem. > > 128 bits AES? > > No manner to attack it is public known (would only deliver $20k, which > is too little anyway, for the risk you take publishing a method). > Yet even if there would be some sort of randomized GA-algorithm that > needs some sort of Pollard-Rho order O( 2 ^ 0.25n ), > then a simple dedicated cpu can already crack it handsdown. > > Obviously most companies are not busy spamming the net what they do with > clusters or do not do. > Keeping it secret for their competitors is just too important, yet if > you ask me i feel big companies do real little > research in areas that do not lead directly to products of them. There > might be 1 or 2 exceptions, like oil companies, > but the question is in how far researchers there can be seen as > employees of that company as they get indirectly > sponsored by government whose interest in everything that happens with > oil is real big. > > If you ask me however, way too little fundamental research gets > sponsored by government. > If you see just in a tiny nation like netherlands; the number of > 'researchers' that work for government is just a very small part of > the number of professors. It's like 1 researcher for each 2 professors; > PHD's not counted. > > Note this is also what a few studies recently showed. > > Just the comparision to intelligence agencies which can put into action > each one of them quarter of a million to millions of people, > and who hire real easy people be it direct or indirect, it is obvious > that the saying: "Better well stolen than bad invented", > gets written in just too big capitals. > > As we're speaking of a 1000 individuals in a nation having 16.5 million > inhabitants, and only very few of those have > time to devote a big part of their time to do research; maybe 5% is? > > Most are too busy with meetings and other organisational work and > posting on the internet. > Realize that same nation also has over a 1000 companies with > more than 1000 employees and half a million charity organisations. > > That really lets fundamental research look like something that gets > total neglected. > > Vincent > > On Oct 7, 2008, at 9:34 AM, > wrote: > >> Andrea, >> >> MPI is of course used by many applications running on commercial >> clusters. >> Two obvious examples are computational chemistry by the drug companies >> and computational fluid dynamics (CFD) for aerospace companies and F1 >> design teams. >> >> These are all long-term 'traditional' uses of MPI for >> scientific/engineering codes. >> >> Is this what you are asking? Or are you thinking of non-traditional >> uses in say computational finance or gaming sites? >> >> Daniel >> >> ------------------------------------------------------------- >> Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger@quadrics.com >> One Bridewell St., Mobile: +44 (0)779 209 1851 >> Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519 >> ----------------------- www.quadrics.com -------------------- >> >> >> >> >> -----Original Message----- >> From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] >> On Behalf Of Andrea Di Blas >> Sent: 13 August 2008 00:37 >> To: beowulf@beowulf.org >> Subject: [Beowulf] large MPI adopters >> >> hello, >> >> >> I am curious about what companies, besides the national labs of course, >> use any implementation of MPI to support large applications of any kind, >> whether only internally (like mapreduce for google, for example) or not. >> >> does anybody know of any cases? >> thank you and best regards, >> >> >> andrea >> >> >> >> >> -- >> Andrea Di Blas, UCSC >> School of Engineering >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From tom.elken at qlogic.com Tue Oct 7 08:01:58 2008 From: tom.elken at qlogic.com (Tom Elken) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: <48EB69B6.5080602@tamu.edu> References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com><3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> <48EB69B6.5080602@tamu.edu> Message-ID: <6DB5B58A8E5AB846A7B3B3BFF1B4315A0269989D@AVEXCH1.qlogic.org> > [mailto:beowulf-bounces@beowulf.org] On Behalf Of Gerry Creager > The list isn't small: As Dan mentions, the drug companies are > big users, > although there's a fair bit of openMPI operation in their codes, too. Gerry, I'm sure you meant OpenMP (standard shared-memory threaded parallelism) So easy for fingers to type "MPI". Even though MPI has far more usage than OpenMP, I have to stick up for it as a former shared-memory advocate from my SGI days and as a member of the OpenMP ARB. -Tom > CFD in aerospace, and auto manufacturers. The US military almost > certainly does parallel processing involving both message passing and > Monte Carlo simulations. I've written code that solves physical and > satellite geodesy problems using PVM (but I was young and that was a > long time ago; today I'd use MPI); a variant of that was apparently > moved to the production world after I changed positions and couldn't > support it anymore. > > I'm not sure to tell you who the biggest corporate users are. > It likely > depends on one's exposure to even have an opinion. However, for the > biggest corporate users I'm aware of in my immediate area, > Schlumberger, > Exxon-Mobil, Shell Exploration, and Chevron come to mind. This is > obviously a very incomplete list. > > > Gerry > >> From: beowulf-bounces@beowulf.org > [mailto:beowulf-bounces@beowulf.org] > >> On Behalf Of Andrea Di Blas > >> Sent: 13 August 2008 00:37 > >> To: beowulf@beowulf.org > >> Subject: [Beowulf] large MPI adopters > >> > >> hello, > >> > >> > >> I am curious about what companies, besides the national > labs of course, > >> use any implementation of MPI to support large > applications of any kind, > >> whether only internally (like mapreduce for google, for > example) or not. > >> > >> does anybody know of any cases? > >> thank you and best regards, > >> > >> > >> andrea > >> > >> > >> > >> > >> -- > >> Andrea Di Blas, UCSC > >> School of Engineering > >> _______________________________________________ > >> Beowulf mailing list, Beowulf@beowulf.org > >> To change your subscription (digest mode or unsubscribe) visit > >> http://www.beowulf.org/mailman/listinfo/beowulf > >> > >> _______________________________________________ > >> Beowulf mailing list, Beowulf@beowulf.org > >> To change your subscription (digest mode or unsubscribe) visit > >> http://www.beowulf.org/mailman/listinfo/beowulf > >> > > > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > -- > Gerry Creager -- gerry.creager@tamu.edu > Texas Mesonet -- AATLT, Texas A&M University > Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 > Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) > visit http://www.beowulf.org/mailman/listinfo/beowulf > From diep at xs4all.nl Tue Oct 7 08:33:52 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: <48EB69B6.5080602@tamu.edu> References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> <48EB69B6.5080602@tamu.edu> Message-ID: On Oct 7, 2008, at 3:52 PM, Gerry Creager wrote: > I'm top posting, 'cause everyone else is. It's easier. > > > Andrea, a lot of scientific computation uses big MPI. Off the top > of my head, and based on examples I'm personally aware of (but > don't work in directly), seismic and reservoir flow models are big > MPI operations. Think "oil company", e.g., Shell, Chevron, Exxon- > Mobil, etc., I do know that, in Houston, all employ big clusters > running MPI on a regular basis. Quantum chemistry is often run as > an MPI job. Weather modeling is a big MPI user, as are ocean and > coastal models run at universities, government facilities (not > solely national labs) and corporate entities. NASA does a fair bit > of MPI work... where did Beowulfery start (Don, that's a cue)? We > have a group using MPI applications in genomics research, and > others using it for physical chemistry modeling, LHC data > operations/reduction/interpretation, and nuclear systems > prognostication. > > The list isn't small: As Dan mentions, the drug companies are big > users, although there's a fair bit of openMPI operation in their > codes, too. CFD in aerospace, and auto manufacturers. The US > military almost certainly does parallel processing involving both > message passing and Monte Carlo simulations. I've written code > that solves physical and satellite geodesy problems using PVM (but > I was young and that was a long time ago; today I'd use MPI); a > variant of that was apparently moved to the production world after > I changed positions and couldn't support it anymore. > > I'm not sure to tell you who the biggest corporate users are. It > likely depends on one's exposure to even have an opinion. However, > for the biggest You raise an interesting question. Military type things not counted, biggest corporate users of generic cpu's is car industry. This is quite logical: safety is everything with cars and a car has far over 50+ processors. Not seldom 100 nowadays. Drug type companies eat typical 0.5% system time. They are far behind, which is not so logical; i would consider it old fashioned to do experiments on monkeys (Bar-ilan brain experiments on monkeys rings a bell?), so doing everything in software is far more logical. The big question is: should one just look to how many generic cpu's one uses? When someone is really in big need for big calculation power, one makes dedicated cpu's for it. It may come... In most embedded devices that badly need high levels of encryption, typically a coprocessor gets put in to do the job. Such co processor is really mighty if one would compare it with how many Tflop such a thing is when putting in generic cpu's. If you look at it from that viewpoint, ISPs are worlds biggest corporate users of hardware. Now go back in your corner and secure the internet :) > corporate users I'm aware of in my immediate area, Schlumberger, > Exxon-Mobil, Shell Exploration, and Chevron come to mind. This is > obviously a very incomplete list. > > Gerry > > Vincent Diepeveen wrote: >> Hi Dan, >> I'd rather guess his question is more: "i want a big list of 10000 >> companies that are actively using clusters". >> That list is not there and if it was there it is utmost secretly >> compiled by some NSA type instance. >> I'm now speaking more for western Europe, not necessarily for USA: >> The research that companies perform is usually applied research: >> the minimum requirement to produce a new product, >> or when forced at gunpoint by government for safety reasons. >> For fundamental research usually most computational resources get >> thrown into battle, which is for nearly 100% sponsored >> by government be it directly or indirectly. >> In fact there is entire industries, especially clean energy >> research, that is 100% paid by government subsidies. >> Investigations after new products, in 99% of cases get funded by >> government; usually the name of a big company is behind it, >> but it's still government dudes who sit in that company doing the >> research. What matters of course is who pays it. >> The exception is persons who have their own company and are some >> sort of big expert at a certain area. Take me. >> I want to do some research after learning algorithms, at a very >> high level (so far they didn't even conclude the right >> value for material for chess, let alone that they were useful at >> anything other than debugging your code, which is a big >> important thing by the way). >> Without getting the help of someone working at an university to >> run such a thing overnight at a 160-200 cores, >> i would of course only be able to run it at a single pc. Now i >> don't get any money for all this from government, >> so the big question is whether i'm supported by government then or >> not. >> What i do know is that all public research here at government >> level is major shit. This is explainable by the fact that a field >> is nonstop >> on the move, only those who have been busy in a field for 10+ >> years have any clue what happens there and are the first to try >> something; >> the PHD's are simply busy with inferior software meanwhile having >> massive hardware that's idle. A big problem is the nonpublishing of >> accomplishments; First of all i'm not publishing *how* i'm doing >> the learning. When some university offers me a job, i might >> publish things, >> i'm not getting paid for publications. Why help "the competition"? >> What might get published is the guy who helps me; one tiny thing >> of all this he'll publish probably. That's maybe 1/100 of the >> conclusions >> drawn. >> Only the NSA type organisations know as they all spy on the >> internet, and you BET that all big >> countries know exactly what i'm doing, they all tap the internet >> like crazy there. >> If 1 researcher is real brilliant and CAN need some big >> computation, there is at least a 100 spies who do know something >> of that field, >> and have to interpret the tapped data. What i do not know, as i >> don't work for any of the N*SA type organisations, is in how far >> *those* >> verify things using massive hardware. >> Most researchers have no clue how big the spying and anti-spying >> is; most might get total scared if they'd realized how much they >> get protected. >> It is easier to steal it than to find an Einstein in your own >> nation figuring it out. >> That is the principle that every nation uses; CIA is a very tiny >> organisation compared to what other nations have, >> note that the weird thing is that CIA is notorious for violating >> agreements with other nations (not spying in friendly nations, >> to give an example; and also passing on information to their >> companies that they got from the information stream they >> got from friendly nations spying onto their own companies). >> The fact that i write this down already means i have not been >> employed in that field nor am; otherwise i would not even >> *mention* the word. >> Or to quote someone who works in that field when i said that there >> gotta be an awful lot of civil servants in Netherlands busy in >> that field, >> as there is 3+ million tourists a year in Netherlands: >> "we have tourism in Spain also". >> Yet nothing on the internet is safe, that's the basic problem >> here. It's too easy to hack internet lines. 1024 bits or 2048 RSA >> or something that >> gets used for SSH? >> 1024 bits is too easy to crack for 1 organisation of each country >> by simply throwing specialized hardware at it. >> 2048 bits RSA is a tad harder, but also not a problem. >> 128 bits AES? >> No manner to attack it is public known (would only deliver $20k, >> which is too little anyway, for the risk you take publishing a >> method). >> Yet even if there would be some sort of randomized GA-algorithm >> that needs some sort of Pollard-Rho order O( 2 ^ 0.25n ), >> then a simple dedicated cpu can already crack it handsdown. >> Obviously most companies are not busy spamming the net what they >> do with clusters or do not do. >> Keeping it secret for their competitors is just too important, yet >> if you ask me i feel big companies do real little >> research in areas that do not lead directly to products of them. >> There might be 1 or 2 exceptions, like oil companies, >> but the question is in how far researchers there can be seen as >> employees of that company as they get indirectly >> sponsored by government whose interest in everything that happens >> with oil is real big. >> If you ask me however, way too little fundamental research gets >> sponsored by government. >> If you see just in a tiny nation like netherlands; the number of >> 'researchers' that work for government is just a very small part of >> the number of professors. It's like 1 researcher for each 2 >> professors; PHD's not counted. >> Note this is also what a few studies recently showed. >> Just the comparision to intelligence agencies which can put into >> action each one of them quarter of a million to millions of people, >> and who hire real easy people be it direct or indirect, it is >> obvious that the saying: "Better well stolen than bad invented", >> gets written in just too big capitals. >> As we're speaking of a 1000 individuals in a nation having 16.5 >> million inhabitants, and only very few of those have >> time to devote a big part of their time to do research; maybe 5% is? >> Most are too busy with meetings and other organisational work and >> posting on the internet. >> Realize that same nation also has over a 1000 companies with >> more than 1000 employees and half a million charity organisations. >> That really lets fundamental research look like something that >> gets total neglected. >> Vincent >> On Oct 7, 2008, at 9:34 AM, >> wrote: >>> Andrea, >>> >>> MPI is of course used by many applications running on commercial >>> clusters. >>> Two obvious examples are computational chemistry by the drug >>> companies >>> and computational fluid dynamics (CFD) for aerospace companies >>> and F1 design teams. >>> >>> These are all long-term 'traditional' uses of MPI for scientific/ >>> engineering codes. >>> >>> Is this what you are asking? Or are you thinking of non- >>> traditional uses in say computational finance or gaming sites? >>> >>> Daniel >>> >>> ------------------------------------------------------------- >>> Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger@quadrics.com >>> One Bridewell St., Mobile: +44 (0)779 209 1851 >>> Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519 >>> ----------------------- www.quadrics.com -------------------- >>> >>> >>> >>> >>> -----Original Message----- >>> From: beowulf-bounces@beowulf.org [mailto:beowulf- >>> bounces@beowulf.org] On Behalf Of Andrea Di Blas >>> Sent: 13 August 2008 00:37 >>> To: beowulf@beowulf.org >>> Subject: [Beowulf] large MPI adopters >>> >>> hello, >>> >>> >>> I am curious about what companies, besides the national labs of >>> course, >>> use any implementation of MPI to support large applications of >>> any kind, >>> whether only internally (like mapreduce for google, for example) >>> or not. >>> >>> does anybody know of any cases? >>> thank you and best regards, >>> >>> >>> andrea >>> >>> >>> >>> >>> -- >>> Andrea Di Blas, UCSC >>> School of Engineering >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >>> >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >>> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > > -- > Gerry Creager -- gerry.creager@tamu.edu > Texas Mesonet -- AATLT, Texas A&M University > Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 > Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 > From kyron at neuralbs.com Tue Oct 7 08:45:52 2008 From: kyron at neuralbs.com (Eric Thibodeau) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> Message-ID: <48EB8430.2070102@neuralbs.com> I've read the entire thread up to now and noticed no one mentioned the parallel bzip2 (pbzip2 http://compression.ca/pbzip2/) utility. Although not strictly compatible with bzip2 when compressing large files, it's still a valid compressor imho. More below... Xu, Jerry wrote: > Hello, > > Currently I generate nearly one TB data every few days and I need to pass it > along enterprise network to the storage center attached to my HPC system, I am > thinking about compressing it (most tiff format image data) as much as I can, as > fast as I can before I send it crossing network ... I performed a test on *uncompressed* tif images (RAW images from a Canon EOS camera iirc) of many 32MB images tarred into a single 2.9Gig file put into /dev/shm (simulating stream). Here is the time characteristics for compressing the 2.9gig file using pbzip2: kyron@kyron ~ $ /usr/bin/time -v pbzip2 -b15 -k /dev/shm/TEST.tar -c >/dev/shm/TEST.tar.bzip Command being timed: "pbzip2 -b15 -k /dev/shm/TEST.tar -c" User time (seconds): 415.95 System time (seconds): 4.89 Percent of CPU this job got: 394% Elapsed (wall clock) time (h:mm:ss or m:ss): 1:46.56 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 0 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 3 Minor (reclaiming a frame) page faults: 677802 Voluntary context switches: 4196 Involuntary context switches: 57332 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 The system has an Intel Quad Core Q6700 with 8Gigs of RAM (forced at 667MHz due to Asus's P35 implementation limitations). Making sense of the figures, the file is 3046748160 bytes, the output file is 1427448276 bytes (2 to 1 ratio...which will depend on image contents of course). It took 106.56 seconds (1:46.56 seconds / 4 cores) to process the 3046748160 which translates into a processing speed of ~ 28E6 bytes/sec processing speed. Modern HDDs and GigE surpass this processing speed comfortably. Since it's been brought up in the thread, here are some figures for a binary database (float vectors of pattern recognition characteristics)...and note how compressible the file ends up being: kyron@kyron ~ $ /usr/bin/time -v pbzip2 -b15 -k ~/1_Files/1_ETS/1_Maitrise/Code/K-Means_Cavalin/featg_row.dat -c >/dev/shm/TEST.tar.bzip Command being timed: "pbzip2 -b15 -k /home/kyron/1_Files/1_ETS/1_Maitrise/Code/K-Means_Cavalin/featg_row.dat -c" User time (seconds): 1096.09 System time (seconds): 7.12 Percent of CPU this job got: 395% Elapsed (wall clock) time (h:mm:ss or m:ss): 4:38.65 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 0 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 135624 Voluntary context switches: 10233 Involuntary context switches: 154397 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 featg_row.dat if 1454853604 bytes and compresses down to 238M (guess there is lots of redundancy in the features ;P ) But still, 1454853604 bytes / 278.65 seconds = 5.2E6 bytes/sec processing speed; still insufficient compared to current hardware. > So, I am wondering whether > anyone is familiar with any hardware based accelerator, which can dramatically > improve the compressing procedure.. suggestion for any file system architecture > will be appreciated too.. I have couple of contacts from some vendors but not > sure whether it works as I expected, so if anyone has experience about it and > want to share, it will be really appreciated ! > > Thanks, > > Jerry > > Jerry Xu PhD > HPC Scientific Computing Specialist > Enterprise Research Infrastructure Systems (ERIS) > Partners Healthcare, Harvard Medical School > http://www.partners.org > > The information transmitted in this electronic communication is intended only > for the person or entity to whom it is addressed and may contain confidential > and/or privileged material. Any review, retransmission, dissemination or other > use of or taking of any action in reliance upon this information by persons or > entities other than the intended recipient is prohibited. If you received this > information in error, please contact the Compliance HelpLine at 800-856-1983 and > properly dispose of this information. > > > HTH Eric Thibodeau PS: Interesting figures, I couldn't resist compressing the same binary DB on a 16Core Opteron (Tyan VX50) machine and was dumbfounded to get horrible results given the same context. The processing speed only came up to 6.4E6 bytes/sec ...for 16 cores, and they were all at 100% during the entire run (FWIW, I tried different block sizes and it does have an impact but this also changes the problem parameters). eric@einstein ~ $ /usr/bin/time -v pbzip2 -b15 -k ~/1_Files/1_ETS/1_Maitrise/Code/K-Means_Cavalin/featg_row.dat -c >/dev/shm/TEST.tar.bzip Command being timed: "pbzip2 -b15 -k /export/livia/home/parallel/eric/1_Files/1_ETS/1_Maitrise/Code/K-Means_Cavalin/featg_row.dat -c" User time (seconds): 3356.42 System time (seconds): 5.20 Percent of CPU this job got: 1481% Elapsed (wall clock) time (h:mm:ss or m:ss): 3:46.94 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 0 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 345519 Voluntary context switches: 4088 Involuntary context switches: 9036 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 From kyron at neuralbs.com Tue Oct 7 11:07:14 2008 From: kyron at neuralbs.com (Eric Thibodeau) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48EB8430.2070102@neuralbs.com> Message-ID: <48EBA552.60305@neuralbs.com> Dmitri Chubarov wrote: > Hello, > > we have got a VX50 down here as well. We have observed very different > scalability on different applications. With an OpenMP molecular > dynamics code we have got over 14 times speedup while on a 2D finite > difference scheme I could not get far beyond 3 fold. > 2D finite difference can be comm intensive is the mesh is too small for each processor to have a fair amount of work to do before needing the neighboring values from a "far" node. > On Tue, Oct 7, 2008 at 10:45 PM, Eric Thibodeau wrote: > >> PS: Interesting figures, I couldn't resist compressing the same binary DB on >> a 16Core Opteron (Tyan VX50) machine and was dumbfounded to get horrible >> results given the same context. The processing speed only came up to 6.4E6 >> bytes/sec ...for 16 cores, and they were all at 100% during the entire run >> (FWIW, I tried different block sizes and it does have an impact but this >> also changes the problem parameters). >> > > Reading your message in the Beowulf list I should say that it looks > interesting and probably shows something happening with the memory > access on the NUMA nodes. Did you try to run the archiver with > different affinity settings? > I don't have affinity control over the app per say. I would have to look/modify pbzip's code. Although, note that the PID's assignment to one processor is governed by the kernel and is thus a scheduler issue. Also note that I have noticed that the kernel doesn't just have fun moving the processes around the cores. > We have observed that the memory architecture shows some strange > behaviour. For instance the latency for a read from the same NUMA node > for different nodes varies significantly. > This is the nature of NUMA. Furthermore, if you have to cross to a far CPU, the latency is also dependent on the CPU's load. > Also on the profiler I often see that x86 instructions that have one > of the operands in memory may > take disproportionally long. I believe that could explain the 100% CPU > load reported by the kernel. > How do you identify the specific instruction using a profiler, this is something that interests me. > From the very little knowledge of this platform that we have got, I > tend to advise the users not to expect good speedup on their > multithreaded applications. Using OpenMP (from GCC 4.3.x) and an embarrassingly parallel problem (computing K-Means on a large database), I do get significant speedup (15-16). > Yet it would be interesting to get a > better understanding of the programming techniques for this sedecimus > and the similar machines. OpenMP is IMHO the easiest one that will bring you the most performance out of 3 lines of #pragma directives. If you manage to get a cluster of VX50s, then learn a bit of MPI to glue all of this together ;) > Even more so due to the QPI systems becoming > commercially available very soon. Don't know that one (QPI)...oh...new Intel stuff...no matter how much I try to stay ahead, I'm always years behind! > At the moment we have got a few > small kernels written in C and Fortran with OpenMP that we use to > evaluate different parallelization strategies. Unfortunately, there > are no tools I would know of that could help to explain what's going > on inside the memory of this machine. > Of course, check out TAU ( http://www.cs.uoregon.edu/research/tau/home.php ), it will at least help you identify bottlenecks and give you an impressive profiling infrastructure. > I am very much interested to hear more about your experience with VX50. > > Best regards, > Dima Chubarov > > -- > Dmitri Chubarov > junior researcher > Siberian Branch of the Russian Academy of Sciences > Institute of Computational Technologies > http://www.ict.nsc.ru/indexen.php > Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081007/bb7f12af/attachment.html From dmitri.chubarov at gmail.com Tue Oct 7 11:34:41 2008 From: dmitri.chubarov at gmail.com (Dmitri Chubarov) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Accelerator for data compressing In-Reply-To: <48EBA552.60305@neuralbs.com> References: <200810021900.m92J05RP012792@bluewest.scyld.com> <552609B844F30844A3EF4B50A11ECF369F8818@PHSXMB32.partners.org> <48EB8430.2070102@neuralbs.com> <48EBA552.60305@neuralbs.com> Message-ID: > > 2D finite difference can be comm intensive is the mesh is too small for each > processor to have a fair amount of work to do before needing the neighboring > values from a "far" node. > Actually it seems that with VX50 the same node may be the "far" node. At least that's what I see from the NUMA Analysis test from TAU Wiki: http://www.nic.uoregon.edu/tau-wiki/Guide:Opteron_NUMA_Analysis Do not have the numbers at hand but that was the impression. > > How do you identify the specific instruction using a profiler, this is > something that interests me. > I am using the Performance Analyzer that comes with Sun Studio 12. It provides a per instruction profile view of the disassembly. From gerry.creager at tamu.edu Wed Oct 8 05:00:36 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: <6DB5B58A8E5AB846A7B3B3BFF1B4315A0269989D@AVEXCH1.qlogic.org> References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com><3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> <48EB69B6.5080602@tamu.edu> <6DB5B58A8E5AB846A7B3B3BFF1B4315A0269989D@AVEXCH1.qlogic.org> Message-ID: <48ECA0E4.4060000@tamu.edu> Tom, I looked at that and *THOUGHT* it looked funny, but I was unable to see the typo. Yeah, OpenMP. Both have their place, which is sometimes integrated into the same application! gerry Tom Elken wrote: >> [mailto:beowulf-bounces@beowulf.org] On Behalf Of Gerry Creager >> The list isn't small: As Dan mentions, the drug companies are >> big users, >> although there's a fair bit of openMPI operation in their codes, too. > Gerry, > I'm sure you meant > OpenMP > (standard shared-memory threaded parallelism) > > So easy for fingers to type "MPI". > > Even though MPI has far more usage than OpenMP, I have to stick up for > it as a former shared-memory advocate from my SGI days and as a member > of the OpenMP ARB. > > -Tom > >> CFD in aerospace, and auto manufacturers. The US military almost >> certainly does parallel processing involving both message passing and >> Monte Carlo simulations. I've written code that solves physical and >> satellite geodesy problems using PVM (but I was young and that was a >> long time ago; today I'd use MPI); a variant of that was apparently >> moved to the production world after I changed positions and couldn't >> support it anymore. >> >> I'm not sure to tell you who the biggest corporate users are. >> It likely >> depends on one's exposure to even have an opinion. However, for the >> biggest corporate users I'm aware of in my immediate area, >> Schlumberger, >> Exxon-Mobil, Shell Exploration, and Chevron come to mind. This is >> obviously a very incomplete list. >> >> >> Gerry >> > >>> From: beowulf-bounces@beowulf.org >> [mailto:beowulf-bounces@beowulf.org] >>>> On Behalf Of Andrea Di Blas >>>> Sent: 13 August 2008 00:37 >>>> To: beowulf@beowulf.org >>>> Subject: [Beowulf] large MPI adopters >>>> >>>> hello, >>>> >>>> >>>> I am curious about what companies, besides the national >> labs of course, >>>> use any implementation of MPI to support large >> applications of any kind, >>>> whether only internally (like mapreduce for google, for >> example) or not. >>>> does anybody know of any cases? >>>> thank you and best regards, >>>> >>>> >>>> andrea >>>> >>>> >>>> >>>> >>>> -- >>>> Andrea Di Blas, UCSC >>>> School of Engineering >>>> _______________________________________________ >>>> Beowulf mailing list, Beowulf@beowulf.org >>>> To change your subscription (digest mode or unsubscribe) visit >>>> http://www.beowulf.org/mailman/listinfo/beowulf >>>> >>>> _______________________________________________ >>>> Beowulf mailing list, Beowulf@beowulf.org >>>> To change your subscription (digest mode or unsubscribe) visit >>>> http://www.beowulf.org/mailman/listinfo/beowulf >>>> >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >> -- >> Gerry Creager -- gerry.creager@tamu.edu >> Texas Mesonet -- AATLT, Texas A&M University >> Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 >> Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) >> visit http://www.beowulf.org/mailman/listinfo/beowulf >> -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From Bogdan.Costescu at iwr.uni-heidelberg.de Wed Oct 8 07:09:36 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> <48EB69B6.5080602@tamu.edu> Message-ID: On Tue, 7 Oct 2008, Vincent Diepeveen wrote: > Drug type companies eat typical 0.5% system time. Could you please explain what you mean here ? That they buy lots of CPUs but only use a fraction of them or only for a fraction of time ? > They are far behind, which is not so logical; i would consider it > old fashioned to do experiments on monkeys (Bar-ilan brain > experiments on monkeys rings a bell?), so doing everything in > software is far more logical. Maybe your knowledge about the pharma industry is far behind ? "Doing everything in software is far more logical" only when you can exactly express biological processes in mathematical terms which is not true in the great majority of cases. F.e. molecular dynamics (often used to simulate interactions at molecular level, f.e. of a drug injected into the body) is based on several approximations and the implementations are subject to numerical errors. Would you take a drug that was produced based only on a simulation with molecular dynamics and found to be good enough ? Similar concerns apply to all levels - the brain, as you mention it, is still very much an enigma, otherwise artificial intelligence would be all around us (or not, if deemed to dangerous...). And when you don't know how an organ or system works, how can you develops drugs only by simulations ? > When someone is really in big need for big calculation power, one > makes dedicated cpu's for it. I tend to agree, provided that the dedicated CPU brings a really big advantage, like an order of more of magnitude. F.e., to stay in the pharma related field, that's the bussiness case for DEShaw Research's molecular dynamics chip (Anton) which is supposed to bring a significant increase in speed for MD simulations compared with software solutions running on general purpose CPUs (or GPUs). -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From diep at xs4all.nl Wed Oct 8 09:32:42 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> <48EB69B6.5080602@tamu.edu> Message-ID: <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl> On Oct 8, 2008, at 4:09 PM, Bogdan Costescu wrote: > On Tue, 7 Oct 2008, Vincent Diepeveen wrote: > >> Drug type companies eat typical 0.5% system time. > > Could you please explain what you mean here ? That they buy lots of > CPUs but only use a fraction of them or only for a fraction of time ? > When you add up others. Note i see in IWR you got a 160 node opteron cluster, what software eats system time on that cluster? There must be graphs about it, want to share one? Also interesting to know is how many cores most jobs eat! Note that i left out 1 important factor which is not a factor in every nation: that's the amount of system time airindustry eats of supercomputers designing new stuff. It is very significant together with car manufacturers. With 160 nodes machine, even if that would be power6 nodes, you can't even design a wing for aircraft. >> They are far behind, which is not so logical; i would consider it >> old fashioned to do experiments on monkeys (Bar-ilan brain >> experiments on monkeys rings a bell?), so doing everything in >> software is far more logical. > > Maybe your knowledge about the pharma industry is far behind ? > "Doing everything in software is far more logical" only when you > can exactly express biological processes in mathematical terms > which is not true in the great majority of cases. F.e. molecular > dynamics (often used to simulate interactions at molecular level, > f.e. of a drug injected into the body) is based on several > approximations and the implementations are subject to numerical > errors. Would you take a drug that was produced based only on a > simulation with molecular dynamics and found to be good enough ? > Well, i sure avoid hospital visits, you? As you know the healthcare commission is basically old men who spread careful chosen sentences in such a manner that everything looks fine on this planet and that there is nothing to worry about; last time i spoke with them on EMF's a guy being spokesman on this subject, he didn't even know how the electricity system in his own house worked (whether on the first floor of his house he had earth). So lucky to introduce to a new drug, there is some burocratic systems. There is several phases of testing that a drug has to pass in order to get accepted as a drug that is legal. Not to confuse with the 'advice' on whether a drug gets prescribed; they want way cheaper drugs for that usually than the ones that might maybe be better (say 0.5% better for a lot of money extra, but if you're that guy that didn't get cured as a result of that, your family will regret it bigtime). Computing power or not, most manufacturers simply want to sell a product, so the amount of systemtime invested on a computer, in order to sell, is total irrelevant to them to get a drug accepted usually. The first and most important part of the show to get a drug accepted is getting invited to the table to show your drug. Computing power is nice to mention here when that is usually not a factor at all to get a new medicine accepted. But it may change, let's hope so... Maybe a new breed of company can do the job. A breed where making profit is not most important :) > Similar concerns apply to all levels - the brain, as you mention > it, is still very much an enigma, otherwise artificial intelligence > would be all around us (or not, if deemed to dangerous...). And > when you don't know how an organ or system works, how can you > develops drugs only by simulations ? > >> When someone is really in big need for big calculation power, one >> makes dedicated cpu's for it. > > I tend to agree, provided that the dedicated CPU brings a really > big advantage, like an order of more of magnitude. F.e., to stay in > the pharma related field, that's the bussiness case for DEShaw > Research's molecular dynamics chip (Anton) which is supposed to > bring a significant increase in speed for MD simulations compared > with software solutions running on general purpose CPUs (or GPUs). Some hardware guys can better comment here. Making your own FPGA chippie is rather cheap. Finding and paying a few good guys to get the chip done is toughest. You basically need 1 good programmer, the rest can get hired parttime in fact. After chip works, to print a 1000 cards, it is like a 100 dollar for pci-card and 50 dollar a chip. So $150k for computing power that total annihilates any supercomputer of now and the NEAR future for your specific application. Real cpu's are a lot faster of course, price for them i do not have at hand. I heard it could be cheaper than this in fact if you regurarly print a batch. Such a batch is producing less cpu's than the FPGA, where you pay for 1 cpu a constant amount. I tend to remember that good cpu designers, not to mention memory controller guys, are rather expensive folks, as they are very popular; on some other mailing list with some very funny lastnames, i see nonstop job offers for that type of folks. Vincent > -- > Bogdan Costescu > > IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany > Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 > E-mail: bogdan.costescu@iwr.uni-heidelberg.de > From Bogdan.Costescu at iwr.uni-heidelberg.de Wed Oct 8 11:00:19 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl> References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> <48EB69B6.5080602@tamu.edu> <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl> Message-ID: [ Trimmed CC list ] On Wed, 8 Oct 2008, Vincent Diepeveen wrote: > When you add up others. It's interesting that you make this statement along with another one saying that many companies that have clusters or other parallel computing resources don't make them public. So how can you arrive to such a figure ??? > Note i see in IWR you got a 160 node opteron cluster, what software > eats system time on that cluster? In my vocabulary, system time is the time used by the system (kernel, maybe daemons) as opposed to user time. What probably interests you is what kind of software is running on HELICS; if so, how come you managed to see the hardware page but missed the projects list ? It's linked from the main HELICS page, but to make it easier for you this is the direct link: http://helics.iwr.uni-heidelberg.de/users/project_overview.php The 160 nodes are only the newer part called HELICS II; and there are also other clusters in IWR :-) > There must be graphs about it, want to share one? Measuring what ? Generally speaking, I don't produce graphs, text output is enough for me to know that the cluster is busy :-) > Also interesting to know is how many cores most jobs eat! Most jobs running on HELICS II start 4 processes per node. There are 4 cores per node, so that's 1 process per core. Some of the software (mostly developed in-house) requires a lot of memory per process, so then less cores per node are being used - it's a tradeoff that was factored in the design. > that's the amount of system time airindustry eats of supercomputers > designing new stuff. It is very significant together with car > manufacturers. And how do you know that ? :-) > With 160 nodes machine, even if that would be power6 nodes, you > can't even design a wing for aircraft. Aircraft design as a research subject can probably easily earn computer time on the many parallel computing resources in Germany that any researcher working in a German university can apply for. Some of the parallel computing resources are better suited for certain tasks (like the NEC SX8 in Stuttgart for vector software), so we're free to choose the one that fits best. Local resources are in some cases preferred because they can be easier controlled (f.e. hardware choices to make it fit for a particular application that is run very often) or getting access to. So the size of the local cluster(s) is not a limiting factor for the science that gets done. Getting non-x86(_64) compatible CPUs (like Power-something) for HELICS was considered but rejected because of the already-mentioned problem of binary-only software from ISVs. > Well, i sure avoid hospital visits, you? I do go to the hospital to visit ill relatives or friends :-) > As you know the healthcare commission is basically old men You seem to have a lot of knowledge about everything, I don't - so I can't comment on your statements. ;-) > Computing power or not, most manufacturers simply want to sell a > product, so the amount of systemtime invested on a computer, in > order to sell, is total irrelevant to them to get a drug accepted > usually. Indeed, the computer is involved usually only in the initial stages, when many compounds are screened and only very few go further. Clinical trials are very expensive and time consuming, so the computer time used initially is only a fraction of the total time needed to put out a new drug. But this is not because the pharma companies would want to keep it so long, it's because all these further tests are supposed to keep the later users of these drugs out of trouble and to provide that (sometimes very long...) list of side-effects they are required to put in the box. It's not the companies who find the computer time irrelevant, it's the whole system that makes it so. > Maybe a new breed of company can do the job. A breed where making > profit is not most important :) "Imagine there's no money" (paraphrasing John Lennon) :-) > Some hardware guys can better comment here. Making your own FPGA > chippie is rather cheap. Finding and paying a few good guys to get > the chip done is toughest. You basically need 1 good programmer, the > rest can get hired parttime in fact. Maybe you should read more about the Anton chip and the team that created it. That would better fit into the 'lot of knowledge about everything' figure that I mentioned above :-) > So $150k for computing power that total annihilates any > supercomputer of now and the NEAR future for your specific > application. Whew, do you wanna become partners so that we can sell this idea to NASA ? :-))) > I tend to remember that good cpu designers, not to mention memory > controller guys, are rather expensive folks Indeed. Then look at the Anton chip and see why your 1+peanuts expert/project idea doesn't fit ;-) -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From csamuel at vpac.org Wed Oct 8 20:40:57 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] FY;) the Helmer cluster In-Reply-To: Message-ID: <2045508708.1313971223523657435.JavaMail.root@mail.vpac.org> ----- "Igor Kozin" wrote: > lol the guy surely is very ambitious - 4 PFLOPS for $0.9M > http://helmer3.sfe.se/ Sadly it looks like that's based on the single precision numbers for graphics cards, rather than double.. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From Dan.Kidger at quadrics.com Thu Oct 9 05:08:22 2008 From: Dan.Kidger at quadrics.com (Dan.Kidger@quadrics.com) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Pretty High Performance Computing In-Reply-To: <9f8092cc0809240805y4e8d56dsc08b0e3ce83fea45@mail.gmail.com> References: <518096.44032.qm@web37908.mail.mud.yahoo.com> <9f8092cc0809240805y4e8d56dsc08b0e3ce83fea45@mail.gmail.com> Message-ID: <0D49B15ACFDF2F46BF90B6E08C90048A0493821198@quadbrsex1.quadrics.com> John, Yes indeed at that Daresbury Machine Evaluation Workshop - quite a while ago - maybe 2001 or 2002 Mike gave out as the freebie on their booth Boxer Shorts with the Streamline logo on. Compared with giving away say t-shirts, I am not sure how much market exposure they got (!) Daniel ------------------------------------------------------------- Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Mobile: +44 (0)779 209 1851 Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519 ----------------------- www.quadrics.com -------------------- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of John Hearns Sent: 24 September 2008 16:06 To: beowulf@beowulf.org Subject: Re: [Beowulf] Pretty High Performance Computing 2008/9/24 Ellis Wilson > This assumes my understanding of middleware is correct in that it is a package or entire system that simplifies things by being somewhat blackboxed and ready to go. Anything canned like tuna is bound to contain too much salt. I believe that that Mike Rudgyard, the chief exec of Streamline Computing, coined the term 'underware' for this. This wa s presentation at one of the Daresbury Lab's Machine Evaluation Workshops, which I can't find via Google at the moment. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081009/4e64b179/attachment.html From Dan.Kidger at quadrics.com Thu Oct 9 06:18:54 2008 From: Dan.Kidger at quadrics.com (Dan.Kidger@quadrics.com) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] FY;) the Helmer cluster In-Reply-To: <2045508708.1313971223523657435.JavaMail.root@mail.vpac.org> References: <2045508708.1313971223523657435.JavaMail.root@mail.vpac.org> Message-ID: <0D49B15ACFDF2F46BF90B6E08C90048A04938211A5@quadbrsex1.quadrics.com> Looking at the CAD drawings - is there any reason why the nodes are built into circular columns, rather than traditional cuboid boxes? If anything such a design does not make full use of the physical space available. And if you did pack the columns as drawn (with vertical water pipes in the gaps) - how do you service the nodes to replace failed components, water leaks et al. ? And also (I do work for an interconnect company after all) how would all this get wired up together as a cluster? Daniel ------------------------------------------------------------- Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Mobile: +44 (0)779 209 1851 Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519 ----------------------- www.quadrics.com -------------------- -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Chris Samuel Sent: 09 October 2008 04:41 To: Beowulf List Subject: Re: [Beowulf] FY;) the Helmer cluster ----- "Igor Kozin" wrote: > lol the guy surely is very ambitious - 4 PFLOPS for $0.9M > http://helmer3.sfe.se/ Sadly it looks like that's based on the single precision numbers for graphics cards, rather than double.. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hearnsj at googlemail.com Thu Oct 9 07:37:30 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Manchester Guardian column on Cray and Windows HPC Message-ID: <9f8092cc0810090737g141a2093l7f054b914483c49d@mail.gmail.com> As a Guardian reader, I have been reading the Thursday Technology supplement for many years. On the train this morning, I opened it to find Jack Schofield has an article on Windows HPC and the Cray deskside supercomputer. http://www.guardian.co.uk/technology/2008/oct/09/computing.microsoft Has Jack been reading the Beowulf list? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081009/5757a76d/attachment.html From james.p.lux at jpl.nasa.gov Thu Oct 9 08:45:07 2008 From: james.p.lux at jpl.nasa.gov (Lux, James P) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Manchester Guardian column on Cray and Windows HPC In-Reply-To: <9f8092cc0810090737g141a2093l7f054b914483c49d@mail.gmail.com> References: <9f8092cc0810090737g141a2093l7f054b914483c49d@mail.gmail.com> Message-ID: From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of John Hearns Sent: Thursday, October 09, 2008 7:38 AM To: beowulf@beowulf.org Subject: [Beowulf] Manchester Guardian column on Cray and Windows HPC As a Guardian reader, I have been reading the Thursday Technology supplement for many years. On the train this morning, I opened it to find Jack Schofield has an article on Windows HPC and the Cray deskside supercomputer. http://www.guardian.co.uk/technology/2008/oct/09/computing.microsoft Has Jack been reading the Beowulf list? --- >From the article: "For example, Faenov says Excel 2007 supports "transparent parallelism", so that an HPC machine can speed up everyday workflows in financial institutions where they have spreadsheets that take hours or even days to run. (Crazy, I know.)" No doubt, those models were used to do interesting things like calculate expected risk of default for the multiple tranches in various securitized debt obligations. I'm not sure that, philosopically, more speed is good in this area. It's one thing for a mathematical proof to be so big only a computer can do it (e.g. 4color map theorem), or when you want to run numerical models of weather or fluid dynamics. At least in those areas, there's a fair amount of work that goes into validation of the underlying numerics. Excel, on the other hand, is probably not the best tool in the world for hard core numerical analysis (what with *interesting* phenomena like the famous 850*77.1= 100000 bug, ... Yes it was fixed, but it's not like there's some rigorous verification of Excel's computations that's available for inspection)http://www.lomont.org/Math/Papers/2007/Excel2007/Excel2007Bug.pdf has an explanation And, as it happens this particular HPC application space (long running spreadsheet calcs) does exist. Many years ago, my sister had a job at a bank where part of the task was running Lotus 1-2-3 worksheets on a PC that took hours to recalc on a PC/AT. I actually contemplated designing and building an ECL IBM PC emulator for this market, but then Intel came out with the 386, etc.. Much better system solution to solve it in silicon. Jim From hearnsj at googlemail.com Thu Oct 9 08:49:08 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] Manchester Guardian column on Cray and Windows HPC In-Reply-To: References: <9f8092cc0810090737g141a2093l7f054b914483c49d@mail.gmail.com> Message-ID: <9f8092cc0810090849o4bc48d46i76029a853440905f@mail.gmail.com> 2008/10/9 Lux, James P > > > > And, as it happens this particular HPC application space (long running > spreadsheet calcs) does exist. Many years ago, my sister had a job at a > bank where part of the task was running Lotus 1-2-3 worksheets on a PC that > took hours to recalc on a PC/AT. Indeed. I have seen job adverts for City of London financial hoouses where the job description was to write Excel macros and run spreadsheets for traders. I'm somewhat glad I didn't apply - could be a tad stressful even without the current downward trend. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081009/9ab66df1/attachment.html From landman at scalableinformatics.com Thu Oct 9 10:59:48 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] OT: OhioLinuxFest this weekend Message-ID: <48EE4694.4070108@scalableinformatics.com> Hi folks In case any beowulfers will be round the Columbus OH area this Saturday, OhioLinuxFest (www.ohiolinux.org) will be on. We (Scalable) will be there at a booth with some nVidia goodness in a deskside unit, and something else as well (big evil grin: ΔV). Andrew Latham (a lurker on this list, and all around good guy) will be there as well. Stop by and say hi if you do attend. If enough people show up, we may try for a Beowulf/Cluster/Storage BOF. Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From rpnabar at gmail.com Thu Oct 9 11:57:39 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] kdump / kexec to optain crash dumps from randomly crashing nodes. Message-ID: On my Centos system I installed kexec/kdump to investigate the cause of some random system-crashes by getting access to a crash-dump. I installed the rpm for kexec and then made the change to grub.conf that reserves the additional memory for the new kernel. Also configured kdump.conf. I start the kexec service.and then I tried to simulate a crash by echo c to sysrq-trigger. The system does crash and then after a while reboots itself. But I see no vmcore when it coms back up. /var/crash is empty. This is when I tried to write to local drive. I also tried a nfs write but then still no success. Any idea what could be missing in my steps? Or any other debug suggestions? Any other kdump users on Beowulf? -- Rahul From rpnabar at gmail.com Thu Oct 9 12:19:20 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] kdump / kexec to optain crash dumps from randomly crashing nodes. In-Reply-To: <48EE567C.3010503@gmail.com> References: <48EE567C.3010503@gmail.com> Message-ID: Hi Paolo, The funny thing is that the console remains blank. We have all these systems connected to a KVM and the kvm shows the system as actually disconnected post the crash. That is what makes it so hard to debug. No screen output at all. -Rahul On Thu, Oct 9, 2008 at 2:07 PM, Paolo Supino wrote: > Hi Rahul > > Did you try to redirect console to a serial port? If a system crashes > and all console messages (including kernel) will be sent to the serial > console that will keep displaying the messages it received until the > system is power cycled ... > > > > > > -- > ttyl > Paolo > > > > Rahul Nabar wrote: >> On my Centos system I installed kexec/kdump to investigate the cause of >> some random system-crashes by getting access to a crash-dump. I installed >> the rpm for kexec and then made the change to grub.conf that reserves the >> additional memory for the new kernel. >> >> Also configured kdump.conf. I start the kexec service.and then I tried to >> simulate a crash by echo c to sysrq-trigger. >> >> The system does crash and then after a while reboots itself. But I see no >> vmcore when it coms back up. /var/crash is empty. This is when I tried to >> write to local drive. >> >> I also tried a nfs write but then still no success. >> >> Any idea what could be missing in my steps? Or any other debug >> suggestions? Any other kdump users on Beowulf? >> > > From rpnabar at gmail.com Thu Oct 9 12:20:52 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes. Message-ID: I have a PowerEdge SC 1435 that has a strange problem. We bought about 23 of these for a cluster and machines have been failing in a somewhat random manner in a peculiar way: (1) Screen is blank. Front blue indicator turns steady orange. (2) Cannot get it to reboot by pressing (or keeping depressed) the power button (3) only way to reboot is to cycle the power. (4) After reboot machine works fine again , till after a few days same failure. Ran the dset and diagnostic CD but nothing relevant. Any tips what could be the faulty component? Or debug ideas? Right now I'm totally lost! Hardware / software? CPU / Motherboard / Power supply? Anoybody knows what exactly makes the indicator LED turn steady orange from its normal blue state? This is not one of the 4 numbered LEDs but the one to their right. I posted this problem on a PowerEdge mailing list but haven't gotten very far yet. Any suggestions are appreciated! -- Rahul From lindahl at pbm.com Thu Oct 9 12:35:07 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <20081009193507.GB2340@bx9.net> On Thu, Oct 09, 2008 at 02:20:52PM -0500, Rahul Nabar wrote: > (2) Cannot get it to reboot by pressing (or keeping depressed) the power button That's a pretty unusual hang. I bet that the reason that you don't get a kernel crash dump is that the kernel doesn't run long enough after the problem happens to create one. Probably your fastest solution is to swap parts until works. Tedious, but... -- greg From rpnabar at gmail.com Thu Oct 9 13:17:18 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] kdump / kexec to optain crash dumps from randomly crashing nodes. In-Reply-To: <48EE6294.5050509@gmail.com> References: <48EE567C.3010503@gmail.com> <48EE6294.5050509@gmail.com> Message-ID: On Thu, Oct 9, 2008 at 2:59 PM, Paolo Supino wrote: > Hi Raul > > Can it be that console messages aren't being sent to VTY (standard PC > monitor), but some other output line? That makes sense. I'll try connecting a dumb monitor to the machine and see what it reports. -- Rahul From paolo.supino at gmail.com Thu Oct 9 12:07:40 2008 From: paolo.supino at gmail.com (Paolo Supino) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] kdump / kexec to optain crash dumps from randomly crashing nodes. In-Reply-To: References: Message-ID: <48EE567C.3010503@gmail.com> Hi Rahul Did you try to redirect console to a serial port? If a system crashes and all console messages (including kernel) will be sent to the serial console that will keep displaying the messages it received until the system is power cycled ... -- ttyl Paolo Rahul Nabar wrote: > On my Centos system I installed kexec/kdump to investigate the cause of > some random system-crashes by getting access to a crash-dump. I installed > the rpm for kexec and then made the change to grub.conf that reserves the > additional memory for the new kernel. > > Also configured kdump.conf. I start the kexec service.and then I tried to > simulate a crash by echo c to sysrq-trigger. > > The system does crash and then after a while reboots itself. But I see no > vmcore when it coms back up. /var/crash is empty. This is when I tried to > write to local drive. > > I also tried a nfs write but then still no success. > > Any idea what could be missing in my steps? Or any other debug > suggestions? Any other kdump users on Beowulf? > From paolo.supino at gmail.com Thu Oct 9 12:59:16 2008 From: paolo.supino at gmail.com (Paolo Supino) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] kdump / kexec to optain crash dumps from randomly crashing nodes. In-Reply-To: References: <48EE567C.3010503@gmail.com> Message-ID: <48EE6294.5050509@gmail.com> Hi Raul Can it be that console messages aren't being sent to VTY (standard PC monitor), but some other output line? I don't know what KVM you're using, but most KVMs keep an open line only to the server they are displaying at the moment (and will send back a signal if they get a probe from an another system they are connected to) and won't buffer display output from the other systems they are connected to. What I suggested is to connect a real serial terminal to the Linux system. Make sure that all console messages (including kernel) are sent to the serial console (the default is to send them to VTY which is the standard display in a PC. If you can't connect a real serial terminal than the closest thing to it would be a PC running putty (or any other terminal emulation software) listening on COM1 and is connected to the serial port of the server. -- ttyl Paolo Rahul Nabar wrote: > Hi Paolo, > > The funny thing is that the console remains blank. We have all these > systems connected to a KVM and the kvm shows the system as actually > disconnected post the crash. > > That is what makes it so hard to debug. No screen output at all. > > -Rahul > > On Thu, Oct 9, 2008 at 2:07 PM, Paolo Supino wrote: >> Hi Rahul >> >> Did you try to redirect console to a serial port? If a system crashes >> and all console messages (including kernel) will be sent to the serial >> console that will keep displaying the messages it received until the >> system is power cycled ... >> >> >> >> >> >> -- >> ttyl >> Paolo >> >> >> >> Rahul Nabar wrote: >>> On my Centos system I installed kexec/kdump to investigate the cause of >>> some random system-crashes by getting access to a crash-dump. I installed >>> the rpm for kexec and then made the change to grub.conf that reserves the >>> additional memory for the new kernel. >>> >>> Also configured kdump.conf. I start the kexec service.and then I tried to >>> simulate a crash by echo c to sysrq-trigger. >>> >>> The system does crash and then after a while reboots itself. But I see no >>> vmcore when it coms back up. /var/crash is empty. This is when I tried to >>> write to local drive. >>> >>> I also tried a nfs write but then still no success. >>> >>> Any idea what could be missing in my steps? Or any other debug >>> suggestions? Any other kdump users on Beowulf? >>> >> From rlinesseagate at gmail.com Thu Oct 9 13:18:09 2008 From: rlinesseagate at gmail.com (Rob Lines) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <69592cde0810091318w70e8507ax4e2dae0aa849fbb5@mail.gmail.com> On Thu, Oct 9, 2008 at 3:20 PM, Rahul Nabar wrote: > I have a PowerEdge SC 1435 that has a strange problem. We bought about 23 of > these for a cluster and machines have been failing in a somewhat random manner > in a peculiar way: > > (1) Screen is blank. Front blue indicator turns steady orange. > > (2) Cannot get it to reboot by pressing (or keeping depressed) the power button > > (3) only way to reboot is to cycle the power. > > (4) After reboot machine works fine again , till after a few days same failure. > > Ran the dset and diagnostic CD but nothing relevant. > > Any tips what could be the faulty component? Or debug ideas? Right now I'm > totally lost! Hardware / software? CPU / Motherboard / Power supply? > Have you checked in the baseboard management log to see if it is throwing an error. Also check on the temperature of the machines. We have had some pretty wierd issues with ram and CPU quirkyness when they reach a high internal temperature. If you can do some poling using ipmi on the nodes to record the current temp and fan data over time so that you could see what it was at just before a crash you might be able to point it to an environmental situation. Hope this helps, Rob From csamuel at vpac.org Thu Oct 9 15:20:54 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <831359607.1340021223590533957.JavaMail.root@mail.vpac.org> Message-ID: <2098513302.1340111223590854452.JavaMail.root@mail.vpac.org> ----- "Rahul Nabar" wrote: > (1) Screen is blank. Front blue indicator turns steady orange. What we do on our CentOS cluster is disable the standard screen blanker in our startup scripts with: echo -e "\033[9;0]" >/dev/console echo -e "\033[13]" >/dev/console Very handy if a node wedges and won't respond to keyboard. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From kyron at neuralbs.com Thu Oct 9 19:52:22 2008 From: kyron at neuralbs.com (Eric Thibodeau) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] FY;) the Helmer cluster In-Reply-To: <0D49B15ACFDF2F46BF90B6E08C90048A04938211A5@quadbrsex1.quadrics.com> References: <2045508708.1313971223523657435.JavaMail.root@mail.vpac.org> <0D49B15ACFDF2F46BF90B6E08C90048A04938211A5@quadbrsex1.quadrics.com> Message-ID: <48EEC366.7030201@neuralbs.com> Dan.Kidger@quadrics.com wrote: > Looking at the CAD drawings - is there any reason why the nodes are built into circular columns, rather than traditional cuboid boxes? > If anything such a design does not make full use of the physical space available. > > And if you did pack the columns as drawn (with vertical water pipes in the gaps) - how do you service the nodes to replace failed components, water leaks et al. ? > > And also (I do work for an interconnect company after all) how would all this get wired up together as a cluster? > Awww com on, don't yo believe in fluffware? I say he goes full wifi :P > Daniel > > ------------------------------------------------------------- > Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger@quadrics.com > One Bridewell St., Mobile: +44 (0)779 209 1851 > Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519 > ----------------------- www.quadrics.com -------------------- > > > > > -----Original Message----- > From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Chris Samuel > Sent: 09 October 2008 04:41 > To: Beowulf List > Subject: Re: [Beowulf] FY;) the Helmer cluster > > > ----- "Igor Kozin" wrote: > > >> lol the guy surely is very ambitious - 4 PFLOPS for $0.9M >> http://helmer3.sfe.se/ >> > > Sadly it looks like that's based on the single precision > numbers for graphics cards, rather than double.. > > -- > Christopher Samuel - (03) 9925 4751 - Systems Manager > The Victorian Partnership for Advanced Computing > P.O. Box 201, Carlton South, VIC 3053, Australia > VPAC is a not-for-profit Registered Research Agency > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081009/f754b250/attachment.html From hearnsj at googlemail.com Thu Oct 9 22:54:03 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:52 2009 Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <9f8092cc0810092254i4cd2c426r3a37281248c5f5ee@mail.gmail.com> 2008/10/9 Rahul Nabar > I > > I posted this problem on a PowerEdge mailing list but haven't gotten > very far yet. Any suggestions are appreciated! > > Yes. (1) Tell your Dell salesman that you have asked for help on this problem on a public mailing list for High Performance Computing. Tell him/her that you need high level Dell support on this. There are Dell customers on this list. (2) Suspect the RAM. Ask some serious questions of your Dell support about RAM compatibility - HPC applications stress the RAM. Ask, and ask again, if the specific RAM chips you have are certified for that motherboard. Use dmidecode to read out the manufacturer codes of the RAM modules - do you have a mix of manufacturers? Ask and ask again about BIOS updates being available for these machines. We had a case once of HP machines - even though the BIOSes were versioned the same on 200 machines, there were some differences - turns out you had to go as far as checking the build date. Get the very latest BIOS version you can. (3) The RAM will be the problem - but if you can keep notes and there are specific machines which crash more than others point this out to Dell and maybe suspect the PSUs being weak on those machines. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081010/c98637a9/attachment.html From hearnsj at googlemail.com Fri Oct 10 01:17:27 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <9f8092cc0810092254i4cd2c426r3a37281248c5f5ee@mail.gmail.com> References: <9f8092cc0810092254i4cd2c426r3a37281248c5f5ee@mail.gmail.com> Message-ID: <9f8092cc0810100117u316b5f46j7d6630e6747c2d8f@mail.gmail.com> 2008/10/10 John Hearns > > (1) Tell your Dell salesman that you have asked for help on this problem > on a public mailing list for High Performance Computing. Tell him/her that > you need high level Dell support on this. > ps. By high level support I mean that you are put in direct contact with one of the engineers responsible for the design of these systems in the factory. You do not want to talk to the normal support chain, and be asked if you have run some diagnostics program downloaded from the Dell site, etc. As a general observation, not particularly aimed at Dell, I have seen this type of behaviour before. Beowulf clusters are built from COTS systems - servers which are disigned for a workload of webserving, or running farms of virtual servers. I really think that the manufacturers would be surprised at the typical HPC workload - running all cores flat out at 100%, having applications which can comfortably use a couple of hundred gigs of RAM. My feeling is that the PSUs are specced to cope with these loads, but any out of spec ones will be on the edge. RAM timings too - if you are stressing all the RAM in a system you'll show up any weaknesses. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081010/98d8da82/attachment.html From rpnabar at gmail.com Fri Oct 10 07:36:53 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. Message-ID: >What we do on our CentOS cluster is disable the standard >screen blanker in our startup scripts with: > >echo -e "\033[9;0]" >/dev/console >echo -e "\033[13]" >/dev/console > >Very handy if a node wedges and won't respond to keyboard. Great idea, Chris. I did not know this. That could be why I don't see any output on my screens later. -- Rahul From rpnabar at gmail.com Fri Oct 10 07:47:32 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. Message-ID: >t's a pretty unusual hang. I bet that the reason that you don't get >a kernel crash dump is that the kernel doesn't run long enough after >the problem happens to create one. Thanks Greg. I suspected that....I am actually curious: when exactly can kdump be useful? If a crash is hardware precipitated the second kernel never gets a chance to do what its supposed to. If it is software related, and the first kernel actually has time to detect the inconsistancy then it might as well "deal" with the offending process. >Probably your fastest solution is to swap parts until works. Tedious, >but... That's exactly what I'm doing so far! :-) Problem is which ones? CPU / MB/ Power supplies /RAM ? I've even received solutions as exotic as re-flashing the BIOS / ESM firmware upgrades / processor reseating. Any bets on the likelihood based on symptoms, intuition, and past experience? We've swapped CPUs and processors on all the offending nodes. Seems to have worked so far (i.e. none of the swapped machines have re-crashed) But I'm hesitant to conclude "problem solved" since all this is only over the last 2 weeks. I'm dreading the day when one of the swapped machines re-crashes! Let's see........ -- Rahul From rpnabar at gmail.com Fri Oct 10 07:54:35 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. Message-ID: >Have you checked in the baseboard management log to see if it is >throwing an error. Apparently the SC1435 does not have OpenManage. "Simple Computing" is too simple to warrant that, I was told. They do have dset to look at the ESM logs but not for CentOS nor Fedora. Redhat is their "validated" [sic] OS. That's the only one they support. So I'm sort of stuck there. > Also check on the temperature of the machines. We >have had some pretty wierd issues with ram and CPU quirkyness when >they reach a high internal temperature. If you can do some poling >using ipmi on the nodes to record the current temp and fan data over >time so that you could see what it was at just before a crash you >might be able to point it to an environmental situation. I'll try ipmi. I was trying lm_sensors but apparantly it does not have a driver for this chipset / motherboard combination. Not sure if its an AMD Opteron specific driver issue or a vendor-not-relesing-motherboard-specs issue (heard both versions on the net). Anybody else had success using lm_sensors on the SC1435? -- Rahul From rpnabar at gmail.com Fri Oct 10 08:05:34 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. Message-ID: >(1) Tell your Dell salesman that you have asked for help on this problem on >a public mailing list for High Performance Computing. Tell him/her that you >need high level Dell support on this. There are Dell customers on this list. Thanks John. I will do that. A question: how likely is it that this is a software issue and not hardware from my symptoms? They keep harping on the fact that I am running a non-validated OS. We used to run Fedora. Now run CentOS. Same issues. They only support RedHat. I have a hard time being 100% certain but the more I see it the more I am convinced it is the hardware. >(2) Suspect the RAM. Ask some serious questions of your Dell support about >RAM compatibility - HPC applications stress the RAM. Ask, and ask again, if >the specific RAM chips you have are certified for that motherboard. Use >dmidecode to read out the manufacturer codes of the RAM modules - do you >have a mix of manufacturers? Very good idea. Never tried that. I will check. I assumed that they were all similar systems and I had compatible RAM since I bought it all packaged together. >Ask and ask again about BIOS updates being available for these machines. >We had a case once of HP machines - even though the BIOSes were versioned >the same on 200 machines, there were some differences - turns out you had to >go as far as checking the build date. >Get the very latest BIOS version you can. I have the latest. But that's only based on the version #. I will dig deeper. Could this be bad BIOS, though, from the symptoms? So, some code somewhere switches the state of that LED from blue to orange and if only I knew what the trigger was supposed to be. Someone had to write that! > >(3) The RAM will be the problem - but if you can keep notes and there are >specific machines which crash more than others point this out to Dell and >maybe suspect the PSUs being weak on those machines. Yes. The crashes seem to be very clustered. We have had 5 specific machines out of 23 crash repeatedly. We swapped the motherboard+cpus on those and they do not seem to have crashed again as yet. But the time scale is only about 2 weeks. So I am not very confident of the statistical significance of my conclusions. -- Rahul From hearnsj at googlemail.com Fri Oct 10 08:22:40 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <9f8092cc0810100822m7c8feb13mfde9879aa1e0cf6b@mail.gmail.com> 2008/10/10 Rahul Nabar > > > Thanks John. I will do that. A question: how likely is it that this is > a software issue and not hardware from my symptoms? They keep harping > on the fact that I am running a non-validated OS. I have had that line from a few companies. Looks like you are talking to first or second line support people - this is what they have been trained to say. You can understand this - it stops idiots calling in to them who are trying to run some spit-and-sawdust distribution they got free with a packet of breakfast serial. Again, just use your salesman or account manager and say that you spent $$$ with Dell to run applications, and that the systems were sold to you as being able to run Linux. > I have the latest. But that's only based on the version #. I will dig > deeper. Could this be bad BIOS, though, from the symptoms? So, some > code somewhere switches the state of that LED from blue to orange and > if only I knew what the trigger was supposed to be. Someone had to > write that! The light coming on indicates a fault. You should be interrogating the onboard management controller (called an IPMI card ort a BMC, or in Dell speak I think a DRAC). Log onto the suspect nodes anr run ipmitool: ipmitool -I open sel elist Do you not get a fault code on the little screen beside the light? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081010/0b874aff/attachment.html From hearnsj at googlemail.com Fri Oct 10 08:29:00 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <9f8092cc0810100829w3feca740j79c1822fcb7749f1@mail.gmail.com> 2008/10/10 Rahul Nabar > > > Yes. The crashes seem to be very clustered. We have had 5 specific > machines out of 23 crash repeatedly. We swapped the motherboard+cpus > on those and they do not seem to have crashed again as yet. But the > time scale is only about 2 weeks. So I am not very confident of the > statistical significance of my conclusions. > > > Rahul, run that past us once more please. You have bought 23 machines and FIVE of them have had CPUs and motherboards replaced? Errrrr... By the way, I live in London, UK. The news is our local government had lots of money in an Icelandic bank, which they've now lost. As a consequence, they are looking to sell off objects of historic significance which they own. I can put you in touch if you are looking for a Victorian-era lifting bridge. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081010/abe6cd2c/attachment.html From hearnsj at googlemail.com Fri Oct 10 08:38:50 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <9f8092cc0810100822m7c8feb13mfde9879aa1e0cf6b@mail.gmail.com> References: <9f8092cc0810100822m7c8feb13mfde9879aa1e0cf6b@mail.gmail.com> Message-ID: <9f8092cc0810100838i4a5f24du32d9dff619ab832f@mail.gmail.com> Actually, let's be fair to Dell here. They have replaced five machines. Those five machines will have an up-to-date BIOS version. There may well have been a problem which has been fixed in this BIOS version. I would use dmidecode, and make sure that all your machines have an identical BIOS version. Any that do not - download and flash the BIOS. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081010/8ad17f42/attachment.html From dnlombar at ichips.intel.com Fri Oct 10 08:55:33 2008 From: dnlombar at ichips.intel.com (Lombard, David N) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <2098513302.1340111223590854452.JavaMail.root@mail.vpac.org> References: <831359607.1340021223590533957.JavaMail.root@mail.vpac.org> <2098513302.1340111223590854452.JavaMail.root@mail.vpac.org> Message-ID: <20081010155533.GA21681@nlxdcldnl2.cl.intel.com> On Thu, Oct 09, 2008 at 03:20:54PM -0700, Chris Samuel wrote: > > What we do on our CentOS cluster is disable the standard > screen blanker in our startup scripts with: To clarify, see console_codes(4): > echo -e "\033[9;0]" >/dev/console Sets screen blank timeout to 0 > echo -e "\033[13]" >/dev/console Unblanks the console -- David N. Lombard, Intel, Irvine, CA I do not speak for Intel Corporation; all comments are strictly my own. From lindahl at pbm.com Fri Oct 10 13:28:10 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <20081010202810.GD11550@bx9.net> On Fri, Oct 10, 2008 at 09:54:35AM -0500, Rahul Nabar wrote: > I'll try ipmi. I was trying lm_sensors but apparantly it does not have > a driver for this chipset / motherboard combination. The CentOS lmsensors package is pretty old. You can get a more up-to-date one (2.10) from http://atrpms.net/dist/el5/lm_sensors/ -- greg From lindahl at pbm.com Fri Oct 10 13:36:21 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <20081010203621.GE11550@bx9.net> On Fri, Oct 10, 2008 at 09:47:32AM -0500, Rahul Nabar wrote: > Thanks Greg. I suspected that....I am actually curious: when exactly > can kdump be useful? For software bugs, but probably only if you're a kernel developer. Otherwise just capturing the oops or printks is most of what you'd want to know. > >Probably your fastest solution is to swap parts until works. Tedious, > >but... > > That's exactly what I'm doing so far! :-) Problem is which ones? All of them, of course. If you can cause the crash fast enough, it's easy. It sounds like your crash happens pretty slowly, which is the hardest kind of problem to diagnose. -- greg From andrea at soe.ucsc.edu Fri Oct 10 00:52:33 2008 From: andrea at soe.ucsc.edu (Andrea Di Blas) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl> References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> <48EB69B6.5080602@tamu.edu> <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl> Message-ID: hi dan, vincent, gerry, bodgan, and all of you, first of all, thank you very much for your answers - I really appreciate. what I understand from your messages is that there isn't what I was looking for - or maybe that my question wasn't clear enough. what I was wondering about is if there are any MPI-based applications that are well known, either because they are being sold as such, or because they are known to be used by (large) companies. for instance, we all know that google uses mapreduce on top of gfs. we know that yahoo uses a version of mapreduce called hadoop. anything similar for MPI? or - are there any commercial products based on MPI that one could buy and run in their own company/institution? for example, let's say I want do do some simulations of whatever kind, and I like matlab. can I go buy a cluster of some kind, and also buy some matlab implementation that will use MPI to run on my cluster? or some other MPI-based tool? is all software like that still for internal use only of the entities that develop it? thanks again, andrea -- Andrea Di Blas UCSC School of Engineering Tel: (831) 459 4193 Fax: (831) 459 4829 On Wed, 8 Oct 2008, Vincent Diepeveen wrote: > > On Oct 8, 2008, at 4:09 PM, Bogdan Costescu wrote: > >> On Tue, 7 Oct 2008, Vincent Diepeveen wrote: >> >>> Drug type companies eat typical 0.5% system time. >> >> Could you please explain what you mean here ? That they buy lots of CPUs >> but only use a fraction of them or only for a fraction of time ? >> > > When you add up others. > > Note i see in IWR you got a 160 node opteron cluster, what software eats > system time on that cluster? > There must be graphs about it, want to share one? Also interesting to know is > how many cores most jobs eat! > > Note that i left out 1 important factor which is not a factor in every > nation: that's the amount of system time airindustry eats of supercomputers > designing new stuff. It is very significant together with car manufacturers. > With 160 nodes machine, even if that would be power6 nodes, > you can't even design a wing for aircraft. > >>> They are far behind, which is not so logical; i would consider it old >>> fashioned to do experiments on monkeys (Bar-ilan brain experiments on >>> monkeys rings a bell?), so doing everything in software is far more >>> logical. >> >> Maybe your knowledge about the pharma industry is far behind ? "Doing >> everything in software is far more logical" only when you can exactly >> express biological processes in mathematical terms which is not true in the >> great majority of cases. F.e. molecular dynamics (often used to simulate >> interactions at molecular level, f.e. of a drug injected into the body) is >> based on several approximations and the implementations are subject to >> numerical errors. Would you take a drug that was produced based only on a >> simulation with molecular dynamics and found to be good enough ? >> > > Well, i sure avoid hospital visits, you? > > As you know the healthcare commission is basically old men who spread careful > chosen sentences in such a manner that > everything looks fine on this planet and that there is nothing to worry > about; last time i spoke with them on EMF's a guy being spokesman > on this subject, he didn't even know how the electricity system in his own > house worked (whether on the first floor of his house > he had earth). > > So lucky to introduce to a new drug, there is some burocratic systems. There > is several phases of testing that a drug > has to pass in order to get accepted as a drug that is legal. Not to confuse > with the 'advice' on whether a drug gets prescribed; > they want way cheaper drugs for that usually than the ones that might maybe > be better (say 0.5% better for a lot of money extra, > but if you're that guy that didn't get cured as a result of that, your family > will regret it bigtime). > > Computing power or not, most manufacturers simply want to sell a product, so > the amount of systemtime invested on a computer, > in order to sell, is total irrelevant to them to get a drug accepted usually. > The first and most important part of the show to get a drug > accepted is getting invited to the table to show your drug. Computing power > is nice to mention here when that is usually not a factor > at all to get a new medicine accepted. > > But it may change, let's hope so... > > Maybe a new breed of company can do the job. A breed where making profit is > not most important :) > >> Similar concerns apply to all levels - the brain, as you mention it, is >> still very much an enigma, otherwise artificial intelligence would be all >> around us (or not, if deemed to dangerous...). And when you don't know how >> an organ or system works, how can you develops drugs only by simulations ? >> >>> When someone is really in big need for big calculation power, one makes >>> dedicated cpu's for it. >> >> I tend to agree, provided that the dedicated CPU brings a really big >> advantage, like an order of more of magnitude. F.e., to stay in the pharma >> related field, that's the bussiness case for DEShaw Research's molecular >> dynamics chip (Anton) which is supposed to bring a significant increase in >> speed for MD simulations compared with software solutions running on >> general purpose CPUs (or GPUs). > > Some hardware guys can better comment here. Making your own FPGA chippie is > rather cheap. > Finding and paying a few good guys to get the chip done is toughest. You > basically need 1 good > programmer, the rest can get hired parttime in fact. After chip works, > to print a 1000 cards, it is like a 100 dollar for pci-card and 50 dollar a > chip. > > So $150k for computing power that total annihilates any supercomputer of now > and the NEAR future for your specific application. > > Real cpu's are a lot faster of course, price for them i do not have at hand. > I heard it could be cheaper > than this in fact if you regurarly print a batch. Such a batch is producing > less cpu's than the FPGA, where you pay for 1 cpu a constant > amount. I tend to remember that good cpu designers, not to mention memory > controller guys, are rather expensive folks, as > they are very popular; on some other mailing list with some very funny > lastnames, i see nonstop job offers for that type of folks. > > Vincent > > >> -- >> Bogdan Costescu >> >> IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany >> Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 >> E-mail: bogdan.costescu@iwr.uni-heidelberg.de > From rlinesseagate at gmail.com Fri Oct 10 08:19:14 2008 From: rlinesseagate at gmail.com (Rob Lines) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <69592cde0810100819w7dbae26bi6e162445cb3ada88@mail.gmail.com> On Fri, Oct 10, 2008 at 10:54 AM, Rahul Nabar wrote: >>Have you checked in the baseboard management log to see if it is >>throwing an error. > > Apparently the SC1435 does not have OpenManage. "Simple Computing" is > too simple to warrant that, I was told. They do have dset to look at > the ESM logs but not for CentOS nor Fedora. Redhat is their > "validated" [sic] OS. That's the only one they support. So I'm sort of > stuck there. We have a cluster of SC1435 without the DRAC card but they still have the baseboard management. It should be accessible right at the end of the POST by hitting ctrl+e and normally it shows a little splash screen with the ip it has been configured with and what keys to press. It gives some great information especially if you are running into ram issues. I don't remember if we had to run on the logging or if it was on from the factory. > > I'll try ipmi. I was trying lm_sensors but apparantly it does not have > a driver for this chipset / motherboard combination. Not sure if its > an AMD Opteron specific driver issue or a > vendor-not-relesing-motherboard-specs issue (heard both versions on > the net). Anybody else had success using lm_sensors on the SC1435? We never tried to deal with lm_sensors on these machines because ipmi had more options. We are using CentOS 5 on our cluster machines. Here are a few links we used when working on ipmi. OpenIPMI http://www.barryodonovan.com/index.php/2007/04/11/dell-ipmi/http://66.102.9.104/search?q=cache:IoPmIimMExQJ:www.barryodonovan.com/index.php/2007/04/11/dell-ipmi/+linux+ipmi+temperature+sensor+read&hl=en&gl=us&strip=1 http://www.hollenback.net/index.php/LinuxServerManagementIpmi http://openipmi.sourceforge.net/ http://ipmitool.sourceforge.net/ http://linux.dell.com/files/presentations/Red_Hat_Summit_May_2006/ipmi_presentation-redhat_summit.pdf http://buttersideup.com/docs/howto/IPMI_on_Debian.html >We used to run Fedora. Now run CentOS. Same issues. They only support RedHat. I have a hard time being 100% certain but the more I see it the more I am convinced it is the hardware. The real key is to go back to the sales person that you purchased them from and tell them the problems you are having and have them help escalate the situation. I am not sure what diagnostics you are running but they have a bootable cd for doing a memory test at the link below. Even when they have tried to point at an OS problem if I can get an error on that cd they really have nothing that they can hide behind. http://support.dell.com/support/downloads/format.aspx?c=us&cs=19&l=en&s=dhs&deviceid=196&libid=13&releaseid=R169189&vercnt=5&formatcnt=0&SystemID=PWE_XEO_1435SC&servicetag=85PG2F1&os=LIN4&osl=en&catid=-1&impid=-1 Best of luck, Rob From lindahl at pbm.com Fri Oct 10 13:53:00 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> <48EB69B6.5080602@tamu.edu> <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl> Message-ID: <20081010205300.GF11550@bx9.net> On Fri, Oct 10, 2008 at 12:52:33AM -0700, Andrea Di Blas wrote: > what I was wondering about is if there are any MPI-based applications > that are well known, either because they are being sold as such, or > because they are known to be used by (large) companies. Many commercial programs like the ones that do CFD or finite element analysis are MPI-based and are used by big companies like Boeing. Fluent, Ansys, etc etc. > for example, let's say I want > do do some simulations of whatever kind, and I like matlab. There are many ways to use Matlab with MPI, including one now distributed by Mathworks. -- greg From jlb17 at duke.edu Fri Oct 10 14:05:53 2008 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: <20081010205300.GF11550@bx9.net> References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> <48EB69B6.5080602@tamu.edu> <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl> <20081010205300.GF11550@bx9.net> Message-ID: On Fri, 10 Oct 2008 at 1:53pm, Greg Lindahl wrote > On Fri, Oct 10, 2008 at 12:52:33AM -0700, Andrea Di Blas wrote: > >> what I was wondering about is if there are any MPI-based applications >> that are well known, either because they are being sold as such, or >> because they are known to be used by (large) companies. > > Many commercial programs like the ones that do CFD or finite element > analysis are MPI-based and are used by big companies like Boeing. And small research labs (my thesis research was done with DYNA). -- Joshua Baker-LePain QB3 Shared Cluster Sysadmin UCSF From hearnsj at googlemail.com Sat Oct 11 01:27:38 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> <48EB69B6.5080602@tamu.edu> <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl> Message-ID: <9f8092cc0810110127w59d1b9cap1db5a15483bf4e51@mail.gmail.com> 2008/10/10 Andrea Di Blas > hi dan, vincent, gerry, bodgan, and all of you, > > > what I was wondering about is if there are any MPI-based applications that > are well known, either because they are being sold as such, or because they > are known to be used by (large) companies. for instance, we all know that > google uses mapreduce on top of gfs. we know that yahoo uses a version of > mapreduce called hadoop. anything similar for MPI? > > or - are there any commercial products based on MPI that one could buy and > run in their own company/institution? You mean like: Fluent www.fluent.com LS-Dyna http://www.ansys.com/products/lsdyna.asp Need I go on? If you're looking at commercial applications, why not look at the documentation on HP-MPI http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801/?ciid=a308a8ea6ce02110a8ea6ce02110275d6e10RCRD and their ISV page: http://h21007.www2.hp.com/portal/site/dspp/menuitem.863c3e4cbcdc3f3515b49c108973a801/?ciid=5aeeb16b907b4110VgnVCM100000275d6e10RCRD 26 companies listed there. (Yeah, I know HP isn't the only kid on the block, but it does tend to be used by big commercial applications) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081011/b731b71e/attachment.html From csamuel at vpac.org Sun Oct 12 20:36:30 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: Message-ID: <1909233229.1425551223868990145.JavaMail.root@mail.vpac.org> ----- "Rahul Nabar" wrote: > >Very handy if a node wedges and won't respond to keyboard. > > Great idea, Chris. I did not know this. That could be why I don't > see any output on my screens later. Real kudos goes to Brett here at VPAC, he dug out the codes, I just suggested it would be a good idea. :-) -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From Dan.Kidger at quadrics.com Mon Oct 13 02:47:36 2008 From: Dan.Kidger at quadrics.com (Dan.Kidger@quadrics.com) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Off topic: equipment sell-off In-Reply-To: <9f8092cc0810100829w3feca740j79c1822fcb7749f1@mail.gmail.com> References: <9f8092cc0810100829w3feca740j79c1822fcb7749f1@mail.gmail.com> Message-ID: <0D49B15ACFDF2F46BF90B6E08C90048A04938211D0@quadbrsex1.quadrics.com> John Hearns wrote: > By the way, I live in London, UK. The news is our local government had lots of money in an Icelandic bank, which they've now lost. > As a consequence, they are looking to sell off objects of historic significance which they own. I can put you in touch if you are looking > for a Victorian-era lifting bridge. We also have financial problems in local government and have a bridge to sell - unfortunately ours does not lift up, but on the plus side it is high enough above the water for most boats to fit under. It will be put on eBay soon, but for now here is a picture: http://www.bristol.ac.uk/university/gallery/city-of-bristol/suspension.html Daniel ------------------------------------------------------------- Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Mobile: +44 (0)779 209 1851 Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519 ----------------------- www.quadrics.com -------------------- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of John Hearns Sent: 10 October 2008 16:29 To: beowulf@beowulf.org Subject: Re: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. 2008/10/10 Rahul Nabar > Yes. The crashes seem to be very clustered. We have had 5 specific machines out of 23 crash repeatedly. We swapped the motherboard+cpus on those and they do not seem to have crashed again as yet. But the time scale is only about 2 weeks. So I am not very confident of the statistical significance of my conclusions. Rahul, run that past us once more please. You have bought 23 machines and FIVE of them have had CPUs and motherboards replaced? Errrrr... By the way, I live in London, UK. The news is our local government had lots of money in an Icelandic bank, which they've now lost. As a consequence, they are looking to sell off objects of historic significance which they own. I can put you in touch if you are looking for a Victorian-era lifting bridge. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081013/bb76838a/attachment.html From mm at yuhu.biz Mon Oct 13 02:58:09 2008 From: mm at yuhu.biz (Marian Marinov) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: MOSIX2 In-Reply-To: <200810011603.10435.mm@yuhu.biz> References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E36012.1070406@rri.sari.ac.uk> <200810011603.10435.mm@yuhu.biz> Message-ID: <200810131258.10041.mm@yuhu.biz> http://linuxpmi.org is alive again. There was a network problem which is now fixed and the site is running again. Regards Marian On Wednesday 01 October 2008 16:03:10 Marian Marinov wrote: > I lost touch with Juri (the guy who started the documentation project of > openMosix) 2-3 months ago. > > More info about LinuxPMI you can find on Freenode #linuxpmi or #openmosix > > As far as I can see the domain is registered until next year and there is > some routing problem New York before the server. > > I'm sorry but I do not know of any mirrors as Juri's machine was the > central storage of the documented patches. > > Hi actually split the openMosix into series of patches so that it can be > ported to 2.6 easier. > > Regards > Marian > > On Wednesday 01 October 2008 14:33:38 Tony Travis wrote: > > Tony Travis wrote: > > > [...] > > > Hello, Marian. > > > > > > Great! I didn't know about LinuxMPI: I've not been following openMosix > > > developments recently. I can't access their web site - Is it mirrored > > > anywhere? > > > > Following up my own message, I meant LinuxPMI of course ;-) > > > > Not much evidence that the LinuxPMI project is still active, though. > > Google reports a lot of hits for the pending DNS de-registration of > > "linuxmpi.com'... > > > > Anyone else know what's happening with LinuxPMI? > > > > Thanks, > > > > Tony. > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From iioleynik at gmail.com Mon Oct 13 16:57:57 2008 From: iioleynik at gmail.com (Ivan Oleynik) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons Message-ID: I am still in process of purchasing a new cluster and consider whether is worth waiting for new Intel Xeons. Does someone know when Intel will start selling Xeons based on Nehalem architecture? They announced desktop launch (Nov 16) but were quiet about server release. Thanks, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081013/243225b7/attachment.html From hbugge at platform.com Mon Oct 13 22:50:32 2008 From: hbugge at platform.com (=?iso-8859-1?Q?H=E5kon?= Bugge) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: References: Message-ID: They are _definitively_ worth waiting for, although I am not familiar with the release timing. But I have been running on a dual-socket with 8 cores and 16 SMTs. And I say they are worth waiting for. H?kon At 01:57 14.10.2008, Ivan Oleynik wrote: >I am still in process of purchasing a new >cluster and consider whether is worth waiting >for new Intel Xeons. Does someone know when >Intel will start selling Xeons based on Nehalem >architecture? They announced desktop launch (Nov >16) but were quiet about server release. > >Thanks, > >Ivan > >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or >unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- H?kon Bugge Chief Technologist mob. +47 92 48 45 14 off. +47 21 37 93 19 fax. +47 22 23 36 66 hbugge@platform.com Skype: hakon_bugge Platform Computing, Inc. From hbugge at platform.com Tue Oct 14 01:21:45 2008 From: hbugge at platform.com (=?iso-8859-1?Q?H=E5kon?= Bugge) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: References: Message-ID: ... and, the statement below is based on comparing Harpertown to Nehalem. I have not tested products from Intel's competitors and compared them to Nehalem. Thanks, H?kon (speaking for himself) At 07:50 14.10.2008, H?kon Bugge wrote: >They are _definitively_ worth waiting for, >although I am not familiar with the release >timing. But I have been running on a dual-socket >with 8 cores and 16 SMTs. And I say they are worth waiting for. > > >H?kon >At 01:57 14.10.2008, Ivan Oleynik wrote: >>I am still in process of purchasing a new >>cluster and consider whether is worth waiting >>for new Intel Xeons. Does someone know when >>Intel will start selling Xeons based on Nehalem >>architecture? They announced desktop launch >>(Nov 16) but were quiet about server release. >> >>Thanks, >> >>Ivan >> >>_______________________________________________ >>Beowulf mailing list, Beowulf@beowulf.org >>To change your subscription (digest mode or >>unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > >-- >H?kon Bugge >Chief Technologist >mob. +47 92 48 45 14 >off. +47 21 37 93 19 >fax. +47 22 23 36 66 >hbugge@platform.com >Skype: hakon_bugge > >Platform Computing, Inc. > > >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or >unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- H?kon Bugge Chief Technologist mob. +47 92 48 45 14 off. +47 21 37 93 19 fax. +47 22 23 36 66 hbugge@platform.com Skype: hakon_bugge Platform Computing, Inc. From ajt at rri.sari.ac.uk Tue Oct 14 02:24:05 2008 From: ajt at rri.sari.ac.uk (Tony Travis) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: MOSIX2 In-Reply-To: <200810131258.10041.mm@yuhu.biz> References: <039801c92260$ab3630f0$1f9886a5@its99fd46g> <48E36012.1070406@rri.sari.ac.uk> <200810011603.10435.mm@yuhu.biz> <200810131258.10041.mm@yuhu.biz> Message-ID: <48F46535.9070708@rri.sari.ac.uk> Marian Marinov wrote: > http://linuxpmi.org is alive again. > > There was a network problem which is now fixed and the site is running again. Hello, Marian. I've been checking it periodically, and I noticed it was back up, but thanks for letting me know. Last time I looked, there had not been any changes for a while, but I've signed up to the new openmosix-dev mailing list to keep upm with progress. I'll continue running openMosix for a while, until I decide what to replace it with - Maybe openMosix :-) Tony. -- Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk mailto:ajt@rri.sari.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt From kus at free.net Tue Oct 14 09:15:19 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: Message-ID: In message from H?kon Bugge (Tue, 14 Oct 2008 07:50:32 +0200): >They are _definitively_ worth waiting for, although I am not familiar >with the release timing. But I have been running on a dual-socket >with 8 cores and 16 SMTs. And I say they are worth waiting for. Q1'2009 - unfortunately, I don't know more exactly :-( Mikhail Kuzminsky Computer Assistance to Chemical Research Center, Zelinsky Institute of Organic Chemistry Moscow > > >H?kon >At 01:57 14.10.2008, Ivan Oleynik wrote: >>I am still in process of purchasing a new cluster and consider >>whether is worth waiting for new Intel Xeons. Does someone know when >>Intel will start selling Xeons based on Nehalem architecture? They >>announced desktop launch (Nov 16) but were quiet about server >>release. >> >>Thanks, >> >>Ivan >> >>_______________________________________________ >>Beowulf mailing list, Beowulf@beowulf.org >>To change your subscription (digest mode or unsubscribe) visit >>http://www.beowulf.org/mailman/listinfo/beowulf > >-- >H?kon Bugge >Chief Technologist >mob. +47 92 48 45 14 >off. +47 21 37 93 19 >fax. +47 22 23 36 66 >hbugge@platform.com >Skype: hakon_bugge > >Platform Computing, Inc. > > >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf From prentice at ias.edu Tue Oct 14 11:13:55 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <0D49B15ACFDF2F46BF90B6E08C90048A049382116E@quadbrsex1.quadrics.com> <3A9122B4-7C80-4EF5-82F7-E5C69F987B28@xs4all.nl> <48EB69B6.5080602@tamu.edu> <57789E59-FA20-4E69-ABA0-8C707CD82E86@xs4all.nl> Message-ID: <48F4E163.4040202@ias.edu> Andrea Di Blas wrote: > > or - are there any commercial products based on MPI that one could buy > and run in their own company/institution? for example, let's say I want > do do some simulations of whatever kind, and I like matlab. can I go buy > a cluster of some kind, and also buy some matlab implementation that > will use MPI to run on my cluster? or some other MPI-based tool? is all > software like that still for internal use only of the entities that > develop it? OpenEye (http://www.eyesopen.com) makes computational chemistry software that uses PVM. While not MPI, it's similar in that it uses a message-passing paradigm. I supported this software at a pharmaceutical company up until about a year ago. There was talk of porting their programs to MPI, since PVM is relatively obsolete these days. I just checked their website, and still see PVM being listed as the technology of choice for FRED, ROCS and OMEGA. MatLab has a parallel computing toolbox that uses MPI-2 interface: http://www.mathworks.com/products/parallel-computing/parallel/ Schrodinger (http://www.schrodinger.com) has a few computationa chemistry applications that use MPI. Desmond and Parallel Jaguar are two. These are all commercial applications that anyone (with enough $$$) can buy, and the use MPI right out of the box. The Schrodinger applications also support the major queuing systems (SGE, PBS/Torge and Platform LSF) right out of the box, too, and even provides very good documentation for setting up your parallel environment/queue to work with their software. -- Prentice From prentice at ias.edu Tue Oct 14 11:19:41 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <48F4E2BD.1030008@ias.edu> Rahul Nabar wrote: > > Thanks John. I will do that. A question: how likely is it that this is > a software issue and not hardware from my symptoms? They keep harping > on the fact that I am running a non-validated OS. We used to run > Fedora. Now run CentOS. Same issues. They only support RedHat. I have > a hard time being 100% certain but the more I see it the more I am > convinced it is the hardware. Dell supports CentOS. They better - that's one of the OSes they will install if you buy a cluster from them. I suspect the technician you are dealing with doesn't even know this. And you can always lie to the support - CentOS, for all practical and technical purposes, *IS* RHEL. -- Prentice From prentice at ias.edu Tue Oct 14 11:25:25 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <9f8092cc0810100822m7c8feb13mfde9879aa1e0cf6b@mail.gmail.com> References: <9f8092cc0810100822m7c8feb13mfde9879aa1e0cf6b@mail.gmail.com> Message-ID: <48F4E415.9060205@ias.edu> John Hearns wrote: > > Do you not get a fault code on the little screen beside the light? Raul, Look on the front of your system, there should be numbers 1, 2, 3, and 4 next to the power switch. If you get an error, some of these will light up to produce an error code. you can give that error code to support to help narrow down the source of the problem. I don't think the meaning of the codes are documented anywhere public. For example, if you have 1, 3 and 4 light up, that is error code 134, which means "something connected to the motherboard is malfunctioning". In my case, that turned out to be one of the processors. -- Prentice From lindahl at pbm.com Tue Oct 14 11:54:38 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <48F4E2BD.1030008@ias.edu> References: <48F4E2BD.1030008@ias.edu> Message-ID: <20081014185438.GA12924@bx9.net> On Tue, Oct 14, 2008 at 02:19:41PM -0400, Prentice Bisbal wrote: > And you can always lie to the support - CentOS, for all practical and > technical purposes, *IS* RHEL. But this requires cleverness -- often install scripts and the like look at /etc/redhat-release or the redhat-release rpm. -- greg From iioleynik at gmail.com Tue Oct 14 12:53:37 2008 From: iioleynik at gmail.com (Ivan Oleynik) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: References: Message-ID: Thanks, H?kon. Did you really hold the Nehalem Xeon chips in your hands? They probably require new motherboards with a new chipset. It would be nice to hear from you some numbers concerning Harpertown vs Nehalem performance. Unfortunately, I can not wait beyond Jan 1 due to financial constraints. Ivan On Tue, Oct 14, 2008 at 1:50 AM, H?kon Bugge wrote: > They are _definitively_ worth waiting for, although I am not familiar with > the release timing. But I have been running on a dual-socket with 8 cores > and 16 SMTs. And I say they are worth waiting for. > > > H?kon > > At 01:57 14.10.2008, Ivan Oleynik wrote: > >> I am still in process of purchasing a new cluster and consider whether is >> worth waiting for new Intel Xeons. Does someone know when Intel will start >> selling Xeons based on Nehalem architecture? They announced desktop launch >> (Nov 16) but were quiet about server release. >> >> Thanks, >> >> Ivan >> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> > > -- > H?kon Bugge > Chief Technologist > mob. +47 92 48 45 14 > off. +47 21 37 93 19 > fax. +47 22 23 36 66 > hbugge@platform.com > Skype: hakon_bugge > > Platform Computing, Inc. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081014/ccdc9e0a/attachment.html From i.n.kozin at googlemail.com Tue Oct 14 14:00:16 2008 From: i.n.kozin at googlemail.com (Igor Kozin) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: References: Message-ID: > > Did you really hold the Nehalem Xeon chips in your hands? They probably > require new motherboards with a new chipset. > absolutely > It would be nice to hear from you some numbers concerning Harpertown vs > Nehalem performance > those who know will not be able tell you because of the NDA. i think it is fair to say that the major difference is expected in the memory bandwidth. > > Unfortunately, I can not wait beyond Jan 1 due to financial constraints. > then make your choice between Barcelona and Harpertown. if you are lucky you can get Shanghai which is expected by the end of the year. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081014/d694409a/attachment.html From bill at cse.ucdavis.edu Tue Oct 14 14:11:57 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <20081014185438.GA12924@bx9.net> References: <48F4E2BD.1030008@ias.edu> <20081014185438.GA12924@bx9.net> Message-ID: <48F50B1D.3040605@cse.ucdavis.edu> Greg Lindahl wrote: > On Tue, Oct 14, 2008 at 02:19:41PM -0400, Prentice Bisbal wrote: > >> And you can always lie to the support - CentOS, for all practical and >> technical purposes, *IS* RHEL. > > But this requires cleverness -- often install scripts and the like > look at /etc/redhat-release or the redhat-release rpm. Indeed, not to mention support is trained to detect centos. In particular there are differences under /proc related to module signatures. I've heard of this happening on several occasions when folks are using popular commercial software with support on centos instead of RHEL. From jcownie at cantab.net Tue Oct 14 14:31:42 2008 From: jcownie at cantab.net (James Cownie) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: References: Message-ID: On 14 Oct 2008, at 22:00, Igor Kozin wrote: > Did you really hold the Nehalem Xeon chips in your hands? They > probably require new motherboards with a new chipset. > absolutely > > > It would be nice to hear from you some numbers concerning Harpertown > vs Nehalem performance > > those who know will not be able tell you because of the NDA. > i think it is fair to say that the major difference is expected in > the memory bandwidth. They may not be able to, but there are presentations from IDF last August which include some information which may be useful, and which is clearly public. Start at https://intel.wingateweb.com/US08/scheduler/controller/catalog and search for presentations which mention Nehalem. (Disclaimer: I work for Intel). -- -- Jim -- James Cownie -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081014/dfc2fee3/attachment.html From lindahl at pbm.com Tue Oct 14 14:42:13 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: References: Message-ID: <20081014214212.GD28823@bx9.net> On Tue, Oct 14, 2008 at 10:00:16PM +0100, Igor Kozin wrote: > then make your choice between Barcelona and Harpertown. > if you are lucky you can get Shanghai which is expected by the end of the > year. Those of us who ordered Shanghai's when first announced should be getting them fairly soon. I'm not sure how bad the backlog became. -- greg From matt at technoronin.com Tue Oct 14 16:53:37 2008 From: matt at technoronin.com (Matt Lawrence) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <48F4E2BD.1030008@ias.edu> References: <48F4E2BD.1030008@ias.edu> Message-ID: On Tue, 14 Oct 2008, Prentice Bisbal wrote: > Dell supports CentOS. They better - that's one of the OSes they will > install if you buy a cluster from them. I suspect the technician you are > dealing with doesn't even know this. Not in my experience. Not only did they refuse to support it, the guy was downright snotty about it. Also, I recommend staying away from the MD3000 storage unit. It has issues such as not being able to present LUNs larger thn 2GB, not supporting RAID6 and requiring a GUI interface to be able to install the command line tools. -- Matt It's not what I know that counts. It's what I can remember in time to use. From kilian.cavalotti.work at gmail.com Tue Oct 14 08:33:51 2008 From: kilian.cavalotti.work at gmail.com (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: References: Message-ID: <48F4BBDF.2020300@gmail.com> Hi H?kon, H?kon Bugge wrote: > ... and, the statement below is based on comparing Harpertown to > Nehalem. I have not tested products from Intel's competitors and > compared them to Nehalem. Do you, by any chance, have any substantial performance figure to make us drool? :) Cheers, -- Kilian From landman at scalableinformatics.com Tue Oct 14 17:36:50 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: <48F4BBDF.2020300@gmail.com> References: <48F4BBDF.2020300@gmail.com> Message-ID: <48F53B22.3080108@scalableinformatics.com> Kilian CAVALOTTI wrote: > Hi H?kon, > > H?kon Bugge wrote: >> ... and, the statement below is based on comparing Harpertown to >> Nehalem. I have not tested products from Intel's competitors and >> compared them to Nehalem. > > Do you, by any chance, have any substantial performance figure to make > us drool? :) Intel has asked that no benchmarks be published by people with units. > > Cheers, -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From xclski at yahoo.com Tue Oct 14 18:34:32 2008 From: xclski at yahoo.com (Ellis Wilson) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons Message-ID: <663549.87679.qm@web37907.mail.mud.yahoo.com> Joe Landman wrote: > Kilian CAVALOTTI wrote: >> Do you, by any chance, have any substantial performance figure to make >> us drool? :) > > Intel has asked that no benchmarks be published by people with units. One wonders why they distributed them in the first place if they didn't intend to excite people about their performance prior to releasing them. With processors I don't think it's for "debugging" or stability checks since that should be well simulated (owing to the high cost of CPU molds costs millions itself). Ellis From landman at scalableinformatics.com Tue Oct 14 18:40:45 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: <663549.87679.qm@web37907.mail.mud.yahoo.com> References: <663549.87679.qm@web37907.mail.mud.yahoo.com> Message-ID: <48F54A1D.8030507@scalableinformatics.com> Ellis Wilson wrote: > Joe Landman wrote: >> Kilian CAVALOTTI wrote: >>> Do you, by any chance, have any substantial performance figure to make >>> us drool? :) >> Intel has asked that no benchmarks be published by people with units. > > One wonders why they distributed them in the first place if they didn't > intend to excite people about their performance prior to releasing them. They are exciting people. I can tell you its fast. I just can't tell you how fast. Its actually pretty good marketing. > With processors I don't think it's for "debugging" or stability checks > since that should be well simulated (owing to the high cost of CPU molds > costs millions itself). Its for (IMO) grass-roots word of mouth marketing. They want the buzz to be there. And it is. > > Ellis > > > > > > > -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From iioleynik at gmail.com Tue Oct 14 19:04:46 2008 From: iioleynik at gmail.com (Ivan Oleynik) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: <48F53B22.3080108@scalableinformatics.com> References: <48F4BBDF.2020300@gmail.com> <48F53B22.3080108@scalableinformatics.com> Message-ID: Joe, If integrators have them, this means that the Nehalem release is imminent? But when? Ivan On Tue, Oct 14, 2008 at 8:36 PM, Joe Landman < landman@scalableinformatics.com> wrote: > Kilian CAVALOTTI wrote: > >> Hi H?kon, >> >> H?kon Bugge wrote: >> >>> ... and, the statement below is based on comparing Harpertown to Nehalem. >>> I have not tested products from Intel's competitors and compared them to >>> Nehalem. >>> >> >> Do you, by any chance, have any substantial performance figure to make us >> drool? :) >> > > Intel has asked that no benchmarks be published by people with units. > > >> Cheers, >> > > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics LLC, > email: landman@scalableinformatics.com > web : http://www.scalableinformatics.com > http://jackrabbit.scalableinformatics.com > phone: +1 734 786 8423 x121 > fax : +1 866 888 3112 > cell : +1 734 612 4615 > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081014/08658a12/attachment.html From landman at scalableinformatics.com Tue Oct 14 19:08:32 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: References: <48F4BBDF.2020300@gmail.com> <48F53B22.3080108@scalableinformatics.com> Message-ID: <48F550A0.50405@scalableinformatics.com> Ivan Oleynik wrote: > Joe, > > If integrators have them, this means that the Nehalem release is imminent? You should ask Intel. We don't know. > > But when? > > Ivan > -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From bill at cse.ucdavis.edu Tue Oct 14 19:19:26 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: <663549.87679.qm@web37907.mail.mud.yahoo.com> References: <663549.87679.qm@web37907.mail.mud.yahoo.com> Message-ID: <48F5532E.7060106@cse.ucdavis.edu> Ellis Wilson wrote: > Joe Landman wrote: >> Kilian CAVALOTTI wrote: >>> Do you, by any chance, have any substantial performance figure to make >>> us drool? :) >> Intel has asked that no benchmarks be published by people with units. > > One wonders why they distributed them in the first place if they didn't > intend to excite people about their performance prior to releasing them. Heh, well they want to excited people... important people... who in exchange sign NDAs. Not to mention providing feedback on performance, stability, bios compatibility, operating system, drivers, compilers, applications etc that intel couldn't hope to replicate all the variations of in the lab. > With processors I don't think it's for "debugging" or stability checks > since that should be well simulated (owing to the high cost of CPU molds > costs millions itself). It's hard to predict when a show stopper will show. Nvidia, AMD, and Intel (and likely most everyone) has had learned hard lessons in this area. Indeed companies do spend big $$$ trying to make sure that each silicon revision is bug free... hardly a guarantee though. In any case if you google around there's a fair bit of performance information on nehalem chips. Stream performance has been mentioned, unlabeled charts with relative performance on Spec CPU among other benchmarks, and preproduction benchmarks on a variety of things. Public info and fuzzy IDF slides seem to conclude: * 2.6, 3.0, and 3.2 clock bins or so * slightly 5-25% higher IPC (per thread) on many workloads * Dramatically better memory system * 4 cores/8 threads first, more variations later. * 3 memory systems per socket. * On chip memory controller * lower memory latency than current intels or opterons. * slightly HIGHER power use per socket than current intel. So nothing really earth shattering for the single socket market, but very healthy competition (unlike the current CPUs) in the 2-4 socket market. Of course that leaves tons of interesting questions, pressure your favorite vendor if you can't wait. Although there is some info at: http://en.wikipedia.org/wiki/Intel_Nehalem http://www.intel.com/technology/architecture-silicon/next-gen/index.htm http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3326 Personally I'm most interested in when hyperthreading helps (hopefully it's a better implementation of SMT than the P4 had) and exactly how the memory system works. Things like how fast does 1,2,4,8,16 threads fetch a random cache line? Sequential? From L1, L2, L3, and main memory. Things like: http://cse.ucdavis.edu/~bill/numa3-smooth.png Current rumors claim the desktop chip (core i7) is due in week 46, but recent news claims week 47 around Nov 17th. No idea when the workstation/server version will be out, at least is should give a good idea where the server version should be performance wise. From himikehawaii1 at yahoo.com Tue Oct 14 19:00:16 2008 From: himikehawaii1 at yahoo.com (MDG) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons lest think about 6 core Dunningtons if just drool factor In-Reply-To: <48F53B22.3080108@scalableinformatics.com> Message-ID: <984250.49699.qm@web54110.mail.re2.yahoo.com> Sun has "leaked" some preformance figorese see http://www.dvhardware.net/article25470.html ? If you really want to drool then Intel's new 6 CORE ? "Dunnington will be a new 45nm processor in the Penryn family, it will feature three dual-core banks and have 16MB of shared L3 cache memory. Each pair of cores can also access 3MB of local L2 cache. The six-core Dunnington processors will be available in the second half of this year and will be pin-compatible with Intel Tigerton processors, this means they'll work in existing Clarksboro chipset based motherboards.: ? well I wonder what rwin Dunnington Xeons would be like, og never enough money or spee and as usual before the latest greateswt product the Behalem Xeons are even available, or? has it obsolete regualarly available, Intel already has the nest product being prepped for released.? Such is the life in trying to buy a top preformance computer, as multu-cores now adding it seems faster then moore's Law of 18 months. --- On Tue, 10/14/08, Joe Landman wrote: From: Joe Landman Subject: Re: [Beowulf] Nehalem Xeons To: "Kilian CAVALOTTI" Cc: "H?kon Bugge" , beowulf@beowulf.org Date: Tuesday, October 14, 2008, 2:36 PM Kilian CAVALOTTI wrote: > Hi H?kon, > > H?kon Bugge wrote: >> ... and, the statement below is based on comparing Harpertown to >> Nehalem. I have not tested products from Intel's competitors and >> compared them to Nehalem. > > Do you, by any chance, have any substantial performance figure to make > us drool? :) Intel has asked that no benchmarks be published by people with units. > > Cheers, -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081014/141da362/attachment.html From jan.heichler at gmx.net Wed Oct 15 00:18:45 2008 From: jan.heichler at gmx.net (Jan Heichler) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: <48F4E2BD.1030008@ias.edu> Message-ID: <1712521297.20081015091845@gmx.net> Hallo Matt, Mittwoch, 15. Oktober 2008, meintest Du: ML> Also, I recommend staying away from the MD3000 storage unit. It has ML> issues such as not being able to present LUNs larger thn 2GB, not ML> supporting RAID6 and The next firmware release will fix those issues. Don't know when it will be out but it should be this year. Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081015/0f733c35/attachment.html From eagles051387 at gmail.com Wed Oct 15 01:06:30 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: <48F5532E.7060106@cse.ucdavis.edu> References: <663549.87679.qm@web37907.mail.mud.yahoo.com> <48F5532E.7060106@cse.ucdavis.edu> Message-ID: hows does one get on the band wagon to test out these newer processors from intel? On Wed, Oct 15, 2008 at 4:19 AM, Bill Broadley wrote: > Ellis Wilson wrote: > >> Joe Landman wrote: >> >>> Kilian CAVALOTTI wrote: >>> >>>> Do you, by any chance, have any substantial performance figure to make >>>> us drool? :) >>>> >>> Intel has asked that no benchmarks be published by people with units. >>> >> >> One wonders why they distributed them in the first place if they didn't >> intend to excite people about their performance prior to releasing them. >> > > Heh, well they want to excited people... important people... who in > exchange sign NDAs. Not to mention providing feedback on performance, > stability, bios compatibility, operating system, drivers, compilers, > applications etc that intel couldn't hope to replicate all the variations of > in the lab. > > With processors I don't think it's for "debugging" or stability checks >> since that should be well simulated (owing to the high cost of CPU molds >> costs millions itself). >> > > It's hard to predict when a show stopper will show. Nvidia, AMD, and Intel > (and likely most everyone) has had learned hard lessons in this area. > Indeed companies do spend big $$$ trying to make sure that each silicon > revision is bug free... hardly a guarantee though. > > In any case if you google around there's a fair bit of performance > information on nehalem chips. Stream performance has been mentioned, > unlabeled charts with relative performance on Spec CPU among other > benchmarks, and preproduction benchmarks on a variety of things. Public > info and fuzzy IDF slides seem to conclude: > * 2.6, 3.0, and 3.2 clock bins or so > * slightly 5-25% higher IPC (per thread) on many workloads > * Dramatically better memory system > * 4 cores/8 threads first, more variations later. > * 3 memory systems per socket. > * On chip memory controller > * lower memory latency than current intels or opterons. > * slightly HIGHER power use per socket than current intel. > > So nothing really earth shattering for the single socket market, but very > healthy competition (unlike the current CPUs) in the 2-4 socket market. > > Of course that leaves tons of interesting questions, pressure your favorite > vendor if you can't wait. Although there is some info at: > http://en.wikipedia.org/wiki/Intel_Nehalem > http://www.intel.com/technology/architecture-silicon/next-gen/index.htm > http://www.anandtech.com/cpuchipsets/intel/showdoc.aspx?i=3326 > > Personally I'm most interested in when hyperthreading helps (hopefully it's > a better implementation of SMT than the P4 had) and exactly how the memory > system works. Things like how fast does 1,2,4,8,16 threads fetch a random > cache line? Sequential? From L1, L2, L3, and main memory. Things like: > http://cse.ucdavis.edu/~bill/numa3-smooth.png > > Current rumors claim the desktop chip (core i7) is due in week 46, but > recent news claims week 47 around Nov 17th. No idea when the > workstation/server version will be out, at least is should give a good idea > where the server version should be performance wise. > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081015/ea636d32/attachment.html From hahn at mcmaster.ca Wed Oct 15 06:01:43 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Nehalem Xeons lest think about 6 core Dunningtons if just drool factor In-Reply-To: <984250.49699.qm@web54110.mail.re2.yahoo.com> References: <984250.49699.qm@web54110.mail.re2.yahoo.com> Message-ID: > If you really want to drool then Intel's new 6 CORE nah. to me, it's just a density play - nothing wrong with that, but not really interesting. if you have small-memory, cache-friendly code, you'll be happy, but putting 6 cores on the same old tired memory is not what I'd call an HPC solution. iirc, the 6-core chips are also quite big/expensive. From prentice at ias.edu Wed Oct 15 06:03:37 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:53 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <20081014185438.GA12924@bx9.net> References: <48F4E2BD.1030008@ias.edu> <20081014185438.GA12924@bx9.net> Message-ID: <48F5EA29.8070400@ias.edu> Greg Lindahl wrote: > On Tue, Oct 14, 2008 at 02:19:41PM -0400, Prentice Bisbal wrote: > >> And you can always lie to the support - CentOS, for all practical and >> technical purposes, *IS* RHEL. > > But this requires cleverness -- often install scripts and the like > look at /etc/redhat-release or the redhat-release rpm. The important thing is that they are *scripts*, which means you can read them, figure out what they are doing, and either modify them to suit your RHEL rebuild, or modify /etc/redhat-release to contain what they are looking for. I never saw one that looked at the redhat-release rpm. The only thing I couldn't get to install on Fedora or CentOS was the EMC PowerPath software. There were unique kernel modules for every RHEL kernel version, and for some reason they refused to install. Since they were closed-source, I couldn't figure out why. But now we've veered off-topic, which I really hate... -- Prentice From prentice at ias.edu Wed Oct 15 06:06:25 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: <48F53B22.3080108@scalableinformatics.com> References: <48F4BBDF.2020300@gmail.com> <48F53B22.3080108@scalableinformatics.com> Message-ID: <48F5EAD1.6000703@ias.edu> Joe Landman wrote: >> Do you, by any chance, have any substantial performance figure to make >> us drool? :) > > Intel has asked that no benchmarks be published by people with units. > Sounds like Intel has something to hide. -- Prentice From laytonjb at att.net Wed Oct 15 06:18:54 2008 From: laytonjb at att.net (Jeff Layton) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: <48F5EAD1.6000703@ias.edu> References: <48F4BBDF.2020300@gmail.com> <48F53B22.3080108@scalableinformatics.com> <48F5EAD1.6000703@ias.edu> Message-ID: <48F5EDBE.5050906@att.net> Prentice Bisbal wrote: > > Sounds like Intel has something to hide. > Nope. They just track people who talk badly about Intel and then they send around the black helicopters and the little blue men and the silver suit dudes will come talk to you :) Actually this makes perfect sense for all chip manufacturers. The reason is that early numbers may or may not be indicative of final performance. If an early number sneaks out and the final performance is less. then people will howl, "Why did you kill performance? You are so evil? I hate you!" If performance is better than perhaps you've turned people off of your new fancy product, "Well blank-blank wasn't smart enough to give out early numbers, especially to me, so I don't ever want to talk to them again. They're evil!" It's indeed a difficult game to play. You want to develop lust for your product (Apple is the master of this) but you don't want to kill popularity for it in either direction. So it makes perfect sense for companies not to release anything early. It doesn't mean they have anything to hide. It means they are trying to build good momentum for their products. Just relax and hang in there. When Intel announces Nehalem I'm sure they will release benchmarks. The same is true for AMD when they announce Shanghai. It's all part of the fun :) Jeff From eagles051387 at gmail.com Wed Oct 15 06:58:51 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Nehalem Xeons lest think about 6 core Dunningtons if just drool factor In-Reply-To: References: <984250.49699.qm@web54110.mail.re2.yahoo.com> Message-ID: i remember reading some where that they are going to be over lapping the cores some how On Wed, Oct 15, 2008 at 3:01 PM, Mark Hahn wrote: > If you really want to drool then Intel's new 6 CORE >> > > nah. to me, it's just a density play - nothing wrong with that, > but not really interesting. if you have small-memory, cache-friendly code, > you'll be happy, but putting 6 cores on the same old tired memory > is not what I'd call an HPC solution. iirc, the 6-core chips are also > quite big/expensive. > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081015/d447c89b/attachment.html From diep at xs4all.nl Wed Oct 15 07:46:41 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: <48F5EAD1.6000703@ias.edu> References: <48F4BBDF.2020300@gmail.com> <48F53B22.3080108@scalableinformatics.com> <48F5EAD1.6000703@ias.edu> Message-ID: <6702FA83-2790-4F7C-A839-5E5D69031BF8@xs4all.nl> They definitely must fear AMD to be faster, with good reasons. Let's check things. Past 12 years that i been watching reports on new cpu's intels performance of new great cpu's always got leaked out long before. All we have now is some posting of a guy from which i cannot confirm he been honest, in face he could've made it all up, as i do not know the guy: by: Vael Jean-Paul Joined: 24 Apr 2008 Posts: 28 subject: Nehalem 8threads running Toga 1.4.2JD-8cpu Posted: Fri Jun 27, 2008 6:04 pm Reply to topic Reply with quote Here you can see TogaII 1.4.2JD-8cpu running on the new Nehalem at 3Ghz! stock speed. I let him just calculate from the begin position...and compare it with your computer how many nodes you get! With my computer a Q6600 at 3.2Ghz i have 4.4Million/nodes and the Nehalem gives 5.8Million/nodes!! So it will be a nice speed gain again! http://img413.imageshack.us/img413/8333/nehalembi4.jpg JP. Back to top View user's profile --- Now i downloaded that executable. Then i ran it on my old dual opteron dual core 2.4Ghz opteron machine with PC2100 RAM. Please note that this old RAM is a big bottleneck for a fast program like Toga, Toga initially was Fruit source code with 50 lines modified. They parallellized it a tad for shared memory machines. Fruit,Toga author = Fabien Letouzey (France) 1/01 0:00 +0.06 1.Na3 (2) 1/01 0:00 +0.34 1.Nc3 (3) 1/01 0:00 +0.40 1.d4 (13) 2/02 0:00 +0.20 1.d4 d5 (45) 3/07 0:00 +0.34 1.d4 d5 2.Nf3 (182) 4/10 0:00 +0.20 1.d4 d5 2.Nf3 Nf6 (537) 5/12 0:00 +0.30 1.d4 d5 2.Nf3 Nf6 3.Nc3 (2.370) 6/12 0:00 +0.20 1.d4 d5 2.Nf3 Nf6 3.Nc3 Nc6 (4.646) 7/14 0:00 +0.20 1.d4 d5 2.Nf3 Nf6 3.Nc3 Nc6 4.Bf4 (11.682) 8/14 0:00 +0.20 1.d4 d5 2.Nf3 Nf6 3.Nc3 Nc6 4.Bf4 Bf5 (23.561) 9/19 0:00 +0.16 1.d4 Nf6 2.Nc3 d5 3.Bf4 Nh5 4.Be5 Nc6 5.Nf3 Nxe5 6.Nxe5 (54.951) 10/20 0:00 +0.23 1.d4 Nf6 2.Nc3 d5 3.Nf3 Qd6 4.Bd2 Nc6 (38.442) 10/21 0:00 +0.25 1.d4 Nf6 2.Nc3 d5 3.Nf3 Nc6 4.Bf4 e6 5.e3 (46.374) 10/20 0:00 +0.26 1.Nc3 d5 2.d4 Nf6 3.Nf3 Nc6 4.Bf4 Bf5 5.Nb5 Rc8 (99.448) 11/21 0:00 +0.13 1.d4 Nf6 2.Nc3 d5 3.Nf3 Nc6 4.Qd3 Bd7 5.a3 Bg4 6.Bf4 (105.957) 11/24 0:00 +0.16 1.Nc3 Nf6 2.Nf3 Nc6 3.e4 d5 4.exd5 Nxd5 5.Bb5 Nxc3 6.bxc3 Qd5 7.Qe2 (104.602) 12/23 0:00 +0.13 1.Nc3 Nf6 2.Nf3 Nc6 3.e4 d5 4.exd5 Nxd5 5.Nxd5 Qxd5 6.Be2 Bf5 7.O-O e5 (192.446) 12/24 0:00 +0.18 1.d4 Nf6 2.Nf3 d5 3.c4 e6 4.cxd5 exd5 5.Nc3 Bb4 6.Qa4+ Nc6 7.Ne5 Bxc3+ 8.bxc3 O-O 9.Nxc6 bxc6 10.Qxc6 (403.232) 12/25 0:00 +0.20 1.Nf3 Nf6 2.e3 Nc6 3.d4 e6 4.Nc3 d5 5.Bb5 Bb4 6.O-O O-O (442.776) 13/30 0:01 +0.15 1.Nf3 Nf6 2.d4 e6 3.e3 d5 4.Bb5+ Bd7 5.Nc3 Bb4 6.Bd3 Nc6 7.O-O O-O (1.088.947) 2958 13/30 0:01 +0.26 1.e4 e5 2.Nf3 Nf6 3.Nxe5 Qe7 4.d4 d6 5.Nf3 Qxe4 + 6.Be2 d5 7.O-O Bb4 (1.203.253) 2958 13/30 0:01 +0.28 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nc4 Nxe4 5.Qe2 d5 6.Nc3 b5 7.Nxe4 bxc4 (1.227.804) 2958 14/34 0:03 +0.26 1.e4 Nf6 2.e5 Nd5 3.Nf3 Nc6 4.Bb5 e6 5.O-O Be7 6.Nc3 Nxc3 7.dxc3 O-O 8.Be3 (2.581.722) 3129 14/33 0:03 +0.34 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Bb5+ c6 6.Be2 Be7 7.d3 Nf6 8.O-O O-O 9.Nc3 (2.553.907) 3129 15/33 0:05 +0.35 1.e4 d5 2.exd5 Nf6 3.Nf3 Nxd5 4.d4 e6 5.Bd3 Nc6 6.O-O Be7 7.Nc3 O-O 8.Re1 Re8 9.Nxd5 exd5 (4.762.459) 3195 15/34 0:05 +0.36 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Bb5+ c6 6.Be2 Be7 7.d3 Nf6 8.O-O O-O 9.Nc3 Be6 (4.809.423) 3195 16/35 0:09 +0.32 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.d3 Nf6 6.d4 Be7 7.Nc3 O-O 8.Bd3 Nc6 9.O-O Be6 (7.746.606) 3317 17/37 0:19 +0.36 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Bd3 Nf6 6.b3 Qe7+ 7.Be2 Nc6 8.O-O Bf5 9.d4 O-O-O 10.Nc3 (17.138.797) 3481 18/45 0:40 +0.27 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Bd3 Nf6 6.O-O Nc6 7.b3 Be7 8.Ba3 O-O 9.Re1 Nd5 10.Nc3 Nxc3 11.dxc3 (37.171.149) 3660 19/45 1:12 +0.26 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Nc6 8.Be3 Be6 9.O-O-O O-O-O 10.Qb5 a6 11.Qa4 d5 (66.063.998) 3654 20/45 2:58 +0.20 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nd3 Nxe4 5.Qe2 Qe7 6.Nf4 Nc6 7.Nd5 Nd4 8.Nxe7 Nxe2 9.Nd5 c6 10.Bxe2 cxd5 11.Nc3 Be6 12.Bb5+ Ke7 (153.338.164) 3596 20/43 2:59 +0.33 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nd3 Nxe4 5.Qe2 Qe7 6.Nf4 Nc6 7.Nd5 Nd4 8.Nxe7 Nxe2 9.Nd5 c6 10.Bxe2 cxd5 11.d3 Nc5 12.Nc3 (144.181.758) 3596 21/48 5:48 +0.21 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Nc6 8.Bf4 a5 9.Nd4 Nxd4 10.cxd4 Bg4 11.f3 Be6 (315.235.900) 3617 21/48 5:48 +0.22 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.d3 Nf6 6.d4 Nc6 7.Nc3 d5 8.Bb5 Bb4 9.Qe2+ Qe7 10.Bg5 Bxc3+ 11.bxc3 Qxe2+ 12.Kxe2 Ne4 13.c4 Nc3+ 14.Kd3 (313.098.050) 3617 22/48 12:51 +0.18 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Nc6 8.Bf4 Qxe2+ 9.Bxe2 Be7 10.O-O-O O-O 11.Rhe1 Be6 12.Kb1 Rfe8 13.Ng5 Bf5 14.Rd5 (715.698.771) 3708 22/48 12:53 +0.20 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Nc6 8.Bf4 Qxe2+ 9.Bxe2 Be7 10.O-O-O O-O 11.Rhe1 Be6 12.Kb1 Rfe8 13.Ng5 Bf5 (642.733.687) 3708 22/50 14:30 +0.22 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Nc6 8.Bf4 Qxe2+ 9.Bxe2 Be7 10.O-O Bd7 11.Rfe1 O- O 12.Bb5 Rfe8 13.Rad1 Bf6 (695.660.943) 3697 23/55 29:51 +0.22 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.Nc3 Nxc3 7.dxc3 Qxe2+ 8.Bxe2 Nc6 9.Bf4 Be7 10.O-O-O O-O 11.Rhe1 Re8 12.Rd5 Bf6 13.h3 Bd7 (1.667.029.770) 3722 23/52 40:30 +0.23 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.d3 Nf6 7.Nc3 Nc6 8.Bg5 Qxe2+ 9.Bxe2 Be7 10.Nb5 Kd8 11.O-O h6 12.Be3 Nd5 13.Nfd4 Nxe3 14.fxe3 (2.039.956.804) 3699 23/58 45:41 +0.28 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.d3 Nf6 7.Nc3 Nc6 8.Bg5 Qxe2+ 9.Bxe2 Be7 10.Nb5 Kd8 11.O-O Nb4 12.Nbd4 Bg4 13.Rfe1 (2.535.957.352) 3697 24/59 72:43 +0.20 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.d3 Nf6 7.Nc3 Nc6 8.Bg5 Qxe2+ 9.Bxe2 Be7 10.Nb5 Kd8 11.Bd2 Bg4 12.Ng5 Ne5 13.d4 Bxe2 14.Kxe2 (4.123.340.979) 3780 24/56 73:16 +0.27 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.d3 Nf6 7.Nc3 Nc6 8.Bg5 Qxe2+ 9.Bxe2 Bd7 10.d4 Be7 11.O-O-O O-O 12.d5 Ne5 13.Nxe5 dxe5 14.Nb5 (3.764.072.477) 3777 24/58 73:58 +0.31 1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nf3 Nxe4 5.Qe2 Qe7 6.d3 Nf6 7.Nc3 Nc6 8.Bg5 Qxe2+ 9.Bxe2 Bd7 10.d4 Be7 11.O-O-O Ng4 12.Rde1 Nxf2 13.Rhf1 Ng4 14.Nd5 (4.123.413.325) 3775 I assumed 128MB hashtable, which is the default. I do not know other settings used on Nehalem so i can only gamble them. Weird is that the nps on nehalem peaks after 16 ply already, so within seconds. That is very weird. Means probably tiny hashtable been used. A program like Toga is not a good program to test on Nehalem with 8 logical cores. But here is extrapolation to 3Ghz of my machine. Again i have PC2100 registered RAM inside this S2881 mainboard. That really sucks. if i extrapolate opteron from 2.4Ghz to 3Ghz that would mean at 22 ply: Nehalem 8 cores : 5.8 mln opteron 3Ghz PC2100 ==> 1.25 * 3.708 = 4.635 5.8 / 4.635 = 1.25 In short the move from PC2100 to on die memory controller DDR3 that should give a memory latency hungry program like Toga already 25%. Then we've got hyperthreading that will bring also a limited speed increase. All this together gives just 25%, very little IMHO. From Phenom i know integers do not run faster on it, but with DDR2 it is a lot faster than this PC2100 i've got. I conclude for my chessprogram Diep, a 4 core AMD might be faster than 8 cores Nehalem. that said, it is possible that the program Toga is bad for 8 cores and that my Diep is doing much better at it. Those guys are not so good in parallellizing software. Alfabeta is hard to parallellize very efficient. The above calculation could be wrong therefore when benchmarking Diep at it. Vincent On Oct 15, 2008, at 3:06 PM, Prentice Bisbal wrote: > Joe Landman wrote: >>> Do you, by any chance, have any substantial performance figure to >>> make >>> us drool? :) >> >> Intel has asked that no benchmarks be published by people with units. >> > > Sounds like Intel has something to hide. > > -- > Prentice > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From maurice at harddata.com Tue Oct 14 21:02:02 2008 From: maurice at harddata.com (Maurice Hilarius) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Re: Nehalem Xeons In-Reply-To: <200810150038.m9F0bqmb002720@bluewest.scyld.com> References: <200810150038.m9F0bqmb002720@bluewest.scyld.com> Message-ID: <48F56B3A.4010006@harddata.com> > then make your choice between Barcelona and Harpertown. > if you are lucky you can get Shanghai which is expected by the end of the > year. > The Shanghai parts are available *now*. Not just "vaporware" They will run in existing chip sets and boards, not some new thing to debug. BIOS for motherboards are not there in a lot of cases, although that seems to be a matter of days away for most. -- With our best regards, //Maurice W. Hilarius Telephone: 01-780-456-9771/ /Hard Data Ltd. FAX: 01-780-456-9772/ /11060 - 166 Avenue email:maurice@harddata.com/ /Edmonton, AB, Canada http://www.harddata.com// / T5X 1Y3/ / -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081014/463bb5e1/attachment.html From tom.elken at qlogic.com Wed Oct 15 09:07:59 2008 From: tom.elken at qlogic.com (Tom Elken) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Nehalem Xeons lest think about 6 core Dunningtons ifjust drool factor In-Reply-To: References: <984250.49699.qm@web54110.mail.re2.yahoo.com> Message-ID: <6DB5B58A8E5AB846A7B3B3BFF1B4315A0269A440@AVEXCH1.qlogic.org> > [mailto:beowulf-bounces@beowulf.org] On Behalf Of Mark Hahn > > > If you really want to drool then Intel's new 6 CORE > > nah. to me, it's just a density play - nothing wrong with that, > but not really interesting. if you have small-memory, > cache-friendly code, > you'll be happy, but putting 6 cores on the same old tired memory > is not what I'd call an HPC solution. To support Mark's statement, at this page, http://www.spec.org/cgi-bin/osgresults?conf=cpu2006&op=form you can enter '6' in the "# Cores Per Chip " field and hit the "Fetch Results" button at the bottom. You will see 36 results as hits -- I think they are all Dunnington. The more interesting are the "CINT2006 Rates" and "CFP2006 Rates" as you scroll down. I think that some of the Integer Rate results are records or near records for x86_64 compatible machines, at least for 4-CPU (4 socket) machines. I think the FP Rate results are very good, but not quite at the level of some 4-socket, 16-core AMD Barcelona results. Since FP is more relevant for HPC (not that INT is totally irrelevant), this supports Mark's point. -Tom > iirc, the 6-core chips > are also > quite big/expensive. From lindahl at pbm.com Wed Oct 15 14:02:50 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <48F5EA29.8070400@ias.edu> References: <48F4E2BD.1030008@ias.edu> <20081014185438.GA12924@bx9.net> <48F5EA29.8070400@ias.edu> Message-ID: <20081015210250.GA19246@bx9.net> > > But this requires cleverness -- often install scripts and the like > > look at /etc/redhat-release or the redhat-release rpm. > > The important thing is that they are *scripts*, which means you can read > them, figure out what they are doing, and either modify them to suit > your RHEL rebuild, Sometimes. Sometimes they're binaries. BTW with binaries you can strace them and generally figure out how to fool them into installing. -- greg From lindahl at pbm.com Wed Oct 15 14:19:14 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: <48F5EAD1.6000703@ias.edu> References: <48F4BBDF.2020300@gmail.com> <48F53B22.3080108@scalableinformatics.com> <48F5EAD1.6000703@ias.edu> Message-ID: <20081015211914.GB19246@bx9.net> > Sounds like Intel has something to hide. Often companies like Intel have an embargo date solely to make sure that all the great new performance news arrives at once, generating a lot of press alongside the formal announcement of availability. I also had one case years ago where I was doing a big benchmark for a bid on a new processor. We had pre-release parts. The release parts were 10% slower, across the board on all of our benchmarks. Memory intensive, cache intensive, fp intensive, int intensive, it was all slower by about the same amount. That was a real head-scratcher, since the cpu clock was the same. AMD sends out pre-release Opterons to partners, often times running at pretty low clock rates. It's very useful for compiler testing, and can provide more opportunity to find chip bugs. Needless to say, no benchmarks from these really slow parts can be published. -- greg From diep at xs4all.nl Wed Oct 15 15:57:10 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: <20081015211914.GB19246@bx9.net> References: <48F4BBDF.2020300@gmail.com> <48F53B22.3080108@scalableinformatics.com> <48F5EAD1.6000703@ias.edu> <20081015211914.GB19246@bx9.net> Message-ID: <6D372C88-8092-45A2-BCA9-618C1C8935D4@xs4all.nl> I feel it's a double edged sword. Usually when intel ships a great new part, all the sites are discussing it and 1 or 2 vague tests show up. AMD seems to use the path you describe below. Suddenly K7-MP 1.2ghz was there crushing P4 Xeon 1.7Ghz at the time. Suddenly opteron was there at 2.2Ghz crushing anything intel had bigtime, including itanium (integer wise). What AMD ships usually gets benchmarked. Intel nowadays seems to release special benchmark chips, eating a 150+ watt and having some unrealistic high price. Look who sells better though. It's obvious that making noise with all drums you have in a nonstop manner, about upcoming great release, is a lot more effective, than AMD's non- existing marketing. For a company worth $10-$20 billion (or what is it?) that's nonstop nearly bankrupt nearly releasing some great new product, it should be possible to pay a good marketing manager from the interest at the bankloan. Marketing is everything. Vincent On Oct 15, 2008, at 11:19 PM, Greg Lindahl wrote: >> Sounds like Intel has something to hide. > > Often companies like Intel have an embargo date solely to make sure > that all the great new performance news arrives at once, generating a > lot of press alongside the formal announcement of availability. > > I also had one case years ago where I was doing a big benchmark for a > bid on a new processor. We had pre-release parts. The release parts > were 10% slower, across the board on all of our benchmarks. Memory > intensive, cache intensive, fp intensive, int intensive, it was all > slower by about the same amount. That was a real head-scratcher, since > the cpu clock was the same. > > AMD sends out pre-release Opterons to partners, often times running at > pretty low clock rates. It's very useful for compiler testing, and can > provide more opportunity to find chip bugs. Needless to say, no > benchmarks from these really slow parts can be published. > > -- greg > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From csamuel at vpac.org Wed Oct 15 17:41:59 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Nehalem Xeons In-Reply-To: Message-ID: <1562607063.1527711224117719542.JavaMail.root@mail.vpac.org> ----- "Jon Aquilina" wrote: > hows does one get on the band wagon to test out these newer processors > from intel? Talk to your local friendly Intel rep I suspect. :-) -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From john.leidel at gmail.com Wed Oct 15 19:38:07 2008 From: john.leidel at gmail.com (John Leidel) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] SC08 LECCIBG Message-ID: <1224124687.4430.308.camel@e521.site> So.... shall I pose the question early this year? Anyone planning the annual LECCIBG? Dr. Eadline? From prentice at ias.edu Thu Oct 16 05:31:21 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <20081015210250.GA19246@bx9.net> References: <48F4E2BD.1030008@ias.edu> <20081014185438.GA12924@bx9.net> <48F5EA29.8070400@ias.edu> <20081015210250.GA19246@bx9.net> Message-ID: <48F73419.7000803@ias.edu> Greg Lindahl wrote: >>> But this requires cleverness -- often install scripts and the like >>> look at /etc/redhat-release or the redhat-release rpm. >> The important thing is that they are *scripts*, which means you can read >> them, figure out what they are doing, and either modify them to suit >> your RHEL rebuild, > > Sometimes. Sometimes they're binaries. BTW with binaries you can > strace them and generally figure out how to fool them into installing. You called them scripts (see above). I was just working off of what you said. From my experience, binary installers in the Unix world are *relatively* rare. Most are simple shell scripts. But now we're off topic again... -- Prentice From niftyompi at niftyegg.com Thu Oct 16 16:11:46 2008 From: niftyompi at niftyegg.com (Nifty niftyompi Mitch) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] large MPI adopters In-Reply-To: <48ECA0E4.4060000@tamu.edu> References: <18076.148.87.1.172.1218584195.squirrel@squirrelmail.soe.ucsc.edu> <48EB69B6.5080602@tamu.edu> <6DB5B58A8E5AB846A7B3B3BFF1B4315A0269989D@AVEXCH1.qlogic.org> <48ECA0E4.4060000@tamu.edu> Message-ID: <20081016231146.GA9957@compegg.wr.niftyegg.com> On Wed, Oct 08, 2008 at 07:00:36AM -0500, Gerry Creager wrote: > > Tom, > > I looked at that and *THOUGHT* it looked funny, but I was unable to see > the typo. Yeah, OpenMP. Both have their place, which is sometimes > integrated into the same application! Also there is a class of applications that are hybrid. i.e. With coarse grain parallel code in MPI and compiler detected fine grain parallel code. In some ways this mix of styles has value with modern multi core CPUs in smallish systems. The application footprint for code and data on memory need not be duplicated and the programmer can focus on the obvious big parallel chunks and let the compiler folk attend to the detailed stuff. I suspect that 90% of MPI clustering is associated with less than fifteen common packages. A bit of research into who purchases or down loads these packages would cover most of the MPI cluster computation sites. In all I suspect two or three weather codes, four or five fluid dynamic packages, a couple thermal modeling and a gaggle of chemistry codes will add up to 90%. More guessing, the last 10% contain the next generation work in progress codes and based on the design choices made will shape the clustering needs in the future. For some of these the science is the hard part and what ever abstraction model the author hooks up to will shape purchase requirements... At the recent Stanford HPC conference there were some very interesting talks on their bio chemistry work (Folding at Home) and how they rethought some of their code designs for orders of magnitude speedups. Some of their Alzheimer related research is astounding. It might be that the next bit for FaH might be multi core aware trickery. Later, mitch -- T o m M i t c h e l l Found me a new hat, now what? From deadline at eadline.org Thu Oct 16 19:41:07 2008 From: deadline at eadline.org (Douglas Eadline) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] SC08 LECCIBG In-Reply-To: <1224124687.4430.308.camel@e521.site> References: <1224124687.4430.308.camel@e521.site> Message-ID: <38524.192.168.1.213.1224211267.squirrel@mail.eadline.org> Yes and it is not me, I have been banded from planning events (a long series of sad stories) Monday night after the Gayla more details to follow. -- Doug > So.... shall I pose the question early this year? Anyone planning the > annual LECCIBG? > > Dr. Eadline? > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Doug From niftyompi at niftyegg.com Fri Oct 17 08:22:30 2008 From: niftyompi at niftyegg.com (Nifty niftyompi Mitch) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <20081017152230.GA14072@compegg.wr.niftyegg.com> On Thu, Oct 09, 2008 at 02:20:52PM -0500, Rahul Nabar wrote: > > I have a PowerEdge SC 1435 that has a strange problem. We bought about 23 of > these for a cluster and machines have been failing in a somewhat random manner > in a peculiar way: > > (1) Screen is blank. Front blue indicator turns steady orange. > > (2) Cannot get it to reboot by pressing (or keeping depressed) the power button > > (3) only way to reboot is to cycle the power. > > (4) After reboot machine works fine again , till after a few days same failure. > > Ran the dset and diagnostic CD but nothing relevant. > > Any tips what could be the faulty component? Or debug ideas? Right now I'm > totally lost! Hardware / software? CPU / Motherboard / Power supply? > > Anoybody knows what exactly makes the indicator LED turn steady orange from its > normal blue state? This is not one of the 4 numbered LEDs but the one to their > right. > > I posted this problem on a PowerEdge mailing list but haven't gotten > very far yet. Any suggestions are appreciated! > Check the baseboard management controller log (Ctrl+E). Tell us what software distribution you are running and any changes that might have been made (no matter how small). What is the default run level (is X11 active/ not active). Are power saving options enabled in the BIOS? Also what hardware monitor software are you running. I have seen system admins add their own package to systems only to find that RHEL has an equivalent package that uses different device drivers for the same hardware with impossible to diagnose results. Custom kernel? Disable cpuspeed, hardware monitor and hardware control software to see if stability changes. What additional hardware is in the chassis? The "poweredge indicator turning orange" tells me that the problem was detected by the system and there should be a hint in the log. The orange state is sticky and needs to be cleared.... -- T o m M i t c h e l l Found me a new hat, now what? From rpnabar at gmail.com Fri Oct 17 08:37:17 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: <20081017152230.GA14072@compegg.wr.niftyegg.com> References: <20081017152230.GA14072@compegg.wr.niftyegg.com> Message-ID: On Fri, Oct 17, 2008 at 10:22 AM, Nifty niftyompi Mitch wrote: > Check the baseboard management controller log (Ctrl+E). > > Tell us what software distribution you are running and any changes that might have > been made (no matter how small). What is the default run level (is X11 active/ not active). > Are power saving options enabled in the BIOS? Distro: Centos 5.2. Linux node03 2.6.18-92.el5 #1 SMP Tue Jun 10 18:51:06 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux No changes made to standard kernel. X11 not active. Power saving not enabled. > Also what hardware monitor software are you running. I have seen system admins add > their own package to systems only to find that RHEL has an equivalent package > that uses different device drivers for the same hardware with impossible to diagnose > results. Custom kernel? I am not sure what you mean by "hardware monitor software". I do not recall installing anything special. > Disable cpuspeed, hardware monitor and hardware control software to see if stability changes. There are a bunch of Dell utilities that come up at boot-time. BMC, RAID, Bradcom-PXE, Remote manage controllers. You want me to disable those? Stability has already changed. After I swapped motherboard+cpu. No more dead nodes in over 2 weeks now (yay!) But I just want to make sure this won't be a recurring problem with these SC1435's before we go in for our next expansion. > What additional hardware is in the chassis? None other than what came with the original Dell units. These are only 2 months old now. They do have dual NICs and no CDROMs. Have disks. Linked to a Dell KVM via a SIP module. No min-n-matching of Hardware. Was a monolithic Dell order. > The "poweredge indicator turning orange" tells me that the problem was detected by the > system and there should be a hint in the log. The orange state is sticky and > needs to be cleared.... Funny. It wasn't sticky for me. When I rebooted the orange light cleared. I did not need to reset it via the BIOS. Unfortunately the SC series does not have the tiny LCD for an error display. -- Rahul From derrick.kondo at inria.fr Thu Oct 16 05:37:55 2008 From: derrick.kondo at inria.fr (Derrick Kondo) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] CFP on Desktop Grids and Volunteer Computing Systems In-Reply-To: <60ec14620810160511k6bbf2026oa3ca2063ed19d404@mail.gmail.com> References: <60ec14620810160508r5195dfc9o3d8089d2c1548eee@mail.gmail.com> <60ec14620810160509k7fb239f8kf718c775f38cfe30@mail.gmail.com> <60ec14620810160511k6bbf2026oa3ca2063ed19d404@mail.gmail.com> Message-ID: <60ec14620810160537v6fd442e2v6a4866f8657b091c@mail.gmail.com> CALL FOR PAPERS Third Workshop on Desktop Grids and Volunteer Computing Systems (PCGrid 2009) held in conjunction with the IEEE International Parallel & Distributed Processing Symposium (IPDPS) May 29, 2009 Submission deadline: November 14, 2008 Rome, Italy web site: http://pcgrid.lri.fr See below for details --------------------------------------------------------------------------------------------------------------------- Journal of Grid Computing Special issue on desktop grids and volunteer computing (Independent of PCGrid workshop) Submission deadline: January 31, 2009 web site: http://mescal.imag.fr/membres/derrick.kondo/cfp_jogc.htm ###################################################################### Third Workshop on Desktop Grids and Volunteer Computing Systems (PCGrid 2009) held in conjunction with the IEEE International Parallel & Distributed Processing Symposium (IPDPS) May 29, 2009 Submission deadline: November 14, 2008 Rome, Italy web site: http://pcgrid.lri.fr Keynote speaker Prof. Jon Weissman, University of Minnesota, USA Desktop grids and volunteer computing systems (DGVCS's) utilize the free resources available in Intranet or Internet environments for supporting large-scale computation and storage. For over a decade, DGVCS's have been one of the largest and most powerful distributed computing systems in the world, offering a high return on investment for applications from a wide range of scientific domains (including computational biology, climate prediction, and high-energy physics). While DGVCS's sustain up to PetaFLOPS of computing power from hundreds of thousands to millions of resources, fully leveraging the platform's computational power is still a major challenge because of the immense scale, high volatility, and extreme heterogeneity of such systems. The purpose of the workshop is to provide a forum for discussing recent advances and identifying open issues for the development of scalable, fault-tolerant, and secure DGVCS's. The workshop seeks to bring desktop grid researchers together from theoretical, system, and application areas to identify plausible approaches for supporting applications with a range of complexity and requirements on desktop environments. Last year's workshop was a great success (see the past program here: http://pcgrid.lri.fr/program08.html). We invite submissions on DGVCS topics including the following: - cloud computing over unreliable enterprise or Internet resources - DGVCS middleware and software infrastructure (including management), with emphasis on virtual machines - incorporation of DGVCS's with Grid infrastructures - DGVCS programming environments and models - modeling, simulation, and emulation of large-scale, volatile environments - resource management and scheduling - resource measurement and characterization - novel DGVCS applications - data management (strategies, protocols, storage) - security on DGVCS's (reputation systems, result verification) - fault-tolerance on shared, volatile resources - peer-to-peer (P2P) algorithms or systems applied to DGVCS's With regard to the last topic, we strongly encourage authors of P2P-related paper submissions to emphasize the applicability to DGVCS's in order to be within the scope of the workshop. The workshop proceedings will be published through the IEEE Computer Society Press as part of the IPDPS CD-ROM. ###################################################################### IMPORTANT DATES Manuscript submission deadline: November 14, 2008 Acceptance Notification: January 16, 2009 Camera-ready paper deadline: February 15, 2009 Workshop: May 29, 2009 ###################################################################### SUBMISSIONS Manuscripts will be evaluated based on their originality, technical strength, quality of presentation, and relevance to the workshop scope. Only manuscripts that have neither appeared nor been submitted previously for publication are allowed. Authors are invited to submit a manuscript of up to 8 pages in IEEE format (10pt font, two-columns, single-spaced). The procedure for electronic submissions will be posted at: http://pcgrid.lri.fr/submission.html ##################################################################### ORGANIZATION General Chairs Franck Cappello, INRIA, France Derrick Kondo, INRIA, France Program Chair Gilles Fedak, INRIA, France Program Committee David Anderson, University of California at Berkeley, USA Artur Andrzejak, Zuse Institute of Berlin, Germany Filipe Araujo, University of Coimbra, Portugal Henri Bal, Vrije Universiteit, The Netherlands Zoltan Balaton, SZTAKI, Hungary Massimo Canonico, University of Piemonte Orientale, Italy Henri Casanova, University of Hawaii at Manoa, USA Abhishek Chandra, University of Minnesota, USA Frederic Desprez, INRIA, France Rudolf Eigenmann, Purdue University, USA Renato Figueiredo, University of Florida, USA Fabrice Huet, University of Nice Sophia Antipolis, France Yang-Suk Kee, University of Southern California, USA Tevfik Kosar, Louisiana State University, USA Arnaud Legrand, CNRS, France Virginia Lo, University of Oregon, USA Grzegorz Malewicz, Google Inc., USA Kevin Reed, World Community Grid, USA Olivier Richard, ID-IMAG, France Mitsuhisa Sato, University of Tsukuba, Japan Luis M. Silva, University of Coimbra, Portugal Jaspal Subhlok, University of Houston, USA Alan Sussman, University of Maryland, USA Michela Taufer, University of Delaware, USA Douglas Thain, University of Notre Dame, USA Bernard Traversat, SUN, USA Carlos Varela, Rensselaer Polytechnic Institute, USA Jon Weissman, University of Minnesota, USA From andrew at moonet.co.uk Fri Oct 17 03:05:39 2008 From: andrew at moonet.co.uk (andrew holway) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes Message-ID: Hi If you wanted to buy an smp machine of 8, 16 or 32 sockets, what would be your options? Thanks From rpnabar at gmail.com Fri Oct 17 09:15:15 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. Message-ID: >Look on the front of your system, there should be numbers 1, 2, 3, and 4 >next to the power switch. If you get an error, some of these will light >up to produce an error code. you can give that error code to support to >help narrow down the source of the problem. I don't think the meaning of >the codes are documented anywhere public. Thanks Prentice. I know about those LEDs. Unfortunately none light up. Only the Power indicator led turns orange from its usual blue. And it sucks that the SC1435's do not have the tiny LCD that flashes an error message (like on my PowerEdge 6248) -- Rahul From rpnabar at gmail.com Fri Oct 17 09:22:09 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. Message-ID: >Dell supports CentOS. They better - that's one of the OSes they will >install if you buy a cluster from them. I suspect the technician you are >dealing with doesn't even know this. You must be right. But I spoke to multiple techs and all of them claim that Fedora and Centos are not "Dell-validated-OSs" [sic]. They told me only RHEL and SUSE are. Not that it matters because I am in no position to upgrade my cluster to RHEL (nor do I think I should need to!) >And you can always lie to the support - CentOS, for all practical and >technical purposes, *IS* RHEL. Maybe! But I wasn't sure "how close". During iffy debugs "misrepresenting" is always tricky since one never knows when it is some small detail that its choking on. Primarily I wanted to rule out that this is hardware and not OS or code (which is what the techs want to always deflect me to). Boils down to this: "if I let you run whatever crappy buggy code and OS that you wanted could you ever get good hardware to go into the "orange light" crashed mode?" I don't know. What do you guys say? -- Rahul From landman at scalableinformatics.com Fri Oct 17 09:26:54 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: References: Message-ID: <48F8BCCE.1030909@scalableinformatics.com> andrew holway wrote: > Hi > > If you wanted to buy an smp machine of 8, 16 or 32 sockets, what would > be your options? x86 architecture? Others? In x86: 8 sockets is possible from a few vendors. 16 and 32 are not yest as single machines. If you don't mind using the ScaleMP software layer you can get up to 32 sockets in an intel flavor. There are AMD flavor versions from other providers that will work as well if you prefer AMD chips. Let me know if you want more info. I don't want to spam the list. Joe > > Thanks > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From skylar at cs.earlham.edu Fri Oct 17 09:36:24 2008 From: skylar at cs.earlham.edu (Skylar Thompson) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <48F8BF08.7040906@cs.earlham.edu> Rahul Nabar wrote: > Maybe! But I wasn't sure "how close". During iffy debugs > "misrepresenting" is always tricky since one never knows when it is > some small detail that its choking on. > Primarily I wanted to rule out that this is hardware and not OS or > code (which is what the techs want to always deflect me to). > > Boils down to this: "if I let you run whatever crappy buggy code and > OS that you wanted could you ever get good hardware to go into the > "orange light" crashed mode?" I don't know. What do you guys say? > Yes. When Intel went from 36-bit to 38-bit physical memory addressing, you had to be running a kernel capable of doing 38-bit addressing or you'd get wacky errors in your system logs. We had a bunch of Poweredge 1950s reporting CPU errors and flashing their orange lights, but never had any system crashes. On the SAGE or LOPSA list, somebody reported similar errors with HP Proliants. Here's the RedHat knowledgebase article on it: http://kbase.redhat.com/faq/FAQ_85_11695 -- -- Skylar Thompson (skylar@cs.earlham.edu) -- http://www.cs.earlham.edu/~skylar/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://www.scyld.com/pipermail/beowulf/attachments/20081017/6c8154d4/signature.bin From prentice at ias.edu Fri Oct 17 10:28:46 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: References: Message-ID: <48F8CB4E.50107@ias.edu> andrew holway wrote: > Hi > > If you wanted to buy an smp machine of 8, 16 or 32 sockets, what would > be your options? > Do you really mean SMP, or just 8, 16, or 32 nodes in the same box, running a single system image (SSI) of the OS? Having a system with 8, 16, or 32 sockets does not necessarily equal SMP. Any Opteron based multiprocessor system is actually a NUMA system, since each Opteron has it's own memory controller on-chip and own bank of DIMMs. If you just want an SSI, and NUMA is acceptable, you can look at the SGI Altix systems. They use SGI's NUMAlink (TM) Architecture to scale up the # of processors while behaving as a single NUMA system. The first Altix systems used Itanium processors which aren't binary compatible with x86 processors (a real inconvenience) if you were planning on running commercial, binary-only x86 software) but there are newer Altix systems (XE series) that use x86-based processors. http://www.sgi.com/products/servers/altix/xe/ If you want that many processors in a single system, you probably do want NUMA, since the single memory controller for that many processors will become a bottleneck. -- Prentice From prentice at ias.edu Fri Oct 17 10:38:20 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <48F8CD8C.2030903@ias.edu> Rahul Nabar wrote: >> Look on the front of your system, there should be numbers 1, 2, 3, and 4 >> next to the power switch. If you get an error, some of these will light >> up to produce an error code. you can give that error code to support to >> help narrow down the source of the problem. I don't think the meaning of >> the codes are documented anywhere public. > > Thanks Prentice. I know about those LEDs. Unfortunately none light up. > Only the Power indicator led turns orange from its usual blue. And it > sucks that the SC1435's do not have the tiny LCD that flashes an error > message (like on my PowerEdge 6248) > I'm in the same boat as you at the moment. I highly recommend you download Dell's DSET or Online-Diagnostics tool to help you figure out what's going on. Of the two, I prefer DSET (Dell System E-Support Tool) much more. It takes 15-20 minutes to run, but once you run it, it gives you a bunch of information, and doesn't require any services/daemons to be running (which online-diags does) The DSET output is a zip file. Unzip it somewhere where you can open the files with a web browser, then open the .hta file in the top level with your web browser. Look at the ESM logs to see what caused your LED to start blinking orange. In my case, it has been intermittent memory errors. The latest version of DSET is v1.6, A00, dated 5/28/2008. -- Prentice From coutinho at dcc.ufmg.br Fri Oct 17 11:44:00 2008 From: coutinho at dcc.ufmg.br (Bruno Coutinho) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <48F8CB4E.50107@ias.edu> References: <48F8CB4E.50107@ias.edu> Message-ID: Sun has a 8 socket AMD machine: http://www.sun.com/servers/x64/4600/ Tyan has a 8 socket opteron server too: http://www.tyan.com/product_barebones_detail.aspx?pid=338 IBM has some 8+ socket x86 systems, it seem they use a NUMAlink like interconnect called ex4: http://www-03.ibm.com/systems/x/hardware/enterprise/index.html HP has 8 socket opteron systems too: http://h10010.www1.hp.com/wwpc/us/en/en/WF04a/15351-15351-3328412-241644-3328423.html There are some high end x86 systems from UNISYS too, but beyond that I think you will have to look for RISC systems. 2008/10/17 Prentice Bisbal > andrew holway wrote: > > Hi > > > > If you wanted to buy an smp machine of 8, 16 or 32 sockets, what would > > be your options? > > > > Do you really mean SMP, or just 8, 16, or 32 nodes in the same box, > running a single system image (SSI) of the OS? > > Having a system with 8, 16, or 32 sockets does not necessarily equal > SMP. Any Opteron based multiprocessor system is actually a NUMA system, > since each Opteron has it's own memory controller on-chip and own bank > of DIMMs. > > > > If you just want an SSI, and NUMA is acceptable, you can look at the SGI > Altix systems. They use SGI's NUMAlink (TM) Architecture to scale up the > # of processors while behaving as a single NUMA system. The first Altix > systems used Itanium processors which aren't binary compatible with x86 > processors (a real inconvenience) if you were planning on running > commercial, binary-only x86 software) but there are newer Altix systems > (XE series) that use x86-based processors. > > http://www.sgi.com/products/servers/altix/xe/ > > If you want that many processors in a single system, you probably do > want NUMA, since the single memory controller for that many processors > will become a bottleneck. > > -- > Prentice > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081017/b3753395/attachment.html From hahn at mcmaster.ca Fri Oct 17 12:13:04 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <48F8CB4E.50107@ias.edu> References: <48F8CB4E.50107@ias.edu> Message-ID: > (different socket), it becomes a NUMA situation. (Is there a name for > this hybrid architecture?) yes, it's called "NUMA" ;) seriously, non-uniform doesn't mean uniformly non-uniform, if you know what I mean. otoh, NUMA should always be quantified, since a software net-shared-memory system like ScaleMP (or other similar ones) is qualitatively different from a 2-socket k8/i7 system. as a WAG, I'd guess remote memory references would cost O(2us) in SW NUMA, ~500ns in a scalable cache-line-based NUMA like SGI or the big-iron systems from IBM/Sun/HP, but ~100 ns in AMD-like systems. as usual, if your workload is cache-friendly, this doesn't matter. > systems used Itanium processors which aren't binary compatible with x86 > processors (a real inconvenience) if you were planning on running > commercial, binary-only x86 software) IMO, the ia64 environment is going to remain different and obscure, and there's little reason to go to it. Intel's hope was to push HPC along the ia64 path, but AMD64 derailed that effort. > but there are newer Altix systems > (XE series) that use x86-based processors. just IB clusters. I don't think any SW NUMA is included, or do they support ScaleMP? From tom.elken at qlogic.com Fri Oct 17 12:21:07 2008 From: tom.elken at qlogic.com (Tom Elken) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <48F8CB4E.50107@ias.edu> References: <48F8CB4E.50107@ias.edu> Message-ID: <6DB5B58A8E5AB846A7B3B3BFF1B4315A02721B4F@AVEXCH1.qlogic.org> > [mailto:beowulf-bounces@beowulf.org] On Behalf Of Prentice Bisbal > > If you just want an SSI, and NUMA is acceptable, you can look at the SGI > Altix systems. They use SGI's NUMAlink (TM) Architecture to scale up the > # of processors while behaving as a single NUMA system. The first Altix > systems used Itanium processors which aren't binary compatible with x86 > processors (a real inconvenience if you were planning on running > commercial, binary-only x86 software) but there are newer Altix systems > (XE series) that use x86-based processors. > > http://www.sgi.com/products/servers/altix/xe/ But the Altix XE and Altix ICE systems with x86_64 processors are essentially clusters with 2 sockets per node -- so not meeting the >= 8-socket SMP box desires of the original poster. > NUMA != SMP is not a universally held conclusion. By some definitions*, SGI's large Altix machines, NUMA with Itanium, are SMP machines. Each processor core in these Altix's can read/write from any memory in the machine (with differing latency), and access the I/O resources of the machine. Press releases certainly trumpet them that way: "NCSA Adds 6.5 Teraflops With SGI Altix SMP System" at http://www.hpcwire.com/offthewire/17869149.html * http://www.webopedia.com/TERM/S/SMP.html "Short for Symmetric Multiprocessing, a computer architecture that provides fast performance by making multiple CPUs available to complete individual processes simultaneously (multiprocessing). Unlike asymmetrical processing, any idle processor can be assigned any task, and additional CPUs can be added to improve performance and handle increased loads. A variety of specialized operating systems and hardware arrangements are available to support SMP. Specific applications can benefit from SMP if the code allows multithreading. SMP uses a single operating system and shares common memory and disk input/output resources. Both UNIX and Windows NT support SMP. " Altix with Itanium satisfies this definiton. -Tom > > If you want that many processors in a single system, you probably do > want NUMA, since the single memory controller for that many processors > will become a bottleneck. > > -- > Prentice > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) > visit http://www.beowulf.org/mailman/listinfo/beowulf > From lindahl at pbm.com Fri Oct 17 12:50:55 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <48F8CB4E.50107@ias.edu> References: <48F8CB4E.50107@ias.edu> Message-ID: <20081017195055.GA23258@bx9.net> On Fri, Oct 17, 2008 at 01:28:46PM -0400, Prentice Bisbal wrote: > Having a system with 8, 16, or 32 sockets does not necessarily equal > SMP. Any Opteron based multiprocessor system is actually a NUMA system, > since each Opteron has it's own memory controller on-chip and own bank > of DIMMs. Note that many people consider SMP to refer to symmetrical OS and not symmetrical memory access, and thus NUMA is a subset of SMP. The Linux kernel, for example, follows this definition... CONFIG_SMP=y is required for CONFIG_NUMA to be set. -- g From prentice at ias.edu Fri Oct 17 12:51:20 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: References: <48F8CB4E.50107@ias.edu> Message-ID: <48F8ECB8.3050808@ias.edu> Mark Hahn wrote: > >> but there are newer Altix systems >> (XE series) that use x86-based processors. > > just IB clusters. I don't think any SW NUMA is included, or do they > support ScaleMP? It is my understanding that the Altix architecture is nothing more than the Origin NUMA architecture modified to accept Intel Itanium and (later) Xeon processors. Not IB at all, but a true expandable NUMA architecture. Am I wrong, or is the XE series that much different? SGI was marketing a ScaleMP vSMP system called the Fusion 1200, that was actually made by another company, VXTECH. When I first learned about it, had SGI logos on it and everything, but it looks like that's no longer the case: http://www.sgi.com/company_info/newsroom/3rd_party/downloads/092606_fusion1200.pdf http://www.vxtech.com/products/product-categories.php?null=3&id1=17&lvl=2 -- Prentice From prentice at ias.edu Fri Oct 17 13:43:35 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <6DB5B58A8E5AB846A7B3B3BFF1B4315A02721B4F@AVEXCH1.qlogic.org> References: <48F8CB4E.50107@ias.edu> <6DB5B58A8E5AB846A7B3B3BFF1B4315A02721B4F@AVEXCH1.qlogic.org> Message-ID: <48F8F8F7.50202@ias.edu> Tom Elken wrote: >> > > NUMA != SMP is not a universally held conclusion. > > By some definitions*, SGI's large Altix machines, NUMA with Itanium, are > SMP machines. Each processor core in these Altix's can read/write from > any memory in the machine (with differing latency), and access the I/O > resources of the machine. > Press releases certainly trumpet them that way: "NCSA Adds 6.5 Teraflops > With SGI Altix SMP System" at > http://www.hpcwire.com/offthewire/17869149.html > > > * http://www.webopedia.com/TERM/S/SMP.html > "Short for Symmetric Multiprocessing, a computer architecture that > provides fast performance by making multiple CPUs available to complete > individual processes simultaneously (multiprocessing). Unlike > asymmetrical processing, any idle processor can be assigned any task, > and additional CPUs can be added to improve performance and handle > increased loads. A variety of specialized operating systems and hardware > arrangements are available to support SMP. Specific applications can > benefit from SMP if the code allows multithreading. > > SMP uses a single operating system and shares common memory and disk > input/output resources. Both UNIX and Windows NT support SMP. " > > Altix with Itanium satisfies this definiton. I always considered SMP to refer the memory access model, purely from a hardware perspective, but as your post and several other replies have indicated, there's differing opinions on this, and definitely no universal consensus. Interestingly, Wikipedia agrees with my earlier statements*, which is in opposition to webopedia's def. http://en.wikipedia.org/wiki/Symmetric_multiprocessing I consulted Hennessy & Patterson** to see what they said, but, they are, uhhhh... to verbose for a Friday afternoon. * I didn't read the *whole* entry, but skimmed it quickly while writing this. **Hennessy, John L, and Patterson, David A., "Computer Architecture: A Quantitative Approach", 3rd Ed., Morgan Kaufman Publishers, 2003 -- Prentice From hahn at mcmaster.ca Fri Oct 17 13:51:42 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <48F8ECB8.3050808@ias.edu> References: <48F8CB4E.50107@ias.edu> <48F8ECB8.3050808@ias.edu> Message-ID: >>> but there are newer Altix systems >>> (XE series) that use x86-based processors. >> >> just IB clusters. I don't think any SW NUMA is included, or do they >> support ScaleMP? > > It is my understanding that the Altix architecture is nothing more than > the Origin NUMA architecture modified to accept Intel Itanium and > (later) Xeon processors. no. the origin architecture (mips) was developed into the ia64-based altix. there is no current numalink-based system other than ia64. it's conceivable that this will change, since intel periodically makes noises about ia64 and x86_64 sharing infrastructure. that seems very attractive, but otoh the market for large SMP is quite small. > Not IB at all, but a true expandable NUMA > architecture. Am I wrong, or is the XE series that much different? it's unrelated. is that different enough? ;) From rpnabar at gmail.com Fri Oct 17 14:10:17 2008 From: rpnabar at gmail.com (Rahul Nabar) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. Message-ID: >ps. By high level support I mean that you are put in direct contact with one >of the engineers responsible for the design of these systems in the factory. >You do not want to talk to the normal support chain, and be asked if you >have run some diagnostics program downloaded from the Dell site, etc. Thanks John! Just that its hard to get that level of support especially since I have "only" 23 servers of this type here. It would be nice though! :-) I guess my best bet is to find and connect with other SC1435 users here and maybe if the problem is real there would be a mass of cases large enough for Dell to take seriously. -- Rahul From hearnsj at googlemail.com Fri Oct 17 16:32:05 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Re: PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: Message-ID: <9f8092cc0810171632n4cd1559ek27abe804fd794449@mail.gmail.com> 2008/10/17 Rahul Nabar > >ps. By high level support I mean that you are put in direct contact with > one > >of the engineers responsible for the design of these systems in the > factory. > >You do not want to talk to the normal support chain, and be asked if you > >have run some diagnostics program downloaded from the Dell site, etc. > > Thanks John! Just that its hard to get that level of support > especially since I have "only" 23 servers of this type here. It would > be nice though! :-) "Just" 23 servers? Emmmm.... no. Customers for hundreds of future sales of Dell machines, form government labs, universities and companies from here to Timbuktu are reading this. And lets not forget Google is carefully archiving this conversation. I rather hope that your Dell contact understands this. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081018/ccfbe46a/attachment.html From niftyompi at niftyegg.com Fri Oct 17 18:20:53 2008 From: niftyompi at niftyegg.com (Nifty niftyompi Mitch) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] PowerEdge SC 1435: Unexplained Crashes. In-Reply-To: References: <20081017152230.GA14072@compegg.wr.niftyegg.com> Message-ID: <20081018012053.GA3562@compegg.wr.niftyegg.com> On Fri, Oct 17, 2008 at 10:37:17AM -0500, Rahul Nabar wrote: ....... > > Stability has already changed. After I swapped motherboard+cpu. No > more dead nodes in over 2 weeks now (yay!) But I just want to make > sure this won't be a recurring problem with these SC1435's before we > go in for our next expansion. If hardware updates help then it is most likely HW.... Keep a good log... dmidecode will often give you hooks to track hardware with scripts. -- T o m M i t c h e l l Found me a new hat, now what? From niftyompi at niftyegg.com Fri Oct 17 18:57:13 2008 From: niftyompi at niftyegg.com (Nifty niftyompi Mitch) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <48F8BCCE.1030909@scalableinformatics.com> References: <48F8BCCE.1030909@scalableinformatics.com> Message-ID: <20081018015713.GB3562@compegg.wr.niftyegg.com> On Fri, Oct 17, 2008 at 12:26:54PM -0400, Joe Landman wrote: > andrew holway wrote: >> Hi >> >> If you wanted to buy an smp machine of 8, 16 or 32 sockets, what would >> be your options? > > x86 architecture? Others? To what purpose? Why so many sockets, for Beowulf? The Sicortex system has lots of cores and some of their smaller development systems and might qualify if the goal is to research SMP, NUMA, and single system memory image. As fast as CPU cores are today, memory and coherency protocols will hobble the system in unexpected ways.... In some cases Infiniband bandwidth between single and dual socket systems is an improvement over large socket count boxes. For some chassis large socket counts permit more RAM... -- T o m M i t c h e l l Found me a new hat, now what? From landman at scalableinformatics.com Fri Oct 17 19:06:02 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <20081018015713.GB3562@compegg.wr.niftyegg.com> References: <48F8BCCE.1030909@scalableinformatics.com> <20081018015713.GB3562@compegg.wr.niftyegg.com> Message-ID: <48F9448A.7040408@scalableinformatics.com> Nifty niftyompi Mitch wrote: > On Fri, Oct 17, 2008 at 12:26:54PM -0400, Joe Landman wrote: >> andrew holway wrote: >>> Hi >>> >>> If you wanted to buy an smp machine of 8, 16 or 32 sockets, what would >>> be your options? >> x86 architecture? Others? > > To what purpose? > Why so many sockets, for Beowulf? You would have to ask Andrew, though we have seen folks needing huge memory spaces as the driving factor for single systems with many sockets and lots of ram. In fact, at least one hedge fund we know went this route. > The Sicortex system has lots of cores and some of their smaller > development systems and might qualify if the goal is to research > SMP, NUMA, and single system memory image. I don't know if you would call them SSI, and I don't remember if they have strict coherency within them. This said, they have a very fast and scalable fabric. It is an interesting system. Not a commodity box, but quite interesting. > As fast as CPU cores are today, memory and coherency protocols will hobble > the system in unexpected ways.... In some cases Infiniband bandwidth Yup. > between single and dual socket systems is an improvement over large socket > count boxes. For some chassis large socket counts permit more RAM... This is the important thing. I still don't see 8GB DIMMs as being affordable for most users. I hope this changes soon. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From larry.stewart at sicortex.com Sun Oct 19 10:41:30 2008 From: larry.stewart at sicortex.com (Lawrence Stewart) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <48F9448A.7040408@scalableinformatics.com> References: <48F8BCCE.1030909@scalableinformatics.com> <20081018015713.GB3562@compegg.wr.niftyegg.com> <48F9448A.7040408@scalableinformatics.com> Message-ID: <2148D1A8-2A6B-410B-A68C-E4F2F4F0C06C@sicortex.com> On Oct 17, 2008, at 10:06 PM, Joe Landman wrote: >> >> The Sicortex system has lots of cores and some of their smaller >> development systems and might qualify if the goal is to research >> SMP, NUMA, and single system memory image. > > I don't know if you would call them SSI, and I don't remember if > they have strict coherency within them. This said, they have a very > fast and scalable fabric. It is an interesting system. Not a > commodity box, but quite interesting. The SiCortex systems are clusters of 6-core SMPs. There is no load/ store access to memory on other nodes, although the interconnect is fast enough to make software access to remote memory quite interesting. All the nodes typically run the same kernel image, but it isn't an SSI system today. I've been coding shmem and GASnet implementations for the SiCortex interconnect recently, and on the older 500 MHz systems a "put" takes about 800 ns and a "get" takes a little under 3 microseconds before any particular effort on performance. These are in line with MPI PingPong times. As far as I can tell, these will scale with clock speed on the 700 MHz systems. On the remote-paging side, I put together a prototype that gets about 2 GB/sec and 64K page fault latencies under 100 microseconds, again, not optimized. I'm kind of hoping some academic departments use the iron for operating systems, programming, and communications research. It's all open source after all. I am kind of amused by all the press and angst about "multicore", like 2, 4, or 8 cores are even an interesting problem. It seems to me that 100s of cores are pretty well understood, but when you get to 1000's, there is a lot more to learn. Of course I had a 5 cpu workstation back in 1986 or so... -Larry/SiCortex From hahn at mcmaster.ca Sun Oct 19 14:15:03 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <2148D1A8-2A6B-410B-A68C-E4F2F4F0C06C@sicortex.com> References: <48F8BCCE.1030909@scalableinformatics.com> <20081018015713.GB3562@compegg.wr.niftyegg.com> <48F9448A.7040408@scalableinformatics.com> <2148D1A8-2A6B-410B-A68C-E4F2F4F0C06C@sicortex.com> Message-ID: > The SiCortex systems are clusters of 6-core SMPs. There is no load/store > access to memory on other nodes, although the interconnect is fast enough to > make software access to remote memory quite interesting. interesting way to put it, since competing interconnects are at least as fast (I'm thinking about the usual IB claim of ~1 us latency). > I've been coding shmem and GASnet implementations for the SiCortex > interconnect recently, and on the older 500 MHz systems a "put" takes about > 800 ns and a "get" takes a little under 3 microseconds before any particular so the 800ns put is half RTT for two nodes doing ping-pong puts? > On the remote-paging side, I put together a prototype that gets about 2 > GB/sec and 64K page fault latencies under 100 microseconds, again, not > optimized. ouch. do you know how much of that time is due to slow MMU interaction? (at 2 GB/s, the wiretime for 64k should be 33 us, if I calculate correctly) > after all. I am kind of amused by all the press and angst about "multicore", well, academics need _something_ to motivate grants, and the gov isn't likley to support all the nation's CS depts on gaming research ;) also, I suspect intel and perhaps sun have encouraged the brouhaha. From larry.stewart at sicortex.com Sun Oct 19 19:20:01 2008 From: larry.stewart at sicortex.com (Lawrence Stewart) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: References: <48F8BCCE.1030909@scalableinformatics.com> <20081018015713.GB3562@compegg.wr.niftyegg.com> <48F9448A.7040408@scalableinformatics.com> <2148D1A8-2A6B-410B-A68C-E4F2F4F0C06C@sicortex.com> Message-ID: On Oct 19, 2008, at 5:15 PM, Mark Hahn wrote: >> The SiCortex systems are clusters of 6-core SMPs. There is no load/ >> store access to memory on other nodes, although the interconnect is >> fast enough to make software access to remote memory quite >> interesting. > > interesting way to put it, since competing interconnects are at > least as fast (I'm thinking about the usual IB claim of ~1 us > latency). I see HPCC results around 1.2 usec for Infinipath.. The submissions at those sort of timings are pretty thin. I am embarassed to say we haven't submitted a full run either. > > >> I've been coding shmem and GASnet implementations for the SiCortex >> interconnect recently, and on the older 500 MHz systems a "put" >> takes about 800 ns and a "get" takes a little under 3 microseconds >> before any particular > so the 800ns put is half RTT for two nodes doing ping-pong puts? No, It is bad scholarship. That figure is back to back puts, which is pretty meaningless. I expect the latency to be similar to MPI Send+Recv which is around 1.4. I'll measure the half RTT. > > >> On the remote-paging side, I put together a prototype that gets >> about 2 GB/sec and 64K page fault latencies under 100 microseconds, >> again, not optimized. > > ouch. do you know how much of that time is due to slow MMU > interaction? > (at 2 GB/s, the wiretime for 64k should be 33 us, if I calculate > correctly) > The 2 GB figure is aggregate bandwidth of multiple requests using multiple rails. Any particular transfer uses only one rail (right now) and the latency includes the server's not particularly polished thought processes. > >> after all. I am kind of amused by all the press and angst about >> "multicore", > well, academics need _something_ to motivate grants, and the gov > isn't likley to support all the nation's CS depts on gaming > research ;) > also, I suspect intel and perhaps sun have encouraged the brouhaha. From lindahl at pbm.com Sun Oct 19 21:04:08 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes Message-ID: <20081020040408.GB23601@bx9.net> On Sun, Oct 19, 2008 at 10:20:01PM -0400, Lawrence Stewart wrote: > I see HPCC results around 1.2 usec for Infinipath.. The submissions at > those sort of timings are pretty thin. > I am embarassed to say we haven't submitted a full run either. Submit one. HPCC isn't perfect, but it's the best that's available, and some of your competitors do really poorly on it. -- greg From patrick at myri.com Mon Oct 20 02:36:21 2008 From: patrick at myri.com (Patrick Geoffray) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <20081020040408.GB23601@bx9.net> References: <20081020040408.GB23601@bx9.net> Message-ID: <48FC5115.9020105@myri.com> Greg Lindahl wrote: > Submit one. HPCC isn't perfect, but it's the best that's available, I disagree. The tests integration sucks and some of the tests themselves are poorly written. For example, last time I looked, bench_lat_bw measures *8* half-RTTs with MPI_Wtime(). If your MPI_Wtime() is not using cycle counting instructions by default (which is not portable), then you are screwed. Getting the timing right is benchmarking 101. Patrick From sp at numascale.com Sat Oct 18 00:32:47 2008 From: sp at numascale.com (Steffen Persvold) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: References: Message-ID: <48F9911F.7020802@numascale.com> andrew holway wrote: > Hi > > If you wanted to buy an smp machine of 8, 16 or 32 sockets, what would > be your options? > http://www.numascale.com. Large scale NUMA architecture at cluster pricing. Disclaimer: Yes I work for them. If you have any questions, I'll be glad to answer them. Cheers, ------------------------------------------------------------------------------------ Steffen Persvold, Chief Architect NumaChip Numascale AS Tel: +47 23 16 71 88 Fax: +47 23 16 71 80 Skype: spersvold From andrew at moonet.co.uk Sat Oct 18 09:15:11 2008 From: andrew at moonet.co.uk (andrew holway) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <48F8CB4E.50107@ias.edu> References: <48F8CB4E.50107@ias.edu> Message-ID: > Do you really mean SMP, or just 8, 16, or 32 nodes in the same box, > running a single system image (SSI) of the OS? Yes, I need a shared memory machine. From Shai at ScaleMP.com Sat Oct 18 20:37:45 2008 From: Shai at ScaleMP.com (Shai Fultheim (Shai@ScaleMP.com)) Date: Wed Nov 25 01:07:54 2009 Subject: [Beowulf] Re: MS Cray In-Reply-To: References: <48D1ABCC.1010605@ldeo.columbia.edu> <48D29631.9080000@ldeo.columbia.edu> Message-ID: <9B14D1490DDECA4E974F6B9FC9EBAB31EE646F0E@VMBX108.ihostexchange.net> Mark, Thanks for the kind comment. ScaleMP does not (yet) support MS-Windows OS. --Shai -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Mark Hahn Sent: Thursday, September 18, 2008 11:22 To: Gus Correa Cc: Beowulf Subject: Re: [Beowulf] Re: MS Cray > Note that for the fully populated enclosure the power supplies are dual, i.e. > 2* 1600W = 3200W. I see that now. question for electrical-code enthusiasts on the list: isn't a normal "15 amp" office circuit actually supposed to only be used up to 12 amps? that would mean 12*120=1440; in other words, each 1600W PS needs a special circuit. also, isn't it pretty common for an office duplex receptacle to be fed by a single (15A) circuit? I guess what really surprises me is that Cray doesn't mention ScaleMP, which would seem to suit this box ideally. if someone wants Windows, I'd very much expect them to want the convenience of ignoring the fact that memory is distributed... _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From hahn at mcmaster.ca Mon Oct 20 12:16:45 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <48F9911F.7020802@numascale.com> References: <48F9911F.7020802@numascale.com> Message-ID: > http://www.numascale.com. > Large scale NUMA architecture at cluster pricing. that's fairly exciting! is this stuff somehow related to the newisys horus system? > Disclaimer: Yes I work for them. If you have any questions, I'll be glad to > answer them. OK, what's the memory profile like? ie, latency for memory from l1 cache hit all the way to all-misses-farthest-node. it would be nice to know a bit more about the actual fabric (dolphinics-sci-like? the whitepaper mentions "PCI Express Gen-2 type signals", but what does that mean? also, the WP seems pretty HT/Opteron-specific; plans for intel support? I know of only a few systems which ship with HTX slots - not an endangered species? also, <1us MPI latency sounds good, but I'm not clear on the bandwidth: what kind of single pair bandwidth is possible, as well as all-pairs? thanks, mark hahn. From hahn at mcmaster.ca Mon Oct 20 12:37:43 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <48F9911F.7020802@numascale.com> References: <48F9911F.7020802@numascale.com> Message-ID: > Disclaimer: Yes I work for them. If you have any questions, I'll be glad to > answer them. oh - one other thing: when is it available, and for what price? From lindahl at pbm.com Mon Oct 20 13:41:30 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: <48FC5115.9020105@myri.com> References: <20081020040408.GB23601@bx9.net> <48FC5115.9020105@myri.com> Message-ID: <20081020204130.GC5662@bx9.net> On Mon, Oct 20, 2008 at 05:36:21AM -0400, Patrick Geoffray wrote: > Greg Lindahl wrote: >> Submit one. HPCC isn't perfect, but it's the best that's available, > > I disagree. The tests integration sucks and some of the tests themselves > are poorly written. I agree with that. But at least it gets at a few useful numbers, unlike relying on just "bandwidth" and "latency" with a single core. I also sent the HPCC folks a few complaints, didn't get much satisfaction, and think they could improve a lot. -- greg From lindahl at pbm.com Mon Oct 20 15:46:42 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: References: <48F9911F.7020802@numascale.com> Message-ID: <20081020224642.GA20408@bx9.net> On Mon, Oct 20, 2008 at 03:16:45PM -0400, Mark Hahn wrote: >> http://www.numascale.com. >> Large scale NUMA architecture at cluster pricing. > > that's fairly exciting! is this stuff somehow related to the newisys > horus system? No, if you read the website, it's Dolphin SCI hooked up to AMD cHT. I'll give you odds on it being late and slow. > also, <1us MPI latency sounds good, Unless Dolphin learned a lot from their previous interface, that MPI number is only achieved with 2 nodes, and goes into the toilet quickly as you add nodes and cores/node. Which is why you shouldn't trust "latency" benchmarks which just use 1 core on each of 2 nodes. -- g From acastillo22 at gmail.com Mon Oct 20 16:18:56 2008 From: acastillo22 at gmail.com (Luis Alejandro Del Castillo Riley) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] mpirun issue Message-ID: <91f4bc370810201618p4c7738fat4d90e8ac675077a6@mail.gmail.com> hi fellows i have a cluster with 1 master 10 nodes with intel Xeon Quad core. Fedora core 6 PGI 7.0-7 mpich 1.2.5.2 machines.x86_64 with a 10 node names when i try to run: mpirun -v -arch x86_64 -keep_pg -nolocal -np 9 mm5.mpp i had no error but when a run with mpirun -v -arch x86_64 -keep_pg -nolocal -np 10 mm5.mpp they take around 40 min to send me and error : bm_list_4667: (1526.781250) wakeup_slave: unable to interrupt slave 0 pid 4666 somebody help -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081020/f0e74cbf/attachment.html From sp at numascale.com Mon Oct 20 16:38:58 2008 From: sp at numascale.com (Steffen Persvold) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: References: <48F9911F.7020802@numascale.com> Message-ID: <48FD1692.9080909@numascale.com> Mark Hahn wrote: >> http://www.numascale.com. >> Large scale NUMA architecture at cluster pricing. > > that's fairly exciting! is this stuff somehow related to the newisys > horus system? Hi Mark, HORUS and NumaChip share the same "vision" I'd say, but there are lots of architectural differences. > >> Disclaimer: Yes I work for them. If you have any questions, I'll be >> glad to answer them. > > OK, what's the memory profile like? ie, latency for memory from l1 > cache hit all the way to all-misses-farthest-node. Worst case scenario is a RMC (Remote Memory Cache) miss, which is ~1.2us for farthest node in a 3D Torus (2 dimension jumps, 4 total). If you have a RMC Hit however you are in practice accessing memory on a remote Opteron on a local node (~150ns). We support a RMC size of 16 GByte. > it would be nice to > know a bit more about the actual fabric (dolphinics-sci-like? > the whitepaper mentions "PCI Express Gen-2 type signals", but what > does that mean? Without going into too much detail, the fabric physical layer is PCIe based using 6 x4(i.e 1GB/s)links. The transport layer however is torus based SCI (as we're using IEEE SCI coherency protocol) with counter rotating rings. > also, the WP seems pretty HT/Opteron-specific; > plans for intel support? HT/AMD-Opteron was a natural interface for us to start with since a) it is available today, and b) the interface is proven and standardized (ever tried interfacing an ASIC with the P6 Processor bus which changes every 3 months ?). However, as we all know, Intel have announced QPI and it would be pretty "un-clever" of us to not look into that platform aswell :) > I know of only a few systems which ship > with HTX slots - not an endangered species? There are only a few systems (as in complete servers from HP/IBM etc) that ship with HTX slots today, yes. There are however quite a few Motherboard vendors that make HTX ready boards (Tyan/Supermicro/Asus etc.). > also, <1us MPI latency > sounds good, but I'm not clear on the bandwidth: what kind of single > pair bandwidth is possible, as well as all-pairs? > Pair bandwidth is today limited by the 1GByte/sec links we're using, meaning that in practice with protocol overhead etc. you're looking at roughly 1.8GByte/sec bidirectional pair bandwidth. Cheers, Steffen From reuti at staff.uni-marburg.de Tue Oct 21 05:53:26 2008 From: reuti at staff.uni-marburg.de (Reuti) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] mpirun issue In-Reply-To: <91f4bc370810201618p4c7738fat4d90e8ac675077a6@mail.gmail.com> References: <91f4bc370810201618p4c7738fat4d90e8ac675077a6@mail.gmail.com> Message-ID: Hi, Am 21.10.2008 um 01:18 schrieb Luis Alejandro Del Castillo Riley: > hi fellows i have a cluster with 1 master 10 nodes with intel Xeon > Quad core. > Fedora core 6 > PGI 7.0-7 > mpich 1.2.5.2 the last version of MPICH from 2005 is 1.2.7p1. For newer installations I would suggest to look into Open MPI. > machines.x86_64 with a 10 node names Means only the 10 nodes? > when i try to run: > mpirun -v -arch x86_64 -keep_pg -nolocal -np 9 mm5.mpp > > i had no error but when a run with > mpirun -v -arch x86_64 -keep_pg -nolocal -np 10 mm5.mpp > > they take around 40 min to send me and error : > bm_list_4667: (1526.781250) wakeup_slave: unable to interrupt slave > 0 pid 4666 With so many time, I would suggest to login to all nodes and check with: $ ps -e f (f w/o -) the ditribution and startup of the porcesses. Is it doing nothing for 40 minutes or running fine until it crashes? -- Reuti From sp at numascale.com Mon Oct 20 16:47:43 2008 From: sp at numascale.com (Steffen Persvold) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] slightly [OT] smp boxes In-Reply-To: References: <48F9911F.7020802@numascale.com> Message-ID: <48FD189F.4090908@numascale.com> Mark Hahn wrote: >> Disclaimer: Yes I work for them. If you have any questions, I'll be >> glad to answer them. > > oh - one other thing: when is it available, and for what price? > Mark, Unfortunately I cannot comment on that at the moment :) Sorry! Cheers, --Steffen From malcolm.croucher at gmail.com Tue Oct 21 01:03:27 2008 From: malcolm.croucher at gmail.com (malcolm croucher) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cluster Architecture Message-ID: <386fa5610810210103l49235afdq8dd90b773b579f45@mail.gmail.com> Hi , Im working on a new project and doing some research for the architecture of the cluster . Can anyone advise any good sites or papers or anything ? Regards Malcolm A.B Croucher -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081021/5aca0e0c/attachment.html From peter.st.john at gmail.com Tue Oct 21 09:25:42 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cluster Architecture In-Reply-To: <386fa5610810210103l49235afdq8dd90b773b579f45@mail.gmail.com> References: <386fa5610810210103l49235afdq8dd90b773b579f45@mail.gmail.com> Message-ID: Malcolm, I'd start with the wiki and its list of links, http://en.wikipedia.org/wiki/Beowulf_(computing)#External_links and RGB's Beowulf page http://www.phy.duke.edu/~rgb/Beowulf/beowulf.php Then maybe you can ask some more specific questions? Good luck Peter On 10/21/08, malcolm croucher wrote: > > Hi , > > Im working on a new project and doing some research for the architecture of > the cluster . Can anyone advise any good sites or papers or anything ? > > Regards > > Malcolm A.B Croucher > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081021/539f5d60/attachment.html From iioleynik at gmail.com Tue Oct 21 15:15:49 2008 From: iioleynik at gmail.com (Ivan Oleynik) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem Message-ID: I have heard that AMD Shanghai will be available in Nov 2008. Does someone know the pricing and performance info and how is it compared with Barcelona? Are there some informal comparisons of Shanghai vs Nehalem? Thanks, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081021/8bde157a/attachment.html From laytonjb at att.net Tue Oct 21 15:28:18 2008 From: laytonjb at att.net (Jeff Layton) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem Message-ID: <950275.39403.qm@web80702.mail.mud.yahoo.com> Those who know, can't say anything :) Be patient grasshopper, be patient. ----- Original Message ---- From: Ivan Oleynik To: beowulf@beowulf.org Sent: Tuesday, October 21, 2008 6:15:49 PM Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem I have heard that AMD Shanghai will be available in Nov 2008. Does someone know the pricing and performance info and how is it compared with Barcelona? Are there some informal comparisons of Shanghai vs Nehalem? Thanks, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081021/48d8b51a/attachment.html From deadline at eadline.org Tue Oct 21 20:19:02 2008 From: deadline at eadline.org (Douglas Eadline) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cluster Architecture In-Reply-To: <386fa5610810210103l49235afdq8dd90b773b579f45@mail.gmail.com> References: <386fa5610810210103l49235afdq8dd90b773b579f45@mail.gmail.com> Message-ID: <41882.192.168.1.213.1224645542.squirrel@mail.eadline.org> http://www.clustermonkey.net is great place to look. -- Doug > Hi , > > Im working on a new project and doing some research for the architecture > of > the cluster . Can anyone advise any good sites or papers or anything ? > > Regards > > Malcolm A.B Croucher > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Doug From bill at cse.ucdavis.edu Wed Oct 22 00:08:34 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem In-Reply-To: <950275.39403.qm@web80702.mail.mud.yahoo.com> References: <950275.39403.qm@web80702.mail.mud.yahoo.com> Message-ID: <48FED172.6070105@cse.ucdavis.edu> Jeff Layton wrote: > Those who know, can't say anything :) Be patient grasshopper, be patient. An increasing amount of hardware seems to be floating around outside of NDAs, the latest of some interest (to me anyways) is: http://www.reghardware.co.uk/2008/10/21/review_motherboard_asus_p6t_deluxe They mention at the beginning: Register Hardware has reported on both the processor and chipset - however, we can?t reveal processor performance figures until the date of the official launch. For some reason they don't count clock speed, power usage, pcmark 05, 3DMark 06, bandwidth, or memory latency. Looks like a rather nice memory system, and not too crazy power usage. Certainly not the benchmarks I'd like to see, but better than nothing. From peter.st.john at gmail.com Wed Oct 22 08:59:22 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] deskside Cray prices Message-ID: Just for fun, I configured the new deskside Cray as if for Christmas, at https://cx1.cray.com I chose one (3 slots of 8) storage node with 8x 10K rpm drives, 5 (1 slot) compute nodes (so 6x32 = 192 GB ram, ECC 800Mhz) with dual socket, quad core 3.0 GHz (so 6x8= 48 cores), and generally took everything (e.g. the battery-backup RAID controller instead of the default), 12 port infiniband kit, but no "visualization node" (which would interest you if you were using HPC for rendering). MS2008 installed would be about a $3500 premium over Red Hat installed; the support price is the same though. Since I'm "splurging" I let them install Red Hat for me. It came out to $89K, which I emphatically don't have :-) if I did, I'd surely call Joe for a quote first. I remember when Bell Labs bought new Crays, they sold their old ones to DuPont. I was in awe of the idea of a hand-me-down supercomputer (DuPont has tremendous resources and does not need to buy second-hand stuff, in general. They built the first commercial scale nuclear reactor, at the Savanah River). So I would indeed love to own a Cray, I feel the appeal. But for 90K I could have a heck of a lot of commodity nodes. But it was fun to look. Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081022/22de5575/attachment.html From kus at free.net Wed Oct 22 10:05:52 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem In-Reply-To: Message-ID: In message from "Ivan Oleynik" (Tue, 21 Oct 2008 18:15:49 -0400): >I have heard that AMD Shanghai will be available in Nov 2008. Does >someone >know the pricing and performance info and how is it compared with >Barcelona? >Are there some informal comparisons of Shanghai vs Nehalem? I beleive that Shanghai performance increase in comparison w/Barcelona will be practically defined only by possible higher Shanghai frequencies. Mikhail > >Thanks, > >Ivan From hahn at mcmaster.ca Wed Oct 22 10:23:08 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem In-Reply-To: References: Message-ID: >> Are there some informal comparisons of Shanghai vs Nehalem? > > I beleive that Shanghai performance increase in comparison w/Barcelona will > be practically defined only by possible higher Shanghai frequencies. is that based on anything hands-on? IMO, AMD needs to get a bit more serious about competing. if I7 ships with ~15 GB/s per socket and working multi-socket scalability, it's hard to imagine why anyone would bother to look at AMD. either: - there is some sort of significant flaw with I7 (runs like a dog in 64b mode or Hf turns into blue cheese after a year, etc). - AMD gets its act together (lower-latency L3, highly efficient ddr3/1333 interface, directory-based coherence). - AMD satisfies itself with bottom-feeding (which probably also means only low-end consumer stuff with little HPC interest). I've had good reason to be an AMD fan in recent-ish years, but if Intel is firing on all cylinders, AMD needs to be the rotary engine or have more cylinders, or something... From kus at free.net Wed Oct 22 10:29:13 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem In-Reply-To: Message-ID: In message from Mark Hahn (Wed, 22 Oct 2008 13:23:08 -0400 (EDT)): >>> Are there some informal comparisons of Shanghai vs Nehalem? >> >> I beleive that Shanghai performance increase in comparison >>w/Barcelona will >> be practically defined only by possible higher Shanghai frequencies. > >is that based on anything hands-on? No, I'm not under NDA - because I don't have Shanghai chips in hands :-) Mikhail > >IMO, AMD needs to get a bit more serious about competing. if I7 >ships with ~15 GB/s per socket and working multi-socket scalability, >it's hard to imagine why anyone would bother to look at AMD. either: > > - there is some sort of significant flaw with I7 (runs like a > dog in 64b mode or Hf turns into blue cheese after a year, etc). > > - AMD gets its act together (lower-latency L3, highly efficient > ddr3/1333 interface, directory-based coherence). > > - AMD satisfies itself with bottom-feeding (which probably > also means only low-end consumer stuff with little HPC interest). > >I've had good reason to be an AMD fan in recent-ish years, but if >Intel >is firing on all cylinders, AMD needs to be the rotary engine or have >more >cylinders, or something... From algomantra at gmail.com Tue Oct 21 20:48:31 2008 From: algomantra at gmail.com (AlgoMantra) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cluster Architecture In-Reply-To: <41882.192.168.1.213.1224645542.squirrel@mail.eadline.org> References: <386fa5610810210103l49235afdq8dd90b773b579f45@mail.gmail.com> <41882.192.168.1.213.1224645542.squirrel@mail.eadline.org> Message-ID: <6171110d0810212048s48f304fdg207f1d78e93ef7f7@mail.gmail.com> On Wed, Oct 22, 2008 at 8:49 AM, Douglas Eadline wrote: > > > http://www.clustermonkey.net is great place to look. > > > Many thanks! I was looking for a nice 101, ya know, a place to start. ------- -.- 1/f ))) --. ------- ... http://www.algomantra.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081022/1f0fb2df/attachment.html From andrew at moonet.co.uk Wed Oct 22 05:03:34 2008 From: andrew at moonet.co.uk (andrew holway) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Scandinavian companies working in the HPC sector Message-ID: Hello, I want to compile a list of Nordic companies engaged with supercomputing technologies. Manufacturers, ISV's, Integrators, Designers etc, Google isn't being very forthcoming. Thanks Andrew From coutinho at dcc.ufmg.br Wed Oct 22 10:34:05 2008 From: coutinho at dcc.ufmg.br (Bruno Coutinho) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem In-Reply-To: References: Message-ID: 2008/10/22 Mikhail Kuzminsky > In message from "Ivan Oleynik" (Tue, 21 Oct 2008 > 18:15:49 -0400): > >> I have heard that AMD Shanghai will be available in Nov 2008. Does someone >> know the pricing and performance info and how is it compared with >> Barcelona? >> Are there some informal comparisons of Shanghai vs Nehalem? >> > > I beleive that Shanghai performance increase in comparison w/Barcelona will > be practically defined only by possible higher Shanghai frequencies. > Mikhail > The larger cache will make difference too. 2MB is too little to four cores. I heard that AMD improved L3 latency. Is it true? They will put HyperTranprt 3.0 in it too. > > >> Thanks, >> >> Ivan >> > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081022/27743e9d/attachment.html From coutinho at dcc.ufmg.br Wed Oct 22 10:43:02 2008 From: coutinho at dcc.ufmg.br (Bruno Coutinho) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem In-Reply-To: References: Message-ID: 2008/10/22 Mark Hahn > Are there some informal comparisons of Shanghai vs Nehalem? >>> >> >> I beleive that Shanghai performance increase in comparison w/Barcelona >> will be practically defined only by possible higher Shanghai frequencies. >> > > is that based on anything hands-on? > > IMO, AMD needs to get a bit more serious about competing. if I7 > ships with ~15 GB/s per socket and working multi-socket scalability, > it's hard to imagine why anyone would bother to look at AMD. either: > > - there is some sort of significant flaw with I7 (runs like a > dog in 64b mode or Hf turns into blue cheese after a year, etc). > > - AMD gets its act together (lower-latency L3, highly efficient > ddr3/1333 interface, directory-based coherence). AMD could MCM 2 shanghai chips, resulting in 6 cores (2 tri-cores) or 8 cores (2 quad cores) and "quad channel" memory interface. But nothing prevents Intel doing it too. > > > - AMD satisfies itself with bottom-feeding (which probably > also means only low-end consumer stuff with little HPC interest). > > I've had good reason to be an AMD fan in recent-ish years, but if Intel > is firing on all cylinders, AMD needs to be the rotary engine or have more > cylinders, or something... > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081022/9dbf230b/attachment.html From hearnsj at googlemail.com Wed Oct 22 10:54:20 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] deskside Cray prices In-Reply-To: References: Message-ID: <9f8092cc0810221054y67e588dcg99f05f6ab6c4e0ec@mail.gmail.com> 2008/10/22 Peter St. John > Just for fun, I configured the new deskside Cray as if for Christmas, at > https://cx1.cray.com > ..... > > But for 90K I could have a heck of a lot of commodity nodes. But it was fun > to look. > Indeed. Doug Eadline's article on "rack em and stack em" personal clusters over on Linux Magazine this week is interesting. Notably, open-MX is now available - so you should be able to spend a lot less $$$ for your Christmas present. But sadly no Cray badge (but hey - do they have badges on their spare parts list, like auto manufacturers :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081022/ba7bde05/attachment.html From reuti at staff.uni-marburg.de Wed Oct 22 11:17:47 2008 From: reuti at staff.uni-marburg.de (Reuti) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] mpirun issue In-Reply-To: <91f4bc370810211321u2796e79aqbd0dc5acd4569d14@mail.gmail.com> References: <91f4bc370810201618p4c7738fat4d90e8ac675077a6@mail.gmail.com> <91f4bc370810211250g2c0e0198mda971cbb0dea832a@mail.gmail.com> <91f4bc370810211321u2796e79aqbd0dc5acd4569d14@mail.gmail.com> Message-ID: <9FB64325-5F77-47A0-B78F-B9FA1D31EBEF@staff.uni-marburg.de> Am 21.10.2008 um 22:21 schrieb Luis Alejandro Del Castillo Riley: > And with the ps -e f shows that is running fine until they crash > with the error broken pipe and killing signal From this I would assume that one processes crashed and you are facing only the follow-up error. Maybe because it ran out of memory or disk space. It might depend on the application, how it will distribute the data and maybe with ten nodes some array or so was getting too big over the runtime of the job. When you can spot the node which crashes, maybe you can find something in /var/log/messages of the node. -- Reuti > On Tue, Oct 21, 2008 at 2:50 PM, Luis Alejandro Del Castillo Riley > wrote: > hi > yes i have 10 nodes each ones with intel xeon quad core so basicaly > are 4 processors per each node > > > > On Tue, Oct 21, 2008 at 7:53 AM, Reuti > wrote: > Hi, > > Am 21.10.2008 um 01:18 schrieb Luis Alejandro Del Castillo Riley: > > > hi fellows i have a cluster with 1 master 10 nodes with intel Xeon > Quad core. > Fedora core 6 > PGI 7.0-7 > mpich 1.2.5.2 > > the last version of MPICH from 2005 is 1.2.7p1. For newer > installations I would suggest to look into Open MPI. > > > machines.x86_64 with a 10 node names > > Means only the 10 nodes? > > > when i try to run: > mpirun -v -arch x86_64 -keep_pg -nolocal -np 9 mm5.mpp > > i had no error but when a run with > mpirun -v -arch x86_64 -keep_pg -nolocal -np 10 mm5.mpp > > they take around 40 min to send me and error : > bm_list_4667: (1526.781250) wakeup_slave: unable to interrupt slave > 0 pid 4666 > > With so many time, I would suggest to login to all nodes and check > with: > > $ ps -e f > > (f w/o -) the ditribution and startup of the porcesses. Is it doing > nothing for 40 minutes or running fine until it crashes? > > -- Reuti > > From jan.heichler at gmx.net Wed Oct 22 11:27:40 2008 From: jan.heichler at gmx.net (Jan Heichler) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem In-Reply-To: References: Message-ID: <264892763.20081022202740@gmx.net> Hallo Mikhail, Mittwoch, 22. Oktober 2008, meintest Du: MK> In message from "Ivan Oleynik" (Tue, 21 Oct 2008 MK> 18:15:49 -0400): >>I have heard that AMD Shanghai will be available in Nov 2008. Does >>someone >>know the pricing and performance info and how is it compared with >>Barcelona? >>Are there some informal comparisons of Shanghai vs Nehalem? MK> I beleive that Shanghai performance increase in comparison w/Barcelona MK> will be practically defined only by possible higher Shanghai MK> frequencies. You can expect to see better performance in SPEC_CPU for Shanghai vs. Barcelona when comparing identical clockspeeds. But of course the increased clockspeed ist a big argument for Shanghai (or the same clockspeed with less energy consumption). And Shanghai has some more features like faster memory and HT3 in some of the later revisions i hope... Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081022/566353ae/attachment.html From bill at cse.ucdavis.edu Wed Oct 22 14:00:48 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem In-Reply-To: References: Message-ID: <48FF9480.2020104@cse.ucdavis.edu> > AMD could MCM 2 shanghai chips, resulting in 6 cores (2 tri-cores) or 8 AMD socket supports 3 hypertransport channels and 2 64 bit memory busses. Somehow you would have to connect these to 2 piece of silicon. I've not heard that AMD has much MCM experience (unlike ibm and intel). Intel on the other hand had a very easy time. Intel's dual die setup was trivial in comparison, each die was designed to sit on a shared bus and the memory controller is offchip. Seems like AMD would have put in a 3rd chip into the MCM, or some how disable some functionality on one die and share the memory bus/HT from the other chip. From rgb at phy.duke.edu Wed Oct 22 15:32:47 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cluster Architecture In-Reply-To: <41882.192.168.1.213.1224645542.squirrel@mail.eadline.org> References: <386fa5610810210103l49235afdq8dd90b773b579f45@mail.gmail.com> <41882.192.168.1.213.1224645542.squirrel@mail.eadline.org> Message-ID: On Tue, 21 Oct 2008, Douglas Eadline wrote: > > > http://www.clustermonkey.net is great place to look. > > I'll second that, just to keep Doug from being quite so shameless. Heck he has a lot of my stuff up there so it MUST be good, doesn't it? On another subject (not quite hijacking the thread, just a comment:-) I just finished reading the Linux Pro mag that replaced the remainder of my Linux Mag subscription, and they have a nifty little article on how a consortium of companies including IBM, RH and Lotus are aggressively going after the linux desktop in business. Apparently MS's Vista debacle has the smell of blood in the water, the bleating of the lamb (in their latest "I am a PC" Mac-responsive ads) excites the tiger, the fact that both Lenovo and Dell are releasing really, really cheap linux laptops ($350-ish), and perhaps a sense of long awaited payback from IBM (royally screwed by MS over OS/2) and Lotus (royally screwed by MS over spreadsheets in general) is helping to drive this. They don't have to succeed in the US to succeed, and in Europe (especially in Russia) business linux boxes are very popular with the government and many businesses who don't have the large prior investment in MS systems to act as ballast. As I've been predicting for years that the day would come, I'm watching with great interest to see if it has at long last arrived...:-) rgb > > -- > Doug > >> Hi , >> >> Im working on a new project and doing some research for the architecture >> of >> the cluster . Can anyone advise any good sites or papers or anything ? >> >> Regards >> >> Malcolm A.B Croucher >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> > > > -- > Doug > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From nixon at nsc.liu.se Wed Oct 22 23:42:48 2008 From: nixon at nsc.liu.se (Leif Nixon) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Re: "hobbyists"es In-Reply-To: <87iqw2uxno.fsf@snark.cb.piermont.com> (Perry E. Metzger's message of "Sat\, 21 Jun 2008 20\:21\:47 -0400") References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> <87abhfzpcw.fsf@snark.cb.piermont.com> <87iqw2uxno.fsf@snark.cb.piermont.com> Message-ID: [reviving a really old thread - sorry] "Perry E. Metzger" writes: > "Robert G. Brown" writes: >>> If they can't use public key auth, give 'em secure ids or something >>> similar. Works fine or such purposes. Passwords are dead. >> >> Yeah, Bill Gates (among others) said something like that back in 2004. >> I confess to being deeply skeptical. Really. The SecureID solution has >> been around for a long time at this point. It was a PITA a decade ago. >> It is a PITA now. Expensive, too. > > It is neither. I use SecureIDs quite regularly and it isn't difficult > at all -- you just look at the device and type in the digits. What's > so hard about that? It isn't that expensive, either, but if you're > minimizing cost there are cheaper competitors and various > challenge-response devices, and even non-hardware solutions. The tokens are pretty expensive, they break, they get lost, they go out of clock sync, they run out of battery and need to be replaced. The support costs are non-negligible. [the rest of this post is a general comment, not necessarily directed at Perry] That said, there are interesting stuff like the YubiKey (http://www.yubico.com/), which is a USB token pretending to be a keyboard. Press a button on it, and it "types" a one-time password. Downside: it uses symmetric crypto, which essentially means you have a shared secret between the token and the auth server. This makes the auth server a fat, juicy target, and if it ever is cracked, you need to replace all your tokens. There are also systems that send out one-time passwords via SMS to the user's cellphone. Rather neat, but you do need to pay for those SMS:es. Soft tokens, like file based client-side certs and private ssh keys, are not necessarily a *huge* improvement over simple passwords. You do become immune against the password-guessing attacks, but private keys can be stolen. We see this happening. And when a private ssh key is stolen, it is a major headache to find all authorized_keys files that contain the corresponding public key. Ssh keys *can* improve your security - encrypt the private key with a good strong passphrase, make sure it never leaves your laptop, and (carefully) use ssh-agent and agent forwarding for your authentication needs. (And add your keys with "ssh-add -c".) However, in practice, this tends to be too complicated for the average user. For a reality check, run grep -L CRYPT /home/*/.ssh/id_{r,d}sa to check how many users that have unencrypted private keys stored on your system. -- Leif Nixon - Systems expert ------------------------------------------------------------ National Supercomputer Centre - Linkoping University ------------------------------------------------------------ From nixon at nsc.liu.se Thu Oct 23 01:53:03 2008 From: nixon at nsc.liu.se (Leif Nixon) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Re: Secure authentication (Was: "hobbyists"es) In-Reply-To: <386fa5610810230137v236e355ei83ef2ff24f710f1b@mail.gmail.com> (malcolm croucher's message of "Thu\, 23 Oct 2008 10\:37\:30 +0200") References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> <87abhfzpcw.fsf@snark.cb.piermont.com> <87iqw2uxno.fsf@snark.cb.piermont.com> <386fa5610810230137v236e355ei83ef2ff24f710f1b@mail.gmail.com> Message-ID: "malcolm croucher" writes: > What about fingerprint readers ..... My brothers got one on his lap top .... > very small , neat and it must work quite well becuase i have tried to use it > but can never log on . Lift his fingerprint from one of the keys on his keyboard, etch it onto a PCB and use that as a matrix for a gelatin fake fingerprint. Or, for some sensors, just print an image of the fingerprint, lick the paper and press it to the sensor. Fingerprint readers suck. Anyway, deploying fancy biometrics stuff is normally only possible if you have control over the users' client hardware. This is often not the case in an academic setting, where you have lots of users working in loose collaborations between sites and departments. -- Leif Nixon - Systems expert ------------------------------------------------------------ National Supercomputer Centre - Linkoping University ------------------------------------------------------------ From kus at free.net Thu Oct 23 05:34:03 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Shanghai vs Barcelona, Shanghai vs Nehalem In-Reply-To: <264892763.20081022202740@gmx.net> Message-ID: In message from Jan Heichler (Wed, 22 Oct 2008 20:27:40 +0200): >Hallo Mikhail, > >Mittwoch, 22. Oktober 2008, meintest Du: > >MK> In message from "Ivan Oleynik" (Tue, 21 Oct >2008 >MK> 18:15:49 -0400): >>>I have heard that AMD Shanghai will be available in Nov 2008. Does >>>someone >>>know the pricing and performance info and how is it compared with >>>Barcelona? >>>Are there some informal comparisons of Shanghai vs Nehalem? > >MK> I beleive that Shanghai performance increase in comparison >w/Barcelona >MK> will be practically defined only by possible higher Shanghai >MK> frequencies. > >You can expect to see better performance in SPEC_CPU for Shanghai vs. >Barcelona when comparing identical clockspeeds. But of course the >increased clockspeed ist a big argument for Shanghai (or the same >clockspeed with less energy consumption). > >And Shanghai has some more features like faster memory and HT3 in >some of the later revisions i hope... Yes, I think HT3 *must* be. It was declared for Barcelona, but really is supported now AFAIK only for desktop chips. Mikhail > >Jan From hearnsj at googlemail.com Thu Oct 23 08:59:07 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Active directory with Linux Message-ID: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> I have to confess my Google skills have failed me. If I'm not wrong, there was a recent discussion in these parts re. using Active Directory with Linux. I think there was a commercial product mentioned, which was quite good. Can anyone remind me please? John Hearns -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081023/b81d2346/attachment.html From peter.st.john at gmail.com Thu Oct 23 09:02:14 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cases for DIY boxen Message-ID: On the subject of Doug's "A Case for Cases" http://www.linux-mag.com/id/7164, I had noticed that the Helmer thing ("bewwulf in an Ikea cabinet") is not really in a wood cabinet (the steel box can be put inside a cabinet). I'm assuming it's unreasonable to actually make a wood cabinet? On account of humidy, or just weight? To me it just sounds easy to build a wooden rack for a bunch of ATX motherboards. And it could look nice. Thermal and electrical insulation would be OK, and humidy controlled with a good paint job on the interior...? Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081023/28826ad2/attachment.html From peter.st.john at gmail.com Thu Oct 23 09:45:19 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Active directory with Linux In-Reply-To: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> References: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> Message-ID: John, This article http://www.securityfocus.com/infocus/1563 is aged (2002) by strikes me as a good place to start, it just seems well organized and has at least a few nice points. This article http://www.linux-watch.com/news/NS9227285361.html is recent (2008) but short, but mentions some products to consider, Likewise and Certify, and points to other reviews. If you don't get a better answer from someone who uses ActiveDirectory (I don't). Peter On 10/23/08, John Hearns wrote: > > I have to confess my Google skills have failed me. > If I'm not wrong, there was a recent discussion in these parts re. using > Active Directory > with Linux. I think there was a commercial product mentioned, which was > quite good. > Can anyone remind me please? > > John Hearns > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081023/229e07ca/attachment.html From tjrc at sanger.ac.uk Thu Oct 23 09:59:20 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Active directory with Linux In-Reply-To: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> References: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> Message-ID: <2DC4B0A9-CF4A-486A-AA60-D49D9303D7A9@sanger.ac.uk> On 23 Oct 2008, at 4:59 pm, John Hearns wrote: > I have to confess my Google skills have failed me. > If I'm not wrong, there was a recent discussion in these parts re. > using > Active Directory > with Linux. I think there was a commercial product mentioned, which > was > quite good. > Can anyone remind me please? If you just want to authenticate against AD, you don't need anything commercial at all. You can just configure PAM on your Linux boxes to authenticate against AD, and configure your nsswitch.conf to obtain its information from AD's LDAP service. We've demonstrated we can do it here but we haven't actually pushed it out on all of our boxes yet, mainly for political reasons. Some versions of Linux even have the runes built-in, so you don't have to do it by hand; SLES10, for example. You can just tell YaST2 to join the AD domain, and that's that, I believe. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From rgb at phy.duke.edu Thu Oct 23 10:15:06 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Active directory with Linux In-Reply-To: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> References: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> Message-ID: On Thu, 23 Oct 2008, John Hearns wrote: > I have to confess my Google skills have failed me. > If I'm not wrong, there was a recent discussion in these parts re. using > Active Directory > with Linux. I think there was a commercial product mentioned, which was > quite good. > Can anyone remind me please? > > John Hearns I'm not sure, but there is a rather long article on it in this month's Linux Pro magazine. However, the article only lists the need for Kerberos, NSS, PAM and Samba to make everything work. It actually is a nice article, and I might give its suggestions and methodology a try in a Windows-server-based site I help with. rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From rgb at phy.duke.edu Thu Oct 23 10:19:16 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cases for DIY boxen In-Reply-To: References: Message-ID: On Thu, 23 Oct 2008, Peter St. John wrote: > On the subject of Doug's "A Case for Cases" > http://www.linux-mag.com/id/7164, I had noticed that the Helmer thing > ("bewwulf in an Ikea cabinet") is not > really in a wood cabinet (the steel box can be put inside a cabinet). I'm > assuming it's unreasonable to actually make a wood cabinet? On account of > humidy, or just weight? To me it just sounds easy to build a wooden rack for > a bunch of ATX motherboards. And it could look nice. Thermal and electrical > insulation would be OK, and humidy controlled with a good paint job on the > interior...? What about fire? Anything electrical can in a worst case pop hot/molten metal before frying and/or blowing a breaker. Capacitors blow up (literally). A wire is badly soldered and pulls free and grounds out, spattering white hot metal. Inside a metal shell, odds are you won't get a REAL fire as there isn't much actively flammable around. In a wooden box, carefully dried by six months of 50C heat... it wouldn't take a lot to get real flames, especially if the box had e.g. a cooling fan mounted to actively fan a hot coal into flames. rgb > > Peter > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From peter.st.john at gmail.com Thu Oct 23 10:33:29 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cases for DIY boxen In-Reply-To: References: Message-ID: Robert, Yes, the consensus (offline) had seemed to be that humidy, thermal insulation, etc are not issues; and the only issue would be flammability. And yeah, I actually watched a capacitor explode under ideal circumstances (it was shadowed dark behind the box where I was looking, wondering why the prototype game box was behaving badly); it shot a beautiful little jet of flame. Incidentally, Sebastian Hyde has pictures of a really beautiful black walnut PC. I think the right word is "baroque". Really beautiful. But yeah consideration would have to made for the fire issue. Peter On 10/23/08, Robert G. Brown wrote: > > On Thu, 23 Oct 2008, Peter St. John wrote: > > On the subject of Doug's "A Case for Cases" >> http://www.linux-mag.com/id/7164, I had noticed that the Helmer thing >> ("bewwulf in an Ikea cabinet") is not >> really in a wood cabinet (the steel box can be put inside a cabinet). I'm >> assuming it's unreasonable to actually make a wood cabinet? On account of >> humidy, or just weight? To me it just sounds easy to build a wooden rack >> for >> a bunch of ATX motherboards. And it could look nice. Thermal and >> electrical >> insulation would be OK, and humidy controlled with a good paint job on the >> interior...? >> > > What about fire? Anything electrical can in a worst case pop hot/molten > metal before frying and/or blowing a breaker. Capacitors blow up > (literally). A wire is badly soldered and pulls free and grounds out, > spattering white hot metal. > > Inside a metal shell, odds are you won't get a REAL fire as there isn't > much actively flammable around. In a wooden box, carefully dried by six > months of 50C heat... it wouldn't take a lot to get real flames, > especially if the box had e.g. a cooling fan mounted to actively fan a > hot coal into flames. > > rgb > > >> Peter >> >> > -- > Robert G. Brown Phone(cell): 1-919-280-8443 > Duke University Physics Dept, Box 90305 > Durham, N.C. 27708-0305 > Web: http://www.phy.duke.edu/~rgb > Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php > Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081023/4f9bbb3a/attachment.html From algomantra at gmail.com Wed Oct 22 16:04:42 2008 From: algomantra at gmail.com (AlgoMantra) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cluster Architecture In-Reply-To: References: <386fa5610810210103l49235afdq8dd90b773b579f45@mail.gmail.com> <41882.192.168.1.213.1224645542.squirrel@mail.eadline.org> Message-ID: <6171110d0810221604h4ac7483bg68a9dff17ecebbed@mail.gmail.com> > > They don't have to succeed in the US to succeed, and in Europe > (especially in Russia) business linux boxes are very popular with the > government and many businesses who don't have the large prior investment > in MS systems to act as ballast. As I've been predicting for years that > the day would come, I'm watching with great interest to see if it has at > long last arrived...:-) You have to see what's happening here in India. There is a large engineering college/university near my house, and Ubuntu is spreading like a forest fire. ------- -.- 1/f ))) --. ------- ... http://www.algomantra.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081023/627e111f/attachment.html From malcolm.croucher at gmail.com Thu Oct 23 01:37:30 2008 From: malcolm.croucher at gmail.com (malcolm croucher) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Re: "hobbyists"es In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> <87abhfzpcw.fsf@snark.cb.piermont.com> <87iqw2uxno.fsf@snark.cb.piermont.com> Message-ID: <386fa5610810230137v236e355ei83ef2ff24f710f1b@mail.gmail.com> What about fingerprint readers ..... My brothers got one on his lap top .... very small , neat and it must work quite well becuase i have tried to use it but can never log on . On Thu, Oct 23, 2008 at 8:42 AM, Leif Nixon wrote: > [reviving a really old thread - sorry] > > "Perry E. Metzger" writes: > > > "Robert G. Brown" writes: > >>> If they can't use public key auth, give 'em secure ids or something > >>> similar. Works fine or such purposes. Passwords are dead. > >> > >> Yeah, Bill Gates (among others) said something like that back in 2004. > >> I confess to being deeply skeptical. Really. The SecureID solution has > >> been around for a long time at this point. It was a PITA a decade ago. > >> It is a PITA now. Expensive, too. > > > > It is neither. I use SecureIDs quite regularly and it isn't difficult > > at all -- you just look at the device and type in the digits. What's > > so hard about that? It isn't that expensive, either, but if you're > > minimizing cost there are cheaper competitors and various > > challenge-response devices, and even non-hardware solutions. > > The tokens are pretty expensive, they break, they get lost, they go > out of clock sync, they run out of battery and need to be replaced. > The support costs are non-negligible. > > [the rest of this post is a general comment, not necessarily directed > at Perry] > > That said, there are interesting stuff like the YubiKey > (http://www.yubico.com/), which is a USB token pretending to be a > keyboard. Press a button on it, and it "types" a one-time password. > > Downside: it uses symmetric crypto, which essentially means you have a > shared secret between the token and the auth server. This makes the > auth server a fat, juicy target, and if it ever is cracked, you need > to replace all your tokens. > > There are also systems that send out one-time passwords via SMS to the > user's cellphone. Rather neat, but you do need to pay for those > SMS:es. > > Soft tokens, like file based client-side certs and private ssh keys, > are not necessarily a *huge* improvement over simple passwords. You do > become immune against the password-guessing attacks, but private keys > can be stolen. We see this happening. And when a private ssh key is > stolen, it is a major headache to find all authorized_keys files that > contain the corresponding public key. > > Ssh keys *can* improve your security - encrypt the private key with a > good strong passphrase, make sure it never leaves your laptop, and > (carefully) use ssh-agent and agent forwarding for your authentication > needs. (And add your keys with "ssh-add -c".) However, in practice, > this tends to be too complicated for the average user. > > For a reality check, run > > grep -L CRYPT /home/*/.ssh/id_{r,d}sa > > to check how many users that have unencrypted private keys stored on > your system. > > -- > Leif Nixon - Systems expert > ------------------------------------------------------------ > National Supercomputer Centre - Linkoping University > ------------------------------------------------------------ > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Malcolm A.B Croucher -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081023/c58c00e0/attachment.html From dave.cunningham at lmco.com Thu Oct 23 10:35:41 2008 From: dave.cunningham at lmco.com (Cunningham, Dave) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Active directory with Linux In-Reply-To: References: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> Message-ID: <3D92CA467E530B4E8295214868F840FE16687493@emss01m12.us.lmco.com> There can be a lot of variation in how AD is set up in various environments. The way yours is set up and your ability to influence it will come in to play. An all open source solution might be preferable for many reasons, but if you are interested in COTS the vintela product from Quest is pretty interesting. Dave -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Robert G. Brown Sent: Thursday, October 23, 2008 10:15 AM To: John Hearns Cc: beowulf@beowulf.org Subject: Re: [Beowulf] Active directory with Linux On Thu, 23 Oct 2008, John Hearns wrote: > I have to confess my Google skills have failed me. > If I'm not wrong, there was a recent discussion in these parts re. > using Active Directory with Linux. I think there was a commercial > product mentioned, which was quite good. > Can anyone remind me please? > > John Hearns I'm not sure, but there is a rather long article on it in this month's Linux Pro magazine. However, the article only lists the need for Kerberos, NSS, PAM and Samba to make everything work. It actually is a nice article, and I might give its suggestions and methodology a try in a Windows-server-based site I help with. rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From james.p.lux at jpl.nasa.gov Thu Oct 23 10:59:44 2008 From: james.p.lux at jpl.nasa.gov (Lux, James P) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cases for DIY boxen In-Reply-To: References: Message-ID: > -----Original Message----- > From: beowulf-bounces@beowulf.org > [mailto:beowulf-bounces@beowulf.org] On Behalf Of Robert G. Brown > Sent: Thursday, October 23, 2008 10:19 AM > To: Peter St. John > Cc: beowulf@beowulf.org > Subject: Re: [Beowulf] Cases for DIY boxen > > On Thu, 23 Oct 2008, Peter St. John wrote: > > > On the subject of Doug's "A Case for Cases" > > http://www.linux-mag.com/id/7164, I had noticed that the > Helmer thing > > ("bewwulf in an Ikea cabinet") is not really in a wood cabinet (the > > steel box can be put inside a cabinet). I'm assuming it's > unreasonable > > to actually make a wood cabinet? On account of humidy, or > just weight? > > To me it just sounds easy to build a wooden rack for a bunch of ATX > > motherboards. And it could look nice. Thermal and electrical > > insulation would be OK, and humidy controlled with a good > paint job on > > the interior...? > > What about fire? Anything electrical can in a worst case pop > hot/molten metal before frying and/or blowing a breaker. > Capacitors blow up (literally). A wire is badly soldered and > pulls free and grounds out, spattering white hot metal. > > Inside a metal shell, odds are you won't get a REAL fire as > there isn't much actively flammable around. In a wooden box, > carefully dried by six months of 50C heat... it wouldn't take > a lot to get real flames, especially if the box had e.g. a > cooling fan mounted to actively fan a hot coal into flames. Well.. This *is* why if you're in a regulated environment, wooden boxes aren't as popular. However, consider that plastic is commonly used in all manner of electronic equipment that gets a UL listing. Typically it requires careful design (since UL labs will try to set it on fire).. Such things as a metal floor plate might be required. (there's also the whole brominated flame retardant issue) On the other hand, at home, maybe this is just an example of "dual function".. Supercomputer by day, room heating woodstove by night. Jim From rgb at phy.duke.edu Thu Oct 23 11:09:37 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cluster Architecture In-Reply-To: <6171110d0810221604h4ac7483bg68a9dff17ecebbed@mail.gmail.com> References: <386fa5610810210103l49235afdq8dd90b773b579f45@mail.gmail.com> <41882.192.168.1.213.1224645542.squirrel@mail.eadline.org> <6171110d0810221604h4ac7483bg68a9dff17ecebbed@mail.gmail.com> Message-ID: On Thu, 23 Oct 2008, AlgoMantra wrote: >> >> They don't have to succeed in the US to succeed, and in Europe >> (especially in Russia) business linux boxes are very popular with the >> government and many businesses who don't have the large prior investment >> in MS systems to act as ballast. As I've been predicting for years that >> the day would come, I'm watching with great interest to see if it has at >> long last arrived...:-) > > > You have to see what's happening here in India. There is a large engineering > > college/university near my house, and Ubuntu is spreading like a forest > fire. Everywhere BUT the US, that seems to be very much the case. Here desktop Linux it spreading, but more like mold. Or perhaps herpes -- it requires a certain amount of intimate contact and seduction to infect a linux virgin with it, but once they try it they can't wait to help spread it themselves. (Sor-ree... couldn't resist that one:-). Remember, if it doesn't exist on TV, it just doesn't exist, at least in the USA. Linux has gotten more visible -- there are some truly hilarious "ads" on YouTube (search for spoofs of the Mac ad that include linux) -- but there still haven't been enough real linux ads to make it exist in the minds of most Americans. Alas, rgb > > ------- -.- > 1/f ))) --. > ------- ... > http://www.algomantra.com > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From prentice at ias.edu Thu Oct 23 11:16:21 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Active directory with Linux In-Reply-To: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> References: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> Message-ID: <4900BF75.5060201@ias.edu> John Hearns wrote: > I have to confess my Google skills have failed me. > If I'm not wrong, there was a recent discussion in these parts re. using > Active Directory > with Linux. I think there was a commercial product mentioned, which was > quite good. > Can anyone remind me please? You're probably thinking of Centrify or Likewise. Don't waste your money either of them. I looked closely at Likewise, and even started to implement it, but then abondoned it, because it doesn't allow you to assign UIDs and GIDs - it generates them on each client using a predictable, repeatable (is that redundant?) hash based on login name, so that there are no ID collisions across machines. The trust is that if you already have and AD installation and the AD controllers have Microsoft Services for Unix (MSSFU, or just SFU) 3.5 or later, you have everything you need to use your AD servers as Kerberos and LDAP masters for your Linux clients. I've successfully done it here on some test systems. It will be rolled out site-wide as soon as I get the time. Believe it or not, Microsoft has this all well-documented, and has very helpful website on the topic: http://blogs.msdn.com/sfu/ I even contacted support through this website, and got a very quickly (less than an hour, I think) If you want to go the other way around, have Linux serve as the AD controllers, you'll need to use Samba, and I haven't had much success with it. What is this world coming to? I'm actually recommending Windows for something! -- Prentice From prentice at ias.edu Thu Oct 23 11:20:28 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cases for DIY boxen In-Reply-To: References: Message-ID: <4900C06C.8040807@ias.edu> Peter St. John wrote: > On the subject of Doug's "A Case for Cases" > http://www.linux-mag.com/id/7164 , I had noticed that the Helmer thing > ("bewwulf in an Ikea cabinet") is not really in a wood cabinet (the > steel box can be put inside a cabinet). I'm assuming it's unreasonable > to actually make a wood cabinet? On account of humidy, or just weight? > To me it just sounds easy to build a wooden rack for a bunch of ATX > motherboards. And it could look nice. Thermal and electrical insulation > would be OK, and humidy controlled with a good paint job on the > interior...? > Wood isn't just heavier, it's also bulkier, since it can't be made as thin as the sheetmetal used to make cases (and still be useful). It's a much better insulator then metal, too. Most high-end gamer cases are now made out of aluminum to take advantage of its superior thermal conductivity properties over steel. -- Prentice From rgb at phy.duke.edu Thu Oct 23 11:20:23 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Cases for DIY boxen In-Reply-To: References: Message-ID: On Thu, 23 Oct 2008, Lux, James P wrote: > On the other hand, at home, maybe this is just an example of "dual > function".. Supercomputer by day, room heating woodstove by night. No, no, no. If it is a supercomputer by day, you just turn off the air conditioning at night to stay warm. Well, turn it down some. Unless you like to be REALLY warm. rgb > > Jim > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From landman at scalableinformatics.com Thu Oct 23 16:11:50 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:55 2009 Subject: [Beowulf] Security issues Message-ID: <490104B6.5040401@scalableinformatics.com> Hi folks: A customer with an [insert unnamed cluster distribution here] cluster had a hack into it yesterday. We looked through it and found some things out. I wrote a post on my blog about it (http://scalability.org/?p=909). No, this is not astro-turfing, we don't derive income from the blog. No ads, no-nothing. We get enough traffic as it is. I thought what I found would be useful to the community. Since it is a long post, I thought it was more courteous to post a link than post the whole thing. Hopefully it will show up on the [insert unnamed cluster distribution here] but I doubt it, after a little exchange offline. Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From csamuel at vpac.org Thu Oct 23 16:31:56 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Cluster Architecture In-Reply-To: <1263464151.1784731224804334515.JavaMail.root@mail.vpac.org> Message-ID: <1383099895.1785141224804716827.JavaMail.root@mail.vpac.org> ----- "Robert G. Brown" wrote: > Everywhere BUT the US, that seems to be very much > the case. Here desktop Linux it spreading, but > more like mold. I think here in Australia we're probably even more reticent than the US, at least Dell US sell systems with Ubuntu pre-installed! Down here we consider ourselves lucky if we can get FreeDOS instead of Windoze and their Inspiron Mini 9 only ships with Windows XP. Our Dell rep is well aware of our feelings on this.. ;-) Even the Linux based Eeepc's are like hens teeth. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From csamuel at vpac.org Thu Oct 23 16:38:38 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Active directory with Linux In-Reply-To: <2DC4B0A9-CF4A-486A-AA60-D49D9303D7A9@sanger.ac.uk> Message-ID: <2120044993.1785371224805118397.JavaMail.root@mail.vpac.org> ----- "Tim Cutts" wrote: > If you just want to authenticate against AD, you don't need anything > commercial at all. You can just configure PAM on your Linux boxes to > authenticate against AD, and configure your nsswitch.conf to obtain > its information from AD's LDAP service. We were trying to do that for one of our members, but were told by the AD admins that we could only use the users credentials to bind to the AD server for queries as they were using lockouts on failed password attempts and so would not provide a "system" style account for queries as locking that out would stop all users from accessing the cluster. It was implied that they couldn't disable lockouts for this particular user. One of our folks tried to get this config to work and failed, so we're now going to a fallback strategy of having our own pukka LDAP server and a web frontend that will authenticate a user correctly against their AD and then let them create a POSIX LDAP account in ours. Suboptimal of course, but we've wasted enough time already banging our heads on this. :-( cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From diep at xs4all.nl Thu Oct 23 20:39:17 2008 From: diep at xs4all.nl (B. Vincent Diepeveen) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: <490104B6.5040401@scalableinformatics.com> References: <490104B6.5040401@scalableinformatics.com> Message-ID: <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> hi Joe, Thanks for your post. Very interesting to see all this. Especially the summary on what the hacker tried. Note i was quite amazed that you mentionned Rocks distribution getting used with you. A few weeks ago i grabbed latest Rocks with the idea to install it for my 1 node cluster. Both attempted with and without quadrics QM400 card. Node being a dual opteron dual core 2.4ghz, tyan S2881 motherboard. All i tried, it didn't help; the distribution crashed during somewhere always. Very weird. From all distributions i tried over a period of a few weeks install at that machine, it was the only distribution that crashed each time somewhere, usually after checking the NIC's (which were not connected to any network, let alone the internet). Sometimes also later on in the procedure; seemed to me the provided kernel in that distribution DVD to be not very good one at all. Now you post here a big story on how your Rocks got hacked. Do i conclude it correctly the problem is that you ran a default Rocks kernel? I'd say: Go take a stand on the Rocks :) Note in my case the only other problem it possibly could've been i guess is that a bunch of insecure encrypted drives are in the box. Vincent p.s. the hacker seems to me from your description some sort of guy with a criminal record. Criminal as in: the type that goes into jail regurarly, so not the Perry E Metzger type :) Note he wouldn't be able to do that anyway as he just runs a few mailing lists; but well, what the hell do i know there. On Oct 24, 2008, at 1:11 AM, Joe Landman wrote: > Hi folks: > > A customer with an [insert unnamed cluster distribution here] > cluster had a hack into it yesterday. We looked through it and > found some things out. I wrote a post on my blog about it (http:// > scalability.org/?p=909). No, this is not astro-turfing, we don't > derive income from the blog. No ads, no-nothing. We get enough > traffic as it is. I thought what I found would be useful to the > community. > > Since it is a long post, I thought it was more courteous to post > a link than post the whole thing. > > Hopefully it will show up on the [insert unnamed cluster > distribution here] but I doubt it, after a little exchange offline. > > Joe > > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics LLC, > email: landman@scalableinformatics.com > web : http://www.scalableinformatics.com > http://jackrabbit.scalableinformatics.com > phone: +1 734 786 8423 x121 > fax : +1 866 888 3112 > cell : +1 734 612 4615 > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From niftyompi at niftyegg.com Fri Oct 24 00:22:24 2008 From: niftyompi at niftyegg.com (Nifty niftyompi Mitch) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> Message-ID: <20081024072224.GA3984@compegg.wr.niftyegg.com> On Fri, Oct 24, 2008 at 05:39:17AM +0200, B. Vincent Diepeveen wrote: ..... > hi Joe, > > Thanks for your post. Very interesting to see all this. Especially the > summary on what the > hacker tried. > > Note i was quite amazed that you mentioned Rocks distribution getting > used with you. > A few weeks ago i grabbed latest Rocks with the idea to install it for my > 1 node cluster. > Both attempted with and without quadrics QM400 card. ........ > > Now you post here a big story on how your Rocks got hacked. Do i > conclude it correctly the > problem is that you ran a default Rocks kernel? The issue is not a ROCKS issue, it is a Linux and system admin issue. ` Recall that ROCKS is based on CentOS/RHEL need have no more and no less out of the box security issues than they do. Over time the list of patches for both grows and grows... The subtle issue is one we all need to take to heart as we build constraints into our environment and make changes to accommodate the community needs. Then there are the foibles of users in general. Each constraint keeps us from patching or updating one thing or another and eventually opens a risk. As Joe's blog notes the hackers hacked their way into the system via a valid account and then began a systematic attack of all the cracks and hacks that they can get their hands on. The suite of tool kits is relentless in that no attack or vector gets forgotten out on the net. Great tools like ROCKS, give a lot and also add some constraints that over time may be a primary, secondary, .... or ....ary problem in the dependency tree that opens the crack the hackers need. In general the most common hacks depend on common user errors to get them started. -- T o m M i t c h e l l Found me a new hat, now what? From eagles051387 at gmail.com Fri Oct 24 00:57:40 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: <20081024072224.GA3984@compegg.wr.niftyegg.com> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> Message-ID: did this person use the ssh exploit that red hat found a few months ago? On Fri, Oct 24, 2008 at 9:22 AM, Nifty niftyompi Mitch < niftyompi@niftyegg.com> wrote: > On Fri, Oct 24, 2008 at 05:39:17AM +0200, B. Vincent Diepeveen wrote: > ..... > > hi Joe, > > > > Thanks for your post. Very interesting to see all this. Especially the > > summary on what the > > hacker tried. > > > > Note i was quite amazed that you mentioned Rocks distribution getting > > used with you. > > A few weeks ago i grabbed latest Rocks with the idea to install it for my > > 1 node cluster. > > Both attempted with and without quadrics QM400 card. > ........ > > > > Now you post here a big story on how your Rocks got hacked. Do i > > conclude it correctly the > > problem is that you ran a default Rocks kernel? > > > The issue is not a ROCKS issue, it is a Linux and system admin issue. > ` > Recall that ROCKS is based on CentOS/RHEL need have no more > and no less out of the box security issues than they do. > > Over time the list of patches for both grows and grows... > > The subtle issue is one we all need to take to heart as we build > constraints into our environment and make changes to accommodate the > community needs. Then there are the foibles of users in general. > Each constraint keeps us from patching or updating one thing or another > and eventually opens a risk. > > As Joe's blog notes the hackers hacked their way into the system via > a valid account and then began a systematic attack of all the cracks > and hacks that they can get their hands on. The suite of tool kits is > relentless in that no attack or vector gets forgotten out on the net. > > Great tools like ROCKS, give a lot and also add some constraints > that over time may be a primary, secondary, .... or ....ary problem > in the dependency tree that opens the crack the hackers need. > > In general the most common hacks depend on common user errors > to get them started. > > -- > T o m M i t c h e l l > Found me a new hat, now what? > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081024/8f5539ea/attachment.html From tjrc at sanger.ac.uk Fri Oct 24 01:21:23 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Active directory with Linux In-Reply-To: <2120044993.1785371224805118397.JavaMail.root@mail.vpac.org> References: <2120044993.1785371224805118397.JavaMail.root@mail.vpac.org> Message-ID: On 24 Oct 2008, at 12:38 am, Chris Samuel wrote: > > ----- "Tim Cutts" wrote: > >> If you just want to authenticate against AD, you don't need anything >> commercial at all. You can just configure PAM on your Linux boxes to >> authenticate against AD, and configure your nsswitch.conf to obtain >> its information from AD's LDAP service. > > We were trying to do that for one of our members, but > were told by the AD admins that we could only use the > users credentials to bind to the AD server for queries > as they were using lockouts on failed password attempts > and so would not provide a "system" style account for > queries as locking that out would stop all users from > accessing the cluster. It was implied that they couldn't > disable lockouts for this particular user. > > One of our folks tried to get this config to work and > failed, so we're now going to a fallback strategy of > having our own pukka LDAP server and a web frontend that > will authenticate a user correctly against their AD and > then let them create a POSIX LDAP account in ours. > > Suboptimal of course, but we've wasted enough time > already banging our heads on this. :-( That's very similar to what we're doing. We're using Sun Directory Server, because there's an additional piece of software for that (whose name escapes me) which can nicely handle data synchronisation between SDS and AD. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From Dan.Kidger at quadrics.com Fri Oct 24 01:31:56 2008 From: Dan.Kidger at quadrics.com (Dan.Kidger@quadrics.com) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Cases for DIY boxen In-Reply-To: References: Message-ID: <0D49B15ACFDF2F46BF90B6E08C90048A0493821300@quadbrsex1.quadrics.com> I am not so sure that wood is as flammable as you think. Hard wood needs sustained heat for a reasonably long period of time to get going. And anyway for a computer system there is no reason why you can't do some fireproofing - get some borates, silicates or other salts to keep the organic matter away from the oxygen. Waterglass (Sodium silicate) is cheap and readily available. Although I guess the main negative factor for us is the presence of a high airflow bringing lots of fresh oxygen. So if you could use oxygen-free cooling air (!) or otherwise shut the airfow off triggered by a smoke detector ... Daniel From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Peter St. John Sent: 23 October 2008 18:33 To: Robert G. Brown Cc: beowulf@beowulf.org Subject: Re: [Beowulf] Cases for DIY boxen Robert, Yes, the consensus (offline) had seemed to be that humidy, thermal insulation, etc are not issues; and the only issue would be flammability. And yeah, I actually watched a capacitor explode under ideal circumstances (it was shadowed dark behind the box where I was looking, wondering why the prototype game box was behaving badly); it shot a beautiful little jet of flame. Incidentally, Sebastian Hyde has pictures of a really beautiful black walnut PC. I think the right word is "baroque". Really beautiful. But yeah consideration would have to made for the fire issue. Peter On 10/23/08, Robert G. Brown > wrote: On Thu, 23 Oct 2008, Peter St. John wrote: On the subject of Doug's "A Case for Cases" http://www.linux-mag.com/id/7164, I had noticed that the Helmer thing ("bewwulf in an Ikea cabinet") is not really in a wood cabinet (the steel box can be put inside a cabinet). I'm assuming it's unreasonable to actually make a wood cabinet? On account of humidy, or just weight? To me it just sounds easy to build a wooden rack for a bunch of ATX motherboards. And it could look nice. Thermal and electrical insulation would be OK, and humidy controlled with a good paint job on the interior...? What about fire? Anything electrical can in a worst case pop hot/molten metal before frying and/or blowing a breaker. Capacitors blow up (literally). A wire is badly soldered and pulls free and grounds out, spattering white hot metal. Inside a metal shell, odds are you won't get a REAL fire as there isn't much actively flammable around. In a wooden box, carefully dried by six months of 50C heat... it wouldn't take a lot to get real flames, especially if the box had e.g. a cooling fan mounted to actively fan a hot coal into flames. rgb Peter -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081024/552680fc/attachment.html From eagles051387 at gmail.com Fri Oct 24 02:23:47 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: <49018D9A.2080303@gmail.com> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> Message-ID: now i see why the sudo approach adopted by debian and the kubuntu line is a good way to go. this is providing me with real motivation to start the development of my own kubuntu derived cluster distro. thing is i would need someone to give lists of pkgs that is used in a cluster and also testers and programmers to help me out seeing as i dont have a cluster. On Fri, Oct 24, 2008 at 10:55 AM, Kilian CAVALOTTI < kilian.cavalotti.work@gmail.com> wrote: > Jon Aquilina wrote: > >> did this person use the ssh exploit that red hat found a few months ago? >> > > Apparently not. From what Joe wrote, "the entry point was via a shared user > account". This account has been compromised, either with brute-force ssh > login attempts, or was socially engineered, it's not clear. > > Nothing seems to indicate (as far as I can tell) that the entry point was > due to some weakness in one of the Rocks components. I second Mitch in > saying that this break-in isn't Rocks specific, but rather the result of > poor (lack of?) administration practices (especially from what I could read > here: http://scalability.org/?p=905, and assuming it's about the same > customer). > > On the other hand, it's true that Rocks' philosophy (which I'm not a big > proponent of) doesn't make updates easy, nor encourage keeping systems > up-to-date. It tends to focus on the Windowsian "reinstall the whole > machine" approach in case of problem. Which makes perfect sense in specific > contexts, where no dedicated administration resources are available, or > where compute time is critical and understanding the root cause of technical > problems not so important. > > But this can also lead to the kind of security problem Joe described, even > if here, I don't think one can blame any of the system's component being > outdated for this intrusion. > > Cheers, > -- > Kilian > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081024/44a70f8d/attachment.html From award at uda.ad Fri Oct 24 02:25:40 2008 From: award at uda.ad (Alan Ward) Date: Wed Nov 25 01:07:56 2009 Subject: RS: [Beowulf] Cases for DIY boxen References: <0D49B15ACFDF2F46BF90B6E08C90048A0493821300@quadbrsex1.quadrics.com> Message-ID: Nowadays many exposition halls, restaurants etc. in Europe have stratified wood beams. These need to be treated with fire retardants by regulation, in which case they are actually more safe than steel (steel bends and bucles when warmed). You could take a look at whatever products they use to treat them. PS: My last DIY box was a TV table on wheels, with a couple of shelves for video casettes underneath beahind glass doors. I took out the shelves and put in 4 motherboards, and there I was with a 70x70x50 cm mobile cluster. Cheers, -Alan -----Missatge original----- De: beowulf-bounces@beowulf.org en nom de Dan.Kidger@quadrics.com Enviat el: dv. 24/10/2008 10:31 Per a: peter.st.john@gmail.com; rgb@phy.duke.edu A/c: Beowulf@beowulf.org Tema: RE: [Beowulf] Cases for DIY boxen I am not so sure that wood is as flammable as you think. Hard wood needs sustained heat for a reasonably long period of time to get going. And anyway for a computer system there is no reason why you can't do some fireproofing - get some borates, silicates or other salts to keep the organic matter away from the oxygen. Waterglass (Sodium silicate) is cheap and readily available. Although I guess the main negative factor for us is the presence of a high airflow bringing lots of fresh oxygen. So if you could use oxygen-free cooling air (!) or otherwise shut the airfow off triggered by a smoke detector ... Daniel From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Peter St. John Sent: 23 October 2008 18:33 To: Robert G. Brown Cc: beowulf@beowulf.org Subject: Re: [Beowulf] Cases for DIY boxen Robert, Yes, the consensus (offline) had seemed to be that humidy, thermal insulation, etc are not issues; and the only issue would be flammability. And yeah, I actually watched a capacitor explode under ideal circumstances (it was shadowed dark behind the box where I was looking, wondering why the prototype game box was behaving badly); it shot a beautiful little jet of flame. Incidentally, Sebastian Hyde has pictures of a really beautiful black walnut PC. I think the right word is "baroque". Really beautiful. But yeah consideration would have to made for the fire issue. Peter On 10/23/08, Robert G. Brown > wrote: On Thu, 23 Oct 2008, Peter St. John wrote: On the subject of Doug's "A Case for Cases" http://www.linux-mag.com/id/7164, I had noticed that the Helmer thing ("bewwulf in an Ikea cabinet") is not really in a wood cabinet (the steel box can be put inside a cabinet). I'm assuming it's unreasonable to actually make a wood cabinet? On account of humidy, or just weight? To me it just sounds easy to build a wooden rack for a bunch of ATX motherboards. And it could look nice. Thermal and electrical insulation would be OK, and humidy controlled with a good paint job on the interior...? What about fire? Anything electrical can in a worst case pop hot/molten metal before frying and/or blowing a breaker. Capacitors blow up (literally). A wire is badly soldered and pulls free and grounds out, spattering white hot metal. Inside a metal shell, odds are you won't get a REAL fire as there isn't much actively flammable around. In a wooden box, carefully dried by six months of 50C heat... it wouldn't take a lot to get real flames, especially if the box had e.g. a cooling fan mounted to actively fan a hot coal into flames. rgb Peter -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081024/5e623715/attachment.html From eagles051387 at gmail.com Fri Oct 24 03:16:30 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> Message-ID: >>my response Im not goign to turn this into a distro war everyone is entitled to their opinions and preferences. that is not the problem packaing or repackaing everything. are there any debian based distros out there? >> from john hearns Oooh, here we go again. Distro wars :-) Jon, if I may be permitted to play the Devils Advocate this morning, any "cluster distro" jsut has the same packages as any other server distro. Kernel, init scripts, ssh daemon, ntp, blah de blah de blah. The 'secret sauce' is in the HPC stack. This consists of the drivers for any high performance interconnect, the libraries for the HPC interconnect, the MPI libraries, the high performance compilers, the maths libraries, the batch shceduler and finally the applications on top. In many cases the above are not packaged as RPMs (sorry - showing which side of the fence I come from). As an aside, all credit should be due here to Pathscale/Qlogic for superbly packaging kernel modules and libraries for Infinipath, enabling you to create a working MPI setup with little more than an 'rpm install' command. Top marks. To create this Debian based cluster distribution you're going to have to package the above software types - and if anyone is going to actually use it, they'll be looking at the most recent versions also. Quite often in a distribution you will find an RPM for (say) LAMMPI or Gridengine - but they're seriously out of date. Let the debate begin. On Fri, Oct 24, 2008 at 11:23 AM, Jon Aquilina wrote: > now i see why the sudo approach adopted by debian and the kubuntu line is a > good way to go. this is providing me with real motivation to start the > development of my own kubuntu derived cluster distro. thing is i would need > someone to give lists of pkgs that is used in a cluster and also testers and > programmers to help me out seeing as i dont have a cluster. > > On Fri, Oct 24, 2008 at 10:55 AM, Kilian CAVALOTTI < > kilian.cavalotti.work@gmail.com> wrote: > >> Jon Aquilina wrote: >> >>> did this person use the ssh exploit that red hat found a few months ago? >>> >> >> Apparently not. From what Joe wrote, "the entry point was via a shared >> user account". This account has been compromised, either with brute-force >> ssh login attempts, or was socially engineered, it's not clear. >> >> Nothing seems to indicate (as far as I can tell) that the entry point was >> due to some weakness in one of the Rocks components. I second Mitch in >> saying that this break-in isn't Rocks specific, but rather the result of >> poor (lack of?) administration practices (especially from what I could read >> here: http://scalability.org/?p=905, and assuming it's about the same >> customer). >> >> On the other hand, it's true that Rocks' philosophy (which I'm not a big >> proponent of) doesn't make updates easy, nor encourage keeping systems >> up-to-date. It tends to focus on the Windowsian "reinstall the whole >> machine" approach in case of problem. Which makes perfect sense in specific >> contexts, where no dedicated administration resources are available, or >> where compute time is critical and understanding the root cause of technical >> problems not so important. >> >> But this can also lead to the kind of security problem Joe described, even >> if here, I don't think one can blame any of the system's component being >> outdated for this intrusion. >> >> Cheers, >> -- >> Kilian >> > > > > -- > Jonathan Aquilina > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081024/486d0205/attachment.html From award at uda.ad Fri Oct 24 03:26:05 2008 From: award at uda.ad (Alan Ward) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues References: <490104B6.5040401@scalableinformatics.com><697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl><20081024072224.GA3984@compegg.wr.niftyegg.com><49018D9A.2080303@gmail.com> Message-ID: Kubuntu-derived? Would Debian not be a better way to go, as in not installing any graphical stuff unless the user needs it? As for testing, if you have a workstation with 1-2 Gigs of RAM, perhaps you could consider a "virtual cluster". Cheers, -Alan -----Original Message----- From: beowulf-bounces@beowulf.org on behalf of Jon Aquilina Sent: Fri 10/24/2008 11:23 AM To: Kilian CAVALOTTI Cc: Nifty niftyompi Mitch; beowulf Subject: Re: [Beowulf] Security issues now i see why the sudo approach adopted by debian and the kubuntu line is a good way to go. this is providing me with real motivation to start the development of my own kubuntu derived cluster distro. thing is i would need someone to give lists of pkgs that is used in a cluster and also testers and programmers to help me out seeing as i dont have a cluster. On Fri, Oct 24, 2008 at 10:55 AM, Kilian CAVALOTTI < kilian.cavalotti.work@gmail.com> wrote: > Jon Aquilina wrote: > >> did this person use the ssh exploit that red hat found a few months ago? >> > > Apparently not. From what Joe wrote, "the entry point was via a shared user > account". This account has been compromised, either with brute-force ssh > login attempts, or was socially engineered, it's not clear. > > Nothing seems to indicate (as far as I can tell) that the entry point was > due to some weakness in one of the Rocks components. I second Mitch in > saying that this break-in isn't Rocks specific, but rather the result of > poor (lack of?) administration practices (especially from what I could read > here: http://scalability.org/?p=905, and assuming it's about the same > customer). > > On the other hand, it's true that Rocks' philosophy (which I'm not a big > proponent of) doesn't make updates easy, nor encourage keeping systems > up-to-date. It tends to focus on the Windowsian "reinstall the whole > machine" approach in case of problem. Which makes perfect sense in specific > contexts, where no dedicated administration resources are available, or > where compute time is critical and understanding the root cause of technical > problems not so important. > > But this can also lead to the kind of security problem Joe described, even > if here, I don't think one can blame any of the system's component being > outdated for this intrusion. > > Cheers, > -- > Kilian > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081024/3a0bcd2f/attachment.html From tjrc at sanger.ac.uk Fri Oct 24 04:17:25 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> Message-ID: <0894519F-A230-4DBB-83D9-7E01C67B0863@sanger.ac.uk> On 24 Oct 2008, at 11:16 am, Jon Aquilina wrote: >>> my response > > Im not goign to turn this into a distro war everyone is entitled to > their > opinions and preferences. that is not the problem packaing or > repackaing > everything. > > are there any debian based distros out there? You don't really need to make a new derived distro - newer releases of Ubuntu and/or Debian, in particular, have a lot of cluster-related gubbins which you can install; here's a sample: Monitoring: ganglia Running commands everywhere: dsh clusterssh Cluster filesystems: gluster lustre 1.6 client ocfs2 cman Configuration management: cfengine Hands-free provisioning: FAI (Fully Automated Install) Those are just the bits and pieces which I either use now or am looking at. I've also noted quite a few bits of HEP community stuff in there; cernlib, for example, and a very large number of bioinformatics codes (which is why I'm using Ubuntu as a platform for providing Bioinformatics training workshops worldwide for the Wellcome Trust; I just create a re-mixed Xubuntu LiveCD with the bioinformatics packages on it required for the particular course, and the students can take the CD home with them afterwards and carry on working on their own machine). We're using this LiveCD for the first time in Bangkok in two weeks' time. Regards, Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From tjrc at sanger.ac.uk Fri Oct 24 04:19:59 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com><697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl><20081024072224.GA3984@compegg.wr.niftyegg.com><49018D9A.2080303@gmail.com> Message-ID: On 24 Oct 2008, at 11:26 am, Alan Ward wrote: > > > Kubuntu-derived? Would Debian not be a better way to go, as in not > installing any graphical stuff unless the user needs it? Ubuntu is a little ahead of Debian in terms of some of the cluster software it contains. For example, gluster is in Ubuntu 8.10, but is not in Debian Lenny. It is in Debian's development tree, but won't hit a production Debian release for God knows how long. :-) You can always install Ubuntu without any of the GUI gubbins. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From carsten.aulbert at aei.mpg.de Fri Oct 24 04:25:09 2008 From: carsten.aulbert at aei.mpg.de (Carsten Aulbert) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Idle core on overloaded CPU? Message-ID: <4901B095.6060004@aei.mpg.de> Hi, (I send this yesterday already to LKML, so sorry if you receive this twice), on Intel X3220 CPU based systems (4 physical cores) I came across the following thing (Debian etch, with vanilla kernel 2.6.25.9): starting the following: $ screen -d -m stress -c 2 $ nice -19 screen -d -m stress -c 4 This causes two cores to be 100% busy in user state and one core to be busy in 100% nice state. However, the remaining core is idle. Even watching it over longer stretches of time, the situation remains static. I guess it's a kernel misconfiguration on my part (config available here: https://n0.aei.uni-hannover.de/linux/kernel/config-2.6.25.9-nodes ), but if not could this be a scheduling bug? Thanks for any hint Cheers Carsten PS: In the mean time we have made similar tests on a dual CPU, dual core opteron machine with the same results (on a different kernel, 2.6.24.x). Most bizarre was this result (thanks Steffen!): 6 jobs with nice 19, one with nice 0: Core 1: 100% with nice 0 Core 2: 100% with single nice 19 Core 3: 2*50% (nice 19) Core 4: 3*33% (nice 19) That's definitely something I would not expect. From eagles051387 at gmail.com Fri Oct 24 04:42:14 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> Message-ID: but why waste time sifting through all 26,000+ pkgs in the repos when u can have a distro with repos focused on clustering pkgs? On Fri, Oct 24, 2008 at 1:19 PM, Tim Cutts wrote: > > On 24 Oct 2008, at 11:26 am, Alan Ward wrote: > > >> >> Kubuntu-derived? Would Debian not be a better way to go, as in not >> installing any graphical stuff unless the user needs it? >> > > Ubuntu is a little ahead of Debian in terms of some of the cluster software > it contains. For example, gluster is in Ubuntu 8.10, but is not in Debian > Lenny. It is in Debian's development tree, but won't hit a production > Debian release for God knows how long. :-) You can always install Ubuntu > without any of the GUI gubbins. > > Tim > > > -- > The Wellcome Trust Sanger Institute is operated by Genome ResearchLimited, > a charity registered in England with number 1021457 and acompany registered > in England with number 2742969, whose registeredoffice is 215 Euston Road, > London, NW1 2BE._______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081024/0ca1b285/attachment.html From carsten.aulbert at aei.mpg.de Fri Oct 24 04:49:40 2008 From: carsten.aulbert at aei.mpg.de (Carsten Aulbert) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> Message-ID: <4901B654.5060003@aei.mpg.de> Hi Jon Jon Aquilina wrote: > but why waste time sifting through all 26,000+ pkgs in the repos when u > can have a distro with repos focused on clustering pkgs? Because you might/will save time later when you hit user requests which want packages which are not pre-packaged in your cluster distro. Cheers Carsten From eagles051387 at gmail.com Fri Oct 24 05:03:04 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: <4901B654.5060003@aei.mpg.de> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> <4901B654.5060003@aei.mpg.de> Message-ID: true but if there is something that isnt in there i would be more then willing to add it to the repo. On Fri, Oct 24, 2008 at 1:49 PM, Carsten Aulbert wrote: > Hi Jon > > Jon Aquilina wrote: > > but why waste time sifting through all 26,000+ pkgs in the repos when u > > can have a distro with repos focused on clustering pkgs? > > Because you might/will save time later when you hit user requests which > want packages which are not pre-packaged in your cluster distro. > > Cheers > > Carsten > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081024/e088c96f/attachment.html From tjrc at sanger.ac.uk Fri Oct 24 05:31:16 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> Message-ID: <895A1B3C-7590-4596-B3D3-6B28ED7D3382@sanger.ac.uk> On 24 Oct 2008, at 12:42 pm, Jon Aquilina wrote: > but why waste time sifting through all 26,000+ pkgs in the repos > when u can > have a distro with repos focused on clustering pkgs? They're grouped according to purpose, so I didn't have to do any such sifting. That list I produced from a cursory glance through the scientific software section, and the few packages I already knew about. I also very much agree with Carsten - there have been any number of occasions when people have asked me for really quite obscure bits and pieces of software, and I've thought "Oh, that's never going to be packaged up already... oh, wait, yes it is... aptitude install really-obscure-package" The point I was countering was the idea of creating [yet another] derived distribution, when it isn't necessary. Maintaining a Linux distribution is an *enormous* amount of effort, especially if you're going to attempt to keep up with security patches and so on, so why anyone would want to do it themselves is beyond me. Just use one of the widely used and well supported distros, and supplement it with a local package repository if you have local custom needs. This is what we do; we use plain old Debian Etch, with a local package repository of a few things we have updated to more recent software versions (cfengine, rsync, multipath, heartbeat and one or two other things). I do the equivalent thing for SLES10, for our Oracle boxes. When we decided to move from Debian Sarge to Etch, it was a really simple move; we had half a dozen or so local packages to rebuild against the updated distro, and that was it, we were up and running. The amount of effort required is pretty small (at least for anyone who's passingly familiar with build RPMs or DEBs from source) Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From rgb at phy.duke.edu Fri Oct 24 05:40:25 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> <4901B654.5060003@aei.mpg.de> Message-ID: On Fri, 24 Oct 2008, Jon Aquilina wrote: > true but if there is something that isnt in there i would be more then > willing to add it to the repo. But what is the ADVANTAGE of reducing the number of packages in a warehouse from which one pulls packages, when the filled warehouse already exists and is free? I don't know about Debian so much, but with RPM repos one can set up the primary full-distro repo, and create as many "local" repos as one wishes on the side (or link in to e.g. livna -- other repos other humans maintain and that you trust). I can only presume that Debianish distros can do the same. So create a "cluster distro" as an OVERLAY on TOP of an existing distro. That way you do far, far less work. All the distro packages are there if you need them. Most of the cluster packages you might need are already there. If you need to rebuild them, augment them with non-distro packages, or e.g. add some custom kernels, build the replacement/augmentations, package them, pop them in an overlay. Yum will (if told to nicely) use them instead of the ones in the regular distro. Don't forget, of course, that then YOU are responsible for maintaining the update stream of any packages you replace -- if the upstream version is patched, you'd better (re-re-)patch your augmented version. This keeps the amount of work to the theoretical minimum required to achieve your goals, costs you "nothing" (what does disk cost per GB these days -- $0.20 or thereabouts? -- so keeping a full distro costs you a few dollars, literally), and makes it extremely easy to track updates and upgrades without YOU doing a ton of work. rgb > > On Fri, Oct 24, 2008 at 1:49 PM, Carsten Aulbert > wrote: > >> Hi Jon >> >> Jon Aquilina wrote: >>> but why waste time sifting through all 26,000+ pkgs in the repos when u >>> can have a distro with repos focused on clustering pkgs? >> >> Because you might/will save time later when you hit user requests which >> want packages which are not pre-packaged in your cluster distro. >> >> Cheers >> >> Carsten >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> > > > > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From d.love at liverpool.ac.uk Fri Oct 24 05:47:00 2008 From: d.love at liverpool.ac.uk (Dave Love) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Re: Active directory with Linux In-Reply-To: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> (John Hearns's message of "Thu, 23 Oct 2008 16:59:07 +0100") References: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> Message-ID: <87wsfy6tyz.fsf@liv.ac.uk> "John Hearns" writes: > I have to confess my Google skills have failed me. > If I'm not wrong, there was a recent discussion in these parts re. using > Active Directory > with Linux. [I assume that means GNU/Linux.] You probably want to be more specific about the value of `using' and what the circumstances are, e.g. whether `you' have control of the AD. The latter is likely the main constraint. > I think there was a commercial product mentioned, which was > quite good. There are several, but you probably don't want them. The ones with which I'm familiar -- too familiar in the case of Centrify -- are aimed at Doze people who think they should/can control everything with AD, specifically its normal interface. From d.love at liverpool.ac.uk Fri Oct 24 05:48:11 2008 From: d.love at liverpool.ac.uk (Dave Love) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Re: Active directory with Linux In-Reply-To: <2120044993.1785371224805118397.JavaMail.root@mail.vpac.org> (Chris Samuel's message of "Fri, 24 Oct 2008 10:38:38 +1100 (EST)") References: <2DC4B0A9-CF4A-486A-AA60-D49D9303D7A9@sanger.ac.uk> <2120044993.1785371224805118397.JavaMail.root@mail.vpac.org> Message-ID: <87vdvi6tx0.fsf@liv.ac.uk> Chris Samuel writes: > We were trying to do that for one of our members, but > were told by the AD admins that we could only use the > users credentials to bind to the AD server for queries > as they were using lockouts on failed password attempts > and so would not provide a "system" style account for > queries as locking that out would stop all users from > accessing the cluster. I don't understand that. If you need LDAP data, as opposed to just Kerberos authentication, and you're not allowed anonymous access to it, you either use a `well-known' password on a special account (which you're probably also not allowed...) or the `machine' account. The latter is what you get from `joining the domain' (e.g. with Samba) and, as far as I remember, is just the system's Kerberos host principal, whose key you stash in a keytab. Obviously avoid AD if you can, though. From d.love at liverpool.ac.uk Fri Oct 24 06:01:28 2008 From: d.love at liverpool.ac.uk (Dave Love) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Re: Active directory with Linux In-Reply-To: <4900BF75.5060201@ias.edu> (Prentice Bisbal's message of "Thu, 23 Oct 2008 14:16:21 -0400") References: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> <4900BF75.5060201@ias.edu> Message-ID: <87r6666tav.fsf@liv.ac.uk> Prentice Bisbal writes: > The trust is that if you already have and AD installation and the AD > controllers have Microsoft Services for Unix (MSSFU, or just SFU) 3.5 or > later, you have everything you need to use your AD servers as Kerberos > and LDAP masters for your Linux clients. You only need that stuff for the NSS databases (passwd, group), not for Kerberos. [I never managed to get the add-on SFE stuff to install -- even after recovering from the server being 0wned whilst it was getting security-patched -- but I guess that's not a general problem.] > If you want to go the other way around, have Linux serve as the AD > controllers, you'll need to use Samba, and I haven't had much success > with it. Samba as an actual AD controller is a Samba 4 thing, which isn't ready yet, as far as I know -- has that changed recently? The canonical way to DTRT is to have a master Kerberos server in the POSIX world, which AD trusts, and populate the POSIX and AD worlds' LDAP separately from one or more accounts databases. Basically you want to keep AD in its own world, and in a network subdomain with a sensible DNS arrangement, since AD wants to control DNS. From gdjacobs at gmail.com Fri Oct 24 06:17:01 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> <4901B654.5060003@aei.mpg.de> Message-ID: <4901CACD.5030706@gmail.com> Robert G. Brown wrote: > On Fri, 24 Oct 2008, Jon Aquilina wrote: > >> true but if there is something that isnt in there i would be more then >> willing to add it to the repo. > > But what is the ADVANTAGE of reducing the number of packages in a > warehouse from which one pulls packages, when the filled warehouse > already exists and is free? > > I don't know about Debian so much, but with RPM repos one can set up the > primary full-distro repo, and create as many "local" repos as one wishes > on the side (or link in to e.g. livna -- other repos other humans > maintain and that you trust). I can only presume that Debianish distros > can do the same. Yes, very much so. Many, many updated packages or packages of dubious legality (libdvdcss anyone?) are available in ancillary repositories. > So create a "cluster distro" as an OVERLAY on TOP of an existing distro. > That way you do far, far less work. All the distro packages are there > if you need them. Most of the cluster packages you might need are > already there. If you need to rebuild them, augment them with > non-distro packages, or e.g. add some custom kernels, build the > replacement/augmentations, package them, pop them in an overlay. Yum > will (if told to nicely) use them instead of the ones in the regular > distro. Or package the source packages and submit them upstream. Volunteer for a life of servitude! > Don't forget, of course, that then YOU are responsible for maintaining > the update stream of any packages you replace -- if the upstream version > is patched, you'd better (re-re-)patch your augmented version. > > This keeps the amount of work to the theoretical minimum required to > achieve your goals, costs you "nothing" (what does disk cost per GB > these days -- $0.20 or thereabouts? -- so keeping a full distro costs > you a few dollars, literally), and makes it extremely easy to track > updates and upgrades without YOU doing a ton of work. Usually when I build a cluster, I make local builds of MPICH2 for each compiler. This does not fit well with the paradigm of really any distro I've ever seen, which is why I leave it as a custom layer on top of e.g. Debian and do not package it. I have yet to see a distro do multiarch really well, so for the moment I try to work around (or perhaps above) the system and avoid using APT/YUM for handling multiple architectures/compiler toolchains. > > rgb > -- Geoffrey D. Jacobs From landman at scalableinformatics.com Fri Oct 24 06:58:24 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Anyone going to SC08 up for an informal BOF on cluster security? Message-ID: <4901D480.1070502@scalableinformatics.com> Probably too late to get it onto the schedule, but I think it could be a useful discussion to get troubles and best practices talked about. Certainly away from areas where people may become unhappy with content or wording, and we can all have frank and open discussions. This would be a good thing IMO. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From award at uda.ad Fri Oct 24 07:04:35 2008 From: award at uda.ad (Alan Ward) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues References: <490104B6.5040401@scalableinformatics.com><697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl><20081024072224.GA3984@compegg.wr.niftyegg.com><49018D9A.2080303@gmail.com><4901B654.5060003@aei.mpg.de> Message-ID: It is much the same in the Debian world. As has been pointed out today, there are some differences between Ubuntu and Debian as to package versions (different kernel versions) and artwork packages, but you can often use packages from a Debian repository with a Ubuntu installation. Perhaps also the other way 'round, though I have not experimented with that. Perhaps an alternative way to go for a cluster install disk would be a bog standard Debian/Ubuntu/Kubuntu/Xubuntu boot iso image, with a customized install script that pulls in a suplementary set of packages over the network and does some extra configuration (NFS server, PXE server etc). My point was that it may be easier to modify a Debian (or Ubuntu server) edition than Kubuntu since Kubuntu is more desktop-oriented. PS: Kubuntu is what I use on my eeePC ;-) -Alan -----Original Message----- From: beowulf-bounces@beowulf.org on behalf of Robert G. Brown Sent: Fri 10/24/2008 2:40 PM To: Jon Aquilina Cc: beowulf@beowulf.org Subject: Re: [Beowulf] Security issues On Fri, 24 Oct 2008, Jon Aquilina wrote: > true but if there is something that isnt in there i would be more then > willing to add it to the repo. But what is the ADVANTAGE of reducing the number of packages in a warehouse from which one pulls packages, when the filled warehouse already exists and is free? I don't know about Debian so much, but with RPM repos one can set up the primary full-distro repo, and create as many "local" repos as one wishes on the side (or link in to e.g. livna -- other repos other humans maintain and that you trust). I can only presume that Debianish distros can do the same. So create a "cluster distro" as an OVERLAY on TOP of an existing distro. That way you do far, far less work. All the distro packages are there if you need them. Most of the cluster packages you might need are already there. If you need to rebuild them, augment them with non-distro packages, or e.g. add some custom kernels, build the replacement/augmentations, package them, pop them in an overlay. Yum will (if told to nicely) use them instead of the ones in the regular distro. Don't forget, of course, that then YOU are responsible for maintaining the update stream of any packages you replace -- if the upstream version is patched, you'd better (re-re-)patch your augmented version. This keeps the amount of work to the theoretical minimum required to achieve your goals, costs you "nothing" (what does disk cost per GB these days -- $0.20 or thereabouts? -- so keeping a full distro costs you a few dollars, literally), and makes it extremely easy to track updates and upgrades without YOU doing a ton of work. rgb > > On Fri, Oct 24, 2008 at 1:49 PM, Carsten Aulbert > wrote: > >> Hi Jon >> >> Jon Aquilina wrote: >>> but why waste time sifting through all 26,000+ pkgs in the repos when u >>> can have a distro with repos focused on clustering pkgs? >> >> Because you might/will save time later when you hit user requests which >> want packages which are not pre-packaged in your cluster distro. >> >> Cheers >> >> Carsten >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> > > > > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081024/c61ae6e9/attachment.html From landman at scalableinformatics.com Fri Oct 24 07:09:29 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: <4901B654.5060003@aei.mpg.de> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> <4901B654.5060003@aei.mpg.de> Message-ID: <4901D719.3060703@scalableinformatics.com> Carsten Aulbert wrote: > Hi Jon > > Jon Aquilina wrote: >> but why waste time sifting through all 26,000+ pkgs in the repos when u >> can have a distro with repos focused on clustering pkgs? > > Because you might/will save time later when you hit user requests which > want packages which are not pre-packaged in your cluster distro. Allow me to expand on this. Some distro packaged stuff is garbage, and broken. Perl in RHEL4 and RHEL5 is notoriously bad (long discussions on this on a few other lists I lurk on). The rational for keeping it bad is compatibility. Which curiously leads to many developers building their own base tools trees. Like the Rocks team does. You can only trust the distro supplied tools so far. Apache2 has greatly improved in RHEL, and Debian/Ubuntu as compared to Apache in RHEL. Php is ancient, as is mysql, postgresql, etc. On the HPC side, tools such as mount are so old you have to pull down the latest sources to get anything close to NFS over RDMA capability even in the mount command. Of course the kernel in RHEL is a (sad) story unto its own. The issue is that any cluster distribution based upon and base distribution inherits all of the underlying issues of the base. And some of those issues are really pretty annoying. In some cases, they are broken. This is why we tend to prefer underlying-OS insensitive systems. As long as the underlying OS works, we don't care what it is. When it doesn't work, this is when we care, and have to figure out if the cost of making it work is worth the effort. The cost is time in this case. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From carsten.aulbert at aei.mpg.de Fri Oct 24 07:09:59 2008 From: carsten.aulbert at aei.mpg.de (Carsten Aulbert) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com><697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl><20081024072224.GA3984@compegg.wr.niftyegg.com><49018D9A.2080303@gmail.com><4901B654.5060003@aei.mpg.de> Message-ID: <4901D737.9040708@aei.mpg.de> Hi Alan, Alan Ward wrote: > My point was that it may be easier to modify a Debian (or Ubuntu server) > edition than Kubuntu since Kubuntu is more desktop-oriented. As a starting point I would always use Ubuntu's server edition when you want to go with Ubuntu. http://www.ubuntu.com/products/whatisubuntu/serveredition For our 'big thing in da basement' we are using Debian Etch and are planning to move to Lenny 'shortly' after the release. Cheers Carsten From hearnsj at googlemail.com Fri Oct 24 07:31:19 2008 From: hearnsj at googlemail.com (John Hearns) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] NFS over RDMA Message-ID: <9f8092cc0810240731l5eb38f18ta4d0040b77e0d9a2@mail.gmail.com> Joe has just mentioned NFS over RDMA, a subject which I have somewhat of a passing interest in. What's the perceived wisdom on it? Biggest, best shiny new thing on the block? Works just fine? Or should it be in the basement inside a locked filing cabinet in a disused lavatory with a sign saying Beware of the Leopard? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081024/b6ed87f6/attachment.html From hahn at mcmaster.ca Fri Oct 24 07:50:54 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> Message-ID: > did this person use the ssh exploit that red hat found a few months ago? are you referring to the event which caused them to change their rpm signing keys? I thought that was not an exploit, but rather that someone's ssh key got exposed one way or other. From tjrc at sanger.ac.uk Fri Oct 24 08:50:09 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: <4901D719.3060703@scalableinformatics.com> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> <4901B654.5060003@aei.mpg.de> <4901D719.3060703@scalableinformatics.com> Message-ID: On 24 Oct 2008, at 3:09 pm, Joe Landman wrote: > Carsten Aulbert wrote: >> Hi Jon >> Jon Aquilina wrote: >>> but why waste time sifting through all 26,000+ pkgs in the repos >>> when u >>> can have a distro with repos focused on clustering pkgs? >> Because you might/will save time later when you hit user requests >> which >> want packages which are not pre-packaged in your cluster distro. > > Allow me to expand on this. > > Some distro packaged stuff is garbage, and broken. Perl in RHEL4 > and RHEL5 is notoriously bad (long discussions on this on a few > other lists I lurk on). The rational for keeping it bad is > compatibility. Which curiously leads to many developers building > their own base tools trees. We do that to an extent, mainly so that machines running different OS's are running a consistent perl environment, for example. But we don't do it because of breakages in the upstream distro. If distros are that broken, we tend to not use them at all. We abandoned pretty much all Red Hat flavours years ago for that reason. For years, large parts of Red Hat were not 64-bit file aware, which was massively infuriating, and as you say, the kernel is in a world of its own (which of course leads to all sorts of fun problems with ISV software, which only supports Red Hat, and then doesn't work on other distros because it's been ported specifically to the Red Hat Broken View of the World) > You can only trust the distro supplied tools so far. Apache2 has > greatly improved in RHEL, and Debian/Ubuntu as compared to Apache in > RHEL. Php is ancient, as is mysql, postgresql, etc. That's always going to happen with any distribution. Ubuntu is, thanks to its 6-month release cycle, usually rather more current than Debian. But it's a trivial matter, usually, if you want something more up to date, to grab the source package from the distro's development tree, and build it on the current stable release. Indeed, there are public repositories (such as etch-backports) where communities are doing just that. But it's easy to do yourself it you want finer control. However, for things like mysql, we tend to do as you describe, and install the versions directly obtained from upstream. > The issue is that any cluster distribution based upon and base > distribution inherits all of the underlying issues of the base. And > some of those issues are really pretty annoying. In some cases, > they are broken. I can't think of any real show-stoppers in the five or so years we've been running Debian. The closest we came to a major snafu there was when Debian made their cock-up with SSH key security. But that was easy enough to put right, and fortunately we hadn't migrated to Etch wholesale when it came to light, so we weren't badly affected. > This is why we tend to prefer underlying-OS insensitive systems. As > long as the underlying OS works, we don't care what it is. When it > doesn't work, this is when we care, and have to figure out if the > cost of making it work is worth the effort. The cost is time in > this case. I agree wholeheartedly with that - time is the most important cost. I also try not to care too much what the underlying OS is, but I also want to minimise the amount of software stack maintenance I have to do, so I tend to ask myself the following questions of the piece of software I'm considering: 1) Does it need to be installed on every machine? 2) Is the precise version present on the machine important? 3) Is the software being rapidly developed, and consequently likely to be out of date in distros? 4) Do I have an ISV support matrix to consider? If the answer is yes to questions 1 and 4, or no to questions 2 and 3, then I tend to lean towards using the distro's packaging. If the answers are the opposite to those, I will tend to use a copy I build and maintain myself, preferably on a central NFS server so I don't have to synchronise it everywhere. There's no hard-and-fast answer to which approach is always best; it's very dependent on the situation. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From rgb at phy.duke.edu Fri Oct 24 08:46:35 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: <4901CACD.5030706@gmail.com> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> <4901B654.5060003@aei.mpg.de> <4901CACD.5030706@gmail.com> Message-ID: On Fri, 24 Oct 2008, Geoff Jacobs wrote: > Or package the source packages and submit them upstream. Volunteer for a > life of servitude! Well, I was thinking more of site-specific custom cuts a la this sort of thing: > Usually when I build a cluster, I make local builds of MPICH2 for each > compiler. This does not fit well with the paradigm of really any distro > I've ever seen, which is why I leave it as a custom layer on top of e.g. > Debian and do not package it. (which you wouldn't, probably, want to submit back upstream:-) but yes... > I have yet to see a distro do multiarch really well, so for the moment I > try to work around (or perhaps above) the system and avoid using APT/YUM > for handling multiple architectures/compiler toolchains. Multiarch isn't that bad -- it requires maintaining twinned repos for the different archs, and I'm sure both rpm and fedora distros do this pretty much transparently for i386 and x86_64 and less so (for no terribly good reason but fewer users) for any of the other archs out there. But multiple compilers -- wow. Never really thought of that one. Maybe I should ask an actual question on the yum list (which, after all, I endure listening to on a daily basis) and see if there are any suggestions for compiler management. What do you do, install particular compilers per system and then need packages to match, or install all the per-compiler packages on all systems and select the one you link to some other way? rgb > >> >> rgb >> > > > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From landman at scalableinformatics.com Fri Oct 24 10:11:22 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] NFS over RDMA In-Reply-To: <9f8092cc0810240731l5eb38f18ta4d0040b77e0d9a2@mail.gmail.com> References: <9f8092cc0810240731l5eb38f18ta4d0040b77e0d9a2@mail.gmail.com> Message-ID: <490201BA.4050103@scalableinformatics.com> John Hearns wrote: > Joe has just mentioned NFS over RDMA, a subject which I have somewhat of > a passing interest in. > What's the perceived wisdom on it? Biggest, best shiny new thing on the > block? Works just fine? > Or should it be in the basement inside a locked filing cabinet in a > disused lavatory with a sign saying Beware of the Leopard? FWIW: we see ~460 MB/s over SDR pairs, going disk to disk (that is real disk on JackRabbit behind the RDMA as opposed to ram disk or null devices which give obscenely fast, and practically unattainable numbers in real world scenarios). We have wedged one machine running it, requiring a LART (reset button press) on the client side. On the server side, it still is finicky but this is mostly on initial mount. Requires mount-utils of 1.1.1 or higher (1.1.2 or higher is recommended). Requires kernels of 2.6.25 or higher on both sides due to in kernel support. Not sure if the OFED stack does some sort of magic here for people. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From niftyompi at niftyegg.com Fri Oct 24 12:47:01 2008 From: niftyompi at niftyegg.com (Nifty niftyompi Mitch) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] NFS over RDMA In-Reply-To: <9f8092cc0810240731l5eb38f18ta4d0040b77e0d9a2@mail.gmail.com> References: <9f8092cc0810240731l5eb38f18ta4d0040b77e0d9a2@mail.gmail.com> Message-ID: <20081024194701.GA13090@compegg.wr.niftyegg.com> On Fri, Oct 24, 2008 at 03:31:19PM +0100, John Hearns wrote: > > Joe has just mentioned NFS over RDMA, a subject which I have somewhat > of a passing interest in. > > What's the perceived wisdom on it? Biggest, best shiny new thing on the > block? Works just fine? There are some hardware cards out there, both Infiniband and 10Ge, that are designed with RDMA as an optimized way to move data. If you have these RDMA optimized cards the new shiny NFS over RDMA may be for you. Others will have to comment on the stability experiences they are having. You will have to match their reports to your distro and decide on the next step. Some source tree searching and web searching will let you discover how new it is and how much special handling it will take at your site. My tolerance indicator tells me that this is a tad new for NFS but new now is not new tomorrow. Clearly the way to think about it is that it is a low level perhaps vendor specific transport optimization that will help or hinder depending on the site situation. My bias is that vendor specific transport tinkering for fundamental resources like NFS is a risk. I would rather the vendor tune foundation things like TCP/IP and not poke at NFS in ways that bypass TCP/IP. > Or should it be in the basement inside a locked filing cabinet in a > disused lavatory with a sign saying Beware of the Leopard? I doubt that it is that bad ;-). Some good vendors are hacking on it. The hard part is evaluation, testing and tomorrows integration with the next interconnect update. -- T o m M i t c h e l l Found me a new hat, now what? From landman at scalableinformatics.com Fri Oct 24 13:38:39 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] OT: readers choice awards Message-ID: <4902324F.3090804@scalableinformatics.com> Don't know if there is interest here, but HPCwire has their annual readers choice awards. Unfortunately they don't have a category for RGB, but they should. Here is the linky to the blog entry ... http://www.hpcwire.com/blogs/2008_HPCwire_Readers_Choice_Nominations_Are_Open.html Vote early ... no seriously, they close shortly ... -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From rgb at phy.duke.edu Fri Oct 24 13:59:29 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] OT: readers choice awards In-Reply-To: <4902324F.3090804@scalableinformatics.com> References: <4902324F.3090804@scalableinformatics.com> Message-ID: On Fri, 24 Oct 2008, Joe Landman wrote: > Don't know if there is interest here, but HPCwire has their annual readers > choice awards. Unfortunately they don't have a category for RGB, but they > should. Here is the linky to the blog entry ... > http://www.hpcwire.com/blogs/2008_HPCwire_Readers_Choice_Nominations_Are_Open.html Awe, surely I could be "Best HPC software product or technology" (rgb-bot, after all:-). Or maybe "Best HPC cluster solution" -- "Ask rgb..." ;-) More seriously, the missing category isn't for me, it is for this list. They should have a category for "Best HPC education/support service". rgb > > Vote early ... no seriously, they close shortly ... > > > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From gdjacobs at gmail.com Fri Oct 24 16:29:05 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> <4901B654.5060003@aei.mpg.de> <4901CACD.5030706@gmail.com> Message-ID: <49025A41.2030705@gmail.com> Robert G. Brown wrote: > On Fri, 24 Oct 2008, Geoff Jacobs wrote: > >> Or package the source packages and submit them upstream. Volunteer for a >> life of servitude! > > Well, I was thinking more of site-specific custom cuts a la this sort > of thing: > >> Usually when I build a cluster, I make local builds of MPICH2 for each >> compiler. This does not fit well with the paradigm of really any distro >> I've ever seen, which is why I leave it as a custom layer on top of e.g. >> Debian and do not package it. > > (which you wouldn't, probably, want to submit back upstream:-) but yes... The layer isn't so much packaging as organization. >> I have yet to see a distro do multiarch really well, so for the moment I >> try to work around (or perhaps above) the system and avoid using APT/YUM >> for handling multiple architectures/compiler toolchains. > > Multiarch isn't that bad -- it requires maintaining twinned repos for > the different archs, and I'm sure both rpm and fedora distros do this > pretty much transparently for i386 and x86_64 and less so (for no > terribly good reason but fewer users) for any of the other archs out > there. But multiple compilers -- wow. Never really thought of that > one. If it were just that, yeah, I could work with different chroots. Unfortunately, the problem is not quite so simple. As I said, different compilers and different build dependencies. > Maybe I should ask an actual question on the yum list (which, after all, > I endure listening to on a daily basis) and see if there are any > suggestions for compiler management. What do you do, install particular > compilers per system and then need packages to match, or install all the > per-compiler packages on all systems and select the one you link to some > other way? I've worked with a trinity of GNU/PGI/Intel. Portland is the most notorious offender in terms of binary incompatibility, so I just make builds for each and shell scripting to allow each user to switch wrappers. It's really very simple, and I haven't found a need to change the method in a few years. The compilers themselves stay more-or-less static, so the builds only need updating if I change the MPI layer. I need to do some more work with OpenMPI to get a real feel for their layout, but from the indications OTW at FSU, for example, OpenMPI can be handled the same. > > rgb -- Geoffrey D. Jacobs From matt at technoronin.com Fri Oct 24 21:01:28 2008 From: matt at technoronin.com (Matt Lawrence) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com><697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl><20081024072224.GA3984@compegg.wr.niftyegg.com><49018D9A.2080303@gmail.com><4901B654.5060003@aei.mpg.de> Message-ID: On Fri, 24 Oct 2008, Alan Ward wrote: > Perhaps an alternative way to go for a cluster install disk would be a > bog standard Debian/Ubuntu/Kubuntu/Xubuntu boot iso image, with a > customized install script that pulls in a suplementary set of packages > over the network and does some extra configuration (NFS server, PXE > server etc). Close. I recommend a very short and simple script that installs a configuration management system such as Puppet, cfengine or bcfg2 and then those do all the other work. Once the infrastructure is set up, it is very easy to do with CentOS and kickstart. > PS: Kubuntu is what I use on my eeePC ;-) I had problems with it so I installed xubuntu and then added the KDE packages. Not bad, but I would prefer to run CentOS since all of the other systems I admin run it. And, no, I am not trying to fan a distro flamewar, I'm just giving my reason for preferring it over any other. -- Matt It's not what I know that counts. It's what I can remember in time to use. From matt at technoronin.com Fri Oct 24 21:29:00 2008 From: matt at technoronin.com (Matt Lawrence) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] OT: readers choice awards In-Reply-To: References: <4902324F.3090804@scalableinformatics.com> Message-ID: On Fri, 24 Oct 2008, Robert G. Brown wrote: > Awe, surely I could be "Best HPC software product or technology" > (rgb-bot, after all:-). Or maybe "Best HPC cluster solution" -- "Ask > rgb..." ;-) Maybe, but I would still rather vote for cymk instead.. -- Matt It's not what I know that counts. It's what I can remember in time to use. From tjrc at sanger.ac.uk Fri Oct 24 23:06:29 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com><697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl><20081024072224.GA3984@compegg.wr.niftyegg.com><49018D9A.2080303@gmail.com><4901B654.5060003@aei.mpg.de> Message-ID: <00CC7E62-779A-4FBD-A63F-4883CC860A74@sanger.ac.uk> On 25 Oct 2008, at 5:01 am, Matt Lawrence wrote: > On Fri, 24 Oct 2008, Alan Ward wrote: > >> Perhaps an alternative way to go for a cluster install disk would >> be a bog standard Debian/Ubuntu/Kubuntu/Xubuntu boot iso image, >> with a customized install script that pulls in a suplementary set >> of packages over the network and does some extra configuration (NFS >> server, PXE server etc). > > Close. I recommend a very short and simple script that installs a > configuration management system such as Puppet, cfengine or bcfg2 > and then those do all the other work. Once the infrastructure is > set up, it is very easy to do with CentOS and kickstart. Yep, that's what we do. We actually use the same Debian FAI installation infrastructure for everything; compute nodes, head nodes, desktops, database servers (except Oracle), you name it. cfengine takes care of tweaking the configuration and installed packages to the actual purpose of the machine. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From eagles051387 at gmail.com Sat Oct 25 00:41:25 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Re: Active directory with Linux In-Reply-To: <87r6666tav.fsf@liv.ac.uk> References: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> <4900BF75.5060201@ias.edu> <87r6666tav.fsf@liv.ac.uk> Message-ID: sry for repost didnt hit reply to all my question though is what is the best way in the linux world to get windows machines to join a linux domain which is being hosted by bind On Fri, Oct 24, 2008 at 3:01 PM, Dave Love wrote: > Prentice Bisbal writes: > > > The trust is that if you already have and AD installation and the AD > > controllers have Microsoft Services for Unix (MSSFU, or just SFU) 3.5 or > > later, you have everything you need to use your AD servers as Kerberos > > and LDAP masters for your Linux clients. > > You only need that stuff for the NSS databases (passwd, group), not for > Kerberos. [I never managed to get the add-on SFE stuff to install -- > even after recovering from the server being 0wned whilst it was getting > security-patched -- but I guess that's not a general problem.] > > > If you want to go the other way around, have Linux serve as the AD > > controllers, you'll need to use Samba, and I haven't had much success > > with it. > > Samba as an actual AD controller is a Samba 4 thing, which isn't ready > yet, as far as I know -- has that changed recently? The canonical way > to DTRT is to have a master Kerberos server in the POSIX world, which AD > trusts, and populate the POSIX and AD worlds' LDAP separately from one > or more accounts databases. Basically you want to keep AD in its own > world, and in a network subdomain with a sensible DNS arrangement, since > AD wants to control DNS. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081025/d73673b0/attachment.html From nixon at nsc.liu.se Fri Oct 24 10:15:10 2008 From: nixon at nsc.liu.se (Leif Nixon) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> (B. Vincent Diepeveen's message of "Fri\, 24 Oct 2008 05\:39\:17 +0200") References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> Message-ID: "B. Vincent Diepeveen" writes: > Now you post here a big story on how your Rocks got hacked. Do i > conclude it correctly the > problem is that you ran a default Rocks kernel? The basic problem seems to be bad account hygiene. That's a hard problem. Users will forever be borrowing each other's accounts, making it difficult to contain security breaches. -- Leif Nixon - Systems expert ------------------------------------------------------------ National Supercomputer Centre - Linkoping University ------------------------------------------------------------ From eagles051387 at gmail.com Sat Oct 25 02:31:28 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> Message-ID: even if you are using a default kernel how often do u hear of a kernel default or not getting hacked? On Fri, Oct 24, 2008 at 7:15 PM, Leif Nixon wrote: > "B. Vincent Diepeveen" writes: > > > Now you post here a big story on how your Rocks got hacked. Do i > > conclude it correctly the > > problem is that you ran a default Rocks kernel? > > The basic problem seems to be bad account hygiene. > > That's a hard problem. Users will forever be borrowing each other's > accounts, making it difficult to contain security breaches. > > -- > Leif Nixon - Systems expert > ------------------------------------------------------------ > National Supercomputer Centre - Linkoping University > ------------------------------------------------------------ > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081025/f58c6ab3/attachment.html From nixon at nsc.liu.se Sun Oct 26 12:36:38 2008 From: nixon at nsc.liu.se (Leif Nixon) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: (Jon Aquilina's message of "Sat\, 25 Oct 2008 11\:31\:28 +0200") References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> Message-ID: "Jon Aquilina" writes: > even if you are using a default kernel how often do u hear of a kernel > default or not getting hacked? I can't parse that. And please stop top-posting. -- Leif Nixon - Systems expert ------------------------------------------------------------ National Supercomputer Centre - Linkoping University ------------------------------------------------------------ From csamuel at vpac.org Sun Oct 26 22:11:08 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Anyone going to SC08 up for an informal BOF on cluster security? In-Reply-To: <1507578205.1874941225084087057.JavaMail.root@mail.vpac.org> Message-ID: <1785098243.1875011225084268384.JavaMail.root@mail.vpac.org> ----- "Joe Landman" wrote: > Probably too late to get it onto the schedule, but I think it could be > a useful discussion to get troubles and best practices talked about. I used to go to Bill Yurcik's Cluster Security Workshops at CCGrid so I'd be interested in this. I bumped into him at SC'07 and he said was thinking about doing something there rather than CCGrid but his old NCSA address bounces. :-( > Certainly away from areas where people may become unhappy with > content or wording, and we can all have frank and open discussions. > This would be a good thing IMO. Amen brother.. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From prentice at ias.edu Mon Oct 27 06:45:53 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Active directory with Linux In-Reply-To: References: <2120044993.1785371224805118397.JavaMail.root@mail.vpac.org> Message-ID: <4905C611.1090201@ias.edu> Tim Cutts wrote: > That's very similar to what we're doing. We're using Sun Directory > Server, because there's an additional piece of software for that (whose > name escapes me) which can nicely handle data synchronisation between > SDS and AD. > Is that SDS the same one that used to be Netscape Directory Server is now Red Hat Directory Server/Fedora Directory Server? If so, read on. I looked at implementing Fedora Directory Server a few months ago to provide LDAP services to our Linux systems and synchronize passwords with our AD servers. To do this, it must store the user passwords in cleartest in the replication logs, where they are in LDIF format, and clearly labelled as clear-text passwords. Even if you shorten the retention time of the replication logs, there is still another log file which, as far as my experimentation detemined, keep the clear-text passwords around forever. I decided this was completely unsafe and abandoned the project. Not long after (the next day, in fact) Slashdot reported that people had been hack into Redhat/Fedora Directory server. -- Prentice From sebastian.hyde at gmail.com Thu Oct 23 21:05:24 2008 From: sebastian.hyde at gmail.com (Sebastian Hyde) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Not all cases need be metal Message-ID: <377dc70d0810232105m19060820g8084c84968ad5bff@mail.gmail.com> http://www.instructables.com/id/Lexan-Computer-Case/ -- Sebastian A. Hyde http://tinyurl.com/sahyde From kilian.cavalotti.work at gmail.com Fri Oct 24 01:55:54 2008 From: kilian.cavalotti.work at gmail.com (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:56 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> Message-ID: <49018D9A.2080303@gmail.com> Jon Aquilina wrote: > did this person use the ssh exploit that red hat found a few months ago? Apparently not. From what Joe wrote, "the entry point was via a shared user account". This account has been compromised, either with brute-force ssh login attempts, or was socially engineered, it's not clear. Nothing seems to indicate (as far as I can tell) that the entry point was due to some weakness in one of the Rocks components. I second Mitch in saying that this break-in isn't Rocks specific, but rather the result of poor (lack of?) administration practices (especially from what I could read here: http://scalability.org/?p=905, and assuming it's about the same customer). On the other hand, it's true that Rocks' philosophy (which I'm not a big proponent of) doesn't make updates easy, nor encourage keeping systems up-to-date. It tends to focus on the Windowsian "reinstall the whole machine" approach in case of problem. Which makes perfect sense in specific contexts, where no dedicated administration resources are available, or where compute time is critical and understanding the root cause of technical problems not so important. But this can also lead to the kind of security problem Joe described, even if here, I don't think one can blame any of the system's component being outdated for this intrusion. Cheers, -- Kilian From malcolm.croucher at gmail.com Fri Oct 24 23:56:30 2008 From: malcolm.croucher at gmail.com (malcolm croucher) Date: Wed Nov 25 01:07:57 2009 Subject: RS: [Beowulf] Cases for DIY boxen In-Reply-To: References: <0D49B15ACFDF2F46BF90B6E08C90048A0493821300@quadbrsex1.quadrics.com> Message-ID: <386fa5610810242356j182b6a30pd15efc0f4197c0e8@mail.gmail.com> I was watching click online (www.bbcworldnews.com/click ) and they mention in the programme how you can build your own boxes into figures like spice racks , robots , cars ect . From the programme It seems you buy pieces of a box which act like lego and then build your own box . quite nifty. Regards Malcolm On Fri, Oct 24, 2008 at 11:25 AM, Alan Ward wrote: > > Nowadays many exposition halls, restaurants etc. in Europe have stratified > wood beams. These need to be treated with fire retardants by regulation, in > which case they are actually more safe than steel (steel bends and bucles > when warmed). > > You could take a look at whatever products they use to treat them. > > PS: My last DIY box was a TV table on wheels, with a couple of shelves for > video casettes underneath beahind glass doors. I took out the shelves and > put in 4 motherboards, and there I was with a 70x70x50 cm mobile cluster. > > Cheers, > -Alan > > > -----Missatge original----- > De: beowulf-bounces@beowulf.org en nom de Dan.Kidger@quadrics.com > Enviat el: dv. 24/10/2008 10:31 > Per a: peter.st.john@gmail.com; rgb@phy.duke.edu > A/c: Beowulf@beowulf.org > Tema: RE: [Beowulf] Cases for DIY boxen > > > I am not so sure that wood is as flammable as you think. > Hard wood needs sustained heat for a reasonably long period of time to get > going. > > And anyway for a computer system there is no reason why you can't do some > fireproofing - get some borates, silicates or other salts to keep the > organic matter away from the oxygen. Waterglass (Sodium silicate) is cheap > and readily available. > > Although I guess the main negative factor for us is the presence of a high > airflow bringing lots of fresh oxygen. So if you could use oxygen-free > cooling air (!) or otherwise shut the airfow off triggered by a smoke > detector ... > > Daniel > > > > From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] > On Behalf Of Peter St. John > Sent: 23 October 2008 18:33 > To: Robert G. Brown > Cc: beowulf@beowulf.org > Subject: Re: [Beowulf] Cases for DIY boxen > > Robert, > > Yes, the consensus (offline) had seemed to be that humidy, thermal > insulation, etc are not issues; and the only issue would be flammability. > And yeah, I actually watched a capacitor explode under ideal circumstances > (it was shadowed dark behind the box where I was looking, wondering why the > prototype game box was behaving badly); it shot a beautiful little jet of > flame. > > Incidentally, Sebastian Hyde has pictures of a really beautiful black > walnut PC. I think the right word is "baroque". Really beautiful. But yeah > consideration would have to made for the fire issue. > > Peter > On 10/23/08, Robert G. Brown >> > wrote: > On Thu, 23 Oct 2008, Peter St. John wrote: > On the subject of Doug's "A Case for Cases" > http://www.linux-mag.com/id/7164, I had noticed that the Helmer thing > ("bewwulf in an Ikea cabinet") is not > really in a wood cabinet (the steel box can be put inside a cabinet). I'm > assuming it's unreasonable to actually make a wood cabinet? On account of > humidy, or just weight? To me it just sounds easy to build a wooden rack > for > a bunch of ATX motherboards. And it could look nice. Thermal and electrical > insulation would be OK, and humidy controlled with a good paint job on the > interior...? > > What about fire? Anything electrical can in a worst case pop hot/molten > metal before frying and/or blowing a breaker. Capacitors blow up > (literally). A wire is badly soldered and pulls free and grounds out, > spattering white hot metal. > > Inside a metal shell, odds are you won't get a REAL fire as there isn't > much actively flammable around. In a wooden box, carefully dried by six > months of 50C heat... it wouldn't take a lot to get real flames, > especially if the box had e.g. a cooling fan mounted to actively fan a > hot coal into flames. > > rgb > > Peter > > -- > Robert G. Brown Phone(cell): 1-919-280-8443 > Duke University Physics Dept, Box 90305 > Durham, N.C. 27708-0305 > Web: http://www.phy.duke.edu/~rgb < > http://www.phy.duke.edu/%7Ergb> > Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php > > Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -- Malcolm A.B Croucher -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081025/1ea78af3/attachment.html From mm at yuhu.biz Sat Oct 25 20:17:27 2008 From: mm at yuhu.biz (Marian Marinov) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> Message-ID: <200810260517.27494.mm@yuhu.biz> On Friday 24 October 2008 20:15:10 Leif Nixon wrote: > "B. Vincent Diepeveen" writes: > > Now you post here a big story on how your Rocks got hacked. Do i > > conclude it correctly the > > problem is that you ran a default Rocks kernel? > > The basic problem seems to be bad account hygiene. > > That's a hard problem. Users will forever be borrowing each other's > accounts, making it difficult to contain security breaches. But if you build a good infrastructure jailing the users within one directory with access to files that do not affect the underlaing OS you will have better chance of leaving such attacks out of your systems. A scheme like that is when all of your users are chrooted to their home folders with the OS for each user mounted from a read-only image. This way it becomes harder for attackers to penetrate the OS security. Also a good security addition will be adding SELinux, RSBAC or GRSecurity to the kernel and actually using any of these. Regards Marian Marinov Head of System Operations at Siteground.com From atchley at myri.com Mon Oct 27 07:48:10 2008 From: atchley at myri.com (Scott Atchley) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Security issues In-Reply-To: <200810260517.27494.mm@yuhu.biz> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <200810260517.27494.mm@yuhu.biz> Message-ID: <04D65A59-0CB4-42E9-8568-1F354DE8F349@myri.com> On Oct 25, 2008, at 11:17 PM, Marian Marinov wrote: > Also a good security addition will be adding SELinux, RSBAC or > GRSecurity to > the kernel and actually using any of these. Bear in mind, that there may be performance trade-offs. Enabling SELinux will cut 2 Gb/s off a 10 Gb/s link as measured by netperf's TCP_STREAM. I am not saying don't use SELinux, but simply be aware of what impact it will have. Scott From landman at scalableinformatics.com Mon Oct 27 07:52:40 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Security issues In-Reply-To: <200810260517.27494.mm@yuhu.biz> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <200810260517.27494.mm@yuhu.biz> Message-ID: <4905D5B8.8050201@scalableinformatics.com> Marian Marinov wrote: > On Friday 24 October 2008 20:15:10 Leif Nixon wrote: >> "B. Vincent Diepeveen" writes: >>> Now you post here a big story on how your Rocks got hacked. Do i >>> conclude it correctly the >>> problem is that you ran a default Rocks kernel? >> The basic problem seems to be bad account hygiene. >> >> That's a hard problem. Users will forever be borrowing each other's >> accounts, making it difficult to contain security breaches. > > But if you build a good infrastructure jailing the users within one directory > with access to files that do not affect the underlaing OS you will have > better chance of leaving such attacks out of your systems. Well, there has been a discussion in the past about using chroot jails for security. My current understanding after following these threads a year or more ago, is that chroot jails are not, in fact, designed with security in mind, and shouldn't be relied upon as a security feature. In fact, there were some chroot tunneling exploits posted a while ago that suggest that chroot for security may be as much security theatre as hard-to-guess-say-speak passwords. > A scheme like that is when all of your users are chrooted to their home > folders with the OS for each user mounted from a read-only image. This way it > becomes harder for attackers to penetrate the OS security. Harder, possibly. Impossible? no. > Also a good security addition will be adding SELinux, RSBAC or GRSecurity to > the kernel and actually using any of these. SElinux has been annoying, even overtly frustrating to use. The things that bother me are there appear to be real things you can do to secure systems (layers), and there is security theatre. Sadly, most people happily talk about security theatre as if it were real security. The best statement I have heard about security is that it is a process, not a feature/function. You can't add more security by adding a product. You can by changing they way something is used. We are in the process of altering how this user uses their cluster. In doing so, we are disabling a number of attack vectors. Does this make their machine more secure? No. It is important to understand, that closing some doors may leave other hidden ones open. So what we try to do is create layers such that in the event we screw up, the damage is contained. > > Regards > Marian Marinov > Head of System Operations at Siteground.com > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From landman at scalableinformatics.com Mon Oct 27 07:59:32 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Security issues In-Reply-To: <49018D9A.2080303@gmail.com> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> Message-ID: <4905D754.7000904@scalableinformatics.com> Kilian CAVALOTTI wrote: > But this can also lead to the kind of security problem Joe described, > even if here, I don't think one can blame any of the system's component > being outdated for this intrusion. It is/was a user issue. We are working to prevent this sort of issue arising again. Sadly, I feel as if we are playing "whack-a-mole" with these issues. No, adding SElinux or other products won't make this any better, they add layers of complexity, and the benefits may not be worth the costs. The issue is, in part, we need to a) prevent sharing of accounts b) control access to ssh logins c) prevent execution of dangerous stuff. "c" is 'easy' (yeah, I know its wrong), but we can disable all suid programs on the machine that are accessible from users accounts. "a" is hard. Academics like to share things. We need to find a way to let them do this. Securely. "b" is interesting. They were using keys for access. Someone loaned their keys to a friend, or their keys were hijacked, or whatever. So we are going to take a different approach. > > Cheers, -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From kspaans at student.math.uwaterloo.ca Mon Oct 27 08:07:05 2008 From: kspaans at student.math.uwaterloo.ca (Kyle Spaans) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Not all cases need be metal In-Reply-To: <377dc70d0810232105m19060820g8084c84968ad5bff@mail.gmail.com> References: <377dc70d0810232105m19060820g8084c84968ad5bff@mail.gmail.com> Message-ID: <20081027150705.GA28275@student.math> On Fri, Oct 24, 2008 at 12:05:24AM -0400, Sebastian Hyde wrote: > http://www.instructables.com/id/Lexan-Computer-Case/ Oh, for sure. "Kids" like me have been putting acrylic windows in our PC cases for years, or even buying/building cases made entirely of acrylic. I'm no handy man though, so I don't know what the difference is between acrylic and lexan. Frankly though, for a personal beowulf, I'd be inclined to rig up some kind of rack that I can screw motherboards straight into, and ignore cases altogether. From eagles051387 at gmail.com Mon Oct 27 08:20:53 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Not all cases need be metal In-Reply-To: <20081027150705.GA28275@student.math> References: <377dc70d0810232105m19060820g8084c84968ad5bff@mail.gmail.com> <20081027150705.GA28275@student.math> Message-ID: i 2nd that On Mon, Oct 27, 2008 at 4:07 PM, Kyle Spaans < kspaans@student.math.uwaterloo.ca> wrote: > On Fri, Oct 24, 2008 at 12:05:24AM -0400, Sebastian Hyde wrote: > > http://www.instructables.com/id/Lexan-Computer-Case/ > > Oh, for sure. "Kids" like me have been putting acrylic windows in our PC > cases for years, or even buying/building cases made entirely of acrylic. > I'm no handy man though, so I don't know what the difference is between > acrylic and lexan. > > Frankly though, for a personal beowulf, I'd be inclined to rig up some kind > of rack that I can screw motherboards straight into, and ignore cases > altogether. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081027/737285e7/attachment.html From Bogdan.Costescu at iwr.uni-heidelberg.de Mon Oct 27 08:23:10 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Security issues In-Reply-To: <200810260517.27494.mm@yuhu.biz> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <200810260517.27494.mm@yuhu.biz> Message-ID: On Sun, 26 Oct 2008, Marian Marinov wrote: >> That's a hard problem. Users will forever be borrowing each other's >> accounts, making it difficult to contain security breaches. > > But if you build a good infrastructure jailing the users Many clusters that I have seen have a very relaxed security policy on the inside network. Having access via a borrowed account to the access node would give a potential attacker the oportunity to use not only 0day exploits but also old-school ones (like those rsh/rexec based) to compromise some nodes or the whole cluster. Jailing users would not help much in this case: they are supposed to be allowed to run whatever software they bring in, so they can also run malicious ones... -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8240, Fax: +49 6221 54 8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From d.love at liverpool.ac.uk Mon Oct 27 09:10:31 2008 From: d.love at liverpool.ac.uk (Dave Love) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Windows client authentication (was: Re: Active directory with Linux) In-Reply-To: (Jon Aquilina's message of "Sat, 25 Oct 2008 09:41:25 +0200") References: <9f8092cc0810230859x4e148b48h70e3aee71603c985@mail.gmail.com> <4900BF75.5060201@ias.edu> <87r6666tav.fsf@liv.ac.uk> Message-ID: <87iqre3too.fsf_-_@liv.ac.uk> "Jon Aquilina" writes: > my question though is what is the best way in the linux world to get windows > machines to join a linux domain which is being hosted by bind I don't understand the question, but it sounds off-topic unless you have a heterogeneous cluster. As I understand it, `joining a domain' is basically sharing an authentication token -- a Kerberos key in the case of AD. (It probably also involves ceding control of the client system to the `domain controller', ? la what Centrify & al will do if you're not careful.) The `domain' in the AD case is basically a Kerberos realm. Realms aren't intrinsically related to DNS, though typically a site's realm is named after its domain; it's just that AD unfortunately conflates them, amongst other things. If you have the misfortune to have nodes running MS Windows and want them to authenticate to a normal Kerberos realm, see e.g. , though I've not done that in a cluster. For ultimate control on clients, you can use the PAM-like system (in MS Windows XP, at least) called GINA. From Bogdan.Costescu at iwr.uni-heidelberg.de Mon Oct 27 09:19:26 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Security issues In-Reply-To: <4905D754.7000904@scalableinformatics.com> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <20081024072224.GA3984@compegg.wr.niftyegg.com> <49018D9A.2080303@gmail.com> <4905D754.7000904@scalableinformatics.com> Message-ID: On Mon, 27 Oct 2008, Joe Landman wrote: > Sadly, I feel as if we are playing "whack-a-mole" with these issues. Well, that's a long time feeling of mine, having to face the same issues independent of the size of the group of scientists that have to use the computer setup (as this is not cluster specific). > "c" is 'easy' (yeah, I know its wrong), but we can disable all suid > programs on the machine that are accessible from users accounts. And if something doesn't need suid to be dangerous ? > "a" is hard. Academics like to share things. We need to find a way > to let them do this. Securely. By all means, let me know if you find something that works - I've been trying for close to 10 years :-) > So we are going to take a different approach. Can you give more details please ? -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8240, Fax: +49 6221 54 8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From rgb at phy.duke.edu Mon Oct 27 12:33:27 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Security issues In-Reply-To: <4905D5B8.8050201@scalableinformatics.com> References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <200810260517.27494.mm@yuhu.biz> <4905D5B8.8050201@scalableinformatics.com> Message-ID: On Mon, 27 Oct 2008, Joe Landman wrote: >> But if you build a good infrastructure jailing the users within one >> directory with access to files that do not affect the underlaing OS you >> will have better chance of leaving such attacks out of your systems. > > Well, there has been a discussion in the past about using chroot jails for > security. My current understanding after following these threads a year or Also to be remembered is that in most situations where one user exploits another's account or takes another's data INSIDE an organization, the best security tool is known as a "sucker rod". Or a hammer. Applied, none too gently, to a user's fingers or the side of their head. Or (if you are in one of this silly environments that frowns on actually causing physical pain to users:-) you can use the "throwing them off of the system, permanently, so hard that they bounce" which can have highly deleterious effects on their ability to e.g. finish a dissertation. Or in a corporate environment, one can "fire them and prosecute". There is therefore something of a difference between security at the oh-so-hard outer layer, where one must repel the great unknown masses intent on penetrating to the soft and chewy interior and with whom you have no reasonable possibility of discovering them and applying sanctions, and the security at the soft and chewy interior. Most clusters have damn-all by way of real security on the inside. Security costs cycles, and cycles are precious. A sucker rod (or any of the less violent but still painful "internal" sanctions) combined with vigilance and a clearly expressed AUA can usually provide adequate internal security in an environment where your office is down the hall from the user's office, where the department chair and policy are on your side, where you have clear ways to identify and punish misbehaving individuals. Once somebody sitting in some internet cafe in Germany and working through three intermediary breakouts has made it through your hard outer layer to userspace therein, then no matter how hard you try to protect your systems you are in some trouble -- at least THAT user's account is compromised from the beginning. Preventing promotion and further compromise in the time window before vigilance detects the intrusion (one hopes), then starts costing something. For some clusters, the data is sufficiently precious that it is worth spending cycles and money to protect it. For others, it is not. Either way, the systems management staff is the "real" security system for any LAN or cluster -- the tools they implement and their vigilance is the fundamental protection of the users, their precious data (if any), and the resource itself. A good systems manager is just a tiny notch this side of being a raving paranoid. Perhaps a "muttering" paranoid. They only rave if they catch you being bad, often carrying a sucker rod...:-) rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From csamuel at vpac.org Mon Oct 27 13:26:34 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Re: Active directory with Linux In-Reply-To: <87vdvi6tx0.fsf@liv.ac.uk> Message-ID: <269510977.1901111225139194459.JavaMail.root@mail.vpac.org> ----- "Dave Love" wrote: > I don't understand that. If you need LDAP data, as opposed to just > Kerberos authentication, and you're not allowed anonymous access to > it, Well we were told that AD doesn't permit anonymous access. Bear in mind we're Linux geeks here, not Windows geeks.. ;-) > you either use a `well-known' password on a special account (which > you're probably also not allowed...) Yup, that's what I described as not being permitted. > or the `machine' account. The latter is what you get from > `joining the domain' (e.g. with Samba) Whilst I couldn't be certain I suspect their security policy would have classed that as just being an implementation of the former, and it too would have been locked out after N failed attempts and hence locked out all users. We got the impression that AD didn't permit them to make an exception to this policy either.. :-( cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From gdjacobs at gmail.com Mon Oct 27 15:10:14 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Not all cases need be metal In-Reply-To: References: <377dc70d0810232105m19060820g8084c84968ad5bff@mail.gmail.com> <20081027150705.GA28275@student.math> Message-ID: <49063C46.2030800@gmail.com> Jon Aquilina wrote: > i 2nd that > > On Mon, Oct 27, 2008 at 4:07 PM, Kyle Spaans > > wrote: > > On Fri, Oct 24, 2008 at 12:05:24AM -0400, Sebastian Hyde wrote: > > http://www.instructables.com/id/Lexan-Computer-Case/ > > Oh, for sure. "Kids" like me have been putting acrylic windows in our PC > cases for years, or even buying/building cases made entirely of acrylic. > I'm no handy man though, so I don't know what the difference is between > acrylic and lexan. > > Frankly though, for a personal beowulf, I'd be inclined to rig up > some kind > of rack that I can screw motherboards straight into, and ignore cases > altogether. This has been covered before. Use a bakery rack with some appropriately sized aluminum plate which has been given a judicious ridge on either side with the brake. Adhere the motherboard and other tackle using double sided foamy tape. 3m has it up to 1/4" thick. http://www.aaacommercialproducts.com/bakeryracks.html http://solutions.3m.com/wps/portal/3M/en_US/Manufacturing/Industry/Product-Catalog/Online-Catalog?PC_7_RJH9U5230GE3E02LECFTDQGLE0_root=GST1T4S9TCgv&PC_7_RJH9U5230GE3E02LECFTDQGLE0_output=html&PC_7_RJH9U5230GE3E02LECFTDQGLE0_gvel=JNXTCSZ9FZgl&PC_7_RJH9U5230GE3E02LECFTDQGLE0_vroot=81LC4NNTD4ge&PC_7_RJH9U5230GE3E02LECFTDQGLE0_node=V2XXLB23V6be&PC_7_RJH9U5230GE3E02LECFTDQGLE0_theme=en_us_manufacturingindustry_portal&PC_7_RJH9U5230GE3E02LECFTDQGLE0_command=CustomizePageHandler -- Geoffrey D. Jacobs From mathog at caltech.edu Mon Oct 27 16:32:14 2008 From: mathog at caltech.edu (David Mathog) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Re: Not all cases need be metal Message-ID: I missed most of this thread, so somebody may have already pointed this out, but cases are metal not just for fire safety reasons. The case is usually made of a good conductor so that it will also shield the radio emissions from the electronics housed inside. Failure to do so could in theory bring the wrath of the FCC down on you (in the U.S.), or at least make you really unpopular with your neighbors. I wonder if the new digital TV tuners are more or less sensitive to this sort of thing than were the old analog tuners? I'm thinking they are likely to be more sensitive, since weak channels we could pick up with varying amounts of "snow" and multipath using an analog tuner tend to be a big nothing with a digital tuner. David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From award at uda.ad Tue Oct 28 02:26:39 2008 From: award at uda.ad (Alan Ward) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Re: Not all cases need be metal References: Message-ID: You are right, of course. This also explains why all laptops have metal lining behind their funky (or not) plastic cases. In my experience of running with cases open, no case at all, or even using a cardboard boot box as a case (yep, a student brought that in once :D) , the worst interference is with loudspeakers. Nothing much is detected with TV sets, radios or cell phones. I guess it's just that loudspeaker wires are so much longer than other equipment. On the other hand, metal cases that are not appropriately isolated from the inboard electronics may act as emitting antennae, actually increasing interference with other equipment. Seen that mainly with some low-end tower boxes. Cheers, -Alan -----Original Message----- From: beowulf-bounces@beowulf.org on behalf of David Mathog Sent: Tue 10/28/2008 12:32 AM To: beowulf@beowulf.org Subject: [Beowulf] Re: Not all cases need be metal I missed most of this thread, so somebody may have already pointed this out, but cases are metal not just for fire safety reasons. The case is usually made of a good conductor so that it will also shield the radio emissions from the electronics housed inside. Failure to do so could in theory bring the wrath of the FCC down on you (in the U.S.), or at least make you really unpopular with your neighbors. I wonder if the new digital TV tuners are more or less sensitive to this sort of thing than were the old analog tuners? I'm thinking they are likely to be more sensitive, since weak channels we could pick up with varying amounts of "snow" and multipath using an analog tuner tend to be a big nothing with a digital tuner. David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20081028/c8bb257f/attachment.html From kilian.cavalotti.work at gmail.com Tue Oct 28 01:52:18 2008 From: kilian.cavalotti.work at gmail.com (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Security issues In-Reply-To: References: <490104B6.5040401@scalableinformatics.com> <697CD0D4-5DC8-4639-8ECF-950B2554A91C@xs4all.nl> <200810260517.27494.mm@yuhu.biz> <4905D5B8.8050201@scalableinformatics.com> Message-ID: <4906D2C2.2030308@gmail.com> Robert G. Brown wrote: > Also to be remembered is that in most situations where one user exploits > another's account or takes another's data INSIDE an organization, the > best security tool is known as a "sucker rod". Or a hammer. Applied, > none too gently, to a user's fingers or the side of their head. > > Or (if you are in one of this silly environments that frowns on actually > causing physical pain to users:-) you can use the "throwing them off of > the system, permanently, so hard that they bounce" which can have highly > deleterious effects on their ability to e.g. finish a dissertation. Or > in a corporate environment, one can "fire them and prosecute". I definitely agree with that. Technical solutions must be adopted in case of technical problems, but technical solutions can't solve non-technical issues. Users sharing an account is not a technical issue, that's a social behavior, which has to be addressed via legal/political/educational measures. > Security costs cycles, and cycles are precious. It's also that security, from the academic users' standpoint, is a useless burden, which sits in their way most of times, and prevent them to do their work. At least that's often their perception. That's why education is so crucial, and helping them understand that the damn sysadmin who puts security checks everywhere is actually working on their side, to keep them safe, to improve their compute tools' uptime, and to prevent that the results of their computations get published before they even have a chance to retrieve their files. And usually, they care about that last one. > security in an environment where your office is down the hall from the > user's office, where the department chair and policy are on your side, > where you have clear ways to identify and punish misbehaving > individuals. Well, yes. But it may also happen that even though the department chair is on your side, punishing certain misbehaving individuals is not that easy... > Once somebody sitting in some internet cafe in Germany Eh! What about Germany? :) > A good systems manager is just a tiny notch this side of being a raving > paranoid. Perhaps a "muttering" paranoid. They only rave if they catch > you being bad, often carrying a sucker rod...:-) That's an issue too. The vigilance windows may be as wide as human resources allow, there will still be periods of time where nobody watches and were malicious users could do their bad things without being noticed. That's where technical measures can help, by limiting the damages a user can do, by restricting the scope of their actions (without preventing them to work either, remember that the main purpose of an HPC system is to produce computations results), and by notifying the sysadmin in case something fishy happens. Anyway, in the case we're talking about, technical solutions would definitely help containing the fire, but to prevent it being lit, there has to be some political will from the user side. Cheers, -- Kilian From niftyompi at niftyegg.com Wed Oct 29 22:21:48 2008 From: niftyompi at niftyegg.com (Nifty niftyompi Mitch) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Not all cases need be metal In-Reply-To: <377dc70d0810232105m19060820g8084c84968ad5bff@mail.gmail.com> References: <377dc70d0810232105m19060820g8084c84968ad5bff@mail.gmail.com> Message-ID: <20081030052148.GA3307@compegg.wr.niftyegg.com> On Fri, Oct 24, 2008 at 12:05:24AM -0400, Sebastian Hyde wrote: > > http://www.instructables.com/id/Lexan-Computer-Case/ > It should be noted that a safety certifications for computer systems pay attention to the vertical flame rating of all the material inside the chassis and the chassis itself. i.e. the ability of the material to sustain a fire. Another agency certification involves radio frequency emissions. In general the radio frequency emissions (in the US FCC part 15) all but forces computers to live in a Faraday cage (metal box). The radio frequency interference issues with modern hardware have moved way up frequency with today's high speed clocks. The common issues are bad cell phone reception, bad WiFI and other stuff. If you sell a product you must measure and comply. Big iron, like the early Crays, often did on site RF measurements and testing at the customers site. The Fire hazard risk should NOT be ignored! Lexan can be made with fire retardants in it to improve its good properties in this regard. But I doubt it is equal to a metal chassis. If you inspect the motherboard you will find an agency symbol that reflects the fire rating of the PCB material. If you sell a product and want to be insured you need to pay attention. In some cities and countries a safety evaluation is not optional! Power level at the wall changes the rules. The more power the more will be required for UL and VDE compliance. To some degree an individual has some wiggle room that a company does not have. However if you are building a "supercomputer" in space you do not own "you will comply" ;-). If I recall the early Google racks were "open frame" designs. It is possible to do some interesting things 'out of the box', as it were. -- T o m M i t c h e l l Found me a new hat, now what? From csamuel at vpac.org Thu Oct 30 20:59:45 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Beowulf bash at SC'08 ? Message-ID: <693809089.2004061225425585868.JavaMail.root@mail.vpac.org> Hi folks, Just a couple of weeks to go and I was wondering if there were plans to do another Beowulf bash at SC this year in Austin ? cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From deadline at eadline.org Fri Oct 31 05:27:19 2008 From: deadline at eadline.org (Douglas Eadline) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Beowulf bash at SC'08 ? In-Reply-To: <693809089.2004061225425585868.JavaMail.root@mail.vpac.org> References: <693809089.2004061225425585868.JavaMail.root@mail.vpac.org> Message-ID: <51947.192.168.1.213.1225456039.squirrel@mail.eadline.org> Short Answer: Yes Medium Answer: Official Announcement coming soon. Longer Answer: Monday, November 17, 9-11:30pm Pete?s Dueling Piano Bar http://www.petesduelingpianobar.com Just a few blocks from the convention center. This is a combined Beowulf Bash/LECCIBG* If you are a vendor, there are still some sponsorship opportunities available. Contact me off the list. -- Doug * LECCIBG http://www.clustermonkey.net//content/view/164/57/ > Hi folks, > > Just a couple of weeks to go and I was wondering if > there were plans to do another Beowulf bash at SC this > year in Austin ? > > cheers, > Chris > -- > Christopher Samuel - (03) 9925 4751 - Systems Manager > The Victorian Partnership for Advanced Computing > P.O. Box 201, Carlton South, VIC 3053, Australia > VPAC is a not-for-profit Registered Research Agency > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Doug From kspaans at student.math.uwaterloo.ca Fri Oct 31 06:00:59 2008 From: kspaans at student.math.uwaterloo.ca (Kyle Spaans) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Beowulf Bash at OSDI'08 ? Message-ID: <20081031130059.GB24378@student.math> I've just received a USENIX student grant to go to OSDI'08, will any other Beowulfers be there? (sorry for the blatant subject-line ripoff ;-) From eugen at leitl.org Fri Oct 31 06:11:00 2008 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] accelerating physics with CUDA/Stream Message-ID: <20081031131100.GB11968@leitl.org> I figured this would be the right place to ask. Has anyone here used CUDA or AMD Stream on toy physics engines, like ODE? I know parallelization has been tackled http://www.cs.cmu.edu/~mpa/ode/final_report.html but I can't find anything about GPGPU approaches. In general, did you find AMD or nVidia architecture a better match? What is the system size one can expect to fit in 1 GByte card memory, without swapping to and from main memory? Thanks, -- Eugen* Leitl leitl http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE From becker at scyld.com Fri Oct 31 09:40:14 2008 From: becker at scyld.com (Donald Becker) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] Beowulf bash at SC'08 ? In-Reply-To: <693809089.2004061225425585868.JavaMail.root@mail.vpac.org> Message-ID: On Fri, 31 Oct 2008, Chris Samuel wrote: > Just a couple of weeks to go and I was wondering if > there were plans to do another Beowulf bash at SC this > year in Austin ? Ooopsss, missing sending out the preliminary announcement to the list. Look for it in my next message. -- Donald Becker becker@scyld.com Penguin Computing / Scyld Software www.penguincomputing.com www.scyld.com Annapolis MD and San Francisco CA From becker at scyld.com Fri Oct 31 09:45:40 2008 From: becker at scyld.com (Donald Becker) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] 10th Annual Beowulf Bash: Announcement and sponsorship opportunity In-Reply-To: <48D92E1F.7000801@hope.edu> Message-ID: Subject: 10th Annual Beowulf Bash: Announcement and sponsorship opportunity Tenth Annual Beowulf Bash And LECCIBG November 17 2008 9pm at Pete's Dueling Piano Bar We are finalizing the plans for this year's combined Beowulf Bash and LECCIBG It will take place, as usual, with the IEEE SC Conference. This year SC08 is in Austin during the week of Nov 17 2008 As in previous years, the attraction is the conversations with other attendees. We will have drinks and light snacks, with a short greeting by the sponsors about 10:15pm. The venue is in the lively area of Austin near 6th street, very close to many of the conference hotels and within walking distance of the rest. November 17 2008 9-11:30pm Monday, Immediately after the SC08 Opening Gala Pete's Dueling Piano Bar http://www.petesduelingpianobar.com If your company (or even you as an individual) would like to help sponsor the event, please contact me, becker@beowulf.org before early November. (We can accommodate last-minute sponsorship, but your name won't be on the printed info.) Our "headlining" sponsor list for 2008 is currently: Penguin/Scyld (organizing sponsor) http://penguincomputing.com AMD http://amd.com NVIDIA http://nvidia.com Sponsors - get their name up in lights (Well, if their sign is lighted. Bring a sign, and we'll do our best to make certain the room is not too dark.) - are part of the brief greeting in the middle of the party. - have the opportunity for technical, hands-on demos at the Bash - will have their logos on the beowulf.org BeoBash 2008 web pages and on the 2008 yearbook page -- Donald Becker Never send mail to beowulf-bait@boewulf.org Penguin Computing / Scyld Software www.penguincomputing.com www.scyld.com Annapolis MD and San Francisco CA From wrankin at duke.edu Fri Oct 31 13:00:25 2008 From: wrankin at duke.edu (Bill Rankin) Date: Wed Nov 25 01:07:57 2009 Subject: [Beowulf] 10th Annual Beowulf Bash: Announcement and sponsorship opportunity In-Reply-To: References: Message-ID: <1225483225.4930.45.camel@yuengling.rankin.org> Very cool. Thanks for the update, Donald! After somewhat of a hiatus, I'm really looking forward to seeing everyone and catching up on things. See ya'll there! :-) -bill On Fri, 2008-10-31 at 09:45 -0700, Donald Becker wrote: > Subject: 10th Annual Beowulf Bash: Announcement and sponsorship > opportunity > > > Tenth Annual Beowulf Bash > And > LECCIBG > > November 17 2008 9pm at Pete's Dueling Piano Bar -- Bill Rankin, Ph.D. Altair Engineering, PBS Gridworks wrankin@altair.com