From Michael.Frese at NumerEx-LLC.com Wed Jun 4 15:52:14 2008 From: Michael.Frese at NumerEx-LLC.com (Michael H. Frese) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] OFED/IB for FC8 Message-ID: <6.2.5.6.2.20080604150239.047ad270@NumerEx-LLC.com> Following Jeff Layton's post to this list [Cheap SDR IB] on January 28, we purchased 8 Infinihost LX's and an 8 port switch, and began trying to get the OpenFabrics (OFED) release of MVAPICH for Fedora Core 6 to run on our new machines. We develop and run a multiphysics code in a relatively fine grain parallel mode where latency dominates the performance scaling, so it seemed like a good thing to try. This is our first exposure to InfiniBand, though we have considerable experience with MPI, both in-memory and over GigE, including using netpipe to measure latency and bandwidth. Those machines have AMD Athlon X2 6000+'s on Asus M2N-SLI Deluxe motherboards with an open PCI Express slot that will handle x4. The main issue is that we are presently running Fedora Core 8 and the 2.6.21 SMP kernel, but there is no OFED release for FC8 yet. Is anyone else working on this? Has anyone succeeded at getting it to work? We started with OFED version 1.2.5 from http://www.openfabrics.org/downloads/OFED/ofed-1.2.5/OFED-1.2.5-RPMS/ We downloaded all the rpms from redhat-release-4AS-6.1 version. In particular the kernel rpms are kernel-ib-devel-1.2-2.6.9_55.ELsmp and kernel-ib-1.2-2.6.9_55.ELsmp. We used the 1.2.5 version because there don't seem to be any rpms for the 1.3 version. All the OFED rpm's for FC6 installed on FC8 without difficulty, except for opensm-3.0.3-0.ppc64.rpm It didn't say "missing dependencies ..." It just got stuck. We had to kill the 'rpm -ivh', remove the lock file and rebuild the rpm database. After that, # lsmod | grep ib shows about 15 IB related kernel mods. Even so, at this point, some of the IB stuff works. We can run ibnetdiscover and see the HCA's on the two machines that have the rpm's installed, and the switch, too. We could use that to make a topology file, but we don't know where to put it, or even if we should put it somewhere. We can run ibchecknet, and though it finds 4 nodes, it says they are all bad. It also reports "lid 0 address resolution: FAILED". We have not succeeded in getting ibping to work, and aren't really sure what how to specify the remote address for it. We found /usr/share/doc/ofed-docs-1.2/README.txt /usr/share/doc/ofed-docs-1.2/OFED_Installation_Guide.txt and, as described there, did # /etc/init.d/openibd start Loading QLogic InfiniPath driver: [FAILED] Loading HCA driver and Access Layer: [ OK ] Setting up InfiniBand network interfaces: Failed to configure IPoIB connected mode for ib0 Bringing up interface ib0: [FAILED] Setting up service network . . . [ done ] Loading ib_sdp [FAILED] Loading ib_vnic [FAILED] Module ib_vnic not loaded. Bringing up VNIC interfaces [FAILED] That mostly looks bad. Does anyone have any suggestions? We are willing to try a build from source, but we are unsure of what challenges might lie down that path. We'd rather not fall back to FC6, but we may have to do that. Thanks for your help. Mike Frese -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080604/085e9134/attachment.html From lindahl at pbm.com Wed Jun 4 17:15:49 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] OFED/IB for FC8 In-Reply-To: <6.2.5.6.2.20080604150239.047ad270@NumerEx-LLC.com> References: <6.2.5.6.2.20080604150239.047ad270@NumerEx-LLC.com> Message-ID: <20080605001549.GE27430@bx9.net> > All the OFED rpm's for FC6 installed on FC8 without difficulty, > except for opensm-3.0.3-0.ppc64.rpm This is the cause of most of your subsequent problems. Without an SM running somewhere on your network, the links don't come fully up. There are mailing lists devoted to OFED that you could ask on. Building the software from scratch is probably the most straight-forward way to get something that works. -- greg From john.hearns at streamline-computing.com Wed Jun 4 22:36:44 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] OFED/IB for FC8 In-Reply-To: <6.2.5.6.2.20080604150239.047ad270@NumerEx-LLC.com> References: <6.2.5.6.2.20080604150239.047ad270@NumerEx-LLC.com> Message-ID: <1212644214.7657.6.camel@Vigor13> On Wed, 2008-06-04 at 16:52 -0600, Michael H. Frese wrote: > Does anyone have any suggestions? > > We are willing to try a build from source, but we are unsure of what > challenges might lie down that path. > I agree with Greg. Build it from source - that way you will have the latest version (1.3), you will learn about the software stack whilst doing it and you will know which switches were used during the configuration process. Depending solely on distribution supplied RPMs for any HPC type software is a bad move IMHO - as you've just seen it might not have a feature you need, or might not install exactly the way you want it. And you'll always have the version supplied by the distribution, and won't be able to update if you hit a bug or need a new feature. Remember, we're in the era of open source. That's why you chose to use Linux - you have control. John Hearns From mfatica at gmail.com Wed Jun 4 16:27:40 2008 From: mfatica at gmail.com (Massimiliano Fatica) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] OFED/IB for FC8 In-Reply-To: <6.2.5.6.2.20080604150239.047ad270@NumerEx-LLC.com> References: <6.2.5.6.2.20080604150239.047ad270@NumerEx-LLC.com> Message-ID: <8e6393ac0806041627k1a0f58c0r319a603154d33068@mail.gmail.com> We have the same switch. I was able to get it to work with the latest OFED 1.3 ( available from the Mellanox web site). They have rpms for RHEL4 and RHEL5. Massimiliano On Wed, Jun 4, 2008 at 3:52 PM, Michael H. Frese < Michael.Frese@numerex-llc.com> wrote: > Following Jeff Layton's post to this list [Cheap SDR IB] on January 28, > we purchased 8 Infinihost LX's and an 8 port switch, and began trying to > get > the OpenFabrics (OFED) release of MVAPICH for Fedora Core 6 to run on our > new > machines. We develop and run a multiphysics code in a relatively fine > grain parallel mode > where latency dominates the performance scaling, so it seemed like a good > thing to try. > > This is our first exposure to InfiniBand, though we have considerable > experience with MPI, both in-memory and over GigE, including using netpipe > to > measure latency and bandwidth. > > Those machines have AMD Athlon X2 6000+'s on Asus M2N-SLI Deluxe > motherboards > with an open PCI Express slot that will handle x4. > > The main issue is that we are presently running Fedora Core 8 and the > 2.6.21 > SMP kernel, but there is no OFED release for FC8 yet. Is anyone else > working > on this? Has anyone succeeded at getting it to work? > > We started with OFED version 1.2.5 from > http://www.openfabrics.org/downloads/OFED/ofed-1.2.5/OFED-1.2.5-RPMS/ > We downloaded all the rpms from redhat-release-4AS-6.1 version. > In particular the kernel rpms are kernel-ib-devel-1.2-2.6.9_55.ELsmp and > kernel-ib-1.2-2.6.9_55.ELsmp. > > We used the 1.2.5 version because there don't seem to be any rpms for the > 1.3 version. > > All the OFED rpm's for FC6 installed on FC8 without difficulty, except for > opensm-3.0.3-0.ppc64.rpm > It didn't say "missing dependencies ..." It just got stuck. We had to kill > the 'rpm -ivh', remove the lock file > and rebuild the rpm database. After that, > > # lsmod | grep ib > > shows about 15 IB related kernel mods. > > Even so, at this point, some of the IB stuff works. We can run > ibnetdiscover and see the HCA's on the > two machines that have the rpm's installed, and the switch, too. We could > use > that to make a topology file, but we don't know where to put it, or even if > we > should put it somewhere. We can run ibchecknet, and though it finds 4 > nodes, > it says they are all bad. It also reports "lid 0 address resolution: > FAILED". We have not succeeded in getting ibping to work, and aren't > really > sure what how to specify the remote address for it. > > We found > > /usr/share/doc/ofed-docs-1.2/README.txt > /usr/share/doc/ofed-docs-1.2/OFED_Installation_Guide.txt > > and, as described there, did > > # /etc/init.d/openibd start > Loading QLogic InfiniPath driver: [FAILED] > Loading HCA driver and Access Layer: [ OK ] > Setting up InfiniBand network interfaces: > Failed to configure IPoIB connected mode for ib0 > Bringing up interface ib0: [FAILED] > Setting up service network . . . [ done ] > Loading ib_sdp [FAILED] > Loading ib_vnic [FAILED] > Module ib_vnic not loaded. > Bringing up VNIC interfaces [FAILED] > > That mostly looks bad. > > Does anyone have any suggestions? > > We are willing to try a build from source, but we are unsure of what > challenges might lie down that path. > > We'd rather not fall back to FC6, but we may have to do that. > > Thanks for your help. > > > Mike Frese > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080604/7b66ecc1/attachment.html From rainer at lfbs.RWTH-Aachen.DE Thu Jun 5 02:38:24 2008 From: rainer at lfbs.RWTH-Aachen.DE (Rainer Finocchiaro) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] OFED/IB for FC8 In-Reply-To: <20080605001549.GE27430@bx9.net> References: <6.2.5.6.2.20080604150239.047ad270@NumerEx-LLC.com> <20080605001549.GE27430@bx9.net> Message-ID: <4847B410.1070202@lfbs.rwth-aachen.de> Hi Michael, Greg Lindahl schrieb: >> All the OFED rpm's for FC6 installed on FC8 without difficulty, >> except for opensm-3.0.3-0.ppc64.rpm > > This is the cause of most of your subsequent problems. Without an SM > running somewhere on your network, the links don't come fully up. > > There are mailing lists devoted to OFED that you could ask > on. Building the software from scratch is probably the most > straight-forward way to get something that works. > > -- greg I completely agree with Greg. I will add some comments, as I have just installed the distribution (the hard way: under Debian). Following your link, I reach a download directory offering only ppc64-RPMs; in fact all precompiled RPMs for OFED-1.2.5 are for Power PC and not for x86. You could try OFED-1.2, where all the precompiled RPMs are for x86_64, which should be suitable for your processor, acutally depending on the type of distribution you installed (32bit vs. 64bit). Much better is to download more up-to-date OFED-1.3 sources. The package includes an install script, which builds and installs the RPMs for you. So you don't have to "fear" to install something which is not controlled by your package management system (RPM). Regards, Rainer From kus at free.net Thu Jun 5 08:42:22 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] Barcelona hardware error: how to detect Message-ID: How is possible to detect, that particular AMD Barcelona CPU has - or doesn't have - known hardware error problem ? To be more exact, Rev. B2 of Opteron 2350 - is it for CPU stepping w/error or w/o error ? Mikhail Kuzminsky Computer Assistance to Chemical Research Center Zelinsky Inst. of Organic Chemistry Moscow From hahn at mcmaster.ca Thu Jun 5 08:57:28 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: References: Message-ID: > To be more exact, Rev. B2 of Opteron 2350 - is it for CPU stepping w/error or > w/o error ? AMD, like Intel, does a reasonable job of disclosing such info: http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/41322.PDF the well-known problem is erattum 298, I think, and fixed in B3. From kus at free.net Thu Jun 5 09:39:02 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: Message-ID: In message from Mark Hahn (Thu, 5 Jun 2008 11:57:28 -0400 (EDT)): >> To be more exact, Rev. B2 of Opteron 2350 - is it for CPU stepping >>w/error or >> w/o error ? > >AMD, like Intel, does a reasonable job of disclosing such info: > >http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/41322.PDF > >the well-known problem is erattum 298, I think, and fixed in B3. Yes, this AMD errata document says that in B3 revision the error "will be fixed". I heard that new CPUs w/o TLB+L3 error are shipped now, but are this CPUs really B3 or may be have some more new release ? Mikhail From hahn at mcmaster.ca Thu Jun 5 10:30:57 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: References: Message-ID: >> http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/41322.PDF >> >> the well-known problem is erattum 298, I think, and fixed in B3. > > Yes, this AMD errata document says that in B3 revision the error "will be > fixed". I believe the absence of 'x' in the B3 column of the table on p 15 means that it _is_ fixed in B3. From kus at free.net Thu Jun 5 10:48:32 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: Message-ID: In message from Mark Hahn (Thu, 5 Jun 2008 13:30:57 -0400 (EDT)): >>> http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/41322.PDF >>> >>> the well-known problem is erattum 298, I think, and fixed in B3. >> >> Yes, this AMD errata document says that in B3 revision the error >>"will be >> fixed". > >I believe the absence of 'x' in the B3 column of the table on p 15 >means that it _is_ fixed in B3. I received just now some preliminary data about Gaussian-03 run problems w/B2 and about absence of this problems w/B3. Yours Mikhail From kus at free.net Thu Jun 5 11:09:58 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: Message-ID: In message from Mark Hahn (Thu, 5 Jun 2008 13:55:01 -0400 (EDT)): >>> I believe the absence of 'x' in the B3 column of the table on p 15 >>> means that it _is_ fixed in B3. >> >> I received just now some preliminary data about Gaussian-03 run >>problems w/B2 >> and about absence of this problems w/B3. > >I'm mystified by this: B2 was broken, so using it without the bios >workaround is just a mistake or masochism. the workaround _did_ >apparently have performance implications, but that's why B3 exists... > >do you mean you know of G03 problems on B2 systems which are operating >_with_ the workaround? I don't know exactly, but I think the crash was under absence of workaround, because I was not informed that there was some kernel patches or BIOS changes. This was interesting for me also, because I have no information how this hardware problem may be affected in the "real life". Mikhail From kus at free.net Thu Jun 5 11:22:36 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: <588c11220806051116i37ff7aa1oec16a85a24009592@mail.gmail.com> Message-ID: In message from "Jason Clinton" (Thu, 5 Jun 2008 13:16:33 -0500): >On Thu, Jun 5, 2008 at 1:09 PM, Mikhail Kuzminsky >wrote: > >> In message from Mark Hahn (Thu, 5 Jun 2008 >>13:55:01 >> -0400 (EDT)): >> >>> I'm mystified by this: B2 was broken, so using it without the bios >>> workaround is just a mistake or masochism. the workaround _did_ >>>apparently >>> have performance implications, but that's why B3 exists... >>> >>> do you mean you know of G03 problems on B2 systems which are >>>operating >>> _with_ the workaround? >>> >> >> I don't know exactly, but I think the crash was under absence of >> workaround, because I was not informed that there was some kernel >>patches or >> BIOS changes. This was interesting for me also, because I have no >> information how this hardware problem may be affected in the "real >>life". >> Mikhail >> > >The B2 BIOS work-around is to disable the L3 cache which gives you a >10-20% >performance hit with no reduction in power consumption. > >The kernel patch is very extensive and, last I heard, under NDA. AMD >has >said publicly that the patch gives you a 1-2% performance hit. This URL is old, but may give some information: https://www.x86-64.org/pipermail/discuss/2007-December/010260.html Mikhail From lindahl at pbm.com Thu Jun 5 11:30:20 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: References: Message-ID: <20080605183020.GA11661@bx9.net> On Thu, Jun 05, 2008 at 10:09:58PM +0400, Mikhail Kuzminsky wrote: > This was interesting for me also, because I > have no information how this hardware problem may be affected in the > "real life". I have 4 chips with the bug, in 2 servers. I see about 1 lockup per month with my workload, which doesn't include any VMs. (VMs are reputed to trigger the bug quickly.) I found a webpage with the details, and indeed this is what I see: | The system may experience a machine check event reporting an L3 | protocol error has occurred. In this case, the MC4 status register | (MSR 0000_0410) will be equal to B2000000_000B0C0F or | BA000000_000B0C0F. The MC4 address register (MSR 0000_0412) will be | equal to 26h.' -- greg From hahn at mcmaster.ca Thu Jun 5 11:43:05 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: <588c11220806051116i37ff7aa1oec16a85a24009592@mail.gmail.com> References: <588c11220806051116i37ff7aa1oec16a85a24009592@mail.gmail.com> Message-ID: > The kernel patch is very extensive and, last I heard, under NDA. AMD has the kernel patch was publicly distributed in dec 07. it appears to add some kernel logic to avoid the specific L3 TLB states which don't behave correctly. the bios-level workaround is different, and appears to disable the L3 TLB - I don't know whether that actually disables the L3 itself... while extremely unfortunate - so much that it clearly threatens the viability of the company, I think AMD responded reasonably. From csamuel at vpac.org Fri Jun 6 04:31:45 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: <971758047.108911212751290783.JavaMail.root@zimbra.vpac.org> Message-ID: <937389196.108931212751905042.JavaMail.root@zimbra.vpac.org> ----- "Mark Hahn" wrote: > the kernel patch was publicly distributed in dec 07. > it appears to add some kernel logic to avoid the specific > L3 TLB states which don't behave correctly. the bios-level > workaround is different, and appears to disable the L3 TLB - > I don't know whether that actually disables the L3 itself... I believe the patch re-enables the L3 cache and then works around the problem in software. When we were running B2 Barcelonas with this patch we didn't hit the errata and didn't see the performance penalty we would have expected if the L3 was disabled. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From csamuel at vpac.org Fri Jun 6 04:33:38 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: Message-ID: <741026114.108961212752018697.JavaMail.root@zimbra.vpac.org> ----- "Mikhail Kuzminsky" wrote: > Yes, this AMD errata document says that in B3 revision the error "will > be fixed". I heard that new CPUs w/o TLB+L3 error are shipped now, > but are this CPUs really B3 or may be have some more new release ? They certainly do exist, we've got 94 nodes of them here and no longer require the kernel patch to work around the errata. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From gerry.creager at tamu.edu Fri Jun 6 08:39:47 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments Message-ID: <48495A43.4060809@tamu.edu> We recently purchased a set of hardware for a cluster from a hardware vendor. We've encountered a couple of interesting issues with bringing the thing up that I'd like to get group comments on. Note that the RFP and negotiations specified this system was for a cluster installation, so there would be no misunderstanding... 1. We specified "No OS" in the purchase so that we could install CentOS as our base. We got a set of systems with a stub OS, and an EULA for the diagnostics embedded on the disk. After clicking thru the EULA, it tells us we have no OS on the disk, but does not fail to PXE. 2. BIOS had a couple of interesting defaults, including warn on keyboard error (Keyboard? Not intentionally. This is a compute node, and should never require a keyboard. Ever.) We also find the BIOS is set to boot from hard disk THEN PXE. But due to item 1, above, we never can fail over to PXE unless we load up a keyboard and monitor, and hit F12 to drop to PXE. In discussions with our sales rep, I'm told that we'd have had to pay extra to get a real bare hard disk, and that, for a fee, they'd have been willing to custom-configure the BIOS. OK, with the BIOS this isn't too unreasonable: They have a standard BIOS for all systems and if you want something special, paying for it's the norm... But, still, this is a CLUSTER installation we were quoted, not a desktop. Also, I'm now told that "almost every customer" ordered their cluster configuration service at several kilobucks per rack. Since the team I'm working with has some degree of experience in configuring and installing hardware and software on computational clusters, now measured in at least 10 separate cluster installations, this seemed like an unnecessary expense. However, we're finding vendor gotchas that are annoying at the least, and sometimes cause significant work-around time/effort. Finally, our sales guy yesterday was somewhat baffled as to why we'd ordered without OS, and further why we were using Linux over Windows for HPC. Not trying to revive the recent rant-fest about Windows HPC capabilities, can anyone cite real HPC applications generally run on significant clusters (I'll accept Cornell's work, although I remain personally convinced that the bulk of their Windows HPC work has been dedicated to maintaining grant funding rather than doing real work)? No, I won't identify the vendor. -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.862.3982 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From dag at sonsorol.org Fri Jun 6 09:15:24 2008 From: dag at sonsorol.org (Chris Dagdigian) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <48495A43.4060809@tamu.edu> References: <48495A43.4060809@tamu.edu> Message-ID: <13F1DDCE-2FE6-4916-B1A5-995237EA80B6@sonsorol.org> Bad job hiding the (obvious) vendor ;) I'm riding the bus back home to Boston after a cluster building gig and your experience exactly matches what I encountered when I walked into the datacenter to start work on a pile of dell 1950 servers. I'll do you one better - 4 nodes out of our "homogenous" cluster had reversed drive cabling which broke our imaging system as we had specific data to place on 2 drives of differing capacity. Regards, Chris /* Sent via phone - apologies for typos & terseness */ On Jun 6, 2008, at 11:39 AM, Gerry Creager wrote: > We recently purchased a set of hardware for a cluster from a > hardware vendor. We've encountered a couple of interesting issues > with bringing the thing up that I'd like to get group comments on. > Note that the RFP and negotiations specified this system was for a > cluster installation, so there would be no misunderstanding... > > 1. We specified "No OS" in the purchase so that we could install > CentOS as our base. We got a set of systems with a stub OS, and an > EULA for the diagnostics embedded on the disk. After clicking thru > the EULA, it tells us we have no OS on the disk, but does not fail > to PXE. > > 2. BIOS had a couple of interesting defaults, including warn on > keyboard error (Keyboard? Not intentionally. This is a compute > node, and should never require a keyboard. Ever.) We also find the > BIOS is set to boot from hard disk THEN PXE. But due to item 1, > above, we never can fail over to PXE unless we load up a keyboard > and monitor, and hit F12 to drop to PXE. > > In discussions with our sales rep, I'm told that we'd have had to > pay extra to get a real bare hard disk, and that, for a fee, they'd > have been willing to custom-configure the BIOS. OK, with the BIOS > this isn't too unreasonable: They have a standard BIOS for all > systems and if you want something special, paying for it's the > norm... But, still, this is a CLUSTER installation we were quoted, > not a desktop. > > Also, I'm now told that "almost every customer" ordered their > cluster configuration service at several kilobucks per rack. Since > the team I'm working with has some degree of experience in > configuring and installing hardware and software on computational > clusters, now measured in at least 10 separate cluster > installations, this seemed like an unnecessary expense. However, > we're finding vendor gotchas that are annoying at the least, and > sometimes cause significant work-around time/effort. > > Finally, our sales guy yesterday was somewhat baffled as to why we'd > ordered without OS, and further why we were using Linux over Windows > for HPC. Not trying to revive the recent rant-fest about Windows > HPC capabilities, can anyone cite real HPC applications generally > run on significant clusters (I'll accept Cornell's work, although I > remain personally convinced that the bulk of their Windows HPC work > has been dedicated to maintaining grant funding rather than doing > real work)? > > No, I won't identify the vendor. > -- > Gerry Creager -- gerry.creager@tamu.edu > Texas Mesonet -- AATLT, Texas A&M University > Cell: 979.229.5301 Office: 979.862.3982 FAX: 979.862.3983 > Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at streamline-computing.com Fri Jun 6 09:43:32 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <48495A43.4060809@tamu.edu> References: <48495A43.4060809@tamu.edu> Message-ID: <1212770622.9679.5.camel@Vigor13> On Fri, 2008-06-06 at 10:39 -0500, Gerry Creager wrote: > 1. We specified "No OS" in the purchase so that we could install CentOS > as our base. We got a set of systems with a stub OS, and an EULA for > the diagnostics embedded on the disk. After clicking thru the EULA, it > tells us we have no OS on the disk, but does not fail to PXE. That sounds normal to me - that's the state we get servers in. > 2. BIOS had a couple of interesting defaults, including warn on > keyboard error (Keyboard? Not intentionally. This is a compute node, > and should never require a keyboard. Ever.) We also find the BIOS is > set to boot from hard disk THEN PXE. But due to item 1, above, we never > can fail over to PXE unless we load up a keyboard and monitor, and hit > F12 to drop to PXE. The "warn on keyboard error" is a bit of a shocker - I haven't see that one for ages. But the rest sound normal, and yes getting the keyboard and monitor out is normal. We get technicians to set the BIOSes on all compute nodes prior to delivery. I just don't see why though that the BIOSes from these major vendors are ALWAYS hard disk first then PXE. A default setting t'other way around would make much more sense. From landman at scalableinformatics.com Fri Jun 6 09:45:27 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <48495A43.4060809@tamu.edu> References: <48495A43.4060809@tamu.edu> Message-ID: <484969A7.4030008@scalableinformatics.com> Hi Gary A first point, before going anywhere else ... you get what you pay for ... most of the time. The vast majority of rack-n-stack vendors do what you describe. They know one thing, any deviation from that leaves them blinking with open mouths ... they deliver what they have a level of comfort delivering. Gerry Creager wrote: > We recently purchased a set of hardware for a cluster from a hardware > vendor. We've encountered a couple of interesting issues with bringing > the thing up that I'd like to get group comments on. Note that the RFP > and negotiations specified this system was for a cluster installation, > so there would be no misunderstanding... > > 1. We specified "No OS" in the purchase so that we could install CentOS > as our base. We got a set of systems with a stub OS, and an EULA for > the diagnostics embedded on the disk. After clicking thru the EULA, it > tells us we have no OS on the disk, but does not fail to PXE. I would say it is likely due to the fact that altering their base cluster construction model is a problem (e.g. costs them money). FWIW: We boot our nodes diskless during testing, and diskful if this is the required state of the cluster. Nothing like actually testing the hardware you are going to deliver in the way your customers are going to use it. This said, it seems ... unlikely ... that this was their purpose. > > 2. BIOS had a couple of interesting defaults, including warn on > keyboard error (Keyboard? Not intentionally. This is a compute node, > and should never require a keyboard. Ever.) We also find the BIOS is > set to boot from hard disk THEN PXE. But due to item 1, above, we never > can fail over to PXE unless we load up a keyboard and monitor, and hit > F12 to drop to PXE. Egad. We (by hand) reconfigure the bios specifically so that there are no issues like this. > In discussions with our sales rep, I'm told that we'd have had to pay > extra to get a real bare hard disk, and that, for a fee, they'd have Heh.... if you want nothing what is it you have to pay? :) > been willing to custom-configure the BIOS. OK, with the BIOS this isn't > too unreasonable: They have a standard BIOS for all systems and if you > want something special, paying for it's the norm... But, still, this is > a CLUSTER installation we were quoted, not a desktop. Agreed. > > Also, I'm now told that "almost every customer" ordered their cluster > configuration service at several kilobucks per rack. Since the team I'm This is standard, it costs money to rack and stack. If you don't want it, you don't have to get it. > working with has some degree of experience in configuring and installing > hardware and software on computational clusters, now measured in at > least 10 separate cluster installations, this seemed like an unnecessary > expense. However, we're finding vendor gotchas that are annoying at the Yeah, in this case, it is unnecessary. If your team has the expertise, you don't need to pay for it. > least, and sometimes cause significant work-around time/effort. For the smaller companies that do cluster setup/installs, the idea is not to mess the customer up. > > Finally, our sales guy yesterday was somewhat baffled as to why we'd > ordered without OS, and further why we were using Linux over Windows for > HPC. Not trying to revive the recent rant-fest about Windows HPC You do understand how hard (e.g. how much money is flowing from) Microsoft is pushing their solution. Money talks. > capabilities, can anyone cite real HPC applications generally run on > significant clusters (I'll accept Cornell's work, although I remain > personally convinced that the bulk of their Windows HPC work has been > dedicated to maintaining grant funding rather than doing real work)? > > No, I won't identify the vendor. :) -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com phone: +1 734 786 8423 fax : +1 734 786 8452 cell : +1 734 612 4615 From john.hearns at streamline-computing.com Fri Jun 6 09:55:05 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <48495A43.4060809@tamu.edu> References: <48495A43.4060809@tamu.edu> Message-ID: <1212771315.9679.15.camel@Vigor13> On Fri, 2008-06-06 at 10:39 -0500, Gerry Creager wrote: > W > Also, I'm now told that "almost every customer" ordered their cluster > configuration service at several kilobucks per rack. Since the team I'm > working with has some degree of experience in configuring and installing > hardware and software on computational clusters, now measured in at > least 10 separate cluster installations, this seemed like an unnecessary > expense. However, we're finding vendor gotchas that are annoying at the > least, and sometimes cause significant work-around time/effort. Somebody has to pay for all those technicians to set you BIOSes. Seriously, we almost always do turn-key clusters for customers. We do what we term as "hardware only" deals - but you wouldn't recognise them as such. The project I'm thinking on consisted of supplying many Intel twin servers, and us setting those BIOSes prior to delivery, racking, labelling and cabling all servers and switches. Providing a loan cluster head node and installing our cluster distribution on there for a week long soak test prior to the customer accepting it and then reinstalling with their own OS. Quite, quite far from leaving a pile of boxes on the loading dock. I don't want to go into this one too deeply, but when we do true hardware-only deals (I'm thinking more of network switches here) you end up supporting things out of goodwill anyway. From bill at cse.ucdavis.edu Fri Jun 6 10:00:29 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <48495A43.4060809@tamu.edu> References: <48495A43.4060809@tamu.edu> Message-ID: <48496D2D.5080801@cse.ucdavis.edu> Gerry Creager wrote: > 1. We specified "No OS" in the purchase so that we could install CentOS > as our base. We got a set of systems with a stub OS, and an EULA for > the diagnostics embedded on the disk. After clicking thru the EULA, it > tells us we have no OS on the disk, but does not fail to PXE. If you want to avoid hooking up a KVM to each node and rebooting it once or twice I'd suggest putting "Nodes must PXE boot by default" in your specifications. > 2. BIOS had a couple of interesting defaults, including warn on > keyboard error (Keyboard? Not intentionally. This is a compute node, > and should never require a keyboard. Ever.) We also find the BIOS is > set to boot from hard disk THEN PXE. But due to item 1, above, we never > can fail over to PXE unless we load up a keyboard and monitor, and hit > F12 to drop to PXE. Very strange standard for a server, let alone a cluster node. > In discussions with our sales rep, I'm told that we'd have had to pay > extra to get a real bare hard disk, and that, for a fee, they'd have > been willing to custom-configure the BIOS. OK, with the BIOS this isn't > too unreasonable: They have a standard BIOS for all systems and if you > want something special, paying for it's the norm... But, still, this is > a CLUSTER installation we were quoted, not a desktop. This whole thing sounds strangely like the vendor has already been picked. Certainly changing any default in the pipeline can cost money, even deleting a floppy, cd/dvd etc can cost money if the machine ships to the integration center with it installed. With that said when someone charges an unreasonable amount for said customizations they lose the bid and someone else wins. > Also, I'm now told that "almost every customer" ordered their cluster > configuration service at several kilobucks per rack. Since the team I'm Not sure of the relevance here. Sounds like the upsell and padding that sales folks love, it is there job to sell equipment preferably high margin at that. Seems way high for a BIOS reset, less so if it includes a cabling harness for power, console, rails premounted, and network. Again if it's a bid process.... > working with has some degree of experience in configuring and installing > hardware and software on computational clusters, now measured in at > least 10 separate cluster installations, this seemed like an unnecessary > expense. However, we're finding vendor gotchas that are annoying at the > least, and sometimes cause significant work-around time/effort. Well there's two choice, either deal with the gotchas, or make them part of the specifications. All vendors have their differences, defaults, and cost structures. Do you want a cluster that could conceivable allow users to start submitting jobs within a day? Or do you want to play BIOS games, testing, and integration that might take a week or two. Every time I order a cluster (well over 10 now) I get vendor queries of the "Sounds like X might mean you need Y which costs $Z". I'm always very clear, it's in the spec, and not meeting the spec will mean the bid isn't considered. Definitely seems like some high margin items end up included... without the margin. > Finally, our sales guy yesterday was somewhat baffled as to why we'd > ordered without OS, and further why we were using Linux over Windows for > HPC. Heh, some sales folks seem to have a right to exert design pressure on cluster design, not sure why your even entertaining that one. If you want to be particularly friendly I'd just point at top500.org and that linux is the standard and not the exception for beowulf clusters. > No, I won't identify the vendor. How about the number of letters in their name ;-). In general I find that the big vendors build in large profits (I.e. negotiating down to 50% of list price is not unusual) and often the preferred cluster defaults often mean higher costs instead of less, despite the typically higher volume purchases, identical compute nodes, don't need a dvd, don't need an OS, don't (typically) need a redundant power supply for compute nodes, etc. The smaller cluster specific shops default (usually) to mostly reasonable cluster configurations, and seem to default to smaller margins. In my experience, writing a spec that welcomes both ends up with the best deals. Even something trivial like specifying 14 or 15 disks in a array (often the max for an external array) instead of 16 (common for direct attached) can be the different to allow a competitive bid from a big vendor. Sometimes Intel or AMD intercedes to get a design win and sometimes a big vendor decides to get more competitive. Of course these specifications directly effect costs and lead to endless discussions on this list. KVM over IP? Serial console? Any console access at all? IMPI or just switched PDUs? But in my experience things like "must boot from PXE" is not a big deal, and not worth several kilobucks. From perry at piermont.com Fri Jun 6 10:45:41 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <48496D2D.5080801@cse.ucdavis.edu> (Bill Broadley's message of "Fri\, 06 Jun 2008 10\:00\:29 -0700") References: <48495A43.4060809@tamu.edu> <48496D2D.5080801@cse.ucdavis.edu> Message-ID: <87hcc6h2yi.fsf@snark.cb.piermont.com> Bill Broadley writes: >> 2. BIOS had a couple of interesting defaults, including warn on >> keyboard error (Keyboard? Not intentionally. This is a compute >> node, and should never require a keyboard. Ever.) We also find the >> BIOS is set to boot from hard disk THEN PXE. But due to item 1, >> above, we never can fail over to PXE unless we load up a keyboard >> and monitor, and hit F12 to drop to PXE. > > Very strange standard for a server, let alone a cluster node. I would be less disturbed about such things if it was trivial to alter the BIOS settings in a semi-automated way -- say by booting some standalone program, or loading a file from a USB thumb drive. Then you could just go up to each box with a USB thumb drive, turn it on, and have it fix itself in a consistent way. However, the fact that you can't generally automate fixing BIOS settings makes all of this far more annoying. Anyone have any cool tricks for how to consistently set the BIOS on large numbers of boxes without requiring steps that humans can screw up easily? -- Perry E. Metzger perry@piermont.com From tjrc at sanger.ac.uk Fri Jun 6 11:35:25 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <87hcc6h2yi.fsf@snark.cb.piermont.com> References: <48495A43.4060809@tamu.edu> <48496D2D.5080801@cse.ucdavis.edu> <87hcc6h2yi.fsf@snark.cb.piermont.com> Message-ID: <9FD5E5B6-BCAF-4C00-A02B-A3678E46C77D@sanger.ac.uk> On 6 Jun 2008, at 6:45 pm, Perry E. Metzger wrote: > > Bill Broadley writes: >>> 2. BIOS had a couple of interesting defaults, including warn on >>> keyboard error (Keyboard? Not intentionally. This is a compute >>> node, and should never require a keyboard. Ever.) We also find the >>> BIOS is set to boot from hard disk THEN PXE. But due to item 1, >>> above, we never can fail over to PXE unless we load up a keyboard >>> and monitor, and hit F12 to drop to PXE. >> >> Very strange standard for a server, let alone a cluster node. > > I would be less disturbed about such things if it was trivial to alter > the BIOS settings in a semi-automated way -- say by booting some > standalone program, or loading a file from a USB thumb drive. Then you > could just go up to each box with a USB thumb drive, turn it on, and > have it fix itself in a consistent way. However, the fact that you > can't generally automate fixing BIOS settings makes all of this far > more annoying. > > Anyone have any cool tricks for how to consistently set the BIOS on > large numbers of boxes without requiring steps that humans can screw > up easily? Nope. :-) This is, in my view, one of the major disadvantages of PC clusters. The crappy old BIOS that we're stuck with. Here, we mostly get around this problem by using blade servers rather than pizza boxes. Or at least using pizza boxes which have some form of command line access to a lights-out management processor that allows us to set the boot order, such as those on HP ProLiants and Sun X**** servers. So with c-Class blades from HP, for example, I don't really have a problem - once the chassis is configured, I make them all PXE boot by ssh'ing into the Onboard administrator and typing: set server boot first pxe all poweron server all Bingo, all 16 machines PXE boot at about 1 second intervals. Job's a good'un. As Joe says, you get what you pay for. I don't think I've *ever* had to futz around with BIOS settings on any recent bladeserver (I used to have to on our old RLX bladeservers, which periodically got confused and lost all the CMOS settings, which required manual fixing in the BIOS). But the IBM and HP stuff we use now, it's very rare indeed. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From matt at technoronin.com Fri Jun 6 15:55:44 2008 From: matt at technoronin.com (Matt Lawrence) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <48495A43.4060809@tamu.edu> References: <48495A43.4060809@tamu.edu> Message-ID: On Fri, 6 Jun 2008, Gerry Creager wrote: > We recently purchased a set of hardware for a cluster from a hardware vendor. > We've encountered a couple of interesting issues with bringing the thing up > that I'd like to get group comments on. Note that the RFP and negotiations > specified this system was for a cluster installation, so there would be no > misunderstanding... > > 1. We specified "No OS" in the purchase so that we could install CentOS as > our base. We got a set of systems with a stub OS, and an EULA for the > diagnostics embedded on the disk. After clicking thru the EULA, it tells us > we have no OS on the disk, but does not fail to PXE. And standing on the perforated floor tiles in front of the system performing this task all day long has left me with sore and very cold feet. The good news is that the cooling in there is working much better. Our student worker is on vacation sailing in the Carribean, so I get the abuse.... -- Matt It's not what I know that counts. It's what I can remember in time to use. From gerry.creager at tamu.edu Fri Jun 6 17:34:53 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <9FD5E5B6-BCAF-4C00-A02B-A3678E46C77D@sanger.ac.uk> References: <48495A43.4060809@tamu.edu> <48496D2D.5080801@cse.ucdavis.edu> <87hcc6h2yi.fsf@snark.cb.piermont.com> <9FD5E5B6-BCAF-4C00-A02B-A3678E46C77D@sanger.ac.uk> Message-ID: <4849D7AD.30605@tamu.edu> Tim Cutts wrote: > > On 6 Jun 2008, at 6:45 pm, Perry E. Metzger wrote: > >> >> Bill Broadley writes: >>>> 2. BIOS had a couple of interesting defaults, including warn on >>>> keyboard error (Keyboard? Not intentionally. This is a compute >>>> node, and should never require a keyboard. Ever.) We also find the >>>> BIOS is set to boot from hard disk THEN PXE. But due to item 1, >>>> above, we never can fail over to PXE unless we load up a keyboard >>>> and monitor, and hit F12 to drop to PXE. >>> >>> Very strange standard for a server, let alone a cluster node. >> >> I would be less disturbed about such things if it was trivial to alter >> the BIOS settings in a semi-automated way -- say by booting some >> standalone program, or loading a file from a USB thumb drive. Then you >> could just go up to each box with a USB thumb drive, turn it on, and >> have it fix itself in a consistent way. However, the fact that you >> can't generally automate fixing BIOS settings makes all of this far >> more annoying. >> >> Anyone have any cool tricks for how to consistently set the BIOS on >> large numbers of boxes without requiring steps that humans can screw >> up easily? > > Nope. :-) This is, in my view, one of the major disadvantages of PC > clusters. The crappy old BIOS that we're stuck with. > > Here, we mostly get around this problem by using blade servers rather > than pizza boxes. Or at least using pizza boxes which have some form of > command line access to a lights-out management processor that allows us > to set the boot order, such as those on HP ProLiants and Sun X**** servers. > > So with c-Class blades from HP, for example, I don't really have a > problem - once the chassis is configured, I make them all PXE boot by > ssh'ing into the Onboard administrator and typing: > > set server boot first pxe all > poweron server all > > Bingo, all 16 machines PXE boot at about 1 second intervals. Job's a > good'un. As Joe says, you get what you pay for. I don't think I've > *ever* had to futz around with BIOS settings on any recent bladeserver > (I used to have to on our old RLX bladeservers, which periodically got > confused and lost all the CMOS settings, which required manual fixing in > the BIOS). But the IBM and HP stuff we use now, it's very rare indeed. Yeah.... Part of the problem. The last several clusters I've worked on, we didn't have to futz with the BIOS, either. HOWEVER, it's been pointed out to me that "You get what you pay for" and part of what you pay for is the competent folks making sure such futzing isn't required. gerry -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.862.3982 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From tjrc at sanger.ac.uk Sat Jun 7 00:14:37 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <4849D7AD.30605@tamu.edu> References: <48495A43.4060809@tamu.edu> <48496D2D.5080801@cse.ucdavis.edu> <87hcc6h2yi.fsf@snark.cb.piermont.com> <9FD5E5B6-BCAF-4C00-A02B-A3678E46C77D@sanger.ac.uk> <4849D7AD.30605@tamu.edu> Message-ID: <70D27FFA-DAD1-4493-9D51-72E149FCA5EF@sanger.ac.uk> On 7 Jun 2008, at 1:34 am, Gerry Creager wrote: > Yeah.... Part of the problem. The last several clusters I've worked > on, we didn't have to futz with the BIOS, either. HOWEVER, it's > been pointed out to me that "You get what you pay for" and part of > what you pay for is the competent folks making sure such futzing > isn't required. Well, there is that. Or at least, paying for people to do the dreary futzing for you. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From gerry.creager at tamu.edu Sat Jun 7 07:49:42 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <70D27FFA-DAD1-4493-9D51-72E149FCA5EF@sanger.ac.uk> References: <48495A43.4060809@tamu.edu> <48496D2D.5080801@cse.ucdavis.edu> <87hcc6h2yi.fsf@snark.cb.piermont.com> <9FD5E5B6-BCAF-4C00-A02B-A3678E46C77D@sanger.ac.uk> <4849D7AD.30605@tamu.edu> <70D27FFA-DAD1-4493-9D51-72E149FCA5EF@sanger.ac.uk> Message-ID: <484AA006.9070009@tamu.edu> And done here. Tim Cutts wrote: > > On 7 Jun 2008, at 1:34 am, Gerry Creager wrote: > >> Yeah.... Part of the problem. The last several clusters I've worked >> on, we didn't have to futz with the BIOS, either. HOWEVER, it's been >> pointed out to me that "You get what you pay for" and part of what you >> pay for is the competent folks making sure such futzing isn't required. > > Well, there is that. Or at least, paying for people to do the dreary > futzing for you. > > Tim > > -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From gerry.creager at tamu.edu Sat Jun 7 07:50:44 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <484AA006.9070009@tamu.edu> References: <48495A43.4060809@tamu.edu> <48496D2D.5080801@cse.ucdavis.edu> <87hcc6h2yi.fsf@snark.cb.piermont.com> <9FD5E5B6-BCAF-4C00-A02B-A3678E46C77D@sanger.ac.uk> <4849D7AD.30605@tamu.edu> <70D27FFA-DAD1-4493-9D51-72E149FCA5EF@sanger.ac.uk> <484AA006.9070009@tamu.edu> Message-ID: <484AA044.1040107@tamu.edu> Sorry about that. Wrong message when I hit "reply all". Time for more coffee. gerry Gerry Creager wrote: > And done here. > > Tim Cutts wrote: >> >> On 7 Jun 2008, at 1:34 am, Gerry Creager wrote: >> >>> Yeah.... Part of the problem. The last several clusters I've worked >>> on, we didn't have to futz with the BIOS, either. HOWEVER, it's been >>> pointed out to me that "You get what you pay for" and part of what >>> you pay for is the competent folks making sure such futzing isn't >>> required. >> >> Well, there is that. Or at least, paying for people to do the dreary >> futzing for you. >> >> Tim >> >> > -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From csamuel at vpac.org Sun Jun 8 17:09:15 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <87hcc6h2yi.fsf@snark.cb.piermont.com> Message-ID: <1110348284.110351212970155947.JavaMail.root@zimbra.vpac.org> ----- "Perry E. Metzger" wrote: > I would be less disturbed about such things if it was > trivial to alter the BIOS settings in a semi-automated > way -- say by booting some standalone program, or loading > a file from a USB thumb drive. Our most recent vendor went to the motherboard manufacturer and said "please can you cut us a BIOS with these default settings" and they did so. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From tjrc at sanger.ac.uk Sun Jun 8 21:05:02 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <1110348284.110351212970155947.JavaMail.root@zimbra.vpac.org> References: <1110348284.110351212970155947.JavaMail.root@zimbra.vpac.org> Message-ID: <69E48321-05BE-4DA1-B5CC-A92A5DF24F56@sanger.ac.uk> On 9 Jun 2008, at 1:09 am, Chris Samuel wrote: > > ----- "Perry E. Metzger" wrote: > >> I would be less disturbed about such things if it was >> trivial to alter the BIOS settings in a semi-automated >> way -- say by booting some standalone program, or loading >> a file from a USB thumb drive. > > Our most recent vendor went to the motherboard manufacturer > and said "please can you cut us a BIOS with these default > settings" and they did so. If you don't mind us asking, roughly how much extra did *that* cost? Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From csamuel at vpac.org Mon Jun 9 00:28:46 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <1237440751.110541212996492356.JavaMail.root@zimbra.vpac.org> Message-ID: <992702936.110561212996526583.JavaMail.root@zimbra.vpac.org> ----- "Tim Cutts" wrote: > On 9 Jun 2008, at 1:09 am, Chris Samuel wrote: > > > Our most recent vendor went to the motherboard manufacturer > > and said "please can you cut us a BIOS with these default > > settings" and they did so. > > If you don't mind us asking, roughly how much extra did *that* cost? Nothing. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From tjrc at sanger.ac.uk Mon Jun 9 01:43:51 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <992702936.110561212996526583.JavaMail.root@zimbra.vpac.org> References: <992702936.110561212996526583.JavaMail.root@zimbra.vpac.org> Message-ID: <484CED47.5050804@sanger.ac.uk> Chris Samuel wrote: > ----- "Tim Cutts" wrote: > > >> On 9 Jun 2008, at 1:09 am, Chris Samuel wrote: >> >> >>> Our most recent vendor went to the motherboard manufacturer >>> and said "please can you cut us a BIOS with these default >>> settings" and they did so. >>> >> If you don't mind us asking, roughly how much extra did *that* cost? >> > > Nothing. > Wow. How many nodes were you buying? And are we allowed to know who the vendor was? Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From csamuel at vpac.org Mon Jun 9 03:30:44 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <222401034.110641213007329196.JavaMail.root@zimbra.vpac.org> Message-ID: <767212096.110661213007444346.JavaMail.root@zimbra.vpac.org> ----- "Tim Cutts" wrote: > Wow. How many nodes were you buying? 95 nodes, each with two Barcelonas, so 760 cores all up. 32GB RAM (4GB/core) and 4x300GB SATA drives (RAID-0) per node. > And are we allowed to know who the vendor was? It's all public, so no reason why not. It was a local Melbourne mob called Xenon Systems, they sell SuperMicro based systems. Kudos to both of them for their support. cheers! Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From apittman at concurrent-thinking.com Mon Jun 9 03:49:07 2008 From: apittman at concurrent-thinking.com (Ashley Pittman) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <48495A43.4060809@tamu.edu> References: <48495A43.4060809@tamu.edu> Message-ID: <1213008547.8064.9.camel@bruce.priv.wark.uk.streamline-computing.com> On Fri, 2008-06-06 at 10:39 -0500, Gerry Creager wrote: > > 2. BIOS had a couple of interesting defaults, including warn on > keyboard error (Keyboard? Not intentionally. This is a compute > node, > and should never require a keyboard. Ever.) We also find the BIOS > is > set to boot from hard disk THEN PXE. But due to item 1, above, we > never > can fail over to PXE unless we load up a keyboard and monitor, and > hit > F12 to drop to PXE. I can think of at least one cluster where the opposite has been true and PXE boot has been the default. The problem with this is if the head node PXE boots on the customers network and gets automatically re-installed as a windows workstation everybody gets egg on their face. Yes even "modern" BIOSes are bad but localboot first is a sensible default. Ashley Pittman. From matt at technoronin.com Mon Jun 9 06:11:53 2008 From: matt at technoronin.com (Matt Lawrence) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <1213008547.8064.9.camel@bruce.priv.wark.uk.streamline-computing.com> References: <48495A43.4060809@tamu.edu> <1213008547.8064.9.camel@bruce.priv.wark.uk.streamline-computing.com> Message-ID: On Mon, 9 Jun 2008, Ashley Pittman wrote: > I can think of at least one cluster where the opposite has been true and > PXE boot has been the default. The problem with this is if the head > node PXE boots on the customers network and gets automatically > re-installed as a windows workstation everybody gets egg on their face. > Yes even "modern" BIOSes are bad but localboot first is a sensible > default. I will have to disagree. Changing the BIOS settings in a single head node is preferable to having to connect to 126 compute nodes and change their BIOS settings. -- Matt It's not what I know that counts. It's what I can remember in time to use. From prentice at ias.edu Mon Jun 9 08:41:29 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] User resource limits Message-ID: <484D4F29.9090704@ias.edu> This topic is slightly off topic, since it's not a beowulf specific problem, but it is HPC-related: I have several fat servers with 4 cores and 32 GB of RAM, for jobs that aren't very parallel and need large amounts of RAM. They are not clustered in any way. At the moment, users ssh into these systems to run large jobs. Eventually, I will have these nodes managed by a queuing system. The problem: Every couple of days, one of these systems become unresponsive due to OOM errors. If we wait long enough, the offending job will complete, and everything will return to normal. Since these are multi-user shared resources, I don't have the luxury of waiting for the systems to clear themselves up, and I often have to hit the power button. I would like to impose some CPU and memory limits on users that are hard limits that can't be changed/overridden by the users. What is the best way to do this? All I know is environment variables or shell commands done as the user (ulimit, for example). -- Prentice From dnlombar at ichips.intel.com Mon Jun 9 09:12:32 2008 From: dnlombar at ichips.intel.com (Lombard, David N) Date: Wed Nov 25 01:07:09 2009 Subject: [Beowulf] User resource limits In-Reply-To: <484D4F29.9090704@ias.edu> References: <484D4F29.9090704@ias.edu> Message-ID: <20080609161232.GB11155@nlxdcldnl2.cl.intel.com> On Mon, Jun 09, 2008 at 11:41:29AM -0400, Prentice Bisbal wrote: > > I would like to impose some CPU and memory limits on users that are hard > limits that can't be changed/overridden by the users. What is the best > way to do this? All I know is environment variables or shell commands > done as the user (ulimit, for example). pam_limits and /etc/security/limits.conf -- David N. Lombard, Intel, Irvine, CA I do not speak for Intel Corporation; all comments are strictly my own. From perry at piermont.com Mon Jun 9 09:53:41 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] User resource limits In-Reply-To: <484D4F29.9090704@ias.edu> (Prentice Bisbal's message of "Mon\, 09 Jun 2008 11\:41\:29 -0400") References: <484D4F29.9090704@ias.edu> Message-ID: <87od6av9be.fsf@snark.cb.piermont.com> Prentice Bisbal writes: > I would like to impose some CPU and memory limits on users that are hard > limits that can't be changed/overridden by the users. What is the best > way to do this? All I know is environment variables or shell commands > done as the user (ulimit, for example). ulimit is not quite "a command done by the user". The user manipulates their ulimits with the shell ulimit command, but the limits are in fact maintained by the kernel, and can be set by the administrator at maximum levels that the user cannot reduce. Read the man page for getrlimit/setrlimit for details on this. ulimits are inherited by a process from its parents, so if the process used at login (like sshd) sets them appropriately, the limits are inherited by the whole session. The administrator can set default ulimit ceilings in various login configuration files -- the file that is used depends on the specific OS you are running. If you say what OS and/or distro you are using, I can be of more specific help. Perry From perry at piermont.com Mon Jun 9 09:55:51 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] User resource limits In-Reply-To: <20080609161232.GB11155@nlxdcldnl2.cl.intel.com> (David N. Lombard's message of "Mon\, 9 Jun 2008 09\:12\:32 -0700") References: <484D4F29.9090704@ias.edu> <20080609161232.GB11155@nlxdcldnl2.cl.intel.com> Message-ID: <87k5gyv97s.fsf@snark.cb.piermont.com> "Lombard, David N" writes: > On Mon, Jun 09, 2008 at 11:41:29AM -0400, Prentice Bisbal wrote: >> >> I would like to impose some CPU and memory limits on users that are hard >> limits that can't be changed/overridden by the users. What is the best >> way to do this? All I know is environment variables or shell commands >> done as the user (ulimit, for example). > > pam_limits and /etc/security/limits.conf You're making assumptions about what OS he's running. He didn't say which flavor of Unix this is. We can only assume it is some POSIX OS because he mentions the ulimit command. Indeed, not even all Linuxes use that file, though many do. Perry From prentice at ias.edu Mon Jun 9 10:38:08 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] User resource limits In-Reply-To: <87k5gyv97s.fsf@snark.cb.piermont.com> References: <484D4F29.9090704@ias.edu> <20080609161232.GB11155@nlxdcldnl2.cl.intel.com> <87k5gyv97s.fsf@snark.cb.piermont.com> Message-ID: <484D6A80.20803@ias.edu> Perry E. Metzger wrote: > "Lombard, David N" writes: >> On Mon, Jun 09, 2008 at 11:41:29AM -0400, Prentice Bisbal wrote: >>> I would like to impose some CPU and memory limits on users that are hard >>> limits that can't be changed/overridden by the users. What is the best >>> way to do this? All I know is environment variables or shell commands >>> done as the user (ulimit, for example). >> pam_limits and /etc/security/limits.conf > > You're making assumptions about what OS he's running. He didn't say > which flavor of Unix this is. We can only assume it is some POSIX OS > because he mentions the ulimit command. Indeed, not even all Linuxes > use that file, though many do. > Yeah, my mistake - I forgot to include that important piece of data. My apologies. I'm running PU_IAS Linux 5.1. PU_IAS is a rebuild of RHEL, so anything that applies to RHEL applies to PU_IAS. http://plug.princeton.edu/linux/ I think David was assuming I was running Linux, and he was correct. thanks for your help. I have to go read some man pages now. Prentice From kus at free.net Mon Jun 9 15:01:40 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] size of swap partition Message-ID: A lot of time ago it was formulated simple rule for swap partition size (equal to main memory size). Currently we all have relative large RAM on the nodes (typically, I beleive, it is 2 or more GB per core; we have 16 GB per dual-socket quad-core Opteron node). What is typical modern swap size today? I understand that it depends from applications ;-) We, in particular, practically don't have jobs which run "out-of-RAM". For single core dual-socket Opteron nodes w/4GB RAM per node and "molecular modelling workload" we used 4 GB swap partition. But what are the reccomendations of modern praxis ? Mikhail Kuzminksy Computer Assistance to Chemical Research Center Zelinsky Inst. of Organic Chemistry Moscow From gerry.creager at tamu.edu Mon Jun 9 15:51:34 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] size of swap partition In-Reply-To: References: Message-ID: <484DB3F6.7010904@tamu.edu> Misha, We have the potential to have to swap whole jobs out of memory on a complete node. As a result, I recommend 1.5-2.0 times memory in swap if this is a consideration. I do know there's likely to be a bit of discussion as this varies widely from site to site and based on requirements. gerry Mikhail Kuzminsky wrote: > A lot of time ago it was formulated simple rule for swap partition size > (equal to main memory size). > > Currently we all have relative large RAM on the nodes (typically, I > beleive, it is 2 or more GB per core; we have 16 GB per dual-socket > quad-core Opteron node). What is typical modern swap size today? > > I understand that it depends from applications ;-) We, in particular, > practically don't have jobs which run "out-of-RAM". For single core > dual-socket Opteron nodes w/4GB RAM per node and "molecular modelling > workload" we used 4 GB swap partition. > > But what are the reccomendations of modern praxis ? > > Mikhail Kuzminksy > Computer Assistance to Chemical Research Center > Zelinsky Inst. of Organic Chemistry > Moscow _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.862.3982 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From kyron at neuralbs.com Mon Jun 9 18:28:28 2008 From: kyron at neuralbs.com (Eric Thibodeau) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] size of swap partition In-Reply-To: <484DB3F6.7010904@tamu.edu> References: <484DB3F6.7010904@tamu.edu> Message-ID: <484DD8BC.10208@neuralbs.com> Mikhail, Somewhat like Gerry said, ballpark figures have always been an arbitrary 1.5*RAM. This is completely ridiculous nowadays and should depend entirely on the applications you run. Typically, you should never swap out memory on a running application. I recommend you perform some metrics collection, doesn't have to be perfect and super-fine-grained. Something like Ganglia should be sufficient to give you an idea of how much swap you need, if ever you actually hit it...but don't! Eric PS: this is a redundant topic on the list ...do a little searching and you'll hit it ;) Gerry Creager wrote: > Misha, > > We have the potential to have to swap whole jobs out of memory on a > complete node. As a result, I recommend 1.5-2.0 times memory in swap > if this is a consideration. I do know there's likely to be a bit of > discussion as this varies widely from site to site and based on > requirements. > > gerry > > Mikhail Kuzminsky wrote: >> A lot of time ago it was formulated simple rule for swap partition size >> (equal to main memory size). >> >> Currently we all have relative large RAM on the nodes (typically, I >> beleive, it is 2 or more GB per core; we have 16 GB per dual-socket >> quad-core Opteron node). What is typical modern swap size today? >> >> I understand that it depends from applications ;-) We, in particular, >> practically don't have jobs which run "out-of-RAM". For single core >> dual-socket Opteron nodes w/4GB RAM per node and "molecular modelling >> workload" we used 4 GB swap partition. >> >> But what are the reccomendations of modern praxis ? >> >> Mikhail Kuzminksy >> Computer Assistance to Chemical Research Center >> Zelinsky Inst. of Organic Chemistry >> Moscow _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > From csamuel at vpac.org Mon Jun 9 20:56:28 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] User resource limits In-Reply-To: <565153331.116881213069858428.JavaMail.root@zimbra.vpac.org> Message-ID: <742070125.116971213070188917.JavaMail.root@zimbra.vpac.org> ----- "Prentice Bisbal" wrote: > I think David was assuming I was running Linux, and he was correct. > thanks for your help. I have to go read some man pages now. Be very aware that there are two different ulimits that affect memory allocations, *depending on the size of the allocation that is asked for*, if you have glibc 2.3 or newer (so most distros still in use). For allocations < 128KB the standard memory limit is applied as it uses brk(), but for allocations greater than that it uses mmap(). Unfortunately the kernel implementation of mmap() doesn't check the maximum memory size (RLIMIT_RSS) or maximum data size (RLIMIT_DATA) limits which were being set, but only the maximum virtual RAM size (RLIMIT_AS) - this is documented in the setrlimit(2) man page. :-( -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From hahn at mcmaster.ca Mon Jun 9 21:58:12 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] size of swap partition In-Reply-To: <484DB3F6.7010904@tamu.edu> References: <484DB3F6.7010904@tamu.edu> Message-ID: > We have the potential to have to swap whole jobs out of memory on a complete > node. that was our intent as well. among other things, this scheme enables running the cluster "split-personality" - mostly shorter/smaller even interactive jobs during the day, with big/long jobs running at night. unfortunately, you need a smart scheduler to do this, and ours is dumb. >> beleive, it is 2 or more GB per core; we have 16 GB per dual-socket >> quad-core Opteron node). What is typical modern swap size today? are you willing to use a node which is actually occupying 16 GB of swap? it is possible to tune how the kernel responds to memory crunches - for instance, you can always avoid OOM with the vm.overcommit_memory=2 sysctl (you'll need to tune vm.overcommit_ratio and the amount of swap to get the desired limits.) in this mode, the kernel tracks how much VM it actually needs (worst-case, reflected in Committed_AS in /proc/meminfo) and compares that to a commit limit that reflects ram and swap. if you don't use overcommit_memory=2, you are basically borrowing VM space in hopes of not needing it. that can still be reasonable, considering how often processes have a lot of shared VM, and how many processes allocate but never touch lots of pages. but you have to ask yourself: would I like a system that was actually _using_ 16 GB of swap? if you have 16x disks, perhaps, but 16G will suck if you only have 1 disk. at least for overcommit_memory != 2, I don't see the point of configuring a lot of swap, since the only time you'd use it is if you were thrashing. sort of a "quality of life" argument. >> But what are the reccomendations of modern praxis ? it depends a lot on the size variance of your jobs, as well as their real/virtual ratio. the kernel only enforces RLIMIT_AS (vsz in ps),assuming a 2.6 kernel - I forget whether 2.4 did RLIMIT_RSS or not. if you use overcommit_memory=2, your desired max VM size determines the amount of swap. otherwise, go with something modest - memory size or so. but given that the smallest reasonable single disk these days is probably about 320GB, it's hard to justify being _too_ tight. From jclinton at advancedclustering.com Thu Jun 5 08:38:42 2008 From: jclinton at advancedclustering.com (Jason Clinton) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] OFED/IB for FC8 In-Reply-To: <4847B410.1070202@lfbs.rwth-aachen.de> References: <6.2.5.6.2.20080604150239.047ad270@NumerEx-LLC.com> <20080605001549.GE27430@bx9.net> <4847B410.1070202@lfbs.rwth-aachen.de> Message-ID: <588c11220806050838t5d2ede3fh3e2e3880869ba4df@mail.gmail.com> On Thu, Jun 5, 2008 at 4:38 AM, Rainer Finocchiaro < rainer@lfbs.rwth-aachen.de> wrote: > Hi Michael, > > Greg Lindahl schrieb: > >> All the OFED rpm's for FC6 installed on FC8 without difficulty, except for >>> opensm-3.0.3-0.ppc64.rpm >>> >> >> This is the cause of most of your subsequent problems. Without an SM >> running somewhere on your network, the links don't come fully up. >> ... > > > ... > Following your link, I reach a download directory offering only ppc64-RPMs; > in fact all precompiled RPMs for OFED-1.2.5 are for Power PC and not for > x86. > > .. > Much better is to download more up-to-date OFED-1.3 sources. The package > includes an install script, which builds and installs the RPMs for you. So > you don't have to "fear" to install something which is not controlled by > your package management system (RPM). As a side note, you've probably gotten yourself in to an unrecoverable state with RPM having already installed all those PPC RPM's on your Fedora 8 x86_64 systems. The easiest thing to do is probably reinstall but if you want, you can try removing them all with something like this: cd /path/to/downloaded/RPMS ls | grep -oP .+?\(\?=.x86_64\\.rpm\) | xargs rpm -e The command will extract the names of the RPM's known by RPM that you installed and then ask RPM to remove them. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080605/7bd7a474/attachment.html From jclinton at advancedclustering.com Thu Jun 5 08:41:14 2008 From: jclinton at advancedclustering.com (Jason Clinton) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] OFED/IB for FC8 In-Reply-To: <588c11220806050838t5d2ede3fh3e2e3880869ba4df@mail.gmail.com> References: <6.2.5.6.2.20080604150239.047ad270@NumerEx-LLC.com> <20080605001549.GE27430@bx9.net> <4847B410.1070202@lfbs.rwth-aachen.de> <588c11220806050838t5d2ede3fh3e2e3880869ba4df@mail.gmail.com> Message-ID: <588c11220806050841r743b0246rea9f940e5c8c7753@mail.gmail.com> On Thu, Jun 5, 2008 at 10:38 AM, Jason Clinton < jclinton@advancedclustering.com> wrote: > ls | grep -oP .+?\(\?=.x86_64\\.rpm\) | xargs rpm -e > Of course, replace "x86_64" with "ppc64" if indeed that is what you installed. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080605/e3be0b3d/attachment.html From jclinton at advancedclustering.com Thu Jun 5 10:46:54 2008 From: jclinton at advancedclustering.com (Jason Clinton) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: References: Message-ID: <588c11220806051046t3ad48a02q449a3ede0e299884@mail.gmail.com> On Thu, Jun 5, 2008 at 11:39 AM, Mikhail Kuzminsky wrote: > In message from Mark Hahn (Thu, 5 Jun 2008 11:57:28 > -0400 (EDT)): > >> To be more exact, Rev. B2 of Opteron 2350 - is it for CPU stepping w/error >>> or w/o error ? >>> >> >> AMD, like Intel, does a reasonable job of disclosing such info: >> >> >> http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/41322.PDF >> >> the well-known problem is erattum 298, I think, and fixed in B3. >> > > Yes, this AMD errata document says that in B3 revision the error "will be > fixed". I heard that new CPUs w/o TLB+L3 error are shipped now, > but are this CPUs really B3 or may be have some more new release ? Yes, what are currently shipping from AMD are B3 revision processors. The TLB-look-aside problem is fixed. There are other less-critical problems with B3, however. Specifically, power-related compatibility issues with various motherboards due to (according to the motherboard manufacturers) AMD changing the TDP late in the release process. I can't give any specific names or models that we know have problems, however. I can say that everyone involved is working on a resolution--usually through PCB revisions of the motherboards. A number of 1U power supplies that have previously worked with all Intel and AMD solutions are now insufficient, as well, due to 12V limitations. B3 pulls a *lot* of power. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080605/fe201ac3/attachment.html From jclinton at advancedclustering.com Thu Jun 5 11:16:33 2008 From: jclinton at advancedclustering.com (Jason Clinton) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: References: Message-ID: <588c11220806051116i37ff7aa1oec16a85a24009592@mail.gmail.com> On Thu, Jun 5, 2008 at 1:09 PM, Mikhail Kuzminsky wrote: > In message from Mark Hahn (Thu, 5 Jun 2008 13:55:01 > -0400 (EDT)): > >> I'm mystified by this: B2 was broken, so using it without the bios >> workaround is just a mistake or masochism. the workaround _did_ apparently >> have performance implications, but that's why B3 exists... >> >> do you mean you know of G03 problems on B2 systems which are operating >> _with_ the workaround? >> > > I don't know exactly, but I think the crash was under absence of > workaround, because I was not informed that there was some kernel patches or > BIOS changes. This was interesting for me also, because I have no > information how this hardware problem may be affected in the "real life". > Mikhail > The B2 BIOS work-around is to disable the L3 cache which gives you a 10-20% performance hit with no reduction in power consumption. The kernel patch is very extensive and, last I heard, under NDA. AMD has said publicly that the patch gives you a 1-2% performance hit. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080605/34e021ab/attachment.html From malallen at indiana.edu Fri Jun 6 11:10:43 2008 From: malallen at indiana.edu (Matt Allen) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <87hcc6h2yi.fsf@snark.cb.piermont.com> References: <48495A43.4060809@tamu.edu> <48496D2D.5080801@cse.ucdavis.edu> <87hcc6h2yi.fsf@snark.cb.piermont.com> Message-ID: <17D5930B-CC49-49CF-A33C-6E7B01401FC1@indiana.edu> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > cool tricks to consistently set the BIOS We had a cluster of systems that supported configuring the BIOS from an image on a bootable floppy. I bought 96 3.5" floppy disks, put one in each node, and then used parallel scp to dd the desired image to each node's floppy from an NFS mount. Then I power-cycled them simultaneously, and listened to the sound of 96 floppy disks being read at the same time (more or less). I'm sure I'll never hear that sound again in my life. I'm not sure how relevant or cool that was (and it did take a few minutes to eject all those disks afterwards), but it took less time than rebooting each node, for sure, and I had a desk full of spare floppy disks for two or three years after that. Matt - -- 812.855.7318 voice Research Technologies - High-Performance Systems hps-admin@iu.edu - http://rtinfo.uits.indiana.edu/hps/ On Jun 6, 2008, at 1:45 PM, Perry E. Metzger wrote: > > Bill Broadley writes: >>> 2. BIOS had a couple of interesting defaults, including warn on >>> keyboard error (Keyboard? Not intentionally. This is a compute >>> node, and should never require a keyboard. Ever.) We also find the >>> BIOS is set to boot from hard disk THEN PXE. But due to item 1, >>> above, we never can fail over to PXE unless we load up a keyboard >>> and monitor, and hit F12 to drop to PXE. >> >> Very strange standard for a server, let alone a cluster node. > > I would be less disturbed about such things if it was trivial to alter > the BIOS settings in a semi-automated way -- say by booting some > standalone program, or loading a file from a USB thumb drive. Then you > could just go up to each box with a USB thumb drive, turn it on, and > have it fix itself in a consistent way. However, the fact that you > can't generally automate fixing BIOS settings makes all of this far > more annoying. > > Anyone have any cool tricks for how to consistently set the BIOS on > large numbers of boxes without requiring steps that humans can screw > up easily? > > -- > Perry E. Metzger perry@piermont.com > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkhJfaMACgkQsHrhTcWK+IZ2GwCeOYae5FD3OrApTAJ3U2hPXfip BtEAnA9Ub3kkoKbFtNOcJgl7vHAi3KO2 =qlG4 -----END PGP SIGNATURE----- From bari at onelabs.com Fri Jun 6 12:14:28 2008 From: bari at onelabs.com (bari) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <9FD5E5B6-BCAF-4C00-A02B-A3678E46C77D@sanger.ac.uk> References: <48495A43.4060809@tamu.edu> <48496D2D.5080801@cse.ucdavis.edu> <87hcc6h2yi.fsf@snark.cb.piermont.com> <9FD5E5B6-BCAF-4C00-A02B-A3678E46C77D@sanger.ac.uk> Message-ID: <48498C94.1010608@onelabs.com> Tim Cutts wrote: > > Nope. :-) This is, in my view, one of the major disadvantages of PC > clusters. The crappy old BIOS that we're stuck with. > Just out of curiosity beside the clusters at LANL and Sandia who here uses coreboot (LinuxBIOS) for BIOS? http://www.coreboot.org If not, why not? Lack of vendor support? -Bari From spambox at emboss.co.nz Sat Jun 7 12:03:07 2008 From: spambox at emboss.co.nz (Michael Brown) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <87hcc6h2yi.fsf@snark.cb.piermont.com> References: <48495A43.4060809@tamu.edu> <48496D2D.5080801@cse.ucdavis.edu> <87hcc6h2yi.fsf@snark.cb.piermont.com> Message-ID: <5ECDF0CF16C448CCA570A7C5DCBDBA71@Forethought> Perry E. Metzger wrote: > Anyone have any cool tricks for how to consistently set the BIOS on > large numbers of boxes without requiring steps that humans can screw > up easily? Get a USB stick that boots into Linux. Set up one machine the way you want, then boot it up using the USB stick. Do: dd if=/dev/nvram of=cmos.bin For each oth the other machines, boot them using the stick and do: dd if=cmos.bin of=/dev/nvram From johnh at streamline-computing.com Mon Jun 9 06:55:17 2008 From: johnh at streamline-computing.com (johnh@streamline-computing.com) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <1213008547.8064.9.camel@bruce.priv.wark.uk.streamline-computing.com> References: <1213008547.8064.9.camel@bruce.priv.wark.uk.streamline-computing.com> Message-ID: <6c91fc1fbdbca86c512bb52ae3cfab4f@87.127.209.200> > On Fri, 2008-06-06 at 10:39 -0500, Gerry Creager wrote: >> > > I can think of at least one cluster where the opposite has been true and > PXE boot has been the default. The problem with this is if the head > node PXE boots on the customers network and gets automatically > re-installed as a windows workstation everybody gets egg on their face. > Yes even "modern" BIOSes are bad but localboot first is a sensible > default. Our clusters are set such that all compute nodes PXE boot first then localboot. All head nodes should have the BIOS set to locaboot first. From maurice at harddata.com Mon Jun 9 13:03:58 2008 From: maurice at harddata.com (Maurice Hilarius) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <200806091657.m59Gum9n021891@bluewest.scyld.com> References: <200806091657.m59Gum9n021891@bluewest.scyld.com> Message-ID: <484D8CAE.8000008@harddata.com> Chris Samuel wrote: > > Our most recent vendor went to the motherboard manufacturer > and said "please can you cut us a BIOS with these default > settings" and they did so. > > cheers, > Chris Some manufacturers do, some do not. Asus , for example, do, for their OEM customers. OTOH, one may buy the BIOS customization software , from AMI, for example, for a support/licensing fee of about $10,000 per year. -- With our best regards, //Maurice W. Hilarius Telephone: 01-780-456-9771/ /Hard Data Ltd. FAX: 01-780-456-9772/ /11060 - 166 Avenue email:maurice@harddata.com/ /Edmonton, AB, Canada http://www.harddata.com// / T5X 1Y3/ / -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080609/858c4f5e/attachment.html From mark.kosmowski at gmail.com Tue Jun 10 06:44:29 2008 From: mark.kosmowski at gmail.com (Mark Kosmowski) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] size of swap partition Message-ID: > Message: 5 > Date: Tue, 10 Jun 2008 00:58:12 -0400 (EDT) > From: Mark Hahn > Subject: Re: [Beowulf] size of swap partition > To: Gerry Creager > Cc: Mikhail Kuzminsky , beowulf@beowulf.org > Message-ID: > > Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed > > > We have the potential to have to swap whole jobs out of memory on a complete > > node. > > that was our intent as well. among other things, this scheme enables > running the cluster "split-personality" - mostly shorter/smaller even > interactive jobs during the day, with big/long jobs running at night. > unfortunately, you need a smart scheduler to do this, and ours is dumb. > > >> beleive, it is 2 or more GB per core; we have 16 GB per dual-socket > >> quad-core Opteron node). What is typical modern swap size today? > > are you willing to use a node which is actually occupying 16 GB of swap? > > it is possible to tune how the kernel responds to memory crunches - > for instance, you can always avoid OOM with the vm.overcommit_memory=2 > sysctl (you'll need to tune vm.overcommit_ratio and the amount of swap > to get the desired limits.) in this mode, the kernel tracks how much VM > it actually needs (worst-case, reflected in Committed_AS in /proc/meminfo) > and compares that to a commit limit that reflects ram and swap. > > if you don't use overcommit_memory=2, you are basically borrowing VM > space in hopes of not needing it. that can still be reasonable, considering > how often processes have a lot of shared VM, and how many processes > allocate but never touch lots of pages. but you have to ask yourself: > would I like a system that was actually _using_ 16 GB of swap? if you > have 16x disks, perhaps, but 16G will suck if you only have 1 disk. > at least for overcommit_memory != 2, I don't see the point of configuring > a lot of swap, since the only time you'd use it is if you were thrashing. > sort of a "quality of life" argument. > > >> But what are the reccomendations of modern praxis ? > > it depends a lot on the size variance of your jobs, as well as > their real/virtual ratio. the kernel only enforces RLIMIT_AS > (vsz in ps),assuming a 2.6 kernel - I forget whether 2.4 did > RLIMIT_RSS or not. > > if you use overcommit_memory=2, your desired max VM size determines > the amount of swap. otherwise, go with something modest - memory size > or so. but given that the smallest reasonable single disk these days > is probably about 320GB, it's hard to justify being _too_ tight. Is anyone using those Gigabyte i-RAM type devices from swap? Or is RAM cheaper? What about using these devices as swap to "add RAM" to older equipment that is at the maximum mobo supported RAM limit? From walid.shaari at gmail.com Tue Jun 10 09:27:43 2008 From: walid.shaari at gmail.com (Walid) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] size of swap partition In-Reply-To: References: Message-ID: Hi, For an 8GB dual socket quad core node, choosing in the kick start file --recommended instead of specifying size RHEL5 allocates 1GB of memory. our developers say that they should not swap as this will cause an overhead, and they try to avoid it as much as possible regards Walid On 10/06/2008, Mark Kosmowski wrote: >> Message: 5 >> Date: Tue, 10 Jun 2008 00:58:12 -0400 (EDT) >> From: Mark Hahn >> Subject: Re: [Beowulf] size of swap partition >> To: Gerry Creager >> Cc: Mikhail Kuzminsky , beowulf@beowulf.org >> Message-ID: >> >> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed >> >> > We have the potential to have to swap whole jobs out of memory on a >> > complete >> > node. >> >> that was our intent as well. among other things, this scheme enables >> running the cluster "split-personality" - mostly shorter/smaller even >> interactive jobs during the day, with big/long jobs running at night. >> unfortunately, you need a smart scheduler to do this, and ours is dumb. >> >> >> beleive, it is 2 or more GB per core; we have 16 GB per dual-socket >> >> quad-core Opteron node). What is typical modern swap size today? >> >> are you willing to use a node which is actually occupying 16 GB of swap? >> >> it is possible to tune how the kernel responds to memory crunches - >> for instance, you can always avoid OOM with the vm.overcommit_memory=2 >> sysctl (you'll need to tune vm.overcommit_ratio and the amount of swap >> to get the desired limits.) in this mode, the kernel tracks how much VM >> it actually needs (worst-case, reflected in Committed_AS in /proc/meminfo) >> and compares that to a commit limit that reflects ram and swap. >> >> if you don't use overcommit_memory=2, you are basically borrowing VM >> space in hopes of not needing it. that can still be reasonable, >> considering >> how often processes have a lot of shared VM, and how many processes >> allocate but never touch lots of pages. but you have to ask yourself: >> would I like a system that was actually _using_ 16 GB of swap? if you >> have 16x disks, perhaps, but 16G will suck if you only have 1 disk. >> at least for overcommit_memory != 2, I don't see the point of configuring >> a lot of swap, since the only time you'd use it is if you were thrashing. >> sort of a "quality of life" argument. >> >> >> But what are the reccomendations of modern praxis ? >> >> it depends a lot on the size variance of your jobs, as well as >> their real/virtual ratio. the kernel only enforces RLIMIT_AS >> (vsz in ps),assuming a 2.6 kernel - I forget whether 2.4 did >> RLIMIT_RSS or not. >> >> if you use overcommit_memory=2, your desired max VM size determines >> the amount of swap. otherwise, go with something modest - memory size >> or so. but given that the smallest reasonable single disk these days >> is probably about 320GB, it's hard to justify being _too_ tight. > > Is anyone using those Gigabyte i-RAM type devices from swap? Or is > RAM cheaper? What about using these devices as swap to "add RAM" to > older equipment that is at the maximum mobo supported RAM limit? > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From kus at free.net Tue Jun 10 10:35:46 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] size of swap partition In-Reply-To: Message-ID: In message from Mark Hahn (Tue, 10 Jun 2008 00:58:12 -0400 (EDT)): ... >for instance, you can always avoid OOM with the vm.overcommit_memory=2 >sysctl (you'll need to tune vm.overcommit_ratio and the amount of swap >to get the desired limits.) in this mode, the kernel tracks how much >VM >it actually needs (worst-case, reflected in Committed_AS in >/proc/meminfo) >and compares that to a commit limit that reflects ram and swap. > >if you don't use overcommit_memory=2, you are basically borrowing VM >space in hopes of not needing it. that can still be reasonable, >considering >how often processes have a lot of shared VM, and how many processes >allocate but never touch lots of pages. but you have to ask yourself: >would I like a system that was actually _using_ 16 GB of swap? if you >have 16x disks, perhaps, but 16G will suck if you only have 1 disk. >at least for overcommit_memory != 2, I don't see the point of >configuring >a lot of swap, since the only time you'd use it is if you were >thrashing. >sort of a "quality of life" argument. > >>> But what are the reccomendations of modern praxis ? > >it depends a lot on the size variance of your jobs, as well as their >real/virtual ratio. the kernel only enforces RLIMIT_AS >(vsz in ps),assuming a 2.6 kernel - I forget whether 2.4 did >RLIMIT_RSS or not. > >if you use overcommit_memory=2, your desired max VM size determines >the amount of swap. otherwise, go with something modest - memory size >or so. but given that the smallest reasonable single disk these days >is probably about 320GB, it's hard to justify being _too_ tight. :-) The disks we use in nodes is SATA WD/10K RPM w/70 GB :-)) We didn't set overcommit_memory=2, but really use strongly restricted scheduling police for SGE batch jobs using only few applications. We have only batch jobs (no interactive), moreover - practically only *long batch jobs*. As a result we have summary VM (requested per node) equal (or lower) than RAM. There is practically zero swap activity. The only exclusion are (seldom executed) small test jobs, non-parallelized, mainly for check of input data. They use small RAM amount. So it looks for me that I may set even lower than 1.5*RAM swap size (I think RAM+4G = 20G will be enough). In message from Walid (Tue, 10 Jun 2008 19:27:43 +0300): >Hi, >For an 8GB dual socket quad core node, choosing in the kick start >file --recommended instead of specifying size RHEL5 allocates 1GB of >memory. our developers say that they should not swap as this will >cause an overhead, and they try to avoid it as much as possible OpenSuSE 10.3 recommends swap size=2 GB only, but I don't know, performs SuSE inst software some estimation of server RAM or no. Yours Mikhail From csamuel at vpac.org Tue Jun 10 18:33:28 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <6c91fc1fbdbca86c512bb52ae3cfab4f@87.127.209.200> Message-ID: <1674199748.126391213148008004.JavaMail.root@zimbra.vpac.org> ----- johnh@streamline-computing.com wrote: > All head nodes should have the BIOS set to locaboot first. We set the interface on the internal cluster network to PXE and the external to not. Mind you, we control the external network too, so even if it did try it shouldn't do anything. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From csamuel at vpac.org Tue Jun 10 18:43:01 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: <436509742.126541213148491810.JavaMail.root@zimbra.vpac.org> Message-ID: <457290452.126561213148581328.JavaMail.root@zimbra.vpac.org> ----- "Jason Clinton" wrote: > The kernel patch is very extensive and, last I heard, under NDA. AMD post the patches publicly to the x86-64 discuss list. The most recent ones covered 2.6.24 and 2.6.25 and were sent out in April. https://www.x86-64.org/pipermail/discuss/2008-April/010398.html -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From Dan.Kidger at quadrics.com Thu Jun 12 04:38:36 2008 From: Dan.Kidger at quadrics.com (Dan.Kidger@quadrics.com) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] A couple of interesting comments In-Reply-To: <1674199748.126391213148008004.JavaMail.root@zimbra.vpac.org> References: <6c91fc1fbdbca86c512bb52ae3cfab4f@87.127.209.200> <1674199748.126391213148008004.JavaMail.root@zimbra.vpac.org> Message-ID: <0D49B15ACFDF2F46BF90B6E08C90048A04884916EC@quadbrsex1.quadrics.com> Chris Samuel wrote: >----- johnh@streamline-computing.com wrote: >> All head nodes should have the BIOS set to localboot first. > >We set the interface on the internal cluster network to >PXE and the external to not. I agree. but note that if you use ROCKS, it insists on the other way round: It wants to always reinstall a node on power up *unless* it has a working bootable partition, and it deliberately trashes the boot sector on a clean boot - only replacing it if the node is shut down cleanly. The key point is that any machine that has state that should be kept (like a head node) should *never* PXE boot by default - possibly not even if the primary HDD won't boot properly - only PXE boot on human intervention. PXE booting is concept of trust - that you trust the machine upstream of you to have full control of you, to delete and reinstall whatever it wants. Daniel ------------------------------------------------------------- Dr. Daniel Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Mobile: +44 (0)779 209 1851 Bristol, BS1 2AA, UK Office: +44 (0)117 915 5519 ----------------------- www.quadrics.com -------------------- From walid.shaari at gmail.com Thu Jun 12 06:32:38 2008 From: walid.shaari at gmail.com (Walid) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] RHEL5 network throughput/scalability Message-ID: Hi All, I have an issue with a new cluster setup where the nodes are RHEL5.1(with the latest 5.2 kernel), when i try to write NFS data, the nodes scale linearly until they reach the 10th node, that is the bandwidth , and throughput seen from the NFS sever on the other side of the nodes shows a liner increment from around 100+Mbyte/sec up to 1Gbyte/sec, however when we add another extra node to the equation the bandwidth/throughput becomes erratic/inconsistent, and drops to around 500-700Mbyte/sec. however if i try the same setup with RHEL4U6 i do not get the same behaviour it sustains the bandwidth at 1Gbyte/sec. the setup is like this 48 nodes sharing 48 port access switch that is up linked using 10g link to a CISCO 6509 switch which is linked to a Clustered NFS File system that consist of eight heads where each head linked using a 10G link to the 6509. the above was a write test, so i thought may be the tcp congestion kicked in, or sliding windows problem, however when i do a read test it gets worse, the scalability now is reduced to 5 nodes that is one node is able to read around 100 MBps, two will read double, and so on until you add the fifth node where the bandwidth drops from around 500+MBps to around 300, and again from RHEL4 the behaviour is different. any pointers? TIA Walid -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080612/4e8f86fa/attachment.html From garantes at iq.usp.br Tue Jun 10 06:53:15 2008 From: garantes at iq.usp.br (Guilherme Menegon Arantes) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] size of swap partition In-Reply-To: <200806101313.m5ADC2t3014136@bluewest.scyld.com> References: <200806101313.m5ADC2t3014136@bluewest.scyld.com> Message-ID: <20080610135315.GA4894@dinamobile> On Tue, Jun 10, 2008 at 06:13:00AM -0700, beowulf-request@beowulf.org wrote: > > Date: Tue, 10 Jun 2008 00:58:12 -0400 (EDT) > From: Mark Hahn > Subject: Re: [Beowulf] size of swap partition > To: Gerry Creager > Cc: Mikhail Kuzminsky , beowulf@beowulf.org > > it is possible to tune how the kernel responds to memory crunches - > for instance, you can always avoid OOM with the vm.overcommit_memory=2 > sysctl (you'll need to tune vm.overcommit_ratio and the amount of swap > to get the desired limits.) in this mode, the kernel tracks how much VM > it actually needs (worst-case, reflected in Committed_AS in /proc/meminfo) > and compares that to a commit limit that reflects ram and swap. > > ... > > their real/virtual ratio. the kernel only enforces RLIMIT_AS > (vsz in ps),assuming a 2.6 kernel - I forget whether 2.4 did > RLIMIT_RSS or not. And that brings me to another related question: Where I can get more information about VM usage/tunning options (such as vm.overcommit_ratio, RLIMIT_AS, etc) and VM metrics (such as vmstat) for the current linux kernels (>= 2.6.18)? I have looked on /usr/src/linux/Documentation/vm but anywhere else with more accessible/digested info? Regards, Guilherme -- Guilherme Menegon Arantes, PhD S?o Paulo, Brasil ______________________________________________________ From raq at cttc.upc.edu Tue Jun 10 10:33:30 2008 From: raq at cttc.upc.edu (Ramiro Alba Queipo) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Infiniband modular switches Message-ID: <1213119210.8051.143.camel@mundo> Hello everybody: We are about to build an HPC cluster with infiniband network starting from 22 dual socket nodes with AMD QUAD core processors and in a year or so we will be having about 120 nodes. We will be using infiniband both for calculation as for storage. The question is that we need a modular solution and we are having 3 candidates: a) Voltaire Grid Director SDR or DDR 288 ports (9988 or 2012 models)-> seems very good and well supported, but very expensive. b) Qlogic SilverStorm 9120 (144 ports) -> no price and support information yet c) Flextronics 10U 144 Port Modular-> very good at price but little support => risky option?. I am in a mess. What is your opinion about this matter? Are you using any of this products. Regards -- Aquest missatge ha estat analitzat per MailScanner a la cerca de virus i d'altres continguts perillosos, i es considera que està net. For all your IT requirements visit: http://www.transtec.co.uk From landman at scalableinformatics.com Thu Jun 12 07:36:32 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <1213119210.8051.143.camel@mundo> References: <1213119210.8051.143.camel@mundo> Message-ID: <48513470.30308@scalableinformatics.com> Ramiro Alba Queipo wrote: > Hello everybody: > > We are about to build an HPC cluster with infiniband network starting > from 22 dual socket nodes with AMD QUAD core processors and in a year or > so we will be having about 120 nodes. We will be using infiniband both > for calculation as for storage. Hi Ramiro: You may experience some contention issues in this case if your code is very latency sensitive, and you do lots of IO. > The question is that we need a modular solution and we are having 3 > candidates: > > a) Voltaire Grid Director SDR or DDR 288 ports (9988 or 2012 models)-> > seems very good and well supported, but very expensive. > > b) Qlogic SilverStorm 9120 (144 ports) -> no price and support > information yet > > c) Flextronics 10U 144 Port Modular-> very good at price but little > support => risky option?. The Flextronics units are Mellanox IP/chips inside (as are, I believe, many/most of the others). That is, the risk is low from a "will it work" view. Flextronics is an ODM, so they may not provide the levels of support around the system that you might get with Voltaire et al. Do you want/need a 1:1 architecture (e.g. all ports are the same number of switch hops from each other), or are you able/willing to look into oversubscribed links? Part of this has to do with your traffic patterns, your code requirements on latency, and your storage bandwidth. The Voltaire units are good, we have used them in units for customers. No complaints. Flextronics should be fine, as should Qlogic. We have customers with all of these. Rarely hear of complaints on IB switches. > > I am in a mess. What is your opinion about this matter? Are you using > any of this products. > > Regards > > -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From djholm at fnal.gov Thu Jun 12 08:08:21 2008 From: djholm at fnal.gov (Don Holmgren) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <1213119210.8051.143.camel@mundo> References: <1213119210.8051.143.camel@mundo> Message-ID: Ramiro - You might want to also consider buying just a single 24-port switch for your 22 nodes, and then when you expand either replace with a larger switch, or build a distributed switch fabric with a number of leaf switches connecting into a central spine switch (or switches). By the time you expand to the larger cluster, switches based on the announced 36-port Mellanox crossbar silicon will be available and perhaps per port prices will have dropped sufficiently to justify the purchase delay and the disruption at the time of expansion. If your applications can tolerate some oversubscription (less than a 1:1 ratio of leaf-to-spine uplinks to leaf-to-node connections), a distributed switch fabric (leaf and spine) has the advantage of shorter (and cheaper) cables between the leaf switches and your nodes, and relatively fewer longer cables from the leaves back to the spine, compared with a single central switch. We have many Flextronics switches - SDR and DDR, 24-port and 144-port - on a pair of large clusters (520 nodes, and 600 nodes) built in 2005 and 2006. No complaints. But, we have been self-supporting, and I would guess you would have very different support structures with Voltaire or Qlogic. With the Flextronics switches you will definitely be using the OFED stack, and you will have to run a subnet manager on one of your nodes (dedicated is probably best). You could optionally buy an embedded subnet manager on the Voltaire or Qlogic switches, depending upon model, though I believe for a large fabric an external subnet manager is still recommended. Don Holmgren Fermilab On Tue, 10 Jun 2008, Ramiro Alba Queipo wrote: > Hello everybody: > > We are about to build an HPC cluster with infiniband network starting > from 22 dual socket nodes with AMD QUAD core processors and in a year or > so we will be having about 120 nodes. We will be using infiniband both > for calculation as for storage. > The question is that we need a modular solution and we are having 3 > candidates: > > a) Voltaire Grid Director SDR or DDR 288 ports (9988 or 2012 models)-> > seems very good and well supported, but very expensive. > > b) Qlogic SilverStorm 9120 (144 ports) -> no price and support > information yet > > c) Flextronics 10U 144 Port Modular-> very good at price but little > support => risky option?. > > I am in a mess. What is your opinion about this matter? Are you using > any of this products. > > Regards From andrew at moonet.co.uk Thu Jun 12 08:51:11 2008 From: andrew at moonet.co.uk (andrew holway) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: References: <1213119210.8051.143.camel@mundo> Message-ID: +1 for the 24 port flextronics switches. They are very cost effective for half bisectional networks upto 32 ports. It starts to get messy after that. I wonder how long we will be waiting for switches based on the 36p asic? On Thu, Jun 12, 2008 at 4:08 PM, Don Holmgren wrote: > > Ramiro - > > You might want to also consider buying just a single 24-port switch for your > 22 nodes, and then when you expand either replace with a larger switch, or > build a distributed switch fabric with a number of leaf switches connecting > into a central spine switch (or switches). By the time you expand to the > larger cluster, switches based on the announced 36-port Mellanox crossbar > silicon will be available and perhaps per port prices will have dropped > sufficiently to justify the purchase delay and the disruption at the time of > expansion. > > If your applications can tolerate some oversubscription (less than a 1:1 > ratio of leaf-to-spine uplinks to leaf-to-node connections), a distributed > switch fabric (leaf and spine) has the advantage of shorter (and cheaper) > cables between the leaf switches and your nodes, and relatively fewer longer > cables from the leaves back to the spine, compared with a single central > switch. > > We have many Flextronics switches - SDR and DDR, 24-port and 144-port - on a > pair of large clusters (520 nodes, and 600 nodes) built in 2005 and 2006. No > complaints. But, we have been self-supporting, and I would guess you would > have very different support structures with Voltaire or Qlogic. With the > Flextronics > switches you will definitely be using the OFED stack, and you will have to > run > a subnet manager on one of your nodes (dedicated is probably best). You > could > optionally buy an embedded subnet manager on the Voltaire or Qlogic > switches, > depending upon model, though I believe for a large fabric an external subnet > manager is still recommended. > > Don Holmgren > Fermilab > > > > > On Tue, 10 Jun 2008, Ramiro Alba Queipo wrote: > >> Hello everybody: >> >> We are about to build an HPC cluster with infiniband network starting >> from 22 dual socket nodes with AMD QUAD core processors and in a year or >> so we will be having about 120 nodes. We will be using infiniband both >> for calculation as for storage. >> The question is that we need a modular solution and we are having 3 >> candidates: >> >> a) Voltaire Grid Director SDR or DDR 288 ports (9988 or 2012 models)-> >> seems very good and well supported, but very expensive. >> >> b) Qlogic SilverStorm 9120 (144 ports) -> no price and support >> information yet >> >> c) Flextronics 10U 144 Port Modular-> very good at price but little >> support => risky option?. >> >> I am in a mess. What is your opinion about this matter? Are you using >> any of this products. >> >> Regards > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From richard.walsh at comcast.net Thu Jun 12 09:12:34 2008 From: richard.walsh at comcast.net (richard.walsh@comcast.net) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Is PowerXCell eDP fully IEEE 754 compliant ... ?? ... the old Cell is/was ... Message-ID: <061220081612.24009.48514AF20003BC1C00005DC92215567074089C040E99D20B9D0E080C079D@comcast.net> All, I have not been able to get an exact answer to this question. The older chip, while much slower in double-precision was fully IEEE compliant I am fairly sure. I believe that IBM has improved the compliance of single-precision in the PowerXCell (although it is still not fully compliant), but its double- precision has fallen back to the single precision level of compliance to allow for the performance boost. This leaves it at close to parity in compliance when compared to the competition (ATI and NVIDIA) in the volume-economics DLP accelerator arena. Could someone that is certain of the answer to this question clarify? Best Regards, rbw -- "Making predictions is hard, especially about the future." Niels Bohr -- Richard Walsh Thrashing River Consulting-- 5605 Alameda St. Shoreview, MN 55126 Phone #: 612-382-4620 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080612/4f5f0370/attachment.html From Shainer at mellanox.com Thu Jun 12 10:01:11 2008 From: Shainer at mellanox.com (Gilad Shainer) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: Message-ID: <9FA59C95FFCBB34EA5E42C1A8573784F0129E5FB@mtiexch01.mti.com> > +1 for the 24 port flextronics switches. They are very cost effective > for half bisectional networks upto 32 ports. It starts to get > messy after that. > > I wonder how long we will be waiting for switches based on > the 36p asic? > Mellanox announced the availability of the switch asic this week, and can provide switch evaluation kits (36 port box and adapters with IB QDR capability) now. My estimation is that the production switches will be out Q3. Gilad. From andrew at moonet.co.uk Thu Jun 12 10:21:43 2008 From: andrew at moonet.co.uk (andrew holway) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <9FA59C95FFCBB34EA5E42C1A8573784F0129E5FB@mtiexch01.mti.com> References: <9FA59C95FFCBB34EA5E42C1A8573784F0129E5FB@mtiexch01.mti.com> Message-ID: > Mellanox announced the availability of the switch asic this week, and > can provide switch evaluation kits (36 port box and adapters with IB QDR > capability) now. My estimation is that the production switches will be > out Q3. Which vendor? From bill at cse.ucdavis.edu Thu Jun 12 11:04:19 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Barcelona hardware error: how to detect In-Reply-To: <588c11220806051046t3ad48a02q449a3ede0e299884@mail.gmail.com> References: <588c11220806051046t3ad48a02q449a3ede0e299884@mail.gmail.com> Message-ID: <48516523.9010406@cse.ucdavis.edu> > Yes, what are currently shipping from AMD are B3 revision processors. The > TLB-look-aside problem is fixed. > > There are other less-critical problems with B3, however. Specifically, > power-related compatibility issues with various motherboards due to > (according to the motherboard manufacturers) AMD changing the TDP late in > the release process. I can't give any specific names or models that we know > have problems, however. I can say that everyone involved is working on a > resolution--usually through PCB revisions of the motherboards. A number of > 1U power supplies that have previously worked with all Intel and AMD > solutions are now insufficient, as well, due to 12V limitations. B3 pulls a > *lot* of power. I've heard reports of b3 pulling more power than b2, not sure if that's just the higher clock speed, or a b3 related change. Has anyone put a dual socket B3 system on a kill-a-watt and tested it under load? From bernard at vanhpc.org Thu Jun 12 12:45:02 2008 From: bernard at vanhpc.org (Bernard Li) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Roadrunner picture Message-ID: Hi all: I am sure most people have seen the following picture for Roadrunner circulating the Net: http://www.cnn.com/2008/TECH/06/09/fastest.computer.ap/index.html?iref=newssearch However, they don't look likes blades to me, more like 2U IBM x series servers. Perhaps those are the I/O nodes? Cheers, Bernard From prentice at ias.edu Thu Jun 12 13:03:07 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] User resource limits In-Reply-To: <742070125.116971213070188917.JavaMail.root@zimbra.vpac.org> References: <742070125.116971213070188917.JavaMail.root@zimbra.vpac.org> Message-ID: <485180FB.2090402@ias.edu> Chris Samuel wrote: > > Unfortunately the kernel implementation of mmap() doesn't check > the maximum memory size (RLIMIT_RSS) or maximum data size (RLIMIT_DATA) > limits which were being set, but only the maximum virtual RAM size > (RLIMIT_AS) - this is documented in the setrlimit(2) man page. > > :-( > Yeah... I was just reading the setrlimit man page. Does that mean that the only way I can limit RAM usage is with RLIMIT_AS? (Or "as" in limits.conf parlance) I would have to limit AS < RAM to keep a user from using all RAM. Since AS includes virtual memory, and VM = RAM + swap, wouldn't I be limiting users a little more than I'd hoped? -- Prentice From prentice at ias.edu Thu Jun 12 13:07:18 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Roadrunner picture In-Reply-To: References: Message-ID: <485181F6.8030904@ias.edu> Bernard Li wrote: > Hi all: > > I am sure most people have seen the following picture for Roadrunner > circulating the Net: > > http://www.cnn.com/2008/TECH/06/09/fastest.computer.ap/index.html?iref=newssearch > > However, they don't look likes blades to me, more like 2U IBM x series > servers. Perhaps those are the I/O nodes? > Perhaps that is a poorly chosen file photo, and not really a photo of Roadrunner? Your I/O node theory is plausible, too. Is IBM getting away from the BlueGene architecture From john.leidel at gmail.com Thu Jun 12 13:09:32 2008 From: john.leidel at gmail.com (John Leidel) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Roadrunner picture In-Reply-To: References: Message-ID: <1213301372.5092.0.camel@e521.site> Also at ComputerWorld: http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9085021&intsrc=news_ts_head On Thu, 2008-06-12 at 12:45 -0700, Bernard Li wrote: > Hi all: > > I am sure most people have seen the following picture for Roadrunner > circulating the Net: > > http://www.cnn.com/2008/TECH/06/09/fastest.computer.ap/index.html?iref=newssearch > > However, they don't look likes blades to me, more like 2U IBM x series > servers. Perhaps those are the I/O nodes? > > Cheers, > > Bernard > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From peter.st.john at gmail.com Thu Jun 12 13:09:43 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Roadrunner picture In-Reply-To: References: Message-ID: Bernard, I'm looking forward to hearing from our resident experts, but meanwhile: http://en.wikipedia.org/wiki/IBM_Roadrunner exlains the architecture some. The buzzword is "triblade", which is 3 blades (with an extension) employing two types of processors (AMD Opteron and IBM Cell) in a hybrid subsystem. I have no idea what a single Triblade looks like. The overallmachine is then composed of zillions of triblades. Wow,imagine a Beowulf of those (jk :-) Peter (designing a Beowulf of abaci to fit his current budget) On Thu, Jun 12, 2008 at 3:45 PM, Bernard Li wrote: > Hi all: > > I am sure most people have seen the following picture for Roadrunner > circulating the Net: > > > http://www.cnn.com/2008/TECH/06/09/fastest.computer.ap/index.html?iref=newssearch > > However, they don't look likes blades to me, more like 2U IBM x series > servers. Perhaps those are the I/O nodes? > > Cheers, > > Bernard > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080612/19388c7f/attachment.html From prentice at ias.edu Thu Jun 12 13:58:54 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Roadrunner picture In-Reply-To: References: Message-ID: <48518E0E.8080604@ias.edu> Bernard Li wrote: > Hi all: > > I am sure most people have seen the following picture for Roadrunner > circulating the Net: > > http://www.cnn.com/2008/TECH/06/09/fastest.computer.ap/index.html?iref=newssearch > > However, they don't look likes blades to me, more like 2U IBM x series > servers. Perhaps those are the I/O nodes? > This might be what your seeing: "Each CU also has access to the Panasas file system through twelve System x3755 machines." - http://en.wikipedia.org/wiki/IBM_Roadrunner From richard.walsh at comcast.net Thu Jun 12 14:05:09 2008 From: richard.walsh at comcast.net (richard.walsh@comcast.net) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Roadrunner picture Message-ID: <061220082105.9640.48518F850002D8B0000025A82215586394089C040E99D20B9D0E080C079D@comcast.net> Skipped content of type multipart/alternative-------------- next part -------------- An embedded message was scrubbed... From: "Peter St. John" Subject: Re: [Beowulf] Roadrunner picture Date: Thu, 12 Jun 2008 20:16:19 +0000 Size: 762 Url: http://www.scyld.com/pipermail/beowulf/attachments/20080612/d2cdc94e/attachment.mht From jan.heichler at gmx.net Thu Jun 12 14:27:56 2008 From: jan.heichler at gmx.net (Jan Heichler) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] MVAPICH2 and osu_latency Message-ID: <6410235104.20080612232756@gmx.net> Dear all! I found this http://mvapich.cse.ohio-state.edu/performance/mvapich2/opteron/MVAPICH2-opteron-gen2-DDR.shtml as reference value for MPI-latency of Infiniband. I try to reproduce those numbers at the moment but i'm stuck with # OSU MPI Latency Test v3.0 # Size Latency (us) 0 3.07 1 3.17 2 3.16 4 3.15 8 3.19 Equipment is two quadsocket Opteron Blades (Supermicro) with Mellanox Ex DDR cards. Single 24 port switch connects them. Can anybody help with suggestions what i can do to lower the latency? Regards, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080612/999953c2/attachment.html From tom.elken at qlogic.com Thu Jun 12 15:04:29 2008 From: tom.elken at qlogic.com (Tom Elken) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] MVAPICH2 and osu_latency In-Reply-To: <6410235104.20080612232756@gmx.net> References: <6410235104.20080612232756@gmx.net> Message-ID: <6DB5B58A8E5AB846A7B3B3BFF1B4315A0214C01D@AVEXCH1.qlogic.org> So you're concerned with the gap between the 2.63 us that OSU measured and your 3.07 us you measured. I wouldn't be too concerned. MPI latency can be quite dependent on the systems you use. OSU used dual-processor 2.8 Ghz processors. Such as system has ~60 ns latency to local memory. On your 4-socket Opteron system, your local memory latency is probably in the 90-100 ns range. Assuming you are also using MVAPICH2, this is probably the main difference for the latency shortfall you are seeing. Another possibility is that the CPU you are running the MPI test on is not the closest CPU to the PCIe chipset. Thus, you may be taking some HT hops on the way to the PCIe bus and adapter card. -Tom ________________________________ From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Jan Heichler Sent: Thursday, June 12, 2008 2:28 PM To: Beowulf Mailing List Subject: [Beowulf] MVAPICH2 and osu_latency Dear all! I found this http://mvapich.cse.ohio-state.edu/performance/mvapich2/opteron/MVAPICH2- opteron-gen2-DDR.shtml as reference value for MPI-latency of Infiniband. I try to reproduce those numbers at the moment but i'm stuck with # OSU MPI Latency Test v3.0 # Size Latency (us) 0 3.07 1 3.17 2 3.16 4 3.15 8 3.19 Equipment is two quadsocket Opteron Blades (Supermicro) with Mellanox Ex DDR cards. Single 24 port switch connects them. Can anybody help with suggestions what i can do to lower the latency? Regards, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080612/16bf3067/attachment.html From hahn at mcmaster.ca Thu Jun 12 15:26:20 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] User resource limits In-Reply-To: <485180FB.2090402@ias.edu> References: <742070125.116971213070188917.JavaMail.root@zimbra.vpac.org> <485180FB.2090402@ias.edu> Message-ID: >> Unfortunately the kernel implementation of mmap() doesn't check >> the maximum memory size (RLIMIT_RSS) or maximum data size (RLIMIT_DATA) >> limits which were being set, but only the maximum virtual RAM size >> (RLIMIT_AS) - this is documented in the setrlimit(2) man page. >> >> :-( I think it's a perfectly reasonable choice. RSS enforcement means accounting and checks on what would otherwise be fast paths. besides, I think it also lacks transparency, since a process's RSS is affected by random other system events, other users, etc. using a memory limit that is triggered on actual allocation events (mmap, brk) makes a lot of sense to me, and that means virtual size, exactly what RLIMIT_AS does... > limits.conf parlance) I would have to limit AS < RAM to keep a user from > using all RAM. Since AS includes virtual memory, and VM = RAM + swap, > wouldn't I be limiting users a little more than I'd hoped? I don't follow that. why would you want to keep a user from using all ram (which assumes the ram is otherwise free/unused/wasted)? the only real trick with RLIMIT_AS and vm.overcommit=2 is that it's hard to predict the vsz of processes. normally, vsz is modestly larger than rss, but sysv shm, mmaped libries perturb this, as well as the dubious practice (more common in fortran I think) of allocating max-sized arrays even if you only ever use a small part. my experience so far is that setting RLIMIT_AS to around ram size is reasonable. we have had good luck with swap=ram (or a little more), vm.overcommit_memory=2 and vm.overcommit_ratio=100. the overcommit settings alone do a poor job - you also need RLIMIT_AS. regards, mark hahn. From kyron at neuralbs.com Thu Jun 12 15:32:13 2008 From: kyron at neuralbs.com (Eric Thibodeau) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] size of swap partition In-Reply-To: <20080610135315.GA4894@dinamobile> References: <200806101313.m5ADC2t3014136@bluewest.scyld.com> <20080610135315.GA4894@dinamobile> Message-ID: <4851A3ED.5030206@neuralbs.com> You can check out the following: http://linux-mm.org/LinuxMM Guilherme Menegon Arantes wrote: > On Tue, Jun 10, 2008 at 06:13:00AM -0700, beowulf-request@beowulf.org wrote: > >> Date: Tue, 10 Jun 2008 00:58:12 -0400 (EDT) >> From: Mark Hahn >> Subject: Re: [Beowulf] size of swap partition >> To: Gerry Creager >> Cc: Mikhail Kuzminsky , beowulf@beowulf.org >> >> it is possible to tune how the kernel responds to memory crunches - >> for instance, you can always avoid OOM with the vm.overcommit_memory=2 >> sysctl (you'll need to tune vm.overcommit_ratio and the amount of swap >> to get the desired limits.) in this mode, the kernel tracks how much VM >> it actually needs (worst-case, reflected in Committed_AS in /proc/meminfo) >> and compares that to a commit limit that reflects ram and swap. >> >> ... >> >> their real/virtual ratio. the kernel only enforces RLIMIT_AS >> (vsz in ps),assuming a 2.6 kernel - I forget whether 2.4 did >> RLIMIT_RSS or not. >> > > > And that brings me to another related question: Where I can get more > information about VM usage/tunning options (such as vm.overcommit_ratio, > RLIMIT_AS, etc) and VM metrics (such as vmstat) for the current linux > kernels (>= 2.6.18)? I have looked on /usr/src/linux/Documentation/vm > but anywhere else with more accessible/digested info? > > Regards, > > Guilherme > > -- > > Guilherme Menegon Arantes, PhD S?o Paulo, Brasil > ______________________________________________________ > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080612/8c8f7ca1/attachment.html From bernard at vanhpc.org Thu Jun 12 15:33:27 2008 From: bernard at vanhpc.org (Bernard Li) Date: Wed Nov 25 01:07:10 2009 Subject: [Beowulf] Roadrunner picture In-Reply-To: <061220082105.9640.48518F850002D8B0000025A82215586394089C040E99D20B9D0E080C079D@comcast.net> References: <061220082105.9640.48518F850002D8B0000025A82215586394089C040E99D20B9D0E080C079D@comcast.net> Message-ID: Dear all: Thanks for all the responses. I was at the Roadrunner booth at SC07. They had a handout explaining the Roadrunner architecture which also has a picture of racks of blades (maybe not of Roadrunner, but blades nevertheless). If I remember correctly they even have the blades on display. John's ComputerWorld link also has some pictures of the blades. So I guess I was just really trying to figure out what nodes those pictures are showing. Most likely the I/O nodes although there is also the off-chance that they are just random racks of servers ;-) Cheers, Bernard On Thu, Jun 12, 2008 at 2:05 PM, wrote: > All, > > Not a expert, but I know a thing or two. The triblade is two CB2 blades > which each hold each two PowerXCell processors in a cc-NUMA arrangement. > They sandwch a LS21 blade that is connected to each through a 16x PCIe to HT > bridge. These three are uni-body constructed. The CB2s resemble the QS22 > blade > that goes into the IBM BladCenter H chassis. They are vertical full-height > blades which fit 14 to an enclosure. The RoadRunner triblade is at least > double-wide > and maybe more. Do not know the measurements. > > The photo confuses me though, because am pretty sure these are vertically > racked. > Another thing to note is that programming the triblade is tri-binary ... > x86, Power, > and SPE. MPI processes are doled out to the Opteron blade. The PowerXCells > are programmed beneath MPI as SIMD accelrators. The systems processing > power is largely resident in the PowerXCell (~200 peak Gflops per CB2), the > Opteron only accounts for about 44 teraflops of the total peak performance > with is > in the vicinity of 1400 teraflops. Linpack runs at about 85% efficient on > the system > and is running on the SPE only I am pretty sure. Running Linpack it > generates 650 > Mflops per watt making it pretty Green I guess ... which is what you would > expect > from a DLP engine. As I recall Blue Gene is about 350 Mflops per watt. But > the > 650 number maybe does not count the LS21 power consumption. Anyway ... > > Hope that was useful ... now, can someone tell about the IEEE-754-ness of > its eDP > units? > > rbw > -- > > "Making predictions is hard, especially about the future." > > Niels Bohr > > -- > > Richard Walsh > Thrashing River Consulting-- > 5605 Alameda St. > Shoreview, MN 55126 > > Phone #: 612-382-4620 > > > -------------- Original message -------------- > From: "Peter St. John" > Bernard, > > I'm looking forward to hearing from our resident experts, but > meanwhile: http://en.wikipedia.org/wiki/IBM_Roadrunner exlains the > architecture some. The buzzword is "triblade", which is 3 blades (with an > extension) employing two types of processors (AMD Opteron and IBM Cell) in > a hybrid subsystem. I have no idea what a single Triblade looks like. The > overallmachine is then composed of zillions of triblades. > Wow,imagine a Beowulf of those (jk :-) > > Peter (designing a Beowulf of abaci to fit his current budget) > > On Thu, Jun 12, 2008 at 3:45 PM, Bernard Li wrote: >> >> Hi all: >> >> I am sure most people have seen the following picture for Roadrunner >> circulating the Net: >> >> >> http://www.cnn.com/2008/TECH/06/09/fastest.computer.ap/index.html?iref=newssearch >> >> However, they don't look likes blades to me, more like 2U IBM x series >> servers. Perhaps those are the I/O nodes? >> >> Cheers, >> >> Bernard >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > > > > ---------- Forwarded message ---------- > From: "Peter St. John" > To: "Bernard Li" > Date: Thu, 12 Jun 2008 20:16:19 +0000 > Subject: Re: [Beowulf] Roadrunner picture > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > From jan.heichler at gmx.net Thu Jun 12 20:06:58 2008 From: jan.heichler at gmx.net (Jan Heichler) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] MVAPICH2 and osu_latency In-Reply-To: <9FA59C95FFCBB34EA5E42C1A8573784F0129E69E@mtiexch01.mti.com> References: <6410235104.20080612232756@gmx.net> <9FA59C95FFCBB34EA5E42C1A8573784F0129E69E@mtiexch01.mti.com> Message-ID: <1745371589.20080613050658@gmx.net> An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080613/c8422ac6/attachment.html From jan.heichler at gmx.net Thu Jun 12 20:11:40 2008 From: jan.heichler at gmx.net (Jan Heichler) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] MVAPICH2 and osu_latency In-Reply-To: <6DB5B58A8E5AB846A7B3B3BFF1B4315A0214C01D@AVEXCH1.qlogic.org> References: <6410235104.20080612232756@gmx.net> <6DB5B58A8E5AB846A7B3B3BFF1B4315A0214C01D@AVEXCH1.qlogic.org> Message-ID: <1788564189.20080613051140@gmx.net> An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080613/a767e25f/attachment.html From apittman at concurrent-thinking.com Fri Jun 13 03:02:07 2008 From: apittman at concurrent-thinking.com (Ashley Pittman) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] MVAPICH2 and osu_latency In-Reply-To: <1788564189.20080613051140@gmx.net> References: <6410235104.20080612232756@gmx.net> <6DB5B58A8E5AB846A7B3B3BFF1B4315A0214C01D@AVEXCH1.qlogic.org> <1788564189.20080613051140@gmx.net> Message-ID: <1213351327.6775.18.camel@bruce.priv.wark.uk.streamline-computing.com> On Fri, 2008-06-13 at 05:11 +0200, Jan Heichler wrote: > > > > > So you're concerned with the gap > between the 2.63 us that OSU > measured and your 3.07 us you > measured. I wouldn't be too > concerned. > > 1st: i get a value of 2.96 with MVAPICH 1.0.0 - this is exactly the > value that i find on the mvapich website ;-) > > > It is not about being concerned not to get "optimal performance" - i > know that such micro-benchmarks are of limited use... but i have a > customer requirement. And since it seems possible it would be helpfull > to get there Rule #1. Don't commit yourself to a latency/bandwidth figure unless you have run the specific micro-benchmark *on the specific chipset/hardware configuration* you intend to deliver this commitment on. 0.4uS is well withing the bounds of fluctuation you get from different PCI chipsets. You're probably best of going back to the hardware vendor to see if there is anything they can tweak (mmtr perhaps?) but beyond that I suspect your problem will have political solution rather than a technical one. > The value is everytime the same. Shouldn't it be different then every > run? And: how can i move the process? numactl or taskset just works on > the local process i assume. How can i move the "remote process" on the > other host? Run numactl on the other host as well. That said it's unusual for the value to the be same every run, this probably won't change anything as all numactl can do is to stabilise the results towards the bottom of the range observed without it. Ashley Pittman. From prentice at ias.edu Fri Jun 13 05:38:13 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] User resource limits In-Reply-To: References: <742070125.116971213070188917.JavaMail.root@zimbra.vpac.org> <485180FB.2090402@ias.edu> Message-ID: <48526A35.6010609@ias.edu> Mark Hahn wrote: >>> Unfortunately the kernel implementation of mmap() doesn't check >>> the maximum memory size (RLIMIT_RSS) or maximum data size (RLIMIT_DATA) >>> limits which were being set, but only the maximum virtual RAM size >>> (RLIMIT_AS) - this is documented in the setrlimit(2) man page. >>> >>> :-( > > I think it's a perfectly reasonable choice. RSS enforcement means > accounting and checks on what would otherwise be fast paths. > besides, I think it also lacks transparency, since a process's RSS is > affected by random other system events, other users, etc. > > using a memory limit that is triggered on actual allocation events > (mmap, brk) makes a lot of sense to me, and that means virtual size, > exactly what RLIMIT_AS does... > >> limits.conf parlance) I would have to limit AS < RAM to keep a user from >> using all RAM. Since AS includes virtual memory, and VM = RAM + swap, >> wouldn't I be limiting users a little more than I'd hoped? > > I don't follow that. why would you want to keep a user from using all > ram (which assumes the ram is otherwise free/unused/wasted)? > Because these are multi-user systems that are not managed by a queuing system, and users are running large jobs on them. Once every couple of days, we have to hard-reboot one of them b/c they become unresponsive when they run out of memory (OOM messages in the logs verify this). I think I explained this in more detail in my original e-mail. > the only real trick with RLIMIT_AS and vm.overcommit=2 is that it's hard > to predict the vsz of processes. normally, vsz is modestly larger than > rss, > but sysv shm, mmaped libries perturb this, as well as the dubious practice > (more common in fortran I think) of allocating max-sized arrays even if > you only ever use a small part. > > my experience so far is that setting RLIMIT_AS to around ram size is > reasonable. we have had good luck with swap=ram (or a little more), > vm.overcommit_memory=2 and vm.overcommit_ratio=100. the overcommit > settings alone do a poor job - you also need RLIMIT_AS. vm.overcommit? never heard of that before. I'm going to google that now. -- Prentice From landman at scalableinformatics.com Fri Jun 13 06:11:33 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] User resource limits In-Reply-To: <48526A35.6010609@ias.edu> References: <742070125.116971213070188917.JavaMail.root@zimbra.vpac.org> <485180FB.2090402@ias.edu> <48526A35.6010609@ias.edu> Message-ID: <48527205.1030400@scalableinformatics.com> Prentice Bisbal wrote: > vm.overcommit? never heard of that before. I'm going to google that now. vm tuning knobs/dials /proc/sys/vm/overcommit_memory /proc/sys/vm/overcommit_ratio or via sysctl landman@lightning:~$ sysctl -a | grep -i overcommit ... vm.overcommit_memory = 0 vm.overcommit_ratio = 50 -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From raq at cttc.upc.edu Fri Jun 13 08:19:28 2008 From: raq at cttc.upc.edu (Ramiro Alba Queipo) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <48513470.30308@scalableinformatics.com> References: <1213119210.8051.143.camel@mundo> <48513470.30308@scalableinformatics.com> Message-ID: <1213370368.5895.46.camel@mundo> On Thu, 2008-06-12 at 10:36 -0400, Joe Landman wrote: > Ramiro Alba Queipo wrote: > > Hello everybody: > > > > We are about to build an HPC cluster with infiniband network starting > > from 22 dual socket nodes with AMD QUAD core processors and in a year or > > so we will be having about 120 nodes. We will be using infiniband both > > for calculation as for storage. > > Hi Ramiro: > > You may experience some contention issues in this case if your code > is very latency sensitive, and you do lots of IO. Our software is home-made on CFD using MPI (lam until now and openmpi from now on) but our solvers are neither very latency sensitive nor do lots of IO at this moment, so I think that now it is a sensible desition ?What do you think? > > > The question is that we need a modular solution and we are having 3 > > candidates: > > > > a) Voltaire Grid Director SDR or DDR 288 ports (9988 or 2012 models)-> > > seems very good and well supported, but very expensive. > > > > b) Qlogic SilverStorm 9120 (144 ports) -> no price and support > > information yet > > > > c) Flextronics 10U 144 Port Modular-> very good at price but little > > support => risky option?. > > The Flextronics units are Mellanox IP/chips inside (as are, I believe, > many/most of the others). That is, the risk is low from a "will it > work" view. Flextronics is an ODM, so they may not provide the levels > of support around the system that you might get with Voltaire et al. > > Do you want/need a 1:1 architecture (e.g. all ports are the same number > of switch hops from each other), or are you able/willing to look into > oversubscribed links? Part of this has to do with your traffic > patterns, your code requirements on latency, and your storage bandwidth. I do not know many details, but we are using nowadays solvers adapted to live with high latencies, and with infiniband we expect to scale better and we expect to run tasks using 500 (8 cores/node) or more cores (thought not right now). In fact we are doing some test at Marenostrum supercomputer in Barcelona with about 1000 cores. The question that worries me is if we will be limited at mid-term by a solution based on 24 ports switches joined by say 4 ports (not a fat-tree topology which waste a lot more of ports), and be loosing a latency/bandwidth that we then will be needing when having 130 ports at the end of next year. By the way: a) How many hops a Flextronics 10U 144 Port Modular is doing? b) And the others? c) How much latency am I loosing in each hop? (In the case of Voltaire switches: ISR 9024 - 24 Ports: 140 ns ; ISR 2004 - 96 ports: 420 ns d) Each port I am using to connect a switch to another one is summing up its bandwidth to the total (20 Gb/s * 4 = 80 Gbs when using 4 ports to connect) The alternatives are: a) Start with a good 24 port swith and grow up loosing latency and bandwidth b) Buy a 48 or 96 ports spending more money to have more ports at full bandwidth/latency c) Use the Flextronix 10U 144 Port Modular solution which will allow us to scale well in a couple years Thanks for your answer Regards -- Aquest missatge ha estat analitzat per MailScanner a la cerca de virus i d'altres continguts perillosos, i es considera que est? net. For all your IT requirements visit: http://www.transtec.co.uk From raq at cttc.upc.edu Fri Jun 13 08:28:06 2008 From: raq at cttc.upc.edu (Ramiro Alba Queipo) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: References: <1213119210.8051.143.camel@mundo> Message-ID: <1213370886.5895.53.camel@mundo> On Thu, 2008-06-12 at 10:08 -0500, Don Holmgren wrote: > Ramiro - > > You might want to also consider buying just a single 24-port switch for your 22 > nodes, and then when you expand either replace with a larger switch, or build a > distributed switch fabric with a number of leaf switches connecting into a > central spine switch (or switches). By the time you expand to the larger > cluster, switches based on the announced 36-port Mellanox crossbar silicon will > be available and perhaps per port prices will have dropped sufficiently to > justify the purchase delay and the disruption at the time of expansion. Could you explain me this solution? I did not know about it > > If your applications can tolerate some oversubscription (less than a 1:1 ratio > of leaf-to-spine uplinks to leaf-to-node connections), a distributed switch > fabric (leaf and spine) has the advantage of shorter (and cheaper) cables > between the leaf switches and your nodes, and relatively fewer longer cables > from the leaves back to the spine, compared with a single central switch. What do you mean with a distributed switch fabric? What is the difference with a modular solution? Thanks for your answer Regards -- Aquest missatge ha estat analitzat per MailScanner a la cerca de virus i d'altres continguts perillosos, i es considera que està net. For all your IT requirements visit: http://www.transtec.co.uk From egan at sense.net Fri Jun 13 08:43:31 2008 From: egan at sense.net (Egan Ford) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Roadrunner picture In-Reply-To: Message-ID: <02f001c8cd6c$3c5c1990$4f833909@oberon> Perhaps this will help: http://www.lanl.gov/roadrunner/ And: http://www.lanl.gov/orgs/hpc/roadrunner/pdfs/Koch%20-%20Roadrunner%20Overvie w/RR%20Seminar%20-%20System%20Overview.pdf Pages 20 - 29 IANS, the triblade is really a quadblade, blade 1 is the Opteron Blade, blade 2 is a bridge, blades 3 and 4 are the Cell blades. Lots of other good stuff here: http://www.lanl.gov/orgs/hpc/roadrunner/rrseminars.shtml > -----Original Message----- > From: beowulf-bounces@beowulf.org > [mailto:beowulf-bounces@beowulf.org] On Behalf Of Bernard Li > Sent: Thursday, June 12, 2008 4:33 PM > To: richard.walsh@comcast.net > Cc: Beowulf Mailing List > Subject: Re: [Beowulf] Roadrunner picture > > > Dear all: > > Thanks for all the responses. I was at the Roadrunner booth > at SC07. They had a handout explaining the Roadrunner > architecture which also has a picture of racks of blades > (maybe not of Roadrunner, but blades nevertheless). If I > remember correctly they even have the blades on display. > > John's ComputerWorld link also has some pictures of the blades. > > So I guess I was just really trying to figure out what nodes > those pictures are showing. Most likely the I/O nodes > although there is also the off-chance that they are just > random racks of servers ;-) > > Cheers, > > Bernard > > On Thu, Jun 12, 2008 at 2:05 PM, wrote: > > All, > > > > Not a expert, but I know a thing or two. The triblade is two CB2 > > blades which each hold each two PowerXCell processors in a cc-NUMA > > arrangement. They sandwch a LS21 blade that is connected to each > > through a 16x PCIe to HT bridge. These three are uni-body > constructed. > > The CB2s resemble the QS22 blade that goes into the IBM > BladCenter H > > chassis. They are vertical full-height blades which fit 14 to an > > enclosure. The RoadRunner triblade is at least double-wide > > and maybe more. Do not know the measurements. > > > > The photo confuses me though, because am pretty sure these are > > vertically racked. Another thing to note is that programming the > > triblade is tri-binary ... x86, Power, > > and SPE. MPI processes are doled out to the Opteron blade. > The PowerXCells > > are programmed beneath MPI as SIMD accelrators. The > systems processing > > power is largely resident in the PowerXCell (~200 peak > Gflops per CB2), the > > Opteron only accounts for about 44 teraflops of the total > peak performance > > with is > > in the vicinity of 1400 teraflops. Linpack runs at about > 85% efficient on > > the system > > and is running on the SPE only I am pretty sure. Running Linpack it > > generates 650 > > Mflops per watt making it pretty Green I guess ... which is > what you would > > expect > > from a DLP engine. As I recall Blue Gene is about 350 > Mflops per watt. But > > the > > 650 number maybe does not count the LS21 power consumption. > Anyway ... > > > > Hope that was useful ... now, can someone tell about the > IEEE-754-ness > > of its eDP units? > > > > rbw > > -- > > > > "Making predictions is hard, especially about the future." > > > > Niels Bohr > > > > -- > > > > Richard Walsh > > Thrashing River Consulting-- > > 5605 Alameda St. > > Shoreview, MN 55126 > > > > Phone #: 612-382-4620 > > > > > > -------------- Original message -------------- > > From: "Peter St. John" > > Bernard, > > > > I'm looking forward to hearing from our resident experts, but > > meanwhile: http://en.wikipedia.org/wiki/IBM_Roadrunner exlains the > > architecture some. The buzzword is "triblade", which is 3 blades > > (with an > > extension) employing two types of processors (AMD Opteron > and IBM Cell) in > > a hybrid subsystem. I have no idea what a single Triblade > looks like. The > > overallmachine is then composed of zillions of triblades. > > Wow,imagine a Beowulf of those (jk :-) > > > > Peter (designing a Beowulf of abaci to fit his current budget) > > > > On Thu, Jun 12, 2008 at 3:45 PM, Bernard Li > > wrote: > >> > >> Hi all: > >> > >> I am sure most people have seen the following picture for > Roadrunner > >> circulating the Net: > >> > >> > >> > http://www.cnn.com/2008/TECH/06/09/fastest.computer.ap/index.html?ire > >> f=newssearch > >> > >> However, they don't look likes blades to me, more like 2U IBM x > >> series servers. Perhaps those are the I/O nodes? > >> > >> Cheers, > >> > >> Bernard > >> _______________________________________________ > >> Beowulf mailing list, Beowulf@beowulf.org > >> To change your subscription (digest mode or unsubscribe) visit > >> http://www.beowulf.org/mailman/listinfo/beowulf > > > > > > > > ---------- Forwarded message ---------- > > From: "Peter St. John" > > To: "Bernard Li" > > Date: Thu, 12 Jun 2008 20:16:19 +0000 > > Subject: Re: [Beowulf] Roadrunner picture > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) > visit http://www.beowulf.org/mailman/listinfo/beowulf From jan.heichler at gmx.net Fri Jun 13 08:55:10 2008 From: jan.heichler at gmx.net (Jan Heichler) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <1213370368.5895.46.camel@mundo> References: <1213119210.8051.143.camel@mundo> <48513470.30308@scalableinformatics.com> <1213370368.5895.46.camel@mundo> Message-ID: <1589632152.20080613175510@gmx.net> Hallo Ramiro, Freitag, 13. Juni 2008, meintest Du: RAQ> By the way: RAQ> a) How many hops a Flextronics 10U 144 Port Modular is doing? 3 RAQ> b) And the others? 3 too. RAQ> c) How much latency am I loosing in each hop? (In the case of Voltaire RAQ> switches: ISR 9024 - 24 Ports: 140 ns ; ISR 2004 - 96 ports: 420 ns 150 ns is the usual value - maybe it is just 140. As far as i know all the switch vendors use the samel 24-port silicon by Mellanox. Because of that you find the same number of switchports everywhere: 24, 144 and 288. 288 is the maximum number of ports you can get when you use a 24port silicon as basis and build a 3-HOP Fat-Tree. RAQ> The alternatives are: RAQ> a) Start with a good 24 port swith and grow up loosing latency and RAQ> bandwidth You can use the 24-port switches to create a full bisectional bandwidth network if you want that. Since all the big switches are based on the 24-port silicon this is no problem. RAQ> c) Use the Flextronix 10U 144 Port Modular solution which will allow us RAQ> to scale well in a couple years But you have to pay now for an expansion that you might never get... Or want a fancy new network technology as soon as you upgrade. Regards, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080613/2af742f7/attachment.html From raq at cttc.upc.edu Fri Jun 13 09:52:45 2008 From: raq at cttc.upc.edu (Ramiro Alba Queipo) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <1589632152.20080613175510@gmx.net> References: <1213119210.8051.143.camel@mundo> <48513470.30308@scalableinformatics.com> <1213370368.5895.46.camel@mundo> <1589632152.20080613175510@gmx.net> Message-ID: <1213375965.5895.66.camel@mundo> On Fri, 2008-06-13 at 17:55 +0200, Jan Heichler wrote: > Hallo Ramiro, > > > RAQ> The alternatives are: > > > RAQ> a) Start with a good 24 port swith and grow up loosing latency > and > > RAQ> bandwidth > > > You can use the 24-port switches to create a full bisectional > bandwidth network if you want that. Since all the big switches are > based on the 24-port silicon this is no problem. Yes, But How many ports must I waste if I do not want to loose to much bandwidth. And about latency, It could be negligible compared to latency at the infiniband card (~3 microsecs in front of 480 nanosecs at switch), right? > > > RAQ> c) Use the Flextronix 10U 144 Port Modular solution which will > allow us > > RAQ> to scale well in a couple years > > > But you have to pay now for an expansion that you might never get... > Or want a fancy new network technology as soon as you upgrade. Well, I most probably will be needing 48 ports in five months and 120 ports in about a year Thanks for your answer Regards -- Aquest missatge ha estat analitzat per MailScanner a la cerca de virus i d'altres continguts perillosos, i es considera que està net. For all your IT requirements visit: http://www.transtec.co.uk From walid.shaari at gmail.com Fri Jun 13 10:31:38 2008 From: walid.shaari at gmail.com (Walid) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] RHEL5 network throughput/scalability In-Reply-To: <588c11220806130856j34c2949dq3ac538ecc4409ca0@mail.gmail.com> References: <588c11220806130856j34c2949dq3ac538ecc4409ca0@mail.gmail.com> Message-ID: 2008/6/13 Jason Clinton : > > We've seen fairly erratic behavior induced by newer drivers for NVidia > NForce-based NIC's with forcedeth. If that's your source NIC in the above > scenario, that could be the source of the issue as congestion timing has > probably changed. Have you tried updating your source NIC driver to > whichever is the newest? Nearly all NIC vendors that are incorporated on > server motherboards put out updated drivers on their websites. > > Jason, The NIC is broadcom (Bnx2 driver), I have, however i had to go to the RHEL5.1 kernel as it does not want to compile on the new kernels. however it did not change much, i have also played with the different congestion settings, and even though Vegas seems to be the most performance capable, it still did not solve it, I had one test that i did not write down where it did perform well, I have saved the sysctl and will check what parameters have made the difference. regards Walid -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080613/b7107d80/attachment.html From jan.heichler at gmx.net Fri Jun 13 10:43:31 2008 From: jan.heichler at gmx.net (Jan Heichler) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <1213375965.5895.66.camel@mundo> References: <1213119210.8051.143.camel@mundo> <48513470.30308@scalableinformatics.com> <1213370368.5895.46.camel@mundo> <1589632152.20080613175510@gmx.net> <1213375965.5895.66.camel@mundo> Message-ID: <9410452169.20080613194331@gmx.net> Hallo Ramiro, Freitag, 13. Juni 2008, meintest Du: RAQ> On Fri, 2008-06-13 at 17:55 +0200, Jan Heichler wrote: >> You can use the 24-port switches to create a full bisectional >> bandwidth network if you want that. Since all the big switches are >> based on the 24-port silicon this is no problem. RAQ> Yes, But How many ports must I waste if I do not want to loose to much RAQ> bandwidth. Exactly 50%. 12 Ports go to clients and 12 ports go to the spine. I can send you a sketch if you are interested. RAQ> And about latency, It could be negligible compared to latency at the RAQ> infiniband card (~3 microsecs in front of 480 nanosecs at switch), RAQ> right? The number of HOPs is always the same... 3 up to 288 ports. With the new 36 port silicon that will change. Up to 648 Clients with 3 HOPs if i'm not wrong. >> RAQ> c) Use the Flextronix 10U 144 Port Modular solution which will >> allow us >> RAQ> to scale well in a couple years >> But you have to pay now for an expansion that you might never get... >> Or want a fancy new network technology as soon as you upgrade. RAQ> Well, I most probably will be needing 48 ports in five months and 120 RAQ> ports in about a year 48 Ports is 6 24 Port Switches. here is a quick overview: client ports #of 24 port switches you need for full bisectional bandwidth 24 1 36 5 48 6 60 8 72 9 84 11 96 12 108 14 120 15 132 16 144 18 Somewhere around 11 and 14 is normally the break even for a 144-port switch - depends on your cost prices of course ;-) I hope i didn't miscalculate ;-) RAQ> Thanks for your answer Any time! Regards, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080613/a7ab7f56/attachment.html From djholm at fnal.gov Fri Jun 13 12:05:22 2008 From: djholm at fnal.gov (Don Holmgren) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <1213370886.5895.53.camel@mundo> References: <1213119210.8051.143.camel@mundo> <1213370886.5895.53.camel@mundo> Message-ID: On Fri, 13 Jun 2008, Ramiro Alba Queipo wrote: > On Thu, 2008-06-12 at 10:08 -0500, Don Holmgren wrote: >> Ramiro - >> >> You might want to also consider buying just a single 24-port switch for your 22 >> nodes, and then when you expand either replace with a larger switch, or build a >> distributed switch fabric with a number of leaf switches connecting into a >> central spine switch (or switches). By the time you expand to the larger >> cluster, switches based on the announced 36-port Mellanox crossbar silicon will >> be available and perhaps per port prices will have dropped sufficiently to >> justify the purchase delay and the disruption at the time of expansion. > > Could you explain me this solution? I did not know about it As far as I know, all currently available commercial Infiniband switches are based on the Mellanox 24-port non-blocking silicon switch chip (InfiniScale III). The 96, 144, and 288 port modular switches from the various companies use a number of these individual chips in a layered (3-hop) design that provides full bisection bandwidth. One can also construct a full bisection bandwidth 144-port (say) switch out of twelve 24-port switches: out of the total 288 switch ports, 144 ports connect to nodes, and 144 ports connect to other switch ports. The latency should be identical to that of a 144-port chassis, as both would use three hops (disregarding the negligible ~ nanosecond per foot of extra cable length delay when using 24-port switches). Usually the per port cost for a large switch is less than the per port cost for a bunch of 24-port switches. When you don't need a full 144-port switch, you can either buy the large chassis and only buy a limited number of blades, or go with a set of 24-port switches. For smaller networks a set of 24-port switches is cheaper. The next generation switch silicon will be 36 ports (InfiniScale IV), rather than 24. Obviously I can't predict for certain that the large switches to be built out of this silicon will be cheaper than the current models, but it is reasonable to guess that this will be the case. > >> >> If your applications can tolerate some oversubscription (less than a 1:1 ratio >> of leaf-to-spine uplinks to leaf-to-node connections), a distributed switch >> fabric (leaf and spine) has the advantage of shorter (and cheaper) cables >> between the leaf switches and your nodes, and relatively fewer longer cables >> from the leaves back to the spine, compared with a single central switch. > > > What do you mean with a distributed switch fabric? > What is the difference with a modular solution? > > Thanks for your answer > > Regards I think both of these questions are answered above. But to be clear, by "distributed" I mean that instead of one large switch chassis one would use a number of 24-port switches. In this case it is very natural to put the individual switches next to their nodes. See, for example, the "A New Approach to Clustering - Distributed Federated Switches" white paper at the Mellanox web site. When the switches are next to the nodes, the cable plant can be a lot easier to deal with. Don't underestimate the pain of having 144 fairly hefty Infiniband cables all terminating into a 10U chassis. One additional item of note when using a distributed fabric: if your typical jobs use a small number of nodes, then it is quite possible to configure your batch scheduler so that the nodes belonging to an individual job all connect to the same leaf switch. This means that your messages only have to go through one switch hop, so latency is reduced compared with going through three hops in a large modular switch chassis (although I seriously doubt that the quarter microsecond of latency difference here matters to many codes). Perhaps of more significance, though, is that you can use oversubscription to lower the cost of your fabric. Instead of connecting 12 ports of a leaf switch to nodes and using the other 12 ports as uplinks, you might get away with 18 nodes and 6 uplinks, or 20 nodes and 4 uplinks. As core counts are increasing, this is becoming more and more viable for some applications. Don From patrick at myri.com Fri Jun 13 12:43:31 2008 From: patrick at myri.com (Patrick Geoffray) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: References: <1213119210.8051.143.camel@mundo> <1213370886.5895.53.camel@mundo> Message-ID: <4852CDE3.4030006@myri.com> Hi Don, Don Holmgren wrote: > latency difference here matters to many codes). Perhaps of more > significance, though, is that you can use oversubscription to lower the > cost of your fabric. Instead of connecting 12 ports of a leaf switch to > nodes and using the other 12 ports as uplinks, you might get away with > 18 nodes and 6 uplinks, or 20 nodes and 4 uplinks. As core counts are > increasing, this is becoming more and more viable for some applications. It's important to note that the "full-bisection" touted by vendors is on paper only. In reality, static routing provides full-bisection for a very small subset of patterns, the average effective bisection on a diameter-3 Clos is ~40% of link rate (adaptive routing improves that a lot, but breaks packet order on the wire which is a requirement for some network protocols). In practice, "paper" full-bisection is near free when using a single enclosure, since all spine cables are on the backplane. For larger networks, where you have to pay for real cables to the spine level, then it may make sense to be oversubscribed if the effective bisection is already bad (static routing), or if your collective communication on large jobs are not bandwidth bounded. However, the later is often false on many-cores. Patrick From walid.shaari at gmail.com Fri Jun 13 12:55:23 2008 From: walid.shaari at gmail.com (Walid) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] RHEL5 network throughput/scalability In-Reply-To: References: <588c11220806130856j34c2949dq3ac538ecc4409ca0@mail.gmail.com> Message-ID: Dear All, It is lame, however i managed to get the following kernel paramter to scale well in terms of both performance per node, and scalability over a high bandwidth low latency network net.ipv4.tcp_workaround_signed_windows = 1 net.ipv4.tcp_congestion_control = vegas net.ipv4.tcp_tso_win_divisor = 8 net.ipv4.tcp_rmem = 4096 87380 174760 net.ipv4.tcp_wmem = 4096 16384 131072 net.ipv4.tcp_mem = 786432 1048576 1572864 net.ipv4.route.max_size = 8388608 net.ipv4.route.gc_thresh = 524288 net.ipv4.icmp_ignore_bogus_error_responses = 0 net.ipv4.icmp_echo_ignore_broadcasts = 0 net.ipv4.tcp_max_orphans = 262144 net.core.netdev_max_backlog = 2000 regards Walid 2008/6/13 Walid : > 2008/6/13 Jason Clinton : > >> >> We've seen fairly erratic behavior induced by newer drivers for NVidia >> NForce-based NIC's with forcedeth. If that's your source NIC in the above >> scenario, that could be the source of the issue as congestion timing has >> probably changed. Have you tried updating your source NIC driver to >> whichever is the newest? Nearly all NIC vendors that are incorporated on >> server motherboards put out updated drivers on their websites. >> >> Jason, > > The NIC is broadcom (Bnx2 driver), I have, however i had to go to the > RHEL5.1 kernel as it does not want to compile on the new kernels. however it > did not change much, i have also played with the different congestion > settings, and even though Vegas seems to be the most performance capable, > it still did not solve it, I had one test that i did not write down where it > did perform well, I have saved the sysctl and will check what parameters > have made the difference. > > regards > > Walid -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080613/8080d153/attachment.html From prentice at ias.edu Fri Jun 13 13:03:49 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] User resource limits In-Reply-To: <20080609161232.GB11155@nlxdcldnl2.cl.intel.com> References: <484D4F29.9090704@ias.edu> <20080609161232.GB11155@nlxdcldnl2.cl.intel.com> Message-ID: <4852D2A5.1050803@ias.edu> Lombard, David N wrote: > On Mon, Jun 09, 2008 at 11:41:29AM -0400, Prentice Bisbal wrote: >> I would like to impose some CPU and memory limits on users that are hard >> limits that can't be changed/overridden by the users. What is the best >> way to do this? All I know is environment variables or shell commands >> done as the user (ulimit, for example). > > pam_limits and /etc/security/limits.conf > In limits.conf, what is the units for as, is it bytes, or kb? It doesn't say in the man page for limits.conf, and the page for setrlimit says bytes RLIMIT_AS. -- Prentice From dnlombar at ichips.intel.com Fri Jun 13 15:58:40 2008 From: dnlombar at ichips.intel.com (Lombard, David N) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] User resource limits In-Reply-To: <4852D2A5.1050803@ias.edu> References: <484D4F29.9090704@ias.edu> <20080609161232.GB11155@nlxdcldnl2.cl.intel.com> <4852D2A5.1050803@ias.edu> Message-ID: <20080613225840.GA8097@nlxdcldnl2.cl.intel.com> On Fri, Jun 13, 2008 at 04:03:49PM -0400, Prentice Bisbal wrote: > Lombard, David N wrote: > > On Mon, Jun 09, 2008 at 11:41:29AM -0400, Prentice Bisbal wrote: > >> I would like to impose some CPU and memory limits on users that are hard > >> limits that can't be changed/overridden by the users. What is the best > >> way to do this? All I know is environment variables or shell commands > >> done as the user (ulimit, for example). > > > > pam_limits and /etc/security/limits.conf > > > In limits.conf, what is the units for as, is it bytes, or kb? It doesn't > say in the man page for limits.conf, and the page for setrlimit says > bytes RLIMIT_AS. As does the kernel's mm/mmap.c and, for >=2.6.25, fs/proc/base.c Note, mmap.c has been validating memory requests against RLIMIT_AS since at least 2.6.3 (earliest kernel sources I have online); 2.6.25 includes a /proc file "limits" that displays the limits (see fs/proc/base.c). HTH -- David N. Lombard, Intel, Irvine, CA I do not speak for Intel Corporation; all comments are strictly my own. From perry at piermont.com Fri Jun 13 19:21:05 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] RHEL5 network throughput/scalability In-Reply-To: (Walid's message of "Fri\, 13 Jun 2008 22\:55\:23 +0300") References: <588c11220806130856j34c2949dq3ac538ecc4409ca0@mail.gmail.com> Message-ID: <87od64lpta.fsf@snark.cb.piermont.com> A number of these seem rather odd, or unrelated to performance. Walid writes: > It is lame, however i managed to get the following kernel paramter to scale > well in terms of both performance per node, and scalability over a high > bandwidth low latency network > > net.ipv4.tcp_workaround_signed_windows = 1 This is a workaround for a buggy remote TCP. If you have a homogeneous network of linux boxes, it will have no effect. > net.ipv4.tcp_congestion_control = vegas I'm under the impression that the Vegas congestion control policy is not well loved by the experts on TCP performance. > net.ipv4.route.max_size = 8388608 This sets the size of the routing cache. You've set it to a rather large and fairly random number. > net.ipv4.icmp_ignore_bogus_error_responses = 0 > net.ipv4.icmp_echo_ignore_broadcasts = 0 Why would paying attention to bogus ICMPs and to ICMP broadcasts help performance? Neither should be prevalent enough to make any difference, and one would naively expect performance to be improved by ignoring such things, not by paying attention to them... > net.ipv4.tcp_max_orphans = 262144 I'm not clear on why this would help unless you were expecting really massive numbers of unattached sockets -- also you're saying that up to 16M of kernel memory can be used for this purpose... > net.core.netdev_max_backlog = 2000 This implies your processes are going to get massive numbers of TCP connections per unit time. Are they? It is certainly not a *general* performance improvement... Perry From csamuel at vpac.org Sat Jun 14 00:25:56 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] User resource limits In-Reply-To: <638248235.163311213426794764.JavaMail.root@zimbra.vpac.org> Message-ID: <511559027.163331213428356775.JavaMail.root@zimbra.vpac.org> ----- "David N Lombard" wrote: > Note, mmap.c has been validating memory requests against RLIMIT_AS > since at least 2.6.3 (earliest kernel sources I have online); Yup, it was glibc that changed from using brk() all the time to using mmap() for allocations > 128KB in 2.3. Looking at mm/mmap.c it seems that sys_brk() only checks RLIMIT_DATA, but I'm wondering if the call out to find_vma_intersection() ends up calling may_expand_vm() and hence enforce RLIMIT_AS as well ? > 2.6.25 includes a /proc file "limits" that displays the limits > (see fs/proc/base.c). Now that could be really handy, here's an example output from my home desktop: $ cat /proc/$$/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited ms Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 71680 71680 processes Max open files 1024 1024 files Max locked memory 32768 32768 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 71680 71680 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From kus at free.net Sat Jun 14 17:26:24 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Powersave on Beowulf nodes Message-ID: What is about using of powersaved (and dbus and HAL daemons) on Beowulf nodes ? Currently I installed SuSE 10.3 where all the corresponding daemons are running (by default) at the runlevel=3. I simple added issuing of powersave -f at the end of booting. /proc/acpi/thermal_zone/ is empty, and powersave can't give me temperature and FANs information. I don't see now any serious advantages of powersaved daemon using in "performance" mode (using performance scheme). We have many jobs in SGE at every time moment, and underload situation (where it's reasonable to decrease CPUs frequency) is not the our danger :-) So I'm thinking about simple stopping of all the corresponding daemons. Mikhail Kuzminsky Computer Assistance to Chemical Research Center Zelinsky Institute of Organic Chemistry Moscow From walid.shaari at gmail.com Sat Jun 14 22:32:29 2008 From: walid.shaari at gmail.com (Walid) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] RHEL5 network throughput/scalability In-Reply-To: <87od64lpta.fsf@snark.cb.piermont.com> References: <588c11220806130856j34c2949dq3ac538ecc4409ca0@mail.gmail.com> <87od64lpta.fsf@snark.cb.piermont.com> Message-ID: 2008/6/14 Perry E. Metzger : > > A number of these seem rather odd, or unrelated to performance. > > Walid writes: > > It is lame, however i managed to get the following kernel paramter to > scale > > well in terms of both performance per node, and scalability over a high > > bandwidth low latency network > > > > net.ipv4.tcp_workaround_signed_windows = 1 > > This is a workaround for a buggy remote TCP. If you have a homogeneous > network of linux boxes, it will have no effect. True, however one of the assumption we have is that the NFS Filer (runing some modifioed version of BSD )is broken > > > net.ipv4.tcp_congestion_control = vegas > > I'm under the impression that the Vegas congestion control policy is > not well loved by the experts on TCP performance. we were working on the assumption that there is congestion involved, and tried the several different algorithms involved, we did even try veno (that is mainly for wirless) just for the sake of testing, and vegas seemed to give us the boost in perfomance. my basic humble research did not show much when it comes to low latency, high bandwidth LAN networks, let me know if you do have any pointers, URLS that are against vegas or recomend otherwise, also i can see that most of these algorthims have options, that i just used thier defaults, not sure if this is the correct way to go about it?! > > > net.ipv4.route.max_size = 8388608 > > This sets the size of the routing cache. You've set it to a rather > large and fairly random number. that's the value in RHEL4U6 system that is working on the same network setup, just made a diff between the RHEL4 and RHEL5 and checked what values are different, and worked by trial and error from thier. > > net.ipv4.icmp_ignore_bogus_error_responses = 0 > > net.ipv4.icmp_echo_ignore_broadcasts = 0 > > Why would paying attention to bogus ICMPs and to ICMP broadcasts help > performance? Neither should be prevalent enough to make any > difference, and one would naively expect performance to be improved by > ignoring such things, not by paying attention to them... > agree, have to test them in isloation, as i said i was working out from trial and error based on the assumption that RHEL4 was working fine in the current environment > > net.ipv4.tcp_max_orphans = 262144 > > I'm not clear on why this would help unless you were expecting really > massive numbers of unattached sockets -- also you're saying that up to > 16M of kernel memory can be used for this purpose... removed, this was again from RHEL4, the default in RHEL5 is actually less > > net.core.netdev_max_backlog = 2000 > > This implies your processes are going to get massive numbers of TCP > connections per unit time. Are they? It is certainly not a *general* > performance improvement... this is what looks like happening actually, and so far that's the one paramter that looks like making it scale better, however i am going to investigate this further, and can share the information back to you, no respone yet from RH. regards Walid -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080615/3ab3dc92/attachment.html From diep at xs4all.nl Sun Jun 15 04:05:56 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Mersenne primes? In-Reply-To: <7899080805011310xe372098y85a3dc01fbeb2be3@mail.gmail.com> References: <7899080805011310xe372098y85a3dc01fbeb2be3@mail.gmail.com> Message-ID: <638550ED-F8C6-484F-9B9A-C12579CE2111@xs4all.nl> hi Mark, Your question is not so much about lucaslehmer, but rather about FFT and DWT. No doubt some attempts have been made to parallellize FFT already to multiply big numbers. DWT, as used by GIMPS, is also taking implicit modulo, basically speeding up a factor 2 nearly. That's what you lose when using some generic approach to multiply. Additionally the DWT from GIMPS has been written in SSE2 assembler, so it only works at x86 hardware that has some sort of SSE2 support. This is very bloody fast code optimized for intel P4 at the time. You won't be able to approach that speed writing C code. There is some code that parallellizes all this, yet it would prove more numbers to be composite when running it embarrassingly parallel. Not because it can't be done, but because that Woltman assembler is so bloody fast. In itself such FFT's can be done a lot faster in integers, just on paper. However all the cpu's seem to be faster for floating point. AMD is by far fastest right now for integer multiplication (oldie K8 beating core2 amazingly). See GMP mailing list for explanation. Intel seems to be faster for running the Woltman code. See benchmarks at mersenneforum.org Those x86/x64 cpu's are rather slow for multiplication. To be honest, finding a mersenne is not so interesting now, already too many join GIMPS. A single test single core is like a month with odds like 1 in a 100k it is a prime. Personally i'm trying to find some numbers p = (2^n + 1) / 3 busy around 1 million bits now. Note those won't win you the $100k, for 2 reasons. First of all they can't get proven prime easily, as they are "industry grade primes", secondly you need that army of 30 + Tflop or so that GIMPS is using. For future it would make sense IMHO to parallellize DWT within 1 socket. That allows better cache reusage. Bandwidth from RAM to the caches already is a problem. Let's not start discussing bandwidth that the highend network cards can deliver :) There is some C codes there that do this (note i have not managed to download it) and there is rumours Woltman also parallellized his own code one day, but most of those codes are indeed only well usable within 1 socket, already losing quite some throughput there. If your intention is to just do a single FFT as fast as possible, i'm sure there must be something out there. Would be fun to optimize code like that, yet unpaid without cluster access it is hard to get it done :) Right now the quickest way to find a real big 100% prime is to join primegrids search for 321 (p = k * 2^n - 1 where k = 3). They recently found another one. It also can use the same transform. For k's above 31, the assembler code runs a lot slower, so you lose big throughput already. Because you've got to test that many numbers to find a single prime, nearly all the dudes in that prime world, when at home, are busy with throughput anyway and not so much highend clusters. Highend clusters in that domain get used for other purposes, those busy with that will shut up and not post here data that is interesting to read. For me it is easy to shout something there as i'm currently unemployed looking for a job (a job abroad outside Netherlands would be cool too, feel free to ask my CV/resume) and in meantime am busy with an innocent computer chess engine. Where GUI's sell real well in computer games, engines are not commercial at all, they just search for the holy grail massively parallel, thereby combining hundreds of algorithms/enhancements. Vincent On May 1, 2008, at 10:10 PM, Mark Reynolds wrote: > I'm fairly new to Beowulfery but am reading TFM from > http://www.phy.duke.edu/~rgb/Beowulf/beowulf_book.php (great book by > the way). Does anyone know of any programs to find Mersenne primes > that are suited to parallelised environments such as a cluster made up > of nodes with multi-core processors? I've found mprimes from > http://www.mersenne.org/ but the FAQ states: > Although, a program could be written for dual-CPU systems (it would be > quite time-consuming), the machine will still get more throughput by > working on separate exponents. > Has anybody tried this and can Lucas-Lehmer testing be effectively > parallalised? Currently, you have to run an instance for each core for > it to get any benefit. Also I'd like to modify it so it could, for > example, search between certain numerical ranges independent of the > work units sent out through the website or to take advantage of 64-bit > hardware. Can someone more knowledgeable about this sort of thing tell > me whether this is practical or if it has been done already? > > Thanks. > > -- > "To win one hundred victories in one hundred battles is not the > highest skill. To subdue the enemy without fighting is the highest > skill."? Sun-Tzu > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From diep at xs4all.nl Sun Jun 15 04:31:12 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <1210016466.4924.1.camel@Vigor13> References: <1210016466.4924.1.camel@Vigor13> Message-ID: Seems the next CELL is 100% confirmed double precision. Yet if you look back in history, Nvidia promises on this can be found years back. The only problem with hardware like Tesla is that it is rather hard to get technical information; like which instructions does Tesla support in hardware? This is crucial to know in order to speedup your code. It is already tough to get realworld codes on GPU's faster than at cpu's. The equivalent CPU code has been optimized real bigtime, knowing everything about hardware. How fast is latency from RAM when all 128 SP's are busy with that? Nvidia gives out zero information and doesn't support anyone either for this. That has to change in order to get GPU calculations more into mainstream. When i calculate on paper for some applications, a GPU can be potentially factor 4-8 faster than a standard quadcore 2.4ghz is right now. Getting that performance out of the GPU is more than a fulltime task however, without having indepth technical hardware data on the GPU. Vincent On May 5, 2008, at 9:40 PM, John Hearns wrote: > On Fri, 2008-05-02 at 14:05 +0100, Ricardo Reis wrote: >> Does anyone knows if/when there will be double floating point on >> those >> little toys from nvidia? >> >> > Ricardo, > I think CUDA is a gret concept, and am starting to work with it at > home. > I recently went to a talk by David Kirk, as part of the "world tour". > I think the answer to your question is Real Soon Now. > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From landman at scalableinformatics.com Sun Jun 15 06:51:44 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: References: <1210016466.4924.1.camel@Vigor13> Message-ID: <48551E70.7070507@scalableinformatics.com> Vincent Diepeveen wrote: > Seems the next CELL is 100% confirmed double precision. > Yet if you look back in history, Nvidia promises on this can be found > years back. [scratches head /] Vincent, it may be possible that some of us on this list may in fact be bound by NDA (non-disclosure agreements), and cannot talk about hardware which has not been announced. > The only problem with hardware like Tesla is that it is rather hard to > get technical information; like which instructions does Tesla support in > hardware? [scratches head /] Hmmm .... www.nvidia.com/cuda is a good starting point. I might suggest http://www.nvidia.com/object/cuda_what_is.html as a start on information. More to the point, you can look at http://www.nvidia.com/object/cuda_develop.html -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From Shainer at mellanox.com Sun Jun 15 09:08:20 2008 From: Shainer at mellanox.com (Gilad Shainer) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <4852CDE3.4030006@myri.com> Message-ID: <9FA59C95FFCBB34EA5E42C1A8573784F0129E83A@mtiexch01.mti.com> > > latency difference here matters to many codes). Perhaps of more > > significance, though, is that you can use oversubscription to lower > > the cost of your fabric. Instead of connecting 12 ports of a leaf > > switch to nodes and using the other 12 ports as uplinks, > you might get > > away with > > 18 nodes and 6 uplinks, or 20 nodes and 4 uplinks. As core > counts are > > increasing, this is becoming more and more viable for some > applications. > > It's important to note that the "full-bisection" touted by > vendors is on paper only. In reality, static routing provides > full-bisection for a very small subset of patterns, the > average effective bisection on a > diameter-3 Clos is ~40% of link rate (adaptive routing > improves that a lot, but breaks packet order on the wire > which is a requirement for some network protocols). > Static routing is the best approach if your pattern is known. In other cases it depends on the applications. LANL and Mellanox have presented a paper on static routing and how to get the maximum of it last ISC. There are cases where adaptive routing will show a benefit, and this is why we see the IB vendors add adaptive routing support as well. But in general, the average effective bandwidth is much much higher than the 40% you claim. > In practice, "paper" full-bisection is near free when using a > single enclosure, since all spine cables are on the > backplane. For larger networks, where you have to pay for > real cables to the spine level, then it may make sense to be > oversubscribed if the effective bisection is already bad > (static routing), or if your collective communication on > large jobs are not bandwidth bounded. However, the later is > often false on many-cores. There are some vendors that uses only the 24 port switches to build very large scale clusters - 3000 nodes and above, without any oversubscription, and they find it more cost effective. Using single enclosures is easier, but the cables are not expensive and you can use the smaller components. I used the 24 ports switches to connect my 96 node cluster. I will replace my current setup with the new 36 InfiniBand port switches this month, since they provide lower latency and adaptive routing capabilities. And if you are bandwidth bounded, using IB QDR will help. You will be able to drive more than 3GB/s from each server. From landman at scalableinformatics.com Sun Jun 15 09:36:15 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <9FA59C95FFCBB34EA5E42C1A8573784F0129E83A@mtiexch01.mti.com> References: <9FA59C95FFCBB34EA5E42C1A8573784F0129E83A@mtiexch01.mti.com> Message-ID: <485544FF.9090101@scalableinformatics.com> On a different note ... Gilad Shainer wrote: > routing capabilities. And if you are bandwidth bounded, using IB QDR > will help. You will be able to drive more than 3GB/s from each server. This is theoretical max, correct? What sort of real observed DMA bandwidth are people seeing with the QDR adapters? That is, how quickly can I copy 32MB from main memory to the PCIe (even gen2) based card? Or how quickly (e.g. what is the real observed data rate) for copying streaming data, say from a nice NFS over RDMA session, to the card? Please correct me if I am wrong, but I suspect that this will be one of rate limiting factors at least in the near term. Not that this is a bad thing, 3GB/s is quite good, but if the practical best case memory->PCIe bandwidth only let us get 2GB/s, or even 1.5 GB/s, this is important to know. If you or someone out there with QDR could measure this, I think a few of us (at least on the storage side) would like to know this. If someone wants to loan us cards and cables (cough cough), we would be happy to do this measurement between pairs of JackRabbits. Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From diep at xs4all.nl Sun Jun 15 10:36:26 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <48551E70.7070507@scalableinformatics.com> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> Message-ID: <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> Joseph, I'm even licensed CUDA developer by Nvidia for Tesla, but even the documents there are very very poor. Knowing latencies is really important. Another big problem is that if you write such a program measuring the latencies, no dude is gonna run it. The promises of nvidia was for videocards long ago released, all of them being single precision. We know that for sure as most of those videocards have been released years ago already. Not to mention the 8800. If you look however to the latest supercomputer announced, 1 peta instructions a second @ 12960 new generation CELL cpu's, that means a tad more than 77 gflop double precision for each CELL. It is a bit weird if you claim to be NDA bound, whereas the news has it in big capitals what the new IBM CELL can deliver. See for example: http://www.cbc.ca/cp/technology/080609/z060917A.html Also i calculated back that all powercosts together it is about 410 watt a node, each node having a dual cell cpu. That's network + harddrives and RAM together of course. I calculated 2.66 MW for the entire machine based upon this article. Sounds reasonably good to me. Now interesting to know is how they are gonna use that cpu power effectively. In any case it is a lot better than what the GPU's can deliver so far in practical situations known to me. So claims by programmers other than the nvidia engineer claims. Especially stuff like matrix calculations, as the weak part of the GPU hardware is the latency to and from the machine RAM (not to confuse with device RAM). From/to 8800 hardware a reliable person i know measured it at 3000 messages a second, which would make it about several hundreds of microseconds of communication latency speed. So a very reasonable question to ask is what the latency is from the stream processors to the device RAM. A 8800 document i read says 600 cycles. It doesn't mention for how many streamprocessors this is though. Also surprising to know i learned that RAM lookups do not get cached. That means a lot of extra work when 128 stream processors hammer regurarly onto the device RAM for stuff that CPU's simply cache in their L1 or L2 cache and todays even L3 caches. So knowing such technical data is total crucial as there is no way to escape the memory controllers latency in a lot of different software that searches for the holy grail. Thanks, Vincent On Jun 15, 2008, at 3:51 PM, Joe Landman wrote: > > > Vincent Diepeveen wrote: >> Seems the next CELL is 100% confirmed double precision. >> Yet if you look back in history, Nvidia promises on this can be >> found years back. > > [scratches head /] > > Vincent, it may be possible that some of us on this list may in > fact be bound by NDA (non-disclosure agreements), and cannot talk > about hardware which has not been announced. > > >> The only problem with hardware like Tesla is that it is rather >> hard to >> get technical information; like which instructions does Tesla >> support in hardware? > > [scratches head /] > > Hmmm .... www.nvidia.com/cuda is a good starting point. > > I might suggest http://www.nvidia.com/object/cuda_what_is.html as a > start on information. More to the point, you can look at http:// > www.nvidia.com/object/cuda_develop.html > > > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics LLC, > email: landman@scalableinformatics.com > web : http://www.scalableinformatics.com > http://jackrabbit.scalableinformatics.com > phone: +1 734 786 8423 > fax : +1 866 888 3112 > cell : +1 734 612 4615 > From hahn at mcmaster.ca Sun Jun 15 10:44:21 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <9FA59C95FFCBB34EA5E42C1A8573784F0129E83A@mtiexch01.mti.com> References: <9FA59C95FFCBB34EA5E42C1A8573784F0129E83A@mtiexch01.mti.com> Message-ID: > Static routing is the best approach if your pattern is known. In other sure, but how often is the pattern actually known? I mean in general: aren't most clusters used for multiple, shifting purposes? > There are some vendors that uses only the 24 port switches to build very > large scale clusters - 3000 nodes and above, without any > oversubscription, and they find it more cost effective. Using single so the switch fabric would be a 'leaf' layer with 12 up and 12 down, and a top layer with 24 down, right? so 3000 nodes means 250 leaves and 125 tops, 9000 total ports so 4500 cables. > enclosures is easier, but the cables are not expensive and you can use > the smaller components. in federated networks, I think cables wind up being 15-20% of the network price. for instance, if we take the simplest possible approach, and equip this 3000-node cluster with a non-blocking federated fabric (assuming just sdr) from colfax's current price list: subtot unit n what 375000 125 3000 ib nic 117000 39 3000 1m host ib cables 148500 99 1500 8m leaf-top ib cables 900000 2400 375 24pt unman switch 1540500 total (cable 17%) I'm still confused about IB pricing, since the street price of nics, cables and switches are dramatically more expensive than colfax. (to the paranoid, colfax would appear to be a mellanox shell company...) for completeness, here's the same bom with "normal" public prices: subtot unit n what 2100000 700 3000 ib nic 330000 110 3000 1m host ib cables 330000 220 1500 8m leaf-top ib cables 1500000 4000 375 24pt unman switch 4260000 total (cable 15%) interestingly, if nodes were about 3700 apiece (about what you'd expect for intel dual-socket quad-core 2G/core), the interconnect winds up being 28% of the cost. From Shainer at mellanox.com Sun Jun 15 14:46:05 2008 From: Shainer at mellanox.com (Gilad Shainer) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <485544FF.9090101@scalableinformatics.com> Message-ID: <9FA59C95FFCBB34EA5E42C1A8573784F0129E864@mtiexch01.mti.com> Joe Landman wrote: > On a different note ... > > Gilad Shainer wrote: > > > routing capabilities. And if you are bandwidth bounded, > using IB QDR > > will help. You will be able to drive more than 3GB/s from > each server. > > This is theoretical max, correct? What sort of real observed > DMA bandwidth are people seeing with the QDR adapters? That > is, how quickly can I copy 32MB from main memory to the PCIe > (even gen2) based card? Or how quickly (e.g. what is the > real observed data rate) for copying streaming data, say from > a nice NFS over RDMA session, to the card? > No, this is the actual number. IB QDR theoretical max is 4GB/s (uni-directional BW) > Please correct me if I am wrong, but I suspect that this will > be one of rate limiting factors at least in the near term. > > Not that this is a bad thing, 3GB/s is quite good, but if the > practical best case memory->PCIe bandwidth only let us get > 2GB/s, or even 1.5 GB/s, this is important to know. > PCIe x8 Gen1 is maxed in around 1.5-1.6GB/s, and PCIe x8 Gen2 in around 3.3-3.4GB/s. There are multiple PCIe Gen2 servers out there from multiple vendors. I am using Supermicro based for my testing, but there are other solutions. Here is some non-tuned MPI results bandwidth (IB QDR on PCIe x8 Gen2 servers): Bytes PingPong (MB/s) SendRecv (MB/s) 512 695.71 707.2 1024 1160.13 1178.18 2048 1664.83 1972.34 4096 1957.76 2597.41 8192 2165.76 3025.58 16384 2642.91 4187.19 32768 2962.5 5076.31 65536 3165.03 5693.37 131072 3274.05 6078.59 262144 3332.28 6287.49 524288 3362.12 6399.46 1048576 3377.28 6458.3 2097152 3385.08 6489.52 4194304 3388.02 6457.68 > If you or someone out there with QDR could measure this, I > think a few of us (at least on the storage side) would like > to know this. If someone wants to loan us cards and cables > (cough cough), we would be happy to do this measurement > between pairs of JackRabbits. I might be able to help you here. Gilad. From james.p.lux at jpl.nasa.gov Sun Jun 15 15:42:27 2008 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:11 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> Message-ID: <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> Quoting Vincent Diepeveen , on Sun 15 Jun 2008 10:36:26 AM PDT: > Joseph, > > It is a bit weird if you claim to be NDA bound, whereas the news has it in > big capitals what the new IBM CELL can deliver. > I don't know that Joseph was claiming to be NDA bound, just that there might be people who could potentially comment who are. But, in any case, more than once I've been in a situation where I was unable to discuss or otherwise acknowledge information that was public because of either classification guidelines or NDAs. There's a big difference between carefully controlled press releases and someone responding to random questions. And, likewise there's a big difference between someone making intelligent speculation about a technology and having certain knowledge. From hahn at mcmaster.ca Sun Jun 15 21:47:21 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> Message-ID: > It is a bit weird if you claim to be NDA bound, whereas the news has it in > big capitals what the new IBM CELL can deliver. I thought he was referring to double-precision on Nvidia gpus, which have indeed not been shipped publicly (afaik). > So a very reasonable question to ask is what the latency is from the stream > processors to the device RAM. sure, they're GPUs, not general-purpose coprocessors. but both AMD and Intel are making noises about changing this. AMD seems to be moving GPU units on-chip, where they would presumably share L3, cache coherency, etc. Intel's Larrabee approach seems to be to add wider vector units to normal x86 cores (and more of them). I personally think the latter is much more promising from an HPC perspective. but then again, both AMD and Nvidia have major cred on the line - they have to deliver competitive levels of the traditional GPU programming model. From diep at xs4all.nl Mon Jun 16 08:11:56 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:11 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> Message-ID: Jim, Reality is that the person who SPECULATES that something is good also hides behind a DNA. This is another typical case of that. On the one hand claiming a NDA, on the other hand implying that is a very good product that will get released. My world is a bit binary there. Especially because of VERY BAD experiences in the past. If NVIDIA clearly indicates towards me that they aren't gonna release any technical data nor support anyone with technical data (the word NDA has not been used by the way by either party, it was a very clear NJET from 'em), and all information i've got so far is very dissapointing, then hiding behind a NDA is considered very bad manners. Either shut up entirely or do not hide behind a NDA. I won't quote persons nor the manufacturer they represent, but i remember such speculations all too well from the past. Hard statistics show that in like 5% of the cases the speculant was correct. In the other 95% of the cases the reason was that they didn't know a fok about what their competitors were gonna show up with. Tunnelvision is common. Good products don't need this type of NDA- promotion. In case of NVIDIA if you google a tad you will figure out that the double precision promise has been done more than once, many many years ago, and each time we got dissappointed. In the end even it was the case that the graphics cards cannot get used for gpu programming, for whatever technical reason (pixel units blocking bandwidth to the RAM or whatever giving a bottleneck of a factor 5 slowdown or so) Then instead of a $200 pci-e card, we needed to buy expensive Tesla's for that, without getting very relevant indepth technical information on how to program for that type of hardware. The few trying on those Tesla's, though they won't ever post this as their job is fulltime GPU programming, report so far very dissappointing numbers for applications that really matter for our nations. Truth is that you can always do a claim of 1 teraflop of computing power. If that doesn't get backupped by technical documents how to get it out of the hardware if your own testprograms show that you can't get that out of the hardware, it is rather useless to start programming for such a platform. It is questionable whether it is interesting to design some algorithms for GPU's; it takes endless testing of every tiny detail to figure out what the GPU can and cannot do and to get accurate timings. By the time you finish with that, you can also implement the same design in FPGA or ASIC/VLSI whatever. As that is of course the type of interested parties in GPU programming; considering the amount of computing power they need, for the same budget they can also make their own CPU's. For other companies that i tried to get interested, there is a lot of hesitation to even *investigate* that hardware, let alone give a contract job to port their software to such hardware. Nvidia for all those civilian and military parties is very very unattractive as of now. IBM now shows up with a working supercomputer using new generation CELL processors which have a sustained 77 Gflop double precision a chip which means a tad more than 150 Gflop for each node. Each rackmount node is relative cheap and performs very well. 1 Petaflop @ 2.66 MW, that's really good. Vincent On Jun 16, 2008, at 12:42 AM, Jim Lux wrote: > Quoting Vincent Diepeveen , on Sun 15 Jun 2008 > 10:36:26 AM PDT: > >> Joseph, >> >> It is a bit weird if you claim to be NDA bound, whereas the news >> has it in >> big capitals what the new IBM CELL can deliver. >> > > I don't know that Joseph was claiming to be NDA bound, just that > there might be people who could potentially comment who are. > > But, in any case, more than once I've been in a situation where I > was unable to discuss or otherwise acknowledge information that was > public because of either classification guidelines or NDAs. > There's a big difference between carefully controlled press > releases and someone responding to random questions. And, likewise > there's a big difference between someone making intelligent > speculation about a technology and having certain knowledge. > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From hahn at mcmaster.ca Mon Jun 16 08:22:55 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Powersave on Beowulf nodes In-Reply-To: References: Message-ID: > What is about using of powersaved (and dbus and HAL daemons) on Beowulf nodes > ? ... > thinking about simple stopping of all the corresponding daemons. I would not even have these desktop-ish things installed. IMO, a cluster compute node should be quite stripped down - no extra daemons, minimal UI stuff. From prentice at ias.edu Mon Jun 16 08:38:44 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: References: <1210016466.4924.1.camel@Vigor13> Message-ID: <48568904.3020307@ias.edu> Vincent Diepeveen wrote: > > That has to change in order to get GPU calculations more into mainstream. > > When i calculate on paper for some applications, a GPU can be potentially > factor 4-8 faster than a standard quadcore 2.4ghz is right now. > > Getting that performance out of the GPU is more than a fulltime task > however, > without having indepth technical hardware data on the GPU. Completely untrue. One of my colleagues, who does a lot of work with GPU processors for astrophysics calculations, was able to increase the performance of the MD5 algorithm by ~100x with about 1.5 days of work. He called this this code that he wrote "(totally unoptimized, a straight CUDA C implementation of Rivest's algorithm". He tinkered some more, adding some optimizations, and I believe he ended up with 350x performance improvement. Here, I quote his e-mail on his first round of coding that he sent me: The other day in NYC on HPC-UG meeting someone mentioned that GPUs would be perfect for password cracking, with which I wholeheartedly agreed (on theoretical grounds). But theory is nothing without experiment :) , so I spent the last night and this morning writing a GPU MD5 hash routine (totally unoptimized, a straight CUDA C implementation of Rivest's algorithm). The results? * GPU (single GeForce 8800 Ultra on cylon): 57,640,967.264473 hash/second * The same algorithm on the CPU (Intel(R) Core(TM)2 Quad CPU Q6700 @ 2.66GHz on cylon): 543,839.652381 hash/second A factor of ~100 difference. Sweet. Another point of comparison: the fastest, assembly-level optimized x86 MD5 code, running on a _dual_ 3.2 GHz Xeon (see http://c3rb3r.openwall.net/mdcrack/) can do 42e6 hash/sec. And remember, I wrote the CUDA code in a day and a half, with _no_ optimization. Nice. In another words, one GPU card with an amateurishly written MD5 code can brute-force crack an 8-character MD5 hashed password consisting of [0-9A-Za-z] in about 6 weeks. Now imagine if someone who knew what they were doing optimized the code, and got a cluster of Tesla's instead of a single gaming card that I used.... Cool :-) . -- Prentice From Craig.Tierney at noaa.gov Mon Jun 16 08:44:58 2008 From: Craig.Tierney at noaa.gov (Craig Tierney) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> Message-ID: <48568A7A.50509@noaa.gov> Mark Hahn wrote: >> It is a bit weird if you claim to be NDA bound, whereas the news has >> it in >> big capitals what the new IBM CELL can deliver. > > I thought he was referring to double-precision on Nvidia gpus, > which have indeed not been shipped publicly (afaik). An article posted today about the GTX280, which is to be release tomorrow, states that the GTX280 has "support for the IEEE-754R double-precision floating-point standard." http://www.maximumpc.com/sites/maximumpc.com/themes/maximumpc/wow.php?back=article/unveiled_nvidias_next_gen_gpu Craig > >> So a very reasonable question to ask is what the latency is from the >> stream processors to the device RAM. > > sure, they're GPUs, not general-purpose coprocessors. but both AMD and > Intel are making noises about changing this. AMD seems to be moving GPU > units on-chip, where they would presumably share L3, cache coherency, > etc. Intel's Larrabee approach seems to be to add wider vector units to > normal x86 cores (and more of them). I personally think the latter is > much more promising from an HPC perspective. but then again, both AMD > and Nvidia have major cred on the line - they have to deliver competitive > levels of the traditional GPU programming model. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Craig Tierney (craig.tierney@noaa.gov) From john.leidel at gmail.com Mon Jun 16 08:56:41 2008 From: john.leidel at gmail.com (John Leidel) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <48568A7A.50509@noaa.gov> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <48568A7A.50509@noaa.gov> Message-ID: <1213631801.6549.0.camel@e521.site> Another article on the same card posted here: http://www.pcper.com/article.php?aid=578 On Mon, 2008-06-16 at 09:44 -0600, Craig Tierney wrote: > Mark Hahn wrote: > >> It is a bit weird if you claim to be NDA bound, whereas the news has > >> it in > >> big capitals what the new IBM CELL can deliver. > > > > I thought he was referring to double-precision on Nvidia gpus, > > which have indeed not been shipped publicly (afaik). > > An article posted today about the GTX280, which is to be release tomorrow, > states that the GTX280 has "support for the IEEE-754R double-precision > floating-point standard." > > http://www.maximumpc.com/sites/maximumpc.com/themes/maximumpc/wow.php?back=article/unveiled_nvidias_next_gen_gpu > > Craig > > > > > >> So a very reasonable question to ask is what the latency is from the > >> stream processors to the device RAM. > > > > sure, they're GPUs, not general-purpose coprocessors. but both AMD and > > Intel are making noises about changing this. AMD seems to be moving GPU > > units on-chip, where they would presumably share L3, cache coherency, > > etc. Intel's Larrabee approach seems to be to add wider vector units to > > normal x86 cores (and more of them). I personally think the latter is > > much more promising from an HPC perspective. but then again, both AMD > > and Nvidia have major cred on the line - they have to deliver competitive > > levels of the traditional GPU programming model. > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > From James.P.Lux at jpl.nasa.gov Mon Jun 16 09:09:04 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:11 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> Message-ID: <6.2.5.6.2.20080616084554.02e4dd18@jpl.nasa.gov> At 08:11 AM 6/16/2008, Vincent Diepeveen wrote: >Jim, > >Reality is that the person who SPECULATES that something is good >also hides behind a DNA. This is another typical case of that. perhaps.. >On the one hand claiming a NDA, on the other hand implying that is a >very good product that will get >released. Perhaps.. It's also that someone might be under NDA, be involved in the technical side of a development, but not be aware of the machinations of the marketing side. I'll bet more than one person has seen features added or removed because of "product positioning" after they last saw the thing they worked on. You might toil on a project, it gets released to internal manufacturing, and 6 to 12 months later, it pops out on the market and has significant differences from what you last saw. ( At a place I used to work, there were always comments about not letting the engineers go to the trade show where the product was being demoed..) >My world is a bit binary there. Especially because of VERY BAD >experiences in the past. With NDAs? (I'm sure lots of people have had less than wonderful experiences in that regard) >Either shut up entirely or do not hide behind a NDA. That's kind of hard when one has expertise in an area, and one wants to correct a misinterpretation or misstatement made by someone without as many facts to hand. (On the other hand, there's always the risk that the commenter is themself missing part of the story...) In the other 95% of the cases the reason was that they didn't know a >fok about what their competitors were gonna show up with. >Tunnelvision is common. Good products don't need this type of NDA- promotion. Actually, they do. In a perfect world, the inherent quality would result in the world beating a path to your door to get your inherently superior product. In a real world, you might not have the resources to bring the product to market as quickly as someone else, so you need time to get IP protection in place, for instance. >In case of NVIDIA if you google a tad you will figure out that the >double precision promise has been done more than once, >many many years ago, and each time we got dissappointed. Well.. I suspect that you didn't actually pay Nvidia for that capability, so the promise is just a marketing pledge, which is of no real legal value. (except as noted below). I can see that Nvidia can make a wise business decision to support or not support some capability based on the cost to provide it vs revenue they'll get. (Note that if you're in a situation of market power, and you announce capabilities or products that you have no real intention of producing, just to scare off the competitors, you can get into trouble. IBM S/360 is a case in point. Proving it is another matter, eh?) >Then instead of a $200 pci-e card, we needed to buy expensive Tesla's >for that, without getting very relevant indepth technical >information on how to program for >that type of hardware. That's the price one pays for being on the fringes of the mainstream. Go out and pay $10-20K for a custom coprocessor card from a small volume company and the mfr will pay a lot more attention to you. For an Nvidia, with a half a billion a year in revenue, the niche supercomputing market is a pimple on a pimple on a pimple of their behind. >The few trying on those Tesla's, though they won't ever post this as >their job is fulltime GPU programming, report so far very >dissappointing numbers for applications that >really matter for our nations. if they really matter, then serious money needs to be thrown at it. While I'm not generally an apologist for the "fiduciary responsibility to the shareholder" mindset, merely because something is interesting or intellectually valuable doesn't get it funded. >Truth is that you can always do a claim of 1 teraflop of computing >power. If that doesn't get backupped by technical documents >how to get it out of the hardware if your own testprograms show that >you can't get that out of the hardware, it is rather useless to start > programming for such a platform. Yep.. that's why *I* always want to see the documents before committing significant development resources to a project. More than once, I've been burned by someone's great idea that didn't pan out. >It is questionable whether it is interesting to design some >algorithms for GPU's; it takes endless testing of every tiny detail >to figure out >what the GPU can and cannot do and to get accurate timings. This can appeal to a certain type of person. It's like tweaking the engine in a car, and one does it, usually, for the challenge, not because it's a cost effective way to solve a problem. It's also attractive to someone who has a lot more time than money. >By the >time you finish with that, you can also implement the same design in >FPGA or ASIC/VLSI whatever. One can, but the cost to make an ASIC is pretty high (figure $1M for a spin). You can buy an awful lot of tinkering and probing time for that that million bucks. (about 8000-10000 hours). FPGAs don't have the flops/watt efficiency that an ASIC can get to, although they are getting better. >As that is of course the type of >interested parties in GPU programming; >considering the amount of computing power they need, for the same >budget they can also make their own CPU's. > >For other companies that i tried to get interested, there is a lot of >hesitation to even *investigate* that hardware, let alone give a >contract job to port their software to such hardware. Nvidia for all those >civilian and military parties is very very unattractive as of now. Yep. And for good reason. Even a big DoD job is still tiny in Nvidia's scale of operations. We face this all the time with NASA work. Semiconductor manufacturers have no real reason to produce special purpose or customized versions of their products for space use, because they can sell all they can make to the consumer market. More than once, I've had a phone call along the lines of this: "Jim: I'm interested in your new ABC321 part." "Rep: Great. I'll just send the NDA over and we can talk about it." "Jim: Great, you have my email and my fax # is..." "Rep: By the way, what sort of volume are you going to be using?" "Jim: Oh, 10-12.." "Rep: thousand per week, excellent..." "Jim: No, a dozen pieces, total, lifetime buy, or at best maybe every year." "Rep: Oh..." {Well, to be fair, it's not that bad, they don't hang up on you.. but that's the idea... and that's before we get into things like lot traceability} From raysonlogin at gmail.com Mon Jun 16 09:13:27 2008 From: raysonlogin at gmail.com (Rayson Ho) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Powersave on Beowulf nodes In-Reply-To: References: Message-ID: <73a01bf20806160913q112b521fvf92cfb4496597c27@mail.gmail.com> Grid Engine can directly control the nodes and do power saving, see: http://wiki.gridengine.info/wiki/index.php/PowerSaving http://code.google.com/p/rocks-solid/wiki/PowerSaveAlgorithm Rayson On Sat, Jun 14, 2008 at 7:26 PM, Mikhail Kuzminsky wrote: > What is about using of powersaved (and dbus and HAL daemons) on Beowulf > nodes ? > > Currently I installed SuSE 10.3 where all the corresponding daemons are > running (by default) at the runlevel=3. I simple added issuing of > > powersave -f > at the end of booting. > /proc/acpi/thermal_zone/ is empty, and powersave can't give me temperature > and FANs information. I don't see now any serious advantages of powersaved > daemon using in "performance" mode (using performance scheme). > We have many jobs in SGE at every time moment, and underload situation > (where it's reasonable to decrease CPUs frequency) is not the our danger :-) > So I'm thinking about simple stopping of all the corresponding daemons. > > Mikhail Kuzminsky > Computer Assistance to Chemical Research Center > Zelinsky Institute of Organic Chemistry > Moscow > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From gerry.creager at tamu.edu Mon Jun 16 09:39:41 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:07:11 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <6.2.5.6.2.20080616084554.02e4dd18@jpl.nasa.gov> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <6.2.5.6.2.20080616084554.02e4dd18@jpl.nasa.gov> Message-ID: <4856974D.1050100@tamu.edu> MOD +2: Informative. Jim Lux wrote: > At 08:11 AM 6/16/2008, Vincent Diepeveen wrote: >> Jim, >> >> Reality is that the person who SPECULATES that something is good >> also hides behind a DNA. This is another typical case of that. > > > perhaps.. > > >> On the one hand claiming a NDA, on the other hand implying that is a >> very good product that will get >> released. > > Perhaps.. It's also that someone might be under NDA, be involved in the > technical side of a development, but not be aware of the machinations of > the marketing side. I'll bet more than one person has seen features > added or removed because of "product positioning" after they last saw > the thing they worked on. You might toil on a project, it gets released > to internal manufacturing, and 6 to 12 months later, it pops out on the > market and has significant differences from what you last saw. > > ( At a place I used to work, there were always comments about not > letting the engineers go to the trade show where the product was being > demoed..) > > > >> My world is a bit binary there. Especially because of VERY BAD >> experiences in the past. > > > With NDAs? (I'm sure lots of people have had less than wonderful > experiences in that regard) > > >> Either shut up entirely or do not hide behind a NDA. > > That's kind of hard when one has expertise in an area, and one wants to > correct a misinterpretation or misstatement made by someone without as > many facts to hand. (On the other hand, there's always the risk that the > commenter is themself missing part of the story...) > > > In the other 95% of the cases the reason was that they didn't know a >> fok about what their competitors were gonna show up with. >> Tunnelvision is common. Good products don't need this type of NDA- >> promotion. > > Actually, they do. In a perfect world, the inherent quality would > result in the world beating a path to your door to get your inherently > superior product. In a real world, you might not have the resources to > bring the product to market as quickly as someone else, so you need time > to get IP protection in place, for instance. > > > > >> In case of NVIDIA if you google a tad you will figure out that the >> double precision promise has been done more than once, >> many many years ago, and each time we got dissappointed. > > Well.. I suspect that you didn't actually pay Nvidia for that > capability, so the promise is just a marketing pledge, which is of no > real legal value. (except as noted below). I can see that Nvidia can > make a wise business decision to support or not support some capability > based on the cost to provide it vs revenue they'll get. > > (Note that if you're in a situation of market power, and you announce > capabilities or products that you have no real intention of producing, > just to scare off the competitors, you can get into trouble. IBM S/360 > is a case in point. Proving it is another matter, eh?) > > > >> Then instead of a $200 pci-e card, we needed to buy expensive Tesla's >> for that, without getting very relevant indepth technical information >> on how to program for >> that type of hardware. > > That's the price one pays for being on the fringes of the mainstream. > Go out and pay $10-20K for a custom coprocessor card from a small volume > company and the mfr will pay a lot more attention to you. For an > Nvidia, with a half a billion a year in revenue, the niche > supercomputing market is a pimple on a pimple on a pimple of their behind. > > >> The few trying on those Tesla's, though they won't ever post this as >> their job is fulltime GPU programming, report so far very >> dissappointing numbers for applications that >> really matter for our nations. > > if they really matter, then serious money needs to be thrown at it. > While I'm not generally an apologist for the "fiduciary responsibility > to the shareholder" mindset, merely because something is interesting or > intellectually valuable doesn't get it funded. > > > >> Truth is that you can always do a claim of 1 teraflop of computing >> power. If that doesn't get backupped by technical documents >> how to get it out of the hardware if your own testprograms show that >> you can't get that out of the hardware, it is rather useless to start >> programming for such a platform. > > Yep.. that's why *I* always want to see the documents before committing > significant development resources to a project. More than once, I've > been burned by someone's great idea that didn't pan out. > > >> It is questionable whether it is interesting to design some >> algorithms for GPU's; it takes endless testing of every tiny detail >> to figure out >> what the GPU can and cannot do and to get accurate timings. > > This can appeal to a certain type of person. It's like tweaking the > engine in a car, and one does it, usually, for the challenge, not > because it's a cost effective way to solve a problem. It's also > attractive to someone who has a lot more time than money. > > > >> By the >> time you finish with that, you can also implement the same design in >> FPGA or ASIC/VLSI whatever. > > One can, but the cost to make an ASIC is pretty high (figure $1M for a > spin). You can buy an awful lot of tinkering and probing time for that > that million bucks. (about 8000-10000 hours). > > FPGAs don't have the flops/watt efficiency that an ASIC can get to, > although they are getting better. > > >> As that is of course the type of >> interested parties in GPU programming; >> considering the amount of computing power they need, for the same >> budget they can also make their own CPU's. >> >> For other companies that i tried to get interested, there is a lot of >> hesitation to even *investigate* that hardware, let alone give a >> contract job to port their software to such hardware. Nvidia for all >> those >> civilian and military parties is very very unattractive as of now. > > Yep. And for good reason. Even a big DoD job is still tiny in Nvidia's > scale of operations. We face this all the time with NASA work. > Semiconductor manufacturers have no real reason to produce special > purpose or customized versions of their products for space use, because > they can sell all they can make to the consumer market. More than once, > I've had a phone call along the lines of this: > > "Jim: I'm interested in your new ABC321 part." > "Rep: Great. I'll just send the NDA over and we can talk about it." > "Jim: Great, you have my email and my fax # is..." > "Rep: By the way, what sort of volume are you going to be using?" > "Jim: Oh, 10-12.." > "Rep: thousand per week, excellent..." > "Jim: No, a dozen pieces, total, lifetime buy, or at best maybe every > year." > "Rep: Oh..." > > {Well, to be fair, it's not that bad, they don't hang up on you.. but > that's the idea... and that's before we get into things like lot > traceability} > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.862.3982 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From christian.bell at qlogic.com Mon Jun 16 10:32:17 2008 From: christian.bell at qlogic.com (Christian Bell) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <9FA59C95FFCBB34EA5E42C1A8573784F0129E83A@mtiexch01.mti.com> References: <4852CDE3.4030006@myri.com> <9FA59C95FFCBB34EA5E42C1A8573784F0129E83A@mtiexch01.mti.com> Message-ID: <20080616173217.GS9479@mv.qlogic.com> On Sun, 15 Jun 2008, Gilad Shainer wrote: > Static routing is the best approach if your pattern is known. In other > cases it depends on the applications. LANL and Mellanox have presented a > paper on static routing and how to get the maximum of it last ISC. There > are cases where adaptive routing will show a benefit, and this is why we > see the IB vendors add adaptive routing support as well. But in general, > the average effective bandwidth is much much higher than the 40% you > claim. As Mark Hahn pointed out, how are so-called "known patterns" representative of any real system? Even a single application with a "known pattern" doesn't translate as-is in practice, even less so on a capacity system. While the "shift all-to-all pattern" referred to in the paper you cite is interesting (on paper) in that it stresses the entire connectivity of a FBB fabric, it remains a simulation carried out in isolation. Sticking to simulation, I find that looking at the observed switch latency at the egress ports as a function of switch loading using random communication patterns to be a more interesting data point. However, it can be even more revealing to try to scale an expensive communication operation on a real system, only to notice that this is where the paper FBB breaks down. 40% looks like a large number but it's not uncommon to see application writers report large speedups by breaking down bandwidth-bound problems into latency-sensitive ones. This looks counter-intuitive because the paper FBB can be available on the fabric, but suggests that systems don't always deliver FBB *even if* the pattern is known and the fabric is otherwise idle. Given a capacity system, there's huge potential in looking at alternative routing methods -- all of which is orthogonal to system size and the ability to describe and understand one's communication pattern. > There are some vendors that uses only the 24 port switches to build very > large scale clusters - 3000 nodes and above, without any > oversubscription, and they find it more cost effective. Using single > enclosures is easier, but the cables are not expensive and you can use > the smaller components. I used the 24 ports switches to connect my 96 > node cluster. I will replace my current setup with the new 36 InfiniBand > port switches this month, since they provide lower latency and adaptive > routing capabilities. And if you are bandwidth bounded, using IB QDR > will help. You will be able to drive more than 3GB/s from each server. Along similar lines (and with less product placement), buying more cores and less IB can help you solve (and scale) larger problems. In a world of quick inferences, one is also permitted to conclude that implementors find it *performance cost effective* to *not* pay top dollar for paper FBB when only a fraction of the FBB performance can be achieved. Don't need the oversubscription for those cases. Based on some of the points provided so far, it's as if "known patterns" are already well served by IB static routing and that the "other cases" will now benefit from newer IB "adaptive routing support". As working for an IB vendor myself, or for anyone out there who spends a lot of their time solving bandwidth-bound problems or working on capacity systems, the discussion can't possibly be reduced to this. The design space for routing methods based on modular switches is much larger that what is currently offered by the Mellanox portfolio. So I'd suggest that *no*, static routing is not necessarily the best approach even if your pattern is known. . . christian -- christian.bell@qlogic.com QLogic Host Solutions Group From landman at scalableinformatics.com Mon Jun 16 11:30:39 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:11 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> Message-ID: <4856B14F.9020104@scalableinformatics.com> Vincent Diepeveen wrote: > Jim, > > Reality is that the person who SPECULATES that something is good > also hides behind a DNA. This is another typical case of that. Hiding behind DNA? Gotta be thin ... (ok ok, I'll keep my day job ...) > On the one hand claiming a NDA, on the other hand implying that is a > very good product that will get > released. An NDA is a device to allow a company to get its act together while getting feedback on the product they want to release. It is quite useful. [...] > If NVIDIA clearly indicates towards me that they aren't gonna release > any technical data nor support anyone with technical er ... have you looked at the site I indicated? > data (the word NDA has not been used by the way by either party, it was > a very clear NJET from 'em), > and all information i've got so far is very dissapointing, then hiding > behind a NDA is considered very bad manners. No, it's not. It seems to me that given the number of applications out there now using their kit, that rumors of a lack of information to use their kit might be ... er ... not well founded. > Either shut up entirely or do not hide behind a NDA. Ah ... Ok. Manners, manners Vincent. [...] > In case of NVIDIA if you google a tad you will figure out that the > double precision promise has been done more than once, Hmmm.... I would suggest you sign an NDA with them, and learn what it is they are talking about to their customers, though I suspect they may not wish to pre-divulge information to you given what you have indicated here. [...] > IBM now shows up with a working supercomputer using new generation CELL > processors which have a sustained 77 Gflop double precision > a chip which means a tad more than 150 Gflop for each node. Each > rackmount node is relative cheap and performs very well. Cell is good. Now try to program it. Yes, it is a SMOP ( http://www.catb.org/jargon/html/S/SMOP.html ) > 1 Petaflop @ 2.66 MW, that's really good. Yup. Exaflop at 2.66 GW. Waaa hoooo!!!! -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From deadline at eadline.org Mon Jun 16 14:40:56 2008 From: deadline at eadline.org (Douglas Eadline) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] A few words about NVidia and HPC Message-ID: <53810.192.168.1.213.1213652456.squirrel@mail.eadline.org> I visited NVidia last week for HPC editors day. I am working on some write-ups, but let me make a few general comments: - NVidia is very serious about HPC. There is of course a limited amount of real estate on the die, so there are trade-offs as to video/HPC - There will be double precision support across the product line (which is) * The GeForce is intended for entertainment market * The Tesla is intended for the HPC market and will use high grade memory for 24x7 HPC use, and support up to 4 GB of on-board RAM * The Quadro is for the high design/graphics market - In the very short time Cuda has been available (about a year) there are some applications that have seen large improvements in performance (not all apps need double precision) - Cuda provides an interesting abstraction layer which hides the dynamic thread execution hardware on the chip - there is more, but then I would be writing an article. Now, should you put your cluster on Ebay and buy video cards? I have no idea. What I do know is the consumer market has provided us with another interesting hardware platform for HPC applications. Plus, the companies (Nvidia, and AMD/ATI) are helping out as well. Interesting times. -- Doug -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From deadline at eadline.org Mon Jun 16 14:47:15 2008 From: deadline at eadline.org (Douglas Eadline) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] June New York/Jersey HPC users meeting Message-ID: <34281.192.168.1.213.1213652835.squirrel@mail.eadline.org> If you live or work in the New York/North Jersey Metropolitan area, mark your calender for this Thursday, June 19th. The NYCA-HUG (New York City Area HPC Users Group) will be trying to answer the ultimate question Torque or Sun Grid Engine? We will be discussing the pros/cons of each scheduler for HPC clusters. Come and add your experiences, wants, and rants. Then you decide. Links for the details http://www.clustermonkey.net//content/view/229/1/ http://www.linux-mag.com/id/4140 -- Doug -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. From patrick at myri.com Mon Jun 16 15:21:51 2008 From: patrick at myri.com (Patrick Geoffray) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <9FA59C95FFCBB34EA5E42C1A8573784F0129E83A@mtiexch01.mti.com> References: <9FA59C95FFCBB34EA5E42C1A8573784F0129E83A@mtiexch01.mti.com> Message-ID: <4856E77F.7080000@myri.com> Gilad Shainer wrote: > Static routing is the best approach if your pattern is known. In other If your pattern is known, and if it is persistent, and it is perfectly synchronized, and if you have a single job running on the fabric, and if you have total control of the process/node mapping and if there is no down/bad links, and if there is no other traffic pattern in the application, then yes static routing is the best. In real life, where there are multiple jobs running at once on various parts of a cluster, where there are always some marginal links, when you cannot guarantee on which nodes a job will be allocated, and applications have multiple communication patterns (collectives) and load is usually unbalanced, static routing is the worst. > cases it depends on the applications. LANL and Mellanox have presented a > paper on static routing and how to get the maximum of it last ISC. There Single app, dedicated machine, total control of the network. Similarly, I could have a pretty good shot at predicting the next lotto numbers if I would know the position (and speed) of all atoms in the universe (Dr Brown, this is for you !). > are cases where adaptive routing will show a benefit, and this is why we > see the IB vendors add adaptive routing support as well. But in general, > the average effective bandwidth is much much higher than the 40% you > claim. Have a look at the slides 17 and 19 of the following set of slides (and slides 21 and 22 to illustrate my point above): http://www.openib.org/archives/spring2007sonoma/Monday%20April%2030/Leininger-Seager-Adaptive-Routing-OFA-Sonoma-2007-v03.pdf Hoefler and al have shown an average effective bisection of ~40% on Infiniband (OMNeT simulations) in a paper submitted to Cluster2008. In a paper to be presented at Hot Interconnects this year, I have measured the effective bisection (SendRecv on random pairs) on a 512-node Myri-10G cluster (single enclosure, 32-port crossbars) under various routing implementations. Below is the link to pretty graphs with static and probing adaptive routing: http://patrick.geoffray.googlepages.com/staticvsadaptiverouting You can see that the worst case static routing goes quickly below 40%, but the average eventually goes there as well. > There are some vendors that uses only the 24 port switches to build very > large scale clusters - 3000 nodes and above, without any > oversubscription, and they find it more cost effective. Using single > enclosures is easier, but the cables are not expensive and you can use Price of cables usually depends on the length (copper and fiber). Using small switches at the edges allows to use very short cables to the hosts (in-rack) but you still have to use the same number of longer cables to connect to the spine. With a single enclosure, you may need longer cables to reach the hosts (different rack), but you don't need cables to the spine as they are on the switch backplane (and PCB is free). Short cables may not be expensive, but they are not free. Furthermore, physical cables are much less reliable than wire on PCB, and they take more space, more power. Patrick From bill at cse.ucdavis.edu Mon Jun 16 17:51:21 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:11 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> Message-ID: <48570A89.3040608@cse.ucdavis.edu> Vincent Diepeveen wrote: > Jim, > > Reality is that the person who SPECULATES that something is good > also hides behind a DNA. This is another typical case of that. Er, who's hiding? Get your credit card out and buy one. CUDA is easily available. Today's new nvidia release has resulted in a handful at www.newegg.com, I'm sure many others will sell them to you as well. > On the one hand claiming a NDA, on the other hand implying that is a > very good product that will get > released. Indeed, there were numerous rumors, even mentions from researchers with pre-release hardware who claimed 2nd half 07 for double precision availability. > If NVIDIA clearly indicates towards me that they aren't gonna release > any technical data nor support anyone with technical > data (the word NDA has not been used by the way by either party, it was > a very clear NJET from 'em), > and all information i've got so far is very dissapointing, then hiding > behind a NDA is considered very bad manners. The CUDA docs look pretty good to me, as usual the definitions of latency, bandwidth, message passing, flops, and overall performance varies. So whatever you have in mind the best bet is to actually try it. Fortunately nvidia makes it rather easy to get going. If you write a microbenchmark I suspect you could get it run. > Then instead of a $200 pci-e card, we needed to buy expensive Tesla's > for that, without getting Even tesla's were single precision I believe, at least until today. > The few trying on those Tesla's, though they won't ever post this as > their job is fulltime GPU programming, > report so far very dissappointing numbers for applications that really > matter for our nations. People seem happy, of course they are far from general purpose, but the progress (especially with CUDA) seems pretty good. > Truth is that you can always do a claim of 1 teraflop of computing > power. If that doesn't get backupped by technical documents You mean like (mentioned on the list previously) http://arxiv.org/abs/0709.3225 > how to get it out of the hardware if your own testprograms show that you > can't get that out of the hardware, > it is rather useless to start programming for such a platform. GPUs are hardly suited for everyone, that doesn't make them useless. Personally finding a port of McCalpin's stream seeing 50GB/sec or so caught my attention. Sure it's not a recompile and go, but at least it's fairly c like. Function Rate (MB/s) Avg time Min time Max time Copy: 50077.8888 0.0013 0.0013 0.0013 Scale: 50637.4974 0.0013 0.0013 0.0013 Add: 51090.5662 0.0019 0.0019 0.0019 Triad: 50527.6617 0.0019 0.0019 0.0019 www.nvidia.com claims said card has a 57GB/sec memory system, today's new 260/280 are in the 130-140GB/sec range (advertised, not stream). > It is questionable whether it is interesting to design some algorithms > for GPU's; it takes endless testing of every tiny detail to figure out > what the GPU can and cannot do and to get accurate timings. By the time > you finish with that, you can also implement the same design in > FPGA or ASIC/VLSI whatever. As that is of course the type of interested Sounds wildly off base to me. But if you implement a FPGA/ASIC + memory system that costs less than $200 qty 1 and can be programmed in a few hours to implement stream at or above 50GB/sec let me know. > parties in GPU programming; > considering the amount of computing power they need, for the same budget > they can also make their own CPU's. Even assuming man months of highly paid programmers I don't see how you get from that cost to the budget for making your own CPU. > For other companies that i tried to get interested, there is a lot of > hesitation to even *investigate* that hardware, let alone give a contract > job to port their software to such hardware. Nvidia for all those > civilian and military parties is very very unattractive as of now. CUDA is the start of practical use of a GPU for non graphics related jobs, seems like a good start to me, sure not everything fits, but it's progress. Today's new chips add double precision which definitely helps. > IBM now shows up with a working supercomputer using new generation CELL > processors which have a sustained 77 Gflop double precision > a chip which means a tad more than 150 Gflop for each node. Each > rackmount node is relative cheap and performs very well. Sure, for rather specialized tasks. Each SPE has what, 32KB of memory? I think of it more as a DSP than a CPU. BTW the previous generation cell did double precision as well, it was just 1/10th as fast as single precision. The new revision is around 100GFlops/sec up from 14 ish. From lindahl at pbm.com Mon Jun 16 18:14:00 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:11 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <48570A89.3040608@cse.ucdavis.edu> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <48570A89.3040608@cse.ucdavis.edu> Message-ID: <20080617011359.GB6176@bx9.net> On Mon, Jun 16, 2008 at 05:51:21PM -0700, Bill Broadley wrote: > Personally finding a port of McCalpin's stream seeing 50GB/sec or so > caught my attention. Well, given that GPUs don't do so hot on Linpack, one should hope that they're close to peak on *something* ! For a while I was trying to incite some friends to go do a CPU that plugged into VRAM, but it didn't get very far. -- greg From perry at piermont.com Mon Jun 16 18:48:55 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <48568904.3020307@ias.edu> (Prentice Bisbal's message of "Mon\, 16 Jun 2008 11\:38\:44 -0400") References: <1210016466.4924.1.camel@Vigor13> <48568904.3020307@ias.edu> Message-ID: <87hcbs96go.fsf@snark.cb.piermont.com> Prentice Bisbal writes: > Completely untrue. One of my colleagues, who does a lot of work with GPU > processors for astrophysics calculations, was able to increase the > performance of the MD5 algorithm by ~100x with about 1.5 days of work. That's rather surprising. MD5 is a pure integer algorithm, and is well known for being unfriendly to vectorization. There is also extensive work by Keromytis et al on the use of GPUs for accelerating cryptographic operations, and I don't think they achieved anything like that sort of performance improvement. I'll point out, by the way... >* GPU (single GeForce 8800 Ultra on cylon): > 57,640,967.264473 hash/second ...that implies moving at least 3.7e9 bytes of data (MD5 operates on blocks of 64 bytes) into the GPU per second, entirely ignoring the 64 Feistel rounds within the GPU. Each round is 4 xors and a rotate, and they can't be done in parallel, so we get a total of about 1.8e10 integer ops (entirely ignoring the world shuffling) per second. That's... rather a lot. Perry -- Perry E. Metzger perry@piermont.com From bill at cse.ucdavis.edu Mon Jun 16 18:58:40 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:11 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <20080617011359.GB6176@bx9.net> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <48570A89.3040608@cse.ucdavis.edu> <20080617011359.GB6176@bx9.net> Message-ID: <48571A50.8020002@cse.ucdavis.edu> Greg Lindahl wrote: > On Mon, Jun 16, 2008 at 05:51:21PM -0700, Bill Broadley wrote: > >> Personally finding a port of McCalpin's stream seeing 50GB/sec or so >> caught my attention. > > Well, given that GPUs don't do so hot on Linpack, one should hope that > they're close to peak on *something* ! Heh. Is there a published linpack for some CUDA based solution? Or possibly code available? > For a while I was trying to incite some friends to go do a CPU that > plugged into VRAM, but it didn't get very far. Well hopefully ATI and AMD will figure out a way to give a $300 CPU as much memory bandwidth as a $200 video card. Seems like with the continuing increase in cores, not to mention the yet again increased width of the SSE registers would make increased bandwidth more usable. I'm pretty sure hypertransport allows for a significant number of outstanding memory transactions, so even a single gpu/cpu hybrid could farm out a 100GB/sec memory system to numerous sockets.... sounds like a good justification for HT3 to me. From perry at piermont.com Mon Jun 16 19:06:49 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:11 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <87hcbs96go.fsf@snark.cb.piermont.com> (Perry E. Metzger's message of "Mon\, 16 Jun 2008 21\:48\:55 -0400") References: <1210016466.4924.1.camel@Vigor13> <48568904.3020307@ias.edu> <87hcbs96go.fsf@snark.cb.piermont.com> Message-ID: <87d4mg95mu.fsf@snark.cb.piermont.com> "Perry E. Metzger" writes: > ...that implies moving at least 3.7e9 bytes of data (MD5 > operates on blocks of 64 bytes) into the GPU per second, entirely > ignoring the 64 Feistel rounds within the GPU. Each round is 4 xors > and a rotate, and they can't be done in parallel, so we get a total of > about 1.8e10 integer ops (entirely ignoring the world shuffling) per > second. That's... rather a lot. By the way, as an aside, dedicated IPSec hardware can keep up with doing HMAC-MD5 at gigabit ethernet speeds -- I don't think anyone has shown hardware capable of doing HMAC-MD5 faster than 10G ethernet. (I'm not even sure anyone has hardware that will keep up on 10GigE). Your friend is claiming he can do faster -- about 30Gbit/sec -- beating custom hardware optimized purely for doing MD5. That would clearly be of a lot of interest to many people if it were true. Perry From shaeffer at neuralscape.com Mon Jun 16 19:52:43 2008 From: shaeffer at neuralscape.com (Karen Shaeffer) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <20080617011359.GB6176@bx9.net> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <48570A89.3040608@cse.ucdavis.edu> <20080617011359.GB6176@bx9.net> Message-ID: <20080617025243.GC8621@synapse.neuralscape.com> On Mon, Jun 16, 2008 at 06:14:00PM -0700, Greg Lindahl wrote: > On Mon, Jun 16, 2008 at 05:51:21PM -0700, Bill Broadley wrote: > > > Personally finding a port of McCalpin's stream seeing 50GB/sec or so > > caught my attention. > > Well, given that GPUs don't do so hot on Linpack, one should hope that > they're close to peak on *something* ! > > For a while I was trying to incite some friends to go do a CPU that > plugged into VRAM, but it didn't get very far. > > -- greg Hi all, I don't want anyone to take this the wrong way. I think Nvidia is very serious about their technology. But they do appear to have some QA issues with their technologies. I might want to consider that, if I was an early adopter of new technology from Nvidia. BTW, I have done no serious evaluation of that statement, although I have experience finding some issues with their technology in the past. Thanks, Karen -- Karen Shaeffer Neuralscape, Palo Alto, Ca. 94306 shaeffer@neuralscape.com http://www.neuralscape.com From hahn at mcmaster.ca Mon Jun 16 20:32:00 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <87hcbs96go.fsf@snark.cb.piermont.com> References: <1210016466.4924.1.camel@Vigor13> <48568904.3020307@ias.edu> <87hcbs96go.fsf@snark.cb.piermont.com> Message-ID: > That's rather surprising. MD5 is a pure integer algorithm, and is well > known for being unfriendly to vectorization. There is also extensive the quoted performance was throughput, not latency, so perhaps they were simply doing a bunch of md5's at once. >> * GPU (single GeForce 8800 Ultra on cylon): >> 57,640,967.264473 hash/second > > ...that implies moving at least 3.7e9 bytes of data (MD5 > operates on blocks of 64 bytes) into the GPU per second, entirely the only comparable number I've heard quoted was in parallel rendering/compositing, where fairly recent systems bragged about reading back the framebuffer at 2 GB/s. From richard.walsh at comcast.net Tue Jun 17 06:52:10 2008 From: richard.walsh at comcast.net (richard.walsh@comcast.net) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? Message-ID: <061720081352.323.4857C18A00084455000001432215568884089C040E99D20B9D0E080C079D@comcast.net> -------------- Original message -------------- From: Vincent Diepeveen > Seems the next CELL is 100% confirmed double precision. Mmm ... "it seems ... 100% confirmed" ... this is confusing, it is either certain or "it seems ...", not both. I have not seen it confirmed anywhere that the PowerXCell (the new chip just announced) is 100% IEEE 754 double-precision compatible. The old generation chip was, but at 1/4 the speed. The new chip runs double precision at ~100 Gflops, and I am guessing that now its double precision has limited IEEE 754 compatibility like single precision did/does... not that this is important to all applications. If you "know" otherwise and can point me to supporting documentation, I would be interested. rbw -- "Making predictions is hard, especially about the future." Niels Bohr -- Richard Walsh Thrashing River Consulting-- 5605 Alameda St. Shoreview, MN 55126 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080617/bedd5a70/attachment.html From james.p.lux at jpl.nasa.gov Tue Jun 17 06:52:42 2008 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <48570A89.3040608@cse.ucdavis.edu> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <48570A89.3040608@cse.ucdavis.edu> Message-ID: <20080617065242.dy9c9t4xco4kw080@webmail.jpl.nasa.gov> Quoting Bill Broadley , on Mon 16 Jun 2008 05:51:21 PM PDT: > > >> parties in GPU programming; >> considering the amount of computing power they need, for the same >> budget they can also make their own CPU's. > > Even assuming man months of highly paid programmers I don't see how you > get from that cost to the budget for making your own CPU. There are some open source CPU cores in VHDL for things like the SPARC V8 around, if you want to try this out. There's also open source VHDL bus interfaces, memory, etc. Get yourself a copy of the appropriate tools and an appropriate FPGA eval board, and you're off and running. Once you have a solid design in FPGA, there are a number of fabs that will take it and turn it into a (faster, smaller, lower power) ASIC, with some form of guarantee (i.e. if it works in FPGA, and you follow their design rules, you're guaranteed a working ASIC). They do the fabs as multiproject wafers (like MOSIS) Typical overall cost is in sub $100K range, from what I understand. From james.p.lux at jpl.nasa.gov Tue Jun 17 07:01:16 2008 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <4856F7EC.7060000@ussarna.se> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> Message-ID: <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> Quoting Linus Harling , on Mon 16 Jun 2008 04:31:56 PM PDT: > Vincent Diepeveen skrev: > >> >> Then instead of a $200 pci-e card, we needed to buy expensive Tesla's >> for that, without getting >> very relevant indepth technical information on how to program for that >> type of hardware. >> >> The few trying on those Tesla's, though they won't ever post this as >> their job is fulltime GPU programming, >> report so far very dissappointing numbers for applications that really >> matter for our nations. > > > Tomography is kind of important to a lot of people: > > http://tech.slashdot.org/tech/08/05/31/1633214.shtml > http://www.dvhardware.net/article27538.html > http://fastra.ua.ac.be/en/index.html > > But of course, that was done with regular $500 cards, not Teslas. Mind you, if you go and get a tomographic scan today, they already use fast hardware to do it. Only researchers on limited budgets tolerate taking days to reduce the data on a desktop PC. And, while the concept of doing faster processing with a <10KEuro box is attractive in that environment, I suspect it's a long way from being commercially viable in that role. The current tomographic technology (e.g. GE Lightspeed) is pretty impressive. They slide you in, and 10-15 seconds later, there's 3 d rendered models and slices on the screen. The equipment is pretty hassle free, the UI straightforward from what I could see, etc. And, of course, people are willing (currently) to pay many millions for a machine to do this. I suspect that the other costs of running a CT scanner (both capital and operating) overwhelm the cost of the computing power, so going from a $100K box to a $20K box is a drop in the bucket. When you're talking MRI, for instance, there's the cost of the liquid helium for the magnets. That's a long way from a bunch of grad students racking up a bunch of PCs. From prentice at ias.edu Tue Jun 17 09:38:12 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <87hcbs96go.fsf@snark.cb.piermont.com> References: <1210016466.4924.1.camel@Vigor13> <48568904.3020307@ias.edu> <87hcbs96go.fsf@snark.cb.piermont.com> Message-ID: <4857E874.7000300@ias.edu> Perry E. Metzger wrote: > Prentice Bisbal writes: >> Completely untrue. One of my colleagues, who does a lot of work with GPU >> processors for astrophysics calculations, was able to increase the >> performance of the MD5 algorithm by ~100x with about 1.5 days of work. > > That's rather surprising. MD5 is a pure integer algorithm, and is well > known for being unfriendly to vectorization. There is also extensive > work by Keromytis et al on the use of GPUs for accelerating > cryptographic operations, and I don't think they achieved anything > like that sort of performance improvement. > > I'll point out, by the way... > >> * GPU (single GeForce 8800 Ultra on cylon): >> 57,640,967.264473 hash/second > > ...that implies moving at least 3.7e9 bytes of data (MD5 > operates on blocks of 64 bytes) into the GPU per second, entirely > ignoring the 64 Feistel rounds within the GPU. Each round is 4 xors > and a rotate, and they can't be done in parallel, so we get a total of > about 1.8e10 integer ops (entirely ignoring the world shuffling) per > second. That's... rather a lot. > > Perry Perry, I was just passing the information along from my colleague. I forwarded your response to him. Maybe he will reply himself and go into more detail about his findings for you. -- Prentice From diep at xs4all.nl Tue Jun 17 10:07:54 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> Message-ID: Jim, I feel you notice a very important thing here. That is that mainly for hobbyists like me a GPU is interesting to program for, or for companies who have a budget to buy less than a dozen of them in total. For ISPs the only thing that matters is the power consumption and for encryption at a low TCP/IP layer it's too easy to equip all those cheapo cpu's with encryption coprocessors which are like 1 watt or so and are delivering enough work to get the 100 mbit/1 gigabit nics fully busy, in case of public key it's in fact at a speed that you won't reach at a GPU when managing to parallellize it and it to work in a great manner. The ISPs look for full scalable stuff of course such machines, quite the opposite of having 1 card @ 250 watt. In fact it would be quite interesting to know how fast you can run RSA on a GPU. Where are the benchmarks there? I tend to remember that i posted some solution to do a fast generic modulo (of course not a new idea, but that you always hear after figuring something out), with a minimum of code under the condition you already have multiplying code. How fast can you multiply big numbers on those GPU's? 4096 x 4096 bits is most interesting there. Then of course take the modulo quickly and repeat this for the entire exponent-squaring. That is the only interesting question IMHO, what amount of throughput does it deliver for RSA4096. I tend to remember a big bug it has in such a case is that the older cards (8800 etc) only can do 16 x 16 bits == 32 bits, whereas at CPU's you can use 64 x 64 bits == 128 bits. BIG difference in speed. Yet those hobbyists who are the interested persons in GPU programming have limited time to get software to run and have a budget far smaller than $10k. They're not even gonna buy as much Tesla's as NASA will. Not a dozen. The state in which gpu programming is now is that some big companies can have a person toy fulltime with 1 gpu, as of course the idea of having a cpu with hundreds of cores is very attractive and looks like a realistic future, so companies must explore that future. Of course every GPU/CPU company is serious in their aim to produce products that perform well, we all do not doubt it. Yet it is only attractive to hobbyists and those hobbyists are not gonna get any interesting technical data needed to get the maximum out of the GPU's from Nvidia. This is a big problem. Those hobbyists have very limited time to get their sparetime products done to do numbercrunching, so being busy fulltime writing testprograms to know everything about 1 specific GPU is not something they all like to do for a hobby. Just having that information will attract the hobbyists as they are willing to take the risk to buy 1 Tesla and spend time there. That produces software. That software will have a certain performance, based upon that performance perhaps some companies might get interested. Intel and AMD will be doing better there i hope. Note that testing CUDA also is suboptimal, it just runs for 5 seconds or so max. You need a machine with a 2nd videocard. That requires a mainboard with at least 2x pci-e 16x. How to cluster that? My cluster cards are pci-x not pci-e. quadrics QM400's. I can get boards @ 139 euro with 1 slot PCI-E 16x and build quadcore Q6600 nodes @ 500 euro, as soon as i have a job again. My macbookpro 17'' has no pci-e 16x slot free though. So for number crunching, a cluster always wins it from a single nvidia videocard. The communication speed over the pci-e from the videocards is too slow latency to parallellize software that is not-embarrassingly parallel. Majority of hobbyists will have a similar problem with nvidia, very sad in itself. A good CUDA setup that can beat a simplistic cluster is not so cheap and easy to program for like building that cluster is. Also the memory scales better in those clusters than it does for the cards. If 1 node can do less work than 1 GPU can, it still means that it's easier to get that exponential speedup by having a shared cache across all nodes (this is true for a lot of modern crunching algorithms). With a GPU you're forced to do all calculation including caching within the GPU and within the limited device RAM. Now in contradiction to what most people tend to believe, usually there is methods to get away with a limited amount of RAM with modern overwriting methods of caching, even when that loses a factor 2 there is ways to overcome that. The biggest limitation is that communication with other nodes is real hard. Scaling to more nodes is just not gonna happen of course as the latency between the nodes is real bad and it is an extra slow hop to latency of course. First from device RAM to RAM then from RAM to card and from card to RAM and from RAM to device RAM. Let's make a list of problems that most clusters here calculate upon and you'll see how much the GPU concept still needs to get matured to get it to work well for most codes. Software that needs low latency interconnects you could get to work within 1 card only therefore, provided the RAM access is not bottlenecked. Yet all reports so far indicate it is. No information there is just not very encouraging and for professional crunching work where companies therefore have a big budget for, building or buying in your own low power co-processor that so to speak even fits into an ipod is just too easy. So in the end i guess some stupid extension of SSE will give a bigger increase in crunching power than the in itself attractive gpgpu hardware. The biggest limitation being development time from hobbyists. Vincent On Jun 17, 2008, at 4:01 PM, Jim Lux wrote: > Quoting Linus Harling , on Mon 16 Jun 2008 > 04:31:56 PM PDT: > >> Vincent Diepeveen skrev: >> >>> >>> Then instead of a $200 pci-e card, we needed to buy expensive >>> Tesla's >>> for that, without getting >>> very relevant indepth technical information on how to program for >>> that >>> type of hardware. >>> >>> The few trying on those Tesla's, though they won't ever post this as >>> their job is fulltime GPU programming, >>> report so far very dissappointing numbers for applications that >>> really >>> matter for our nations. >> >> >> Tomography is kind of important to a lot of people: >> >> http://tech.slashdot.org/tech/08/05/31/1633214.shtml >> http://www.dvhardware.net/article27538.html >> http://fastra.ua.ac.be/en/index.html >> >> But of course, that was done with regular $500 cards, not Teslas. > > > Mind you, if you go and get a tomographic scan today, they already > use fast hardware to do it. Only researchers on limited budgets > tolerate taking days to reduce the data on a desktop PC. And, while > the concept of doing faster processing with a <10KEuro box is > attractive in that environment, I suspect it's a long way from > being commercially viable in that role. > > The current tomographic technology (e.g. GE Lightspeed) is pretty > impressive. They slide you in, and 10-15 seconds later, there's 3 > d rendered models and slices on the screen. The equipment is > pretty hassle free, the UI straightforward from what I could see, etc. > > And, of course, people are willing (currently) to pay many millions > for a machine to do this. I suspect that the other costs of > running a CT scanner (both capital and operating) overwhelm the > cost of the computing power, so going from a $100K box to a $20K > box is a drop in the bucket. When you're talking MRI, for > instance, there's the cost of the liquid helium for the magnets. > > That's a long way from a bunch of grad students racking up a bunch > of PCs. > > > > > > > From James.P.Lux at jpl.nasa.gov Tue Jun 17 10:43:07 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> Message-ID: <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> At 10:07 AM 6/17/2008, Vincent Diepeveen wrote: >Jim, > >I feel you notice a very important thing here. > >That is that mainly for hobbyists like me a GPU is interesting to >program for, or for companies who have a budget to buy less than a dozen >of them in total. And, as you say, that's a hobby/R&D market where you're willing to spend more in labor than in hardware. >For ISPs the only thing that matters is the power consumption and for >encryption at a low TCP/IP layer it's too easy to equip all those >cheapo cpu's with encryption coprocessors which are like 1 watt or so >and are delivering enough work to get the 100 mbit/1 gigabit nics >fully busy, in case of public key it's in fact at a speed that you >won't reach at a GPU when managing to parallellize it and it to work >in a great >manner. The ISPs look for full scalable stuff of course such >machines, quite the opposite of having 1 card @ 250 watt. I don't know much about the economies of running an ISP. While electrical power (and cooling,etc.) might be a big chunk of their budget, I suspect that mundane business stuff like advertising, billing, account management, etc. might actually be a bigger slice. For instance, do co-lo facilities charge you for power, or is it like office space, where you rent it by the square foot, and an assumed amount of power and HVAC comes with the price. >Yet those hobbyists who are the interested persons in GPU programming >have limited time >to get software to run and have a budget far smaller than $10k. >They're not even gonna buy as much Tesla's as NASA will. >Not a dozen. There, I think you're wrong. There's lots of hobbyists and tinkerer's of one sort or another out there. I'd bet they sell at least thousands of them. >The state in which gpu programming is now is that some big companies >can have a person toy fulltime with 1 gpu, >as of course the idea of having a cpu with hundreds of cores is very >attractive and looks like a realistic future, >so companies must explore that future. The various flavors of multi-core in a field of RAM have been around for decades, because it's (superficially?) attractive from a scalability standpoint. The problem, as everyone on this list is aware, is effectively using such an architecture.. parallelizing isn't trivial. There's a reason they still sell mainframe computers, but, hope does spring eternal. >Of course every GPU/CPU company is serious in their aim to produce >products that perform well, we all do not doubt it. Not necessarily, unless your performance metric is shareholder return. It is the rare company that can make a business of selling solely on top-end performance (e.g. Cray). There's also several strategies and target markets. If you have good manufacturing capability for large quantities, you adjust your performance to what consumers will buy at a price you can make money on. If you're in a more "fee for service" model, then you likely are doing smaller unit volumes, but the units cost a lot more (I suspect that most of the cluster vendors on this list fall in this category), but still, in the long run the cost to do the job MUST be less than what the customer is willing to pay (unless the owner is some sort of philanthropist, naive, or a fool) >Yet it is only attractive to hobbyists and those hobbyists are not >gonna get any interesting technical data needed to get the maximum >out of the GPU's from Nvidia. This is a big problem. Those hobbyists >have very limited time to get their sparetime products done >to do numbercrunching, So it's basically an investment decision. How much value do you want to get out of your investment of time or money? If you're only willing to spend a few hours, then you must not value the end state of the work very highly (or, more correctly, you value something else more highly...). > so being busy fulltime writing testprograms to >know everything about 1 specific GPU is not something they >all like to do for a hobby. Just having that information will attract >the hobbyists as they are willing to take the risk to buy 1 Tesla and >spend time there. That produces software. That software will have a >certain performance, >based upon that performance perhaps some companies might get interested. To a certain extent, this is the "build it and they will come" model. It's not one that is going to make any real difference to Nvidia's bottom line, so they're unlikely to invest more than a token amount in it. >So in the end i guess some stupid extension of SSE will give a bigger >increase in crunching power than the in itself attractive gpgpu >hardware. >The biggest limitation being development time from hobbyists. And HPC hobbyists are a very tiny market, not worth very much commercially. From lindahl at pbm.com Tue Jun 17 11:23:56 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> Message-ID: <20080617182356.GA28059@bx9.net> On Tue, Jun 17, 2008 at 10:43:07AM -0700, Jim Lux wrote: > For instance, do co-lo facilities charge you for power, or is > it like office space, where you rent it by the square foot, and an > assumed amount of power and HVAC comes with the price. The typical colo charges by space, with a certain amount of power per rack, but in practice the limit is power. As an example, the new colo I'm moving into only gives me enough power for 16 machines per rack. But older colos are only 2/3 of that. These are HPCish nodes. I was pretty surprised by this, as most HPC machine rooms allow a significantly higher power density. > but still, in > the long run the cost to do the job MUST be less than what the > customer is willing to pay (unless the owner is some sort of > philanthropist, naive, or a fool) Mmmmmmmmmmmmf. MMMMF! -- greg From James.P.Lux at jpl.nasa.gov Tue Jun 17 11:41:34 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <20080617182356.GA28059@bx9.net> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> <20080617182356.GA28059@bx9.net> Message-ID: <6.2.5.6.2.20080617113801.02f7ca70@jpl.nasa.gov> At 11:23 AM 6/17/2008, Greg Lindahl wrote: >On Tue, Jun 17, 2008 at 10:43:07AM -0700, Jim Lux wrote: > > > For instance, do co-lo facilities charge you for power, or is > > it like office space, where you rent it by the square foot, and an > > assumed amount of power and HVAC comes with the price. > >The typical colo charges by space, with a certain amount of power per >rack, but in practice the limit is power. As an example, the new colo >I'm moving into only gives me enough power for 16 machines per >rack. But older colos are only 2/3 of that. These are HPCish nodes. > >I was pretty surprised by this, as most HPC machine rooms allow a >significantly higher power density. > > > but still, in > > the long run the cost to do the job MUST be less than what the > > customer is willing to pay (unless the owner is some sort of > > philanthropist, naive, or a fool) > >Mmmmmmmmmmmmf. MMMMF! > >-- greg Well.. to be fair, there were (and still are) businesses out there (particularly a few years ago) that didn't fully understand the concept of needing net profit. (ah yes, the glory days of startups "buying market share" in the dot-com bubble) And, some folks made a fine living in the mean time. (But, then, those folks weren't the owners, were they, or if they were, in a limited sense, they now have some decorative wallpaper..) From lindahl at pbm.com Tue Jun 17 11:53:08 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <48571A50.8020002@cse.ucdavis.edu> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <48570A89.3040608@cse.ucdavis.edu> <20080617011359.GB6176@bx9.net> <48571A50.8020002@cse.ucdavis.edu> Message-ID: <20080617185307.GB28059@bx9.net> On Mon, Jun 16, 2008 at 06:58:40PM -0700, Bill Broadley wrote: > Heh. Is there a published linpack for some CUDA based solution? Or > possibly code available? http://www.hp.com/techservers/hpccn/hpccollaboration/ADCatalyst/downloads/accelerating-HPCUsing-GPUs.pdf says that some folks at Berkeley wrote some SGEMM code and that it achieves ~ 165 Gflops out of a 4-GPU Tesla setup. That's waaaaay down from the alleged peak of that box. > I'm pretty sure hypertransport allows for a significant number of > outstanding memory transactions, so even a single gpu/cpu hybrid could farm > out a 100GB/sec memory system to numerous sockets.... sounds like a good > justification for HT3 to me. Hypertransport is just a bus, it's the northbridge+cpu that determines how many outstanding transactions you can have. -- greg From richard.walsh at comcast.net Tue Jun 17 12:00:58 2008 From: richard.walsh at comcast.net (richard.walsh@comcast.net) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? Message-ID: <061720081900.1570.485809EA0009C560000006222214756402089C040E99D20B9D0E080C079D@comcast.net> -------------- Original message -------------- From: Jim Lux > Well.. to be fair, there were (and still are) businesses out there > (particularly a few years ago) that didn't fully understand the > concept of needing net profit. (ah yes, the glory days of startups > "buying market share" in the dot-com bubble) And, some folks made a > fine living in the mean time. (But, then, those folks weren't the > owners, were they, or if they were, in a limited sense, they now have > some decorative wallpaper..) > Hey Jim, Gold rushes are good (and greed I guess too ... ;-) ...) ... there IS often gold in them there hills, it is just that very few if anyone knows exactly where. So, the less risk averse among us and those with more money than sense (thankfully, I say) starting digging. Most of their trials end in error, but the rest of us benefit from the few that are lucky/smart enough to find it. I think you are assuming that the futures are far more predictable than it in fact is, even for the best and brightest like yourself ... what percentage of the HPC market will accelerators have at this time next year? Regards, rbw -- "Making predictions is hard, especially about the future." Niels Bohr -- Richard Walsh Thrashing River Consulting-- 5605 Alameda St. Shoreview, MN 55126 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080617/8dd5911c/attachment.html From hahn at mcmaster.ca Tue Jun 17 12:05:33 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <20080617182356.GA28059@bx9.net> References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> <20080617182356.GA28059@bx9.net> Message-ID: > I'm moving into only gives me enough power for 16 machines per > rack. But older colos are only 2/3 of that. These are HPCish nodes. I guess I'm not surprised, but then again, it does make me scratch my head about vendors who are still in love with blades. you know, headlines like "HP puts 1000 cores in a rack". and who buys them? I picture PHB's buying blade chassis and then putting just one per rack for heat-density reasons. maybe blades are creating a giant market for blanking panels ;) From shaeffer at neuralscape.com Tue Jun 17 12:59:34 2008 From: shaeffer at neuralscape.com (Karen Shaeffer) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <6.2.5.6.2.20080617113801.02f7ca70@jpl.nasa.gov> References: <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> <20080617182356.GA28059@bx9.net> <6.2.5.6.2.20080617113801.02f7ca70@jpl.nasa.gov> Message-ID: <20080617195934.GB20890@synapse.neuralscape.com> On Tue, Jun 17, 2008 at 11:41:34AM -0700, Jim Lux wrote: > > Well.. to be fair, there were (and still are) businesses out there > (particularly a few years ago) that didn't fully understand the > concept of needing net profit. (ah yes, the glory days of startups > "buying market share" in the dot-com bubble) And, some folks made a > fine living in the mean time. (But, then, those folks weren't the > owners, were they, or if they were, in a limited sense, they now have > some decorative wallpaper..) > Hi Jim, I think you have the common view about this. The reality is many of those same companies would be making money today. They were just ahead of their time -- which is very common here in Silicon Valley. Having good ideas or exceptional technology is not enough -- you need to time it right -- or you don't survive. I love the example of IP telephony. I first heard of friends in IP telephony startups in 1993-4 time frame. They and a long line of others went bankrupt, until Skype hit it big. The .com bust is about as hyped as you can get. They used to love to bash San Francisco as the symbol of the bust. Take a look at this month's cover of San Francisco magazine. Timing is everything in life. (smiles ;) http://www.sanfranmag.com/ And even worse, they lumped the failure of big iron companies and especially Sun Microsystems during that time with the .com bust, when, in reality, everyone was moving to commodity hardware and Linux during that time frame -- the business press just didn't want to report that at the time... Go figure. And timing isn't just about being to early. A good example today is all those new Internet search companies that are litering Silicon Valley today. Most of them will fail, because they are too late. Some will get bought out -- but most are going to fail. Timing works both ways. Just my view from the trenches of Silicon Valley, Karen -- Karen Shaeffer Neuralscape, Palo Alto, Ca. 94306 shaeffer@neuralscape.com http://www.neuralscape.com From James.P.Lux at jpl.nasa.gov Tue Jun 17 13:25:49 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <20080617195934.GB20890@synapse.neuralscape.com> References: <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> <20080617182356.GA28059@bx9.net> <6.2.5.6.2.20080617113801.02f7ca70@jpl.nasa.gov> <20080617195934.GB20890@synapse.neuralscape.com> Message-ID: <6.2.5.6.2.20080617131658.02f713e8@jpl.nasa.gov> At 12:59 PM 6/17/2008, Karen Shaeffer wrote: >On Tue, Jun 17, 2008 at 11:41:34AM -0700, Jim Lux wrote: > > > > Well.. to be fair, there were (and still are) businesses out there > > (particularly a few years ago) that didn't fully understand the > > concept of needing net profit. (ah yes, the glory days of startups > > "buying market share" in the dot-com bubble) And, some folks made a > > fine living in the mean time. (But, then, those folks weren't the > > owners, were they, or if they were, in a limited sense, they now have > > some decorative wallpaper..) > > > >Hi Jim, >I think you have the common view about this. The reality is many of >those same companies would be making money today. They were just >ahead of their time -- which is very common here in Silicon Valley. Hmmm.. I don't know about that.. having a solution with no problem that needs to be solved isn't a valid business model. You could equally well argue that a company with lots of smart employees, but no ideas, is also ahead of it's time. Either you have all the elements, or you don't. >Having good ideas or exceptional technology is not enough -- you >need to time it right -- or you don't survive. I love the example >of IP telephony. I first heard of friends in IP telephony startups >in 1993-4 time frame. They and a long line of others went bankrupt, >until Skype hit it big. Back in 93, the telcos were still talking ISDN and ATM to the desktop as their model..whether it delivered IP packets (X.25, anyone?) or circuit switched digital voice was sort of immaterial. Big firms already mixed voice/data on their data connections (e.g. Micom was building such data/voice switches for decades by then), but what made IP telephony take off, I think, was sufficient installed density of sufficiently high speed internet access with flat rate billing. It's the flat rate billing that really did it, because it got around the "pay per minute" model for standard telephony, which was already on the way out for big customers, but very entrenched for the consumer. The long distance resellers really cashed in, because they could buy flat rate bulk and resell by the minute for 5c/minute or less. >The .com bust is about as hyped as you can get. They used to love >to bash San Francisco as the symbol of the bust. Take a look at this >month's cover of San Francisco magazine. Timing is everything in >life. (smiles ;) > >http://www.sanfranmag.com/ > >And even worse, they lumped the failure of big iron companies >and especially Sun Microsystems during that time with the .com >bust, when, in reality, everyone was moving to commodity hardware >and Linux during that time frame -- the business press just didn't >want to report that at the time... Go figure. To a certain extent, the big iron companies had troubles, though, because of the .com bubble in general. All this cash flooding into the market looking for investments, so really, really speculative ventures got funded (sort of like cash flooding into the collateralized debt obligation/mortgage backed securities market), and that funding drove deposits on equipment from the big iron companies, etc. From landman at scalableinformatics.com Tue Jun 17 14:12:00 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <6.2.5.6.2.20080617131658.02f713e8@jpl.nasa.gov> References: <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> <20080617182356.GA28059@bx9.net> <6.2.5.6.2.20080617113801.02f7ca70@jpl.nasa.gov> <20080617195934.GB20890@synapse.neuralscape.com> <6.2.5.6.2.20080617131658.02f713e8@jpl.nasa.gov> Message-ID: <485828A0.5040701@scalableinformatics.com> Jim Lux wrote: > At 12:59 PM 6/17/2008, Karen Shaeffer wrote: >> On Tue, Jun 17, 2008 at 11:41:34AM -0700, Jim Lux wrote: >> > >> > Well.. to be fair, there were (and still are) businesses out there >> > (particularly a few years ago) that didn't fully understand the >> > concept of needing net profit. (ah yes, the glory days of startups >> > "buying market share" in the dot-com bubble) And, some folks made a >> > fine living in the mean time. (But, then, those folks weren't the >> > owners, were they, or if they were, in a limited sense, they now have >> > some decorative wallpaper..) >> > >> >> Hi Jim, >> I think you have the common view about this. The reality is many of >> those same companies would be making money today. They were just >> ahead of their time -- which is very common here in Silicon Valley. > > Hmmm.. I don't know about that.. having a solution with no problem that > needs to be solved isn't a valid business model. You could equally well [hmmm.... how do we get Jim to lecture to VC's about funding the next social wannabe linky-linky site? ] [...] >> And even worse, they lumped the failure of big iron companies >> and especially Sun Microsystems during that time with the .com >> bust, when, in reality, everyone was moving to commodity hardware >> and Linux during that time frame -- the business press just didn't >> want to report that at the time... Go figure. Yup. Most of them still don't want to admit it. > To a certain extent, the big iron companies had troubles, though, > because of the .com bubble in general. All this cash flooding into the > market looking for investments, so really, really speculative ventures > got funded (sort of like cash flooding into the collateralized debt > obligation/mortgage backed securities market), and that funding drove > deposits on equipment from the big iron companies, etc. Well, there were secondary affects. Cisco having a $0.5T valuation was one of those. At SGI, one of the "oh feces" moments we had was when TJ, who wasn't throwing up in a hotel pool somewhere, noted that some SGI gear made for great web servers in 1995 or so. Then we (while I was there) managed to completely miss that market. Coulda rode the bubble up ... More seriously, getting the business model right is hard. Making it work is hard. Getting investments from groups that understand these things is hard. There was too much speculation, not enough focus upon the model. I would argue there is lots of that today with the social bits. How many social websites are there? What are they really worth? What is their revenue model? How will they make money (apart from being sold to larger companies)? Why are VCs funding them? -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From diep at xs4all.nl Tue Jun 17 14:28:24 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <061720081900.1570.485809EA0009C560000006222214756402089C040E99D20B9D0E080C079D@comcast.net> References: <061720081900.1570.485809EA0009C560000006222214756402089C040E99D20B9D0E080C079D@comcast.net> Message-ID: Richard, The big question that we should raise to answer your HPC market is whether assembler coded x64 SSE2 code also counts. Nothing as weird as all those bizarre SSE2 instructions. There is no way you can make an objective analysis why some weirdo instructions are in and why some very useful stuff is not in. Let's zoom in into one detail. What we really need in SSE2 to really speed up bigtime some FFT type transforms as well as make it attractive to all kind of small sized codes is to have an instruction lowbitsmultiply. If we represent a register as 2 x 64 bits integers: A:B and we multiply with C:D. Then we want to be able to execute the next 2 instructions, especially at intel processors: lowbitsmultiply : A:B * C:D ===> AC (mod 2^64) : BD (mod 2^64) highbitsmultiply : A:B * C:D ===> AC >> 64 : BD >> 64 where '>>' means shiftright 64 Basically in floating point the itanium could have such instructions as the above, so why not the PC processors also for 64 bits? Of course we need these 2 instructions a tad faster than currently multiplying an integer is in the integer units. We want a throughput of 1 cycle without blocking other execution units of course. When further vectorizing the SSE units it is of course not nice when a single instruction in one of the execution units blocks all others. There is no real perfect hardware from HPC viewpoint. Fiddling with SSE2 instructions is something very few programmers are good at in order to get their thing done in a cryptic manner, as algorithms and caches haven't become simpler since the 80s. So i would argue that the number crunching market already gets dominated by SSE2 acceleration and will see even more of that type of stuff, which basically means a return to the 80s in some sense for programmers who want to get the maximum out of a CPU. Interesting now is when Intel comes with something and AMD with yet unother crunching koncept (YUCK), taking care more cash gets pocketted by programmers. So that is good news for the jobless low level programmers (me, says the fool) :) Note if GPU's get equipped with this type of instructions, even when just 32 bits, that would already be a big step, as that allows CRT (Chinese Remainder Theorem) to get the job done, which is a manner to currently solve it at the PC. Point is that register size and which type of instructions your hardware supports matters. It especially matters to release which ones you support. Videocards simply have the luck that our coordinate system can get expressed easily with 32 bits floating points. When needed even 16 bits. So i'd not wait for a GPU that is entirely 64 bits double precision and/or 64 bits integers, as that would slow them down for their biggest market. Note that it would on paper be possible right now to make a chessprogram within a GPU run real fast, if the RAM accesses would be latency optimized (so 2 random acceses of all stream processors, each 10000 cycles to the RAM, first access being a read, the second one a write). All other theoretic problems i solved on paper. Yet this small 'problem' is dominating it that much that i'm not gonna gamble programming in order to find out perhaps that my assumption is correct :) However to reveal some of the math to compare the GPU's versus the CPU's when i would design a new 'braindead' chessprogram: Let's say we have 240 cores 32 bits at 0.675 Ghz Let's assume now we need 10k cycles for each node at each streamprocessor: (1 nps = 1 node per second = 1 chessposition a second searching for the holy grail) I assume on average 1 instruction a cycle (dangerous assumption considering there is 2 RAM hits also) 675000k / 10k = 67.5k nps 240 * 67.5k nps = 16.2 million nps Now if we compare with what i fight against in tournaments, that is a skulltrail mainboard with 2 xeon cpu's overclocked to 4Ghz, so 8 cores in total with big RAM. If i see what practical nps is that fast chessprograms get at it right now, then that is about 20 million nps. PC faster than GPU. I will skip all kind of technical discussions, such as where PC loses something; its memory controller is not near fast enough to serve every node, so it loses bigtime there last plies, at least 20-30%, and that we assume the same speedup for 8 cores versus 240 cores, where game tree search is one of the hardest to parallellize challenges; so you effectively will lose a lot more at 240 cores than at 8 cores. In fact i would guess it's 30% efficiency for the GPU versus 87.5% for the PC. Yet there is things to discover there which make up for an interesting challenge so i skip that algorithmic discussion entirely here, for now. In short for problems that the past were latency oriented, the dominating factor is: "how many instructions per second can you execute". The PC simply wins it from the GPU here, this for a typical 32 bits problem (in case of my chess software). The PC simply can effectively execute more instructions a cycle, when also counting the instructions it can skip by taking branches. Note that the PC also wins it in powerconsumption at 4Ghz @ 2 sockets. It has taken me many months to redo the above math, trying to find a solution to get somehow GPU faster than PC. Didn't manage so far. The biggest 2 differences between a CPU and a GPU when doing such math is the low clock of the GPU versus the high clock of the CPU. CPU at events already wins a factor 6 nearly just based upon clockspeed. I've showed up at a world championship in 2003 with a supercomputer with 500Mhz cpu's and fought against opponents at 2.8Ghz MP Xeon cpu's. Also a factor 6 difference nearly already in clockspeed. That is really a big dang in your selfconfidence. Maybe it is good that Greg Lindahl didn't join in that event, he would've googled a tad and showed up with that one-liner of Seymour Cray. Vincent On Jun 17, 2008, at 9:00 PM, richard.walsh@comcast.net wrote: > > -------------- Original message -------------- > From: Jim Lux > > > Well.. to be fair, there were (and still are) businesses out there > > (particularly a few years ago) that didn't fully understand the > > concept of needing net profit. (ah yes, the glory days of startups > > "buying market share" in the dot-com bubble) And, some folks made a > > fine living in the mean time. (But, then, those folks weren't the > > owners, were they, or if they were, in a limited sense, they now > have > > some decorative wallpaper..) > > > > Hey Jim, > > Gold rushes are good (and greed I guess too ... ;-) ...) ... there > IS often gold in them there hills, it is just that very few if > anyone knows exactly where. So, the less risk averse among us and > those with more money than sense (thankfully, I say) starting > digging. Most of their trials end in error, but the rest of us > benefit from the few that are lucky/smart enough to find it. I > think you are assuming that the futures are far more predictable > than it in fact is, even for the best and brightest like > yourself ... what percentage of the HPC market will accelerators > have at this time next year? > > Regards, > > rbw > > -- > > "Making predictions is hard, especially about the future." > > Niels Bohr > > -- > > Richard Walsh > Thrashing River Consulting-- > 5605 Alameda St. > Shoreview, MN 55126 > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From shaeffer at neuralscape.com Tue Jun 17 14:51:34 2008 From: shaeffer at neuralscape.com (Karen Shaeffer) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <6.2.5.6.2.20080617131658.02f713e8@jpl.nasa.gov> References: <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> <20080617182356.GA28059@bx9.net> <6.2.5.6.2.20080617113801.02f7ca70@jpl.nasa.gov> <20080617195934.GB20890@synapse.neuralscape.com> <6.2.5.6.2.20080617131658.02f713e8@jpl.nasa.gov> Message-ID: <20080617215134.GA21429@synapse.neuralscape.com> On Tue, Jun 17, 2008 at 01:25:49PM -0700, Jim Lux wrote: > At 12:59 PM 6/17/2008, Karen Shaeffer wrote: > >Hi Jim, > >I think you have the common view about this. The reality is many of > >those same companies would be making money today. They were just > >ahead of their time -- which is very common here in Silicon Valley. > > Hmmm.. I don't know about that.. having a solution with no problem > that needs to be solved isn't a valid business model. You could > equally well argue that a company with lots of smart employees, but > no ideas, is also ahead of it's time. > > Either you have all the elements, or you don't. Have you ever worked in Silicon Valley? Much of the thinking is about what is emerging, and how long it will take to emerge. This is the mindset for the big winners, but it is also the highest risk. Once all the elements are in place, then that is the secondary market of what I call the carpet baggers. Actually there are a lot of those folks too. > >The .com bust is about as hyped as you can get. They used to love > >to bash San Francisco as the symbol of the bust. Take a look at this > >month's cover of San Francisco magazine. Timing is everything in > >life. (smiles ;) > > > >http://www.sanfranmag.com/ > > > >And even worse, they lumped the failure of big iron companies > >and especially Sun Microsystems during that time with the .com > >bust, when, in reality, everyone was moving to commodity hardware > >and Linux during that time frame -- the business press just didn't > >want to report that at the time... Go figure. > > > To a certain extent, the big iron companies had troubles, though, > because of the .com bubble in general. All this cash flooding into > the market looking for investments, so really, really speculative > ventures got funded (sort of like cash flooding into the > collateralized debt obligation/mortgage backed securities market), > and that funding drove deposits on equipment from the big iron companies, > etc. I think you nailed that. Silicon Valley is driven by greed. And no group is more full of greed than the investors of Silicon Valley -- as it should be. We all live by the golden rule -- Those with the gold make the rule. What is almost comical is how they engage in herd investing. They watch what their counterparts are investing in -- and they often need to invest in a competing company to avoid looking clueless. There is cover in that model -- If it hits big, you can say you were playing in the market, even if your investment lost. And if that market flops, you can say that everyone else made the same mistake as you. This is very common here in Silicon Valley. And it definitely was at play in the hyper-investments that set up the .com bust -- herd mentality that required everyone to be dumping money into the emerging Internet paradigm -- early. You gotta love Silicon Valley. I wouldn't live any other place in the world. (smiles ;) Thanks for your comments. They are always interesting. I actually have work to get done today. Gotta go. Karen -- Karen Shaeffer Neuralscape, Palo Alto, Ca. 94306 shaeffer@neuralscape.com http://www.neuralscape.com From James.P.Lux at jpl.nasa.gov Tue Jun 17 15:18:02 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <485828A0.5040701@scalableinformatics.com> References: <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> <20080617182356.GA28059@bx9.net> <6.2.5.6.2.20080617113801.02f7ca70@jpl.nasa.gov> <20080617195934.GB20890@synapse.neuralscape.com> <6.2.5.6.2.20080617131658.02f713e8@jpl.nasa.gov> <485828A0.5040701@scalableinformatics.com> Message-ID: <6.2.5.6.2.20080617141600.02fa7660@jpl.nasa.gov> At 02:12 PM 6/17/2008, Joe Landman wrote: >Jim Lux wrote: >>At 12:59 PM 6/17/2008, Karen Shaeffer wrote: >>>On Tue, Jun 17, 2008 at 11:41:34AM -0700, Jim Lux wrote: >>> > >>> > Well.. to be fair, there were (and still are) businesses out there >>> > (particularly a few years ago) that didn't fully understand the >>> > concept of needing net profit. (ah yes, the glory days of startups >>> > "buying market share" in the dot-com bubble) And, some folks made a >>> > fine living in the mean time. (But, then, those folks weren't the >>> > owners, were they, or if they were, in a limited sense, they now have >>> > some decorative wallpaper..) >>> > >>> >>>Hi Jim, >>>I think you have the common view about this. The reality is many of >>>those same companies would be making money today. They were just >>>ahead of their time -- which is very common here in Silicon Valley. >>Hmmm.. I don't know about that.. having a solution with no problem >>that needs to be solved isn't a valid business model. You could equally well > >[hmmm.... how do we get Jim to lecture to VC's about funding the >next social wannabe linky-linky site? ] Ahh... those sites have advertising, which is their business model. The links are just there to get you to see the ads and are more socially acceptable and allow a narrower tailoring to specific target markets allowing higher ad rates, than something of broad appeal (mentos & coke or porn). >Well, there were secondary affects. Cisco having a $0.5T valuation >was one of those. At SGI, one of the "oh feces" moments we had was >when TJ, who wasn't throwing up in a hotel pool somewhere, noted >that some SGI gear made for great web servers in 1995 or so. Then >we (while I was there) managed to completely miss that >market. Coulda rode the bubble up ... > >More seriously, getting the business model right is hard. Making it >work is hard. Getting investments from groups that understand these >things is hard. There was too much speculation, not enough focus >upon the model. I would argue there is lots of that today with the >social bits. How many social websites are there? What are they >really worth? What is their revenue model? How will they make >money (apart from being sold to larger companies)? Why are VCs funding them? The VCs funding them do so because it's a high risk gamble that can potentially pay off big. Not everyone wants to invest in CDs or mutual funds. You could, for instance, invest in something that has a probability of paying off of 0.01, but has a return of 1500% and come out ahead, in an expected value sense. There's also all manner of money to be made in the process of doing the investing (i.e. fees). As they say in the horse racing business, the guys shoveling the manure always make money, even the horse owners lose money overall. From James.P.Lux at jpl.nasa.gov Tue Jun 17 15:24:54 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: <20080617215134.GA21429@synapse.neuralscape.com> References: <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> <20080617182356.GA28059@bx9.net> <6.2.5.6.2.20080617113801.02f7ca70@jpl.nasa.gov> <20080617195934.GB20890@synapse.neuralscape.com> <6.2.5.6.2.20080617131658.02f713e8@jpl.nasa.gov> <20080617215134.GA21429@synapse.neuralscape.com> Message-ID: <6.2.5.6.2.20080617151932.02fc5a58@jpl.nasa.gov> At 02:51 PM 6/17/2008, Karen Shaeffer wrote: >On Tue, Jun 17, 2008 at 01:25:49PM -0700, Jim Lux wrote: > > At 12:59 PM 6/17/2008, Karen Shaeffer wrote: > > >Hi Jim, > > >I think you have the common view about this. The reality is many of > > >those same companies would be making money today. They were just > > >ahead of their time -- which is very common here in Silicon Valley. > > > > Hmmm.. I don't know about that.. having a solution with no problem > > that needs to be solved isn't a valid business model. You could > > equally well argue that a company with lots of smart employees, but > > no ideas, is also ahead of it's time. > > > > Either you have all the elements, or you don't. > >Have you ever worked in Silicon Valley? Much of the thinking >is about what is emerging, and how long it will take to emerge. Thinking yes, profit/revenue no. >This is the mindset for the big winners, but it is also the >highest risk. I would say that it's the folks who think and identify what the elements are and when they will be in place that have the highest returns. >Once all the elements are in place, then that is >the secondary market of what I call the carpet baggers. Actually >there are a lot of those folks too. I believe the term is "fast followers" > > > > To a certain extent, the big iron companies had troubles, though, > > because of the .com bubble in general. All this cash flooding into > > the market looking for investments, so really, really speculative > > ventures got funded (sort of like cash flooding into the > > collateralized debt obligation/mortgage backed securities market), > > and that funding drove deposits on equipment from the big iron companies, > > etc. > >I think you nailed that. Silicon Valley is driven by greed. And no group is >more full of greed than the investors of Silicon Valley -- as it should be. I suspect that there are folks in other places who are substantially greedier and who think that "they're the smartest guys in the room", and who play in far more ephemeral and speculative venues than actual ideas and products. Structured Investment Vehicles with multiple tranches are just an example.. entirely a construct of paper and one economic model against another to hopefully have a positive return. No technology ideas or better mousetraps needed. It's all moving future money from one place to another, betting that your risk model and corresponding asset allocation strategy is better than someone else's. >We all live by the golden rule -- Those with the gold make the rule. What >is almost comical is how they engage in herd investing. They watch what >their counterparts are investing in -- and they often need to invest in >a competing company to avoid looking clueless. There is cover in that >model -- If it hits big, you can say you were playing in the market, >even if your investment lost. And if that market flops, you can say that >everyone else made the same mistake as you. This is very common here in >Silicon Valley. And in the entertainment business. Look how many clone "high concept" movies/TV shows there are. From perry at piermont.com Tue Jun 17 18:26:17 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:12 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: (Mark Hahn's message of "Tue\, 17 Jun 2008 15\:05\:33 -0400 \(EDT\)") References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> <4856F7EC.7060000@ussarna.se> <20080617070116.nqbmzub87zks8084@webmail.jpl.nasa.gov> <6.2.5.6.2.20080617102708.02fa63a0@jpl.nasa.gov> <20080617182356.GA28059@bx9.net> Message-ID: <87d4mf7cue.fsf@snark.cb.piermont.com> Mark Hahn writes: >> I'm moving into only gives me enough power for 16 machines per >> rack. But older colos are only 2/3 of that. These are HPCish nodes. > > I guess I'm not surprised, but then again, it does make me scratch > my head about vendors who are still in love with blades. you know, > headlines like "HP puts 1000 cores in a rack". and who buys them? > I picture PHB's buying blade chassis and then putting just one per > rack for heat-density reasons. I have worked with companies using them quite successfully, in fully populated racks. If you have a machine room in an office tower in midtown Manhattan (and some people have no choice but to do that), your main concern is space, not power. There are places where you can buy more power far more cheaply than more space. -- Perry E. Metzger perry@piermont.com From sean at duke.edu Wed Jun 18 07:37:31 2008 From: sean at duke.edu (Sean Dilda) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] HPC Position available in Durham, NC Message-ID: <48591DAB.80902@duke.edu> (I'm resending this as the first version had an html attachment that was too big for this list) If there are any experience HPC Sysadmins in the Raleigh-Durham area or looking to move to it, please take a look at this job posting. We're stepping up our HPC and Research Computing efforts and are looking for an extra Systems Administrator to help with the extra load. If you have any questions about the position, please feel free to contact me off-list. Thanks, Sean *If you are interested in this position, please apply on-line at http://www.hr.duke.edu/jobs/external.html and use 400214967 in the requisition number field. POSITION AVAILABLE POSITION TITLE: Analyst, IT (High Performance Computing) JOB CODE: 2423 JOB BAND / LEVEL: C JOB FAMILY: 08 (Information Technology) WORK SCHEDULE: Normal hours worked DEPARTMENT CONTACT: Stephen Galla (galla@duke.edu) 919-668-4236 POSITION SUMMARY: High Performance Computing is an important component of shared services provided by Duke?s Office of Information Technology (OIT). Work in this position involves planning, implementing, and supporting Duke?s Shared Cluster Resource (DSCR) and other high performance computing environments in support of research computing. The DSCR is a High Performance Computing Cluster (HPCC) designed for parallel and single-threaded jobs. The DSCR runs Linux and Sun Grid Engine (batch scheduler). This position reports to the Sr. Manager of Collaborative Systems and will work closely with OIT?s management, systems administrators and researchers to provide operational support of high performance computing. A working knowledge of Red Hat Enterprise Linux (RHEL) or CentOS is required. Familiarity with programming languages (including Shell, Perl, Java, PHP, and Ruby) is preferred. DUTIES & WORK PERFORMED: ? Administration of Duke?s Shared Cluster Resource (DSCR) and other high performance computing environments. ? Interfacing with peer administrators supporting Linux based systems. ? Interfacing with researchers to plan and implement participation within the cluster. ? Proactively initiate measures to ensure operational availability and performance of the DSCR. ? Respond, troubleshoot, resolve and document system incidents. ? Maintain monitoring, logging, backup and restoration of the DSCR infrastructure. SOFT SKILLS: ? Ability to work within a team in a demanding, fast-paced environment. ? Must have good planning and organizational skills, strong verbal and written communication skills. ? Highly motivated individual able to drive projects to completion. ? Ability to handle multiple concurrent activities and have a flexible, positive attitude. ? Demonstrated ability to track, organize, prioritize and execute project and operational workloads. ? Excellent analytical and problem solving skills. QUALIFICATIONS & EXPERIENCE: The most qualified candidates will exhibit demonstrated ability or experience in the following areas. ? Solid working knowledge of Red Hat Enterprise Linux (RHEL) or CentOS, networking and account management. ? Experience packaging, installing and supporting 3rd party applications. ? Experience with scripting and programming languages (Shell, Perl, Java, PHP, and Ruby). ? Experience with kick start and the management of a large number of systems. ? Experience with NFS and NIS. ? Experience with NetApp filers. ? Ability to write end-user documentation for non-technical staff. ? Experience developing project plans and time estimates. ? Proven design and debugging skills. ? Familiarity with SGE (Sun Grid Engine) and other distributed computing tools. EDUCATION & EXPERIENCE: Required: ? Equivalent combination of relevant education and experience to a BA or BS in Math or Computer Science or related field. DATE OF POSTING: June 17, 2008 Pos#: 50468910 Req#: 400214967 From prentice at ias.edu Wed Jun 18 07:51:04 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" Message-ID: <485920D8.2030309@ias.edu> I seem to have muddied the waters of the original NVIDIA/CUDA post. Someone made the inaccurate statement that CUDA programming is difficult and time-consuming. I cited the MD5-example of my colleague as an example of how easy it is to port the code, and how significant the performance improvements could be. Some subscribers to this list questions these results,and the discussion quicky turned away from NVIDIA/CUDA/GPUs to MD5. I forward the e-mails to my colleague. Since he doesn't subscribe to this list, I'm replying based on information he has provided me. He hasn't asked me to reply on his behalf, I'm doing this on my onw to contribute to the discussion. My colleague who did this work, Mario Juric, is a member at the Institute for Advanced Study (member = postdoc) studying Astrophysics. Contrary to the assertion by Vincent that CUDA/GPUs are only for hobbyists, Mario is very interested in using GPUs to speed up his astrophysics research. The biggest hindrance to doing "real" work with GPUs is the lack of dual-precision capabilities. As we all know, that hindrance was eliminated yesterday. In November of 2007, Mario organized the AstroGPU Workshop here at the Institute to discuss the use of GPUs in Astrophysics (http://www.astrogpu.org/). This workshop serves as proof that there are scientists serious about using GPUs for real work (not just hobbyists). Now about that MD5 discussion... I forwarded Mario the replies to my post, and he replied thusly: Hi Prentice, The guy ... is right -- it's throughput for a lot of simultaneous computations, not for a computation of a single hash. If you attempt to compute a single hash on an entire card, you won't get any improvement. Same as you wouldn't if you tried it on a single vs. quad core CPU. But if you compute four hashes, than single vs. quad makes a huge difference. And the GPU cards are effectively 128 core CPUs, so when you need to compute millions of hashes... Feel free to e-mail them my online writeup at: http://majuric.org/software/cudamd5/ The full source code is there as well. That should clear a lot of things up. For those of you with questions about the MD5 performance, please see the link above, and better yet, take a look at Mario's code. The link includes lots of pretty graphs ;) Elcomsoft did the same thing, and sells it commercially: http://www.engadget.com/2007/10/24/elcomsoft-turns-your-pc-into-a-password-cracking-supercomputer/ NVIDIA has promised us some new GPUs through their Professor Partner program. I'm sure once we get our hands on them, we'll do more coding/benchmarking. Not sure if they'll be DP-capable units. -- Prentice Bisbal Linux Software Support Specialist/System Administrator School of Natural Sciences Institute for Advanced Study Princeton, NJ From kus at free.net Wed Jun 18 08:25:51 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Tyan S2932 and lm_sensors Message-ID: Sorry, do somebody have correct sensors.conf file for Tyan S2932 motherboard ? There is no lm_sensors configuration file for this mobos on Tyan site :-( Yours Mikhail Kuzminsky Computer Assistance to Chemical Research Center Zelinsky Institute of Organic Chemistry Moscow From diep at xs4all.nl Wed Jun 18 13:14:28 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <485920D8.2030309@ias.edu> References: <485920D8.2030309@ias.edu> Message-ID: Prentice, No one doubts your collegue. Taking a md5 is very fast on PC processors. The code is very simple. Just a few lines of code it is. At my raid10 array i take a lot of md5sums for big files (chess endgame table bases), the limiting factor is the i/o speed. I/O delivers oh in my case a 100MB/s. Running 128 parallel sessions of md5sum is not so interesting at all, we all believe this can be done fast. Even if you would have a harddrive that can deliver 20 GB/s, still the limiting factor taking md5sum with a GPU is the PCI-e bandwidth of what is it say 2GB/s or so? So you can never stream enough data to the videocard to declare that it is useful to handle your md5sums, if your main processor, being a cheapo PC processor from years ago, already is doing more than fine there, considering the bottleneck. So this experiment of your high esteemed collegue is an example of an application that you do not want to buy those cards for. Of course the reason he did do it, is obvious. He wanted to know something about the throughput it can deliver. Yet the limitation of that throughput, namely that you have that throughput only within its registerfile/local cache, means basically bad news. Maybe it is an idea for your collegue to measure how many instructions per second he managed to get executed. For the new cards getting close to say 0.15 Tera instructions per cycle would be interesting information to have. Knowing the effective number of gigabytes per second it can process for md5 is however not interesting at all. Because the above is so trivial, i guess that caused some here. Interesting of course is to parallellize taking a md5. Yet being able to parallellize taking a md5sum of course defeats the original purpose of the algorithm, as it means of course by deduction of that fact that it is possible to fool md5sum (modifying a file such that it contains the content you want it to have meanwhile giving the same md5sum output). MD5 is hopelessly outdated for that reason, as it is very insecure, giving online hackers opportunities to do bad things at your file system. Vincent On Jun 18, 2008, at 4:51 PM, Prentice Bisbal wrote: > I seem to have muddied the waters of the original NVIDIA/CUDA post. > Someone made the inaccurate statement that CUDA programming is > difficult > and time-consuming. I cited the MD5-example of my colleague as an > example of how easy it is to port the code, and how significant the > performance improvements could be. > > Some subscribers to this list questions these results,and the > discussion > quicky turned away from NVIDIA/CUDA/GPUs to MD5. I forward the e-mails > to my colleague. Since he doesn't subscribe to this list, I'm replying > based on information he has provided me. He hasn't asked me to > reply on > his behalf, I'm doing this on my onw to contribute to the discussion. > > My colleague who did this work, Mario Juric, is a member at the > Institute for Advanced Study (member = postdoc) studying Astrophysics. > Contrary to the assertion by Vincent that CUDA/GPUs are only for > hobbyists, Mario is very interested in using GPUs to speed up his > astrophysics research. The biggest hindrance to doing "real" work with > GPUs is the lack of dual-precision capabilities. As we all know, that > hindrance was eliminated yesterday. > > In November of 2007, Mario organized the AstroGPU Workshop here at the > Institute to discuss the use of GPUs in Astrophysics > (http://www.astrogpu.org/). This workshop serves as proof that > there are > scientists serious about using GPUs for real work (not just > hobbyists). > > Now about that MD5 discussion... I forwarded Mario the replies to my > post, and he replied thusly: > > > > Hi Prentice, > The guy ... is right -- it's throughput for a lot of simultaneous > computations, not for a computation of a single hash. If you > attempt to > compute a single hash on an entire card, you won't get any > improvement. > Same as you wouldn't if you tried it on a single vs. quad core CPU. > But > if you compute four hashes, than single vs. quad makes a huge > difference. And the GPU cards are effectively 128 core CPUs, so > when you > need to compute millions of hashes... > > Feel free to e-mail them my online writeup at: > > http://majuric.org/software/cudamd5/ > > The full source code is there as well. That should clear a lot of > things > up. > > > > For those of you with questions about the MD5 performance, please see > the link above, and better yet, take a look at Mario's code. The link > includes lots of pretty graphs ;) > > > > Elcomsoft did the same thing, and sells it commercially: > > http://www.engadget.com/2007/10/24/elcomsoft-turns-your-pc-into-a- > password-cracking-supercomputer/ > > > > NVIDIA has promised us some new GPUs through their Professor Partner > program. I'm sure once we get our hands on them, we'll do more > coding/benchmarking. Not sure if they'll be DP-capable units. > > -- > Prentice Bisbal > Linux Software Support Specialist/System Administrator > School of Natural Sciences > Institute for Advanced Study > Princeton, NJ > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From prentice at ias.edu Wed Jun 18 14:13:30 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> Message-ID: <48597A7A.9070101@ias.edu> Vincent Diepeveen wrote: > > Running 128 parallel sessions of md5sum is not so interesting at all, we > all believe this can be done fast. > Vincent, That is the whole point of my original posting. The point was NEVER to demonstrate the use of GPUs for streaming MD5-encrypted data. This was the point of my posting: 1. To prove that CUDA programming is NOT as difficult as you made it out to be. 2. To demonstrate the performance improvement you can get by parallelizing an operation using CUDA. The MD5 algorithm was perfect for this. No claims were ever made as to the need for parallelizing MD5. There is value, however, if your goal is to recover (discover?) an MD5-hashed password through a brute-force attack. Last time I checked, MD5 password s are the default for most Linux distros. 3. To show that more than just "hobbyists" are investigating GPUs. -- Prentice Bisbal Linux Software Support Specialist/System Administrator School of Natural Sciences Institute for Advanced Study Princeton, NJ From lindahl at pbm.com Wed Jun 18 14:46:04 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <485920D8.2030309@ias.edu> References: <485920D8.2030309@ias.edu> Message-ID: <20080618214603.GC31594@bx9.net> On Wed, Jun 18, 2008 at 10:51:04AM -0400, Prentice Bisbal wrote: > Someone made the inaccurate statement that CUDA programming is difficult > and time-consuming. One data point cannot prove that CUDA is easy. There are people out there claiming that FPGAs are easy to program, because they're one of the 7 people on the planet for whom programming an FPGA is easy. I've looked over CUDA and some examples, and while it's better looking than some of the other GPU programming languages out there, it's clear that it is more difficult and time-consuming than using traditional languages on traditional cpus. -- greg From kus at free.net Wed Jun 18 14:54:18 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Tyan S2932 and lm_sensors In-Reply-To: Message-ID: In message from "Seth Bardash" (Wed, 18 Jun 2008 10:32:17 -0600): >ftp://ftp.tyan.com/softwave/lms/2932.sensors.conf > > >Seth Bardash > >Integrated Solutions and Systems >1510 Old North Gate Road >Colorado Springs, CO 80921 > >719-495-5866 >719-495-5870 Fax >719-337-4779 Cell > >http://www.integratedsolutions.org > >Failure can not cope with knowledge and perseverance! > >-----Original Message----- >From: beowulf-bounces@beowulf.org >[mailto:beowulf-bounces@beowulf.org] >On Behalf Of Mikhail Kuzminsky >Sent: Wednesday, June 18, 2008 9:26 AM >To: beowulf@beowulf.org >Subject: [Beowulf] Tyan S2932 and lm_sensors > >Sorry, do somebody have correct sensors.conf file for Tyan S2932 >motherboard ? There is no lm_sensors configuration file for this >mobos > >on Tyan site :-( > >Yours >Mikhail Kuzminsky >Computer Assistance to Chemical Research Center >Zelinsky Institute of Organic Chemistry >Moscow >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf >No virus found in this incoming message. >Checked by AVG. >Version: 8.0.100 / Virus Database: 270.4.0/1507 - Release Date: >6/18/2008 7:09 AM > From landman at scalableinformatics.com Wed Jun 18 14:57:12 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <48597A7A.9070101@ias.edu> References: <485920D8.2030309@ias.edu> <48597A7A.9070101@ias.edu> Message-ID: <485984B8.2050409@scalableinformatics.com> Prentice Bisbal wrote: > Vincent Diepeveen wrote: >> Running 128 parallel sessions of md5sum is not so interesting at all, we >> all believe this can be done fast. >> > > Vincent, > > That is the whole point of my original posting. The point was NEVER to > demonstrate the use of GPUs for streaming MD5-encrypted data. This was > the point of my posting: > > 1. To prove that CUDA programming is NOT as difficult as you made it out > to be. Hi Prentis: There is a general impression that CUDA is hard. I am not sure precisely where this is coming from, but this is what I am hearing from multiple quarters. Usually from people whom have not tried it. > > 2. To demonstrate the performance improvement you can get by > parallelizing an operation using CUDA. The MD5 algorithm was perfect for > this. No claims were ever made as to the need for parallelizing MD5. > There is value, however, if your goal is to recover (discover?) an > MD5-hashed password through a brute-force attack. Last time I checked, > MD5 password s are the default for most Linux distros. > > 3. To show that more than just "hobbyists" are investigating GPUs. I think I can comment on some papers submitted to various conferences. I am privy to some work not yet published, so I can't recount that. In short, we have seen papers on segmentation of medical image data sets (Liver segmentation to be precise) on CUDA platforms, getting ~70x performance over a single machine. I am aware of some unnamed bioinformatics applications seeing ... nice ... speedups on CUDA. None of these are hobbyist things. The CUDA eco-system is growing rapidly, with real users. We have 3 CUDA machines in house, one of them my laptop :). I just need to get on a few planes so I can spend that time coding ... -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From kus at free.net Wed Jun 18 14:58:51 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Tyan S2932 and lm_sensors In-Reply-To: Message-ID: In message from "Seth Bardash" (Wed, 18 Jun 2008 10:32:17 -0600): >ftp://ftp.tyan.com/softwave/lms/2932.sensors.conf > > >Seth Bardash Thank you very much !! It's strange, but I didn't find this file on Tyan archive! Mikhail > >Integrated Solutions and Systems >1510 Old North Gate Road >Colorado Springs, CO 80921 > >719-495-5866 >719-495-5870 Fax >719-337-4779 Cell > >http://www.integratedsolutions.org > >Failure can not cope with knowledge and perseverance! > >-----Original Message----- >From: beowulf-bounces@beowulf.org >[mailto:beowulf-bounces@beowulf.org] >On Behalf Of Mikhail Kuzminsky >Sent: Wednesday, June 18, 2008 9:26 AM >To: beowulf@beowulf.org >Subject: [Beowulf] Tyan S2932 and lm_sensors > >Sorry, do somebody have correct sensors.conf file for Tyan S2932 >motherboard ? There is no lm_sensors configuration file for this >mobos > >on Tyan site :-( > >Yours >Mikhail Kuzminsky >Computer Assistance to Chemical Research Center >Zelinsky Institute of Organic Chemistry >Moscow >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf >No virus found in this incoming message. >Checked by AVG. >Version: 8.0.100 / Virus Database: 270.4.0/1507 - Release Date: >6/18/2008 7:09 AM > From lindahl at pbm.com Wed Jun 18 15:11:54 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] SuperMicro and lm_sensors Message-ID: <20080618221153.GA24788@bx9.net> Speaking of lm_sensors, does anyone have configs for recent SuperMicro mobos? My SuperMicro support contact doesn't have ay idea, and running sensors-detect leaves me with lots of readings which are miscalibrated. -- greg From kilian at stanford.edu Wed Jun 18 15:21:46 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <20080618214603.GC31594@bx9.net> References: <485920D8.2030309@ias.edu> <20080618214603.GC31594@bx9.net> Message-ID: <200806181521.47115.kilian@stanford.edu> On Wednesday 18 June 2008 02:46:04 pm Greg Lindahl wrote: > One data point cannot prove that CUDA is easy. There are people out > there claiming that FPGAs are easy to program, because they're one of > the 7 people on the planet for whom programming an FPGA is easy. > > I've looked over CUDA and some examples, and while it's better > looking than some of the other GPU programming languages out there, > it's clear that it is more difficult and time-consuming than using > traditional languages on traditional cpus. I'll add my 2 cents here. I've recently attended a "CUDA technical training session" given by NVIDIA in Santa Clara, CA. The presentation came along with practical exercises (laptops were provided for the hands-on session, sweet). It wasn't about going too deep into details, but from what I've seen, they've made a lot of effort to make the language understandable and usable for whoever has some programming experience. CUDA is regular C, with a small set of extensions. You definitely have to wrap your mind around some concepts (loops are irrelevant, for instance, since everything GPU-related is implicitely parallel ; you have to divide your work into smaller units which can fit in the CUDA programming model, and which are susceptible to keep the GPU pipes busy ; you don't really control how the work units are scheduled to the processors, etc.), but nothing fundamentaly different from your average parallel programming. So, yes, of course, there's a learning curve, and I only scratched the surface, but it doesn't seem to me that any experienced programmer would need more than a couple days to begin writing efficient CUDA code. We've also encountered somme oddities, like CUDA code freezing a machine running X.org (and using the proprietary NVIDIA driver), compiler segfaults or code returning incoherent results. We didn't spend too much time on debugging those, so they may very well have been related to the code itself, and not to the compiler or the driver. And moreover, it was version 1.1, and I think they released a version 2.0 since then. Anyway, CUDA is definitely not an UML-like language (it's not enough to draw boxes, you still have to write code, and NVIDIA already ported a bunch of standard libs to help you porting your applications), but it definitely looks easier than what ATI used to call CTM. Cheers, -- Kilian From jakob at unthought.net Wed Jun 18 15:45:23 2008 From: jakob at unthought.net (Jakob Oestergaard) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> Message-ID: <20080618224523.GA18052@unthought.net> On Wed, Jun 18, 2008 at 10:14:28PM +0200, Vincent Diepeveen wrote: > Prentice, > > No one doubts your collegue. > Ok the following has nothing to do with any part of the discussion, but I just needed to get this out :) ... > So this experiment of your high esteemed collegue is an example of an > application that you do not want to buy those cards for. For attacks against password hashes (and similar) this is very useful. You generate strings, hash them, compare the hash to your target value. If it matches, you've found your source string (password). Disks are never involved. Your brute-force of md5 (or something similar) will run as fast as the string generators (processors) allow. Some time ago I looked into buying an FPGA card to speed up a bitsliced 3des to brute force certain hashes, but never actually got around to doing this. From an algorithmic point of view, though, this could have rocked seriously on even a low end FPGA :) The point being; this is a useful application. Even though the md5 implementation may just have been "for show" and not intended as a final product, it is very close to something that has direct uses. And sure, no, people won't buy GPUs to speed up the occational file hashing with md5sum, and I'm sure that wasn't the intention either :) -- / jakob From diep at xs4all.nl Wed Jun 18 16:16:27 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <20080618224523.GA18052@unthought.net> References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> Message-ID: <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> Jakob, You are speaking here of a hobby project of a system administrator to crack a few passwords in the slowest possible way? In general very stupid algorithms do not benefit much from caching things in RAM, let alone caches and are lightyears slower than what is possible. Game tree search is a great example of this, but lately also in weather research they've found more sophisticated methods if i read this list correctly. Apart from all this. See www.wassenaar.org That is a treaty that USA wanted and many nations signed it. Called the Wassenaar treaty. Maybe therefore a new embassy from USA gets built now in the town Wassenaar (close to The Hague). Treaty, if i read it well, makes problems using public key above 512 bits. So according to that treaty, if i read it correct, trying to do anything with a SSH which uses usually 1024 bits RSA, is already meaning you are a criminal category 5 (highest category). In fact programming that SSH 1024 bits code already makes you a big enemy of the state as it is above 512 bits. So your posting rises the question with me, without thinking yet about the CUDA component of your solution, Is it legal what your friend is doing for his hobby in his sparetime? Thanks, Vincent checkout: http://www.primenumbers.net/prptop/prptop.php On Jun 19, 2008, at 12:45 AM, Jakob Oestergaard wrote: > On Wed, Jun 18, 2008 at 10:14:28PM +0200, Vincent Diepeveen wrote: >> Prentice, >> >> No one doubts your collegue. >> > > Ok the following has nothing to do with any part of the discussion, > but I just > needed to get this out :) > > ... >> So this experiment of your high esteemed collegue is an example of an >> application that you do not want to buy those cards for. > > For attacks against password hashes (and similar) this is very useful. > > You generate strings, hash them, compare the hash to your target > value. If it > matches, you've found your source string (password). > > Disks are never involved. > > Your brute-force of md5 (or something similar) will run as fast as > the string > generators (processors) allow. Some time ago I looked into buying > an FPGA card > to speed up a bitsliced 3des to brute force certain hashes, but > never actually > got around to doing this. From an algorithmic point of view, > though, this > could have rocked seriously on even a low end FPGA :) > > The point being; this is a useful application. Even though the md5 > implementation may just have been "for show" and not intended as a > final > product, it is very close to something that has direct uses. > > And sure, no, people won't buy GPUs to speed up the occational file > hashing > with md5sum, and I'm sure that wasn't the intention either :) > > -- > > / jakob > > From jlforrest at berkeley.edu Wed Jun 18 16:31:21 2008 From: jlforrest at berkeley.edu (Jon Forrest) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <200806181521.47115.kilian@stanford.edu> References: <485920D8.2030309@ias.edu> <20080618214603.GC31594@bx9.net> <200806181521.47115.kilian@stanford.edu> Message-ID: <48599AC9.3040100@berkeley.edu> Kilian CAVALOTTI wrote: > We've also encountered somme oddities, like CUDA code freezing a machine > running X.org (and using the proprietary NVIDIA driver), I'm glad you mentioned this. I've read through much of the information on their web site and I still don't understand the usage model for CUDA. By that I mean, on a desktop machine, are you supposed to have 2 graphics cards, 1 for running CUDA code and one for regular graphics? If you only need 1 card for both, how do you avoid the problem you mentioned, which was also mentioned in the documentation? Or, if you have a compute node that will sit in a dark room, you aren't going to be running an X server at all, so you won't have to worry about anything hanging? How does this behavior change, if at all, when running Windows? I'm planning on starting a pilot program to get the chemists in my department to use CUDA, but I'm waiting for V2 of the SDK to come out. Cordially, -- Jon Forrest Research Computing Support College of Chemistry 173 Tan Hall University of California Berkeley Berkeley, CA 94720-1460 510-643-1032 jlforrest@berkeley.edu From bill at cse.ucdavis.edu Wed Jun 18 16:34:59 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <20080618214603.GC31594@bx9.net> References: <485920D8.2030309@ias.edu> <20080618214603.GC31594@bx9.net> Message-ID: <48599BA3.1050105@cse.ucdavis.edu> Greg Lindahl wrote: > On Wed, Jun 18, 2008 at 10:51:04AM -0400, Prentice Bisbal wrote: > >> Someone made the inaccurate statement that CUDA programming is difficult >> and time-consuming. > > One data point cannot prove that CUDA is easy. There are people out > there claiming that FPGAs are easy to program, because they're one of > the 7 people on the planet for whom programming an FPGA is easy. *chuckle*. > I've looked over CUDA and some examples, and while it's better looking > than some of the other GPU programming languages out there, it's clear > that it is more difficult and time-consuming than using traditional > languages on traditional cpus. Agreed. Traditional languages are easier, but don't express parallelism well. One approach is of course openMP, a few pragmas, and parallel friendly (loop index independent) loops and you can get reasonable speedups on SMP machines. Cuda seems to take a different approach, instead of trying to auto-parallelize a loop, it requires a function pointer to the code, and the function must declare it's exit condition. CUDA seems rather similar to openMP. Massimiliano Fatica of nvidia did a stream port, and I'll quote pieces of his code below. So instead of: for (j=0; j>>(d_b, d_c, scalar, N); cudaThreadSynchronize(); times[1][k]= mysecond() - times[1][k]; Instead of: static double a[N+OFFSET] You get: cudaMalloc((void**)&d_a, sizeof(float)*N); Instead of: for (j=0; j>>(d_a, 2.f, N); set_array<<>>(d_b, .5f, N); set_array<<>>(d_c, .5f, N); So yes, it's a change, but it does seem pretty reasonable. From perry at piermont.com Wed Jun 18 16:37:02 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> (Vincent Diepeveen's message of "Thu\, 19 Jun 2008 01\:16\:27 +0200") References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> Message-ID: <87r6auiach.fsf@snark.cb.piermont.com> Not that this mailing list is an appropriate place to discuss this, but... Vincent Diepeveen writes: > Treaty, if i read it well, makes problems using public key above 512 > bits. > > So according to that treaty, if i read it correct, trying to do anything > with a SSH which uses usually 1024 bits RSA, is already meaning you > are a criminal category 5 (highest category). Er, no. That's totally false in every way. There may be countries where this is true, but certainly in the United States, there is no problem implementing and using crypto with any length keys you like. You are also allowed to make available arbitrary open source crypto software for export so long as you send one email to alert a particular agency that you're doing so, and if software is available commercially it can also be exported with minimal trouble. Ten or fifteen years ago, things were different, but that was ten or fifteen years ago. And yes, I am an expert on this subject. I run a very large mailing list on security and cryptography. If you want to discuss this, I suspect this is not the right place to do it. Perry -- Perry E. Metzger perry@piermont.com From kilian at stanford.edu Wed Jun 18 16:57:59 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <48599AC9.3040100@berkeley.edu> References: <485920D8.2030309@ias.edu> <200806181521.47115.kilian@stanford.edu> <48599AC9.3040100@berkeley.edu> Message-ID: <200806181657.59782.kilian@stanford.edu> On Wednesday 18 June 2008 04:31:21 pm you wrote: > I'm glad you mentioned this. I've read through much of the > information on their web site and I still don't understand the usage > model for CUDA. By that I mean, on a desktop machine, are you > supposed to have 2 graphics cards, 1 for running CUDA code and one > for regular graphics? Well, we used laptops for the hands-on session, so one graphics card is sufficient. Everything is handled by the driver. I have no clue about the internals, but somehow, the CUDA code you compile generates GPU PTX assembly, which is passed to the driver for execution. That what I meant by mentioning you don't really control the scheduling of your threads: I have no idea how the GPU decides if it should better render an 3D object for your screensaver, or execute your code. It seems to be able to do both at the same time, though, since we've been shown some CUDA applications involving OpenGL rendering. And we didn't even mention the (Tri)SLI setups. I'm actually curious to know how their Tesla boxes work. Are the four graphics card treated as independent processing units (meaning there should be a scheduler somewhere to apportion the work amongst the GPUs). Or are they treated as a single GPU, ? la SLI? > If you only need 1 card for both, how do you > avoid the problem you mentioned, which was also mentioned in the > documentation? Well, you can't. :) It's not fundamentaly different from what you do with a regular CPU: if your code locks it up, your whole machine is dead, along with the other running applications. Althought it seems a bit easier to lock up a GPU rather than a CPU. Except for those which are specifically designed for this, of course (MIPS-X, anyone? hsc instruction, page 65 in [1]). > How does this behavior change, if at all, when running Windows? I think that's pretty much the same. Since proper execution of your code depends on the graphical driver's goodwill, and given their reputation on the Windows platform regarding stability, you'd better take that into account. > I'm planning on starting a pilot program to get the > chemists in my department to use CUDA, but I'm waiting > for V2 of the SDK to come out. Looks like v2 beta2 is out ([2]). [1]ftp://reports.stanford.edu/pub/cstr/reports/csl/tr/86/289/CSL-TR-86-289.pdf [2]http://www.nvidia.com/object/cuda_get.html Cheers, -- Kilian From James.P.Lux at jpl.nasa.gov Wed Jun 18 17:03:16 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> Message-ID: <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> At 04:16 PM 6/18/2008, Vincent Diepeveen wrote: >Jakob, > > >Is it legal what your friend is doing for his hobby in his sparetime? > >Thanks, >Vincent There are many things that one might do, perhaps with a cluster, that can get you into trouble with various treaties and agreements. For example, the ITAR describes a heck of a lot of really interesting things that if inappropriately transferred to someone might result in some attention from the authorities. In general, fundamental research is not subject to export controls, so if you frame your problem in terms of abstract mathematical problems, you're not going to be treading on any toes. However, start distributing it as "Jim Lux's superduper encryptor/password cracker, now with 1024 bit capability!" and it's moved from fundamental research to a product. "Defense service" is a particularly tricky area since it's pretty vague in definition. There's tricky guidelines like this: Providing guidance or instruction to a foreign person on where to find data or information related to controlled items may be a defense service even if the data is in the public domain, if the data addresses a specific problem and is being provided to help with that problem. So.. if your (foreign person) buddy is designing thermonuclear devices in their garage, and they complain about how slow it is to run the hydrocodes to simulate stuff, better not hand them that old copy of Sterling, et al., or even worse, give them rgb's website. (the latter would be too suspicious, since rgb *is* a physicist, doing monte carlo simulations no less, while Tom Sterling is *just a computer scientist*) From bernard at vanhpc.org Wed Jun 18 17:03:53 2008 From: bernard at vanhpc.org (Bernard Li) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] SuperMicro and lm_sensors In-Reply-To: <20080618221153.GA24788@bx9.net> References: <20080618221153.GA24788@bx9.net> Message-ID: Hi Greg: On Wed, Jun 18, 2008 at 3:11 PM, Greg Lindahl wrote: > Speaking of lm_sensors, does anyone have configs for recent SuperMicro > mobos? My SuperMicro support contact doesn't have ay idea, and running > sensors-detect leaves me with lots of readings which are > miscalibrated. Have you tried updating lm_sensors to the 3.x series? Supermicro also has their own system health monitoring tool called superodoctor which you can get here: ftp://ftp.supermicro.com/utility/Supero_Doctor_II/Linux/ Cheers, Bernard From mathog at caltech.edu Wed Jun 18 17:04:23 2008 From: mathog at caltech.edu (David Mathog) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] OT: LTO Ultrium (3) throughput? Message-ID: If any of you have an LTO Ultrium-3 drive what kind of speeds are you observing? On one Linux system here (kernel 2.6.24-19) we have an HP Ultrium-3 attached to an Adaptec ASC-29320ALP U320 controller. There is nothing else on that SCSI bus, termination and cable seem good. Getting into scsi-select from the BIOS shows everything set to 320. No error messages or warnings are appearing. Yet: dd if=/dev/zero of=/dev/nst0 bs=8192 count=10000 only moves 21.3MB/sec. The HP documentation indicates 432GB/hour, (compressed) which is 120 Mb/sec, so we're off by 6X (or maybe 3X for 2:1 compression, either way, a lot). The system's CPU and memory aren't rate limiting as dd if=/dev/zero of=/dev/null bs=8192 count=10000 moves 6.9GB/sec. Any thoughts where the bottleneck might be? Thanks, David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From landman at scalableinformatics.com Wed Jun 18 17:19:14 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> Message-ID: <4859A602.2040900@scalableinformatics.com> Jim Lux wrote: > So.. if your (foreign person) buddy is designing thermonuclear devices > in their garage, and they complain about how slow it is to run the > hydrocodes to simulate stuff, better not hand them that old copy of > Sterling, et al., or even worse, give them rgb's website. (the latter > would be too suspicious, since rgb *is* a physicist, doing monte carlo > simulations no less, while Tom Sterling is *just a computer scientist*) Hold on ... does this mean RGB is a munition? man .... -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From landman at scalableinformatics.com Wed Jun 18 17:24:20 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] OT: LTO Ultrium (3) throughput? In-Reply-To: References: Message-ID: <4859A734.8090401@scalableinformatics.com> David Mathog wrote: > If any of you have an LTO Ultrium-3 drive what kind of speeds are you > observing? On one Linux system here (kernel 2.6.24-19) we have an > HP Ultrium-3 attached to an Adaptec ASC-29320ALP U320 controller. > There is nothing else on that SCSI bus, termination and cable seem good. > Getting into scsi-select from the BIOS shows everything set to 320. > No error messages or warnings are appearing. Yet: > > dd if=/dev/zero of=/dev/nst0 bs=8192 count=10000 > > only moves 21.3MB/sec. The HP documentation indicates 432GB/hour, > (compressed) which is 120 Mb/sec, so we're off by 6X (or maybe 3X for > 2:1 compression, either way, a lot). The system's CPU and memory > aren't rate limiting as > > dd if=/dev/zero of=/dev/null bs=8192 count=10000 > > moves 6.9GB/sec. > > Any thoughts where the bottleneck might be? Have you adjusted the block size up? maybe try dd if=/dev/zero of=/dev/nst0 bs=8M count=1000 this is about 8 GB, so it hopefully shouldn't take too long. At 21 MB/s, this is about 50 seconds per gigabyte, so if this takes 400 seconds, you may have a problem. This said, the tape drive numbers we see quoted are "best case" scenarios, with optimal block sizes, a strong wind at the tapes back, and no monopole flux nearby. Burning incense may or may not help the speed, YMMV... This said, have you tried the "obvious" tests, such as what does lsscsi report, and lscpi -v for that adapter, and ... -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From perry at piermont.com Wed Jun 18 17:41:47 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> (Jim Lux's message of "Wed\, 18 Jun 2008 17\:03\:16 -0700") References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> Message-ID: <87iqw6i7ck.fsf@snark.cb.piermont.com> Jim Lux writes: > In general, fundamental research is not subject to export controls, so > if you frame your problem in terms of abstract mathematical problems, > you're not going to be treading on any toes. However, start > distributing it as "Jim Lux's superduper encryptor/password cracker, > now with 1024 bit capability!" and it's moved from fundamental > research to a product. Under current rules, provided it is open source or generally available to any buyer, you can distribute and export cryptographic code quite freely. There are some minimal reporting requirements, but they're barely worth mentioning. That's the reason you can freely distribute things like Kerberos, OpenSSL, pgp/gpg, etc. Perry -- Perry E. Metzger perry@piermont.com From jlb17 at duke.edu Wed Jun 18 17:45:29 2008 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] OT: LTO Ultrium (3) throughput? In-Reply-To: <4859A734.8090401@scalableinformatics.com> References: <4859A734.8090401@scalableinformatics.com> Message-ID: On Wed, 18 Jun 2008 at 8:24pm, Joe Landman wrote > David Mathog wrote: >> If any of you have an LTO Ultrium-3 drive what kind of speeds are you >> observing? On one Linux system here (kernel 2.6.24-19) we have an >> HP Ultrium-3 attached to an Adaptec ASC-29320ALP U320 controller. ISTR hearing some rumblings at some point about some LTO tape drives not liking Adapted SCSI. Mine run on LSI boards, and run quite nicely. >> There is nothing else on that SCSI bus, termination and cable seem good. >> Getting into scsi-select from the BIOS shows everything set to 320. >> No error messages or warnings are appearing. Yet: >> >> dd if=/dev/zero of=/dev/nst0 bs=8192 count=10000 >> >> only moves 21.3MB/sec. The HP documentation indicates 432GB/hour, >> (compressed) which is 120 Mb/sec, so we're off by 6X (or maybe 3X for >> 2:1 compression, either way, a lot). The system's CPU and memory >> aren't rate limiting as >> >> dd if=/dev/zero of=/dev/null bs=8192 count=10000 >> >> moves 6.9GB/sec. >> Any thoughts where the bottleneck might be? > > Have you adjusted the block size up? maybe try > > dd if=/dev/zero of=/dev/nst0 bs=8M count=1000 A very good suggestion. In my testing, 32KB blocks (the default for amanda) yielded only about 27MB/s (in line with your 8KB blocks above) while 2M blocks got me to 60MB/s (all testing done with tar and a very fast disk system). Note that I couldn't go much bigger than 2M or I started getting errors like "/dev/nst1: Cannot write: Value too large for defined data type". > this is about 8 GB, so it hopefully shouldn't take too long. At 21 MB/s, > this is about 50 seconds per gigabyte, so if this takes 400 seconds, you may > have a problem. > > This said, the tape drive numbers we see quoted are "best case" scenarios, > with optimal block sizes, a strong wind at the tapes back, and no monopole > flux nearby. Burning incense may or may not help the speed, YMMV... In addition, 2x hardware compression is almost always a lovely fairly tale with absolutely no basis in reality. That said, the LTO hardware compression is rather nice, in that it recognizes uncompressible data and doesn't try to further compress it. And, as always, be sure you used the proper color goat in your ceremonies. -- Joshua Baker-LePain QB3 Shared Cluster Sysadmin UCSF From mathog at caltech.edu Wed Jun 18 17:56:43 2008 From: mathog at caltech.edu (David Mathog) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Re: [OT: LTO Ultrium (3) throughput? Message-ID: Joe Landman wrote: > Have you adjusted the block size up? maybe try > > dd if=/dev/zero of=/dev/nst0 bs=8M count=1000 The tape has a 64Mb on board buffer and somewhere in the manual it said that it shouldn't matter. But no, I have not tried that yet and will do so tomorrow. > This said, have you tried the "obvious" tests, such as what does lsscsi > report, and lscpi -v for that adapter, and ... Maybe it will mean more to you than it does to me... % lsscsi --long [8:0:0:0] tape HP Ultrium 3-SCSI D21D /dev/st0 state=running queue_depth=32 scsi_level=4 type=1 device_blocked=0 timeout=900 lspci -v 04:04.0 SCSI storage controller: Adaptec ASC-29320ALP U320 (rev 10) Subsystem: Adaptec ASC-29320LPE PCIe U320 Flags: bus master, 66MHz, slow devsel, latency 32, IRQ 19 I/O ports at 4400 [disabled] [size=256] Memory at d8400000 (64-bit, non-prefetchable) [size=8K] I/O ports at 4000 [disabled] [size=256] [virtual] Expansion ROM at d8600000 [disabled] [size=512K] Capabilities: [dc] Power Management version 2 Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 Enable- Capabilities: [94] PCI-X non-bridge device /proc/interrupts shows that this controller is the only thing on IRQ 19. Thanks, David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From diep at xs4all.nl Wed Jun 18 18:02:26 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] OT: LTO Ultrium (3) throughput? In-Reply-To: References: Message-ID: <929B39C3-7FCE-4890-BAEE-22E32D8221AD@xs4all.nl> Hi Dave, Probably the i/o is your limiting factor. Using raid10 or something? Ignore that factor 2 compression by the way, that's just marketing paper. My LTO-2 gets about 28MB/s uncompressed streamspeed, for compressed files (7-zip is far superior to anything else currently under linux). Usually i also store longterm data with 10% par2 data (free linux tool is there for it as well), so that i can recover bit errors later on. That 28MB/s is at least what i observe, could be tad optimistic :) The initial problems i had also with streamspeed and also reading back was all software based. A lot of backup software was real ugly bad simply. What mattered at the raid10 array also was the file system used and how far loaded it already was. Reading files from a raid10 is not necessarily fast. Harddrives are relative fast when reading/writing blocks of 128-512KB or so, whereas filesystems use far smaller blocksizes to store in reality. Very bad to stream a couple of hundreds of GB to tape. The software used is important. Default backup software was dubious in my case and the biggest reason for problems. After that was fixed, the i/o speed is a problem when storing tiny files at tape. But well i'm a total layman with backups. Perhaps you already figured out your problem. Vincent On Jun 19, 2008, at 2:04 AM, David Mathog wrote: > If any of you have an LTO Ultrium-3 drive what kind of speeds are you > observing? On one Linux system here (kernel 2.6.24-19) we have an > HP Ultrium-3 attached to an Adaptec ASC-29320ALP U320 controller. > There is nothing else on that SCSI bus, termination and cable seem > good. > Getting into scsi-select from the BIOS shows everything set to 320. > No error messages or warnings are appearing. Yet: > > dd if=/dev/zero of=/dev/nst0 bs=8192 count=10000 > > only moves 21.3MB/sec. The HP documentation indicates 432GB/hour, > (compressed) which is 120 Mb/sec, so we're off by 6X (or maybe 3X for > 2:1 compression, either way, a lot). The system's CPU and memory > aren't rate limiting as > > dd if=/dev/zero of=/dev/null bs=8192 count=10000 > > moves 6.9GB/sec. > > Any thoughts where the bottleneck might be? > > Thanks, > > David Mathog > mathog@caltech.edu > Manager, Sequence Analysis Facility, Biology Division, Caltech > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From landman at scalableinformatics.com Wed Jun 18 18:09:30 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Re: [OT: LTO Ultrium (3) throughput? In-Reply-To: References: Message-ID: <4859B1CA.9070006@scalableinformatics.com> David Mathog wrote: > Joe Landman wrote: [...] > Maybe it will mean more to you than it does to me... > > % lsscsi --long > [8:0:0:0] tape HP Ultrium 3-SCSI D21D /dev/st0 > state=running queue_depth=32 scsi_level=4 type=1 device_blocked=0 > timeout=900 Hmmm... queue_depth of 32? On a tape? Weird. There has been some strangeness in the TCQ/NCQ layers. Might want to try to set this to 1 and see if it helps echo 1 > /sys/block/st0/device/queue_depth (note: some drivers don't support changing the queue_depth) > lspci -v > 04:04.0 SCSI storage controller: Adaptec ASC-29320ALP U320 (rev 10) > Subsystem: Adaptec ASC-29320LPE PCIe U320 > Flags: bus master, 66MHz, slow devsel, latency 32, IRQ 19 > I/O ports at 4400 [disabled] [size=256] > Memory at d8400000 (64-bit, non-prefetchable) [size=8K] > I/O ports at 4000 [disabled] [size=256] > [virtual] Expansion ROM at d8600000 [disabled] [size=512K] > Capabilities: [dc] Power Management version 2 > Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1 > Enable- > Capabilities: [94] PCI-X non-bridge device Looks like a PCI-x adapter. This shouldn't be an issue though. Try changing the block size. 2M to 8M, and see what happens. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From lindahl at pbm.com Wed Jun 18 18:14:20 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] SuperMicro and lm_sensors In-Reply-To: <20080618221153.GA24788@bx9.net> References: <20080618221153.GA24788@bx9.net> Message-ID: <20080619011419.GA7040@bx9.net> On Wed, Jun 18, 2008 at 03:11:54PM -0700, Greg Lindahl wrote: > Speaking of lm_sensors, does anyone have configs for recent SuperMicro > mobos? My SuperMicro support contact doesn't have ay idea, and running > sensors-detect leaves me with lots of readings which are > miscalibrated. Thanks to the 3 people who wrote me with suggestions. It appears that the magic is this: * Upgrade to a newer version of lm_sensors (atrpms has 2.10.6, dunno if you can find lm_sensors3 rpms anywhere) * Make sure your kernel has a driver for what sensors-detect detects Since I'm on CentOS 5.1, I was behind in both respects, and the result is that "sensors" prints nonsense -- it doesn't help that all of my boxes have 2 winbond chips in them, with the older one being detected and having a driver and having none of the interesting signals. With both updates I get reasonable numbers. -- greg From diep at xs4all.nl Wed Jun 18 18:35:21 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <87iqw6i7ck.fsf@snark.cb.piermont.com> References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <87iqw6i7ck.fsf@snark.cb.piermont.com> Message-ID: <6D3FFF5E-20CA-4AFC-88B2-18AA74C7EB28@xs4all.nl> USA and allowing encryption beyond the cracking capabilities of a 1st year computer science student... ...hmmm I remember how i did do big efforts to get vista ultimate bitlocker to work. On paper it sounds ok. AES 256 bits CBC. The idea is : your usb stick has the encryption key and only that thing has it. So no one can decrypt the partition without that usb stick. In case you forget to load that usb stick having the encryption key, there is a general manner to unlock the machine by feeding it a long code. Looks great isn't it? So far the paper... But now the usual bug; the implementation that practical was allowed by one those guys on the Perry-Sport mailing list. Of course we don't want to tire ourselves too much typing too long unlock codes... That unlock code, stored at a different USB stick is 48 digits. By the way 48 digits is how many bits? Right, that's less than 48 * (log 10/log 2) = 160 bits. So the problem for our first year student has been brought back from 256 to 160 bits already. Not that it is a hobby mine, but one day i rebooted the machine and had forgotten to put in the USB encryption stick. By accident i mistyped the key. Windows then told me: "you made a mistake somewhere in those 5 digits of the 48 digit key, please retype them". Doh. So my guess is that soon i do not need to worry when by accident i lose that USB stick... Vincent On Jun 19, 2008, at 2:41 AM, Perry E. Metzger wrote: > > Jim Lux writes: >> In general, fundamental research is not subject to export >> controls, so >> if you frame your problem in terms of abstract mathematical problems, >> you're not going to be treading on any toes. However, start >> distributing it as "Jim Lux's superduper encryptor/password cracker, >> now with 1024 bit capability!" and it's moved from fundamental >> research to a product. > > Under current rules, provided it is open source or generally available > to any buyer, you can distribute and export cryptographic code quite > freely. There are some minimal reporting requirements, but they're > barely worth mentioning. > > That's the reason you can freely distribute things like Kerberos, > OpenSSL, pgp/gpg, etc. > > > Perry > -- > Perry E. Metzger perry@piermont.com From shaeffer at neuralscape.com Wed Jun 18 18:47:44 2008 From: shaeffer at neuralscape.com (Karen Shaeffer) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] SuperMicro and lm_sensors In-Reply-To: <20080619011419.GA7040@bx9.net> References: <20080618221153.GA24788@bx9.net> <20080619011419.GA7040@bx9.net> Message-ID: <20080619014744.GA10075@synapse.neuralscape.com> On Wed, Jun 18, 2008 at 06:14:20PM -0700, Greg Lindahl wrote: > On Wed, Jun 18, 2008 at 03:11:54PM -0700, Greg Lindahl wrote: > > > Speaking of lm_sensors, does anyone have configs for recent SuperMicro > > mobos? My SuperMicro support contact doesn't have ay idea, and running > > sensors-detect leaves me with lots of readings which are > > miscalibrated. > > Thanks to the 3 people who wrote me with suggestions. > > It appears that the magic is this: > > * Upgrade to a newer version of lm_sensors (atrpms has 2.10.6, > dunno if you can find lm_sensors3 rpms anywhere) > * Make sure your kernel has a driver for what sensors-detect > detects > > Since I'm on CentOS 5.1, I was behind in both respects, and the result > is that "sensors" prints nonsense -- it doesn't help that all of my > boxes have 2 winbond chips in them, with the older one being detected > and having a driver and having none of the interesting signals. > > With both updates I get reasonable numbers. > > -- greg Hi Greg, I have some experience with lm_sensors. And you are correct that the 2.10.x release is required for many currrent system boards and recent kernels. But that isn't enough, depending on your system board. In particular, I have seen some cases where the temperatures are mislabeled. Unless the system board manufacturer gives you a configuration file, you might need to verify the mapping of the temperature labels to specific devices. YMMV. Thanks, Karen -- Karen Shaeffer Neuralscape, Palo Alto, Ca. 94306 shaeffer@neuralscape.com http://www.neuralscape.com From john.hearns at streamline-computing.com Thu Jun 19 00:17:07 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <48599AC9.3040100@berkeley.edu> References: <485920D8.2030309@ias.edu> <20080618214603.GC31594@bx9.net> <200806181521.47115.kilian@stanford.edu> <48599AC9.3040100@berkeley.edu> Message-ID: <1213859837.4784.9.camel@Vigor13> On Wed, 2008-06-18 at 16:31 -0700, Jon Forrest wrote: > Kilian CAVALOTTI wrote: > I'm glad you mentioned this. I've read through much of the information > on their web site and I still don't understand the usage model for > CUDA. By that I mean, on a desktop machine, are you supposed to have > 2 graphics cards, 1 for running CUDA code and one for regular > graphics? If you only need 1 card for both, how do you avoid the > problem you mentioned, which was also mentioned in the documentation? Actually, I should imagine Kilian is referring to something else, not the inbuilt timeout which is in the documentation. But I can't speak for im. > Or, if you have a compute node that will sit in a dark room, > you aren't going to be running an X server at all, so you won't > have to worry about anything hanging? I don't work for Nvidia, so I can't say! But the usage model is as you say - you can prototype applications which will run for a short time on the desktop machine, but long runs are meant to be done on a dedicated back-end machine. If you want a totally desk-side solution, they sell a companion box which goes alongside and attaches via a ribbon cable. I guess the art here is finding a motherboard with the right number and type of PCI-express slots to take both the companion box and a decent graphics card for X use. > > I'm planning on starting a pilot program to get the > chemists in my department to use CUDA, but I'm waiting > for V2 of the SDK to come out. > Why wait? The hardware will be the same, and you can dip your toe in the water now. From john.hearns at streamline-computing.com Thu Jun 19 00:24:38 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] SuperMicro and lm_sensors In-Reply-To: <20080618221153.GA24788@bx9.net> References: <20080618221153.GA24788@bx9.net> Message-ID: <1213860288.4784.15.camel@Vigor13> On Wed, 2008-06-18 at 15:11 -0700, Greg Lindahl wrote: > Speaking of lm_sensors, does anyone have configs for recent SuperMicro > mobos? My SuperMicro support contact doesn't have ay idea, and running > sensors-detect leaves me with lots of readings which are Can't help you on the lm_sensors front, we always use IPMI these days. But hearing of miscalibrated values does not surprise me. From dnlombar at ichips.intel.com Thu Jun 19 06:50:20 2008 From: dnlombar at ichips.intel.com (Lombard, David N) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] SuperMicro and lm_sensors In-Reply-To: <20080619011419.GA7040@bx9.net> References: <20080618221153.GA24788@bx9.net> <20080619011419.GA7040@bx9.net> Message-ID: <20080619135020.GA8918@nlxdcldnl2.cl.intel.com> On Wed, Jun 18, 2008 at 06:14:20PM -0700, Greg Lindahl wrote: > On Wed, Jun 18, 2008 at 03:11:54PM -0700, Greg Lindahl wrote: > > > Speaking of lm_sensors, does anyone have configs for recent SuperMicro > > mobos? My SuperMicro support contact doesn't have ay idea, and running > > sensors-detect leaves me with lots of readings which are > > miscalibrated. > > Thanks to the 3 people who wrote me with suggestions. > > It appears that the magic is this: > > * Upgrade to a newer version of lm_sensors (atrpms has 2.10.6, > dunno if you can find lm_sensors3 rpms anywhere) > * Make sure your kernel has a driver for what sensors-detect > detects Did you look for /proc/acpi/thermal_zone/*/temperature The glob is for your BIOS-defined ID. If it does exist, that's the value that drives /proc/acpi/thermal_zone/*/trip_points See also /proc/acpi/thermal_zone/*/polling_frequency -- David N. Lombard, Intel, Irvine, CA I do not speak for Intel Corporation; all comments are strictly my own. From shaeffer at neuralscape.com Thu Jun 19 07:40:38 2008 From: shaeffer at neuralscape.com (Karen Shaeffer) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] SuperMicro and lm_sensors In-Reply-To: <1213860288.4784.15.camel@Vigor13> References: <20080618221153.GA24788@bx9.net> <1213860288.4784.15.camel@Vigor13> Message-ID: <20080619144038.GA18151@synapse.neuralscape.com> On Thu, Jun 19, 2008 at 08:24:38AM +0100, John Hearns wrote: > On Wed, 2008-06-18 at 15:11 -0700, Greg Lindahl wrote: > > Speaking of lm_sensors, does anyone have configs for recent SuperMicro > > mobos? My SuperMicro support contact doesn't have ay idea, and running > > sensors-detect leaves me with lots of readings which are > > Can't help you on the lm_sensors front, we always use IPMI these days. > But hearing of miscalibrated values does not surprise me. Hi John, Yes, miscalibrated values and even the mapping of labels to expected devices can be in error, even when the lm_sensors subsystem is fully operational. This is inherent in the architecture of the system, if you will. The configuration file must have information specific to the system board implementation, rather than merely related to specific sensor chips. The generic configuration files found in distributions often are not correct (even if they look reasonable) for specific system boards. And, if you are attempting to support an appliance product line, where there are numerous system boards in various appliances, then this becomes quite a chore to manage, due to the architecture of lm_sensors subsystem. IPMI is clearly a better way to go. But lm_sensors can always be made to work, it just requires some in-house expertise in many cases. Thanks, Karen -- Karen Shaeffer Neuralscape, Palo Alto, Ca. 94306 shaeffer@neuralscape.com http://www.neuralscape.com From rgb at phy.duke.edu Thu Jun 19 06:58:44 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:12 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> Message-ID: On Wed, 18 Jun 2008, Jim Lux wrote: > So.. if your (foreign person) buddy is designing thermonuclear devices in > their garage, and they complain about how slow it is to run the hydrocodes to > simulate stuff, better not hand them that old copy of Sterling, et al., or > even worse, give them rgb's website. (the latter would be too suspicious, > since rgb *is* a physicist, doing monte carlo simulations no less, while Tom > Sterling is *just a computer scientist*) Ah, taking my name in vain I see...;-) Don't forget, at one time, the shocking subversive rgb had a mirror of the NWFAQ: http://nuclearweaponarchive.org/Nwfaq/Nfaq0.html on his personal website, which contains nearly all the instructions for making nuclear bombs one could ever need even without a cluster. At this point the whole idea of computers (clusters or not) as weapons is just plain silly. My cell phone and PDA are "weapons" by the standards set a mere fifteen years ago -- as far as being capable of running fast enough complex enough codes to solve the problems of designing "good enough" nuclear weapons is concerned. Well, I'm not sure about my cell phone, but my PDA has 64 MB of memory and a 400 MHz processor -- that is more than enough. Or my son's PS3 playstation, if you don't like the floating point on my PDA. Who cares? Compare the computing available to the people who actually DESIGNED the early thermonuclear devices to a single dual CPU, quad core (8 core total) 64 bit, over the counter compute "node" with 16 GB of memory (total cost WAY less than what I paid for my original IBM PC with its 15 MB of attached storage even ignoring the significant inflation accumulation since 1982). A joke, man, a joke. My laptop -- even my old one and not my new one or my still newer one that should get here tomorrow or Monday -- can design nuclear bombs, thermo or otherwise, and any halfway competent computational physicist can write the code needed to do so using the excellent numerical libraries that are readily available within any distribution of linux. Remember, building nuclear bombs ranges from VERY easy to not terribly difficult, until you want to build a BIG one, or a small one, or one that has to be "precise" in its performance. Terrorists need none of these things -- sloppy to the point of being a fizzle of sorts is still more than good enough. Building a kiloton-range U-235 suicide bomb (given the U-235) I could do in roughly a day, reusing a 12 gauge shotgun already in the house. I'd probably have to buy a metal lathe or small metal smelter, I admit, depending on how the Uranium was delivered to me. To make a really GOOD (efficient) bomb and get into the 10 KT range might take me a week or more to run down a few more materials -- a neutron source and reflector, some concrete, a remote trigger (suicide not being all that appealing to me), a superior propellant to gunpowder (e.g. a lump of TNT, homemade or otherwise). Handling or making the explosives would be the most dangerous or difficult part of the process as I'm not a chemist and it is very easy to blow yourself up, but I'm certain that it is still quite easy to get TNT and commercial triggers if you really really want to and have timescale months to acquire them legally and have no criminal record and have a legitimate construction project of some sort that requires blowing up some rocks on your property in the country. Building a plutonium bomb (given the plutonium) is considerably more difficult and not a project for MY garage even with a lathe. Plutonium is downright dangerous to handle, and the construction requires shaped lensing charges, which in turn requires an ability to make precision casts of at least two different explosives with differential burn rates and to set them off with high speed triggers at exactly the same time. However, "exactly the same time" very probably means something quite different now from what it did in 1945. Again, my PC contains nanosecond clocks; over the counter electronics can probably provide enough switching speed and power to get within the range that will suffice for an implosion device, especially one slightly overengineered in other respects. Sure, the government controls known fast switches, but I very much doubt that they control the knowledge of how to make them, and I doubt that they are that hard to make. I'd say a plutonium bomb (given the plutonium) is a project that would cost somewhere in the $100K range up to a few million, for a small team that includes an explosive expert, an electronics expert, a physicist, and a computer geek, to get (again) to the 10+ KT range. Thermonuclear fusion and the 100+ KT to MT range are similarly straightforward. From what I recall, one can just monkey around with building bigger bombs surrounded by more fissile material and get close to the latter, adding fusile material such as tritium (expensive and dangerous) or deuterium (plentiful and harmless) and lithium into spaces between trigger and a U-238 casing. To get to MT, one has to build a proper dual implosion device so that the trigger causes both heating, compression, and neutrons to all happen at the same time to a significant volume of fusile material. The NWFAQ doesn't contain engineering specs (as it proudly and irrelevantly announces). So it is entirely possible that one could try for MT and only end up with a 100 KT or so. However, range of total destruction scales like the 1/3 power or thereabouts of the energy, so a 1 MT device only does roughly twice the damage of a 100 KT device anyway. Code and my laptop might up the odds of making a MT+ device work the first time, but... ...from the point of view of strategic war or tactical war these differences matter, I suppose. A 100 KT "fizzle" might let a hard shelter survive where a 1 MT non-fizzle would kill it. Getting too big or two small an explosion can either kill your own troops or not kill all of the enemy on an actual battlefield. Tactical devices like neutron bombs require significant engineering and experimentation to achieve and are not garage projects, I suspect -- get them wrong on the one side and they're thermonuclear devices that are far more powerful than you anticipate, get them wrong on the other and they don't make a significant flux of neutrons and the enemy soldiers overrun your position. However, from the point of view of terrorist bombs NOBODY CARES -- or should care -- a 1 KT "near fizzle" bomb is the moral equivalent of two million pounds of TNT, 100 panel trucks loaded full of TNT and set off all at once. Set off in the right place, it would do billions of dollars in damage and kill as many as hundreds of thousands of people, especially if it were surrounded by e.g. a ton or so of cobalt. I could do that with my shotgun. A plutonium bomb that COMPLETELY fizzled would fizzle by prematurely fission of imperfectly compressed ball of plutonium blowing apart before a large "enough" fraction of the total mass fissions, spreading highly radioactive plutonium and various radioactive fission byproducts over a few hundred acres of presumably expensive and densely populated real estate and STILL would likely achieve at least billions of dollars in damages and a death toll in the thousands to tens of thousands, the latter spread out over ten or twenty years and from horrible things like leukemia and other cancers. "Chernobyl" in the middle of the city of your choice. The only solution to the threat of nuclear terrorism is to change the conditions that give rise to terrorism in the first place. Poverty, ignorance, scriptural theistic religion (with its absolutes and myths that lead to irrational action), hopelessness, nationalism, overpopulation, scarcity, human greed. If we as a species fail to do this, then SOONER or LATER, somebody is going to end up with few chunks of sufficiently pure U-235, a garage, and a shotgun, or more likely will end up being provided a properly engineered and tested design plutonium bomb all "ready to use" and will then -- not terribly surprisingly -- use it on us. Bad as it is, this beats the hell out of the condition I grew up in -- living just outside of DC but well within the radius of TD expected for a 10 MT airburst over the Washington Monument, with MIRV'd ICBMs targeted on both sides and a single moment of insanity away from MAD -- but it isn't terribly desireable. No matter what the holding action, no matter what the defenses, it is just too easy. Put a shotgun-bomb into a freighter as machine parts or a nameless lump of concrete down in the bilge, sail it up the Saint Clair river, there goes Detroit, or maybe Chicago. SF, NY, Miami, New Orleans, Washington, Baltimore, Boston -- all vulnerable. Or unload it, put it on a truck, and anyplace is a target. The government could uncover and stop thirty such plans in a row, but the thirty-first loses a city. Guantanamo isn't the answer, because it doesn't address the right questions, eliminate the root cause, it is at best a finger in the dyke that creates still more leaks from its own nontrivial contribution to the sheer injustice of it all. Poverty, despair, scriptural religion, social injustice on a global scale ... nothing else will do but eliminating them, and it will take fifty years of CORRECTLY directed action to do much about them, all the while sticking fingers into the dyke whereever we discover a leak. In the meantime, well, one day we'll lose a city. Or worse. All it takes for evil to triumph is for humans of goodwill to do nothing, and we've been doing the WRONG thing (much of it being nothing) for far too long to escape unscathed. Depressingly and politically yours (and not ENTIRELY off topic -- I did talk for a BIT about computing at the very beginning:-), rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From rgb at phy.duke.edu Thu Jun 19 07:26:45 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <6D3FFF5E-20CA-4AFC-88B2-18AA74C7EB28@xs4all.nl> References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <87iqw6i7ck.fsf@snark.cb.piermont.com> <6D3FFF5E-20CA-4AFC-88B2-18AA74C7EB28@xs4all.nl> Message-ID: On Thu, 19 Jun 2008, Vincent Diepeveen wrote: > USA and allowing encryption beyond the cracking capabilities of a 1st year > computer science student... ...hmmm Dear Vincent, IIRC almost any of the high-end encryption routines available within linux are effectively uncrackable, certainly uncrackable to somebody with less than NSA-class resources. Most of the routines scale -- if 1024 bit keys still leave you worried, use a larger one. Brute force searches of keyspace are easily driven past the capabilities of any computer; all that remains are solving certain "hard" problems in e.g. number theory implicit to the method. I haven't looked at the literature recently, but to the best of my knowledge e.g the integer factorization problem cannot be solved in polynomial time for any known algorithm, and factoring a single 663 bit integer in a test that took ballpark of a GFLOP-century of effort for the record as of 2005. NP means that a 2048 bit key would very likely require TFLOP-centuries of effort if not more -- maybe NSA trying really really hard could do it, maybe IBM using big blue could do it, maybe the top ten of the top-500 could do it -- in a year or ten of unbroken effort -- or maybe not. Moore's Law and the possible advent of quantum computing (where there is reportedly a P-time algorithm but no hardware to run it on) might change this, a truly significant advance in number theory or human cleverness might change it -- training a humongous neural network to perform 2048 bit factorizations well enough that its partial and horrendously parallel and nonlinear solutions serve to reduce the search space to where NP searches can find it in human-accessible time, for example. Barring that, you're pretty safe with any of the open source encryption methods with a large key. rgb > I remember how i did do big efforts to get vista ultimate bitlocker to work. > On paper it sounds ok. AES 256 bits CBC. AES is a "good" encryption IIRC. The weakness, of course, is guarding the key (as always). ssh is quite secure, but not if you have both of my public/private keys. Symmetric methods are quite secure as well, but not if someone takes your key. And on Vista, it doesn't matter whether or not the key is on a USB stick if somebody cracks the OS, which is the REAL weak link in the chain, and inserts code to copy your key the next time you insert the USB stick and then decrypt and transmit the entire contents of your encrypted drive. It takes NSA resources, perhaps, to find long keys or factor long keys or solve elliptical or discrete logarithm problems in number theory that have only NP algorithms that scale badly as the keysize gets large. Does anyone seriously think that it takes NSA resources to crack Vista itself? Especially in a consumer environment (that is, systems run by anyone less knowledgeable than a computer systems engineer or sysadmin)? rgb > > The idea is : your usb stick has the encryption key and only that thing has > it. > So no one can decrypt the partition without that usb stick. > > In case you forget to load that usb stick having the encryption key, > there is a general manner to unlock the machine by feeding it a long code. > > Looks great isn't it? > > So far the paper... > > But now the usual bug; the implementation that practical was allowed by one > those guys on the Perry-Sport mailing list. > Of course we don't want to tire ourselves too much typing too long unlock > codes... > > That unlock code, stored at a different USB stick is 48 digits. > > By the way 48 digits is how many bits? > > Right, that's less than 48 * (log 10/log 2) = 160 bits. > > So the problem for our first year student has been brought back from 256 to > 160 bits already. > > Not that it is a hobby mine, but one day i rebooted the machine and had > forgotten to put in the USB encryption stick. > > By accident i mistyped the key. Windows then told me: > "you made a mistake somewhere in those 5 digits of the 48 digit key, > please retype them". > > Doh. > > So my guess is that soon i do not need to worry when by accident i lose that > USB stick... > > Vincent > > > > On Jun 19, 2008, at 2:41 AM, Perry E. Metzger wrote: > >> >> Jim Lux writes: >>> In general, fundamental research is not subject to export controls, so >>> if you frame your problem in terms of abstract mathematical problems, >>> you're not going to be treading on any toes. However, start >>> distributing it as "Jim Lux's superduper encryptor/password cracker, >>> now with 1024 bit capability!" and it's moved from fundamental >>> research to a product. >> >> Under current rules, provided it is open source or generally available >> to any buyer, you can distribute and export cryptographic code quite >> freely. There are some minimal reporting requirements, but they're >> barely worth mentioning. >> >> That's the reason you can freely distribute things like Kerberos, >> OpenSSL, pgp/gpg, etc. >> >> >> Perry >> -- >> Perry E. Metzger perry@piermont.com > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From kilian at stanford.edu Thu Jun 19 09:45:21 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> Message-ID: <200806190945.21604.kilian@stanford.edu> On Thursday 19 June 2008 06:58:44 am Robert G. Brown wrote: > Getting too big > or two small an explosion can either kill your own troops or not kill > all of the enemy on an actual battlefield. To add some more OT stuff to this thread, I don't think a nuclear weapon has ever been used (or even considered being used) to kill troops on a battlefield. Some cluster bombs (hey, back on topic! :)) are probably enough for this purpose. IMHO, a nuclear weapon is mainly a dissuasion weapon, ie, one you claim you own to make your ennemies think twice before they strike you. Or that you use against civilians to make your point louder, and let your ennemies understand they'd better surrender. That's why I find the association between "nuclear weapon" and "battlefield" a bit irrelevant. Other than that, pretty interesting stuff. I'm unfortunately supporting your conclusions. Cheers, -- Kilian From peter.st.john at gmail.com Thu Jun 19 09:52:26 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <1213859837.4784.9.camel@Vigor13> References: <485920D8.2030309@ias.edu> <20080618214603.GC31594@bx9.net> <200806181521.47115.kilian@stanford.edu> <48599AC9.3040100@berkeley.edu> <1213859837.4784.9.camel@Vigor13> Message-ID: I dug up this pdf from Nvidia: http://www.nvidia.com/docs/IO/43395/tesla_product_overview_dec.pdf Since I can't imagine coding a graphics card while it serves my X :-) I supposed one might put the PCIE card in a box with a cheap SVGA for the least-cost CUDA experiment (one GPU, 128 "thread processors" per GPU). The deskside is 2 GPU with 3 GB, the rackable is 4 GPU with 6 GB; they have PCIE adapter cards to talk to your workstation. I think one plan might be like the GraPE (Gravity Pipe); maybe one of the rackables alternating with a CPU MB acting as net host and fileserver, so each (2-board) node has 4 GPU for array computing. Peter P.S. incidentally, while I was browsing Nvidia, I spec'd out a fantasy gaming rig. $23K, 256GB of solid state "disk", 3x 1GB video cards, 180W just to liquid-cool the CPU :-) Maybe next year. On 6/19/08, John Hearns wrote: > > On Wed, 2008-06-18 at 16:31 -0700, Jon Forrest wrote: > > > Kilian CAVALOTTI wrote: > > > I'm glad you mentioned this. I've read through much of the information > > on their web site and I still don't understand the usage model for > > CUDA. By that I mean, on a desktop machine, are you supposed to have > > 2 graphics cards, 1 for running CUDA code and one for regular > > graphics? If you only need 1 card for both, how do you avoid the > > problem you mentioned, which was also mentioned in the documentation? > > Actually, I should imagine Kilian is referring to something else, > not the inbuilt timeout which is in the documentation. But I can't speak > for im. > > > > > > Or, if you have a compute node that will sit in a dark room, > > you aren't going to be running an X server at all, so you won't > > have to worry about anything hanging? > > > I don't work for Nvidia, so I can't say! > But the usage model is as you say - you can prototype applications which > will run for a short time on the desktop machine, but long runs are > meant to be done on a dedicated back-end machine. > If you want a totally desk-side solution, they sell a companion box > which goes alongside and attaches via a ribbon cable. I guess the art > here is finding a motherboard with the right number and type of > PCI-express slots to take both the companion box and a decent graphics > card for X use. > > > > > > > I'm planning on starting a pilot program to get the > > chemists in my department to use CUDA, but I'm waiting > > for V2 of the SDK to come out. > > > > > Why wait? The hardware will be the same, and you can dip your toe in the > water now. > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080619/31c25251/attachment.html From kilian at stanford.edu Thu Jun 19 10:05:15 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <1213859837.4784.9.camel@Vigor13> References: <485920D8.2030309@ias.edu> <48599AC9.3040100@berkeley.edu> <1213859837.4784.9.camel@Vigor13> Message-ID: <200806191005.15794.kilian@stanford.edu> On Thursday 19 June 2008 12:17:07 am John Hearns wrote: > Actually, I should imagine Kilian is referring to something else, > not the inbuilt timeout which is in the documentation. But I can't > speak for im. I don't know about this timeout. As I said we didn't really had time nor the resources to investigate the crashes. But I've definitely seen a machine freeze when launching a binary containing CUDA code. > I guess the art > here is finding a motherboard with the right number and type of > PCI-express slots to take both the companion box and a decent > graphics card for X use. AFAIK, the multi GPU Tesla boxes contain up to 4 Tesla processors, but are hooked to the controlling server with only 1 PCIe link, right? Does this spell like "bottleneck" to anyone? Sure, moving data between host memory and GPU memory is not what you do the most often, but still. Cheers, -- Kilian From glen.beane at jax.org Thu Jun 19 10:11:18 2008 From: glen.beane at jax.org (Glen Beane) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <200806190945.21604.kilian@stanford.edu> References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <200806190945.21604.kilian@stanford.edu> Message-ID: <485A9336.8010303@jax.org> Kilian CAVALOTTI wrote: > On Thursday 19 June 2008 06:58:44 am Robert G. Brown wrote: >> Getting too big >> or two small an explosion can either kill your own troops or not kill >> all of the enemy on an actual battlefield. > > To add some more OT stuff to this thread, I don't think a nuclear weapon > has ever been used (or even considered being used) to kill troops on a > battlefield. look up "tactical nukes". These were the USA's only hope of defending Europe from a Soviet ground invasion. -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153 From landman at scalableinformatics.com Thu Jun 19 10:19:28 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <200806190945.21604.kilian@stanford.edu> References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <200806190945.21604.kilian@stanford.edu> Message-ID: <485A9520.2080508@scalableinformatics.com> Kilian CAVALOTTI wrote: > On Thursday 19 June 2008 06:58:44 am Robert G. Brown wrote: >> Getting too big >> or two small an explosion can either kill your own troops or not kill >> all of the enemy on an actual battlefield. > > To add some more OT stuff to this thread, I don't think a nuclear weapon > has ever been used (or even considered being used) to kill troops on a > battlefield. Some cluster bombs (hey, back on topic! :)) are probably > enough for this purpose. Tactical nukes (aimed at armies) were on the table for a few of the NATO scenarios involving responses to Soviet invasion of western Europe (based upon some of the historical reading, though I am not sure how serious they were). The western Europeans were understandably un-enthusiastic about such scenarios. > IMHO, a nuclear weapon is mainly a dissuasion weapon, ie, one you claim > you own to make your ennemies think twice before they strike you. Or > that you use against civilians to make your point louder, and let your > ennemies understand they'd better surrender. This makes a number of fundamental presumptions, which may not be true in all cases. First and foremost, it assumes that the potential recipient of the attack or response to attack is a rational actor. As seen on the world stage, this isn't always the case. A rational head of government will balance the lives lost in an attack against any possible gains. An irrational apocalyptic head of government will not likely make that calculation, but an alternative one in which they somehow come out a winner regardless of the events. Second, it assumes that the potential recipient of the attack or response to attack is state based. This has also been demonstrated to be problematic on today's world stage. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From peter.st.john at gmail.com Thu Jun 19 10:22:50 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <200806190945.21604.kilian@stanford.edu> References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <200806190945.21604.kilian@stanford.edu> Message-ID: Actually, nuclear weapons have indeed been considered for killing troops on the battlefield. At one time, the possibility of the Soviet Union invading western Europe seemed not so remote. Here is a link (wiki) to basically a bazooka launched fission bomb: http://en.wikipedia.org/wiki/Davy_Crockett_%28nuclear_device%29 with pictures. ("recoiless gun" means a rocket fired from a tube). The idea would be to for a relatively small force to cope with a relatively large number of tanks. Peter On 6/19/08, Kilian CAVALOTTI wrote: > > On Thursday 19 June 2008 06:58:44 am Robert G. Brown wrote: > > Getting too big > > or two small an explosion can either kill your own troops or not kill > > all of the enemy on an actual battlefield. > > > To add some more OT stuff to this thread, I don't think a nuclear weapon > has ever been used (or even considered being used) to kill troops on a > battlefield. Some cluster bombs (hey, back on topic! :)) are probably > enough for this purpose. > > IMHO, a nuclear weapon is mainly a dissuasion weapon, ie, one you claim > you own to make your ennemies think twice before they strike you. Or > that you use against civilians to make your point louder, and let your > ennemies understand they'd better surrender. > > That's why I find the association between "nuclear weapon" > and "battlefield" a bit irrelevant. > > Other than that, pretty interesting stuff. I'm unfortunately supporting > your conclusions. > > Cheers, > -- > > Kilian > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080619/54ddac50/attachment.html From perry at piermont.com Thu Jun 19 10:27:41 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: (Robert G. Brown's message of "Thu\, 19 Jun 2008 10\:26\:45 -0400 \(EDT\)") References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <87iqw6i7ck.fsf@snark.cb.piermont.com> <6D3FFF5E-20CA-4AFC-88B2-18AA74C7EB28@xs4all.nl> Message-ID: <87k5glfi7m.fsf@snark.cb.piermont.com> Again, this is not a cryptography list, but I'll correct a few small things... "Robert G. Brown" writes: > I haven't looked at the literature recently, but to the best of my > knowledge e.g the integer factorization problem cannot be solved in > polynomial time for any known algorithm, Correct. > and factoring a single 663 bit integer in a test that took > ballpark of a GFLOP-century of effort for the record as of 2005. I don't remember the record, but at this point it is considered (theoretically) feasible to attack 1024 bit RSA keys using GNFS and similar methods -- Dan Bernstein has published on this, you can doubtless find the paper on his site. Serious users are using 2048 bit RSA keys at this point. There aren't nearly such good methods for attacking elliptic curve based systems, and many people have migrated to those for performance reasons -- you can use shorter keys with (it is believed) equal security. > ssh is quite secure, but not if you have both of my public/private > keys. That depends on what you mean by "secure". There are two forms of security provided by SSH. One is protection from people trying to break in to your account, the other is protection from people reading your traffic over the network. I can log in using your credentials if I have your private key and you are using SSH with public key authentication. However, even if I have both of your private and public keys, the ephemeral key used for a particular session is agreed to using Diffie-Hellman key exchange, and mere knowledge of your long term keys will not allow anyone to read your session traffic. This property is known as "Perfect Forward Secrecy." (Technically, this is only true of sshv2 -- sshv1 used random nonces exchanged under RSA for the key material, but sshv1 is no longer in wide use because it has a number of security issues.) -- Perry E. Metzger perry@piermont.com From jmdavis1 at vcu.edu Thu Jun 19 10:39:41 2008 From: jmdavis1 at vcu.edu (Mike Davis) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <200806190945.21604.kilian@stanford.edu> References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <200806190945.21604.kilian@stanford.edu> Message-ID: <485A99DD.1030707@vcu.edu> To continue to be OT. Look up the Davey Crockett. http://en.wikipedia.org/wiki/Davy_Crockett_(nuclear_device) It is a weapon meant exclusively for battlefield use. It is also almost a suicide weapon. Kilian CAVALOTTI wrote: > On Thursday 19 June 2008 06:58:44 am Robert G. Brown wrote: > >> Getting too big >> or two small an explosion can either kill your own troops or not kill >> all of the enemy on an actual battlefield. >> > > To add some more OT stuff to this thread, I don't think a nuclear weapon > has ever been used (or even considered being used) to kill troops on a > battlefield. Some cluster bombs (hey, back on topic! :)) are probably > enough for this purpose. > > IMHO, a nuclear weapon is mainly a dissuasion weapon, ie, one you claim > you own to make your ennemies think twice before they strike you. Or > that you use against civilians to make your point louder, and let your > ennemies understand they'd better surrender. > > That's why I find the association between "nuclear weapon" > and "battlefield" a bit irrelevant. > > Other than that, pretty interesting stuff. I'm unfortunately supporting > your conclusions. > > Cheers, > From kilian at stanford.edu Thu Jun 19 10:50:50 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485A9520.2080508@scalableinformatics.com> References: <485920D8.2030309@ias.edu> <200806190945.21604.kilian@stanford.edu> <485A9520.2080508@scalableinformatics.com> Message-ID: <200806191050.51115.kilian@stanford.edu> On Thursday 19 June 2008 10:19:28 am you wrote: > This makes a number of fundamental presumptions, which may not be > true in all cases. > > First and foremost, it assumes that the potential recipient of the > attack or response to attack is a rational actor. That's a very valid point. It's all been described in deterrence theories, and the paradgim shifted from the Cold War's "deterrence of the strong by the weak" to today's "deterrence of the crazy by the weak". > As seen on the > world stage, this isn't always the case. A rational head of > government will balance the lives lost in an attack against any > possible gains. An irrational apocalyptic head of government will > not likely make that calculation, but an alternative one in which > they somehow come out a winner regardless of the events. > > Second, it assumes that the potential recipient of the attack or > response to attack is state based. This has also been demonstrated > to be problematic on today's world stage. Agreed. Then how adequate is a tactical nuke against military troops in today's world stage? Cheers, -- Kilian From vanallsburg at hope.edu Thu Jun 19 10:52:50 2008 From: vanallsburg at hope.edu (Paul Van Allsburg) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485A99DD.1030707@vcu.edu> References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <200806190945.21604.kilian@stanford.edu> <485A99DD.1030707@vcu.edu> Message-ID: <485A9CF2.1050408@hope.edu> Mike Davis wrote: > To continue to be OT. Look up the Davey Crockett. > > > http://en.wikipedia.org/wiki/Davy_Crockett_(nuclear_device) > > > > It is a weapon meant exclusively for battlefield use. It is also > almost a suicide weapon. > and a live video at http://www.sonicbomb.com/modules.php?name=Content&pa=showpage&pid=56 -- Paul Van Allsburg Computational Science & Modeling Facilitator Natural Sciences Division, Hope College 35 East 12th Street Holland, Michigan 49423 616-395-7292 http://www.hope.edu/academic/csm/ From kilian at stanford.edu Thu Jun 19 11:00:28 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485A9336.8010303@jax.org> References: <485920D8.2030309@ias.edu> <200806190945.21604.kilian@stanford.edu> <485A9336.8010303@jax.org> Message-ID: <200806191100.29026.kilian@stanford.edu> On Thursday 19 June 2008 10:11:18 am you wrote: > > To add some more OT stuff to this thread, I don't think a nuclear > > weapon has ever been used (or even considered being used) to kill > > troops on a battlefield. > > look up "tactical nukes". These were the USA's only hope of > defending Europe from a Soviet ground invasion. Well, what would have been the effect of launching nuclear weapons to defend Europe in case of a Soviet invasion? They would have been either launched to where the Soviet troops actually were, ie, on Europe, with the main effect of wiping up the countries they were supposed to protect. Not so appealing. Or, and it's probably the most plausible scenario, they would have been aimed to USSR, and likely to major cities, where they would have killed mostly civilians, not troops. With the hope that the Soviet government would withdraw from Europe. That's why I think nuclear weapons are hardly a mean to kill military troops on a battlefield. I concede that tactical nukes are still weapons, and that the main purpose of a weapon is to hurt your ennemy. But not only: building and showing off bigger weapons can also be a way to frighten him, hoping that it will be enough to dissuade him to attack. Cheers, -- Kilian From James.P.Lux at jpl.nasa.gov Thu Jun 19 11:08:48 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> Message-ID: <6.2.5.6.2.20080619105301.02cc8fd8@jpl.nasa.gov> At 06:58 AM 6/19/2008, Robert G. Brown wrote: >On Wed, 18 Jun 2008, Jim Lux wrote: > >>So.. if your (foreign person) buddy is designing thermonuclear >>devices in their garage, and they complain about how slow it is to >>run the hydrocodes to simulate stuff, better not hand them that old >>copy of Sterling, et al., or even worse, give them rgb's website. >>(the latter would be too suspicious, since rgb *is* a physicist, >>doing monte carlo simulations no less, while Tom Sterling is *just >>a computer scientist*) > >Ah, taking my name in vain I see...;-) hardly in vain.. (hoping you actually get to read this, in some venue other than your detention hearing, ) >Remember, building nuclear bombs ranges from VERY easy to not terribly >difficult, until you want to build a BIG one, or a small one, or one >that has to be "precise" in its performance. Terrorists need none of >these things -- sloppy to the point of being a fizzle of sorts is still >more than good enough. Nicholas Freeling, "Gadget" is a thriller/detective story about someone (like rgb) being kidnapped and forced to design a smallish device for some nefarious sorts. A better story, in my opinion, than, say, Clancy's "sum of all fears". Building a plutonium bomb (given the plutonium) is considerably more >difficult and not a project for MY garage even with a lathe. Plutonium >is downright dangerous to handle, and the construction requires shaped >lensing charges, which in turn requires an ability to make precision >casts of at least two different explosives with differential burn rates >and to set them off with high speed triggers at exactly the same time. One can look at the dual-use list at ORNL and see what sort of precision is considered "dangerous" to bound your design effort. >However, "exactly the same time" very probably means something quite >different now from what it did in 1945. Again, my PC contains >nanosecond clocks; over the counter electronics can probably provide >enough switching speed and power to get within the range that will >suffice for an implosion device, especially one slightly overengineered >in other respects. Sure, the government controls known fast switches, >but I very much doubt that they control the knowledge of how to make >them, and I doubt that they are that hard to make. Simultaneous is easy.. equal length coax, for instance. It's the switches that are tough (and export controlled) A bit trickier than you might think, to get the jitter down in the subnanosecond range, and to have the high peak currents needed. And no, you can't use spare xenon flashtubes you've scavenged out of disposable cameras. If you're worried about small (always an issue when thinking about nanoseconds.. 30cm/nanosecond for free space propagation), there's also the energy storage to consider. To optimize the yield, too, the timing of your zipper relative to your compression is important. >Thermonuclear fusion and the 100+ KT to MT range are similarly >straightforward. From what I recall, one can just monkey around with >building bigger bombs surrounded by more fissile material and get close >to the latter, adding fusile material such as tritium (expensive and >dangerous) or deuterium (plentiful and harmless) and lithium into spaces >between trigger and a U-238 casing. Uh.. probably not.. That's the "alarm clock" or "layer cake" design and has some issues. I refer interested readers to Morland's 1977 article in "The Progressive" for more information. > To get to MT, one has to build a >proper dual implosion device so that the trigger causes both heating, >compression, and neutrons to all happen at the same time to a >significant volume of fusile material. The NWFAQ doesn't contain >engineering specs (as it proudly and irrelevantly announces). So it is >entirely possible that one could try for MT and only end up with a 100 >KT or so. However, range of total destruction scales like the 1/3 power >or thereabouts of the energy, so a 1 MT device only does roughly twice >the damage of a 100 KT device anyway. Code and my laptop might up the >odds of making a MT+ device work the first time, but... cube scaling is the rule in explosives (all the way down to sub kilo amounts), however, for very large devices, where the "point source" assumption is invalid (e.g. that 10km fireball), it does break down. There's also the issue of the propagation of a shock wave in air and air not being a linear medium. >...from the point of view of strategic war or tactical war these >differences matter, I suppose. A 100 KT "fizzle" might let a hard >shelter survive where a 1 MT non-fizzle would kill it. Getting too big >or two small an explosion can either kill your own troops or not kill >all of the enemy on an actual battlefield. Or, knocking the lid off a silo, which requires both precision targeting AND suitable yield AND something that fits in a package that can be appropriately delivered. >Tactical devices like >neutron bombs require significant engineering and experimentation to >achieve and are not garage projects, I suspect -- get them wrong on the >one side and they're thermonuclear devices that are far more powerful >than you anticipate, get them wrong on the other and they don't make a >significant flux of neutrons and the enemy soldiers overrun your >position. Hence the interest in things like the National Ignition Facility. >However, from the point of view of terrorist bombs NOBODY CARES -- or >should care -- a 1 KT "near fizzle" bomb is the moral equivalent of two >million pounds of TNT, 100 panel trucks loaded full of TNT and set off >all at once. Set off in the right place, it would do billions of >dollars in damage and kill as many as hundreds of thousands of people, >especially if it were surrounded by e.g. a ton or so of cobalt. The study done a couple years ago that postulated a "nominal yield"(e.g. 20kT) fission device in Los Angeles/Long Beach port showed that the majority of damage and death resulted from things like traffic jams and accidents in the crowds fleeing and overloading hospitals. The actual damage radius is fairly small (in the context of Los Angeles, which is >100km across) and the fallout (from a particularly dirty surface burst.. i.e. the shipping container on the dock sucking up the dirt) wasn't all that bad, in terms of dose. The panic, on the other hand, is lethal. > I could >do that with my shotgun. Hmm. Mcrit for good quality U235 is fairly high, especially unreflected. I don't know if your shotgun has sufficient oomph to assemble it quickly enough without predetonation or a fizzle. I've seen numbers for required assy speed in the 1000 m/sec sort of range (i.e. you've got to move from a noncrit to a crit configuration in the amount of time between spontaneous neutrons appearing to get things rolling). Accelerating 10kg, say, to 1000 m/sec, takes a heap o'joules >Bad as it is, this beats the hell out of the condition I grew up in -- >living just outside of DC but well within the radius of TD expected for >a 10 MT airburst over the Washington Monument, with MIRV'd ICBMs >targeted on both sides and a single moment of insanity away from MAD -- >but it isn't terribly desireable. No matter what the holding action, no >matter what the defenses, it is just too easy. Put a shotgun-bomb into >a freighter as machine parts or a nameless lump of concrete down in the >bilge, sail it up the Saint Clair river, there goes Detroit, or maybe >Chicago. SF, NY, Miami, New Orleans, Washington, Baltimore, Boston -- >all vulnerable. Or unload it, put it on a truck, and anyplace is a >target. why worry about ICBMs when DHL/FedEx will deliver it to your selected doorstep? From peter.st.john at gmail.com Thu Jun 19 11:09:09 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485A99DD.1030707@vcu.edu> References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <200806190945.21604.kilian@stanford.edu> <485A99DD.1030707@vcu.edu> Message-ID: War is hell, but I wouldn't call it a "suicide device". The range of the rocket is on the order of 1000 meters, and the effective radius at the target is on the order of 100 m. You wouldn't want to shoot yourself in the foot with one of these, you'd take out your own batallion, but you are aiming at a valley or a hillside, not an oncoming tank. Peter On 6/19/08, Mike Davis wrote: > > To continue to be OT. Look up the Davey Crockett. > > > http://en.wikipedia.org/wiki/Davy_Crockett_(nuclear_device) > > > > It is a weapon meant exclusively for battlefield use. It is also almost a > suicide weapon. > > > > > > Kilian CAVALOTTI wrote: > >> On Thursday 19 June 2008 06:58:44 am Robert G. Brown wrote: >> >> >>> Getting too big >>> or two small an explosion can either kill your own troops or not kill >>> all of the enemy on an actual battlefield. >>> >>> >> >> To add some more OT stuff to this thread, I don't think a nuclear weapon >> has ever been used (or even considered being used) to kill troops on a >> battlefield. Some cluster bombs (hey, back on topic! :)) are probably enough >> for this purpose. >> >> IMHO, a nuclear weapon is mainly a dissuasion weapon, ie, one you claim >> you own to make your ennemies think twice before they strike you. Or that >> you use against civilians to make your point louder, and let your ennemies >> understand they'd better surrender. >> >> That's why I find the association between "nuclear weapon" and >> "battlefield" a bit irrelevant. >> >> Other than that, pretty interesting stuff. I'm unfortunately supporting >> your conclusions. >> >> Cheers, >> >> > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080619/6e59caf4/attachment.html From glen.beane at jax.org Thu Jun 19 11:13:36 2008 From: glen.beane at jax.org (Glen Beane) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <200806191100.29026.kilian@stanford.edu> References: <485920D8.2030309@ias.edu> <200806190945.21604.kilian@stanford.edu> <485A9336.8010303@jax.org> <200806191100.29026.kilian@stanford.edu> Message-ID: <485AA1D0.1080209@jax.org> Kilian CAVALOTTI wrote: > On Thursday 19 June 2008 10:11:18 am you wrote: >>> To add some more OT stuff to this thread, I don't think a nuclear >>> weapon has ever been used (or even considered being used) to kill >>> troops on a battlefield. >> look up "tactical nukes". These were the USA's only hope of >> defending Europe from a Soviet ground invasion. > > Well, what would have been the effect of launching nuclear weapons to > defend Europe in case of a Soviet invasion? They would have been either > launched to where the Soviet troops actually were, ie, on Europe, with > the main effect of wiping up the countries they were supposed to > protect. Not so appealing. > Or, and it's probably the most plausible scenario, they would have been > aimed to USSR, and likely to major cities, where they would have killed > mostly civilians, not troops. With the hope that the Soviet government > would withdraw from Europe. This is terribly off topic, but you are thinking of strategic nuclear weapons. You couldn't aim a tactical nuke at a city in the USSR unless you were a mile or so from the city. These were small rocket or artillery fired warheads of less than 100 pounds. Tactical nukes are not large enough to destroy cities. The main effect would not be wiping out the countries they were meant to protect, the main effect would be to wipe out large tank formations and make small areas temporarily irradiated to block the movement of troops. > > That's why I think nuclear weapons are hardly a mean to kill military > troops on a battlefield. Strategic nukes, no. Tactical nukes, yes. -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153 From jmdavis1 at vcu.edu Thu Jun 19 11:25:05 2008 From: jmdavis1 at vcu.edu (Mike Davis) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <200806190945.21604.kilian@stanford.edu> <485A99DD.1030707@vcu.edu> Message-ID: <485AA481.2030000@vcu.edu> According to one weapons designer the only safe way to use it was to fire from a hilltop into a valley from a jeep and then drive like hell into the next valley. If you are within 150m you will receive an instantly lethal dose. Within 400m you will receive up to 600 rem with is lethal. Even at to 1000m you would likely get enough radiation to interfere with your warfighting capacity. Peter St. John wrote: > War is hell, but I wouldn't call it a "suicide device". The range of > the rocket is on the order of 1000 meters, and the effective radius at > the target is on the order of 100 m. You wouldn't want to shoot > yourself in the foot with one of these, you'd take out your own > batallion, but you are aiming at a valley or a hillside, not an > oncoming tank. > Peter > > On 6/19/08, *Mike Davis* > > wrote: > > To continue to be OT. Look up the Davey Crockett. > > > http://en.wikipedia.org/wiki/Davy_Crockett_(nuclear_device) > > > > > It is a weapon meant exclusively for battlefield use. It is also > almost a suicide weapon. > > > > > > Kilian CAVALOTTI wrote: > > On Thursday 19 June 2008 06:58:44 am Robert G. Brown wrote: > > > Getting too big > or two small an explosion can either kill your own troops > or not kill > all of the enemy on an actual battlefield. > > > > To add some more OT stuff to this thread, I don't think a > nuclear weapon has ever been used (or even considered being > used) to kill troops on a battlefield. Some cluster bombs > (hey, back on topic! :)) are probably enough for this purpose. > > IMHO, a nuclear weapon is mainly a dissuasion weapon, ie, one > you claim you own to make your ennemies think twice before > they strike you. Or that you use against civilians to make > your point louder, and let your ennemies understand they'd > better surrender. > > That's why I find the association between "nuclear weapon" and > "battlefield" a bit irrelevant. > > Other than that, pretty interesting stuff. I'm unfortunately > supporting your conclusions. > > Cheers, > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > From bernard at vanhpc.org Thu Jun 19 11:28:08 2008 From: bernard at vanhpc.org (Bernard Li) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] SuperMicro and lm_sensors In-Reply-To: <20080619135020.GA8918@nlxdcldnl2.cl.intel.com> References: <20080618221153.GA24788@bx9.net> <20080619011419.GA7040@bx9.net> <20080619135020.GA8918@nlxdcldnl2.cl.intel.com> Message-ID: Hi David: On Thu, Jun 19, 2008 at 6:50 AM, Lombard, David N wrote: > Did you look for /proc/acpi/thermal_zone/*/temperature The glob is for > your BIOS-defined ID. If it does exist, that's the value that drives > /proc/acpi/thermal_zone/*/trip_points > > See also /proc/acpi/thermal_zone/*/polling_frequency I have always wondered about /proc/acpi/thermal_zone. I noticed that on some servers, the files exist, but on others, that directory is empty. I guess this is dependent on whether the BIOS exposes the information to the kernel? Or are there modules that I need to install to get it working? Thanks, Bernard From jmdavis1 at vcu.edu Thu Jun 19 11:29:40 2008 From: jmdavis1 at vcu.edu (Mike Davis) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <200806191100.29026.kilian@stanford.edu> References: <485920D8.2030309@ias.edu> <200806190945.21604.kilian@stanford.edu> <485A9336.8010303@jax.org> <200806191100.29026.kilian@stanford.edu> Message-ID: <485AA594.1060601@vcu.edu> The plan as per my limited understanding was to engage at Fulda Gap, slow the advance and then hit them with tactical nukes in E. Germany and further behind the lines. the Davey Crockett could be used in multiples in such a situation (and in combination with helicopter and ground anti-tank units) to stall the advance. The tactical and theatre weapons were to be used on the battlefield, and concentrations of troops behind the lines. Mike Davis Kilian CAVALOTTI wrote: > On Thursday 19 June 2008 10:11:18 am you wrote: > >>> To add some more OT stuff to this thread, I don't think a nuclear >>> weapon has ever been used (or even considered being used) to kill >>> troops on a battlefield. >>> >> look up "tactical nukes". These were the USA's only hope of >> defending Europe from a Soviet ground invasion. >> > > Well, what would have been the effect of launching nuclear weapons to > defend Europe in case of a Soviet invasion? They would have been either > launched to where the Soviet troops actually were, ie, on Europe, with > the main effect of wiping up the countries they were supposed to > protect. Not so appealing. > > Or, and it's probably the most plausible scenario, they would have been > aimed to USSR, and likely to major cities, where they would have killed > mostly civilians, not troops. With the hope that the Soviet government > would withdraw from Europe. > > That's why I think nuclear weapons are hardly a mean to kill military > troops on a battlefield. I concede that tactical nukes are still > weapons, and that the main purpose of a weapon is to hurt your ennemy. > But not only: building and showing off bigger weapons can also be a way > to frighten him, hoping that it will be enough to dissuade him to > attack. > > Cheers, > From kilian at stanford.edu Thu Jun 19 11:31:13 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485AA1D0.1080209@jax.org> References: <485920D8.2030309@ias.edu> <200806191100.29026.kilian@stanford.edu> <485AA1D0.1080209@jax.org> Message-ID: <200806191131.13502.kilian@stanford.edu> On Thursday 19 June 2008 11:13:36 am you wrote: > Strategic nukes, no. Tactical nukes, yes. That's right, I think I've been confused over the terms. Cheers, -- Kilian From kus at free.net Thu Jun 19 11:50:25 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] SuperMicro and lm_sensors In-Reply-To: Message-ID: In message from "Bernard Li" (Thu, 19 Jun 2008 11:28:08 -0700): >Hi David: > >On Thu, Jun 19, 2008 at 6:50 AM, Lombard, David N > wrote: > >> Did you look for /proc/acpi/thermal_zone/*/temperature The glob is >>for >> your BIOS-defined ID. If it does exist, that's the value that >>drives >> /proc/acpi/thermal_zone/*/trip_points >> >> See also /proc/acpi/thermal_zone/*/polling_frequency > >I have always wondered about /proc/acpi/thermal_zone. I noticed that >on some servers, the files exist, but on others, that directory is >empty. I guess this is dependent on whether the BIOS exposes the >information to the kernel? Or are there modules that I need to >install to get it working? AFAIK it depends from BIOS. On my Tyan S2932 w/last BIOS version this directory is empty. Mikhail > >Thanks, > >Bernard >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pbm.com Thu Jun 19 11:57:39 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <48599BA3.1050105@cse.ucdavis.edu> References: <485920D8.2030309@ias.edu> <20080618214603.GC31594@bx9.net> <48599BA3.1050105@cse.ucdavis.edu> Message-ID: <20080619185737.GB4491@bx9.net> On Wed, Jun 18, 2008 at 04:34:59PM -0700, Bill Broadley wrote: > Cuda seems to take a different approach, instead of trying to > auto-parallelize > a loop, it requires a function pointer to the code, and the function must > declare it's exit condition. In the end, CUDA doesn't end up being that much weirder than array processors. Which were considered to be significantly harder to program than, say, vector processors. Your post talked about how difficult it was to write code that worked. Well, the harder part is not getting some simple code written, it's getting your actual algorithm into that straight-jacket, and getting it to run fast. If I had more free time, I'd experiment with getting my pretty simple loopy hydrocode working. I suspect it would be pretty tedious, since the 1D operator at the heart of the algorithm is 4,000 lines long. -- greg From lindahl at pbm.com Thu Jun 19 12:08:02 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] SuperMicro and lm_sensors In-Reply-To: References: Message-ID: <20080619190801.GD4491@bx9.net> On Thu, Jun 19, 2008 at 10:50:25PM +0400, Mikhail Kuzminsky wrote: > AFAIK it depends from BIOS. On my Tyan S2932 w/last BIOS version this > directory is empty. On the Opteron box that I got lm_sensors working on, the k8temp module was required for some of the temperatures. But that directory is empty. "hit or miss" comes to mind... -- greg From peter.st.john at gmail.com Thu Jun 19 12:14:51 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <200806190945.21604.kilian@stanford.edu> <485A99DD.1030707@vcu.edu> Message-ID: It's off-topic, certainly. The wiki entry indicates ranges from 2 to 4 thousand meters (depending on model) and effective radius in hundreds of meters. Certainly dangerous for everyone, but so is a howitzer battery barrage. By "order" I meant nearest order of magnitude (factor of ten). The range of the weapon is around ten times as great as the effective killing radius. Peter On 6/19/08, R P Herrold wrote: > > On Thu, 19 Jun 2008, Peter St. John wrote: > > War is hell, but I wouldn't call it a "suicide device". The range of the >> rocket is on the order of 1000 meters, and the effective radius at the >> target is on the order of 100 m. >> > > the video posted indicated 4 kliks range, and a greater lethal radius ... > > but how is this on topic here? > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080619/034a176a/attachment.html From rgb at phy.duke.edu Thu Jun 19 10:14:09 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <200806190945.21604.kilian@stanford.edu> References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <200806190945.21604.kilian@stanford.edu> Message-ID: On Thu, 19 Jun 2008, Kilian CAVALOTTI wrote: > On Thursday 19 June 2008 06:58:44 am Robert G. Brown wrote: >> Getting too big >> or two small an explosion can either kill your own troops or not kill >> all of the enemy on an actual battlefield. > > To add some more OT stuff to this thread, I don't think a nuclear weapon > has ever been used (or even considered being used) to kill troops on a > battlefield. Some cluster bombs (hey, back on topic! :)) are probably > enough for this purpose. Never used, but then, only two have been used on humans at all, both strategic. This is a GOOD thing...;-) However, since the early fifties or sixties, MOST of the nuclear weapons on the planet were, and probably remain, "tactical" nukes that are indeed intended (at least nominally and in terms of their delivery systems) for use on battlefields or to achieve tactical advantage in hypothesized less-than-total wars, not strategic dissuasion from those wars happening. NATO and the Warsaw pact were basically a row of armies supported by tactical nukes on both sides for most of my lifetime up to the fall of the Soviet Union (which wasn't all that long ago). You might think your tanks were better, but you knew that the other side would nuke your tanks if they ever broke through the line of defense-in-depth opposing them EVEN IF they shrank from a total MAD strategic exchange. Enhanced radiation weapons (neutron bombs) were designed almost totally as tank (or small city, small fortification) killers -- you detonate one over a concentration of armor, the neutrons go right through the tank and both bodies in the tank (killing the latter by paralyzing them in a matter of minutes) and an hour later your tanks roar past all the more or less undamaged armor (seizing it for your own use, dumping the bodies, dead yet or not, as desired) with little danger from residual radiation. Or ditto with city-killing -- kill the people, preserve much of the infrastructure. > IMHO, a nuclear weapon is mainly a dissuasion weapon, ie, one you claim > you own to make your ennemies think twice before they strike you. Or > that you use against civilians to make your point louder, and let your > ennemies understand they'd better surrender. > > That's why I find the association between "nuclear weapon" > and "battlefield" a bit irrelevant. I'm merely reciting the doctrine that was openly avowed government policy for most of my life. Strategic nuclear arms were and are primarily connected to long range delivery systems. At one point they were megaton bombs on B-52s run by SAC (strategic air command), at one point ICBMs in silos with multiple warheads scattered all over the midwest, at one point submarine launched ballistic missiles, at one point air or sub or ship launched cruise missiles (that could swing either tactical or strategic where ballistic missiles were pretty much strategic). Mutual Assured Destruction of both military and non-military targets a long ways away on both sides. HOWEVER, at the same time MAD was in place, the east and west fought a number of wars and went through a number of crises associated with those wars. Korea, Viet Nam, Warsaw pact rebellions in Poland and Czeckoslovakia, the Cuba Missile Crisis. And the US and USSR were faced off in Europe that entire time. There was a very strong feeling that the USSR would invade Europe and try to conquer it the first moment they thought they could get away with it. There was a very strong feeling that either side, if they got to the point where they had an overwhelming strategic advantage in nukes, would be tempted to launch a first strike and incapacitate the strategic deterrence of other side, then achieve tactical victory at their leisure -- the tactical nukes made the prospect of success in the latter negligible in any war that would leave something worth invading unbombed in the strategic war. I used to study all this, seriously. I was obsessed with nuclear war from roughly when I was nine or ten until, well, I still am. I understood fission and knew how to build a Uranium bomb by the time I was ten or eleven, and knew "how" to build a plutonium bomb at about the same, although I didn't learn all the relevant details of the latter well enough to understand until I read through the NWFAQ. I recall writing a paper in fifth grade on how to build bombs, radius of total destruction, and so on. I visited Hiroshima when I was ten and went through the atom bomb museum there with the negative images of human fingers scorched onto farm tools and the tattered observatory. So I'd have to disagree. IIRC, most of the remaining nuclear bombs on the planet are TACTICAL, not strategic, or at least are mixed use -- aim at army and they are tactical, aim at a city and they are strategic, with very few "hot mounted" on strategic (long range) delivery systems. The biggest ones have been dismantled, as have many of the means of immediate long-range delivery that were the basis of the MAD strategy. I'm guessing that a large fraction of the 10,000 or so left in the US and Russia are cruise missile mounted weapons in the 10-50 KT range (where most cruise missiles have a range of less than 1000 miles, making them unsuitable for use as truly strategic weapons, although of course subs or stealth aircraft can make up the difference). The thousand or so in the possession of the rest of the nuclear club are similarly very likely to be MOSTLY tactically mounted, although most of these countries could probably remount in strategic delivery vehicles quickly enough if tensions were to mount for any reason. This is more than sUFFICIENT to guarantee that no land, sea or air based assault on the major powers or nuclear club nations (even the smaller ones) can succeed even if they DON'T resort to strategic punishment of the attacker (with serious reasons not to do the latter even if attacked). Nuclear subs and stealth bombers still give us more than ample strategic OR tactical delivery capabilities, and I'm sure Russia and China and England and France have very similar capabilities on a decreasing scale. Then there are the mini-nuclear-powers in opposition. Israel has tactical scale nukes that are ready for either strategic or tactical use. Tactical scale because fallout doesn't remain at home and the middle east is tiny -- nuke syria or iraq or iran the wrong time of year and you'll shower your own farmland with fallout if you're not careful and maybe even if you are. One ten megaton bomb on Jerusalem and there wouldn't be much Israel left -- or Jordan, or Giza, and depending on the wind I wouldn't want to be living in Egypt or Syria or Iraq. No army can invade Israel and succeed, and everybody knows it. At the MOMENT Israel can invade any of its neighbors with conventional arms and PROBABLY dominate them with relative impunity, but if/when Iran gets two or three nukes, that won't be true. Pakistan and India are a classical case. India has had (probably small) nukes for a rather long time, giving them again both a strategic and a tactical advantage over Pakistan in their eternal war (it was going on when I lived in India in the 60's and continues today). Pakistan had basically no chance of tactical victory in Kashmir or elsewhere as long as India could tactically nuke an armored invasion through the mountains, which would be slow and vulnerable for days. Pakistan has its own small nukes, and while they probably lack reliable ranged strategic delivery systems and are still heavily outnumbered and outgunned in any likely tactical conflict, it changes the balance of power considerably, in part because there are always extremists (Islamic and otherwise) in the Pakistani military who are just crazy enough to engage in first use under non-emergent circumstances. India would then proceed to crush Pakistan both strategically and tactically (so far cooler heads in Pakistan and India recognize this and if anything the new balance has them moving closer to peace, at least THIS year:-) but it is hard to envision any sort of strategic use of nukes in this theater that didn't kill millions of people. North Korea is yet another theater of interest as it "probably" has nukes. South can't invade the North with or without the US's help because China unfortunately endorses the North, giving them a strategic deterrent. Tactically, at this point an invasion of the North might be over in a matter of days even without nukes -- our weapons systems have improved that much -- just as they would have been over in Viet Nam in a matter of months if it weren't for the fact that we "couldn't" invade the north while the north could invade the south. So far, NK has no nukes of its own. If it did, well, once again you have a batshit crazy government with nuclear arms, and NK is already working hard on strategic delivery systems, not just tactical (the difference is primarily range -- tactical delivery is order of 10 to 100 miles, targeting e.g. concentrations of military forces of immediate interest, which may or may not be cities but probably not as they would be used to eliminate threats, not make points or punish, at least as long as there were immediate threats to be faced). If NK can deliver a nuke to Tokyo, though, they can say "do as we say or we'll shoot this dog" and their very insanity becomes a bluff nobody would be eager to call. It isn't clear how MANY they have -- it might be none -- but even ten small nukes become a formidable tactical weapon in a country that geographically concentrates any invading force, and with long range delivery they can make the cost of invasion (human and otherwise) very high indeed. We could, of course, nuke them into oblivion in a five minute long time on target cruise-missile based attack and almost certainly eliminate their nuclear arms and delivery systems in the process (and this is a not unlikely response to any first use or use at all on their part) but with China backing them and with world opinion something that MATTERS to us, it is almost certain we would do this short of nuclear provocation and MAYBE not even then. If China repudiated them on a first use (they have far too few weapons to be a credible strategic OR tactical threat) we could probably eliminate them on a tactical battlefield in short order IF they squandered their tactical nukes on strategic targets. To conclude todays briefing, the level of strategic nuclear risk globally continues to be quite low (as it has been for maybe 15 years now). No major conflicts appear immanent between the major powers, although Russia has been doing a bit of sabre rattling as they dream of their former "glory" as an empire and live through the slow process of building a truly stable and robust modern society. The level of nuclear risk in the Koreas is high, with a small risk of strategic escalation to engagement between the US and China (who both have STRONG reasons to avoid it). The risk of nuclear conflict between India and Pakistan is momentarily low, but chaos in the government of Pakistan continues and I am certain that there are extremist factions in the military and society in general that could at any time seize de facto control of their small nuclear arsenal or launch a destabilizing military action in Kashmir or back an assasination of Indian government officials. IMO a war there would be won by India in a matter of days, probably won by the systematic elimination of Pakistan's cities and military forces (India has over 100 nukes and both strategic and tactical delivery systems that are known to work). It would be at a huge cost -- I wouldn't be surprised at millions to tens of millions of deaths, the latter if Pakistan managed to reach any large Indian city e.g. Mumbai or New Delhi. India might forbear to use their weapons against the most densely populated cities in Pakistan if their major cities were not reached, but even limited use against "mostly tactical" targets would kill millions in that densely populated part of the world. Similarly, at the moment Irael's 100+ nukes are unopposed, making it simply incredible that they are at any sort of serious tactical risk. Nobody can invade Israel and everybody knows it. If it ever looked like they were actually succeeding, the invasion would end, permanently, a matter of hours later and it would end in such a way that civilization would have to re-emerge from the ashes of their former enemies lands before they ever again became a credible threat. This makes it actually quite stable in the region -- much posturing and a high risk of conventional miniwars, but otherwise little risk of Syria, Egypt, Iraq, Jordan, Lebanon actually mounting an attack. I can't quite predict what Iran's immanent nuclear arsenal will do in the region. On the one hand, there are plenty of wackos (as there are in Pakistan). On the other hand, there are plenty of well-educated and sane people who have absolutely no interest in launching a nuclear war over Iraq and Jordan into Israel in the certain knowledge that it would mean the end of Iran when Israel retaliated. I >>think<< that it would primarily give Iran a strategic advantage in its dealings with the surrounding ISLAMIC countries, a strong tactical advantage in the event of anything like a US-led invasion (where they could nuke our invading forces, but where politically we couldn't nuke them back). One possibility is that in five years it could actually have a positive influence in the region as a new equilibrium is reached. Another is that nukes are transferred to "terrorists". Terrorists and extremists are the major nuclear risk. In the modern world (as opposed to those whose heads are somehow stuck in the 70's or 80's) control of nuclear arms means nothing more nor less than controlling access to fissile bomb-grade material. Any country that has research-grade physics taught and performed in real universities, that has an engineering school, that has even modest manufacturing, chemical, and electronic manufacturing capabilities can build a nuclear bomb given bomb grade material Highly advanced countries like the US extend this bomb-building capability to sufficiently motivated private citizens. Furthermore, sorry, it isn't that difficult to accumulate bomb-grade fissile material or enriched Uranium capable of being cooked into Plutonium if one can get Uranium at all. Israel (and the US, Russia, and I expect China) has managed it a clever way I will not discuss that costs far less than 1940's technology with its huge centrifuges etc. I SUSPECT that they have worked out the means of producing pure Pu 239 (as opposed to 20% Pu 240 reactor grade) Plutonium. If this is true, then a shotgun design becomes possible for Uranium and building a bomb is trivial once again. Countries with Uranium deposits are all potential members of the nuclear club. I estimate the probability density of nuclear terrorism to be order of a few percent per year, making it nearly certain that at least one nuclear device will be used as a terrorist act over the next century UNLESS the conditions that could lead to it are changed. Fortunately, there isn't that much Uranium on the planet, and we're burning it at a furious rate. We could conceivably deplete the currently known reserves in a matter of decades; it is NOT a long term solution to the problem of fossil fuel use. rgb > > Other than that, pretty interesting stuff. I'm unfortunately supporting > your conclusions. > > Cheers, > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From peter.st.john at gmail.com Thu Jun 19 12:14:51 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <200806190945.21604.kilian@stanford.edu> <485A99DD.1030707@vcu.edu> Message-ID: It's off-topic, certainly. The wiki entry indicates ranges from 2 to 4 thousand meters (depending on model) and effective radius in hundreds of meters. Certainly dangerous for everyone, but so is a howitzer battery barrage. By "order" I meant nearest order of magnitude (factor of ten). The range of the weapon is around ten times as great as the effective killing radius. Peter On 6/19/08, R P Herrold wrote: > > On Thu, 19 Jun 2008, Peter St. John wrote: > > War is hell, but I wouldn't call it a "suicide device". The range of the >> rocket is on the order of 1000 meters, and the effective radius at the >> target is on the order of 100 m. >> > > the video posted indicated 4 kliks range, and a greater lethal radius ... > > but how is this on topic here? > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080619/034a176a/attachment-0001.html From rgb at phy.duke.edu Thu Jun 19 10:26:49 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <87k5glfi7m.fsf@snark.cb.piermont.com> References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <87iqw6i7ck.fsf@snark.cb.piermont.com> <6D3FFF5E-20CA-4AFC-88B2-18AA74C7EB28@xs4all.nl> <87k5glfi7m.fsf@snark.cb.piermont.com> Message-ID: On Thu, 19 Jun 2008, Perry E. Metzger wrote: > I can log in using your credentials if I have your private key and you > are using SSH with public key authentication. However, even if I have > both of your private and public keys, the ephemeral key used for a > particular session is agreed to using Diffie-Hellman key exchange, and > mere knowledge of your long term keys will not allow anyone to read > your session traffic. This property is known as "Perfect Forward > Secrecy." (Technically, this is only true of sshv2 -- sshv1 used > random nonces exchanged under RSA for the key material, but sshv1 > is no longer in wide use because it has a number of security issues.) They do enable man in the middle attacks, however, so that while your connection cannot be snooped "passively", somebody in the middle (say, in possession of any intermediary router) can pretend to be both sides by establishing simulations of the connections requested and forwarding the traffic. Similarly, if somebody has both my public and private keys they very likely can get into my system and insert trojans into it and directly snoop everything I do, access kmem, own my view of the universe through completely bugged network and peripheral eyes. Encryption is never any better than the physical and network and systems security on which it is implemented, as it is a weak-link problem. But otherwise sure. Similar things for WPA vs WEP as I recall -- WEP doesn't change the ephemeral keys. But encryption is more a hobby associated with my interest in information theory and random numbers than a speciality. I didn't realize that they'd made 1024 bit keys vulnerable at this point. I'm guessing that "vulnerable" still means "vulnerable to people with obscene amounts of free computer time and not enough to do" as opposed to "vulnerable" as in airsnort makes WEP vulnerable to pimply faced kids with old laptops, but still, worth knowing, thanks! rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From landman at scalableinformatics.com Thu Jun 19 12:39:43 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] found this amusing Message-ID: <485AB5FF.70705@scalableinformatics.com> Not wholly OT ... Running some benchmarks for a customer, and doing a little baseline performance gathering. This is Windows 2008 RC2 32 bit on one of our JackRabbit units. Well, it appears that there is either a bug in the performance counter, or the JackRabbit is simply too fast ... Have a look at http://scalability.org/images/Screenshot-small-1.png . The other measures seem to be working properly. Average is ~1.35 GB (writes) but it bounces around a bit. Looks like it overflows at 2^31-1 or so ... So we hit these bursty regions we get into that ... negative ... performance :) Either that or we are doing something horribly wrong (won't discount that either). At least it is amusing. Back to your regularly scheduled cluster. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From perry at piermont.com Thu Jun 19 12:43:19 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: (Robert G. Brown's message of "Thu\, 19 Jun 2008 13\:26\:49 -0400 \(EDT\)") References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <87iqw6i7ck.fsf@snark.cb.piermont.com> <6D3FFF5E-20CA-4AFC-88B2-18AA74C7EB28@xs4all.nl> <87k5glfi7m.fsf@snark.cb.piermont.com> Message-ID: <87skv9dxd4.fsf@snark.cb.piermont.com> "Robert G. Brown" writes: >> I can log in using your credentials if I have your private key and you >> are using SSH with public key authentication. However, even if I have >> both of your private and public keys, the ephemeral key used for a >> particular session is agreed to using Diffie-Hellman key exchange, and >> mere knowledge of your long term keys will not allow anyone to read >> your session traffic. This property is known as "Perfect Forward >> Secrecy." (Technically, this is only true of sshv2 -- sshv1 used >> random nonces exchanged under RSA for the key material, but sshv1 >> is no longer in wide use because it has a number of security issues.) > > They do enable man in the middle attacks, however, so that while your > connection cannot be snooped "passively", somebody in the middle (say, > in possession of any intermediary router) can pretend to be both sides > by establishing simulations of the connections requested and forwarding > the traffic. Not quite. If they only have your key, but not the remote host's key, they can pretend to be you to the remote host, but they can't pretend to be the remote host to you. (Similarly if they've stolen the host's key but not yours.) > Similarly, if somebody has both my public and private keys they very > likely can get into my system I said that. See "I can log in using your credentials", above. > But otherwise sure. Similar things for WPA vs WEP as I recall -- WEP > doesn't change the ephemeral keys. The latter is true (in that WEP has no such mechanisms at all), but WPA in pre shared key mode doesn't change the base keys either, and the TKIP keys are derived from it, so there is no perfect forward secrecy. In "enterprise" mode, a shared key is created for each user, but I'm not sure that all modes of the protocol end up providing perfect forward secrecy. > I didn't realize that they'd made 1024 bit keys vulnerable at this > point. I'm guessing that "vulnerable" still means "vulnerable to > people with obscene amounts of free computer time and not enough to > do" I think the NSA types think of this sort of activity as quite justified, as do their equivalents in other nations and in the, er, "private sector". In particular, if you spend any of your time working for banks, the issue is not ignorable because if money is at stake, people will spend money to get at it. If you are mostly concerned about your family members reading your email, the situation is quite different. That said, typing "2048" instead of "1024" in to PGP or OpenSSL is no more expensive, so there is no point in using shorter keys even if you have little to worry about. > as opposed to "vulnerable" as in airsnort makes WEP vulnerable to > pimply faced kids with old laptops, but still, worth knowing, > thanks! Perry -- Perry E. Metzger perry@piermont.com From rgb at phy.duke.edu Thu Jun 19 10:51:15 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <6.2.5.6.2.20080619105301.02cc8fd8@jpl.nasa.gov> References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <6.2.5.6.2.20080619105301.02cc8fd8@jpl.nasa.gov> Message-ID: On Thu, 19 Jun 2008, Jim Lux wrote: > Uh.. probably not.. That's the "alarm clock" or "layer cake" design and has > some issues. > I refer interested readers to Morland's 1977 article in "The Progressive" for > more information. Fusion boosting of fission, OTOH, is relatively easy to accomplish and a common practice, and recall that the yield range I was speaking of was LESS than a MT, which is the upper bound of layer cakes in any event. But my point was that throwing in a bunch of deuterium and/or tritium is almost certainly going to increase your yield, moreso for a "bad design" (near fizzle) than for a really good implosion device. The extra neutrons basically fission a lot of stuff that would otherwise be blown apart without fissioning, and you do get SOME increased yield from fusion, although it may well be small unless you do the "good" designs. For strategic weapons, this matters. For tactical weapons, it totally matters. For terrorist-class weapons or homemade weapons, getting a "good fusion design" doesn't matter, because boosting a KT to 5 KT or 10 KT might as well be infinity -- a blast capable of causing kilodeaths and gigabuck damages is more important than details of the yield. > The study done a couple years ago that postulated a "nominal yield"(e.g. > 20kT) fission device in Los Angeles/Long Beach port showed that the majority > of damage and death resulted from things like traffic jams and accidents in > the crowds fleeing and overloading hospitals. The actual damage radius is > fairly small (in the context of Los Angeles, which is >100km across) and the > fallout (from a particularly dirty surface burst.. i.e. the shipping > container on the dock sucking up the dirt) wasn't all that bad, in terms of > dose. The panic, on the other hand, is lethal. Sure, maybe. OTOH Hiroshima with a 20KT device killed order of 10^5 people. 20KT in a dense population of any sort surrounded by wood frame dwellings that can play in a fire storm might not make a megadeath, but it would dwarf 9/11. > Hmm. Mcrit for good quality U235 is fairly high, especially unreflected. I > don't know if your shotgun has sufficient oomph to assemble it quickly enough > without predetonation or a fizzle. I've seen numbers for required assy speed > in the 1000 m/sec sort of range (i.e. you've got to move from a noncrit to a > crit configuration in the amount of time between spontaneous neutrons > appearing to get things rolling). Accelerating 10kg, say, to 1000 m/sec, > takes a heap o'joules I can only quote Oppenheimer, that if you drop a subcrit piece of U235 off of a table to assemble with one on the floor you are likely to get a creditable nuclear blast. I don't know what he meant by 'creditable', but I'm assuming KT range, about the same as the davy crockett (nice movie:-), and that even if he was exaggerating about the table, a shotgun is better than a table;-) But sure, real explosives would be better, a real gun barrel would be useful, and both are readily available. I would argue that a U235 device is within "garage" assembly either way. "Pure" Pu-239 gun designs are likely to be a lot trickier, but still fairly accessible. It's getting rid fo the Pu-240 that is the problem -- it puts you right back in the isotope separation game only worse, as the mass difference is even smaller. > why worry about ICBMs when DHL/FedEx will deliver it to your selected > doorstep? Well, even a small bomb would be pretty heavy. Kind of at the boundary of what FedEx will delivery;-) rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From perry at piermont.com Thu Jun 19 12:53:37 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <6.2.5.6.2.20080619105301.02cc8fd8@jpl.nasa.gov> (Jim Lux's message of "Thu\, 19 Jun 2008 11\:08\:48 -0700") References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <6.2.5.6.2.20080619105301.02cc8fd8@jpl.nasa.gov> Message-ID: <87mylhdwvy.fsf@snark.cb.piermont.com> Jim Lux writes: >>Thermonuclear fusion and the 100+ KT to MT range are similarly >>straightforward. From what I recall, one can just monkey around with >>building bigger bombs surrounded by more fissile material and get close >>to the latter, adding fusile material such as tritium (expensive and >>dangerous) or deuterium (plentiful and harmless) and lithium into spaces >>between trigger and a U-238 casing. > > Uh.. probably not.. That's the "alarm clock" or "layer cake" design > and has some issues. I refer interested readers to Morland's 1977 > article in "The Progressive" for more information. "Dark Sun" by Richard Rhodes is probably better (and more entertaining). It describes the Teller-Ulam mechanism in some detail. I will point out that, if the description of the initial Ivy Mike test is to be taken as an indication, it actually requires quite a bit of effort to put together such a thing -- one need worry about amateurs building fission devices, but probably no fusion... Perry -- Perry E. Metzger perry@piermont.com From perry at piermont.com Thu Jun 19 12:55:48 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] SuperMicro and lm_sensors In-Reply-To: <20080619190801.GD4491@bx9.net> (Greg Lindahl's message of "Thu\, 19 Jun 2008 12\:08\:02 -0700") References: <20080619190801.GD4491@bx9.net> Message-ID: <87iqw5dwsb.fsf@snark.cb.piermont.com> Greg Lindahl writes: > On Thu, Jun 19, 2008 at 10:50:25PM +0400, Mikhail Kuzminsky wrote: > >> AFAIK it depends from BIOS. On my Tyan S2932 w/last BIOS version this >> directory is empty. > > On the Opteron box that I got lm_sensors working on, the k8temp module > was required for some of the temperatures. But that directory is > empty. "hit or miss" comes to mind... And what does this have to do with nuclear weapons? You would think this was an HPC mailing list or something... Perry From rgb at phy.duke.edu Thu Jun 19 10:59:06 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <87mylhdwvy.fsf@snark.cb.piermont.com> References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <6.2.5.6.2.20080619105301.02cc8fd8@jpl.nasa.gov> <87mylhdwvy.fsf@snark.cb.piermont.com> Message-ID: On Thu, 19 Jun 2008, Perry E. Metzger wrote: > "Dark Sun" by Richard Rhodes is probably better (and more > entertaining). It describes the Teller-Ulam mechanism in some > detail. I will point out that, if the description of the initial Ivy > Mike test is to be taken as an indication, it actually requires quite > a bit of effort to put together such a thing -- one need worry about > amateurs building fission devices, but probably no fusion... Not "good" as in high yield fusion, agreed. I was really just pointing out that fusion or neutron enhancement in the less than MT range is still quite accessible. rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From jcownie at cantab.net Thu Jun 19 13:02:37 2008 From: jcownie at cantab.net (James Cownie) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <87skv9dxd4.fsf@snark.cb.piermont.com> References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <87iqw6i7ck.fsf@snark.cb.piermont.com> <6D3FFF5E-20CA-4AFC-88B2-18AA74C7EB28@xs4all.nl> <87k5glfi7m.fsf@snark.cb.piermont.com> <87skv9dxd4.fsf@snark.cb.piermont.com> Message-ID: Returning the thread slightly nearer its original topic, I'm surprised that no one mentioned this... http://www.engadget.com/2008/05/21/open-source-ogd1-graphics-card-up-for-pre-order/ as a good place for hobbyists who want to play with FPGAs for calculation. -- -- Jim -- James Cownie -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080619/ea709304/attachment.html From peter.st.john at gmail.com Thu Jun 19 13:03:33 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <200806190945.21604.kilian@stanford.edu> Message-ID: Regarding: [RGB] Fortunately, there isn't that much Uranium on the planet, and we're burning it at a furious rate. We could conceivably deplete the currently known reserves in a matter of decades; it is NOT a long term solution to the problem of fossil fuel use. rgb Isn't there a scheme for creating fissionable material from more common stuff (hafnium?) by bombarding it with neutrons excited out of their nuclei by laster pulses? So one might make bombs without U235? (See e.g. http://en.wikipedia.org/wiki/Hafnium#Applications) Peter P.S. Geothermal is a long term solution, maybe; see http://geothermal.inel.gov/publications/future_of_geothermal_energy.pdf but I'm still hoping for practicable fusion. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080619/5016ab2c/attachment.html From James.P.Lux at jpl.nasa.gov Thu Jun 19 13:45:40 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <87iqw6i7ck.fsf@snark.cb.piermont.com> <6D3FFF5E-20CA-4AFC-88B2-18AA74C7EB28@xs4all.nl> <87k5glfi7m.fsf@snark.cb.piermont.com> <87skv9dxd4.fsf@snark.cb.piermont.com> Message-ID: <6.2.5.6.2.20080619133800.02ccc488@jpl.nasa.gov> At 01:02 PM 6/19/2008, James Cownie wrote: >Returning the thread slightly nearer its original topic, I'm >surprised that no one mentioned this... > >http://www.engadget.com/2008/05/21/open-source-ogd1-graphics-card-up-for-pre-order/ > >as a good place for hobbyists who want to play with FPGAs for calculation. > >-- Indeed.. There's also a variety of eval cards from the manufacturers in this sort of around a kilobuck price range. Xilinx has a Spartan-3 PCIe with the XC3S1000 part (1/4 size of the one on the card above) for $350 From john.hearns at streamline-computing.com Thu Jun 19 14:22:02 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <20080618214603.GC31594@bx9.net> <200806181521.47115.kilian@stanford.edu> <48599AC9.3040100@berkeley.edu> <1213859837.4784.9.camel@Vigor13> Message-ID: <1213910532.7651.4.camel@Vigor13> On Thu, 2008-06-19 at 12:52 -0400, Peter St. John wrote: > I dug up this pdf from Nvidia: > http://www.nvidia.com/docs/IO/43395/tesla_product_overview_dec.pdf > Since I can't imagine coding a graphics card while it serves my X :-) > I supposed one might put the PCIE card in a box with a cheap SVGA for > the least-cost CUDA experiment One thing I've never understood, and hopefully someone on here can explain clearly, is why the onboard graphics is normally disabled when you add a PCI-e card. If this was an option in the BIOS it would be useful in this case (my own workstation is a Sun Ultra 20). From dnlombar at ichips.intel.com Thu Jun 19 14:23:04 2008 From: dnlombar at ichips.intel.com (Lombard, David N) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] SuperMicro and lm_sensors In-Reply-To: References: <20080618221153.GA24788@bx9.net> <20080619011419.GA7040@bx9.net> <20080619135020.GA8918@nlxdcldnl2.cl.intel.com> Message-ID: <20080619212304.GB8918@nlxdcldnl2.cl.intel.com> On Thu, Jun 19, 2008 at 11:28:08AM -0700, Bernard Li wrote: > Hi David: Hey Bernard! > On Thu, Jun 19, 2008 at 6:50 AM, Lombard, David N > wrote: > > > Did you look for /proc/acpi/thermal_zone/*/temperature The glob is for > > your BIOS-defined ID. If it does exist, that's the value that drives > > /proc/acpi/thermal_zone/*/trip_points > > > > See also /proc/acpi/thermal_zone/*/polling_frequency > > I have always wondered about /proc/acpi/thermal_zone. I noticed that > on some servers, the files exist, but on others, that directory is > empty. I guess this is dependent on whether the BIOS exposes the > information to the kernel? Or are there modules that I need to > install to get it working? Yes, a specific config is needed; but, a given kernel that sees it on one system, it will see it all systems IFF the BIOS sets it up. For any distro, it's a BIOS issue. -- David N. Lombard, Intel, Irvine, CA I do not speak for Intel Corporation; all comments are strictly my own. From john.hearns at streamline-computing.com Thu Jun 19 14:38:59 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <20080618224523.GA18052@unthought.net> <5819D187-01A1-4623-9839-3D2771ADAF3F@xs4all.nl> <6.2.5.6.2.20080618164843.02b1bd30@jpl.nasa.gov> <6.2.5.6.2.20080619105301.02cc8fd8@jpl.nasa.gov> Message-ID: <1213911549.7651.10.camel@Vigor13> On Thu, 2008-06-19 at 13:51 -0400, Robert G. Brown wrote: > > > why worry about ICBMs when DHL/FedEx will deliver it to your selected > > doorstep? > > Well, even a small bomb would be pretty heavy. Kind of at the boundary > of what FedEx will delivery;-) Recall that the first British nuclear test was a device on board a ship, the frigate HMS Plym. This reflected fears that the British had of a nuclear weapon being smuggled into a harbour aboard a ship. The British, being a strong naval power at the time, were also of course interested in the effects of nuclear weapons on ships. ps. any chance of hauling this discussion back on track by mentioning Roadrunner? Does anyone know if there are projected ASCI type machines using Nvidia cards? I'd very much suppose that if anyone DOES have an answer to that one then a cell is waiting in Leavenworth, or the Tower. From kilian at stanford.edu Thu Jun 19 14:39:42 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <1213910532.7651.4.camel@Vigor13> References: <485920D8.2030309@ias.edu> <1213910532.7651.4.camel@Vigor13> Message-ID: <200806191439.43134.kilian@stanford.edu> On Thursday 19 June 2008 02:22:02 pm John Hearns wrote: > One thing I've never understood, and hopefully someone on here can > explain clearly, is why the onboard graphics is normally disabled > when you add a PCI-e card. It probably comes from the times when dual-head configurations were still bleeding-edge, and when BIOS manufacturers assumed that if you had a AGP/PCIe graphics card on your bus, you wanted to use it, rather than the usually cheapo onboard chip. Nowadays, NVIDIA's Hybrid-SLI [1] precisely aims to aggregate graphical power from the onboard chip and add-in card(s) in heterogenous configurations. So hopefully this trend will come to an end. At least on NV mobos. [1]http://www.nvidia.com/object/hybrid_sli.html Cheers, -- Kilian From csamuel at vpac.org Thu Jun 19 16:32:11 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <447096711.34491213918066691.JavaMail.root@zimbra.vpac.org> Message-ID: <1381552971.34571213918331904.JavaMail.root@zimbra.vpac.org> ----- "Kilian CAVALOTTI" wrote: > AFAIK, the multi GPU Tesla boxes contain up to 4 Tesla processors, but > are hooked to the controlling server with only 1 PCIe link, right? > Does this spell like "bottleneck" to anyone? The nVidia website says: http://www.nvidia.com/object/tesla_tech_specs.html # 6 GB of system memory (1.5 GB dedicated memory per GPU) [...] # Connects to host via cabling to a low power PCI # Express x8 or x16 adapter card So my guess is that you'd be using local RAM not the host systems RAM whilst computing. I took a photo of an open Tesla box at SC'07: http://flickr.com/photos/chrissamuel/2267613381/in/set-72157603919719911/ (click on "All sizes" for a larger version), my guess is that the DIMMS are hidden under the shrouds. There's a lot of fans there.. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From csamuel at vpac.org Thu Jun 19 16:41:31 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <980339235.34611213918666184.JavaMail.root@zimbra.vpac.org> Message-ID: <1169577922.34631213918891004.JavaMail.root@zimbra.vpac.org> ----- "Robert G. Brown" wrote: > IIRC almost any of the high-end encryption routines available within > linux are effectively uncrackable, certainly uncrackable to somebody > with less than NSA-class resources. As long as the implementation is correct.. Debian SSL. :-) Humans are always the weak links in these things, whether that be implementation, crypto security or just doing plain dumb things like sending an email confirmation in the clear containing plain text passwords that were submitted over SSL. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From landman at scalableinformatics.com Thu Jun 19 17:08:43 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <1169577922.34631213918891004.JavaMail.root@zimbra.vpac.org> References: <1169577922.34631213918891004.JavaMail.root@zimbra.vpac.org> Message-ID: <485AF50B.2080807@scalableinformatics.com> Chris Samuel wrote: > ----- "Robert G. Brown" wrote: > >> IIRC almost any of the high-end encryption routines available within >> linux are effectively uncrackable, certainly uncrackable to somebody >> with less than NSA-class resources. > > As long as the implementation is correct.. Debian SSL. :-) N-tro-PEE? We dont need no steen-keen N-tro-PEE! Get yer fresh hot bits here, all 15 of them. > Humans are always the weak links in these things, > whether that be implementation, crypto security or > just doing plain dumb things like sending an email > confirmation in the clear containing plain text > passwords that were submitted over SSL. People spend lots of time and effort on security theater. Make up odd rules for passwords. Make them hard to guess and crack. Well, is that the vector for break-ins? Weak passwords? I saw a linux machine (a cluster) rooted. It was rooted because of a person with a windows laptop that happened to catch a key logger. Crackers had been attempting to break in to that machine for a long time, and here goes a grad student, and gives them the password. Worse, this grad student acted in a way we advised against, and ran jobs from root. Yeah, I know. Security theater is troubling. It gives us sheep the appearance of being secure, without any real additional value. Opie and multi-factor are hard to beat. And no theater needed. Even better, no worries about replay attacks with opie, or with a multi-factor that disables a password upon use. But even with these, you still need good *real* practices. A non-security theater practice would limit the damage one can do in a non-privileged setting. SElinux and Apparmor try to limit the damage even in a secure setting, though I am not sure how well they do there. Joe -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From kilian at stanford.edu Thu Jun 19 17:16:41 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <1381552971.34571213918331904.JavaMail.root@zimbra.vpac.org> References: <1381552971.34571213918331904.JavaMail.root@zimbra.vpac.org> Message-ID: <200806191716.41743.kilian@stanford.edu> On Thursday 19 June 2008 04:32:11 pm Chris Samuel wrote: > ----- "Kilian CAVALOTTI" wrote: > > AFAIK, the multi GPU Tesla boxes contain up to 4 Tesla processors, > > but are hooked to the controlling server with only 1 PCIe link, > > right? Does this spell like "bottleneck" to anyone? > > The nVidia website says: > > http://www.nvidia.com/object/tesla_tech_specs.html > > # 6 GB of system memory (1.5 GB dedicated memory per GPU) The latest S1070 has even more than that: 4GB per GPU as it seems, according to [1]. But I think this refers to the "global memory", as decribed in [1] (slide 12, "Kernel Memory Access"). It's the graphics card main memory, the kind of one which is used to store textures in games, for instance. Each GPU core also has what they call "shared memory" and which is really only shared between threads on the same core (it's more like a L2 cache actually). > So my guess is that you'd be using local RAM not the > host systems RAM whilst computing. Right, but at some point, you do need to transfer data from the host memory to the GPU memory, and back. That's where there's probably a bottleneck if all 4 GPUs want to read/dump data from/to the host at the same time. Moreover, I don't think that the different GPUs can work together, ie. exchange data and participate to the same parallel computation. Unless they release something along the lines of a CUDA-MPI, those 4 GPUs sitting in the box would have to be considered as independent processing units. So as I understand it, the scaling benefits from your application's parallelization would be limited to one GPU, no matter how many you got hooked to your machine. I don't even know how you choose (or even if you can choose) on which GPU you want your code to be executed. It has to be handled by the driver on the host machine somehow. > There's a lot of fans there.. They probably get hot. At least the G80 do. They say "Typical Power Consumption: 700W" for the 4 GPUs box. Given that a modern gaming rig featuring a pair of 8800GTX in SLI already requires a 1kW PSU, I would put this on the optimistic side. [1]http://www.nvidia.com/object/tesla_s1070.html [2]http://www.mathematik.uni-dortmund.de/~goeddeke/arcs2008/C1_CUDA.pdf Cheers, -- Kilian From kilian at stanford.edu Thu Jun 19 17:25:22 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485AF50B.2080807@scalableinformatics.com> References: <1169577922.34631213918891004.JavaMail.root@zimbra.vpac.org> <485AF50B.2080807@scalableinformatics.com> Message-ID: <200806191725.23005.kilian@stanford.edu> On Thursday 19 June 2008 05:08:43 pm Joe Landman wrote: > SElinux and Apparmor try to limit the damage > even in a secure setting, though I am not sure how well they do > there. If you want/need to use things like Lustre, for instance, you can forgot about SELinux and AppArmor, it simply doesn't work. Isn't it a common practice in HPC to keep security rules relatively relaxed *inside* a cluster (passwordless logins between compute nodes for instance), whilst trying to harden the links to the external world? I mean, most of the scientific applications haven't precisely been designed with security as their first concern, have they? Cheers -- Kilian From landman at scalableinformatics.com Thu Jun 19 17:33:40 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <200806191725.23005.kilian@stanford.edu> References: <1169577922.34631213918891004.JavaMail.root@zimbra.vpac.org> <485AF50B.2080807@scalableinformatics.com> <200806191725.23005.kilian@stanford.edu> Message-ID: <485AFAE4.1050705@scalableinformatics.com> Kilian CAVALOTTI wrote: > On Thursday 19 June 2008 05:08:43 pm Joe Landman wrote: >> SElinux and Apparmor try to limit the damage >> even in a secure setting, though I am not sure how well they do >> there. > > If you want/need to use things like Lustre, for instance, you can forgot > about SELinux and AppArmor, it simply doesn't work. In the Lustre release notes: "Do not laugh in the presence of Lustre, for it is subtle and quick to anger". (for those who aren't sure, this is an attempt at humor ... no need to skewer me over "anti-Lustre" comments ...) > > Isn't it a common practice in HPC to keep security rules relatively > relaxed *inside* a cluster (passwordless logins between compute nodes > for instance), whilst trying to harden the links to the external world? Yes. Leave the window wide open while bolting the door tight ... :( > > I mean, most of the scientific applications haven't precisely been > designed with security as their first concern, have they? I would be happy if they had better code than if (!(fdopen( ... )) { printf "I died\n"; exit(-1); } > > Cheers -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From csamuel at vpac.org Thu Jun 19 17:43:59 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485AF50B.2080807@scalableinformatics.com> Message-ID: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> ----- "Joe Landman" wrote: > People spend lots of time and effort on security theater. Make up odd > rules for passwords. Make them hard to guess and crack. Well, is > that the vector for break-ins? Weak passwords? Yeah - sadly.. :-( -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From csamuel at vpac.org Thu Jun 19 17:52:57 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: Message-ID: <1193904974.35781213923177281.JavaMail.root@zimbra.vpac.org> ----- "Peter St. John" wrote: > War is hell, but I wouldn't call it a "suicide device". http://www.damninteresting.com/?p=783 # The Davy Crockett's timer allowed a minimum shot # distance of about 1,000 feet, but such inept use # of the weapon would certainly result in the deaths # of the firing team. In most cases, the approaching # Soviets would be at least one mile away, leaving the # Atomic Battle Group personnel outside of the hazard # zone. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From rgb at phy.duke.edu Thu Jun 19 19:33:47 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> Message-ID: On Fri, 20 Jun 2008, Chris Samuel wrote: > > ----- "Joe Landman" wrote: > >> People spend lots of time and effort on security theater. Make up odd >> rules for passwords. Make them hard to guess and crack. Well, is >> that the vector for break-ins? Weak passwords? > > Yeah - sadly.. :-( Do you have an recent contemporary evidence for that? I mean, back in the 80's and 90's, when I could use ypx to grab anybody's encrypted password files and run crack on them and get a dozen hits in a few hours of work, sure, but since MD5 became near-universal and since /etc/shadow was invented and since they fixed the worst of the holes that let "anybody" get at the encrypted password list, since password changing programs no longer let you use a REALLY bad password (or at least bitch about it if they do), since sysadmins started routinely running crack on the encrypted list defensively and forcing the change of particularly weak ones, since most systems can beconfigured with tools that bitch or slow down or flag repeated brute force attacks, I'd have thought that wasn't so true anymore. We run log scanners that count the attacks on our systems in a 24 hour period and break them down by e.g. originating IP number and so on, and truth be told they are nearly continuous, but I haven't heard of any of those attacks SUCCEEDING on any linux box run by any non-complete-idiot for years now. Password TRAPS are a pretty common vector; the only cases I tend to hear of at all commonly anymore for crackings (of linux boxes, not Windows systems that are cracked or infected almost at will) tend to be somebody who goes home for the summer, uses an infected, trojanned, vile spewpot of a Windows box to login back at duke from home via e.g. putty or some other related interface, and has their keystrokes logged as they do. Quite a lot of the Windows viruses install trojan spyware that does full keystroke logging and so on; I got to watch one attempt this on one of my kids boxes when it was infected, and had to change one of my passwords after cleaning it up because (sigh) I had to use it to get Duke to get the site license software I needed to do the cleaning. There are also still -- relatively rarely -- buffer overwrite attacks discovered. Most coders "get it" that one shalt not use the non-n string commands to manipulate buffers these days, although there is still legacy code in existence (I'm sure) that has it. I personally last got nailed by the slammer attack, because I got lazy about updates (this was barely pre-yum) and didn't patch my web software in time. Kernel bugs, and MAYBE a rare race condition, still sometimes allow promotion to root. But weak passwords that are brute force guessed or cracked from the shadow file? Only on a poorly managed network, one where the sysadmin doesn't bother to check and fails to inform the users of how to choose a good one, AND where users manage to gain access to the shadow file in the first place. rgb (of course MY passwd is just rgbbgr -- that's secure enough don't you think...;-) -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From csamuel at vpac.org Thu Jun 19 20:15:31 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: Message-ID: <1277978866.37731213931731379.JavaMail.root@zimbra.vpac.org> ----- "Robert G. Brown" wrote: > Do you have an recent contemporary evidence for that? Not since we moved to LDAP, but a few years back the cluster that I inherited (and that was configured by a large vendor who shall remain nameless) was still running vanilla YP. Although we ran (and still do run) regular brute force attacks against the hundreds of users we have there was still a window of opportunity between a new user setting a dumb password and us breaking it and locking the account. It would have been great if we could have enforced good passwords through cracklib, but from what I remember yppasswd didn't appear to want to play at that time (RH7.3). My memory also tells me that the logs at the time showed people brute forcing their account prior to gaining access, but I have a fairly high bit error rate so please apply 2D6 pinches of salt. cheers! Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From perry at piermont.com Thu Jun 19 21:03:04 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485AFAE4.1050705@scalableinformatics.com> (Joe Landman's message of "Thu\, 19 Jun 2008 20\:33\:40 -0400") References: <1169577922.34631213918891004.JavaMail.root@zimbra.vpac.org> <485AF50B.2080807@scalableinformatics.com> <200806191725.23005.kilian@stanford.edu> <485AFAE4.1050705@scalableinformatics.com> Message-ID: <87wskkwy6f.fsf@snark.cb.piermont.com> Joe Landman writes: >> Isn't it a common practice in HPC to keep security rules relatively >> relaxed *inside* a cluster (passwordless logins between compute >> nodes for instance), whilst trying to harden the links to the >> external world? > > Yes. Leave the window wide open while bolting the door tight ... :( I don't think that's "leaving the window open" since there is no window. If the only way to get to the cluster network is through a head node that is actually secure, then it is just fine to treat the cluster network as if it were the backplane of a single box and leave the cluster nodes fairly open. The head node can dispatch arbitrary jobs to all the boxes anyway, and once you've got code running on a box you can often break root anyway. One has to be reasonable about these things. Perry -- Perry E. Metzger perry@piermont.com From perry at piermont.com Thu Jun 19 21:15:39 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: (Robert G. Brown's message of "Thu\, 19 Jun 2008 22\:33\:47 -0400 \(EDT\)") References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> Message-ID: <87skv8wxlg.fsf@snark.cb.piermont.com> "Robert G. Brown" writes: > On Fri, 20 Jun 2008, Chris Samuel wrote: >> ----- "Joe Landman" wrote: >> >>> People spend lots of time and effort on security theater. Make up odd >>> rules for passwords. Make them hard to guess and crack. Well, is >>> that the vector for break-ins? Weak passwords? >> >> Yeah - sadly.. :-( > > Do you have an recent contemporary evidence for that? Yes, Run a box with sshd on it connected to the internet and watch your logs for a few days. You will find numerous attempts to try thousands of possible account names and passwords -- brute force cracking. Here is an extract from the log on a real machine, one of mine, from last night: Jun 19 20:56:53 smaug sshd[2577]: Invalid user secretariat from 70.90.14.154 Jun 19 20:56:54 smaug sshd[2522]: Invalid user secretar from 70.90.14.154 Jun 19 20:56:55 smaug sshd[23949]: Invalid user present from 70.90.14.154 Jun 19 20:56:56 smaug sshd[3440]: Invalid user test from 70.90.14.154 Jun 19 20:56:57 smaug sshd[8809]: Invalid user test from 70.90.14.154 Jun 19 20:56:58 smaug sshd[21600]: Invalid user teste from 70.90.14.154 Jun 19 20:56:59 smaug sshd[314]: Invalid user teste from 70.90.14.154 It goes on and on and on. There are countermeasures you can run to block the zombies trying to guess passwords, but I rarely bother since none of my machines allow password based login so their attempts are useless anyway. These attacks are done by automated malware that spreads itself around from machine to machine for nefarious purposes -- good luck trying to track down the puppet masters. I've tracked the bad guys down a few times but they're always somewhere like Bucharest anyway, and the locals don't care to arrest them. It is true that this is only one of many modern attack vectors and that buffer overflows, drive by malware downloads into browsers, etc., are all far more common ways in, but you will indeed get hacked by automated systems if you leave an sshd on and have accounts with weak passwords. > There are also still -- relatively rarely -- buffer overwrite attacks > discovered. Rarely? You haven't been reading full-disclosure lately I see. There are a half dozen new such vulns found a day. > Most coders "get it" No, most of them don't. I've done a lot of code audit in my day. The average C programmer turned out these days really thinks you use system("rm filename") to unlink a file, and that's the good part of their code. For a laugh, google for the "daily wtf" and start reading some of the stuff you see. > But weak passwords that are brute force guessed[...]? > Only on a poorly managed network, That would be 95% of networks. I've done a lot of network audits in my day, too. -- Perry E. Metzger perry@piermont.com From rgb at phy.duke.edu Thu Jun 19 23:41:42 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <87skv8wxlg.fsf@snark.cb.piermont.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> Message-ID: On Fri, 20 Jun 2008, Perry E. Metzger wrote: > > "Robert G. Brown" writes: >> On Fri, 20 Jun 2008, Chris Samuel wrote: >>> ----- "Joe Landman" wrote: >>> >>>> People spend lots of time and effort on security theater. Make up odd >>>> rules for passwords. Make them hard to guess and crack. Well, is >>>> that the vector for break-ins? Weak passwords? >>> >>> Yeah - sadly.. :-( >> >> Do you have an recent contemporary evidence for that? > > Yes, Run a box with sshd on it connected to the internet and watch your > logs for a few days. You will find numerous attempts to try thousands > of possible account names and passwords -- brute force cracking. Well, yeah, sure, I know about that as I DO watch my logs -- I just haven't heard of one of these attacks SUCCEEDING in pretty much forever, for obvious reasons. These attacks run through a list of "likely user names" (including well known system names like "root") and then run through a short list of "likely passwords" such as user=test password=test. Short because ssh enforces a roughly 1 second interval between tries, which limits the total number of attempts that can be made to < 100,000/day, with a lot less actual informational volume to the attempted space. In fact, to me it looks like most of the attacks are generated by more or less the same code and probably reuse most of the same names and password guesses so that the actual full repertoire tested is really pretty miniscule, even compared to a dictionary attack. To hit they have to a) match a username; and b) determine the password that goes with it. On a site that runs any sort of a password checker and avoids the dumbest of dumb passwords (almost by definition, the contents of their list:-) their probability of success is minute. They only pluck the very lowest of the low hanging fruit, fruit that is so low it is basically rotting in the gutter attracting flies. > Here is an extract from the log on a real machine, one of mine, from > last night: > > Jun 19 20:56:53 smaug sshd[2577]: Invalid user secretariat from 70.90.14.154 > Jun 19 20:56:54 smaug sshd[2522]: Invalid user secretar from 70.90.14.154 > Jun 19 20:56:55 smaug sshd[23949]: Invalid user present from 70.90.14.154 > Jun 19 20:56:56 smaug sshd[3440]: Invalid user test from 70.90.14.154 > Jun 19 20:56:57 smaug sshd[8809]: Invalid user test from 70.90.14.154 > Jun 19 20:56:58 smaug sshd[21600]: Invalid user teste from 70.90.14.154 > Jun 19 20:56:59 smaug sshd[314]: Invalid user teste from 70.90.14.154 Sure, it goes on and on. I don't really LIKE seeing this, especially on a server with sensitive information, but that is precisely why one configures such servers with tight controls and runs a password checker. > These attacks are done by automated malware that spreads itself around > from machine to machine for nefarious purposes -- good luck trying to > track down the puppet masters. I've tracked the bad guys down a few > times but they're always somewhere like Bucharest anyway, and the > locals don't care to arrest them. Agreed. All part of the Russian Diabolical Supercomputer we discussed six months ago or so -- a cloud of unknown proportions and evil intent:-) > It is true that this is only one of many modern attack vectors and > that buffer overflows, drive by malware downloads into browsers, etc., > are all far more common ways in, but you will indeed get hacked by > automated systems if you leave an sshd on and have accounts with weak > passwords. Agreed as well. Which is why professionals generally check for weak passwords (as do most of the password tools nowadays). As I said, this sort of attack only checks for weak combinations in a tiny, tiny fraction of the phase space, and usually they only run a their dictionary once (or for at most a day or two) and then move on. So many IP numbers to try to crack, so little time. BTW, one can prevent nearly all of the attacks like the ones above just by running sshd on a nonstandard port. They are almost never driven by a real portscanner-driven attack, the attackers are mindless droids doing what droids do, which is looking for stupid passwords on a handful of possible accounts on the standard sshd port and then moving on. Ssh running on port umpty-eleven ensures that your server will almost never get touched (compared to almost always), but if one adds a few simple lines to your .ssh/config file you can ssh to it completely transparently. That's what I did to eliminate most of the "noise" like that above, which is harmless I suppose but does make me nervous...;-) Oh, and don't forget the port you used or you'll have to use nmap to find it again from the outside...;-) Moving the port also reduces the network load on your servers by a tiny amount; it costs resources just to say nope, no, not here, wrong username, sorry, try again later 10,000 or so times a day, and it clogs up log monitoring programs and /var/log as well. It doesn't help much on user workstations or login servers, obviously -- they tend to have to have standard port numbers so people can find them -- but boy does it help with the protected/restricted access servers and the like. >> There are also still -- relatively rarely -- buffer overwrite attacks >> discovered. > > Rarely? You haven't been reading full-disclosure lately I see. There > are a half dozen new such vulns found a day. Not like in the old days, when lpd nearly always had one, syslogd had one for years, etc. The core daemons run by a standard linux box get fairly strongly scrutinized. That doesn't mean that there aren't any, only that ones in tools that are at all likely to be managing a public port are discovered relatively infrequently and patched very quickly, in linux at least (and I don't care about Windows). nmap a box and unless somebody is doing something really fancy with it you're likely to get what, anywhere from 0 to 10 ports, the high end of that only if it is some sort of server? As always, with exceptions, especially for novices who run all sorts of things including e.g. games servers without much of a clue. And I have no idea what is discovered in the Windows tree these days. I'm not trying to minimize the threat posed by buffer overwrite attacks, BTW. They are very serious, obviously. However, again, the risk is very, very significantly mitigated by tools such as yum or apt. In the linux community it isn't that uncommon for an exploit to be discovered in some important app or daemon Wednesday morning, for it to be closed up by Wednesday afternoon, for the patched code to be dropped into the major toplevel repos by Wednesday evening, to be mirrored overnight and automatically updated on user desktops between Thursday and Saturday morning depending on your distance down the distribution/repo tree. Not a lot of room for traction there, either. >> Most coders "get it" > > No, most of them don't. I've done a lot of code audit in my day. The > average C programmer turned out these days really thinks you use > system("rm filename") to unlink a file, and that's the good part of > their code. For a laugh, google for the "daily wtf" and start reading > some of the stuff you see. Very funny! My own favorite bits of coding humor are a draw between the Tao of Programming and the fake C++ interview (both on my website, although I didn't write either of them:-). I can see why this site would make a cynic out of you, though. And I have no doubt that its examples are real. But no surprise, really -- I teach coding. To be fair, you have to realize that most schools don't teach C programming at all any more, and many haven't taught a variant of C such as C++ for quite a while either. There aren't even many competent TEACHERS of C left at schools -- they've been superceded by people teaching java, C++, pascal, visual basic, etc. Consequently, I find myself teaching C to independent study students at Duke -- quite bright kids, usually, often have some idea of how to code in some language or other -- maybe java (which they do currently teach, sigh). The sad truth of the matter is that programming, especially in C, is really quite difficult. It takes: a) a certain kind of personality -- most computer geeks are self-selected into the group because they have the right kind. Here I am, typing away at 2:30 am because everybody else is in bed and I can CODE. I have the right kind too. Trying to teach a non-coder type to code is effectively impossible. b) a certain amount of native intelligence. After all, you're building a machine out of words, so to speak. In one sense, it is easier than calculus and differential geometry; in another it embraces the difficulty of both as one often has to code problems in calculus or even differential geometry (or worse, linear algebra:-). Not for the stupid. c) some mix of mentoring by a guru and working your ass off. A bright and motivated person can teach themselves C, but I'll be damned if I think that they can teach themselves to code C >>well<< in less than years of painful mistake-driven lessons unless they have a guru or some other way of learning some of the simple wisdom that makes it possible to turn out decent, readable, code. And even with a guru and working quite hard, it still takes years, just fewer years. I taught myself C with no guru (I already had taught myself fortran, pascal, basic, and had learned PL/1 and a couple of assember languages in school) and hence had to learn everything the hardest of ways (and have made spectacular mistakes over the years). I still am learning, and still the hard way -- just this week I had to solve a really nasty "impossible" bug of my own creation where a variable changed values literally between lines of code where nothing touched it (when manipulating pointers nothing is impossible, including impossible bugs;-). C is very, very difficult and there is No Safety Net. You can perform sublime miracles of coding or make a complete, utter mess with almost no help at all either way from the compiler or linker. So I would never mock the poor souls that have yet to learn of unlink or fstat, only enlighten them. And pray for deliverance myself from my own ignorance as I did so, as C is >>complex<< (oh, wait, no, it is double:-) and I'm certain I still am doing at least some things stupidly or non-optimally for all that I've written a few thousand lines of C in the last week and a half in a 20,000 line program (with 2/3 of it comments and it is running, thank you;-). BUT, with all of that said -- the end of man strcpy these days contains gems like: BUGS If the destination string of a strcpy() is not large enough (that is, if the programmer was stupid or lazy, and failed to check the size before copying) then anything might happen. Overflowing fixed length strings is a favorite cracker technique. It helps to have a guru to bop you upside the head and tell you "use a command with an 'n' in it to copy, move, manipulate buffers and here's why" but in the end, as MY guru once told me, RTFM. Then read it again. Don't be lazy and stupid! >> But weak passwords that are brute force guessed[...]? >> Only on a poorly managed network, > > That would be 95% of networks. I've done a lot of network audits in my > day, too. Well, Duke's are pretty good, or the people being cracked are deeply ashamed and I'm not hearing of them. Which was not always the case, BTW -- I used to help people debrief cracking attempts and helped write two distinct revisions of its security guidelines and policy. So I'm not sure I'd agree with your 95% of all networks are poorly managed -- that is a rather large and possibly hyperbolic assignment of incompetence, after all, more rhetoric than substance. However, my experience is far from universal, and there IS certainly plenty of incompetence out there. And there is no doubt, five users in a hundred WILL choose incredibly dumb passwords if left to their own devices and two of them will get angry at you for telling them they have to change them to something more difficult to guess because then they have to remember something difficult to guess instead of their own birthday. We actually have had a really competent "security czar" on campus for a decade or so as well as good sysadmin groups and a few really bright lights to lead them, so the mentorship here is perhaps better than average. That's what really makes the difference in the long run. People who take their job seriously and work in a supportive environment will usually do it decently and avoid the most obvious mistakes. People who aren't really bright enough to be a computer geek, who don't have the right temperament to be a computer geek, and who've had no mentoring, people who basically chose computing as a career to make money -- they become MCSEs, don't they? Not linux people...;-) rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From john.hearns at streamline-computing.com Fri Jun 20 00:38:17 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:07:13 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <200806191716.41743.kilian@stanford.edu> References: <1381552971.34571213918331904.JavaMail.root@zimbra.vpac.org> <200806191716.41743.kilian@stanford.edu> Message-ID: <1213947507.4713.2.camel@Vigor13> On Thu, 2008-06-19 at 17:16 -0700, Kilian CAVALOTTI wrote: > I don't even know how you choose (or even if you can choose) on which > GPU you want your code to be executed. It has to be handled by the > driver on the host machine somehow. There are functions for discovering the properties of the GPU units on the system, and for selecting the device. You call the function cudaSetDevice, from the manual: cudaSetDevice() is used to select the device associated to the host thread and: cudaGetDeviceCount() and cudaGetDeviceProperties() provide a way to enumerate these devices and retrieve their properties: int deviceCount; cudaGetDeviceCount(&deviceCount); int device; for (device = 0; device < deviceCount; ++device) { cudaDeviceProp deviceProp; cudaGetDeviceProperties(&deviceProp, device); From perry at piermont.com Fri Jun 20 05:14:01 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: (Robert G. Brown's message of "Fri\, 20 Jun 2008 02\:41\:42 -0400 \(EDT\)") References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> Message-ID: <87mylgwbg6.fsf@snark.cb.piermont.com> "Robert G. Brown" writes: >> Yes, Run a box with sshd on it connected to the internet and watch your >> logs for a few days. You will find numerous attempts to try thousands >> of possible account names and passwords -- brute force cracking. > > Well, yeah, sure, I know about that as I DO watch my logs -- I just > haven't heard of one of these attacks SUCCEEDING in pretty much forever, > for obvious reasons. The worms that do that attack self-spread, so the fact that you're seeing it in your logs means that they've succeeded. > These attacks run through a list of "likely user names" (including well > known system names like "root") and then run through a short list of > "likely passwords" such as user=test password=test. Short because ssh > enforces a roughly 1 second interval between tries, which limits the > total number of attempts that can be made to < 100,000/day, That limits the number of attempts that may be made against your particular machine. At the same time that they're attacking your machine, that one instance is attacking a vast number of other randomly selected boxes. There are also a vast number of the things running out there, so in the long run, they succeed quite a bit of the time. There are many other similar sorts of things out there that you are less aware of -- things knob-twisting on ebay, yahoo and gmail accounts (I would explain why people want to steal something that's free but its a long story), online banking, etc. The deal is, this is a sort of biological ecosystem. Every possible niche is filled. If you can imagine an attack, someone out there is doing it at this point. > In fact, to me it looks like most of the attacks are generated by > more or less the same code and probably reuse most of the same names > and password guesses Don't be overly confident because of how primitive such things seem. They succeed constantly. As I noted, if they didn't, you wouldn't see them in your logs all the time. > Sure, it goes on and on. I don't really LIKE seeing this, especially on > a server with sensitive information, but that is precisely why one > configures such servers with tight controls and runs a password checker. I don't run a password checker. I simply have no mechanisms in use that *use* passwords, so it is irrelevant. Then again, I'm far more paranoid than most people are. Professional hazard. >> These attacks are done by automated malware that spreads itself around >> from machine to machine for nefarious purposes -- good luck trying to >> track down the puppet masters. I've tracked the bad guys down a few >> times but they're always somewhere like Bucharest anyway, and the >> locals don't care to arrest them. > > Agreed. All part of the Russian Diabolical Supercomputer we discussed > six months ago or so -- a cloud of unknown proportions and evil > intent:-) A few times, friends of mine have suggested I should rent botnet time to run computational chemistry problems. It would certainly be vastly cheaper than my own clusters, but unfortunately it is also against my morals. >> It is true that this is only one of many modern attack vectors and >> that buffer overflows, drive by malware downloads into browsers, etc., >> are all far more common ways in, but you will indeed get hacked by >> automated systems if you leave an sshd on and have accounts with weak >> passwords. > > Agreed as well. Which is why professionals generally check for weak > passwords (as do most of the password tools nowadays). Nah. Professionals don't even use passwords any more, as I said. (I'm forced to use them on most e-Commerce sites because just about no one does client cert based authentication, but my machines don't use passwords. Most of my buddies machines don't use them either.) > As I said, this sort of attack only checks for weak combinations in > a tiny, tiny fraction of the phase space, and usually they only run > a their dictionary once (or for at most a day or two) and then move > on. So many IP numbers to try to crack, so little time. Any given instance only tries a small random subset of the phase space at a time on a given machine. However, the worms are pretty good about picking random subsets. Over time, you will find they slowly will find most of the more weakly defended boxes, just like water wearing down a stone. > BTW, one can prevent nearly all of the attacks like the ones above just > by running sshd on a nonstandard port. People do all sorts of things -- port knocking, filters that autoblacklist IPs that have too many errors, etc. It is simpler and safer just not to use passwords. > They are almost never driven by a real portscanner-driven attack, Only because they're not attacking you in particular. If someone had reason to attack you in particular, then you would have more to worry about. Again, I find it is better just to make sure that there is nothing to attack. > Moving the port also reduces the network load on your servers by a tiny > amount; it costs resources just to say nope, no, not here, wrong > username, sorry, try again later 10,000 or so times a day, and it clogs > up log monitoring programs and /var/log as well. I have professional reasons to want to know that this sort of thing is happening. If it irritates you try sshd_sentry, sshd-blacklist, or any one of fifty other little scripts that will stop the kneebiters. >>> There are also still -- relatively rarely -- buffer overwrite attacks >>> discovered. >> >> Rarely? You haven't been reading full-disclosure lately I see. There >> are a half dozen new such vulns found a day. > > Not like in the old days, when lpd nearly always had one, syslogd had > one for years, etc. No, really, it is still like the old days. lpd and syslogd probably still have plenty (just read the code), but nowadays no one smart runs them exposed on the net. Syslogd has long since had patches added so that it doesn't need to listen to the net by default. Does that mean the syslogd code base is great now? No. Remember when X servers ran listening to all comers? One of my employees, at my request, was the one who contributed the "-nolisten tcp" patch to the X consortium that is now more or less the default on all installations. Does that mean that X has no buffer overflows? Certainly not. It just means that they're not exploitable by arbitrary people these days so it isn't a big target. It is, to some extent, a question of how many people are interested in a particular attack vector. Internet Explorer is a major attack vector for people who make money at this, so they work hard finding the bugs in it, of which there are an apparent endless number. I believe that more than 250 days last year, Internet Explorer had a known but as yet unpatched vulnerability. That's why the overwhelming majority of Windows boxes are zombies, including almost certainly most of yours unless you are a really unusual sysadmin. I encourage trying to read full-disclosure, bugtraq, and the like. You'll get depressed very fast. Here is just the May 1 traffic on Bugtraq, cut and pasted from the mailing list archive (http://seclists.org/bugtraq/2008/May/): # XSS in AstroCam Steffen Wendzel (May 01 2008) # iDefense Security Advisory 04.30.08: Akamai Download Manager Arbitrary Program Execution Vulnerability iDefense Labs (May 01 2008) # [SECURITY] [DSA 1564-1] New wordpress packages fix several vulnerabilities Thijs Kinkhorst (May 01 2008) # Team SHATTER Security Advisory: Oracle Database Buffer Overflow in SYS.DBMS_AQJMS_INTERNAL (DB15) Team SHATTER (May 01 2008) # mjguest 6.7 (ALL VERSION) Xss & Redirection Vuln irancrash_at_gmail.com (May 01 2008) # vlBook 1.21 (ALL VERSION) irancrash_at_gmail.com (May 01 2008) # Team SHATTER Security Advisory: Oracle Database SQL Injection in SYS.DBMS_CDC_UTILITY.LOCK_CHANGE_SET (DB02) Team SHATTER (May 01 2008) # Team SHATTER Security Advisory: Oracle Database Buffer Overflow in SYS.KUPF$FILE_INT.GET_FULL_FILENAME (DB11) Team SHATTER (May 01 2008) # [SECURITY] [DSA 1565-1] New Linux 2.6.18 packages fix several vulnerabilities dann frazier (May 01 2008) # php-addressbook v2.0 Multiple Remote Vulnerabilities (LFI/XSS) irancrash_at_gmail.com (May 01 2008) # Re: netOffice Dwins 1.3 Remote code execution. luiswang_at_user.sourceforge.net (May 01 2008) # BlackBook v1.0 Multiple XSS Vulnerabilities irancrash_at_gmail.com (May 01 2008) A large number of new vulnerabilities a day, half a dozen or more of which are always new buffer overflow exploits. > The core daemons run by a standard linux box get fairly strongly > scrutinized. If you're smart, you're listening on: * DNS, with bind configured to run chrooted and unprivileged * sshd running with priv sep * ntpd running chrooted and unprived (though not all OSes will allow you to do that.) * maybe SMTP via postfix, which runs chrooted and unprived * and NOTHING ELSE. And if you're really smart, those daemons are further tied down with various bondage and discipline equipment like apparmor or SE Linux or what have you. Everything you listen on is another attack vector that will be exploited one day. > That doesn't mean that there aren't any, only that ones in tools > that are at all likely to be managing a public port are discovered > relatively infrequently and patched very quickly, I think that the real trick is that most machines aren't paying attention to very many ports. > I'm not trying to minimize the threat posed by buffer overwrite attacks, > BTW. They are very serious, obviously. However, again, the risk is > very, very significantly mitigated by tools such as yum or apt. If you're lucky. The problem is that we find out about most of these things after someone is already mass exploiting it. If you're smart, you'll make sure that it is unlikely anyone can exploit your boxes in the first place by being exceptionally mean about how you run the very few things you run. There is doubtless some way to exploit bind if it is running chrooted, unprived and with some sort of ACLs preventing use of unneeded syscalls, but it is much less likely that you'll be the one that gets hit if you do all of that when many people don't. > In the linux community it isn't that uncommon for an exploit to be > discovered in some important app or daemon Wednesday morning, for it > to be closed up by Wednesday afternoon, for the patched code to be > dropped into the major toplevel repos by Wednesday evening, to be > mirrored overnight and automatically updated on user desktops > between Thursday and Saturday morning depending on your distance > down the distribution/repo tree. Not a lot of room for traction > there, either. Not true. At this point, there are automated attack engines that can have any new zero-day exploit dropped in and will be on hundreds of thousands of boxes the same hour they're found. Really. 24 hours is too long. If you don't believe me on all of this, I suggest reading the literature. It is very grim. There is also time to register for Usenix Security if you would like to hear a very long parade of exceptionally depressing papers being presented about a month from now. >>> But weak passwords that are brute force guessed[...]? >>> Only on a poorly managed network, >> >> That would be 95% of networks. I've done a lot of network audits in my >> day, too. > > Well, Duke's are pretty good, or the people being cracked are deeply > ashamed and I'm not hearing of them. Most people who get cracked never even know, so why would they say anything? University networks are among the worst -- they're filled with boxes with no proper management. Even Jeff Schiller at MIT will cop to his net being filled with exploited boxes and that's a place where most of the infrastructure is hardened by world class experts. If you really believe your local net is very good, run a sniffer on it for a while -- or talk to someone who's job is to run one. > So I'm not sure I'd agree with your 95% of all networks are poorly > managed -- that is a rather large and possibly hyperbolic assignment > of incompetence, after all, more rhetoric than substance. If you want to believe that, sure. Most networks are in small offices where there isn't even a professional sysadmin. The average network has no one associated with it who knows anything much about computers at all, and was set up by a consultant who barely knews anything either. Think about what your dentist's office or the network at a small town city hall is like. 95% is being generous. It is probably higher. > However, my experience is far from universal, I've done computer security consulting for most of the last couple of decades. (So why am I on this list, you ask? Well, a very long story, but for the last few years I've been much more interested in scientific research than in my putative day job, which I've scaled back to only about a third of my time. Having a computer science background has enormous pluses when you find yourself doing scientific computing in mid life...) > We actually have had a really competent "security czar" on campus for a > decade or so as well as good sysadmin groups and a few really bright > lights to lead them, so the mentorship here is perhaps better than > average. That's what really makes the difference in the long run. Your "security czar" probably spends 2/3rds of her time horribly depressed at the futility of the whole thing. Most of the people in that job that I know are very very depressed about how well it can be done in a world where the average box is a zombie. -- Perry E. Metzger perry@piermont.com From smulcahy at aplpi.com Fri Jun 20 05:32:08 2008 From: smulcahy at aplpi.com (stephen mulcahy) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <87mylgwbg6.fsf@snark.cb.piermont.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> Message-ID: <485BA348.5020303@aplpi.com> Hi, I resisted the urge to join in on the nuclear tangent but this one proved too much (and we are indirectly back to talking about the security of the clusters we look after right?). Besides, we don't have any nukes in Ireland. Perry E. Metzger wrote: > It is, to some extent, a question of how many people are interested in > a particular attack vector. Internet Explorer is a major attack vector > for people who make money at this, so they work hard finding the bugs > in it, of which there are an apparent endless number. I believe that > more than 250 days last year, Internet Explorer had a known but as yet > unpatched vulnerability. That's why the overwhelming majority of > Windows boxes are zombies, including almost certainly most of yours > unless you are a really unusual sysadmin. I'm reading this to mean that you think most Windows boxes on most networks are zombies - is that right? As one of my many roles, I babysit our company network and I'd love to know how to avoid the scenario you're painting - other than the usual stuff of keeping the machines up to date, ensuring people don't run the latest .exe they receive in a spam and not exposing Windows boxes to the internet. Maybe I should get MS certified (joke, joke ;) While suggestions to install Linux on all of them are constructive, I'm afraid we can't avoid running some Windows boxen on our network. > If you're smart, you're listening on: > > * DNS, with bind configured to run chrooted and unprivileged > * sshd running with priv sep > * ntpd running chrooted and unprived (though not all OSes will allow > you to do that.) > * maybe SMTP via postfix, which runs chrooted and unprived > * and NOTHING ELSE. > > And if you're really smart, those daemons are further tied down with > various bondage and discipline equipment like apparmor or SE Linux or > what have you. Ouch, it's a never-ending battle isn't it? I think you're largely right about the level of expertise out there for managing networks though - small companies don't pay someone to manage their network. Either they have some internal guy who has half a dozen other jobs or they outsource it, and unfortunately they'll usually outsource it to the cheapest guy ... who's cheap for a reason. > If you really believe your local net is very good, run a sniffer on it > for a while -- or talk to someone who's job is to run one. I'd love to know how anyone with skype running on their network manages to see much of anything from the firehose that is a packet trace (and our network is small). Again, maybe it's just a question of time. -stephen -- Stephen Mulcahy, Applepie Solutions Ltd., Innovation in Business Center, GMIT, Dublin Rd, Galway, Ireland. +353.91.751262 http://www.aplpi.com Registered in Ireland, no. 289353 (5 Woodlands Avenue, Renmore, Galway) From landman at scalableinformatics.com Fri Jun 20 06:28:40 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> Message-ID: <485BB088.7060406@scalableinformatics.com> Robert G. Brown wrote: > On Fri, 20 Jun 2008, Perry E. Metzger wrote: > >> >> "Robert G. Brown" writes: >>> On Fri, 20 Jun 2008, Chris Samuel wrote: >>>> ----- "Joe Landman" wrote: >>>> >>>>> People spend lots of time and effort on security theater. Make up odd >>>>> rules for passwords. Make them hard to guess and crack. Well, is >>>>> that the vector for break-ins? Weak passwords? >>>> >>>> Yeah - sadly.. :-( >>> >>> Do you have an recent contemporary evidence for that? >> >> Yes, Run a box with sshd on it connected to the internet and watch your >> logs for a few days. You will find numerous attempts to try thousands >> of possible account names and passwords -- brute force cracking. > > Well, yeah, sure, I know about that as I DO watch my logs -- I just > haven't heard of one of these attacks SUCCEEDING in pretty much forever, > for obvious reasons. Run pam_abl on your machine, and you can pretty much guarantee that the brute force attacks will not work, even if they miraculously guess the right password. This presumes more than some small number of previous login failures. [...] >> Here is an extract from the log on a real machine, one of mine, from >> last night: >> >> Jun 19 20:56:53 smaug sshd[2577]: Invalid user secretariat from >> 70.90.14.154 >> Jun 19 20:56:54 smaug sshd[2522]: Invalid user secretar from 70.90.14.154 >> Jun 19 20:56:55 smaug sshd[23949]: Invalid user present from 70.90.14.154 >> Jun 19 20:56:56 smaug sshd[3440]: Invalid user test from 70.90.14.154 >> Jun 19 20:56:57 smaug sshd[8809]: Invalid user test from 70.90.14.154 >> Jun 19 20:56:58 smaug sshd[21600]: Invalid user teste from 70.90.14.154 >> Jun 19 20:56:59 smaug sshd[314]: Invalid user teste from 70.90.14.154 > > Sure, it goes on and on. I don't really LIKE seeing this, especially on > a server with sensitive information, but that is precisely why one > configures such servers with tight controls and runs a password checker. Use pam_abl. Really. Even if the password were weak, and they guessed it on the 57th try, pam_abl will stop the login. Read the manual. Adjust the config settings. Our ssh logs are scary, have been for a while. They aren't the scariest of our logs. Even paranoids have enemies. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From smulcahy at aplpi.com Fri Jun 20 06:36:14 2008 From: smulcahy at aplpi.com (stephen mulcahy) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485BB088.7060406@scalableinformatics.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <485BB088.7060406@scalableinformatics.com> Message-ID: <485BB24E.7030807@aplpi.com> Joe Landman wrote: > Use pam_abl. Really. Even if the password were weak, and they guessed > it on the 57th try, pam_abl will stop the login. Read the manual. > Adjust the config settings. > > Our ssh logs are scary, have been for a while. They aren't the scariest > of our logs. DenyHosts works on a similar principle - http://denyhosts.sourceforge.net/ although pam_abl's pretending to accept logins even when the attacker has been blacklisted sounds downright sneaky! Could make for an interesting Monday morning pre-coffee login session though - as you wonder why your initially mistyped password still isn't working after 50 login attempts :) > Even paranoids have enemies. or "I've only been paranoid since they started watching me"? -stephen -- Stephen Mulcahy, Applepie Solutions Ltd., Innovation in Business Center, GMIT, Dublin Rd, Galway, Ireland. +353.91.751262 http://www.aplpi.com Registered in Ireland, no. 289353 (5 Woodlands Avenue, Renmore, Galway) From landman at scalableinformatics.com Fri Jun 20 06:52:54 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485BB24E.7030807@aplpi.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <485BB088.7060406@scalableinformatics.com> <485BB24E.7030807@aplpi.com> Message-ID: <485BB636.5060106@scalableinformatics.com> stephen mulcahy wrote: > > > Joe Landman wrote: >> Use pam_abl. Really. Even if the password were weak, and they >> guessed it on the 57th try, pam_abl will stop the login. Read the >> manual. Adjust the config settings. >> >> Our ssh logs are scary, have been for a while. They aren't the >> scariest of our logs. > > DenyHosts works on a similar principle - > http://denyhosts.sourceforge.net/ although pam_abl's pretending to > accept logins even when the attacker has been blacklisted sounds > downright sneaky! Yup! I wrote a tool called "danger" that parses the ssh logs, the /etc/hosts.deny logs, and makes ... recommondations about what to add. Based upon who has been attacking you. This pre-dated denyhosts by a bit. I still run it, and it gives me a nightly summary of the bad guys. > > Could make for an interesting Monday morning pre-coffee login session > though - as you wonder why your initially mistyped password still isn't > working after 50 login attempts :) Yuppers. Had that happen once. Learned to get coffee before doing anything important. > >> Even paranoids have enemies. > > or "I've only been paranoid since they started watching me"? > > -stephen > -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From tjrc at sanger.ac.uk Fri Jun 20 06:51:51 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485BA348.5020303@aplpi.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> <485BA348.5020303@aplpi.com> Message-ID: On 20 Jun 2008, at 1:32 pm, stephen mulcahy wrote: > I'd love to know how anyone with skype running on their network > manages to see much of anything from the firehose that is a packet > trace (and our network is small). Again, maybe it's just a question > of time. That'll be why we don't allow Skype, except on one network which, from a security perspective, is considered to be outside the Institute, and just as untrusted as the rest of the Internet. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From Craig.Tierney at noaa.gov Fri Jun 20 07:13:52 2008 From: Craig.Tierney at noaa.gov (Craig Tierney) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <200806191716.41743.kilian@stanford.edu> References: <1381552971.34571213918331904.JavaMail.root@zimbra.vpac.org> <200806191716.41743.kilian@stanford.edu> Message-ID: <485BBB20.8070408@noaa.gov> Kilian CAVALOTTI wrote: > On Thursday 19 June 2008 04:32:11 pm Chris Samuel wrote: >> ----- "Kilian CAVALOTTI" wrote: >>> AFAIK, the multi GPU Tesla boxes contain up to 4 Tesla processors, >>> but are hooked to the controlling server with only 1 PCIe link, >>> right? Does this spell like "bottleneck" to anyone? >> The nVidia website says: >> >> http://www.nvidia.com/object/tesla_tech_specs.html >> >> # 6 GB of system memory (1.5 GB dedicated memory per GPU) > > The latest S1070 has even more than that: 4GB per GPU as it seems, > according to [1]. > > But I think this refers to the "global memory", as decribed in [1] > (slide 12, "Kernel Memory Access"). It's the graphics card main memory, > the kind of one which is used to store textures in games, for instance. > Each GPU core also has what they call "shared memory" and which is > really only shared between threads on the same core (it's more like a > L2 cache actually). > >> So my guess is that you'd be using local RAM not the >> host systems RAM whilst computing. > > Right, but at some point, you do need to transfer data from the host > memory to the GPU memory, and back. That's where there's probably a > bottleneck if all 4 GPUs want to read/dump data from/to the host at the > same time. > > Moreover, I don't think that the different GPUs can work together, ie. > exchange data and participate to the same parallel computation. Unless > they release something along the lines of a CUDA-MPI, those 4 GPUs > sitting in the box would have to be considered as independent > processing units. So as I understand it, the scaling benefits from your > application's parallelization would be limited to one GPU, no matter > how many you got hooked to your machine. > You can integrate MPI with CUDA and create parallel applications. CUDA is just a preprocessor that uses the local C compiler (gcc for Linux by default). I have seen some messages on the MVAPICH mailing list talking about users doing this. Since the memory is on the card, you have to transfer back to the host before you can send it via an MPI call. However, if your entire model can fit in the GPU's memory (which is why the 4GB S1070 Tesla card is useful), then you should be able to pull down the portion of memory from the GPU you want to send out, then send it. Or at least that's how I understand it. When I get my systems I will get to figure out the "real" details. Craig > I don't even know how you choose (or even if you can choose) on which > GPU you want your code to be executed. It has to be handled by the > driver on the host machine somehow. > >> There's a lot of fans there.. > > They probably get hot. At least the G80 do. They say "Typical Power > Consumption: 700W" for the 4 GPUs box. Given that a modern gaming rig > featuring a pair of 8800GTX in SLI already requires a 1kW PSU, I would > put this on the optimistic side. > > [1]http://www.nvidia.com/object/tesla_s1070.html > [2]http://www.mathematik.uni-dortmund.de/~goeddeke/arcs2008/C1_CUDA.pdf > > > Cheers, -- Craig Tierney (craig.tierney@noaa.gov) From shaeffer at neuralscape.com Fri Jun 20 07:36:21 2008 From: shaeffer at neuralscape.com (Karen Shaeffer) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> Message-ID: <20080620143621.GA31402@synapse.neuralscape.com> On Fri, Jun 20, 2008 at 02:41:42AM -0400, Robert G. Brown wrote: > > c) some mix of mentoring by a guru and working your ass off. A bright > and motivated person can teach themselves C, but I'll be damned if I > think that they can teach themselves to code C >>well<< in less than > years of painful mistake-driven lessons unless they have a guru or some > other way of learning some of the simple wisdom that makes it possible > to turn out decent, readable, code. And even with a guru and working > quite hard, it still takes years, just fewer years. Today it is very straight forward. The 2.6.x linux kernel is an exemplary body of superb C programming code. All you need is motivation and a network connection. And the linux kernel email development lists have hundreds of programming guru's pontificating and standing on soap boxes daily. Thanks, Karen -- Karen Shaeffer Neuralscape, Palo Alto, Ca. 94306 shaeffer@neuralscape.com http://www.neuralscape.com From john.leidel at gmail.com Fri Jun 20 07:42:09 2008 From: john.leidel at gmail.com (John Leidel) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <485BBB20.8070408@noaa.gov> References: <1381552971.34571213918331904.JavaMail.root@zimbra.vpac.org> <200806191716.41743.kilian@stanford.edu> <485BBB20.8070408@noaa.gov> Message-ID: <1213972929.5684.52.camel@e521.site> Craig [et.al], this is also how I understand. One could realistically wrap standard MPI calls to do this for you: MPI_GPU_Bcast(...){ malloc_some_stuff pull_mem_from_gpu MPI_Bcast(...) free_some_stuff } ...just a random thought though... On Fri, 2008-06-20 at 08:13 -0600, Craig Tierney wrote: > Kilian CAVALOTTI wrote: > > On Thursday 19 June 2008 04:32:11 pm Chris Samuel wrote: > >> ----- "Kilian CAVALOTTI" wrote: > >>> AFAIK, the multi GPU Tesla boxes contain up to 4 Tesla processors, > >>> but are hooked to the controlling server with only 1 PCIe link, > >>> right? Does this spell like "bottleneck" to anyone? > >> The nVidia website says: > >> > >> http://www.nvidia.com/object/tesla_tech_specs.html > >> > >> # 6 GB of system memory (1.5 GB dedicated memory per GPU) > > > > The latest S1070 has even more than that: 4GB per GPU as it seems, > > according to [1]. > > > > But I think this refers to the "global memory", as decribed in [1] > > (slide 12, "Kernel Memory Access"). It's the graphics card main memory, > > the kind of one which is used to store textures in games, for instance. > > Each GPU core also has what they call "shared memory" and which is > > really only shared between threads on the same core (it's more like a > > L2 cache actually). > > > >> So my guess is that you'd be using local RAM not the > >> host systems RAM whilst computing. > > > > Right, but at some point, you do need to transfer data from the host > > memory to the GPU memory, and back. That's where there's probably a > > bottleneck if all 4 GPUs want to read/dump data from/to the host at the > > same time. > > > > Moreover, I don't think that the different GPUs can work together, ie. > > exchange data and participate to the same parallel computation. Unless > > they release something along the lines of a CUDA-MPI, those 4 GPUs > > sitting in the box would have to be considered as independent > > processing units. So as I understand it, the scaling benefits from your > > application's parallelization would be limited to one GPU, no matter > > how many you got hooked to your machine. > > > > You can integrate MPI with CUDA and create parallel applications. CUDA is > just a preprocessor that uses the local C compiler (gcc for Linux by default). > I have seen some messages on the MVAPICH mailing list talking about users > doing this. > > Since the memory is on the card, you have to transfer back to the host > before you can send it via an MPI call. However, if your entire model > can fit in the GPU's memory (which is why the 4GB S1070 Tesla card is > useful), then you should be able to pull down the portion of memory > from the GPU you want to send out, then send it. > > Or at least that's how I understand it. When I get my systems I will > get to figure out the "real" details. > > Craig > > > > > > > I don't even know how you choose (or even if you can choose) on which > > GPU you want your code to be executed. It has to be handled by the > > driver on the host machine somehow. > > > >> There's a lot of fans there.. > > > > They probably get hot. At least the G80 do. They say "Typical Power > > Consumption: 700W" for the 4 GPUs box. Given that a modern gaming rig > > featuring a pair of 8800GTX in SLI already requires a 1kW PSU, I would > > put this on the optimistic side. > > > > [1]http://www.nvidia.com/object/tesla_s1070.html > > [2]http://www.mathematik.uni-dortmund.de/~goeddeke/arcs2008/C1_CUDA.pdf > > > > > > Cheers, > > From jmdavis1 at vcu.edu Fri Jun 20 08:12:06 2008 From: jmdavis1 at vcu.edu (Mike Davis) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <20080620143621.GA31402@synapse.neuralscape.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> Message-ID: <485BC8C6.4080800@vcu.edu> Karen Shaeffer wrote: > On Fri, Jun 20, 2008 at 02:41:42AM -0400, Robert G. Brown wrote: > >> c) some mix of mentoring by a guru and working your ass off. A bright >> and motivated person can teach themselves C, but I'll be damned if I >> think that they can teach themselves to code C >>well<< in less than >> years of painful mistake-driven lessons unless they have a guru or some >> other way of learning some of the simple wisdom that makes it possible >> to turn out decent, readable, code. And even with a guru and working >> quite hard, it still takes years, just fewer years. >> > > Today it is very straight forward. The 2.6.x linux kernel is an exemplary > body of superb C programming code. All you need is motivation and a network > connection. And the linux kernel email development lists have hundreds of > programming guru's pontificating and standing on soap boxes daily. > > Thanks, > Karen > Linus made this statement or a very similar one at a Linux Expo at Duke back in 1998. In his version, he included the advice that if you wanted to write good code, you should start by reading good code. Then he went on to recommend the linux kernel. Reading code is IMHO one of the most important parts of learning to write good code. It doesn't matter if the target language is f77, C, or even (gulp) java. Reading well written code will help one to understand how to construct it. Mike From matt at technoronin.com Fri Jun 20 08:32:28 2008 From: matt at technoronin.com (Matt Lawrence) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] How to do a single network install Message-ID: I needed a quick way to signal back to the kickstart server that my install was done and to boot from the hard drive next time. Or, PXEboot and then fall through to booting off the hard drive. So, I wrote a quick and simple script. In the %post clause, I have: echo "done" | telnet 192.168.200.8 46758 Here's the script: #!/usr/bin/ruby require 'socket' server = TCPserver.new('0.0.0.0', 46758) while (session = server.accept) text = session.gets puts text # p session.peeraddr addr = "" session.peeraddr[3].split('.').each { |n| addr += sprintf("%02.2X", n.to_i) } p addr begin base = "/tftpboot/pxelinux.cfg/#{addr}" File.rename(base, base + ".done") rescue SystemCallError puts "Error occurred renaming #{base}" end session.close end Yeah, lots of stuff hardcoded. Enjoy. -- Matt It's not what I know that counts. It's what I can remember in time to use. From mark.kosmowski at gmail.com Fri Jun 20 09:30:26 2008 From: mark.kosmowski at gmail.com (Mark Kosmowski) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] security for small, personal clusters Message-ID: What kind of security is recommended for the owner of a small personal cluster? Where should the owner of a small, personal cluster go to learn about security? Doing searches tends to give a few "head in the sand" sites but predominantly seem to be oriented for the security professional. I maintain a small 3 node personal cluster used for my part-time PhD studies in chemistry. So far I've just kind of been hoping that I'm too small to bother with and that the firewalls (Linux and switch) between my boxes and the cable internet do something. I do use ssh for cluster communications, and have disabled the reknowned unstable services such as ftp. RGB mentioned running services on non-standard ports. This seems like a good idea to further reduce the probability of successful attack. I'll look into this in a couple weeks when I'm laid off and will have more time on the IT side of things. Are there any good sites aimed at the small user to slowly learn a little bit about security? Or are the only choices to do security full-time or forget about it entirely? What sorts of security do other here have on their personal clusters? Is maybe the simplest idea to just disable TCP/IP across the internet network while not in use and enable it when it needs to be up? Thanks, Mark E. Kosmowski From dnlombar at ichips.intel.com Fri Jun 20 10:07:38 2008 From: dnlombar at ichips.intel.com (Lombard, David N) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <87skv8wxlg.fsf@snark.cb.piermont.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> Message-ID: <20080620170738.GA12172@nlxdcldnl2.cl.intel.com> On Fri, Jun 20, 2008 at 12:15:39AM -0400, Perry E. Metzger wrote: > > "Robert G. Brown" writes: > > Do you have an recent contemporary evidence for that? > > Yes, Run a box with sshd on it connected to the internet and watch your > logs for a few days. You will find numerous attempts to try thousands > of possible account names and passwords -- brute force cracking. > > Here is an extract from the log on a real machine, one of mine, from > last night: > > Jun 19 20:56:53 smaug sshd[2577]: Invalid user secretariat from 70.90.14.154 > Jun 19 20:56:54 smaug sshd[2522]: Invalid user secretar from 70.90.14.154 > Jun 19 20:56:55 smaug sshd[23949]: Invalid user present from 70.90.14.154 > Jun 19 20:56:56 smaug sshd[3440]: Invalid user test from 70.90.14.154 > Jun 19 20:56:57 smaug sshd[8809]: Invalid user test from 70.90.14.154 > Jun 19 20:56:58 smaug sshd[21600]: Invalid user teste from 70.90.14.154 > Jun 19 20:56:59 smaug sshd[314]: Invalid user teste from 70.90.14.154 Yeah, I get that all the time too, I use an /etc/hosts.allow filter to temporarily block those idiots after three such attempts. > It goes on and on and on. There are countermeasures you can run to > block the zombies trying to guess passwords, but I rarely bother since > none of my machines allow password based login so their attempts are > useless anyway. Same here, so agree to the futility. But, why suffer the endless churn? If left alone, some will pound away for hours. > > But weak passwords that are brute force guessed[...]? > > Only on a poorly managed network, > > That would be 95% of networks. I've done a lot of network audits in my > day, too. Yup. Just fire up any Wifi kit and look at the visible networks. Also don't forget SC's wall of shame... -- David N. Lombard, Intel, Irvine, CA I do not speak for Intel Corporation; all comments are strictly my own. From john.hearns at streamline-computing.com Fri Jun 20 10:09:27 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] security for small, personal clusters In-Reply-To: References: Message-ID: <1213981777.5022.17.camel@Vigor13> On Fri, 2008-06-20 at 12:30 -0400, Mark Kosmowski wrote: > What kind of security is recommended for the owner of a small personal > cluster? Where should the owner of a small, personal cluster go to > learn about security? Doing searches tends to give a few "head in the > sand" sites but predominantly seem to be oriented for the security > professional. The best book I have on this subject is "Hacking Exposed" http://books.google.co.uk/books?hl=en&id=J5CDISzdA1EC&dq=hackingexposed&printsec=frontcover&source=web&ots=P_IReeKpK6&sig=96wpzidcNtiY1mvQqHBSeVTNLsA&sa=X&oi=book_result&resnum=1&ct=result http://www.amazon.com/Hacking-Exposed-5th/dp/0072260815/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1213981724&sr=1-1 Run, do not walk, to your nearest bookstore and purchase this. From tjrc at sanger.ac.uk Fri Jun 20 10:37:34 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485BC8C6.4080800@vcu.edu> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> Message-ID: <596BD9EE-8C63-4BF1-BC73-2895AACA028A@sanger.ac.uk> On 20 Jun 2008, at 4:12 pm, Mike Davis wrote: > Reading code is IMHO one of the most important parts of learning to > write good code. It doesn't matter if the target language is f77, C, > or even (gulp) java. Reading well written code will help one to > understand how to construct it. That's true. I thought I wrote reasonably well structured and maintainable C until I saw some code written by an ex-colleague of mine, and realised just how good some people are! It was far better than the majority of code I've seen, and that includes almost all the open source stuff I've had to grub around in. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From kilian at stanford.edu Fri Jun 20 10:55:38 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <485BA348.5020303@aplpi.com> Message-ID: <200806201055.38590.kilian@stanford.edu> On Friday 20 June 2008 06:51:51 am Tim Cutts wrote: > That'll be why we don't allow Skype, except on one network which, > from a security perspective, is considered to be outside the > Institute, and just as untrusted as the rest of the Internet. I think that's a wise decision. Skype is a giant black box. Fabrice Desclaux published a fair amount of cryptanalysis papers about Skype, each one more frightening than the previous ([1], [2] and [3]) [1]http://www.secdev.org/conf/skype_BHEU06.handout.pdf [2]http://recon.cx/en/f/vskype-part1.pdf [3]http://recon.cx/en/f/vskype-part2.pdf In 2005, an official recommendation has been issued by the French authorities to prohibit usage of Skype in the National Center for Scientific Research (CNRS)'s networks (see [4], and [5] for the machine-translated version) [4]http://www.urec.cnrs.fr/rubrique216.html [5]http://translate.google.com/translate?u=http%3A%2F%2Fwww.urec.cnrs.fr%2Frubrique216.html&hl=en&ie=UTF8&sl=fr&tl=en -- Kilian From peter.st.john at gmail.com Fri Jun 20 10:59:08 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Plan 9 Message-ID: I was surprised to see Plan 9 deployed on a Blue Gene, http://en.wikipedia.org/wiki/Blue_Gene#Plan_9_support. I think of Plan 9 as the direct descendant of System V; Bell Labs wanted to integrate distributed processing (I believe thinking, everything on the net can be an appliance to anything else) in the "new" (latter day) unix. I've never run it myself, but hosting MPI sounds plausible this way. Has anybody tried it? Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080620/54856226/attachment.html From larry.stewart at sicortex.com Fri Jun 20 11:03:25 2008 From: larry.stewart at sicortex.com (Lawrence Stewart) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485BC8C6.4080800@vcu.edu> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> Message-ID: <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> On Jun 20, 2008, at 11:12 AM, Mike Davis wrote: > Karen Shaeffer wrote: >> On Fri, Jun 20, 2008 at 02:41:42AM -0400, Robert G. Brown wrote: >> >>> c) some mix of mentoring by a guru and working your ass off. A >>> bright >>> and motivated person can teach themselves C, but I'll be damned if I >>> think that they can teach themselves to code C >>well<< in less than >>> years of painful mistake-driven lessons unless they have a guru or >>> some >>> other way of learning some of the simple wisdom that makes it >>> possible >>> to turn out decent, readable, code. And even with a guru and >>> working >>> quite hard, it still takes years, just fewer years. >>> >> >> Today it is very straight forward. The 2.6.x linux kernel is an >> exemplary >> body of superb C programming code. All you need is motivation and a >> network >> connection. And the linux kernel email development lists have >> hundreds of >> programming guru's pontificating and standing on soap boxes daily. >> >> Thanks, >> Karen >> > > Linus made this statement or a very similar one at a Linux Expo at > Duke back in 1998. In his version, he included the advice that if > you wanted to write good code, you should start by reading good > code. Then he went on to recommend the linux kernel. > > Reading code is IMHO one of the most important parts of learning to > write good code. It doesn't matter if the target language is f77, C, > or even (gulp) java. Reading well written code will help one to > understand how to construct it. > > > > Mike > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf There are many positive things to say about the Linux kernel, but "good code" is not one of them. Well that is too broad an indictment - There is good code in there, but the level varies widely. Have you <> at the TTY drivers? Have you counted the number of "<< 9" s in the block code? Tried to figure out which includes are actually active? Tried to figure out which of the 17 ways to do something is the "approved" one? I have one of the copies of the "Lions" book - the annoted sources for V6. That is good code. Written by one or a very few exceptional programmers, not an agglomeration of a zillion patches. And don't get me started about the ways in which Linux is ill suited to HPC. . . . well actually that would be a pretty good debate for this forum. The kernel is very reliable compared to pretty much anything, and that is worth a lot, but it isn't the place to learn programming style. -Larry From prentice at ias.edu Fri Jun 20 11:20:53 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> Message-ID: <485BF505.6000904@ias.edu> Lawrence Stewart wrote: > And don't get me started about the ways in which Linux is ill suited to > HPC. . . . well actually that > would be a pretty good debate for this forum. > -Larry > But, Larry, SiCortex products are based on Linux. It say so in nice big letters right there on your company's main page: "Linux-based super-computing, designed from the silicon up." - http://www.sicortex.com/ -- Prentice From larry.stewart at sicortex.com Fri Jun 20 12:33:19 2008 From: larry.stewart at sicortex.com (Lawrence Stewart) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485BF505.6000904@ias.edu> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> Message-ID: <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> On Jun 20, 2008, at 2:20 PM, Prentice Bisbal wrote: > Lawrence Stewart wrote: > >> And don't get me started about the ways in which Linux is ill >> suited to >> HPC. . . . well actually that >> would be a pretty good debate for this forum. > >> -Larry >> > > But, Larry, SiCortex products are based on Linux. It say so in nice > big > letters right there on your company's main page: > > "Linux-based super-computing, designed from the silicon up." > > - http://www.sicortex.com/ > > -- > Prentice > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf Linux is the worst operating system anywhere...except for all the others. Kind of like democracy. Now who was it at SiCortex who chose Linux? Oh yes, it was me. Linux is more reliable than anything else, more amenable to porting, more adaptable, there is more available stuff that works with it, customers like it, it pretty much works good. And it is open source. But it isn't elegant, except in patches. More specifically for HPC, linux seems designed for the desktop, and for small memory machines. Many of the decisions that made good sense in the '90s make no sense now. 512 byte disk sectors? 4096 byte pages? Randomly allocated physical memory? "large page" patches on small pages? All sorts of dinky little tasks waking up and generating OS noise to clobber your collectives? -L From rgb at phy.duke.edu Fri Jun 20 11:18:04 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <596BD9EE-8C63-4BF1-BC73-2895AACA028A@sanger.ac.uk> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <596BD9EE-8C63-4BF1-BC73-2895AACA028A@sanger.ac.uk> Message-ID: On Fri, 20 Jun 2008, Tim Cutts wrote: > > On 20 Jun 2008, at 4:12 pm, Mike Davis wrote: > >> Reading code is IMHO one of the most important parts of learning to write >> good code. It doesn't matter if the target language is f77, C, or even >> (gulp) java. Reading well written code will help one to understand how to >> construct it. > > That's true. I thought I wrote reasonably well structured and maintainable C > until I saw some code written by an ex-colleague of mine, and realised just > how good some people are! > > It was far better than the majority of code I've seen, and that includes > almost all the open source stuff I've had to grub around in. Also, use version control and NEVER throw ANYTHING you write away. My view of "object oriented design" and "code reuse" is to gradually build a large code base of code you wrote, however good, that solves various "frequently encountered problems" in one segment or another of the program. If you have a decent memory, eventually this becomes a fragment code bank. I'm constantly saying to myself "gee, I wrote a program a few years ago where I had to parse down a decision tree just like this one -- let me go grab what I did then and at least use it as a starting place." Sometimes I learn that my earlier code had bugs or sucked and I rewrite from scratch anyway. More often (by now) I just grab the code, change the variables and rearrange a bit (or import a subroutine set for e.g. managing linked lists the way I prefer them or the like) and I'm done. And I would often KILL for a good, simple piece of code to accomplish certain tasks even today. When I was getting started with writing daemons and raw networking, some pieces of example code saved my life. I've looked repeatedly for a userspace "ping" fragment that I could use to check for a host being up in a lightweight way and without suid root (feel free to direct me -- nmap has a "sort of" solution, but I'm still looking:-). Examples of code that use various library calls are often worth a week of staring at often indifferent documentation. rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From rgb at phy.duke.edu Fri Jun 20 11:45:42 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <20080620143621.GA31402@synapse.neuralscape.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> Message-ID: On Fri, 20 Jun 2008, Karen Shaeffer wrote: > On Fri, Jun 20, 2008 at 02:41:42AM -0400, Robert G. Brown wrote: >> >> c) some mix of mentoring by a guru and working your ass off. A bright >> and motivated person can teach themselves C, but I'll be damned if I >> think that they can teach themselves to code C >>well<< in less than >> years of painful mistake-driven lessons unless they have a guru or some >> other way of learning some of the simple wisdom that makes it possible >> to turn out decent, readable, code. And even with a guru and working >> quite hard, it still takes years, just fewer years. > > Today it is very straight forward. The 2.6.x linux kernel is an exemplary > body of superb C programming code. All you need is motivation and a network > connection. And the linux kernel email development lists have hundreds of > programming guru's pontificating and standing on soap boxes daily. I can't tell if you're joking or not...;-) I usually start newbies off on things that are a bit simpler, like forming the frequency histogram of the letters in an input text file, trying to compute the date of Easter for the next hundred years, solving a set of linear equations, integrating an ODE and writing out the result for plotting, stuff like that. After a bit I might move them up to some code that does some pointer magic or struct magic or both. I do make them get e.g. King Ables and Graham Glass or Kernighan and Pike because it is hard to write good C without some knowledge of Unixoid operating systems and pipes, forks, sockets, systems calls. But having them look over kernel source to learn to program -- isn't that a bit like teaching a five year old to learn to read with the complete works of Shakespeare or a three inch thick guide to writing, grammar and style? Yes, arguably it's all there, but it does help to creep up on it gradually...:-) rgb > > Thanks, > Karen > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From kilian at stanford.edu Fri Jun 20 14:08:04 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485AA481.2030000@vcu.edu> References: <485920D8.2030309@ias.edu> <485AA481.2030000@vcu.edu> Message-ID: <200806201408.04839.kilian@stanford.edu> On Thursday 19 June 2008 11:25:05 am Mike Davis wrote: > According to one weapons designer the only safe way to use it was to > fire from a hilltop into a valley from a jeep and then drive like > hell into the next valley. Or... there's also the so-called "home appliance nuke evasion manoeuvre", as demonstrated in "Indiana Jones and the Fridge of Nuclear Doom". But, obviously, you need a fridge. http://blog.uwinnipeg.ca/ius/archives/003477.html -- Kilian From peter.st.john at gmail.com Fri Jun 20 14:31:49 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <200806201408.04839.kilian@stanford.edu> References: <485920D8.2030309@ias.edu> <485AA481.2030000@vcu.edu> <200806201408.04839.kilian@stanford.edu> Message-ID: The destructive radius of Little Boy was about total, up to about one mile radius, and tapered down to light at about two miles. So being in a lead-lined steel container at 2000 meters might be OK for Indiana. In all action movies, blasts throw people unhurt for long distances; when that much force (to impart that much momentum) would kill you. That part is just conventional Hollywood. I could teach RGB to kick me so that I fly through the air as in a Bruce Lee movie; it's a stunt, and real kicks reallly hitting drop you like a sack of potatoes, I've seen it. But not in movies. Similarly bullets, they drill holes in you, if they pushed you through the air the recoil would do the same to the shooter. As for the scene's good taste I can't say, I haven't seen the movie yet :-) Peter On 6/20/08, Kilian CAVALOTTI wrote: > > On Thursday 19 June 2008 11:25:05 am Mike Davis wrote: > > According to one weapons designer the only safe way to use it was to > > fire from a hilltop into a valley from a jeep and then drive like > > hell into the next valley. > > > Or... there's also the so-called "home appliance nuke evasion > manoeuvre", as demonstrated in "Indiana Jones and the Fridge of Nuclear > Doom". But, obviously, you need a fridge. > > http://blog.uwinnipeg.ca/ius/archives/003477.html > > -- > > Kilian > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080620/e8881c16/attachment.html From perry at piermont.com Fri Jun 20 15:05:18 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485BA348.5020303@aplpi.com> (stephen mulcahy's message of "Fri\, 20 Jun 2008 13\:32\:08 +0100") References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> <485BA348.5020303@aplpi.com> Message-ID: <87iqw3zrs1.fsf@snark.cb.piermont.com> stephen mulcahy writes: > Perry E. Metzger wrote: >> It is, to some extent, a question of how many people are interested in >> a particular attack vector. Internet Explorer is a major attack vector >> for people who make money at this, so they work hard finding the bugs >> in it, of which there are an apparent endless number. I believe that >> more than 250 days last year, Internet Explorer had a known but as yet >> unpatched vulnerability. That's why the overwhelming majority of >> Windows boxes are zombies, including almost certainly most of yours >> unless you are a really unusual sysadmin. > > I'm reading this to mean that you think most Windows boxes on most > networks are zombies - is that right? Most are infected with something -- spyware, keyloggers, etc. I may be slightly wrong on zombies -- that number might be a bit smaller. I haven't checked the stats in a while. But, yes, it is an insane fraction of the machines out there. >> If you're smart, you're listening on: >> >> * DNS, with bind configured to run chrooted and unprivileged >> * sshd running with priv sep >> * ntpd running chrooted and unprived (though not all OSes will allow >> you to do that.) >> * maybe SMTP via postfix, which runs chrooted and unprived >> * and NOTHING ELSE. >> >> And if you're really smart, those daemons are further tied down with >> various bondage and discipline equipment like apparmor or SE Linux or >> what have you. > > Ouch, it's a never-ending battle isn't it? Yup. >> If you really believe your local net is very good, run a sniffer on it >> for a while -- or talk to someone who's job is to run one. > > I'd love to know how anyone with skype running on their network > manages to see much of anything from the firehose that is a packet > trace (and our network is small). Again, maybe it's just a question of > time. You tell tcpdump to filter out the skype packets. :) More seriously, if you want to analyze a trace from a big network, you record a couple of minutes worth and you have automated tools tease it apart. You don't try to do it by hand. There's lots of good open source stuff out there now that will do what you need. Perry From perry at piermont.com Fri Jun 20 15:07:54 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <20080620170738.GA12172@nlxdcldnl2.cl.intel.com> (David N. Lombard's message of "Fri\, 20 Jun 2008 10\:07\:38 -0700") References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620170738.GA12172@nlxdcldnl2.cl.intel.com> Message-ID: <87ej6rzrnp.fsf@snark.cb.piermont.com> "Lombard, David N" writes: > Yeah, I get that all the time too, I use an /etc/hosts.allow filter to > temporarily block those idiots after three such attempts. If you really care to do that, there are scripts that automate black holeing the kneebiters for you. As I said, I have professional reasons for not running them. Perry From perry at piermont.com Fri Jun 20 15:57:35 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: (Robert G. Brown's message of "Fri\, 20 Jun 2008 11\:19\:16 -0400 \(EDT\)") References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> Message-ID: <87abhfzpcw.fsf@snark.cb.piermont.com> "Robert G. Brown" writes: > On Fri, 20 Jun 2008, Perry E. Metzger wrote: >> That limits the number of attempts that may be made against your >> particular machine. At the same time that they're attacking your >> machine, that one instance is attacking a vast number of other >> randomly selected boxes. There are also a vast number of the things >> running out there, so in the long run, they succeed quite a bit of the >> time. > > Yes, but only rarely, on a site that is actually registered with > nameservice, I don't understand what you mean by that... >>From the evidence, they almost never succeed in the US, A few days ago I informed an ISP in Florida that one of their servers was running an ssh brute force agent, and I find that sort of thing often enough that I don't think you're correct. > when they do they almost NEVER succeed on a machine that is > professionally managed, The ISP seemed reasonably professional. Unfortunately they have to let their web hosting customers log in with passwords... > It isn't clear how many of the machines that are cracked and > participating in the botswarm attack are linux based even globally. A lot, but it is hard to say because the number of Linux boxes is so small compared to Windows boxes. I could ask someone who knows statistics if you're interested -- let me know. > Of course a "real" professional ubercracker is damn near invisible once > they get in and encapsulate, even on a Unix or Linux box. Almost all root kits are professional these days -- they're written by funded full time developers, and they're usually very good. > In memoriam I to this day do try to watch network traffic from passive > third party systems from time to time, especially if a system is > behaving oddly, although with a switched network this is somewhat more > difficult than it used to be That's why they make passive taps out there. You put one on your uplink and run Snort or something similar on a box attached to it. > All of which is a cost. Solving the cost-benefit equation for security > is nontrivial and often leads to a fairly unique "best practices" for > pros such as "read your logs", "get to know 'normal' on your systems", > "watch for anomalies, deviations from normal", "read your logs", "watch > your users and educate them gently", "tolerate not evil that lives in It is actually a lot easier than that. I read my logs mostly to find out what's new and not to keep my machines defended. Unfortunately, living that way requires a lot of smarts to make things run safely. I usually recommend automating the heck out of everything, and I also don't recommend trying to educate users -- it is much like trying to shovel the ocean. However, this isn't a security list so it is portably not appropriate to discuss this much here. >> There are many other similar sorts of things out there that you are >> less aware of -- things knob-twisting on ebay, yahoo and gmail >> accounts (I would explain why people want to steal something that's >> free but its a long story), online banking, etc. > > No need to explain. There are clear advantages to bot-spammers in > autogenerating accounts on systems that can be used to launch bot-spam > attacks (or can be convinced to accept the mail messages that make up > bot-spam). Actually, it is mostly so that account reputations can be stolen. On eBay that's an obvious win -- it turns out that there are reasons to want yahoo accounts that have been open for a while too. For spamming via yahoo, people just use captcha crackers or the porn site mechanical turk trick. >>> Sure, it goes on and on. I don't really LIKE seeing this, especially on >>> a server with sensitive information, but that is precisely why one >>> configures such servers with tight controls and runs a password checker. >> >> I don't run a password checker. I simply have no mechanisms in use >> that *use* passwords, so it is irrelevant. Then again, I'm far more >> paranoid than most people are. Professional hazard. > > Well, in many locations networks have to "serve the public", that is, a > set of users. Users who want to be able to work system to system at > work, work system to system from home, work from remote sites in China > where they happen to be attending a conference. If they can't use public key auth, give 'em secure ids or something similar. Works fine or such purposes. Passwords are dead. > Cracking happens. Such is life. Almost nothing you can do on an open > network with hundreds of users will completely prevent it, although if > you want to spend money like water you can significantly reduce it. You can make it rare enough not to worry much if you are willing to do fairly mundane things, but most people don't. >>>> It is true that this is only one of many modern attack vectors and >>>> that buffer overflows, drive by malware downloads into browsers, etc., >>>> are all far more common ways in, but you will indeed get hacked by >>>> automated systems if you leave an sshd on and have accounts with weak >>>> passwords. >>> >>> Agreed as well. Which is why professionals generally check for weak >>> passwords (as do most of the password tools nowadays). >> >> Nah. Professionals don't even use passwords any more, as I said. (I'm >> forced to use them on most e-Commerce sites because just about no one >> does client cert based authentication, but my machines don't use >> passwords. Most of my buddies machines don't use them either.) > > I meant "professionals managing networks or clusters of systems for > users", not "professionals running their own servers or running their > own network or cluster for themselves". It is fairly rare in the circles I travel in for people to use password based remote access. Hardware tokens and multi-factor auth took over years ago. I'm talking about systems with tens of thousands of users doing remote access, too. > I avoid passwords myself when I can and choose strong ones when I > can't and cross my fingers either way. But a professional sysadmin > managing a corporate, university, private, public network almost > invariably has to support userid/password based access, Not really, no. Tokens are cheap for remote access. > usually to "most" of the network they manage. How would you suggest > that 15,000 students and 20,000 full time employees authenticate to > access Duke's resources from absolutely anonymous hosts that could be > anywhere on the global internet at any point in time Hardware tokens, and multi-factor auth. The tokens these days fit on a key ring. I know places with more users than you have and they're happy with the solution. It is reasonably economical. I also realize it won't happen on your network, but that's probably not because it is economically infeasible. >> Any given instance only tries a small random subset of the phase space >> at a time on a given machine. However, the worms are pretty good about >> picking random subsets. Over time, you will find they slowly will find >> most of the more weakly defended boxes, just like water wearing down a >> stone. > > The phase space is enormous. I mean really, really enormous, I can multiply too. It is not "big enough", though, and educating users doesn't work. > IMO, we are quite possibly moving towards a "healthy world" on the > internet. The problem we face is understandable, the linux solution is > remarkably robust (and could be and is being made even more so). I have my doubts. The problem appears to be getting much worse with time from where I stand. I probably see more horror on a regular basis than you do, though. >> People do all sorts of things -- port knocking, filters that >> autoblacklist IPs that have too many errors, etc. It is simpler and >> safer just not to use passwords. > > Please explain how this works to me. You keep saying that you don't use > passwords. How then do you and your users access your systems from > remote anonymous sites (home or whereever)? I talked about site wide solutions above. For myself, I personally am too paranoid to use a keyboard I've left out of my control for more than a trivial amount of time. I use ssh with public key auth only. >>> They are almost never driven by a real portscanner-driven attack, >> >> Only because they're not attacking you in particular. If someone had >> reason to attack you in particular, then you would have more to worry >> about. Again, I find it is better just to make sure that there is >> nothing to attack. > > There is always something to attack. You can narrow the aperture a lot. Anything you don't run can't be exploited. No one can guess passwords if you don't allow people to use them. The principle can be extended a lot... > No arguments, except that in a lot of cases I'd cut this down to sshd > and nothing else for a user client. However, we're back to the > tradeoffs required by real world management. Web servers ARE useful and > are generally mandatory in many environments. Mine run chrooted, unprived and heavily caged, and I generally don't run Apache. > And so is NFS. And NFS often comes with a certain amount of > baggage, although most people will block NFS-related ports at the > firewall and limit their exposure to internal exposure. I'm a believer in a different kind of firewall -- the kind that blocks everything except the small number of things you know you need to let through. One wants a firewall, not a firesieve... :) > We have some pretty good people at Duke, actually. I do NOT think that > the physics department is in any sense "filled with exploited boxes" and > the few that turn up can be instantly understood in terms of their > distance from the centrally managed and monitored core (Windows > machines, in other words). If you exclude the overwhelmingly most popular OS, you're doing fine, in other words. :) The problem is, Windows takes all the hit these days because most people have it so the pros who get paid attack Windows. If most people ran OS X, the pros would be attacking OS X and you'ld be talking about how secure you were except for the Macs... > I think that our problem is that I have been prepending the word LINUX > mentally to our discussions. LINUX networks are not so commonly set up > by people who know nothing. Ubuntu is rapidly helping with that. :) Perry -- Perry E. Metzger perry@piermont.com From perry at piermont.com Fri Jun 20 18:44:20 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485BB636.5060106@scalableinformatics.com> (Joe Landman's message of "Fri\, 20 Jun 2008 09\:52\:54 -0400") References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <485BB088.7060406@scalableinformatics.com> <485BB24E.7030807@aplpi.com> <485BB636.5060106@scalableinformatics.com> Message-ID: <87y74zzhmz.fsf@snark.cb.piermont.com> Joe Landman writes: > I wrote a tool called "danger" that parses the ssh logs, the > /etc/hosts.deny logs, and makes ... recommondations about what to > add. Based upon who has been attacking you. This pre-dated denyhosts > by a bit. I still run it, and it gives me a nightly summary of the > bad guys. As my post said earlier, there are probably 30 such tools out there already. Generally speaking, I don't think they're worth using, except perhaps as a way to keep your logs a bit emptier. The easiest way to get safety is just to turn off password based login via sshd and only allow public key, kerberos, or other methods that do not involve reusable credentials that go over the wire. Perry -- Perry E. Metzger perry@piermont.com From perry at piermont.com Fri Jun 20 18:49:05 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> (Lawrence Stewart's message of "Fri\, 20 Jun 2008 14\:03\:25 -0400") References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> Message-ID: <87tzfnzhf2.fsf@snark.cb.piermont.com> Lawrence Stewart writes: > There are many positive things to say about the Linux kernel, but > "good code" is not one of them. I tend to agree. The quality levels tend to be very mixed. > Well that is too broad an indictment - There is good code in there, > but the level varies widely. > Have you <> at the TTY drivers? Have you counted the number of > "<< 9" s in the block code? > Tried to figure out which includes are actually active? Tried to > figure out which of the 17 ways to do > something is the "approved" one? There are worse examples, sadly. :( > I have one of the copies of the "Lions" book - the annoted sources > for V6. That is good code. Written by one or a very few exceptional > programmers, not an agglomeration of a zillion patches. Unfortunately, also in a pretty old C dialect, and written for an age where catching edge conditions wasn't worth it because you had to run in 64k and had no attackers to deal with. :( That said, the Lions book is in a prominent place on my shelf. There's a great book out there called "Code Reading" by Diomidis Spinellis that I'd recommend ahead of just jumping in to the Linux kernel. -- Perry E. Metzger perry@piermont.com From perry at piermont.com Fri Jun 20 18:51:30 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] How to do a single network install In-Reply-To: (Matt Lawrence's message of "Fri\, 20 Jun 2008 10\:32\:28 -0500 \(CDT\)") References: Message-ID: <87prqbzhb1.fsf@snark.cb.piermont.com> Matt Lawrence writes: > I needed a quick way to signal back to the kickstart server that my > install was done and to boot from the hard drive next time. Or, > PXEboot and then fall through to booting off the hard drive. So, I > wrote a quick and simple script. In the %post clause, I have: > > echo "done" | telnet 192.168.200.8 46758 > > Here's the script: Just an FYI, this is the sort of thing that netcat was built for. If you don't know about it, you might want to check it out. Perry From perry at piermont.com Fri Jun 20 19:01:05 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] security for small, personal clusters In-Reply-To: (Mark Kosmowski's message of "Fri\, 20 Jun 2008 12\:30\:26 -0400") References: Message-ID: <87lk0zzgv2.fsf@snark.cb.piermont.com> "Mark Kosmowski" writes: > What kind of security is recommended for the owner of a small personal > cluster? Where should the owner of a small, personal cluster go to > learn about security? Doing searches tends to give a few "head in the > sand" sites but predominantly seem to be oriented for the security > professional. Sadly, as a security professional I don't have very good book recommendations to give because I'm not the sort of person the books you want are aimed at. However, let me suggest that "simpler is better". Don't bother with anything exotic. Set your cluster up on the other side of your head node, allow access to the head node only over SSH with public key auth, and run very few services on the outside interface on the head node (preferably SSH and almost nothing else) and there will be very few ways to attack the thing. > I maintain a small 3 node personal cluster used for my part-time PhD > studies in chemistry. So far I've just kind of been hoping that I'm > too small to bother with and that the firewalls (Linux and switch) > between my boxes and the cable internet do something. There is no such thing as "too small to bother with" because the attacks these days are all automated and essentially target people at random. However, if you are behind a NAT box attached to a cable modem, and you haven't configured any incoming ports, and you keep your firmware in the NAT box reasonably up to date and keep the machines reasonably well patched, I wouldn't worry too much. > I do use ssh for cluster communications, and have disabled the > reknowned unstable services such as ftp. FTP isn't "unstable", it is just not secure. :) > RGB mentioned running services on non-standard ports. This seems like > a good idea to further reduce the probability of successful attack. Eh, if you're not allowing login except with public key auth (i.e. everything else is turned off in the sshd_config), I wouldn't bother. Here is one favorite trick of mine: run "netstat -A inet -a" on your boxes. Turn off every service you find listening on the network that you don't actually need -- mDNS, cups, etc, etc. If you get the machine down to essentials, you've taken care of half the problem. -- Perry E. Metzger perry@piermont.com From perry at piermont.com Fri Jun 20 19:03:04 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:14 2009 Subject: [Beowulf] Plan 9 In-Reply-To: (Peter St. John's message of "Fri\, 20 Jun 2008 13\:59\:08 -0400") References: Message-ID: <87hcbnzgrr.fsf@snark.cb.piermont.com> "Peter St. John" writes: > I was surprised to see Plan 9 deployed on a Blue Gene, > http://en.wikipedia.org/wiki/Blue_Gene#Plan_9_support. > I think of Plan 9 as the direct descendant of System V; Not even remotely related to System V, and Rob Pike would probably giggle uproariously at the idea. In fact, Plan 9 is sort of the anti-SysV. > Has anybody tried it? It is neat, but it is even more fringe than the normal fringe choices... Perry From diep at xs4all.nl Fri Jun 20 19:55:23 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] security for small, personal clusters In-Reply-To: References: Message-ID: <4C48DB6D-B5D4-42FB-9D16-507115B772B8@xs4all.nl> Just don't worry and live a happy life. Vincent On Jun 20, 2008, at 6:30 PM, Mark Kosmowski wrote: > What kind of security is recommended for the owner of a small personal > cluster? Where should the owner of a small, personal cluster go to > learn about security? Doing searches tends to give a few "head in the > sand" sites but predominantly seem to be oriented for the security > professional. > > I maintain a small 3 node personal cluster used for my part-time PhD > studies in chemistry. So far I've just kind of been hoping that I'm > too small to bother with and that the firewalls (Linux and switch) > between my boxes and the cable internet do something. I do use ssh > for cluster communications, and have disabled the reknowned unstable > services such as ftp. > > RGB mentioned running services on non-standard ports. This seems like > a good idea to further reduce the probability of successful attack. > I'll look into this in a couple weeks when I'm laid off and will have > more time on the IT side of things. > > Are there any good sites aimed at the small user to slowly learn a > little bit about security? Or are the only choices to do security > full-time or forget about it entirely? > > What sorts of security do other here have on their personal clusters? > Is maybe the simplest idea to just disable TCP/IP across the internet > network while not in use and enable it when it needs to be up? > > Thanks, > > Mark E. Kosmowski > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From gdjacobs at gmail.com Fri Jun 20 22:54:03 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485AA1D0.1080209@jax.org> References: <485920D8.2030309@ias.edu> <200806190945.21604.kilian@stanford.edu> <485A9336.8010303@jax.org> <200806191100.29026.kilian@stanford.edu> <485AA1D0.1080209@jax.org> Message-ID: <485C977B.6020102@gmail.com> Glen Beane wrote: > > > Kilian CAVALOTTI wrote: >> On Thursday 19 June 2008 10:11:18 am you wrote: >>>> To add some more OT stuff to this thread, I don't think a nuclear >>>> weapon has ever been used (or even considered being used) to kill >>>> troops on a battlefield. >>> look up "tactical nukes". These were the USA's only hope of >>> defending Europe from a Soviet ground invasion. >> >> Well, what would have been the effect of launching nuclear weapons to >> defend Europe in case of a Soviet invasion? They would have been >> either launched to where the Soviet troops actually were, ie, on >> Europe, with the main effect of wiping up the countries they were >> supposed to protect. Not so appealing. >> Or, and it's probably the most plausible scenario, they would have >> been aimed to USSR, and likely to major cities, where they would have >> killed mostly civilians, not troops. With the hope that the Soviet >> government would withdraw from Europe. > > > This is terribly off topic, but you are thinking of strategic nuclear > weapons. You couldn't aim a tactical nuke at a city in the USSR unless > you were a mile or so from the city. These were small rocket or > artillery fired warheads of less than 100 pounds. Tactical nukes are > not large enough to destroy cities. The main effect would not be wiping > out the countries they were meant to protect, the main effect would be > to wipe out large tank formations and make small areas temporarily > irradiated to block the movement of troops. Many (most?) scenarios had the Soviets making tactical and subtactical use of chemical munitions. With a largely unprotected civilian population, there wouldn't be many people left to collaterally kill anyway. >> That's why I think nuclear weapons are hardly a mean to kill military >> troops on a battlefield. > > Strategic nukes, no. Tactical nukes, yes. Now find an effective way of preventing a tactical exchange from escalating to a strategic exchange. -- Geoffrey D. Jacobs From geoff at galitz.org Sat Jun 21 00:44:36 2008 From: geoff at galitz.org (Geoff Galitz) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485C977B.6020102@gmail.com> References: <485920D8.2030309@ias.edu><200806190945.21604.kilian@stanford.edu> <485A9336.8010303@jax.org><200806191100.29026.kilian@stanford.edu> <485AA1D0.1080209@jax.org> <485C977B.6020102@gmail.com> Message-ID: <461CB2E232CA45A6981BD3207EBF444A@geoffPC> The MAD doctrine still applies. Attacking advancing formations with tactical nukes is still a far cry from a full-scale nuke exchange. The former is a battlefield tactic and places limited (friendly) military units in danger while the latter will destroy your labor force, production capabilities and so on. In spite of what we might think of how crazy those guys are behind the big red button the generals and politicians tend to realize these facts. If I might complete devolve the thread and go waaaay off-topic. Does anyone remember the movie Failsafe? http://www.imdb.com/title/tt0058083/ http://classicfilms.suite101.com/article.cfm/fail_safe Geoff Galitz Blankenheim NRW, Deutschland http://www.galitz.org -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Geoff Jacobs Sent: Samstag, 21. Juni 2008 07:54 To: Glen Beane Cc: Beowulf List Subject: Re: [Beowulf] Re: "hobbyists" [stuff snipped] >> That's why I think nuclear weapons are hardly a mean to kill military >> troops on a battlefield. > > Strategic nukes, no. Tactical nukes, yes. Now find an effective way of preventing a tactical exchange from escalating to a strategic exchange. -- Geoffrey D. Jacobs _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From eoin.mchugh at ichec.ie Sun Jun 15 05:11:10 2008 From: eoin.mchugh at ichec.ie (Eoin McHugh) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Powersave on Beowulf nodes In-Reply-To: References: Message-ID: <20080615121110.GA11985@ichec.ie> On Sun, Jun 15, 2008 at 04:26:24AM +0400, Mikhail Kuzminsky wrote: > We have many jobs in SGE at every time moment, and underload situation > (where it's reasonable to decrease CPUs frequency) is not the our > danger :-) So I'm thinking about simple stopping of all the > corresponding daemons. Hi Mikhail, There's not much point to running a powersave daemon. Even if a job is not CPU bound you'll find that lower power states will probably have a negative effect on your interconnect performance too. I don't run a daemon but I use the Torque prologue and epilogue to set the max and min power states using the /proc cpufreq interface before and after jobs. This results in a power saving when a queued jobs are waiting for additional nodes to free up. It's a measurably large difference when the system is being drained prior to a downtime. Our nodes are never shared beteen jobs making this easy to implement. Regards, -- Eoin McHugh ICHEC Systems From linus at ussarna.se Mon Jun 16 16:31:56 2008 From: linus at ussarna.se (Linus Harling) Date: Wed Nov 25 01:07:15 2009 Subject: NDAs Re: [Beowulf] Nvidia, cuda, tesla and... where's my double floating point? In-Reply-To: References: <1210016466.4924.1.camel@Vigor13> <48551E70.7070507@scalableinformatics.com> <4AF41375-3A13-4691-A2A1-D5B853FEC3A4@xs4all.nl> <20080615154227.u8fwdpn08ww4c40k@webmail.jpl.nasa.gov> Message-ID: <4856F7EC.7060000@ussarna.se> Vincent Diepeveen skrev: > > Then instead of a $200 pci-e card, we needed to buy expensive Tesla's > for that, without getting > very relevant indepth technical information on how to program for that > type of hardware. > > The few trying on those Tesla's, though they won't ever post this as > their job is fulltime GPU programming, > report so far very dissappointing numbers for applications that really > matter for our nations. Tomography is kind of important to a lot of people: http://tech.slashdot.org/tech/08/05/31/1633214.shtml http://www.dvhardware.net/article27538.html http://fastra.ua.ac.be/en/index.html But of course, that was done with regular $500 cards, not Teslas. From ktaka at clustcom.com Tue Jun 17 02:16:58 2008 From: ktaka at clustcom.com (Kimitoshi Takahashi) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] ECC support on motherboards? In-Reply-To: <87mymu6s3y.fsf@snark.cb.piermont.com> References: <87bq3bwd1n.fsf@snark.cb.piermont.com> <4828D537.2000200@scalableinformatics.com> <87prrrukib.fsf@snark.cb.piermont.com> <20080513165520.C5FC735A8E5@mail.scali.no> <87mymu6s3y.fsf@snark.cb.piermont.com> Message-ID: <4857810A.3050005@clustcom.com> Hi Perry, > So another question is, how can you reliably test any of this stuff? > It isn't like you can reliably induce single bit errors and see if the > hardware catches them. (A special memory module that let you test > would be a wonderful thing, but I've never even heard of such a thing.) > We scrathed off one of the plates to reliably induce single bit errors, so that we could submit EDAC patch for i3000. http://www.clustcom.com/content/view/89/32/ It's not in English, but I hope you'll know what you can do. Using the same way, my collegue is writing edac code for i3200. Of course, there is non destructive way: http://bluesmoke.sourceforge.net/testing.html But we wanted to be sure if it's really causing bit errors. If both edac and memtest86 didn't support the tested chipset, we wouldn't be able to tell if we really masked right pins and hence reliably causing bit errors. That was the case for i3000 and i3200. Please note that, pin arrangement is different for different type of memories. And we are not sure if this technique applies to different types of memory. -- Kimitoshi Takahashi From d.love at liverpool.ac.uk Fri Jun 20 09:04:42 2008 From: d.love at liverpool.ac.uk (Dave Love) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: SuperMicro and lm_sensors References: <20080618221153.GA24788@bx9.net> <1213860288.4784.15.camel@Vigor13> Message-ID: <87iqw4hz39.fsf@liv.ac.uk> John Hearns writes: > Can't help you on the lm_sensors front, we always use IPMI these days. For what it's worth, IPMI doesn't work in-band on SuperMicros Streamline-configured with opensuse 10.3. This fixes it after unloading the ipmi modules or rebooting: # ssh lvinfi100 cat /etc/modprobe.d/ipmi # The ipmi startup fails without this, and then ipmitool doesn't work. # module parameters are from # ftp://ftp.supermicro.com/utility/Supero_Doctor_II/Linux/README-IPMI.htm options ipmi_si type=kcs ports=0xca8 regspacings=4 From jbardin at bu.edu Fri Jun 20 09:21:13 2008 From: jbardin at bu.edu (James Bardin) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] How to do a single network install In-Reply-To: References: Message-ID: <485BD8F9.9080901@bu.edu> You might want to take a look at cobbler. http://cobbler.et.redhat.com/ IIRC, cobbler won't serve a pxe boot image to machine once it's been provisioned, until you set a flag to reinstall. -jim Matt Lawrence wrote: > I needed a quick way to signal back to the kickstart server that my > install was done and to boot from the hard drive next time. Or, PXEboot > and then fall through to booting off the hard drive. So, I wrote a > quick and simple script. In the %post clause, I have: > > echo "done" | telnet 192.168.200.8 46758 > > Here's the script: > > #!/usr/bin/ruby > > require 'socket' > > server = TCPserver.new('0.0.0.0', 46758) > while (session = server.accept) > text = session.gets > puts text > # p session.peeraddr > addr = "" > session.peeraddr[3].split('.').each { |n| addr += sprintf("%02.2X", > n.to_i) } > p addr > begin > base = "/tftpboot/pxelinux.cfg/#{addr}" > File.rename(base, base + ".done") > rescue SystemCallError > puts "Error occurred renaming #{base}" > end > session.close > end > > Yeah, lots of stuff hardcoded. Enjoy. > > -- Matt > It's not what I know that counts. > It's what I can remember in time to use. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- /* * * * * * * * * * * * * * * * * * * * * * * * * James Bardin - Systems Analyst / Administrator I Boston University Department of Electrical and Computer Engineering 8 Saint Mary's St, Room 305, Boston, MA 02215 Ph:617-358-2785 http://www.bu.edu/ece/it * * * * * * * * * * * * * * * * * * * * * * * * */ From gregory.warnes at rochester.edu Fri Jun 20 12:53:23 2008 From: gregory.warnes at rochester.edu (Gregory R. Warnes, Ph.D.) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Best training for rusty HPC skills? Message-ID: <0FBB78A3-6B0E-48FD-9455-E38C182BE482@rochester.edu> Hi All, I've just been appointed to head an acadmeic computing center, after an absence from the HPC arena affor 10 years. What conferences/ training/seminars/books/etc would be best for refreshing my awareness of the technologies and issues? Thanks! -Greg From anandvaidya.ml at gmail.com Fri Jun 20 18:13:06 2008 From: anandvaidya.ml at gmail.com (Anand Vaidya) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] NFS v3/v4 Benchmarks Message-ID: <200806210913.06648.anandvaidya.ml@gmail.com> Article on comparison between NFS v3 and v4 performance: http://www.linux.com/feature/138453 Regards Anand From brett at worth.id.au Wed Jun 18 18:31:02 2008 From: brett at worth.id.au (Brett Worth) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: OT: LTO Ultrium (3) throughput? In-Reply-To: <200806190019.m5J0J2SV014661@bluewest.scyld.com> References: <200806190019.m5J0J2SV014661@bluewest.scyld.com> Message-ID: <4859B6D6.2040004@worth.id.au> "David Mathog" wrote: > > dd if=/dev/zero of=/dev/nst0 bs=8192 count=10000 > >only moves 21.3MB/sec. David, Are you using LTO-3 tapes? An older gen tape could do it. Also I have seen instances where /dev/zero is no longer a device but has turned into a file. Maybe a big file due to an incorrect "dd of=" option at some stage. From then on your dd tests would be measuring your root disk performance. Regards Brett Worth From charliep at cs.earlham.edu Sat Jun 21 04:53:39 2008 From: charliep at cs.earlham.edu (Charlie Peck) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Best training for rusty HPC skills? In-Reply-To: <0FBB78A3-6B0E-48FD-9455-E38C182BE482@rochester.edu> References: <0FBB78A3-6B0E-48FD-9455-E38C182BE482@rochester.edu> Message-ID: <608154F6-8053-4530-9F6F-BABA99F2B1A8@cs.earlham.edu> version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on quark.cs.earlham.edu On Jun 20, 2008, at 1:53 PM, Gregory R. Warnes, Ph.D. wrote: > I've just been appointed to head an acadmeic computing center, > after an absence from the HPC arena affor 10 years. What > conferences/training/seminars/books/etc would be best for > refreshing my awareness of the technologies and issues? Consider the SC Conference's Education Program workshops and on-site program: http://sc08.sc-education.org/ charlie From csamuel at vpac.org Sat Jun 21 05:10:20 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] How to do a single network install In-Reply-To: Message-ID: <1540063496.42381214050220795.JavaMail.root@zimbra.vpac.org> ----- "Matt Lawrence" wrote: > I needed a quick way to signal back to the kickstart server that my > install was done and to boot from the hard drive next time. Our "installnode" script on the management node just watches the DHCP logs for the node requesting an address and sets it back to localboot N seconds after it sees it. If something goes wrong in the install we just rerun the installnode script and reboot the node. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From eugen at leitl.org Sat Jun 21 06:18:13 2008 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> Message-ID: <20080621131813.GR9875@leitl.org> On Fri, Jun 20, 2008 at 03:33:19PM -0400, Lawrence Stewart wrote: > More specifically for HPC, linux seems designed for the desktop, and > for small > memory machines. Many of the decisions that made good sense in the '90s In case of embedded-RAM nodes 'small' most likely would be ~MByte sized. In that case, Linux is indeed quite unsuitable for small memory machines. You'd end up burning almost all of your silicon real estate just for the OS, which would also be a large overkill for the kind of asynchronous message passing object soup you'd have on that hardware. (A minimal Forth or Lisp OS would be probably more suitable). > make no sense now. 512 byte disk sectors? 4096 byte pages? Randomly In case of embedded-RAM nodes, most nodes would have no disk at all. Nor swap. > allocated physical memory? "large page" patches on small pages? > All sorts of dinky little tasks waking up and generating OS noise to > clobber > your collectives? -- Eugen* Leitl leitl http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE From john.leidel at gmail.com Sat Jun 21 06:31:11 2008 From: john.leidel at gmail.com (John Leidel) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Best training for rusty HPC skills? In-Reply-To: <0FBB78A3-6B0E-48FD-9455-E38C182BE482@rochester.edu> References: <0FBB78A3-6B0E-48FD-9455-E38C182BE482@rochester.edu> Message-ID: <1214055071.5684.66.camel@e521.site> I would definitely recommend heading to Supercomputing [http://sc08.supercomp.org/]. The SC education committee has also setup a program geared toward educators that holds seminars on specific subjects. There are a few within driving distance of NY this summer. http://sc08.sc-education.org/workshops/index.php George Washington University in DC has several training courses in HPC. I believe they also have an "official" certificate program as well. http://www.gridswatch.com/ For a broad series of topics, I suggest going through some of the training materials provided by Ohio Supercomputing. [ http://www.osc.edu/supercomputing/training/ ] Read the news: www.hpcwire.com && www.insidehpc.com [selfish plug on that one...] && www.clustermonkey.net cheers john On Fri, 2008-06-20 at 15:53 -0400, Gregory R. Warnes, Ph.D. wrote: > Hi All, > > I've just been appointed to head an acadmeic computing center, after > an absence from the HPC arena affor 10 years. What conferences/ > training/seminars/books/etc would be best for refreshing my awareness > of the technologies and issues? > > Thanks! > > -Greg > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Sat Jun 21 05:27:40 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <200806201408.04839.kilian@stanford.edu> References: <485920D8.2030309@ias.edu> <485AA481.2030000@vcu.edu> <200806201408.04839.kilian@stanford.edu> Message-ID: On Fri, 20 Jun 2008, Kilian CAVALOTTI wrote: > On Thursday 19 June 2008 11:25:05 am Mike Davis wrote: >> According to one weapons designer the only safe way to use it was to >> fire from a hilltop into a valley from a jeep and then drive like >> hell into the next valley. > > Or... there's also the so-called "home appliance nuke evasion > manoeuvre", as demonstrated in "Indiana Jones and the Fridge of Nuclear > Doom". But, obviously, you need a fridge. > > http://blog.uwinnipeg.ca/ius/archives/003477.html Gawds, some people just don't get black humor. Or blackened smoking radioactive humor...;-) rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From rgb at phy.duke.edu Sat Jun 21 05:33:11 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <485AA481.2030000@vcu.edu> <200806201408.04839.kilian@stanford.edu> Message-ID: On Fri, 20 Jun 2008, Peter St. John wrote: > The destructive radius of Little Boy was about total, up to about one mile > radius, and tapered down to light at about two miles. So being in a > lead-lined steel container at 2000 meters might be OK for Indiana. > > In all action movies, blasts throw people unhurt for long distances; when > that much force (to impart that much momentum) would kill you. That part is > just conventional Hollywood. I could teach RGB to kick me so that I fly > through the air as in a Bruce Lee movie; it's a stunt, and real kicks Ooo, you can? When can we start? What if I, um, "miss"? > reallly hitting drop you like a sack of potatoes, I've seen it. But not in > movies. Similarly bullets, they drill holes in you, if they pushed you > through the air the recoil would do the same to the shooter. Yes, one does have to work a bit to suspend disbelief when watching movies these days. But not TOO much. Years of watching Jackie Chan, James Bond, Bruce Lee, Angelina Jolie, etc have gradually broken us into the idea that there are two completely distinct realities. The one where you can get hit on the head with a two by four and continue fighting, and the one where you can get hit on the head with a two by four and bits of your skull and brains spatter out onto the walls. They're even both hollywood realities! > As for the scene's good taste I can't say, I haven't seen the movie yet :-) Yah. Maybe this week. Maybe I'll wait for the video. rgb > > Peter > > > > On 6/20/08, Kilian CAVALOTTI wrote: >> >> On Thursday 19 June 2008 11:25:05 am Mike Davis wrote: >>> According to one weapons designer the only safe way to use it was to >>> fire from a hilltop into a valley from a jeep and then drive like >>> hell into the next valley. >> >> >> Or... there's also the so-called "home appliance nuke evasion >> manoeuvre", as demonstrated in "Indiana Jones and the Fridge of Nuclear >> Doom". But, obviously, you need a fridge. >> >> http://blog.uwinnipeg.ca/ius/archives/003477.html >> >> -- >> >> Kilian >> > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From gerry.creager at tamu.edu Sat Jun 21 07:50:11 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <461CB2E232CA45A6981BD3207EBF444A@geoffPC> References: <485920D8.2030309@ias.edu><200806190945.21604.kilian@stanford.edu> <485A9336.8010303@jax.org><200806191100.29026.kilian@stanford.edu> <485AA1D0.1080209@jax.org> <485C977B.6020102@gmail.com> <461CB2E232CA45A6981BD3207EBF444A@geoffPC> Message-ID: <485D1523.5000007@tamu.edu> Geoff Galitz wrote: > > > The MAD doctrine still applies. Attacking advancing formations with > tactical nukes is still a far cry from a full-scale nuke exchange. The > former is a battlefield tactic and places limited (friendly) military units > in danger while the latter will destroy your labor force, production > capabilities and so on. In spite of what we might think of how crazy those > guys are behind the big red button the generals and politicians tend to > realize these facts. While I'm less confident of our politicians (although I do have faith in our current Secretary of Defense: He left our University to take that slot from Rumsfeld and I knew him as a smart, thoughtful, insightful guy), I have a lot of faith in our generals. > If I might complete devolve the thread and go waaaay off-topic. Does anyone > remember the movie Failsafe? > > http://www.imdb.com/title/tt0058083/ > http://classicfilms.suite101.com/article.cfm/fail_safe Slim Pickens was a MASTER! > -----Original Message----- > From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On > Behalf Of Geoff Jacobs > Sent: Samstag, 21. Juni 2008 07:54 > To: Glen Beane > Cc: Beowulf List > Subject: Re: [Beowulf] Re: "hobbyists" > > [stuff snipped] > >>> That's why I think nuclear weapons are hardly a mean to kill military >>> troops on a battlefield. >> Strategic nukes, no. Tactical nukes, yes. > Now find an effective way of preventing a tactical exchange from > escalating to a strategic exchange. > -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From rgb at phy.duke.edu Sat Jun 21 14:09:37 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists"es In-Reply-To: <87abhfzpcw.fsf@snark.cb.piermont.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> <87abhfzpcw.fsf@snark.cb.piermont.com> Message-ID: On Fri, 20 Jun 2008, Perry E. Metzger wrote: > > "Robert G. Brown" writes: >> On Fri, 20 Jun 2008, Perry E. Metzger wrote: >>> That limits the number of attempts that may be made against your >>> particular machine. At the same time that they're attacking your >>> machine, that one instance is attacking a vast number of other >>> randomly selected boxes. There are also a vast number of the things >>> running out there, so in the long run, they succeed quite a bit of the >>> time. >> >> Yes, but only rarely, on a site that is actually registered with >> nameservice, > > I don't understand what you mean by that... A significant fraction of the sites that attack do not resolve as hostnames. For example: rgb@lucifer|B:838#host 61.144.122.38 ;; connection timed out; no servers could be reached If I work very hard (WAY harder than it is worth -- every minute one spends on this is "cost" in the security game) I can run traceroute, whois, and so on and determine that this is PROBABLY an "unused" address on a block of addresses in Beijing (61.144.122.109 resolves to qinghuishiye.com in Beijing). Perhaps it is an address belonging to this company; perhaps it is an illegal tap and is an address being hand set to route by some enterprising young Chinese person. Who can say? Who really cares? >>> From the evidence, they almost never succeed in the US, > > A few days ago I informed an ISP in Florida that one of their servers > was running an ssh brute force agent, and I find that sort of thing > often enough that I don't think you're correct. I might be wrong, sure. Anecdotal evidence, and truthfully it isn't worth my time to SERIOUSLY analyze logs from enough machines for enough time to come up with an authoritative or even statistically valid answer. On the one host I can easily check up on at this moment without wasting still more time, making a single pass on the current /var/log/secure, four out of six attacker addresses have no reverse mapping (that is, either "unregistered" altogether or at the end of some ISP's DHCP space where they don't bother establishing a reverse lookup). One of these, as I establish above, is PROBABLY in Beijing judging by the routing. Two have addresses with a reverse lookup. Of the addresses with reverse maps established and a discoverable name and whois record, one is in ecuador, the other in korea. I would bet five bucks even money that all six of today's attackers are outside the US. Still, that doesn't mean that the never succeed in the US -- that's poorly put. It means that -- assuming that the casual sample above is reasonably extrapolable -- the odds are perhaps 10 to 1 (or 3 to 1, or 8 to 1, or 15 to 1 -- small sample, big expected variance) that any given attacking host is outside the US. That may still leave a large NUMBER of attacking hosts inside the US, of course. Given the probable prevalence of computers inside the US to outside the US -- especially in the relatively small, relatively poor countries typically represented in my anecdotal attacker pool, either the probability of success in the US is in fact dramatically reduced (as I expect that there may well be as many computers on the Internet in the US and Europe as there are computers in all the more prevalent attackers combined) or some assumption made above is dramatically wrong. For example (to do the Bayesian analysis in more detail) if we assume that there are 3x as many internet-connected reverse-lookup resolvable hosts in the US and Western Europe as there are in the complementary set or countries (whether or not their addresses can be resolved), but 90% of all attacks com from the latter set, then if I'm doing my arithmetic correctly -- always subject to doubt;-) -- it is 27x more likely for a non-US-WEurope host to be attacking than it is for a US-WEurope host. That doesn't mean that the e.g. Chinese or Korean hosts are compromised, of course. They could easily be primary -- people working this as a job. $5 for every host you crack in the US and successfully insert the following spambots selling the following products. The point is that even with their indifferent skills and determination, every ISP in the US has to sign a whole set of AUAs to join the Internet at all, and every one of those from the toplevel backbone sites on down has explicit rules and sanctions associated with spam and bots originating on subnetting hosts. Those rules and whois make it "mandatory" to try to police your hosts, with the threat of removal (effectively death, to an ISP) if you can't do an acceptable job. And people DO try to police their hosts, again within reason. Sure, there are networks that are bleeding wounds, but they fairly quickly end up on mail blacklists, customers complain, the ISP is shocked into taking more professional action, the problem clears up, and things work again. Problems in the US with a modest amount of responsibility and accountability TEND to have a reasonably short lifetime, because people like you and the many other sysadmins I know WILL take the three minutes it takes to contact the whois person of the responsible entity IF we can find that entity in no more than two minutes of effort, which is usually the case for reverse-resolvable names and is a complete waste of time trying if not (even if occasionally you can run one down). My own touchstone process isn't /var/log/secure, actually. It is SPAM. I have the "honor" of having had a spambot-grazable email address fairly prominently represented on this very list for eleven or twelve years now, and while I'm probably not a world record holder I get a shitcan full of spam every day. The tiny fraction that makes it past spamassassin is still tens to low hundreds of messages a day. My highwater mark was something like 10 MB of spam a day (with days of 20 MB, just to little old me). Since then, I've been rebuilding my personal webpages with my email address obfuscated and Duke has been prescanning and eliminating the worst of the blacklist spam before it gets to SA, and I'm down to only a MB a day or thereabouts, plus leakage. To see where the viral spambots currently live, I just have to toggle on headers for a second before killing a spam message, or filter out my spam garbage can from the last month. Here, I could generate impeccable statistics as I am a virtual coal-mine canary, (although at the moment my sample is unfortunately biased as I'm no longer getting 80% of the spam even to where I can filter it) but it is still in the why bother category. 22% of what gets through has probably forged addresses. 31% is unknown (not resolveable). 10% is from Russia (and registered). 7% is from Brazil. 3% from Argentina. 3% from India. Tiny fractions come from China, Korea, Taiwan (Ha! -- look at those unresolvable and forged addresses...;-). An amazing 12% comes from Italy, of all places, and a solid 6% comes from Turkey. Just adding up the fractions from THESE registered foreign addresses (there are more) and we've got nearly all of the spam that COMES from a registered address at all coming from a registered FOREIGN address. This is no doubt a biased sample at this point because I don't know the prefiltering parameters -- maybe they're removing all the US addresses and letting through only unregistered, forged, and foreign addresses. If, OTOH, the filter isn't biased w.r.t country, I think that it is safe to say that AT LEAST 90% of all viral spam originates outside of the US and most of Western Europe (for shame, Italy!). Given the probable bias in the numbers of connected systems (surprisingly difficult to find aggregate numbers, sorry -- Korea has the highest percentage of personal ownership but the US has the raw numbers even before counting the corporate machines) I think that it is fairly safe to conclude that -- unsurprisingly, really -- all of the problems associated with unmanaged machines: virus infection, spambot infection, spyware, worms, phishing, snooping, man-in-the-middle -- are far, far worse in countries other than the US, Canada and Mexico (not a lot of Mexico, perhaps surprisingly), most of Western Europe, Australia -- the "first world" of yesteryear. Personally, I wouldn't be suprised if Windows is sold pre-hacked in e.g. China -- get your copy of stolen Windows with our own special spyware and spambots preinstalled to use your network when you aren't, only a dollar...;-) The conclusion is that the antispam measures taken in this country are, for the most part, actually working. There are doubtless many incidents, but if so they have a short lifetime and are relatively quickly x'd out of the internet until they are resolved. Antispyware measures I can't address -- linux being mostly not vulnerable and easy to audit, I've never had reason to think my machines are infected, and Windows (even my own Windows systems) I truthfully don't give a rat's ass about. I'd believe any evil rumor until it was proven wrong by an unbiased, expert, third party. And hey, maybe it is true that there ARE thirty million unwilling US machines in the russian mafia supercomputer that we kicked around (that's close to as many as there are in all of Korea) just waiting to to be turned to Evil Purposes, but if true, I'll bet that a tiny, tiny fraction of them run Linux. >> when they do they almost NEVER succeed on a machine that is >> professionally managed, > > The ISP seemed reasonably professional. Unfortunately they have to let > their web hosting customers log in with passwords... "Have to let"? > If they can't use public key auth, give 'em secure ids or something > similar. Works fine or such purposes. Passwords are dead. Yeah, Bill Gates (among others) said something like that back in 2004. I confess to being deeply skeptical. Really. The SecureID solution has been around for a long time at this point. It was a PITA a decade ago. It is a PITA now. Expensive, too. And then, people have to authenticate to so MANY things nowadays. I have to authenticate to my cell phone. To order pizza. To do banking online. To shop at X, Y or Z. Then there is logging onto systems I work on -- something that IS possible for me without a password. The problem there is that many of the systems I'm logging in from are laptops (I have two personally, about to make that three). The laptops themselves then become a security risk if they are stolen, so I tend to LEAVE passwords on WITH ssh on servers that are likely to have sensitive data on them -- a cracker may get my keys, but they will likely not be able to exploit them in the window before I change them even if I don't discover the theft for a day or a week. I personally have no good crystal ball feeling for where authentication is going. The password, flawed as it may be, has been flawed in exactly the ways it is currently flawed in more or less forever -- the only difference is in how many characters you have to have to be immune to brute force attacks, which alas Moore's law eats into. All the alternatives I've heard of, however, either a) don't scale, at least without something really scary like an international personal keyserver system -- oh, wait, that doesn't scale well and has its OWN set of problems; or b) are horribly inconvenient and expensive; or both. Like SecureID. Passwords scale well, give people immediate and direct control, are cheap, are convenient. I think that there will be tremendous resistance to the point of just plain ignoring any attempt to change, but I could be wrong. We'll see. Maybe something really new will emerge. If you think differently, please advise and explain. Ideally with a discussion of scaling and cost-benefit analysis. >> Cracking happens. Such is life. Almost nothing you can do on an open >> network with hundreds of users will completely prevent it, although if >> you want to spend money like water you can significantly reduce it. > > You can make it rare enough not to worry much if you are willing to > do fairly mundane things, but most people don't. It IS rare enough not to worry much, except on consumer machines. Humans are innately lazy, natural optimizing systems. In any managed network, the sysadmins expend precisely enough effort (time and/or money, most of it other people's money when possible) to reduce the rate of successful cracking to where a) it covers their ass with their bosses; b) incidents are rare enough not to be a PAIN in their ass that never stops; c) the effort to prevent still more successful attacks starts to exceed the time saved by the prevented attacks. Of course people don't usually think about it quite that way. They just do it. Getting cracked all the time? Boss getting mad? Better up the spending on security, learn how to stop the attacks, institute a rigid policy against e.g. browsing the web from work or opening non-internal attachments, invest in mail prescanning and filtering. Maybe hire a guru. Never get attacked, everything fine (as far as you know)? You might be wrong, but as long as nobody complains and the work gets done and as long as you can document that you're doing what you CAN do or SHOULD be doing according to standard of practice, you don't really care. Or you do care, but don't think it is happening strongly enough to spend a month of effort and lots of money to find out. Cost-benefit is all that matters, not "security". Probability of loss times expected cost of loss, marginal cost of each preventive measure balanced against it weighted by expected marginal savings from fewer incidents. There are nominal exceptions to this -- mostly legally mandated ones -- but even there you simply add more costs (going to jail, massive fines, lawsuits) to the same process. I'm guessing that in professionally managed Unix/Linux operations -- like those maintained by most of the readers on this list -- most of the managers DO a lot of those mundane things already. Probably not precisely the same mix that you use, but then there are many ways to skin this particular cat, and variance is a good thing as it permits a more dynamic adaptation to the targeted variations tried out by hackers -- when one line of defense fails, there are others that the hacker may not have managed to cancel as effectively. Linux "out of the box" is far, far more secure these days than it used to be -- default rather closed, rather than open, selinux enabled, only sshd open if that, nag to install a prescanned root password and at least one prescanned password protected account. If one doesn't openly work to defeat its default security (or try to much, too ignorantly) a "normal" user of even an UNMANAGED (by a knowledgeable professional) box will not be terribly easy to crack from outside, I think. If they leave yum's nightly update on (the default) for e.g. fedora or centos and the box is left on, on a high speed internet connection (to maximally facilitate access) and completely unattended I doubt that anybody could crack it from the wire side in a year of trying. It's not IMPOSSIBLE, of course -- sshd can have its exploits too -- but again, the window would be open for a very short time and it's a very boring box. Not that I'd recommend this for a bank server or so on, where somebody might be lurking and trying hard to open the door and just waiting for an ssh exploit to give them the key -- of course not. I'm just pointing out that even a default numb-nut linux is pretty darn secure compared to any other operating system that has ever existed to be installed by individual users, and that in the hands of a professional the security level is LIKELY to only go up. Unless, of course, the professional requirement is to install a wiki, a blog server, a mysql database, a webserver, an NFS server, and six other open port applications on the single server that (naturally) contains all the personal information of all the participants, including their SSN and credit card numbers;-) I >>know<< that there are people that are just this dumb out there. The Duke Law School just permitted the SSN's of a bunch of law school applicants to be stolen, no so much because of a failure in SYSTEM security as because a staff person didn't realize that when you post them to certain web-based group chat tools (shared by a committee) they are publically available and bound to be grazed, indexed, and rendered instantly searchable overnight by webbots galore. Even with the best security model in the universe, it is hard to idiot-proof the world (even by educating them so they are no longer idiots). However, one has to think of it as an ecology as you noted. Evolution in action. Survival of the fittest, correlary death of the unfit. Self-correcting system -- lots of negative feedback. Even closing a barn door after a horse gets away may well keep you from losing future horses, but the loss of a horse doesn't always justify installing GPS-trackers in all your horses and hiring personal nannies carrying submachine guns to take them to water. It MIGHT justify installing a spring and latch on the door, and it definitely justifies publicly buffeting the ears of the careless horse-barn-door-non-closer and uttering in deep tones "Next time, close the damn door so the horse doesn't get out!" > It is fairly rare in the circles I travel in for people to use > password based remote access. Hardware tokens and multi-factor auth > took over years ago. I'm talking about systems with tens of thousands > of users doing remote access, too. > >> I avoid passwords myself when I can and choose strong ones when I >> can't and cross my fingers either way. But a professional sysadmin >> managing a corporate, university, private, public network almost >> invariably has to support userid/password based access, > > Not really, no. Tokens are cheap for remote access. I'll have to revisit this. I do know that people use them. There is one in the house for a particular site. My impression was a minimal cost of $10's of dollars per user on up, per site you want to access this way, plus a whack for the server side stuff. Is this reduced? Or is this what you call "cheap"? > Hardware tokens, and multi-factor auth. The tokens these days fit on a > key ring. I know places with more users than you have and they're > happy with the solution. It is reasonably economical. I also realize > it won't happen on your network, but that's probably not because it is > economically infeasible. It hasn't HAPPENED on Duke's general network. The last time I looked at e.g. secureid it was prohibitively expensive and a total PITA, but that was years ago. I know that they've reduced size and made the fobs fit on keyrings, but it is still another piece of hardware to carry around and track, times the number of places (distinct domains) you have to access in this way. Multifactor ties it to yet another piece of hardware, doesn't it, e.g. a cell phone? Both of these assume a strong, centralized, organization wide authentication system, which is likely reasonable for a corporation but rare at Universities. Duke has kicked this around before (I helped do the kicking) but I don't mess with enterprise security much anymore -- it is depressing and as far as I'm concerned the enterprise solution starts by saying "use only linux" anyway, and everybody else says NO, we have to use WINDOWS for this application or that application, and the security game is more or less lost before it properly begins. Fortunately Duke doesn't MANDATE the use of Microsoft products (even if Melinda Gates IS a Duke graduate:-) and a lot of Duke -- nearly all of the sciences and engineering, a lot of the students, a few other departments here and there -- runs almost exclusively linux. Which is why I keep saying -- we just don't have much of a problem here, and it isn't because we are ignorant fools who are all cracked and don't know it. There are some very smart and extremely paranoid systems people who work on campus, with enterprise-level tools looking for problems. Windows boxes get cracked all the time and become e.g. suppurating wounds of warez and copyright violations. They are typically discovered, the systems admins responsible are informed (if there are any -- Duke has a half-inside/half-outside wireless network required to facilitate student access and lots and lots of students and the vast majority of all campus incidents are in the dorms with student run Windows boxes, with a lesser number coming out of departments with minimal or shared windows management) and they are taken off the network until they are fixed within hours of discovery. Not quite minutes -- but then, we aren't really centrally managed, which gives us considerable defensive robustness at the expense of less immediate control. It may be time to kick the fob idea around again. I still think it will end up costing Duke a million dollars a year, easy, to implement it -- we're talking 30 to 50 thousand users of all classes, in an multi-operating system distributed management environment that LACKS a flat userid space (scalability). Issuing a fob per department one has access rights in would be insane. Duke DOES have a centralized auth facility (used to authenticate a variety of access rights to confidential material) and it would very likely be quite reasonable to use it there), but I don't know if the linux folks would really trust single-sign on authentication for regular logins from this facility, presuming the userid mapping problem could be or has been solved. IIRC linux still uses 16 bit uids which is an obvious and immediate problem without some degree of domain segmentation and id translation. >> IMO, we are quite possibly moving towards a "healthy world" on the >> internet. The problem we face is understandable, the linux solution is >> remarkably robust (and could be and is being made even more so). > > I have my doubts. The problem appears to be getting much worse with > time from where I stand. I probably see more horror on a regular basis > than you do, though. It sounds like it;-) I hope you don't mind my debating with you and disagreeing on some of the things you say, by the way. I'm not trying to flame or fight a war to prove I'm right, I'm picking your brains (in part, by seeing how you refute some of the things I say, in part by just listening to them). And I'm guessing others on the list are interested as well -- it may not be "specifically" cluster oriented, but clusters are nearly invariably openly accessible through at least one portal, and represent a potentially valuable resource once one gets through the portal(s) even if they don't contain valuable data per se (and sometimes they do). You actually sound like precisely the kind of wild-eyed paranoid that can be extremely valuable to any organization that is concerned about enterprise level security. It sounds like you have a security consulting business. Is that what you do? I may have a potential client for you if so, contact me offline. The one thing I haven't heard you address is the cost-benefit associated with any particular set of security measures, especially on a broad basis. For example, as I noted, Duke is quite heterogeneous, and I personally would have to declare jihad on anyone that recommended that we adopt a central management scheme as the first step towards enterprise security for reasons too numerous to mention (but ones that if you've been around a while I don't HAVE to mention as they are common to many enterprises:-). CBA requires numbers, numbers that justify your paranoia. In a "typical" university department (or corporate department, or small business, or whatever) running linux, how many successful attacks are there per registered IP number (one exposed to the whole internet) or whatever other metric you like? How many of the attacks succeed because of e.g. open ports with buffer overwrite attacks, sniffed/stolen passwords (in my anecdotal experience the number one point of entry on linux boxes), how many root promotions succeed, rootkits are detected on a boot from a USB key followed by a security scan, etc? What is the cost (average, estimated, whatever) of these incidents, including detection and repair? What is the cost (real, potential, whatever) of not discovering them? And so on. So far as my own experience is concerned, I've seen a fair number of attacks that succeeded over 20+ years through many channels, with stolen passwords overwhelmingly dominating over that time (although the problem was tremendously ameliorated when Duke went to ssh-only or ssl-only -- bidirectional encryption including the authentication sequence -- for remote access across the entire campus as rigid policy). That dropped the problem from several a year to a maybe one every few years. The worst incident I recall in our own department involved an exploit of the portmap maybe a decade ago -- this was pre-yum. Our sysadmin had quit to take a new job, I had taken over (again) because I could while we searched for a new admin (that would turn out to be Seth Vidal, hooray:-), and he'd left me with four unpatched systems that got nailed and rootkitted. This worst-case crack -- for our scale of operation -- took me two days to discover, deconstruct across the department, and clean up. Maybe another week of intense paranoia as I looked for leftovers, forced user password changes, and multiply scanned the servers (which were not cracked). Root promotion on our clients didn't really get you much unless it was on my system, and I may not sound like it but I'm a bit paranoid too. I've personally been cracked twice, lifetime, to my knowledge -- the first time by the TRULY ancient emacs bug and it wasn't my fault or on my system -- I actually caught the perp, tracked him back home (he was at Duke), contacted his sysadmin (a friend of mine) and got his wrist SOLIDLY whacked, by golly, but they wouldn't kick out a computer science grad student even for nefarious evil, sigh. The second time was my own sloppiness on my home network -- I ran an externally available webserver on my DSL line just for the fun of it, and failed to update apache and got slammed. But then yum came along and took the guesswork out of staying patched. Now, two days of my time -- or even a week -- cost Duke a truly pitiful amount (I'm embarrassed to say:-). Most cracks take even less time to resolve. A server crack might cost more, but we've never had one in our department, and the ones I've helped deconstruct in other departments WERE more expensive (in part because they spent as much as a week of my time then, along with a couple of other people's time to boot:-) but we're still talking a few thousand in opportunity cost time. Damn near "free" in real dollars, in other words. The ante has gone up with the stricter security requirements and legal liability issues of modern times, but at the department level we don't have too much exposure, especially since we can easily demonstrate due diligence and then some. So the CBA as >>I<< see it is that using SecureID for departmental access would cost some thousands of dollars for the initial setup and hardware, some thousands of dollars of "effort" to get it all going, and some (hundreds?) of dollars in ongoing hardware replacement and opportunity cost labor for maintenence a year. This would save us (assuming it prevented 100% of all successful cracks) -- a few hundred dollars in expended opportunity cost labor a year, based on historical costs. Plus, of course, an empirically small chance of a much more expensive cracking incident that penetrated our servers or caused real losses somehow (security rule number one being "keep good backups", hopefully making that probability rather small). So I'm still not seeing it. Every user is also inconvenienced to some degree by the system, having to carry their fob with them (and inevitably losing or forgetting it or breaking it), which is a nontrivial and poorly scaling cost right there. If empirically this changed -- cracking rates due to getting passwords went up -- we wouldn't have to be TOLD it implement something more expensive to reduce the rate. We'd do it just because at that time it would make sense to do it. IMO, the appropriate level and expenditure of security is individual to each individual network and group of users; it isn't just "everybody should use secureid" or any other particular measure. > For myself, I personally am too paranoid to use a keyboard I've left > out of my control for more than a trivial amount of time. I use ssh > with public key auth only. > I'm a believer in a different kind of firewall -- the kind that blocks > everything except the small number of things you know you need to let > through. One wants a firewall, not a firesieve... :) Amen, brother! Give us an Amen, everybody! Amen! Drill one hole through on port 5002 or the like that leads straight to the sshd and is otherwise used for a moribund or little used application (or nothing at all). Instant peace and quiet. Block all access to internal (LAN) ports like NFS from the outside, and keep a sucker rod handy for anybody that messes with it on the inside. Ditto for e.g. webservers. If possible, put them on the external network. Otherwise, drill a hole. If possible, virtualize. If not, be prepared to be vigilant(er) and paranoid(er). Add ports and measures and costs as makes sense, with the maxim being less exposure is always better, and ssh tunnels ports so why would anybody need anything more? (Not quite true or reasonable, of course, but true for experts, anyway...:-) >> I think that our problem is that I have been prepending the word LINUX >> mentally to our discussions. LINUX networks are not so commonly set up >> by people who know nothing. > Ubuntu is rapidly helping with that. :) > > > Perry I actually don't agree that linux would prove anywhere nearly as vulnerable as Windows has been historically even if they switched market share tomorrow, and Ubuntu was the only version of linux used. After all, as YOU pointed out, MS left Explorer unpatched for 9 months. NINE MONTHS! Say what? Find a similar exploit in (say) Firefox. Now MAYBE it wouldn't, in fact, be found on Wednesday and get patched everywhere by Friday. It MIGHT take to the following Monday, or even the following Wednesday. But who seriously thinks that it would take nine months? There are reasons why one would expect Linux to respond optimally rapidly for pretty much any exploit -- literally as rapidly as it is possible to respond. A huge base of world class programmers who use it, for example, many of whom would fix the damn bug themselves if it didn't get fixed in a timely way by others, if they relied on the tool or service. The fact that the code is right there and open for scrutiny. The fact that many of those world class programmers and system and network administrators sleep with a pistol under their pillows and sit with their backs to a wall in restaurants (and require occassional medication) as an external expression of their paranoia (ensuring that it is actually not that likely that an broad exploit would remain undetected for long -- I'll bet that there are plenty of places that maintain sentinal sites and externally monitor the hell out of them, a thing that VMware makes possible with a SINGLE SYSTEM -- you can actually run a network server in one VM, run its traffic monitor in another, and offer no ports at all in the toplevel OS). Virtualization is going to change everything, by the way. Very, very soon. I predict that the standard linux install will be: Linux (VM manager, also monitor VM). Think "this is my expert system monitor" that does absolutely nothing but watch network traffic patterns from slave VMs and manage access to resources as requested. | |-Linux (network services VM, completely disjoint from trusted space, chroot on serious steroids, quite possibly read only VM image and certainly snapshotted for instant restore to a known clean state). | |-Linux (userspace VM). Accesses network services as a trusted host on a completely imaginary internal network with one way trust. No way back to this VM from the services VM, lots of bells that ring on an attempt). | |-Linux (optional/additional VMs). Even a Windows VM. This is the only way I run Windows these days, when >>I<< have a choice. It is lovely. Boot by clicking a VMware button. Do what I need to do. Shut the sucker down. Decide whether or not to snapshot it, or update it, or just leave it frozen (annoying, actually, as it fails to update and write back to the system so it always complains about being out of date). Use Explorer only if your life depends on it -- firefox or galeon is Ctrl-Alt and a couple of clicks away. In fact, use ANY software under Windows only if life or money depend on it. And who cares -- if it is cracked as all hell, you can easily see it on the toplevel/router system by tracking its connections, you can easily fix it by backing up to the clean original image, and if you limit its access to userspace to a tiny window on essential shared storage, you leave virtually no probability of a windows exploit opening a path back to your linux workspace. If you are truly paranoid, I'd recommend giving this a try. It's easy to set up, costs you a tiny bit of performance (truly irrelevant on most multicores), and lets you set up and run your very own Raving Paranoid Gateway network traffic monitor on the toplevel linux install and otherwise leave anything from absolutely nothing open to sshd on any linux level or windows level you choose. It's really, really hard to crack a site invisibly when every IP number that talks to it or that it talks to is isolated in real time and compared to a list you set up and control and sets off all sorts of alarms if any sort of anomalous or unapproved pattern occurs. In fact, this isn't a bad setup for a cluster gateway system. There. It's even OT...;-) rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From rgb at phy.duke.edu Sat Jun 21 15:07:24 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <485C977B.6020102@gmail.com> References: <485920D8.2030309@ias.edu> <200806190945.21604.kilian@stanford.edu> <485A9336.8010303@jax.org> <200806191100.29026.kilian@stanford.edu> <485AA1D0.1080209@jax.org> <485C977B.6020102@gmail.com> Message-ID: On Sat, 21 Jun 2008, Geoff Jacobs wrote: >> Strategic nukes, no. Tactical nukes, yes. > Now find an effective way of preventing a tactical exchange from > escalating to a strategic exchange. Use them on an enemy that can't strike back in a strategic way and that nobody that CAN strike back in a strategic way cares about. For example (just for example, mind you): North Korea is stressed internally. They seek an external conflict and prepare to invade South Korea. The US prepares to defend it, and starts to lean on China to control NK. China looks at the monetary value of their relationship with NK (negative) and the monetary value of their relationship with us (enormous). They lean heavily on NK to pipe down and behave. NK, feeling betrayed, goes berserk. They blackmail one and all by threatening a strategic nuclear strike against Japan or Alaska or whoever they think they can reach. They get their ass kicked at the SK border. China stands silent, but NK pushes more and more forces at the border and threaten to overrun it before we can get enough forces there to block them. We draw a line in the sand and threaten the use of tactical nukes to stop them cold if they cross it. We REALLY lean on China to control their client. China tries, and is kicked out of NK altogether. They publically renounce NK (because of some sub rosa discussions about our 5000 nukes vs their 200, with Russia waiting to pick up the pieces of China and unlikely to be of any help). NK nuke's Tokyo. The world stands shocked. The president releases tactical nukes for theater use. Two days later, NK falls, its government captured. rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From perry at piermont.com Sat Jun 21 17:21:47 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists"es In-Reply-To: (Robert G. Brown's message of "Sat\, 21 Jun 2008 17\:09\:37 -0400 \(EDT\)") References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> <87abhfzpcw.fsf@snark.cb.piermont.com> Message-ID: <87iqw2uxno.fsf@snark.cb.piermont.com> "Robert G. Brown" writes: >> If they can't use public key auth, give 'em secure ids or something >> similar. Works fine or such purposes. Passwords are dead. > > Yeah, Bill Gates (among others) said something like that back in 2004. > I confess to being deeply skeptical. Really. The SecureID solution has > been around for a long time at this point. It was a PITA a decade ago. > It is a PITA now. Expensive, too. It is neither. I use SecureIDs quite regularly and it isn't difficult at all -- you just look at the device and type in the digits. What's so hard about that? It isn't that expensive, either, but if you're minimizing cost there are cheaper competitors and various challenge-response devices, and even non-hardware solutions. > And then, people have to authenticate to so MANY things nowadays. I > have to authenticate to my cell phone. To order pizza. To do > banking online. To shop at X, Y or Z. That's why they created client certs. Unfortunately few people use them. MIT does some good things with them, though. > Then there is logging onto systems I work on -- something that IS > possible for me without a password. The problem there is that many of > the systems I'm logging in from are laptops (I have two personally, > about to make that three). The laptops themselves then become a > security risk if they are stolen, That's why they invented encrypted partitions, and why ssh lets you encrypt your public key credentials. I haven't used password based credentials with any services in about ten years. I deal with a lot of machines, and I don't find the lack of passwords inconvenient. Between SecureID, public key credentials, kerberos, etc., there is really not much cause to use passwords over a network any more. >> Not really, no. Tokens are cheap for remote access. > > I'll have to revisit this. I do know that people use them. There is > one in the house for a particular site. My impression was a minimal > cost of $10's of dollars per user on up, per site you want to access > this way, plus a whack for the server side stuff. Is this reduced? Or > is this what you call "cheap"? $10-$50 a user is cheap compared to the salaries of even university sysadmins multiplied over all the hours of trouble that breakins cause. As I said, there are competitors that are cheaper -- if you really need a $5 solution, they exist. If you think you're spending less than $5 a user on related security problems over the amortization life of a device, you're kidding yourself in any real organization. Of course, if you are still over budget, one can also use mechanisms like Kerberos, and in a university environment that's quite achievable, and has no associated hardware costs at all. >>> IMO, we are quite possibly moving towards a "healthy world" on the >>> internet. The problem we face is understandable, the linux solution is >>> remarkably robust (and could be and is being made even more so). >> >> I have my doubts. The problem appears to be getting much worse with >> time from where I stand. I probably see more horror on a regular basis >> than you do, though. > > It sounds like it;-) > > I hope you don't mind my debating with you and disagreeing on some of > the things you say, by the way. I'm hard to rattle on this sort of thing. :) > I'm not trying to flame or fight a war to prove I'm right, I'm > picking your brains (in part, by seeing how you refute some of the > things I say, in part by just listening to them). You'll pardon me for not replying in greater detail -- you've written quite a lot and I haven't answered all of it. I'm on vacation this weekend and I'm attempting not to read overly much email -- my spouse would get mad if I spent more time on it. > You actually sound like precisely the kind of wild-eyed paranoid that > can be extremely valuable to any organization that is concerned about > enterprise level security. It sounds like you have a security > consulting business. Is that what you do? It is how I pay for things like my HPC habit, yes. > The one thing I haven't heard you address is the cost-benefit > associated with any particular set of security measures, especially > on a broad basis. Well, all security is about economics. There are certainly measures that are appropriate at a bank and utterly inappropriate at a university, both for reasons of the willingness of the user community to put up with various kinds of inconvenience and because of raw cost. This sort of thing becomes a very, very long discussion, though, and again, I'm on vacation. :) > Now, two days of my time -- or even a week -- cost Duke a truly pitiful > amount (I'm embarrassed to say:-). I doubt even you are paid less than minimum wage, though, and you should keep in mind any hour you spend away from your real job costs the University the labor you were doing on your actual work as well. >>> I think that our problem is that I have been prepending the word >>> LINUX mentally to our discussions. LINUX networks are not so >>> commonly set up by people who know nothing. > >> Ubuntu is rapidly helping with that. :) > > I actually don't agree that linux would prove anywhere nearly as > vulnerable as Windows has been historically even if they switched market > share tomorrow, I tend to agree that Unix has a much better overall architecture, but the big problem right now is the professionalism of the attackers. People are making a lot of money attacking Windows because it is the majority platform, and they tend to concentrate their efforts there. Linux would receive quite a bit more unwanted attention from such folks if it was profitable for them, and the quality of the codebase is quite variable. There *are* holes. > and Ubuntu was the only version of linux used. After all, as YOU > pointed out, MS left Explorer unpatched for 9 months. NINE MONTHS! > Say what? > > Find a similar exploit in (say) Firefox. There have been a number. I'd go and look at the CVE database. That said, fewer of them have gone unpatched for quite so long -- the pressure there from the community tends to be higher for whatever reason. > Virtualization is going to change everything, by the way. There are whole sessions at security conferences on ways to break out of VMs. Some of the techniques are quite amazing. I'd look at recent conference proceedings. I'm not claiming that VMs aren't a good thing -- they are. They're just not foolproof. > It's really, really hard to crack a site invisibly when every IP number > that talks to it or that it talks to is isolated in real time and > compared to a list you set up and control and sets off all sorts of > alarms if any sort of anomalous or unapproved pattern occurs. There are some really, really clever exploits out there. If you want to terrify yourself, start reading up on ethernet card firmware exploits. You can do astonishing things once you own the ethernet card on a modern machine -- you have a real processor with DMA access to the whole of main memory at your disposal. If you want to get even worse, here is the paper from a month ago about designing exploits in to microprocessors directly... Perry -- Perry E. Metzger perry@piermont.com From rgb at phy.duke.edu Sun Jun 22 05:33:09 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <461CB2E232CA45A6981BD3207EBF444A@geoffPC> References: <485920D8.2030309@ias.edu><200806190945.21604.kilian@stanford.edu> <485A9336.8010303@jax.org><200806191100.29026.kilian@stanford.edu> <485AA1D0.1080209@jax.org> <485C977B.6020102@gmail.com> <461CB2E232CA45A6981BD3207EBF444A@geoffPC> Message-ID: On Sat, 21 Jun 2008, Geoff Galitz wrote: > If I might complete devolve the thread and go waaaay off-topic. Does anyone > remember the movie Failsafe? > > http://www.imdb.com/title/tt0058083/ > http://classicfilms.suite101.com/article.cfm/fail_safe Of course, book and movie. Or even better, Dr. Strangelove: http://en.wikipedia.org/wiki/Dr._Strangelove_or:_How_I_Learned_to_Stop_Worrying_and_Love_the_Bomb Peter Sellers in three roles including the incredible Dr. Strangelove himself. George C. Scott as a general. James Earl Jones' film debut. Slim Pickens riding a fusion bomb down to start the holocaust. The Doomsday Device. What's not to like? Then there was Alas Babylon, a GREAT book and mediocre movie: http://en.wikipedia.org/wiki/Alas,_Babylon This was more or less a template for what might have happened if the Soviets hadn't backed down during the Cuban Missile Crisis, or if any of the many little conflicts around the world had heated up to boiling, or if the MAD policy ever became unbalanced because one side or the other gained too much advantage. No nuclear winter (even in florida) though -- it hadn't been hypothesized yet. There was the BBC "civil defense movie" The War Game that was made to illustrate the dangers of a nuclear war to the public that is even today scarier in its own way than the Texas Chainsaw Massacre: http://en.wikipedia.org/wiki/The_War_Game It describes a scenario where China invades South Vietnam in the mid 60's, the US authorizes the use of tactical nukes to stop them, the USSR threatens to start the long awaited war in Europe by taking Berlin if they do, they do, they do, the US tries to retake Berlin with Nato forces in an overland invasion but get their ass kicked, they use tactical nukes there, and a "limited" strategic/tactical war begins in Europe (leaving the US and USSR out of it per se). The movie covers the physical consequences of a bomb that strikes a city as a "miss" of an attack on a nearby strategic military target. There is a whole literature on this -- apocalyptic and post-apocalyptic views of nuclear war. On The Beach. Five Signs from Ruby. A Boy and his Dog. Damnation Alley. Mad Max. Daybreak, 2024. The Day After. One of my favorites: A Canticle for Liebowitz. I read 'em, or watched 'em, all. The executive summary: Nuclear war is a really bad idea. It is hard on children and pets. Nobody is likely to "win", and worse, once the possibility exists at all, sooner or later it will happen, then happen again, because we can't/won't learn, because we are flawed creatures, because even if 99.999% of the population acts against it it only takes one nut case seeking purity of essence acting for it to make it happen. It won't QUITE end the human race altogether -- On the Beach was pretty much the only one to allege that it would and it was almost certainly wrong -- but it could knock us back to anywhere from the stone age to the dark ages for thousands of years and kill anywhere up to 90% of the global population and leave huge tracts of land uninhabitable for those thousands of years. Friends don't let friends use nuclear bombs. rgb > > > > Geoff Galitz > Blankenheim NRW, Deutschland > http://www.galitz.org > > > -----Original Message----- > From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On > Behalf Of Geoff Jacobs > Sent: Samstag, 21. Juni 2008 07:54 > To: Glen Beane > Cc: Beowulf List > Subject: Re: [Beowulf] Re: "hobbyists" > > [stuff snipped] > >>> That's why I think nuclear weapons are hardly a mean to kill military >>> troops on a battlefield. >> >> Strategic nukes, no. Tactical nukes, yes. > Now find an effective way of preventing a tactical exchange from > escalating to a strategic exchange. > > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From spambox at emboss.co.nz Sun Jun 22 00:50:04 2008 From: spambox at emboss.co.nz (Michael Brown) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists"es In-Reply-To: <87iqw2uxno.fsf@snark.cb.piermont.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org><87skv8wxlg.fsf@snark.cb.piermont.com><87mylgwbg6.fsf@snark.cb.piermont.com><87abhfzpcw.fsf@snark.cb.piermont.com> <87iqw2uxno.fsf@snark.cb.piermont.com> Message-ID: <5DE3883A2FD1411AA798772E37563449@Forethought> "Perry E. Metzger" wrote: > "Robert G. Brown" writes: >>> If they can't use public key auth, give 'em secure ids or something >>> similar. Works fine or such purposes. Passwords are dead. >> >> Yeah, Bill Gates (among others) said something like that back in 2004. >> I confess to being deeply skeptical. Really. The SecureID solution has >> been around for a long time at this point. It was a PITA a decade ago. >> It is a PITA now. Expensive, too. > > It is neither. I use SecureIDs quite regularly and it isn't difficult > at all -- you just look at the device and type in the digits. What's > so hard about that? The biggest problem comes when everybody wants to use them. I already have to carry around three SecurID cards, and that number could easily hit a dozen even if I only included networks that I log into on a nearly daily basis and online banking sites. What is needed is the ability to securely share a single physical token between multiple networks. [...] >> Then there is logging onto systems I work on -- something that IS >> possible for me without a password. The problem there is that many of >> the systems I'm logging in from are laptops (I have two personally, >> about to make that three). The laptops themselves then become a >> security risk if they are stolen, > > That's why they invented encrypted partitions, and why ssh lets you > encrypt your public key credentials. In some sense, encrypted keys are more of a security problem than passwords. To break a password-based login requires an easily detected online attack. Breaking the password on a ssh key file can be done offline, and can have orders of magnitude more attempts thrown at it. Both depend on the user choosing a sufficiently secure password. You have to make sure that difficulty in obtaining the key file makes up for the easier breaking of the password. -- Michael Brown Add michael@ to emboss.co.nz ---+--- My inbox is always open From rgb at phy.duke.edu Sun Jun 22 08:02:44 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists"es In-Reply-To: <87iqw2uxno.fsf@snark.cb.piermont.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> <87abhfzpcw.fsf@snark.cb.piermont.com> <87iqw2uxno.fsf@snark.cb.piermont.com> Message-ID: On Sat, 21 Jun 2008, Perry E. Metzger wrote: >> It's really, really hard to crack a site invisibly when every IP number >> that talks to it or that it talks to is isolated in real time and >> compared to a list you set up and control and sets off all sorts of >> alarms if any sort of anomalous or unapproved pattern occurs. > > There are some really, really clever exploits out there. If you want > to terrify yourself, start reading up on ethernet card firmware > exploits. You can do astonishing things once you own the ethernet card > on a modern machine -- you have a real processor with DMA access to > the whole of main memory at your disposal. If you want to get even > worse, here is the paper from a month ago about designing exploits in > to microprocessors directly... No, no, no. If I went and read all of that, I'd have to start adding lexipro to my morning orange juice, and it makes the vodka taste horrible. Pardon me, I have to go pull the plug on my home DSL modem now, install dual-isolation on my system power supply, stop using wireless altogether, and install a self-destruct circuit on the USB fob I use to boot my working system...;-) Seriously, you've convinced me that it is time to revising the secureid issue, although to me the PITA isn't the pain of typing it in, it is the pain of carrying it and having to have it with you in order to access "the network". And possibly the PITA of having to install it on an enterprise basis in a complex and heterogeneous environment so one doesn't end up having to carry a dozen of the damn things. I will dutifully mention it to the powers at Duke. If we can architect it so that just one central auth system works for the entire campus and medical center (which is as much a POLITICAL problem as it is a technical problem) without altering the balance of power as it were, and it weren't horribly expensive, maybe it might fly. Alas, though, I'm no longer good buddies with the security czar. In fact, I'm not sure we have a security czar at all at this moment. I'm now abandoning the thread, BTW, in deference to your being on vacation and my having an insane amount of work to do before leaving on my, um, "teach at the beach" summer that will mix vacation and work in equal measure, I hope. It has been very interesting. And useful. rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From peter.st.john at gmail.com Sun Jun 22 15:04:30 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Plan 9 In-Reply-To: <87hcbnzgrr.fsf@snark.cb.piermont.com> References: <87hcbnzgrr.fsf@snark.cb.piermont.com> Message-ID: Perry, Point taken; by "descendant" I meant the same developers as SysV, but yeah they don't think of it as merely a unix flavor at all. Peter On Fri, Jun 20, 2008 at 10:03 PM, Perry E. Metzger wrote: > > "Peter St. John" writes: > > I was surprised to see Plan 9 deployed on a Blue Gene, > > http://en.wikipedia.org/wiki/Blue_Gene#Plan_9_support. > > I think of Plan 9 as the direct descendant of System V; > > Not even remotely related to System V, and Rob Pike would probably > giggle uproariously at the idea. In fact, Plan 9 is sort of the > anti-SysV. > > > Has anybody tried it? > > It is neat, but it is even more fringe than the normal fringe > choices... > > Perry > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080622/887d6790/attachment.html From gdjacobs at gmail.com Sun Jun 22 16:11:00 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Plan 9 In-Reply-To: References: <87hcbnzgrr.fsf@snark.cb.piermont.com> Message-ID: <485EDC04.1000304@gmail.com> Peter St. John wrote: > Perry, > Point taken; by "descendant" I meant the same developers as SysV, but > yeah they don't think of it as merely a unix flavor at all. > Peter Yeah, Plan 9 was always considered to be the next evolution in the UNIX tradition. -- Geoffrey D. Jacobs From perry at piermont.com Sun Jun 22 18:08:15 2008 From: perry at piermont.com (Perry E. Metzger) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists"es In-Reply-To: (Robert G. Brown's message of "Sun\, 22 Jun 2008 11\:02\:44 -0400 \(EDT\)") References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> <87abhfzpcw.fsf@snark.cb.piermont.com> <87iqw2uxno.fsf@snark.cb.piermont.com> Message-ID: <87k5ghx8jk.fsf@snark.cb.piermont.com> "Robert G. Brown" writes: > Seriously, you've convinced me that it is time to revising the secureid > issue, although to me the PITA isn't the pain of typing it in, it is the > pain of carrying it and having to have it with you in order to access > "the network". And possibly the PITA of having to install it on an > enterprise basis in a complex and heterogeneous environment so one > doesn't end up having to carry a dozen of the damn things. Well, there are a dozen other solutions out there. Kerberos is well loved by many people. There are various new EKE-like mechanisms out there that are apparently not patented. There's all sorts of stuff. SecureID was merely something I mentioned as a proof that you don't have to use passwords. -- Perry E. Metzger perry@piermont.com From franz.marini at mi.infn.it Mon Jun 23 03:24:20 2008 From: franz.marini at mi.infn.it (Franz Marini) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <200806190945.21604.kilian@stanford.edu> <485A9336.8010303@jax.org> <200806191100.29026.kilian@stanford.edu> <485AA1D0.1080209@jax.org> <485C977B.6020102@gmail.com> <461CB2E232CA45A6981BD3207EBF444A@geoffPC> Message-ID: <1214216660.15270.10.camel@merlino.mi.infn.it> On Sun, 2008-06-22 at 08:33 -0400, Robert G. Brown wrote: > There is a whole literature on this -- apocalyptic and post-apocalyptic > views of nuclear war. On The Beach. Five Signs from Ruby. A Boy and > his Dog. Damnation Alley. Mad Max. Daybreak, 2024. The Day After. > One of my favorites: A Canticle for Liebowitz. I read 'em, or watched > 'em, all. I would like to add to that list a somewhat obscure BBC television movie from 1984, Threads, which is, easily, the most scaring and horrifying play I've ever watched, much more than The Day After. No good ending in this one. No hope. No future. The last scene is downright nightmare-inspiring. By the way, it does depict a nuclear winter. As far as I know, no other post nuclear war movie ever did. I suggest everyone get hold of a copy (be careful, there are two versions around, one reedited and cut, and another one, the DVD one, uncut, get this one) and watch it. F. --------------------------------------------------------- Franz Marini Prof. R. A. Broglia Theoretical Physics of Nuclei, Atomic Clusters and Proteins Research Group Dept. of Physics, University of Milan, Italy. email : franz.marini@mi.infn.it phone : +39 02 50317226 --------------------------------------------------------- From tjrc at sanger.ac.uk Mon Jun 23 04:31:06 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists"es In-Reply-To: <87iqw2uxno.fsf@snark.cb.piermont.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <87mylgwbg6.fsf@snark.cb.piermont.com> <87abhfzpcw.fsf@snark.cb.piermont.com> <87iqw2uxno.fsf@snark.cb.piermont.com> Message-ID: On 22 Jun 2008, at 1:21 am, Perry E. Metzger wrote: > $10-$50 a user is cheap compared to the salaries of even university > sysadmins multiplied over all the hours of trouble that breakins > cause. As I said, there are competitors that are cheaper -- if you > really need a $5 solution, they exist. If you think you're spending > less than $5 a user on related security problems over the amortization > life of a device, you're kidding yourself in any real organization. Of > course, if you are still over budget, one can also use mechanisms like > Kerberos, and in a university environment that's quite achievable, and > has no associated hardware costs at all. Unfortunately, of course, many managers won't accept that and put the lock on the door until *after* the horse has bolted. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From kus at free.net Mon Jun 23 07:01:29 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) Message-ID: I'm testing my 1st dual-socket quad-core Opteron 2350-based server. Let me assume that the RAM used by kernel and system processes is zero, there is no physical RAM fragmentation, and the affinity of processes to CPU cores is maintained. I assume also that both the nodes are populated w/equal number of the same DIMMs. If I run thread- parallelized (for example, w/OpenMP) application w/8 threads (8 = number of server CPU cores), the ideal case for all the ("equal") threads is: the shared memory used by each of 2 CPUs (by each of 2 processes "quads") should be divided equally between 2 nodes, and the local memory used by each process should be mapped analogically. Theoretically like ideal case may be realized if my application (8 threads) uses practically all the RAM and uses only shared memory (I assume here also that all the RAM addresses have the same load, and the size of program codes is zero :-) ). The questions are 1) Is there some way to distribute analogously the local memory of threads (I assume that it have the same size for each thread) using "reasonable" NUMA allocation ? 2) Is it right that using of numactl for applications may gives improvements of performance for the following case: the number of application processes is equal to the number of cores of one CPU *AND* the necessary (for application) RAM amount may be placed on one node DIMMs (I assume that RAM is allocated "continously"). What will be w/performance (at numactl using) for the case if RAM size required is higher than RAM available per one node, and therefore the program will not use the possibility of (load balanced) simultaneous using of memory controllers on both CPUs ? (I also assume also that RAM is allocated continously). 3) Is there some reason to use things like mpirun -np N /usr/bin/numactl my_application ? 4) If I use malloc() and don't use numactl, how to understand - from which node Linux will begin the real memory allocation ? (I remember that I assume that all the RAM is free) And how to understand where are placed the DIMMs which will corresponds to higher RAM addresses or lower RAM addresses ? 5) In which cases is it reasonable to switch on "Node memory interleaving" (in BIOS) for the application which uses more memory than is presented on the node ? And BTW: if I use taskset -c CPU1,CPU2, ... and the program_file creates some new processes, will all this processes run only on the same CPUs defined in taskset command ? Mikhail Kuzminsky Computer Assistance to Chemical Research Center, Zelinsky Institute of Organic Chemistry Moscow From hahn at mcmaster.ca Mon Jun 23 08:25:28 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: References: Message-ID: > The questions are > 1) Is there some way to distribute analogously the local memory of threads (I > assume that it have the same size for each thread) using "reasonable" NUMA > allocation ? that is, not surprisingly, the default. generally, on all NUMA machines, the starting rule is that memory is allocated for a thread upon "first touch". that is, the first thread to touch it, causing a page fault and triggering the actual allocation. (if you allocate memory but never touch it, it remains purely virtual, ignoring any book-keeping by your memory allocation library, if any.) > 2) Is it right that using of numactl for applications may gives improvements > of performance for the following case: > the number of application processes is equal to the number of cores of one > CPU *AND* the necessary (for application) RAM amount may be placed on one > node DIMMs (I assume that RAM is allocated "continously"). you certainly don't want to _deliberately_ create imbalances. "numactl --hardware" is interesting to see the state of memory allocation. of course, it reflects only size and free (where free means "wasted" to the kernel, not the same as "freeable".) > What will be w/performance (at numactl using) for the case if RAM size > required is higher than RAM available per one node, and therefore the program > will not use the possibility of (load balanced) simultaneous using of memory > controllers on both CPUs ? non-local memory is modestly slower than local - not dramatically. > (I also assume also that RAM is allocated > continously). I'm not sure what that means - continuously in time? or contiguously? the latter is definitely not true - the allocated memory map for a task will normally be pretty chopped up, and the virtual addresses will have little relation to physical addresses. > 3) Is there some reason to use things like > mpirun -np N /usr/bin/numactl my_application ? not that I know. > 4) If I use malloc() and don't use numactl, how to understand - from which > node Linux will begin the real memory allocation ? (I remember that I assume if there is free memory on the node where the thread is running, that's where the physical page will be allocated. > that all the RAM is free) And how to understand where are placed the DIMMs > which will corresponds to higher RAM addresses or lower RAM addresses ? I don't see why userspace would need to know that. the main question is whether non-local allocations are allowed or not, and you set that policy with numactl --localalloc (or override with --preferred, etc) > 5) In which cases is it reasonable to switch on "Node memory interleaving" > (in BIOS) for the application which uses more memory than is presented on the > node ? I leave it off, since numactl --interleave lets you get the same effect from user-space. I'm not sure I've ever seen it be a win. > And BTW: if I use taskset -c CPU1,CPU2, ... > and the program_file creates some new processes, will all this processes run > only on the same CPUs defined in taskset command ? afaik, scheduler settings like this are indeed inherited across clone, possibly also fork. From diep at xs4all.nl Mon Jun 23 09:41:21 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: References: Message-ID: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> I would add to this: "how sure are we that a process (or thread) that allocated and initialized and writes to memory at a single specific memory node, also keeps getting scheduled at a core on that memory node?" It seems to me that sometimes (like every second or so) threads jump from 1 memory node to another. I could be wrong, but i certainly have that impression with the linux kernels. That said, it has improved a lot, now all we need is a better compiler for linux. GCC is for my chessprogram generating an executable that gets 22% slower positions per second than visual c++ 2005 is. Thanks, Vincent On Jun 23, 2008, at 4:01 PM, Mikhail Kuzminsky wrote: > I'm testing my 1st dual-socket quad-core Opteron 2350-based server. > Let me assume that the RAM used by kernel and system processes is > zero, there is no physical RAM fragmentation, and the affinity of > processes to CPU cores is maintained. I assume also that both the > nodes are populated w/equal number of the same DIMMs. > > If I run thread- parallelized (for example, w/OpenMP) application w/ > 8 threads (8 = number of server CPU cores), the ideal case for all > the ("equal") threads is: the shared memory used by each of 2 CPUs > (by each of 2 processes "quads") should be divided equally between > 2 nodes, and the local memory used by each process should be mapped > analogically. > Theoretically like ideal case may be realized if my application (8 > threads) uses practically all the RAM and uses only shared memory > (I assume here also that all the RAM addresses have the same load, > and the size of program codes is zero :-) ). > > The questions are > 1) Is there some way to distribute analogously the local memory of > threads (I assume that it have the same size for each thread) using > "reasonable" NUMA allocation ? > > 2) Is it right that using of numactl for applications may gives > improvements of performance for the following case: > the number of application processes is equal to the number of cores > of one CPU *AND* the necessary (for application) RAM amount may be > placed on one node DIMMs (I assume that RAM is allocated > "continously"). > > What will be w/performance (at numactl using) for the case if RAM > size required is higher than RAM available per one node, and > therefore the program will not use the possibility of (load > balanced) simultaneous using of memory controllers on both CPUs ? > (I also assume also that RAM is allocated continously). > > 3) Is there some reason to use things like > mpirun -np N /usr/bin/numactl my_application ? > > 4) If I use malloc() and don't use numactl, how to understand - > from which node Linux will begin the real memory allocation ? (I > remember that I assume that all the RAM is free) And how to > understand where are placed the DIMMs which will corresponds to > higher RAM addresses or lower RAM addresses ? > > 5) In which cases is it reasonable to switch on "Node memory > interleaving" (in BIOS) for the application which uses more memory > than is presented on the node ? > And BTW: if I use taskset -c CPU1,CPU2, ... > and the program_file creates some new processes, will all this > processes run only on the same CPUs defined in taskset command ? > > Mikhail Kuzminsky > Computer Assistance to Chemical Research Center, > Zelinsky Institute of Organic Chemistry > Moscow > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From Bogdan.Costescu at iwr.uni-heidelberg.de Mon Jun 23 09:44:34 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <485920D8.2030309@ias.edu> References: <485920D8.2030309@ias.edu> Message-ID: On Wed, 18 Jun 2008, Prentice Bisbal wrote: > The biggest hindrance to doing "real" work with GPUs is the lack of > dual-precision capabilities. I think that the biggest hindrance is a unified API or language for all these accelerators (taking into account not only the GPUs !). Many developers are probably scared that their code depends on the whim of the accelerator producer in terms of long-term compatibility of the source code with the API or language or of the binary code with the available hardware; sure, you can't prevent the hardware being obsoleted or the company from going out of bussiness, but if you're only one recompilation away it's manageable. At the last week's ISC'08, after the ATI/AMD and NVidia talks, someone asked a NVidia guy about any plans of unification with at least ATI/AMD on this front and the answer was "we're not there yet"... while the ATI/AMD presentation went on to say "we learnt from mistakes with our past implementations and we present you now with OpenCL" - yet another way of programming their GPU... I see this situation very similar to the SSE vs. 3Dnow of some years ago or the one before MPI came to replace all the proprietary communication libraries. Anybody else shares this view ? -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From Bogdan.Costescu at iwr.uni-heidelberg.de Mon Jun 23 09:54:54 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <48597A7A.9070101@ias.edu> References: <485920D8.2030309@ias.edu> <48597A7A.9070101@ias.edu> Message-ID: On Wed, 18 Jun 2008, Prentice Bisbal wrote: > There is value, however, if your goal is to recover (discover?) an > MD5-hashed password through a brute-force attack. Last time I > checked, MD5 password s are the default for most Linux distros. The above paragraph can be misleading: while Linux distros use MD5 passwords, the Linux passwords are not simple MD5 hashes of the password text, which means that using a GPGPU to precompute MD5 hash tables will not help you reverse the passwords. There are already such tables available on the 'net, search for 'rainbow table', probably computed before any GPGPU acceleration existed and we'd all be using something else by now if they could be used to directly get the password text... -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From Bogdan.Costescu at iwr.uni-heidelberg.de Mon Jun 23 09:57:03 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: <1213910532.7651.4.camel@Vigor13> References: <485920D8.2030309@ias.edu> <20080618214603.GC31594@bx9.net> <200806181521.47115.kilian@stanford.edu> <48599AC9.3040100@berkeley.edu> <1213859837.4784.9.camel@Vigor13> <1213910532.7651.4.camel@Vigor13> Message-ID: On Thu, 19 Jun 2008, John Hearns wrote: > One thing I've never understood, and hopefully someone on here can > explain clearly, is why the onboard graphics is normally disabled > when you add a PCI-e card. Conflict of resources like I/O ports to emulate old VGA (or earlier) comes to mind. -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From peter.st.john at gmail.com Mon Jun 23 10:50:51 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <485AA481.2030000@vcu.edu> <200806201408.04839.kilian@stanford.edu> Message-ID: RGB, (re kicking) Sure, and you won't miss. Basically, you lift your foot up and place it on my chest, with your knee bent. I bend my knees. We say "one two three go!" and you push with your leg while I jump backwards. If people practice doing that *quickly* it looks kinda like you are kicking me through the air. It helps if I have a balsa-wood table to land on :-) Or one of those sugar-glass windows to crash through. I should be down to Chapel Hill this Christmas (last year we went to SF to meet newest grand-nephew). Peter On 6/21/08, Robert G. Brown wrote: > > On Fri, 20 Jun 2008, Peter St. John wrote: > > The destructive radius of Little Boy was about total, up to about one mile >> radius, and tapered down to light at about two miles. So being in a >> lead-lined steel container at 2000 meters might be OK for Indiana. >> >> In all action movies, blasts throw people unhurt for long distances; when >> that much force (to impart that much momentum) would kill you. That part >> is >> just conventional Hollywood. I could teach RGB to kick me so that I fly >> through the air as in a Bruce Lee movie; it's a stunt, and real kicks >> > > Ooo, you can? When can we start? What if I, um, "miss"? > > reallly hitting drop you like a sack of potatoes, I've seen it. But not in >> movies. Similarly bullets, they drill holes in you, if they pushed you >> through the air the recoil would do the same to the shooter. >> > > Yes, one does have to work a bit to suspend disbelief when watching > movies these days. But not TOO much. Years of watching Jackie Chan, > James Bond, Bruce Lee, Angelina Jolie, etc have gradually broken us into > the idea that there are two completely distinct realities. The one > where you can get hit on the head with a two by four and continue > fighting, and the one where you can get hit on the head with a two by > four and bits of your skull and brains spatter out onto the walls. > They're even both hollywood realities! > > As for the scene's good taste I can't say, I haven't seen the movie yet >> :-) >> > > Yah. Maybe this week. Maybe I'll wait for the video. > > rgb > > >> Peter >> >> >> >> On 6/20/08, Kilian CAVALOTTI wrote: >> >>> >>> On Thursday 19 June 2008 11:25:05 am Mike Davis wrote: >>> >>>> According to one weapons designer the only safe way to use it was to >>>> fire from a hilltop into a valley from a jeep and then drive like >>>> hell into the next valley. >>>> >>> >>> >>> Or... there's also the so-called "home appliance nuke evasion >>> manoeuvre", as demonstrated in "Indiana Jones and the Fridge of Nuclear >>> Doom". But, obviously, you need a fridge. >>> >>> http://blog.uwinnipeg.ca/ius/archives/003477.html >>> >>> -- >>> >>> Kilian >>> >>> >> > -- > Robert G. Brown Phone(cell): 1-919-280-8443 > Duke University Physics Dept, Box 90305 > Durham, N.C. 27708-0305 > Web: http://www.phy.duke.edu/~rgb > Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php > Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080623/bb71035f/attachment.html From kus at free.net Mon Jun 23 11:24:59 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> Message-ID: In message from Vincent Diepeveen (Mon, 23 Jun 2008 18:41:21 +0200): >I would add to this: > >"how sure are we that a process (or thread) that allocated and > initialized and writes to memory at a single specific memory node, >also keeps getting scheduled at a core on that memory node?" > >It seems to me that sometimes (like every second or so) threads jump > from 1 memory node to another. I could be wrong, >but i certainly have that impression with the linux kernels. Dear Vincent, do I understand you correctly that simple using of taskset is not enough to prevent process migration to other core/node ?? Mikhail > >That said, it has improved a lot, now all we need is a better > compiler for linux. GCC is for my chessprogram generating an >executable that gets 22% slower positions per second than visual c++ > 2005 is. > >Thanks, >Vincent > > > >On Jun 23, 2008, at 4:01 PM, Mikhail Kuzminsky wrote: > >> I'm testing my 1st dual-socket quad-core Opteron 2350-based server. >> Let me assume that the RAM used by kernel and system processes is >> zero, there is no physical RAM fragmentation, and the affinity of >> processes to CPU cores is maintained. I assume also that both the >> nodes are populated w/equal number of the same DIMMs. >> >> If I run thread- parallelized (for example, w/OpenMP) application w/ >> 8 threads (8 = number of server CPU cores), the ideal case for all >> the ("equal") threads is: the shared memory used by each of 2 CPUs >> (by each of 2 processes "quads") should be divided equally between >> 2 nodes, and the local memory used by each process should be mapped >> >> analogically. >> Theoretically like ideal case may be realized if my application (8 >> threads) uses practically all the RAM and uses only shared memory >> (I assume here also that all the RAM addresses have the same load, >> and the size of program codes is zero :-) ). >> >> The questions are >> 1) Is there some way to distribute analogously the local memory of >> threads (I assume that it have the same size for each thread) using >> >> "reasonable" NUMA allocation ? >> >> 2) Is it right that using of numactl for applications may gives >> improvements of performance for the following case: >> the number of application processes is equal to the number of cores >> >> of one CPU *AND* the necessary (for application) RAM amount may be >> placed on one node DIMMs (I assume that RAM is allocated >> "continously"). >> >> What will be w/performance (at numactl using) for the case if RAM >> size required is higher than RAM available per one node, and >> therefore the program will not use the possibility of (load >> balanced) simultaneous using of memory controllers on both CPUs ? >> (I also assume also that RAM is allocated continously). >> >> 3) Is there some reason to use things like >> mpirun -np N /usr/bin/numactl my_application >> ? >> >> 4) If I use malloc() and don't use numactl, how to understand - >> from which node Linux will begin the real memory allocation ? (I >> remember that I assume that all the RAM is free) And how to >> understand where are placed the DIMMs which will corresponds to >> higher RAM addresses or lower RAM addresses ? >> >> 5) In which cases is it reasonable to switch on "Node memory >> interleaving" (in BIOS) for the application which uses more memory >> than is presented on the node ? >> And BTW: if I use taskset -c CPU1,CPU2, ... >> and the program_file creates some new processes, will all this >> processes run only on the same CPUs defined in taskset command ? >> >> Mikhail Kuzminsky >> Computer Assistance to Chemical Research Center, >> Zelinsky Institute of Organic Chemistry >> Moscow >> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> > >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf From hahn at mcmaster.ca Mon Jun 23 12:12:48 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> References: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> Message-ID: > "how sure are we that a process (or thread) that allocated and initialized > and writes to memory at a single specific memory node, > also keeps getting scheduled at a core on that memory node?" numactl --cpubind=0 --membind=0 > It seems to me that sometimes (like every second or so) threads jump from 1 > memory node to another. I could be wrong, > but i certainly have that impression with the linux kernels. you can always tie a thread to a core. for non-bound threads, the question is really how long the kernel should leave a runnable thread "on" a busy cpu before running it on another (idle) cpu. the kernel does try to avoid this, but how hard has in the past depended on the kernel's guess about the cache footprint of the thread and its "natural" timeslice (how long it typically runs before yielding.) From jac67 at georgetown.edu Mon Jun 23 14:21:24 2008 From: jac67 at georgetown.edu (Jess Cannata) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Best training for rusty HPC skills? In-Reply-To: <1214055071.5684.66.camel@e521.site> References: <0FBB78A3-6B0E-48FD-9455-E38C182BE482@rochester.edu> <1214055071.5684.66.camel@e521.site> Message-ID: <486013D4.6000707@georgetown.edu> Greg, I agree that SuperComputing is a great place to go to catch up on the latest technology. And yes, here at Georgetown University (not be confused with our neighbor George Washington University) we offer HPC training courses (www.gridswatch.com, Training). We have taught a fair number of Linux/Unix admins with HPC experience who just needed a refresher course. You can see the current offerings at http://www.gridswatch.com/index.php?option=com_content&task=view&id=25&Itemid=16 We have several classes coming in the next few months on a variety of topics. If you have any questions, please feel free to contact me directly. Jess -- Jess Cannata Advanced Research Computing Georgetown University 202-687-3661 John Leidel wrote: > I would definitely recommend heading to Supercomputing > [http://sc08.supercomp.org/]. The SC education committee has also setup > a program geared toward educators that holds seminars on specific > subjects. There are a few within driving distance of NY this summer. > http://sc08.sc-education.org/workshops/index.php > > George Washington University in DC has several training courses in HPC. > I believe they also have an "official" certificate program as well. > http://www.gridswatch.com/ > > For a broad series of topics, I suggest going through some of the > training materials provided by Ohio Supercomputing. > [ http://www.osc.edu/supercomputing/training/ ] > > Read the news: www.hpcwire.com && www.insidehpc.com > [selfish plug on that one...] > && www.clustermonkey.net > > > cheers > john > > On Fri, 2008-06-20 at 15:53 -0400, Gregory R. Warnes, Ph.D. wrote: > >> Hi All, >> >> I've just been appointed to head an acadmeic computing center, after >> an absence from the HPC arena affor 10 years. What conferences/ >> training/seminars/books/etc would be best for refreshing my awareness >> of the technologies and issues? >> >> Thanks! >> >> -Greg >> >> > From diep at xs4all.nl Mon Jun 23 14:57:06 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> Message-ID: <3E2E0923-1E83-427E-A383-EFEBEA390F1F@xs4all.nl> Not really, The architecture of AMD versus Nvidia is quite different. I would encourage each manufacturer to have their own system. So to speak AMD is a low clocked core2 supercomputer @ 64 cores, versus nvidia a 240 processor mips supercomputer. I feel the real limitation is that the achievement of the GPU's only exist on marketing paper. I can also claim my cars engine is capable of driving 20000 miles per hour. Sure it is, in space! A PC processors is just as good as its caches and memory controller is. There is too little technical data about GPU's with respect to bottlenecks, whereas bottlenecks will dominate such hardware of course. If you KNOW what is a bottleneck, then in a theoretic model you can work around it. The few reports from individuals who work fulltime with GPU's who tried writing some number crunching code for it, yes even 32 bits number crunching codes, the practical odds for succes for an individual programmer is too small currently for GPU's. It is a fact that to program those things well, you first need to be hell of a programmer. Those hell of programmers know very well that you need full technical information, even if that means bad news for Nvidia and AMD as suddenly the GPU's look a lot weaker then. If there is technical specifications that also show the bottlenecks very well, then the algorithmic strong among us (trying to not look too much outside of the window), will find some clever solutions to get a specific thing done. This is all paper work. If there is on paper an algorithmic solution, or even a method of how to get something done, then there will be programmers implementing it, as they see no risks. they just see that solution that is gonna give them more crunching power. It is all about risk assesment from programmers viewpoint. Right now the only thing he knows is big bragging stories, he of course realizes that if you do something within the register files of it, that this can be fast, other than that he knows that in the first place it is a GPU meant for displaying graphics and not especially designed just to do numbercrunching. If you go for a platform into the deep, so without information, you just don't do it. At the time, if you went for SSE/SSE2 assembler code, you knew full specs of it, every instruction, every latency of every instruction and so on. To take the step to CONSIDER writing something on a GPU means that the programmer in question is already a total hardcore addict; you really want to get the ultimate achievement out of the hardware to achieve your numbercrunching. The same is true for SSE/SSE2. I would argue writing SSE2 code is tougher than writing for a GPU, from implementation viewpoint seen, under the condition that you DO have a parallel model how to get things done on a GPU. The number of people who know how to write a parallel model on paper that theoretical works and gets the maximum out of crunching hardware that is non trivial to parallellize is just real little. If within a specific specialism that is more than a dozen, that's a lot already. The number of good programmers who can write you that code, in whatever language, is little compared to the total number of programmers, but real huge compared to algorithmic designers who are expert in your field. It will not take long until such solutions are simply posted on the net. That might increase the number of people who toy with GPU's. In itself Seymour Crays statement, "If you were plowing a field, which would you rather use? Two strong oxen or 1024 chickens?" is very true from practical viewpoint; it is simpler to work with 4 cores than with 64, let alone 240, but objectively of course such a majority should be able to beat 2 strong oxen. Doesn't mean it is simple to do it. So the number of persons who start writing solutions there you can really count on a few hands. Most of them currently really are a few students who tried at a card which delivers already in marketing value such tiny amounts of single precision gflops, that it really must get seen as a hobby project of a student who just learns advanced programming a tad better, as their quadcore with existing highly optimized free software is for sure faster. In itself that is very weird, as in itself there is not really anyone who doubts that in the long run many tiny processors are gonna win it for number crunching. Vincent On Jun 23, 2008, at 6:44 PM, Bogdan Costescu wrote: > On Wed, 18 Jun 2008, Prentice Bisbal wrote: > >> The biggest hindrance to doing "real" work with GPUs is the lack >> of dual-precision capabilities. > > I think that the biggest hindrance is a unified API or language for > all these accelerators (taking into account not only the GPUs !). > Many developers are probably scared that their code depends on the > whim of the accelerator producer in terms of long-term > compatibility of the source code with the API or language or of the > binary code with the available hardware; sure, you can't prevent > the hardware being obsoleted or the company from going out of > bussiness, but if you're only one recompilation away it's manageable. > > At the last week's ISC'08, after the ATI/AMD and NVidia talks, > someone asked a NVidia guy about any plans of unification with at > least ATI/AMD on this front and the answer was "we're not there > yet"... while the ATI/AMD presentation went on to say "we learnt > from mistakes with our past implementations and we present you now > with OpenCL" - yet another way of programming their GPU... > > I see this situation very similar to the SSE vs. 3Dnow of some > years ago or the one before MPI came to replace all the proprietary > communication libraries. Anybody else shares this view ? > > -- > Bogdan Costescu > > IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany > Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 > E-mail: bogdan.costescu@iwr.uni-heidelberg.de > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From lindahl at pbm.com Mon Jun 23 14:59:25 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: References: Message-ID: <20080623215924.GC14844@bx9.net> > 3) Is there some reason to use things like > mpirun -np N /usr/bin/numactl my_application ? Your MPI (and OpenMP) should do this for you. -- greg From diep at xs4all.nl Mon Jun 23 16:27:54 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: References: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> Message-ID: <54DF7C95-DE01-4DB8-9A5D-1546D721012D@xs4all.nl> On Jun 23, 2008, at 9:12 PM, Mark Hahn wrote: >> "how sure are we that a process (or thread) that allocated and >> initialized and writes to memory at a single specific memory node, >> also keeps getting scheduled at a core on that memory node?" > > numactl --cpubind=0 --membind=0 > >> It seems to me that sometimes (like every second or so) threads >> jump from 1 memory node to another. I could be wrong, >> but i certainly have that impression with the linux kernels. > > you can always tie a thread to a core. for non-bound threads, > the question is really how long the kernel should leave a runnable > thread "on" a busy cpu before running it on another (idle) cpu. > the kernel > does try to avoid this, but how hard has in the past depended on > the kernel's guess about the cache footprint of the thread and its > "natural" > timeslice (how long it typically runs before yielding.) > ______ Mark, thanks for your input. I've tried that numactl several times to no avail. It kept doing wrong. Though this is from a few years ago, last time i toyed a few days with numactl, it could have been improved by now, maybe. It is in itself a very relevant topic that Michael Kuzminsky adresses, as when a thread allocates a lot of memory, it is really relevant. Now i assume what i'm doing is on paper the ideal situation. If an AMD machine with 2 to 4 memory controllers has say 4 GB of ram, i give each process (memory - 500MB) / 4. So that is quite a tad of RAM. This ram gets nonstop hammered upon storing better ways to achieve finding the holy grail. It's writing about 150k entries a second to RAM on each core to memory controller at my AMD dual opteron dual core 2.4ghz machine (probably by most of you considered nowadays as old energy wasting junk, but well). If a cpu has say 750MB ram that means to get a loading factor alpha of 0.5 into memory (ignoring the chaining that happens a lot by the way as taking the efficiencydecrease by chaining is faster thanks to how latency to RAM works) is roughly 0.5 * 750M / (20 bytes * 0.150M/s) = 0.5 * 750 / 3 = 125 seconds A game can last for an hour or 6. So even a single switch within that 6 hours to another memory node is very bad. I tried to lock with commands each process to a different core (4 processes, 4 cores). I still saw a flip sometimes. Now of course at big clusters/supers some software support from manufacturers allows automigration of nodes and memory, with good reasons. So i guess for this type of scheduling we speak at a different level. Avoiding latency of RAM over the network by scheduling nodes closer to each other is really important. Yet within 1 node it is a different story. Suppose we've got 4 search processes P0..P3 and we have 4 cores C0..C3 I am guessing this happens, please tell me it is wrong: some OS-service gets a timeslice at C1, searchprocess P1 gets pushed backwards in the queue. C2's timeslice at memory node 1 finishes. P2 gets pushed back in FIFO queue. P1 is before P2 in the queue, so P1 runs on C2. What i want is in fact that P2 starts to run on C2 and P1 still keeps in queue ntil the service timeslice finished. Note the above is based purely on guessing based upon a 100 assumptions from something i *thought* i saw this or that; i didn't look in kernel code for it. Compared to that Perry is not paranoia at all. When getting in a further state you only consider someone paranoia when a person is paranoia with respect to future occasions; seeing some sort of ghost or bottleneck will be there as "it has to suck". RGB don't give up yet! Do your public duty and take care that Perry speaks out on those subjects, as we all have to deal with it, some more than others! Maybe tempt him post on how he deals with potential nuke builders? Best Regards, Vincent > _________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From csamuel at vpac.org Mon Jun 23 22:56:34 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <20080623215924.GC14844@bx9.net> Message-ID: <851570955.61161214286994032.JavaMail.root@zimbra.vpac.org> ----- "Greg Lindahl" wrote: > > 3) Is there some reason to use things like > > mpirun -np N /usr/bin/numactl my_application > ? > > Your MPI (and OpenMP) should do this for you. Although not always correctly, it may assume that it can allocate from core 0 onwards leading to odd performance issues if you happen to get two 4 CPU jobs running on the same node.. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From lindahl at pbm.com Mon Jun 23 23:38:12 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <851570955.61161214286994032.JavaMail.root@zimbra.vpac.org> References: <20080623215924.GC14844@bx9.net> <851570955.61161214286994032.JavaMail.root@zimbra.vpac.org> Message-ID: <20080624063811.GA5872@bx9.net> > > Your MPI (and OpenMP) should do this for you. > > Although not always correctly, it may assume that it can > allocate from core 0 onwards leading to odd performance > issues if you happen to get two 4 CPU jobs running on the > same node.. Most clusters I've seen aren't used that way (whole nodes only), but sure, you can always consult the manual in that case. Scali has posted here several times about how they handle it, and I believe the PathScale OpenMP and InfiniPath MPI manuals explicitly discuss it. OpenMPI discusses how they don't yet handle this case at: http://www.open-mpi.de/faq/?category=tuning#using-paffinity -- greg From m.janssens at opencfd.co.uk Tue Jun 24 01:17:34 2008 From: m.janssens at opencfd.co.uk (Mattijs Janssens) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] NVIDIA GPUs, CUDA, MD5, and "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> Message-ID: <200806240917.34579.m.janssens@opencfd.co.uk> On Monday 23 June 2008 17:44, Bogdan Costescu wrote: > On Wed, 18 Jun 2008, Prentice Bisbal wrote: > > The biggest hindrance to doing "real" work with GPUs is the lack of > > dual-precision capabilities. > > I think that the biggest hindrance is a unified API or language for > all these accelerators (taking into account not only the GPUs !). Many > developers are probably scared that their code depends on the whim of > the accelerator producer in terms of long-term compatibility of the > source code with the API or language or of the binary code with the > available hardware; sure, you can't prevent the hardware being > obsoleted or the company from going out of bussiness, but if you're > only one recompilation away it's manageable. > > At the last week's ISC'08, after the ATI/AMD and NVidia talks, someone > asked a NVidia guy about any plans of unification with at least > ATI/AMD on this front and the answer was "we're not there yet"... > while the ATI/AMD presentation went on to say "we learnt from mistakes > with our past implementations and we present you now with OpenCL" - > yet another way of programming their GPU... > > I see this situation very similar to the SSE vs. 3Dnow of some years > ago or the one before MPI came to replace all the proprietary > communication libraries. Anybody else shares this view ? Most certainly! We're looking at all these exciting technologies (gpu, cell, etc.) but all these programming difficulties are definitely holding us back. A standardised solution like mpi would be great. There is a second outcome though - one of the manufacturers gets a virtual monopoly. (and my guess is it will be the manufacturer with the latest chip technology and closest integration with the processor ;-) -- Mattijs Janssens From Hakon.Bugge at scali.com Tue Jun 24 02:57:24 2008 From: Hakon.Bugge at scali.com (=?iso-8859-1?Q?H=E5kon?= Bugge) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: References: Message-ID: <20080624095726.CA12C35A660@mail.scali.no> Hi Mark, At 17:25 23.06.2008, Mark Hahn wrote: >>5) In which cases is it reasonable to switch on >>"Node memory interleaving" (in BIOS) for the >>application which uses more memory than is presented on the node ? > >I leave it off, since numactl --interleave lets >you get the same effect from user-space. I'm >not sure I've ever seen it be a win. numactl --interleave will for sure use (at least) page resolution performing the interleave. In my fantasy, I imagine the hardware interleave to be on a cache-line boundary. If that is the case, you might see differences though, for example if the threads/processes tend to generate cache misses to a single page, but different lines within that page. Probably an unlikely scenario for real applications... H?kon From Hakon.Bugge at scali.com Tue Jun 24 03:15:11 2008 From: Hakon.Bugge at scali.com (=?iso-8859-1?Q?H=E5kon?= Bugge) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <851570955.61161214286994032.JavaMail.root@zimbra.vpac.org> References: <20080623215924.GC14844@bx9.net> <851570955.61161214286994032.JavaMail.root@zimbra.vpac.org> Message-ID: <20080624101512.A0C2435A90B@mail.scali.no> At 07:56 24.06.2008, Chris Samuel wrote: > > Your MPI (and OpenMP) should do this for you. > >Although not always correctly, it may assume that it can >allocate from core 0 onwards leading to odd performance >issues if you happen to get two 4 CPU jobs running on the >same node.. If the MPI is doing that it is real stupid. Physical core IDs are physical ones. Users and applications should stay away from binding to physical resources. They can be used by others! These physical IDs varies greatly, and it has even been observed different enumeration on the same mothbd between boots. Imagine how to operate a cluster of these when users enumerate these IDs specifically ;-) IMHO, the MPI should virtualize these resources and relieve the end-user/application programmer from the burden. Here's one example using Scali MPI Connect running two jobs, each with four MPI processes on a dual-socket, quad-core Barcelona system: First job: (submon-1@barcelona-1) Affinity 'automatic' policy BANDWIDTH granularity CORE nprocs 4 (submon-1@barcelona-1) Will bind process 0 with mask=0000000000000001 [(board=0, socket=0, core=0, execunit=0)] (submon-1@barcelona-1) Will bind process 1 with mask=0000000000010000 [(board=0, socket=1, core=0, execunit=0)] (submon-1@barcelona-1) Will bind process 2 with mask=0000000000000010 [(board=0, socket=0, core=1, execunit=0)] (submon-1@barcelona-1) Will bind process 3 with mask=0000000000100000 [(board=0, socket=1, core=1, execunit=0)] Second job: (submon-1@barcelona-1) Affinity 'automatic' policy BANDWIDTH granularity CORE nprocs 4 (submon-1@barcelona-1) Will bind process 0 with mask=0000000000000100 [(board=0, socket=0, core=2, execunit=0)] (submon-1@barcelona-1) Will bind process 1 with mask=0000000001000000 [(board=0, socket=1, core=2, execunit=0)] (submon-1@barcelona-1) Will bind process 2 with mask=0000000000001000 [(board=0, socket=0, core=3, execunit=0)] (submon-1@barcelona-1) Will bind process 3 with mask=0000000010000000 [(board=0, socket=1, core=3, execunit=0)] and, this is with default settings. Other policies and 'resolutions' as we call it can be applied. H?kon From apittman at concurrent-thinking.com Tue Jun 24 03:33:17 2008 From: apittman at concurrent-thinking.com (Ashley Pittman) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] security for small, personal clusters In-Reply-To: References: Message-ID: <1214303597.6823.11.camel@bruce.priv.wark.uk.streamline-computing.com> On Fri, 2008-06-20 at 12:30 -0400, Mark Kosmowski wrote: > What kind of security is recommended for the owner of a small personal > cluster? Most likely perimeter security. I very much doubt there is information on the compute nodes which isn't on the head node and the compute nodes won't have any internet connection/bandwidth the frontend doesn't also have so they won't work as jumping-off points. If the front-end is compromised there is little or nothing to be gained from compromising the compute nodes so it's not worth the hassle or inconvenience to protect them. The one exception to this is passwords, don't use any service on the system which use plain-text passwords as these can be sniffed and people often use the same password for many accounts. > Where should the owner of a small, personal cluster go to > learn about security? Doing searches tends to give a few "head in the > sand" sites but predominantly seem to be oriented for the security > professional. It's no different than securing a standard Desktop machine or your laptop. Disable as many services on the external interface as you can. Ashley Pittman. From apittman at concurrent-thinking.com Tue Jun 24 05:23:00 2008 From: apittman at concurrent-thinking.com (Ashley Pittman) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <20080623215924.GC14844@bx9.net> References: <20080623215924.GC14844@bx9.net> Message-ID: <1214310180.6823.33.camel@bruce.priv.wark.uk.streamline-computing.com> On Mon, 2008-06-23 at 14:59 -0700, Greg Lindahl wrote: > > 3) Is there some reason to use things like > > mpirun -np N /usr/bin/numactl my_application ? > > Your MPI (and OpenMP) should do this for you. Of course how you want the binding to be done can depend on the communication pattern of the application, at least in the case when you aren't using all cpu's per node. Your MPI should pick sensible defaults for you. Ashley Pittman. From lynesh at cardiff.ac.uk Mon Jun 23 02:25:00 2008 From: lynesh at cardiff.ac.uk (Huw Lynes) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Powersave on Beowulf nodes In-Reply-To: <20080615121110.GA11985@ichec.ie> References: <20080615121110.GA11985@ichec.ie> Message-ID: <485F6BEC.4000407@cardiff.ac.uk> Eoin McHugh wrote: > I don't run a daemon but I use the Torque prologue and epilogue to set > the max and min power states using the /proc cpufreq interface before > and after jobs. This results in a power saving when a queued jobs are > waiting for additional nodes to free up. It's a measurably large > difference when the system is being drained prior to a downtime. Our > nodes are never shared beteen jobs making this easy to implement. We've been looking at power-saving measures on our new cluster. I've gradually come around to the idea that a compute node only really has two modes of efficiency: 1) running flat out 2) switched off at the wall In our first year of production we are unlikely to have jobs keeping the cluster full much of the time. So I'm considering writing something to watch the job queue and power down nodes that are not needed. The trick here being to work out some logic such that nodes aren't constantly being power-cycled. Obviously this isn't an option unless you have some form of lights-out control (e.g IPMI) on your nodes. Cheers, Huw -- Huw Lynes | Advanced Research Computing HEC Sysadmin | Cardiff University | Redwood Building, Tel: +44 (0) 29208 70626 | King Edward VII Avenue, CF10 3NB From kspaans at student.math.uwaterloo.ca Mon Jun 23 11:41:15 2008 From: kspaans at student.math.uwaterloo.ca (Kyle Spaans) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> Message-ID: <20080623184115.GA27571@student.math> On Fri, Jun 20, 2008 at 03:33:19PM -0400, Lawrence Stewart wrote: > More specifically for HPC, linux seems designed for the desktop, and > for small memory machines. That's funny, because I've heard people get scared that it was the complete opposite. That Linux was driven by Big Iron, and that no one cared about the "little desktop guy" (Con Kolivas is an interesting history example). From eugen at leitl.org Tue Jun 24 08:45:40 2008 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] FYI: HPC Server 2008 hits top 25 Message-ID: <20080624154540.GR9875@leitl.org> http://www.eweek.com/c/a/Infrastructure/Fastestever-Windows-HPC-Cluster/?kc=EWKNLENT062008STR1 Fastest-ever Windows HPC Cluster By Chris Preimesberger 2008-06-18 Running a beta of Windows HPC Server 2008, supercomputer hits 68.5 teraflops. Microsoft fastest-yet homegrown supercomputer, running the U.S. company's new Windows HPC Server 2008, debuted in the top 25 of the world's top 500 fastest supercomputers, as tested and operated by the National Center for Supercomputing Applications. The supercomputer, built and maintained in Urbana-Champaign, Illinois, ranked No. 23 in the world with a problem-solving performance of 68.5 teraflops. The announcement was made June 18 at the International Supercomputing conference in Dresden, Germany. The NCSA used the beta version of Windows HPC Server 2008 to reach the 68.5-teraflop (68.5 trillion floating point operations per second) level. It runs on commodity hardware and reported 77.7 percent application efficiency on 9,472 cores, making this facility one of the most powerful supercomputing systems in the world and the fastest Windows cluster to date. Most of the cores were made up of Intel Xeon quad-core chips. Storage for the system was about 6 terabytes. "When we deployed Windows on our cluster, which has more than 1,000 nodes, we went from bare metal to running the Linpack benchmark programs in just four hours," said Robert Pennington, deputy director of the NCSA. "The performance of Windows HPC Server 2008 has yielded efficiencies that are among the highest we've seen for this class of machine," Pennington said. Microsoft announced that the final release candidate version of Windows HPC Server 2008 will be available for download during the last week of June. Key features of Windows HPC Server 2008 include: high-speed networking; new, scalable cluster management tools; advanced failover capabilities; a service-oriented architecture job scheduler; and support for clustered file systems from some of Microsoft partners. Microsoft, which got rich producing PC operating systems and applications, has been investing heavily in research and development in high-performance computing for about the last eight years. In fact, the world's largest software company currently employs hundreds of engineers whose sole job it is to conceptualize and develop new products for the burgeoning HPC market, which research company IDC reports is the fast-growing sector in IT, at 19.5 percent per year. According to IDC, during the next five years the HPC server market is projected to show healthy, steady growth. The researcher expects revenue for the total HPC server market to expand at a compound annual growth rate of 9.2 percent to reach $18 billion by 2012. From apittman at concurrent-thinking.com Tue Jun 24 09:04:20 2008 From: apittman at concurrent-thinking.com (Ashley Pittman) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <20080623184115.GA27571@student.math> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> Message-ID: <1214323460.6823.69.camel@bruce.priv.wark.uk.streamline-computing.com> On Mon, 2008-06-23 at 14:41 -0400, Kyle Spaans wrote: > On Fri, Jun 20, 2008 at 03:33:19PM -0400, Lawrence Stewart wrote: > > More specifically for HPC, linux seems designed for the desktop, and > > for small memory machines. > > That's funny, because I've heard people get scared that it was the > complete opposite. That Linux was driven by Big Iron, and that no one > cared about the "little desktop guy" (Con Kolivas is an interesting > history example). Well that depends if you're at a Linux conference or a HPC conference, I regularly attend both and the Linux folks are convinced they spend too much time tending to Big Iron and the HPC folks are convinced that they spend too much time worrying about their mp3's skipping. I think Andrew Morton hit the nail on the head when he pointed out that from a kernel perspective Scientific Computing is more akin to embedded computing than it is to heavy server workloads and in a lot of cases HPC doesn't benefit from the attention that Big Iron does get. The solution for this would be for the HPC industry to employ more core kernel hackers however it would seem they are thin on the ground which is ironic as HPC is probably the one industry where Linux has the biggest market share. Ashley Pittman. From peter.st.john at gmail.com Tue Jun 24 09:20:25 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:15 2009 Subject: [Beowulf] Go-playing machines Message-ID: Programming a computer to play Go (an Asian strategy boardgame) has been difficult; some people say it's proof that Go is better or harder than chess, since computers can beat masters at chess but struggle at Go. (I think that statistically a game of go is about equivalent to a two-game match of chess; both games empty your brain quickly of course). My view is that while go may be somewhat harder to reduce to tree-searching, the main advantage of computer chess was an early start, e.g. von Neumann. This article: http://www.usgo.org/resources/downloads/CogApdx%20II-2.pdf describes recent trends in computer Go and mentions a 32-node cluster, 8 cores per node. Apparently MPI parallelization is recent for them and they are making good progress. Peter The game Go: http://en.wikipedia.org/wiki/Go_%28game%29 AGA (American Go Association): http://www.usgo.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080624/373d2814/attachment.html From James.P.Lux at jpl.nasa.gov Tue Jun 24 09:25:19 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <485920D8.2030309@ias.edu> <485AA481.2030000@vcu.edu> <200806201408.04839.kilian@stanford.edu> Message-ID: <6.2.5.6.2.20080624091802.02c11380@jpl.nasa.gov> At 02:31 PM 6/20/2008, Peter St. John wrote: >The destructive radius of Little Boy was about total, up to about >one mile radius, and tapered down to light at about two miles. So >being in a lead-lined steel container at 2000 meters might be OK for Indiana. > >In all action movies, blasts throw people unhurt for long distances; >when that much force (to impart that much momentum) would kill you. >That part is just conventional Hollywood. I could teach RGB to kick >me so that I fly through the air as in a Bruce Lee movie; it's a >stunt, and real kicks reallly hitting drop you like a sack of >potatoes, I've seen it. But not in movies. Similarly bullets, they >drill holes in you, if they pushed you through the air the recoil >would do the same to the shooter. > >As for the scene's good taste I can't say, I haven't seen the movie yet :-) > >Peter As someone who used to work in the business of doing this sort of thing (e.g. physical effects) for movies, TV shows, and commercials, you can assume that whatever you see on screen is specifically designed to "look like" what the director thinks will create the correct impression in the viewer. (e.g. real rain is invisible on film, for all intents and purposes..) For blasts (or kicks, etc.) flinging folks about, they use what is known as a "jerk vest" (for the high third derivative of position, not to describe the wearer) and bungee cords, springs, hydraulic winches, etc. Note well that the effects tech just runs the gear. A stunt person (aka human sandbag) survives the loads (and gives thanks to Stapp). To fling things about, we used a variety of things.. air power is popular, so is gunpowder. (look under a car that flips over for the piece of telephone pole used as the piston in a one-shot internal combustion engine.) Speaking of refrigerators, air pressure is just fine for a hundred meter or so launch. From kus at free.net Tue Jun 24 09:32:15 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Timers and TSC behaviour on SMP/x86 Message-ID: As I remember, TSCs in SMP/x86 are synchronized by Linux kernels at the boot process. But the only message (about TSC) I see after Linux boot in dmesg (or /var/log/messages) in SuSE 10.3 w/2.6.22 default kernel on quad-core dual socket Opteron serever is "Marking TSC unstable due to TSCs unsynchronized" Does it means that RDTSC-based timer (I use it for microbenchmarks) will give wrong results ? :-( Some additional information, according Software Optimization Guide for AMD Familly 10h Processors (quad-core) from Apr 4th, 2008: Early each AMD core had own TSC. Now quad-core processors have one common clock source in NorthBridge (BTW, is it in this case integrated into CPU chip - i.e. includes integrated memory controller and support of HT links ? - M.K.) - for all the TSCs of CPUs (cores ? - M.K.). The synchronization accuracy should be few tens cycles. Mikhail Kuzminsky Computer Assistance to Chemical Research Center Zelinsky Institute of Organic Chemistry Moscow From tjrc at sanger.ac.uk Tue Jun 24 09:46:17 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <20080623184115.GA27571@student.math> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> Message-ID: On 23 Jun 2008, at 7:41 pm, Kyle Spaans wrote: > On Fri, Jun 20, 2008 at 03:33:19PM -0400, Lawrence Stewart wrote: >> More specifically for HPC, linux seems designed for the desktop, and >> for small memory machines. > > That's funny, because I've heard people get scared that it was the > complete opposite. That Linux was driven by Big Iron, and that no > one cared about the "little desktop guy" (Con Kolivas is an > interesting history example). I think it depends on which bit of "Linux" you're talking about. I think Lawrence was really referring to the kernel (Which is of course the only bit that can actually be called Linux anyway). The impression is indeed the other way around when you look at the whole distribution, which is easier to target at server tasks than it is at a novice desktop user, and attempts to make the desktop really accessible to non-nerds are still very much in their infancy, IMHO. I guess the kernel really was developed for desktop machines, if we consider the sort of thing people did with Linux back in 1992 (which is when I first used it). Later people realised they could use this to save a bunch of money on big-iron traditional server tasks, and it did very nicely thank-you, and no-one really cared whether the performance was top-notch. After all, performance per dollar was superb (and still is). Even now, our little HPC community are probably pretty much the only people who really care that much about the performance of the OS kernel itself. Oh, OK, maybe some of the embedded guys do as well, given they're trying to run it on really tiny hardware. And in my little corner of the HPC world, genomics, where the vast majority of the crap run on my clusters is written in perl, python, ruby, java and R, I find it pretty difficult to get too worked up about kernel performance, memory latencies and cache miss rates. It's fairly irrelevant when I can't even get users to write code in a compiled language, where they can even attempt to control the layout of data in memory and get performance close to anything I would consider acceptable. You guys don't realise how lucky you are :-) Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From shaeffer at neuralscape.com Tue Jun 24 09:52:09 2008 From: shaeffer at neuralscape.com (Karen Shaeffer) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <1214323460.6823.69.camel@bruce.priv.wark.uk.streamline-computing.com> References: <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> <1214323460.6823.69.camel@bruce.priv.wark.uk.streamline-computing.com> Message-ID: <20080624165209.GB5427@synapse.neuralscape.com> On Tue, Jun 24, 2008 at 05:04:20PM +0100, Ashley Pittman wrote: > > On Mon, 2008-06-23 at 14:41 -0400, Kyle Spaans wrote: > > On Fri, Jun 20, 2008 at 03:33:19PM -0400, Lawrence Stewart wrote: > > > More specifically for HPC, linux seems designed for the desktop, and > > > for small memory machines. > > > > That's funny, because I've heard people get scared that it was the > > complete opposite. That Linux was driven by Big Iron, and that no one > > cared about the "little desktop guy" (Con Kolivas is an interesting > > history example). > > Well that depends if you're at a Linux conference or a HPC conference, I > regularly attend both and the Linux folks are convinced they spend too > much time tending to Big Iron and the HPC folks are convinced that they > spend too much time worrying about their mp3's skipping. I think Andrew > Morton hit the nail on the head when he pointed out that from a kernel > perspective Scientific Computing is more akin to embedded computing than > it is to heavy server workloads and in a lot of cases HPC doesn't > benefit from the attention that Big Iron does get. The solution for > this would be for the HPC industry to employ more core kernel hackers > however it would seem they are thin on the ground which is ironic as HPC > is probably the one industry where Linux has the biggest market share. > > Ashley Pittman. Hi Ashley, I think you got it exactly right. Linux is open to anyone who has an interest in extending the code base to support their special needs. What we are seeing is that the enterprise corporations have all hired teams of kernel developers to ensure their needs are being addressed. And the hpc market needs to do the same, if they want their interests to be addressed in the kernel as well. It's just that simple. What isn't simple is that folks often think their needs are more important than the other special interest groups. And since the enterprise corporations are paying all the key kernel developers like Andrew Morton, these same key kernel developers are protecting the interest of their employers. And Con Kolivas is a good example, when you look at it from that perspective. Money talks in the linux kernel today. How much influence any particular special interest group has in the kernel today is directly proportional to how many kernel developers are paid by that particular special interest group. I totally agree with your assessment. Thanks, Karen -- Karen Shaeffer Neuralscape, Palo Alto, Ca. 94306 shaeffer@neuralscape.com http://www.neuralscape.com From James.P.Lux at jpl.nasa.gov Tue Jun 24 10:19:20 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:16 2009 Subject: jerk vests, flying refrigerators, etc.Re: [Beowulf] Re: "hobbyists" In-Reply-To: <6.2.5.6.2.20080624091802.02c11380@jpl.nasa.gov> References: <485920D8.2030309@ias.edu> <485AA481.2030000@vcu.edu> <200806201408.04839.kilian@stanford.edu> <6.2.5.6.2.20080624091802.02c11380@jpl.nasa.gov> Message-ID: <6.2.5.6.2.20080624100951.02c6fd58@jpl.nasa.gov> At 09:25 AM 6/24/2008, Jim Lux wrote: >At 02:31 PM 6/20/2008, Peter St. John wrote: >>The destructive radius of Little Boy was about total, up to about >>one mile radius, and tapered down to light at about two miles. So >>being in a lead-lined steel container at 2000 meters might be OK for Indiana. >> >>In all action movies, blasts throw people unhurt for long >>distances; when that much force (to impart that much momentum) >>would kill you. That part is just conventional Hollywood. I could >>teach RGB to kick me so that I fly through the air as in a Bruce >>Lee movie; it's a stunt, and real kicks reallly hitting drop you >>like a sack of potatoes, I've seen it. But not in movies. Similarly >>bullets, they drill holes in you, if they pushed you through the >>air the recoil would do the same to the shooter. >> >>As for the scene's good taste I can't say, I haven't seen the movie yet :-) >> >>Peter > >As someone who used to work in the business of doing this sort of >thing (e.g. physical effects) for movies, TV shows, and commercials, >you can assume that whatever you see on screen is specifically >designed to "look like" what the director thinks will create the >correct impression in the viewer. (e.g. real rain is invisible on >film, for all intents and purposes..) > >For blasts (or kicks, etc.) flinging folks about, they use what is >known as a "jerk vest" (for the high third derivative of position, >not to describe the wearer) and bungee cords, springs, hydraulic >winches, etc. Note well that the effects tech just runs the >gear. A stunt person (aka human sandbag) survives the loads (and >gives thanks to Stapp). > >To fling things about, we used a variety of things.. air power is >popular, so is gunpowder. (look under a car that flips over for the >piece of telephone pole used as the piston in a one-shot internal >combustion engine.) Speaking of refrigerators, air pressure is just >fine for a hundred meter or so launch. For some examples: jerk vest test http://www.reelefx.com/index.php?c=effect.view&id=260 refrigerator launch http://www.reelefx.com/index.php?c=effect.view&id=124 http://www.reelefx.com/ gives a lot of examples http://www.reelefx.com/index.php?c=effect.list&id=15 has variety of specific effect tests. Search for 'air mortar' for launches Naturally, I'm particularly proud of the tornado and multicam, since I helped make them... http://www.reelefx.com/index.php?c=effect.list&id=13 tornados http://www.reelefx.com/index.php?c=effect.view&id=153 from Swordfish http://www.reelefx.com/index.php?c=effect.view&id=169 (stuff done more recently, now with digital cameras, which makes life MUCH easier..) Eadweard Muybridge would be proud of us. From hvidal at tesseract-tech.com Tue Jun 24 10:20:07 2008 From: hvidal at tesseract-tech.com (H.Vidal, Jr.) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Timers and TSC behaviour on SMP/x86 In-Reply-To: References: Message-ID: <48612CC7.2030203@tesseract-tech.com> If memory serves, TSC sync is an option when configuring a custom 2.4x series kernel.....perhaps this is so for 2.6x as well? hv Mikhail Kuzminsky wrote: > As I remember, TSCs in SMP/x86 are synchronized by Linux kernels at the > boot process. > But the only message (about TSC) I see after Linux boot in dmesg (or > /var/log/messages) in SuSE 10.3 w/2.6.22 default kernel on quad-core > dual socket Opteron serever is > > "Marking TSC unstable due to TSCs unsynchronized" > > Does it means that RDTSC-based timer (I use it for microbenchmarks) will > give wrong results ? :-( > > Some additional information, > according Software Optimization Guide for AMD Familly 10h Processors > (quad-core) from Apr 4th, 2008: > > Early each AMD core had own TSC. Now quad-core processors have one > common clock source in NorthBridge (BTW, is it in this case integrated > into CPU chip - i.e. includes integrated memory controller and support > of HT links ? - M.K.) - for all the TSCs of CPUs (cores ? - M.K.). > > The synchronization accuracy should be few tens cycles. > > Mikhail Kuzminsky > Computer Assistance to Chemical Research Center > Zelinsky Institute of Organic Chemistry > Moscow > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From i.kozin at dl.ac.uk Tue Jun 24 10:39:54 2008 From: i.kozin at dl.ac.uk (Kozin, I (Igor)) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] FYI: HPC Server 2008 hits top 25 In-Reply-To: <20080624154540.GR9875@leitl.org> Message-ID: > In fact, the world's largest software company currently employs hundreds of > engineers whose sole job it is to conceptualize and develop new products for > the burgeoning HPC market Oh, really? I hardly see any truth in this sentence. Perhaps "the world's largest"? From peter.st.john at gmail.com Tue Jun 24 10:47:36 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:16 2009 Subject: jerk vests, flying refrigerators, etc.Re: [Beowulf] Re: "hobbyists" In-Reply-To: <6.2.5.6.2.20080624100951.02c6fd58@jpl.nasa.gov> References: <485920D8.2030309@ias.edu> <485AA481.2030000@vcu.edu> <200806201408.04839.kilian@stanford.edu> <6.2.5.6.2.20080624091802.02c11380@jpl.nasa.gov> <6.2.5.6.2.20080624100951.02c6fd58@jpl.nasa.gov> Message-ID: Jim, Well I was thinking of low-budget, Hong Kong in the 70's type special effects, and just meant that things don't really fly around as much as we see in the movies. That said, I'm a huge fan of stuff flying around, so thanks :-) Peter On 6/24/08, Jim Lux wrote: > > At 09:25 AM 6/24/2008, Jim Lux wrote: > >> At 02:31 PM 6/20/2008, Peter St. John wrote: >> >>> The destructive radius of Little Boy was about total, up to about one >>> mile radius, and tapered down to light at about two miles. So being in a >>> lead-lined steel container at 2000 meters might be OK for Indiana. >>> >>> In all action movies, blasts throw people unhurt for long distances; when >>> that much force (to impart that much momentum) would kill you. That part is >>> just conventional Hollywood. I could teach RGB to kick me so that I fly >>> through the air as in a Bruce Lee movie; it's a stunt, and real kicks >>> reallly hitting drop you like a sack of potatoes, I've seen it. But not in >>> movies. Similarly bullets, they drill holes in you, if they pushed you >>> through the air the recoil would do the same to the shooter. >>> >>> As for the scene's good taste I can't say, I haven't seen the movie yet >>> :-) >>> >>> Peter >>> >> >> As someone who used to work in the business of doing this sort of thing >> (e.g. physical effects) for movies, TV shows, and commercials, you can >> assume that whatever you see on screen is specifically designed to "look >> like" what the director thinks will create the correct impression in the >> viewer. (e.g. real rain is invisible on film, for all intents and >> purposes..) >> >> For blasts (or kicks, etc.) flinging folks about, they use what is known >> as a "jerk vest" (for the high third derivative of position, not to describe >> the wearer) and bungee cords, springs, hydraulic winches, etc. Note well >> that the effects tech just runs the gear. A stunt person (aka human >> sandbag) survives the loads (and gives thanks to Stapp). >> >> To fling things about, we used a variety of things.. air power is popular, >> so is gunpowder. (look under a car that flips over for the piece of >> telephone pole used as the piston in a one-shot internal combustion engine.) >> Speaking of refrigerators, air pressure is just fine for a hundred meter or >> so launch. >> > > For some examples: > jerk vest test > http://www.reelefx.com/index.php?c=effect.view&id=260 > > refrigerator launch > http://www.reelefx.com/index.php?c=effect.view&id=124 > > http://www.reelefx.com/ gives a lot of examples > > > http://www.reelefx.com/index.php?c=effect.list&id=15 > > has variety of specific effect tests. Search for 'air mortar' for launches > > > Naturally, I'm particularly proud of the tornado and multicam, since I > helped make them... > > http://www.reelefx.com/index.php?c=effect.list&id=13 tornados > http://www.reelefx.com/index.php?c=effect.view&id=153 from Swordfish > http://www.reelefx.com/index.php?c=effect.view&id=169 (stuff done more > recently, now with digital cameras, which makes life MUCH easier..) > Eadweard Muybridge would be proud of us. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080624/8c0350cc/attachment.html From peter.st.john at gmail.com Tue Jun 24 10:54:05 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] FYI: HPC Server 2008 hits top 25 In-Reply-To: References: <20080624154540.GR9875@leitl.org> Message-ID: MS isn't famous for new development (as scientists see "new development") but a few hundred engineers would not be such a big investment for them. I'd be more interested in a comparison of that many whatever, quad core xeon? running linux vs MS's running Cluster Server 2008. CP/M could be a pretty powerful OS if it ran on enough nodes :-) and of course I think of XP as CP/M v.99 (approximately) (although that isn't fair to VMS, the forebear of NT) Peter On 6/24/08, Kozin, I (Igor) wrote: > > > > In fact, the world's largest software company currently employs > hundreds of > > engineers whose sole job it is to conceptualize and develop new > products for > > the burgeoning HPC market > > > Oh, really? I hardly see any truth in this sentence. > Perhaps "the world's largest"? > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080624/4fb47f8c/attachment.html From dnlombar at ichips.intel.com Tue Jun 24 11:01:17 2008 From: dnlombar at ichips.intel.com (Lombard, David N) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> Message-ID: <20080624180117.GA26194@nlxdcldnl2.cl.intel.com> On Tue, Jun 24, 2008 at 05:46:17PM +0100, Tim Cutts wrote: > > On 23 Jun 2008, at 7:41 pm, Kyle Spaans wrote: > > >On Fri, Jun 20, 2008 at 03:33:19PM -0400, Lawrence Stewart wrote: > >>More specifically for HPC, linux seems designed for the desktop, and > >>for small memory machines. > > > >That's funny, because I've heard people get scared that it was the > >complete opposite. That Linux was driven by Big Iron, and that no > >one cared about the "little desktop guy" (Con Kolivas is an > >interesting history example). > > I think it depends on which bit of "Linux" you're talking about. Doesn't it always? > Even now, our little HPC community are probably pretty much the only > people who really care that much about the performance of the OS > kernel itself. Oh, OK, maybe some of the embedded guys do as well, > given they're trying to run it on really tiny hardware. HPC use very few system calls and make little use of those calls beyond large-block sequential I/O and our application networks; there are few execution threads to stress the scheduler and memory is usually very carefully managed. Transaction workloads--think TPCx--spend a LOT more time in the kernel than pretty much anything HPC. Also consider the IOPS (I/O per second) metrics that the transaction and non-HPC storage providers live and die by, and that we just don't care about. -- David N. Lombard, Intel, Irvine, CA I do not speak for Intel Corporation; all comments are strictly my own. From peter.st.john at gmail.com Tue Jun 24 11:52:01 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] FYI: HPC Server 2008 hits top 25 In-Reply-To: <356685033.681601214331159293.JavaMail.root@zimbra-1.ncsa.uiuc.edu> References: <557589079.681451214331034355.JavaMail.root@zimbra-1.ncsa.uiuc.edu> <356685033.681601214331159293.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: The latter shows "Windows Server 2008/Red Hat Linux"; I assume both were used with that hardware, seperately, and it made no difference at all? So the application overwhelmed the impact of the OS, to as much precision as say 89.59? Wow. Thanks, Peter On 6/24/08, Galen Arnold wrote: > > linux > #14 at 1st appearance on top500 at : > http://www.top500.org/list/2007/11/100 > > windows > #23 at 2nd appearance on top500: http://www.top500.org/list/2008/06/100 > > compare the performance numbers...you didn't see anything. > > > Galen Arnold > system engineer > NCSA > > > ----- Original Message ----- > From: "Peter St. John" > To: "I Kozin (Igor)" > Cc: Beowulf@beowulf.org > Sent: Tuesday, June 24, 2008 12:54:05 PM GMT -06:00 US/Canada Central > Subject: Re: [Beowulf] FYI: HPC Server 2008 hits top 25 > > MS isn't famous for new development (as scientists see "new development") > but a few hundred engineers would not be such a big investment for them. > > I'd be more interested in a comparison of that many whatever, quad core > xeon? running linux vs MS's running Cluster Server 2008. CP/M could be a > pretty powerful OS if it ran on enough nodes :-) and of course I think of > XP > as CP/M v.99 (approximately) (although that isn't fair to VMS, the forebear > of NT) > > Peter > > On 6/24/08, Kozin, I (Igor) wrote: > > > > > > > In fact, the world's largest software company currently employs > > hundreds of > > > engineers whose sole job it is to conceptualize and develop new > > products for > > > the burgeoning HPC market > > > > > > Oh, really? I hardly see any truth in this sentence. > > Perhaps "the world's largest"? > > > > > > > > > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080624/8584669d/attachment.html From prentice at ias.edu Tue Jun 24 12:26:55 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> References: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> Message-ID: <48614A7F.2080606@ias.edu> Vincent Diepeveen wrote: > That said, it has improved a lot, now all we need is a better compiler > for linux. GCC is for my chessprogram generating an > executable that gets 22% slower positions per second than visual c++ > 2005 is. > > Thanks, > Vincent > GCC is a free compiler, and Visual C++ is a commercial product. That's not really a fair comparison. Stop being a cheapskate and buy a commercial compiler for Linux. There are plenty available that promise better performance than GCC: Intel Compilers: http://www.intel.com/cd/software/products/asmo-na/eng/compilers/284264.htm Portland Group Compilers: http://www.pgroup.com/products/workpgi.htm Pathscale Compiler Suite: http://www.pathscale.com/ Sun Studio 12: http://developers.sun.com/sunstudio/ -- Prentice From diep at xs4all.nl Tue Jun 24 12:49:53 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: Message-ID: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> Go has a bigger branching factor than chess, as it starts with an empty board of 19x19, versus chess a loaded board of 8x8. The first few moves in go decide the outcome of the game already, as the rest is just a 'playout' of the first few moves. So what matters most is the first few moves in the game. It is easier to search selective in go than it is in chess. In chess selective searching is really tough to get to work well. In go you can throw away majority of the moves with near 100% sureness, some even with 100% sureness. Reason why chessprograms play so well is simply money and popularity of the game. Chess computers in the 80s and start 90s, used to export to 106 countries. I remember talking about producing a dedicated chesscomputer, and usually 100000 of them get printed. A minimum of 20000 pieces is needed to heat up the production line (Hong Kong, China). There is no go computers AFAIK, for simple reason that the only nation where you can sell your product is Japan. The 3 main nations where go gets played is China, Korea, Japan. So only Japan you could sell some if you have entrance to its very close market. In fact there is even a company that claims to have the rights on all human go games. At my chat is someone, Gian-Carlo Pascutto, whose program Leela you can buy. It is as we speak the strongest commercial go program on planet earth that you can buy. His engine focuses upon search, its knowlede is rather simplistic. He has a normal job just like you have one. So this is a sparetime written engine. Computerchess engines used to be fulltime work. When someone is jobless like me, you again work for a few months fulltime at it. There is 500 chess engines to compete with or so. In go the competition is very limited, only recently more engines are there. Most programmed by non-asian programmers. Not even from Asian decent. It's all about how much money you want to put in research. Would go have been the game been played in 106 nations and chess in just 3 from which only 1 has money and is a closed market, then we would be speaking now about a computer-go world champion program and wondering what makes computer chess so hard. In that case I would write then that if more money had been put at chess, that those engines would be stronger than the go engines. Don't count at it that the big supercomputers make any chance in go, neither in chess. The quality of the program is most important. As soon as you massively parallellize a strong engine, now *that* makes sense. Vincent On Jun 24, 2008, at 6:20 PM, Peter St. John wrote: > Programming a computer to play Go (an Asian strategy boardgame) has > been difficult; some people say it's proof that Go is better or > harder than chess, since computers can beat masters at chess but > struggle at Go. (I think that statistically a game of go is about > equivalent to a two-game match of chess; both games empty your > brain quickly of course). My view is that while go may be somewhat > harder to reduce to tree-searching, the main advantage of computer > chess was an early start, e.g. von Neumann. > > This article: > http://www.usgo.org/resources/downloads/CogApdx%20II-2.pdf > describes recent trends in computer Go and mentions a 32-node > cluster, 8 cores per node. Apparently MPI parallelization is recent > for them and they are making good progress. > > Peter > > The game Go: http://en.wikipedia.org/wiki/Go_%28game%29 > AGA (American Go Association): http://www.usgo.org > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From peter.st.john at gmail.com Tue Jun 24 13:15:42 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> References: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> Message-ID: Vincent, I found your reply very agreable except this: "The first few moves in go decide the outcome of the game already, as the rest is just a 'playout' of the first few moves. So what matters most is the first few moves in the game." Many professional games are decided in the endgame. When I lose at chess, it's almost always resignation within 50 moves; when I lose at go, it's frequently necessary to count, less frequently resignation (but handicaps make more games close), and I think I never resign before move 150 or so. In fact, some crazy mathematicians proved that the endgame in go is very tricky, see http://senseis.xmp.net/?TemperatureCGT (using "temperature" from combinatorial game theory). As it happens I'm really bad at fuseki, and often have to catch up with fighting in the middle game, which often leads to resentful squeezing of the yose :-) So I rarely have quick games. Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080624/43719cf2/attachment.html From diep at xs4all.nl Tue Jun 24 13:21:01 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <48614A7F.2080606@ias.edu> References: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> <48614A7F.2080606@ias.edu> Message-ID: <520EC3EF-536A-4BD8-9806-492C23D2A058@xs4all.nl> Not long ago GCP and me did do a comparision, and visual c++ kicked all of them by a rather large margin. intel c++ obviously is close to visual studio. Within 0.5% to 1.5% range (depending upon flags and hidden flags that you managed to get from someone). Intel C++ is free for researchers such as me. The PG compiler and especially pathscale compiler are doing rather well at benchmarks, especially that last, yet at our codes they're real ugly. Maybe they do better for floating point oriented workloads, which doesn't describe game tree search. What strikes me is that visual studio/intel c++ produce a far tinier executable than either of those compilers and that the fastest executable (in case of Diep) is a 32 bits one. 32 bits visual c++ executable with pgo is roughly 1.5% faster than its 64 bits counterpart. The main datastructures are using integers and a lot of pointers get used. Seems in 64 bits this hurts a tad. Vincent On Jun 24, 2008, at 9:26 PM, Prentice Bisbal wrote: > Vincent Diepeveen wrote: > >> That said, it has improved a lot, now all we need is a better >> compiler >> for linux. GCC is for my chessprogram generating an >> executable that gets 22% slower positions per second than visual c++ >> 2005 is. >> >> Thanks, >> Vincent >> > > GCC is a free compiler, and Visual C++ is a commercial product. That's > not really a fair comparison. Stop being a cheapskate and buy a > commercial compiler for Linux. There are plenty available that promise > better performance than GCC: > > Intel Compilers: > http://www.intel.com/cd/software/products/asmo-na/eng/compilers/ > 284264.htm > > Portland Group Compilers: > http://www.pgroup.com/products/workpgi.htm > > Pathscale Compiler Suite: > http://www.pathscale.com/ > > Sun Studio 12: > http://developers.sun.com/sunstudio/ > > -- > Prentice > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From diep at xs4all.nl Tue Jun 24 13:29:20 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> Message-ID: Hi Peter, At the risk of being off topic. A few points. First of all we must distinguish computers and humans. For chess and go this is completely the same: the choices that get made at start of the game decide its outcome with a high degree of certainty; In computer chess i launched the slogan a few years ago: "the move that matters most for the game is the first move out of book" Being a titled player myself (FIDE master), this was of course easy to judge and it still is the case. This might get explained especially because of the limited knowledge that a program has as compared to a human playing. At human level, if we look to what happens in go at the highest level, is that nearly all time gets spent the first few moves of the game. Chess and Go and 10x0 international checkers are all the same in that respect. The real strong titled players are that strong that it is relative easy for them to play the last few moves based upon technique to assure the win, as opposed to the start of the game when 'book theory' ends. In go of course that last is rather soon. Vincent On Jun 24, 2008, at 10:15 PM, Peter St. John wrote: > Vincent, > > I found your reply very agreable except this: > > "The first few moves in go decide the outcome of the game already, > as the rest is just a 'playout' of the first few moves. So what > matters most is the first few moves in the game." > > Many professional games are decided in the endgame. When I lose at > chess, it's almost always resignation within 50 moves; when I lose > at go, it's frequently necessary to count, less frequently > resignation (but handicaps make more games close), and I think I > never resign before move 150 or so. In fact, some crazy > mathematicians proved that the endgame in go is very tricky, see > http://senseis.xmp.net/?TemperatureCGT (using "temperature" from > combinatorial game theory). > > As it happens I'm really bad at fuseki, and often have to catch up > with fighting in the middle game, which often leads to resentful > squeezing of the yose :-) So I rarely have quick games. > > Peter > > From peter.st.john at gmail.com Tue Jun 24 13:46:43 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> Message-ID: Vincent, FIDE Master is very cool :-) I'm only 2000; shodan in go. The first move out of book may indeed be the move that matters most (in chess) but in Go, the connection between the end of fuseki and the technique of exploiting yose seems very remote. Since I can beat all computers at Go, I assume it's remote for the machines also :-) because the correct valuation of "influence" vs "territory" seems intractable. 1.e4 Peter On 6/24/08, Vincent Diepeveen wrote: > > Hi Peter, > > At the risk of being off topic. A few points. > > First of all we must distinguish computers and humans. For chess and go > this is completely the same: > the choices that get made at start of the game decide its outcome with a > high degree of certainty; > > In computer chess i launched the slogan a few years ago: > "the move that matters most for the game is the first move out of > book" > > Being a titled player myself (FIDE master), > this was of course easy to judge and it still is the case. > > This might get explained especially because of the limited knowledge that a > program has as compared to a human playing. > > At human level, if we look to what happens in go at the highest level, is > that nearly all time gets spent the first few moves of the game. > > Chess and Go and 10x0 international checkers are all the same in that > respect. > > The real strong titled players are that strong that it is relative easy for > them to play the last few moves based upon technique to assure the win, > as opposed to the start of the game when 'book theory' ends. In go of > course that last is rather soon. > > Vincent > > On Jun 24, 2008, at 10:15 PM, Peter St. John wrote: > > Vincent, >> >> I found your reply very agreable except this: >> >> "The first few moves in go decide the outcome of the game already, as the >> rest is just a 'playout' of the first few moves. So what matters most is the >> first few moves in the game." >> >> Many professional games are decided in the endgame. When I lose at chess, >> it's almost always resignation within 50 moves; when I lose at go, it's >> frequently necessary to count, less frequently resignation (but handicaps >> make more games close), and I think I never resign before move 150 or so. In >> fact, some crazy mathematicians proved that the endgame in go is very >> tricky, see http://senseis.xmp.net/?TemperatureCGT (using "temperature" >> from combinatorial game theory). >> >> As it happens I'm really bad at fuseki, and often have to catch up with >> fighting in the middle game, which often leads to resentful squeezing of the >> yose :-) So I rarely have quick games. >> >> Peter >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080624/c5d01acf/attachment.html From arnoldg at ncsa.uiuc.edu Tue Jun 24 11:12:39 2008 From: arnoldg at ncsa.uiuc.edu (Galen Arnold) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] FYI: HPC Server 2008 hits top 25 In-Reply-To: <557589079.681451214331034355.JavaMail.root@zimbra-1.ncsa.uiuc.edu> Message-ID: <356685033.681601214331159293.JavaMail.root@zimbra-1.ncsa.uiuc.edu> linux #14 at 1st appearance on top500 at : http://www.top500.org/list/2007/11/100 windows #23 at 2nd appearance on top500: http://www.top500.org/list/2008/06/100 compare the performance numbers...you didn't see anything. Galen Arnold system engineer NCSA ----- Original Message ----- From: "Peter St. John" To: "I Kozin (Igor)" Cc: Beowulf@beowulf.org Sent: Tuesday, June 24, 2008 12:54:05 PM GMT -06:00 US/Canada Central Subject: Re: [Beowulf] FYI: HPC Server 2008 hits top 25 MS isn't famous for new development (as scientists see "new development") but a few hundred engineers would not be such a big investment for them. I'd be more interested in a comparison of that many whatever, quad core xeon? running linux vs MS's running Cluster Server 2008. CP/M could be a pretty powerful OS if it ran on enough nodes :-) and of course I think of XP as CP/M v.99 (approximately) (although that isn't fair to VMS, the forebear of NT) Peter On 6/24/08, Kozin, I (Igor) wrote: > > > > In fact, the world's largest software company currently employs > hundreds of > > engineers whose sole job it is to conceptualize and develop new > products for > > the burgeoning HPC market > > > Oh, really? I hardly see any truth in this sentence. > Perhaps "the world's largest"? > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From lindahl at pbm.com Tue Jun 24 14:07:06 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <520EC3EF-536A-4BD8-9806-492C23D2A058@xs4all.nl> References: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> <48614A7F.2080606@ias.edu> <520EC3EF-536A-4BD8-9806-492C23D2A058@xs4all.nl> Message-ID: <20080624210705.GB17964@bx9.net> On Tue, Jun 24, 2008 at 10:21:01PM +0200, Vincent Diepeveen wrote: > The PG compiler and especially pathscale compiler are doing rather > well at benchmarks, > especially that last, yet at our codes they're real ugly. Maybe they > do better for floating point > oriented workloads, which doesn't describe game tree search. There are certainly unusual codes out there, and PathScale has gotten a lot of examples sent in by customers, thanks to the "if we're slower than someone else, it's a bug" philosophy. This allowed us to improve the compiler on a lot of non-benchmark codes. In your case, I'd suggest that you use pathopt to search for better flags. -- greg From diep at xs4all.nl Tue Jun 24 14:28:46 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <20080624210705.GB17964@bx9.net> References: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> <48614A7F.2080606@ias.edu> <520EC3EF-536A-4BD8-9806-492C23D2A058@xs4all.nl> <20080624210705.GB17964@bx9.net> Message-ID: <764CE5DC-C6B2-463A-AC27-85E932D50A67@xs4all.nl> What would interest me is if you describe how you get your information on how instructions pair and what weak sequences are at the processor. Like for example the by now old AMD K8 used to have the feature that if you do an integer multiplication, that the first and last cycle of the latency, it blocks all other execution units. Besides that this still kicks butt compared to intel's implementation of integer multiplication (proof of this statement: Most of GMP's functions are integer multiplication dominated and AMD k8 already murders core2 as a result of that), this is total crucial to know when building a compiler. Did this information already sit in Pathscales database of "processor behaviour"? If so, where did you get the knowledge? Vincent On Jun 24, 2008, at 11:07 PM, Greg Lindahl wrote: > On Tue, Jun 24, 2008 at 10:21:01PM +0200, Vincent Diepeveen wrote: > >> The PG compiler and especially pathscale compiler are doing rather >> well at benchmarks, >> especially that last, yet at our codes they're real ugly. Maybe they >> do better for floating point >> oriented workloads, which doesn't describe game tree search. > > There are certainly unusual codes out there, and PathScale has gotten > a lot of examples sent in by customers, thanks to the "if we're slower > than someone else, it's a bug" philosophy. This allowed us to improve > the compiler on a lot of non-benchmark codes. > > In your case, I'd suggest that you use pathopt to search for better > flags. > > -- greg > > From hahn at mcmaster.ca Tue Jun 24 15:20:35 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <20080623184115.GA27571@student.math> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> Message-ID: >> More specifically for HPC, linux seems designed for the desktop, and >> for small memory machines. the only justice I can see in that is that there hasn't been all that much effort to get bigpages widely/easily used. in particular, I don't see that scheduler or general memory-management issues in linux are particularly biased for desktop or against HPC. > That's funny, because I've heard people get scared that it was the complete > opposite. That Linux was driven by Big Iron, and that no one cared about > the "little desktop guy" (Con Kolivas is an interesting history example). Con didn't play the game right - you have to have the right combination of social engineering (especially timing and proactive response) and good tech kungfoo. kernel people are biased towards a certain aesthetic that doesn't punish big-picture redesigns from scratch, but _does_ punish solutions in search of a problem. so the question is, if you had a magic wand, what would you change in the kernel (or perhaps libc or other support libs, etc)? most of the things I can think of are not clear-cut. I'd like to be able to give better info from perf counters to our users (but I don't think Linux is really in the way). I suspect we lose some performance due to jitter injected by the OS (and/or our own monitoring) and would like to improve, but again, it's hard to blame Linux. I'd love to have better options for cluster-aware filesystems. kernel-assisted network shared memory? From csamuel at vpac.org Tue Jun 24 16:19:37 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <20080624063811.GA5872@bx9.net> Message-ID: <1472215019.65571214349577120.JavaMail.root@zimbra.vpac.org> ----- "Greg Lindahl" wrote: > > > Your MPI (and OpenMP) should do this for you. > > > > Although not always correctly, it may assume that it can > > allocate from core 0 onwards leading to odd performance > > issues if you happen to get two 4 CPU jobs running on the > > same node.. > > Most clusters I've seen aren't used that way (whole nodes only), I don't know of a single cluster in Australia that's managed that way, presumably because there's far less money for such systems than elsewhere and we have to try and do a lot more with a lot less (we can't afford to waste idle CPUs). The Australian state of the art peak computing facility in Canberra just dropped out of the Top500 for instance. > but sure, you can always consult the manual in that case. IIRC this was with MVAPICH and we had to consult the source to find the undocumented environment variable to fix it. :) We switched to OpenMPI and use Torque's 2.6 kernel cpuset support instead, with local patches. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From csamuel at vpac.org Tue Jun 24 16:23:08 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <20080624101512.A0C2435A90B@mail.scali.no> Message-ID: <674893953.65671214349788737.JavaMail.root@zimbra.vpac.org> ----- "H?kon Bugge" wrote: > IMHO, the MPI should virtualize these resources > and relieve the end-user/application programmer > from the burden. IMHO the resource manager (Torque, SGE, LSF, etc) should be setting up cpusets for the jobs based on what the scheduler has told it to use and the MPI shouldn't get a choice in the matter. :-) Also helps when newbies run OpenMP codes thinking they're single CPU codes and get 3 or 4 on the same 8 CPU node. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From csamuel at vpac.org Tue Jun 24 16:39:45 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Powersave on Beowulf nodes In-Reply-To: <498008047.65831214350692356.JavaMail.root@zimbra.vpac.org> Message-ID: <921466112.65851214350785152.JavaMail.root@zimbra.vpac.org> ----- "Huw Lynes" wrote: Syt mae Huw, /* exhausts first year Welsh */ > In our first year of production we are unlikely to have jobs keeping > the cluster full much of the time. So I'm considering writing something to > watch the job queue and power down nodes that are not needed. The > trick here being to work out some logic such that nodes aren't constantly > being power-cycled. Obviously this isn't an option unless you have > some form of lights-out control (e.g IPMI) on your nodes. FWIW the Moab scheduler supports power managing cluster nodes via callouts to utilities (e.g. the IPMI tools) to do such things, including hysteresis for nodes to prevent over-eager cycling. I know that Swinburne Uni here uses it for that on their Dell cluster "Green", as you can see from their public Ganglia interface: http://green.ssi.swin.edu.au/ganglia/ cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From larry.stewart at sicortex.com Tue Jun 24 18:11:06 2008 From: larry.stewart at sicortex.com (Lawrence Stewart) Date: Wed Nov 25 01:07:16 2009 Subject: Linux magic wand - was Re: [Beowulf] Re: "hobbyists" In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> Message-ID: <48619B2A.8030604@sicortex.com> Mark Hahn wrote: > > so the question is, if you had a magic wand, what would you change in > the kernel (or perhaps libc or other support libs, etc)? most of the > things I can think of are not clear-cut. I'd like to be able to give > better info from perf counters to our users (but I don't think Linux > is really in the way). I suspect we lose some performance due to jitter > injected by the OS (and/or our own monitoring) and would like to improve, > but again, it's hard to blame Linux. I'd love to have better options > for cluster-aware filesystems. kernel-assisted network shared memory? > _______________________________________________ There's a good rant to be written for Usenix or the Ottowa Linux Symposium I suspect. VM - 4096 is small now. In 1976 a page was 512 bytes. It moved to 4096 in the mid '90s? I forget. Since then computers and memory bandwidths are much bigger and faster. The telling point for me was that I took a look at a running system and there were only a couple of VM areas in service, so page breakage amounts to almost nothing. We run with 64K pages and plan to experiment with much larger ones. One could argue about thread stacks, but I think that threads and HPC don't mix well, so there won't be that many. I am aware of the great debate about the right way to program high core-count nodes, but I doubt that more threads than processors is the right answer. Linux also has pretty poor mechanisms for keeping physical memory contiguous, the slabs tend to get fragmented, which is why the big page stuff and things like bigphysarea get preallocated. There's no good reason why you couldn't compact memory on the fly. The VM system is also in the way of OS bypass RDMA NICs - you either get large kernel patches like Quadrics to let virtual RDMA work, or you get pinning and registration and other performance sapping cruft. The new external-pager stuff may help a lot here, I haven't looked at it yet. I/O system The block device layer has 512 byte sectors wired in, and is solely useful for devices that you own exclusively. You've got queueing going on at multiple levels, I think because the architecture has assumptions about cpu/disk performance ratios baked in. And the segments of a bio have to complete in order, what's that about? A little one we ran into here is that the block I/O system doesn't know if an I/O is to satisfy an I stream page fault or a D stream page fault. Consequently if your L1 Icache is not coherent (and few are) you have to flush it on all read completions. A little book keeping would solve that. (I hope I am wrong about this one!) File systems Agree complelely about cluster aware FS. We struggle with the Lustre patch sets, which may be an extreme case. Performance stuff We are big users of the PAPI infrastructure, which is pretty good, but once you step off that train you have to deal with things like sysfs. So we're trying to read hardware counters without undue disturbance to running HPC applications, and the advice of Linux is to make a system call for each value, converted to ascii. This makes sense for slow admin stuff but not for performance data. At least it isn't XML. Runtime system I tend towards thinking we would be better off without shared libraries. Memory is big, programs are generally small. There is a lot of complexity here, to which I am allergic. To the extent that shared libraries make the program slower (due to separate segments for library data, for example), lets get rid of them. Two arguments in favor are when the library is implementing a system service chosen by the admin, rather than the programmer (PAM modules), and there is this talk about MPI ABIs, so applications can use alternate packages without relinking. I think that is a bad idea too, but it is off-topic. OS noise This becomes a big issue in large systems. There's way too much stuff running in linux, each piece separately designed, each thread with its own notions of timing and periodic wakeups. Maybe the OS should run on a separate node altogether, and you communicate with it via RDMA. All that is left behind is maybe memory management. -L From rgb at phy.duke.edu Tue Jun 24 18:22:17 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: Message-ID: On Tue, 24 Jun 2008, Peter St. John wrote: > Programming a computer to play Go (an Asian strategy boardgame) has been > difficult; some people say it's proof that Go is better or harder than > chess, since computers can beat masters at chess but struggle at Go. (I > think that statistically a game of go is about equivalent to a two-game > match of chess; both games empty your brain quickly of course). My view is > that while go may be somewhat harder to reduce to tree-searching, the main > advantage of computer chess was an early start, e.g. von Neumann. I thought that Go was combinatorially vastly, vastly more difficult than chess. Both because of the extremely large number of move permutations that make it very difficult to follow trees and because Go is a fundamentally nonlocal game -- one can imagine a "perfect" Go strategy in which pieces are never played close to each other as the board gradually fills in with higher and higher still disconnected density, culminating in the winner completely unravelling the loser or precipitating out to win by a single small amount. Yet nobody can even come close to playing that way -- most games start out that way for a while and then transform into local dogfights that are STILL often resolved by long range connections. > This article: > http://www.usgo.org/resources/downloads/CogApdx%20II-2.pdf > describes recent trends in computer Go and mentions a 32-node cluster, 8 > cores per node. Apparently MPI parallelization is recent for them and they > are making good progress. Interesting. I'll have to look. Last time I read up on this (in the context of a conversation with you, IIRC:-) nobody could make a computer go player that could beat even a very bad human player, where computers back to maybe 386's have been able to play chess well enough to win at least sometimes against weak humans (and win "often" against weak human players by now). I suck at chess (but can still beat a lot of the people I -- rarely -- play) and modern computers can beat me pretty easily. I suck at Go, too -- but I can't even find a BAD go player to run on a PC to try to get better. rgb > > Peter > > The game Go: http://en.wikipedia.org/wiki/Go_%28game%29 > AGA (American Go Association): http://www.usgo.org > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From peter.st.john at gmail.com Tue Jun 24 18:35:03 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: Message-ID: That go is "vastly" more complex is a matter of degree; but there are bounds on e.g. the number of possible games, and if chess is googleplex times 2, go is up-arrow-googleplex. That's not my official mathematician estimage :-) but yeah go is horribly complex, and more horribly than chess, but still tractable to folks who think as we do. Go programs can beat even masters, now, on the 9x9 board, and I think recently they play well enough to beat a master on the full 19x19, *occasionally*, at *fast time controls*, and with handicaps. When I played Many Faces of Go several years ago I could beat it at 13 stone handicap, but it could beat me at 17 (Edward Lasker said that 3 stones was like knight odds in chess). But now, if I play a "bot" on KGS, I have difficulty giving it a 6 stone handicap. That's a huge improvement (to me). There is still a lone way to go. On a finite board, the game eventually becomes local; in fact "contact plays" are common after about the first dozen moves, but they are inescapable later. But the possibilities are horrible, the numbers are huge, for tree-searching. The new thing is monte carlo; to consider a move, the machine actually plays that position against itself a few hundred times (with a trivial, not recursive, algorithm) and weighs the move by the percentage of outcomes. It's weird, to me. Peter On Tue, Jun 24, 2008 at 9:22 PM, Robert G. Brown wrote: > On Tue, 24 Jun 2008, Peter St. John wrote: > > Programming a computer to play Go (an Asian strategy boardgame) has been >> difficult; some people say it's proof that Go is better or harder than >> chess, since computers can beat masters at chess but struggle at Go. (I >> think that statistically a game of go is about equivalent to a two-game >> match of chess; both games empty your brain quickly of course). My view is >> that while go may be somewhat harder to reduce to tree-searching, the main >> advantage of computer chess was an early start, e.g. von Neumann. >> > > I thought that Go was combinatorially vastly, vastly more difficult than > chess. Both because of the extremely large number of move permutations > that make it very difficult to follow trees and because Go is a > fundamentally nonlocal game -- one can imagine a "perfect" Go strategy > in which pieces are never played close to each other as the board > gradually fills in with higher and higher still disconnected density, > culminating in the winner completely unravelling the loser or > precipitating out to win by a single small amount. Yet nobody can even > come close to playing that way -- most games start out that way for a > while and then transform into local dogfights that are STILL often > resolved by long range connections. > > This article: >> http://www.usgo.org/resources/downloads/CogApdx%20II-2.pdf >> describes recent trends in computer Go and mentions a 32-node cluster, 8 >> cores per node. Apparently MPI parallelization is recent for them and they >> are making good progress. >> > > Interesting. I'll have to look. Last time I read up on this (in the > context of a conversation with you, IIRC:-) nobody could make a computer > go player that could beat even a very bad human player, where computers > back to maybe 386's have been able to play chess well enough to win at > least sometimes against weak humans (and win "often" against weak human > players by now). I suck at chess (but can still beat a lot of the > people I -- rarely -- play) and modern computers can beat me pretty > easily. I suck at Go, too -- but I can't even find a BAD go player to > run on a PC to try to get better. > > rgb > > > >> Peter >> >> The game Go: http://en.wikipedia.org/wiki/Go_%28game%29 >> AGA (American Go Association): http://www.usgo.org >> >> > -- > Robert G. Brown Phone(cell): 1-919-280-8443 > Duke University Physics Dept, Box 90305 > Durham, N.C. 27708-0305 > Web: http://www.phy.duke.edu/~rgb > Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php > Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080624/f89752a9/attachment.html From csamuel at vpac.org Tue Jun 24 19:02:13 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] 2.6.25.9 out - fixes VM regression In-Reply-To: <15997090.68381214359233025.JavaMail.root@zimbra.vpac.org> Message-ID: <709603870.68411214359333816.JavaMail.root@zimbra.vpac.org> http://lwn.net/Articles/287339/ # The 2.6.25.9 stable kernel update is available. # This one contains a relatively small set of patches; # one of them might be security-related, and another # addresses a serious virtual memory performance regression The problem is explained in a previous version of the patch: http://lwn.net/Articles/287342/ # In particular, the removal of ZERO_PAGE effectively # removed the core file writing optimization where we # would skip writing pages that had not been populated # at all, and increased memory pressure a lot by allocating # all those useless newly zeroed pages. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From matt at technoronin.com Tue Jun 24 19:22:41 2008 From: matt at technoronin.com (Matt Lawrence) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] FYI: HPC Server 2008 hits top 25 In-Reply-To: References: <20080624154540.GR9875@leitl.org> Message-ID: On Tue, 24 Jun 2008, Peter St. John wrote: > MS isn't famous for new development (as scientists see "new development") > but a few hundred engineers would not be such a big investment for them. > > I'd be more interested in a comparison of that many whatever, quad core > xeon? running linux vs MS's running Cluster Server 2008. CP/M could be a > pretty powerful OS if it ran on enough nodes :-) and of course I think of XP > as CP/M v.99 (approximately) (although that isn't fair to VMS, the forebear > of NT) I'm sorely tempted to try to run a cluster of OS/360 systems just to say it has been done. Should be pretty easy using Hercules/390. Since my boss reads this list, I'm not sure if he will be amused or appalled at the idea. probably both. -- Matt It's not what I know that counts. It's what I can remember in time to use. From rgb at phy.duke.edu Tue Jun 24 19:15:59 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> References: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> Message-ID: On Tue, 24 Jun 2008, Vincent Diepeveen wrote: > Go has a bigger branching factor than chess, as it starts with an empty board > of 19x19, versus chess a loaded board of 8x8. > > The first few moves in go decide the outcome of the game already, as the rest > is just a 'playout' of the first few moves. So what matters > most is the first few moves in the game. This is simply not correct. Go is a highly nonlocal game in both space on the game board and time. Chess is (in contrast) relatively linear -- the restricted moves available to each piece and the ability to deduce at least some relatively simple payoff functions make it much, much less complex. Sure, a "perfect chess game" is still quite complex, but its complexity fits in its entirety in a tiny fraction of the space occupied by Go. > Reason why chessprograms play so well is simply money and popularity of the > game. Again, no. The reason is that the decision trees in chess are countable by enough, big enough, fast enough, computers, and one can compute objective functions on even a modest number of looks ahead that will exceed what many people can do. Go's are not. I won't say they never will be, but they won't be anytime soon even with help from Moore's law. You might look at the comments here: http://en.wikipedia.org/wiki/Go_%28board_game%29#Computers_and_Go Note that it is far more complex BOTH because the combinatorics are vastly larger, where "vastly" is an understatement, and because crucial tactical situations can be resolved by seemingly unrelated plays made hundreds of turns earlier far away on the board. Go also is very difficult to tactically estimate -- one can play out seemingly balanced tactical situations for forty or fifty plys before a tiny weakness in one player's position -- or one of those distant pieces placed early in the game -- causes their entire effort to "unravel" and turn into a disaster. That's almost twice the number of plys in an entire chess game, and is still only the first third or so of the game. If you or your friend disagree with this, well, feel free to edit the wikipedia article(s) with examples that contradict it, but the mathematics and difficulty of pruning the Go trees suggest that it isn't. rgb > > Chess computers in the 80s and start 90s, used to export to 106 countries. I > remember talking about producing a dedicated chesscomputer, > and usually 100000 of them get printed. A minimum of 20000 pieces is needed > to heat up the production line (Hong Kong, China). > > There is no go computers AFAIK, for simple reason that the only nation where > you can sell your product is Japan. The 3 main nations where > go gets played is China, Korea, Japan. So only Japan you could sell some if > you have entrance to its very close market. > > In fact there is even a company that claims to have the rights on all human > go games. > > At my chat is someone, Gian-Carlo Pascutto, whose program Leela you can buy. > It is as we speak the strongest commercial go program on planet earth that > you can buy. > His engine focuses upon search, its knowlede is rather simplistic. > > He has a normal job just like you have one. > > So this is a sparetime written engine. > > Computerchess engines used to be fulltime work. When someone is jobless like > me, you again work for a few months fulltime at it. > There is 500 chess engines to compete with or so. > > In go the competition is very limited, only recently more engines are there. > Most programmed by non-asian programmers. > Not even from Asian decent. > > It's all about how much money you want to put in research. Would go have been > the game been played in 106 nations and chess in just 3 from which only 1 has > money and is a closed market, then we would be speaking now about a > computer-go world champion program and wondering what makes computer chess so > hard. > > In that case I would write then that if more money had been put at chess, > that those engines would be stronger than the go engines. > > Don't count at it that the big supercomputers make any chance in go, neither > in chess. The quality of the program is most important. > As soon as you massively parallellize a strong engine, now *that* makes > sense. > > Vincent > > On Jun 24, 2008, at 6:20 PM, Peter St. John wrote: > >> Programming a computer to play Go (an Asian strategy boardgame) has been >> difficult; some people say it's proof that Go is better or harder than >> chess, since computers can beat masters at chess but struggle at Go. (I >> think that statistically a game of go is about equivalent to a two-game >> match of chess; both games empty your brain quickly of course). My view is >> that while go may be somewhat harder to reduce to tree-searching, the main >> advantage of computer chess was an early start, e.g. von Neumann. >> >> This article: >> http://www.usgo.org/resources/downloads/CogApdx%20II-2.pdf >> describes recent trends in computer Go and mentions a 32-node cluster, 8 >> cores per node. Apparently MPI parallelization is recent for them and they >> are making good progress. >> >> Peter >> >> The game Go: http://en.wikipedia.org/wiki/Go_%28game%29 >> AGA (American Go Association): http://www.usgo.org >> >> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From rgb at phy.duke.edu Tue Jun 24 20:17:04 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: Message-ID: On Tue, 24 Jun 2008, Peter St. John wrote: > On a finite board, the game eventually becomes local; in fact "contact > plays" are common after about the first dozen moves, but they are > inescapable later. But the possibilities are horrible, the numbers are huge, > for tree-searching. The new thing is monte carlo; to consider a move, the > machine actually plays that position against itself a few hundred times > (with a trivial, not recursive, algorithm) and weighs the move by the > percentage of outcomes. It's weird, to me. It makes sense to me. It's in some sense more like humans play. In fact, if one can come up with any (even very simple) "local" rules to semi-prune the tree, you can imagine an importance-sampling monte carlo algorithm. However, if I were going to try, it would be genetic algorithm all the way. Don't explore trees exhaustively; too many of them. Don't sample them (especially if you can't create a canonical weight to do importance sampling); too many of them. Breed them. There are still too many, but again, if one can generate ANY sort of fitness function, maybe one can get the N^3 and exponential growth of favorable possibilities that are like "making a decision". Fortunately, I'm not going to try. Neural nets are much more fun. And MIGHT be able to manage the complexity -- especially one built with a GA...;-) rgb > > Peter > > On Tue, Jun 24, 2008 at 9:22 PM, Robert G. Brown wrote: > >> On Tue, 24 Jun 2008, Peter St. John wrote: >> >> Programming a computer to play Go (an Asian strategy boardgame) has been >>> difficult; some people say it's proof that Go is better or harder than >>> chess, since computers can beat masters at chess but struggle at Go. (I >>> think that statistically a game of go is about equivalent to a two-game >>> match of chess; both games empty your brain quickly of course). My view is >>> that while go may be somewhat harder to reduce to tree-searching, the main >>> advantage of computer chess was an early start, e.g. von Neumann. >>> >> >> I thought that Go was combinatorially vastly, vastly more difficult than >> chess. Both because of the extremely large number of move permutations >> that make it very difficult to follow trees and because Go is a >> fundamentally nonlocal game -- one can imagine a "perfect" Go strategy >> in which pieces are never played close to each other as the board >> gradually fills in with higher and higher still disconnected density, >> culminating in the winner completely unravelling the loser or >> precipitating out to win by a single small amount. Yet nobody can even >> come close to playing that way -- most games start out that way for a >> while and then transform into local dogfights that are STILL often >> resolved by long range connections. >> >> This article: >>> http://www.usgo.org/resources/downloads/CogApdx%20II-2.pdf >>> describes recent trends in computer Go and mentions a 32-node cluster, 8 >>> cores per node. Apparently MPI parallelization is recent for them and they >>> are making good progress. >>> >> >> Interesting. I'll have to look. Last time I read up on this (in the >> context of a conversation with you, IIRC:-) nobody could make a computer >> go player that could beat even a very bad human player, where computers >> back to maybe 386's have been able to play chess well enough to win at >> least sometimes against weak humans (and win "often" against weak human >> players by now). I suck at chess (but can still beat a lot of the >> people I -- rarely -- play) and modern computers can beat me pretty >> easily. I suck at Go, too -- but I can't even find a BAD go player to >> run on a PC to try to get better. >> >> rgb >> >> >> >>> Peter >>> >>> The game Go: http://en.wikipedia.org/wiki/Go_%28game%29 >>> AGA (American Go Association): http://www.usgo.org >>> >>> >> -- >> Robert G. Brown Phone(cell): 1-919-280-8443 >> Duke University Physics Dept, Box 90305 >> Durham, N.C. 27708-0305 >> Web: http://www.phy.duke.edu/~rgb >> Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php >> Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 >> > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From diep at xs4all.nl Tue Jun 24 21:10:21 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> Message-ID: On Jun 25, 2008, at 4:15 AM, Robert G. Brown wrote: > On Tue, 24 Jun 2008, Vincent Diepeveen wrote: > >> Go has a bigger branching factor than chess, as it starts with an >> empty board of 19x19, versus chess a loaded board of 8x8. >> >> The first few moves in go decide the outcome of the game already, >> as the rest is just a 'playout' of the first few moves. So what >> matters >> most is the first few moves in the game. > > This is simply not correct. Go is a highly nonlocal game in both > space > on the game board and time. Chess is (in contrast) relatively > linear -- > the restricted moves available to each piece and the ability to deduce > at least some relatively simple payoff functions make it much, much > less > complex. Sure, a "perfect chess game" is still quite complex, but its > complexity fits in its entirety in a tiny fraction of the space > occupied > by Go. > In fact it is the opposite: Go has more local effects than Chess. This is empirically provable by the fact that you can safely prune in a hard manner in search moves in Go - the same is not possible in chess at all. In fact some of the top programs just below the level of Leela even today still prune in computer go already at the root. The same in computer chess would result in a player that is *real* weak. In fact weaker than the go programs in question doing it. Historically Go software (think of the old handtalk) consisted out of a global search and a fast assembler local search. In computerchess this idea was abandonned around end of 70s and exchanged for a global search. If you think about it, it makes sense. Just do a full factorization - GCD of {Chess,GO}, then it factors out that the nonshared factor is that in chess all pieces can move, whereas in Go once you make a move somewhere, it will keep there large areas. So for the only thing that really matters in computer go - the dead and alive evaluation - you can nearly local figure this out. That's exactly what software used to do. >> Reason why chessprograms play so well is simply money and >> popularity of the game. > > Again, no. The reason is that the decision trees in chess are > countable > by enough, big enough, fast enough, computers, and one can compute > objective functions on even a modest number of looks ahead that will > exceed what many people can do. Go's are not. I won't say they never > will be, but they won't be anytime soon even with help from Moore's > law. > You might look at the comments here: > A few years ago a few chessprogrammers switched to computer-go. See what has been achieved last few years. A huge increase in playstrength. When i spoke to some computer-go authors a few years ago, they all made real fundamental mistakes in search. > http://en.wikipedia.org/wiki/Go_%28board_game%29#Computers_and_Go > > Note that it is far more complex BOTH because the combinatorics are > vastly larger, where "vastly" is an understatement, and because > crucial What matters for progress is how many brilliant guys are interested in building a competing software engine, and put fulltime work in it, *that* progresses a field. In computer-go this incentive is not there simply. The huge easy money that was there to win in the 80s and 90s in computer-chess, has caused a big number of real brilliant minds to be busy with the problem. That amount gets less and less now, which is sad in itself. If we consider that all ideas in computer go now get completely dominated by remnants left over from computer chess, in fact ex-computer chess authors, not a single algorithm been invented in China, Japan nor Korea that kicks butt for computer-go, then obviously the point is easy to prove. What i foresee in computer-go is that the field will keep get dominated for the coming years by authors from Western Europe and USA. If you realize how many people live in those countries, Japan, China and Korea, that is in itself an interesting observation. End this year there is a computer-olympiads, so both world championship computer chess and go in China, Beijing. It is very easy in such events, which yearly happen, to speak to the different authors. It is very difficult to motivate someone to start writing a go engine if he doesn't even know how to make a ton of money with it. Out of respect for authors i won't quote you the number of copies of computer-go programs sold, but if you compare that with just 2 chess interfaces that sold each millions and millions of copies (just chessmaster already 4 million copies), and that this was already long after the decline of income from dedicated chesscomputers (as you can't copy those easily!) which were exported to 106 nations and manufacturers made hundreds of millions a year, just on a game that in itself has very few fanatic practioners, and a potential of "only" 200 million people when the Chinese get not counted (as chess is so similar in itself to chinese chess, and the huge step they make there now to play chess means that another half billion people will know the chess rules by now); compare that with computer-go which never sold really well. If there is 1 product on a console that sold a 100k copies one day that is already *really* a lot, and this only and only in Japan, whereas the current top programs *hardly* sell a copy to Japan. Economy gets driven by sales; if economy demands a better product, it will be there. I'd argue that the top go programs would be professional dan level by now when authors would be able to fulltime work on those programs and have a lot of competition. AFAIK not a single computer-go author works fulltime on his engine as of now (not to confuse with GUI that sells). > tactical situations can be resolved by seemingly unrelated plays made > hundreds of turns earlier far away on the board. Go also is very > difficult to tactically estimate -- one can play out seemingly > balanced > If you realize how much effort has been put in chess to make kick butt evaluation functions, and slowly by now people learn how to do it. None of those techniques has been applied yet to computer go. All that happens very slowly now. Of course within 10 years usually things leak from computer chess to computer go. That's how things happen. > tactical situations for forty or fifty plys before a tiny weakness in There are clear estimates on the tactical depth of chess and go. In computer-chess the average depth for a trick is about 12 ply for 'advanced' level. Say 'national master level' in USA. Tricks at my level usually are within that depth also, and the more advanced tricks are usually less than 20 ply. About a ply or 8 you get 'for free', as simple extensions can extend you that line. So todays chess software with their ultimate thin searches, see those tricks at around 16 ply. In computer-go, the deepest tactics is about 30 ply. About 20 ply out of that is totally forced moves. Right now the thin searches in computer-go are a 1 to 1 match nearly with computerchess. That's weird in itself. How to extend they slowly figure out, because the authors in question are real weak in computer-go. Compare with chess where even PHD's have been achieved by persons just on "how to extend?". None of all that in computer-go. I'd argue that if you get selectively about 20 ply deep in computer- go, that a good evaluation function and good computer-go driven extensions will get the engine in itself to the same level like in computerchess; as soon as you remove the openingsbook of todays top chess programs, they still look like beginners in the eye of titled chessplayers. Go has a huge historic disadvantage there; go games hardly ever get written down on paper. Chess has a 100 years advantage in the opening there. Or some trainers will argue 50 years advantage, as honor, glory and not being creative, and dying just for not admitting you were wrong, was before world war 2 real normal in chess; short after world war 2 that all changed. Go is such a local Asian game, as a result it is totally impossible if you do not speak Mandarin nor one of the Japanese languages, to get access to go knowledge ready to implement in your go program. In chess this has all been much easier for authors. This problem will keep there. Working indirectly via translation using a go player is tough too; not so long ago i had asked a Polish girl to translate a small text where a Polish chess author wrote down some algorithmic ideas he had. I lost big time there again as it made zero sense the pseudo-C code versus the translated text. The Polish programmer posted a day later the pseudo-C code into an English forum and it appeared the translation was total wrong. Not even a little, just completely. That confusion is there still in the go world. The culture of China and Japan and Korea causes Go to be at the state of where Chess was before world war 2. Like here i have a book from a 9-dan go. The idea being: "attack your enemy where he is strongest". Just to be sure i contacted a few strong go players whether this is true. They confirmed my feeling that this lemma is total wrong. Punished bigtime also in world war 1. In English there is this saying: "over the top" derived from that type of warfare. (getting over the top out of a trench straight into the machine guns). Basically that is not helping computer go much either. When busy with a go program end of 90s, I concluded that only with a good go player next to me, i would make a chance of writing a good go program, as the start of the game where in Go having a book doesn't help much, is requiring some positional knowledge. In computerchess we cover up for idiocy of programs there by a book, something that is hard in go. All that makes it of course extra surprising that there is no strong Chinese/Korean/Japanese go programs at this moment that make any chance against programs from here. If i draw a circle from here with a 300 miles radius, i have a big number of world top go engines, under which the number 1 and number 2. Perhaps even the top 3. > one player's position -- or one of those distant pieces placed > early in > the game -- causes their entire effort to "unravel" and turn into a > disaster. That's almost twice the number of plys in an entire chess > game, and is still only the first third or so of the game. > See above for correct calculation. > If you or your friend disagree with this, well, feel free to edit the > wikipedia article(s) with examples that contradict it, but the > mathematics and difficulty of pruning the Go trees suggest that it > isn't. > See above for disproof of that. Vincent > rgb > >> >> Chess computers in the 80s and start 90s, used to export to 106 >> countries. I remember talking about producing a dedicated >> chesscomputer, >> and usually 100000 of them get printed. A minimum of 20000 pieces >> is needed to heat up the production line (Hong Kong, China). >> >> There is no go computers AFAIK, for simple reason that the only >> nation where you can sell your product is Japan. The 3 main >> nations where >> go gets played is China, Korea, Japan. So only Japan you could >> sell some if you have entrance to its very close market. >> >> In fact there is even a company that claims to have the rights on >> all human go games. >> >> At my chat is someone, Gian-Carlo Pascutto, whose program Leela >> you can buy. >> It is as we speak the strongest commercial go program on planet >> earth that you can buy. >> His engine focuses upon search, its knowlede is rather simplistic. >> >> He has a normal job just like you have one. >> >> So this is a sparetime written engine. >> >> Computerchess engines used to be fulltime work. When someone is >> jobless like me, you again work for a few months fulltime at it. >> There is 500 chess engines to compete with or so. >> >> In go the competition is very limited, only recently more engines >> are there. Most programmed by non-asian programmers. >> Not even from Asian decent. >> >> It's all about how much money you want to put in research. Would >> go have been the game been played in 106 nations and chess in just >> 3 from which only 1 has money and is a closed market, then we >> would be speaking now about a computer-go world champion program >> and wondering what makes computer chess so hard. >> >> In that case I would write then that if more money had been put at >> chess, that those engines would be stronger than the go engines. >> >> Don't count at it that the big supercomputers make any chance in >> go, neither in chess. The quality of the program is most important. >> As soon as you massively parallellize a strong engine, now *that* >> makes sense. >> >> Vincent >> >> On Jun 24, 2008, at 6:20 PM, Peter St. John wrote: >> >>> Programming a computer to play Go (an Asian strategy boardgame) >>> has been difficult; some people say it's proof that Go is better >>> or harder than chess, since computers can beat masters at chess >>> but struggle at Go. (I think that statistically a game of go is >>> about equivalent to a two-game match of chess; both games empty >>> your brain quickly of course). My view is that while go may be >>> somewhat harder to reduce to tree-searching, the main advantage >>> of computer chess was an early start, e.g. von Neumann. >>> This article: >>> http://www.usgo.org/resources/downloads/CogApdx%20II-2.pdf >>> describes recent trends in computer Go and mentions a 32-node >>> cluster, 8 cores per node. Apparently MPI parallelization is >>> recent for them and they are making good progress. >>> Peter >>> The game Go: http://en.wikipedia.org/wiki/Go_%28game%29 >>> AGA (American Go Association): http://www.usgo.org >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > > -- > Robert G. Brown Phone(cell): 1-919-280-8443 > Duke University Physics Dept, Box 90305 > Durham, N.C. 27708-0305 > Web: http://www.phy.duke.edu/~rgb > Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php > Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 > From Shainer at mellanox.com Tue Jun 24 21:38:25 2008 From: Shainer at mellanox.com (Gilad Shainer) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: Message-ID: <9FA59C95FFCBB34EA5E42C1A8573784F0129EEC4@mtiexch01.mti.com> Mark Hahn wrote: > > Static routing is the best approach if your pattern is > known. In other > > sure, but how often is the pattern actually known? I mean in general: > aren't most clusters used for multiple, shifting purposes? > There will be many arguments in each side. There are enough cases where the patterns are know, or that the majority of the applications use the same pattern, and there are cases that they are not. As I see it the key is flexibility - choose what works best for you, and if you want to tune it - that is possible as well > > There are some vendors that uses only the 24 port switches to build > > very large scale clusters - 3000 nodes and above, without any > > oversubscription, and they find it more cost effective. Using single > > so the switch fabric would be a 'leaf' layer with 12 up and > 12 down, and a top layer with 24 down, right? so 3000 nodes > means 250 leaves and 125 tops, 9000 total ports so 4500 cables. > The number of switch layers, or tiers, depends on the cluster size and the oversubscription. For full non blocking, 2 layer of 24 switches get you up to 288 ports switch, and with 36 ports up to 648 ports. With 3 layers you can build 3456 ports or more than 11K ports (24 or 36 ports building blocks). Assume that you use 2 layers, you need the same number of cables as number of nodes. > > enclosures is easier, but the cables are not expensive and > you can use > > the smaller components. > > in federated networks, I think cables wind up being 15-20% of > the network price. for instance, if we take the simplest > possible approach, and equip this 3000-node cluster with a > non-blocking federated fabric (assuming just sdr) from > colfax's current price list: > > subtot unit n what > 375000 125 3000 ib nic > 117000 39 3000 1m host ib cables > 148500 99 1500 8m leaf-top ib cables > 900000 2400 375 24pt unman switch > 1540500 total (cable 17%) > > I'm still confused about IB pricing, since the street price > of nics, cables and switches are dramatically more expensive > than colfax. > (to the paranoid, colfax would appear to be a mellanox shell > company...) This is totally wrong. Colfax is one of many distributers and is in business much before Mellanox. Mellanox do not need to have shell companies. If you check with othersm you might see same pricing. > for completeness, here's the same bom with "normal" public prices: > > subtot unit n what > 2100000 700 3000 ib nic > 330000 110 3000 1m host ib cables > 330000 220 1500 8m leaf-top ib cables > 1500000 4000 375 24pt unman switch > 4260000 total (cable 15%) > I can not comment on any pricing from my customers and partners. > interestingly, if nodes were about 3700 apiece (about what > you'd expect for intel dual-socket quad-core 2G/core), the > interconnect winds up being 28% of the cost. > To the smart user, if your application require high speed interconnect, than it worth the money (assuming your math is correct). Buying 300 servers and being able to utilize only half, not to mention the electric bill - suddenly the interconnect seems very cheap. Gilad. From rgb at phy.duke.edu Tue Jun 24 22:14:06 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> Message-ID: On Wed, 25 Jun 2008, Vincent Diepeveen wrote: >> one player's position -- or one of those distant pieces placed early in >> the game -- causes their entire effort to "unravel" and turn into a >> disaster. That's almost twice the number of plys in an entire chess >> game, and is still only the first third or so of the game. >> > > See above for correct calculation. > >> If you or your friend disagree with this, well, feel free to edit the >> wikipedia article(s) with examples that contradict it, but the >> mathematics and difficulty of pruning the Go trees suggest that it >> isn't. >> > > See above for disproof of that. Very educational and interesting. But I was also serious -- if you can (and it sounds like you can) you should consider editing the wikipedia articles. I'm not COMPLETELY convinced -- I'll be a lot more convinced when I can get a GPL Go engine that will play a decent game on my laptop...;-) rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From bill at cse.ucdavis.edu Tue Jun 24 23:05:10 2008 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <520EC3EF-536A-4BD8-9806-492C23D2A058@xs4all.nl> References: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> <48614A7F.2080606@ias.edu> <520EC3EF-536A-4BD8-9806-492C23D2A058@xs4all.nl> Message-ID: <4861E016.9040803@cse.ucdavis.edu> Vincent Diepeveen wrote: > intel c++ obviously is close to visual studio. Within 0.5% to 1.5% range > (depending upon flags I believe Microsoft licensed the intel optimization technology, so the similarity is hardly surprising. > and hidden flags that you managed to get from someone). Intel C++ is > free for researchers such as > me. Last I checked it was fine for research in compilers, but as a tool to help facilitate research you have to pay, even in an academic environment. Maybe I misread the license, what exactly let you to believe that it's "free for researchers"? > The PG compiler and especially pathscale compiler are doing rather well > at benchmarks, In my experience, real applications as well, things like nwchem for instance. Granted the difference (at least in the past) was much larger for g77 vs commercial fortran than it was for gcc/g++ vs commercial c compilers. I've heard gfortran has gotten much better and I've even occasionally seen SPEC numbers for GNU compilers which looked encouraging. In any case I've often seen commercial compilers justified on a pure cost/benefit basis. I.e. $1k on compilers gets me more performance than another $1k on hardware. > especially that last, yet at our codes they're real ugly. Maybe they do > better for floating point > oriented workloads, which doesn't describe game tree search. I've written codes that do mostly pointer chasing, compilers didn't make any difference whatsoever. gcc -O1 matched any optimization flag of any compiler I tried (pathcc, pgcc, icc). > What strikes me is that visual studio/intel c++ produce a far tinier > executable than either of those compilers > and that the fastest executable (in case of Diep) is a 32 bits one. 32 > bits visual c++ executable with pgo > is roughly 1.5% faster than its 64 bits counterpart. Ah, I don't find 1.5% very exciting, nor the size of the binary. Certainly if you have many pointers and are close to falling into the next level of the memory hierarchy smaller pointers can by a significant help. Did you compare stripped binaries with similar flags? Where was the difference? /usr/bin/size (on most linux distros) can be handy for looking at different section sizes of a binary. > The main datastructures are using integers and a lot of pointers get > used. Seems in 64 bits this hurts a tad. The main differences I know of are: * 64 bit pointers can cause more cache/memory pressure for pointer intensive codes. * I believe on most (all?) core2 chips some optimizations are disabled in 64 bit mode (nehalem is supposed to fix this). I don't believe AMD disables any optimizations in 32 bit mode. Thus the performance difference for AMD vs Intel can depend on which mode you use, a slight advantage for intel at 32 bit, and a slight advantage for amd at 64 bit. This varies hugely depending on the code of course. * Some codes benefit from the additional (8-> 16) registers only available in 64 bit mode, which takes some pressure off of the L1, and creates opportunities to keep more functional units busy for 2 reasons. Instruction issue is less of an issue (you can have more register references than pending loads) and there are more read/write ports to the register file than L1. From eagles051387 at gmail.com Tue Jun 24 23:43:34 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> Message-ID: how much does a sugar glass window cost now that sugar and other things are being used for bio fuels? On Wed, Jun 25, 2008 at 12:20 AM, Mark Hahn wrote: > > More specifically for HPC, linux seems designed for the desktop, and >>> for small memory machines. >>> >> > the only justice I can see in that is that there hasn't been all that much > effort to get bigpages widely/easily used. in particular, I don't > see that scheduler or general memory-management issues in linux are > particularly biased for desktop or against HPC. > > That's funny, because I've heard people get scared that it was the complete >> opposite. That Linux was driven by Big Iron, and that no one cared about >> the "little desktop guy" (Con Kolivas is an interesting history example). >> > > Con didn't play the game right - you have to have the right combination of > social engineering (especially timing and proactive response) and good tech > kungfoo. kernel people are biased towards a certain aesthetic that doesn't > punish big-picture redesigns from scratch, but _does_ punish solutions in > search of a problem. > > so the question is, if you had a magic wand, what would you change in the > kernel (or perhaps libc or other support libs, etc)? most of the things I > can think of are not clear-cut. I'd like to be able to give better info > from perf counters to our users (but I don't think Linux is really in the > way). I suspect we lose some performance due to jitter > injected by the OS (and/or our own monitoring) and would like to improve, > but again, it's hard to blame Linux. I'd love to have better options for > cluster-aware filesystems. kernel-assisted network shared memory? > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080625/bec8e8cc/attachment.html From csamuel at vpac.org Wed Jun 25 04:39:42 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <1843262915.73441214393713442.JavaMail.root@zimbra.vpac.org> Message-ID: <175463834.73461214393982940.JavaMail.root@zimbra.vpac.org> ----- "Bill Broadley" wrote: > Maybe I misread the license, what exactly let you > to believe that it's "free for researchers"? The Intel website is pretty clear on the matter and I'm pretty sure this is the case since they began their free non-commercial license program. # Q. I am engaged in research projects. Can I qualify to # use the noncommercial product? # # A. If you, as an individual, are receiving any form of # compensation for the research project (i.e., you receive # a salary, or funding, etc.) you do not qualify for a # non-commercial license. http://www.intel.com/cd/software/products/asmo-na/eng/download/download/219771.htm # This offering is provided to developers who are # developing software on their own time without # compensation. [...] # Note that academic use of the products does not qualify # for a non-commercial license. Intel offers heavily # discounted licenses to academic developers through # our Academic Developer Program. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From diep at xs4all.nl Wed Jun 25 06:02:52 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> Message-ID: <7F09EDD7-0A8A-4B05-BD72-F3A984E39CFC@xs4all.nl> Yes i realize very well what you mean Robert, though i partly agree the problem is 2 fold: a) i don't get paid to do it, if one were part of an university that is different b) having a chessprogram i'm used to hear wrong information all the time, games are so so popular, everyone likes to do statements about game tree search, ANN's, GA's, Monte Carlo, automatic parameter tuning and whatever falls under my Artificial Intelligence expertises; if i would try to correct it all that is more than a fulltime job. Just correcting the nonsense that Thesis spit as being the truth, is already total impossible. To give a popular example of misconecption in Thesis, not a single world top engine can use the MTD algorithm from Aske Plaat, as it is worse than other forms of search. In fact it is that bad, that the difference in playstrength is real huge. Most use something like PVS or some windowed form of alfabeta as the starting algorithm. Difference between windowed forms of alfabeta versus PVS is hardly measurable for some programs, but the difference with MTD is so huge, that a program plays significantly worse using it; it is pretty easy also to know why. I've listen on the net several times a few arguments why MTD performs so bad, yet Aske moved from Netherlands to the institute "MiT", so they keep quoting him. I could quote tens of things like that. Fact is i'm not paid to publish articles. A minimal TV crew was here in my backyarden (1 cameraman also doing sound and a reporter), their TV channel a tiny one (RTV), but they had plans also to sell it to National Geographic. As usual a very stubborn setup they wanted to show thing X and were not interested in all kind of interesting things Y, which IMHO is more interesting to show the audience; i'd argue for a reporter doing this type of documentary it is far better to start interviewing and while interviewing you hear from the experts such interesting stuff that you can make a far better documentary with more relevant questions; so letting experts talk rather than the reporter, which defines the typical reporters problem. It is this reporters problem that causes a lot of nonsense to get spit into the ether. They started spitting information they had been given by other persons (a professor). Now Jaap van den Herik, didn't do too bad there actually. Relevant for this mailing list is this piece of information they had for processing power. The quote i got from them being more or less: "chess has more than 10^40 positions, that is more than there is particles in planet earth, so processors can never get that amount of calculation power, therefore it is impossible to create the unbeatable player in chess". Realize this is from someone who decides also on what the next supercomputer of Dutch Government is gonna be for scientists. The above statement is wrong for several reasons. Regarding the number of particles that are in planet earth, i'll skip that discussion. I'm no expert on that, so i cannot do a statement there. Chess has indeed more than 10^40 positions. Latest mathematical calculation as published in ICGA journal came to 10^43. There is 2 points: a) playing the best move doesn't require searching the entire search space, so who knows maybe searching 10^18 nodes with a huge hashtable and good evaluation function, we might already play the best move. b) To get to 10^43 calculation power, we do not need a processor that exists out of 10^43 particles. All we need is a processor that is hosting 10^43 internal movements, which is a lot easier to get acomplished, than building that cpu of 10^43 in size. In technical areas that people like to hear about, it is impossible as a single person to fight unpaid all the desinformation. The biggest problem there is professors who seek publicity, or cover up for total nonsense thesis of their students. I tried to have some sort of discussion with Jonathan Schaeffer about a student of him quoting MTD is ok in his thesis. What i skipped in fact is the total nonsense about parallel speedup in game tree searching from his student idea being: make the most inefficient searcher on planet earth, then parallellize it and of course because even random move orderings would improve your program then, also the parallellism gets a decent speedup as a result of all that inefficiency; the classical approach. Also MiT is notorious bad in game tree search. I will give yet 1 more example. I remember they claimed in articles a great speedup (50% or so even) in game tree search at supercomputers. Now my chessprogram has a lot of chessknowledge and also is doing all sorts of mobility, which requires expensive scans. That slows down the nodes per second that my chessprogram Diep can get. The search frame from MiT slows down the searchspeed of their chessengine by a staggering factor 40. At the same SGI Origin3800 hardware in fact my slow Diep engine gets MORE nodes per second, and a lot more, than MiT's cilkchess chess engine. Not because single core mine is any faster. In fact single core cilkchess without cilkframe used to be factor 20. Additionally there is nowhere anything statistical significant done there. All speedups of supercomputers are never carrying a table with what is its speedup with 95% sureness. Doing publications just to show how it should be done and disprove and prove a number of algorithms (especially disprove nonsense), is already quite a tough challenge and i can be busy for years there, as there is hundreds of such publications, just from the past few years to disprove and show the better alternative. It's a technical science, there is not many who really are expert there, progress goes *real* fast. In the end what matters for engines is not how strong it plays against you, it matters how strong you play against competitors :) On Jun 25, 2008, at 7:14 AM, Robert G. Brown wrote: > On Wed, 25 Jun 2008, Vincent Diepeveen wrote: > >>> one player's position -- or one of those distant pieces placed >>> early in >>> the game -- causes their entire effort to "unravel" and turn into a >>> disaster. That's almost twice the number of plys in an entire chess >>> game, and is still only the first third or so of the game. >> >> See above for correct calculation. >> >>> If you or your friend disagree with this, well, feel free to edit >>> the >>> wikipedia article(s) with examples that contradict it, but the >>> mathematics and difficulty of pruning the Go trees suggest that it >>> isn't. >> >> See above for disproof of that. > > Very educational and interesting. But I was also serious -- if you > can > (and it sounds like you can) you should consider editing the wikipedia > articles. > > I'm not COMPLETELY convinced -- I'll be a lot more convinced when I > can > get a GPL Go engine that will play a decent game on my laptop...;-) > > rgb > > -- > Robert G. Brown Phone(cell): 1-919-280-8443 > Duke University Physics Dept, Box 90305 > Durham, N.C. 27708-0305 > Web: http://www.phy.duke.edu/~rgb > Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php > Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 > From dnlombar at ichips.intel.com Wed Jun 25 06:33:26 2008 From: dnlombar at ichips.intel.com (Lombard, David N) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Re: "hobbyists"es In-Reply-To: References: <87abhfzpcw.fsf@snark.cb.piermont.com> Message-ID: <20080625133326.GA29168@nlxdcldnl2.cl.intel.com> On Sat, Jun 21, 2008 at 02:09:37PM -0700, Robert G. Brown wrote: > > > It would appear the rgb-bot ran amuk! -- David N. Lombard, Intel, Irvine, CA I do not speak for Intel Corporation; all comments are strictly my own. From hahn at mcmaster.ca Wed Jun 25 06:47:39 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: <9FA59C95FFCBB34EA5E42C1A8573784F0129EEC4@mtiexch01.mti.com> References: <9FA59C95FFCBB34EA5E42C1A8573784F0129EEC4@mtiexch01.mti.com> Message-ID: >> so the switch fabric would be a 'leaf' layer with 12 up and >> 12 down, and a top layer with 24 down, right? so 3000 nodes >> means 250 leaves and 125 tops, 9000 total ports so 4500 cables. > > The number of switch layers, or tiers, depends on the cluster size and > the oversubscription. For full non blocking, 2 layer of 24 switches get > you up to 288 ports switch, and with 36 ports up to 648 ports. I guess I'm confused. wouldn't a non-blocking fabric with 288 host ports be one layer of 24x 24pt switches (12 up, 12 down to hosts), with _half_ as many top-level switches (ie, 12 switches with 24 down each)? From Shainer at mellanox.com Wed Jun 25 06:51:31 2008 From: Shainer at mellanox.com (Gilad Shainer) Date: Wed Nov 25 01:07:16 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: Message-ID: <9FA59C95FFCBB34EA5E42C1A8573784F0129EF14@mtiexch01.mti.com> >>> so the switch fabric would be a 'leaf' layer with 12 up and >>> 12 down, and a top layer with 24 down, right? so 3000 nodes means >>> 250 leaves and 125 tops, 9000 total ports so 4500 cables. >> >> The number of switch layers, or tiers, depends on the cluster size and >> the oversubscription. For full non blocking, 2 layer of 24 switches >> get you up to 288 ports switch, and with 36 ports up to 648 ports. > > I guess I'm confused. wouldn't a non-blocking fabric with 288 host > ports be one layer of 24x 24pt switches (12 up, 12 down to hosts), > with _half_ as many top-level switches (ie, 12 switches with 24 down each)? Yes, 2 tiers, 1st with 24X24p and the 2nd with 12x24p From jan.heichler at gmx.net Wed Jun 25 06:58:14 2008 From: jan.heichler at gmx.net (Jan Heichler) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Infiniband modular switches In-Reply-To: References: <9FA59C95FFCBB34EA5E42C1A8573784F0129EEC4@mtiexch01.mti.com> Message-ID: <1492419345.20080625155814@gmx.net> Hallo Mark, Mittwoch, 25. Juni 2008, meintest Du: >>> so the switch fabric would be a 'leaf' layer with 12 up and >>> 12 down, and a top layer with 24 down, right? so 3000 nodes >>> means 250 leaves and 125 tops, 9000 total ports so 4500 cables. >> The number of switch layers, or tiers, depends on the cluster size and >> the oversubscription. For full non blocking, 2 layer of 24 switches get >> you up to 288 ports switch, and with 36 ports up to 648 ports. MH> I guess I'm confused. wouldn't a non-blocking fabric with 288 host ports MH> be one layer of 24x 24pt switches (12 up, 12 down to hosts), with _half_ MH> as many top-level switches (ie, 12 switches with 24 down each)? That is how it is build normally. Regards, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080625/6a8bc146/attachment.html From diep at xs4all.nl Wed Jun 25 07:06:58 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> Message-ID: <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> Even worse, Why is there subsidy on bio fuels that get produced out of food eaten by poor people? This causes as we speak people dying as they can no longer for a cent or so buy food made out of it; the prices have doubled if not more for such types of cheap food because of subsidy in the 1st world countries for this. I assume EU will take measures to turn back those subsidies on bio fuels that get produced out of food that feeds billions, who now hardly can afford to buy food anymore as it gets burned for energy in first world countries, whereas commercially spoken it cannot get burned, it is just because of subsidy it can exist. Even better i would be in favour of a ban on bio fuels that are outright food products in 3d world countries. When i just walked previous week into a shop and my sister was interested in a new washing machine, i pointed her to the fact that the thing she was interested in, was eating 3.8 kW, versus the 100 euro more expensive thing next to it was eating 1.14 kW. It is something that only very few will notice. It is easy to cheaply produce equipment that eats more power than equipment of competitors, that is the fundamental problem. Vincent - speaking for himself On Jun 25, 2008, at 8:43 AM, Jon Aquilina wrote: > how much does a sugar glass window cost now that sugar and other > things are being used for bio fuels? > > On Wed, Jun 25, 2008 at 12:20 AM, Mark Hahn wrote: > > More specifically for HPC, linux seems designed for the desktop, and > for small memory machines. > > the only justice I can see in that is that there hasn't been all > that much effort to get bigpages widely/easily used. in > particular, I don't > see that scheduler or general memory-management issues in linux are > particularly biased for desktop or against HPC. > > > That's funny, because I've heard people get scared that it was the > complete > opposite. That Linux was driven by Big Iron, and that no one cared > about > the "little desktop guy" (Con Kolivas is an interesting history > example). > > Con didn't play the game right - you have to have the right > combination of social engineering (especially timing and proactive > response) and good tech > kungfoo. kernel people are biased towards a certain aesthetic that > doesn't > punish big-picture redesigns from scratch, but _does_ punish > solutions in search of a problem. > > so the question is, if you had a magic wand, what would you change > in the kernel (or perhaps libc or other support libs, etc)? most > of the things I can think of are not clear-cut. I'd like to be > able to give better info from perf counters to our users (but I > don't think Linux is really in the way). I suspect we lose some > performance due to jitter > injected by the OS (and/or our own monitoring) and would like to > improve, > but again, it's hard to blame Linux. I'd love to have better > options for cluster-aware filesystems. kernel-assisted network > shared memory? > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > > > -- > Jonathan Aquilina > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From rgb at phy.duke.edu Wed Jun 25 07:10:31 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: <7F09EDD7-0A8A-4B05-BD72-F3A984E39CFC@xs4all.nl> References: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> <7F09EDD7-0A8A-4B05-BD72-F3A984E39CFC@xs4all.nl> Message-ID: On Wed, 25 Jun 2008, Vincent Diepeveen wrote: > Chess has indeed more than 10^40 positions. Latest mathematical calculation > as published in ICGA journal came to 10^43. > > There is 2 points: > a) playing the best move doesn't require searching the entire search > space, > so who knows maybe searching 10^18 nodes with a huge hashtable and > good evaluation function, > we might already play the best move. :-) That's true and funny at the same time. "Only" 10^18 positions, at "only" a nanosecond each takes "only" a gigasecond, which since a year is \pi x 10^7 seconds IIRC, is "only" around \pi x 10 Gigasearch-years. It is a difficult problem, in other words. It's what makes it fun, I suppose. And equally obviously, human brains don't work anything like this way on it. To me it is more fun to speculate on how the human brain does it WITHOUT anything like the capability to process giga-giga-trees. It clearly prunes things down to some much more finite order extremely quickly by discarding nearly all of this space from the beginning, and YET the space that remains is often the good one to the best that even Big Blue can parse out otherwise. > b) To get to 10^43 calculation power, we do not need a processor that > exists out of 10^43 particles. All we need is > a processor that is hosting 10^43 internal movements, which is a lot easier > to get acomplished, than building that > cpu of 10^43 in size. I'm not sure what you mean by that. All silicon processing has the usual serial and parallel components and accomplishes work (per processor) with a gigahertz scale serial barrier, leaving it many, many orders of magnitude short even in parallel. Yes, neural network processing in the brain has way fewer (maybe 10^11) neurons, but each neuron has thousands of synaptic connections to other neurons and the network formed has staggering complexity. Big enough that the chess problem could actually fit in its entirety inside with room enough for the go problem left over. And still be nearly "empty", and work in an extremely parallel fashion. There aren't a whole lot of candidate architectures for a parallel processing system that can do this sort of thing, but I agree that this is "all" we need. It's just not terribly easy to accomplish, because the problem is hard (and fun). One encounters numbers that scale like this in physics (stat mech) quite often, actually. If you work out the states in e.g. Monte Carlo sampling of a very simple 100x100x100 simply cubic Ising ferromagnet lattice (where each site contains a spin that can hold one of two states) the number of states in the system is 2^(1,000,000). This is "a big number" at least at first glance. If one then traces through "moves" through the system from any given starting state through all the various intermediary states to all the possible final states, one creates a space with e.g. [2^1000000]^N configurational trajectories for N steps. Ooo, maybe THAT is a big number, at least if N is reasonably large (say, a few thousand). Out of all of those states, a MUCH MUCH SMALLER set of states represents the only ones that are "likely" to be seen if the system is in equilibrium at any given temperature. How to find this needle in a very "large" (and yet tiny -- we're talking a mere million atoms when physical systems big enough to see contain, um, a lot more than this and have continuous spins instead of discrete ones) haystack? Importance sampling Monte Carlo lets you discard nearly all moves in a Markov process that takes you from any starting configuration to the "relevant" part of phase space rather quickly (considering) and then samples that space effectively. It rejects far, far, far more states than there are particles in the Universe with a metaphorical "sniff" as being absurdly unlikely and homes right in on the likely suspects. I think that this is a much better model, or metaphor, for how humans think than search trees. Humans really suck at search trees, with their fundamentally serial attention focus and conscious processing speed measured in tens of transactions per second on a good day. We excel at "intuition", at the instantaneous parallel pruning of enormous spaces to the relevant part. rgb > > In technical areas that people like to hear about, it is impossible as a > single person to fight unpaid all the desinformation. > The biggest problem there is professors who seek publicity, or cover up for > total nonsense thesis of their students. > > I tried to have some sort of discussion with Jonathan Schaeffer about a > student of him quoting MTD is ok in his thesis. > > What i skipped in fact is the total nonsense about parallel speedup in game > tree searching from his student idea being: > make the most inefficient searcher on planet earth, then parallellize it and > of course because even random move orderings > would improve your program then, also the parallellism gets a decent speedup > as a result of all that inefficiency; > the classical approach. > > Also MiT is notorious bad in game tree search. > > I will give yet 1 more example. > > I remember they claimed in articles a great speedup (50% or so even) in game > tree search at supercomputers. > > Now my chessprogram has a lot of chessknowledge and also is doing all sorts > of mobility, which requires expensive > scans. That slows down the nodes per second that my chessprogram Diep can > get. > > The search frame from MiT slows down the searchspeed of their chessengine by > a staggering factor 40. > > At the same SGI Origin3800 hardware in fact my slow Diep engine gets MORE > nodes per second, and a lot more, > than MiT's cilkchess chess engine. > > Not because single core mine is any faster. In fact single core cilkchess > without cilkframe used to be factor 20. > > Additionally there is nowhere anything statistical significant done there. > All speedups of supercomputers are never carrying > a table with what is its speedup with 95% sureness. > > Doing publications just to show how it should be done and disprove and prove > a number of algorithms (especially disprove nonsense), > is already quite a tough challenge and i can be busy for years there, as > there is hundreds of such publications, just from the past few > years to disprove and show the better alternative. > > It's a technical science, there is not many who really are expert there, > progress goes *real* fast. > > In the end what matters for engines is not how strong it plays against you, > it matters how strong you play against competitors :) > > On Jun 25, 2008, at 7:14 AM, Robert G. Brown wrote: > >> On Wed, 25 Jun 2008, Vincent Diepeveen wrote: >> >>>> one player's position -- or one of those distant pieces placed early in >>>> the game -- causes their entire effort to "unravel" and turn into a >>>> disaster. That's almost twice the number of plys in an entire chess >>>> game, and is still only the first third or so of the game. >>> >>> See above for correct calculation. >>> >>>> If you or your friend disagree with this, well, feel free to edit the >>>> wikipedia article(s) with examples that contradict it, but the >>>> mathematics and difficulty of pruning the Go trees suggest that it >>>> isn't. >>> >>> See above for disproof of that. >> >> Very educational and interesting. But I was also serious -- if you can >> (and it sounds like you can) you should consider editing the wikipedia >> articles. >> >> I'm not COMPLETELY convinced -- I'll be a lot more convinced when I can >> get a GPL Go engine that will play a decent game on my laptop...;-) >> >> rgb >> >> -- >> Robert G. Brown Phone(cell): 1-919-280-8443 >> Duke University Physics Dept, Box 90305 >> Durham, N.C. 27708-0305 >> Web: http://www.phy.duke.edu/~rgb >> Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php >> Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From diep at xs4all.nl Wed Jun 25 07:39:49 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> <7F09EDD7-0A8A-4B05-BD72-F3A984E39CFC@xs4all.nl> Message-ID: <3925DE45-F3A5-4073-A9B5-295A85744283@xs4all.nl> You raise an interesting question there, which is: "how are humans doing it". When i discussed this recentely with a brain researcher (so the type of researcher that really puts brains in a 3-Tesla MRI scanner) i also quoted the old fashioned theories as that the brain works at like 12Hz. His answer was very surprising (apologies to not know the English word for it) that the central part at where all the brain cells connect to, is at far higher speed than what i assumed it is taking decisions. Something in the order of 200 kHz to 400 Mhz. The thing unknown in how it exactly works to researchers is this high speed central nerve system. Thousands of times faster than i assumed it worked at, which was my big surprise. If we multiply that speed by a few billion brain cells then obviously we should revisit our idea on how the brain works. Maybe a new entry in wiki is needed for them as well :) As for the 1000^3 cube, there is of course a lot of searching algorithms that are yet there waiting for us to get invented. Simple ideas sometimes work rather well when combined with a few others. The classical example open to invention still is multiplication of big numbers and matrix calculations; to avoid questions there and zoom in: if it is possible to first do some slow conversion to fourier space, then multiply it there and then convert it back to radix2, obviously it is also possible to have a direct O (n log n) algorithm that can achieve a bigger throughput of bits per limbs multiplied. The real fundamental problem there has more to do with: "what do companies pay for". Companies hardly ever pay for research, they let government pay for that in general. Companies are there to produce products and earn on them. Any form of research objectively seen therefore is overhead unnecessary, where they only invest in when it is absolutely necessary. Additionally most departments of universities where sometimes a few PHD's are doing research, they have little experience yet in their field. Fields are so advanced nowadays, it not seldom takes 10 years to master it. Most professors are fulltime busy with meetings and have no time to even keep up with the latest theories and developments in their fields. The number of persons that really can do research therefore is really limited and fundamental research like investigating new manners of doing a matrix calculation more efficient, is hardly getting sponsored, even when there is a direct military reason to do so. As i figured out the hard way, first proving you can do better than other scientists and THEN asking for funding, is not clever either. So to get back on the original brain question: "how are humans doing it" My answer is that the majority messes up so bigtime, that it is unclear how *some* are doing it. Vincent On Jun 25, 2008, at 4:10 PM, Robert G. Brown wrote: > On Wed, 25 Jun 2008, Vincent Diepeveen wrote: > >> Chess has indeed more than 10^40 positions. Latest mathematical >> calculation as published in ICGA journal came to 10^43. >> >> There is 2 points: >> a) playing the best move doesn't require searching the entire >> search space, >> so who knows maybe searching 10^18 nodes with a huge >> hashtable and good evaluation function, >> we might already play the best move. > > :-) That's true and funny at the same time. "Only" 10^18 > positions, at > "only" a nanosecond each takes "only" a gigasecond, which since a year > is \pi x 10^7 seconds IIRC, is "only" around \pi x 10 Gigasearch- > years. > > It is a difficult problem, in other words. It's what makes it fun, I > suppose. And equally obviously, human brains don't work anything like > this way on it. To me it is more fun to speculate on how the human > brain does it WITHOUT anything like the capability to process > giga-giga-trees. It clearly prunes things down to some much more > finite > order extremely quickly by discarding nearly all of this space from > the > beginning, and YET the space that remains is often the good one to the > best that even Big Blue can parse out otherwise. > >> b) To get to 10^43 calculation power, we do not need a processor >> that exists out of 10^43 particles. All we need is >> a processor that is hosting 10^43 internal movements, which is a >> lot easier to get acomplished, than building that >> cpu of 10^43 in size. > > I'm not sure what you mean by that. All silicon processing has the > usual serial and parallel components and accomplishes work (per > processor) with a gigahertz scale serial barrier, leaving it many, > many > orders of magnitude short even in parallel. Yes, neural network > processing in the brain has way fewer (maybe 10^11) neurons, but each > neuron has thousands of synaptic connections to other neurons and the > network formed has staggering complexity. Big enough that the chess > problem could actually fit in its entirety inside with room enough for > the go problem left over. And still be nearly "empty", and work > in an extremely parallel fashion. > > There aren't a whole lot of candidate architectures for a parallel > processing system that can do this sort of thing, but I agree that > this > is "all" we need. It's just not terribly easy to accomplish, because > the problem is hard (and fun). > > One encounters numbers that scale like this in physics (stat mech) > quite > often, actually. If you work out the states in e.g. Monte Carlo > sampling of a very simple 100x100x100 simply cubic Ising ferromagnet > lattice (where each site contains a spin that can hold one of two > states) the number of states in the system is 2^(1,000,000). This > is "a > big number" at least at first glance. > > If one then traces through "moves" through the system from any given > starting state through all the various intermediary states to all the > possible final states, one creates a space with e.g. [2^1000000]^N > configurational trajectories for N steps. Ooo, maybe THAT is a big > number, at least if N is reasonably large (say, a few thousand). > > Out of all of those states, a MUCH MUCH SMALLER set of states > represents > the only ones that are "likely" to be seen if the system is in > equilibrium at any given temperature. How to find this needle in a > very > "large" (and yet tiny -- we're talking a mere million atoms when > physical systems big enough to see contain, um, a lot more than > this and > have continuous spins instead of discrete ones) haystack? > > Importance sampling Monte Carlo lets you discard nearly all moves in a > Markov process that takes you from any starting configuration to the > "relevant" part of phase space rather quickly (considering) and then > samples that space effectively. It rejects far, far, far more states > than there are particles in the Universe with a metaphorical > "sniff" as > being absurdly unlikely and homes right in on the likely suspects. > > I think that this is a much better model, or metaphor, for how humans > think than search trees. Humans really suck at search trees, with > their > fundamentally serial attention focus and conscious processing speed > measured in tens of transactions per second on a good day. We > excel at > "intuition", at the instantaneous parallel pruning of enormous > spaces to > the relevant part. > > rgb > >> >> In technical areas that people like to hear about, it is >> impossible as a single person to fight unpaid all the desinformation. >> The biggest problem there is professors who seek publicity, or >> cover up for total nonsense thesis of their students. >> >> I tried to have some sort of discussion with Jonathan Schaeffer >> about a student of him quoting MTD is ok in his thesis. >> >> What i skipped in fact is the total nonsense about parallel >> speedup in game tree searching from his student idea being: >> make the most inefficient searcher on planet earth, then >> parallellize it and of course because even random move orderings >> would improve your program then, also the parallellism gets a >> decent speedup as a result of all that inefficiency; >> the classical approach. >> >> Also MiT is notorious bad in game tree search. >> >> I will give yet 1 more example. >> >> I remember they claimed in articles a great speedup (50% or so >> even) in game tree search at supercomputers. >> >> Now my chessprogram has a lot of chessknowledge and also is doing >> all sorts of mobility, which requires expensive >> scans. That slows down the nodes per second that my chessprogram >> Diep can get. >> >> The search frame from MiT slows down the searchspeed of their >> chessengine by a staggering factor 40. >> >> At the same SGI Origin3800 hardware in fact my slow Diep engine >> gets MORE nodes per second, and a lot more, >> than MiT's cilkchess chess engine. >> >> Not because single core mine is any faster. In fact single core >> cilkchess without cilkframe used to be factor 20. >> >> Additionally there is nowhere anything statistical significant >> done there. All speedups of supercomputers are never carrying >> a table with what is its speedup with 95% sureness. >> >> Doing publications just to show how it should be done and disprove >> and prove a number of algorithms (especially disprove nonsense), >> is already quite a tough challenge and i can be busy for years >> there, as there is hundreds of such publications, just from the >> past few >> years to disprove and show the better alternative. >> >> It's a technical science, there is not many who really are expert >> there, progress goes *real* fast. >> >> In the end what matters for engines is not how strong it plays >> against you, it matters how strong you play against competitors :) >> >> On Jun 25, 2008, at 7:14 AM, Robert G. Brown wrote: >> >>> On Wed, 25 Jun 2008, Vincent Diepeveen wrote: >>>>> one player's position -- or one of those distant pieces placed >>>>> early in >>>>> the game -- causes their entire effort to "unravel" and turn >>>>> into a >>>>> disaster. That's almost twice the number of plys in an entire >>>>> chess >>>>> game, and is still only the first third or so of the game. >>>> See above for correct calculation. >>>>> If you or your friend disagree with this, well, feel free to >>>>> edit the >>>>> wikipedia article(s) with examples that contradict it, but the >>>>> mathematics and difficulty of pruning the Go trees suggest that it >>>>> isn't. >>>> See above for disproof of that. >>> Very educational and interesting. But I was also serious -- if >>> you can >>> (and it sounds like you can) you should consider editing the >>> wikipedia >>> articles. >>> I'm not COMPLETELY convinced -- I'll be a lot more convinced when >>> I can >>> get a GPL Go engine that will play a decent game on my laptop...;-) >>> >>> rgb >>> -- >>> Robert G. Brown Phone(cell): >>> 1-919-280-8443 >>> Duke University Physics Dept, Box 90305 >>> Durham, N.C. 27708-0305 >>> Web: http://www.phy.duke.edu/~rgb >>> Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/ >>> Lilith.php >>> Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 >> > > -- > Robert G. Brown Phone(cell): 1-919-280-8443 > Duke University Physics Dept, Box 90305 > Durham, N.C. 27708-0305 > Web: http://www.phy.duke.edu/~rgb > Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php > Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 > From geoff at galitz.org Wed Jun 25 07:50:42 2008 From: geoff at galitz.org (Geoff Galitz) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org><87skv8wxlg.fsf@snark.cb.piermont.com><20080620143621.GA31402@synapse.neuralscape.com><485BC8C6.4080800@vcu.edu><5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com><485BF505.6000904@ias.edu><7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com><20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> Message-ID: I've never really bought the argument that biofuels are causing a food shortage considering that there is still so much unused farmland in the US and farming practices here in the EU. I must admit this out of my field so I have no real evidence to support my suspicion, but there have been a trickle of articles from publications like Der Spiegel and the NY Times which indicate there is some suspicion these shortages have just as much to do with commodity speculation and manipulation. http://www.iht.com/articles/2008/06/12/business/speculate.php http://www.spiegel.de/international/world/0,1518,559550,00.html (in English) Geoff Galitz Blankenheim NRW, Deutschland http://www.galitz.org -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Vincent Diepeveen Sent: Mittwoch, 25. Juni 2008 16:07 To: Jon Aquilina Cc: Beowulf Mailing List; Mark Hahn Subject: Re: [Beowulf] Re: "hobbyists" Even worse, Why is there subsidy on bio fuels that get produced out of food eaten by poor people? This causes as we speak people dying as they can no longer for a cent or so buy food made out of it; the prices have doubled if not more for such types of cheap food because of subsidy in the 1st world countries for this. I assume EU will take measures to turn back those subsidies on bio fuels that get produced out of food that feeds billions, who now hardly can afford to buy food anymore as it gets burned for energy in first world countries, whereas commercially spoken it cannot get burned, it is just because of subsidy it can exist. Even better i would be in favour of a ban on bio fuels that are outright food products in 3d world countries. When i just walked previous week into a shop and my sister was interested in a new washing machine, i pointed her to the fact that the thing she was interested in, was eating 3.8 kW, versus the 100 euro more expensive thing next to it was eating 1.14 kW. It is something that only very few will notice. It is easy to cheaply produce equipment that eats more power than equipment of competitors, that is the fundamental problem. Vincent - speaking for himself On Jun 25, 2008, at 8:43 AM, Jon Aquilina wrote: > how much does a sugar glass window cost now that sugar and other > things are being used for bio fuels? > > On Wed, Jun 25, 2008 at 12:20 AM, Mark Hahn wrote: > > More specifically for HPC, linux seems designed for the desktop, and > for small memory machines. > > the only justice I can see in that is that there hasn't been all > that much effort to get bigpages widely/easily used. in > particular, I don't > see that scheduler or general memory-management issues in linux are > particularly biased for desktop or against HPC. > > > That's funny, because I've heard people get scared that it was the > complete > opposite. That Linux was driven by Big Iron, and that no one cared > about > the "little desktop guy" (Con Kolivas is an interesting history > example). > > Con didn't play the game right - you have to have the right > combination of social engineering (especially timing and proactive > response) and good tech > kungfoo. kernel people are biased towards a certain aesthetic that > doesn't > punish big-picture redesigns from scratch, but _does_ punish > solutions in search of a problem. > > so the question is, if you had a magic wand, what would you change > in the kernel (or perhaps libc or other support libs, etc)? most > of the things I can think of are not clear-cut. I'd like to be > able to give better info from perf counters to our users (but I > don't think Linux is really in the way). I suspect we lose some > performance due to jitter > injected by the OS (and/or our own monitoring) and would like to > improve, > but again, it's hard to blame Linux. I'd love to have better > options for cluster-aware filesystems. kernel-assisted network > shared memory? > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > > > -- > Jonathan Aquilina > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From prentice at ias.edu Wed Jun 25 07:52:14 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <4861E016.9040803@cse.ucdavis.edu> References: <5CC60DD0-53FD-4D5D-A7A5-8638E54F3F17@xs4all.nl> <48614A7F.2080606@ias.edu> <520EC3EF-536A-4BD8-9806-492C23D2A058@xs4all.nl> <4861E016.9040803@cse.ucdavis.edu> Message-ID: <48625B9E.3090802@ias.edu> Bill Broadley wrote: > Vincent Diepeveen wrote: >> intel c++ obviously is close to visual studio. Within 0.5% to 1.5% >> range (depending upon flags > > I believe Microsoft licensed the intel optimization technology, so the > similarity is hardly surprising. > >> and hidden flags that you managed to get from someone). Intel C++ is >> free for researchers such as >> me. > > Last I checked it was fine for research in compilers, but as a tool to > help facilitate research you have to pay, even in an academic > environment. Maybe I misread the license, what exactly let you to > believe that it's "free for researchers"? If you get paid to do your research in any way(as an employee of a university, non-profit, govt agency, big oil, small bio-tech, graduate student or post-doc receiving a stipend), you must pay Intel for a license. Degree-granting academic institutions get a discount. If you are programming for a personal project on your own accord and aren't being compensated in any way for it, it's free. -- Prentice From rgb at phy.duke.edu Wed Jun 25 07:57:24 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists"es In-Reply-To: <20080625133326.GA29168@nlxdcldnl2.cl.intel.com> References: <87abhfzpcw.fsf@snark.cb.piermont.com> <20080625133326.GA29168@nlxdcldnl2.cl.intel.com> Message-ID: On Wed, 25 Jun 2008, Lombard, David N wrote: > On Sat, Jun 21, 2008 at 02:09:37PM -0700, Robert G. Brown wrote: >> >> >> > > It would appear the rgb-bot ran amuk! The rgb-bot just got a new Dell XPS M1530. It is only sold with Vista, and Duke gets Vista Business. It got here Monday. I took it out of the box, booted it up into Vista, and started to "dress" it in Duke's software packages -- VPN clients, AV, a few site license packages -- and went through the mandatory updates and setup needed even for Windows that I was planning to use pretty much to watch movies and that's it (although I was going to give it an honest try before deciding whether or not to retroactively install XP Pro in a VM). Well, the fingerprint scanner wouldn't work at all, but I postponed worrying about that. I did about an hour and a half of work (through a couple of reboots) and decided to defragment the system (now that it was "done") preparatory to installing linux on it. I figured that if I defragged it it would minimize the work gparted would have to do. It threw an error in the defrag and forced a reboot. The reboot reported that Vista had eaten its own partition table, irreparably. Five or ten passes through its recovery tools indicate the latter. I sigh, decide that it is "good luck" as it makes it really easy to repartition for linux if I have to reinstall Vista anyway, and do a Fedora 8 install into a solid half of the 320 BG disk. Totally uneventful install, then a short while mucking around on ITS post install update, and then I start to move in. I login, everything is completely normal -- a few devices I have to tweak, some bios settings that conflict with linux (the bios wants to manage wireless, bluetooth and cell, and linux can't cope unless this is all turned off so that the devices are just devices). The GUI works fine, X works fine, the mouse works fine. I reboot, and the touchpad goes berserk. At first it just doesn't work. Then it is clear -- it has "pad disease". That's where one touches the wristrest or the mouse and the mouse whips off to the lower left corner and issues some random click sequence. Usually my laptops get this when the laptop is two years old. Then the keyboard gradually gets keyboard disease, where if I press on the left wristrest with any weight at all it gets "stuck" and goes nuts for a second or ten. All covered by Dell's Custom Care etc, so they'll replace it all as soon as possible, but in the meantime the rgb-bot can generate interesting noise if his thumb brushes against the pad or either wrist actually rests on the laptop. Sigh. rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From diep at xs4all.nl Wed Jun 25 08:05:14 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org><87skv8wxlg.fsf@snark.cb.piermont.com><20080620143621.GA31402@synapse.neuralscape.com><485BC8C6.4080800@vcu.edu><5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com><485BF505.6000904@ias.edu><7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com><20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> Message-ID: <4B831460-3895-433D-A399-8243B3E6C069@xs4all.nl> It is not about what we do in the EU and/or what happens in USA. That is the typical short sighted view of politicians who decided on the subsidy. What matters is that the 'biofuel' of course does get imported from South America and Africa and that it causes in THOSE countries a huge inflation of food. Some 3d world country managers are begging to adress this issue: "My nations people die, as your bio fuel raises our food prices, the poor are so poor here, they use that stuff as food and cannot afford it now". USA nor Europe can *never* produce that stuff as cheap as 3d world countries can. Agriculture in USA and Europe only exists because of protectionism (without me taking a viewpoint on protectionistic measures). For biofuels there is always manners to import all those different types of food, creating at its LOCAL market a huge food shortage and inflation of food prices for the poor, a lot of whom die as a result. Not a single EU rule will be able to avoid that it gets imported and later on somehow subsidy on bio-fuel causes it to get burned for energy, whereas it is of course a pathetic expensive form of energy. There is always a manner to get the stuff into the country. For example you want to produce product X of it. By accident after import production line of X gets left with 95% waste product, which then you burn for bio-fuel. So food of the poor ends up as biofuel, good for only a very tiny % of all energy produced and "by accident" always a smaller % than the % of energy that gets heavily subsidized for being bio-fuel! Yet if you deduce macro-economically, it is LOGICAL that the stuff that gets used to produce the cheapest food for the poor, also has a price low enough to get cheaply imported from those 3d world countries to our nation, to get burned as bio-fuel. The poorest of the poorest can afford that stuff only because it is the cheapest thing on planet earth, and we are taking it away from them just for some stupid subsidized bio-energy! Vincent On Jun 25, 2008, at 4:50 PM, Geoff Galitz wrote: > > > I've never really bought the argument that biofuels are causing a food > shortage considering that there is still so much unused farmland in > the US > and farming practices here in the EU. I must admit this out of my > field so > I have no real evidence to support my suspicion, but there have been a > trickle of articles from publications like Der Spiegel and the NY > Times > which indicate there is some suspicion these shortages have just as > much to > do with commodity speculation and manipulation. > > > http://www.iht.com/articles/2008/06/12/business/speculate.php > http://www.spiegel.de/international/world/0,1518,559550,00.html (in > English) > > > > Geoff Galitz > Blankenheim NRW, Deutschland > http://www.galitz.org > > > -----Original Message----- > From: beowulf-bounces@beowulf.org [mailto:beowulf- > bounces@beowulf.org] On > Behalf Of Vincent Diepeveen > Sent: Mittwoch, 25. Juni 2008 16:07 > To: Jon Aquilina > Cc: Beowulf Mailing List; Mark Hahn > Subject: Re: [Beowulf] Re: "hobbyists" > > Even worse, > Why is there subsidy on bio fuels that get produced out of food eaten > by poor people? > > This causes as we speak people dying as they can no longer for a cent > or so buy food made out > of it; the prices have doubled if not more for such types of cheap > food because of subsidy in the > 1st world countries for this. > > I assume EU will take measures to turn back those subsidies on bio > fuels > that get produced out of food that feeds billions, who now hardly can > afford to buy food anymore as > it gets burned for energy in first world countries, whereas > commercially spoken it cannot get burned, > it is just because of subsidy it can exist. > > Even better i would be in favour of a ban on bio fuels that are > outright food products in 3d world countries. > > When i just walked previous week into a shop and my sister was > interested in a new washing machine, > i pointed her to the fact that the thing she was interested in, was > eating 3.8 kW, versus the 100 euro more expensive > thing next to it was eating 1.14 kW. It is something that only very > few will notice. > > It is easy to cheaply produce equipment that eats more power than > equipment of competitors, that is the fundamental problem. > > Vincent - speaking for himself > > On Jun 25, 2008, at 8:43 AM, Jon Aquilina wrote: > >> how much does a sugar glass window cost now that sugar and other >> things are being used for bio fuels? >> >> On Wed, Jun 25, 2008 at 12:20 AM, Mark Hahn wrote: >> >> More specifically for HPC, linux seems designed for the desktop, and >> for small memory machines. >> >> the only justice I can see in that is that there hasn't been all >> that much effort to get bigpages widely/easily used. in >> particular, I don't >> see that scheduler or general memory-management issues in linux are >> particularly biased for desktop or against HPC. >> >> >> That's funny, because I've heard people get scared that it was the >> complete >> opposite. That Linux was driven by Big Iron, and that no one cared >> about >> the "little desktop guy" (Con Kolivas is an interesting history >> example). >> >> Con didn't play the game right - you have to have the right >> combination of social engineering (especially timing and proactive >> response) and good tech >> kungfoo. kernel people are biased towards a certain aesthetic that >> doesn't >> punish big-picture redesigns from scratch, but _does_ punish >> solutions in search of a problem. >> >> so the question is, if you had a magic wand, what would you change >> in the kernel (or perhaps libc or other support libs, etc)? most >> of the things I can think of are not clear-cut. I'd like to be >> able to give better info from perf counters to our users (but I >> don't think Linux is really in the way). I suspect we lose some >> performance due to jitter >> injected by the OS (and/or our own monitoring) and would like to >> improve, >> but again, it's hard to blame Linux. I'd love to have better >> options for cluster-aware filesystems. kernel-assisted network >> shared memory? >> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> >> >> >> -- >> Jonathan Aquilina >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From john.hearns at streamline-computing.com Wed Jun 25 08:39:32 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com><485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> Message-ID: <1214408382.4979.9.camel@Vigor13> On Wed, 2008-06-25 at 16:50 +0200, Geoff Galitz wrote: > > I've never really bought the argument that biofuels are causing a food > shortage considering that there is still so much unused farmland in the US > and farming practices here in the EU. I must admit this out of my field so > I have no real evidence to support my suspicion, but there have been a > trickle of articles from publications like Der Spiegel and the NY Times > which indicate there is some suspicion these shortages have just as much to > do with commodity speculation and manipulation. Hauling this thread back closer to the track, there is an article in this week's Economist magazine entitled "How computers can help to cut carbon emissions" http://www.economist.com/business/displayStory.cfm?source=hptextfeature&story_id=11585208 Me, I think looking at the amount of watts that a typical cluster uses we're more likely to ADD to global warming. From jmdavis1 at vcu.edu Wed Jun 25 09:04:38 2008 From: jmdavis1 at vcu.edu (Mike Davis) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" still OT In-Reply-To: <4B831460-3895-433D-A399-8243B3E6C069@xs4all.nl> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org><87skv8wxlg.fsf@snark.cb.piermont.com><20080620143621.GA31402@synapse.neuralscape.com><485BC8C6.4080800@vcu.edu><5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com><485BF505.6000904@ias.edu><7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com><20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> <4B831460-3895-433D-A399-8243B3E6C069@xs4all.nl> Message-ID: <48626C96.8060305@vcu.edu> Vincent Diepeveen wrote: > > Some 3d world country managers are begging to adress this issue: "My > nations people die, > as your bio fuel raises our food prices, the poor are so poor here, > they use that stuff as food > and cannot afford it now". > > USA nor Europe can *never* produce that stuff as cheap as 3d world > countries can. > Since my uncle held a patent on an Alcohol Fuel still, and my grandfather grew corn, I have a somewhat different view. I'm not sure where your information is coming from. According to http://www.earth-policy.org/, the US produced 414 Million tons of grain last year. Of that total 81 Million tons were used for fuel and 106 Million tons were exported. While it might be less expensive to produce grain in the 3rd world, the top exporter of Corn, Rice and Wheat for the world is the US. http://www.earth-policy.org/Updates/2008/Update69_data.htm#table10 http://www.earth-policy.org/Updates/2008/Update69_data.htm#table9 Now since corn prices have been rather stagnant until recently for almost 30 years and since I know farmers that have burned corn for heat rather than sell at a loss. I see the increase in grain prices as neutral. Yes, there are bad results for the third world. But those farmers growing grain deserve a fair price for the work and uncertainty of agriculture. From kus at free.net Wed Jun 25 09:34:53 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: Message-ID: Let me assume now the following situation. I have OpenMP-parallelized application which have the number of processes equal to number of CPU cores per server. And let me assume that this application uses not too more virtual memory, so all the real memory used may be placed in RAM of *one* node. It's not the abstract question - a lot of Gaussian-03 jobs we have fit to this situation, and all the 8 cores for dual socket quad core Opteron server will be "well loaded". Is it right that all the application memory (w/o using of numactl) will be allocated (by Linux kernel) in *one* node ? Then only one memory controller will be used. OK, then if I have the same server but w/2 times more small memory size (it's enough for run of this Gaussian-03 job !) and DIMMs are populating both nodes, then the performance of this server will be higher ! - because both memory controllers (and therefore more memory channels) will work simultaneously. Is it right - that more cheap server will have higher performance for like cases ?? Mikhail Kuzminsky Computer Assistance to Chemical Research Center Zelinsky Institute of Organic Chemistry Moscow From diep at xs4all.nl Wed Jun 25 09:52:56 2008 From: diep at xs4all.nl (Vincent Diepeveen) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" still OT In-Reply-To: <48626C96.8060305@vcu.edu> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org><87skv8wxlg.fsf@snark.cb.piermont.com><20080620143621.GA31402@synapse.neuralscape.com><485BC8C6.4080800@vcu.edu><5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com><485BF505.6000904@ias.edu><7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com><20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> <4B831460-3895-433D-A399-8243B3E6C069@xs4all.nl> <48626C96.8060305@vcu.edu> Message-ID: <6392137C-B548-45BC-8762-FF8B36E010D2@xs4all.nl> Take me correct, myself living outside of the big cities here into a city which is surrounded by farmers who have been hit very hard, i'm not taking a viewpoint on the subsidied exporting of Corn, Rice and Wheat. What i find bad is that these types of products which get produced a lot cheaper in 3d world countries, not subsidized, now are getting used to get subsidy on bio-fuel, as there has been set a target on which percentage of energy "must" be green energy. Even a tiny nation like Netherlands that is not a production nation (therefore we consume less power than production nations such as France, Germany, USA, as factories eat 90% of all power), at its peek is consuming about 10000 megawatt of power. I'm not sure whether the entire production of wheat, corn and rice from the USA is even closely enough to produce enough energy for a tiny nation like the Netherlands, with just 16.5 million inhabitants. Now calculate how many farmers are involved and how much they need to make a year to have a living. Doesn't just transport costs of Wheat and Corn already make it more expensive than the equivalent in coals or uranium? Note that here in Europe in case of coals they get transported to their destiny by boat usually. That minimum cost price for all that wheat, corn and rice is having a costprice of tens of billions, whereas the total production on planet earth in potatoes, rice, wheat and corn is not that much bigger than what we need to feed ourselves. Yet from all that wheet and corn produced in entire USA for that huge price of tens of billions, you can't even power a tiny nation from that price that is a big multiple for what energy centrals buy in their coals, gas or low enriched uranium (U235/U238). Let's be objective whether an energy centrals which can buy in coals for just a few dollars for each ton of coals (1 ton = 1000 kilo), can *benefit* from buying in wheat/corn to burn for their energy central to produce. They do it at a big loss as EU rules force them here, idemdito in USA. Logically therefore only subsidy on bio-fuel causes a lot of Corn/ Wheat from 3d world nations to get bought in, as it gets produced a lot cheaper there. That causes a big increase in price, if there is still corn/wheat left in those nations. The subsidy on a single kilowatt hour of bio fuel is a lot more than what a poor man in a 3d world country can afford to pay for its wheat. However the poor man needs to eat just a little, and that's where production has been focussing upon past thousands of years. Hardly producing enough to feed the world population. Now you want to burn all that for a few pennies of subsidy? To produce those tens of thousands of megawatts you can never produce enough corn. It is a total dead end to use corn and wheat for bio-fuel. Even solar panels are better than that and have more future, as the sun radiates around 1300 watt of power on each square meter. When i sat in the platform high voltage powerlines, you soon figure out one thing and that is that the Energy world is a very complex technical world. It is hard to get experts speak out. I also spoke to an Australian researcher who has this viewpoint: "coals is so evil, many people die digging up the coals in Australia, we really must close those coal mines and just produce nuclear fuel". A very valid viewpoint. Most experts keep their mouth shut bigtime, afraid for this or that interest group. When one or 2 persons say a word, they directly get from government told that they should shut up as they have signed a secrecy act, and that the documents in question were given a stamp 'secret'. We have dealt with that as well. It is very hard to fight against. That's what limits this discussion too. In meantime people die in South America as they cannot buy food as it has gone too expensive. You could argue 30% of that inflation is caused by socialistic regimes and mismanagement perhaps, but the price of it has gone up dramatically more than that. If i add into account also what RGB quoted here: "we are soon out of U238 as well", then that is of course not a very optimistic thing to tell to your citizens. Have faith, scientists will find a solution is what i told in front of the camera. Yet bio-fuel is not the solution. That just kills right now the poor in 3d world nations. In front of that same camera a director of an energy company did do a statement on 'green energy'. Camera's were running, but those fragments never were broadcasted. Not even used for a newsreport. Didn't pass censorship i assume. Vincent On Jun 25, 2008, at 6:04 PM, Mike Davis wrote: > Vincent Diepeveen wrote: >> >> Some 3d world country managers are begging to adress this issue: >> "My nations people die, >> as your bio fuel raises our food prices, the poor are so poor >> here, they use that stuff as food >> and cannot afford it now". >> >> USA nor Europe can *never* produce that stuff as cheap as 3d world >> countries can. >> > Since my uncle held a patent on an Alcohol Fuel still, and my > grandfather grew corn, I have a somewhat different view. I'm not > sure where your information is coming from. According to http:// > www.earth-policy.org/, the US produced 414 Million tons of grain > last year. Of that total 81 Million tons were used for fuel and 106 > Million tons were exported. While it might be less expensive to > produce grain in the 3rd world, the top exporter of Corn, Rice and > Wheat for the world is the US. http://www.earth-policy.org/Updates/ > 2008/Update69_data.htm#table10 > > > http://www.earth-policy.org/Updates/2008/Update69_data.htm#table9 > > Now since corn prices have been rather stagnant until recently for > almost 30 years and since I know farmers that have burned corn for > heat rather than sell at a loss. I see the increase in grain prices > as neutral. Yes, there are bad results for the third world. But > those farmers growing grain deserve a fair price for the work and > uncertainty of agriculture. From peter.st.john at gmail.com Wed Jun 25 10:03:26 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: Message-ID: RGB, My hypothetical future Recursive Genetic Algorithm Go Player will crush your Hypothetical Future NeuralNet Go Player! Peter On 6/24/08, Robert G. Brown wrote: > > On Tue, 24 Jun 2008, Peter St. John wrote: > > On a finite board, the game eventually becomes local; in fact "contact >> plays" are common after about the first dozen moves, but they are >> inescapable later. But the possibilities are horrible, the numbers are >> huge, >> for tree-searching. The new thing is monte carlo; to consider a move, the >> machine actually plays that position against itself a few hundred times >> (with a trivial, not recursive, algorithm) and weighs the move by the >> percentage of outcomes. It's weird, to me. >> > > It makes sense to me. It's in some sense more like humans play. In > fact, if one can come up with any (even very simple) "local" rules to > semi-prune the tree, you can imagine an importance-sampling monte carlo > algorithm. > > However, if I were going to try, it would be genetic algorithm all the > way. Don't explore trees exhaustively; too many of them. Don't sample > them (especially if you can't create a canonical weight to do importance > sampling); too many of them. Breed them. There are still too many, but > again, if one can generate ANY sort of fitness function, maybe one can > get the N^3 and exponential growth of favorable possibilities that are > like "making a decision". > > Fortunately, I'm not going to try. Neural nets are much more fun. And > MIGHT be able to manage the complexity -- especially one built with a > GA...;-) > > rgb > > >> Peter >> >> On Tue, Jun 24, 2008 at 9:22 PM, Robert G. Brown >> wrote: >> >> On Tue, 24 Jun 2008, Peter St. John wrote: >>> >>> Programming a computer to play Go (an Asian strategy boardgame) has been >>> >>>> difficult; some people say it's proof that Go is better or harder than >>>> chess, since computers can beat masters at chess but struggle at Go. (I >>>> think that statistically a game of go is about equivalent to a two-game >>>> match of chess; both games empty your brain quickly of course). My view >>>> is >>>> that while go may be somewhat harder to reduce to tree-searching, the >>>> main >>>> advantage of computer chess was an early start, e.g. von Neumann. >>>> >>>> >>> I thought that Go was combinatorially vastly, vastly more difficult than >>> chess. Both because of the extremely large number of move permutations >>> that make it very difficult to follow trees and because Go is a >>> fundamentally nonlocal game -- one can imagine a "perfect" Go strategy >>> in which pieces are never played close to each other as the board >>> gradually fills in with higher and higher still disconnected density, >>> culminating in the winner completely unravelling the loser or >>> precipitating out to win by a single small amount. Yet nobody can even >>> come close to playing that way -- most games start out that way for a >>> while and then transform into local dogfights that are STILL often >>> resolved by long range connections. >>> >>> This article: >>> >>>> http://www.usgo.org/resources/downloads/CogApdx%20II-2.pdf >>>> describes recent trends in computer Go and mentions a 32-node cluster, 8 >>>> cores per node. Apparently MPI parallelization is recent for them and >>>> they >>>> are making good progress. >>>> >>>> >>> Interesting. I'll have to look. Last time I read up on this (in the >>> context of a conversation with you, IIRC:-) nobody could make a computer >>> go player that could beat even a very bad human player, where computers >>> back to maybe 386's have been able to play chess well enough to win at >>> least sometimes against weak humans (and win "often" against weak human >>> players by now). I suck at chess (but can still beat a lot of the >>> people I -- rarely -- play) and modern computers can beat me pretty >>> easily. I suck at Go, too -- but I can't even find a BAD go player to >>> run on a PC to try to get better. >>> >>> rgb >>> >>> >>> >>> Peter >>>> >>>> The game Go: http://en.wikipedia.org/wiki/Go_%28game%29 >>>> AGA (American Go Association): http://www.usgo.org >>>> >>>> >>>> -- >>> Robert G. Brown Phone(cell): 1-919-280-8443 >>> Duke University Physics Dept, Box 90305 >>> Durham, N.C. 27708-0305 >>> Web: http://www.phy.duke.edu/~rgb >>> Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php< >>> http://www.phy.duke.edu/%7Ergb/Lilith/Lilith.php> >>> Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 >>> >>> >> > -- > Robert G. Brown Phone(cell): 1-919-280-8443 > Duke University Physics Dept, Box 90305 > Durham, N.C. 27708-0305 > Web: http://www.phy.duke.edu/~rgb > Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php > Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080625/94251d73/attachment.html From rgb at phy.duke.edu Wed Jun 25 10:09:28 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <87skv8wxlg.fsf@snark.cb.piermont.com> <20080620143621.GA31402@synapse.neuralscape.com> <485BC8C6.4080800@vcu.edu> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> Message-ID: On Wed, 25 Jun 2008, Vincent Diepeveen wrote: > When i just walked previous week into a shop and my sister was interested in > a new washing machine, > i pointed her to the fact that the thing she was interested in, was eating > 3.8 kW, versus the 100 euro more expensive > thing next to it was eating 1.14 kW. It is something that only very few will > notice. Hey V. But that's not the important question; only a factor to it. The important question is how much energy total is used, not the rate used, and that may depend on lots of things. If you have to wash your clothes twice (or several times as long) to get them as clean (lower power means lower SOMETHING -- less agitation, less heating of the water, smaller capacity so you have to run more loads) then you have to factor in power times TIME, really the system time, your own time, and the energy and monetary cost of doubling the wear on your clothes. > It is easy to cheaply produce equipment that eats more power than equipment > of competitors, that is the fundamental problem. And it is equally easy to produce an "energy saving" product that saves energy by not working as well, or at all. I'm not really arguing, by the way -- only pointing out that you should look beyond JUST the energy RATE of draw and make sure that the product will actually perform just as well and still save total energy and do what you need it to do. It isn't always easy to do that, and it goes beyond just looking at the power consumption rate. Lower power could translate into "wearing slightly dirty clothes all the time" or it could translate into "getting a load that is just as large of just as many clothes just as clean -- but saving 2.5 KW every hour you use it." Then to REALLY do the CBA -- once you've established that it is all things being equal comparison -- you have to count the expected number of hours of operation over the duty cycle of the machine -- if you use it (say) three hours a week, that's around 150 hours a year, times ten years is 1500 hours. At a generous dime a KW-hour, that's a savings of maybe 350 Euros over ten years at a cost of an extra $100 now. Not bad, but not spectacular, either. If you use it more than 3 hours a week (and it draws the 2.5+ KW power differential for all of that time, which I doubt unless it is heating the water, in which case you have a variety of alternative solutions because heating water to a given temperature can't be done more or less efficiently, really) then you might save more. To put it another way, if it really is talking about PEAK draw, not CONTINUOUS draw -- a 3.5 KW washer couldn't even be plugged in over here and 3.5 KW is a huge draw for any appliance that doesn't heat water -- you might not save anything at all beyond the fact that you don't need a special circuit to run it on. If it IS heating water, then you need to be very careful. It takes a certain number of joules to heat water, period. Most washers let you choose to run a cold water wash, or select any degree of warmth. In the US they tend to get hot water from a hot water heater that delivers it for the whole house (so they only need a few hundred watts to run, enough to run the washer motor and a pump). If there is a power differential but they heat the same amount of water to the same temperature, they use the same amount of energy to do it (within a hair) period. They probably use very similar -- and much lower -- amounts of power to do the actual washing, or use higher power for spins and so on for brief periods of time. In this case there is almost certainly NO marginal cost advantage in the lower power unit except the need for cheaper receptacles to plug it into, and there might even be a total energy loss. Yet by putting a big notice on the machine, they can fool people into spending an extra hundred Euros. I'd ask lots of questions of the salespeople and fully understand the energy budget of the machine, not just the peak draw, before concluding that it is more or less "green". So this sort of CBA is a very good idea, but it has to be carefully done. It's miles per gallon that matters, not gallons per minute, so to speak. Amount of work done per delivered/purchased joule, not number of joules per second drawn to do possibly differential amounts of work with possibly differential quality in possibly differential amounts of time. rgb -- > > Vincent - speaking for himself > > On Jun 25, 2008, at 8:43 AM, Jon Aquilina wrote: > >> how much does a sugar glass window cost now that sugar and other things are >> being used for bio fuels? >> >> On Wed, Jun 25, 2008 at 12:20 AM, Mark Hahn wrote: >> >> More specifically for HPC, linux seems designed for the desktop, and >> for small memory machines. >> >> the only justice I can see in that is that there hasn't been all that much >> effort to get bigpages widely/easily used. in particular, I don't >> see that scheduler or general memory-management issues in linux are >> particularly biased for desktop or against HPC. >> >> >> That's funny, because I've heard people get scared that it was the complete >> opposite. That Linux was driven by Big Iron, and that no one cared about >> the "little desktop guy" (Con Kolivas is an interesting history example). >> >> Con didn't play the game right - you have to have the right combination of >> social engineering (especially timing and proactive response) and good tech >> kungfoo. kernel people are biased towards a certain aesthetic that doesn't >> punish big-picture redesigns from scratch, but _does_ punish solutions in >> search of a problem. >> >> so the question is, if you had a magic wand, what would you change in the >> kernel (or perhaps libc or other support libs, etc)? most of the things I >> can think of are not clear-cut. I'd like to be able to give better info >> from perf counters to our users (but I don't think Linux is really in the >> way). I suspect we lose some performance due to jitter >> injected by the OS (and/or our own monitoring) and would like to improve, >> but again, it's hard to blame Linux. I'd love to have better options for >> cluster-aware filesystems. kernel-assisted network shared memory? >> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> >> >> >> -- >> Jonathan Aquilina >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From rgb at phy.duke.edu Wed Jun 25 10:28:23 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: <3925DE45-F3A5-4073-A9B5-295A85744283@xs4all.nl> References: <04AB4FA1-5562-4E39-B46C-9DD4236D0505@xs4all.nl> <7F09EDD7-0A8A-4B05-BD72-F3A984E39CFC@xs4all.nl> <3925DE45-F3A5-4073-A9B5-295A85744283@xs4all.nl> Message-ID: On Wed, 25 Jun 2008, Vincent Diepeveen wrote: > His answer was very surprising (apologies to not know the English word for > it) that the central part at where all the brain cells connect to, is at far > higher > speed than what i assumed it is taking decisions. Something in the order > of 200 kHz to 400 Mhz. The thing unknown in how it exactly works to > researchers > is this high speed central nerve system. ???? I'm deeply, deeply skeptical. The rate of synapse firing is well known and long since measured. However, many synapses fire in parallel -- webs of cascading fires ripple around all the time. The aggregate rate of synapse firing may well be very high, then, but the clock per synapse very low (and synapses do other odd things, like saturate if you drive it too hard, get "tired", and so on). > Thousands of times faster than i assumed it worked at, which was my big > surprise. > > If we multiply that speed by a few billion brain cells then obviously we > should revisit > our idea on how the brain works. Maybe a new entry in wiki is needed for them > as well :) No, you can't multiply it out. This is already aggregated over those hundred billion cells, each with all of those synapses. Think about the energetics of it. Your brain is already using roughly 1/3 of your total energy budget as it is. 100 billion 400 Mhz clocks, with biological switching energies of any reasonable magnitude would cook your brain. > So to get back on the original brain question: "how are humans doing it" > My answer is that the majority messes up so bigtime, that it is unclear how > *some* are doing it. Humans probably do it lots of ways, starting with pursuing simple trees. The "messing up bigtime" is the people who are still trying to think linearly, the way we are programming the computers. Then suddenly they gain experience, and prune 90% of the trees away from the very beginning, and are a bit better. Then they develop elaborate pre-pruned trees -- a repertoire of "openings" -- and from them abstract some very nonlinear patterns that become a mix of "intuition" that causes them to not even glance at all the "obviously stupid" moves in any given situation and an ability to "guess" how certain very long sequences will play out even without working out ALL of the possibilities, and they suddenly become a decent player. Then they gain a lot more experience, and can "see" very far indeed down the relevant tracks, and see enough alternatives that they can be deceptive with people that can see less far or who still have a flawed intuition about certain sets of tracks. The same question applies to the Ramanujans and Riemanns. Some people can just "see" two numbers to know what their product is. They don't "compute it" by using a serial algorithm. They don't memorize it. They just know. I think that this is a really, really interesting question, BTW. An equally interesting one is whether or not we can "learn" this kind of intuition, and if so what would the training/learning process look like? Probably NOT teaching people long ways of algorithmically multiplying. Probably NOT asking them to memorize large tables. Something different. We probably don't even have the right task decomposition to begin to experiment. rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From rgb at phy.duke.edu Wed Jun 25 10:33:55 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Go-playing machines In-Reply-To: References: Message-ID: On Wed, 25 Jun 2008, Peter St. John wrote: > RGB, > My hypothetical future Recursive Genetic Algorithm Go Player will crush your > Hypothetical Future NeuralNet Go Player! Will not! Hypothetically, of course...;-) rgb > Peter > > On 6/24/08, Robert G. Brown wrote: >> >> On Tue, 24 Jun 2008, Peter St. John wrote: >> >> On a finite board, the game eventually becomes local; in fact "contact >>> plays" are common after about the first dozen moves, but they are >>> inescapable later. But the possibilities are horrible, the numbers are >>> huge, >>> for tree-searching. The new thing is monte carlo; to consider a move, the >>> machine actually plays that position against itself a few hundred times >>> (with a trivial, not recursive, algorithm) and weighs the move by the >>> percentage of outcomes. It's weird, to me. >>> >> >> It makes sense to me. It's in some sense more like humans play. In >> fact, if one can come up with any (even very simple) "local" rules to >> semi-prune the tree, you can imagine an importance-sampling monte carlo >> algorithm. >> >> However, if I were going to try, it would be genetic algorithm all the >> way. Don't explore trees exhaustively; too many of them. Don't sample >> them (especially if you can't create a canonical weight to do importance >> sampling); too many of them. Breed them. There are still too many, but >> again, if one can generate ANY sort of fitness function, maybe one can >> get the N^3 and exponential growth of favorable possibilities that are >> like "making a decision". >> >> Fortunately, I'm not going to try. Neural nets are much more fun. And >> MIGHT be able to manage the complexity -- especially one built with a >> GA...;-) >> >> rgb >> >> >>> Peter >>> >>> On Tue, Jun 24, 2008 at 9:22 PM, Robert G. Brown >>> wrote: >>> >>> On Tue, 24 Jun 2008, Peter St. John wrote: >>>> >>>> Programming a computer to play Go (an Asian strategy boardgame) has been >>>> >>>>> difficult; some people say it's proof that Go is better or harder than >>>>> chess, since computers can beat masters at chess but struggle at Go. (I >>>>> think that statistically a game of go is about equivalent to a two-game >>>>> match of chess; both games empty your brain quickly of course). My view >>>>> is >>>>> that while go may be somewhat harder to reduce to tree-searching, the >>>>> main >>>>> advantage of computer chess was an early start, e.g. von Neumann. >>>>> >>>>> >>>> I thought that Go was combinatorially vastly, vastly more difficult than >>>> chess. Both because of the extremely large number of move permutations >>>> that make it very difficult to follow trees and because Go is a >>>> fundamentally nonlocal game -- one can imagine a "perfect" Go strategy >>>> in which pieces are never played close to each other as the board >>>> gradually fills in with higher and higher still disconnected density, >>>> culminating in the winner completely unravelling the loser or >>>> precipitating out to win by a single small amount. Yet nobody can even >>>> come close to playing that way -- most games start out that way for a >>>> while and then transform into local dogfights that are STILL often >>>> resolved by long range connections. >>>> >>>> This article: >>>> >>>>> http://www.usgo.org/resources/downloads/CogApdx%20II-2.pdf >>>>> describes recent trends in computer Go and mentions a 32-node cluster, 8 >>>>> cores per node. Apparently MPI parallelization is recent for them and >>>>> they >>>>> are making good progress. >>>>> >>>>> >>>> Interesting. I'll have to look. Last time I read up on this (in the >>>> context of a conversation with you, IIRC:-) nobody could make a computer >>>> go player that could beat even a very bad human player, where computers >>>> back to maybe 386's have been able to play chess well enough to win at >>>> least sometimes against weak humans (and win "often" against weak human >>>> players by now). I suck at chess (but can still beat a lot of the >>>> people I -- rarely -- play) and modern computers can beat me pretty >>>> easily. I suck at Go, too -- but I can't even find a BAD go player to >>>> run on a PC to try to get better. >>>> >>>> rgb >>>> >>>> >>>> >>>> Peter >>>>> >>>>> The game Go: http://en.wikipedia.org/wiki/Go_%28game%29 >>>>> AGA (American Go Association): http://www.usgo.org >>>>> >>>>> >>>>> -- >>>> Robert G. Brown Phone(cell): 1-919-280-8443 >>>> Duke University Physics Dept, Box 90305 >>>> Durham, N.C. 27708-0305 >>>> Web: http://www.phy.duke.edu/~rgb >>>> Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php< >>>> http://www.phy.duke.edu/%7Ergb/Lilith/Lilith.php> >>>> Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 >>>> >>>> >>> >> -- >> Robert G. Brown Phone(cell): 1-919-280-8443 >> Duke University Physics Dept, Box 90305 >> Durham, N.C. 27708-0305 >> Web: http://www.phy.duke.edu/~rgb >> Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php >> Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 >> > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From kilian at stanford.edu Wed Jun 25 10:36:09 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> Message-ID: <200806251036.10244.kilian@stanford.edu> [Wow, this thread is really out of control. Nukes, geopolitics, stunts, and now biofuels. And all this because of CUDA! :)] > -----Original Message----- > From: beowulf-bounces@beowulf.org > [mailto:beowulf-bounces@beowulf.org] On Behalf Of Vincent Diepeveen > This causes as we speak people dying as they can no longer for a cent > or so buy food made out of it; the prices have doubled if not more for > such types of cheap food because of subsidy in the 1st world countries > for this. Yup. See http://en.wikipedia.org/wiki/Food_vs_fuel In that sense, we're lucky we've escaped the water-powered car (yet). It always scared me to death to imagine that such a vital element for life could be used as a fuel to propel cars, and be subject to all kinds of blind speculation. > I assume EU will take measures to turn back those subsidies on bio > fuels that get produced out of food that feeds billions, who now > hardly can afford to buy food anymore as > it gets burned for energy in first world countries, whereas > commercially spoken it cannot get burned, > it is just because of subsidy it can exist. Well, yeah, even without mentioning biofuels' efficiency and their energy balance. There are numerous articles stating that it requires more energy to produce biofuels than it produces. > When i just walked previous week into a shop and my sister was > interested in a new washing machine, > i pointed her to the fact that the thing she was interested in, was > eating 3.8 kW, versus the 100 euro more expensive > thing next to it was eating 1.14 kW. It is something that only very > few will notice. Don't you get those? http://en.wikipedia.org/wiki/European_Union_energy_label They are mandatorily displayed under appliances on shelves, in France, and people are paying more and more attention. On Wednesday 25 June 2008 07:50:42 am Geoff Galitz wrote: > I've never really bought the argument that biofuels are causing a > food shortage considering that there is still so much unused farmland > in the US and farming practices here in the EU. """ Worldwide production of vegetable oil and animal fat is not yet sufficient to replace liquid fossil fuel use. Furthermore, some object to the vast amount of farming and the resulting fertilization, pesticide use, and land use conversion that would be needed to produce the additional vegetable oil. The estimated transportation diesel fuel and home heating oil used in the United States is about 160 million tonnes (350 billion pounds) according to the Energy Information Administration, US Department of Energy - [38]. [...] If the entire arable land area of the USA (470 million acres, or 1.9 million square kilometers) were devoted to biodiesel production from soy, this would just about provide the 160 million tonnes required (assuming an optimistic 98 gpa of biodiesel). """ http://en.wikipedia.org/wiki/Biodiesel > I must admit this > out of my field so I have no real evidence to support my suspicion, > but there have been a trickle of articles from publications like Der > Spiegel and the NY Times which indicate there is some suspicion these > shortages have just as much to do with commodity speculation and > manipulation. I do also believe that those prices rises are mainly due to speculation, as oil price increases are, and that supply and demand regulations have been long forgotten. But would anybody bother speculating to that scale on the food markets if it wasn't for the biofuels, and the potential benefits one can foresee in this fossil oil shortage context? Cheers, -- Kilian From kilian at stanford.edu Wed Jun 25 10:57:59 2008 From: kilian at stanford.edu (Kilian CAVALOTTI) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> Message-ID: <200806251057.59304.kilian@stanford.edu> On Wednesday 25 June 2008 10:09:28 am Robert G. Brown wrote: > (lower power means lower SOMETHING -- less agitation, less heating of > the water, smaller capacity so you have to run more loads) Or just better energy conversion, less energy wasted in a transformer, use of higher-grade components (hence the higher price tag)... This kind of thing can also explain the differences. > And it is equally easy to produce an "energy saving" product that > saves energy by not working as well, or at all. I'm not really > arguing, by the way -- only pointing out that you should look beyond > JUST the energy RATE of draw and make sure that the product will > actually perform just as well Can you actually "try" washing machines in a store? :) > and still save total energy and do what > you need it to do. It isn't always easy to do that, and it goes > beyond just looking at the power consumption rate. Lower power could > translate into "wearing slightly dirty clothes all the time" or it > could translate into "getting a load that is just as large of just as > many clothes just as clean -- but saving 2.5 KW every hour you use > it." That's precisely what Energy Labels are for: """ For washing machines the energy efficiency scale is calculated using a cotton cycle at 60?C (140?F) with a maximum declared load. This load is typically 6 kg. The energy efficiency index is in kW?h per kilogramme of washing. """ http://en.wikipedia.org/wiki/European_Union_energy_label > If it IS heating water, then you need to be very careful. It takes a > certain number of joules to heat water, period. Yes, but again, to produce that certain amount of heat, it may require more or less electric energy depending on the efficiency of the conversion. Although I assume most components probably do about the same, ball park. I would think efficiency is more a determinant factor in motors, for instance. Cheers, -- Kilian From rgb at phy.duke.edu Wed Jun 25 11:34:21 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <200806251057.59304.kilian@stanford.edu> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> <200806251057.59304.kilian@stanford.edu> Message-ID: On Wed, 25 Jun 2008, Kilian CAVALOTTI wrote: > For washing machines the energy efficiency scale is calculated using a > cotton cycle at 60?C (140?F) with a maximum declared load. This load is > typically 6 kg. The energy efficiency index is in kW?h per kilogramme > of washing. > """ > http://en.wikipedia.org/wiki/European_Union_energy_label Yeah, I saw your post. Vincent left off the "hour" from kW-hours. That makes more sense. >> If it IS heating water, then you need to be very careful. It takes a >> certain number of joules to heat water, period. > > Yes, but again, to produce that certain amount of heat, it may require > more or less electric energy depending on the efficiency of the > conversion. Although I assume most components probably do about the > same, ball park. I would think efficiency is more a determinant factor > in motors, for instance. All energy conversion for heating water with electricity is joule heating, almost certainly. So it is one to one -- one joule of energy as electricity to add one joule of heat to the water. Efficiency variation only associated with variations in e.g. insulation of the reservoir. The motors might have differential efficiency, but a factor of three is a LOT. That a suggests completely different process. rgb > > Cheers, > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From prentice at ias.edu Wed Jun 25 11:39:49 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" still OT In-Reply-To: <48626C96.8060305@vcu.edu> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org><87skv8wxlg.fsf@snark.cb.piermont.com><20080620143621.GA31402@synapse.neuralscape.com><485BC8C6.4080800@vcu.edu><5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com><485BF505.6000904@ias.edu><7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com><20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> <4B831460-3895-433D-A399-8243B3E6C069@xs4all.nl> <48626C96.8060305@vcu.edu> Message-ID: <486290F5.70707@ias.edu> > Vincent Diepeveen wrote: >> >> Some 3d world country managers are begging to adress this issue: "My >> nations people die, >> as your bio fuel raises our food prices, the poor are so poor here, >> they use that stuff as food >> and cannot afford it now". All this discussion of politics is completely off topic for this list, but with all this talk about food shortages, I feel I need to state the obvious facts that everyone overlooks. I'm sure I'll get flamed for this, but I don't care: The United States alone produces enough grain to feed the entire world. This was true when I was taught this in 3rd grade, and with modern technology (genetic engineering, automated farm equipment, etc.), that is even more true today. People are starving in other parts of the world because of political or economic factors. There have been multiple cases of the US or UN providing food to starving countries only to have the tyrannical dictator hoard the food for his ruling class and not distribute it to the general population. And then there's Myanmar... The government refused aid, and watched as its people died, rather than accept help. Those are political examples. For economics, consider foreign governments not bothering to provide aid to starving countries b/c that country has nothing to offer in return. Watch the movie "Hotel Rwanda". I guarantee you that if Rwanda had exploitable oil reserves, the US would have intervened almost immediately. What happened there was *MUCH* worse than what the Iraqis suffered under Hussein. The difference? Iraq has oil, and Rwanda does not. To reitierate: The US alone can produce more than enough grain to feed the whole world. If people are starving, it's because of politics and economics, not an inability of the world's farmer to grow enough crops. All this talk about biofuels causing people to starve is a load of crap. I won't even speculate with BS like this originates. >> >> USA nor Europe can *never* produce that stuff as cheap as 3d world >> countries can. >> Vincent, you are completely off-base here. The US (and the rest of the modern world) can produce crops cheaper than anyone else. Modern machinery, which 3rd world farmers don't have, reduce labor costs drastically and can run around the clock. While human labor might be significantly cheaper somewhere else, what 2 US farmers can accomplish with modern machinery would take hundreds (thousands?) of manual laborers to produce elsewhere. For example, John Deere make a tractor that runs on GPS. A farmer can program it, and then let it do it's thing. Since it runs by GPS, it runs itself 24 hours a day, doesn't miss any of the crop, nor overlap, for maximum efficiency. Someone justs has to make sure it doesn't run out of gas. That's much easier than hundreds of laborers swinging scyths somewhere in Africa or Asia, and I bet it's much cheaper per pound of grain produced per unit of time. -- Prentice From peter.st.john at gmail.com Wed Jun 25 13:17:46 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] FYI: HPC Server 2008 hits top 25 In-Reply-To: References: <20080624154540.GR9875@leitl.org> Message-ID: Matt, First, I don't understand how a "technoronin" can have a boss :-) I only saw OS360 through Brook's Mythical Man Month; by the time I did fortran on big iron it was VM/CMS. But if you have the hardware drawing power already, then sure, cluster 'em :-) Peter On 6/24/08, Matt Lawrence wrote: > > On Tue, 24 Jun 2008, Peter St. John wrote: > > MS isn't famous for new development (as scientists see "new development") >> but a few hundred engineers would not be such a big investment for them. >> >> I'd be more interested in a comparison of that many whatever, quad core >> xeon? running linux vs MS's running Cluster Server 2008. CP/M could be a >> pretty powerful OS if it ran on enough nodes :-) and of course I think of >> XP >> as CP/M v.99 (approximately) (although that isn't fair to VMS, the >> forebear >> of NT) >> > > I'm sorely tempted to try to run a cluster of OS/360 systems just to say it > has been done. Should be pretty easy using Hercules/390. > > Since my boss reads this list, I'm not sure if he will be amused or > appalled at the idea. probably both. > > -- Matt > It's not what I know that counts. > It's what I can remember in time to use. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080625/a17331a5/attachment.html From carlossegurag at gmail.com Wed Jun 25 05:34:24 2008 From: carlossegurag at gmail.com (Carlos Segura) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <175463834.73461214393982940.JavaMail.root@zimbra.vpac.org> References: <1843262915.73441214393713442.JavaMail.root@zimbra.vpac.org> <175463834.73461214393982940.JavaMail.root@zimbra.vpac.org> Message-ID: >In my experience, real applications as well, things like nwchem for instance. Granted the difference (at least in the past) was much >larger for g77 vs commercial fortran than it was for gcc/g++ vs commercial c compilers. I've >heard gfortran has gotten much better and I've even occasionally seen SPEC >numbers for GNU compilers which looked encouraging. In any case I've often seen commercial compilers justified on a pure cost/benefit >basis. I.e. $1k on compilers gets me more performance than another $1k on hardware. Some benchmarks about fortran compilers can be found on http://www.polyhedron.com/pb05-linux-f90bench_p40html The intel compiler have very good results. I also think that in the case of fortran compilers there is more difference that in the case of C compilers, although I haven`t seen comparissons of C compilers lately. Carlos 2008/6/25 Chris Samuel : > > ----- "Bill Broadley" wrote: > > > Maybe I misread the license, what exactly let you > > to believe that it's "free for researchers"? > > The Intel website is pretty clear on the matter and > I'm pretty sure this is the case since they began their > free non-commercial license program. > > # Q. I am engaged in research projects. Can I qualify to > # use the noncommercial product? > # > # A. If you, as an individual, are receiving any form of > # compensation for the research project (i.e., you receive > # a salary, or funding, etc.) you do not qualify for a > # non-commercial license. > > > http://www.intel.com/cd/software/products/asmo-na/eng/download/download/219771.htm > > # This offering is provided to developers who are > # developing software on their own time without > # compensation. > > [...] > > # Note that academic use of the products does not qualify > # for a non-commercial license. Intel offers heavily > # discounted licenses to academic developers through > # our Academic Developer Program. > > -- > Christopher Samuel - (03) 9925 4751 - Systems Manager > The Victorian Partnership for Advanced Computing > P.O. Box 201, Carlton South, VIC 3053, Australia > VPAC is a not-for-profit Registered Research Agency > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080625/3caaf2c2/attachment.html From kspaans at student.math.uwaterloo.ca Wed Jun 25 05:41:25 2008 From: kspaans at student.math.uwaterloo.ca (Kyle Spaans) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Ottawa Linux Symposium, anyone going? Message-ID: <20080625124125.GA14641@student.math> Just out of curiosity, is anyone else from this list going to be attending OLS? Which talks would you say are relevant to beowulfery? From alsimao at gmail.com Wed Jun 25 07:49:54 2008 From: alsimao at gmail.com (Alcides Simao) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] A simple cluster Message-ID: <7be8c36b0806250749t3fc55b4fpaa589bedc3d21f5c@mail.gmail.com> Hi all! I am thinking of build a small cluster, fit for my calculations ( I am a spectroscopist, so I often do QM calculations in some molecules, which can be quite intensive for a singles machine) I was thinking of trying to build a cluster initially using 'scrap' pc's, just to get acquainted to the methodology and for testing purposes. I have some old crates lying around that I can use. I would like to know how can I assembly such equipment, and how to configurate it! Best, Alcides -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080625/d67af831/attachment.html From saville at comcast.net Wed Jun 25 10:31:13 2008 From: saville at comcast.net (saville@comcast.net) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" Message-ID: <062520081731.12441.486280E10007F0080000309922069997350A040407900E9C@comcast.net> I respectfully request that you take conversations about washing machines and other non_beowulf related topics off to some other mailing list. I have plenty of email to delete without having the load increased by irrelevant discussions on this one. Many thanks, Gregg -- Audentes Fortuna Juvant -------------- Original message ---------------------- From: "Robert G. Brown" > On Wed, 25 Jun 2008, Vincent Diepeveen wrote: > > > When i just walked previous week into a shop and my sister was interested in > > a new washing machine, > > i pointed her to the fact that the thing she was interested in, was eating > > 3.8 kW, versus the 100 euro more expensive > > thing next to it was eating 1.14 kW. It is something that only very few will > > notice. From peter.st.john at gmail.com Wed Jun 25 16:32:10 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <4B831460-3895-433D-A399-8243B3E6C069@xs4all.nl> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <5C489EB5-185D-4DEA-9DE5-66ADB5EABE4E@sicortex.com> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> <4B831460-3895-433D-A399-8243B3E6C069@xs4all.nl> Message-ID: I just wanted to mention that there is a plan afoot to make alchohol (fuel) from cellulose (e.g. grass, I think the plan is corn stalks) along the lines of http://en.wikipedia.org/wiki/Cellulose#Cellulose_source_and_energy_crops I read somewhere a claim to be immently commercially viable (with corn stalks) but I don't recall that specific link. Of course, even corn stalks have food value, when they are ploughed back into the land after harvesting, preparing the land for next season. So it's not as simple as growing food and fuel together with linear efficiency. But it's a positive approach, as presumably merely ploughing under cornstalks is not the most efficient form of fertilization. Also, grass grows in places, such as hillsides, that are awkward for food. Sheep are not the most efficient form of food per acre (but they take advantage of underutilized hillsides). It's not an easy computation. Peter On 6/25/08, Vincent Diepeveen wrote: > > It is not about what we do in the EU and/or what happens in USA. > > That is the typical short sighted view of politicians who decided on the > subsidy. > > What matters is that the 'biofuel' of course does get imported from > South America and Africa and that it causes in THOSE countries a huge > inflation of food. > > Some 3d world country managers are begging to adress this issue: "My > nations people die, > as your bio fuel raises our food prices, the poor are so poor here, they > use that stuff as food > and cannot afford it now". > > USA nor Europe can *never* produce that stuff as cheap as 3d world > countries can. > > Agriculture in USA and Europe only exists because of protectionism (without > me taking a viewpoint > on protectionistic measures). > > For biofuels there is always manners to import all those different types of > food, creating at its > LOCAL market a huge food shortage and inflation of food prices for the > poor, a lot of whom die > as a result. > > Not a single EU rule will be able to avoid that it gets imported and later > on somehow subsidy on > bio-fuel causes it to get burned for energy, whereas it is of course a > pathetic expensive form of > energy. There is always a manner to get the stuff into the country. For > example you want to produce > product X of it. By accident after import production line of X gets left > with 95% waste product, > which then you burn for bio-fuel. > > So food of the poor ends up as biofuel, good for only a very tiny % of all > energy produced and "by accident" > always a smaller % than the % of energy that gets heavily subsidized for > being bio-fuel! > > Yet if you deduce macro-economically, it is LOGICAL that the stuff that > gets used to produce the cheapest > food for the poor, also has a price low enough to get cheaply imported from > those 3d world countries to > our nation, to get burned as bio-fuel. > > The poorest of the poorest can afford that stuff only because it is the > cheapest thing on planet earth, > and we are taking it away from them just for some stupid subsidized > bio-energy! > > Vincent > > On Jun 25, 2008, at 4:50 PM, Geoff Galitz wrote: > > >> >> I've never really bought the argument that biofuels are causing a food >> shortage considering that there is still so much unused farmland in the US >> and farming practices here in the EU. I must admit this out of my field >> so >> I have no real evidence to support my suspicion, but there have been a >> trickle of articles from publications like Der Spiegel and the NY Times >> which indicate there is some suspicion these shortages have just as much >> to >> do with commodity speculation and manipulation. >> >> >> http://www.iht.com/articles/2008/06/12/business/speculate.php >> http://www.spiegel.de/international/world/0,1518,559550,00.html (in >> English) >> >> >> >> Geoff Galitz >> Blankenheim NRW, Deutschland >> http://www.galitz.org >> >> >> -----Original Message----- >> From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On >> Behalf Of Vincent Diepeveen >> Sent: Mittwoch, 25. Juni 2008 16:07 >> To: Jon Aquilina >> Cc: Beowulf Mailing List; Mark Hahn >> Subject: Re: [Beowulf] Re: "hobbyists" >> >> Even worse, >> Why is there subsidy on bio fuels that get produced out of food eaten >> by poor people? >> >> This causes as we speak people dying as they can no longer for a cent >> or so buy food made out >> of it; the prices have doubled if not more for such types of cheap >> food because of subsidy in the >> 1st world countries for this. >> >> I assume EU will take measures to turn back those subsidies on bio fuels >> that get produced out of food that feeds billions, who now hardly can >> afford to buy food anymore as >> it gets burned for energy in first world countries, whereas >> commercially spoken it cannot get burned, >> it is just because of subsidy it can exist. >> >> Even better i would be in favour of a ban on bio fuels that are >> outright food products in 3d world countries. >> >> When i just walked previous week into a shop and my sister was >> interested in a new washing machine, >> i pointed her to the fact that the thing she was interested in, was >> eating 3.8 kW, versus the 100 euro more expensive >> thing next to it was eating 1.14 kW. It is something that only very >> few will notice. >> >> It is easy to cheaply produce equipment that eats more power than >> equipment of competitors, that is the fundamental problem. >> >> Vincent - speaking for himself >> >> On Jun 25, 2008, at 8:43 AM, Jon Aquilina wrote: >> >> how much does a sugar glass window cost now that sugar and other >>> things are being used for bio fuels? >>> >>> On Wed, Jun 25, 2008 at 12:20 AM, Mark Hahn wrote: >>> >>> More specifically for HPC, linux seems designed for the desktop, and >>> for small memory machines. >>> >>> the only justice I can see in that is that there hasn't been all >>> that much effort to get bigpages widely/easily used. in >>> particular, I don't >>> see that scheduler or general memory-management issues in linux are >>> particularly biased for desktop or against HPC. >>> >>> >>> That's funny, because I've heard people get scared that it was the >>> complete >>> opposite. That Linux was driven by Big Iron, and that no one cared >>> about >>> the "little desktop guy" (Con Kolivas is an interesting history >>> example). >>> >>> Con didn't play the game right - you have to have the right >>> combination of social engineering (especially timing and proactive >>> response) and good tech >>> kungfoo. kernel people are biased towards a certain aesthetic that >>> doesn't >>> punish big-picture redesigns from scratch, but _does_ punish >>> solutions in search of a problem. >>> >>> so the question is, if you had a magic wand, what would you change >>> in the kernel (or perhaps libc or other support libs, etc)? most >>> of the things I can think of are not clear-cut. I'd like to be >>> able to give better info from perf counters to our users (but I >>> don't think Linux is really in the way). I suspect we lose some >>> performance due to jitter >>> injected by the OS (and/or our own monitoring) and would like to >>> improve, >>> but again, it's hard to blame Linux. I'd love to have better >>> options for cluster-aware filesystems. kernel-assisted network >>> shared memory? >>> >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >>> >>> >>> >>> -- >>> Jonathan Aquilina >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >>> >> >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> >> > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080625/63052f2a/attachment.html From peter.st.john at gmail.com Wed Jun 25 16:38:42 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" still OT In-Reply-To: <486290F5.70707@ias.edu> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> <4B831460-3895-433D-A399-8243B3E6C069@xs4all.nl> <48626C96.8060305@vcu.edu> <486290F5.70707@ias.edu> Message-ID: Prentice, Yes. You may get flamed, but the US (and Canada) produce remarkable food per acre and per man-hour, compared to anyone else. Technology, the heavy industry and infrastructure for the tech to proliferate, fertilizer, irrigation, and a continent worth of arable land. Our food is actually pretty cheap. One guy breaking his back for an acre in the third world works the acre cheaper than we can, but he does not produce as much food per acre as we do, so he can't feed his neighbors. But to do this in the third world takes not just the technology, but the infrastructure to support it, and social changes. In a society where more food directly means more babies (because if more babies, there is a higher chance one of them will live long enough to take care of me in my old age), there can never be enough food. Alot has to change, socially, politically, technologically, industrially. Plus not everyone has as much arable land as in North America. Europe has all the tech but is older, more crowded, less arable land per capita. Everything is complicated. Peter On 6/25/08, Prentice Bisbal wrote: > > > Vincent Diepeveen wrote: > >> > >> Some 3d world country managers are begging to adress this issue: "My > >> nations people die, > >> as your bio fuel raises our food prices, the poor are so poor here, > >> they use that stuff as food > >> and cannot afford it now". > > > > All this discussion of politics is completely off topic for this list, > but with all this talk about food shortages, I feel I need to state the > obvious facts that everyone overlooks. I'm sure I'll get flamed for > this, but I don't care: > > The United States alone produces enough grain to feed the entire world. > This was true when I was taught this in 3rd grade, and with modern > technology (genetic engineering, automated farm equipment, etc.), that > is even more true today. > > People are starving in other parts of the world because of political or > economic factors. There have been multiple cases of the US or UN > providing food to starving countries only to have the tyrannical > dictator hoard the food for his ruling class and not distribute it to > the general population. > > And then there's Myanmar... The government refused aid, and watched as > its people died, rather than accept help. > > Those are political examples. For economics, consider foreign > governments not bothering to provide aid to starving countries b/c that > country has nothing to offer in return. > > Watch the movie "Hotel Rwanda". I guarantee you that if Rwanda had > exploitable oil reserves, the US would have intervened almost > immediately. What happened there was *MUCH* worse than what the Iraqis > suffered under Hussein. The difference? Iraq has oil, and Rwanda does not. > > To reitierate: The US alone can produce more than enough grain to feed > the whole world. If people are starving, it's because of politics and > economics, not an inability of the world's farmer to grow enough crops. > > All this talk about biofuels causing people to starve is a load of crap. > I won't even speculate with BS like this originates. > > > >> > >> USA nor Europe can *never* produce that stuff as cheap as 3d world > >> countries can. > >> > > > Vincent, you are completely off-base here. The US (and the rest of the > modern world) can produce crops cheaper than anyone else. Modern > machinery, which 3rd world farmers don't have, reduce labor costs > drastically and can run around the clock. While human labor might be > significantly cheaper somewhere else, what 2 US farmers can accomplish > with modern machinery would take hundreds (thousands?) of manual > laborers to produce elsewhere. > > For example, John Deere make a tractor that runs on GPS. A farmer can > program it, and then let it do it's thing. Since it runs by GPS, it runs > itself 24 hours a day, doesn't miss any of the crop, nor overlap, for > maximum efficiency. Someone justs has to make sure it doesn't run out of > gas. That's much easier than hundreds of laborers swinging scyths > somewhere in Africa or Asia, and I bet it's much cheaper per pound of > grain produced per unit of time. > > > -- > > Prentice > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080625/bebd497c/attachment.html From peter.st.john at gmail.com Wed Jun 25 16:42:31 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Ottawa Linux Symposium, anyone going? In-Reply-To: <20080625124125.GA14641@student.math> References: <20080625124125.GA14641@student.math> Message-ID: Kyle, You mean the Ottawa Linux Symposium? You have a link for them? Peter On 6/25/08, Kyle Spaans wrote: > > Just out of curiosity, is anyone else from this list going to be attending > OLS? > Which talks would you say are relevant to beowulfery? > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080625/a96f15d4/attachment.html From peter.st.john at gmail.com Wed Jun 25 16:46:56 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] A simple cluster In-Reply-To: <7be8c36b0806250749t3fc55b4fpaa589bedc3d21f5c@mail.gmail.com> References: <7be8c36b0806250749t3fc55b4fpaa589bedc3d21f5c@mail.gmail.com> Message-ID: Alcides, I think a short answer is: get a switch, plug all the boxes (throught the ethernet ports on the motherboards) to the switch, install Ubuntu and use OpenMP. Longer answers will be forthcoming, but I bet they will start with questions about the specific hardware you have, the budget, whether your applications are latency or compute time dependent, how parallelizable it is, shared memory model... Peter On 6/25/08, Alcides Simao wrote: > > Hi all! > > I am thinking of build a small cluster, fit for my calculations ( I am a > spectroscopist, so I often do QM calculations in some molecules, which can > be quite intensive for a singles machine) > > I was thinking of trying to build a cluster initially using 'scrap' pc's, > just to get acquainted to the methodology and for testing purposes. I have > some old crates lying around that I can use. > > I would like to know how can I assembly such equipment, and how to > configurate it! > > Best, > > Alcides > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080625/0f6f3631/attachment.html From gerry.creager at tamu.edu Wed Jun 25 18:49:34 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] FYI: HPC Server 2008 hits top 25 In-Reply-To: References: <20080624154540.GR9875@leitl.org> Message-ID: <4862F5AE.1020906@tamu.edu> His boss wrote fortran on OS/360 (and liked it!). WatFOR, WatFIV, IBM FORT-G and FORT-H, with intermediate step manual optimization. atmospheric models and radar propagation models, plus a few system hacks (how high can the paper arch coming out of the line printer at full-rate pate-eject?). I doubt we could convince someone to free up enough systems and time to load Hercules up and run that way but it might be fun. In an odd, perverted sorta way. Peter St. John wrote: > Matt, > First, I don't understand how a "technoronin" can have a boss :-) > I only saw OS360 through Brook's Mythical Man Month; by the time I did > fortran on big iron it was VM/CMS. But if you have the hardware drawing > power already, then sure, cluster 'em :-) > Peter > > On 6/24/08, *Matt Lawrence* > wrote: > > On Tue, 24 Jun 2008, Peter St. John wrote: > > MS isn't famous for new development (as scientists see "new > development") > but a few hundred engineers would not be such a big investment > for them. > > I'd be more interested in a comparison of that many whatever, > quad core > xeon? running linux vs MS's running Cluster Server 2008. CP/M > could be a > pretty powerful OS if it ran on enough nodes :-) and of course I > think of XP > as CP/M v.99 (approximately) (although that isn't fair to VMS, > the forebear > of NT) > > > I'm sorely tempted to try to run a cluster of OS/360 systems just to > say it has been done. Should be pretty easy using Hercules/390. > > Since my boss reads this list, I'm not sure if he will be amused or > appalled at the idea. probably both. > > -- Matt > It's not what I know that counts. > It's what I can remember in time to use. > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.862.3982 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From james.p.lux at jpl.nasa.gov Wed Jun 25 20:41:40 2008 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] FYI: HPC Server 2008 hits top 25 In-Reply-To: <4862F5AE.1020906@tamu.edu> References: <20080624154540.GR9875@leitl.org> <4862F5AE.1020906@tamu.edu> Message-ID: <20080625204140.4xtkdwtgyskc8kw0@webmail.jpl.nasa.gov> Quoting Gerry Creager , on Wed 25 Jun 2008 06:49:34 PM PDT: > His boss wrote fortran on OS/360 (and liked it!). WatFOR, WatFIV, IBM > FORT-G and FORT-H, with intermediate step manual optimization. > atmospheric models and radar propagation models, plus a few system > hacks (how high can the paper arch coming out of the line printer at > full-rate pate-eject?). no, no, no.. it's two other printer games... if you overprint a line of underscores, can you get the paper to break (royally POing the operators) or, what sequence of characters if printed cause all the hammers to hit simultaneously on the chain making the printer move around. Actually, there's a great pleasure in coding for a mainframe of that era.. you really own the machine... Jim From csamuel at vpac.org Wed Jun 25 20:43:15 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" still OT In-Reply-To: <486290F5.70707@ias.edu> Message-ID: <1953527533.79861214451795299.JavaMail.root@zimbra.vpac.org> ----- "Prentice Bisbal" wrote: > The United States alone produces enough grain to feed the entire > world. It is probably worth pointing out that, as a recent New Scientist article mentioned, a major part for the rise in grain prices is due the rising demand for meat from around the world. This is, of course, a very inefficient conversion method of solar power into human food. There is also the fact that yield growth in food is rising at a slower rate than population, which is largely the result of the developing world cutting back on agri research. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From john.hearns at streamline-computing.com Wed Jun 25 21:28:37 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" still OT In-Reply-To: <1953527533.79861214451795299.JavaMail.root@zimbra.vpac.org> References: <1953527533.79861214451795299.JavaMail.root@zimbra.vpac.org> Message-ID: <1214454527.4979.35.camel@Vigor13> On Thu, 2008-06-26 at 13:43 +1000, Chris Samuel wrote: > It is probably worth pointing out that, as a recent > New Scientist article mentioned, a major part for the > rise in grain prices is due the rising demand for meat > from around the world. > > This is, of course, a very inefficient conversion method > of solar power into human food. George Monbiot agrees with you. http://www.monbiot.com/archives/2008/04/15/the-pleasures-of-the-flesh/ "If you care about hunger, eat less meat." There. I've now outed myself as a Guardian reader. (The Manchester Guardian - daily newspaper in the UK beloved of pinkos, geography teachers and social workers. I would make a droll comment about bearded, sandal wearing cyclists but would get beaten about the head with a rolled up Guardian at my next LUG meeting). From eagles051387 at gmail.com Wed Jun 25 23:34:03 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" still OT In-Reply-To: <48626C96.8060305@vcu.edu> References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> <4B831460-3895-433D-A399-8243B3E6C069@xs4all.nl> <48626C96.8060305@vcu.edu> Message-ID: taking this thread off on another tangent here though. using bio fules might be good for now but is actually creating lots of problems. the end all solution would to be to use hydrogen as the fuel source. put water in the car gets broken down through hydrolysis and the water which is exhaust is recycled back into the system. no need for fueling stations because a convenient source of water would be your outside your door On Wed, Jun 25, 2008 at 6:04 PM, Mike Davis wrote: > Vincent Diepeveen wrote: > >> >> Some 3d world country managers are begging to adress this issue: "My >> nations people die, >> as your bio fuel raises our food prices, the poor are so poor here, they >> use that stuff as food >> and cannot afford it now". >> >> USA nor Europe can *never* produce that stuff as cheap as 3d world >> countries can. >> >> Since my uncle held a patent on an Alcohol Fuel still, and my grandfather > grew corn, I have a somewhat different view. I'm not sure where your > information is coming from. According to http://www.earth-policy.org/, the > US produced 414 Million tons of grain last year. Of that total 81 Million > tons were used for fuel and 106 Million tons were exported. While it might > be less expensive to produce grain in the 3rd world, the top exporter of > Corn, Rice and Wheat for the world is the US. > http://www.earth-policy.org/Updates/2008/Update69_data.htm#table10 > > > http://www.earth-policy.org/Updates/2008/Update69_data.htm#table9 > > Now since corn prices have been rather stagnant until recently for almost > 30 years and since I know farmers that have burned corn for heat rather than > sell at a loss. I see the increase in grain prices as neutral. Yes, there > are bad results for the third world. But those farmers growing grain deserve > a fair price for the work and uncertainty of agriculture. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080626/79d6d261/attachment.html From rgb at phy.duke.edu Thu Jun 26 00:30:17 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Re: "hobbyists" still OT In-Reply-To: References: <2031586795.35641213922639112.JavaMail.root@zimbra.vpac.org> <485BF505.6000904@ias.edu> <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> <4B831460-3895-433D-A399-8243B3E6C069@xs4all.nl> <48626C96.8060305@vcu.edu> Message-ID: On Thu, 26 Jun 2008, Jon Aquilina wrote: > taking this thread off on another tangent here though. using bio fules might > be good for now but is actually creating lots of problems. the end all > solution would to be to use hydrogen as the fuel source. put water in the > car gets broken down through hydrolysis and the water which is exhaust is > recycled back into the system. no need for fueling stations because a > convenient source of water would be your outside your door I'm trying to let the thread die because it is way OT and because I'm packing to leave for my "teach at the beach" summer. But note well -- the fundamental issue isn't really "burn this" or "burn that" sythetic fuel (defined quite loosely as a fuel that is manufactured or assembled by humans out of parts that are not themselves fuel, e.g. hydrogen (fuel) produced out of water. The issue is what is the source of free energy that will be used to do the synthesis, put the energy INTO the fuel. It takes energy to electrolyze water. It takes energy to grow corn and turn it into alcohol. It takes energy to grow oilseed crops and turn them into biodiesel. The free energy source might be the sun, it might be fossil fuels (indirectly the sun), it might be plants (the sun), wind (the sun), hydroelectric (the sun) -- hmmm, with the exception of geothermal and nuclear, looks like nearly everything is the sun. Stored sunlight is in finite supply in fossil fuels. Plants grow slowly and require more than JUST sunlight to produce energy, so a fair bit of energy yield has to be turned right back into fertilizer, fuel for tractors or oxen, humans to grown and transport it, the refining or extraction process. Using THIS fuel to produce hydrogen would simply layer inefficiency on inefficiency, as one would throw maybe 70% of it away in the process, and then throw a fair bit of what remained away in the final stage of energy release as work. The laws of thermodynamics cannot be broken, not really. This suggests that in the long run, the only free energy sources likely to be able to keep up with human demand are the sun, implemented as DIRECT conversion through solar cells, concentrators, turbines, and perhaps thermonuclear fusion (not fission). The deuterium or helium supply of the planet is relatively inexhaustible compared to other fossil fuels, and more could be mined from other planets in the solar system if need be. So sure, hydrogen is great, once you have the free energy source. But FIRST you need that, as hydrogen doesn't WANT to come out of water molecules. You have to make it come off. rgb > > On Wed, Jun 25, 2008 at 6:04 PM, Mike Davis wrote: > >> Vincent Diepeveen wrote: >> >>> >>> Some 3d world country managers are begging to adress this issue: "My >>> nations people die, >>> as your bio fuel raises our food prices, the poor are so poor here, they >>> use that stuff as food >>> and cannot afford it now". >>> >>> USA nor Europe can *never* produce that stuff as cheap as 3d world >>> countries can. >>> >>> Since my uncle held a patent on an Alcohol Fuel still, and my grandfather >> grew corn, I have a somewhat different view. I'm not sure where your >> information is coming from. According to http://www.earth-policy.org/, the >> US produced 414 Million tons of grain last year. Of that total 81 Million >> tons were used for fuel and 106 Million tons were exported. While it might >> be less expensive to produce grain in the 3rd world, the top exporter of >> Corn, Rice and Wheat for the world is the US. >> http://www.earth-policy.org/Updates/2008/Update69_data.htm#table10 >> >> >> http://www.earth-policy.org/Updates/2008/Update69_data.htm#table9 >> >> Now since corn prices have been rather stagnant until recently for almost >> 30 years and since I know farmers that have burned corn for heat rather than >> sell at a loss. I see the increase in grain prices as neutral. Yes, there >> are bad results for the third world. But those farmers growing grain deserve >> a fair price for the work and uncertainty of agriculture. >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> > > > > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From Hakon.Bugge at scali.com Thu Jun 26 02:05:50 2008 From: Hakon.Bugge at scali.com (=?iso-8859-1?Q?H=E5kon?= Bugge) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <674893953.65671214349788737.JavaMail.root@zimbra.vpac.org> References: <20080624101512.A0C2435A90B@mail.scali.no> <674893953.65671214349788737.JavaMail.root@zimbra.vpac.org> Message-ID: <20080626090551.7EC2A35A8CC@mail.scali.no> At 01:23 25.06.2008, Chris Samuel wrote: > > IMHO, the MPI should virtualize these resources > > and relieve the end-user/application programmer > > from the burden. > >IMHO the resource manager (Torque, SGE, LSF, etc) should >be setting up cpusets for the jobs based on what the >scheduler has told it to use and the MPI shouldn't >get a choice in the matter. :-) I am inclined to agree with you in a perfect world. But, from my understanding the resource managers does not know the relationship between the cores. E.g., does core 3 and core 5 share a cache? Do they share a north-bridge bus, or are they located on different sockets? This is information we're using to optimize how pnt-to-pnt communication is implemented. The code-base involved is fairly complicated and I do not expect resource management systems to cope with it. I posted some measurement of the benefit of this methods some time ago and I include it here as a reference: http://www.scali.com/info/SHM-perf-8bytes-2007-12-20.htm If you look at the ping-ping numbers, you will se a nearly constant message rate, independent of placement of the processes. This contrary to other MPIs which (apparently) does not use this technique. So, in a practical world I go for performance, not perfect layering ;-) >Also helps when newbies run OpenMP codes thinking they're >single CPU codes and get 3 or 4 on the same 8 CPU node. Not sure I read you here. Do you mean pure OMP or hybrid models? Thanks, H?kon From Hakon.Bugge at scali.com Thu Jun 26 02:16:17 2008 From: Hakon.Bugge at scali.com (=?iso-8859-1?Q?H=E5kon?= Bugge) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: References: Message-ID: <20080626091618.6C1DD35A760@mail.scali.no> At 18:34 25.06.2008, Mikhail Kuzminsky wrote: >Let me assume now the following situation. I >have OpenMP-parallelized application which have >the number of processes equal to number of CPU >cores per server. And let me assume that this >application uses not too more virtual memory, so >all the real memory used may be placed in RAM of *one* node. >It's not the abstract question - a lot of >Gaussian-03 jobs we have fit to this situation, >and all the 8 cores for dual socket quad core >Opteron server will be "well loaded". > >Is it right that all the application memory (w/o >using of numactl) will be allocated (by Linux kernel) in *one* node ? Guess the answer is, it depends. The memory will be allocated on the node where the thread first touching it is running. But you could use numastat to investigate the issue. H?kon From gdjacobs at gmail.com Thu Jun 26 06:16:33 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:07:17 2009 Subject: [Beowulf] A simple cluster In-Reply-To: References: <7be8c36b0806250749t3fc55b4fpaa589bedc3d21f5c@mail.gmail.com> Message-ID: <486396B1.1040706@gmail.com> Peter St. John wrote: > Alcides, > > I think a short answer is: get a switch, plug all the boxes (throught > the ethernet ports on the motherboards) to the switch, install Ubuntu > and use OpenMP. > > Longer answers will be forthcoming, but I bet they will start with > questions about the specific hardware you have, the budget, whether > your applications are latency or compute time dependent, how > parallelizable it is, shared memory model... > > Peter > OpenMPI, maybe? From prentice at ias.edu Thu Jun 26 07:03:53 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:07:18 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <062520081731.12441.486280E10007F0080000309922069997350A040407900E9C@comcast.net> References: <062520081731.12441.486280E10007F0080000309922069997350A040407900E9C@comcast.net> Message-ID: <4863A1C9.8010605@ias.edu> saville@comcast.net wrote: > I respectfully request that you take conversations about washing machines and other non_beowulf related topics off to some other mailing list. I have plenty of email to delete without having the load increased by irrelevant discussions on this one. > > Many thanks, > > Gregg Look out, Gregg! I made a statement similar to this once, and I got flamed for days. This list always goes off-topic. :( -- Prentice From peter.st.john at gmail.com Thu Jun 26 08:08:52 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:18 2009 Subject: [Beowulf] A simple cluster In-Reply-To: <486396B1.1040706@gmail.com> References: <7be8c36b0806250749t3fc55b4fpaa589bedc3d21f5c@mail.gmail.com> <486396B1.1040706@gmail.com> Message-ID: Well I was really thinking that OpenMP might be easiest, the fastest from setup to application running for someone new (which would include me, this is hypothetical for me, but I think for myself I'll use MPI not OpenMP, because I want to micromanage the message passing) but I"m no expert by any any means. OpenMP is http://en.wikipedia.org/wiki/OpenMP (wiki) Peter On 6/26/08, Geoff Jacobs wrote: > > Peter St. John wrote: > > Alcides, > > > > I think a short answer is: get a switch, plug all the boxes (throught > > the ethernet ports on the motherboards) to the switch, install Ubuntu > > and use OpenMP. > > > > Longer answers will be forthcoming, but I bet they will start with > > questions about the specific hardware you have, the budget, whether > > your applications are latency or compute time dependent, how > > parallelizable it is, shared memory model... > > > > Peter > > > > > OpenMPI, maybe? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080626/fe488080/attachment.html From kus at free.net Thu Jun 26 09:10:43 2008 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:07:18 2009 Subject: [Beowulf] Again about NUMA (numactl and taskset) In-Reply-To: <20080626091618.6C1DD35A760@mail.scali.no> Message-ID: In message from H?kon Bugge (Thu, 26 Jun 2008 11:16:17 +0200): Numastat statistics before Gaussian-03 run (OpenMP, 8 threads, 8 cores, requires 512 Mbytes shared memory plus something more, may be fitted in memory of any node - I have 8 GB per node, 6- GB free in node0 and 7+ GB free in node1) node0: numa_hit 14594588 numa_miss 0 numa_foreign 0 interleave_hit 14587 local_node 14470168 other_node 124420 node1: numa_hit 11743071 numa_miss 0 numa_foreign 0 interleave_hit 14584 local_node 11727424 other_node 15647 ------------------------------------------- Statistics after run: node0: numa_hit 15466972 numa_miss 0 numa_foreign 0 interleave_hit 14587 local_node 15342552 other_node 124420 node1: numa_hit 12960452 numa_miss 0 numa_foreign 0 interleave_hit 14584 local_node 12944805 other_node 15647 ------------------------------------------- Unfortunately I don't know, what exactly means this lines !! :-( (BTW, do somebody know ?!) But intuitive it looks (taking into account the increase of numa_hit and local_node values), that the allocation of RAM was performed from BOTH nodes (and more RAM was allocated from node1 memory - node1 had initially more free RAM). It is in opposition w/my expectations of "continuous" RAM allocation from the RAM of one node ! Mikhail Kuzminsky, Computer Assistance to Chemical Research Zelinsky Institute of Organic Chemistry Moscow >At 18:34 25.06.2008, Mikhail Kuzminsky wrote: >>Let me assume now the following situation. I have OpenMP-parallelized >>application which have the number of processes equal to number of CPU >>cores per server. And let me assume that this application uses not >>too more virtual memory, so all the real memory used may be placed in >>RAM of *one* node. >>It's not the abstract question - a lot of Gaussian-03 jobs we have >>fit to this situation, and all the 8 cores for dual socket quad core >>Opteron server will be "well loaded". >> >>Is it right that all the application memory (w/o using of numactl) >>will be allocated (by Linux kernel) in *one* node ? > >Guess the answer is, it depends. The memory will be allocated on the >node where the thread first touching it is running. But you could use >numastat to investigate the issue. > > >H?kon > > >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf From shaeffer at neuralscape.com Thu Jun 26 09:32:29 2008 From: shaeffer at neuralscape.com (Karen Shaeffer) Date: Wed Nov 25 01:07:18 2009 Subject: [Beowulf] Re: "hobbyists" still OT In-Reply-To: References: <7FC50571-BBDF-45D1-A0B8-AB27D42B66E4@sicortex.com> <20080623184115.GA27571@student.math> <0BB98210-7B52-4456-9DDA-BCE044C7C924@xs4all.nl> <4B831460-3895-433D-A399-8243B3E6C069@xs4all.nl> <48626C96.8060305@vcu.edu> Message-ID: <20080626163229.GA32548@synapse.neuralscape.com> On Thu, Jun 26, 2008 at 03:30:17AM -0400, Robert G. Brown wrote: > > Stored sunlight is in finite supply in fossil fuels. Plants grow slowly > and require more than JUST sunlight to produce energy, so a fair bit of > energy yield has to be turned right back into fertilizer, fuel for > tractors or oxen, humans to grown and transport it, the refining or > extraction process. Using THIS fuel to produce hydrogen would simply > layer inefficiency on inefficiency, as one would throw maybe 70% of it > away in the process, and then throw a fair bit of what remained away in > the final stage of energy release as work. The laws of thermodynamics > cannot be broken, not really. > > This suggests that in the long run, the only free energy sources likely > to be able to keep up with human demand are the sun, implemented as > DIRECT conversion through solar cells, concentrators, turbines, and > perhaps thermonuclear fusion (not fission). The deuterium or helium > supply of the planet is relatively inexhaustible compared to other > fossil fuels, and more could be mined from other planets in the solar > system if need be. > > So sure, hydrogen is great, once you have the free energy source. But > FIRST you need that, as hydrogen doesn't WANT to come out of water > molecules. You have to make it come off. > > rgb Indeed. And there is tremendous activity in Solar cell technology today. Intel and IBM both have announced major projects to bring the technology into the mainstream in the past month. And I believe there is considerable news coming out of Japan on this issue these days. And if you want to secure venture capital in Silicon Valley these days, just sprinkle the words green tech into your business plan... Biofuels are just another loosing proposition by the old vested interests attempting to make more money at the expense of the health of the planet. I am glad you pointed out that it requires a lot of energy to produce all these biofuels -- and it all is contributing to the destruction of our environment as we knew it. Old thinking. Its dead. And cars in the future will be battery powered from the sun; There won't be any combustion engines in just a few decades. Combustion engines are going the way of the horse and buggy. -- Karen Shaeffer Neuralscape, Palo Alto, Ca. 94306 shaeffer@neuralscape.com http://www.neuralscape.com From peter.st.john at gmail.com Thu Jun 26 10:15:25 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:07:18 2009 Subject: [Beowulf] Re: "hobbyists" In-Reply-To: <4863A1C9.8010605@ias.edu> References: <062520081731.12441.486280E10007F0080000309922069997350A040407900E9C@comcast.net> <4863A1C9.8010605@ias.edu> Message-ID: Some of us are latency bound and can't handle the extra bandwidth, but some of us are compute intensive and don't mind :-) Peter On 6/26/08, Prentice Bisbal wrote: > > saville@comcast.net wrote: > > I respectfully request that you take conversations about washing machines > and other non_beowulf related topics off to some other mailing list. I have > plenty of email to delete without having the load increased by irrelevant > discussions on this one. > > > > Many thanks, > > > > Gregg > > > Look out, Gregg! I made a statement similar to this once, and I got > flamed for days. This list always goes off-topic. :( > > -- > > Prentice > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080626/e212665d/attachment.html From gdjacobs at gmail.com Thu Jun 26 10:17:09 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:07:18 2009 Subject: [Beowulf] A simple cluster In-Reply-To: Referen