From csamuel at vpac.org Sat Mar 1 12:25:25 2008 From: csamuel at vpac.org (csamuel@vpac.org) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] Cluster Monitoring Tool In-Reply-To: <19973042.101204403003202.JavaMail.csamuel@ubuntu> Message-ID: <12219323.121204403093384.JavaMail.csamuel@ubuntu> ----- "Cally" wrote: > Hi everyone, Hi Cally, > Is there some kinda tool to use, say if I want to see how much of memory > is being used just for a rendering process. You don't mention which O/S you're using, so I'm presuming it's Linux based. The Linux kernel memory accounting is, umm, sub-optimal for this sort of stuff, to get an accurate picture you probably want to look into something like exmap which is both a kernel module and a user space program. Ubuntu describes it as: Exmap is a memory analysis tool which allows you to accurately determine how much physical memory and swap is used by individual processes and shared libraries on a running system. In particular, it accounts for the sharing of memory and swap between different processes. Help is at hand though - 2.6.25 will include [1] Matt Mackall's page map patches [2] to improve the memory accounting in the kernel. [1] - http://lwn.net/Articles/267849/ [2] - http://lwn.net/Articles/230975/ cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From csamuel at vpac.org Sat Mar 1 13:02:02 2008 From: csamuel at vpac.org (csamuel@vpac.org) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] Open source Job Scheduler for Apple Leopard 10.5.2 server that will work with Open Directory? In-Reply-To: <15809818.381204405232710.JavaMail.csamuel@ubuntu> Message-ID: <27647177.401204405286579.JavaMail.csamuel@ubuntu> ----- "Prakashan Korambath" wrote: > Anyone knows an Open source job scheduler for Apple Leopard 10.5.2 > server that will work with Open Directory? SGE seems to have > intermittent problems, Condor and Torque supports only 10.4. Tiger. It might be worth asking on the Torque development list [1] about whether anyone there is working with Leopard, I know that they've just got the SVN trunk version building with no warnings on OSX, but don't know which version they're testing with (I've just asked). I guess there's not much demand for OSX clusters these days.. [1] - http://www.supercluster.org/mailman/listinfo/torquedev cheers! Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From reuti at staff.uni-marburg.de Sun Mar 2 01:45:06 2008 From: reuti at staff.uni-marburg.de (Reuti) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] Re: python2.4 error when loose MPICH2 TI with Grid Engine In-Reply-To: References: Message-ID: Hi, Am 22.02.2008 um 09:23 schrieb Sangamesh B: > Dear Reuti & members of beowulf, > > I need to execute a parallel job thru grid engine. > > MPICH2 is installed with Process Manager:mpd. > > Added a parallel environment MPICH2 into SGE: > > $ qconf -sp MPICH2 > pe_name MPICH2 > slots 999 > user_lists NONE > xuser_lists NONE > start_proc_args /share/apps/MPICH2/startmpi.sh -catch_rsh > $pe_hostfile > stop_proc_args /share/apps/MPICH2/stopmpi.sh > allocation_rule $pe_slots > control_slaves FALSE > job_is_first_task TRUE > urgency_slots min > > > Added this PE to the default queue: all.q. > > mpdboot is done. mpd's are running on two nodes. > > The script for submitting this job thru sge is: > > $ cat subsamplempi.sh > #!/bin/bash > > #$ -S /bin/bash > > #$ -cwd > > #$ -N Samplejob > > #$ -q all.q > > #$ -pe MPICH2 4 > > #$ -e ERR_$JOB_NAME.$JOB_ID > > #$ -o OUT_$JOB_NAME.$JOB_ID > > date > > hostname > > /opt/MPI_LIBS/MPICH2-GNU/bin/mpirun -np $NSLOTS -machinefile > $TMP_DIR/machines ./samplempi > > echo "Executed" > > exit 0 > > > The job is getting submitted, but not executing. The error and > output file contain: > > cat ERR_Samplejob.192 > /usr/bin/env: python2.4: No such file or directory > > $ cat OUT_Samplejob.192 > -catch_rsh /opt/gridengine/default/spool/compute-0-0/active_jobs/ > 192.1/pe_hostfile > compute-0-0 > compute-0-0 > compute-0-0 > compute-0-0 > Fri Feb 22 12:57:18 IST 2008 > compute-0-0.local > Executed > > So the problem is coming for python2.4. > > $ which python2.4 > /opt/rocks/bin/python2.4 > > I googled this error. Then created a symbolic link: > > # ln -sf /opt/rocks/bin/python2.4 /bin/python2.4 > > After this also same error is coming. > > I guess the problem might be different. i.e. gridengine might not > getting the link to running mpd. > > And the procedure followed by me to configure PE might be wrong. > > So, I expect from you to clear my doubts and help me to resolve > this error. > > 1. Is the PE configuration of MPICH2 + grid engine right? if you want to integrate MPICH2 with MPD it's similar to a PVM setup. The daemons must be started in start_proc_args on every node with a dedicated port number per job. You don't say what your startmpi.sh is doing. > 2. Without Tight integration, is there a way to run a MPICh2(mpd) > based job using gridengine? Yes. > 3. In smpd-daemon based and daemonless MPICH2 tight integration, > which one is better? Depends: if you have just one mpirun per job which will run for days, I would go for the daemonless startup. But if you issue many mpirun calls in your jobscript which will just run for seconds I would go for the daemon based startup, as the mpirun will be distributed to the slaves faster. > 4. Can we do mvapich2 tight integration with SGE? Any differences > with process managers wrt MVAPICH2? Maybe, if the startup is similar to standard MPICH2. -- Reuti > Thanks & Best Regards, > Sangamesh B -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080302/a3484f8f/attachment.html From reuti at staff.uni-marburg.de Sun Mar 2 01:48:53 2008 From: reuti at staff.uni-marburg.de (Reuti) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] Three questions on a new Beowulf Cluster In-Reply-To: <47BEEF3A.9020806@sps.aero> References: <47BEEF3A.9020806@sps.aero> Message-ID: <00C9756F-C215-4934-BF18-498997FEBB4A@staff.uni-marburg.de> Hi, Am 22.02.2008 um 16:50 schrieb John P. Kosky, PhD: > My company is taking it's first foray into the world of HPC with an > expandable architecture, 16 processor (comprised of quad core > Opterons), one header node cluster using Infiniband interconnects. > OS has tentatively been selected as SUSE 64-bit Linux. The > principal purpose of the cluster is as a tool for spacecraft and > propulsion design support. The cluster will therefore be running > the most recent versions of commercially available software - > initially for FEA and CFD using COMSOL Multiphysics and associated > packages, NASTRAN, MatLab modules, as well as an internally > modified and expanded commercial code for materials properties > prediction,with emphasis on polymer modeling (Accelrys Materials > Studio). Since we will be repetitively running standard modeling > codes on this system, we are trying to make the system as user > friendly as possible... most of our scientists and engineers want > to use this as a tool, and not have to become cluster experts. The > company WILL be hiring an IT Sys Admin with good cluster experience > to support the system, however... > > Question 1: > 1) Does anyone here know of any issues that have arisen running the > above named commercial packages on clusters using infiniband? > > Question 2: > 2) As far as the MPI for the system is concerned, for the system > and application requirements described above, would OpenMPI or > MvApich be better for managing node usage? none of them will manage the node usage - you have to assemble a node list for every run by hand. What you might be looking for is a resource manager like SGE, Torque, LSF, Condor,... und run parallel jobs under their supervision. -- Reuti > ANY help or advice would be greatly appreciated. > > Thanks in advance > > John > > John P. Kosky, PhD > Director of Technical Development > Space Propulsion Systems > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From hahn at mcmaster.ca Sun Mar 2 11:40:28 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] IB + 10G eth integrated switches? Message-ID: Hi all, has anyone experienced switches that have integrated both IB and 10Geth? I guess I've seen IB switch vendors talking about EoIB modules for a while, but I'm curious about how well they work. mainly I'm curious about providing high-speed IP connectivity for a cluster that has only IB networking. I presume this would work by configuring an EoIB interface on the node, and having the IB switch act like an eth switch (L2?). thanks, mark hahn. From kalpana0611 at gmail.com Sun Mar 2 02:21:29 2008 From: kalpana0611 at gmail.com (Cally) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] Re: Cluster Monitoring Tool Message-ID: Yeah, I am using OpenSuse for the 2 node thing, bt the cluster in the lab uses Redhat, thanks alot. I am checking out ur mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080302/c8e3c327/attachment.html From glen.beane at jax.org Sun Mar 2 17:21:46 2008 From: glen.beane at jax.org (Glen Beane) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] Open source Job Scheduler for Apple Leopard 10.5.2 server that will work with Open Directory? In-Reply-To: <43F64E86355A744E9D51506B6C6783B9021AE68C@EM2.ad.ucla.edu> References: <02A63D14-3E34-4C0E-A012-D491922AC023@ee.duke.edu> <220FE1C2-C27A-4B94-8060-D4D78DFCF50A@staff.uni-marburg.de> <43F64E86355A744E9D51506B6C6783B9021AE68C@EM2.ad.ucla.edu> Message-ID: <47CB52AA.9070306@jax.org> Korambath, Prakashan wrote: > Anyone knows an Open source job scheduler for Apple Leopard 10.5.2 > server that will work with Open Directory? SGE seems to have > intermittent problems, Condor and Torque supports only 10.4. Tiger. Thanks. Hello Prakashan, I am a TORQUE developer and Apple user. TORQUE 2.2.x and up should work on Leopard. Although 2.2.x requires passing --disable-gcc-warnings to configure, as we compile with -Wall -pedantic -Werror by default and there are a few harmless warnings generated. If you are compiling the latest 2.3.0 (development) snapshot then these warnings are fixed (some were Apples fault, the code was fine, and this has been fixed in the 2.5 and 3.0 Developer Tools). An official 2.3.0 release should be out soon. If you have any problems running TORQUE on OS X Leopard I would like to know so I can improve our support for OS X -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153 From Shainer at mellanox.com Sun Mar 2 22:34:44 2008 From: Shainer at mellanox.com (Gilad Shainer) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] IB + 10G eth integrated switches? In-Reply-To: Message-ID: <9FA59C95FFCBB34EA5E42C1A8573784FF826F6@mtiexch01.mti.com> Hi Mark, > > Hi all, > has anyone experienced switches that have integrated both IB > and 10Geth? > I guess I've seen IB switch vendors talking about EoIB > modules for a while, but I'm curious about how well they > work. mainly I'm curious about providing high-speed IP > connectivity for a cluster that has only IB networking. I > presume this would work by configuring an EoIB interface on > the node, and having the IB switch act like an eth switch (L2?). > Most of the IB switch vendors have gateway solution to connect InfiniBand based clusters to 10GigE networks, and the gateway change L2 of IB to L2 of Eth. Those gateways have been installed in many places and work great. Gilad. > thanks, mark hahn. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org To change your > subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From hahn at mcmaster.ca Sun Mar 2 22:43:35 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] IB + 10G eth integrated switches? In-Reply-To: <9FA59C95FFCBB34EA5E42C1A8573784FF826F6@mtiexch01.mti.com> References: <9FA59C95FFCBB34EA5E42C1A8573784FF826F6@mtiexch01.mti.com> Message-ID: > of IB to L2 of Eth. Those gateways have been installed in many places > and work great. thanks, but that's not what I'm asking for. I am interested in actual experience from purchasers and users, not vendor/marketing praise. for instance, if you stream TCP through such a route, can you saturate the link? how well does NFS work through such a network? From deadline at eadline.org Mon Mar 3 06:23:08 2008 From: deadline at eadline.org (Douglas Eadline) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] NYCA-HUG Meeting (geo specific) Message-ID: <40232.192.168.1.1.1204554188.squirrel@mail.eadline.org> Ignore this message if you are not in the New York City Metro area. For those interested, this months meeting will be on March 6th at Le Figaro Cafe. Mario Juric from the School of Natural Sciences, Institute for Advanced Study at Princeton will be talking about computing with GPGPU chips. (General Purpose Graphical Processing Units i.e. video cards) More information, directions, and links at http://www.linux-mag.com/nyca-hug See you there. -- Doug From ppk at ats.ucla.edu Mon Mar 3 06:36:44 2008 From: ppk at ats.ucla.edu (Korambath, Prakashan) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] Open source Job Scheduler for Apple Leopard 10.5.2 server that will work with Open Directory? References: <02A63D14-3E34-4C0E-A012-D491922AC023@ee.duke.edu> <220FE1C2-C27A-4B94-8060-D4D78DFCF50A@staff.uni-marburg.de> <43F64E86355A744E9D51506B6C6783B9021AE68C@EM2.ad.ucla.edu> <47CB52AA.9070306@jax.org> Message-ID: <43F64E86355A744E9D51506B6C6783B9021AE6C6@EM2.ad.ucla.edu> Thank you Glen. I will try Torque and let you know if there are any problems. Prakashan -----Original Message----- From: Glen Beane [mailto:glen.beane@jax.org] Sent: Sun 3/2/2008 5:21 PM To: Korambath, Prakashan Cc: Beowulf Mailing List Subject: Re: [Beowulf] Open source Job Scheduler for Apple Leopard 10.5.2 server that will work with Open Directory? Korambath, Prakashan wrote: > Anyone knows an Open source job scheduler for Apple Leopard 10.5.2 > server that will work with Open Directory? SGE seems to have > intermittent problems, Condor and Torque supports only 10.4. Tiger. Thanks. Hello Prakashan, I am a TORQUE developer and Apple user. TORQUE 2.2.x and up should work on Leopard. Although 2.2.x requires passing --disable-gcc-warnings to configure, as we compile with -Wall -pedantic -Werror by default and there are a few harmless warnings generated. If you are compiling the latest 2.3.0 (development) snapshot then these warnings are fixed (some were Apples fault, the code was fine, and this has been fixed in the 2.5 and 3.0 Developer Tools). An official 2.3.0 release should be out soon. If you have any problems running TORQUE on OS X Leopard I would like to know so I can improve our support for OS X -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080303/6b9d8250/attachment.html From glen.beane at jax.org Mon Mar 3 07:29:37 2008 From: glen.beane at jax.org (Glen Beane) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] Open source Job Scheduler for Apple Leopard 10.5.2 server that will work with Open Directory? In-Reply-To: <43F64E86355A744E9D51506B6C6783B9021AE6C6@EM2.ad.ucla.edu> References: <02A63D14-3E34-4C0E-A012-D491922AC023@ee.duke.edu> <220FE1C2-C27A-4B94-8060-D4D78DFCF50A@staff.uni-marburg.de> <43F64E86355A744E9D51506B6C6783B9021AE68C@EM2.ad.ucla.edu> <47CB52AA.9070306@jax.org> <43F64E86355A744E9D51506B6C6783B9021AE6C6@EM2.ad.ucla.edu> Message-ID: <47CC1961.7080505@jax.org> Korambath, Prakashan wrote: > Thank you Glen. I will try Torque and let you know if there are any > problems. please note that the job memory usage reported by Torque will be incorrect on OS X - I just got some fixes for that into subversion a few days ago. These fixes will be released with 2.3.0 (and in the next 2.3.0 development snapshot) I also backported these job memory reporting fixes to the 2.2 branch and those will be available to 2.2.x users when 2.2.2 is released. -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153 From mathog at caltech.edu Mon Mar 3 12:47:06 2008 From: mathog at caltech.edu (David Mathog) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] mysterious slow disk Message-ID: One node has developed a "slow disk", although at this point I'm not at all sure that the disk is actually at fault. Have any of you ever seen something like this: 1. hdparm -t -T /dev/hda on the slow node is typically: Timing cached reads: 504 MB in 2.00 seconds = 251.61 MB/sec Timing buffered disk reads: 104 MB in 3.55 seconds = 29.29 MB/sec but the second line varies A LOT, down to 24Mb and up to over 30. However the same test on the other nodes varies only a little: Timing cached reads: 514 MB in 2.01 seconds = 256.03 MB/sec Timing buffered disk reads: 124 MB in 3.04 seconds = 40.74 MB/sec (plus or minus about 1MB/sec on the second line). 2. hdparm -i -v and -i -m are identical on all machines, except for serial number. Here is -i -v on the slow one: /dev/hda: multcount = 16 (on) IO_support = 1 (32-bit) unmaskirq = 1 (on) using_dma = 1 (on) keepsettings = 0 (off) readonly = 0 (off) readahead = 256 (on) geometry = 65535/16/63, sectors = 78165360, start = 0 Model=WDC WD400BB-00DEA0, FwRev=05.03E05, SerialNo=WD-WMAD11736294 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } RawCHS=16383/16/63, TrkSize=57600, SectSize=600, ECCbytes=40 BuffType=DualPortCache, BuffSize=2048kB, MaxMultSect=16, MultSect=16 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=78165360 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 udma3 udma4 *udma5 AdvancedPM=no WriteCache=enabled Drive conforms to: Unspecified: ATA/ATAPI-1 ATA/ATAPI-2 ATA/ATAPI-3 ATA/ATAPI-4 ATA/ATAPI-5 3. Western Digital diagnostics and smartctl show nothing wrong with the disk. It has no bad sectors or other errors logged. The smartctl tests do take longer to complete than on the other disks. 4. DMA is working (at least partially), since turning that off drops the hdparm test down to about 4Mb/sec. 5. Opened the case and all jumpers were as they should be, Power supply tested good. 6. dmesg from the slow node and a normal node shows no significant differences. (Changes in the 3rd digit after the decimal of the bogoMIPS value, for instance.) The only oddity I've found was that the setting "32 bit I/O" in the BIOS was disabled for some reason on the slow node. Changing it to enabled made no difference (even after several reboots, cold and warm.) Is it possible that the OS has the earlier setting hidden away somewhere and is still using it? This was particularly weird because hdparm showed IO_Support = 1 (32-bit) even when this BIOS bit was disabled. The speed issue initially turned up in a run where a certain program was required to allocate about 1.6 Gb of memory (at least .6GB of which had to come out of the 2GB swap, since there was only 1GB of RAM.) That large region was then ordered with qsort(). Bizarrely, this took forever on one node (hours longer than on any other node, with similar sized data), and when it finished the sort the resulting binary data was written to disk at only 0.5 Mb/sec. Yes, 500kilobytes/sec. Nothing else was using CPU time. Prior to that one run this node did nothing to draw my attention to it as a "slow node". This is on one of the notorious Tyan S2466 boards. I'm beginning to wonder if perhaps it now has a bit stuck somewhere in the BIOS, in which case maybe wiping the BIOS settings and redoing them will fix it. I already tried powering it off for 15 minutes unplugged, but that did not help. Also the whole cluster had to be powered down for about 15 minutes in the morning before this started for A/C service. If the battery on that board is iffy it might explain how the "32 bit I/O" became disabled. However, it did not reset again on a subsequent long power down. Thanks, David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From mathog at caltech.edu Mon Mar 3 14:25:24 2008 From: mathog at caltech.edu (David Mathog) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] Re: mysterious slow disk Message-ID: Bruno Coutinho wrote: > What is the Reallocated_Sector_Ct in smartctl? If it's non zero, you will > have badblocks soon. > > How is the Seek_Error_Rate is compared with orher HDs? Here is part of smartctl -a for the slow one, with some columns edited out so that it will fit without wrapping. The disk is old but it isn't throwing any conventional sorts of errors: ID# ATTRIBUTE_NAME FLAG UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b Always - 0 3 Spin_Up_Time 0x0007 Always - 2658 4 Start_Stop_Count 0x0032 Always - 88 5 Reallocated_Sector_Ct 0x0033 Always - 0 7 Seek_Error_Rate 0x000b Always - 0 9 Power_On_Hours 0x0032 Always - 44036 10 Spin_Retry_Count 0x0013 Always - 0 11 Calibration_Retry_Count 0x0013 Always - 0 12 Power_Cycle_Count 0x0032 Always - 88 196 Reallocated_Event_Count 0x0032 Always - 0 197 Current_Pending_Sector 0x0012 Always - 0 198 Offline_Uncorrectable 0x0012 Always - 0 199 UDMA_CRC_Error_Count 0x000a Always - 0 200 Multi_Zone_Error_Rate 0x0009 Offline - 0 Regards, David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From carsten.aulbert at aei.mpg.de Mon Mar 3 22:57:24 2008 From: carsten.aulbert at aei.mpg.de (Carsten Aulbert) Date: Wed Nov 25 01:06:55 2009 Subject: [Beowulf] mysterious slow disk In-Reply-To: References: Message-ID: <47CCF2D4.9090701@aei.mpg.de> Hi David, David Mathog wrote: > 2. hdparm -i -v and -i -m are identical on all machines, except > for serial number. Here is -i -v on the slow one: Looks good. Can you run hdparm -I /dev/hda? And there please have a look at the line with acoustic mgmt: Recommended acoustic management value: 128, current value: 128 (this line is from my laptop). Run it on the faster node and compare those. Maybe it's ust that somehow acoustic mgmt is enabled on one and not on the other. HTH Carsten From asabigue at fing.edu.uy Tue Mar 4 02:14:19 2008 From: asabigue at fing.edu.uy (ariel sabiguero yawelak) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] mysterious slow disk In-Reply-To: References: Message-ID: <47CD20FB.2000605@fing.edu.uy> Hi! I found a situation pretty similar to this a few years ago in a system with shared video memory. Whenever you allocate less than 32MB of video memory (even without X server running) the performance drops incredibly low, without any kind of error, warning or whatever. System seems to stalls erratically, but when it continues working, everythig seems ok. The performance problem was related to every I/O operation we performed, not only HDD related. We were able to detect the problem because we were "tunning" the configuration, trying to free as much memory and resources. After we tunned the system, It just started to crawl and when we rolled back, we realized the source of the problem: "If it works, don't fix it" I hope this helps. best regards ariel David Mathog escribi?: > One node has developed a "slow disk", although at this point I'm not > at all sure that the disk is actually at fault. Have any of you ever > seen something like this: > > 1. hdparm -t -T /dev/hda on the slow node is typically: > > Timing cached reads: 504 MB in 2.00 seconds = 251.61 MB/sec > Timing buffered disk reads: 104 MB in 3.55 seconds = 29.29 MB/sec > > but the second line varies A LOT, down to 24Mb and up to over > 30. However the same test on the other nodes varies only a little: > > Timing cached reads: 514 MB in 2.01 seconds = 256.03 MB/sec > Timing buffered disk reads: 124 MB in 3.04 seconds = 40.74 MB/sec > (plus or minus about 1MB/sec on the second line). > > 2. hdparm -i -v and -i -m are identical on all machines, except > for serial number. Here is -i -v on the slow one: > > /dev/hda: > multcount = 16 (on) > IO_support = 1 (32-bit) > unmaskirq = 1 (on) > using_dma = 1 (on) > keepsettings = 0 (off) > readonly = 0 (off) > readahead = 256 (on) > geometry = 65535/16/63, sectors = 78165360, start = 0 > > Model=WDC WD400BB-00DEA0, FwRev=05.03E05, SerialNo=WD-WMAD11736294 > Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } > RawCHS=16383/16/63, TrkSize=57600, SectSize=600, ECCbytes=40 > BuffType=DualPortCache, BuffSize=2048kB, MaxMultSect=16, MultSect=16 > CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=78165360 > IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} > PIO modes: pio0 pio1 pio2 pio3 pio4 > DMA modes: mdma0 mdma1 mdma2 > UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5 udma3 udma4 *udma5 > AdvancedPM=no WriteCache=enabled > Drive conforms to: Unspecified: ATA/ATAPI-1 ATA/ATAPI-2 ATA/ATAPI-3 > ATA/ATAPI-4 ATA/ATAPI-5 > > 3. Western Digital diagnostics and smartctl show nothing wrong with > the disk. It has no bad sectors or other errors logged. The smartctl > tests do take longer to complete than on the other disks. > > 4. DMA is working (at least partially), since turning that off drops > the hdparm test down to about 4Mb/sec. > > 5. Opened the case and all jumpers were as they should be, Power supply > tested good. > > 6. dmesg from the slow node and a normal node shows no significant > differences. (Changes in the 3rd digit after the decimal of the > bogoMIPS value, for instance.) > > The only oddity I've found was that the setting "32 bit I/O" in the BIOS > was disabled for some reason on the slow node. Changing it to enabled > made no difference (even after several reboots, cold and warm.) Is it > possible that the OS has the earlier setting hidden away somewhere and > is still using it? This was particularly weird because hdparm showed > IO_Support = 1 (32-bit) > even when this BIOS bit was disabled. > > The speed issue initially turned up in a run where a certain program was > required to allocate about 1.6 Gb of memory (at least .6GB of which had > to come out of the 2GB swap, since there was only 1GB of RAM.) That > large region was then ordered with qsort(). Bizarrely, this took > forever on one node (hours longer than on any other node, with similar > sized data), and when it finished the sort the resulting binary data was > written to disk at only 0.5 Mb/sec. Yes, 500kilobytes/sec. Nothing > else was using CPU time. Prior to that one run this node did > nothing to draw my attention to it as a "slow node". > > This is on one of the notorious Tyan S2466 boards. I'm beginning to > wonder if perhaps it now has a bit stuck somewhere in the BIOS, in which > case maybe wiping the BIOS settings and redoing them will fix it. > I already tried powering it off for 15 minutes unplugged, but that > did not help. > > Also the whole cluster had to be powered down for about 15 minutes > in the morning before this started for A/C service. If the battery > on that board is iffy it might explain how the "32 bit I/O" became > disabled. However, it did not reset again on a subsequent long power down. > > Thanks, > > David Mathog > mathog@caltech.edu > Manager, Sequence Analysis Facility, Biology Division, Caltech > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > From steffen.grunewald at aei.mpg.de Tue Mar 4 00:39:18 2008 From: steffen.grunewald at aei.mpg.de (Steffen Grunewald) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Re: mysterious slow disk In-Reply-To: References: Message-ID: <20080304083918.GE23076@casco.aei.mpg.de> On Mon, Mar 03, 2008 at 02:25:24PM -0800, David Mathog wrote: > Bruno Coutinho wrote: > > > What is the Reallocated_Sector_Ct in smartctl? If it's non zero, you will > > have badblocks soon. > > > > How is the Seek_Error_Rate is compared with orher HDs? > > Here is part of smartctl -a for the slow one, with some columns > edited out so that it will fit without wrapping. The disk is old but > it isn't throwing any conventional sorts of errors: > > ID# ATTRIBUTE_NAME FLAG UPDATED WHEN_FAILED RAW_VALUE > 1 Raw_Read_Error_Rate 0x000b Always - 0 > 3 Spin_Up_Time 0x0007 Always - 2658 > 4 Start_Stop_Count 0x0032 Always - 88 > 5 Reallocated_Sector_Ct 0x0033 Always - 0 > 7 Seek_Error_Rate 0x000b Always - 0 > 9 Power_On_Hours 0x0032 Always - 44036 > 10 Spin_Retry_Count 0x0013 Always - 0 > 11 Calibration_Retry_Count 0x0013 Always - 0 > 12 Power_Cycle_Count 0x0032 Always - 88 > 196 Reallocated_Event_Count 0x0032 Always - 0 > 197 Current_Pending_Sector 0x0012 Always - 0 > 198 Offline_Uncorrectable 0x0012 Always - 0 > 199 UDMA_CRC_Error_Count 0x000a Always - 0 > 200 Multi_Zone_Error_Rate 0x0009 Offline - 0 Looks clean. (Sorry, but I probably missed part of the thread:) Did you check the (U)DMA mode? (hdparm -di /dev/hd*) Would a full reboot (power down, power up) change the performance? Steffen -- Steffen Grunewald * MPI Grav.Phys.(AEI) * Am M?hlenberg 1, D-14476 Potsdam Cluster Admin * http://pandora.aei.mpg.de/merlin/ * http://www.aei.mpg.de/ * e-mail: steffen.grunewald(*)aei.mpg.de * +49-331-567-{fon:7233,fax:7298} No Word/PPT mails - http://www.gnu.org/philosophy/no-word-attachments.html From p2s2-chairs at mcs.anl.gov Tue Mar 4 13:03:37 2008 From: p2s2-chairs at mcs.anl.gov (p2s2-chairs@mcs.anl.gov) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] [p2s2-announce] CFP: Workshop on Parallel Programming Models and Systems Software for High-end Computing (P2S2) Message-ID: <20080304210337.BAC4246218@shakey.mcs.anl.gov> CALL FOR PAPERS =============== First International Workshop on Parallel Programming Models and Systems Software for High-end Computing (P2S2) (http://www.mcs.anl.gov/events/workshops/p2s2) Sep. 8th, 2008 To be held in conjunction with ICPP-08: The 27th International Conference on Parallel Processing Sep. 8-12, 2008 Portland, Oregon, USA SCOPE ----- The goal of this workshop is to bring together researchers and practitioners in parallel programming models and systems software for high-end computing systems. Please join us in a discussion of new ideas, experiences, and the latest trends in these areas at the workshop. TOPICS OF INTEREST ------------------ The focus areas for this workshop include, but are not limited to: * Programming models and their high-performance implementations o MPI, Sockets, OpenMP, Global Arrays, X10, UPC, Chapel o Other Hybrid Programming Models * Systems software for scientific and enterprise computing o Communication sub-subsystems for high-end computing o High-performance File and storage systems o Fault-tolerance techniques and implementations o Efficient and high-performance virtualization and other management mechanisms * Tools for Management, Maintenance, Coordination and Synchronization o Software for Enterprise Data-centers using Modern Architectures o Job scheduling libraries o Management libraries for large-scale system o Toolkits for process and task coordination on modern platforms * Performance evaluation, analysis and modeling of emerging computing platforms SUBMISSION INSTRUCTIONS ----------------------- Submissions should be in PDF format in U.S. Letter size paper. They should not exceed 8 pages (all inclusive). Submissions will be judged based on relevance, significance, originality, correctness and clarity. DATES AND DEADLINES ------------------- Paper Submission: April 11th, 2008 Author Notification: May 20th, 2008 Camera Ready: June 2nd, 2008 PROGRAM CHAIRS -------------- * Pavan Balaji (Argonne National Laboratory) * Sayantan Sur (IBM Research) STEERING COMMITTEE ------------------ * William D. Gropp (University of Illinois Urbana-Champaign) * Dhabaleswar K. Panda (Ohio State University) * Vijay Saraswat (IBM Research) PROGRAM COMMITTEE ----------------- * David Bernholdt (Oak Ridge National Laboratory) * Ron Brightwell (Sandia National Laboratory) * Wu-chun Feng (Virginia Tech) * Richard Graham (Oak Ridge National Laboratory) * Hyun-wook Jin (Konkuk University, South Korea) * Sameer Kumar (IBM Research) * Doug Lea (State University of New York at Oswego) * Jarek Nieplocha (Pacific Northwest National Laboratory) * Scott Pakin (Los Alamos National Laboratory) * Vivek Sarkar (Rice University) * Rajeev Thakur (Argonne National Laboratory) * Pete Wyckoff (Ohio Supercomputing Center) If you have any questions, please contact us at p2s2-chairs@mcs.anl.gov ------------------------------------------------------------------------- If you do not want to receive any more announcements about the P2S2 workshop, please send an email to majordomo@mcs.anl.gov with the email body "unsubscribe p2s2-announce". ------------------------------------------------------------------------- From csamuel at vpac.org Wed Mar 5 05:55:59 2008 From: csamuel at vpac.org (csamuel@vpac.org) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] mysterious slow disk In-Reply-To: Message-ID: <502938.641204725340009.JavaMail.csamuel@ubuntu> ----- "David Mathog" wrote: > One node has developed a "slow disk", although at this point I'm not > at all sure that the disk is actually at fault. As an extreme suggestion - how about swapping the drive with a known good node and rebuilding both ? If the problem moves with the drive then you'll know it's that, and if it stays then it's likely to be the MB or the cabling. Of course there's always the chance that the problem will disappear completely or you'll have two slow drives instead of one.. :-) cheers! Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From Bogdan.Costescu at iwr.uni-heidelberg.de Wed Mar 5 07:00:30 2008 From: Bogdan.Costescu at iwr.uni-heidelberg.de (Bogdan Costescu) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] mysterious slow disk In-Reply-To: <502938.641204725340009.JavaMail.csamuel@ubuntu> References: <502938.641204725340009.JavaMail.csamuel@ubuntu> Message-ID: On Thu, 6 Mar 2008, csamuel@vpac.org wrote: > If the problem moves with the drive then you'll know it's that, and > if it stays then it's likely to be the MB or the cabling. Sorry for the late jump... IIRC, the OP had dual Athlon based nodes; there used to be some issues of slow I/O related to the usage of energy saving features of the CPUs and/or chipset. Have you already checked for such a situation ? Another thing that comes to my mind: I have seen some SATA disks (I think from Hitachi) which have developed some bad sectors, which could be seen in the smartctl output. I have run the DFT on them and after choosing to rewrite the whole disk (I have forgotten the exact option name...) the disk was shown as good and there was no sign of bad sectors in the smartctl output. However, the access to disk was slower, especially the writes, for which the speed was aproximately halved. So, is it sure that the disk was never subjected to a manufacturer "diagnose and repair" tool ? -- Bogdan Costescu IWR, University of Heidelberg, INF 368, D-69120 Heidelberg, Germany Phone: +49 6221 54 8869/8240, Fax: +49 6221 54 8868/8850 E-mail: bogdan.costescu@iwr.uni-heidelberg.de From mathog at caltech.edu Wed Mar 5 11:44:59 2008 From: mathog at caltech.edu (David Mathog) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Re: mysterious slow disk Message-ID: I don't think I'm going to solve this one :-(. Bruno Coutinho wrote: >I noticed that some disks can't get full bandwidth at once. >A way to monitor disk throughput for a longer time is to use dd >to copy a partition to /dev/null. The disk is also slow for long dd operations by about the same factor. SLOW: % sync; \ TIME=`accudate -t0` ; \ dd if=/dev/zero count=4000000 of=/scratch/tmp/foo.dat ; \ sync; \ accudate -ds $TIME 4000000+0 records in 4000000+0 records out 2048000000 bytes (2.0 GB) copied, 72.4961 seconds, 28.2 MB/s 0000105.343 vs. FAST: % sync; \ TIME=`accudate -t0` ; \ dd if=/dev/zero count=4000000 of=/scratch/tmp/foo.dat ; \ sync; \ accudate -ds $TIME 4000000+0 records in 4000000+0 records out 2048000000 bytes (2.0 GB) copied, 59.6326 seconds, 34.3 MB/s 0000075.422 (27Mb/sec sustained vs. 19.4Mb/sec sustained). Carsten Aulbert wrote: >Looks good. Can you run hdparm -I /dev/hda? >And there please have a look at the line with acoustic mgmt: hdparm -I was identical on fast and slow systems (except for serial numbers). hdparm -I /dev/hda /dev/hda: ATA device, with non-removable media Model Number: WDC WD400BB-00DEA0 Serial Number: WD-WMAD11736294 Firmware Revision: 05.03E05 Standards: Supported: 5 4 3 Likely used: 6 Configuration: Logical max current cylinders 16383 16383 heads 16 16 sectors/track 63 63 -- CHS current addressable sectors: 16514064 LBA user addressable sectors: 78165360 device size with M = 1024*1024: 38166 MBytes device size with M = 1000*1000: 40020 MBytes (40 GB) Capabilities: LBA, IORDY(can be disabled) bytes avail on r/w long: 40 Standby timer values: spec'd by Standard, with device specific minimum R/W multiple sector transfer: Max = 16 Current = 16 Recommended acoustic management value: 128, current value: 254 DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 Cycle time: min=120ns recommended=120ns PIO: pio0 pio1 pio2 pio3 pio4 Cycle time: no flow control=120ns IORDY flow control=120ns Commands/features: Enabled Supported: * SMART feature set Security Mode feature set * Power Management feature set * Write cache * Look-ahead * Host Protected Area feature set * WRITE_BUFFER command * READ_BUFFER command * DOWNLOAD_MICROCODE SET_MAX security extension Automatic Acoustic Management feature set * Device Configuration Overlay feature set * SMART error logging * SMART self-test Security: supported not enabled not locked frozen not expired: security count not supported: enhanced erase HW reset results: CBLID- above Vih Device num = 0 determined by CSEL Checksum: correct All three Acoustic management settings of 0, 128, 254 were tried. 128 made it slightly slower still, 0 or 254 were apparently equivalent, and it was at 0 to start with. (The only difference between 0 and 254 is that the former unchecks the "Automatic Acoustic Management feature set" line.) ariel sabiguero yawelak wrote: >I found a situation pretty similar to this a few years ago in a system >with shared video memory. The system has a separate graphics card. As a final shot the case was opened again and all of the following checked, none of which made any difference and/or were different from other systems: 1. motherboard and disk jumpers 2. IDE cable 3. voltage on the power connector to the drive 4. checked power supply with two testers (both showed PS in spec) 5. cleared the BIOS with the CMOS jumper, loaded defaults, changed the few settings that were not at default to match the other systems. 6. moved cable from first IDE primary IDE to secondary IDE socket 7. Observed the exposed running disk. The amount of vibration and temperature were typical, and there were no unusual noises. Ran the Stream benchmark on both the slow and normal systems and it scored the same. The hdparm -T test is also the same on the slow system as on the fast ones, it is only -t which is slow. Seems like there is something on the disk itself which is slow and it isn't a CPU speed, memory speed, or even IDE bus speed issue. The disk is apparently going, but it is an odd way to fail. I almost wonder if it has not stepped down from 7200 RPM to 5400 RPM. That ratio is 0.75, and the speed ratio in the dd test was .72, which is pretty close. It spins up with no problems though. Thanks all, David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From timattox at open-mpi.org Thu Mar 6 12:59:26 2008 From: timattox at open-mpi.org (Tim Mattox) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Re: Setting up a new Beowulf cluster In-Reply-To: <5721d9d70802190700m2a356e5fo3d637308e2d9a34d@mail.gmail.com> References: <5721d9d70802190700m2a356e5fo3d637308e2d9a34d@mail.gmail.com> Message-ID: One important consideration that I didn't see mentioned when skimming this thread on the beowulf list, is what software to use to manage the cluster. You will save yourself a LOT of time using good cluster management software. I recommend PERCEUS (http://www.perceus.org/) but there are others you can try, such as OSCAR (http://oscar.openclustergroup.org/). On Tue, Feb 19, 2008 at 10:00 AM, Berkley Starks wrote: > Thank you all for the help and support here. With what has been presented > here, and sound considerations, we have decided on a home for our Beowulf > cluster. The room is already sound proofed, and well air conditioned. As > for people worrying about noise, it will be housed with out vacuum chamber, > so those going into the room and doing stuff are already used to a little > bit of noise. > > The floor is rated to hold more than enough computers and the AC in there is > phenomenal. I just finished meeting with campus physical facilities the > other day and have got the budget requestioned and approved to allow us > independent AC control of the room. > > Right now we are seeing how much money can be appropriated for the actual > construction of the cluster. > > Thank you all so much for your input and support so far. It has helped a > lot. > > Berkley Starks > > > > > On Feb 14, 2008 9:39 AM, David Mathog wrote: > > > > Jim Lux wrote: > > > > > >>quiet down a rack because to first order sound insulation == heat > > > >>insulation. \ > > > > > > Actually, no.. good acoustic isolation is not good thermal > > > isolation. Sure, things like fiberglass batts provide thermal > > > insulation and also (slightly) attenuate high frequencies. > > > > I guess I should have used => or some other "implies". Sound insulators > > tend to be good heat insulators, heat insulators are generally not good > > sound insulators. > > > > I spent way too long trying to quiet down a rack when it had to live in > > a classroom. Mass loaded vinyl on all 4 sides worked fairly well > > to stop the noise coming out that way, but then it just turned into a > > big speaker enclosure and directed nearly as much sound out the fan > > holes, where it bounced off the ceiling and floor. And the rack exhaust > > fans (2 very high capacity 120mm fans on the top) were not able to keep > > it cool when it was fully sound insulated. The rated capacity > > of those two fans was more than the sum of all the little ones in the > > nodes, but the air flow was too restricted, I think mostly by the narrow > > space between the node's front panels and the front insulator panel. > > Thankfully it finally moved to a machine room and the noise problem went > > away. > > > > Anyway, it is a much easier to sound insulate a room than it is a single > > noisy rack. > > > > > > David Mathog > > mathog@caltech.edu > > Manager, Sequence Analysis Facility, Biology Division, Caltech > > _______________________________________________ > > > > > > > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -- Tim Mattox, Ph.D. - http://homepage.mac.com/tmattox/ tmattox@gmail.com || timattox@open-mpi.org I'm a bright... http://www.the-brights.net/ From peter.skomoroch at gmail.com Fri Mar 7 08:21:00 2008 From: peter.skomoroch at gmail.com (Peter Skomoroch) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] VMC - Virtual Machine Console Message-ID: I used Doug's BPS package to benchmark a virtual cluster on Amazon EC2, and was hoping the beowulf list could give their feedback on the results and feasibility of using this for on-demand clusters. This approach is currently being used to run some MPI code which is tolerant of poor latency, i.e. MPIBlast, monte carlo runs, etc. You get gigabit ethernet on EC2, but the latency from netpipe seems to be an order of magnitude higher than Doug's Kronos example on the Cluster Monkey page: Amazon EC2 Latency: 0.000492 (microseconds) Kronos Latency: 0.000029 (microseconds) Full Results/Charts for a "small" cluster of two extra-large nodes here (I just used the default BPS config with MPICH2): http://www.datawrangling.com/media/BPS-AmazonEC2-xlarge-run-1/index.html http://www.datawrangling.com/media/BPS-AmazonEC2-xlarge-run-2/index.html The unixbench results are misleading on VM, so I left those out. Others have verified the performance mentioned in the EC2 documentation: "One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor." Some bonnie results are here: http://blog.dbadojo.com/2007/10/bonnie-io-benchmark-vs-ec2.html The cluster is launched and configured using some python scripts and a custom beowulf Amazon Machine Image (AMI), which is basically a Xen image configured to run on EC2. You end up paying 80 cents/hour for 8 cores with15GB RAM, and can scale that up to 100 or more if you need to. I'm cleaning up the code, and will post it on my blog if anyone wants to try it out. I think this could be a cost effective path for people, who for whatever reason, can't build/use a dedicated cluster. Here are the specifications for each instance: Extra Large Instance: 15 GB memory 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each) 1,690 GB instance storage (4 x 420 GB plus 10 GB root partition) 64-bit platform I/O Performance: High Price: $0.80 per instance hour -Pete > There are plenty of parallel chores that are tolerant of poor latency -- > the whole world of embarrassingly parallel computations plus some > extension up into merely coarse grained, not terribly synchronous real > parallel computations. > > VMs can also be wonderful for TEACHING clustering and for managing > "political" problems. ... Having any sort of access to a high-latency Linux VM > node running on a Windows box beats the hell out of having no node at > all or having to port one's code to work under Windows. > > > > We can therefore see that there are clearly environments where the bulk > of the work being done is latency tolerant and where VMs may well have > benefits in administration and security and fault tolerance and local > politics that make them a great boon in clustering, just as there are > without question computations for which latency is the devil and any > suggestion of adding a layer of VM latency on top of what is already > inherent to the device and minimal OS will bring out the peasants with > pitchforks and torches. Multiboot systems, via grub and local > provisioning or PXE and remote e.g. NFS provisioning is also useful but > is not always politically possible or easy to set up. > > It is my hope that folks working on both sorts of multienvironment > provisioning and sysadmin environments work hard and produce spectacular > tools. I've done way more work than I care to setting up both of these > sorts of things. It is not easy, and requires a lot of expertise. > Hiding this detail and expertise from the user would be a wonderful > contribution to practical clustering (and of course useful in the HA > world as well). > > -- Peter N. Skomoroch peter.skomoroch@gmail.com http://www.datawrangling.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080307/db2f89e3/attachment.html From landman at scalableinformatics.com Fri Mar 7 08:57:35 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] VMC - Virtual Machine Console In-Reply-To: References: Message-ID: <47D173FF.6030001@scalableinformatics.com> Peter Skomoroch wrote: > Extra Large Instance: > > 15 GB memory > 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each) > 1,690 GB instance storage (4 x 420 GB plus 10 GB root partition) > 64-bit platform > I/O Performance: High Note: minor criticism, but overall, nice results. Looking over your bonnie results is worth a quick comment. Any time you have bonnie or IOzone (or other IO benchmarks) which are testing file sizes less than ram size, you are not actually measuring disk IO. This is cache speed pure and simple. Either page/buffer cache, or RAID cache, or whatever. We have had people tell us to our face that their 2GB file results (on a 16 GB RAM machine) were somehow indicative of real file performance, when, if they walked over to the units they were testing, they would have noticed the HD lights simply not blinking ... Yeah, an amusing beer story (the longer version of it), but a problem none-the-less. Your 1GB file data is likely more representative, but with 15 GB ram, you need to be testing 30-60 GB files. Not trying to be a marketing guy here or anything like that ... we test our JackRabbit units with 80GB to 1.3TB sized files. We see (sustained) 750 MB/s - 1.3 GB/s in these tests. We also note some serious issues with the linux buffer cache and multiple RAID controllers (buffer cache appears to serialize access). We do this as we actually want to measure disk performance, and not buffer cache performance. That criticism aside, nice results. It shows what a "cloud" can do. > Price: $0.80 per instance hour -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From peter.skomoroch at gmail.com Fri Mar 7 09:05:03 2008 From: peter.skomoroch at gmail.com (Peter Skomoroch) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] VMC - Virtual Machine Console In-Reply-To: <47D173FF.6030001@scalableinformatics.com> References: <47D173FF.6030001@scalableinformatics.com> Message-ID: Joe, thanks for the feedback. The bonnie results were not actually mine, I was just pointing to some numbers run by Paul Moen. Your 1GB file data is likely more representative, but with 15 GB ram, > you need to be testing 30-60 GB files. > I'll try to tweak the BPS bonnie tests to run some large files... On Fri, Mar 7, 2008 at 11:57 AM, Joe Landman < landman@scalableinformatics.com> wrote: > Peter Skomoroch wrote: > > > Extra Large Instance: > > > > 15 GB memory > > 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units > each) > > 1,690 GB instance storage (4 x 420 GB plus 10 GB root partition) > > 64-bit platform > > I/O Performance: High > > Note: minor criticism, but overall, nice results. > > Looking over your bonnie results is worth a quick comment. Any time you > have bonnie or IOzone (or other IO benchmarks) which are testing file > sizes less than ram size, you are not actually measuring disk IO. This > is cache speed pure and simple. Either page/buffer cache, or RAID > cache, or whatever. > > We have had people tell us to our face that their 2GB file results (on a > 16 GB RAM machine) were somehow indicative of real file performance, > when, if they walked over to the units they were testing, they would > have noticed the HD lights simply not blinking ... Yeah, an amusing > beer story (the longer version of it), but a problem none-the-less. > > Your 1GB file data is likely more representative, but with 15 GB ram, > you need to be testing 30-60 GB files. > > Not trying to be a marketing guy here or anything like that ... we test > our JackRabbit units with 80GB to 1.3TB sized files. We see (sustained) > 750 MB/s - 1.3 GB/s in these tests. We also note some serious issues > with the linux buffer cache and multiple RAID controllers (buffer cache > appears to serialize access). We do this as we actually want to measure > disk performance, and not buffer cache performance. > > That criticism aside, nice results. It shows what a "cloud" can do. > > > Price: $0.80 per instance hour > > > -- > Joseph Landman, Ph.D > Founder and CEO > Scalable Informatics LLC, > email: landman@scalableinformatics.com > web : http://www.scalableinformatics.com > http://jackrabbit.scalableinformatics.com > phone: +1 734 786 8423 > fax : +1 866 888 3112 > cell : +1 734 612 4615 > -- Peter N. Skomoroch peter.skomoroch@gmail.com http://www.datawrangling.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080307/a043bfa0/attachment.html From peter.skomoroch at gmail.com Fri Mar 7 09:49:46 2008 From: peter.skomoroch at gmail.com (Peter Skomoroch) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] VMC - Virtual Machine Console In-Reply-To: References: <47D173FF.6030001@scalableinformatics.com> Message-ID: I'm running bonnie++ on a xlarge instance right now with 30 GB files on /mnt. I'll post the results when it finishes. I also have Ganglia set up on the node, so you can check that out until I shut the instance down: http://ec2-72-44-53-20.compute-1.amazonaws.com/ganglia On Fri, Mar 7, 2008 at 12:05 PM, Peter Skomoroch wrote: > Joe, thanks for the feedback. The bonnie results were not actually mine, > I was just pointing to some numbers run by Paul Moen. > > Your 1GB file data is likely more representative, but with 15 GB ram, > > you need to be testing 30-60 GB files. > > > > I'll try to tweak the BPS bonnie tests to run some large files... > > > > On Fri, Mar 7, 2008 at 11:57 AM, Joe Landman < > landman@scalableinformatics.com> wrote: > > > Peter Skomoroch wrote: > > > > > Extra Large Instance: > > > > > > 15 GB memory > > > 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units > > each) > > > 1,690 GB instance storage (4 x 420 GB plus 10 GB root partition) > > > 64-bit platform > > > I/O Performance: High > > > > Note: minor criticism, but overall, nice results. > > > > Looking over your bonnie results is worth a quick comment. Any time you > > have bonnie or IOzone (or other IO benchmarks) which are testing file > > sizes less than ram size, you are not actually measuring disk IO. This > > is cache speed pure and simple. Either page/buffer cache, or RAID > > cache, or whatever. > > > > We have had people tell us to our face that their 2GB file results (on a > > 16 GB RAM machine) were somehow indicative of real file performance, > > when, if they walked over to the units they were testing, they would > > have noticed the HD lights simply not blinking ... Yeah, an amusing > > beer story (the longer version of it), but a problem none-the-less. > > > > Your 1GB file data is likely more representative, but with 15 GB ram, > > you need to be testing 30-60 GB files. > > > > Not trying to be a marketing guy here or anything like that ... we test > > our JackRabbit units with 80GB to 1.3TB sized files. We see (sustained) > > 750 MB/s - 1.3 GB/s in these tests. We also note some serious issues > > with the linux buffer cache and multiple RAID controllers (buffer cache > > appears to serialize access). We do this as we actually want to measure > > disk performance, and not buffer cache performance. > > > > That criticism aside, nice results. It shows what a "cloud" can do. > > > > > Price: $0.80 per instance hour > > > > > > -- > > Joseph Landman, Ph.D > > Founder and CEO > > Scalable Informatics LLC, > > email: landman@scalableinformatics.com > > web : http://www.scalableinformatics.com > > http://jackrabbit.scalableinformatics.com > > phone: +1 734 786 8423 > > fax : +1 866 888 3112 > > cell : +1 734 612 4615 > > > > > > -- > Peter N. Skomoroch > peter.skomoroch@gmail.com > http://www.datawrangling.com > -- Peter N. Skomoroch peter.skomoroch@gmail.com http://www.datawrangling.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080307/760a28fd/attachment.html From peter.st.john at gmail.com Fri Mar 7 10:18:49 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] VMC - Virtual Machine Console In-Reply-To: References: <47D173FF.6030001@scalableinformatics.com> Message-ID: Peter Sk, Where is the blog you mentioned (where you'll be posting followups)? Thanks, Peter St On Fri, Mar 7, 2008 at 12:49 PM, Peter Skomoroch wrote: > I'm running bonnie++ on a xlarge instance right now with 30 GB files on > /mnt. I'll post the results when it finishes. I also have Ganglia set up > on the node, so you can check that out until I shut the instance down: > > http://ec2-72-44-53-20.compute-1.amazonaws.com/ganglia > > > On Fri, Mar 7, 2008 at 12:05 PM, Peter Skomoroch < > peter.skomoroch@gmail.com> wrote: > > > Joe, thanks for the feedback. The bonnie results were not actually > > mine, I was just pointing to some numbers run by Paul Moen. > > > > Your 1GB file data is likely more representative, but with 15 GB ram, > > > you need to be testing 30-60 GB files. > > > > > > > I'll try to tweak the BPS bonnie tests to run some large files... > > > > > > > > On Fri, Mar 7, 2008 at 11:57 AM, Joe Landman < > > landman@scalableinformatics.com> wrote: > > > > > Peter Skomoroch wrote: > > > > > > > Extra Large Instance: > > > > > > > > 15 GB memory > > > > 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units > > > each) > > > > 1,690 GB instance storage (4 x 420 GB plus 10 GB root > > > partition) > > > > 64-bit platform > > > > I/O Performance: High > > > > > > Note: minor criticism, but overall, nice results. > > > > > > Looking over your bonnie results is worth a quick comment. Any time > > > you > > > have bonnie or IOzone (or other IO benchmarks) which are testing file > > > sizes less than ram size, you are not actually measuring disk IO. > > > This > > > is cache speed pure and simple. Either page/buffer cache, or RAID > > > cache, or whatever. > > > > > > We have had people tell us to our face that their 2GB file results (on > > > a > > > 16 GB RAM machine) were somehow indicative of real file performance, > > > when, if they walked over to the units they were testing, they would > > > have noticed the HD lights simply not blinking ... Yeah, an amusing > > > beer story (the longer version of it), but a problem none-the-less. > > > > > > Your 1GB file data is likely more representative, but with 15 GB ram, > > > you need to be testing 30-60 GB files. > > > > > > Not trying to be a marketing guy here or anything like that ... we > > > test > > > our JackRabbit units with 80GB to 1.3TB sized files. We see > > > (sustained) > > > 750 MB/s - 1.3 GB/s in these tests. We also note some serious issues > > > with the linux buffer cache and multiple RAID controllers (buffer > > > cache > > > appears to serialize access). We do this as we actually want to > > > measure > > > disk performance, and not buffer cache performance. > > > > > > That criticism aside, nice results. It shows what a "cloud" can do. > > > > > > > Price: $0.80 per instance hour > > > > > > > > > -- > > > Joseph Landman, Ph.D > > > Founder and CEO > > > Scalable Informatics LLC, > > > email: landman@scalableinformatics.com > > > web : http://www.scalableinformatics.com > > > http://jackrabbit.scalableinformatics.com > > > phone: +1 734 786 8423 > > > fax : +1 866 888 3112 > > > cell : +1 734 612 4615 > > > > > > > > > > > -- > > Peter N. Skomoroch > > peter.skomoroch@gmail.com > > http://www.datawrangling.com > > > > > > -- > Peter N. Skomoroch > peter.skomoroch@gmail.com > http://www.datawrangling.com > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080307/d378d961/attachment.html From peter.skomoroch at gmail.com Fri Mar 7 10:30:11 2008 From: peter.skomoroch at gmail.com (Peter Skomoroch) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] VMC - Virtual Machine Console In-Reply-To: References: <47D173FF.6030001@scalableinformatics.com> Message-ID: My blog is here: http://www.datawrangling.com/ I had a post last year describing launching a cluster of the small 32 bit instances: http://www.datawrangling.com/mpi-cluster-with-python-and-amazon-ec2-part-2-of-3.html Since then, Amazon upgraded to 64 bit "extra-large" instances with full gigabit ethernet which might make this feasible. Essentially you are getting a full physical box, where with the small instances you would be sharing network, disk, etc. I built a set of new Fedora images and config scripts for the extra-large instances which include NFS,mpich,lam,openmpi,ganglia etc. I'd like to use a standard cluster distribution, but it would take some hacking to get the networking to work properly within EC2. Amazon uses a custom firewall setup where autodiscovery won't work, also multicast is not supported and subnets are randomly assigned. On Fri, Mar 7, 2008 at 1:18 PM, Peter St. John wrote: > Peter Sk, > Where is the blog you mentioned (where you'll be posting followups)? > Thanks, > Peter St > > On Fri, Mar 7, 2008 at 12:49 PM, Peter Skomoroch < > peter.skomoroch@gmail.com> wrote: > > > I'm running bonnie++ on a xlarge instance right now with 30 GB files on > > /mnt. I'll post the results when it finishes. I also have Ganglia set up > > on the node, so you can check that out until I shut the instance down: > > > > http://ec2-72-44-53-20.compute-1.amazonaws.com/ganglia > > > > > > On Fri, Mar 7, 2008 at 12:05 PM, Peter Skomoroch < > > peter.skomoroch@gmail.com> wrote: > > > > > Joe, thanks for the feedback. The bonnie results were not actually > > > mine, I was just pointing to some numbers run by Paul Moen. > > > > > > Your 1GB file data is likely more representative, but with 15 GB ram, > > > > you need to be testing 30-60 GB files. > > > > > > > > > > I'll try to tweak the BPS bonnie tests to run some large files... > > > > > > > > > > > > On Fri, Mar 7, 2008 at 11:57 AM, Joe Landman < > > > landman@scalableinformatics.com> wrote: > > > > > > > Peter Skomoroch wrote: > > > > > > > > > Extra Large Instance: > > > > > > > > > > 15 GB memory > > > > > 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute > > > > Units each) > > > > > 1,690 GB instance storage (4 x 420 GB plus 10 GB root > > > > partition) > > > > > 64-bit platform > > > > > I/O Performance: High > > > > > > > > Note: minor criticism, but overall, nice results. > > > > > > > > Looking over your bonnie results is worth a quick comment. Any time > > > > you > > > > have bonnie or IOzone (or other IO benchmarks) which are testing > > > > file > > > > sizes less than ram size, you are not actually measuring disk IO. > > > > This > > > > is cache speed pure and simple. Either page/buffer cache, or RAID > > > > cache, or whatever. > > > > > > > > We have had people tell us to our face that their 2GB file results > > > > (on a > > > > 16 GB RAM machine) were somehow indicative of real file performance, > > > > when, if they walked over to the units they were testing, they would > > > > have noticed the HD lights simply not blinking ... Yeah, an amusing > > > > beer story (the longer version of it), but a problem none-the-less. > > > > > > > > Your 1GB file data is likely more representative, but with 15 GB > > > > ram, > > > > you need to be testing 30-60 GB files. > > > > > > > > Not trying to be a marketing guy here or anything like that ... we > > > > test > > > > our JackRabbit units with 80GB to 1.3TB sized files. We see > > > > (sustained) > > > > 750 MB/s - 1.3 GB/s in these tests. We also note some serious > > > > issues > > > > with the linux buffer cache and multiple RAID controllers (buffer > > > > cache > > > > appears to serialize access). We do this as we actually want to > > > > measure > > > > disk performance, and not buffer cache performance. > > > > > > > > That criticism aside, nice results. It shows what a "cloud" can do. > > > > > > > > > Price: $0.80 per instance hour > > > > > > > > > > > > -- > > > > Joseph Landman, Ph.D > > > > Founder and CEO > > > > Scalable Informatics LLC, > > > > email: landman@scalableinformatics.com > > > > web : http://www.scalableinformatics.com > > > > http://jackrabbit.scalableinformatics.com > > > > phone: +1 734 786 8423 > > > > fax : +1 866 888 3112 > > > > cell : +1 734 612 4615 > > > > > > > > > > > > > > > > -- > > > Peter N. Skomoroch > > > peter.skomoroch@gmail.com > > > http://www.datawrangling.com > > > > > > > > > > > -- > > Peter N. Skomoroch > > peter.skomoroch@gmail.com > > http://www.datawrangling.com > > > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > -- Peter N. Skomoroch peter.skomoroch@gmail.com http://www.datawrangling.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080307/383d5e5b/attachment.html From peter.skomoroch at gmail.com Fri Mar 7 11:07:51 2008 From: peter.skomoroch at gmail.com (Peter Skomoroch) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] VMC - Virtual Machine Console In-Reply-To: References: <47D173FF.6030001@scalableinformatics.com> Message-ID: Here are the results from one pass: [root@ip-10-251-166-127 bonnie++-1.03a]# ./bonnie++ -d /mnt/bonnie -s 30000 -n 1 -m Fedora6 -x 3 -r 15000 -u lamuser Using uid:500, gid:500. name,file_size,putc,putc_cpu,put_block,put_block_cpu,rewrite,rewrite_cpu,getc,getc_cpu,get_block,get_block_cpu,seeks,seeks_cpu,num_files,seq_create,seq_create_cpu,seq_stat,seq_stat_cpu,seq_del,seq_del_cpu,ran_create,ran_create_cpu,ran_stat,ran_stat_cpu,ran_del,ran_del_cpu Writing with putc()...done Writing intelligently...done Rewriting...done Reading with getc()...done Reading intelligently...done start 'em...done...done...done... Create files in sequential order...done. Stat files in sequential order...done. Delete files in sequential order...done. Create files in random order...done. Stat files in random order...done. Delete files in random order...done. Fedora6,30000M,14984,28,54544,15,22190,1,41469,58,55526,0,176.0 ,0,1,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ On Fri, Mar 7, 2008 at 12:49 PM, Peter Skomoroch wrote: > I'm running bonnie++ on a xlarge instance right now with 30 GB files on > /mnt. I'll post the results when it finishes. I also have Ganglia set up > on the node, so you can check that out until I shut the instance down: > > http://ec2-72-44-53-20.compute-1.amazonaws.com/ganglia > > > On Fri, Mar 7, 2008 at 12:05 PM, Peter Skomoroch < > peter.skomoroch@gmail.com> wrote: > > > Joe, thanks for the feedback. The bonnie results were not actually > > mine, I was just pointing to some numbers run by Paul Moen. > > > > Your 1GB file data is likely more representative, but with 15 GB ram, > > > you need to be testing 30-60 GB files. > > > > > > > I'll try to tweak the BPS bonnie tests to run some large files... > > > > > > > > On Fri, Mar 7, 2008 at 11:57 AM, Joe Landman < > > landman@scalableinformatics.com> wrote: > > > > > Peter Skomoroch wrote: > > > > > > > Extra Large Instance: > > > > > > > > 15 GB memory > > > > 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units > > > each) > > > > 1,690 GB instance storage (4 x 420 GB plus 10 GB root > > > partition) > > > > 64-bit platform > > > > I/O Performance: High > > > > > > Note: minor criticism, but overall, nice results. > > > > > > Looking over your bonnie results is worth a quick comment. Any time > > > you > > > have bonnie or IOzone (or other IO benchmarks) which are testing file > > > sizes less than ram size, you are not actually measuring disk IO. > > > This > > > is cache speed pure and simple. Either page/buffer cache, or RAID > > > cache, or whatever. > > > > > > We have had people tell us to our face that their 2GB file results (on > > > a > > > 16 GB RAM machine) were somehow indicative of real file performance, > > > when, if they walked over to the units they were testing, they would > > > have noticed the HD lights simply not blinking ... Yeah, an amusing > > > beer story (the longer version of it), but a problem none-the-less. > > > > > > Your 1GB file data is likely more representative, but with 15 GB ram, > > > you need to be testing 30-60 GB files. > > > > > > Not trying to be a marketing guy here or anything like that ... we > > > test > > > our JackRabbit units with 80GB to 1.3TB sized files. We see > > > (sustained) > > > 750 MB/s - 1.3 GB/s in these tests. We also note some serious issues > > > with the linux buffer cache and multiple RAID controllers (buffer > > > cache > > > appears to serialize access). We do this as we actually want to > > > measure > > > disk performance, and not buffer cache performance. > > > > > > That criticism aside, nice results. It shows what a "cloud" can do. > > > > > > > Price: $0.80 per instance hour > > > > > > > > > -- > > > Joseph Landman, Ph.D > > > Founder and CEO > > > Scalable Informatics LLC, > > > email: landman@scalableinformatics.com > > > web : http://www.scalableinformatics.com > > > http://jackrabbit.scalableinformatics.com > > > phone: +1 734 786 8423 > > > fax : +1 866 888 3112 > > > cell : +1 734 612 4615 > > > > > > > > > > > -- > > Peter N. Skomoroch > > peter.skomoroch@gmail.com > > http://www.datawrangling.com > > > > > > -- > Peter N. Skomoroch > peter.skomoroch@gmail.com > http://www.datawrangling.com > -- Peter N. Skomoroch peter.skomoroch@gmail.com http://www.datawrangling.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080307/e5938597/attachment.html From erwan at seanodes.com Sat Mar 8 15:07:02 2008 From: erwan at seanodes.com (Erwan) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Re: mysterious slow disk In-Reply-To: References: Message-ID: <47D31C16.50600@seanodes.com> David Mathog wrote: > I don't think I'm going to solve this one :-( > Here come some ideas. If you invert a disk from a good node and a bad node, does the good node becomes bad ? If yes, the disk is the problem, not the host. If no, the host is the problem. This test will help us searching in the right direction. Maybe the pci latency isn't the same, could you diff a lspci -vvv between a good and a bad host ? Could you also diff the dmidecode of a good and a bad host ? Maybe we missed something. Could you provide a "smartctl -A /dev/" output for a good and a bad disk ? Maybe their proprietary tools could diagnose something we can't see directly: http://websupport.wdc.com/rd.asp?p=sw30&t=122&lang=fr&s=http://support.wdc.com/download/dlg/Diag504cCD.iso I'm out of ideas for the moment ;o) Erwan, -------------------------------------------------------------------------------- Les opinions et prises de position emises par le signataire du present message lui sont propres et ne sauraient engager la responsabilite de la societe SEANODES. Ce message ainsi que les eventuelles pieces jointes constituent une correspondance privee et confidentielle a l'attention exclusive du destinataire designe ci-dessus. Si vous n'etes pas le destinataire du present message ou une personne susceptible de pouvoir le lui delivrer, il vous est signifie que toute divulgation, distribution ou copie de cette transmission est strictement interdite. Si vous avez recu ce message par erreur, nous vous remercions d'en informer l'expediteur par telephone ou de lui retourner le present message, puis d'effacer immediatement ce message de votre systeme. The views and opinions expressed by the author of this message are personal. SEANODES shall assume no liability, express or implied for such message. This e-mail and any attachments is a confidential correspondence intended only for use of the individual or entity named above. If you are not the intended recipient or the agent responsible for delivering the message to the intended recipient, you are hereby notified that any disclosure, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender by phone or by replying this message, and then delete this message from your system. From mathog at caltech.edu Mon Mar 10 11:10:43 2008 From: mathog at caltech.edu (David Mathog) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Re: mysterious slow disk Message-ID: >Erwan wrote > If you invert a disk from a good node and a bad node, does the good node > becomes bad ? It is on the list of things to try. > Maybe the pci latency isn't the same, could you diff a lspci -vvv > between a good and a bad host ? Already looked at that, they were the same. > Could you also diff the dmidecode of a good and a bad host ? Maybe we > missed something. That's interesting, dmidecode on the slow node shows only: # dmidecode 2.9 # No SMBIOS nor DMI entry point found, sorry. The other nodes actually have dmidecode output. dmesg shows on the slow one: DMI not present or invalid. the others show DMI 2.3 present. This is probably not the problem though, since it seems to have come up after I tried reloading the BIOS last week. Prior to that I did diff dmesg and this difference was not present. Thanks, David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From xclski at yahoo.com Wed Mar 12 20:07:40 2008 From: xclski at yahoo.com (Ellis Wilson) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Live Implementation for Clusters Message-ID: <993205.49661.qm@web37909.mail.mud.yahoo.com> Hi all, I've been working lately developing a LiveCD/LiveUSB which will dynamically bring up a cluster for a few professors at my university. Their issue is that this (LaSalle U.) being largely a business and liberal arts university, all their funds go towards instruments not geared towards computers (or really research in general, this is largely a "teaching" school). However, a few are really into research in Chemistry (specifically using ORCA, Gaussian is too expensive and actually doesn't do what they need). They've run their tasks in the past for weeks at a time on their home computers or old computers turned over by the school following student use. I asked, why not use the computers for the students when they can't use them? This sparked my LiveCD endeavor. Now, granted, using gigabit ethernet is a huge drawback (and again, money is the real issue, so I can't buy a switch and install nice NICs into the computers for my own use in those off hours), but the task at hand won't even allow propagation beyond 12 computers (hard coded limitation in ORCA). Therefore, our speedup is somewhat CPU bound (at night, obviously a non-dedicated switch during the day would create really terrible problems since students are mucking around on it simultaneously). I'd like to hear concerns/comments from the community on this. For reference I've built the CD based on a slightly stripped down Gentoo, but kept it fairly run of the mill so I can use it for some other applications afterwards by simply unmerging the chemistry applications and reburning it with my new configuration. MPI 1.27 is utilized because ORCA (again very picky and unfortunately closed source) requires it. I've got a small folder of scripts that create the computer as a Node or Master with more or less one command. Oh, and no, IT here is really, really bad and super Windows friendly. There's no way they'd let me install onto another partition. Feel free to rip away, I know its not a perfect solution, but I'm not sure under the heavy money circumstances a better one exists (please prove me wrong!). Ellis ____________________________________________________________________________________ Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ From john.hearns at streamline-computing.com Wed Mar 12 23:20:31 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Live Implementation for Clusters In-Reply-To: <993205.49661.qm@web37909.mail.mud.yahoo.com> References: <993205.49661.qm@web37909.mail.mud.yahoo.com> Message-ID: <1205389241.10217.7.camel@Vigor13> On Wed, 2008-03-12 at 20:07 -0700, Ellis Wilson wrote: > Hi all, > > > Feel free to rip away, Why should we? This sounds an exciting and innovative project. Write it up. Doug Eadline - sounds like a perfect article for the Clustermonkey site. Actually, your post is quite topical for me. I'm doing a talk on Saturday, and was looking at the Cluster Knoppix site, as I'll probably be asked about how to go about getting a taster of cluster building. http://clusterknoppix.sw.be/ I was a bit surprised how out of date this is. Best of British with the project, and as I say write it up. John Hearns Senior HPC Engineer Streamline Computing From john.hearns at streamline-computing.com Wed Mar 12 23:42:21 2008 From: john.hearns at streamline-computing.com (John Hearns) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Live Implementation for Clusters In-Reply-To: <993205.49661.qm@web37909.mail.mud.yahoo.com> References: <993205.49661.qm@web37909.mail.mud.yahoo.com> Message-ID: <1205390551.10217.23.camel@Vigor13> On Wed, 2008-03-12 at 20:07 -0700, Ellis Wilson wrote: > This sparked my LiveCD endeavor. Now, granted, using > gigabit ethernet > is a huge drawback Hey, don't say that. We build and install clusters for customers every day with gigabit ethernet as the main interconnect. (And Infiniband; Infinipath; Myrinet; Quadrics or anything else your budget will stretch to). An enhancement you might like to think of is wiring any second on-board gigabit ports to a dedicated switch for MPI traffic. (See Beowulf passim for discussions on the merits of various switches) People are using our gig ethernet connected clusters right now for serious computational chemistry work all over the UK. John Hearns Senior HPC Engineer Streamline Computing From xclski at yahoo.com Thu Mar 13 07:43:42 2008 From: xclski at yahoo.com (Ellis Wilson) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Live Implementation for Clusters Message-ID: <933254.46423.qm@web37910.mail.mud.yahoo.com> > An enhancement you might like to think of is wiring any second on-board > gigabit ports to a dedicated switch for MPI traffic. > (See Beowulf passim for discussions on the merits of various switches) Actually, its funny you say that because as of right now the script serves in a (very rough) DHCP capacity that seems to work best on a dedicated switch. My work today was to change that, but it might be good to make two functions so one might choose. Anyhow, the way that it currently works in my script is: 1. Fire up the master node, which configures itself as 192.168.0.250 and starts an NFS server in its home folder as .ssh 2. Fire up each node, running the script, which gets an IP from the standard DHCP server, connects to the NFS server, grabs a file called IP from it and assigns its IP based on that and increments the IP file on the masters NFS server. Unmount NFS, restart network, remount NFS, run SSH passwordless stuff (which is placed into the .ssh folder, handily available now to any PC connected to the NFS server) and is ready to go. I was going to change this so that it doesn't grab an IP file but rather simply appends its standard assigned ip address to a file on the Masters NFS server and when all your nodes are up, one issues a command from the Master to refresh all the Hosts and Machine files on the nodes and master using this file with all the random ip addresses in it (obviously Node1 in hosts is the first IP, and so on). This way I can take advantage of my school using a non 192.168.***.*** network. I tried the old system using this, the master sets its ip as 192.168.0.250, and yet the nodes don't seem to find it. Not sure why. Ellis ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping From garantes at iq.usp.br Fri Mar 14 05:55:03 2008 From: garantes at iq.usp.br (Guilherme Menegon Arantes) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] some tips on new cluster hardware Message-ID: <20080314125503.GA9770@dinamobile> Hi folks, We are buindling a new baby and would like some tips on inexpensive hardware. Any help is much appreciated: 1) Suggestions on MoBo's that support a 1333MHz FSB to plug Intel's Q9450, with video and jumbo-frames capable Gigabit on-board; 2) NIC PCI Gigabit card, jumbo-frames capable (because I might be asking too much on item 1); 3) 16-port Gigabit Switch, jumbo-frames capable, with Flow Control. The most likely brands we can find here in Brazil are Intel, 3COM, D-Link, BCM and TrendWare. Thanks a lot for your opinions/hints, G -- Guilherme Menegon Arantes, PhD S?o Paulo, Brasil ______________________________________________________ From laytonjb at charter.net Sat Mar 15 16:27:00 2008 From: laytonjb at charter.net (Jeffrey B. Layton) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] some tips on new cluster hardware In-Reply-To: <20080314125503.GA9770@dinamobile> References: <20080314125503.GA9770@dinamobile> Message-ID: <47DC5B44.40907@charter.net> Guilherme Menegon Arantes wrote: > Hi folks, > > We are buindling a new baby and would like some tips on inexpensive > hardware. Any help is much appreciated: > > 1) Suggestions on MoBo's that support a 1333MHz FSB to plug Intel's > Q9450, with video and jumbo-frames capable Gigabit on-board; > To get a good idea, look at www.newegg.com and look for motherboards that people like and also have a fair number of reviews. You might want to think about looking at a newer chipset instead of really old ones while you're at it (just my opinion). > 2) NIC PCI Gigabit card, jumbo-frames capable (because I might be > asking too much on item 1); > I would think about using a simple PCI-e x1 GigE card. Intel has a good one that's not too expensive. Doug Eadline has done some testing and these NICs look a bit better than simple PCI NICs. Jeff From hahn at mcmaster.ca Sun Mar 16 15:25:47 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] some tips on new cluster hardware In-Reply-To: <20080314125503.GA9770@dinamobile> References: <20080314125503.GA9770@dinamobile> Message-ID: > 1) Suggestions on MoBo's that support a 1333MHz FSB to plug Intel's > Q9450, with video and jumbo-frames capable Gigabit on-board; > 2) NIC PCI Gigabit card, jumbo-frames capable (because I might be > asking too much on item 1); > 3) 16-port Gigabit Switch, jumbo-frames capable, with Flow Control. it sounds to me as if you've done some research, and just want help shopping. how thorough was your research? often people start with preconceptions that limit the range of solutions they look at, unnecessarily. for instance, you have clearly decided you want Gb+jumbo - how did you decide that? jumbo is mainly about reducing the cpu overhead of high-bandwidth apps, which does not necessarily correspond with typical cluster workloads. if you're concerned about bandwidth, might you be better off with some other interconnect? IB has gotten a lot cheaper recently. to me, cluster hardware should really take one of two approaches: high road or low. the high road includes IB or 10Ge, and tends to also include rackmount and dual or quad socket. this is approach provides an excellent platform for demanding (high-bandwidth and/or tight-coupled) workloads. the low road is more "beowulfy" - to leverage commodity parts. that means gigabit, and excludes IB. but it also means PC-style cases, probably single-socket mATX boards with integrated video and gbit. these parts are a LOT cheaper than high-road "server" boards. main downsides of the low road: - no real managability. cheap boards don't have IPMI, though nowadays they will at least PXE boot. - at most 4cores and one memory controller per node. - gigabit isn't exactly low-latency or high-bandwidth, so this kind of cluster is mostly appropriate for serial or small/loose parallel jobs. > The most likely brands we can find here in Brazil are Intel, 3COM, > D-Link, BCM and TrendWare. I think you should look closely at using motherboard-integrated gigabit. besides saving cost, it reduces the complexity of the system, and if you're feeling DIYish, you can mount boards caseless for extra savings (and often better control of airflow!) I also think you should avoid falling in love with your ethernet switch ;) I just checked the price on the DGS-1216t, which appears to be what you're looking for, and it's $Cdn 270 or so (cheaper than one cpu). you should probably consider getting a modestly larger switch - 24 is still commodity, and would let you expand or add a trunked uplink to fileserver(s). From garantes at iq.usp.br Sun Mar 16 17:27:38 2008 From: garantes at iq.usp.br (Guilherme Menegon Arantes) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] some tips on new cluster hardware In-Reply-To: References: <20080314125503.GA9770@dinamobile> Message-ID: <20080317002738.GA4265@dinamobile> On Sun, Mar 16, 2008 at 06:25:47PM -0400, Mark Hahn wrote: > > it sounds to me as if you've done some research, and just want help > shopping. Yes. We don't have much cheap hardware options around here. Nor well-educated sellers that can help finding solutions, so I was just asking to see if you could give me some ideas. > how thorough was your research? often people start with preconceptions > that limit the range of solutions they look at, unnecessarily. for > instance, > you have clearly decided you want Gb+jumbo - how did you decide that? Based on previous experience. It helps considerably with one of our applications, which has loads of disk I/O (previous and new cluster will be diskless). > if you're concerned about bandwidth, might you be better off with some > other interconnect? IB has gotten a lot cheaper recently. We are aware of that. Actually, we are in touch with Colfax to import some chips. > to me, cluster hardware should really take one of two approaches: > high road or low. the high road includes IB or 10Ge, and tends to Agreed. We are in the low side at the mo, because of scarce funds and little experience. But, want to learn/get experience (with IB, for example) and move towards the higher end road. > feeling DIYish, you can mount boards caseless for extra savings (and often > better control of airflow!) No, thanks. We are going rackmount. > I also think you should avoid falling in love with your ethernet switch ;) Don't worry, I won't change my girlfriend for that ;-), but a 24 port (as suggested by somelse on Beowulf before) switch is something I am considering. Just a bit unsure of D-Link brand, since I had problems with it before (wireless ap and NIC). Thanks for the suggestions/attention, G -- Guilherme Menegon Arantes, PhD S?o Paulo, Brasil ______________________________________________________ From garantes at iq.usp.br Sun Mar 16 17:30:26 2008 From: garantes at iq.usp.br (Guilherme Menegon Arantes) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] some tips on new cluster hardware In-Reply-To: <47DC5B44.40907@charter.net> References: <20080314125503.GA9770@dinamobile> <47DC5B44.40907@charter.net> Message-ID: <20080317003026.GB4265@dinamobile> On Sat, Mar 15, 2008 at 06:27:00PM -0500, Jeffrey B. Layton wrote: > > I would think about using a simple PCI-e x1 GigE card. Intel has a good > one that's not too expensive. Doug Eadline has done some testing and > these NICs look a bit better than simple PCI NICs. URL? I searched Clustermonkey withou success... Thanks, G -- Guilherme Menegon Arantes, PhD S?o Paulo, Brasil ______________________________________________________ From midair77 at gmail.com Mon Mar 17 11:49:14 2008 From: midair77 at gmail.com (Steven Truong) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] NMI (Non maskable interrupts) Message-ID: <28bb77d30803171149p396f515cs124f1ce045ceeea7@mail.gmail.com> Dear, all. We recently bought some dual quadcore AMD Barcelona nodes with Asus KFSN4-DRE motherboard and installed Rocks Cluster 4.3, CentOS 5.1 on these machines. What we found have irked us in terms of the number of NMI generated. #cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 11468997 11550969 11551029 11550982 11550374 11549932 11549991 11553108 IO-APIC-edge timer 8: 0 0 0 0 0 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 0 0 0 0 IO-APIC-level acpi 177: 0 0 0 0 0 0 0 0 IO-APIC-level ohci_hcd 185: 0 0 0 0 0 0 0 0 IO-APIC-level ehci_hcd 193: 0 149313 392 38104 36736 0 1 77870 IO-APIC-level libata 201: 0 0 0 0 0 0 0 0 IO-APIC-level libata 233: 29715082 0 0 0 0 0 0 0 PCI-MSI eth0 NMI: 658519 686187 682474 687981 690017 689957 685692 588203 LOC: 92324693 92324694 92324693 92324692 92324692 92324687 92324689 92324681 ERR: 0 MIS: 0 # uptime 11:38:50 up 10 days, 16:27, 1 user, load average: 7.99, 7.98, 7.99 >From my understanding, NMI is not good since the processors really have to handle these interrupts right away and these might degrade the performance of the nodes. From what I read, NMI are usually generated by bad hardwares or memory issues and I would like to know how to find out what causes these NMI... Could you please point me to the right direction in finding out more about this? Thank you. From hahn at mcmaster.ca Mon Mar 17 15:02:24 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] NMI (Non maskable interrupts) In-Reply-To: <28bb77d30803171149p396f515cs124f1ce045ceeea7@mail.gmail.com> References: <28bb77d30803171149p396f515cs124f1ce045ceeea7@mail.gmail.com> Message-ID: > From my understanding, NMI is not good since the processors really > have to handle these interrupts right away and these might degrade the > performance of the nodes. I think you're mistaken - NMI's of the sort you're talking about will result in a panic. these NMI's are probably just low-level kernel synchronization like where one CPU needs to cause others to immediately do something like changing the status of a page in their MMUs. for instance, I notice that more recent kernels classify interrupts more finely: [root@experiment ~]# cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 68 0 0 0 IO-APIC-edge timer 1: 0 0 0 10 IO-APIC-edge i8042 4: 0 0 0 2 IO-APIC-edge 8: 0 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-fasteoi acpi 12: 0 0 0 4 IO-APIC-edge i8042 14: 0 0 0 0 IO-APIC-edge ide0 17: 0 0 0 0 IO-APIC-fasteoi sata_nv 18: 0 0 0 0 IO-APIC-fasteoi sata_nv 19: 123229 148 514 4698 IO-APIC-fasteoi sata_nv 362: 127524168 5281605 236961 121506 PCI-MSI-edge eth1 377: 519748 12731137 607115 42573852 PCI-MSI-edge eth0:MSI-X-2-RX 378: 109154 80191 302109913 6487104 PCI-MSI-edge eth0:MSI-X-1-TX NMI: 0 0 0 0 Non-maskable interrupts LOC: 300446104 300446082 300446060 300446038 Local timer interrupts RES: 2698262 44102 2234502 3677120 Rescheduling interrupts CAL: 4135 4379 4460 415 function call interrupts TLB: 14018 15088 4079 7251 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 Threshold APIC interrupts SPU: 0 0 0 0 Spurious interrupts ERR: 0 I suspect that all the counts listed after RES are, in earlier kernels, lumped into NMI. obviously, rescheduling, function call and TLB shootdowns are perfectly normal, not indicating any error (though you might want to minimize them as well...) how about trying a new kernel? the above is 2.6.24.3. note that there are important security fixes that you might be missing if you're running certain ranges of old kernels... From csamuel at vpac.org Tue Mar 18 05:53:32 2008 From: csamuel at vpac.org (csamuel@vpac.org) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] NMI (Non maskable interrupts) In-Reply-To: Message-ID: <20740902.661205814821402.JavaMail.csamuel@ubuntu> ----- "Mark Hahn" wrote: > how about trying a new kernel? the above is 2.6.24.3. note that > there are important security fixes that you might be missing if > you're running certain ranges of old kernels... We also saw performance improvements going to the mainline kernel (2.6.24.*) from the CentOS 5 kernels. Don't forget to make sure you're not disabling ACPI on Barcelona by booting with noacpi, if you do then the kernel won't learn that it's a NUMA box as the old K8 table hack doesn't work on K10h and you'll get much worse memory bandwidth. cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From midair77 at gmail.com Tue Mar 18 11:55:21 2008 From: midair77 at gmail.com (Steven Truong) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] NMI (Non maskable interrupts) In-Reply-To: References: <28bb77d30803171149p396f515cs124f1ce045ceeea7@mail.gmail.com> Message-ID: <28bb77d30803181155n3f52aeb6k6d92ac2fb5a90322@mail.gmail.com> On Mon, Mar 17, 2008 at 3:02 PM, Mark Hahn wrote: > > From my understanding, NMI is not good since the processors really > > have to handle these interrupts right away and these might degrade the > > performance of the nodes. > > I think you're mistaken - NMI's of the sort you're talking about will > result in a panic. these NMI's are probably just low-level kernel > synchronization like where one CPU needs to cause others to immediately do > something like changing the status of a page in their MMUs. > > for instance, I notice that more recent kernels classify interrupts > more finely: > > [root@experiment ~]# cat /proc/interrupts > CPU0 CPU1 CPU2 CPU3 > 0: 68 0 0 0 IO-APIC-edge timer > 1: 0 0 0 10 IO-APIC-edge i8042 > 4: 0 0 0 2 IO-APIC-edge > 8: 0 0 0 0 IO-APIC-edge rtc > 9: 0 0 0 0 IO-APIC-fasteoi acpi > 12: 0 0 0 4 IO-APIC-edge i8042 > 14: 0 0 0 0 IO-APIC-edge ide0 > 17: 0 0 0 0 IO-APIC-fasteoi sata_nv > 18: 0 0 0 0 IO-APIC-fasteoi sata_nv > 19: 123229 148 514 4698 IO-APIC-fasteoi sata_nv > 362: 127524168 5281605 236961 121506 PCI-MSI-edge eth1 > 377: 519748 12731137 607115 42573852 PCI-MSI-edge eth0:MSI-X-2-RX > 378: 109154 80191 302109913 6487104 PCI-MSI-edge eth0:MSI-X-1-TX > NMI: 0 0 0 0 Non-maskable interrupts > LOC: 300446104 300446082 300446060 300446038 Local timer interrupts > RES: 2698262 44102 2234502 3677120 Rescheduling interrupts > CAL: 4135 4379 4460 415 function call interrupts > TLB: 14018 15088 4079 7251 TLB shootdowns > TRM: 0 0 0 0 Thermal event interrupts > THR: 0 0 0 0 Threshold APIC interrupts > SPU: 0 0 0 0 Spurious interrupts > ERR: 0 > > I suspect that all the counts listed after RES are, in earlier kernels, > lumped into NMI. obviously, rescheduling, function call and TLB shootdowns > are perfectly normal, not indicating any error (though you might want to > minimize them as well...) > > how about trying a new kernel? the above is 2.6.24.3. note that there are > important security fixes that you might be missing if you're running certain > ranges of old kernels... > Hi, Mark. Yes, I was wrong. I also found a very informative discussion of NMI. http://x86vmm.blogspot.com/2005/10/linux-nmis-on-intel-64-bit-hardware.html Thank you. From carsten.aulbert at aei.mpg.de Tue Mar 18 12:35:13 2008 From: carsten.aulbert at aei.mpg.de (Carsten Aulbert) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Which Xeon supports NUMA? Message-ID: <47E01971.6050108@aei.mpg.de> Hi, given that many core Xeons (especially quad and/or many socket systems) have some memory speed issues. With NUMA the kernel seems to be able to optimize this somehow. However, I have two questions: (1) Which EM64T Xeon supports NUMA? I've searched a bit, but I have not found a definitive answer so far. Is this visible from /proc/cpuinfo? (2) More importantly, has someone measured (how?) if this improves performance? Thanks for a brief answer Carsten From hahn at mcmaster.ca Tue Mar 18 12:42:01 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] NMI (Non maskable interrupts) In-Reply-To: <20740902.661205814821402.JavaMail.csamuel@ubuntu> References: <20740902.661205814821402.JavaMail.csamuel@ubuntu> Message-ID: > We also saw performance improvements going to the mainline > kernel (2.6.24.*) from the CentOS 5 kernels. interesting - in what area? thanks, mark hahn. From john.leidel at gmail.com Tue Mar 18 12:50:30 2008 From: john.leidel at gmail.com (John Leidel) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Which Xeon supports NUMA? In-Reply-To: <47E01971.6050108@aei.mpg.de> References: <47E01971.6050108@aei.mpg.de> Message-ID: <1205869830.28737.13.camel@e521.site> This should be an option in your kernel. Check to see if you have this enabled [assuming you are running some sort of Linux varient] On Tue, 2008-03-18 at 20:35 +0100, Carsten Aulbert wrote: > Hi, > > given that many core Xeons (especially quad and/or many socket systems) > have some memory speed issues. With NUMA the kernel seems to be able to > optimize this somehow. However, I have two questions: > > (1) Which EM64T Xeon supports NUMA? I've searched a bit, but I have not > found a definitive answer so far. Is this visible from /proc/cpuinfo? > > (2) More importantly, has someone measured (how?) if this improves > performance? > > Thanks for a brief answer > > Carsten > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From carsten.aulbert at aei.mpg.de Tue Mar 18 12:54:37 2008 From: carsten.aulbert at aei.mpg.de (Carsten Aulbert) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Which Xeon supports NUMA? In-Reply-To: <1205869830.28737.13.camel@e521.site> References: <47E01971.6050108@aei.mpg.de> <1205869830.28737.13.camel@e521.site> Message-ID: <47E01DFD.6030800@aei.mpg.de> Hi John, John Leidel wrote: > This should be an option in your kernel. Check to see if you have this > enabled [assuming you are running some sort of Linux varient] > Sorry, I came up with this question because when compiling 2.6.24.3 (linux kernel) I came across these lines: For x86_64 this is recommended on all multiprocessor Opteron systems. If the system is EM64T, you should say N unless your system is EM64T NUMA. Therefore the question if our Xeon E5345 or Xeon X3220 are EM64T NUMA or just EM64T. Sorry for not being more clear initially. Cheers Carsten From lindahl at pbm.com Tue Mar 18 13:16:45 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Which Xeon supports NUMA? In-Reply-To: <47E01971.6050108@aei.mpg.de> References: <47E01971.6050108@aei.mpg.de> Message-ID: <20080318201645.GB12358@bx9.net> On Tue, Mar 18, 2008 at 08:35:13PM +0100, Carsten Aulbert wrote: > (1) Which EM64T Xeon supports NUMA? The typical chipsets we use around here are not NUMA with Xeons -- all of main memory is the same distance from all the cpus. It's my impression that the grouping of cores to sockets and such isn't part of the NUMA code in the kernel. (Anyone?) In the future, QPI Xeons will be NUMA, similar to Opterons. There are also some exotic chipsets that do NUMA with Xeons today, but you probably don't own one. -- greg From jan.heichler at gmx.net Tue Mar 18 13:18:09 2008 From: jan.heichler at gmx.net (Jan Heichler) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Which Xeon supports NUMA? In-Reply-To: <47E01DFD.6030800@aei.mpg.de> References: <47E01971.6050108@aei.mpg.de> <1205869830.28737.13.camel@e521.site> <47E01DFD.6030800@aei.mpg.de> Message-ID: <366299804.20080318211809@gmx.net> Hallo Carsten, Dienstag, 18. M?rz 2008, meintest Du: CA> Hi John, CA> John Leidel wrote: >> This should be an option in your kernel. Check to see if you have this >> enabled [assuming you are running some sort of Linux varient] CA> Sorry, I came up with this question because when compiling 2.6.24.3 CA> (linux kernel) I came across these lines: CA> For x86_64 this is recommended on all multiprocessor Opteron systems. CA> CA> If the system is EM64T, you should say N unless your system is CA> CA> EM64T NUMA. CA> Therefore the question if our Xeon E5345 or Xeon X3220 are EM64T NUMA or CA> just EM64T. CA> Sorry for not being more clear initially. Could it be that there is already code for the upcoming Nehalem CPU in the kernel? Is intel maybe providing code? I asked a few people who have good knowledge about technologies and linux and this was the only explanation that came up. Cheers, Jan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080318/4545776e/attachment.html From hahn at mcmaster.ca Tue Mar 18 14:00:10 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Which Xeon supports NUMA? In-Reply-To: <47E01971.6050108@aei.mpg.de> References: <47E01971.6050108@aei.mpg.de> Message-ID: > given that many core Xeons (especially quad and/or many socket systems) have > some memory speed issues. With NUMA the kernel seems to be able to optimize > this somehow. I don't believe so. Intel currently still uses a single memory controller (MCH), which means that memory access is, in the NUMA sense, uniform. I don't believe that Intel's recent use of multiple socket-MCH links, or multiple independent FBDIMM channels off the MCH change this. here's an Intel It2 chipset: http://www.intel.com/products/i/chipsets/e8870sp/e8870_blkdiag_8way_800.jpg you can see that there are two FSB's with 4cpus each. a CPU on the left will have non-uniform access to a memory bank which happens to be on the right side of the system. I don't believe any of the Intel x86 chipsets provide this kind of design, though several other companies have done numa x86 chipsets (IBM for one). the interesting thing is that Intel has decided to embrace the numa-oriented system architecture of AMD (et al). it'll be very interesting to see how this plays out with Nehalem/QPI. obviously, AMD really, really needs to wake up and try a little harder to complete... > (2) More importantly, has someone measured (how?) if this improves > performance? usually, tuning for NUMA just means trying to keep a process near its memory. in the chipset above, if a proc starts on the left half, make an effort to allocate its memory on the left as well, and keep scheduling it on left cpus. the kernel does contain code that tries to understand this topology - the most common machines that use it are multi-socket opteron boxes. but systems like SGI Altix depend on this sort of thing quite heavily. following is a trivial measurement of the effect. I'm running the stream benchmark on a single thread. in the first case, I force the process and memory to be on the same socket. then the "wrong" socket. [hahn@rb17 ~]$ numactl --membind=0 --cpubind=0 ./s ... The total memory requirement is 1144 MB You are running each test 11 times ... Function Rate (MB/s) Avg time Min time Max time Copy: 5298.8324 0.1515 0.1510 0.1520 Scale: 5334.1523 0.1504 0.1500 0.1510 Add: 5455.4020 0.2200 0.2200 0.2200 Triad: 5455.3902 0.2200 0.2200 0.2200 ... [hahn@rb17 ~]$ numactl --membind=0 --cpubind=1 ./s ... Function Rate (MB/s) Avg time Min time Max time Copy: 3556.1072 0.2253 0.2250 0.2260 Scale: 3620.4688 0.2213 0.2210 0.2220 Add: 3647.9716 0.3305 0.3289 0.3310 Triad: 3659.0890 0.3305 0.3280 0.3310 note that NUMA optimizations are a wonderful thing, but hardly a panacea. for instance, a busy system might not be able to put all a proc's memory on a particular node. or perhaps the cpus of that node are busy. and then think about multithreaded programs. on top of that, consider caches, which these days are variously per-core, per-chip and per-socket. > Thanks for a brief answer oh, sorry ;) From amjad11 at gmail.com Wed Mar 19 00:58:44 2008 From: amjad11 at gmail.com (amjad ali) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Three questions on a new Beowulf Cluster In-Reply-To: <47BEEF3A.9020806@sps.aero> References: <47BEEF3A.9020806@sps.aero> Message-ID: <428810f20803190058r4ea2ced4vce9f809b1b0b5b03@mail.gmail.com> Hello, If you dont want to indulge into cluster details and want to use it as "black box" kind of tool, Then I would suggest you to use some commercial cluster scheduler/manager like Aspen Beowulf Cluster, PlatformLSF, Scali Manage etc.. or you may go for ROCKS , OSCAR etc This way perhaps you could have easier simpler life. Also have a look in using PETSc (that make use of MPI internally). regards, Amjad Ali. On Fri, Feb 22, 2008 at 8:50 PM, John P. Kosky, PhD wrote: > My company is taking it's first foray into the world of HPC with an > expandable architecture, 16 processor (comprised of quad core Opterons), > one header node cluster using Infiniband interconnects. OS has > tentatively been selected as SUSE 64-bit Linux. The principal purpose of > the cluster is as a tool for spacecraft and propulsion design support. > The cluster will therefore be running the most recent versions of > commercially available software - initially for FEA and CFD using COMSOL > Multiphysics and associated packages, NASTRAN, MatLab modules, as well > as an internally modified and expanded commercial code for materials > properties prediction,with emphasis on polymer modeling (Accelrys > Materials Studio). Since we will be repetitively running standard > modeling codes on this system, we are trying to make the system as user > friendly as possible... most of our scientists and engineers want to use > this as a tool, and not have to become cluster experts. The company WILL > be hiring an IT Sys Admin with good cluster experience to support the > system, however... > > Question 1: > 1) Does anyone here know of any issues that have arisen running the > above named commercial packages on clusters using infiniband? > > Question 2: > 2) As far as the MPI for the system is concerned, for the system and > application requirements described above, would OpenMPI or MvApich be > better for managing node usage? > > ANY help or advice would be greatly appreciated. > > Thanks in advance > > John > > John P. Kosky, PhD > Director of Technical Development > Space Propulsion Systems > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080319/a48ad5bb/attachment.html From csamuel at vpac.org Wed Mar 19 03:44:33 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] NMI (Non maskable interrupts) In-Reply-To: <1655697996.235421205896688535.JavaMail.root@zimbra.vpac.org> Message-ID: <875929636.241761205923473284.JavaMail.root@zimbra.vpac.org> ----- "Mark Hahn" wrote: > > We also saw performance improvements going to the mainline > > kernel (2.6.24.*) from the CentOS 5 kernels. > > interesting - in what area? I think it was NAMD, from memory. -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From csamuel at vpac.org Wed Mar 19 04:04:30 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Which Xeon supports NUMA? In-Reply-To: <6581730.241861205924318549.JavaMail.root@zimbra.vpac.org> Message-ID: <741569291.241881205924670773.JavaMail.root@zimbra.vpac.org> ----- "Carsten Aulbert" wrote: > Sorry, I came up with this question because when compiling 2.6.24.3 > (linux kernel) I came across these lines: > > For x86_64 this is recommended on all multiprocessor Opteron systems. > > > > If the system is EM64T, you should say N unless your system is > EM64T NUMA. Interestingly that text dates back to a modification for 2.6.14-rc5 back in 2005: http://osdir.com/ml/linux.ports.x86-64.general/2005-10/msg00140.html with a comment that says: # CONFIG_K8_NUMA is not needed for Intel EM64T NUMA boxes So either this was in the early days of CSI, or there are/were other Intel AMD64 NUMA boxes out there that none of us know about.. cheers! Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From csamuel at vpac.org Wed Mar 19 04:08:46 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Which Xeon supports NUMA? In-Reply-To: <741569291.241881205924670773.JavaMail.root@zimbra.vpac.org> Message-ID: <400907048.241911205924926660.JavaMail.root@zimbra.vpac.org> ----- "Chris Samuel" wrote: > So either this was in the early days of CSI, or there are/were > other Intel AMD64 NUMA boxes out there that none of us know about.. Aha - to answer myself this even predates the core architecture, rolling our clocks back to 2005 we find this message from an IBM kernel developer correcting Andi Kleen (then the x86_64 maintainer) on the issue: This is quoted from http://lkml.org/lkml/2005/10/17/183 : ----------------8< snip snip 8<---------------- On Mon, Oct 17, 2005 at 05:40:56PM +0200, Andi Kleen wrote: [...] > Intel NUMA x86 machines are really rare No they are not. IBM X460s are generally available machines and the bug affects those boxes. ----------------8< snip snip 8<---------------- So there you go! cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From surs at us.ibm.com Wed Mar 19 11:54:33 2008 From: surs at us.ibm.com (Sayantan Sur) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] [p2s2-announce] UPDATED CFP: Workshop on Parallel Programming Models and Systems Software for High-end Computing (P2S2) Message-ID: CALL FOR PAPERS =============== First International Workshop on Parallel Programming Models and Systems Software for High-end Computing (P2S2) (http://www.mcs.anl.gov/events/workshops/p2s2) Sep. 8th, 2008 To be held in conjunction with ICPP-08: The 27th International Conference on Parallel Processing Sep. 8-12, 2008 Portland, Oregon, USA SCOPE ----- The goal of this workshop is to bring together researchers and practitioners in parallel programming models and systems software for high-end computing systems. Please join us in a discussion of new ideas, experiences, and the latest trends in these areas at the workshop. TOPICS OF INTEREST ------------------ The focus areas for this workshop include, but are not limited to: * Programming models and their high-performance implementations o MPI, Sockets, OpenMP, Global Arrays, X10, UPC, Chapel o Other Hybrid Programming Models * Systems software for scientific and enterprise computing o Communication sub-subsystems for high-end computing o High-performance File and storage systems o Fault-tolerance techniques and implementations o Efficient and high-performance virtualization and other management mechanisms * Tools for Management, Maintenance, Coordination and Synchronization o Software for Enterprise Data-centers using Modern Architectures o Job scheduling libraries o Management libraries for large-scale system o Toolkits for process and task coordination on modern platforms * Performance evaluation, analysis and modeling of emerging computing platforms PROCEEDINGS ----------- Proceedings of this workshop will be published by the IEEE Computer Society (together with the ICPP conference proceedings) in CD format only and will be available at the conference. SUBMISSION INSTRUCTIONS ----------------------- Submissions should be in PDF format in U.S. Letter size paper. They should not exceed 8 pages (all inclusive). Submissions will be judged based on relevance, significance, originality, correctness and clarity. DATES AND DEADLINES ------------------- Paper Submission: April 11th, 2008 Author Notification: May 20th, 2008 Camera Ready: June 2nd, 2008 PROGRAM CHAIRS -------------- * Pavan Balaji (Argonne National Laboratory) * Sayantan Sur (IBM Research) STEERING COMMITTEE ------------------ * William D. Gropp (University of Illinois Urbana-Champaign) * Dhabaleswar K. Panda (Ohio State University) * Vijay Saraswat (IBM Research) PROGRAM COMMITTEE ----------------- * David Bernholdt (Oak Ridge National Laboratory) * Ron Brightwell (Sandia National Laboratory) * Wu-chun Feng (Virginia Tech) * Richard Graham (Oak Ridge National Laboratory) * Hyun-wook Jin (Konkuk University, South Korea) * Sameer Kumar (IBM Research) * Doug Lea (State University of New York at Oswego) * Jarek Nieplocha (Pacific Northwest National Laboratory) * Scott Pakin (Los Alamos National Laboratory) * Vivek Sarkar (Rice University) * Rajeev Thakur (Argonne National Laboratory) * Pete Wyckoff (Ohio Supercomputing Center) If you have any questions, please contact us at p2s2-chairs@mcs.anl.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080319/b81e432a/attachment.html From saville at comcast.net Wed Mar 19 16:31:21 2008 From: saville at comcast.net (Gregg Germain) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Simple MPI programs hang Message-ID: <47E1A249.5030206@comcast.net> Hi everyone, I've created a 2 node cluster running FC8. I've installed MPICH2 1.0.6pl on both (not NFS'd). The Master, Ragnar, is a 64 bit; olaf is a 32 bit. I set up the ring, and mpdtrace shows: $ mpdtrace -l Ragnar_37601 (192.168.0.2) olaf_45530 (192.168.0.5) $ I run a VERY simple MPI program and it hangs: #include "mpi.h" #include #include #include int main( int argc, char *argv[] ) { MPI_Init(&argc,&argv); printf("Hello!\n"); MPI_Finalize(); return 0; } The program outputs the two lines for the two nodes and hangs. I have to CNTRL-C out of it: [gregg@Ragnar ~/BEOAPPS]$ mpiexec -l -n 2 mpibase 0: Hello! 1: Hello! It would sit there forever if I didn't bail. Other simple tests work fine: Running a simple "hostname" test works fine: $ mpiexec -l -n 2 hostname 0: Ragnar 1: olaf $ Now I run a Hello World (no MPI): #include #include int main(int argc,char *argv[]) { printf("\nHello World!\n %d \n", n); } $ mpiexec -l -n 2 ../HelloWorld 0: 0: Hello World! 1: 1: Hello World! $ Any help with this would be appreciated Gregg From gdjacobs at gmail.com Wed Mar 19 18:13:56 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Simple MPI programs hang In-Reply-To: <47E1A249.5030206@comcast.net> References: <47E1A249.5030206@comcast.net> Message-ID: <47E1BA54.4080406@gmail.com> Gregg Germain wrote: > Hi everyone, > > I've created a 2 node cluster running FC8. I've installed MPICH2 > 1.0.6pl on both (not NFS'd). > > The Master, Ragnar, is a 64 bit; olaf is a 32 bit. > > I set up the ring, and mpdtrace shows: > > $ mpdtrace -l > Ragnar_37601 (192.168.0.2) > olaf_45530 (192.168.0.5) > $ > > I run a VERY simple MPI program and it hangs: > #include "mpi.h" > #include > #include > #include > > int main( int argc, char *argv[] ) > { > MPI_Init(&argc,&argv); > printf("Hello!\n"); > MPI_Finalize(); > return 0; > } > > The program outputs the two lines for the two nodes and hangs. I have to > CNTRL-C out of it: > > [gregg@Ragnar ~/BEOAPPS]$ mpiexec -l -n 2 mpibase > 0: Hello! > 1: Hello! > > It would sit there forever if I didn't bail. Other simple tests work fine: > > Running a simple "hostname" test works fine: > > $ mpiexec -l -n 2 hostname > 0: Ragnar > 1: olaf > $ > > Now I run a Hello World (no MPI): > #include > #include > > int main(int argc,char *argv[]) > { > printf("\nHello World!\n %d \n", n); > } > > $ mpiexec -l -n 2 ../HelloWorld > 0: > 0: Hello World! > 1: > 1: Hello World! > $ > > Any help with this would be appreciated > > Gregg Last time I checked, MPICH2 does not permit heterogeneous machine architectures. If Ragnar is using an AMD64 build of MPICH2 and Olaf using MPICH2 targeted on IA32, you are most likely seeing an ABI conflict. You can get around this by using a 32 bit compiler and MPICH library on Ragnar, or a 32 bit development environment residing in a chroot, or a hosted 32 bit VM image, or just reinstall Ragnar as 32 bit only. Or you can go shopping for a different MPI library. The Open MPI people look like they're actively working on this functionality, for example. -- Geoffrey D. Jacobs From wrankin at ee.duke.edu Thu Mar 20 15:51:54 2008 From: wrankin at ee.duke.edu (Bill Rankin) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] Simple MPI programs hang In-Reply-To: <47E1A249.5030206@comcast.net> References: <47E1A249.5030206@comcast.net> Message-ID: <41DA62C4-678F-401F-AA4C-83BA221F7A79@ee.duke.edu> On Mar 19, 2008, at 7:31 PM, Gregg Germain wrote: > Hi everyone, > > I've created a 2 node cluster running FC8. I've installed MPICH2 > 1.0.6pl on both (not NFS'd). > > The Master, Ragnar, is a 64 bit; olaf is a 32 bit. > > I set up the ring, and mpdtrace shows: > > $ mpdtrace -l > Ragnar_37601 (192.168.0.2) > olaf_45530 (192.168.0.5) > $ > > I run a VERY simple MPI program and it hangs: > #include "mpi.h" > #include > #include > #include > > int main( int argc, char *argv[] ) > { > MPI_Init(&argc,&argv); > printf("Hello!\n"); > MPI_Finalize(); > return 0; > } > If it's not a 32/64 bit issue (as the other poster mentioned) you should try putting an MPI_Barrier() call immediately before the MPI_Finalize(). This will at least make sure that both instances have made it complete through MPI_Init() before the one or the other one exits. -bill From eagles051387 at gmail.com Thu Mar 20 12:32:49 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:06:56 2009 Subject: [Beowulf] bonic projects on a cluster Message-ID: has anyone setup a cluster to be used with the analysis of decently sized boinc project data sets. do they integrate nicely into a cluster environment? -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080320/beb902e5/attachment.html From carsten.aulbert at aei.mpg.de Fri Mar 21 00:11:33 2008 From: carsten.aulbert at aei.mpg.de (Carsten Aulbert) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] bonic projects on a cluster In-Reply-To: References: Message-ID: <47E35FA5.6050702@aei.mpg.de> Hi, Jon Aquilina wrote: > has anyone setup a cluster to be used with the analysis of decently > sized boinc project data sets. do they integrate nicely into a cluster > environment? Yes, Einstein @ Home has been used for this. Either as a backfill when no job was running on the node(core) or as a continuous job maximally niced in the background. Usually the memory footprint is not too harsh, thus there is not much of a problem there. However, if running Linux, be advised to stay away from the completely fair scheduler. this tends to give too many cycles to the job if other jobs are running at the same time (as a different user). Another note: Just make sure that the nodes can talk to the servers, i.e. either give the nodes a gateway to talk to the world or provide them with a http-proxy. HTH Carsten From rgb at phy.duke.edu Fri Mar 21 07:24:55 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] bonic projects on a cluster In-Reply-To: <47E35FA5.6050702@aei.mpg.de> References: <47E35FA5.6050702@aei.mpg.de> Message-ID: On Fri, 21 Mar 2008, Carsten Aulbert wrote: > Hi, > > Jon Aquilina wrote: >> has anyone setup a cluster to be used with the analysis of decently sized >> boinc project data sets. do they integrate nicely into a cluster >> environment? What exactly is bonic/boinc? rgb > > Yes, Einstein @ Home has been used for this. Either as a backfill when no job > was running on the node(core) or as a continuous job maximally niced in the > background. Usually the memory footprint is not too harsh, thus there is not > much of a problem there. > > However, if running Linux, be advised to stay away from the completely fair > scheduler. this tends to give too many cycles to the job if other jobs are > running at the same time (as a different user). > > Another note: Just make sure that the nodes can talk to the servers, i.e. > either give the nodes a gateway to talk to the world or provide them with a > http-proxy. > > HTH > > Carsten > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From carsten.aulbert at aei.mpg.de Fri Mar 21 08:22:35 2008 From: carsten.aulbert at aei.mpg.de (Carsten Aulbert) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] bonic projects on a cluster In-Reply-To: References: <47E35FA5.6050702@aei.mpg.de> Message-ID: <47E3D2BB.1080604@aei.mpg.de> Robert G. Brown wrote: >> Jon Aquilina wrote: >>> has anyone setup a cluster to be used with the analysis of decently >>> sized boinc project data sets. do they integrate nicely into a >>> cluster environment? > > What exactly is bonic/boinc? First hit with Google: http://boinc.berkeley.edu/ Cheers Carsten From rgb at phy.duke.edu Fri Mar 21 08:52:59 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] bonic projects on a cluster In-Reply-To: <47E3D2BB.1080604@aei.mpg.de> References: <47E35FA5.6050702@aei.mpg.de> <47E3D2BB.1080604@aei.mpg.de> Message-ID: On Fri, 21 Mar 2008, Carsten Aulbert wrote: > > > Robert G. Brown wrote: >>> Jon Aquilina wrote: >>>> has anyone setup a cluster to be used with the analysis of decently sized >>>> boinc project data sets. do they integrate nicely into a cluster >>>> environment? >> >> What exactly is bonic/boinc? > > First hit with Google: > > http://boinc.berkeley.edu/ Ah. I was wondering if it was supposed to be "bionic" misspelled for a while there. Very interesting. RC5 and SETI@home take on new life as boinc. Hard to find out what boinc stands for, though. Maybe just itself... rgb > > Cheers > > Carsten > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From peter.st.john at gmail.com Fri Mar 21 09:12:55 2008 From: peter.st.john at gmail.com (Peter St. John) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] bonic projects on a cluster In-Reply-To: References: <47E35FA5.6050702@aei.mpg.de> <47E3D2BB.1080604@aei.mpg.de> Message-ID: Berkeley Open Infrastructure for Network Computing, and their old logo was cuter (preserved at the wiki article). And RGB, nice win last night :-) My heart was in my throat. Peter On 3/21/08, Robert G. Brown wrote: > > On Fri, 21 Mar 2008, Carsten Aulbert wrote: > > > > > > > > Robert G. Brown wrote: > >>> Jon Aquilina wrote: > >>>> has anyone setup a cluster to be used with the analysis of decently > sized > >>>> boinc project data sets. do they integrate nicely into a cluster > >>>> environment? > >> > >> What exactly is bonic/boinc? > > > > First hit with Google: > > > > http://boinc.berkeley.edu/ > > > Ah. I was wondering if it was supposed to be "bionic" misspelled for a > while there. Very interesting. RC5 and SETI@home take on new life as > boinc. > > Hard to find out what boinc stands for, though. Maybe just itself... > > > rgb > > > > > > Cheers > > > > Carsten > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > -- > Robert G. Brown Phone(cell): 1-919-280-8443 > Duke University Physics Dept, Box 90305 > Durham, N.C. 27708-0305 > Web: http://www.phy.duke.edu/~rgb > Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php > Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080321/5c918c32/attachment.html From glen.beane at jax.org Fri Mar 21 09:18:39 2008 From: glen.beane at jax.org (Glen Beane) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] bonic projects on a cluster In-Reply-To: References: <47E35FA5.6050702@aei.mpg.de> <47E3D2BB.1080604@aei.mpg.de> Message-ID: <8751161E-BE83-4C04-B92E-2BD56CC7D01F@jax.org> On Mar 21, 2008, at 11:52 AM, Robert G. Brown wrote: > On Fri, 21 Mar 2008, Carsten Aulbert wrote: > >> >> >> Robert G. Brown wrote: >>>> Jon Aquilina wrote: >>>>> has anyone setup a cluster to be used with the analysis of >>>>> decently sized boinc project data sets. do they integrate >>>>> nicely into a cluster environment? >>> What exactly is bonic/boinc? >> >> First hit with Google: >> >> http://boinc.berkeley.edu/ > > Ah. I was wondering if it was supposed to be "bionic" misspelled > for a > while there. Very interesting. RC5 and SETI@home take on new life as > boinc. > > Hard to find out what boinc stands for, though. Maybe just itself... Berkeley Open Infrastructure for Network Computing at least thats what they used to stay it stood for -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153 From mathog at caltech.edu Fri Mar 21 12:19:32 2008 From: mathog at caltech.edu (David Mathog) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Re: bonic projects on a cluster Message-ID: Carsten Aulbert wrote > Robert G. Brown wrote: > > What exactly is bonic/boinc? > > First hit with Google: > > http://boinc.berkeley.edu/ I have a nit to pick with them. Their web site implies (but does not explicitly state) that giving them access to your wasted computing resources costs you nothing, that everybody wins and nobody loses. That is simply not true and is one reason why some sites ban the installation of these sorts of programs. For some rough cost numbers (it was for a desktop, but the difference does not matter much here) see: http://saf.bio.caltech.edu/saving_power.html#costs My point being, with respect to the original poster, letting something like boinc run on a cluster for outside use could easily end up costing the cluster's owner many thousands of dollars a year. Regards, David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From deadline at eadline.org Fri Mar 21 12:35:48 2008 From: deadline at eadline.org (Douglas Eadline) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] bonic projects on a cluster In-Reply-To: References: <47E35FA5.6050702@aei.mpg.de> <47E3D2BB.1080604@aei.mpg.de> Message-ID: <33078.192.168.1.1.1206128148.squirrel@mail.eadline.org> > And RGB, nice win last night :-) My heart was in my throat. RGB, Physicist, Beowulfer, Scrivener, Point Guard -- what a guy! -- Doug > Peter > > On 3/21/08, Robert G. Brown wrote: >> >> On Fri, 21 Mar 2008, Carsten Aulbert wrote: >> >> > >> > >> >> > Robert G. Brown wrote: >> >>> Jon Aquilina wrote: >> >>>> has anyone setup a cluster to be used with the analysis of decently >> sized >> >>>> boinc project data sets. do they integrate nicely into a cluster >> >>>> environment? >> >> >> >> What exactly is bonic/boinc? >> > >> > First hit with Google: >> > >> > http://boinc.berkeley.edu/ >> >> >> Ah. I was wondering if it was supposed to be "bionic" misspelled for a >> while there. Very interesting. RC5 and SETI@home take on new life as >> boinc. >> >> Hard to find out what boinc stands for, though. Maybe just itself... >> >> >> rgb >> >> >> > >> > Cheers >> > >> > Carsten >> > _______________________________________________ >> > Beowulf mailing list, Beowulf@beowulf.org >> > To change your subscription (digest mode or unsubscribe) visit >> > http://www.beowulf.org/mailman/listinfo/beowulf >> > >> >> >> -- >> Robert G. Brown Phone(cell): 1-919-280-8443 >> Duke University Physics Dept, Box 90305 >> Durham, N.C. 27708-0305 >> Web: http://www.phy.duke.edu/~rgb >> Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php >> Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 >> _______________________________________________ >> >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> > > > !DSPAM:47e3def165341735719037! > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > > !DSPAM:47e3def165341735719037! > -- Doug From rgb at phy.duke.edu Fri Mar 21 14:59:02 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] bonic projects on a cluster In-Reply-To: References: <47E35FA5.6050702@aei.mpg.de> <47E3D2BB.1080604@aei.mpg.de> Message-ID: On Fri, 21 Mar 2008, Peter St. John wrote: > Berkeley Open Infrastructure for Network Computing, and their old logo was > cuter (preserved at the wiki article). > And RGB, nice win last night :-) My heart was in my throat. Ah, so. And "nice" doesn't quite describe it. Although the referees certainly did their bit to help keep it close. A couple of "blocking fouls" in replay showed our player standing still in the lane so long he probably got bored waiting to get hit, and there was a strong tendency to call touch fouls in a unidirectional manner. Not that I'm biased or anything...;-) rgb > Peter > > On 3/21/08, Robert G. Brown wrote: >> >> On Fri, 21 Mar 2008, Carsten Aulbert wrote: >> >>> >>> >> >>> Robert G. Brown wrote: >>>>> Jon Aquilina wrote: >>>>>> has anyone setup a cluster to be used with the analysis of decently >> sized >>>>>> boinc project data sets. do they integrate nicely into a cluster >>>>>> environment? >>>> >>>> What exactly is bonic/boinc? >>> >>> First hit with Google: >>> >>> http://boinc.berkeley.edu/ >> >> >> Ah. I was wondering if it was supposed to be "bionic" misspelled for a >> while there. Very interesting. RC5 and SETI@home take on new life as >> boinc. >> >> Hard to find out what boinc stands for, though. Maybe just itself... >> >> >> rgb >> >> >>> >>> Cheers >>> >>> Carsten >>> _______________________________________________ >>> Beowulf mailing list, Beowulf@beowulf.org >>> To change your subscription (digest mode or unsubscribe) visit >>> http://www.beowulf.org/mailman/listinfo/beowulf >>> >> >> >> -- >> Robert G. Brown Phone(cell): 1-919-280-8443 >> Duke University Physics Dept, Box 90305 >> Durham, N.C. 27708-0305 >> Web: http://www.phy.duke.edu/~rgb >> Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php >> Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 >> _______________________________________________ >> >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From rgb at phy.duke.edu Fri Mar 21 15:37:39 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Re: bonic projects on a cluster In-Reply-To: References: Message-ID: On Fri, 21 Mar 2008, David Mathog wrote: > Carsten Aulbert wrote >> Robert G. Brown wrote: >>> What exactly is bonic/boinc? >> >> First hit with Google: >> >> http://boinc.berkeley.edu/ > > I have a nit to pick with them. Their web site implies (but does not > explicitly state) that giving them access to your wasted computing > resources costs you nothing, that everybody wins and nobody loses. That > is simply not true and is one reason why some sites ban the installation > of these sorts of programs. For some rough cost numbers (it was for a > desktop, but the difference does not matter much here) see: > > http://saf.bio.caltech.edu/saving_power.html#costs > > My point being, with respect to the original poster, letting something > like boinc run on a cluster for outside use could easily end up costing > the cluster's owner many thousands of dollars a year. I typically estimate $1/watt/year because in generaly you have to pay to remove those watts once you release them into the wild of a server room, and it adds anywhere from 20% to 40% to the cost of the power alone depending on the COP of your AC unit and the outside temperature and so on. In the case of cluster users, one rarely turns off the nodes so the issue is much more idle consumption vs non-idle consumption, a marginal cost on the order of the $20-30 or so differential between his case 2 and his case 3, at my slightly higher power cost estimate because cluster nodes would definitely require cooling. As far as CRT vs flatpanel monitors, I'm not sure that I agree with this site's conclusions, if one does NOT turn off the monitors, a running CRT costs more like the full $85-90/year vs $30-35 (according to MY kill-a-watt readings) for a CRT. However, this is really a tricky question as "green" monitors and computers may -- as noted -- really turn off the monitor IF the monitor supports it. Most flatpanels do. Older CRTs do not. CRTs are much more likely to draw at least a trickle even when "off" to reduce the time lag for the electron gun to start working. When I replaced my OLD CRT monitors several years ago, I assumed a 3-5 year payoff (not 20) and still feel that that is about right, although flatpanels have gotten cheaper since then (the CRTs they're replacing are also probably more efficient as well). And at this point the question is moot because you'd have to be crazy to buy a CRT >>over<< a flatpanel when you have to replace a monitor -- the differential marginal cost at my payoff rate of $50-$60/year will pay for the difference in a very short time, and CRTs eat desktop, weigh a ton, and are full of lead (where yes, the flatpanels have mercury and both have arsenic so it is a matter of choosing your toxin). Overall the environmental kindness of flatpanels beats that of CRTs, I would judge. The final issue is that IF one signs up one's cluster AND one has an application to contribute to the application pool, one can obtain the usual dual advantage of sharing a large resource. True, you pay for somebody else's computation and energy and any heat related maintenance and the risk associated with third parties running unknown applications on your machines inside what might otherwise be a sound firewall. If that application turns out to be "an encryption engine for securely distributing kiddie porn" -- overcoming the barriers that I'm sure are there to this sort of thing being so distributed -- that latter risk my not seem as trivial as otherwise it might. Still, you in principle can access far greater PEAK performance for your embarrassingly parallel application in exchange for contributing your otherwise idle cycles -- an efficiency that many big cluster centers rely on today. At fixed maintenance and human costs, increasing duty cycle is pure return, and it doesn't take much access to time to compensate for the additional modest cost in energy relative to idle IF your work flow has the appropriate pattern. For example, if you run a simulation that requires 100 node-months to complete, and then need to spend at least a month writing it up and analyzing results before starting the next one, then contributing your 50 nodes in exchange for access to the boincweb (or whatever they call it) MIGHT let you run for two weeks on 200 nodes and let others use your 50 nodes for six weeks. You complete a work cycle in two months, able to take a two week vacation in there. Otherwise it takes you two months to run your application and a third one to write it up. There is plenty of room for a win-win here. So while I agree that the docs are misleading, that doesn't mean that a real cost-benefit analysis will show that it is always a bad idea to share resources. Many grids and shared clusters do. Aside from the difficult-to-quantify risk factor, I'd argue that even the BOINC cluster might be a worthwhile thing for CLUSTER people to participate in (just as are many grids that do exactly the same thing without the SETI@Home flavor). Individual systems belonging to private individuals, well, there I agree. Even at a marginal differential cost of $30/year, they should be upfront about this (and note that it might be much higher if you DO turn off your systems at home every night -- I don't). To me the real question is, what is the risk? How much bandwidth does it consume (a very definitely scarce resource on thin-pipe home ADSL links)? Do I trust a third party being granted access to my compute resources inside a firewall? What happens if that permits "bad things" to happen inside my trusted domain, and who pays for fixing them or gets fired for being stupid? It's one thing to put an application up on my kids probably already-virus-ridden WinXX boxes used primary for gaming, another to put it on a system that might have passwords or other data stored as "encrypted" data for a browser (encrypted in quotes as it is trivial to decrypt), or that I use to access my bank. If trolls take over a system, they can read every keystroke. I'd have to REALLY TRUST the maker of a sandbox before I let just anyone play in it... rgb > > Regards, > > David Mathog > mathog@caltech.edu > Manager, Sequence Analysis Facility, Biology Division, Caltech > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From mathog at caltech.edu Fri Mar 21 16:26:48 2008 From: mathog at caltech.edu (David Mathog) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Re: bonic projects on a cluster Message-ID: Robert G. Brown wrote: > On Fri, 21 Mar 2008, David Mathog wrote: > > > I have a nit to pick with them. > > My point being, with respect to the original poster, letting something > > like boinc run on a cluster for outside use could easily end up costing > > the cluster's owner many thousands of dollars a year. > > I typically estimate $1/watt/year because in generaly you have to pay to > remove those watts once you release them into the wild of a server room. Right. The original analysis was for PCs which might be at home, where an open window would suffice. $1/watt/year is certainly easier to calculate and more accurate for a room with A/C. > In the case of cluster users, one rarely turns off the nodes so the > issue is much more idle consumption vs non-idle consumption, a marginal > cost on the order of the $20-30 or so differential between his case 2 > and his case 3, at my slightly higher power cost estimate because > cluster nodes would definitely require cooling. Under pressure from the big computing farm users, and to some extent, the government, computers are getting better at dropping to lower power usage when idle. In this (and virtually nothing else) the Via C7 is the best game in town, since its idle power consumption is a tiny 0.1 watts, vs. 20 watts peak. The move to multiple cores has not helped in this cause though, since as far as I know none of them can fully power down N-1 cores and those N-1 cores' support circuitry, resulting in N idling cores instead of just 1. > at this point the question is moot because you'd have to be crazy to buy a > CRT >>over<< a flatpanel when you have to replace a monitor Right, assuming you could even find one. The trickiest thing about the LCDs is that it can be hard to distinguish between blanked and idle. With a kill-a-watt it is clear, but not many people have those. The other way is to wait a while and see if it cools off, but some of them run so cool in the first place that that is a difficult test. > So while I agree that the docs are misleading, that doesn't mean that a > real cost-benefit analysis will show that it is always a bad idea to > share resources. I didn't mean to imply that it was. Regards, David Mathog mathog@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech From James.P.Lux at jpl.nasa.gov Fri Mar 21 16:52:41 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Re: bonic projects on a cluster In-Reply-To: References: Message-ID: <6.2.3.4.2.20080321163144.031d5718@mail.jpl.nasa.gov> At 03:37 PM 3/21/2008, Robert G. Brown wrote: And at >this point the question is moot because you'd have to be crazy to buy a >CRT >>over<< a flatpanel when you have to replace a monitor -- the >differential marginal cost at my payoff rate of $50-$60/year will pay >for the difference in a very short time, and CRTs eat desktop, weigh a >ton, and are full of lead But what if you wanted to use the CRTs as shielding, saving you money on shielding that x-ray source you're testing. Or you needed the mass of the CRTs to hold your desk down in a seismic event. Or, you live in a cold climate, and the CRT is being used as " workspace heat source". There's also the carbon footprint of you (or some vendor) shlepping those CRTs off to the dump, the cost of hazmat disposal of the CRT, and then the carbon footprint and cost of bringing the new flatpanel display home, etc. One really does have to consider these other "system" issues... (That's what I tell my wife about why I still have all those big CRT monitors out in the garage... they're providing radiation shielding) >(where yes, the flatpanels have mercury and >both have arsenic so it is a matter of choosing your toxin). Overall >the environmental kindness of flatpanels beats that of CRTs, I would >judge. In terms of toxin content of your output display device, I might point out that an ASR33 teletype (http://en.wikipedia.org/wiki/ASR33) uses rolls of inexpensive newsprint type paper (preferred by beat poets! no interrupting the stream of consciousness with switching pages as you create), produced from a renewable resource (trees) and can be recycled after use, either as fuel or feedstock to make more teletype paper. They also draw almost no power when not actually typing (assuming you have the autostart for the motor turned on). Not much persistent toxic stuff in a teletype, just good old 19th century mechanical components made of steel, grease, etc. Nobody worries about fireproofing the housing either with brominated compounds.. they're made of steel as well. If you were to use one of those newfangled LA36 DECwriters or my own favorite, the TI 820KSR, which could print at a blistering 150cps, you're into the plastic case era (although probably not fireproof, but certainly lots of chlorinated hydrocarbons involved in the manufacture) (Why, yes, I have one of those in the garage too, the TI810, without keyboard, also providing shielding from radiation) If you feel compelled to have computer generated graphs, plotters also use paper as their output media, and one could make the ink from oak galls, squid, etc., which is a quite sustainable source. Just how much output do you really need to know that the answer is 42 (or 41 or 43)? Spend those precious joules on computation, not eye candy. Jim From James.P.Lux at jpl.nasa.gov Fri Mar 21 16:54:47 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Re: bonic projects on a cluster In-Reply-To: References: Message-ID: <6.2.3.4.2.20080321165318.031afcc8@mail.jpl.nasa.gov> At 04:26 PM 3/21/2008, David Mathog wrote: > > at this point the question is moot because you'd have to be crazy to >buy a > > CRT >>over<< a flatpanel when you have to replace a monitor > >Right, assuming you could even find one. The trickiest thing about the >LCDs is that it can be hard to distinguish between blanked and idle. >With a kill-a-watt it is clear, but not many people have those. Anyone on this list probably has one, if not more than one. Those of you who don't, shame! It's a great toy, if nothing else. Jim From mark.kosmowski at gmail.com Sat Mar 22 15:20:30 2008 From: mark.kosmowski at gmail.com (Mark Kosmowski) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request Message-ID: Could someone please provide a website or two for NUMA information? I'm about to upgrade the RAM of my cluster and might want to experiment with using NUMA to try too eek out a little speed. I am definitely more of a cluster end-user than a developer, so NUMA admin / usage sites would be preferred to development sites. Thanks! (My background: Chemistry graduate student using plane wave software, 3 node dual cpu single-core opteron cluster soon to each have 8 Gb RAM on Linux (OpenSUSE 10.2 / 64 bit) with OpenMPI.) From xclski at yahoo.com Sat Mar 22 16:42:25 2008 From: xclski at yahoo.com (Ellis Wilson) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] bonic projects on a cluster In-Reply-To: <47E3D2BB.1080604@aei.mpg.de> Message-ID: <344918.80024.qm@web37911.mail.mud.yahoo.com> I'm very interested to hear of some applications for such loosely connected "clusters" such as boinc would create. My crappy 100mb connection is alright for the few embarrassingly parallel applications I've applied to it which do not have a large origin dataset, but again, these are very few. I would imagine the connection (with hops and an even slower/less dedicated interconnect at each hop) for boinc "nodes" would be far worse. I looked for examples on the boinc site briefly, but cannot seem to find any. Any notable embarrassingly parallel problems you all have seen or worked with? Sorry for my lack of experience in this area. Ellis ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping From rgb at phy.duke.edu Sat Mar 22 20:26:14 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] bonic projects on a cluster In-Reply-To: <344918.80024.qm@web37911.mail.mud.yahoo.com> References: <344918.80024.qm@web37911.mail.mud.yahoo.com> Message-ID: On Sat, 22 Mar 2008, Ellis Wilson wrote: > I'm very interested to hear of some applications for > such loosely connected "clusters" such as boinc would > create. My crappy 100mb connection is alright for the > few embarrassingly parallel applications I've applied > to it which do not have a large origin dataset, but > again, these are very few. I would imagine the > connection (with hops and an even slower/less > dedicated interconnect at each hop) for boinc "nodes" > would be far worse. I looked for examples on the > boinc site briefly, but cannot seem to find any. > > Any notable embarrassingly parallel problems you all > have seen or worked with? Sorry for my lack of > experience in this area. Almost all importance sampling Monte Carlo computations fit into the category of stuff that works wonderfully on anything from sneakernet on up. I used to distribute jobs on with tcl/expect scripts running over rsh and 10 Mbps ethernet back in the early to mid 90's. Lots of other physics computations are one CPU, one job, but a large parameter space to be investigated with many jobs. Random number generator testing could be parallelized nicely with very low bandwidth connections (although I haven't even thought seriously about parallelizing dieharder yet -- it's difficult enough to mess with the code as serial modular tests so far). Lots of graphical processing fits in a category that will work, although some does start getting up into bw intensive as well -- it depends as always on the granularity and ratio of computation to communication for the specific task. In general code that requires only "small data" for startup and that spans a large parametric space with that small data, that takes a substantial amount of time to run a computation from that data, and that returns (say) a small vector of numbers to a central aggregator in a master-slave sort of computation is a very reasonable candidate for this sort of cluster. I ran code for years that I used to joke about distributing on OnSpin3d@Home...;-) rgb > > Ellis > > > ____________________________________________________________________________________ > Looking for last minute shopping deals? > Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From kyron at neuralbs.com Mon Mar 24 08:40:19 2008 From: kyron at neuralbs.com (Eric Thibodeau) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: References: Message-ID: <47E7CB63.1080609@neuralbs.com> Mark, NUMA is an acronym meaning Non Uniform Memory Access. This is a hardware constraint and is not a "performance" switch you turn on. Under the Linux kernel there is an option that is meant to tell the kernel to be conscious about that hardware fact and attempt to help it optimize the way it maps the memory allocation to a task Vs the processor the given task will be using (processor affinity, check out taskset (in recent util-linux implementations, ie: 2.13+). In your specific case, you would have 4Gigs per CPU and would want to make sure each task (assuming one per CPU) stays on the same CPU all the time and would want to make sure each task fits within the "local" 4Gig. Here is a link that should help you out with that respect: http://www.nic.uoregon.edu/tau-wiki/Guide:Opteron_NUMA_Analysis Eric Mark Kosmowski wrote: > Could someone please provide a website or two for NUMA information? > I'm about to upgrade the RAM of my cluster and might want to > experiment with using NUMA to try too eek out a little speed. I am > definitely more of a cluster end-user than a developer, so NUMA admin > / usage sites would be preferred to development sites. > > Thanks! > > (My background: Chemistry graduate student using plane wave software, > 3 node dual cpu single-core opteron cluster soon to each have 8 Gb RAM > on Linux (OpenSUSE 10.2 / 64 bit) with OpenMPI.) > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > From prentice at ias.edu Mon Mar 24 09:04:40 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: <47E7CB63.1080609@neuralbs.com> References: <47E7CB63.1080609@neuralbs.com> Message-ID: <47E7D118.2050603@ias.edu> Eric Thibodeau wrote: > Mark, > > NUMA is an acronym meaning Non Uniform Memory Access. This is a > hardware constraint and is not a "performance" switch you turn on. Under > the Linux kernel there is an option that is meant to tell the kernel to > be conscious about that hardware fact and attempt to help it optimize > the way it maps the memory allocation to a task Vs the processor the > given task will be using (processor affinity, check out taskset (in > recent util-linux implementations, ie: 2.13+). > > In your specific case, you would have 4Gigs per CPU and would want to > make sure each task (assuming one per CPU) stays on the same CPU all the > time and would want to make sure each task fits within the "local" 4Gig. > > Here is a link that should help you out with that respect: > > http://www.nic.uoregon.edu/tau-wiki/Guide:Opteron_NUMA_Analysis > For a more general (and detailed) discussion of NUMA, you might be able for find some computer architecture books at the library where you are a grad student. Might be a little *too* much info, though. This text book is a popular comp. architecture text book. you might even say it's the "gold standard" I'm pretty sure it discusses NUMA somewhere in between it's covers. http://www.amazon.com/Computer-Architecture-Quantitative-Approach-Kaufmann/dp/1558605967 Prentice Bisbal Linux Software Support Specialist/System Administrator School of Natural Sciences Institute for Advanced Study Princeton, NJ From hahn at mcmaster.ca Mon Mar 24 13:24:15 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: <47E7CB63.1080609@neuralbs.com> References: <47E7CB63.1080609@neuralbs.com> Message-ID: > NUMA is an acronym meaning Non Uniform Memory Access. This is a hardware > constraint and is not a "performance" switch you turn on. Under the Linux I don't agree. NUMA is indeed a description of hardware. I'm not sure what you meant by "constraint" - NUMA is not some kind of shortcoming. > kernel there is an option that is meant to tell the kernel to be conscious > about that hardware fact and attempt to help it optimize the way it maps the > memory allocation to a task Vs the processor the given task will be using > (processor affinity, check out taskset (in recent util-linux implementations, > ie: 2.13+). the kernel has had various forms of NUMA and socket affinity for a long time, and I suspect most any distro will install kernel which has the appropriate support (surely any x86_64 kernel would have NUMA support). I usually use numactl rather than taskset. I'm not sure of the history of those tools. as far as I can tell, taskset only addresses numactl --cpubind, though they obviously approach things differently. if you're going to use taskset, you'll want to set cpu affinity to multiple cpus (those local to a socket, or 'node' in numactl terms.) > In your specific case, you would have 4Gigs per CPU and would want to make > sure each task (assuming one per CPU) stays on the same CPU all the time and > would want to make sure each task fits within the "local" 4Gig. "numactl --localalloc". but you should first verify that your machines actually do have the 8GB split across both nodes. it's not that uncommon to see an inexperienced assembler fill up one node before going onto the next, and there have even been some boards which provided no memory to the second node. From csamuel at vpac.org Mon Mar 24 20:48:16 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: <1760697696.8991206416309797.JavaMail.root@zimbra.vpac.org> Message-ID: <1038684506.9661206416896484.JavaMail.root@zimbra.vpac.org> ----- "Mark Hahn" wrote: > I usually use numactl rather than taskset. Some of the MPI variants have CPU affinity support (though that can be a poisoned chalice if you have two MPI jobs sharing nodes) and there is also the kernels cpuset support which is now part of the more generic control group system, see Documentation/cgroups.txt for more info (in the current kernel). So, for instance, Torque 2.3.x now supports Linux cpusets (make sure /dev/cpuset is mounted before starting pbs_mom) for simple CPU affinity allocations for PBS jobs, but it does not (yet) tackle the much harder problem of binding memory nodes to cpusets as well as processor cores, it just lets them access any memory node (and leaves where that comes from to the kernel). cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From kyron at neuralbs.com Mon Mar 24 21:17:04 2008 From: kyron at neuralbs.com (Eric Thibodeau) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: References: <47E7CB63.1080609@neuralbs.com> Message-ID: <47E87CC0.7000000@neuralbs.com> Mark Hahn wrote: >> NUMA is an acronym meaning Non Uniform Memory Access. This is a >> hardware constraint and is not a "performance" switch you turn on. >> Under the Linux > > I don't agree. NUMA is indeed a description of hardware. I'm not > sure what you meant by "constraint" - NUMA is not some kind of > shortcoming. Mark is right, my choice of words is misleading. By constraint I meant that you have to be conscious of what ends up where (that was the point of the link I added in my e-mail ;P ) >> kernel there is an option that is meant to tell the kernel to be >> conscious about that hardware fact and attempt to help it optimize >> the way it maps the memory allocation to a task Vs the processor the >> given task will be using (processor affinity, check out taskset (in >> recent util-linux implementations, ie: 2.13+). > the kernel has had various forms of NUMA and socket affinity for a > long time, > and I suspect most any distro will install kernel which has the > appropriate support (surely any x86_64 kernel would have NUMA support). My point of view on distro kernels is that they are to be scrutinized unless they are specifically meant to be used as computation nodes (ie: don't expect CONFIG_HZ=100 to be set on "typical" distros). Also, NUMA is only applicable to Opteron architecture (internal MMU with HyperTransport), not the Intel flavor of multi-core CPUs (external MMU, which can be a single bus or any memory access scheme as dictated by the motherboard manufacturer). > > I usually use numactl rather than taskset. I'm not sure of the > history of those tools. as far as I can tell, taskset only addresses > numactl --cpubind, > though they obviously approach things differently. if you're going to > use taskset, you'll want to set cpu affinity to multiple cpus (those > local to a socket, or 'node' in numactl terms.) > >> In your specific case, you would have 4Gigs per CPU and would want >> to make sure each task (assuming one per CPU) stays on the same CPU >> all the time and would want to make sure each task fits within the >> "local" 4Gig. > > "numactl --localalloc". > > but you should first verify that your machines actually do have the 8GB > split across both nodes. it's not that uncommon to see an > inexperienced assembler fill up one node before going onto the next, > and there have even > been some boards which provided no memory to the second node. Mark (Hahn) is right (again !), I ASSumed the tech would load the memory banks appropriately, don't make that mistake ;) And numactl is indeed more appropriate in this case (thanks Mr. Hahn ;) ). Note that the kernel (configured with NUMA) _will_ attempt to allocate the memory to "'local nodes" before offloading to memory "abroad". Eric From mark.kosmowski at gmail.com Tue Mar 25 03:58:58 2008 From: mark.kosmowski at gmail.com (Mark Kosmowski) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: <47E87CC0.7000000@neuralbs.com> References: <47E7CB63.1080609@neuralbs.com> <47E87CC0.7000000@neuralbs.com> Message-ID: On Tue, Mar 25, 2008 at 12:17 AM, Eric Thibodeau wrote: > > Mark Hahn wrote: > >> NUMA is an acronym meaning Non Uniform Memory Access. This is a > >> hardware constraint and is not a "performance" switch you turn on. > >> Under the Linux > > > > I don't agree. NUMA is indeed a description of hardware. I'm not > > sure what you meant by "constraint" - NUMA is not some kind of > > shortcoming. > Mark is right, my choice of words is misleading. By constraint I meant > that you have to be conscious of what ends up where (that was the point > of the link I added in my e-mail ;P ) > > >> kernel there is an option that is meant to tell the kernel to be > >> conscious about that hardware fact and attempt to help it optimize > >> the way it maps the memory allocation to a task Vs the processor the > >> given task will be using (processor affinity, check out taskset (in > >> recent util-linux implementations, ie: 2.13+). > > the kernel has had various forms of NUMA and socket affinity for a > > long time, > > and I suspect most any distro will install kernel which has the > > appropriate support (surely any x86_64 kernel would have NUMA support). > My point of view on distro kernels is that they are to be scrutinized > unless they are specifically meant to be used as computation nodes (ie: > don't expect CONFIG_HZ=100 to be set on "typical" distros). > Also, NUMA is only applicable to Opteron architecture (internal MMU with > HyperTransport), not the Intel flavor of multi-core CPUs (external MMU, > which can be a single bus or any memory access scheme as dictated by the > motherboard manufacturer). > > > > > I usually use numactl rather than taskset. I'm not sure of the > > history of those tools. as far as I can tell, taskset only addresses > > numactl --cpubind, > > though they obviously approach things differently. if you're going to > > use taskset, you'll want to set cpu affinity to multiple cpus (those > > local to a socket, or 'node' in numactl terms.) > > > >> In your specific case, you would have 4Gigs per CPU and would want > >> to make sure each task (assuming one per CPU) stays on the same CPU > >> all the time and would want to make sure each task fits within the > >> "local" 4Gig. > > > > "numactl --localalloc". > > > > but you should first verify that your machines actually do have the 8GB > > split across both nodes. it's not that uncommon to see an > > inexperienced assembler fill up one node before going onto the next, > > and there have even > > been some boards which provided no memory to the second node. > Mark (Hahn) is right (again !), I ASSumed the tech would load the memory > banks appropriately, don't make that mistake ;) And numactl is indeed > more appropriate in this case (thanks Mr. Hahn ;) ). Note that the > kernel (configured with NUMA) _will_ attempt to allocate the memory to > "'local nodes" before offloading to memory "abroad". > > Eric > The memory will be installed by myself correctly - that is, distributing the memory according to cpu. However, it appears that one of my nodes (my first Opteron machine) may well be one that has only one bank of four DIMM slots assigned to cpu 0 and shared by cpu 1. It uses a Tyan K8W Tiger s2875 motherboard. My other two nodes use Arima HDAMA motherboards with SATA support - each cpu has a bank of 4 DIMMs associated with it. The Tyan node is getting 4 @ 2 Gb DIMMs, one of the HDAMA nodes is getting 8 @ 1 Gb (both instances fully populating the available DIMM slots) and the last machine is going to get 4 @ 1 Gb DIMMs for one cpu and 2 @ 2 Gb for the other. It looks like I may want to upgrade my motherboard before exploring NUMA / affinity then. This discussion as well as reading about NUMA and affinity elsewhere leads to another question - what is the difference between using numactl or using the affinity options of my parallelization software (in my case openmpi)? Thanks, Mark (Kosmowski) From fumie.costen at manchester.ac.uk Tue Mar 25 04:21:54 2008 From: fumie.costen at manchester.ac.uk (f.costen@cs.man.ac.uk) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] a postdoc in Canada Message-ID: <47E8E052.9060104@cs.man.ac.uk> Dear All, One of my friends is looking for a person who can take up a postdoc for one year in Canada ( 1500 canadian dollars including tax per month ) This may be able to be extended subject to successive grant availability. The successful candidate will have the following conditions: Strong mathematics good publication record good knowledge in parallel computing The postdoc job involves Theoretical formulation of the radar cross section calculation in the inhonogeneous media of 2D and 3D targets and the algorithm implementation and computation If you need more information please contact: Hosam El-Ocla: Department of Computer Science Lakehead University 955 Oliver Road, Thunder Bay, On P7B 5E1, Canada Tel: 1-807-343-8270 hosam@lakeheadu.ca Thank you very much Fumie From james.p.lux at jpl.nasa.gov Tue Mar 25 06:34:27 2008 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <47E8E052.9060104@cs.man.ac.uk> References: <47E8E052.9060104@cs.man.ac.uk> Message-ID: <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> Quoting "f.costen@cs.man.ac.uk" , on Tue 25 Mar 2008 04:21:54 AM PDT: > Dear All, > > One of my friends is looking for a person > who can take up a postdoc for one year in Canada > ( 1500 canadian dollars including tax per month ) > This may be able to be extended subject to > successive grant availability. Just curious, since I'm not in academia, but is this a typical compensation ($18,000/yr). Is this for a full time job? Or is the person going to be out getting some other funds, and the subject research is just part of their overall work. Or do they get some other stipend/living allowance. (Here in California, this would be just about minimum wage: $8/hr = $16k/yr) Jim Lux From fumie.costen at manchester.ac.uk Tue Mar 25 07:09:05 2008 From: fumie.costen at manchester.ac.uk (f.costen@cs.man.ac.uk) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> Message-ID: <47E90781.7060805@cs.man.ac.uk> Dear Jim, >> One of my friends is looking for a person >> who can take up a postdoc for one year in Canada >> ( 1500 canadian dollars including tax per month ) >> This may be able to be extended subject to >> successive grant availability. > Just curious, since I'm not in academia, but is this a typical > compensation ($18,000/yr). Is this for a full time job? Or is the > person going to be out getting some other funds, and the subject > research is just part of their overall work. Or do they get some other > stipend/living allowance. (Here in California, this would be just about > minimum wage: $8/hr = $16k/yr) To be honest with you, I can not comment on the salary in Canadian dollars as I am based in the U.K. So if there is anything to my post, please send your query to Thank you very much indeed for your input. Best wishes, Fumie From gdjacobs at gmail.com Tue Mar 25 07:30:04 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> Message-ID: <47E90C6C.4050104@gmail.com> Jim Lux wrote: > Quoting "f.costen@cs.man.ac.uk" , on Tue > 25 Mar 2008 04:21:54 AM PDT: > >> Dear All, >> >> One of my friends is looking for a person >> who can take up a postdoc for one year in Canada >> ( 1500 canadian dollars including tax per month ) >> This may be able to be extended subject to >> successive grant availability. > > Just curious, since I'm not in academia, but is this a typical > compensation ($18,000/yr). Is this for a full time job? Or is the > person going to be out getting some other funds, and the subject > research is just part of their overall work. Or do they get some other > stipend/living allowance. (Here in California, this would be just about > minimum wage: $8/hr = $16k/yr) > > Jim Lux No, that's a starvation wage. This must be just the portion of the grant allocated for an additional post-doc. It might be filled out with an NSERC grant or a teaching position at Lakehead. -- Geoffrey D. Jacobs From prentice at ias.edu Tue Mar 25 08:12:43 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> Message-ID: <47E9166B.3040700@ias.edu> Jim Lux wrote: > Quoting "f.costen@cs.man.ac.uk" , on Tue > 25 Mar 2008 04:21:54 AM PDT: > >> Dear All, >> >> One of my friends is looking for a person >> who can take up a postdoc for one year in Canada >> ( 1500 canadian dollars including tax per month ) >> This may be able to be extended subject to >> successive grant availability. > > Just curious, since I'm not in academia, but is this a typical > compensation ($18,000/yr). Is this for a full time job? Or is the > person going to be out getting some other funds, and the subject > research is just part of their overall work. Or do they get some other > stipend/living allowance. (Here in California, this would be just about > minimum wage: $8/hr = $16k/yr) > As someone who works in academia and has interacted with postdocs from time-to-time, that sounds right. Some consider postdoc positions to be legalized slavery. You're supposed to work for the honor of working with a great researcher and doing great work, and getting "an education" at the same time. Who needs money when you can have that? ;-) I was once working for an extremely wealthy commercial company (a big oil co. Need I say more?) And I met a postdoc there. It was his 3rd or 4th postdoc in a row, and he was living at just about the poverty level while working for one of the richest companies in the world. Prentice From dnlombar at ichips.intel.com Tue Mar 25 08:28:04 2008 From: dnlombar at ichips.intel.com (Lombard, David N) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Re: bonic projects on a cluster In-Reply-To: References: Message-ID: <20080325152804.GA7457@nlxdcldnl2.cl.intel.com> On Fri, Mar 21, 2008 at 12:19:32PM -0700, David Mathog wrote: > Carsten Aulbert wrote > > Robert G. Brown wrote: > > > What exactly is bonic/boinc? > > > > First hit with Google: > > > > http://boinc.berkeley.edu/ > > I have a nit to pick with them. Their web site implies (but does not > explicitly state) that giving them access to your wasted computing > resources costs you nothing, that everybody wins and nobody loses. ... > My point being, with respect to the original poster, letting something > like boinc run on a cluster for outside use could easily end up costing > the cluster's owner many thousands of dollars a year. Along these lines, I had a prime number search running in the background on my home desktop. It ran so well--99% utilization with no impact on interactive responsiveness--that I completely forgot about it. I only rediscovered it when I wondered why my CPU was running so hot. -- David N. Lombard, Intel, Irvine, CA I do not speak for Intel Corporation; all comments are strictly my own. From dnlombar at ichips.intel.com Tue Mar 25 08:58:06 2008 From: dnlombar at ichips.intel.com (Lombard, David N) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: References: <47E7CB63.1080609@neuralbs.com> <47E87CC0.7000000@neuralbs.com> Message-ID: <20080325155806.GC7457@nlxdcldnl2.cl.intel.com> On Tue, Mar 25, 2008 at 06:58:58AM -0400, Mark Kosmowski wrote: > > This discussion as well as reading about NUMA and affinity elsewhere > leads to another question - what is the difference between using > numactl or using the affinity options of my parallelization software > (in my case openmpi)? OpenMPI will handle to affinity for you for each rank of your job within the context of your MPI job; using the external utils will require you to manage this yourself on a process- by-process basis. -- David N. Lombard, Intel, Irvine, CA I do not speak for Intel Corporation; all comments are strictly my own. From glen.beane at jax.org Tue Mar 25 09:30:38 2008 From: glen.beane at jax.org (Glen Beane) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> Message-ID: <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> On Mar 25, 2008, at 9:34 AM, Jim Lux wrote: > Quoting "f.costen@cs.man.ac.uk" , on > Tue 25 Mar 2008 04:21:54 AM PDT: > >> Dear All, >> >> One of my friends is looking for a person >> who can take up a postdoc for one year in Canada >> ( 1500 canadian dollars including tax per month ) >> This may be able to be extended subject to >> successive grant availability. > > Just curious, since I'm not in academia, but is this a typical > compensation ($18,000/yr). Is this for a full time job? Or is the > person going to be out getting some other funds, and the subject > research is just part of their overall work. Or do they get some > other stipend/living allowance. (Here in California, this would be > just about minimum wage: $8/hr = $16k/yr) I'm at a private non-profit research lab doing cancer/genetics research and I am pretty sure we pay post docs according to the NIH scale which starts at around 37k for someone with 0-1 years experience and goes to about 51k for someone with 7+ years (if you want to do some googling you can find the NIH postdoc stipend scale). These stipends are exempt from FICA tax. If the income paid is determined to be subject to FICA tax then the postdoc is considered an employee and they get paid a salary that is a little higher than the stipends -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153 From kyron at neuralbs.com Tue Mar 25 09:40:32 2008 From: kyron at neuralbs.com (kyron@neuralbs.com) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: References: <47E7CB63.1080609@neuralbs.com> <47E87CC0.7000000@neuralbs.com> Message-ID: <44238.142.137.232.82.1206463232.squirrel@mail.neuralbs.com> > On Tue, Mar 25, 2008 at 12:17 AM, Eric Thibodeau > wrote: >> >> Mark Hahn wrote: >> >> NUMA is an acronym meaning Non Uniform Memory Access. This is a >> >> hardware constraint and is not a "performance" switch you turn on. >> >> Under the Linux >> > >> > I don't agree. NUMA is indeed a description of hardware. I'm not >> > sure what you meant by "constraint" - NUMA is not some kind of >> > shortcoming. >> Mark is right, my choice of words is misleading. By constraint I meant >> that you have to be conscious of what ends up where (that was the point >> of the link I added in my e-mail ;P ) >> >> >> kernel there is an option that is meant to tell the kernel to be >> >> conscious about that hardware fact and attempt to help it optimize >> >> the way it maps the memory allocation to a task Vs the processor the >> >> given task will be using (processor affinity, check out taskset (in >> >> recent util-linux implementations, ie: 2.13+). >> > the kernel has had various forms of NUMA and socket affinity for a >> > long time, >> > and I suspect most any distro will install kernel which has the >> > appropriate support (surely any x86_64 kernel would have NUMA >> support). >> My point of view on distro kernels is that they are to be scrutinized >> unless they are specifically meant to be used as computation nodes (ie: >> don't expect CONFIG_HZ=100 to be set on "typical" distros). >> Also, NUMA is only applicable to Opteron architecture (internal MMU >> with >> HyperTransport), not the Intel flavor of multi-core CPUs (external MMU, >> which can be a single bus or any memory access scheme as dictated by >> the >> motherboard manufacturer). >> >> > >> > I usually use numactl rather than taskset. I'm not sure of the >> > history of those tools. as far as I can tell, taskset only addresses >> > numactl --cpubind, >> > though they obviously approach things differently. if you're going >> to >> > use taskset, you'll want to set cpu affinity to multiple cpus (those >> > local to a socket, or 'node' in numactl terms.) >> > >> >> In your specific case, you would have 4Gigs per CPU and would want >> >> to make sure each task (assuming one per CPU) stays on the same CPU >> >> all the time and would want to make sure each task fits within the >> >> "local" 4Gig. >> > >> > "numactl --localalloc". >> > >> > but you should first verify that your machines actually do have the >> 8GB >> > split across both nodes. it's not that uncommon to see an >> > inexperienced assembler fill up one node before going onto the next, >> > and there have even >> > been some boards which provided no memory to the second node. >> Mark (Hahn) is right (again !), I ASSumed the tech would load the >> memory >> banks appropriately, don't make that mistake ;) And numactl is indeed >> more appropriate in this case (thanks Mr. Hahn ;) ). Note that the >> kernel (configured with NUMA) _will_ attempt to allocate the memory to >> "'local nodes" before offloading to memory "abroad". >> >> Eric >> > The memory will be installed by myself correctly - that is, > distributing the memory according to cpu. However, it appears that > one of my nodes (my first Opteron machine) may well be one that has > only one bank of four DIMM slots assigned to cpu 0 and shared by cpu > 1. It uses a Tyan K8W Tiger s2875 motherboard. My other two nodes > use Arima HDAMA motherboards with SATA support - each cpu has a bank > of 4 DIMMs associated with it. The Tyan node is getting 4 @ 2 Gb > DIMMs, one of the HDAMA nodes is getting 8 @ 1 Gb (both instances > fully populating the available DIMM slots) and the last machine is > going to get 4 @ 1 Gb DIMMs for one cpu and 2 @ 2 Gb for the other. That last scheme might give you some unbalanced performance but that is something to look up with the MB's instruction manual (ie: you might be better off installing the RAM as 1G+1G+2G for both CPUs instead of 4x1G + 2x2G). > It looks like I may want to upgrade my motherboard before exploring > NUMA / affinity then. If you're getting into "upgrading" (ie: trowing money at) anything, then you're getting into the slippery slope of the hardware selection debate ;) > This discussion as well as reading about NUMA and affinity elsewhere > leads to another question - what is the difference between using > numactl or using the affinity options of my parallelization software > (in my case openmpi)? numactl is an application to help nudge processes in the correct direction. Implementing cpuaffinity within your code makes your code explicitally aware that it will run on an SMP machine (ie: it's hardcoded and you don't need to call a script to change your processe's affinity). In that regards Chris Samuel replied with the mention of Torque and PBS which would support affinity assignment. IMHO, that would be the most logical place to control affinity (as long as one can provide some memory access hints, ie: same options as seen in numactl's manpage) > Thanks, > > Mark (Kosmowski) > Eric Thibodeau From kyron at neuralbs.com Tue Mar 25 09:49:58 2008 From: kyron at neuralbs.com (kyron@neuralbs.com) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: <47E7D118.2050603@ias.edu> References: <47E7CB63.1080609@neuralbs.com> <47E7D118.2050603@ias.edu> Message-ID: <54552.142.137.232.82.1206463798.squirrel@mail.neuralbs.com> > > > Eric Thibodeau wrote: >> Mark, >> >> NUMA is an acronym meaning Non Uniform Memory Access. This is a >> hardware constraint and is not a "performance" switch you turn on. Under >> the Linux kernel there is an option that is meant to tell the kernel to >> be conscious about that hardware fact and attempt to help it optimize >> the way it maps the memory allocation to a task Vs the processor the >> given task will be using (processor affinity, check out taskset (in >> recent util-linux implementations, ie: 2.13+). >> >> In your specific case, you would have 4Gigs per CPU and would want to >> make sure each task (assuming one per CPU) stays on the same CPU all the >> time and would want to make sure each task fits within the "local" 4Gig. >> >> Here is a link that should help you out with that respect: >> >> http://www.nic.uoregon.edu/tau-wiki/Guide:Opteron_NUMA_Analysis >> > > For a more general (and detailed) discussion of NUMA, you might be able > for find some computer architecture books at the library where you are a > grad student. Might be a little *too* much info, though. > > This text book is a popular comp. architecture text book. you might even > say it's the "gold standard" I'm pretty sure it discusses NUMA somewhere > in between it's covers. > > http://www.amazon.com/Computer-Architecture-Quantitative-Approach-Kaufmann/dp/1558605967 Indeed an excellent book but note that the 4th edition is out. I would only recommend reading it if you need to decide between buying Intel Itanium, Sun T1/2, Opteron , IBM P5, etc., processors and need to know the the detailed architecture of your systems. You'll learn that Sun T1 is incredibly fast...but only for integer use, not for floating point. If you're only starting in HPC and want to do some parallel coding without having the luxury of choosing your hardware, I strongly recommend "Software Optimization for High Performance Computing: Creating Faster Applications (HP Professional Series)". Even if it is a little dated (2000), I find it to be an excellent reference for anyone getting into HPC. http://www.amazon.com/Software-Optimization-High-Performance-Computing/dp/0130170089 > > Prentice Bisbal > Linux Software Support Specialist/System Administrator > School of Natural Sciences > Institute for Advanced Study > Princeton, NJ Eric Thibodeau From mark.kosmowski at gmail.com Wed Mar 26 02:33:08 2008 From: mark.kosmowski at gmail.com (Mark Kosmowski) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: <44238.142.137.232.82.1206463232.squirrel@mail.neuralbs.com> References: <47E7CB63.1080609@neuralbs.com> <47E87CC0.7000000@neuralbs.com> <44238.142.137.232.82.1206463232.squirrel@mail.neuralbs.com> Message-ID: On Tue, Mar 25, 2008 at 12:40 PM, wrote: > > > On Tue, Mar 25, 2008 at 12:17 AM, Eric Thibodeau > > wrote: > >> > >> Mark Hahn wrote: > >> >> NUMA is an acronym meaning Non Uniform Memory Access. This is a > >> >> hardware constraint and is not a "performance" switch you turn on. > >> >> Under the Linux > >> > > >> > I don't agree. NUMA is indeed a description of hardware. I'm not > >> > sure what you meant by "constraint" - NUMA is not some kind of > >> > shortcoming. > >> Mark is right, my choice of words is misleading. By constraint I meant > >> that you have to be conscious of what ends up where (that was the point > >> of the link I added in my e-mail ;P ) > >> > >> >> kernel there is an option that is meant to tell the kernel to be > >> >> conscious about that hardware fact and attempt to help it optimize > >> >> the way it maps the memory allocation to a task Vs the processor the > >> >> given task will be using (processor affinity, check out taskset (in > >> >> recent util-linux implementations, ie: 2.13+). > >> > the kernel has had various forms of NUMA and socket affinity for a > >> > long time, > >> > and I suspect most any distro will install kernel which has the > >> > appropriate support (surely any x86_64 kernel would have NUMA > >> support). > >> My point of view on distro kernels is that they are to be scrutinized > >> unless they are specifically meant to be used as computation nodes (ie: > >> don't expect CONFIG_HZ=100 to be set on "typical" distros). > >> Also, NUMA is only applicable to Opteron architecture (internal MMU > >> with > >> HyperTransport), not the Intel flavor of multi-core CPUs (external MMU, > >> which can be a single bus or any memory access scheme as dictated by > >> the > >> motherboard manufacturer). > >> > >> > > >> > I usually use numactl rather than taskset. I'm not sure of the > >> > history of those tools. as far as I can tell, taskset only addresses > >> > numactl --cpubind, > >> > though they obviously approach things differently. if you're going > >> to > >> > use taskset, you'll want to set cpu affinity to multiple cpus (those > >> > local to a socket, or 'node' in numactl terms.) > >> > > >> >> In your specific case, you would have 4Gigs per CPU and would want > >> >> to make sure each task (assuming one per CPU) stays on the same CPU > >> >> all the time and would want to make sure each task fits within the > >> >> "local" 4Gig. > >> > > >> > "numactl --localalloc". > >> > > >> > but you should first verify that your machines actually do have the > >> 8GB > >> > split across both nodes. it's not that uncommon to see an > >> > inexperienced assembler fill up one node before going onto the next, > >> > and there have even > >> > been some boards which provided no memory to the second node. > >> Mark (Hahn) is right (again !), I ASSumed the tech would load the > >> memory > >> banks appropriately, don't make that mistake ;) And numactl is indeed > >> more appropriate in this case (thanks Mr. Hahn ;) ). Note that the > >> kernel (configured with NUMA) _will_ attempt to allocate the memory to > >> "'local nodes" before offloading to memory "abroad". > >> > >> Eric > >> > > The memory will be installed by myself correctly - that is, > > distributing the memory according to cpu. However, it appears that > > one of my nodes (my first Opteron machine) may well be one that has > > only one bank of four DIMM slots assigned to cpu 0 and shared by cpu > > 1. It uses a Tyan K8W Tiger s2875 motherboard. My other two nodes > > use Arima HDAMA motherboards with SATA support - each cpu has a bank > > of 4 DIMMs associated with it. The Tyan node is getting 4 @ 2 Gb > > DIMMs, one of the HDAMA nodes is getting 8 @ 1 Gb (both instances > > fully populating the available DIMM slots) and the last machine is > > going to get 4 @ 1 Gb DIMMs for one cpu and 2 @ 2 Gb for the other. > > That last scheme might give you some unbalanced performance but that is > something to look up with the MB's instruction manual (ie: you might be > better off installing the RAM as 1G+1G+2G for both CPUs instead of 4x1G + > 2x2G). On my Opteron systems, wouldn't 3 DIMMs per CPU drop me into 64-bit memory bandwidth rather than the allowed 128-bit memory bandwidth when each CPU has an even number of DIMMs? > > > > It looks like I may want to upgrade my motherboard before exploring > > NUMA / affinity then. > > If you're getting into "upgrading" (ie: trowing money at) anything, then > you're getting into the slippery slope of the hardware selection debate ;) Slippery indeed. At this point, I think I may just install the RAM to bring my current calculation out of swap and be done with the cluster for now. Given that I think one of my nodes uses hypertransport for all of cpu 1 memory access, would it hurt anything to use affinity when only 2 out of 3 nodes can benefit from affinity? > > > > This discussion as well as reading about NUMA and affinity elsewhere > > leads to another question - what is the difference between using > > numactl or using the affinity options of my parallelization software > > (in my case openmpi)? > > numactl is an application to help nudge processes in the correct > direction. Implementing cpuaffinity within your code makes your code > explicitally aware that it will run on an SMP machine (ie: it's hardcoded > and you don't need to call a script to change your processe's affinity). > > In that regards Chris Samuel replied with the mention of Torque and PBS > which would support affinity assignment. IMHO, that would be the most > logical place to control affinity (as long as one can provide some memory > access hints, ie: same options as seen in numactl's manpage) > > > Thanks, > > > > Mark (Kosmowski) > > > > Eric Thibodeau > > Again, thank you for this discussion - I'm learning quite a bit! From kyron at neuralbs.com Wed Mar 26 07:58:38 2008 From: kyron at neuralbs.com (kyron@neuralbs.com) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: References: <47E7CB63.1080609@neuralbs.com> <47E87CC0.7000000@neuralbs.com> <44238.142.137.232.82.1206463232.squirrel@mail.neuralbs.com> Message-ID: <30343.74.56.208.220.1206543518.squirrel@mail.neuralbs.com> > On Tue, Mar 25, 2008 at 12:40 PM, wrote: >> >> > On Tue, Mar 25, 2008 at 12:17 AM, Eric Thibodeau >> > wrote: >> >> >> >> Mark Hahn wrote: >> >> >> NUMA is an acronym meaning Non Uniform Memory Access. This is >> a >> >> >> hardware constraint and is not a "performance" switch you turn >> on. >> >> >> Under the Linux >> >> > >> >> > I don't agree. NUMA is indeed a description of hardware. I'm >> not >> >> > sure what you meant by "constraint" - NUMA is not some kind of >> >> > shortcoming. >> >> Mark is right, my choice of words is misleading. By constraint I >> meant >> >> that you have to be conscious of what ends up where (that was the >> point >> >> of the link I added in my e-mail ;P ) >> >> >> >> >> kernel there is an option that is meant to tell the kernel to be >> >> >> conscious about that hardware fact and attempt to help it >> optimize >> >> >> the way it maps the memory allocation to a task Vs the processor >> the >> >> >> given task will be using (processor affinity, check out taskset >> (in >> >> >> recent util-linux implementations, ie: 2.13+). >> >> > the kernel has had various forms of NUMA and socket affinity for >> a >> >> > long time, >> >> > and I suspect most any distro will install kernel which has the >> >> > appropriate support (surely any x86_64 kernel would have NUMA >> >> support). >> >> My point of view on distro kernels is that they are to be >> scrutinized >> >> unless they are specifically meant to be used as computation nodes >> (ie: >> >> don't expect CONFIG_HZ=100 to be set on "typical" distros). >> >> Also, NUMA is only applicable to Opteron architecture (internal MMU >> >> with >> >> HyperTransport), not the Intel flavor of multi-core CPUs (external >> MMU, >> >> which can be a single bus or any memory access scheme as dictated >> by >> >> the >> >> motherboard manufacturer). >> >> >> >> > >> >> > I usually use numactl rather than taskset. I'm not sure of the >> >> > history of those tools. as far as I can tell, taskset only >> addresses >> >> > numactl --cpubind, >> >> > though they obviously approach things differently. if you're >> going >> >> to >> >> > use taskset, you'll want to set cpu affinity to multiple cpus >> (those >> >> > local to a socket, or 'node' in numactl terms.) >> >> > >> >> >> In your specific case, you would have 4Gigs per CPU and would >> want >> >> >> to make sure each task (assuming one per CPU) stays on the same >> CPU >> >> >> all the time and would want to make sure each task fits within >> the >> >> >> "local" 4Gig. >> >> > >> >> > "numactl --localalloc". >> >> > >> >> > but you should first verify that your machines actually do have >> the >> >> 8GB >> >> > split across both nodes. it's not that uncommon to see an >> >> > inexperienced assembler fill up one node before going onto the >> next, >> >> > and there have even >> >> > been some boards which provided no memory to the second node. >> >> Mark (Hahn) is right (again !), I ASSumed the tech would load the >> >> memory >> >> banks appropriately, don't make that mistake ;) And numactl is >> indeed >> >> more appropriate in this case (thanks Mr. Hahn ;) ). Note that the >> >> kernel (configured with NUMA) _will_ attempt to allocate the memory >> to >> >> "'local nodes" before offloading to memory "abroad". >> >> >> >> Eric >> >> >> > The memory will be installed by myself correctly - that is, >> > distributing the memory according to cpu. However, it appears that >> > one of my nodes (my first Opteron machine) may well be one that has >> > only one bank of four DIMM slots assigned to cpu 0 and shared by cpu >> > 1. It uses a Tyan K8W Tiger s2875 motherboard. My other two nodes >> > use Arima HDAMA motherboards with SATA support - each cpu has a bank >> > of 4 DIMMs associated with it. The Tyan node is getting 4 @ 2 Gb >> > DIMMs, one of the HDAMA nodes is getting 8 @ 1 Gb (both instances >> > fully populating the available DIMM slots) and the last machine is >> > going to get 4 @ 1 Gb DIMMs for one cpu and 2 @ 2 Gb for the other. >> >> That last scheme might give you some unbalanced performance but that is >> something to look up with the MB's instruction manual (ie: you might be >> better off installing the RAM as 1G+1G+2G for both CPUs instead of 4x1G >> + >> 2x2G). > > On my Opteron systems, wouldn't 3 DIMMs per CPU drop me into 64-bit > memory bandwidth rather than the allowed 128-bit memory bandwidth when > each CPU has an even number of DIMMs? Hence "look up with the MB's instruction manual" but you're probably right. > >> >> >> > It looks like I may want to upgrade my motherboard before exploring >> > NUMA / affinity then. >> >> If you're getting into "upgrading" (ie: trowing money at) anything, >> then >> you're getting into the slippery slope of the hardware selection debate >> ;) > > Slippery indeed. At this point, I think I may just install the RAM to > bring my current calculation out of swap and be done with the cluster > for now. Given that I think one of my nodes uses hypertransport for > all of cpu 1 memory access, would it hurt anything to use affinity > when only 2 out of 3 nodes can benefit from affinity? Start by getting that RAM in and worry about affinity once your code runs in parallel, the kernel will probably play nice with you and take that worry away from you. Numactl can help you determine if your process is attempting to access another node's memory space by "crashing" your process: (from the numactl manpage): --membind=nodes, -m nodes Only allocate memory from nodes. Allocation will fail when there is not enough memory available on these nodes. >> >> >> > This discussion as well as reading about NUMA and affinity elsewhere >> > leads to another question - what is the difference between using >> > numactl or using the affinity options of my parallelization software >> > (in my case openmpi)? >> >> numactl is an application to help nudge processes in the correct >> direction. Implementing cpuaffinity within your code makes your code >> explicitally aware that it will run on an SMP machine (ie: it's >> hardcoded >> and you don't need to call a script to change your processe's >> affinity). >> >> In that regards Chris Samuel replied with the mention of Torque and PBS >> which would support affinity assignment. IMHO, that would be the most >> logical place to control affinity (as long as one can provide some >> memory >> access hints, ie: same options as seen in numactl's manpage) >> >> > Thanks, >> > >> > Mark (Kosmowski) >> > >> >> Eric Thibodeau >> >> > Again, thank you for this discussion - I'm learning quite a bit! > No prob. Eric From csamuel at vpac.org Wed Mar 26 15:12:12 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] NUMA info request In-Reply-To: Message-ID: <1476416350.116711206569532670.JavaMail.root@zimbra.vpac.org> ----- "Mark Kosmowski" wrote: > This discussion as well as reading about NUMA and affinity elsewhere > leads to another question - what is the difference between using > numactl or using the affinity options of my parallelization software > (in my case openmpi)? Our experiences with MVAPICH2 was unpleasant because of our workload (lots of users with their own codes doing very different - and occasionally wrong - things). It has a naive implementation where if you have say an 8 way system and run 2 x 4-way MPI jobs on it then this happens: Job 1 starts and each MPI thread sets affinity starting from core 0, so it allocates core 0, 1, 2 and 3. Job 2 starts and each MPI thread sets affinity starting from core 0, so it allocates core 0, 1, 2 and 3. This is a Bad Thing(tm). Suddenly the user wonders why their performance has just halved. The sysadmin looks at the node and wonders what the code is doing wrong if the load average is so high but there's so much idle CPU time available. This is why I'm a big fan of the queueing system doing this for you, especially if it can (as Torque does when you use a compatible MPI job launcher) allocate each MPI process onto its own core. Of course it's game over if you're using a legacy MPI launcher that uses ssh or rsh. :-( cheers, Chris -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From u048450 at squ.edu.om Tue Mar 25 07:08:33 2008 From: u048450 at squ.edu.om (Eiman Mohammed Masoud Al-Rajhi) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Need help Message-ID: Hello every body; Excuse me we need a program for image processing written using C program with MPI in FreeBSD operating system. Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080325/945d7262/attachment.html From eagles051387 at gmail.com Tue Mar 25 09:13:38 2008 From: eagles051387 at gmail.com (Jon Aquilina) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Re: bonic projects on a cluster In-Reply-To: <20080325152804.GA7457@nlxdcldnl2.cl.intel.com> References: <20080325152804.GA7457@nlxdcldnl2.cl.intel.com> Message-ID: thats always nice. i have a duel core machine on this laptop and when i have boinc projects utilizing 100% of my processor with no impact on responsiveness wchi is nice. off topic but i noticed u work for intel. are you guys looking for people to test future chips? On Tue, Mar 25, 2008 at 4:28 PM, Lombard, David N wrote: > On Fri, Mar 21, 2008 at 12:19:32PM -0700, David Mathog wrote: > > Carsten Aulbert wrote > > > Robert G. Brown wrote: > > > > What exactly is bonic/boinc? > > > > > > First hit with Google: > > > > > > http://boinc.berkeley.edu/ > > > > I have a nit to pick with them. Their web site implies (but does not > > explicitly state) that giving them access to your wasted computing > > resources costs you nothing, that everybody wins and nobody loses. > ... > > My point being, with respect to the original poster, letting something > > like boinc run on a cluster for outside use could easily end up costing > > the cluster's owner many thousands of dollars a year. > > Along these lines, I had a prime number search running in the background > on my home desktop. It ran so well--99% utilization with no impact on > interactive responsiveness--that I completely forgot about it. I only > rediscovered it when I wondered why my CPU was running so hot. > > -- > David N. Lombard, Intel, Irvine, CA > I do not speak for Intel Corporation; all comments are strictly my own. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Jonathan Aquilina -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080325/e57a9a59/attachment.html From jmack at wm7d.net Tue Mar 25 11:15:40 2008 From: jmack at wm7d.net (Joseph Mack NA3T) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> Message-ID: On Tue, 25 Mar 2008, Glen Beane wrote: > These stipends are exempt from FICA tax. This is just a way of short changing the post-doc by not giving them the same pension benefits that every other working person gets. Joe -- Joseph Mack NA3T EME(B,D), FM05lw North Carolina jmack (at) wm7d (dot) net - azimuthal equidistant map generator at http://www.wm7d.net/azproj.shtml Homepage http://www.austintek.com/ It's GNU/Linux! From rreis at aero.ist.utl.pt Thu Mar 27 03:41:28 2008 From: rreis at aero.ist.utl.pt (Ricardo Reis) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] visualization machine Message-ID: Hi all I beg to take advantage of your experience although the topic isn't completly cluster thing. I got some money to buy a new machine, at least 8Gb and I'm thinking between a 2 x dual core or a 1 x quad (or even 2x quads). It must be one machine because it must (urgh) be able to use it's 8Gb in serial codes (don't ask). Anyway, I've been experiencing with paraview for parallel visualization and was wondering on your opinion on... buying a ultra-duper-cool state-of-the-art graphic card (Nvidia) or 2 graphic cards? thanks for your time, Ricardo Reis 'Non Serviam' PhD student @ Lasef Computational Fluid Dynamics, High Performance Computing, Turbulence http://www.lasef.ist.utl.pt & Cultural Instigator @ R?dio Zero http://www.radiozero.pt http://www.flickr.com/photos/rreis/ From landman at scalableinformatics.com Thu Mar 27 11:22:14 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> Message-ID: <47EBE5D6.7050702@scalableinformatics.com> Joseph Mack NA3T wrote: > On Tue, 25 Mar 2008, Glen Beane wrote: > >> These stipends are exempt from FICA tax. > > This is just a way of short changing the post-doc by not giving them the > same pension benefits that every other working person gets. Hmmm.... deviating quite a bit from beowulfery ... but ... not "every other working person" gets pension benefits. Last I checked here in the US, there isn't a national pension system (yeah we have social security ... anyone want to try to live off of that?). Small companies, and most newer companies don't do pensions (for a very good reason), rather using 401k's and similar devices. That is the sidebar to the OT. When I started looking at postdoc (eek ... that long ago?) positions, it was not uncommon to see theoretical physics PhDs earning ~16-19k/year US. No FICA exemptions. For a family of 4, this put you under the poverty line. Somehow, I am not quite convinced that the science departments offering these grasped how the invisible hand of the market would decimate their supply of new PhD's for these positions. You can fight the market, or accept that there is a market and deal with it. The former is not a wise course of action (though it had been standard practice when I was looking at postdocs). The old Young Scientists Network had formed then to discuss some of the abuses of the market and attempts to manipulate the demand/supply curves to oversupply the market with talent, thus keeping the price of talent low. There is a cost to every decision, and flooding the market (then) has had longer term effects that are being observed today. Back to your regularly scheduled cluster ... Joe (a free-market capitalist) -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From cousins at umit.maine.edu Thu Mar 27 13:58:09 2008 From: cousins at umit.maine.edu (Steve Cousins) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Re: visualization machine (Ricardo Reis) In-Reply-To: <200803271900.m2RJ0BsT005729@bluewest.scyld.com> References: <200803271900.m2RJ0BsT005729@bluewest.scyld.com> Message-ID: > Hi all > > I beg to take advantage of your experience although the topic isn't > completly cluster thing. I got some money to buy a new machine, at least > 8Gb and I'm thinking between a 2 x dual core or a 1 x quad (or even 2x > quads). It must be one machine because it must (urgh) be able to use it's > 8Gb in serial codes (don't ask). Anyway, I've been experiencing with > paraview for parallel visualization and was wondering on your opinion > on... buying a ultra-duper-cool state-of-the-art graphic card (Nvidia) or > 2 graphic cards? We've been happy with just a single nVidia 8800GTX (state of the art at the time) card in a quad-socket Opteron machine. This machine is capable of 128 GB of RAM and two 8800GTX cards although currently we only have 16 GB of RAM, two CPU's (four cores) and one video card. It has been great at showing our ocean model data in 3D with a customized version of Vis5D. Steve ______________________________________________________________________ Steve Cousins, Ocean Modeling Group Email: cousins@umit.maine.edu Marine Sciences, 452 Aubert Hall http://rocky.umeoce.maine.edu Univ. of Maine, Orono, ME 04469 Phone: (207) 581-4302 From kalpana0611 at gmail.com Thu Mar 27 18:26:54 2008 From: kalpana0611 at gmail.com (Kalpana Kanthasamy) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Confused over the term cluster Message-ID: Hi, I am kinda confused with term cluster and now am sure if this where I can post my question. What exactly can be defined as a cluster ?, We have a 32 node cluster in our lab. and the number 32 here is referred to 32 separate machines ( each with its own processor ). What about the new machines like, dual core processors or the quad core processors, are those considered clusters also. For example, should we refer to a quad core processor as a cluster of 4 processors. Hope someone out there could explain. Thanks -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080328/6af977ec/attachment.html From jac67 at georgetown.edu Thu Mar 27 19:47:35 2008 From: jac67 at georgetown.edu (Jess Cannata) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: References: Message-ID: <47EC5C47.6080302@georgetown.edu> I would call a cluster with 32 nodes and 8 cores in each node a 32 node, 256 CPU cluster. Jess Kalpana Kanthasamy wrote: > Hi, > > I am kinda confused with term cluster and now am sure if this where I can > post my question. What exactly can be defined as a cluster ?, We have a 32 > node cluster in our lab. and the number 32 here is referred to 32 separate > machines ( each with its own processor ). What about the new machines > like, > dual core processors or the quad core processors, are those considered > clusters also. For example, should we refer to a quad core processor as a > cluster of 4 processors. Hope someone out there could explain. Thanks > ------------------------------------------------------------------------ > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > From hahn at mcmaster.ca Thu Mar 27 20:32:22 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: <47EC5C47.6080302@georgetown.edu> References: <47EC5C47.6080302@georgetown.edu> Message-ID: > I would call a cluster with 32 nodes and 8 cores in each node a 32 node, 256 > CPU cluster. yes, definitely. node implies a boundary of memory addressing: after all, it's a cluster because we need to use distributed-memory programming techniques like MPI to harness it. often nodes that have more cores are called "fat nodes". >> I am kinda confused with term cluster and now am sure if this where I can >> post my question. What exactly can be defined as a cluster ?, We have a 32 >> node cluster in our lab. and the number 32 here is referred to 32 separate >> machines ( each with its own processor ). What about the new machines >> like, >> dual core processors or the quad core processors, are those considered >> clusters also. For example, should we refer to a quad core processor as a >> cluster of 4 processors. Hope someone out there could explain. Thanks From gdjacobs at gmail.com Thu Mar 27 20:53:35 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Need help In-Reply-To: References: Message-ID: <47EC6BBF.4030507@gmail.com> Eiman Mohammed Masoud Al-Rajhi wrote: > Hello every body; > > Excuse me we need a program for image processing written using C program > with MPI in FreeBSD operating system. Not much is really different. Do you require help with the MPI part of your project, or are you looking for advice on how to organize the architecture? Perhaps you could be more specific on what you're looking for? Maybe you're looking for a tin can solution? I can't find the source, but a quick google found some interesting publications: http://citeseer.ist.psu.edu/230149.html -- Geoffrey D. Jacobs From gdjacobs at gmail.com Thu Mar 27 21:02:07 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] visualization machine In-Reply-To: References: Message-ID: <47EC6DBF.1080705@gmail.com> Ricardo Reis wrote: > > Hi all > > I beg to take advantage of your experience although the topic isn't > completly cluster thing. I got some money to buy a new machine, at least > 8Gb and I'm thinking between a 2 x dual core or a 1 x quad (or even 2x > quads). It must be one machine because it must (urgh) be able to use > it's 8Gb in serial codes (don't ask). Anyway, I've been experiencing > with paraview for parallel visualization and was wondering on your > opinion on... buying a ultra-duper-cool state-of-the-art graphic card > (Nvidia) or 2 graphic cards? > > thanks for your time, > > Ricardo Reis No nvidia for me unless they open up driver specs. Open drivers have a history of being very, very stable if not as fast. Nvidia does have good drivers, but I have definitely seen them experience problems. -- Geoffrey D. Jacobs From amjad11 at gmail.com Thu Mar 27 22:46:09 2008 From: amjad11 at gmail.com (amjad ali) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: References: <47EC5C47.6080302@georgetown.edu> Message-ID: <428810f20803272246q49f853efr782194a7360b0ae0@mail.gmail.com> More general term one could use is the "processing element". For example if a cluster consists of 2 nodes with each having 1 single core CPU, 2 nodes with each having 1 dual core CPU, 2 nodes with each having 1 quad core CPU, 2 nodes with each having 2 dual core CPUs, and 2 nodes with each having 2 quad core CPUs. Then we could better say that there are 38 "pocessing elements" in this 10-nodes cluster. Note that there are 10 motherboards in this cluster. On Fri, Mar 28, 2008 at 8:32 AM, Mark Hahn wrote: > > I would call a cluster with 32 nodes and 8 cores in each node a 32 node, > 256 > > CPU cluster. > > yes, definitely. node implies a boundary of memory addressing: after all, > it's a cluster because we need to use distributed-memory programming > techniques like MPI to harness it. often nodes that have more cores > are called "fat nodes". > > >> I am kinda confused with term cluster and now am sure if this where I > can > >> post my question. What exactly can be defined as a cluster ?, We have a > 32 > >> node cluster in our lab. and the number 32 here is referred to 32 > separate > >> machines ( each with its own processor ). What about the new machines > >> like, > >> dual core processors or the quad core processors, are those considered > >> clusters also. For example, should we refer to a quad core processor as > a > >> cluster of 4 processors. Hope someone out there could explain. Thanks > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080328/6b284376/attachment.html From kalpana0611 at gmail.com Thu Mar 27 23:00:28 2008 From: kalpana0611 at gmail.com (Kalpana Kanthasamy) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: <428810f20803272246q49f853efr782194a7360b0ae0@mail.gmail.com> References: <47EC5C47.6080302@georgetown.edu> <428810f20803272246q49f853efr782194a7360b0ae0@mail.gmail.com> Message-ID: Thanks guys, that was really helpful. On 3/28/08, amjad ali wrote: > > More general term one could use is the "processing element". > For example if a cluster consists of > 2 nodes with each having 1 single core CPU, > 2 nodes with each having 1 dual core CPU, > 2 nodes with each having 1 quad core CPU, > 2 nodes with each having 2 dual core CPUs, and > 2 nodes with each having 2 quad core CPUs. > > Then we could better say that there are 38 "pocessing elements" in this > 10-nodes cluster. > Note that there are 10 motherboards in this cluster. > > > On Fri, Mar 28, 2008 at 8:32 AM, Mark Hahn wrote: > > > > I would call a cluster with 32 nodes and 8 cores in each node a 32 > > node, 256 > > > CPU cluster. > > > > > > yes, definitely. node implies a boundary of memory addressing: after > > all, > > it's a cluster because we need to use distributed-memory programming > > techniques like MPI to harness it. often nodes that have more cores > > are called "fat nodes". > > > > >> I am kinda confused with term cluster and now am sure if this where I > > can > > >> post my question. What exactly can be defined as a cluster ?, We have > > a 32 > > >> node cluster in our lab. and the number 32 here is referred to 32 > > separate > > >> machines ( each with its own processor ). What about the new machines > > >> like, > > >> dual core processors or the quad core processors, are those > > considered > > >> clusters also. For example, should we refer to a quad core processor > > as a > > >> cluster of 4 processors. Hope someone out there could explain. Thanks > > _______________________________________________ > > Beowulf mailing list, Beowulf@beowulf.org > > To change your subscription (digest mode or unsubscribe) visit > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080328/31bd71df/attachment.html From rgb at phy.duke.edu Fri Mar 28 05:45:41 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <47EBE5D6.7050702@scalableinformatics.com> References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> <47EBE5D6.7050702@scalableinformatics.com> Message-ID: On Thu, 27 Mar 2008, Joe Landman wrote: > When I started looking at postdoc (eek ... that long ago?) positions, it was > not uncommon to see theoretical physics PhDs earning ~16-19k/year US. No > FICA exemptions. For a family of 4, this put you under the poverty line. Um, but that was (I'm guessing) a fairly long time ago. Our grad students make that now. I think a minimum postdoc is more like $30K, and I don't think that is just at high end institutions like Duke. State institutions often have even deeper pockets. It is off topic so I manfully resisted, but I'm glad Jim or whoever asked this question as I don't think one could live in the US on a salary of $1500/month unless it were completely tax free. Post tax this is likely no more than $1100. Driving a car is likely to cost $200-300 a month, assuming that you already own one and don't have to make either payments or pay excessive taxes on it. An apartment is perhaps $600-1000/month unless you share it (far more in certain locales), and postdocs shouldn't "have" to share to survive. And then food, even for a single person, is almost certainly going to cost $10/day or more. Add it up and you're already spending your salary on room and board and transportation, leaving one nothing for clothes, fees and taxes, incidental expenses, car payments or repairs, entertainment (yes, even postdocs need vacations and entertainment). I made almost this as a miserably paid postdoc back in maybe 1983, and 25 years haven't been kind to the cost of living. > Somehow, I am not quite convinced that the science departments offering these > grasped how the invisible hand of the market would decimate their supply of > new PhD's for these positions. You can fight the market, or accept that > there is a market and deal with it. The former is not a wise course of > action (though it had been standard practice when I was looking at postdocs). > The old Young Scientists Network had formed then to discuss some of the > abuses of the market and attempts to manipulate the demand/supply curves to > oversupply the market with talent, thus keeping the price of talent low. > There is a cost to every decision, and flooding the market (then) has had > longer term effects that are being observed today. Honestly, I think it more likely that this posted salary is a typo of some sort. As I said, graduate students make that in the US, and they ARE paid "just enough" to cover room and board with the assumption that they are working 80 hours a week and don't need entertainment, that they'll supplement a bit with tutoring or TA or grading work, and that they still DO have parents or other resources they can tap to cover things like car repairs or emergency incidentals. I'm pretty sure postdocs make close to twice this much almost anywhere in the US. I'd predict negative responses to this offer. In fact, this discussion IS a negative response to the offer. It's all the more crazy given that anybody capable of running the cluster could work as a sysadmin for an absolute MINIMUM of $35-40K/year, and that as a glorified gofer working in University IT somewhere -- real sysadmins with even a year or two of experience and anything like skill certification would add $5-15K to that. Pretty much at LEAST twice as much, that is. The only possible exceptions I can think of would be a position at a tiny place out in the middle of the wilds in a community that is so small and depressed that the cost of living is half that in the US. But either way, the market being what it is, you get what you pay for. In this case, you pay for nothing, and nothing is what they're likely to get. > Back to your regularly scheduled cluster ... I'm not sure this is truly irrelevant. Non-technical, sure, but the economics of clusters is a wholistic endeavor; one of the most often omitted factors in the discussion of cluster cost-benefit is the human cost of running it. At $18K canadian (which is currently within a percent or so exchange value with the USD) this is a low-water mark for the estimated cost of a human to run a cluster, actually CHEAPER than a graduate student who would have to make this plus (somewhere, even as a bookkeeping entry ) the cost of tuition. This is order of magnitude of $100/node/year for cluster sizes of 50-200 nodes for management, down there with the cost of power and a maintenance contract, an even better deal of the postdoc ever did any real "research" on the side. I'd be very interested in whether or not they fill the position at this price. > Joe (a free-market capitalist) rgb (ditto, but remember Adam Smith's invisible hand WILL just "work") -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From hahn at mcmaster.ca Fri Mar 28 05:54:28 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: <428810f20803272246q49f853efr782194a7360b0ae0@mail.gmail.com> References: <47EC5C47.6080302@georgetown.edu> <428810f20803272246q49f853efr782194a7360b0ae0@mail.gmail.com> Message-ID: > More general term one could use is the "processing element". > For example if a cluster consists of > 2 nodes with each having 1 single core CPU, > 2 nodes with each having 1 dual core CPU, > 2 nodes with each having 1 quad core CPU, > 2 nodes with each having 2 dual core CPUs, and > 2 nodes with each having 2 quad core CPUs. > > Then we could better say that there are 38 "pocessing elements" in this > 10-nodes cluster. I find "processing element" pretentious. I'd say core, cpu or processor. how many cores happen to share a socket is largely a packaging issue, and one I'd address by discussing the app's memory size or BW needs, not packaging... From landman at scalableinformatics.com Fri Mar 28 06:37:24 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:06:57 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> <47EBE5D6.7050702@scalableinformatics.com> Message-ID: <47ECF494.5010402@scalableinformatics.com> Robert G. Brown wrote: > On Thu, 27 Mar 2008, Joe Landman wrote: > >> When I started looking at postdoc (eek ... that long ago?) positions, >> it was not uncommon to see theoretical physics PhDs earning >> ~16-19k/year US. No FICA exemptions. For a family of 4, this put you >> under the poverty line. > > Um, but that was (I'm guessing) a fairly long time ago. Our grad > students make that now. I think a minimum postdoc is more like $30K, When I was in grad school, stipends (pre-tax) were 8k-10k$ US for teaching, and maybe a little more for research. > and I don't think that is just at high end institutions like Duke. > State institutions often have even deeper pockets. Heh... not where I went :( [...] >> Somehow, I am not quite convinced that the science departments >> offering these grasped how the invisible hand of the market would >> decimate their supply of new PhD's for these positions. You can fight >> the market, or accept that there is a market and deal with it. The >> former is not a wise course of action (though it had been standard >> practice when I was looking at postdocs). The old Young Scientists >> Network had formed then to discuss some of the abuses of the market >> and attempts to manipulate the demand/supply curves to oversupply the >> market with talent, thus keeping the price of talent low. There is a >> cost to every decision, and flooding the market (then) has had longer >> term effects that are being observed today. > > Honestly, I think it more likely that this posted salary is a typo of > some sort. As I said, graduate students make that in the US, and they ... then many departments made similar typos. The departments I was in regularly lost graduate students to other departments (medical physics was the big draw in the late 80s early 90s), and many quit simply due to the ridiculous economics of the situation. Work your behind off for 5-10 years to finish up a Ph.D., then go slave away for another (several!) postdocs trying to fatten up your CV, so you have a fighting chance at a job which may become "permanent" shortly after your 35th birthday or so... and the students you teach will be making more than you within months of graduating ... When you say this out loud, and in sequence, you start to realize that you really must want to pursue this and sacrifice economically. You do a mental CBA and start asking yourself "why am I here" ? and "is it really worth it?" For some it is. For others, well, ... > ARE paid "just enough" to cover room and board with the assumption that > they are working 80 hours a week and don't need entertainment, that > they'll supplement a bit with tutoring or TA or grading work, and that > they still DO have parents or other resources they can tap to cover > things like car repairs or emergency incidentals. I'm pretty sure > postdocs make close to twice this much almost anywhere in the US. These days, postdocs (at least in CS/EE) get reasonable compensation. Don't know about physics these days. > I'd predict negative responses to this offer. In fact, this discussion > IS a negative response to the offer. It's all the more crazy given that I agree. Slave/indentured servant wages make no sense in a competitive market for talent, unless the group offering these wages *wants* to fail to attract staff. > anybody capable of running the cluster could work as a sysadmin for an > absolute MINIMUM of $35-40K/year, and that as a glorified gofer working > in University IT somewhere -- real sysadmins with even a year or two of > experience and anything like skill certification would add $5-15K to > that. Pretty much at LEAST twice as much, that is. > > The only possible exceptions I can think of would be a position at a > tiny place out in the middle of the wilds in a community that is so > small and depressed that the cost of living is half that in the US. But heh ... > either way, the market being what it is, you get what you pay for. In Yup. One customer wants Ph.D. level people with 20+ years of experience for the same as they pay their help desk people. Go figure. > this case, you pay for nothing, and nothing is what they're likely to > get. See above. Like bad RFP T&C, it is not a good idea to take a bad deal. > >> Back to your regularly scheduled cluster ... > > I'm not sure this is truly irrelevant. Non-technical, sure, but the > economics of clusters is a wholistic endeavor; one of the most often > omitted factors in the discussion of cluster cost-benefit is the human > cost of running it. At $18K canadian (which is currently within a ... when I speak with University types, this and support are often completely ignored. Often to the significant detriment of the system. FWIW: when I drove through Ontario yesterday (Buffalo to Detroit), the USD = 0.96 CDN (that and the Tim Horton's coffee isn't that bad ... if you want bad, try McD's). -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From glen.beane at jax.org Fri Mar 28 07:00:15 2008 From: glen.beane at jax.org (Glen Beane) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <47ECF494.5010402@scalableinformatics.com> References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> <47EBE5D6.7050702@scalableinformatics.com> <47ECF494.5010402@scalableinformatics.com> Message-ID: On Mar 28, 2008, at 9:37 AM, Joe Landman wrote: > > > FWIW: when I drove through Ontario yesterday (Buffalo to Detroit), > the USD = 0.96 CDN (that and the Tim Horton's coffee isn't that > bad ... if you want bad, try McD's). we have quite few Tim Horton's here in Maine and one thing I've noticed is that a Tim Horton's coffee in Canada is a size smaller than the "same size" Tim Horton's small coffee in the States. So a Canadian small is smaller than the US small, a Canadian medium = US small, Canadian large = US Medium so if you take this post doc in Canada not only will you put up with a slave wage, but you'll have to make more trips to Tim Horton's for your caffeine fix -- Glen L. Beane Software Engineer The Jackson Laboratory Phone (207) 288-6153 From landman at scalableinformatics.com Fri Mar 28 07:04:49 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> <47EBE5D6.7050702@scalableinformatics.com> <47ECF494.5010402@scalableinformatics.com> Message-ID: <47ECFB01.1040000@scalableinformatics.com> Glen Beane wrote: > so if you take this post doc in Canada not only will you put up with a > slave wage, but you'll have to make more trips to Tim Horton's for your > caffeine fix Sadly I have developed a taste for/addiction to Kona coffee (and many other high quality beans grown in limited supply), which requires wages far beyond those of most postdocs .... So simply for that reason alone .... > > > -- > Glen L. Beane > Software Engineer > The Jackson Laboratory > Phone (207) 288-6153 > > -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From rgb at phy.duke.edu Fri Mar 28 07:00:38 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: References: Message-ID: On Fri, 28 Mar 2008, Kalpana Kanthasamy wrote: > Hi, > > I am kinda confused with term cluster and now am sure if this where I can > post my question. What exactly can be defined as a cluster ?, We have a 32 > node cluster in our lab. and the number 32 here is referred to 32 separate > machines ( each with its own processor ). What about the new machines > like, > dual core processors or the quad core processors, are those considered > clusters also. For example, should we refer to a quad core processor as a > cluster of 4 processors. Hope someone out there could explain. Thanks A "cluster" generally refers to a cluster of separate machines using a network as an interconnect. A machine with multiple processors (or multiple cores, or multiple multiple core processors) isn't a cluster per se UNLESS it is internally architected on top of what amounts to a network. There have been supercomputer designs that are -- IBM's SP series for example, or the CM series in the old days -- and "bus" oriented designs that put the various processors on a parallel structure that isn't precisely "a network" as well. Nowadays the nature of single motherboards has become even more architecturally blurred as e.g. Hypertransport has features that look like like a network as much as a bus, so one could argue that AMD systems with many cores that use it are "a cluster" from a certain architectural point of view. However, on THIS list, the issue of what is or isn't "a cluster node" has to be taken in context. A "node" (I think) most commonly refers to a single physical box (motherboard) with one or more network connections. By common usage, we don't consider a single multicore box to be a cluster per se, although we certainly wouldn't care if somebody did as the issues facing the user of such a box are ALMOST ALL the same issues that would need to be faced for multiple node clusters. Ultimately, we view "a cluster" as ultimately being a collection of CPU cores that can be in two or more physical boxes containing UP systems (one core, one processor per motherboard/box) all the way up to 16-way SMP (four quad cores per motherboard/box) or whatever the current max-pack boxes are, connected by some sort of COTS network, which historically may well be not terribly OTS, unless you consider single-source direct purchases to be "shelves". The point is that PROGRAMMING a cluster efficiently requires viewing the entire collection of available processor cores as the resource, with a differential matrix of interprocessor communications (IPCs) with very specific costs/speeds (latencies and bandwidths) associated with each link, with nonlinear aspects of communication such as just how IPCs onboard work vs IPCs between nodes all accounted for. Multicores and large SMP systems have actually rather complicated cluster design, because other bottlenecks emerge that are strongly differential for various cluster purposes, such as how memory bandwidth and latency fares when many cores are trying to access a large shared memory resource with or without locking, processor affinity, NUMA, and more. With a pile of UP PC's on a single network, there were really just a handful of really important rates -- latency and bw to main memory in streaming and non-streaming (cache defeating) memory access patterns, raw CPU clock and its relation to various integer and floating point rates, and latency and bw between nodes. Now one has to worry about core to core on the same die, core to core on different dies, core to "local" memory with affinity, core to memory that "belongs" to other cores or dies -- quite a range of possible bottlenecking rates between cores on ONE PHYSICAL MOTHERBOARD -- and THEN one still has to worry about the internode network and network-based IPCs, which in turn depends on how the cores write to e.g. shared network devices! My advice is to START with the simple model for a cluster -- one CPU core/node, one network -- to get to where you understand the BASIC principles of bottlenecks, the differential effect of latency and bandwidth for different kinds of parallel code, and so on. When you've mastered the ideas at this level (with the help of the list as needed), THEN this list is an excellent resource for learning more advanced topics as well. The ongoing discussion of NUMA and processor affinity has been most instructive, for example, although I lack the time to really experiment with a lot of what was discussed. rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From james.p.lux at jpl.nasa.gov Fri Mar 28 07:24:46 2008 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> <47EBE5D6.7050702@scalableinformatics.com> Message-ID: <20080328072446.b2v69zlbkscsk0g8@webmail.jpl.nasa.gov> Quoting "Robert G. Brown" , on Fri 28 Mar 2008 05:45:41 AM PDT: Some tongue in cheek comments below > It is off topic so I manfully resisted, but I'm glad Jim or whoever > asked this question as I don't think one could live in the US on a > salary of $1500/month unless it were completely tax free. Post tax this > is likely no more than $1100. Driving a car is likely to cost $200-300 > a month, assuming that you already own one and don't have to make either > payments or pay excessive taxes on it. Off campus apartment in the student ghetto within walking/public transit distance? An apartment is perhaps > $600-1000/month unless you share it (far more in certain locales), and > postdocs shouldn't "have" to share to survive. Why not? The carwasheros working for tips and migrant farmworkers following the crops do it. Research work and grants are just another crop, and your fingernails don't get as dirty, but you don't get to spend time in the healthy outdoors. Adversity inspires creativity, or something like that. And then food, even for > a single person, is almost certainly going to cost $10/day or more. Here, the migrant farmworker DOES have an advantage since they're standing in the midst of the food. A 70 pound sack of oats runs about $15-20 at the feed store (2x-3x times that at the health food store), and I can speak from personal experience that one can eat oatmeal for many, many days from that sack. And what about Ramen noodles? Add > it up and you're already spending your salary on room and board and > transportation, leaving one nothing for clothes, One really needs to buy your bulk oats in cloth bags, so you can wear them to the lab. The modern trend towards those sort of poly fabric materials is really putting a crimp in "dustbowl farm chic" clothing. No more soft muslin flour sacks or burlap sacks. fees and taxes, > incidental expenses, car payments or repairs, What car? entertainment (yes, even > postdocs need vacations and entertainment). The sheer joy of research and creation aren't enough? Back to the salt mines, you slacker. At least you're not digging Emeralds in Colombia. > > Honestly, I think it more likely that this posted salary is a typo of > some sort. I thought the same. that's why I asked. > I'm not sure this is truly irrelevant. Non-technical, sure, but the > economics of clusters is a wholistic endeavor; one of the most often > omitted factors in the discussion of cluster cost-benefit is the human > cost of running it. At $18K canadian (which is currently within a > percent or so exchange value with the USD) this is a low-water mark for > the estimated cost of a human to run a cluster, actually CHEAPER than a > graduate student who would have to make this plus (somewhere, even as a > bookkeeping entry ) the cost of tuition This is order of magnitude of > $100/node/year for cluster sizes of 50-200 nodes for management, down > there with the cost of power and a maintenance contract, an even better > deal of the postdoc ever did any real "research" on the side. I'd be > very interested in whether or not they fill the position at this price. And this is why a standard sort of "per desktop computer" fee of a couple hundred bucks a month in most companies isn't all that unreasonable. > >> Joe (a free-market capitalist) > > rgb (ditto, but remember Adam Smith's invisible hand WILL just "work") But not necessarily in a way that will be pleasant or desirable for YOU. And I don't know that Smith contemplated the concept of multiple hands with mutual interactions. Jim From james.p.lux at jpl.nasa.gov Fri Mar 28 07:29:41 2008 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> <47EBE5D6.7050702@scalableinformatics.com> <47ECF494.5010402@scalableinformatics.com> Message-ID: <20080328072941.5owyc60rok4gok0s@webmail.jpl.nasa.gov> Quoting Glen Beane , on Fri 28 Mar 2008 07:00:15 AM PDT: > > so if you take this post doc in Canada not only will you put up with a > slave wage, but you'll have to make more trips to Tim Horton's for your > caffeine fix At that wage, you're going to be making hot beverages from the bark you strip from trees on campus, heated over a fire made from the wood from those trees. Coffee would be something you hope to drink if you ever get funded to go to a conference where it's provided free. From landman at scalableinformatics.com Fri Mar 28 07:38:17 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <20080328072941.5owyc60rok4gok0s@webmail.jpl.nasa.gov> References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> <47EBE5D6.7050702@scalableinformatics.com> <47ECF494.5010402@scalableinformatics.com> <20080328072941.5owyc60rok4gok0s@webmail.jpl.nasa.gov> Message-ID: <47ED02D9.9030508@scalableinformatics.com> Jim Lux wrote: > Quoting Glen Beane , on Fri 28 Mar 2008 07:00:15 AM > PDT: > >> >> so if you take this post doc in Canada not only will you put up with a >> slave wage, but you'll have to make more trips to Tim Horton's for your >> caffeine fix > > At that wage, you're going to be making hot beverages from the bark you > strip from trees on campus, heated over a fire made from the wood from > those trees. Coffee would be something you hope to drink if you ever > get funded to go to a conference where it's provided free. I seem to remember some of my fellow students looking at how to avoid buying food by going to the assorted colloquium on campus, and eating (and stashing) the food. Sadly, the food they put out was high fat/high sugar, low nutritional value, which led to badly malnourished graduate students ... who were twitching badly from all the free (bad) coffee ... -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From jlb17 at duke.edu Fri Mar 28 08:16:00 2008 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: <47EC6DBF.1080705@gmail.com> References: <47EC6DBF.1080705@gmail.com> Message-ID: On Thu, 27 Mar 2008 at 11:02pm, Geoff Jacobs wrote > Ricardo Reis wrote: >> >> Hi all >> >> I beg to take advantage of your experience although the topic isn't >> completly cluster thing. I got some money to buy a new machine, at least >> 8Gb and I'm thinking between a 2 x dual core or a 1 x quad (or even 2x >> quads). It must be one machine because it must (urgh) be able to use >> it's 8Gb in serial codes (don't ask). Anyway, I've been experiencing >> with paraview for parallel visualization and was wondering on your >> opinion on... buying a ultra-duper-cool state-of-the-art graphic card >> (Nvidia) or 2 graphic cards? >> >> thanks for your time, >> >> Ricardo Reis > > No nvidia for me unless they open up driver specs. Open drivers have a > history of being very, very stable if not as fast. Nvidia does have good > drivers, but I have definitely seen them experience problems. For truly high-end 3D, there is (still, unfortunately) really no alternative to closed source drivers. nouveua (for nvidia hardware) isn't there yet, and the open source radeon stuff doesn't support any of their newer hardware. Also, AFAIK, neither project supports the professional series of cards (Quadro/FireGL). Touch wood, I've had mostly good experience with the nvidia drivers where I've had to use them. -- Joshua Baker-LePain QB3 Shared Cluster Sysadmin UCSF From hahn at mcmaster.ca Fri Mar 28 09:26:09 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: References: <47EC6DBF.1080705@gmail.com> Message-ID: > Also, AFAIK, neither project supports the professional series of cards > (Quadro/FireGL). I'm pretty sure I normally just install the unified NV driver for quadros. (course, afaikt the "pro" distinction is just marketing spew.) From jlb17 at duke.edu Fri Mar 28 09:46:53 2008 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: References: <47EC6DBF.1080705@gmail.com> Message-ID: On Fri, 28 Mar 2008 at 12:26pm, Mark Hahn wrote >> Also, AFAIK, neither project supports the professional series of cards >> (Quadro/FireGL). > > I'm pretty sure I normally just install the unified NV driver for quadros. Sure, but that driver doesn't do hardware-accelerated 3D. > (course, afaikt the "pro" distinction is just marketing spew.) The nvidia man page lists features only supported on Quadro cards (and which I've also seen listed as used by "pro" apps (high-end CAD/CAM, etc)). But I've also never seen a benchmark showing how much (if any) difference they make. -- Joshua Baker-LePain QB3 Shared Cluster Sysadmin UCSF From hahn at mcmaster.ca Fri Mar 28 10:40:57 2008 From: hahn at mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: References: <47EC6DBF.1080705@gmail.com> Message-ID: >>> Also, AFAIK, neither project supports the professional series of cards >>> (Quadro/FireGL). >> >> I'm pretty sure I normally just install the unified NV driver for quadros. > > Sure, but that driver doesn't do hardware-accelerated 3D. ah - I wasn't thinking. by NV, I meant to abbreviate the company name (and refer to the closed driver that calls itself "nvidia") rather than the xorg driver which calls itself "nv"... the nvidia driver does provide hw accell on quadro cards, which are just minor variants of non-quadro versions. I haven't had problems with either nv or nvidia drivers. From gdjacobs at gmail.com Fri Mar 28 11:57:52 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <20080328072446.b2v69zlbkscsk0g8@webmail.jpl.nasa.gov> References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> <47EBE5D6.7050702@scalableinformatics.com> <20080328072446.b2v69zlbkscsk0g8@webmail.jpl.nasa.gov> Message-ID: <47ED3FB0.2010205@gmail.com> Jim Lux wrote: > Quoting "Robert G. Brown" , on Fri 28 Mar 2008 > 05:45:41 AM PDT: > > Some tongue in cheek comments below > > >> It is off topic so I manfully resisted, but I'm glad Jim or whoever >> asked this question as I don't think one could live in the US on a >> salary of $1500/month unless it were completely tax free. Post tax this >> is likely no more than $1100. Driving a car is likely to cost $200-300 >> a month, assuming that you already own one and don't have to make either >> payments or pay excessive taxes on it. > > Off campus apartment in the student ghetto within walking/public transit > distance? > > An apartment is perhaps >> $600-1000/month unless you share it (far more in certain locales), and >> postdocs shouldn't "have" to share to survive. > > Why not? The carwasheros working for tips and migrant farmworkers > following the crops do it. Research work and grants are just another > crop, and your fingernails don't get as dirty, but you don't get to > spend time in the healthy outdoors. Adversity inspires creativity, or > something like that. > > And then food, even for >> a single person, is almost certainly going to cost $10/day or more. > > Here, the migrant farmworker DOES have an advantage since they're > standing in the midst of the food. A 70 pound sack of oats runs about > $15-20 at the feed store (2x-3x times that at the health food store), > and I can speak from personal experience that one can eat oatmeal for > many, many days from that sack. And what about Ramen noodles? > > > Add >> it up and you're already spending your salary on room and board and >> transportation, leaving one nothing for clothes, > > One really needs to buy your bulk oats in cloth bags, so you can wear > them to the lab. The modern trend towards those sort of poly fabric > materials is really putting a crimp in "dustbowl farm chic" clothing. No > more soft muslin flour sacks or burlap sacks. > > > fees and taxes, >> incidental expenses, car payments or repairs, > > What car? > > entertainment (yes, even >> postdocs need vacations and entertainment). > > The sheer joy of research and creation aren't enough? Back to the salt > mines, you slacker. At least you're not digging Emeralds in Colombia. > >> >> Honestly, I think it more likely that this posted salary is a typo of >> some sort. > > I thought the same. that's why I asked. > > > > >> I'm not sure this is truly irrelevant. Non-technical, sure, but the >> economics of clusters is a wholistic endeavor; one of the most often >> omitted factors in the discussion of cluster cost-benefit is the human >> cost of running it. At $18K canadian (which is currently within a >> percent or so exchange value with the USD) this is a low-water mark for >> the estimated cost of a human to run a cluster, actually CHEAPER than a >> graduate student who would have to make this plus (somewhere, even as a >> bookkeeping entry ) the cost of tuition > > > This is order of magnitude of >> $100/node/year for cluster sizes of 50-200 nodes for management, down >> there with the cost of power and a maintenance contract, an even better >> deal of the postdoc ever did any real "research" on the side. I'd be >> very interested in whether or not they fill the position at this price. > > And this is why a standard sort of "per desktop computer" fee of a > couple hundred bucks a month in most companies isn't all that unreasonable. > > > > > >> >>> Joe (a free-market capitalist) >> >> rgb (ditto, but remember Adam Smith's invisible hand WILL just "work") > > But not necessarily in a way that will be pleasant or desirable for YOU. > And I don't know that Smith contemplated the concept of multiple hands > with mutual interactions. > > Jim Have you ever considered a turn on the comedy circuit? This is brilliant sardonic humour. -- Geoffrey D. Jacobs From lindahl at pbm.com Fri Mar 28 15:25:36 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: References: <47EC6DBF.1080705@gmail.com> Message-ID: <20080328222536.GA1389@bx9.net> On Fri, Mar 28, 2008 at 01:40:57PM -0400, Mark Hahn wrote: > I haven't had problems with either nv or nvidia drivers. nvidia is somewhat annoying to update to new versions or new kernels. If you use the arguments "-a --no-x-check -K -k" it will run in a window, but you still have to install the first time manually, and updates have to be run one by one... although I suppose I could run it in a screen or something... -- greg From jlb17 at duke.edu Fri Mar 28 15:41:20 2008 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: <20080328222536.GA1389@bx9.net> References: <47EC6DBF.1080705@gmail.com> <20080328222536.GA1389@bx9.net> Message-ID: On Fri, 28 Mar 2008 at 3:25pm, Greg Lindahl wrote > On Fri, Mar 28, 2008 at 01:40:57PM -0400, Mark Hahn wrote: > >> I haven't had problems with either nv or nvidia drivers. > > nvidia is somewhat annoying to update to new versions or new kernels. > > If you use the arguments "-a --no-x-check -K -k" it will run in a > window, but you still have to install the first time manually, and > updates have to be run one by one... although I suppose I could run it > in a screen or something... Depending on your distribution, there are various folks packaging up the drivers in more convenient forms than nvidia's installer. The livna RPMs for Fedora, e.g., work very well. There's also the option of tying it into DKMS. -- Joshua Baker-LePain QB3 Shared Cluster Sysadmin UCSF From lindahl at pbm.com Fri Mar 28 17:47:13 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: References: <47EC6DBF.1080705@gmail.com> <20080328222536.GA1389@bx9.net> Message-ID: <20080329004713.GA24211@bx9.net> On Fri, Mar 28, 2008 at 06:41:20PM -0400, Joshua Baker-LePain wrote: > Depending on your distribution, there are various folks packaging up the > drivers in more convenient forms than nvidia's installer. The livna RPMs > for Fedora, e.g., work very well. Alas, I haven't found such a thing for RHEL/CentOS. -- greg From jlb17 at duke.edu Fri Mar 28 18:22:15 2008 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: <20080329004713.GA24211@bx9.net> References: <47EC6DBF.1080705@gmail.com> <20080328222536.GA1389@bx9.net> <20080329004713.GA24211@bx9.net> Message-ID: On Fri, 28 Mar 2008 at 5:47pm, Greg Lindahl wrote > On Fri, Mar 28, 2008 at 06:41:20PM -0400, Joshua Baker-LePain wrote: > >> Depending on your distribution, there are various folks packaging up the >> drivers in more convenient forms than nvidia's installer. The livna RPMs >> for Fedora, e.g., work very well. > > Alas, I haven't found such a thing for RHEL/CentOS. Scientific Linux seems to provide packages for it (via dkms). Not having used them myself, I'd be interested to hear if/how well they work in CentOS. -- Joshua Baker-LePain QB3 Shared Cluster Sysadmin UCSF From csamuel at vpac.org Fri Mar 28 22:40:05 2008 From: csamuel at vpac.org (Chris Samuel) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <46622920.198071206769055096.JavaMail.root@zimbra.vpac.org> Message-ID: <307286415.198091206769205947.JavaMail.root@zimbra.vpac.org> ----- "Jim Lux" wrote: > At that wage, you're going to be making hot beverages from the bark > you strip from trees on campus, heated over a fire made from the wood > from those trees. That seems to describe most of what passes for coffee that I have encountered in the US, which always surprises me given that you folks have such a mythos around it. :-) Perhaps I'm just spoiled being in the second[0] most livable city in the world[1] where good coffee is taken very seriously [2][3]. cheers, Chris [0] - Damn you Vancouver, we used to be No.1! [1] - http://www.economist.com/markets/rankings/displaystory.cfm?story_id=8908454 [2] - http://www.melbournecoffeereview.com/ [3] - http://www.cremamagazine.com.au/page/vic.html -- Christopher Samuel - (03) 9925 4751 - Systems Manager The Victorian Partnership for Advanced Computing P.O. Box 201, Carlton South, VIC 3053, Australia VPAC is a not-for-profit Registered Research Agency From landman at scalableinformatics.com Sat Mar 29 05:12:42 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <307286415.198091206769205947.JavaMail.root@zimbra.vpac.org> References: <307286415.198091206769205947.JavaMail.root@zimbra.vpac.org> Message-ID: <47EE323A.9050402@scalableinformatics.com> Chris Samuel wrote: > ----- "Jim Lux" wrote: > >> At that wage, you're going to be making hot beverages from the bark >> you strip from trees on campus, heated over a fire made from the wood >> from those trees. > > That seems to describe most of what passes for coffee that > I have encountered in the US, which always surprises me > given that you folks have such a mythos around it. :-) :( There is good coffee, you just have to search for it. Austin Texas (for those going to attend SC08) has some good coffee places, though they are a little ways away from the convention center. > Perhaps I'm just spoiled being in the second[0] most > livable city in the world[1] where good coffee is taken > very seriously [2][3]. Please understand that one of the verbs applied to coffee here is "ground", which, when tasting some of it, makes me thing it may actually be a noun ... -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From 3lucid at gmail.com Sat Mar 29 06:53:41 2008 From: 3lucid at gmail.com (Kyle Spaans) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <47EE323A.9050402@scalableinformatics.com> References: <307286415.198091206769205947.JavaMail.root@zimbra.vpac.org> <47EE323A.9050402@scalableinformatics.com> Message-ID: <5a1205b30803290653p12eafbf1n2533b03866f476eb@mail.gmail.com> I'd work for that much, granted I'm only in my second year of undergrad studies. It'd only do it as a labour of love though, since I make more money than that per month on my 4-month co-op jobs doing various monkey work in various places. From ascheinine at tuffmail.us Sat Mar 29 09:09:10 2008 From: ascheinine at tuffmail.us (Alan Louis Scheinine) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: References: <47EC5C47.6080302@georgetown.edu> <428810f20803272246q49f853efr782194a7360b0ae0@mail.gmail.com> Message-ID: <47EE69A6.6040504@tuffmail.us> I recommend giving two facts number of nodes number of cores A node is a box of CPUs that share memory. A core is, well, what everyone calls a core. In my opinion, the most ambiguous term is the number of CPU's because some people use the term CPU to refer to the number of sockets and others use the term to refer to the number of cores. For a more fine-grained description you can cite, in addition, the number of sockets and the number of cores per socket. best regards, Alan From tjrc at sanger.ac.uk Sun Mar 30 02:35:46 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: <47EE69A6.6040504@tuffmail.us> References: <47EC5C47.6080302@georgetown.edu> <428810f20803272246q49f853efr782194a7360b0ae0@mail.gmail.com> <47EE69A6.6040504@tuffmail.us> Message-ID: <37121373-67A3-4D26-A719-91A5B99154AB@sanger.ac.uk> On 29 Mar 2008, at 4:09 pm, Alan Louis Scheinine wrote: > I recommend giving two facts > number of nodes > number of cores > > A node is a box of CPUs that share memory. > A core is, well, what everyone calls a core. > In my opinion, the most ambiguous term is the > number of CPU's because some people use the term CPU > to refer to the number of sockets and others use > the term to refer to the number of cores. > > For a more fine-grained description you can > cite, in addition, the number of sockets and > the number of cores per socket. Just to be awkward, there are of course machines like the SGI Altix. Is it a cluster, or is it a node? From the programmer's perspective it's the latter, from the architectural perspective, the former. There's no real dividing line; there are machines across the entire spectrum. But I'm just being difficult, generally I agree with what the others have said. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From i.kozin at dl.ac.uk Sun Mar 30 10:54:45 2008 From: i.kozin at dl.ac.uk (Kozin, I (Igor)) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: <37121373-67A3-4D26-A719-91A5B99154AB@sanger.ac.uk> Message-ID: > Just to be awkward, there are of course machines like the SGI Altix. > Is it a cluster, or is it a node? From the programmer's perspective > it's the latter, from the architectural perspective, the former. > > There's no real dividing line; there are machines across the entire > spectrum. But I'm just being difficult, generally I agree with what > the others have said. I'd say definitely the former - a cluster where the nodes (two sockets) are connected using Numalink. I have been thinking long and hard about those classification issues when I was designing sufficiently generic but consistent machine description for our benchmarking database. The view that I took is that there is a "node" which is a reasonably independent entity and you take those nodes and glue them together using some sort of "interconnect". So far this architectural view has not failed me but it is not impossible that future machines might have more intricate "node" or "interconnect" structure and therefore require more sophisticated approach. Programmer's view can't be used for classification purposes because there can be several views on the same hardware at the same time. The Altix example is admittedly blurred but even a traditional cluster can be programmed a la SMP using Cluster OpenMP or other similar approaches which obviously do not make the underlying hardware any different. As for the terms processing element (PE) and core, PE was clearly preferable for machine description rather than core because PE is more generic whereas the term "core" usually implies a fully functional processor that has been shrank. Terminologically PE may include processor cores, SPEs in Cell BE, GPGPUs attached to the node or accelerating co-processors. From rgb at phy.duke.edu Sun Mar 30 11:59:55 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: References: Message-ID: On Sun, 30 Mar 2008, Kozin, I (Igor) wrote: > > >> Just to be awkward, there are of course machines like the SGI Altix. >> Is it a cluster, or is it a node? From the programmer's perspective >> it's the latter, from the architectural perspective, the former. >> >> There's no real dividing line; there are machines across the entire >> spectrum. But I'm just being difficult, generally I agree with what >> the others have said. > > I'd say definitely the former - a cluster where the nodes (two sockets) > are connected using Numalink. > > I have been thinking long and hard about those classification issues > when I was designing sufficiently generic but consistent machine > description for our benchmarking database. The view that I took is that > there is a "node" which is a reasonably independent entity and you take > those nodes and glue them together using some sort of "interconnect". So > far this architectural view has not failed me but it is not impossible > that future machines might have more intricate "node" or "interconnect" > structure and therefore require more sophisticated approach. > > Programmer's view can't be used for classification purposes because > there can be several views on the same hardware at the same time. The > Altix example is admittedly blurred but even a traditional cluster can > be programmed a la SMP using Cluster OpenMP or other similar approaches > which obviously do not make the underlying hardware any different. This is really the crucial point. There are machines that are clusters, nodes that are clusters, clusters of clusters, clusters of nodes -- all of that is nomenclature that in some sense "doesn't matter". One can fairly easily specify the architecture of any given pile of PCs (one of the most primitive views of a cluster:-) in direct descriptive detail. Processor core(s), packaging/socketing, low level bus and/or net, motherboard design, network and other peripherals. Naming something "a node" or "a processor" or "a core" unaccompanied by an in-context definition of what the terms mean for THIS cluster is not necessary. The real point is that whatever you call it, that pile of PCs is designed/intended to do parallel work. The tools for writing programs for doing that parallel work impose various hardware related constraints on those programs that have to be accomodated by a programmer for each particular cluster (where how much it MATTERS depends of course on the nature of the parallel work being done and how it "fits" the hardware. A 16-core single motherboard machine may or may not be "a cluster", but you're very probably going to program your parallel applications on it using the same tools you'd use to program it on four four-core machines, or 16 uniprocessor machines. You might write the code very differently; the balance of processor speed to IPCs and bottlenecks to critical resources might be quite different, but we have learned the hard way that programming to particular architectures at the deep hardware level is a good way to go broke fast as it is obsoleted by Moore's law, while code written generically for a good COTS cluster architecture can ride from one cluster to another (with tweaks to serious rewrites, sure, but using the same toolset and mental picture of what's going on) as hardware and networks evolve. So I personally agree with the consensus view here -- the "best" use of nomenclature is that nodes are connected to nodes by COTS networks, and anything in a single box with a proprietary internal network interconnect on a single motherboard is a node, even though its ARCHITECTURE on the motherboard may for all practical purposes itself be "a cluster". I've taken to calling what USED to be a processor "a core" because processor seems to refer to a single package (as in "multicore processor"). Just how the cores are packaged into processors, put onto motherboards and connected to memory and perpherials, and motherboards interconnected into clusters using networks -- all of that is just part of the generic (now more complex) cluster description. > As for the terms processing element (PE) and core, PE was clearly > preferable for machine description rather than core because PE is more > generic whereas the term "core" usually implies a fully functional > processor that has been shrank. Terminologically PE may include > processor cores, SPEs in Cell BE, GPGPUs attached to the node or > accelerating co-processors. Fair enough. Viewing this as a "proposal" or "RFC" rather than a de facto truth, it seems reasonable to differentiate at this level, but since cores abound and PEs that are not also cores do not, don't expect most people to not just talk about cores and processors. I will try to use PE (suitably defined) in future discussions that involve a wider array of PE options than "just" cores. But I'm guessing people will still know what I'm talking about from context if I forget. rgb -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From deadline at eadline.org Sun Mar 30 12:27:39 2008 From: deadline at eadline.org (Douglas Eadline) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: References: Message-ID: <49170.192.168.1.1.1206905259.squirrel@mail.eadline.org> My short operational definition of a cluster is collection of independent systems (motherboards) that communicate over a private network/interconnect, none of which are dependent on the other for operating, with excetion of the head/master/gateway node. Under this definition, a multi-core node is not a cluster nor is any design whose operation is dependent upon shared memory between nodes, even though the nodes are separate systems. Of course this is my definition. BTW, I believe the following is also used: N=nodes P=cores/processors per node P>=N = cluster N

Hi, > > I am kinda confused with term cluster and now am sure if this where I can > post my question. What exactly can be defined as a cluster ?, We have a 32 > node cluster in our lab. and the number 32 here is referred to 32 separate > machines ( each with its own processor ). What about the new machines > like, > dual core processors or the quad core processors, are those considered > clusters also. For example, should we refer to a quad core processor as a > cluster of 4 processors. Hope someone out there could explain. Thanks > > > !DSPAM:47ec49f7195876865219710! > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > > !DSPAM:47ec49f7195876865219710! > -- Doug From gerry.creager at tamu.edu Mon Mar 31 05:41:11 2008 From: gerry.creager at tamu.edu (Gerry Creager) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] Confused over the term cluster In-Reply-To: <49170.192.168.1.1.1206905259.squirrel@mail.eadline.org> References: <49170.192.168.1.1.1206905259.squirrel@mail.eadline.org> Message-ID: <47F0DBE7.6060302@tamu.edu> Didn't we hash through this a few years ago, too? I seem to recall a similar definition, but one that specifically indicated that shared memory wasn't an element of a Beowulf Cluster? Didn't that make it into Kragen's FAQ? gerry Douglas Eadline wrote: > My short operational definition of a cluster is collection of > independent systems (motherboards) that communicate > over a private network/interconnect, none of which > are dependent on the other for operating, with > excetion of the head/master/gateway node. > > Under this definition, a multi-core node is not a cluster > nor is any design whose operation is dependent > upon shared memory between nodes, even though the > nodes are separate systems. Of course this is my > definition. > > BTW, I believe the following is also used: > > N=nodes > P=cores/processors per node > > P>=N = cluster > N

> > -- > Doug > > >> Hi, >> >> I am kinda confused with term cluster and now am sure if this where I can >> post my question. What exactly can be defined as a cluster ?, We have a 32 >> node cluster in our lab. and the number 32 here is referred to 32 separate >> machines ( each with its own processor ). What about the new machines >> like, >> dual core processors or the quad core processors, are those considered >> clusters also. For example, should we refer to a quad core processor as a >> cluster of 4 processors. Hope someone out there could explain. Thanks >> >> >> !DSPAM:47ec49f7195876865219710! >> _______________________________________________ >> Beowulf mailing list, Beowulf@beowulf.org >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf >> >> >> !DSPAM:47ec49f7195876865219710! >> > > > -- > Doug > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From prentice at ias.edu Mon Mar 31 05:51:03 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: <47EC6DBF.1080705@gmail.com> References: <47EC6DBF.1080705@gmail.com> Message-ID: <47F0DE37.9080300@ias.edu> Geoff Jacobs wrote: > Ricardo Reis wrote: >> Hi all >> >> I beg to take advantage of your experience although the topic isn't >> completly cluster thing. I got some money to buy a new machine, at least >> 8Gb and I'm thinking between a 2 x dual core or a 1 x quad (or even 2x >> quads). It must be one machine because it must (urgh) be able to use >> it's 8Gb in serial codes (don't ask). Anyway, I've been experiencing >> with paraview for parallel visualization and was wondering on your >> opinion on... buying a ultra-duper-cool state-of-the-art graphic card >> (Nvidia) or 2 graphic cards? >> >> thanks for your time, >> >> Ricardo Reis > > No nvidia for me unless they open up driver specs. Open drivers have a > history of being very, very stable if not as fast. Nvidia does have good > drivers, but I have definitely seen them experience problems. > Ricard,o If you want the best visual performance on Linux, you can't do better than an nvidia card. The nvidia-supplied closed-source linux driver provide performance just as good as you'd get with Windows. Not sure what the current state of the ATI linux drivers are. Geoff, what problems did you have with the nvidia drivers? In my previous position (which I was at until only a few months ago), I used nvidia's closed source drivers to provide 3D hardware stereo (LCD shutter glasses with IR emitters), to provide 3D visualization of protein molecules. I used nvidia QuadroFX 3450 cards with nvidia's proprietary linux drivers. In 3.5 years, never had a single driver problem. Even when using advanced features like overlay and FSAA, I never had a problem. What's wrong with NVidia's closed source drivers? NVidia was the first graphics company to provide high-performance drivers themselves for their video cards on Linux. For years, if you wanted decent 3D acceleration on Linux, nvidia cards with the nvidia-supplied drivers was the only way to go. nvidia was the first video card co. to really support Linux. I think nvidia should be applauded for that. Prentice From prentice at ias.edu Mon Mar 31 06:09:09 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: References: <47EC6DBF.1080705@gmail.com> Message-ID: <47F0E275.6000202@ias.edu> Joshua Baker-LePain wrote: > On Fri, 28 Mar 2008 at 12:26pm, Mark Hahn wrote > >>> Also, AFAIK, neither project supports the professional series of >>> cards (Quadro/FireGL). >> >> I'm pretty sure I normally just install the unified NV driver for >> quadros. > > Sure, but that driver doesn't do hardware-accelerated 3D. > >> (course, afaikt the "pro" distinction is just marketing spew.) > > The nvidia man page lists features only supported on Quadro cards (and > which I've also seen listed as used by "pro" apps (high-end CAD/CAM, > etc)). But I've also never seen a benchmark showing how much (if any) > difference they make. Nvidiai's "Pro" cards are the only models that support 3D hardware stereo. If you need that, that's a big performance difference. Prentice From prentice at ias.edu Mon Mar 31 06:14:28 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: <20080328222536.GA1389@bx9.net> References: <47EC6DBF.1080705@gmail.com> <20080328222536.GA1389@bx9.net> Message-ID: <47F0E3B4.4050208@ias.edu> Greg Lindahl wrote: > On Fri, Mar 28, 2008 at 01:40:57PM -0400, Mark Hahn wrote: > >> I haven't had problems with either nv or nvidia drivers. > > nvidia is somewhat annoying to update to new versions or new kernels. > This is easily fixed if you use HP's rpm of the drivers, which includes the /etc/init.d/nvconfig script which detects if the there is a driver module for the current version at boot time. If not, it downloads or compiles it. Or, you can use Dell's DKMS system, which uses a different approach to solve the same method. Both of these systems will work on non-HP or non-Dell systems, too. I used the HP system for 3.5 years at a previous employer without a single hiccup. I'm using the Dell DKMS method, since November, and I have no complaints. Prentice From prentice at ias.edu Mon Mar 31 06:16:19 2008 From: prentice at ias.edu (Prentice Bisbal) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: <20080329004713.GA24211@bx9.net> References: <47EC6DBF.1080705@gmail.com> <20080328222536.GA1389@bx9.net> <20080329004713.GA24211@bx9.net> Message-ID: <47F0E423.8080802@ias.edu> Greg Lindahl wrote: > On Fri, Mar 28, 2008 at 06:41:20PM -0400, Joshua Baker-LePain wrote: > >> Depending on your distribution, there are various folks packaging up the >> drivers in more convenient forms than nvidia's installer. The livna RPMs >> for Fedora, e.g., work very well. > > Alas, I haven't found such a thing for RHEL/CentOS. > > -- greg > DKMS works with RHEL/CentOS, as does the HP's /etc/init.d/nvconfig script which comes with the HP-supplied RPM. Prentice From landman at scalableinformatics.com Mon Mar 31 06:17:02 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: <47EC6DBF.1080705@gmail.com> References: <47EC6DBF.1080705@gmail.com> Message-ID: <47F0E44E.90800@scalableinformatics.com> Geoff Jacobs wrote: > No nvidia for me unless they open up driver specs. Open drivers have a > history of being very, very stable if not as fast. Nvidia does have good > drivers, but I have definitely seen them experience problems. Silicon Image SATA drivers are great examples of open drivers that have been (in the past) terrible. The tg3 drivers have been very bad (start sending lots of packets and watch your CSW climb and climb and climb). Forcedeth are still terrible, and are open. The nv driver in X often breaks on newer hardware (the laptop I type this on is a testament to this problem). No, open-ness doesn't equate to good-ness. Open-ness equates to portability, ability to hunt for problems on your own and correct them if need be. For the platforms I have used them on the nVidia drivers (the closed source ones) have been *fantastic* in performance, compatibility, and so forth. I have not (ever) had a good experience with ATI graphics drivers ... so much so that I conciously avoid buying their products (which is hard in the laptop space, as nVidia doesn't have as many design wins). This said, one area we would have liked to have seen open nVidia drivers is on our Itanium2 box. Yeah, it is a relic in the making, and no, I don't blame nVidia for dropping support for it. It would be nice to be able to see graphics on it though ... well ... accelerated graphics (there is a nice Quadro FX 1100 in there now). Oh well (move that card somewhere else). Just my $0.01 (used to be $0.02, but the dollar is dropping in value). -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From Ron.Jerome at nrc-cnrc.gc.ca Fri Mar 28 06:51:11 2008 From: Ron.Jerome at nrc-cnrc.gc.ca (Jerome, Ron) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: References: Message-ID: I use a Linux (RHEL 5) machine with 16G ram, two dual core Opterons and two Nvidia Quadro cards to drive a pair of Panasonic projectors lighting up my 10 foot diagonal passive stereo display wall. It works quite well. _________________________________________ Ron Jerome Programmer/Analyst National Research Council Canada M-2, 1200 Montreal Road, Ottawa, Ontario K1A 0R6 Government of Canada Phone: 613-993-5346 FAX: 613-941-1571 _________________________________________ > -----Original Message----- > From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] > On Behalf Of Ricardo Reis > Sent: March 27, 2008 6:41 AM > To: beowulf@beowulf.org > Subject: [Beowulf] visualization machine > > > Hi all > > I beg to take advantage of your experience although the topic isn't > completly cluster thing. I got some money to buy a new machine, at > least > 8Gb and I'm thinking between a 2 x dual core or a 1 x quad (or even 2x > quads). It must be one machine because it must (urgh) be able to use > it's > 8Gb in serial codes (don't ask). Anyway, I've been experiencing with > paraview for parallel visualization and was wondering on your opinion > on... buying a ultra-duper-cool state-of-the-art graphic card (Nvidia) > or > 2 graphic cards? > > thanks for your time, > > Ricardo Reis > > 'Non Serviam' > > PhD student @ Lasef > Computational Fluid Dynamics, High Performance Computing, Turbulence > http://www.lasef.ist.utl.pt > > & > > Cultural Instigator @ R?dio Zero > http://www.radiozero.pt > > http://www.flickr.com/photos/rreis/ From jmack at wm7d.net Fri Mar 28 08:09:18 2008 From: jmack at wm7d.net (Joseph Mack NA3T) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: <20080328072446.b2v69zlbkscsk0g8@webmail.jpl.nasa.gov> References: <47E8E052.9060104@cs.man.ac.uk> <20080325063427.ykaynvwtckk0ock8@webmail.jpl.nasa.gov> <476F940B-5048-4C98-8947-7106EEE9518B@jax.org> <47EBE5D6.7050702@scalableinformatics.com> <20080328072446.b2v69zlbkscsk0g8@webmail.jpl.nasa.gov> Message-ID: On Fri, 28 Mar 2008, Jim Lux wrote: >> Honestly, I think it more likely that this posted salary is a typo of >> some sort. obviously missing a zero Joe -- Joseph Mack NA3T EME(B,D), FM05lw North Carolina jmack (at) wm7d (dot) net - azimuthal equidistant map generator at http://www.wm7d.net/azproj.shtml Homepage http://www.austintek.com/ It's GNU/Linux! From rreis at aero.ist.utl.pt Fri Mar 28 08:22:10 2008 From: rreis at aero.ist.utl.pt (Ricardo Reis) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: References: <47EC6DBF.1080705@gmail.com> Message-ID: On Fri, 28 Mar 2008, Joshua Baker-LePain wrote: > For truly high-end 3D, there is (still, unfortunately) really no alternative > to closed source drivers. nouveua (for nvidia hardware) isn't there yet, and > the open source radeon stuff doesn't support any of their newer hardware. > Also, AFAIK, neither project supports the professional series of cards > (Quadro/FireGL). I've though NVIDIA because of ATI bad rep on their linux drivers. Yes, shure, they aren't open source (yet? maybe AMD buying ATI will change this in the long run) but don't know anyone with bad experiences on them. I will gladly admit having a biased population for this statistic but that's my perception. I think I'll end building a ultra gamers PC... Something that gives me the puzzles is whats the different between consumer and server Quad processors for Intel (for the money you're paying, I mean)... greets, Ricardo Reis 'Non Serviam' PhD student @ Lasef Computational Fluid Dynamics, High Performance Computing, Turbulence http://www.lasef.ist.utl.pt & Cultural Instigator @ R?dio Zero http://www.radiozero.pt http://www.flickr.com/photos/rreis/ From rreis at aero.ist.utl.pt Fri Mar 28 12:12:17 2008 From: rreis at aero.ist.utl.pt (Ricardo Reis) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada Message-ID: Sorry for the plug but it seems to me that many of you are just repeating strips from PHD comics: http://www.phdcomics.com/comics.php a small amusement on PhDs lifes... Ricardo Reis 'Non Serviam' PhD student @ Lasef Computational Fluid Dynamics, High Performance Computing, Turbulence http://www.lasef.ist.utl.pt & Cultural Instigator @ R?dio Zero http://www.radiozero.pt http://www.flickr.com/photos/rreis/ From andrew.robbie at gmail.com Sun Mar 30 07:17:41 2008 From: andrew.robbie at gmail.com (Andrew Robbie (GMail)) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: References: Message-ID: On Thu, Mar 27, 2008 at 9:41 PM, Ricardo Reis wrote: > > Hi all > > I beg to take advantage of your experience although the topic isn't > completly cluster thing. I got some money to buy a new machine, at least > 8Gb and I'm thinking between a 2 x dual core or a 1 x quad (or even 2x > quads). It must be one machine because it must (urgh) be able to use it's > 8Gb in serial codes (don't ask). Just be aware that most of the machines designed to be number crunchers have shortcomings in board layout or bus design that make them suck for visualization. Not that many that will be happy with 8GB for starters. So few machines are actually ever populated with big dimms that you almost always get issues. So you end up going for machines with lots of ram slots, ecc support etc, which is all good. These are almost always at least dual socket. But many of those motherboards aren't designed to take a 16x PCIe graphics card and only have PCIe 8x buses. Also, graphics cards have an extra retaining lug which extends further than the PCIe slot; this is commonly blocked on server motherboards by some capacitor. High end graphics cards always take up two slots and require additional power; on the Quadro 5600 and other cards this connector enters from the top not the end, hence making it impossible to fit them in a 3U case. Oh yeah -- don't think about one of these for under your desk unless you want to wear earmuffs in the office. > Anyway, I've been experiencing with > paraview for parallel visualization and was wondering on your opinion > on... buying a ultra-duper-cool state-of-the-art graphic card (Nvidia) or > 2 graphic cards? Depends -- is performance critical *now*? If so, buy the fastest Quadro. If you want to maximize performance over time, just upgrade the graphics card every six months with the sweet spot on the price/performance curve. Quadros are the first low-yield parts from the fab; the same chips, with slightly slower/cheaper memory hierarchy, become mass market later. Don't bother with SLI, you won't notice any speedup unless you invest lots of tuning time. And since your viz app is 3rd party, probably no speedup at all. ATI vs nVidia: ATI drivers really really suck. nVidia drivers are generally stable unless you are on the bleeding edge (eg brand new part) or a corner case (eg quad buffered stereo on a 2.2 kernel but with recent hardware & drivers). nVidia developer support sucks too unless you are a major game author or eg industrial light & magic. ATI developer support is non-existent under linux; under windows I'm told they can be ok about fixing windows bugs. Regards, Andrew (flight simulation geek) > > thanks for your time, > > Ricardo Reis > > 'Non Serviam' > > PhD student @ Lasef > Computational Fluid Dynamics, High Performance Computing, Turbulence > http://www.lasef.ist.utl.pt > > & > > Cultural Instigator @ R?dio Zero > http://www.radiozero.pt > > http://www.flickr.com/photos/rreis/ > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080331/13d40354/attachment.html From israsilva at gmail.com Sun Mar 30 10:12:10 2008 From: israsilva at gmail.com (Israel Lehnen Silva) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] FREEBSD CLUSTER Message-ID: <5ce468b90803301012g1aeecb86h492c6cd266995a2e@mail.gmail.com> Hello friends! I am Brazilian, and i am doing a Beowulf Cluster in OS FreeBSD for a project of University where i study the networking Infrastructure of the link between the nodes. If someone has other material that can help me, send me for me to increase my project! Counting on the cooperation and support of the brothers. -- Att. Israel Lehnen Silva From landman at scalableinformatics.com Mon Mar 31 08:40:55 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] a postdoc in Canada In-Reply-To: References: Message-ID: <47F10607.8050309@scalableinformatics.com> Hola Ricardo Ricardo Reis wrote: > > Sorry for the plug but it seems to me that many of you are just > repeating strips from PHD comics: Sadly, no. > > http://www.phdcomics.com/comics.php > > a small amusement on PhDs lifes... -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From raysonlogin at gmail.com Mon Mar 31 11:14:38 2008 From: raysonlogin at gmail.com (Rayson Ho) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] FREEBSD CLUSTER In-Reply-To: <5ce468b90803301012g1aeecb86h492c6cd266995a2e@mail.gmail.com> References: <5ce468b90803301012g1aeecb86h492c6cd266995a2e@mail.gmail.com> Message-ID: <73a01bf20803311114m502d9a85t4b35b09ecea78c1@mail.gmail.com> You can get detailed information on setting up a BSD HPC cluster at: http://people.freebsd.org/~brooks/papers/ Grid Engine was ported and maintained by the author... Rayson On Sun, Mar 30, 2008 at 12:12 PM, Israel Lehnen Silva wrote: > Hello friends! > > I am Brazilian, and i am doing a Beowulf Cluster in OS FreeBSD for a > project of University where i study the networking Infrastructure of > the link between the nodes. > If someone has other material that can help me, send me for me to > increase my project! > > Counting on the cooperation and support of the brothers. > -- > > Att. Israel Lehnen Silva > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > From jlforrest at berkeley.edu Mon Mar 31 13:36:46 2008 From: jlforrest at berkeley.edu (Jon Forrest) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] "Code" vs. "Codes" Message-ID: <47F14B5E.7010904@berkeley.edu> Sometime long ago I first heard the term "codes" used in the same way I would use the term "programs". For example, someone might have said "The codes from Berkeley were very tricky". The first time I heard this, I thought maybe this term came from someone who wasn't a native speaker of English, who was trying to pluralize the term "code". This often happens with words like "information" and "documentation". But, now I regularly hear native speakers of English using "codes" to mean "programs", especially in the scientific realm, such as what many of us deal with regularly. Does anybody know how this usage first came about? Cordially, -- Jon Forrest Research Computing Support College of Chemistry 173 Tan Hall University of California Berkeley Berkeley, CA 94720-1460 510-643-1032 jlforrest@berkeley.edu From lindahl at pbm.com Mon Mar 31 14:23:15 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] "Code" vs. "Codes" In-Reply-To: <47F14B5E.7010904@berkeley.edu> References: <47F14B5E.7010904@berkeley.edu> Message-ID: <20080331212315.GB27222@bx9.net> On Mon, Mar 31, 2008 at 01:36:46PM -0700, Jon Forrest wrote: > Does anybody know how this usage first came about? Why not try the OED? -- g From geoff at galitz.org Mon Mar 31 14:36:18 2008 From: geoff at galitz.org (Geoff Galitz) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] "Code" vs. "Codes" In-Reply-To: <47F14B5E.7010904@berkeley.edu> Message-ID: <200803312136.m2VLaScm002619@b.mail.sonic.net> You got me curious about this so I did a little looking around, and found the following: ------------------------------------------------------ Free On-line Dictionary of Computing - Cite This Source - Share This codes 1. Programs. This usage is common among scientific computing people who use supercumputers for heavy-duty number crunching. 2. Something to do with cryptography. [ The Jargon File] (1994-10-28) The Free On-line Dictionary of Computing, ? 1993-2007 Denis Howe Jargon File - Cite This Source - Share This codes n. [scientific computing] Programs. This usage is common in people who hack supercomputers and heavy-duty number-crunching, rare to unknown elsewhere (if you say "codes" to hackers outside scientific computing, their first association is likely to be "and cyphers"). Jargon File 4.2.0 ------------------------------------------------------ Citation: codes. Dictionary.com. The American Heritage? Science Dictionary. Houghton Mifflin Company. http://dictionary.reference.com/browse/codes (accessed: March 31, 2008). It does not answer your question as to when or how it got started, but apparently it has been with us a while. -geoff -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Jon Forrest Sent: Montag, 31. M?rz 2008 22:37 To: beowulf@beowulf.org Subject: [Beowulf] "Code" vs. "Codes" Sometime long ago I first heard the term "codes" used in the same way I would use the term "programs". For example, someone might have said "The codes from Berkeley were very tricky". The first time I heard this, I thought maybe this term came from someone who wasn't a native speaker of English, who was trying to pluralize the term "code". This often happens with words like "information" and "documentation". But, now I regularly hear native speakers of English using "codes" to mean "programs", especially in the scientific realm, such as what many of us deal with regularly. Does anybody know how this usage first came about? Cordially, -- Jon Forrest Research Computing Support College of Chemistry 173 Tan Hall University of California Berkeley Berkeley, CA 94720-1460 510-643-1032 jlforrest@berkeley.edu _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080331/5a63ee33/attachment.html From tjrc at sanger.ac.uk Mon Mar 31 14:40:23 2008 From: tjrc at sanger.ac.uk (Tim Cutts) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] "Code" vs. "Codes" In-Reply-To: <47F14B5E.7010904@berkeley.edu> References: <47F14B5E.7010904@berkeley.edu> Message-ID: <9DD7EDB2-5274-4B5F-A105-A40DC861FEA3@sanger.ac.uk> On 31 Mar 2008, at 9:36 pm, Jon Forrest wrote: > Sometime long ago I first heard the term "codes" > used in the same way I would use the term > "programs". For example, someone might have > said "The codes from Berkeley were very tricky". > > The first time I heard this, I thought maybe > this term came from someone who wasn't a native > speaker of English, who was trying to pluralize > the term "code". This often happens with > words like "information" and "documentation". > > But, now I regularly hear native speakers > of English using "codes" to mean "programs", > especially in the scientific realm, such as > what many of us deal with regularly. > > Does anybody know how this usage first came about? I don't know, but I'd be curious to find out too. It's a usage I've only ever encountered in the HPC field, and when I first heard it it was from people who've been in the business a long time, so I suspect it's quite old. Tim -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From rgb at phy.duke.edu Mon Mar 31 15:45:42 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] "Code" vs. "Codes" In-Reply-To: <200803312136.m2VLaScm002619@b.mail.sonic.net> References: <200803312136.m2VLaScm002619@b.mail.sonic.net> Message-ID: On Mon, 31 Mar 2008, Geoff Galitz wrote: I would have used this one: http://en.wikipedia.org/wiki/Source_code which I think is perhaps more accurate. "Computer code" has been a synonym for "program" forever -- at least since the 60's or 70's -- and not just in HPC. I can't even remember a time when I didn't use it or short versions such as code or codes. Also note the term "codesmith" as a synonym for "programmer", although alas Google shows that the terms has been eaten alive by marketing droids. Some of these terms are indeed used mostly by the 1337, but hardly (as alleged by the references below) only the "scientific computing community". I'd say the general programming community, with some perhaps more likely to use it than others. rgb > You got me curious about this so I did a little looking around, and found > the following: > > > > ------------------------------------------------------ > > > > Free > On-line Dictionary of Computing - > Cite This > Source - > Share This > > codes > 1. Programs. This usage is common among scientific computing people who use > supercumputers for > heavy-duty > number crunching. > 2. Something to do with > cryptography. > [ The > Jargon File] > (1994-10-28) > > > The Free On-line Dictionary of Computing, ? 1993-2007 Denis Howe > > Jargon > File - Cite > This Source - > Share > This > > codes > > n. [scientific computing] Programs. This usage is common in people who hack > supercomputers and heavy-duty > number-crunching, > rare to unknown elsewhere (if you say "codes" to hackers outside scientific > computing, their first association is likely to be "and cyphers"). > > > Jargon File 4.2.0 > > > > ------------------------------------------------------ > > > > > > Citation: > > > > codes. Dictionary.com. The American Heritage? Science Dictionary. Houghton > Mifflin Company. > http://dictionary.reference.com/browse/codes (accessed: March 31, 2008). > > > > > > > > It does not answer your question as to when or how it got started, but > apparently it has been with us a while. > > > > > > -geoff > > > > > > > > > > > > > > -----Original Message----- > From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On > Behalf Of Jon Forrest > Sent: Montag, 31. M?rz 2008 22:37 > To: beowulf@beowulf.org > Subject: [Beowulf] "Code" vs. "Codes" > > > > Sometime long ago I first heard the term "codes" > > used in the same way I would use the term > > "programs". For example, someone might have > > said "The codes from Berkeley were very tricky". > > > > The first time I heard this, I thought maybe > > this term came from someone who wasn't a native > > speaker of English, who was trying to pluralize > > the term "code". This often happens with > > words like "information" and "documentation". > > > > But, now I regularly hear native speakers > > of English using "codes" to mean "programs", > > especially in the scientific realm, such as > > what many of us deal with regularly. > > > > Does anybody know how this usage first came about? > > > > Cordially, > > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From gdjacobs at gmail.com Mon Mar 31 16:03:51 2008 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: <47F0E44E.90800@scalableinformatics.com> References: <47EC6DBF.1080705@gmail.com> <47F0E44E.90800@scalableinformatics.com> Message-ID: <47F16DD7.9040506@gmail.com> Joe Landman wrote: > Geoff Jacobs wrote: > >> No nvidia for me unless they open up driver specs. Open drivers have a >> history of being very, very stable if not as fast. Nvidia does have good >> drivers, but I have definitely seen them experience problems. > > Silicon Image SATA drivers are great examples of open drivers that have > been (in the past) terrible. The tg3 drivers have been very bad (start > sending lots of packets and watch your CSW climb and climb and climb). > Forcedeth are still terrible, and are open. The nv driver in X often > breaks on newer hardware (the laptop I type this on is a testament to > this problem). There is no magic bullet :) I was referring specifically to graphics drivers. Sorry. I'm surprised the nv driver has problems, as it is contributed to by NVidia. > No, open-ness doesn't equate to good-ness. Open-ness equates to > portability, ability to hunt for problems on your own and correct them > if need be. I find that open-ness tends to allow better integration (less problems when switching between kernel framebuffer and X, for example). As well, I don't like the idea of having hardware orphaned because it just got EOLed and NVidia or ATI is not going to support it in future driver releases. > For the platforms I have used them on the nVidia drivers (the closed > source ones) have been *fantastic* in performance, compatibility, and so > forth. I have not (ever) had a good experience with ATI graphics > drivers ... so much so that I conciously avoid buying their products > (which is hard in the laptop space, as nVidia doesn't have as many > design wins). I don't use the ATI closed source drivers on Linux. > This said, one area we would have liked to have seen open nVidia drivers > is on our Itanium2 box. Yeah, it is a relic in the making, and no, I > don't blame nVidia for dropping support for it. It would be nice to be > able to see graphics on it though ... well ... accelerated graphics > (there is a nice Quadro FX 1100 in there now). Oh well (move that card > somewhere else). I wouldn't blame them either -- it's a reality of business. Still, it sucks from the users perspective. > Just my $0.01 (used to be $0.02, but the dollar is dropping in value). -- Geoffrey D. Jacobs From James.P.Lux at jpl.nasa.gov Mon Mar 31 16:10:50 2008 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] "Code" vs. "Codes" In-Reply-To: <47F14B5E.7010904@berkeley.edu> References: <47F14B5E.7010904@berkeley.edu> Message-ID: <6.2.3.4.2.20080331153851.0289a130@mail.jpl.nasa.gov> At 01:36 PM 3/31/2008, Jon Forrest wrote: >Sometime long ago I first heard the term "codes" >used in the same way I would use the term >"programs". For example, someone might have >said "The codes from Berkeley were very tricky". > >The first time I heard this, I thought maybe >this term came from someone who wasn't a native >speaker of English, who was trying to pluralize >the term "code". This often happens with >words like "information" and "documentation". > >But, now I regularly hear native speakers >of English using "codes" to mean "programs", >especially in the scientific realm, such as >what many of us deal with regularly. > >Does anybody know how this usage first came about? It's quite old (in computer age), I think. One sees references to various hydrodynamic codes (meaning things like numerical computation of Navier-Stokes, etc.) in reports from Los Alamos in the 1950s. Probably comes from the fact that "coders" did "coding" to implement various models and algorithms, and what would the process of coding produce but a code. Jim Some references: at http://history.siam.org/articles.htm one can find links to, e.g. http://history.siam.org/pdf/cwgear.pdf mentions that early implementations of RungeKutta methods were, for example, "implemented by Wheeler as code #27 for the ILLIAC I" ILLIAC 1 became operational in 1952. (page 10 of the pdf) .. however, I'm not sure if they referred to them as codes then, or if that usage is from the author of the article. One might want to get references 33,34, and 35 from that article (from 1950,51, 54, respectively) There's also the transcript of an oral history by Cody, where he talks about the "Argonne Code Center", and the interviewer (Haigh) asks Cody: Cody: ... The Argonne Code Center was a repository for nuclear codes, codes that had been written to perform nuclear computations, design of reactors, whatever. Haigh: And code in this context basically means program? Cody: Program, the complete program. http://stinet.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=AD0842402 Accession Number : AD0842402 Title : Comparison Between a Von Neumann-Richtmyer Hydrocode (AFWL's PUFF) and a Lax-Wendroff Hydrocode. Descriptive Note : Technical rept. Aug 67-Aug 68, Corporate Author : AIR FORCE WEAPONS LAB KIRTLAND AFB NM Personal Author(s) : Hicks, Darrell ; Pelzl, Robert Report Date : OCT 1968 There are 7 TR's in DTIC with dates before 1970 and the word hydrocode appearing somewhere. The earliest one is from March 1967 Accession Number: AD0817429 Full Text (pdf) Availability: Size: 28 MB Handle / proxy Url: http://handle.dtic.mil/100.2/AD817429 Citation Status: ACTIVE Title: THEORETICAL CALCULATIONS OF THE PHENOMENOLOGY OF DISTANT PLAIN EVENT 6 Abstract:Abstract: Theoretical calculations of the phenomenology of an atmospheric high- explosive detonation are presented. The charge was a 100-ton sphere (radius, 240.5cm) of TNT loading density, 1.56 gms/cc whose center was at an altitude of 646.405 meters. The ground, upon which the spherical charge rested, was 644 meters above sea level. The numerical calculations taken out to 6 seconds were performed on the CDC 6600 digital computer using SHELL2, a two-material (version of the SHELL-OIL code), two-dimensional pure Eulerian hydrodynamic code. Air and the detonation products of TNT were the two materials considered in the calculation. The analytic, self-similar solution for the detonation wave in TNT provided the initial conditions. Included are pressure and density contours, velocity vector plots, and wave forms for 19 test stations. This calculation is a representation of the air blast of Event 6 of the DISTANT PLAIN test series to be fired in Canada, July 1967. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20080331/233ed448/attachment.html From landman at scalableinformatics.com Mon Mar 31 16:17:59 2008 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: <47F16DD7.9040506@gmail.com> References: <47EC6DBF.1080705@gmail.com> <47F0E44E.90800@scalableinformatics.com> <47F16DD7.9040506@gmail.com> Message-ID: <47F17127.3040705@scalableinformatics.com> Geoff Jacobs wrote: > Joe Landman wrote: >> Geoff Jacobs wrote: >> >>> No nvidia for me unless they open up driver specs. Open drivers have a >>> history of being very, very stable if not as fast. Nvidia does have good >>> drivers, but I have definitely seen them experience problems. >> Silicon Image SATA drivers are great examples of open drivers that have >> been (in the past) terrible. The tg3 drivers have been very bad (start >> sending lots of packets and watch your CSW climb and climb and climb). >> Forcedeth are still terrible, and are open. The nv driver in X often >> breaks on newer hardware (the laptop I type this on is a testament to >> this problem). > There is no magic bullet :) ... nor are there free lunches ... :) > > I was referring specifically to graphics drivers. Sorry. Whoops... the ATI drivers have been well ... um ... not that good (the open ones) for a long time. > I'm surprised the nv driver has problems, as it is contributed to by > NVidia. Yeah, kinda shocked me ... laptop with a Quadro FX/360, and whammo ... no display. Had to load the nvidia ones by hand to get a display. Hopefully (crosses fingers) Ubuntu 8.04 will fix this ... hopefully ... fix (as in not break!) this ... > >> No, open-ness doesn't equate to good-ness. Open-ness equates to >> portability, ability to hunt for problems on your own and correct them >> if need be. > I find that open-ness tends to allow better integration (less problems > when switching between kernel framebuffer and X, for example). As well, > I don't like the idea of having hardware orphaned because it just got > EOLed and NVidia or ATI is not going to support it in future driver > releases. Yeah, this does bug me too. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 fax : +1 866 888 3112 cell : +1 734 612 4615 From jlforrest at berkeley.edu Mon Mar 31 16:33:30 2008 From: jlforrest at berkeley.edu (Jon Forrest) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] "Code" vs. "Codes" In-Reply-To: <6.2.3.4.2.20080331153851.0289a130@mail.jpl.nasa.gov> References: <47F14B5E.7010904@berkeley.edu> <6.2.3.4.2.20080331153851.0289a130@mail.jpl.nasa.gov> Message-ID: <47F174CA.3070304@berkeley.edu> I appreciate everyone's comments so far, but please keep in mind that I only meant to comment on the use of "codes" (plural). The use of "code" in place of "program" feels normal. I still suspect that somewhere this started with a non-native speaker of English. The same thing has happened recently with "e-mail", which didn't used to exist in common usage in plural form. Now you see "e-mails" used by all kinds of people. Cordially, -- Jon Forrest Research Computing Support College of Chemistry 173 Tan Hall University of California Berkeley Berkeley, CA 94720-1460 510-643-1032 jlforrest@berkeley.edu From rgb at phy.duke.edu Mon Mar 31 17:12:04 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] "Code" vs. "Codes" In-Reply-To: <47F174CA.3070304@berkeley.edu> References: <47F14B5E.7010904@berkeley.edu> <6.2.3.4.2.20080331153851.0289a130@mail.jpl.nasa.gov> <47F174CA.3070304@berkeley.edu> Message-ID: On Mon, 31 Mar 2008, Jon Forrest wrote: > I appreciate everyone's comments so far, but please > keep in mind that I only meant to comment on the > use of "codes" (plural). The use of "code" in place > of "program" feels normal. I think Jim had the right of it. If a program can be referred to as a code, then some programs can be referred to as some codes. I think I understand what you're objecting to -- "code" is somewhat like "deer" -- either singular or plural so it can feel odd to refer to "codes". But it isn't EXACTLY like deer. Try it on with an adjective: "I gave my student the group's hydrodynamics codes." vs "I gave my student the group's hydrodynamic code." These two sentences have slightly different meaning. By the former it is clearly meant that I handed the student not (the source for) one program but for a suite of programs for doing hydrodynamics computations. Perhaps a simulation code, a direct numerical solution code, a code for analyzing results of hydrodynamics experiments, perhaps even auxiliary codes for display and visualization. Each of these is a distinct package of (source) code (the plural/collective form for a single program's worth of lines of code). The latter COULD mean that, but it would be a slightly odd usage. To me it means that I've given the student a big package of code (that might well have lots of subroutines and modules and lines of code and so on) that ultimately builds a single program. So I doubt that it is non-native speakers of English -- just people forming the natural plural of (source) code for a (single) program, into the (source) codes for (multiple) programs. > I still suspect that somewhere this started with > a non-native speaker of English. The same thing > has happened recently with "e-mail", which didn't > used to exist in common usage in plural form. > Now you see "e-mails" used by all kinds of people. And I think that it's the same kind of thing. "e-mail" is number indeterminate, and it is "expensive" to indicate number with a full quantifier. I got e-mail could be one or ten messages, sort of like the deer in my front yard (how many are there?). I could speak of a bunch of deer, a herd of deer, lots of deer, or a single deer to clarify, but English being a marvelously flexible language, I could also half-kidding-use the deliberate malconstruction "the deers in my front yard" (or "mooses in the forest" or "e-mails in my mailbox") to save a quantifier and still unambiguously indicate that I mean the plural vs singular form (or catch it with a verb form elsewhere, but this won't work when the deer are the sentence's object or off in a clause). Plenty of native speakers of English might do this. Some of them might even be pretty literate and doing it deliberately in jocular or informal conversation (but not in an English paper if they know what's good for them). It isn't that uncommon for these plurals to make it into "real English". Pardon me, I've got to go put away the leftover fishes from my table. Otherwise the deers from the yard might sneak in to get them and make a mess of my floor. ;-) rgb > > Cordially, > > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From lindahl at pbm.com Mon Mar 31 17:28:24 2008 From: lindahl at pbm.com (Greg Lindahl) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] "Code" vs. "Codes" In-Reply-To: References: <47F14B5E.7010904@berkeley.edu> <6.2.3.4.2.20080331153851.0289a130@mail.jpl.nasa.gov> <47F174CA.3070304@berkeley.edu> Message-ID: <20080401002824.GA15924@bx9.net> English has super-plurals: http://d.a.nicolas.free.fr/research/Linnebo-Nicolas-Superplurals.pdf rgb knows this, but he was pretending like he doesn't use "all-y'all" all the time... Anyway, the reason I suggested the OED is that they often have early usage quotes, which may well include examples of 'codes'. I can only access the free version, which shows that they do have the definition, but doesn't include the examples. -- greg From rgb at phy.duke.edu Mon Mar 31 18:12:53 2008 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] "Code" vs. "Codes" In-Reply-To: <20080401002824.GA15924@bx9.net> References: <47F14B5E.7010904@berkeley.edu> <6.2.3.4.2.20080331153851.0289a130@mail.jpl.nasa.gov> <47F174CA.3070304@berkeley.edu> <20080401002824.GA15924@bx9.net> Message-ID: On Mon, 31 Mar 2008, Greg Lindahl wrote: > English has super-plurals: > > http://d.a.nicolas.free.fr/research/Linnebo-Nicolas-Superplurals.pdf > > rgb knows this, but he was pretending like he doesn't use "all-y'all" > all the time... Y'all find some mighty odd things out there on the web, y'know Greg? I hadn't ever heard of super-plurals -- very interesting. All y'all should look at this, because the bit on the analysis of certain super-plural constructs providing a lead to second and third (and speculatively higher order) logics is mighty interesting. Still, God knows that it is difficult enough to keep my verbs all tensed up and correctly quantified without superplural constructs. Now Whitehead and Russell, Whitehead and Hilbert, is a superplural construct in a self-referential sentence that asks whether the verb "is" that refers to Whitehead and Russell, Whitehead and Hilbert, should correctly be singular super-super-plural or if I should have used "are" to indicate ordinary super-plural (and while I'm at it, whether or not I can use "used" as a self-referential past tense of a usage in a single sentence with an implicit temporal order of expression right before switching back to present tense). And only Kilpatrick Knows if I should have put a question mark at the end...;-) rgb > Anyway, the reason I suggested the OED is that they often have early > usage quotes, which may well include examples of 'codes'. I can only > access the free version, which shows that they do have the definition, > but doesn't include the examples. I have VPN problems at home these days (too complicated to describe, sigh) but when I get to Duke tomorrow I'll see if the University subscribes to the OED so I can use it for free. rgb > > -- greg > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown Phone(cell): 1-919-280-8443 Duke University Physics Dept, Box 90305 Durham, N.C. 27708-0305 Web: http://www.phy.duke.edu/~rgb Book of Lilith Website: http://www.phy.duke.edu/~rgb/Lilith/Lilith.php Lulu Bookstore: http://stores.lulu.com/store.php?fAcctID=877977 From franz.marini at mi.infn.it Mon Mar 31 13:40:30 2008 From: franz.marini at mi.infn.it (Franz Marini) Date: Wed Nov 25 01:06:58 2009 Subject: [Beowulf] visualization machine In-Reply-To: References: Message-ID: <1206996030.6926.29.camel@merlino.mi.infn.it> On Mon, 2008-03-31 at 01:17 +1100, Andrew Robbie (GMail) wrote: > Just be aware that most of the machines designed to be number > crunchers > have shortcomings in board layout or bus design that make them suck > for > visualization. > > Not that many that will be happy with 8GB for starters. So few > machines are > actually ever populated with big dimms that you almost always get > issues. > So you end up going for machines with lots of ram slots, ecc support > etc, which > is all good. These are almost always at least dual socket. But many of > those > motherboards aren't designed to take a 16x PCIe graphics card and only > have > PCIe 8x buses. Also, graphics cards have an extra retaining lug which > extends further than the PCIe slot; this is commonly blocked on server > motherboards by some capacitor. > High end graphics cards always take up two slots and require > additional power; on > the Quadro 5600 and other cards this connector enters from the top not > the end, > hence making it impossible to fit them in a 3U case. > Oh yeah -- don't think about > one of these for under your desk unless you want to wear earmuffs in > the office. Only if you don't know how to choose your components. Right now, I have (under my desk) a box with the following specs: - Cooler Master Cosmos 1000 case - Supermicro X7DAE+ motherboard - 2 x Intel Xeon E5345 - Asus Extreme N8800 GTX - 3 x WD Raptor 150GB - 1 x WD Caviar RE2 400GB 16MB SATA-II - 4 x Kingston KVR667D2D4F5K2/4G (2 x 2GB FB-DIMM each, for a total of 16GB) - Corsair HX620W This is barely audible. The case has sound dampening foam on the side panels, and silent 12cm fans to cool the components. Moreover, both the Xeons fans and the 8800GTX one are quite quiet. The HDs are mounted on vibration dampening staffs, with rubber grommets, this helps keeping the sound level low, too. It *is* possible to build a quiet workstation powerhouse. You just have to carefully choose your components. And by the way, that motherboard has a 16x PCIe slot. As do most workstation-class motherboards from Supermicro. I guess you're confusing workstation motherboards with server ones. > Depends -- is performance critical *now*? If so, buy the fastest > Quadro. If you want > to maximize performance over time, just upgrade the graphics card > every six months > with the sweet spot on the price/performance curve. Quadros are the > first low-yield > parts from the fab; the same chips, with slightly slower/cheaper > memory hierarchy, > become mass market later. Actually, NVidia usually puts out the "gaming" parts first, and then follows with the Quadro class. The only difference between the two that I can recall are the (usually) different cooling solution and hardware support for antialiased points and lines, OpenGL logic operations, multiple clip regions, accelerated clip planes and overlay planes. Unless you're doing heavy (and I mean *heavy*) CAD/CAM/CAE, most of these things won't matter to you (sure, they can result in slightly higher speed in VMD, but not so much). The other differences (the bigger ones) lie in the drivers, which have specific application optimizations (for, e.g., Maya, Autocad, Softimage, Avid, etc., almost nothing, as far as I know, for scientific applications). The Quadro FX 4600 for example uses a G80GL chip, which is a slightly modified G80 chip (the one in the 8800 GTX). In fact, I'm quite sure it's the same chip (maybe from a better bin), with the hardware support for the operations I mentioned before enabled. Should there be some engineer from NVidia reading the list, and should I be wrong somewhere, please feel free to correct me (but spare me the marketing crap, I said engineer, no marketing people is allowed in the discussion ;) :D). I use the 8800GTX for VMD, Chimera, and in-house developed applications, and it works like a charm. Franz --------------------------------------------------------- Franz Marini Prof. R. A. Broglia Theoretical Physics of Nuclei, Atomic Clusters and Proteins Research Group Dept. of Physics, University of Milan, Italy. email : franz.marini@mi.infn.it phone : +39 02 50317226 --------------------------------------------------------- From eemoore at fyndo.com Mon Mar 31 21:57:10 2008 From: eemoore at fyndo.com (Eric Moore) Date: Wed Nov 25 01:06:59 2009 Subject: [Beowulf] "Code" vs. "Codes" References: <47F14B5E.7010904@berkeley.edu> <6.2.3.4.2.20080331153851.0289a130@mail.jpl.nasa.gov> <47F174CA.3070304@berkeley.edu> Message-ID: <873aq62mvt.fsf@nyarlothotep.fyndo.com> Jon Forrest writes: > I appreciate everyone's comments so far, but please > keep in mind that I only meant to comment on the > use of "codes" (plural). The use of "code" in place > of "program" feels normal. >From the WWII German surrender documents at: http://www.law.ou.edu/ushistory/germsurr.shtml I found this: This German Flag Officer is to be accompanied by a Communications Officer who is familiar with the German Naval W/T organization and who is to bring with him the current naval communications Orders, including allocation of frequencies, list of W/T and R/T call signs in force, and a list of all codes and cyphers in use, and intended to be brought into use. So certainly the use of "codes" as a plural for "code" was preexisting in the English language by the time "code" became a synonym for "program", so if "code" for "program" is accepted, there's no reason to look to non-native speakers for "codes" for "programs". (The surrender documents are not English translations of something in German, they make the point several times that the English version is authoritative). -- Eric