From becker at scyld.com Tue May 11 09:05:37 2004 From: becker at scyld.com (Donald Becker) Date: Wed Nov 25 01:03:08 2009 Subject: bwbug: BUBUG Meeting this Tuesday May 11 th in McLean Virginia (fwd) Message-ID: From: "Fitzmaurice, Michael" To: "'bwbug@pbm.com'" Subject: bwbug: BUBUG Meeting this Tuesday May 11 th in McLean Virginia A quick reminder. The next BWBUG meeting will be tomorrow Tuesday May 11th at the usual time at the Northrop Grumman building in McLean, VA. The speaker will be Marshall Presser talking about Oracle 9i RAC and 10g on Linux. This will be a great talk. I look forward to see you there. As always, for more information please go to http://www.bwbug.org T. Michael Fitzmaurice, Jr. BWBUG Coordinator 8110 Gatehouse Road, Suite 400W Falls Church, VA 22042 703-205-3132 office 240-475-7877 cell Email michael.fitzmaurice@ngc.com From richard_fryer at charter.net Tue May 11 16:00:55 2004 From: richard_fryer at charter.net (Richard Fryer) Date: Wed Nov 25 01:03:08 2009 Subject: Reliable CPU fans Message-ID: I have had a fair amount of maintenance on a fairly small cluster (~16nodes) with Cooler Master fans. In various workstations I've tried getting Alpha and other brand coolers but the real reliability issue has been of course the fans. When specified I've always chosen a ball bearing (some say 'dual' ball bearing) fans, but this has not led to spectacular reliability either. Thermaltake seems to be the best I've used but lacking hours-on data for the workstations, I doubt the data to be significant. Has anyone discovered a good resource regarding CPU cooler reliability to point me to? Or - taken enough data to have a good opinion? Thank you Richard Fryer From rgb at phy.duke.edu Tue May 11 10:55:19 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:08 2009 Subject: ping Message-ID: Are we up yet? (Sent 05/11/04, 13:56 and change EST). rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From bryce at jfet.net Wed May 12 11:12:19 2004 From: bryce at jfet.net (Bryce Bockman) Date: Wed Nov 25 01:03:08 2009 Subject: ping In-Reply-To: Message-ID: ICMP Echo Reply! On Tue, 11 May 2004, Robert G. Brown wrote: > Are we up yet? (Sent 05/11/04, 13:56 and change EST). > > rgb > > -- Bryce Bockman bryce@jfet.net "My brains the cliff and my heart's the bitter buffalo, My brains the weak heart and my heart's the long stairs." --Modest Mouse From jeffrey.b.layton at lmco.com Wed May 12 10:22:40 2004 From: jeffrey.b.layton at lmco.com (Jeff Layton) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Re: ping In-Reply-To: References: Message-ID: <40A25D60.5090109@lmco.com> pong >Are we up yet? (Sent 05/11/04, 13:56 and change EST). > > rgb > > > -- Dr. Jeff Layton Aerodynamics and CFD Lockheed-Martin Aeronautical Company - Marietta From alvin at Mail.Linux-Consulting.com Wed May 12 11:36:30 2004 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Re: ping - pong In-Reply-To: Message-ID: On Tue, 11 May 2004, Robert G. Brown wrote: > Are we up yet? (Sent 05/11/04, 13:56 and change EST). pong c ya alvin -- anybody know what happened ... -- and if hw is a problem ... tons of free hw floating around From alvin at ns.Linux-Consulting.com Wed May 12 11:55:14 2004 From: alvin at ns.Linux-Consulting.com (Alvin Oga) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Re: Reliable CPU fans In-Reply-To: ; from Richard Fryer on Tue, May 11, 2004 at 04:00:55PM -0700 References: Message-ID: <20040512115514.A10597@Maggie.Linux-Consulting.com> hi ya richard On Tue, May 11, 2004 at 04:00:55PM -0700, Richard Fryer wrote: > I have had a fair amount of maintenance on a fairly small cluster (~16nodes) > with Cooler Master fans. are we talking about midtower cases or 1Us .. 2Us ... it makes a difference if midtower, etc, i guess we'd be using the default fans that comes with the cpu if 1Us .. than we have some issues > In various workstations I've tried getting Alpha and other brand coolers but > the real reliability issue has been of course the fans. When specified I've > always chosen a ball bearing (some say 'dual' ball bearing) fans, but this > has not led to spectacular reliability either. alpha is a heatsink manufacturer heatsinks and fans ... http://www.linux-1u.net/HeatSinkFans/ > Thermaltake seems to be the best I've used but lacking hours-on data for the > workstations, I doubt the data to be significant. thermaltake fans ( any 1U model ) has been the worst fan for us we usually get vantec p4 fans for 1U sometimes we get the coolermaster ones specific models .. depends on the month ... each time is different as models keep changing and whats in stock at the distributors - we'll try most of um ... but ... thermaltakes was the ones we throw away the most ... the fan on it is worthless and if you watch the fan rpm ... its bad ... stops and goes even though its supposed to be running at full rpm to see which fans NOT to use .... - to to the local colo and see if you can see dead fan labels and do NOT buy those ... > Has anyone discovered a good resource regarding CPU cooler reliability to > point me to? Or - taken enough data to have a good opinion? we use indek fans for chassis cooling and cpu cooling when we only have a passive heatsink on the cpu ... and not many people carry sanyo/nmb fans ... indek is as good - the other problem is most fan manufacturers do not give us any usable MTBF we also use about 4-6 fans per cpu(1U box) .... including one for cooling each disk - fans are right next to the cpu on the side of the mb the fan above the cpu that is almost flush to the 1U cover is worthless fan as one needs say 0.5" - 1.0" of air gap for the fans blades to do its job and the air that blows on top of the cpu doesnt bend at 90degrees when it hits the cpu ( bottom of the heatsink ) we do custom cooling as needed to get the cpu temp as low as possible c ya alvin From mikee at mikee.ath.cx Wed May 12 11:55:22 2004 From: mikee at mikee.ath.cx (Mike Eggleston) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] pong In-Reply-To: References: Message-ID: <20040512185522.GB18664@mikee.ath.cx> On Tue, 11 May 2004, Robert G. Brown wrote: > Are we up yet? (Sent 05/11/04, 13:56 and change EST). yes From wrankin at ee.duke.edu Wed May 12 12:22:08 2004 From: wrankin at ee.duke.edu (Bill Rankin) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Re: ping In-Reply-To: <200405121900.i4CJ03HO018448@bluewest.scyld.com> References: <200405121900.i4CJ03HO018448@bluewest.scyld.com> Message-ID: <1084389728.5587.11.camel@rohgun.csem.duke.edu> > ICMP Echo Reply! Gotta do something about that latency issue... :-) -b From solszew7887 at yahoo.com Wed May 12 17:23:56 2004 From: solszew7887 at yahoo.com (steve olszewski) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] admin test message - please discard Message-ID: <20040513002356.12110.qmail@web13302.mail.yahoo.com> test __________________________________ Do you Yahoo!? Yahoo! Movies - Buy advance tickets for 'Shrek 2' http://movies.yahoo.com/showtimes/movie?mid=1808405861 From solszew7887 at yahoo.com Wed May 12 17:51:11 2004 From: solszew7887 at yahoo.com (steve olszewski) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] admin test message - please discard Message-ID: <20040513005111.33413.qmail@web13301.mail.yahoo.com> test __________________________________ Do you Yahoo!? Yahoo! Movies - Buy advance tickets for 'Shrek 2' http://movies.yahoo.com/showtimes/movie?mid=1808405861 From wathey at salk.edu Wed May 12 18:31:57 2004 From: wathey at salk.edu (Jack Wathey) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Re: Reliable CPU fans In-Reply-To: References: Message-ID: <20040512181715.G54918@gauss.snl.salk.edu> Here's my favorite combination for keeping 2GHz Athlons happy: Fan: Delta model AFB0612EH size: 60x60x25mm cfm: 38 rpm: 6800 noise: 46.5db mtbf: 60k hours http://www.bestbyte.net/Product.cfm?ProductID=502&CategoryID=8&Keyword= (There are several other models of Delta fans available on the bestbyte website.) Heat sink: All-copper Thermalright SK6+ http://www.bestbyte.net/Product.cfm?ProductID=941&CategoryID=6&Keyword= Thermal compound: Arctic Silver 3 http://www.bestbyte.net/Product.cfm?ProductID=736&CategoryID=7&Keyword= I have been using 198 coolers of this kind for about half a year with no problem. That's not much time, I know, but the Delta fans are popular among the serious overclockers, judging from the online reviews I've read. I don't know how trustworthy the 60k h mtbf number is. The SK6+ heatsink may be discontinued by now, but Thermalright makes lots of other excellent heatsinks. I'm also using two of their SLK800 heatsinks in my server. They are truly amazing. The disadvantage to the Delta fans is that they are loud. If, like most cluster people, you cram everything into a cluster room, close the door and work somewhere else, that's not such a big deal. Best wishes, Jack On Tue, 11 May 2004, Richard Fryer wrote: > I have had a fair amount of maintenance on a fairly small cluster (~16nodes) > with Cooler Master fans. > > In various workstations I've tried getting Alpha and other brand coolers but > the real reliability issue has been of course the fans. When specified I've > always chosen a ball bearing (some say 'dual' ball bearing) fans, but this > has not led to spectacular reliability either. > > Thermaltake seems to be the best I've used but lacking hours-on data for the > workstations, I doubt the data to be significant. > > Has anyone discovered a good resource regarding CPU cooler reliability to > point me to? Or - taken enough data to have a good opinion? > > Thank you > Richard Fryer From daniel at labtie.mmt.upc.es Thu May 13 04:47:49 2004 From: daniel at labtie.mmt.upc.es (Daniel Fernandez) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Athlon64 / Opteron test Message-ID: <1084448869.17067.55.camel@qeldroma.cttc.org> Hi, We're studying the possibility of installing a 64 bit testing node. The main purpose is about getting impressions related to the performance increase we would obtain in our particular scenario, computational fluid dynamics. In order to do the test, we have no doubt about the OS: Red Hat Enterprise 3, but we are a bit confused about the harware of choice: Athlon64 Opteron As far as we know, Opteron has two main differences: - A wider memory interface (128 bit in front of 64) - A larger L2 cache memory (1 Mb) Before doing any test, the questions are: 1) Which is the theoretical maximum performance gain using full 64 bit architecture in front of 32 bit, taking into account that our computations are done in double precision floating point using really big matrices? 2) Is it nowadays the 64 bit solution using Linux ready for production? If this is the case, which problems may we have to deal with in order to compile and run our code in a full 64 bit environment? 3) Which is the most mature solution: AMD Opteron or Intel Itanium? -- Daniel Fernandez Heat And Mass Transfer Center - CTTC www.cttc.upc.edu c/ Colom n?11 UPC Campus Industrials Terrassa , Edifici TR4 From rgb at phy.duke.edu Thu May 13 04:02:38 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Re: ping In-Reply-To: <1084389728.5587.11.camel@rohgun.csem.duke.edu> Message-ID: On Wed, 12 May 2004, Bill Rankin wrote: > > > ICMP Echo Reply! > > Gotta do something about that latency issue... I think it is improving... my first message was held for hand-proctoring, which will really crimp conversation on the list if it persists, but Doug tells me that Don tells him that the new list server is just about up and shaken down with spam assassin sitting above mailman. Inshallah as it is demonstrated that the DDDoS (Damn Distributed Denial of Service) attack that is SPAM is being mostly-effectively filtered the need for proctoring will go away and we can actually talk again. Personally, I think it is a miracle -- I couldn't have picked a better few weeks for the list to go away than right through finals. I got SO much more work done...:-) rgb > > :-) > > -b > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From canon at nersc.gov Thu May 13 07:36:38 2004 From: canon at nersc.gov (canon@nersc.gov) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Re: Reliable CPU fans In-Reply-To: Your message of "Tue, 11 May 2004 16:00:55 PDT." Message-ID: <200405131436.i4DEacVd018618@pookie.nersc.gov> Richard, We've recently changed all of our Athlon based machines over to fans similar to this.... http://www.dynatron-corp.com/products/cpucooler/cpucooler_model.asp?id=47# It is a squirrel cage design which seems to move more air and has better reliability (from our experience) than the vertical style fans from the same manufacturer. Unfortunately, the manufacturer doesn't supply MTBF's for this fan. I can say this... The vertical fans from Dynatron were very reliable at dieing for us. It finally got to a point where we decided to replace all the fans on 60+ dual nodes to this squirrel cage design. No regrets so far. Another 96 Athlon nodes already had the squirrel cage design and all have been reliable to this point. Having said this, I would consider using a CPU that can run with a passive heat sink (Xeon's and Opterons), since CPU fans seem to compete with disks for being the most unreliable component in a commodity system. --Shane ------------------------------------------------------------------------ Shane Canon voice: 510-486-6981 PSDF Project Lead fax: 510-486-7520 National Energy Research Scientific Computing Center 1 Cyclotron Road Mailstop 943-256 Berkeley, CA 94720 canon@nersc.gov ------------------------------------------------------------------------ From daniel.kidger at quadrics.com Thu May 13 14:24:49 2004 From: daniel.kidger at quadrics.com (Dan Kidger) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: <1084448869.17067.55.camel@qeldroma.cttc.org> References: <1084448869.17067.55.camel@qeldroma.cttc.org> Message-ID: <200405132224.49615.daniel.kidger@quadrics.com> On Thursday 13 May 2004 12:47 pm, Daniel Fernandez wrote: > 3) Which is the most mature solution: AMD Opteron or Intel Itanium? I would suggest that Opteron is both the newest and also the most mature. :-) Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- From rgb at phy.duke.edu Thu May 13 17:06:04 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: <1084448869.17067.55.camel@qeldroma.cttc.org> Message-ID: On Thu, 13 May 2004, Daniel Fernandez wrote: > Hi, > > We're studying the possibility of installing a 64 bit testing node. > > The main purpose is about getting impressions related to the performance > increase we would obtain in our particular scenario, computational fluid > dynamics. > > In order to do the test, we have no doubt about the OS: Red Hat > Enterprise 3, but we are a bit confused about the harware of choice: > > Athlon64 > Opteron > > As far as we know, Opteron has two main differences: > > - A wider memory interface (128 bit in front of 64) > - A larger L2 cache memory (1 Mb) > > Before doing any test, the questions are: > > 1) > > Which is the theoretical maximum performance gain using full 64 bit > architecture in front of 32 bit, taking into account that our > computations are done in double precision floating point using really > big matrices? I don't know what theoretical maximum gain means -- something with multiplying out clocks, pipeline form factors, datapath widths, and assuming perfect state? Never happens, more or less useless number. I do have a moderate amount of faith in what process-specific microbenchmarks like stream and lmbench tell one (not as much as one might think, but some useful data points). I also have a LOT of faith in using your own application as a benchmark on a loaner machine (or machines). So much so that this is one of the points of focus of my next couple of Cluster World columns -- the importance of understanding your application and the deep significance of "YMMV" when asking other people what to buy. If stream or stream-like microbenchmarks would be of any use to you, I have Opterons but not Athlon64's handy. Maybe somebody else can do the other for you. Remember, though, that even these results may vary significantly with compiler, with link libraries, with WHICH Opteron motherboard you get and how memory is loaded onto the motherboard. It is very, very difficult to get portable numbers without a prototyping machine that is "like" the architecture you are considering in very specific form (ideally, identical). > 2) > > Is it nowadays the 64 bit solution using Linux ready for production? > If this is the case, which problems may we have to deal with in order to > compile and run our code in a full 64 bit environment? I'm using dual Opterons (Penguin Altus 1000E's) running Fedora right now without any major problems. Your two problems are likely to be rebuilding things that aren't for some reason in the Fedora core and already built for the Opteron. So far I haven't encountered any major problems compiling gcc-based code linked to the GSL and other useful/standard libraries. YMMV, but a lot of people have these boxes now and together they form a powerful force pushing for full and clean distros, so even if you find something that doesn't work it likely WILL work and at worst you might have to help make it work in the best of open source tradition, if it is in your key work pathway. We're likely to buy nothing else for cluster nodes in the medium run. So far they are by far the cost-benefit winner. > 3) > > Which is the most mature solution: AMD Opteron or Intel Itanium? Did you mean mature or moribund;-)? I'm only half kidding. Itanium is dead as a doornail as technology goes -- overpriced, underperforming, incompatible. Intel is migrating to a (more or less) Opteron compatible 64 bit processor as fast as they can get there, as Major Software Companies (MSC) have announced that they aren't going to do major ports to new chips with new machine languages and compilers anymore if they can possibly avoid it. If Intel dropped the price of an Itanium to slightly LESS than that of an Opteron, I think they'd still have trouble maintaining a market, because Opterons are relatively easy to port to and will in principle run i386 code (badly, of course) native. Sometimes. I haven't had a lot of luck with it, of course, because you can't mix i386 code and 64 bit DLLs and we installed a 64 bit version of the OS from the start, but theoretically it will work. The good news is that Opterons are surprisingly fast for MY applications for their relatively pokey CPU clocks, and some benchmarks show that they can be really quite fast indeed for memory intensive applications relative to e.g. an Athlon or P4 clock. They also run much cooler than regular Athlons (again for my application). I draw ballpark of 185 watts loaded (dual CPU Opteron) vs 230 Watts or so loaded (dual CPU Athlon) running more or less the same code. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From clwang at csis.hku.hk Thu May 13 22:54:49 2004 From: clwang at csis.hku.hk (Cho Li Wang) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] CFP : ISPA-2004 Message-ID: <40A45F29.EF12EBEA@csis.hku.hk> ========================== ISPA-2004 CALL FOR PAPERS =========================== Second International Symposium on Parallel and Distributed Processing and Applications (ISPA-2004) Hong Kong, China, 13 - 15 Dec. 2004 http://www.comp.polyu.edu.hk/ISPA04 With the advance of computer networks and hardware technology, parallel and distributed processing has become a key technology which plays an important part in determining future research and development activities in many academic and industrial branches. It provides a means to solve computationally intensive problems by improving processing speed. It is also the only viable approach to building highly reliable and inherently distributed applications. Following the success of the first conference ISPAˇ¦2003, held in Aizu-Wakamatsu City, Japan,ISPAˇ¦04 will provide a forum for scientists and engineers in academia and industry to exchange and discuss their experiences, new ideas, research results, and applications about all aspects of parallel and distributed computing. ISPA'04 will feature session presentations, workshops,and keynote speeches. Topics of particular interest include, but are not limited to : * Computer networks * Network routing and communication algorithms * Parallel/distributed system architectures * Parallel/distributed algorithms * Wireless networks, mobile and pervasive computing * Internet computing and Web technologies * Peer-to-peer computing * Grid and cluster computing * Reliability, fault-tolerance, and security * Performance evaluation and measurements * Tools and environments for software development * Distributed systems and applications * High-performance scientific and engineering computing * Database applications and data mining * Biological/molecular computing SUBMISSION GUIDELINES --------------------- Submissions should include an abstract, key words, the e-mail address of the correspondingauthor, and must not exceed 15 pages, including tables and figures, with PDF, PostScript, or MSWord format. Electronic submission through the submission website is strongly encouraged. Hard copies will be accepted only if electronic submission is not possible. Submission of apaper should be regarded as an undertaking that, should the paper be accepted,at least one ofthe authors will register and attend the conference to present the work. IMPORTANT DATES --------------------- Workshop proposals due: 15 June 2004 Paper submission due: 1 July 2004 Acceptance notification: 30 Aug 2004 Camera-ready due: 30 Sept. 2004 Conference: 13-15 Dec 2004 PUBLICATION --------------------- The proceedings of the symposium will be published in Springer's Lecture Notes in Computer Science. A selection of the best papers for the conference will be published in a special issueof The Journal of Supercomputing and International Journal of High Performance Computing andNetworking (IJHPCN). BEST PAPER AWARDS --------------------- The program committee will select one winner for the Best Paper Award (all regular papers areeligible) and one winner for the Best Student Paper Award (only the regular papers whose firstauthor is a full-time student are eligible). Each winner will be presented at the conferencewith a certificate and US$200. KEYNOTE SPEECHES: --------------------- Prof. Jack Dongarra, University of Tennessee, USA - "Present and Future Supercomputer Architectures" Prof. Linoel Ni, Hong Kong University of Science and Technology - "Challenges in P2P Computing" CONFERENCE WEB SITE --------------------- http://www.comp.polyu.edu.hk/ISPA04 ORGANIZING AND PROGRAM COMMITTES -------------------------------- General Co-Chairs: Minyi Guo, Univ. of Aizu, Japan Francis Lau, Univ. of Hong Kong, H.K. Program Co-Chairs: Jiannong Cao, Hong Kong Polytechnic Univ, H.K. Laurence T. Yang, St. Francis Xavier Univ. Canada Vice Program Co-Chairs Rajkumar Buyya, Univ of Melbourne, Australia Weijia Jia, City Univ. of Hong Kong, H.K. B.D. Martino, Second Univ. of Naples, Italy Workshop Chair Hong-Va Leong, Hong Kong Polytechnic University Publicity Chair Cho-Li Wang, Univ. of Hong Kong, H.K. Local Organisation Chair Allan K.Y. Wong, H.K. PloyU, Hong Kong Publication Chair Alvin T.S. Chan, Hong Kong Polytechnic University Registration Chair Joseph K.Y. Ng, Hong Kong Baptist University Steering Committee Minyi Guo, Univ. of Aziu, Japan Jiannong Cao, H.K. PolyU, H.K. Yi Pan, George State Univ. USA Jie Wu, Florida Altantic Univ. USA Li Xie, Nanjing Univ. Program Committee David Abramson, Monash Univ. Australia Selim G. Akl, Queens University, Canada Giuseppe Anastasi, University of Pisa, Italy Hamid R. Arabnia, University of Georgia, USA Amy Apon, Univ. of Arkansas, USA David A. Bader, University of New Mexico, USA Mark Baker, University of Portsmouth, U.K. Virendra C. Bhavsar, University of New Brunswick, Canada Wentong Cai, Nanyang Technological University, Singapore Daoxu Chen, Nanjing University, China Xing Cai, University of Oslo, Norway Kessler Christoph - Linkoping University, Sweden Kranzlmueller Dieter - Linz University, Austria A. Goscinski, Deakin U. Australia Yanxiang He, Wuhan University, China Hung-Chang Hsiao, National Tsing-Hua University, Taiwan Ching-Hsien Hsu, Chung Hua University, Taiwan Chun-Hsi Huang, University of Connecticut, USA Xiaohua Jia, City University of Hong Kong Beihong Jin, Institute of Software, CAS, China Hai Jin, Huazhong University of Sciene and Technology, China Hatsuhiko Kato, Shonan Institute of Technology, Japan Sy-Yen Kuo, National Taiwan University, Taiwan Tau Leng, Scalable System Group, Dell Inc., USA Jie Li, University of Tskuba, Japan Lei Li, Hosei University, Japan Minglu Li, Shanghai Jiaotong University, China Wenjun Li, UT Southwestern Medical Center, USA Xiaoming Li, Peking University, China Wei-keng Liao - Northwestern University, Chicago, USA Man Lin, St Francis Xavier University, Canada Jiangchuan Liu, Chinese University of Hong Kong, Hong Kong Jian Lu, Nanjing University, China Paul Lu, Univ. of Alberta, Canada Jianhua Ma, Hosei University, Japan Rodrigo Mello, University of Sao Paulo, Brazil Schellekens Michel, National University of Ireland, Cork, Ireland Jun Ni, University of Iowa, USA Y. Pan, George State Univ., USA Yuzhong Sun, Institute of Computing Technology, CAS, China Sabin Tabirca, University College Cork, Ireland Ruppa K. Thulasiram, University of Manitoba, Canada Parimala Thulasiram, University of Manitoba, Canada Dhaene Tom - University of Antwerpen, Belgium Sudharshan Vazhkudai, ORNL, USA Guojung Wang, Central South University, China Andrew L Wendelborn, University of Adelaide, Australia Jie Wu, Florida Altanlic University, USA Bing Xiao, Hong Kong Polytechnic University, Hong Kong Cheng-Zhong Xu, Wayne State Universuty, USA Ming Xu, National University of Defence Technology, China Zhiwei Xu, Institute of Computing Technology, CAS, China Jun Zhang, University of Kentucky, USA Xiaodong Zhang, William & Mary College, USA Wei Zhao, Texas T&M University, USA Weimin Zheng, Tsinghua University, Beijing Bingbing Zhou, University of Sydney, Australia Wanlei Zhou, Deakin University, Australia Xiaobao Zhou, University of Colorado at Colorado Springs, USA Ming Zhu, Drexel University, USA A.Y. Zomaya, Univ. of Sydney, Australia For more information, please contact: Dr. Jiannong Cao Deartment of Computing Hong Kong Polytechnic University Hung Hom, KLW, Hong Kong Email: csjcao@comp.polyu.edu.hk Prof. Laurence T. Yang Department of Computer Science St. Francis Xavier University Antigonish, B2G 2W5, NS, Canada Email: lyang@stfx.ca From alvin at Mail.Linux-Consulting.com Thu May 13 11:27:17 2004 From: alvin at Mail.Linux-Consulting.com (Alvin Oga) Date: Wed Nov 25 01:03:08 2009 Subject: squirrel cage - Re: [Beowulf] Re: Reliable CPU fans In-Reply-To: <200405131436.i4DEacVd018618@pookie.nersc.gov> Message-ID: hi ya shane On Thu, 13 May 2004 canon@nersc.gov wrote: > Richard, > > We've recently changed all of our Athlon based machines over > to fans similar to this.... > > http://www.dynatron-corp.com/products/cpucooler/cpucooler_model.asp?id=47# > > It is a squirrel cage design which seems to move more air and has > better reliability (from our experience) than the vertical style > fans from the same manufacturer. Unfortunately, the manufacturer > doesn't supply MTBF's for this fan. I can say this... The vertical as far as i know, dynatron is just a reseller like thousands of folks selling fans squirrel cages ( aka cross blow fans ) are a LOT more air ... and LOT quieter http://www.linux-1u.net/HeatSinkFans/Lateral/ ( panasonic/nmb would be our better choices ) - you can even order custom squirrel cages if you wanna make a jet engine quality fan blades ( good for finite element simulations of the airflow ( and fan blade desing to cool itself(the cluster) later the regular fans most people ( 99% ) see are axial fans and noisy when forcing air though the itty-bitty holes in the chassis http://www.linux-1u.net/HeatSinkFans/Axial/ c ya alvin From bill at cse.ucdavis.edu Thu May 13 11:48:33 2004 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: <1084448869.17067.55.camel@qeldroma.cttc.org> References: <1084448869.17067.55.camel@qeldroma.cttc.org> Message-ID: <20040513184833.GA24354@cse.ucdavis.edu> > In order to do the test, we have no doubt about the OS: Red Hat > Enterprise 3, but we are a bit confused about the harware of choice: Just FYI, I run RHEL-3 on both Itanium 2, P4, and Opteron and haven't noticed any practical differences. > As far as we know, Opteron has two main differences: > > - A wider memory interface (128 bit in front of 64) > - A larger L2 cache memory (1 Mb) Athlon FX, and opteron have a 128 bit memory interface. Athlon FX, and opteron have a 1 MB L2 cache. Some athlon 64's have 1 MB L2 cache, some have 512 KB l2 cache. > Which is the theoretical maximum performance gain using full 64 bit > architecture in front of 32 bit, taking into account that our > computations are done in double precision floating point using really > big matrices? I didn't quite parse that. "32 bit" and "64 bit" have basically nothing to do with double precision floating point. Accelerated double FP has been common in "32 bit" chips for quite some time. In any case the best way to decide on nodes, is to take all the nodes that meet your needs (run the needed compilers, OS, and applications), fit any weight, heat, space and heat constraints, then take the application that justifies the cluster purchase and measure both performance and price/performance. Buying nodes on "theoretical maximum performance" is much like buying a car based on what the maximum RPM of the engine is. > Is it nowadays the 64 bit solution using Linux ready for production? IMO, yes. > If this is the case, which problems may we have to deal with in order to > compile and run our code in a full 64 bit environment? Well written code should "just work" with or without a recompile. Well written means never assuming that a pointer and an integer is the same size (by assigning one to the other), or assuming (with pointer arithmetic) that a pointer is 4 bytes. Often porting an application to a new architecture exposes bugs in the original code that affected the original architecture as well. > Which is the most mature solution: AMD Opteron or Intel Itanium? I've run both without noticing a lack of maturity. What exactly do you mean by mature? If you mean OS, Compiler, Application availability I'd definitely say opteron. If you mean units shipped, again I'd say opteron. If you mean something else I'm not sure. -- Bill Broadley Computational Science and Engineering UC Davis From edwardsa at plk.af.mil Thu May 13 12:03:41 2004 From: edwardsa at plk.af.mil (Arthur H. Edwards) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Vector processing on AMD 64 Message-ID: <200405131903.i4DJ3gla004603@knockout.kirtland.af.mil> AMD has advertised that their 64 bit architectures have vector registers. This was initially very exciting because eigensolvers are not easily paralllized for medium-sized problems, but vectorize very well. I say this based on experience with the CRAY-YMP where we saw speed-ups ofrr 100 when we used a verctorized eigensolver. The Y-MP had, I think 128 vector registers, which could easily account for the 100 speed up. Does anyone know 1. How many vector registers each of the AMD 64 bit CPU's has 2. Is there a source for the old CRAY vector library? Art Edwards -- Art Edwards Senior Research Physicist Air Force Research Laboratory Electronics Foundations Branch KAFB, New Mexico (505) 853-6042 (v) (505) 846-2290 (f) From cresio at dcc.fesurv.br Thu May 13 12:52:48 2004 From: cresio at dcc.fesurv.br (cresio@dcc.fesurv.br) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Help! Message-ID: <1084477968.40a3d210c0e02@dcc.fesurv.br> Hello, Where I can do download of the beowulf? What is the URL? Thanks! ------------------------------- Departamento de Ci?ncia da Computa??o - DCC-FESURV (www.dcc.fesurv.br) From moor007 at bellsouth.net Thu May 13 14:51:57 2004 From: moor007 at bellsouth.net (Tim Moore) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Re: Athlon64 / Opteron test References: <200405131900.i4DJ02Lw017764@bluewest.scyld.com> Message-ID: <000b01c43934$833f1a90$6f01a8c0@timlaptop> Hello - We have been running CFD codes on Opterons for approximately one year. We have been experimenting with Itaniums since Thanksgiving. We have 3 clusters: [1] Production Cluster: 16 CPU Opteron (8 nodes)/Myrinet/SLES 8 for AMD64. [2] Opteron Testbed: 4 CPU (2 nodes)/Dual Port Myrinet/SLES 8 AMD64 [3] Itanium 2 Testbed: 4 CPU (2 nodes)/Dual Port Myrinet/ SLES 8 for IA64 I have not tested RH ES3 for AMD64 so I will not make any comments. However, SLES 8 was the first OS for AMD64 and gives excellent performance. Q1: I have no response. Q2: I think it is ready for production mode. I have a production cluster running 1 CFD code, 1 Structural Dynamics Code, 2 Particle Codes, and 1 Solid Dynamics Code. The CFD and Particle Codes run in 64 bit over Myrinet. The Structural Dynamics Code and Solid Dynamics Code run in 32 bit over myrinet. The Structural Dynamics code runs over 32 bit because the development platform continues to be RH7.3. I have a dual athlon machine running 7.3 for compilation that links to the gm libs for AMD64/Myrinet and it works great. The solid dynamics code is quite large and am unsure whether it is the Portland Group Compilers or the code itself. I do not think the code developers know. Anyway, it compiles and runs in 32 bit mode. The issue, at least to me are the compilers. I have to use the Portland Group because of the Cray pointers. GCC works fine for codes that do not require the pointers. Q3: Maturity <<<<>>>> goes to Intel. However, I have had great difficulty compiling some of the codes in 64 bit mode. (Intel is either 64 bit or nothing, unless you want them to emulate 32 bit which is a performance reduction). The main issue with Intel is the compilers. We ran benchmarks of the CFD code compiled with gcc and Intel to find that the Intel compilers generate code 3X faster than that of gcc. On the other hand, the same CFD code ran on the opterons (compiled with gcc) only 8% slower than the Intel/I2 version. To summarize my response to 3... We build our own opteron nodes and purchase our I2 nodes from Promicro. We build a 1U/2U dual opteron node with myrinet for less than 1/3 the cost of a I2/1.4 GHz/Myrinet node. You can spend 3 times the money and get an 8% boost in performance. I am putting together a paper that documents workshop results and you may have a draft if you would like. We will be posting the results on our website soon. I hope that I have helped some. Tim Moore Chief Scientist, TCG > > Hi, > > We're studying the possibility of installing a 64 bit testing node. > > The main purpose is about getting impressions related to the performance > increase we would obtain in our particular scenario, computational fluid > dynamics. > > In order to do the test, we have no doubt about the OS: Red Hat > Enterprise 3, but we are a bit confused about the harware of choice: > > Athlon64 > Opteron > > As far as we know, Opteron has two main differences: > > - A wider memory interface (128 bit in front of 64) > - A larger L2 cache memory (1 Mb) > > Before doing any test, the questions are: > > 1) > > Which is the theoretical maximum performance gain using full 64 bit > architecture in front of 32 bit, taking into account that our > computations are done in double precision floating point using really > big matrices? > > 2) > > Is it nowadays the 64 bit solution using Linux ready for production? > If this is the case, which problems may we have to deal with in order to > compile and run our code in a full 64 bit environment? > > 3) > > Which is the most mature solution: AMD Opteron or Intel Itanium? > From moor007 at bellsouth.net Thu May 13 15:01:47 2004 From: moor007 at bellsouth.net (Tim Moore) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Re: Athlon64 / Opteron test References: <200405131900.i4DJ02Lw017764@bluewest.scyld.com> Message-ID: <001701c43935$e2beedf0$6f01a8c0@timlaptop> I forgot to respond about Opteron vs Athlon 64... I have an Athlon 64 laptop and have not been particularly impressed with it as I have the Opterons. The Athlon 64 does have some serious thermal issues. Now, I must caveat that statement by saying that it is one of the mobile processors. (Athlon64 3400+). I run a dual boot with Suse Professional for AMD64 and Win2K. I get OK performance when running alpha and beta codes on the linux side. Linux reports the processor as an 800 MHz CPU. It seems to get better performance under Win2K... I do not know that it does, it is just my perception. My recommendation is to spend your money on opterons when choosing between them and Athlon64. Tim Moore Chief Scientist, TCG > In order to do the test, we have no doubt about the OS: Red Hat > Enterprise 3, but we are a bit confused about the harware of choice: > > Athlon64 > Opteron > From Andrew.Cannon at nnc.co.uk Fri May 14 00:56:32 2004 From: Andrew.Cannon at nnc.co.uk (Cannon, Andrew) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Athlon64 / Opteron test Message-ID: > See answers indicated by the > Hi, We're studying the possibility of installing a 64 bit testing node. The main purpose is about getting impressions related to the performance increase we would obtain in our particular scenario, computational fluid dynamics. In order to do the test, we have no doubt about the OS: Red Hat Enterprise 3, but we are a bit confused about the harware of choice: Athlon64 Opteron As far as we know, Opteron has two main differences: - A wider memory interface (128 bit in front of 64) - A larger L2 cache memory (1 Mb) Before doing any test, the questions are: 1) Which is the theoretical maximum performance gain using full 64 bit architecture in front of 32 bit, taking into account that our computations are done in double precision floating point using really big matrices? > I don't know about this one, but I think you expect about a 20% speed increase over the equivalent 32 bit program on the Athlon 64. (i.e. compile a code for 32 bit and then compile the same code for 64 bit and run them on the same processor, and I've read that you get about a 20% speed increase). 2) Is it nowadays the 64 bit solution using Linux ready for production? If this is the case, which problems may we have to deal with in order to compile and run our code in a full 64 bit environment? > The Linux kernel has had 64 bit support for the Hammer family of processors since before the processors actually came out. It is the supporting programs and drivers that may cause problems. (We are having display driver problems in X). gcc has got 64 bit support and the NAG compilers are available for x86_64. (I received NAG Fortran v5 a couple of days ago). 3) Which is the most mature solution: AMD Opteron or Intel Itanium? > It depends on what you want. Itanium is more mature, but doesn't give the performance increase you would expect given the price. There is a good thread on the AMD forums that is looking at the relative speeds of Itanium systems and Opteron systems running the same code. It can be found at http://forums.amd.com/index.php?showtopic=14239 and compares a couple of runs. It is not exactly benchmarking, but is a real world application. Personally, I would choose the Opteron. Itanium is too expensive for the performance you get from it. > Also, some Athlon 64 processors have 1MB cache. The difference between the Athlon64 and the Opteron is not that big. The major advantage of the Opteron is that you can have it in dual, quad or 8 way systems. Hope that helps. Andy NNC's UK Operating Companies : NNC Holdings Limited (no. 3725076), NNC Limited (no. 1120437), National Nuclear Corporation Limited (no. 2290928), STATS-NNC Limited (no. 4339062) and Technica-NNC Limited (no. 235856). The registered office of each company is at Booths Hall, Chelford Road, Knutsford, Cheshire WA16 8QZ except for Technica-NNC Limited whose registered office is at 6 Union Row, Aberdeen AB10 1DQ. NNC's head office and principal address is Booths Hall and the switchboard number is 01565 633800. The NNC website is www.nnc.co.uk Any information or opinions in this message which do not relate to our business are not authorised by any of the above companies. Where this message does relate to our business the content is sent to you by the relevant company (as above) and is confidential and intended for the use of the individual or entity to whom it is addressed. The contents do not form part of any contract unless so stated. If you have received this e-mail in error please notify the NNC system manager by email at eadm@nnc.co.uk. From tegner at nada.kth.se Fri May 14 02:16:20 2004 From: tegner at nada.kth.se (Jon Tegner) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Regarding wulfstat Message-ID: <40A48E64.40503@nada.kth.se> Hi, Trying to use wulfstat on a RH7.3 system. Installed xmlsysd and wulfstat from the source-rpms and have the following problems: When specifying the names through b%03d 1 2 the two nodes (b001 and b002) are marked as down. However, they seem to be resolved to the correct ip-addresses, and if they are specified by b001 b002 they are handled correctly. Furthermore, when using 192.168.1.1 192.168.1.2 wulfstat dies with a segmentation fault. Any help/hints on this would be appreciated. And Thank You Robert for a nice little program! Regards, /jon From agrajag at dragaera.net Fri May 14 06:43:57 2004 From: agrajag at dragaera.net (Sean Dilda) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: <1084448869.17067.55.camel@qeldroma.cttc.org> References: <1084448869.17067.55.camel@qeldroma.cttc.org> Message-ID: <1084542236.4478.57.camel@pel> On Thu, 2004-05-13 at 07:47, Daniel Fernandez wrote: > > Which is the most mature solution: AMD Opteron or Intel Itanium? Despite what Intel claims, its the consensus of everyone I know in the industry that the (t)itanium has already sunk. If you need proof, just look at the fact that Intel is now releasing a chip that's compatible with the Opteron instruction set. From rgb at phy.duke.edu Fri May 14 07:49:07 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Help! In-Reply-To: <1084477968.40a3d210c0e02@dcc.fesurv.br> Message-ID: On Thu, 13 May 2004 cresio@dcc.fesurv.br wrote: > Hello, > > Where I can do download of the beowulf? What is the URL? > > Thanks! There is no single "beowulf" to download. A beowulf is a particular style of cluster computer used for High Performance Computing (HPC) -- numerical computations in parallel, basically. You can read about beowulfs in an online book and other online resources here: http://www.phy.duke.edu/brahma (look under the Resources link). There are lots of links under the Links link to other major beowulf/cluster computing sites and resources. You can build a beowulf or GRID or general purpose parallel compute cluster on top of any Linux distribution. There are also several "special" distributions and software collections (such as free OSCAR and commercial Scyld) that more directly support the construction of a beowulf-style cluster but that often lag the primary distributions significantly in how up to date they are (which may or may not matter to you). So, you're homework assignment is: Read stuff you find here and on related links and come back to ask a more focussed question next week, if the online material doesn't answer all your questions. rgb > > > > ------------------------------- > Departamento de Ci?ncia da Computa??o - DCC-FESURV (www.dcc.fesurv.br) > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From jcownie at etnus.com Fri May 14 07:57:53 2004 From: jcownie at etnus.com (James Cownie) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: Message from "Robert G. Brown" of "Thu, 13 May 2004 20:06:04 EDT." Message-ID: <1BOe8H-3Ef-00@etnus.com> > I think they'd still have trouble maintaining a market, because > Opterons are relatively easy to port to and will in principle run > i386 code (badly, of course) native. The "(badly, of course)" should definitely be removed. The mass market for the AMD64 chips at the moment is gamers running that other OS which so far _only_ runs in 32 bit mode. The gamers like the AMD64 chips because they're the fastest 32 bit x86 chips you can buy. (Check any of the benchmark sites for proof.) You may have been confusing the AMD64 bit chips with the Itanic, which does have a very slow 32 bit x86 mode. -- -- Jim -- James Cownie Etnus, LLC. +44 117 9071438 http://www.etnus.com From scheinin at crs4.it Fri May 14 08:11:05 2004 From: scheinin at crs4.it (Alan Scheinine) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Re: Athlon64 / Opteron test Message-ID: <200405141511.i4EFB5R03080@dali.crs4.it> In an article in the beowulf mailing list, Dr. Moore writes "Linux reports the processor as an 800 MHz CPU." There have been many messages posted concerning this problem in http://lists.suse.com/archive/suse-amd64/ in the May set of messages. Alan Scheinine From rgb at phy.duke.edu Fri May 14 08:20:12 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Regarding wulfstat In-Reply-To: <40A48E64.40503@nada.kth.se> Message-ID: On Fri, 14 May 2004, Jon Tegner wrote: > Hi, I reproduce the iprange behavior here, so that is just a plain old bug and I'll fix it shortly, I hope. The other I'm having a hard time reproducing, although I don't have a lot of RH7.3 systems left to play with. While I'm trying, check that wulfstat is current (I just now posted 0.5.7 to both brahma and personal sites). It has been substantially rewritten with the most important change being that it now is based on a library shared with both wulflogger and indirectly wulfweb (to facilitate further development of client tools). It also has better debugging. If you build it from e.g. the tarball for testing purposes, and put your failing wulfhosts specs into e.g. wulfhosts.tst, then running: ./wulfstat -f wulfhosts.tst -v 1 2> log should BOTH let you see wulfstat AND create a maximally verbose log of what it is doing. Let it run for maybe one update and quit wulfstat, then look at the log. It SHOULD tell you just what is failing in a segment near the top like: D_READHOST: Entering hostrange loop Validating hostname = r00, port = 0 D_READHOST: Validating host: D_READHOST: Starting name = |r00| hostip = || inetaddress = |33b60398| port = |0| D_READHOST: Looking up ip number for host r00. D_READHOST: Got host_id = 42134b58. D_READHOST: Setting ip_tmp to 192.168.182.50 D_READHOST: Reset hostip from D_READHOST: hostip is now 192.168.182.50 D_READHOST: Setting inetaddr from hostip 192.168.182.50 D_READHOST: Setting host r00's port = 7887 D_READHOST: Cleaned up host r00 (we hope). D_READHOST: Ending name = r00 hostip = 192.168.182.50 inetaddress = 32b6a8c0 port = 7887 Validating hostname = r01, port = 0 D_READHOST: Validating host: D_READHOST: Starting name = |r01| hostip = || inetaddress = |32b6a8c0| port = |0| D_READHOST: Looking up ip number for host r01. D_READHOST: Got host_id = 42134b58. D_READHOST: Setting ip_tmp to 192.168.182.51 D_READHOST: Reset hostip from D_READHOST: hostip is now 192.168.182.51 D_READHOST: Setting inetaddr from hostip 192.168.182.51 D_READHOST: Setting host r01's port = 7887 D_READHOST: Cleaned up host r01 (we hope). D_READHOST: Ending name = r01 hostip = 192.168.182.51 inetaddress = 33b6a8c0 port = 7887 Obviously it works here, and should clearly indicate it if the problem is e.g. something not resolving. > Any help/hints on this would be appreciated. And Thank You Robert for a > nice little program! You're welcome and I'm glad you are enjoying it. You should check out wulflogger and wulfweb (the latter is pretty heavily beta still, but works) if you haven't. You can see what wulfweb can do (very crudely still -- don't have time to really pretty this up yet) on: http://www.phy.duke.edu/resources/computing/brahma/wulfweb/wulfweb.html Our local users like this better than wulfstat in a lot of cases, although I don't have any of the other views done yet. I don't think it is as useful for admins or serious users, though -- I tend to run it at a time granularity of around a minute because updates bollix old/stupid browsers. I'll get back to you when I've fixed iprange. This is an egregious failure, so it is probably a simple enough bug. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From joe.griffin at mscsoftware.com Fri May 14 08:25:12 2004 From: joe.griffin at mscsoftware.com (Joe Griffin) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: References: Message-ID: <40A4E4D8.9010001@mscsoftware.com> Hi All, Originally this thread was about the choice of Athlon vs. Opteron. But the comparison between Opteron/Intel was brought up. I wish to state that the best choice is highly dependent on YOUR application. I test various CFD and FEM engineering applications. I have not only seen differences when comparing different application programs, but also when comparing different uses of the same program (say if a person changes a job from statics to dynamics). The biggest question should be how YOUR application is used. Below is a web site comparing IA32, IA64 (linux and HPUX), Opteron and an IBM P655 running AIX. The site should only be used to compare hardare platforms when running our software. I am sure that Fluent, LSTC/Dyna, Star-CD have similar sites. I recomend finding out about the software that you will be using. MSC.Nastran Hardware comparison: http://www.mscsoftware.com/support/prod_support/nastran/performance/v04_sngl.cfm Regards, Joe Griffin Robert G. Brown wrote: >>In order to do the test, we have no doubt about the OS: Red Hat >>Enterprise 3, but we are a bit confused about the harware of choice: >> >> Athlon64 >> Opteron >> >>As far as we know, Opteron has two main differences: >> >> - A wider memory interface (128 bit in front of 64) >> - A larger L2 cache memory (1 Mb) >> >> >>3) >> >>Which is the most mature solution: AMD Opteron or Intel Itanium? >> >> > >Did you mean mature or moribund;-)? > >I'm only half kidding. Itanium is dead as a doornail as technology goes >-- overpriced, underperforming, incompatible. Intel is migrating to a >(more or less) Opteron compatible 64 bit processor as fast as they can >get there, as Major Software Companies (MSC) have announced that they >aren't going to do major ports to new chips with new machine languages >and compilers anymore if they can possibly avoid it. If Intel dropped >the price of an Itanium to slightly LESS than that of an Opteron, I >think they'd still have trouble maintaining a market, because Opterons >are relatively easy to port to and will in principle run i386 code >(badly, of course) native. Sometimes. I haven't had a lot of luck with >it, of course, because you can't mix i386 code and 64 bit DLLs and we >installed a 64 bit version of the OS from the start, but theoretically >it will work. > >The good news is that Opterons are surprisingly fast for MY applications >for their relatively pokey CPU clocks, and some benchmarks show that >they can be really quite fast indeed for memory intensive applications >relative to e.g. an Athlon or P4 clock. They also run much cooler than >regular Athlons (again for my application). I draw ballpark of 185 >watts loaded (dual CPU Opteron) vs 230 Watts or so loaded (dual CPU >Athlon) running more or less the same code. > > rgb > > > From rgb at phy.duke.edu Fri May 14 08:35:57 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: <40A4E4D8.9010001@mscsoftware.com> Message-ID: On Fri, 14 May 2004, Joe Griffin wrote: > Hi All, > > Originally this thread was about the choice of Athlon vs. Opteron. > But the comparison between Opteron/Intel was brought up. > > I wish to state that the best choice is highly dependent > on YOUR application. I test various CFD and FEM As I said earlier in the same response (and have said for years, will write in the July CW Cluster Kickstart column and will say yet again, I'm sure:-) YMMV, the most useful "benchmark" is your own application, and even more, your own application compiled with the actual compilers and linked to the actual libraries you plan to use on production and run on a system that an exact prototype of the systems you plan to use in production and as you note you STILL can easily see sigificant variations in performance for many applications that can shift cost/benefit optima by simply changing the scale of the computation! A benchmark running out of cache in a "small" run may scale very nonlinearly in speed compared to the same benchmark running out of main memory in a "large" run, or a shift in parameters can cause a good algorithm to go bad. The ATLAS library is a wonderful example of the tremendous benefits that can be had just optimizing algorithm for a given architecture, and serves equally well to illustrate that running a benchmark with an UNoptimized BLAS may well not give you anything like an accurate idea of real comparative system performance. So absolutely, I agree! On the other hand, to be fair, a lot of times one's application is bottlenecked in just one or two core loops with a relatively simple and repetitive structure, and it isn't that uncommon to find performance scaling nearly perfectly with system CPU clock within an architecture family across many variations in motherboard, bus speed, cache size, and so forth. But YMMV! rgb > engineering applications. I have not only seen differences > when comparing different application programs, but also > when comparing different uses of the same program (say if a > person changes a job from statics to dynamics). The biggest question > should be how YOUR application is used. > > Below is a web site comparing IA32, IA64 (linux and HPUX), Opteron > and an IBM P655 running AIX. The site should only be used to > compare hardare platforms when running our software. I am sure > that Fluent, LSTC/Dyna, Star-CD have similar sites. I recomend > finding out about the software that you will be using. > > MSC.Nastran Hardware comparison: > > http://www.mscsoftware.com/support/prod_support/nastran/performance/v04_sngl.cfm > > Regards, > Joe Griffin > > > Robert G. Brown wrote: -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb at phy.duke.edu Fri May 14 08:49:34 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:08 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: <1BOe8H-3Ef-00@etnus.com> Message-ID: On Fri, 14 May 2004, James Cownie wrote: > > > I think they'd still have trouble maintaining a market, because > > Opterons are relatively easy to port to and will in principle run > > i386 code (badly, of course) native. > > The "(badly, of course)" should definitely be removed. > > The mass market for the AMD64 chips at the moment is gamers running > that other OS which so far _only_ runs in 32 bit mode. > > The gamers like the AMD64 chips because they're the fastest 32 bit x86 > chips you can buy. > > (Check any of the benchmark sites for proof.) > > You may have been confusing the AMD64 bit chips with the Itanic, which > does have a very slow 32 bit x86 mode. Actually, I was referring to the following: rgb@ganesh|B:1117>uname -a Linux ganesh 2.4.20-30.9 #1 Wed Feb 4 20:45:39 EST 2004 i686 athlon i386 GNU/Linux rgb@ganesh|B:1118>pwd /home/einstein/rgb/Src/Ospin rgb@ganesh|B:1119>make make: Nothing to be done for `all'. rgb@ganesh|B:1121>ssh s02 rgb@s02|B:1001>cd Src/Ospin rgb@s02|B:1002>uname -a Linux s02 2.4.22-1.2188.nptlsmp #1 SMP Wed Apr 21 20:12:16 EDT 2004 x86_64 x86_64 x86_64 GNU/Linux rgb@s02|B:1003>./Ospin -bash: ./Ospin: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory rgb@s02|B:1004> Where on a regular Athlon (basically RH9 i386) I compile Ospin (my current application) and then I ssh to s02 (fedora x86_64) to run the binary, it barfs. Probably missing a compatibility library, maybe the library is around somewhere but not installed but I don't care much BECAUSE... ...it also refers to anecdotal accounts that numerical performance significantly degrades if one runs i386 code compared to recompiled x86_64 code. That may leave it plenty fast enough for gamers (who may be running memory bus bound vector float transformation code a lot and hence get benefit from the fast memory without getting much advantage from the CPU), but still slower than the optimum one would like running numerical applications that take hours to days to complete. So OK, "badly, of course" is perhaps overly harsh. "Better than Itaniums" would be fair, I think. So would "more slowly than native 64 bit recompiles" which is what most people care about anyway. I stand corrected;-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From douglas at pop3.shore.net Fri May 14 09:12:06 2004 From: douglas at pop3.shore.net (douglas@pop3.shore.net) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] RE: Vector processing on AMD 64 Message-ID: <54360-22004551416126933@M2W084.mail2web.com> Arthur: I think you are refering to the SSE2 instructions available in (pretty much) all x86 CPUs theses days. There are also the ACML math libraries for Opterons which are optimized for BLAS routines. You may here these refered to as the 'vectorized' libraries. Go to developer.amd.com for links to the libraries and various presentations. In particular you might be interested in some of Tim Wilken's presentations there. doug ======= Message: 6 Date: Thu, 13 May 2004 13:03:41 -0600 From: "Arthur H. Edwards" Subject: [Beowulf] Vector processing on AMD 64 To: beowulf@beowulf.org Cc: "Arthur H. Edwards" Message-ID: <200405131903.i4DJ3gla004603@knockout.kirtland.af.mil> Content-Type: text/plain; charset=us-ascii AMD has advertised that their 64 bit architectures have vector registers. This was initially very exciting because eigensolvers are not easily paralllized for medium-sized problems, but vectorize very well. I say this based on experience with the CRAY-YMP where we saw speed-ups ofrr 100 when we used a verctorized eigensolver. The Y-MP had, I think 128 vector registers, which could easily account for the 100 speed up. Does anyone know 1. How many vector registers each of the AMD 64 bit CPU's has 2. Is there a source for the old CRAY vector library? Art Edwards -- Art Edwards Senior Research Physicist Air Force Research Laboratory Electronics Foundations Branch KAFB, New Mexico (505) 853-6042 (v) (505) 846-2290 (f) -------------------------------------------------------------------- mail2web - Check your email from the web at http://mail2web.com/ . From rbw at ahpcrc.org Fri May 14 09:13:42 2004 From: rbw at ahpcrc.org (Richard Walsh) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Vector processing on AMD 64 In-Reply-To: <200405131903.i4DJ3gla004603@knockout.kirtland.af.mil> Message-ID: <20040514161342.CFF046EB07@clam.ahpcrc.org> Art Edwards wrote: >AMD has advertised that their 64 bit architectures have vector >registers. This was initially very exciting because eigensolvers are not >easily paralllized for medium-sized problems, but vectorize very well. >I say this based on experience with the CRAY-YMP where we saw speed-ups >ofrr 100 when we used a verctorized eigensolver. The Y-MP had, I think >128 vector registers, which could easily account for the 100 speed up. Vector computing on the microprocessor is almost a different beast from traditional vector computing in that vectors are defined width-wise rather than length-wise. As such the SSE1/2/3 offer instructions that load, compute, and store either 4, 32 bit or 2, 64 entities at a time (not all micros load and store the full 128-bits). Unlike the Cray ISA, which loads words in sequence up to the vector length (64x64 or 128x64) from memory with a single instruction, the microprocessors load and computes on one wide word at a time (prefetching functions as a pseudo-vector load) ... that makes the vector length/ width of your microprocessor 2 or 4 depending on the word size. The totality of the Cray ISA and architecture is only poorly emulated by the vector operations on the various "vectorizing" microprocessoers. It is a good thing, but not very close to the original concept yet. >Does anyone know? >1. How many vector registers each of the AMD 64 bit CPU's has It has 16x128-bit regs, but these do not reflect vector length. The Cray X1 has 32x64x64-bit regs per SSP. >2. Is there a source for the old CRAY vector library? Not sure what is meant here, unless your are refering to SCILIB which is merely an API and can be emulated by any processor that can do FLOPS. Remember it is the ISA (and supporting hardware) that supports the libraries that is key. Someone may have written a SCILUB based on SSE instructions. Regards, rbw PS Great to have the reflector operational again! #--------------------------------------------------- # Richard Walsh # Project Manager, Cluster Computing, Computational # Chemistry and Finance # netASPx, Inc. # 1200 Washington Ave. So. # Minneapolis, MN 55415 # VOX: 612-337-3467 # FAX: 612-337-3400 # EMAIL: rbw@networkcs.com, richard.walsh@netaspx.com # rbw@ahpcrc.org # #--------------------------------------------------- # "What you can do, or dream you can, begin it; # Boldness has genius, power, and magic in it." # -Goethe #--------------------------------------------------- # "Without mystery, there can be no authority." # -Charles DeGaulle #--------------------------------------------------- # "Why waste time learning when ignornace is # instantaneous?" -Thomas Hobbes #--------------------------------------------------- From Karl.Bellve at umassmed.edu Fri May 14 09:26:10 2004 From: Karl.Bellve at umassmed.edu (Karl Bellve) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: References: Message-ID: <40A4F322.7090803@umassmed.edu> We have a project that I have timed on various CPUs, and compilers. It makes heavy use of FFTW (single precision). Running native i386 code does very well for us on the Opteron, unlike the Itanium 2. *Itanium 2 1.0 GHz on Red Hat Linux Advanced Workstation release 2.1AW* 234 seconds (IA64 code and ECC, -fno-alias -IPF_fma -Ob2 -ipo -restrict) 1515 seconds (i386 code and GCC) *AMD Opteron 1.4 GHz on United Linux 1.0* 189 seconds (gcc, x86_64 code) 221 seconds (gcc, i386 code) *Athlon 1.5 GHz (AMD Athlon(tm) MP Processor 1800+) on Redhat 8.0* 190 seconds (gcc, i386 code) *Intel Xeon 1.7 GHz on Redhat 8.0* 267 seconds (gcc, i386 code) *Sun Intel Xeon 2.8 GHz on Redhat 7.3* 150 seconds (gcc, i386 code) *Alpha 667 GHz* 430 seconds (Compaq compiler for both, optimized code) *Intel 2.2Ghz Xeon on RH 7.3* 208 seconds (gcc, i386 code) -- Cheers, Karl Bellve -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20040514/74bfde1a/attachment.html From heckendo at uidaho.edu Fri May 14 09:44:01 2004 From: heckendo at uidaho.edu (Robert B Heckendorn) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] 64bit comparisons In-Reply-To: <200405141538.i4EFcQpd015889@bluewest.scyld.com> Message-ID: <200405141644.i4EGi1Aq023213@marvin.ibest.uidaho.edu> > 1. Re: Athlon64 / Opteron test (Tim Moore) > 2. RE: Athlon64 / Opteron test (Cannon, Andrew) > 3. Regarding wulfstat (Jon Tegner) > 4. Re: Athlon64 / Opteron test (Sean Dilda) > 5. Re: Help! (Robert G. Brown) > 6. Re: Athlon64 / Opteron test (James Cownie) > 7. Re: Athlon64 / Opteron test (Alan Scheinine) > 8. Re: Regarding wulfstat (Robert G. Brown) > 9. Re: Athlon64 / Opteron test (Joe Griffin) > 10. Re: Athlon64 / Opteron test (Robert G. Brown) One of the options we are strongly considering for our next cluster is going with Apple X-servers. There performance is purported to be good and their power consumption is small. We have ordered a tiny test cluster to do some checking. Can people comment on any comparisons betwee Apple and (Athlon64 or Opteron)? thanks, -- | Robert Heckendorn | "The belief that there is only | heckendo@uidaho.edu | one truth, and that oneself is | http://marvin.ibest.uidaho.edu/~heckendo | in possession of it, is the root | CS Dept, University of Idaho | of all evil in the world." | Moscow, Idaho, USA 83844-1010 | -- Max Born From tegner at nada.kth.se Fri May 14 10:00:22 2004 From: tegner at nada.kth.se (Jon Tegner) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Regarding wulfstat In-Reply-To: References: Message-ID: <40A4FB26.5000103@nada.kth.se> Thanks a lot for info! After doing the verbose test I realized that the nodes that use hostrange got the wrong port number. After specifying 7887 it worked as expected. Thanks again, much apreciated! /jon Robert G. Brown wrote: >On Fri, 14 May 2004, Jon Tegner wrote: > > > >>Hi, >> >> > >I reproduce the iprange behavior here, so that is just a plain old bug >and I'll fix it shortly, I hope. > >The other I'm having a hard time reproducing, although I don't have a >lot of RH7.3 systems left to play with. > >While I'm trying, check that wulfstat is current (I just now posted >0.5.7 to both brahma and personal sites). It has been substantially >rewritten with the most important change being that it now is based on a >library shared with both wulflogger and indirectly wulfweb (to >facilitate further development of client tools). > >It also has better debugging. If you build it from e.g. the tarball for >testing purposes, and put your failing wulfhosts specs into e.g. >wulfhosts.tst, then running: > > ./wulfstat -f wulfhosts.tst -v 1 2> log > >should BOTH let you see wulfstat AND create a maximally verbose log of >what it is doing. Let it run for maybe one update and quit wulfstat, >then look at the log. It SHOULD tell you just what is failing in a >segment near the top like: > >D_READHOST: Entering hostrange loop >Validating hostname = r00, port = 0 >D_READHOST: Validating host: >D_READHOST: Starting name = |r00| hostip = || inetaddress = |33b60398| port = |0| >D_READHOST: Looking up ip number for host r00. >D_READHOST: Got host_id = 42134b58. >D_READHOST: Setting ip_tmp to 192.168.182.50 >D_READHOST: Reset hostip from >D_READHOST: hostip is now 192.168.182.50 >D_READHOST: Setting inetaddr from hostip 192.168.182.50 >D_READHOST: Setting host r00's port = 7887 >D_READHOST: Cleaned up host r00 (we hope). >D_READHOST: Ending name = r00 hostip = 192.168.182.50 inetaddress = 32b6a8c0 port = 7887 >Validating hostname = r01, port = 0 >D_READHOST: Validating host: >D_READHOST: Starting name = |r01| hostip = || inetaddress = |32b6a8c0| port = |0| >D_READHOST: Looking up ip number for host r01. >D_READHOST: Got host_id = 42134b58. >D_READHOST: Setting ip_tmp to 192.168.182.51 >D_READHOST: Reset hostip from >D_READHOST: hostip is now 192.168.182.51 >D_READHOST: Setting inetaddr from hostip 192.168.182.51 >D_READHOST: Setting host r01's port = 7887 >D_READHOST: Cleaned up host r01 (we hope). >D_READHOST: Ending name = r01 hostip = 192.168.182.51 inetaddress = 33b6a8c0 port = 7887 > >Obviously it works here, and should clearly indicate it if the problem >is e.g. something not resolving. > > > >>Any help/hints on this would be appreciated. And Thank You Robert for a >>nice little program! >> >> > >You're welcome and I'm glad you are enjoying it. You should check out >wulflogger and wulfweb (the latter is pretty heavily beta still, but >works) if you haven't. You can see what wulfweb can do (very crudely >still -- don't have time to really pretty this up yet) on: > > http://www.phy.duke.edu/resources/computing/brahma/wulfweb/wulfweb.html > >Our local users like this better than wulfstat in a lot of cases, >although I don't have any of the other views done yet. I don't think it >is as useful for admins or serious users, though -- I tend to run it at >a time granularity of around a minute because updates bollix old/stupid >browsers. > >I'll get back to you when I've fixed iprange. This is an egregious >failure, so it is probably a simple enough bug. > > rgb > > > From landman at scalableinformatics.com Fri May 14 10:32:33 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: <40A4E4D8.9010001@mscsoftware.com> References: <40A4E4D8.9010001@mscsoftware.com> Message-ID: <20040514164955.M14725@scalableinformatics.com> Hi Joe: Quite surprised to see these numbers, as they don't seem to line up well with what my customers are seeing. Interesting data points though. I see you are using an older (ancient) kernel for the e325. I wonder if you have all of your memory physically attached to one versus two processors. Lots of folks see slowdowns of some sort when they don't set up banking. I do know that some compilers (Intel specifically) check for processor identity, and switch in or out the SSE2 versions of loops depending upon the return code. I also have heard cases where this (dramatically negatively) impacts Opteron performance until it is "patched". Greg Lindahl wrote about this in a recent ClusterWorld article. Measurements we have done on non-engineering codes using the GCC, PGI, and other compilers have shown that Xeons and Opterons are generally similar for 32 bit codes, to within a few percent. When you recompile with the -m64 option that you get some 5-30% advantages. As you said, YMMV. Its also quite easy to misconfigure these machines, and I have seen this in a number of benchmarks. The memory configuration can (drastically) impact performance. I might also suggest updating to a modern kernel, one with at least some NUMA aware functionality. 2.4.18/2.4.19 was used in the RedHat GinGin64 series. This was not a good OS platform for benchmarking. Just some thoughts. Joe On Fri, 14 May 2004 08:25:12 -0700, Joe Griffin wrote > Hi All, > > Originally this thread was about the choice of Athlon vs. Opteron. > But the comparison between Opteron/Intel was brought up. > > I wish to state that the best choice is highly dependent > on YOUR application. I test various CFD and FEM > engineering applications. I have not only seen differences > when comparing different application programs, but also > when comparing different uses of the same program (say if a > person changes a job from statics to dynamics). The biggest question > should be how YOUR application is used. > > Below is a web site comparing IA32, IA64 (linux and HPUX), Opteron > and an IBM P655 running AIX. The site should only be used to > compare hardare platforms when running our software. I am sure > that Fluent, LSTC/Dyna, Star-CD have similar sites. I recomend > finding out about the software that you will be using. > > MSC.Nastran Hardware comparison: > > http://www.mscsoftware.com/support/prod_support/nastran/performance/v04_sngl.cfm > > Regards, > Joe Griffin > > Robert G. Brown wrote: > > >>In order to do the test, we have no doubt about the OS: Red Hat > >>Enterprise 3, but we are a bit confused about the harware of choice: > >> > >> Athlon64 > >> Opteron > >> > >>As far as we know, Opteron has two main differences: > >> > >> - A wider memory interface (128 bit in front of 64) > >> - A larger L2 cache memory (1 Mb) > >> > >> > >>3) > >> > >>Which is the most mature solution: AMD Opteron or Intel Itanium? > >> > >> > > > >Did you mean mature or moribund;-)? > > > >I'm only half kidding. Itanium is dead as a doornail as technology goes > >-- overpriced, underperforming, incompatible. Intel is migrating to a > >(more or less) Opteron compatible 64 bit processor as fast as they can > >get there, as Major Software Companies (MSC) have announced that they > >aren't going to do major ports to new chips with new machine languages > >and compilers anymore if they can possibly avoid it. If Intel dropped > >the price of an Itanium to slightly LESS than that of an Opteron, I > >think they'd still have trouble maintaining a market, because Opterons > >are relatively easy to port to and will in principle run i386 code > >(badly, of course) native. Sometimes. I haven't had a lot of luck with > >it, of course, because you can't mix i386 code and 64 bit DLLs and we > >installed a 64 bit version of the OS from the start, but theoretically > >it will work. > > > >The good news is that Opterons are surprisingly fast for MY applications > >for their relatively pokey CPU clocks, and some benchmarks show that > >they can be really quite fast indeed for memory intensive applications > >relative to e.g. an Athlon or P4 clock. They also run much cooler than > >regular Athlons (again for my application). I draw ballpark of 185 > >watts loaded (dual CPU Opteron) vs 230 Watts or so loaded (dual CPU > >Athlon) running more or less the same code. > > > > rgb > > > > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 From kus at free.net Fri May 14 11:24:05 2004 From: kus at free.net (Mikhail Kuzminsky) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: <40A4E4D8.9010001@mscsoftware.com> from "Joe Griffin" at May 14, 4 08:25:12 am Message-ID: <200405141827.WAA12362@nocserv.free.net> According to Joe Griffin > > ... > Below is a web site comparing IA32, IA64 (linux and HPUX), Opteron > and an IBM P655 running AIX. The site should only be used to > compare hardare platforms when running our software. I am sure > that Fluent, LSTC/Dyna, Star-CD have similar sites. I recomend > finding out about the software that you will be using. > > MSC.Nastran Hardware comparison: > > http://www.mscsoftware.com/support/prod_support/nastran/performance/v04_sngl.cfm > > Regards, > Joe Griffin > This page contains very interesting tables w/description of hardware used, but at first look I found only the data about OSes, not about compilers/run time libraries used. The (relative bad) data for IBM e325/Opteron 2 Ghz looks "nontrivial"; I beleive some interptretation of "why?" will be helpful. May be some applications used are relative cache-friendly and have working set placing in large Itanium 2 cache? May be it depends from compiler and Math library used ? BTW, for LGQDF test: I/O is relative small (compare pls elapsed and CPU times which are very close); but Windows time for Dell P4/3.2 Ghz (4480 sec) is much more worse than for Linux on the same hardware (3713 sec). IMHO, in this case they must be very close in the case of using same comlilers&libraries (I don't like Windows, but this result is too bad for this OS :-)) Yours Mikhail Kuzminsky Zelinsky Institute of Organic Chemistry Moscow From bill at cse.ucdavis.edu Fri May 14 11:48:21 2004 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] 64bit comparisons In-Reply-To: <200405141644.i4EGi1Aq023213@marvin.ibest.uidaho.edu> References: <200405141538.i4EFcQpd015889@bluewest.scyld.com> <200405141644.i4EGi1Aq023213@marvin.ibest.uidaho.edu> Message-ID: <20040514184821.GF31642@cse.ucdavis.edu> On Fri, May 14, 2004 at 09:44:01AM -0700, Robert B Heckendorn wrote: > One of the options we are strongly considering for our next cluster is > going with Apple X-servers. There performance is purported to be good Careful to benchmark both processors at the same time if that is your intended usage pattern. Are the dual-g5's shipping yet? Last I heard yield problems were resulting in only uniprocessor shipments. My main concern that despite the marketing blurb of 2 10GB/sec CPU interfaces or similar that there is a shared 6.4 GB/sec memory bus. > and their power consumption is small. Has anyone measured a dual g5 xserv with a kill-a-watt or similar? > Can people comment on any comparisons betwee Apple and (Athlon64 > or Opteron)? Personally I've had problems, I need to spend more time resolving them, things like: * Need to tweak /etc/rc to allow Mpich to use shared memory * Latency between two mpich processes on the same node is 10-20 times the linux latency. I've yet to try LAM. * Differences in semaphores requires a rewrite for some linux code I had * Difference in the IBM fortran compiler required a rewrite compared to code that ran on Intel's, portland group's, and GNU's fortran compiler. Given all that I'm still interested to see what the G5 is good at and under what workloads the G5 wins perf/price or perf/watt. -- Bill Broadley Computational Science and Engineering UC Davis From david.n.lombard at intel.com Fri May 14 13:28:49 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Athlon64 / Opteron test Message-ID: <187D3A7CAB42A54DB61F1D05F0125722025F580A@orsmsx402.jf.intel.com> From: Joe Landman; Friday, May 14, 2004 10:33 AM [deletia] > I see you are using an older (ancient) kernel for the e325. I wonder if > you > have all of your memory physically attached to one versus two processors. > Lots > of folks see slowdowns of some sort when they don't set up banking. While I don't know about these specific results, I was at MSC.Software when Opteron benchmarks were being performed on AMD-provided systems. MSC, AMD, and IBM all spent a lot of time and effort on the benchmarks, with AMD providing hardware, OS, and support to ensure the results were as good as possible. The kernel was specifically NUMA-capable, as provided by AMD, with memory arranged per AMD specifications, with much experimentation by all involved to ensure best result. I can only assume that IBM and AMD worked equally well on the benchmarks presented on the page. > Measurements we have done on non-engineering codes using the GCC, PGI, > and > other compilers have shown that Xeons and Opterons are generally similar > for 32 > bit codes, to within a few percent. When you recompile with the -m64 > option > that you get some 5-30% advantages. Over the many years I worked at MSC, specifically concerned w/ Nastran performance, I saw many disconnects between performance on non-CAE applications and Nastran. Through the combined efforts of MSC.Software's internal experts, like Joe Griffin, and vendor optimization experts, MSC.Nastran usually did a fairly good job of achieving maximal performance -- that was certainly my personal goal. For the record, one particular platform actually achieved about 98% peak theoretical performance on a dense vector-matrix operation. Note I wrote usually, MSC.Nastran demands a lot from a computer. It could also highlight weaknesses, especially those related to "machine balance". One relatively unknown, but quite notorious, example from the '90s had an order-of-magnitude difference between CPU and elapsed time due to a customer's specific benchmarking requirements and an OEM's poor I/O performance. > As you said, YMMV. Its also quite easy to misconfigure these machines, > and I > have seen this in a number of benchmarks. The memory configuration can > (drastically) impact performance. I might also suggest updating to a > modern > kernel, one with at least some NUMA aware functionality. 2.4.18/2.4.19 > was used > in the RedHat GinGin64 series. This was not a good OS platform for > benchmarking. As mentioned above, AMD, IBM, and MSC worked long and hard on the benchmarks, with all having the goal of obtaining the best showing. While I'm in no position to guarantee such na?ve problems did not occur, especially as I am no longer at MSC.Software, too many experts were too interested in the results for that to be likely. -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. From david.n.lombard at intel.com Fri May 14 13:32:45 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Athlon64 / Opteron test Message-ID: <187D3A7CAB42A54DB61F1D05F0125722025F580B@orsmsx402.jf.intel.com> From: Mikhail Kuzminsky > According to Joe Griffin [deletia] > > > > MSC.Nastran Hardware comparison: > > > http://www.mscsoftware.com/support/prod_support/nastran/performance/v04_ sn > gl.cfm > > > This page contains very interesting tables w/description of hardware > used, but at first look I found only the data about OSes, not about > compilers/run time libraries used. The (relative bad) data for IBM > e325/Opteron 2 Ghz > looks "nontrivial"; I beleive some interptretation of "why?" will be > helpful. > May be some applications used are relative cache-friendly and have working > set > placing in large Itanium 2 cache? For the record, on these single-processor MSC.Nastran jobs, FP performance, memory bandwidth, and I/O bandwidth are most significant; memory latency is irrelevant. MSC.Nastran is decidedly not cache-friendly; having said that, numerical kernels generally use rank-n updates, n>1, with the actual rank size being selected via processor-specific tuning. > May be it depends from compiler and Math library used ? BTW, for LGQDF > test: > I/O is relative small (compare pls elapsed and CPU times which are very > close); LGQDF does a TB of I/O, as shown in the table at the top. > but Windows time for Dell P4/3.2 Ghz (4480 sec) is much more worse than > for Linux on the same hardware (3713 sec). IMHO, in this case they > must be very close in the case of using same comlilers&libraries > (I don't like Windows, but this result is too bad for this OS :-)) Note: I left MSC.Software last year; I ceased running the Porting organization (i.e., having direct contact with how MSC.Nastran was built) in '00. Same machine differences like the Dell Linux/Windows results are more often OS related. See especially XXAFST, a statics job dominated by a single math kernel, with relatively low I/0 (yes, 77GB is "low" for MSC.Nastran), where the Windows and Linux CPU times are very close. Contrast that with the three jobs with the greatest Windows/Linux time difference, they do the largest amounts of I/O. -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. From lindahl at pathscale.com Fri May 14 13:35:50 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Re: Athlon64 / Opteron test In-Reply-To: <000b01c43934$833f1a90$6f01a8c0@timlaptop> References: <200405131900.i4DJ02Lw017764@bluewest.scyld.com> <000b01c43934$833f1a90$6f01a8c0@timlaptop> Message-ID: <20040514203550.GG2415@greglaptop.internal.keyresearch.com> On Thu, May 13, 2004 at 05:51:57PM -0400, Tim Moore wrote: > The issue, at least to me are the compilers. I have to use the Portland > Group because of the Cray pointers. GCC works fine for codes that do not > require the pointers. Tim, The PathScale Fortran compiler also supports Cray pointers on Opteron. -- greg From lindahl at pathscale.com Fri May 14 13:43:51 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: References: <1BOe8H-3Ef-00@etnus.com> Message-ID: <20040514204351.GH2415@greglaptop.internal.keyresearch.com> On Fri, May 14, 2004 at 11:49:34AM -0400, Robert G. Brown wrote: > rgb@s02|B:1003>./Ospin > -bash: ./Ospin: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory > rgb@s02|B:1004> Pilot error. You have to install a couple of additional rpms to run 32-bit stuff on an Opteron. The annoying thing about it is that they have the same names as x86_64 packages, grrrr. In this case you need glibc-*.i686.rpm. > ...it also refers to anecdotal accounts that numerical performance > significantly degrades if one runs i386 code compared to recompiled > x86_64 code. The hard thing about anecdotes is that they're hard to disprove. However, if you look at SPECfp scores for Opteron using the Intel compiler (32-bit mode) and the PathScale compiler (64-bit mode), I don't think that you'd come to that conclusion. It is an apples to oranges comparison: our compiler has a better optimization framework. But the benefit from 64-bit mode is clearly moderate. I talk about it in a recent ClusterWorld article. -- greg From joe.griffin at mscsoftware.com Fri May 14 14:00:38 2004 From: joe.griffin at mscsoftware.com (Joe Griffin) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: <200405141827.WAA12362@nocserv.free.net> References: <200405141827.WAA12362@nocserv.free.net> Message-ID: <40A53376.7080505@mscsoftware.com> Mikhail, Compilers aren't mentioned on the performance page. Lot's of the CPU time is spent in a few routines which are often hand tuned in assembler. An interesting note for this group is the dominance of Linux in the charts. When performance numbers for our last release were made, MSC Posted Tru64, UNICOS, Fujitsu/UXP, HPUX, AIX, Linux, NEC/SuperUX, IRIX, and Solaris numbers: http://www.mscsoftware.com/support/prod_support/nastran/performance/v0109_sngl.cfm but with this last release, most of the hardware companies submitted linux numbers: http://www.mscsoftware.com/support/prod_support/nastran/performance/v04_sngl.cfm I think this shift is worthy of notation. Joe Mikhail Kuzminsky wrote: >According to Joe Griffin > > >> ... >>Below is a web site comparing IA32, IA64 (linux and HPUX), Opteron >>and an IBM P655 running AIX. The site should only be used to >>compare hardare platforms when running our software. I am sure >>that Fluent, LSTC/Dyna, Star-CD have similar sites. I recomend >>finding out about the software that you will be using. >> >>MSC.Nastran Hardware comparison: >> >>http://www.mscsoftware.com/support/prod_support/nastran/performance/v04_sngl.cfm >> >>Regards, >>Joe Griffin >> >> >> > This page contains very interesting tables w/description of hardware >used, but at first look I found only the data about OSes, not about compilers/run time libraries used. The (relative bad) data for IBM e325/Opteron 2 Ghz >looks "nontrivial"; I beleive some interptretation of "why?" will be helpful. >May be some applications used are relative cache-friendly and have working set >placing in large Itanium 2 cache? > >May be it depends from compiler and Math library used ? BTW, for LGQDF test: >I/O is relative small (compare pls elapsed and CPU times which are very close); >but Windows time for Dell P4/3.2 Ghz (4480 sec) is much more worse than >for Linux on the same hardware (3713 sec). IMHO, in this case they > must be very close in the case of using same comlilers&libraries > (I don't like Windows, but this result is too bad for this OS :-)) > >Yours >Mikhail Kuzminsky >Zelinsky Institute of Organic Chemistry >Moscow > > > > > From orion at cora.nwra.com Fri May 14 15:49:40 2004 From: orion at cora.nwra.com (Orion Poplawski) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Opteron and 32-bit libraries In-Reply-To: <20040514204351.GH2415@greglaptop.internal.keyresearch.com> References: <1BOe8H-3Ef-00@etnus.com> <20040514204351.GH2415@greglaptop.internal.keyresearch.com> Message-ID: <40A54D04.90307@cora.nwra.com> Greg Lindahl wrote: > On Fri, May 14, 2004 at 11:49:34AM -0400, Robert G. Brown wrote: > > >>rgb@s02|B:1003>./Ospin >>-bash: ./Ospin: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory >>rgb@s02|B:1004> > > > Pilot error. You have to install a couple of additional rpms to run > 32-bit stuff on an Opteron. The annoying thing about it is that they > have the same names as x86_64 packages, grrrr. In this case you need > glibc-*.i686.rpm. > Does anyone know how to do this automatically on Fedora using kickstart or yum (or ?)? This is a pain in the butt to do by hand - esp. for a cluster. -- Orion Poplawski System Administrator 303-415-9701 x222 Colorado Research Associates/NWRA FAX: 303-415-9702 3380 Mitchell Lane, Boulder CO 80301 http://www.co-ra.com From becker at scyld.com Fri May 14 16:33:32 2004 From: becker at scyld.com (Donald Becker) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: <20040514204351.GH2415@greglaptop.internal.keyresearch.com> Message-ID: [[ The mailing list should now be back up and running for all subscribers. I'll write up the long story of the Beowulf Mailing List problems over the weekend, assuming that everything continues running. - DJB]] On Fri, 14 May 2004, Greg Lindahl wrote: > On Fri, May 14, 2004 at 11:49:34AM -0400, Robert G. Brown wrote: > > > rgb@s02|B:1003>./Ospin > > -bash: ./Ospin: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory > > rgb@s02|B:1004> > > Pilot error. You have to install a couple of additional rpms to run > 32-bit stuff on an Opteron. The annoying thing about it is that they > have the same names as x86_64 packages, grrrr. In this case you need > glibc-*.i686.rpm. That makes it sound easier than it really is... We distribute 32 bit library RPMs with the Scyld Beowulf for AMD64 distribution. But those libraries are not just a simple duplication of the x86 32 bit RPMs. The 32 bit library RPMs must contain only the libraries, not other configuration and documentation files. The libraries must placed in the proper directories or otherwise made to have non-conflicting names. Some libraries that exist only as 32 bit versions must be placed in the LSB-standard location. And all of this get extra complicated with 3rd party compilers. We started out with an ad hoc approach, using the 32 bit RPMs, but quickly decided that it had too many exceptions to support for end users. > > ...it also refers to anecdotal accounts that numerical performance > > significantly degrades if one runs i386 code compared to recompiled > > x86_64 code. > ...I don't think that you'd come to that conclusion. It is an apples to > oranges comparison There were list postings in March that pointed to cases where the 64 bit instruction set was more compact and efficient than x86 32 bit native mode, thanks largely to more general purpose registers. The only part that was surprising was the "more compact" part. -- Donald Becker becker@scyld.com Scyld Software Scyld Beowulf cluster systems 914 Bay Ridge Road, Suite 220 www.scyld.com Annapolis MD 21403 410-990-9993 From rgb at phy.duke.edu Fri May 14 20:41:27 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Regarding wulfstat In-Reply-To: <40A4FB26.5000103@nada.kth.se> Message-ID: On Fri, 14 May 2004, Jon Tegner wrote: > Thanks a lot for info! > > After doing the verbose test I realized that the nodes that use > hostrange got the wrong port number. After specifying 7887 it worked as > expected. This was a bug (you shouldn't need to specify a port for iprange or hostrange) but I >>think<< it is fixed in 0.5.8, posted on the website(s) now. At this point I can't get it to either crash or use the wrong port number. Hopefully it is all better again. rgb > > Thanks again, much apreciated! > > /jon > > > Robert G. Brown wrote: > > >On Fri, 14 May 2004, Jon Tegner wrote: > > > > > > > >>Hi, > >> > >> > > > >I reproduce the iprange behavior here, so that is just a plain old bug > >and I'll fix it shortly, I hope. > > > >The other I'm having a hard time reproducing, although I don't have a > >lot of RH7.3 systems left to play with. > > > >While I'm trying, check that wulfstat is current (I just now posted > >0.5.7 to both brahma and personal sites). It has been substantially > >rewritten with the most important change being that it now is based on a > >library shared with both wulflogger and indirectly wulfweb (to > >facilitate further development of client tools). > > > >It also has better debugging. If you build it from e.g. the tarball for > >testing purposes, and put your failing wulfhosts specs into e.g. > >wulfhosts.tst, then running: > > > > ./wulfstat -f wulfhosts.tst -v 1 2> log > > > >should BOTH let you see wulfstat AND create a maximally verbose log of > >what it is doing. Let it run for maybe one update and quit wulfstat, > >then look at the log. It SHOULD tell you just what is failing in a > >segment near the top like: > > > >D_READHOST: Entering hostrange loop > >Validating hostname = r00, port = 0 > >D_READHOST: Validating host: > >D_READHOST: Starting name = |r00| hostip = || inetaddress = |33b60398| port = |0| > >D_READHOST: Looking up ip number for host r00. > >D_READHOST: Got host_id = 42134b58. > >D_READHOST: Setting ip_tmp to 192.168.182.50 > >D_READHOST: Reset hostip from > >D_READHOST: hostip is now 192.168.182.50 > >D_READHOST: Setting inetaddr from hostip 192.168.182.50 > >D_READHOST: Setting host r00's port = 7887 > >D_READHOST: Cleaned up host r00 (we hope). > >D_READHOST: Ending name = r00 hostip = 192.168.182.50 inetaddress = 32b6a8c0 port = 7887 > >Validating hostname = r01, port = 0 > >D_READHOST: Validating host: > >D_READHOST: Starting name = |r01| hostip = || inetaddress = |32b6a8c0| port = |0| > >D_READHOST: Looking up ip number for host r01. > >D_READHOST: Got host_id = 42134b58. > >D_READHOST: Setting ip_tmp to 192.168.182.51 > >D_READHOST: Reset hostip from > >D_READHOST: hostip is now 192.168.182.51 > >D_READHOST: Setting inetaddr from hostip 192.168.182.51 > >D_READHOST: Setting host r01's port = 7887 > >D_READHOST: Cleaned up host r01 (we hope). > >D_READHOST: Ending name = r01 hostip = 192.168.182.51 inetaddress = 33b6a8c0 port = 7887 > > > >Obviously it works here, and should clearly indicate it if the problem > >is e.g. something not resolving. > > > > > > > >>Any help/hints on this would be appreciated. And Thank You Robert for a > >>nice little program! > >> > >> > > > >You're welcome and I'm glad you are enjoying it. You should check out > >wulflogger and wulfweb (the latter is pretty heavily beta still, but > >works) if you haven't. You can see what wulfweb can do (very crudely > >still -- don't have time to really pretty this up yet) on: > > > > http://www.phy.duke.edu/resources/computing/brahma/wulfweb/wulfweb.html > > > >Our local users like this better than wulfstat in a lot of cases, > >although I don't have any of the other views done yet. I don't think it > >is as useful for admins or serious users, though -- I tend to run it at > >a time granularity of around a minute because updates bollix old/stupid > >browsers. > > > >I'll get back to you when I've fixed iprange. This is an egregious > >failure, so it is probably a simple enough bug. > > > > rgb > > > > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb at phy.duke.edu Fri May 14 21:16:50 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: Message-ID: On Fri, 14 May 2004, Donald Becker wrote: > > [[ The mailing list should now be back up and running for all > subscribers. I'll write up the long story of the Beowulf Mailing List > problems over the weekend, assuming that everything continues running. > - DJB]] > > On Fri, 14 May 2004, Greg Lindahl wrote: > > > On Fri, May 14, 2004 at 11:49:34AM -0400, Robert G. Brown wrote: > > > > > rgb@s02|B:1003>./Ospin > > > -bash: ./Ospin: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory > > > rgb@s02|B:1004> > > > > Pilot error. You have to install a couple of additional rpms to run > > 32-bit stuff on an Opteron. The annoying thing about it is that they > > have the same names as x86_64 packages, grrrr. In this case you need > > glibc-*.i686.rpm. > > That makes it sound easier than it really is... > > We distribute 32 bit library RPMs with the Scyld Beowulf for AMD64 > distribution. But those libraries are not just a simple duplication of > the x86 32 bit RPMs. > > The 32 bit library RPMs must contain only the libraries, not other > configuration and documentation files. The libraries must placed > in the proper directories or otherwise made to have non-conflicting > names. Some libraries that exist only as 32 bit versions must be placed > in the LSB-standard location. And all of this get extra complicated > with 3rd party compilers. > > We started out with an ad hoc approach, using the 32 bit RPMs, but > quickly decided that it had too many exceptions to support for end users. We just haven't messed with trying to (build and/or) install the compatibility libraries on our Opterons (yet). It's not that big a deal to do a recompile or rebuild to 64 bit code (for cleanly written applications), and we're currently using all our Opterons as relatively thin compute nodes so it is typically only one or two strictly numerical applications that need the recompile. If/when they start showing up on the desktop, perhaps this will change. I haven't looked at the latest version of the fedora64 core to see if somebody has built and contributed the libraries in a way that will kickstart/yum (sheer laziness on my part, let's see: rgb@s02|B:1001>yum list glibc\* Gathering header information file(s) from server(s) Server: Fedora Core 1 - x86_64 - base Server: Physics RHL Server: Fedora Core 1 - x86_64 - updates Finding updated packages Downloading needed headers Looking in Available Packages: Name Arch Version Repo -------------------------------------------------------------------------------- glibc i686 2.3.2-101.4 base glibc-debug x86_64 2.3.2-101.4 base glibc-devel i386 2.3.2-101.4 base glibc-profile x86_64 2.3.2-101.4 base glibc-utils x86_64 2.3.2-101.4 base Looking in Installed Packages: Name Arch Version Repo -------------------------------------------------------------------------------- glibc x86_64 2.3.2-101.4 db glibc-common x86_64 2.3.2-101.4 db glibc-devel x86_64 2.3.2-101.4 db glibc-headers x86_64 2.3.2-101.4 db glibc-kernheaders x86_64 2.4-8.36 db Hmmm, looks like there is a glibc i686. I'm home, and not eager to try a secondary glibc install from here because messing with glibc on a running system HAS been known to crash systems;-) but perhaps next week I'll experiment with this some If this fedora rpm just "works", it should be fairly simple to get it to install in kickstart (if necessary in the %post), but I'm not sure how to get it to install in yum. I'll ask Seth about it -- if this is a compatibility library rpm, it should probably have a name like glibc_i686 or glibc_32 to make it easier to install and update. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From eugen at leitl.org Sat May 15 00:31:28 2004 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] memory prefetch on Athlon64 Message-ID: <20040515073128.GL25728@leitl.org> Does anyone have benchmarks of code from http://leitl.org/docs/AMD_block_prefetch_paper.pdf on Athlon64/Opteron? If yes, can you please post the data here? Thanks, -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20040515/6dff4fae/attachment.bin From toon at moene.indiv.nluug.nl Sun May 16 12:36:47 2004 From: toon at moene.indiv.nluug.nl (Toon Moene) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Vector processing on AMD 64 In-Reply-To: <200405131903.i4DJ3gla004603@knockout.kirtland.af.mil> References: <200405131903.i4DJ3gla004603@knockout.kirtland.af.mil> Message-ID: <40A7C2CF.7050601@moene.indiv.nluug.nl> Arthur H. Edwards wrote: > AMD has advertised that their 64 bit architectures have vector > registers. This was initially very exciting because eigensolvers are not > easily paralllized for medium-sized problems, but vectorize very well. Perhaps you're interested in talks like Dorit Naishlos': "Autovectorization in GCC". at the GCC summit (http://www.gccsummit.org). -- Toon Moene - mailto:toon@moene.indiv.nluug.nl - phoneto: +31 346 214290 Saturnushof 14, 3738 XG Maartensdijk, The Netherlands Maintainer, GNU Fortran 77: http://gcc.gnu.org/onlinedocs/g77_news.html GNU Fortran 95: http://gcc.gnu.org/fortran/ (under construction) From lindahl at pathscale.com Sun May 16 19:59:13 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] memory prefetch on Athlon64 In-Reply-To: <20040515073128.GL25728@leitl.org> References: <20040515073128.GL25728@leitl.org> Message-ID: <20040517025913.GA2830@greglaptop.greghome.keyresearch.com> > Does anyone have benchmarks of code from > http://leitl.org/docs/AMD_block_prefetch_paper.pdf > on Athlon64/Opteron? Sections 5.5 and 5.11 of the AMD Opteron Software Optimization Guide are a modern version of this paper. -- greg From robl at mcs.anl.gov Mon May 17 07:08:13 2004 From: robl at mcs.anl.gov (Robert Latham) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] 64bit comparisons In-Reply-To: <20040514184821.GF31642@cse.ucdavis.edu> References: <200405141538.i4EFcQpd015889@bluewest.scyld.com> <200405141644.i4EGi1Aq023213@marvin.ibest.uidaho.edu> <20040514184821.GF31642@cse.ucdavis.edu> Message-ID: <20040517140810.GW1940@mcs.anl.gov> On Fri, May 14, 2004 at 11:48:21AM -0700, Bill Broadley wrote: > Personally I've had problems, I need to spend more time resolving them, > things like: > * Need to tweak /etc/rc to allow Mpich to use shared memory > * Latency between two mpich processes on the same node is 10-20 times the > linux latency. I've yet to try LAM. > * Differences in semaphores requires a rewrite for some linux code I had > * Difference in the IBM fortran compiler required a rewrite compared to code > that ran on Intel's, portland group's, and GNU's fortran compiler. This might sound like a troll, but I'm serious: why not run linux on the G5 Xserves? That would address point 1, 2, and 3 and you would have a much better underlying kernel. The Darwin kernel is a terrible performance dog, and linux outperforms it in context switches, file system access, page faults and just about every other kernel-specific metric by an order of magnatude. It's a cluster, so you're not using the OS X GUI. You've probably already got a cluster management infrastrucutre for your linux clusters, so you won't have an oddball Darwin cluster screwing things up. I don't know how possible it is to get 3rd party software for linux-ppc. Aside from that one issue, linux on G5 clusters seems to make a lot of sense from a portability, performance, and administration perspective. ==rob -- Rob Latham Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF Argonne National Labs, IL USA B29D F333 664A 4280 315B From daniel at labtie.mmt.upc.es Mon May 17 07:01:50 2004 From: daniel at labtie.mmt.upc.es (Daniel Fernandez) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: <40A4F322.7090803@umassmed.edu> References: <40A4F322.7090803@umassmed.edu> Message-ID: <1084802510.7573.20.camel@qeldroma.cttc.org> Those results are a bit surprising, it seems that the performance gains from 32bit (Athlon) to 64bit (Opteron) is nearly 0.5% . Considering that is using only single precision and x87 operations run equally fast at same clockspeed on both processors ( not at all ) it's not bandwith limited ... ? How about double precision performance results ? -- Daniel Fernandez Heat And Mass Transfer Center - CTTC www.cttc.upc.edu c/ Colom n?11 UPC Campus Industrials Terrassa , Edifici TR4 On Fri, 2004-05-14 at 18:26, Karl Bellve wrote: > > > We have a project that I have timed on various CPUs, and compilers. > It makes heavy use of FFTW (single precision). > > Running native i386 code does very well for us on the Opteron, unlike > the Itanium 2. > > > Itanium 2 1.0 GHz on Red Hat Linux Advanced Workstation release 2.1AW > 234 seconds (IA64 code and ECC, -fno-alias -IPF_fma -Ob2 -ipo > -restrict) > 1515 seconds (i386 code and GCC) > > AMD Opteron 1.4 GHz on United Linux 1.0 > 189 seconds (gcc, x86_64 code) > 221 seconds (gcc, i386 code) > > Athlon 1.5 GHz (AMD Athlon(tm) MP Processor 1800+) on Redhat 8.0 > 190 seconds (gcc, i386 code) > > Intel Xeon 1.7 GHz on Redhat 8.0 > 267 seconds (gcc, i386 code) > > Sun Intel Xeon 2.8 GHz on Redhat 7.3 > 150 seconds (gcc, i386 code) > > Alpha 667 GHz > 430 seconds (Compaq compiler for both, optimized code) > > Intel 2.2Ghz Xeon on RH 7.3 > 208 seconds (gcc, i386 code) > > -- > Cheers, > > > > Karl Bellve > > ______________________________________________________________________ > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From daniel at labtie.mmt.upc.es Mon May 17 11:27:15 2004 From: daniel at labtie.mmt.upc.es (Daniel Fernandez) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Athlon64 / Opteron test In-Reply-To: References: Message-ID: <1084818435.7573.72.camel@qeldroma.cttc.org> What kind of applications are you running ? Just only to get a reference regarding our "theorical" future performance running our code on an Opteron. As a side note there's the power that the Opteron processor draws from the power supply , supposedly less than an Athlon at same clockspeed. I think a standard ATX 300W one should suffice for one Opteron, memory and a small harddisk. ? Are you using any other kind of power supply for AMD64 platforms ? Thanks. -- Daniel Fernandez Heat And Mass Transfer Center - CTTC www.cttc.upc.edu c/ Colom n?11 UPC Campus Industrials Terrassa , Edifici TR4 On Fri, 2004-05-14 at 02:06, Robert G. Brown wrote: > On Thu, 13 May 2004, Daniel Fernandez wrote: > > > Hi, > > > > We're studying the possibility of installing a 64 bit testing node. > > > > The main purpose is about getting impressions related to the performance > > increase we would obtain in our particular scenario, computational fluid > > dynamics. > > > > In order to do the test, we have no doubt about the OS: Red Hat > > Enterprise 3, but we are a bit confused about the harware of choice: > > > > Athlon64 > > Opteron > > > > As far as we know, Opteron has two main differences: > > > > - A wider memory interface (128 bit in front of 64) > > - A larger L2 cache memory (1 Mb) > > > > Before doing any test, the questions are: > > > > 1) > > > > Which is the theoretical maximum performance gain using full 64 bit > > architecture in front of 32 bit, taking into account that our > > computations are done in double precision floating point using really > > big matrices? > > I don't know what theoretical maximum gain means -- something with > multiplying out clocks, pipeline form factors, datapath widths, and > assuming perfect state? Never happens, more or less useless number. > > I do have a moderate amount of faith in what process-specific > microbenchmarks like stream and lmbench tell one (not as much as one > might think, but some useful data points). I also have a LOT of faith > in using your own application as a benchmark on a loaner machine (or > machines). So much so that this is one of the points of focus of my > next couple of Cluster World columns -- the importance of understanding > your application and the deep significance of "YMMV" when asking other > people what to buy. > > If stream or stream-like microbenchmarks would be of any use to you, I > have Opterons but not Athlon64's handy. Maybe somebody else can do the > other for you. Remember, though, that even these results may vary > significantly with compiler, with link libraries, with WHICH Opteron > motherboard you get and how memory is loaded onto the motherboard. It > is very, very difficult to get portable numbers without a prototyping > machine that is "like" the architecture you are considering in very > specific form (ideally, identical). > > > 2) > > > > Is it nowadays the 64 bit solution using Linux ready for production? > > If this is the case, which problems may we have to deal with in order to > > compile and run our code in a full 64 bit environment? > > I'm using dual Opterons (Penguin Altus 1000E's) running Fedora right now > without any major problems. Your two problems are likely to be > rebuilding things that aren't for some reason in the Fedora core and > already built for the Opteron. So far I haven't encountered any major > problems compiling gcc-based code linked to the GSL and other > useful/standard libraries. YMMV, but a lot of people have these boxes > now and together they form a powerful force pushing for full and clean > distros, so even if you find something that doesn't work it likely WILL > work and at worst you might have to help make it work in the best of > open source tradition, if it is in your key work pathway. > > We're likely to buy nothing else for cluster nodes in the medium run. > So far they are by far the cost-benefit winner. > > > 3) > > > > Which is the most mature solution: AMD Opteron or Intel Itanium? > > Did you mean mature or moribund;-)? > > I'm only half kidding. Itanium is dead as a doornail as technology goes > -- overpriced, underperforming, incompatible. Intel is migrating to a > (more or less) Opteron compatible 64 bit processor as fast as they can > get there, as Major Software Companies (MSC) have announced that they > aren't going to do major ports to new chips with new machine languages > and compilers anymore if they can possibly avoid it. If Intel dropped > the price of an Itanium to slightly LESS than that of an Opteron, I > think they'd still have trouble maintaining a market, because Opterons > are relatively easy to port to and will in principle run i386 code > (badly, of course) native. Sometimes. I haven't had a lot of luck with > it, of course, because you can't mix i386 code and 64 bit DLLs and we > installed a 64 bit version of the OS from the start, but theoretically > it will work. > > The good news is that Opterons are surprisingly fast for MY applications > for their relatively pokey CPU clocks, and some benchmarks show that > they can be really quite fast indeed for memory intensive applications > relative to e.g. an Athlon or P4 clock. They also run much cooler than > regular Athlons (again for my application). I draw ballpark of 185 > watts loaded (dual CPU Opteron) vs 230 Watts or so loaded (dual CPU > Athlon) running more or less the same code. > > rgb From rgb at phy.duke.edu Tue May 18 07:51:00 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] wulfstat, wulflogger fix, new features Message-ID: Karl Bellve posted a bug in wulflogger that caused it to miss connecting to the first host in the wulfhosts list until the second pass. He also requested a feature that would let wulflogger execute only a single time and then exit so that it could be used in e.g. a cron script to graze for downed hosts in a cluster easily. I found the bug (a legacy from wulfstat where I closed stdin pre-curses, which caused the first port SUCCESSFULLY returned from socket() to be returned as 0 (being reused) which actually doesn't work. This is a bug in socket() I personally would say, but either way, when I eliminated the close statement wulflogger now connects to the first host with no problem first try. I implemented the request by adding a -c count flag to both wulflogger and wulfstat. -c 1 is the behavior requested, but somebody may have use for the greater flexibility permitted by it being a variable. I also updated both Usage and the man page for both applications, in the case of wulflogger including an example fragment that might go into a cron job to graze for down hosts and in the both cases adding a short section on debugging (I've written the code to be tremendously self-debugging to make it relatively easy to maintain or augment). Still to implement: a) I want to add a ping to the connection engine to precede the xmlsysd connection attempt. ping actually is a bit of a pain -- the usual iputils implementation requires suid root. nmap, however, has three or four distinct ways of "pinging" that don't require root privileges, and eventually I'll try stealing one although the code is a lot more complex than I'd like for a simple task. Anybody with a SHORT/SIMPLE version of userspace (e.g. ack) ping in C should feel free to let me know where to find it. b) I need to do something about tracking running jobs in wulflogger, and figure out a better display for them in wulfstat. c) I still have fantasies of writing gwulfstat on top of gtk. This could be a very cool application. d) And wulfweb needs love as well, although that is straightforward web programming at this point -- wulflogger is the real tool involved. Anyway, those of you who are using it, enjoy. Those who aren't, consider giving xmlsysd/wulf[stat,logger,web] a try. It is a fairly simple way to monitor an entire cluster (tested with order <100 hosts, don't know how or if it scales to ~1000) in a lightweight fashion with adjustable time granularity. Those of you who are also LAN managers might consider using it to monitor your LAN status as well. The default wulfstat/wulflogger display is something like: # Name Status Timestamp load1 load5 load15 rx byts tx byts si so pi po ctxt intr users lilith up 1084891476.44 0.01 0.04 0.01 9761 7171 0 0 0 22 148 170 asixteencharname up 1084891476.44 0.01 0.04 0.01 9761 7171 0 0 0 22 148 170 lucifer up 1084891610.24 0.00 0.02 0.00 226 709 0 0 0 9 135 104 uriel up 1084887238.42 0.00 0.00 0.00 1030 1672 0 0 0 5 36 114 caine down eve up 1084888284.75 0.00 0.00 0.00 685 1168 0 0 0 11 21 109 serpent up 1084877687.98 0.00 0.00 0.00 1116 1707 0 0 0 6 41 187 tyrial up 1084891762.44 0.00 0.00 0.00 3146 3064 0 0 0 9 208 218 abel down archangel up 1084888715.71 0.00 0.00 0.00 119 1376 0 0 0 30 28 105 (used to look at my home cluster, with one machine turned off and one machine down awaiting a reinstall.) There is a display that only looks at load, a display only for network traffic, one for network usage, even one that tells you uptime and duty cycle (cpu cycles used/cpu cycles available) from the last boot. All GPL v2b... http://www.phy.duke.edu/~rgb/Beowulf/beowulf.php I suggest rebuilding the source rpm or working from tarball, although people running RH 9 can probably install the binary rpms without disaster. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb at phy.duke.edu Tue May 18 08:08:27 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] wulfware list Message-ID: To get some of the wulfstat/xmlsysd traffic off of this list, I've set up a mailman list here: http://lists.phy.duke.edu/mailman/listinfo/wulfware Users (current or potential) are encouraged to subscribe and ask questions or report bugs there. I'll probably report only major changes or announcements on the beowulf list proper in the future. Anybody who wants to work on developing new clients or adding things on to existing clients is ESPECIALLY encouraged to join... the toolset could do a lot more than it does so far. With the right interface, one could pretty much run a cluster from it, likely including a simple scheduler and more, as xmlsysd can provide all the information required by a scheduler. However, I've got pretty strict limits on the time I can devote to it. ;-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From eugen at leitl.org Wed May 19 08:01:44 2004 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from jle@world.std.com) Message-ID: <20040519150144.GD16732@leitl.org> ----- Forwarded message from Joe M Leonard ----- From: Joe M Leonard Date: Wed, 19 May 2004 08:06:49 -0400 (EDT) To: chemistry@ccl.net Subject: CCL:Question regarding Mac G5 performance Folks, has anybody run QM or MD benchmarks on a dual-processor 2GHz G5 that they care to share? I'm particularly interested in how they stack up to the newest/newer Opteron and Pentium4 machines. If you've worked with the newer Mac's, (a) is it possible to get good, useful 2x processor performance for double-precision code? (b) what compilers are required to get (a)? (c) is Altivec useful for double-precision code, or merely for int/float work? A quick glance at apple.com suggests these machines go for about $3200 USD, which isn't that different from numbers gained from a quick glance at dell.com. I know IBM chips were number-crunchers in the good old days, and I assume that's still the case now (which is why I'm looking to CCL for guidance :-). Thanks in advance! Joe Leonard jle@theworld.com -= This is automatically added to each message by the mailing script =- To send e-mail to subscribers of CCL put the string CCL: on your Subject: line and send your message to: CHEMISTRY@ccl.net Send your subscription/unsubscription requests to: CHEMISTRY-REQUEST@ccl.net HOME Page: http://www.ccl.net | Jobs Page: http://www.ccl.net/jobs If your mail is bouncing from CCL.NET domain send it to the maintainer: Jan Labanowski, jlabanow@nd.edu (read about it on CCL Home Page) -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ----- End forwarded message ----- -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20040519/319b90d3/attachment.bin From eugen at leitl.org Wed May 19 13:02:04 2004 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from mmccallum@pacific.edu) Message-ID: <20040519200204.GO16732@leitl.org> ----- Forwarded message from Mike McCallum ----- From: Mike McCallum Date: Wed, 19 May 2004 10:49:14 -0700 To: Joe M Leonard Cc: chemistry@ccl.net Subject: CCL:Question regarding Mac G5 performance X-Mailer: Apple Mail (2.613) On May 19, 2004, at 05:06, Joe M Leonard wrote: >Folks, has anybody run QM or MD benchmarks on a dual-processor >2GHz G5 that they care to share? I'm particularly interested >in how they stack up to the newest/newer Opteron and Pentium4 >machines. If you've worked with the newer Mac's, (a) is it >possible to get good, useful 2x processor performance for >double-precision code? (b) what compilers are required to get >(a)? (c) is Altivec useful for double-precision code, or >merely for int/float work? > I had done some comparisons between price/performance, and I found dual G5s to be at or near the best in price/performance, especially if things are recompiled with the IBM compilers (between 8-10 % speed increase over pre-compiled (with apple gcc) versions of NAMD2, and using standard gcc for charmm). I would expect that things like gaussian03 run very well (I believe gaussian uses the IBM compilers for macOS). For MD, the speedup seems to be due to the on-chip square root eval. The built-in Gigabit enet is attractive, also, as charmm and NAMD scale very well with gigabit, and it makes myrinet less price-effective (when used on any platform, wintel included, see http://biobos.nih.gov/apps/charmm/charmmdoc/Bench/c30b1.html for example). I decided that dual G5 xserve cluster nodes with gigabit switches were much more cost-effective for me than any other processor, especially any high-bandwidth specialty comm method (apple's gigabit has a pretty low latency also). Additional considerations for us were the BSD environment which is more secure than windows, and the OS is arguably more stable and supported than a linux (though you can pretend to be linux with the fink layer). Being much more familiar and happy with OS X than windows or linux sealed it. It is my impression that opterons, PIVs, G5s all have their advantages, and disadvantages, so your own comfort level may be a non-trivial consideration. I don't feel there is any net performance penalty, and often a benefit from using G5s. >A quick glance at apple.com suggests these machines go for about >$3200 USD, which isn't that different from numbers gained from a >quick glance at dell.com. I know IBM chips were number-crunchers >in the good old days, and I assume that's still the case now (which >is why I'm looking to CCL for guidance :-). I got my current dual 2.0 G5 for $2200. This is acadmic pricing, and I'm sure your rep would give you a similar price, especially now that a new processor version is close to (?) production. I was looking at at least $2500 for a similarly configured opteron node (w/o gigabit, though). Hope this helps, Mike > >Thanks in advance! > >Joe Leonard >jle@theworld.com > > >To send e-mail to subscribers of CCL put the string CCL: on your >Subject: line > >Send your subscription/unsubscription requests to: >CHEMISTRY-REQUEST@ccl.net >HOME Page: http://www.ccl.net | Jobs Page: http://www.ccl.net/jobs > >-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- >+-+ > > > > > > -- C. Michael McCallum http://chem.cop.uop.edu/cmmccallum.html Associate Professor Department of Chemistry, UOP mmccallum .at. pacific .dot. edu (209) 946-2636 v / (209) 946-2607 fax -= This is automatically added to each message by the mailing script =- To send e-mail to subscribers of CCL put the string CCL: on your Subject: line and send your message to: CHEMISTRY@ccl.net Send your subscription/unsubscription requests to: CHEMISTRY-REQUEST@ccl.net HOME Page: http://www.ccl.net | Jobs Page: http://www.ccl.net/jobs If your mail is bouncing from CCL.NET domain send it to the maintainer: Jan Labanowski, jlabanow@nd.edu (read about it on CCL Home Page) -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ----- End forwarded message ----- -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20040519/561263ef/attachment.bin From bill at cse.ucdavis.edu Wed May 19 15:13:32 2004 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from mmccallum@pacific.edu) In-Reply-To: <20040519200204.GO16732@leitl.org> References: <20040519200204.GO16732@leitl.org> Message-ID: <20040519221332.GA29085@cse.ucdavis.edu> > I had done some comparisons between price/performance, and I found dual > G5s to be at or near the best in price/performance, especially if > things are recompiled with the IBM compilers (between 8-10 % speed > increase over pre-compiled (with apple gcc) versions of NAMD2, and > using standard gcc for charmm). I would expect that things like > gaussian03 run very well (I believe gaussian uses the IBM compilers for > macOS). For MD, the speedup seems to be due to the on-chip square root > eval. Any actual numbers would be very useful, ideally with the compilers, compiler options, motherboard, and similar. Did you compare 1 job per node vs 1 job per node? Or 2 jobs per node vs 2 jobs per node? 4 or 8 dimms? Which kind of memory PC2700? PC3200? > The built-in Gigabit enet is attractive, also, as charmm and NAMD scale > very well with gigabit, and it makes myrinet less price-effective (when > used on any platform, wintel included, see > http://biobos.nih.gov/apps/charmm/charmmdoc/Bench/c30b1.html for > example). I decided that dual G5 xserve cluster nodes with gigabit I come to a different conclusion based on those graphs. In the first graph myrinet improves by a factor of 2.6 (250 -> 95 seconds) from 2 processors to 8, where gige improves by only 20% (255 -> 210). In the second graph gigE gets SLOWER from 2 to 8 processors. Do you think in either case the 8 node (let alone 16) gige cluster would have better price/performance then a myrinet cluster? Seems like for applications like shown on that page you shouldn't really bother with a cluster over 2 nodes with gigE, not many people would be willing to pay a factor of 4 more for 20% or even negative scaling. > switches were much more cost-effective for me than any other processor, Cost effective = price/performance? Can you make any numbers available? > especially any high-bandwidth specialty comm method (apple's gigabit > has a pretty low latency also). Oh? Can you share the apple gigabit numbers? What is "pretty low"? > Additional considerations for us were the BSD environment which is more > secure than windows, and the OS is arguably more stable and supported I'd agree with more stable and supported for use as a desktop, I'd disagree with stable and supported as computational node. OSX is the new player on the block in this space. Do you really think you would get a good response from calling apple's tech support line when the scheduler or network stack isn't performing to your performance expectations? Certainly a very reasonable thing. > It is my impression that opterons, PIVs, G5s all have their advantages, Agreed, thus the value in sharing performance actual results for specific codes in specific environments. -- Bill Broadley Computational Science and Engineering UC Davis From moor007 at bellsouth.net Wed May 19 13:28:09 2004 From: moor007 at bellsouth.net (Tim Moore) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] Re: Question regarding Mac G5 performance References: <200405191900.i4JJ06vQ029798@bluewest.scyld.com> Message-ID: <004601c43ddf$cd0ba3f0$6f01a8c0@timlaptop> I posted the other day about Opteron vs. Itanium...the CFD code we use is being ported to the MAC and will be tested in the next few weeks. I am sorry it does not answer your question...but will be able to post results later. ----- Original Message ----- From: To: Sent: Wednesday, May 19, 2004 3:00 PM Subject: Beowulf Digest, Vol 3, Issue 13 > Send Beowulf mailing list submissions to > beowulf@beowulf.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://www.beowulf.org/mailman/listinfo/beowulf > or, via email, send a message with subject or body 'help' to > beowulf-request@beowulf.org > > You can reach the person managing the list at > beowulf-owner@beowulf.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Beowulf digest..." > > > Today's Topics: > > 1. CCL:Question regarding Mac G5 performance (fwd from > jle@world.std.com) (Eugen Leitl) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 19 May 2004 17:01:44 +0200 > From: Eugen Leitl > Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from > jle@world.std.com) > To: Beowulf@beowulf.org > Message-ID: <20040519150144.GD16732@leitl.org> > Content-Type: text/plain; charset="us-ascii" > > ----- Forwarded message from Joe M Leonard ----- > > From: Joe M Leonard > Date: Wed, 19 May 2004 08:06:49 -0400 (EDT) > To: chemistry@ccl.net > Subject: CCL:Question regarding Mac G5 performance > > Folks, has anybody run QM or MD benchmarks on a dual-processor > 2GHz G5 that they care to share? I'm particularly interested > in how they stack up to the newest/newer Opteron and Pentium4 > machines. If you've worked with the newer Mac's, (a) is it > possible to get good, useful 2x processor performance for > double-precision code? (b) what compilers are required to get > (a)? (c) is Altivec useful for double-precision code, or > merely for int/float work? > > A quick glance at apple.com suggests these machines go for about > $3200 USD, which isn't that different from numbers gained from a > quick glance at dell.com. I know IBM chips were number-crunchers > in the good old days, and I assume that's still the case now (which > is why I'm looking to CCL for guidance :-). > > Thanks in advance! > > Joe Leonard > jle@theworld.com > > > -= This is automatically added to each message by the mailing script =- > To send e-mail to subscribers of CCL put the string CCL: on your Subject: line > and send your message to: CHEMISTRY@ccl.net > > Send your subscription/unsubscription requests to: CHEMISTRY-REQUEST@ccl.net > HOME Page: http://www.ccl.net | Jobs Page: http://www.ccl.net/jobs > > If your mail is bouncing from CCL.NET domain send it to the maintainer: > Jan Labanowski, jlabanow@nd.edu (read about it on CCL Home Page) > -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > > > > > ----- End forwarded message ----- > -- > Eugen* Leitl leitl > ______________________________________________________________ > ICBM: 48.07078, 11.61144 http://www.leitl.org > 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE > http://moleculardevices.org http://nanomachines.net > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: not available > Type: application/pgp-signature > Size: 198 bytes > Desc: not available > Url : http://www.scyld.com/pipermail/beowulf/attachments/20040519/319b90d3/attachment-0001.bin > > ------------------------------ > > _______________________________________________ > Beowulf mailing list > Beowulf@beowulf.org > http://www.beowulf.org/mailman/listinfo/beowulf > > > End of Beowulf Digest, Vol 3, Issue 13 > ************************************** > From mathiasbrito at yahoo.com.br Thu May 20 04:49:24 2004 From: mathiasbrito at yahoo.com.br (=?iso-8859-1?q?Mathias=20Brito?=) Date: Wed Nov 25 01:03:09 2009 Subject: [Beowulf] About Cluster Performance... Message-ID: <20040520114924.92225.qmail@web12208.mail.yahoo.com> hello folks, Well, i would like to know how to organize the nodes to abtain the best performance of my cluster, for now i'm using a star topology to my 16 nodes cluster, i heard something about using a 4x4 grid instead 1x16, is it better? why? And i also would like to know a way to calculate(predict) the performance of a cluster, for example, i have a 16 nodes(using fast ethernet) cluster and i`m not sure about the maximum performace i should get, to compare with the HPL benchmark results. Thanks Mathias Brito ===== Mathias Brito Universidade Estadual de Santa Cruz - UESC Departamento de Ci?ncias Exatas e Tecnol?gicas Estudante do Curso de Ci?ncia da Computa??o ______________________________________________________________________ Yahoo! Messenger - Fale com seus amigos online. Instale agora! http://br.download.yahoo.com/messenger/ From hunting at ix.netcom.com Wed May 19 23:05:12 2004 From: hunting at ix.netcom.com (Michael Huntingdon) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from mmccallum@pacific.edu) In-Reply-To: <20040519221332.GA29085@cse.ucdavis.edu> References: <20040519200204.GO16732@leitl.org> <20040519221332.GA29085@cse.ucdavis.edu> Message-ID: <6.0.3.0.2.20040519222707.05023d08@popd.ix.netcom.com> I've spent some time sifting through the attached numbers. Though not each lends itself to hp Itanium 2, there appears to be a very balanced trend. For clusters, scale down the hp rx2600 to the rx1600 dual CPU with fewer PCI-X slots, comparable performance, HP-UX/linux/OVMS, in box upgrades, excellent services, and education price under $3,000. I'm at a loss as to why there is not a great deal more conversation around these systems. hptc clusters should lend themselves to easy additions and upgrades. An investment in Pentuim, SPARC, Athlon, or Opteron yesterday, should easily include G5, Itanuim 2 or futures toward cluster enhancements. From the attachments, it looks to me as though the Itanium 2 numbers are (at least) compelling, with a long term commitment to the hardware technology as well as the operating systems. Cheers Michael At 03:13 PM 5/19/2004, Bill Broadley wrote: > > I had done some comparisons between price/performance, and I found dual > > G5s to be at or near the best in price/performance, especially if > > things are recompiled with the IBM compilers (between 8-10 % speed > > increase over pre-compiled (with apple gcc) versions of NAMD2, and > > using standard gcc for charmm). I would expect that things like > > gaussian03 run very well (I believe gaussian uses the IBM compilers for > > macOS). For MD, the speedup seems to be due to the on-chip square root > > eval. > >Any actual numbers would be very useful, ideally with the compilers, >compiler options, motherboard, and similar. Did you compare 1 job per >node vs 1 job per node? Or 2 jobs per node vs 2 jobs per node? 4 or 8 >dimms? Which kind of memory PC2700? PC3200? > > > The built-in Gigabit enet is attractive, also, as charmm and NAMD scale > > very well with gigabit, and it makes myrinet less price-effective (when > > used on any platform, wintel included, see > > http://biobos.nih.gov/apps/charmm/charmmdoc/Bench/c30b1.html for > > example). I decided that dual G5 xserve cluster nodes with gigabit > >I come to a different conclusion based on those graphs. In the first >graph myrinet improves by a factor of 2.6 (250 -> 95 seconds) from 2 >processors to 8, where gige improves by only 20% (255 -> 210). In the >second graph gigE gets SLOWER from 2 to 8 processors. Do you think in >either case the 8 node (let alone 16) gige cluster would have better >price/performance then a myrinet cluster? > >Seems like for applications like shown on that page you shouldn't >really bother with a cluster over 2 nodes with gigE, not many people >would be willing to pay a factor of 4 more for 20% or even negative >scaling. > > > switches were much more cost-effective for me than any other processor, > >Cost effective = price/performance? Can you make any numbers available? > > > especially any high-bandwidth specialty comm method (apple's gigabit > > has a pretty low latency also). > >Oh? Can you share the apple gigabit numbers? What is "pretty low"? > > > Additional considerations for us were the BSD environment which is more > > secure than windows, and the OS is arguably more stable and supported > >I'd agree with more stable and supported for use as a desktop, I'd >disagree with stable and supported as computational node. OSX is the >new player on the block in this space. Do you really think you would >get a good response from calling apple's tech support line when the >scheduler or network stack isn't performing to your performance >expectations? > >Certainly a very reasonable thing. > > > It is my impression that opterons, PIVs, G5s all have their advantages, > >Agreed, thus the value in sharing performance actual results for >specific codes in specific environments. > >-- >Bill Broadley >Computational Science and Engineering >UC Davis >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit >http://www.beowulf.org/mailman/listinfo/beowulf -------------- next part -------------- A non-text attachment was scrubbed... Name: Benchmark.daresbury.pdf Type: application/pdf Size: 337664 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20040519/5461552c/Benchmark.daresbury.pdf -------------- next part -------------- A non-text attachment was scrubbed... Name: compchem.pdf Type: application/pdf Size: 1454990 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20040519/5461552c/compchem.pdf -------------- next part -------------- ********************************************************************* Systems Performance Consultants Michael Huntingdon Higher Education Technology Office (408) 294-6811 131-A Stony Circle, Suite 500 Cell (707) 478-0226 Santa Rosa, CA 95401 fax (707) 577-7419 Web: <http://www.spcnet.com> hunting@ix.netcom.com ********************************************************************* From rgb at phy.duke.edu Thu May 20 11:16:58 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] About Cluster Performance... In-Reply-To: <20040520114924.92225.qmail@web12208.mail.yahoo.com> Message-ID: On Thu, 20 May 2004, Mathias Brito wrote: > hello folks, > > Well, i would like to know how to organize the nodes > to abtain the best performance of my cluster, for now Attain the best performance doing what? The best (and cheapest) "general purpose" design is to put your 16 nodes on a fast ethernet switch as you are doing. It works fine for moderately coarse grained tasks and is inexpensive to upgrade to gigabit ethernet now or later for improved bandwidth if need be. However, decisions concerning network and topology tend to be driven by the application (set) you desire to run. If you are doing Monte Carlo (as I am) in an embarrassingly parallel configuration then don't change. Who cares how "fast" you run some benchmark if that benchmark has nothing to do with the structure of YOUR code? If you are doing something medium to fine grained parallel, where there are quite a few network communications that have to take place for your computation to advance a step, then consider a faster network. If the communications tend to be sending relatively few BIG messages between the nodes, gigabit ethernet is a reasonable (and still cheap) choice. If they tend to be sending lots of itty bitty messages between the nodes all the time, then you will need to look at the really expensive, racehorse networks. Myrinet and SCI are both expensive (quite possibly more expensive than the nodes themselves, per node) but offer latency on the order of microseconds, where ethernet latencies tend to be on the tens to hundreds of microseconds. They also offer quite high peak bandwidths. I honestly don't know a lot about firewire -- perhaps somebody on the list could summarize experiences with a firewire connected network. > i'm using a star topology to my 16 nodes cluster, i > heard something about using a 4x4 grid instead 1x16, > is it better? why? And i also would like to know a way > to calculate(predict) the performance of a cluster, > for example, i have a 16 nodes(using fast ethernet) It is "in principle" possible to predict parallel performance, but it isn't easy and to have a good shot at getting it right requires extensive study of your hardware resources, your operating system and libraries, and your application. By "study" I mean both learning all you can about it (and about parallel task scaling in general) from real textbooks, manuals, and informational resources and making real world measurements. For example, you'll need to know things like (and this is by no means a complete list): a) Raw CPU clock, and how fast the processor does all the things it can do. The structure of the processor -- how and if it is pipelined, number and kind of registers -- it all matters. b) Size and structure and latencies associated with CPU registers, L1 and L2 (and L3, if any) cache. c) Size and structure and (clock and) latency of main memory. Structure includes a lot -- datapath widths, how the memory is interfaced to e.g. peripherals and one or more CPUs. d) The bus(es) on the motherboard, and how they are interfaced with the CPU and memory and each other. Again, things like clock, datapath width and so forth are important, but there is more. e) The devices attached to the bus(es). Obviously and especially the network hardware, but also disk hardware and even humble slow peripheral hardware can matter, as its mere existence CAN (has in the past) significantly alter the performance of things you care about. a') Operating system. Kernels are different -- schedulers, interrupt mechanisms, locking mechanisms, device drivers -- all of it goes into how well and stably a system will perform overall. b') Compilers. Just changing compilers can make a huge difference in the speed of your code. Some compilers support SSE2 instructions, and some code speeds up if it uses them. Others (in both cases) don't. c') Libraries. You don't really write your whole program (any program) -- you write PART of it, and rely on the kindness and skill of strangers for the rest of it, in the form of the many libraries you link to. Whether it is core libraries such as libc and libm or esoteric/advanced libraries like libgsl, libxml2, libpvm -- if your code uses these libraries your performance will be affected by their quality (which can vary with all of the above). a") Your application. The ATLAS project is a perfect demonstration of how naive implementation of even the most mundane tasks (such as vector and matrix operations) can fail to achieve the speed available to them by factors of two and three. ATLAS (automatically tuned linear algebra system) provides linear algebra libraries that are custom built for particular systems (a-e above) and operating system environments (a'-c') above and does things like change algorithms altogether and alter the blocking of the code according to the size and MEASURED speed of the CPU and its various layers of cache. There are whole books devoted to parallel programming and algorithms, and performance on any given task depends on the algorithms chosen. b") Your application. Yes, it bears repeating. You cannot predict "performance" of piece of hardware with a given operating system and compiler and set of libraries. You can only predict performance of an application. So you have to fully understand it, in particular where and how it is bottlenecked. c") Your application yet again. The more you understand about it and how it uses the hardware resources available to it, the better your chances of both predicting its performance and optimizing that performance on any given box. Profiling tools, debuggers, and code instrumentation are all your friends. Most people I know (even the ones that know enough that they COULD make a creditable stab at all of the points above) tend to NOT try to predict performance per se -- they prefer to measure it. There are a very few microbenchmarks that can be useful to people who are bottlenecked in one of the "standard" ways -- at the network layer, or memory layer, or raw CPU layer, for example. These people care about netpipe numbers, they care about stream results, they care about lmbench or cpu_rate numbers. Most applications are a complicated mix of integer and floating point code where "improving" any one thing may not affect the overall performance at all! For years I ran my Monte Carlo code and discovered that within a processor family it scaled almost perfectly with CPU clock and nothing else. Fast expensive memory or slow blodgy memory, didn't matter. Big CPU L2 cache or little cache, didn't matter. Fast FSB or slow FSB, fast network or slow network -- all about the same, but double CPU clock and it finishes twice as fast. Not all jobs are like that, and scaling BETWEEN processor families (e.g. AMD and Intel) can be completely different even for this job. That's why everybody will conclude any answer about speed, benchmarks, performance with --- YMMV (Your Mileage May Vary). Don't trust benchmarks, don't trust what people tell you about system performance. Don't compute it if you can avoid it. Measure it. It's the only sure way, and even MEASURING it you have work to do with e.g. compiler and library selection, and algorithm optimization. rgb > cluster and i`m not sure about the maximum performace > i should get, to compare with the HPL benchmark > results. > > Thanks > Mathias Brito > > > > ===== > Mathias Brito > Universidade Estadual de Santa Cruz - UESC > Departamento de Ci?ncias Exatas e Tecnol?gicas > Estudante do Curso de Ci?ncia da Computa??o > > ______________________________________________________________________ > > Yahoo! Messenger - Fale com seus amigos online. Instale agora! > http://br.download.yahoo.com/messenger/ > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From hunting at ix.netcom.com Thu May 20 17:26:13 2004 From: hunting at ix.netcom.com (Michael Huntingdon) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from mmccallum@pacific.edu) In-Reply-To: <1085084616.2581.44.camel@protein.scalableinformatics.com> References: <20040519200204.GO16732@leitl.org> <20040519221332.GA29085@cse.ucdavis.edu> <6.0.3.0.2.20040519222707.05023d08@popd.ix.netcom.com> <1085084616.2581.44.camel@protein.scalableinformatics.com> Message-ID: <6.0.3.0.2.20040520141728.05023d08@popd.ix.netcom.com> At 01:23 PM 5/20/2004, Joe Landman wrote: >On Thu, 2004-05-20 at 02:05, Michael Huntingdon wrote: > > I've spent some time sifting through the attached numbers. Though not each > >Any paper that starts out praising spec as a "good" predictive benchmark >is suspect. Benchmarking is difficult to do right, in large part >because of deceptively simple scoring functions (time, frequency (not of >the CPU, but number of iterations per unit time), ...). Although SPEC was included among others, I didn't see where Martyn Guest "praised" SPEC. >Further, looking these over, I did not see much of a discussion (though >it is implied by the use of certain compilers) of the effects of things >like SSE2 in the P4, memory alignment, 32/64 bit >compilation/optimization, use of tuned libraries where available... >Given the sheer number of machines tested, it is unlikely that they used >up to date compilers (latest gcc's are better than earlier gcc's for >performance), or recompiled the binary for all the different platforms >to run native. The ifc results seem to indicate that they used SSE2 on >P4's but probably used plain old 32 bit code on Opterons. I wouldn't begin to speculate; however, would hope Daresbury Laboratory and Martyn Guest were working to advance research, using the best technology available for each platform. I didn't see anything in their mission statement which leads me to think otherwise. > > lends itself to hp Itanium 2, there appears to be a very balanced trend. > >... in a specific set of operations relevant for specific classes of >calculation. The tables cover a wide range of benchmarks specific to the interests of those working in computational chemistry. With respect to this, the rx2600 (Itanium 2 based) ranked among the top ten (with the exception of table 4 where it was ranked #11). Averaged out, the tables reflect an overall rating of 5.86 among the 400 platforms tested. My initial conclusion may have been less than scientific, but I'll stay with it for now. >Not everyone in HPC does matrix work, eigenvalue extraction, etc. Some >of us do things like string/db searching (informatics). There, the >numbers look quite different. My comments referenced numerically intensive research rather than I/O intensive environments. I'm surprised 8GB of memory is enough to sustain superior performance when searching very large data sets normally associated with bio-informatics. Ciao~ Michael >[... snip ...] > >-- >Joseph Landman, Ph.D >Scalable Informatics LLC, >email: landman@scalableinformatics.com >web : http://scalableinformatics.com >phone: +1 734 612 4615 From hunting at ix.netcom.com Thu May 20 20:41:02 2004 From: hunting at ix.netcom.com (Michael Huntingdon) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from mmccallum@pacific.edu) In-Reply-To: <1085101265.3308.87.camel@protein.scalableinformatics.com> References: <20040519200204.GO16732@leitl.org> <20040519221332.GA29085@cse.ucdavis.edu> <6.0.3.0.2.20040519222707.05023d08@popd.ix.netcom.com> <1085084616.2581.44.camel@protein.scalableinformatics.com> <6.0.3.0.2.20040520141728.05023d08@popd.ix.netcom.com> <1085101265.3308.87.camel@protein.scalableinformatics.com> Message-ID: <6.0.3.0.2.20040520181415.05050658@popd.ix.netcom.com> At 06:01 PM 5/20/2004, Joe Landman wrote: >On Thu, 2004-05-20 at 20:26, Michael Huntingdon wrote: > > At 01:23 PM 5/20/2004, Joe Landman wrote: > > >On Thu, 2004-05-20 at 02:05, Michael Huntingdon wrote: > > > > I've spent some time sifting through the attached numbers. Though > not each > > > > > >Any paper that starts out praising spec as a "good" predictive benchmark > > >is suspect. Benchmarking is difficult to do right, in large part > > >because of deceptively simple scoring functions (time, frequency (not of > > >the CPU, but number of iterations per unit time), ...). > > Although SPEC was included among others, I didn't see where Martyn Guest > > "praised" SPEC. > >Page 5: "One of the most useful indicators of CPU performance is >provided by the SPEC ... benchmarks" > >Many folks would take issue with the exact utility of these benchmarks. Again...SPEC was included, among a number of others. Would you happen to have a cross section of code relevant to computational chemistry that might offer a fair example of platform performance? I'd enjoy a chance to get it into the labs and see how Itanuim 2 might stack up. It's hard to overlook the bias in all the assumptions suggested. Those funding the project that led to the papers referenced, as well as those who took part in the testing/reporting, by way of their careers must have followed procedures to ensure the system/os/compiled code represented a fair and balanced environment. We support Pentium, Xeon, Opteron, PA, Alpha and Itanuim 2 solutions for higher education, so I have no particular ax to grind. Puzzled though, in as much as Martyn is well published over many years, how his results find this type skepticism. I hope you have an opportunity to contact him directly. regards michael >At least he points out later in the document (page 6) some of the >serious flaws in the benchmark. The problem is that he continues to use >it as a valid scoring function. We can argue and debate over this, but >the numbers are of highly dubious value at best. > > > > > >Further, looking these over, I did not see much of a discussion (though > > >it is implied by the use of certain compilers) of the effects of things > > >like SSE2 in the P4, memory alignment, 32/64 bit > > >compilation/optimization, use of tuned libraries where available... > > >Given the sheer number of machines tested, it is unlikely that they used > > >up to date compilers (latest gcc's are better than earlier gcc's for > > >performance), or recompiled the binary for all the different platforms > > >to run native. The ifc results seem to indicate that they used SSE2 on > > >P4's but probably used plain old 32 bit code on Opterons. > > > > I wouldn't begin to speculate; however, would hope Daresbury Laboratory > and > > Martyn Guest were working to advance research, using the best technology > > available for each platform. I didn't see anything in their mission > > statement which leads me to think otherwise. > >No one is implying that they would do anything less than advance the >state of knowledge. It is important to note that little information (I >may have missed it, so please do point it out if you find it) exists on >the use of the -m64 gcc compilation for Opterons (gets you a nice >performance boost in many cases, and in a number of chemistry >applications I have worked with), or the ACML libraries for high >performance *GEMM operations on AMD, or the Altivec compilation/math >libraries, or the SGI performance libraries, ... etc. That is, as I >implied, it would be quite difficult for the lab to a) test all the >machines, b) test all the machines optimally. In fact, they >specifically indicate that they could not do so (see page 4) due to time >constraints. > >While the information in here does appear to be useful (and I did not >state otherwise), it does not constitute an exhaustive investigation of >machine performance characteristics. It does appear to compare how well >some programs ran on limited time loaner machines, donated hardware, >etc. Which means the operative issue is to get results quickly and hope >you can do some fast optimization. > >It would be dangerous to draw conclusions beyond the text which the >authors specifically caution against. > > > > > lends itself to hp Itanium 2, there appears to be a very balanced > trend. > > > > > >... in a specific set of operations relevant for specific classes of > > >calculation. > > > > The tables cover a wide range of benchmarks specific to the interests of > > those working in computational chemistry. With respect to this, the rx2600 > > (Itanium 2 based) ranked among the top ten (with the exception of table 4 > > where it was ranked #11). Averaged out, the tables reflect an overall > > rating of 5.86 among the 400 platforms tested. My initial conclusion may > > have been less than scientific, but I'll stay with it for now. > >Thats fine. You of course are entitled to your opinion. > >You asked a simple question as to why there is not more discussion of >this in these and other circles. Well, other people are entitled to >their opinions, and it appears the market is indeed deciding between >competitors. > >Aside from this, "benchmarks" are problematic to do right, in a >completely non-biased manner. These benchmarks are interesting, but >there was not enough detail given of the systems for others to try to >replicate the work. For example, which OS, specific compiler versions, >patches were used? For the non-spec codes, which compilation options >were used? For the chips with SIMD capability, was it used (P4, >Opteron, G5)? How was memory laid out? Was any attention paid to >processor affinity and related scheduling? > >Remember that using the ifc/efc compilers with the Itanium chips as well >as the Pentium chips gives you a significant leg up in performance as >compared to using the gcc system on similar architectures. Moreover, >there is a performance penalty to be paid for not picking the compiler >options carefully under gcc or ifc. > > > >Not everyone in HPC does matrix work, eigenvalue extraction, etc. Some > > >of us do things like string/db searching (informatics). There, the > > >numbers look quite different. > > > > My comments referenced numerically intensive research rather than I/O > > intensive environments. I'm surprised 8GB of memory is enough to sustain > > superior performance when searching very large data sets normally > > associated with bio-informatics. > >8GB is enough for some, not enough for others. Some projects I have >worked on >(http://www.sgi.com/newsroom/press_releases/2001/january/msi.html) have >used a few processors and a little memory. > >Again, as indicated by many others (and myself), the only things that >matter are your (the end users) tests, with real data. I am intrigued >by Martyn's chemistry tests, and when I get a free moment, I will send a >note about possibly including a more protein oriented set of tests into >BBS (http://bioinformatics.org/bbs) v2 due out soon (yeah I know I keep >saying that, but it is soon...) > > > > > Ciao~ > > Michael > > > > > > >[... snip ...] > > > > > >-- > > >Joseph Landman, Ph.D > > >Scalable Informatics LLC, > > >email: landman@scalableinformatics.com > > >web : http://scalableinformatics.com > > >phone: +1 734 612 4615 >-- >Joseph Landman, Ph.D >Scalable Informatics LLC, >email: landman@scalableinformatics.com >web : http://scalableinformatics.com >phone: +1 734 612 4615 From cpignol at seismiccity.com Thu May 20 11:42:37 2004 From: cpignol at seismiccity.com (Claude Pignol) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] ARIMA HDAMA dual opteron motherboard Message-ID: <40ACFC1D.50701@seismiccity.com> Greetings I have noticed a very high failure rate on motherboard dual opteron HDAMA revision D5 or D6 from ARIMA and would like to know if I have a very bad batch of boards or if it's well known problem. It is always the same component (CPU 0 or CPU 1 mosfet) that failed. Thanks Claude From landman at scalableinformatics.com Thu May 20 13:23:36 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from mmccallum@pacific.edu) In-Reply-To: <6.0.3.0.2.20040519222707.05023d08@popd.ix.netcom.com> References: <20040519200204.GO16732@leitl.org> <20040519221332.GA29085@cse.ucdavis.edu> <6.0.3.0.2.20040519222707.05023d08@popd.ix.netcom.com> Message-ID: <1085084616.2581.44.camel@protein.scalableinformatics.com> On Thu, 2004-05-20 at 02:05, Michael Huntingdon wrote: > I've spent some time sifting through the attached numbers. Though not each Any paper that starts out praising spec as a "good" predictive benchmark is suspect. Benchmarking is difficult to do right, in large part because of deceptively simple scoring functions (time, frequency (not of the CPU, but number of iterations per unit time), ...). Further, looking these over, I did not see much of a discussion (though it is implied by the use of certain compilers) of the effects of things like SSE2 in the P4, memory alignment, 32/64 bit compilation/optimization, use of tuned libraries where available... Given the sheer number of machines tested, it is unlikely that they used up to date compilers (latest gcc's are better than earlier gcc's for performance), or recompiled the binary for all the different platforms to run native. The ifc results seem to indicate that they used SSE2 on P4's but probably used plain old 32 bit code on Opterons. > lends itself to hp Itanium 2, there appears to be a very balanced trend. ... in a specific set of operations relevant for specific classes of calculation. Not everyone in HPC does matrix work, eigenvalue extraction, etc. Some of us do things like string/db searching (informatics). There, the numbers look quite different. [... snip ...] -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 From lindahl at pathscale.com Thu May 20 13:40:11 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from mmccallum@pacific.edu) In-Reply-To: <6.0.3.0.2.20040519222707.05023d08@popd.ix.netcom.com> References: <20040519200204.GO16732@leitl.org> <20040519221332.GA29085@cse.ucdavis.edu> <6.0.3.0.2.20040519222707.05023d08@popd.ix.netcom.com> Message-ID: <20040520204011.GE2586@greglaptop.internal.keyresearch.com> On Wed, May 19, 2004 at 11:05:12PM -0700, Michael Huntingdon wrote: > For clusters, scale down the hp rx2600 to the rx1600 dual CPU with fewer > PCI-X slots, comparable performance, HP-UX/linux/OVMS, in box upgrades, > excellent services, and education price under $3,000. The only benchmarks for rx1600 are at 1 Ghz, and SPECfp peak is only 1382. If the $3,000 system is only 1 Ghz, that's not so great price performance, compared to say a 2.2 Ghz Opteron with a SPECfp peak of 1691. Perhaps HP does have a faster CPU for the rx1600 and didn't submit a SPEC score for it? But then what's the price? -- greg From scrumb at mcs.anl.gov Thu May 20 15:05:21 2004 From: scrumb at mcs.anl.gov (Steve Crumb) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] Upcoming Global Grid Forum Events Message-ID: <001601c43eb6$8bc5c2e0$9412dd8c@ggflaptop2> GGF11 - June 6-10, 2004, Honolulu, Hawaii (Oahu) The Global Grid Forum (GGF) will be holding its next event (GGF11) at the Kalia Tower Mid-Pacific Conference Center in Honolulu, Hawaii, June 6-10, 2004. This event will held in conjunction with HPDC-13 and will focus on "Building Foundations for Enterprise Grids". Specifically, the program will include the following highlights: - 8 tutorials (http://www.ggf.org/Meetings/GGF11/GGF11_Tutorials_b.htm) - 4 workshops (http://www.extreme.indiana.edu/groc/index.html) - Keynotes and invited talks by Ian Foster, Hiro Kishimoto, Yoshio Tanaka, Andrew Grimshaw, David Snelling, Marty Humphrey, and others. - Several panels including a standards body collaboration panel with representatives from OASIS, DMTF, IETF, and W3C. - 64 working group or research group sessions - 7 birds-of-a-feather (BOF) sessions Please visit http://www.ggf.org for more details or register at http://www.regonline.com/Checkin.asp?EventId=13537. GGF's Second Telecom Focus Session, June 21, 2004 GGF has selected Supercomm2004 (McCormick Place, Chicago) as the venue for its second telecom focus session. This meeting will continue the discussions started at GGF10 in Berlin where several large telecom providers, vendors, and researchers met to explore the impact of grid computing on telecommunication companies. The two primary focuses of the meeting will be: - Business value of grids in multiple vertical markets - Requirements for Grid Services: Telcos as suppliers of commercial grid services. This meeting is free but you must RSVP to Steve Crumb (scrumb@ggf.org) prior to June 4, 2004. The Global Grid Forum (GGF) is a community-initiated forum of thousands of individuals from industry and research, leading the global standardization effort for grid computing. GGF's primary objectives are to promote and support the development, deployment, and implementation of grid technologies and applications via the creation and documentation of "best practices" - technical specifications, user experiences, and implementation guidelines. From landman at scalableinformatics.com Thu May 20 18:01:06 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from mmccallum@pacific.edu) In-Reply-To: <6.0.3.0.2.20040520141728.05023d08@popd.ix.netcom.com> References: <20040519200204.GO16732@leitl.org> <20040519221332.GA29085@cse.ucdavis.edu> <6.0.3.0.2.20040519222707.05023d08@popd.ix.netcom.com> <1085084616.2581.44.camel@protein.scalableinformatics.com> <6.0.3.0.2.20040520141728.05023d08@popd.ix.netcom.com> Message-ID: <1085101265.3308.87.camel@protein.scalableinformatics.com> On Thu, 2004-05-20 at 20:26, Michael Huntingdon wrote: > At 01:23 PM 5/20/2004, Joe Landman wrote: > >On Thu, 2004-05-20 at 02:05, Michael Huntingdon wrote: > > > I've spent some time sifting through the attached numbers. Though not each > > > >Any paper that starts out praising spec as a "good" predictive benchmark > >is suspect. Benchmarking is difficult to do right, in large part > >because of deceptively simple scoring functions (time, frequency (not of > >the CPU, but number of iterations per unit time), ...). > Although SPEC was included among others, I didn't see where Martyn Guest > "praised" SPEC. Page 5: "One of the most useful indicators of CPU performance is provided by the SPEC ... benchmarks" Many folks would take issue with the exact utility of these benchmarks. At least he points out later in the document (page 6) some of the serious flaws in the benchmark. The problem is that he continues to use it as a valid scoring function. We can argue and debate over this, but the numbers are of highly dubious value at best. > > >Further, looking these over, I did not see much of a discussion (though > >it is implied by the use of certain compilers) of the effects of things > >like SSE2 in the P4, memory alignment, 32/64 bit > >compilation/optimization, use of tuned libraries where available... > >Given the sheer number of machines tested, it is unlikely that they used > >up to date compilers (latest gcc's are better than earlier gcc's for > >performance), or recompiled the binary for all the different platforms > >to run native. The ifc results seem to indicate that they used SSE2 on > >P4's but probably used plain old 32 bit code on Opterons. > > I wouldn't begin to speculate; however, would hope Daresbury Laboratory and > Martyn Guest were working to advance research, using the best technology > available for each platform. I didn't see anything in their mission > statement which leads me to think otherwise. No one is implying that they would do anything less than advance the state of knowledge. It is important to note that little information (I may have missed it, so please do point it out if you find it) exists on the use of the -m64 gcc compilation for Opterons (gets you a nice performance boost in many cases, and in a number of chemistry applications I have worked with), or the ACML libraries for high performance *GEMM operations on AMD, or the Altivec compilation/math libraries, or the SGI performance libraries, ... etc. That is, as I implied, it would be quite difficult for the lab to a) test all the machines, b) test all the machines optimally. In fact, they specifically indicate that they could not do so (see page 4) due to time constraints. While the information in here does appear to be useful (and I did not state otherwise), it does not constitute an exhaustive investigation of machine performance characteristics. It does appear to compare how well some programs ran on limited time loaner machines, donated hardware, etc. Which means the operative issue is to get results quickly and hope you can do some fast optimization. It would be dangerous to draw conclusions beyond the text which the authors specifically caution against. > > > lends itself to hp Itanium 2, there appears to be a very balanced trend. > > > >... in a specific set of operations relevant for specific classes of > >calculation. > > The tables cover a wide range of benchmarks specific to the interests of > those working in computational chemistry. With respect to this, the rx2600 > (Itanium 2 based) ranked among the top ten (with the exception of table 4 > where it was ranked #11). Averaged out, the tables reflect an overall > rating of 5.86 among the 400 platforms tested. My initial conclusion may > have been less than scientific, but I'll stay with it for now. Thats fine. You of course are entitled to your opinion. You asked a simple question as to why there is not more discussion of this in these and other circles. Well, other people are entitled to their opinions, and it appears the market is indeed deciding between competitors. Aside from this, "benchmarks" are problematic to do right, in a completely non-biased manner. These benchmarks are interesting, but there was not enough detail given of the systems for others to try to replicate the work. For example, which OS, specific compiler versions, patches were used? For the non-spec codes, which compilation options were used? For the chips with SIMD capability, was it used (P4, Opteron, G5)? How was memory laid out? Was any attention paid to processor affinity and related scheduling? Remember that using the ifc/efc compilers with the Itanium chips as well as the Pentium chips gives you a significant leg up in performance as compared to using the gcc system on similar architectures. Moreover, there is a performance penalty to be paid for not picking the compiler options carefully under gcc or ifc. > >Not everyone in HPC does matrix work, eigenvalue extraction, etc. Some > >of us do things like string/db searching (informatics). There, the > >numbers look quite different. > > My comments referenced numerically intensive research rather than I/O > intensive environments. I'm surprised 8GB of memory is enough to sustain > superior performance when searching very large data sets normally > associated with bio-informatics. 8GB is enough for some, not enough for others. Some projects I have worked on (http://www.sgi.com/newsroom/press_releases/2001/january/msi.html) have used a few processors and a little memory. Again, as indicated by many others (and myself), the only things that matter are your (the end users) tests, with real data. I am intrigued by Martyn's chemistry tests, and when I get a free moment, I will send a note about possibly including a more protein oriented set of tests into BBS (http://bioinformatics.org/bbs) v2 due out soon (yeah I know I keep saying that, but it is soon...) > > Ciao~ > Michael > > > >[... snip ...] > > > >-- > >Joseph Landman, Ph.D > >Scalable Informatics LLC, > >email: landman@scalableinformatics.com > >web : http://scalableinformatics.com > >phone: +1 734 612 4615 -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 From hahn at physics.mcmaster.ca Fri May 21 06:14:09 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from mmccallum@pacific.edu) In-Reply-To: <20040520204011.GE2586@greglaptop.internal.keyresearch.com> Message-ID: > The only benchmarks for rx1600 are at 1 Ghz, and SPECfp peak is only > 1382. If the $3,000 system is only 1 Ghz, that's not so great price > performance, compared to say a 2.2 Ghz Opteron with a SPECfp peak of > 1691. right. I did a little meta-analysis of specFP recently, trying to get at the fact that some machines owe much of their score to a small number of spec components. I simply sorted the individual spec scores for a machine, and recomputed specFP omitting the top 1, top 2, etc. http://sharcnet.mcmaster.ca/~hahn/circumspec.png as a user, what you'd like to see is a long, flat section in the middle. I thought a couple things were interesting about this. first, for It2, the high scores correspond with components that have tiny working sets, and fit in cache. you can also see this reflected in how the two it2/1300 scores diverge. the other interesting thing is that the opteron (148, I think) is actually faster than the it2/1300 for a significant subset of specFP. that said, it seems pretty clear that: - ppc970's are great if your code is cache-friendly and can launch two vector mul-add's per cycle. - it2 is great if you can afford them and your code is cache-friendly and your code is amenable to the kind of pipelining required by EPIC. - opterons are great if your working set makes caches less effective. of course, most installations will also need to consider power - though no one seems to know how much the 90 nm ppc970 dissipates... From landman at scalableinformatics.com Fri May 21 06:34:53 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] updated x86_64 mpiBLAST rpms available Message-ID: <1085146492.2551.18.camel@protein.scalableinformatics.com> Hi folks: I updated the mpiBLAST (http://mpiblast.lanl.gov) RPMs that we are hosting off of http://scalableinformatics.com . The 1.2.1-1 source RPM would not build cleanly on an x86_64 platform. This has been fixed, and we built binaries for SUSE 9.0 AMD. If there is interest, we may build some for Fedora Core 2, and other platforms, though the source RPM is there for you as well. See http://downloads.scalableinformatics.com/downloads/mpiblast/ for the updated packages (1.2.1-2). Older packages will be archived by next week. Joe -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 From jlb17 at duke.edu Fri May 21 05:25:52 2004 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] ARIMA HDAMA dual opteron motherboard In-Reply-To: <40ACFC1D.50701@seismiccity.com> References: <40ACFC1D.50701@seismiccity.com> Message-ID: On Thu, 20 May 2004 at 1:42pm, Claude Pignol wrote > I have noticed a very high failure rate on motherboard dual opteron > HDAMA revision D5 or D6 from ARIMA > and would like to know if I have a very bad batch of boards or if it's > well known problem. > It is always the same component (CPU 0 or CPU 1 mosfet) that failed. Very small data point: I've got 6 of these (well, I'm not sure which revision) in production since February with no problems. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University From konstantin_kudin at yahoo.com Fri May 21 12:49:47 2004 From: konstantin_kudin at yahoo.com (Konstantin Kudin) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] Good 1 Gbit switches - which ones? Message-ID: <20040521194947.27584.qmail@web21204.mail.yahoo.com> Hi there, Can anyone offer an insight with respect to 1 Gbit switches for a Beowulf cluster? There are all these reports that a lot of inexpensive switches on the market tend to choke under heavy internal traffic. Can anyone suggest an affordable switch with good internal bandwidth, which was tested under heavy load, and actually worked well? Thanks! Konstantin __________________________________ Do you Yahoo!? Yahoo! Domains – Claim yours for only $14.70/year http://smallbusiness.promotions.yahoo.com/offer From eugen at leitl.org Fri May 21 12:49:34 2004 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:running GAMESS using Beowulf cluster (fwd from orbital_jc@yahoo.co.uk) Message-ID: <20040521194933.GO16732@leitl.org> ----- Forwarded message from Jeff C ----- From: Jeff C Date: Fri, 21 May 2004 16:12:20 +0100 (BST) To: CHEMISTRY@ccl.net Subject: CCL:running GAMESS using Beowulf cluster Dear Comp Chem Experts, I am new to this field so I will apologize in advance if the following question is too trivial to the skilled researchers. Currently I am trying to learn how to use gamess. The procedure is simple when comes to running gamess with single CPU. However, I am clueless when trying to run gamess on our newly built 16 nodes Beowulf cluster using MPI. I hope someone out there can give me the step-by-step guide. Much appreciated. Jeff Chen ____________________________________________________________ Yahoo! Messenger - Communicate instantly..."Ping" your friends today! Download Messenger Now http://uk.messenger.yahoo.com/download/index.html -= This is automatically added to each message by the mailing script =- To send e-mail to subscribers of CCL put the string CCL: on your Subject: line and send your message to: CHEMISTRY@ccl.net Send your subscription/unsubscription requests to: CHEMISTRY-REQUEST@ccl.net HOME Page: http://www.ccl.net | Jobs Page: http://www.ccl.net/jobs If your mail is bouncing from CCL.NET domain send it to the maintainer: Jan Labanowski, jlabanow@nd.edu (read about it on CCL Home Page) -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ----- End forwarded message ----- -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20040521/dc9f69f4/attachment.bin From lindahl at pathscale.com Fri May 21 13:56:48 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from mmccallum@pacific.edu) In-Reply-To: References: <20040520204011.GE2586@greglaptop.internal.keyresearch.com> Message-ID: <20040521205648.GA1888@greglaptop.internal.keyresearch.com> On Fri, May 21, 2004 at 09:14:09AM -0400, Mark Hahn wrote: > right. I did a little meta-analysis of specFP recently, trying to get at > the fact that some machines owe much of their score to a small number of > spec components. I simply sorted the individual spec scores for a machine, > and recomputed specFP omitting the top 1, top 2, etc. This is not a valid approach: The scores are not normalized in a fashion that makes their absolute values useful. A more valid approach is to look at how many components of SPECfp each system wins on. That doesn't depend on the absolute values of the scores, only the relative values. > - it2 is great if you can afford them [...] What does this mean? People say it all the time, but it doesn't mean anything. If we're talking clusters, and systems built with processor X cost 3 times as much as systems built with processor Y, you can "afford" X unless you're spending so little you can't get a whole X. Do people commonly build clusters out of 2 machines? I don't think so... Conversely, even if I have a lot of money, I am still going to spend it to get the most bang for the buck. So it doesn't matter if I have a lot of money, it's my problem and the performance of systems on my problem that dictate what I buy. Funny that I'm complaining about the same issue twice over: absolute values vs. relative ones ;-) -- greg From konstantin_kudin at yahoo.com Fri May 21 14:23:48 2004 From: konstantin_kudin at yahoo.com (Konstantin Kudin) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance Message-ID: <20040521212348.90590.qmail@web21202.mail.yahoo.com> > Again...SPEC was included, among a number of others. Would you happen > to have a cross section of code relevant to computational chemistry > that might offer a fair example of platform performance? I'd enjoy a > chance to get it into the labs and see how Itanuim 2 might stack up. While I can't speak for every code out there, for Gaussian03 I2 seems to work extremely well: http://www.princeton.edu/~kkudin/g03_b5_tests_1.txt Note that these are ~1 hour long jobs, which surely do not fit in cache. The interesting part is that for some real life codes I2 performance matches what SPEC numbers would predict. The cost, of course, is a different issue altogether. Konstantin __________________________________ Do you Yahoo!? Yahoo! Domains – Claim yours for only $14.70/year http://smallbusiness.promotions.yahoo.com/offer From wharman at prism.net Fri May 21 14:06:36 2004 From: wharman at prism.net (William Harman) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] Good 1 Gbit switches - which ones? In-Reply-To: <20040521194947.27584.qmail@web21204.mail.yahoo.com> Message-ID: <200405212100.i4LL0BXv029826@bluewest.scyld.com> I like the HP Procurve Products: Procurve 2824, 24-ports of GigE (4 ports can be either copper or fiber) List $2,499 look for a street price around $1,970 Procurve 2848, 48-ports of GigE (4 ports can be either copper of fiber) List $4,899 look for a street price around $3,855 They are both 1U form factor, stackable; the 2824 has a switching capacity of 48Gbps and the 2848 has a switching capacity of 96Gbps Bill -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Konstantin Kudin Sent: Friday, May 21, 2004 1:50 PM To: beowulf@beowulf.org Subject: [Beowulf] Good 1 Gbit switches - which ones? Hi there, Can anyone offer an insight with respect to 1 Gbit switches for a Beowulf cluster? There are all these reports that a lot of inexpensive switches on the market tend to choke under heavy internal traffic. Can anyone suggest an affordable switch with good internal bandwidth, which was tested under heavy load, and actually worked well? Thanks! Konstantin __________________________________ Do you Yahoo!? Yahoo! Domains  Claim yours for only $14.70/year http://smallbusiness.promotions.yahoo.com/offer _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From eugen at leitl.org Fri May 21 14:38:15 2004 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance (fwd from konstantin_kudin@yahoo.com) Message-ID: <20040521213815.GV16732@leitl.org> ----- Forwarded message from Konstantin Kudin ----- From: Konstantin Kudin Date: Fri, 21 May 2004 14:23:48 -0700 (PDT) To: beowulf@beowulf.org Subject: [Beowulf] CCL:Question regarding Mac G5 performance > Again...SPEC was included, among a number of others. Would you happen > to have a cross section of code relevant to computational chemistry > that might offer a fair example of platform performance? I'd enjoy a > chance to get it into the labs and see how Itanuim 2 might stack up. While I can't speak for every code out there, for Gaussian03 I2 seems to work extremely well: http://www.princeton.edu/~kkudin/g03_b5_tests_1.txt Note that these are ~1 hour long jobs, which surely do not fit in cache. The interesting part is that for some real life codes I2 performance matches what SPEC numbers would predict. The cost, of course, is a different issue altogether. Konstantin __________________________________ Do you Yahoo!? Yahoo! Domains ? Claim yours for only $14.70/year http://smallbusiness.promotions.yahoo.com/offer _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ----- End forwarded message ----- -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20040521/e88fb899/attachment.bin From roger at ERC.MsState.Edu Fri May 21 15:24:22 2004 From: roger at ERC.MsState.Edu (Roger L. Smith) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] (no subject) Message-ID: I have a visualization cluster that has four systems connected to the four DVI inputs on an IBM T221 monitor. Each system drives one quadrant of the monitor. I'd like to set things up such that a user only has to log into one of the systems and then be able to display X applications on the other three without having to manually log into the other three systems. For example, the user logs into node1, then starts dmx and it controls all four systems. It doesn't matter to me if they would have to run a script that would ssh onto the other nodes and start X or set authority keys or something on the other nodes. I'd like to keep some semblance of security in the process, but there would be a fairly limited number of users needing this, and I am potentially willing to grant them some sort of elevated privileges, if necessary. Does anyone have any ideas of a good way to do this? _\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_ | Roger L. Smith Phone: 662-325-3625 | | Sr. Systems Administrator FAX: 662-325-7692 | | roger@ERC.MsState.Edu http://WWW.ERC.MsState.Edu/~roger | | Mississippi State University | |____________________________________ERC__________________________________| From gerry.creager at tamu.edu Fri May 21 15:35:00 2004 From: gerry.creager at tamu.edu (Gerry Creager N5JXS) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] Good 1 Gbit switches - which ones? In-Reply-To: <20040521194947.27584.qmail@web21204.mail.yahoo.com> References: <20040521194947.27584.qmail@web21204.mail.yahoo.com> Message-ID: <40AE8414.3060901@tamu.edu> I just bought Foundry EdgeIrons at about $3K for 24 ports. Excellent performance, and stellar small packet performance as tested in my network engineering lab (Anritsu MD1230). The latest variant of HP ProCurve looks good by the specs, but I'd been able to test the EdgeIrons... Konstantin Kudin wrote: > Hi there, > > Can anyone offer an insight with respect to 1 Gbit switches for a > Beowulf cluster? There are all these reports that a lot of inexpensive > switches on the market tend to choke under heavy internal traffic. Can > anyone suggest an affordable switch with good internal bandwidth, which > was tested under heavy load, and actually worked well? > > Thanks! > Konstantin > > > > > __________________________________ > Do you Yahoo!? > Yahoo! Domains ? Claim yours for only $14.70/year > http://smallbusiness.promotions.yahoo.com/offer > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Gerry Creager -- gerry.creager@tamu.edu Network Engineering -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578 Page: 979.228.0173 Office: 903A Eller Bldg, TAMU, College Station, TX 77843 From jmdavis at mail2.vcu.edu Fri May 21 17:37:39 2004 From: jmdavis at mail2.vcu.edu (Mike Davis) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:running GAMESS using Beowulf cluster (fwd from orbital_jc@yahoo.co.uk) In-Reply-To: <20040521194933.GO16732@leitl.org> References: <20040521194933.GO16732@leitl.org> Message-ID: <40AEA0D3.2050805@mail2.vcu.edu> I had one of the gamess linux MPI port people (Charles Castevens) working for me and we still had problems with the mpi version of gamess. We wound up using the sockets version along with a script to submit the jobs to pbs. For this to work you should have gamess (preferably installed in a common location on each node), PBS up and running (a far bigger task usually), modified comp files and rungms, and a script to handle pbs submission. I'm sorry but we never had nearly the time required to do a step by step guide. I will attach a copy of the script and note the modifications: ****************** script: #!/bin/tcsh -f # $cwd -- current working directory # $0 -- program name and path to its location # $1 -- gamess version # $2 -- input file name -- w/o inp # $3 -- nodes (optional) # $4 -- queue (optional) # set wd = $cwd set version = $1 set name = $2 if (null$version == null ) goto usage if (null$name == null ) goto usage set nodes = $3 if (null$nodes == null) set nodes=1 set queue = $4 if (null$queue == null ) set queue=p2p cat <${name}.job #!/bin/csh -f #PBS -q $queue #PBS -l nodes=$nodes #alias gms "/usr/local/gamess/seqgms" echo "Starting GAMESS with file $name at \`date\` on \`hostname\`" cd $wd #printenv /usr/u2/gamess16feb02r1/new_rungms $name $version $nodes >& $name.log mail $USER < ${name}.err mv /scr/$USER/$name.dat ~/scr/$name.dat mv /scr/$USER/$name.irc ~/scr/$name.irc echo "GAMESS Done with status \$status at \`date\` on \`hostname\`" END chmod u+x ${name}.job # submit the new script qsub ${name}.job #nohup ./${name}.job >& ${name}.err & unalias rm #rm $name.job echo "Success!" exit ************************** rungms mods: if ( `hostname` == "nameofyourcluster" ) then echo "creating hostlist using PBS environment PBS_NODEFILE:" echo $PBS_NODEFILE cat $PBS_NODEFILE set HOSTLIST='hostname' # # get the actual value of hostname via rsh for gamess # foreach i (`cat $PBS_NODEFILE`) if ( $i != "hydra1.vcu.edu" ) set HOSTLIST=($HOSTLIST `rsh $i hostname`) end # # if we are not on node001 and we are using more than 1 node # build a hostlist and use rsh to get the actual value returned # by hostname # else if ( -e $PBS_NODEFILE ) then echo "creating hostlist using PBS environment PBS_NODEFILE:" cat $PBS_NODEFILE # zero out HOSTLIST set HOSTLIST='' # get the actual value of hostname via rsh for gamess foreach i (`cat $PBS_NODEFILE`) set HOSTLIST=($HOSTLIST `rsh $i hostname`) end endif Eugen Leitl wrote: >----- Forwarded message from Jeff C ----- > >From: Jeff C >Date: Fri, 21 May 2004 16:12:20 +0100 (BST) >To: CHEMISTRY@ccl.net >Subject: CCL:running GAMESS using Beowulf cluster > >Dear Comp Chem Experts, > >I am new to this field so I will apologize in advance >if the following question is too trivial to the >skilled researchers. > >Currently I am trying to learn how to use gamess. The >procedure is simple when comes to running gamess with >single CPU. However, I am clueless when trying to run >gamess on our newly built 16 nodes Beowulf cluster >using MPI. I hope someone out there can give me the >step-by-step guide. > >Much appreciated. > >Jeff Chen > > > > > > >____________________________________________________________ >Yahoo! Messenger - Communicate instantly..."Ping" >your friends today! Download Messenger Now >http://uk.messenger.yahoo.com/download/index.html > > >-= This is automatically added to each message by the mailing script =- >To send e-mail to subscribers of CCL put the string CCL: on your Subject: line >and send your message to: CHEMISTRY@ccl.net > >Send your subscription/unsubscription requests to: CHEMISTRY-REQUEST@ccl.net >HOME Page: http://www.ccl.net | Jobs Page: http://www.ccl.net/jobs > >If your mail is bouncing from CCL.NET domain send it to the maintainer: >Jan Labanowski, jlabanow@nd.edu (read about it on CCL Home Page) >-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > > > > > >----- End forwarded message ----- > > >------------------------------------------------------------------------ > >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > From hanulec at hanulec.com Fri May 21 23:04:09 2004 From: hanulec at hanulec.com (Michael Hanulec) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] Good 1 Gbit switches - which ones? In-Reply-To: <200405212100.i4LL0BXv029826@bluewest.scyld.com> References: <200405212100.i4LL0BXv029826@bluewest.scyld.com> Message-ID: i'd recommend the Nortel BayStack 5510 series: http://www.nortelnetworks.com/products/02/bstk/switches/baystack_5510/index.html it supports 160 Gbps in one switch, 1280 Gbps when eight (8) switches are stacked together. in small cluster implementations you can 'stack' two cabinets together, one in each cabinet, using their 3' cables if the cabinets are placed next to each other. in large implementations you can use a combo of this two cabinet stack along w/ a four port trunk upto a master switch. http://www.nortelnetworks.com/products/02/bstk/switches/baystack_5510/techspecs.html -mike -- hanulec@hanulec.com cell: 858.518.2647 && 516.410.4478 https://secure.hanulec.com EFnet irc && aol im: hanulec On Fri, 21 May 2004, William Harman wrote: > I like the HP Procurve Products: > Procurve 2824, 24-ports of GigE (4 ports can be either copper or fiber) List > $2,499 look for a street price around $1,970 > Procurve 2848, 48-ports of GigE (4 ports can be either copper of fiber) List > $4,899 look for a street price around $3,855 > > They are both 1U form factor, stackable; the 2824 has a switching capacity > of 48Gbps and the 2848 has a switching capacity of 96Gbps > > Bill > > -----Original Message----- > From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On > Behalf Of Konstantin Kudin > Sent: Friday, May 21, 2004 1:50 PM > To: beowulf@beowulf.org > Subject: [Beowulf] Good 1 Gbit switches - which ones? > > Hi there, > > Can anyone offer an insight with respect to 1 Gbit switches for a Beowulf > cluster? There are all these reports that a lot of inexpensive switches on > the market tend to choke under heavy internal traffic. Can anyone suggest an > affordable switch with good internal bandwidth, which was tested under heavy > load, and actually worked well? > > Thanks! > Konstantin > > > > > __________________________________ > Do you Yahoo!? > Yahoo! Domains  Claim yours for only $14.70/year > http://smallbusiness.promotions.yahoo.com/offer > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org To change your subscription > (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > From Per.Lindstrom at me.chalmers.se Sat May 22 01:57:33 2004 From: Per.Lindstrom at me.chalmers.se (Per R M =?iso-8859-1?Q?Lindstr=F6m?=) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] ARIMA HDAMA dual opteron motherboard In-Reply-To: <40ACFC1D.50701@seismiccity.com> References: <40ACFC1D.50701@seismiccity.com> Message-ID: <61335.81.228.235.7.1085216253.squirrel@mail.medic.chalmers.se> Hi Claude, I have heard about some sort of calculation problem with one or more Opteron machines here at the school of Mechanical Engineering, Chalmers. I'm not sure about the exact details. But I will come back in the end of next week with all details found. Best regards Per ;) > Greetings > > I have noticed a very high failure rate on motherboard dual opteron > HDAMA revision D5 or D6 from ARIMA > and would like to know if I have a very bad batch of boards or if it's > well known problem. > It is always the same component (CPU 0 or CPU 1 mosfet) that failed. > > Thanks > Claude > > > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From landman at scalableinformatics.com Fri May 21 18:30:43 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance In-Reply-To: <20040521212348.90590.qmail@web21202.mail.yahoo.com> References: <20040521212348.90590.qmail@web21202.mail.yahoo.com> Message-ID: <1085189442.2461.20.camel@protein.scalableinformatics.com> On Fri, 2004-05-21 at 17:23, Konstantin Kudin wrote: > > Again...SPEC was included, among a number of others. Would you happen > > > to have a cross section of code relevant to computational chemistry > > > that might offer a fair example of platform performance? I'd enjoy a > > chance to get it into the labs and see how Itanuim 2 might stack up. > > While I can't speak for every code out there, for Gaussian03 I2 seems > to work extremely well: > http://www.princeton.edu/~kkudin/g03_b5_tests_1.txt Looks interesting. From the text, "All Intel compatible hardware ran under RedHat 9.0, in 32 bits." I presume you ran the I2 as a 64 bit OS using pgf77? or g77/ifc? Also you used the Atlas libs for p4. Same ones for O16 and O20? I would be curious to see the performance with a 64 bit compilation running under a 64 bit OS (SuSE 9.x) > Note that these are ~1 hour long jobs, which surely do not fit in > cache. Well, length of time doesn't indicate cache fit. Gaussian (depending upon the module) will significantly overflow cache, and sometimes main memory. Disk I/O bandwidth and organization matter for performance on the larger analysis, though anything spilling to disk is going to be slow. > The interesting part is that for some real life codes I2 performance > matches what SPEC numbers would predict. The cost, of course, is a > different issue altogether. The I2 is an interesting beast. Then again, so is the Opteron, and the G5. -- Joe Landman From hahn at physics.mcmaster.ca Sat May 22 14:24:25 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] Good 1 Gbit switches - which ones? In-Reply-To: <200405212100.i4LL0BXv029826@bluewest.scyld.com> Message-ID: > I like the HP Procurve Products: > Procurve 2824, 24-ports of GigE (4 ports can be either copper or fiber) List > $2,499 look for a street price around $1,970 > Procurve 2848, 48-ports of GigE (4 ports can be either copper of fiber) List > $4,899 look for a street price around $3,855 > > They are both 1U form factor, stackable; the 2824 has a switching capacity > of 48Gbps and the 2848 has a switching capacity of 96Gbps HP seems to make fine products, as does SMC. SMC's 8624 for instance, is widely used in clusters, also claims 48 Gbps, and sells for about $1000. I saw a 48 or 50pt version at a tradeshow ~6 months ago, but I don't see signs of it on the net... offhand, I'd guess that even cheaper/smaller switches from no-name vendors are not disasterous in performance any more, simply because all the vendors probably use the same basic multi-port chips to implement their switches... From hahn at physics.mcmaster.ca Sun May 23 10:40:30 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] harmonic mitigators vs PFC? Message-ID: Hi all. we're doing a new machineroom, which will dissipate around 250 KW. I'm having a hard time addressing the question of whether such a load (all modern PFC computers) needs harmonic mitigation. the electricians all just say "your loads are nonlinear so you need HM". I can't help think that the cost of HM hardware translates to more than a couple more nodes! and merely because the PSU is implemented with nonlinear components doesn't mean that the load is noisy. to my thinking, the active PFC found in current computers means precisely that HM is not necessary, since a PFC of .97 (so says my KillAWatt) indicates that only 3% of the power is drawn outside the ideal sine envelope. further, we're talking about O(800) seperate power supplies, and their harmonics are probably not perfectly synchronized, and so sub-additive. can anyone offer advice or references on this? we can apparently get a full answer by hiring a power consultant to bring in some kind of fancy power digitizer which will give us a plot of our load waveforms and presumably also a spectrum. so far, we're going along with the HM plan, but mainly because it'll clean up the power coming in, and perhaps permit us to ride out some flickers (the compute nodes won't have UPS-protected power...) thanks, mark hahn. From rgb at phy.duke.edu Mon May 24 05:29:46 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] harmonic mitigators vs PFC? In-Reply-To: Message-ID: On Sun, 23 May 2004, Mark Hahn wrote: > Hi all. > we're doing a new machineroom, which will dissipate around 250 KW. > I'm having a hard time addressing the question of whether such a load > (all modern PFC computers) needs harmonic mitigation. the electricians > all just say "your loads are nonlinear so you need HM". I can't help > think that the cost of HM hardware translates to more than a couple more nodes! > and merely because the PSU is implemented with nonlinear components doesn't > mean that the load is noisy. > > to my thinking, the active PFC found in current computers means precisely > that HM is not necessary, since a PFC of .97 (so says my KillAWatt) indicates > that only 3% of the power is drawn outside the ideal sine envelope. > further, we're talking about O(800) seperate power supplies, and their > harmonics are probably not perfectly synchronized, and so sub-additive. > > can anyone offer advice or references on this? we can apparently get a full > answer by hiring a power consultant to bring in some kind of fancy power > digitizer which will give us a plot of our load waveforms and presumably also > a spectrum. so far, we're going along with the HM plan, but mainly because > it'll clean up the power coming in, and perhaps permit us to ride out some > flickers (the compute nodes won't have UPS-protected power...) You're lucky your electricians are that clueful. Ours weren't... There is a HM link on the brahma resources page, let's see: http://www.mirusinternational.com/ Their FAQ is one of the best explanations of HM and nonlinear loads associated with switching power supplies that I've seen. When last I looked at this (not too long ago) not so many power supplies were PFC and the ones that were cost more (as one would expect). The argument advanced by Mirus is that putting a high quality HM transformer at the room itself saves you this marginal cost (x N nodes) and keeps you from having to make a PFC power supply a design specification for future nodes. Being a low suspicious sort of person, I'd also put a scope on the lines and look at the waveforms for the voltage and current (ideally simultaneously) before concluding that the current draw and voltage are in sync and that there are no e.g. 180 Hz harmonics. I'd guess that this is what the fancy consultant is offering to sell you, but for what you're likely to have to pay them you can probably buy an oscilloscope and do it yourself. I trust the kill-a-watt, sort of, but I trust my eyes more. You also should think of the whole picture (as it looks like you are doing) and not just the PFC issue. I believe that it is desirable to have the server room transformer as local as possible and have the distribution panel grounded to building steel (there are code rules on just where and how things have to be grounded) to avoid ground loops and excessively long runs to ground. If you need a transformer anyway, then you are really looking at the MARGINAL cost of the HM on the transformer you get, and I don't think it is all that great. HM can save you operational money if you DO have non-PFC supplies -- the peak currents drawn in the middle third of the phase a) cause the primary transformer to run hot; b) heat is money; c) it can actually cause the primary transformer to burn out prematurely which is a LOT of money; d) if you run circuits at max capacity without correction, you typically brown out the voltage during the peak due to resistive drops on the supply line, which in turn causes the supply to run with the capacitor not properly charged, which in turn reduces the system's fault tolerance. Shoot your electrical contractor if they even THINK of sharing a common neutral among three phases, and get them to use at least 12/2 throughout even for "short" runs. Thick wire stays cooler -- enough to pay for the difference in cost several times over, I believe, over the ~10+ year lifetime of such a space running 24x7 -- and minimizes the voltage on the neutral at the receptacle relative to ground. They'll likely bitch, because bending thicker wire through conduits is somewhat more difficult than thinner wire. I think I read somewhere in my researches on power that it is desirable in server room designs to keep the length of the runs between distribution panel and receptacle down to 25 feet. I don't know about the riding out flickers thing, but I would guess that the HM transformers would do a pretty good job of buffering at least certain classes of spikes and supply line wobbles. rgb > > thanks, mark hahn. > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From Jonathan.Ennis-King at csiro.au Sun May 23 23:14:09 2004 From: Jonathan.Ennis-King at csiro.au (Jonathan Ennis-King) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] licensing issues for Fortran compiler on cluster Message-ID: <40B192B1.2020107@csiro.au> I'm contemplating building a small cluster (10-12 nodes), and want to clarify the current situtation for licensing of Fortran compilers, so that I can compile some Fortran90 MPI code. From the archives I see that early last year there was discussion about the license for the Lahey Fortran compiler restricting one to running on clusters of a certain designated size (depending on cost of the license). Now looking at the Absoft website, they seem to have license costs for clusters of various sizes as well. So the question: Which Fortran90 compiler vendors (if any) allow one to compile on a single machine, link again relevant libraries and run on a cluster of arbitrary size, WITHOUT having to pay based on the size of the cluster? -- Jonathan Ennis-King email: Jonathan.Ennis-King@csiro.au post: CSIRO Petroleum, Private Bag 10, Clayton South, Victoria, 3169, Australia ph: +61-3-9545 8355 fax: +61-3-9545 8380 From konstantin_kudin at yahoo.com Mon May 24 10:45:32 2004 From: konstantin_kudin at yahoo.com (Konstantin Kudin) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance Message-ID: <20040524174532.14547.qmail@web21203.mail.yahoo.com> Joe Landman wrote: >> While I can't speak for every code out there, for Gaussian03 I2 seems >> to work extremely well: >> http://www.princeton.edu/~kkudin/g03_b5_tests_1.txt > Looks interesting. From the text, "All Intel compatible hardware ran > under RedHat 9.0, in 32 bits." I presume you ran the I2 as a 64 bit OS > using pgf77? or g77/ifc? I think I2 was using ifc compiler or something like that under a 64-bit OS. It does not seem like PGF or G77 are supported on I2 in 64-bits by Gaussian. > Also you used the Atlas libs for p4. Same > ones for O16 and O20? Yep, it was the same binary version running on all these machines. After a couple of optimization attempts it appeared that Opteron ran code optimized for P4 just as quickly as the code that was supposedly optimized for Opteron, so I simplified the whole thing. > I would be curious to see the performance with a 64 bit compilation > running under a 64 bit OS (SuSE 9.x) Me too :-) According to this CCL message (and other reliable sources), the official 64-bit support is just around the corner: http://www.ccl.net/cgi-bin/ccl/message.cgi?2004+04+28+013 (see reply #2) It is unlikely that one can gain much speed from going to 64 bits, but the support for larger memory and unlimited scratch files is very worthwhile in itself. Konstantin __________________________________ Do you Yahoo!? Yahoo! Domains – Claim yours for only $14.70/year http://smallbusiness.promotions.yahoo.com/offer From landman at scalableinformatics.com Mon May 24 07:07:08 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] licensing issues for Fortran compiler on cluster In-Reply-To: <40B192B1.2020107@csiro.au> References: <40B192B1.2020107@csiro.au> Message-ID: <1085407628.5412.88.camel@protein.scalableinformatics.com> Hi Jonathan: Portland Group (www.pgroup.com) Intel and probably a few others. Joe On Mon, 2004-05-24 at 02:14, Jonathan Ennis-King wrote: > I'm contemplating building a small cluster (10-12 nodes), and want to > clarify the current situtation for licensing of Fortran compilers, so > that I can compile some Fortran90 MPI code. From the archives I see that > early last year there was discussion about the license for the Lahey > Fortran compiler restricting one to running on clusters of a certain > designated size (depending on cost of the license). Now looking at the > Absoft website, they seem to have license costs for clusters of various > sizes as well. So the question: > > Which Fortran90 compiler vendors (if any) allow one to compile on a > single machine, link again relevant libraries and run on a cluster of > arbitrary size, WITHOUT having to pay based on the size of the cluster? -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 From atp at piskorski.com Mon May 24 07:08:04 2004 From: atp at piskorski.com (Andrew Piskorski) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] Re: harmonic mitigators vs PFC? In-Reply-To: <200405241312.i4ODCJW8017104@bluewest.scyld.com> References: <200405241312.i4ODCJW8017104@bluewest.scyld.com> Message-ID: <20040524140804.GA77504@piskorski.com> On Mon, May 24, 2004 at 06:12:46AM -0700, beowulf-request@beowulf.org wrote: > Date: Mon, 24 May 2004 08:29:46 -0400 (EDT) > From: "Robert G. Brown" > When last I looked at this (not too long ago) not so many power supplies > were PFC and the ones that were cost more (as one would expect). The I think most current desktops and the like DO still ship with non-PFC supplies, but I'm not sure. But FYI, newegg.com sells the Thermaltake Purpower 420W w/ Active PFC for only $50. I've only used it in one AMD Athlon box, but it did work nicely while a generic cheapo "500 W" supply caused all sorts of instability. (Which I've also seen before with Athlons.) Newegg is also currently selling at least one 300W Active PFC supply for only $21. With those kinds of prices, I don't see much reason to ever use a non-PFC supply. http://www.newegg.com/app/ViewProduct.asp?&catalog=58&description=Active+PFC&Order=price -- Andrew Piskorski http://www.piskorski.com/ From james.p.lux at jpl.nasa.gov Mon May 24 07:13:59 2004 From: james.p.lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] harmonic mitigators vs PFC? References: Message-ID: <000a01c44199$5bf39bb0$32a8a8c0@LAPTOP152422> ----- Original Message ----- From: "Mark Hahn" To: Sent: Sunday, May 23, 2004 10:40 AM Subject: [Beowulf] harmonic mitigators vs PFC? > Hi all. > we're doing a new machineroom, which will dissipate around 250 KW. > I'm having a hard time addressing the question of whether such a load > (all modern PFC computers) needs harmonic mitigation. the electricians > all just say "your loads are nonlinear so you need HM". I can't help > think that the cost of HM hardware translates to more than a couple more nodes! > and merely because the PSU is implemented with nonlinear components doesn't > mean that the load is noisy. You mention 800 nodes, etc., So you're talking about around 0.2% of your total hardware cost? Seems like a no-brainer to go with the recommendation of the people who are concerned, if only for political reasons. > > to my thinking, the active PFC found in current computers means precisely > that HM is not necessary, since a PFC of .97 (so says my KillAWatt) indicates > that only 3% of the power is drawn outside the ideal sine envelope. > further, we're talking about O(800) seperate power supplies, and their > harmonics are probably not perfectly synchronized, and so sub-additive. Actually, the mechanism is such that the harmonics ARE synchronized. It's not the harmonics of the switching element, it's the harmonics of the non-sinusoidal line current being drawn by a diode feeding a capacitor input filter, which looks very peaky, particularly at light loads. Those harmonics are multiples of the line frequency, and, since your 250 kVA load is almost certainly being fed from three phase, the neutral or circulating currents for the 3 x line frequency (called triplen) harmonics are of great concern. > > can anyone offer advice or references on this? we can apparently get a full > answer by hiring a power consultant to bring in some kind of fancy power > digitizer which will give us a plot of our load waveforms and presumably also > a spectrum. so far, we're going along with the HM plan, but mainly because > it'll clean up the power coming in, and perhaps permit us to ride out some > flickers (the compute nodes won't have UPS-protected power...) Got an oscilloscope? Just hook it up to measure the current going into one of your nodes. If it's a digitizing scope (any will do, because you're looking at lowish frequencies), grab a few cycles worth of the data, and run a transform to look at the power spectrum. Somewhere, about 3 or 4 months ago, someone posted a link to a very nice explanation of managing harmonics in this sort of situation. From landman at scalableinformatics.com Mon May 24 07:49:06 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] NCBI-2.2.9 rpms available Message-ID: <1085410145.5412.98.camel@protein.scalableinformatics.com> Folks: RPM's (including the source RPM) of NCBI Toolkit 2.2.9 are available from http://downloads.scalableinformatics.com/downloads/ncbi . These RPMs are for i686 (pentium 4 and higher), athlon, x86_64, and source. They include our Opteron patch to compile the x86_64 binaries. The athlon and x86_64 RPMs were built on SuSE 9.0, and the i686 was built on RHEL 3.0. Please report any bugs to us. Joe -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 From lindahl at pathscale.com Mon May 24 13:14:43 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Wed Nov 25 01:03:10 2009 Subject: [Beowulf] licensing issues for Fortran compiler on cluster In-Reply-To: <1085407628.5412.88.camel@protein.scalableinformatics.com> References: <40B192B1.2020107@csiro.au> <1085407628.5412.88.camel@protein.scalableinformatics.com> Message-ID: <20040524201443.GB5129@greglaptop.internal.keyresearch.com> On Mon, May 24, 2004 at 10:07:08AM -0400, Joe Landman wrote: > Portland Group (www.pgroup.com) > Intel > > and probably a few others. PathScale also falls into this category -- we charge for support according to the # of users and # of nodes compiled on. Compiled binaries can be run anywhere without any limitation. This is the way most compilers in the non-Linux world are charged for. -- greg From konstantin_kudin at yahoo.com Mon May 24 15:10:43 2004 From: konstantin_kudin at yahoo.com (Konstantin Kudin) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance In-Reply-To: <20040524205514.M61533@scalableinformatics.com> Message-ID: <20040524221043.51913.qmail@web21204.mail.yahoo.com> --- Joe Landman wrote: > > It is unlikely that one can gain much speed from going to 64 bits, > but > > the support for larger memory and unlimited scratch files is very > > worthwhile in itself. > > I have seen in md43, moldy, and a few others, about 20-30% under gcc > recompilation with -m64 on Opteron. For informatics codes it was > about the same. Well, bioinformatics codes presumably run mostly integer operations. This is very different from heavy floating point calculations in G03 which are double precision in all architectures and run with approximately the same speed regardless of 32/64 bitness. > Your mileage will vary of course, but I expect with Gaussian and > others that > overflow memory, the overall system design will be as important (if > not more so) > to the overall performance than the CPU architecture, unless you can > somehow > isolate the computation to never spill to disk. With G03, some types of jobs will mostly be compute bound, and others will be mostly I/O bound. This is reasonably trivial to predict beforehand. I've tested jobs which were compute bound because testing the other side of the equation is more difficult due to more factors. For I/O bound jobs a box with loads of RAM and fast sequential I/O is the best. Something like dual-quad Opteron with 16-32Gb of RAM and 2-4 ATA disks with RAID0 (striping) is a good choice these days. Konstantin __________________________________ Do you Yahoo!? Friends. Fun. Try the all-new Yahoo! Messenger. http://messenger.yahoo.com/ From konstantin_kudin at yahoo.com Tue May 25 05:38:47 2004 From: konstantin_kudin at yahoo.com (Konstantin Kudin) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Opteron & G03 - official (was Re: CCL:Question regarding Mac G5 performance) In-Reply-To: <40B279ED.4000003@scalableinformatics.com> Message-ID: <20040525123847.90866.qmail@web21204.mail.yahoo.com> --- Joe Landman wrote: > > > >>Your mileage will vary of course, but I expect with Gaussian and > >>others that > >>overflow memory, the overall system design will be as important (if > >>not more so) > >>to the overall performance than the CPU architecture, unless you > can > >>somehow > >>isolate the computation to never spill to disk. > >> > >> > > > > With G03, some types of jobs will mostly be compute bound, and > others > >will be mostly I/O bound. This is reasonably trivial to predict > >beforehand. I've tested jobs which were compute bound because > testing > >the other side of the equation is more difficult due to more > factors. > > > > For I/O bound jobs a box with loads of RAM and fast sequential I/O > is > >the best. Something like dual-quad Opteron with 16-32Gb of RAM and > 2-4 > >ATA disks with RAID0 (striping) is a good choice these days. > > > > > > :) I might suggest the 3ware folks for their controllers. Just > pick > your file systems and stripe width carefully. Enough idle chat :-) The G03.C1 with Opteron support seems to be finally out, so it is possible to just run it and find out how it performs in 64-bits vs 32: http://www.gaussian.com/g_tech/g03_rel.htm New Hardware Support in Rev. C.01 Opteron, with 64-bit integers and pointers; Linda is available for this system Konstantin __________________________________ Do you Yahoo!? Friends. Fun. Try the all-new Yahoo! Messenger. http://messenger.yahoo.com/ From landman at scalableinformatics.com Mon May 24 13:58:02 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance In-Reply-To: <20040524174532.14547.qmail@web21203.mail.yahoo.com> References: <20040524174532.14547.qmail@web21203.mail.yahoo.com> Message-ID: <20040524205514.M61533@scalableinformatics.com> On Mon, 24 May 2004 10:45:32 -0700 (PDT), Konstantin Kudin wrote [...] > > I would be curious to see the performance with a 64 bit compilation > > running under a 64 bit OS (SuSE 9.x) > > Me too :-) According to this CCL message (and other reliable sources), > the official 64-bit support is just around the corner: > > http://www.ccl.net/cgi-bin/ccl/message.cgi?2004+04+28+013 > (see reply #2) > > It is unlikely that one can gain much speed from going to 64 bits, but > the support for larger memory and unlimited scratch files is very > worthwhile in itself. I have seen in md43, moldy, and a few others, about 20-30% under gcc recompilation with -m64 on Opteron. For informatics codes it was about the same. Your mileage will vary of course, but I expect with Gaussian and others that overflow memory, the overall system design will be as important (if not more so) to the overall performance than the CPU architecture, unless you can somehow isolate the computation to never spill to disk. -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 From daniel at labtie.mmt.upc.es Mon May 24 12:52:07 2004 From: daniel at labtie.mmt.upc.es (Daniel Fernandez) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] High quality hardware Message-ID: <1085428326.4151.26.camel@qeldroma.cttc.org> Hi you all, We've recently suffered some unsuspected random failures during runs of our CFD cases. Our subsequent mpi tests also showed random results, the first were countinuous "lamd" daemoun hangups. But the next day all nodes ran almost fine with identical test suites. Moreover, now it's giving very few checksum errors or "lamd" hangups in a really reduced set of all nodes, really weird. This headache makes us think that could be: -Continuous run 24h a day of common hardware not prepared to. -Defective mainboard/memory -External inteferences/noise The last argument seemed to gain our attention, in that case what would be the best case material for shielding ? Also, what kind of mainboard manufacturers do you trust most ? I'm referring mainly to Socket A platform, we're currently using Asus and MSI but alse Tyan seems a good option. -- Daniel Fernandez Heat And Mass Transfer Center - CTTC www.cttc.upc.edu c/ Colom n?11 UPC Campus Industrials Terrassa , Edifici TR4 From landman at scalableinformatics.com Mon May 24 15:40:45 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] CCL:Question regarding Mac G5 performance In-Reply-To: <20040524221043.51913.qmail@web21204.mail.yahoo.com> References: <20040524221043.51913.qmail@web21204.mail.yahoo.com> Message-ID: <40B279ED.4000003@scalableinformatics.com> Konstantin Kudin wrote: >--- Joe Landman wrote: > > > >>> It is unlikely that one can gain much speed from going to 64 bits, >>> >>> >>but >> >> >>>the support for larger memory and unlimited scratch files is very >>>worthwhile in itself. >>> >>> >>I have seen in md43, moldy, and a few others, about 20-30% under gcc >>recompilation with -m64 on Opteron. For informatics codes it was >>about the same. >> >> > > Well, bioinformatics codes presumably run mostly integer operations. >This is very different from heavy floating point calculations in G03 >which are double precision in all architectures and run with >approximately the same speed regardless of 32/64 bitness. > > md43 and moldy are molecular dynamics codes. I had been thinking of running some tests with GAMESS, or some other codes just so I have a sense of how electronic structure codes do on the system. For computationally intensive codes, going to 64 bits on the Opteron gets you a) double the number of general purpose registers, b) double the number of SSE registers. This means that the optimizer, if it is dealing with a heavy register pressure code, can do a better job of scheduling the resources. It also means that some codes may be able to leverage more instructions in flight per cycle because of resource availability. The address space is also flat as compared to the segmented space of the 32 bit mode. It is most definitely not a simple case of there being just a 32 vs 64 bit address space. That advantage is there, but it is not the only one. One of the interesting side effects of the NUMA systems has to do with memory bandwidth per CPU as you increase the number of CPUs in a node. For a correctly populated system (e.g. memory evenly distributed among the CPUs), each CPU has full bandwidth to its local memory, and an additional latency hop to remote memory on the same node. If you stack all the memory on a single CPU (as I have seen many folks do, then run benchmarks, and report their results), you share memory bandwidth. In this case, you get the sort of results we see occasionally reported here. Similar results occur if you have a kernel (say an ancient one like 2.4.18) that doesn't know much about NUMA and related scheduling. > > > >>Your mileage will vary of course, but I expect with Gaussian and >>others that >>overflow memory, the overall system design will be as important (if >>not more so) >>to the overall performance than the CPU architecture, unless you can >>somehow >>isolate the computation to never spill to disk. >> >> > > With G03, some types of jobs will mostly be compute bound, and others >will be mostly I/O bound. This is reasonably trivial to predict >beforehand. I've tested jobs which were compute bound because testing >the other side of the equation is more difficult due to more factors. > > For I/O bound jobs a box with loads of RAM and fast sequential I/O is >the best. Something like dual-quad Opteron with 16-32Gb of RAM and 2-4 >ATA disks with RAID0 (striping) is a good choice these days. > > :) I might suggest the 3ware folks for their controllers. Just pick your file systems and stripe width carefully. Joe From douglas at shore.net Mon May 24 16:12:16 2004 From: douglas at shore.net (Douglas O'Flaherty) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Gaussian Support In-Reply-To: <200405241900.i4OJ04Hl026055@bluewest.scyld.com> References: <200405241900.i4OJ04Hl026055@bluewest.scyld.com> Message-ID: <40B28150.6050902@shore.net> There were questions about Gaussian support for Opterons. The release of Gaussian '03 C.01 was Friday and it supports Opterons. see: http://www.gaussian.com No idea about performance or anything else about it. doug From bill at cse.ucdavis.edu Mon May 24 17:12:20 2004 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Good 1 Gbit switches - which ones? In-Reply-To: <20040521194947.27584.qmail@web21204.mail.yahoo.com> References: <20040521194947.27584.qmail@web21204.mail.yahoo.com> Message-ID: <20040525001220.GA29350@cse.ucdavis.edu> On Fri, May 21, 2004 at 12:49:47PM -0700, Konstantin Kudin wrote: > Hi there, > > Can anyone offer an insight with respect to 1 Gbit switches for a > Beowulf cluster? There are all these reports that a lot of inexpensive > switches on the market tend to choke under heavy internal traffic. Can > anyone suggest an affordable switch with good internal bandwidth, which > was tested under heavy load, and actually worked well? I've written a small benchmark which allows testing various number of MPI_INTs in a message between a variable number of pairs of nodes. With a 32 node dual opteron cluster and a Nortel Baystack 470 48 port switch: # of MPI_INT BetweenPairs of Wallclock Latency Bandwidth ============================================================================== size= 1, 131072 hops, 8 nodes in 7.04 sec ( 53.7 us/hop) 73 KB/sec size= 1, 131072 hops, 16 nodes in 7.46 sec ( 56.9 us/hop) 69 KB/sec size= 1, 131072 hops, 24 nodes in 7.51 sec ( 57.3 us/hop) 68 KB/sec size= 1, 131072 hops, 32 nodes in 8.44 sec ( 64.4 us/hop) 61 KB/sec (19% or so drop) size= 10, 131072 hops, 8 nodes in 7.15 sec ( 54.5 us/hop) 716 KB/sec size= 10, 131072 hops, 16 nodes in 7.39 sec ( 56.4 us/hop) 693 KB/sec size= 10, 131072 hops, 24 nodes in 7.59 sec ( 57.9 us/hop) 674 KB/sec size= 10, 131072 hops, 32 nodes in 8.06 sec ( 61.5 us/hop) 635 KB/sec (13% or so drop) size= 1000, 16384 hops, 8 nodes in 1.93 sec (117.8 us/hop) 33163 KB/sec size= 1000, 16384 hops, 16 nodes in 1.96 sec (119.6 us/hop) 32652 KB/sec size= 1000, 16384 hops, 24 nodes in 1.98 sec (120.6 us/hop) 32400 KB/sec size= 1000, 16384 hops, 32 nodes in 2.20 sec (134.1 us/hop) 29129 KB/sec (13% or so drop) size= 10000, 16384 hops, 8 nodes in 9.71 sec (592.5 us/hop) 65930 KB/sec size= 10000, 16384 hops, 16 nodes in 9.92 sec (605.2 us/hop) 64543 KB/sec size= 10000, 16384 hops, 24 nodes in 10.13 sec (618.4 us/hop) 63164 KB/sec size= 10000, 16384 hops, 32 nodes in 17.47 sec (1066.4 us/hop) 36629 KB/sec (80% or so drop) size=100000, 16384 hops, 8 nodes in 100.00 sec (6103.5 us/hop) 64000 KB/sec size=100000, 16384 hops, 16 nodes in 104.72 sec (6391.3 us/hop) 61118 KB/sec size=100000, 16384 hops, 24 nodes in 103.68 sec (6328.0 us/hop) 61730 KB/sec size=100000, 16384 hops, 32 nodes in 134.14 sec (8187.3 us/hop) 47711 KB/sec (34% or so drop) Seems like in all cases I'm seeing a substantial drop off by the time I keep 32 ports busy, I suspect the drop off at 48 would be even worse. Does this seem like a reasonable way to benchmark switches? Anyone have suggested improvments or better tools? If people think this would be valuable I could clean up the source and provide a central location for storing benchmark results. -- Bill Broadley Computational Science and Engineering UC Davis From pjs at eurotux.com Tue May 25 03:56:00 2004 From: pjs at eurotux.com (Paulo Silva) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] MPICH problem Message-ID: <1085482560.14090.6.camel@valen> Hi, I'm having some problems running some mpi programs in a beowulf cluster. The cluster is composed of 12 Linux machines and the compilation of the mpich libraries run well. I've also configured the machines.LINUX file so that it lists all machines available in the cluster. When I try to run some program I get the following error: $ mpirun -np 3 cpi rm_924: p4_error: rm_start: net_conn_to_listener failed: 33064 p0_22381: p4_error: Child process exited while making connection to remote process on a01: 0 /opt/mpich/bin/mpirun: line 1: 22381 Broken pipe /nfshome/ex/cpi -p4pg /nfshome/ex/PI22264 -p4wd /nfshome/ex The /nfshome is a nfs shared directory. The a01 is accessible by rsh. Can someone help me with this error? -- Paulo Silva -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://www.scyld.com/pipermail/beowulf/attachments/20040525/e2062a11/attachment.bin From eugen at leitl.org Tue May 25 06:44:15 2004 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Redmond is at it, again Message-ID: <20040525134414.GU1105@leitl.org> Yeah, think of all those power Excel hydro code users who're going to switch... http://zdnet.com.com/2102-1103_2-5219282.html?tag=printthis Microsoft creating Windows for supercomputers By Stephen Shankland and Ina Fried CNET News.com May 24, 2004, 12:30 PM PT URL: http://zdnet.com.com/2100-1103-5219282.html Microsoft has launched an effort to produce a version of Windows for high-performance computing, a move seen as a direct attack on a Linux stronghold. Bottom line: For now, Linux has the upper hand, owing to its affinity with Unix--the OS environment the high-performance crowd is most comfortable with--and the open-source model, which lets users turn directly to source code for answers to problems. But a Microsoft product would theoretically integrate better with Windows desktop machines, and if the company can serve up an impressive offering, Linux could be in for a tussle. High-performance computing once required massive, expensive, exotic machines from companies such as Cray, but the field is being remade by the arrival of clusters of low-end machines. While the trend could be considered an opportunity for Microsoft, which has long been the leading operating-system company, Linux has actually become the favored software used on these clusters. Now Microsoft has begun its response, forming its High Performance Computing team and planning a new OS version called Windows Server HPC Edition. Kyril Faenov is director of the effort, and Microsoft is hiring new managers, programmers, testers and others. The Redmond, Wash.-based software colossus has its work cut out in the market--and knows it. "Winning in this important space against entrenched Linux/open-source software competition requires creativity, innovation, speed of execution, and deep engagements with hardware, software and academic partners," reads a job posting for a program manager responsible for setting up the team's academic partnerships. In a recent interview, Bob Muglia, a Microsoft senior vice president who leads the development of Windows Server, said the company is interested in two particular areas: building high-performance computing clusters and harvesting the unused processing power of PCs. Although Microsoft is a comparative newcomer to the market, the company could bring several advantages: . Machines running Windows HPC Edition could seamlessly connect to desktop computers, providing instant power for someone such as a financial analyst performing calculations on an Excel spreadsheet, said David Lifka, chief technology officer for the Cornell Theory Center, Microsoft's premier high-performance computing partner. . Microsoft could create a specialized version of its widely praised programming tools, said Phil Papadopoulos, director of the grids and clusters program at the San Diego Supercomputing Center. "Windows could make that much easier with their integrated development environment. They have the manpower to do that piece of the puzzle." . Microsoft could also adapt its popular SQL Server database software to run on high-performance systems. The company has already said the next major version of SQL Server, code-named Yukon and due next year, will include better support for very large databases and for running on clustered systems. . And Microsoft could build software into its desktop version of Windows to harness the power of PCs, letting companies get more value from their computers. It's a technology that's applicable to tasks such as drug discovery and microchip design. The business imperative The high-performance effort doesn't mark the first time Microsoft has tried to head off Linux's progress. With Windows Server 2003, Microsoft released a lower-priced Web server edition, as Linux was growing popular for use on the machines that host Web sites. "The Windows Server group is really focused on countering Linux," said Rob Helm, an analyst with Directions on Microsoft. "They've identified specific areas where Linux has the most traction." The HPC Edition is also an example of a Microsoft strategy to increase revenue by creating versions of Windows tailored for specific market segments--for example, Windows for tablet PCs, digital TV recorders and storage servers. "Another way for them to keep Windows sales moving is to roll out more of these editions," Helm said. "When you've got a product that you need to keep moving, one way to do it is to segment it. You introduce Tarter Control Windows Server and Sensitive Teeth Windows Server." High-performance computing is a lucrative market, with sales that increased 14 percent to $5.3 billion in 2003, according to IDC. And "bright clusters," Linux servers that manufacturers know will be used in a cluster, had sales of $384 million in the fourth quarter. Beating the incumbent But for once, Microsoft is the newcomer, and Linux is the incumbent. Linux got its first foothold in academia and research labs, which already had expertise and software for the functionally similar Unix operating system. "The majority of people doing high-performance computing are a lot more comfortable and efficient inside a Unix environment," a category that includes Linux, the SDSC's Papadopoulos said. To convince people to invest the time and money to switch, Microsoft will have to offer something much better, he said. Linux, boosted by low-cost servers using processors from Intel and Advanced Micro Devices, now is used on prestigious machines. Thunder, a machine at the Lawrence Livermore National Laboratory with 512 Linux servers running Red Hat Enterprise Linux, can perform more than 19 trillion calculations per second, second only to Japan's Earth Simulator. Dozens of machines in a list of the 500 fastest supercomputers run Linux, including five of the top 10. Only two on the list are identified as Windows machines. One reason Windows has been slow to catch on is that Unix and Linux were bred to be administered remotely, a necessary feature for managing a cluster with dozens or hundreds of computers. In Windows, "the notion of remote computing is significantly more difficult than in Unix," Papadopoulos said. "Because Windows was born out of the desktop, (it is) deeply ingrained in the Microsoft culture that you have somebody sitting in front of the machine to do work." Management is on Microsoft's agenda, though. The company is hiring one programmer to work on a "graphical and script-based user interface for efficient job and resource management across large clusters" and another to create "automated infrastructure to uncover performance and reliability problems with high performance, large-scale server applications." Linux adds another advantage: It's open-source software, meaning that anybody may see and modify its underlying source code. Most business customers aren't interested, but high-performance technical computing users need to extract every bit of performance and track down difficult bugs. "The nice thing is that because everything is open, if you have a problem, you can get at the root of the problem in terms of the software. That moves things along quite a bit faster," Papadopoulos said. That openness also makes it easier to accommodate the multitude of different technologies used in the high-performance market but not necessarily in the mainstream computing market, said Brian Stevens, vice president of operating system development for Linux seller Red Hat. Releasing a product Microsoft declined to share schedule information about the HPC Edition, but work is already under way. For example, a software developer kit for HPC Edition will include support for the Message Passing Interface, or MPI, widely used software to let computers in a cluster communicate with one another. The Cornell Theory Center's Lifka believes that an early software development kit for the HPC Edition could arrive as soon as this fall. The center is helping Microsoft develop and test the new software. Microsoft has several upcoming server releases, to which an HPC version of Windows could be added. Service Pack 1 of Windows Server 2003 is due later this year, followed by a more substantive upgrade, code-named R2, slated for 2005. The next major update to Windows, code-named Longhorn, is scheduled to arrive in server form in 2007. According to job postings, Microsoft is adapting MPI to Microsoft's .Net infrastructure. A key foundation of .Net is the C# programming language and the Common Language Runtime, or CLR, which lets C# programs run on a multitude of different systems. Lifka said the first phase will use a version of MPI written for a specific operating system and hardware type. The next foundation will be a version of MPI for the CLR that will let administrators run the same programs on a wide variety of different Windows machines--for example, those using Xeon, Opteron or Itanium processors. So far, programs written for the CLR and .Net aren't as fast as those written for a specific machine, "but we see constant improvement in that," Lifka added. Another area that needs work is security and easy patch installation, he said. Overall, Lifka is a fan of Windows for high-performance computing. The biggest reason for his enthusiasm is that it can dovetail easily with other versions of Windows in a company. And companies are more familiar with Windows than Linux, he added. "Moving to Windows has allowed us to have a greater number and quality of corporate relationships," Lifka said. Microsoft takes a long-term view of the challenge. Muglia often discusses technology moving from possible to practical to seamless, as it matures. High-performance computing on Windows today is in the possible stage, he said, but the goal is to make it practical. "That is something that will happen in the next few years," Muglia said. "There is an opportunity to make this better." -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20040525/7d770ac3/attachment.bin From hahn at physics.mcmaster.ca Tue May 25 09:14:52 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] High quality hardware In-Reply-To: <1085428326.4151.26.camel@qeldroma.cttc.org> Message-ID: > countinuous "lamd" daemoun hangups. But the next day all nodes ran > almost fine with identical test suites. Moreover, now it's giving very but do you have monitoring of, for instance, temperature? lack of repeatability will make your life very difficult. > -Continuous run 24h a day of common hardware not prepared to. which generally equates to temperature. > -Defective mainboard/memory but what you describe is a degredation, no? that is, it used to work fine, but now, sometimes intermittently fails? > -External inteferences/noise are these systems based on bare boards? or multiple boards per chassis? > The last argument seemed to gain our attention, in that case what would > be the best case material for shielding ? mu-metal, I suppose. but that's rather extreme! are you proposing some kind of EMF interference *through* the case? or some kind of exotic noise > Also, what kind of mainboard manufacturers do you trust most ? I'm I go with "A-list" vendors: recognizable vendors, preferably not entirely focused on either low-end or gamer markets. asus, tyan, supermicro, msi, celestica, hp, ibm, apple, dell, etc. > referring mainly to Socket A platform, ah. I wonder if that's your problem, then. socket A has always had a rep for being somewhat fiddly to run stably, and to keep cool. the latter is presumably just because the chips dissipate a fair amount of power, and need rather good contact with a rather good heatsink. unlike intel or recent AMD systems, which have builtin heatspreaders. still, if you have properly mounted, fan-working, copper heatsinks with good through-case airflow, I'd think you could expect stable behavior. > we're currently using Asus and > MSI but alse Tyan seems a good option. those work for me. oddly, I'm getting feedback from OEM channels that Tyan is having trouble stocking/delivering products. that's kind of worrisome, since I tend to like their products... regards, mark hahn. From James.P.Lux at jpl.nasa.gov Tue May 25 09:47:42 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Redmond is at it, again In-Reply-To: <20040525134414.GU1105@leitl.org> Message-ID: <5.2.0.9.2.20040525094418.017b2e90@mailhost4.jpl.nasa.gov> At 03:44 PM 5/25/2004 +0200, Eugen Leitl wrote: >Management is on Microsoft's agenda, though. The company is hiring one >programmer to work on a "graphical and script-based user interface for >efficient job and resource management across large clusters" and another to >create "automated infrastructure to uncover performance and reliability >problems with high performance, large-scale server applications." Wow! one whole body to work on graphical and script based user interfaces! Bill must really be quaking in his boots to invest $200K in this. >According to job postings, Microsoft is adapting MPI to Microsoft's .Net >infrastructure. A key foundation of .Net is the C# programming language and >the Common Language Runtime, or CLR, which lets C# programs run on a >multitude of different systems. Fascinating... A C# binding for MPI? James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 From rmyers1400 at comcast.net Tue May 25 10:03:57 2004 From: rmyers1400 at comcast.net (Robert Myers) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Redmond is at it, again In-Reply-To: <20040525134414.GU1105@leitl.org> References: <20040525134414.GU1105@leitl.org> Message-ID: <40B37C7D.308@comcast.net> Eugen Leitl wrote: >Yeah, think of all those power Excel hydro code users who're going to >switch... > >http://zdnet.com.com/2102-1103_2-5219282.html?tag=printthis > > > >For now, Linux has the upper hand, owing to its affinity with Unix--the OS >environment the high-performance crowd is most comfortable with--and the >open-source model, which lets users turn directly to source code for answers >to problems. But a Microsoft product would theoretically integrate better >with Windows desktop machines, and if the company can serve up an impressive >offering, Linux could be in for a tussle. > Gates will have to explain that personally to the graduate students and postdocs who actually do the work, I would think. I wouldn't put it beyond him to try to do that, actually, and it could conceivably be worth his trouble. > >In a recent interview, Bob Muglia, a Microsoft senior vice president who >leads the development of Windows Server, said the company is interested in >two particular areas: building high-performance computing clusters and >harvesting the unused processing power of PCs. > >Although Microsoft is a comparative newcomer to the market, the company could >bring several advantages: > >. Machines running Windows HPC Edition could seamlessly connect to desktop >computers, providing instant power for someone such as a financial analyst >performing calculations on an Excel spreadsheet, said David Lifka, chief >technology officer for the Cornell Theory Center, Microsoft's premier >high-performance computing partner. > > >. And Microsoft could build software into its desktop version of Windows to >harness the power of PCs, letting companies get more value from their >computers. It's a technology that's applicable to tasks such as drug >discovery and microchip design. > If there is a way to take this seriously, I thought, it is in "harvesting the unused processing power of PC's." The annual market for HPC is about $5 billion and for PC's about $200 billion. Taking the market price of an "HPC CPU" to be ten times that of a "PC CPU" (starting from an estimated $100 million for 10000 Red Storm processors), that would put 400 times as many "PC CPU's" entering service each year as "HPC CPU's". From the point of view of HPC, 400 is not a very interesting number, especially if you account for even the tiniest bit of the difficulty you'd encounter in trying to harness any serious fraction of so many more available CPU's. Even if the price disparity between a "PC CPU" and an "HPC CPU" were more like a hundred, roughly the difference between a mainframe CPU and a PC CPU, that still only makes 4000 times as many available CPU's--still not a very interesting number. Were I Microsoft, I'd be aiming to tell the world that Microsoft is not making money, it is solving the problems of science with the untapped power of PC's. We will probably see something along those lines from Microsoft. If it stood up to scrutiny, it might be in the interests of science, even if not in the interests of those who have built their careers around linux clusters. It's a great marketing line, and, if anyone would have the resources to make such a thing happen, it would be Microsoft. Thinking again rather cynically in the interests of science, having Microsoft adopt such a marketing posture and making it stick to any extent at all might actually be a good thing, since it would be difficult for Redmond to be a fairly obvious market predator at the same time as it is being the salvation of mankind. Had Microsoft the wisdom to see it though, it could probably work around the graduate student/postdoc problem by artful use of Windows Services for Unix. A _really_ smooth Microsoft would even give the appearance of supporting the use of Linux for science while in reality using that support as a way of keeping Linux contained. RM From lathama at yahoo.com Tue May 25 10:05:19 2004 From: lathama at yahoo.com (Andrew Latham) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Good 1 Gbit switches - which ones? In-Reply-To: <20040525001220.GA29350@cse.ucdavis.edu> Message-ID: <20040525170519.57986.qmail@web60302.mail.yahoo.com> Nice work I think we could really use a "Mikes Hardware" of the HPC world. I use the list to figure this stuff out but I think that a (unbiased) hardware review and comparison website centered around HPC products would be great. I will help out on that. --- Bill Broadley wrote: > On Fri, May 21, 2004 at 12:49:47PM -0700, Konstantin Kudin wrote: > > Hi there, > > > > Can anyone offer an insight with respect to 1 Gbit switches for a > > Beowulf cluster? There are all these reports that a lot of inexpensive > > switches on the market tend to choke under heavy internal traffic. Can > > anyone suggest an affordable switch with good internal bandwidth, which > > was tested under heavy load, and actually worked well? > > I've written a small benchmark which allows testing various number of > MPI_INTs in a message between a variable number of pairs of nodes. > > With a 32 node dual opteron cluster and a Nortel Baystack 470 48 port > switch: > > # of > MPI_INT BetweenPairs of Wallclock Latency Bandwidth > ============================================================================== > size= 1, 131072 hops, 8 nodes in 7.04 sec ( 53.7 us/hop) 73 KB/sec > size= 1, 131072 hops, 16 nodes in 7.46 sec ( 56.9 us/hop) 69 KB/sec > size= 1, 131072 hops, 24 nodes in 7.51 sec ( 57.3 us/hop) 68 KB/sec > size= 1, 131072 hops, 32 nodes in 8.44 sec ( 64.4 us/hop) 61 KB/sec > (19% or so drop) > > size= 10, 131072 hops, 8 nodes in 7.15 sec ( 54.5 us/hop) 716 KB/sec > size= 10, 131072 hops, 16 nodes in 7.39 sec ( 56.4 us/hop) 693 KB/sec > size= 10, 131072 hops, 24 nodes in 7.59 sec ( 57.9 us/hop) 674 KB/sec > size= 10, 131072 hops, 32 nodes in 8.06 sec ( 61.5 us/hop) 635 KB/sec > (13% or so drop) > > size= 1000, 16384 hops, 8 nodes in 1.93 sec (117.8 us/hop) 33163 KB/sec > size= 1000, 16384 hops, 16 nodes in 1.96 sec (119.6 us/hop) 32652 KB/sec > size= 1000, 16384 hops, 24 nodes in 1.98 sec (120.6 us/hop) 32400 KB/sec > size= 1000, 16384 hops, 32 nodes in 2.20 sec (134.1 us/hop) 29129 KB/sec > (13% or so drop) > > size= 10000, 16384 hops, 8 nodes in 9.71 sec (592.5 us/hop) 65930 KB/sec > size= 10000, 16384 hops, 16 nodes in 9.92 sec (605.2 us/hop) 64543 KB/sec > size= 10000, 16384 hops, 24 nodes in 10.13 sec (618.4 us/hop) 63164 KB/sec > size= 10000, 16384 hops, 32 nodes in 17.47 sec (1066.4 us/hop) 36629 KB/sec > (80% or so drop) > > size=100000, 16384 hops, 8 nodes in 100.00 sec (6103.5 us/hop) 64000 KB/sec > size=100000, 16384 hops, 16 nodes in 104.72 sec (6391.3 us/hop) 61118 KB/sec > size=100000, 16384 hops, 24 nodes in 103.68 sec (6328.0 us/hop) 61730 KB/sec > size=100000, 16384 hops, 32 nodes in 134.14 sec (8187.3 us/hop) 47711 KB/sec > (34% or so drop) > > Seems like in all cases I'm seeing a substantial drop off by the time > I keep 32 ports busy, I suspect the drop off at 48 would be even worse. > > Does this seem like a reasonable way to benchmark switches? Anyone > have suggested improvments or better tools? If people think this would > be valuable I could clean up the source and provide a central location > for storing benchmark results. > > -- > Bill Broadley > Computational Science and Engineering > UC Davis > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ===== *----------------------------------------------------------* Andrew Latham AKA: LATHAMA (lay-th-ham-eh) - LATHAMA.COM LATHAMA@LATHAMA.COM - LATHAMA@YAHOO.COM If yahoo.com is down we have bigger problems than my email! *----------------------------------------------------------* From eugen at leitl.org Tue May 25 11:07:24 2004 From: eugen at leitl.org (Eugen Leitl) Date: Wed Nov 25 01:03:11 2009 Subject: CCL:[Beowulf] Question regarding Mac G5 performance (fwd from mmccallum@pacific.edu) Message-ID: <20040525180724.GY1105@leitl.org> ----- Forwarded message from Mike McCallum ----- From: Mike McCallum Date: Tue, 25 May 2004 10:40:55 -0700 To: chemistry@ccl.net Cc: Eugen Leitl Subject: Re: CCL:[Beowulf] Question regarding Mac G5 performance X-Mailer: Apple Mail (2.613) I'm sorry that this response is so long in coming --- I had forgotten that I was filtering messages to a different mbox! On May 20, 2004, at 01:54, Eugen Leitl wrote: > >Any actual numbers would be very useful, ideally with the compilers, >compiler options, motherboard, and similar. Did you compare 1 job per >node vs 1 job per node? Or 2 jobs per node vs 2 jobs per node? 4 or 8 >dimms? Which kind of memory PC2700? PC3200? > The numbers I have are for NAMD2, using one of my own jobs. This was with the stock PC3200, 512M. I'm asking Rick V. for his input files for his benchmarks, so we can make a more direct comparison in the types of stuff I'm interested in. I'm going to try and borrow another dual G5 or two so I can test the gigabit enet scaling. >>The built-in Gigabit enet is attractive, also, as charmm and NAMD >>scale >>very well with gigabit, and it makes myrinet less price-effective >>(when >>used on any platform, wintel included, see >>http://biobos.nih.gov/apps/charmm/charmmdoc/Bench/c30b1.html for >>example). I decided that dual G5 xserve cluster nodes with gigabit > >I come to a different conclusion based on those graphs. In the first >graph myrinet improves by a factor of 2.6 (250 -> 95 seconds) from 2 >processors to 8, where gige improves by only 20% (255 -> 210). In the >second graph gigE gets SLOWER from 2 to 8 processors. Do you think in >either case the 8 node (let alone 16) gige cluster would have better >price/performance then a myrinet cluster? I realize after actually looking at that URL that that wasn't the data I was thinking about when I wrote that message. I'm struggling to find the URL I was thinking of --- it mentioned the very poor performance of wintel systems with gigE, and mentioned that the G5 system architecture didn't suffer from the same. I realize this is useful information, so I'll find it! > >Seems like for applications like shown on that page you shouldn't >really bother with a cluster over 2 nodes with gigE, not many people >would be willing to pay a factor of 4 more for 20% or even negative >scaling. > >>switches were much more cost-effective for me than any other >>processor, > >Cost effective = price/performance? Can you make any numbers >available? It boiled down to the cost of the smallest myrinet switch + cards cost almost as much as 5 more 2.0GHz G5s. Add that to the comparable price of the wintel hardware (P4s, I'm not talking itaniums here, those guys are even more expensive), and it was a no brainer, even with the slightly poorer scaling. Also realize that I'm talking about small clusters, here, not > 16 nodes. > >>especially any high-bandwidth specialty comm method (apple's gigabit >>has a pretty low latency also). > >Oh? Can you share the apple gigabit numbers? What is "pretty low"? > >>Additional considerations for us were the BSD environment which is >>more >>secure than windows, and the OS is arguably more stable and supported > >I'd agree with more stable and supported for use as a desktop, I'd >disagree with stable and supported as computational node. OSX is the >new player on the block in this space. Do you really think you would >get a good response from calling apple's tech support line when the >scheduler or network stack isn't performing to your performance >expectations? > >Certainly a very reasonable thing. > Actually, Apple has been very responsive and interested in making things work. I've already been over to Cupertino twice, and met with engineers here(on my campus) twice. I don't even have a cluster yet. To be fair, I don't have the resources (time, students or $$) to analyze the code to the nth degree, and find out why the network stack isn't doing 100 mph. I'm pretty much forced to analyze stock situations, because that is all my situation allows me to do. This clearly puts much more weight on the "familiarity" aspect, as I can't afford a whole lot of time spent under the hood, so to speak. Cheers, Mike -- C. Michael McCallum http://chem.cop.uop.edu/cmmccallum.html Associate Professor Department of Chemistry, UOP mmccallum .at. pacific .dot. edu (209) 946-2636 v / (209) 946-2607 fax ----- End forwarded message ----- -- Eugen* Leitl leitl ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20040525/06448eda/attachment.bin From idooley at isaacdooley.com Tue May 25 12:06:24 2004 From: idooley at isaacdooley.com (Isaac Dooley) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] MPICH problem In-Reply-To: <200405251842.i4PIgpSq005667@bluewest.scyld.com> References: <200405251842.i4PIgpSq005667@bluewest.scyld.com> Message-ID: <40B39930.6060408@isaacdooley.com> I've gotten similar problems on a linux Xeon cluster with ethernet, and mpich2-0.96p2. I ended up just using mpich-1.2.5.2. Both were compiled from source with gcc(for c mpi programs). With version 2-0.96p2 I could not get any sample program to run on more than a single node(which incidently worked), even those that just initialize MPI and don't do any real message passing. Which version are you using? Isaac Dooley >I'm having some problems running some mpi programs in a beowulf cluster. >The cluster is composed of 12 Linux machines and the compilation of the >mpich libraries run well. I've also configured the machines.LINUX file >so that it lists all machines available in the cluster. When I try to >run some program I get the following error: > >$ mpirun -np 3 cpi >rm_924: p4_error: rm_start: net_conn_to_listener failed: 33064 >p0_22381: p4_error: Child process exited while making connection to >remote process on a01: 0 >/opt/mpich/bin/mpirun: line 1: 22381 Broken >pipe /nfshome/ex/cpi -p4pg /nfshome/ex/PI22264 >-p4wd /nfshome/ex > >The /nfshome is a nfs shared directory. The a01 is accessible by rsh. >Can someone help me with this error? From atp at piskorski.com Tue May 25 12:37:45 2004 From: atp at piskorski.com (Andrew Piskorski) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] how useful is bproc, and what does Scyld cost? Message-ID: <20040525193745.GB75378@piskorski.com> I'm tentatively planning a small cluster that might or might not actually get built. My current plan is somewhere from 5-20 nodes, 1-2 x86 CPUs per node (exact CPU flavor undecided), gigabit ethernet, and all nodes either entirely diskless, or using 1 IDE disk solely for swap and /tmp. I would prefer to have as much as posible of the cluster software infrastructure Just Work, rather than having to spend lots of time rolling my own. (I will be spending enough time on the custom software I actually want to RUN on the cluster as is.) I am, of course, quite willing to select hardware in order to make the software job easier on myself. Since I want to go diskless anyway, so far I am also leaning towards a bproc based cluster. I only know of two bproc-based cluster distributions, Scyld and Clustermatic. Scyld is commerical and costs money, Clustermatic is not and does not. Are there any others? In particular, are there any Debian based systems that play nicely out of the box with bproc? How much time and effort is Scyld actually going to save me over using Clustermatic? How much is either going to save me over completely rolling my own, preferably using Debian rather than the old and outdated versions of Red Hat that Scyld and Clustermatic seem to use? Also, are there any major drawbacks or snafus I should worry about in going down the bproc route? Finally, just what DOES Scyld actually cost? Can anyone give me a rough idea? >From Scyld's website, I can't tell whether they charge 50 cents or $5,000 per node, and the Scyld/Penguin salesman seemed unable to spit out any kind of ballpark price at all. AFAICT, Scyld seems to expect you to first actually build your cluster, and then send them your cluster's complete hardware specs, down to the smallest detail, in order to get any kind of quote! Thanks! -- Andrew Piskorski http://www.piskorski.com/ From rgb at phy.duke.edu Tue May 25 14:20:51 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Redmond is at it, again In-Reply-To: <5.2.0.9.2.20040525094418.017b2e90@mailhost4.jpl.nasa.gov> Message-ID: On Tue, 25 May 2004, Jim Lux wrote: > At 03:44 PM 5/25/2004 +0200, Eugen Leitl wrote: > > >Management is on Microsoft's agenda, though. The company is hiring one > >programmer to work on a "graphical and script-based user interface for > >efficient job and resource management across large clusters" and another to > >create "automated infrastructure to uncover performance and reliability > >problems with high performance, large-scale server applications." > > Wow! one whole body to work on graphical and script based user > interfaces! Bill must really be quaking in his boots to invest $200K in this. > > > >According to job postings, Microsoft is adapting MPI to Microsoft's .Net > >infrastructure. A key foundation of .Net is the C# programming language and > >the Common Language Runtime, or CLR, which lets C# programs run on a > >multitude of different systems. > > Fascinating... A C# binding for MPI? There is a common and obvious thread to M$ choices under nearly all circumstances. It can easily be used to interpret the specific decisions to make MPI "their own" and eschew TCP/IP, vendor network drivers, ANSI standard C or C++ or f-whatever, and especially to interface it with their own version of core network services: Standards are bad, unless we own them and can alter them on a whim to effectively clone, co-opt, and eliminate all competition from any products ever developed that are based on those "standards" and actually make money. The key words here being "make money" of course. Microsoft won't ever jump on a cluster computing bandwagon unless it is running on wheels they own and all of the seats have a highly proprietary "eject" button somewhere they can use to get rid of all the rest of the instruments. A corollary of the above is that "Java is Evil" (an opinion shared by at least some people who don't work for M$:-). Of course, so is nearly everything else used in modern networks for the transport layer and most of the layers above transport in the good old ISO/OSI scheme, so is http, html, xml, php, perl, python and so forth, and Open Source, Open Standard software is The Devil Himself -- to M$. Current systems corporations are thinking "license/lease" and "renewable revenue stream" instead of "software sales" as the latter era of the computer revolution draws to a slow end. Red Hat has gone there. Sun is well along the road. Microsoft is stuck between a fervent wish to go there and somehow preserve their high margins and the realities of the expectations of their PC customers that they are "buying a copy" of the operating system, not the right to use it for a year with automated prepaid updates. In the corporate world, however, they've moved a long ways there and I feel confident that they'll go the rest of the way soon. So the interesting question is "where's the money" in this move. The HPC market hasn't proven to be exactly a get-rich-quick proposition for any of the various groups that have prospected in it. I suspect (given the reference to "resource management" and "server applications") that it is seeking to plunder the relatively deep pockets and relatively non-computer-savvy researchers in the biocluster and medicluster market, with their bet backed by the proposition that they can reuse their efforts in a pinch in the HA market (those "server applications"). Bioclusters and biogrids as a market probably does total millions of dollars per year -- chickenfeed for M$ at this point -- but it also has some growth potential and integration potential for the future, when we reach the point where e.g. individuals are bioassayed in real time in their Dr.'s office, their genes are matched against a huge database scanning for defects and predispositions and damage and who knows what else, and genetically matched and tuned therapies are prescribed, where dog pedigrees and likelyhood of developing hip dysplasia are similarly determined in real time. Of course, Moore's law and so forth may eat this market before it fully matures -- in three or four years at expected rates desktop PCs will have (possibly multiple) TB size LOCAL SINGLE disks, multiGB of RAM standard, and at least gigabit interfaces standard with 10 Gbps not uncommon. With 10+ GHz CPU clocks, 64 bit and better data paths, really slick memory and device management, and perhaps some onboard parallelism, would one NEED a cluster to run such a (relatively simple) application? If not, it leaves one back with just the research market, and that is, I think, a hopeless case for M$ to make back their not inconsiderable development and sales costs in without a hedge. They would at best break even and at worst would both lose money and worse, lose money EMBARRASSINGLY and publically. As this list well knows, real, transparent, "windows-level" scalability is an elusive beast in the parallel HPC world, although you can find niches where it is possible. And there is no mass market here, per se -- even if you can find somebody crazy enough to pay out LARGE software costs that scale per node, there simply aren't that many customers, and most of those customers always have several proven alternatives with costs that DON'T scale per node. Those customers will pay for support, they'll pay for service, but they don't like paying for software and REALLY hate paying for software and getting lousy service and poor support to go with it. Plenty of people make money on service and support, but they don't get (unduly) rich and they work for a living earning what they earn. rgb > > > James Lux, P.E. > Spacecraft Telecommunications Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From daniel at labtie.mmt.upc.es Tue May 25 12:40:25 2004 From: daniel at labtie.mmt.upc.es (Daniel Fernandez) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] High quality hardware In-Reply-To: References: Message-ID: <1085514025.5872.9.camel@qeldroma.cttc.org> On Tue, 2004-05-25 at 18:14, Mark Hahn wrote: > > countinuous "lamd" daemoun hangups. But the next day all nodes ran > > almost fine with identical test suites. Moreover, now it's giving very > > but do you have monitoring of, for instance, temperature? lack of > repeatability will make your life very difficult. I'm running continuous tests, and saving results , there's only one node out of range running a CPU temperature of 60?C ( crashed powersource fan ) but its *not* on the group of nodes that causes most of problems. > > -Continuous run 24h a day of common hardware not prepared to. > > which generally equates to temperature. > > > -Defective mainboard/memory > > but what you describe is a degredation, no? that is, it used to work fine, > but now, sometimes intermittently fails? > > > -External inteferences/noise > > are these systems based on bare boards? or multiple boards per chassis? > > > The last argument seemed to gain our attention, in that case what would > > be the best case material for shielding ? > > mu-metal, I suppose. but that's rather extreme! are you proposing some > kind of EMF interference *through* the case? or some kind of exotic noise > > > Also, what kind of mainboard manufacturers do you trust most ? I'm > > I go with "A-list" vendors: recognizable vendors, preferably not entirely > focused on either low-end or gamer markets. asus, tyan, supermicro, msi, > celestica, hp, ibm, apple, dell, etc. > > > referring mainly to Socket A platform, > > ah. I wonder if that's your problem, then. socket A has always had a rep > for being somewhat fiddly to run stably, and to keep cool. the latter is > presumably just because the chips dissipate a fair amount of power, and need > rather good contact with a rather good heatsink. unlike intel or recent AMD > systems, which have builtin heatspreaders. still, if you have properly > mounted, fan-working, copper heatsinks with good through-case airflow, > I'd think you could expect stable behavior. Our CPUs ( XP 2600+ Thoroughbred/Barton ) are working at a temperature range of 44-48 C? and our ambient temperature is about 21-23 ?C ... I don't consider it too high, maybe I am wrong and this range of temperatures don't assure a 100% fail-free 365x24h run. > > we're currently using Asus and > > MSI but alse Tyan seems a good option. > > those work for me. oddly, I'm getting feedback from OEM channels that Tyan > is having trouble stocking/delivering products. that's kind of worrisome, > since I tend to like their products... > > regards, mark hahn. > > Cheers. -- Daniel Fernandez Heat And Mass Transfer Center - CTTC www.cttc.upc.edu c/ Colom n?11 UPC Campus Industrials Terrassa , Edifici TR4 From agrajag at dragaera.net Tue May 25 13:56:55 2004 From: agrajag at dragaera.net (Sean Dilda) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] how useful is bproc, and what does Scyld cost? In-Reply-To: <20040525193745.GB75378@piskorski.com> References: <20040525193745.GB75378@piskorski.com> Message-ID: <1085518615.4397.20.camel@pel> There are some pros and cons to bproc (just like everything else). My experience with bproc is limited to some older versions of Scyld. I've never used clustermatic. In my opinion, if you are already used the tricks to simplify maintaining a decent sized number machines, then bproc hurts your more than it helps you. However, if you're not used to using things like kickstart and yum to quickly add nodes and keep them up to date, then a bproc system can make your life much easier. There are definately some pitfalls to BProc. As it was when I last used it (late 2002), BProc systems read the binary off the master node, then caused it to be run on the slave node (I'm simplifying things here) without copying the binary onto the slave node. This means there's not really any filesystem you need to keep up on the slave node, which is a big plus. On the down side, there's no file system on the slave node. This means that if there are any files your program wants to access, you're out of luck. You have to know what they are beforehand and either copy them over, or have them exported by NFS. In your case, you're writing a custom app, so its possible that you can work around the BProc limitations without much trouble. But then again, your problem may not allow that. BProc's downfalls tend to manifest the most with 3rd party apps. As far as I know, Scyld and Clustermatic are the only distros with BProc. At one point Progeny was talking about doing some stuff with BProc, but I don't know what ever came of that. Another thing to note is that clustermatic and scyld probably don't have the same version of bproc, beoboot, etc. Some time back new versions of the software was coming out of Los Alamos, but Scyld was reluctant to use those newer versions, and tended to keep hacking on older versions. So Scyld may have some features clustermatic doesn't have, and vice versa. I don't think I can speak on how much Scyld costs. If you're more interesting in having a turn-key cluster solution, I have some friends who are happy with Rocks (http://www.rocksclusters.org/), which is free. However, I don't know how well that'll work with the diskless setup you want. As far as I know, Scyld/clustermatic are the only ones that'll have a diskless setup "just work". If that's what's most important to you, then I say give them a try. Otherwise, you may want to look around. Again I'll note, all my info is about a year and a half old. Things may (or may not) have changed since then. From laytonjb at charter.net Tue May 25 14:42:06 2004 From: laytonjb at charter.net (Jeffrey B. Layton) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Redmond is at it, again In-Reply-To: <20040525134414.GU1105@leitl.org> References: <20040525134414.GU1105@leitl.org> Message-ID: <40B3BDAE.4090400@charter.net> For some people, me being one of them, this is a HUGE threat. Let me explain. Redhat, Suse, etc. have gone to a per-CPU price for license and support driving up the costs of installing the OS on a cluster. In fact, the last quote I got for a 72 node cluster averaged about $400 a node! Now management has started to compare this price to Windows and guess what? - Windows is cheaper on a per node basis. This includes purchasing and support! We already have a massive license with MS so we get Windows pretty cheap, but I can get Windows myself cheaper than $400 a node. We started to talk to management about the support costs, etc. but they argue that they already have a whole bunch of admins trained in Windows so support costs should be low (I tried to explain the difference between admining a cluster and admining a Windows server). Plus we have an infrastructure to push updates to Windows machines, so they though updates would be a no-brainer as well. At the ClusterWorld conference I tried to talk to various distro vendors about pricing, but they weren't listening. I don't mind paying a relatively small fee to purchase a distro, but paying a support costs for each CPU is nuts! If I have a problem on one node, I have a problem on ALL nodes. The distro companies have yet to understand this. Novell is starting to come around a bit, but they still aren't there yet. We are encouraging cluster vendors to support at least one open distro. That way we aren't held hostage to these companies that don't understand HPC. I don't know if we're making any head way, but I constantly hope we are. We're also trying to push IT management into "approving" a couple of open distros, but they are reluctant because they don't see a company behind them (they insist on the "one throat to choke" model of support). However, I'm still hopefully. If you work for a disro company and you are reading this, all I can say is, WAKE UP! You are going down the tubes fast in the HPC market. If you work for a cluster vendor and you are reading this, please push your management hard to adopt at least one open distro. We'll pay for support for it, but not using the current pricing scheme that the Redhat's, Suse, etc. are charging. If you work for a company that sells commercial software for Linux (e.g. compilers, MPI, etc.), please support more than RHEL and SLES! Think seriously about supporting an open distro and also supporting a kernel/glibc combo. I'm sure I'm not the only one who feels this way. Let's tell the distro companies that we won't stand for it! Thanks! Jeff >Yeah, think of all those power Excel hydro code users who're going to >switch... > >http://zdnet.com.com/2102-1103_2-5219282.html?tag=printthis > >Microsoft creating Windows for supercomputers >By Stephen Shankland and Ina Fried >CNET News.com >May 24, 2004, 12:30 PM PT >URL: http://zdnet.com.com/2100-1103-5219282.html > >Microsoft has launched an effort to produce a version of Windows for >high-performance computing, a move seen as a direct attack on a Linux >stronghold. > >Bottom line: >For now, Linux has the upper hand, owing to its affinity with Unix--the OS >environment the high-performance crowd is most comfortable with--and the >open-source model, which lets users turn directly to source code for answers >to problems. But a Microsoft product would theoretically integrate better >with Windows desktop machines, and if the company can serve up an impressive >offering, Linux could be in for a tussle. > >High-performance computing once required massive, expensive, exotic machines >from companies such as Cray, but the field is being remade by the arrival of >clusters of low-end machines. While the trend could be considered an >opportunity for Microsoft, which has long been the leading operating-system >company, Linux has actually become the favored software used on these >clusters. > >Now Microsoft has begun its response, forming its High Performance Computing >team and planning a new OS version called Windows Server HPC Edition. Kyril >Faenov is director of the effort, and Microsoft is hiring new managers, >programmers, testers and others. > >The Redmond, Wash.-based software colossus has its work cut out in the >market--and knows it. > >"Winning in this important space against entrenched Linux/open-source >software competition requires creativity, innovation, speed of execution, and >deep engagements with hardware, software and academic partners," reads a job >posting for a program manager responsible for setting up the team's academic >partnerships. > >In a recent interview, Bob Muglia, a Microsoft senior vice president who >leads the development of Windows Server, said the company is interested in >two particular areas: building high-performance computing clusters and >harvesting the unused processing power of PCs. > >Although Microsoft is a comparative newcomer to the market, the company could >bring several advantages: > >. Machines running Windows HPC Edition could seamlessly connect to desktop >computers, providing instant power for someone such as a financial analyst >performing calculations on an Excel spreadsheet, said David Lifka, chief >technology officer for the Cornell Theory Center, Microsoft's premier >high-performance computing partner. > >. Microsoft could create a specialized version of its widely praised >programming tools, said Phil Papadopoulos, director of the grids and clusters >program at the San Diego Supercomputing Center. "Windows could make that much >easier with their integrated development environment. They have the manpower >to do that piece of the puzzle." > >. Microsoft could also adapt its popular SQL Server database software to run >on high-performance systems. The company has already said the next major >version of SQL Server, code-named Yukon and due next year, will include >better support for very large databases and for running on clustered systems. > >. And Microsoft could build software into its desktop version of Windows to >harness the power of PCs, letting companies get more value from their >computers. It's a technology that's applicable to tasks such as drug >discovery and microchip design. > >The business imperative >The high-performance effort doesn't mark the first time Microsoft has tried >to head off Linux's progress. With Windows Server 2003, Microsoft released a >lower-priced Web server edition, as Linux was growing popular for use on the >machines that host Web sites. > >"The Windows Server group is really focused on countering Linux," said Rob >Helm, an analyst with Directions on Microsoft. "They've identified specific >areas where Linux has the most traction." > >The HPC Edition is also an example of a Microsoft strategy to increase >revenue by creating versions of Windows tailored for specific market >segments--for example, Windows for tablet PCs, digital TV recorders and >storage servers. > >"Another way for them to keep Windows sales moving is to roll out more of >these editions," Helm said. "When you've got a product that you need to keep >moving, one way to do it is to segment it. You introduce Tarter Control >Windows Server and Sensitive Teeth Windows Server." > >High-performance computing is a lucrative market, with sales that increased >14 percent to $5.3 billion in 2003, according to IDC. And "bright clusters," >Linux servers that manufacturers know will be used in a cluster, had sales of >$384 million in the fourth quarter. > >Beating the incumbent >But for once, Microsoft is the newcomer, and Linux is the incumbent. Linux >got its first foothold in academia and research labs, which already had >expertise and software for the functionally similar Unix operating system. > >"The majority of people doing high-performance computing are a lot more >comfortable and efficient inside a Unix environment," a category that >includes Linux, the SDSC's Papadopoulos said. To convince people to invest >the time and money to switch, Microsoft will have to offer something much >better, he said. > >Linux, boosted by low-cost servers using processors from Intel and Advanced >Micro Devices, now is used on prestigious machines. Thunder, a machine at the >Lawrence Livermore National Laboratory with 512 Linux servers running Red Hat >Enterprise Linux, can perform more than 19 trillion calculations per second, >second only to Japan's Earth Simulator. > >Dozens of machines in a list of the 500 fastest supercomputers run Linux, >including five of the top 10. Only two on the list are identified as Windows >machines. > >One reason Windows has been slow to catch on is that Unix and Linux were bred >to be administered remotely, a necessary feature for managing a cluster with >dozens or hundreds of computers. > >In Windows, "the notion of remote computing is significantly more difficult >than in Unix," Papadopoulos said. "Because Windows was born out of the >desktop, (it is) deeply ingrained in the Microsoft culture that you have >somebody sitting in front of the machine to do work." > >Management is on Microsoft's agenda, though. The company is hiring one >programmer to work on a "graphical and script-based user interface for >efficient job and resource management across large clusters" and another to >create "automated infrastructure to uncover performance and reliability >problems with high performance, large-scale server applications." > >Linux adds another advantage: It's open-source software, meaning that anybody >may see and modify its underlying source code. Most business customers aren't >interested, but high-performance technical computing users need to extract >every bit of performance and track down difficult bugs. > >"The nice thing is that because everything is open, if you have a problem, >you can get at the root of the problem in terms of the software. That moves >things along quite a bit faster," Papadopoulos said. > >That openness also makes it easier to accommodate the multitude of different >technologies used in the high-performance market but not necessarily in the >mainstream computing market, said Brian Stevens, vice president of operating >system development for Linux seller Red Hat. > >Releasing a product >Microsoft declined to share schedule information about the HPC Edition, but >work is already under way. > >For example, a software developer kit for HPC Edition will include support >for the Message Passing Interface, or MPI, widely used software to let >computers in a cluster communicate with one another. > >The Cornell Theory Center's Lifka believes that an early software development >kit for the HPC Edition could arrive as soon as this fall. The center is >helping Microsoft develop and test the new software. > >Microsoft has several upcoming server releases, to which an HPC version of >Windows could be added. Service Pack 1 of Windows Server 2003 is due later >this year, followed by a more substantive upgrade, code-named R2, slated for >2005. The next major update to Windows, code-named Longhorn, is scheduled to >arrive in server form in 2007. > >According to job postings, Microsoft is adapting MPI to Microsoft's .Net >infrastructure. A key foundation of .Net is the C# programming language and >the Common Language Runtime, or CLR, which lets C# programs run on a >multitude of different systems. > >Lifka said the first phase will use a version of MPI written for a specific >operating system and hardware type. The next foundation will be a version of >MPI for the CLR that will let administrators run the same programs on a wide >variety of different Windows machines--for example, those using Xeon, Opteron >or Itanium processors. > >So far, programs written for the CLR and .Net aren't as fast as those written >for a specific machine, "but we see constant improvement in that," Lifka >added. Another area that needs work is security and easy patch installation, >he said. > >Overall, Lifka is a fan of Windows for high-performance computing. The >biggest reason for his enthusiasm is that it can dovetail easily with other >versions of Windows in a company. > >And companies are more familiar with Windows than Linux, he added. "Moving to >Windows has allowed us to have a greater number and quality of corporate >relationships," Lifka said. > >Microsoft takes a long-term view of the challenge. > >Muglia often discusses technology moving from possible to practical to >seamless, as it matures. High-performance computing on Windows today is in >the possible stage, he said, but the goal is to make it practical. > >"That is something that will happen in the next few years," Muglia said. >"There is an opportunity to make this better." > > > >------------------------------------------------------------------------ > >_______________________________________________ >Beowulf mailing list, Beowulf@beowulf.org >To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > From ed at ednevitible.co.uk Tue May 25 15:04:51 2004 From: ed at ednevitible.co.uk (Ed) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Possibly OT Message-ID: <20040525230451.6411350e@barton> Hi, Im very new to clustering. I have a perhaps very simple question: I have to provide a webhost, but with low spec computers which we have in abndance. Now, I assume that the webservice will use the single IP address x.x.x.x. However, I have not got a clue how to go about this. Should I aim to have two single computers linked via a serial interface, or should I do something more adventurous such as link them via a network interface. Am I asking on a list where Beowulf is a API and only certain programs will work with Beowulf? This is probably a question that has been asked many times. But if Beowulf can control computers and spawn instances of daemons for mail as mail arrives it would also solve many of the mail problems which we currently face in the small hosting organisation where I work. Currently I spend many hours researching this question, however fruitless. From landman at scalableinformatics.com Tue May 25 16:57:04 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Redmond is at it, again In-Reply-To: References: Message-ID: <1085529423.2873.76.camel@protein.scalableinformatics.com> On Tue, 2004-05-25 at 17:20, Robert G. Brown wrote: > > Fascinating... A C# binding for MPI? > > There is a common and obvious thread to M$ choices under nearly all > circumstances. It can easily be used to interpret the specific > decisions to make MPI "their own" and eschew TCP/IP, vendor network > drivers, ANSI standard C or C++ or f-whatever, and especially to > interface it with their own version of core network services: > > Standards are bad, unless we own them and can alter them on a whim to > effectively clone, co-opt, and eliminate all competition from any > products ever developed that are based on those "standards" and > actually make money. Not an apologist for anyone/any corp Somehow I have the sense that many folks simply attribute nefarious intent to various corporate entities, though there are simpler explanations. Profit motivation drives most companies, apart from those whose business model involves giving away their value to their consumers in an uncompensated manner. Microsoft, like all corporations whose goals are to make money, needs to review from time to time, its mix of product, its market, etc. Failure to do so can be (and has been for some companies) fatal. Heinlein said something apt about this which could be paraphrased, Never attribute to malicious intent, that which can be explained by cluelessness. Corporations are not (for the most part) malevolent entities out to destroy all. They (for the most part) have easy to discern goals and objectives. In fact they are eminently predictable (for the most part). > The key words here being "make money" of course. Microsoft won't ever > jump on a cluster computing bandwagon unless it is running on wheels > they own and all of the seats have a highly proprietary "eject" button > somewhere they can use to get rid of all the rest of the instruments. Is this in any way different from GE's strategy of being number 1 or number 2 in every market they are in? All corporations (the ones that want to make money) seek to maximize profitable revenue. There is nothing wrong with this as a goal. For the larger folks (not the smaller fish like us), this means controlling the market. While controlling the market is great for the shareholders, it isn't so great for the consumers. Competition in a market is always a good thing. > A corollary of the above is that "Java is Evil" (an opinion shared by at > least some people who don't work for M$:-). Of course, so is nearly heh... more than a few. "If all you have is a hammer, then every problem looks like a nail." > everything else used in modern networks for the transport layer and most > of the layers above transport in the good old ISO/OSI scheme, so is > http, html, xml, php, perl, python and so forth, and Open Source, Open > Standard software is The Devil Himself -- to M$. Oddly enough, it should not be. Open standards relinquish some level of control. But they provide a common platform for a much larger group of developers. No one can make a decision, change an ABI, etc. which will prevent me or my company from releasing a Perl, Python, ... based application. The same is not true for closed platform languages, of which the previously mentioned is one, though oddly enough, C# is not. I can run my C# code on mono on most systems, and on .Net. Go figure. With this in mind it is very much to Microsoft's advantage to support Mono. It gets rather hard to accuse someone of embrace and extend with the tool when the platform runs everywhere and is open (and standardized). > Current systems corporations are thinking "license/lease" and "renewable > revenue stream" instead of "software sales" as the latter era of the > computer revolution draws to a slow end. Red Hat has gone there. Sun They have to. Shareholders (such as the folks in the US with 401k, 403(*), and other retirement bits from TIAA/CREF and elsewhere) have this preference to get a return on their investments. These investments, in mutual funds, or stocks, etc. only really come about (ignoring bubbles) when earnings rise. Earnings rise of course when a) you sell more, b) you sell more profitably, c) your competitors leave the market d) your revenue streams become more predictable. We are the ones who compensate these folks by their success at maximizing the profitability function of the corporate entity. So we shouldn't be surprised when they come up with "new" models to "monetize" assets. Sure, as a model, subscriptions are boring. You sign up, you pay, you get something, you keep paying, you keep getting things. Over the lifetime of the thing you use/buy, you keep paying, often as much if not more than before, and in a regular manner. This makes revenue streams more predictable, smooths out business, and generally makes Wall street happier (smooth == nice), which has positive effects upon the stock price. See the feedback loop there? > is well along the road. Microsoft is stuck between a fervent wish to go > there and somehow preserve their high margins and the realities of the > expectations of their PC customers that they are "buying a copy" of the > operating system, not the right to use it for a year with automated > prepaid updates. In the corporate world, however, they've moved a long > ways there and I feel confident that they'll go the rest of the way > soon. > > So the interesting question is "where's the money" in this move. The > HPC market hasn't proven to be exactly a get-rich-quick proposition for > any of the various groups that have prospected in it. I suspect (given HPC has never been a large market. Heck, many of the Sun machines on the top 500 list in the past have been database servers, or corporate "mainframes" and not supercomputers in any traditional sense. > the reference to "resource management" and "server applications") that > it is seeking to plunder the relatively deep pockets and relatively > non-computer-savvy researchers in the biocluster and medicluster market, > with their bet backed by the proposition that they can reuse their > efforts in a pinch in the HA market (those "server applications"). Hmmm. Me-thinks those who think that the biocluster folks have deep pockets might be mistaken. And some of them are quite computer-saavy. The module that launched a thousand web sites, CGI.pm was written by Lincoln Stein, who has since done some other very interesting stuff. It would be a mistake to mis-characterize these folks, and paint them with broad strokes. Yes, I am biased, as this is where my companies play. They (the scientists and end users) have some neat problems, and they are computationally expensive and getting more so by the day. > Bioclusters and biogrids as a market probably does total millions of > dollars per year -- chickenfeed for M$ at this point -- but it also has > some growth potential and integration potential for the future, when we There are other very serious problems that simply building clusters and grids will not overcome. We are starting to run into the edge of them now, and they will only be exacerbated over time. There are problems of commercial importance in here. > reach the point where e.g. individuals are bioassayed in real time in > their Dr.'s office, their genes are matched against a huge database > scanning for defects and predispositions and damage and who knows what > else, and genetically matched and tuned therapies are prescribed, where > dog pedigrees and likelyhood of developing hip dysplasia are similarly > determined in real time. This of course depends upon cheap/fast sequencing, clinical micro-arrays and other technologies, and the folks at 454 and a few other places may indeed get us there. At that point you will need some serious processing horsepower. Even if Moore's law can hold out for a while, it will not get us where we need to be for this. If you want to know more, contact me offline. > Of course, Moore's law and so forth may eat this market before it fully > matures -- in three or four years at expected rates desktop PCs will > have (possibly multiple) TB size LOCAL SINGLE disks, multiGB of RAM likely ... > standard, and at least gigabit interfaces standard with 10 Gbps not likely ... (nVidia motherboards have built in gigabit as do a few others now, for the consumer market, which they bill as "download from the internet faster" ... :o ) > uncommon. With 10+ GHz CPU clocks, 64 bit and better data paths, really There are limits to feature size, clock speed, and thermal dissipation. With a small enough feature size, the thermodynamics of defect formation becomes ever more important, and hence cooling and stabilization against diffusion become even more critical. We are just going to have to stop computing using fermions and start using bosons. Talk about natural parallelism... :) How many photons can you have on a single "wire" ? > slick memory and device management, and perhaps some onboard > parallelism, would one NEED a cluster to run such a (relatively simple) > application? If not, it leaves one back with just the research market, > and that is, I think, a hopeless case for M$ to make back their not > inconsiderable development and sales costs in without a hedge. They > would at best break even and at worst would both lose money and worse, > lose money EMBARRASSINGLY and publically. > > As this list well knows, real, transparent, "windows-level" scalability > is an elusive beast in the parallel HPC world, although you can find > niches where it is possible. And there is no mass market here, per se This is true. > -- even if you can find somebody crazy enough to pay out LARGE software > costs that scale per node, there simply aren't that many customers, and > most of those customers always have several proven alternatives with > costs that DON'T scale per node. Those customers will pay for support, > they'll pay for service, but they don't like paying for software and > REALLY hate paying for software and getting lousy service and poor I don't know that people hate paying for software per se, rather they hate not having the control over their software (e.g. cannot use it they way they need to due to onerous licensing, binary only releases, etc.), cannot fix problems with it, even if they know what they are. > support to go with it. Plenty of people make money on service and > support, but they don't get (unduly) rich and they work for a living > earning what they earn. Support is not easy. It is very hard to build a company out of support. It is impossible as far as I can tell to get any sort of external funding (be it equity, SBIR/grant, etc) to do support. It is not a bad business, just nothing more than a small one. Large consulting houses are amalgamations of small consulting groups .... Aside from that, there is nothing wrong with reaping the rewards of your hard labor, be they research recognition, awards, or remuneration. If your work/research/product is bad, well, maybe there is something wrong with it ... :) What annoys me personally these days are the folks who simply repackage and resell others open source work without contributing back to the folks who did all the hard work in the first place. Not plagiarism, but really a failure to acknowledge and attribute the source of innovation at minimum, and support it. I see lots of that in the market. Real innovation is hard. Those who innovate should be recognized for it. Those who repackage should indicate that is what they do. > > rgb Joe -- Joseph Landman, Ph.D Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://scalableinformatics.com phone: +1 734 612 4615 From gerry.creager at tamu.edu Tue May 25 19:07:58 2004 From: gerry.creager at tamu.edu (Gerry Creager N5JXS) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Redmond is at it, again In-Reply-To: <20040525134414.GU1105@leitl.org> References: <20040525134414.GU1105@leitl.org> Message-ID: <40B3FBFE.60407@tamu.edu> Excerpting, > Now Microsoft has begun its response, forming its High Performance Computing > team and planning a new OS version called Windows Server HPC Edition. Kyril > Faenov is director of the effort, and Microsoft is hiring new managers, > programmers, testers and others. Does "and others" mean users, too? gerry Eugen Leitl wrote: > Yeah, think of all those power Excel hydro code users who're going to > switch... > > http://zdnet.com.com/2102-1103_2-5219282.html?tag=printthis > > Microsoft creating Windows for supercomputers > By Stephen Shankland and Ina Fried > CNET News.com > May 24, 2004, 12:30 PM PT > URL: http://zdnet.com.com/2100-1103-5219282.html > > Microsoft has launched an effort to produce a version of Windows for > high-performance computing, a move seen as a direct attack on a Linux > stronghold. > > Bottom line: > For now, Linux has the upper hand, owing to its affinity with Unix--the OS > environment the high-performance crowd is most comfortable with--and the > open-source model, which lets users turn directly to source code for answers > to problems. But a Microsoft product would theoretically integrate better > with Windows desktop machines, and if the company can serve up an impressive > offering, Linux could be in for a tussle. > > High-performance computing once required massive, expensive, exotic machines > from companies such as Cray, but the field is being remade by the arrival of > clusters of low-end machines. While the trend could be considered an > opportunity for Microsoft, which has long been the leading operating-system > company, Linux has actually become the favored software used on these > clusters. > > Now Microsoft has begun its response, forming its High Performance Computing > team and planning a new OS version called Windows Server HPC Edition. Kyril > Faenov is director of the effort, and Microsoft is hiring new managers, > programmers, testers and others. > > The Redmond, Wash.-based software colossus has its work cut out in the > market--and knows it. > > "Winning in this important space against entrenched Linux/open-source > software competition requires creativity, innovation, speed of execution, and > deep engagements with hardware, software and academic partners," reads a job > posting for a program manager responsible for setting up the team's academic > partnerships. > > In a recent interview, Bob Muglia, a Microsoft senior vice president who > leads the development of Windows Server, said the company is interested in > two particular areas: building high-performance computing clusters and > harvesting the unused processing power of PCs. > > Although Microsoft is a comparative newcomer to the market, the company could > bring several advantages: > > . Machines running Windows HPC Edition could seamlessly connect to desktop > computers, providing instant power for someone such as a financial analyst > performing calculations on an Excel spreadsheet, said David Lifka, chief > technology officer for the Cornell Theory Center, Microsoft's premier > high-performance computing partner. > > . Microsoft could create a specialized version of its widely praised > programming tools, said Phil Papadopoulos, director of the grids and clusters > program at the San Diego Supercomputing Center. "Windows could make that much > easier with their integrated development environment. They have the manpower > to do that piece of the puzzle." > > . Microsoft could also adapt its popular SQL Server database software to run > on high-performance systems. The company has already said the next major > version of SQL Server, code-named Yukon and due next year, will include > better support for very large databases and for running on clustered systems. > > . And Microsoft could build software into its desktop version of Windows to > harness the power of PCs, letting companies get more value from their > computers. It's a technology that's applicable to tasks such as drug > discovery and microchip design. > > The business imperative > The high-performance effort doesn't mark the first time Microsoft has tried > to head off Linux's progress. With Windows Server 2003, Microsoft released a > lower-priced Web server edition, as Linux was growing popular for use on the > machines that host Web sites. > > "The Windows Server group is really focused on countering Linux," said Rob > Helm, an analyst with Directions on Microsoft. "They've identified specific > areas where Linux has the most traction." > > The HPC Edition is also an example of a Microsoft strategy to increase > revenue by creating versions of Windows tailored for specific market > segments--for example, Windows for tablet PCs, digital TV recorders and > storage servers. > > "Another way for them to keep Windows sales moving is to roll out more of > these editions," Helm said. "When you've got a product that you need to keep > moving, one way to do it is to segment it. You introduce Tarter Control > Windows Server and Sensitive Teeth Windows Server." > > High-performance computing is a lucrative market, with sales that increased > 14 percent to $5.3 billion in 2003, according to IDC. And "bright clusters," > Linux servers that manufacturers know will be used in a cluster, had sales of > $384 million in the fourth quarter. > > Beating the incumbent > But for once, Microsoft is the newcomer, and Linux is the incumbent. Linux > got its first foothold in academia and research labs, which already had > expertise and software for the functionally similar Unix operating system. > > "The majority of people doing high-performance computing are a lot more > comfortable and efficient inside a Unix environment," a category that > includes Linux, the SDSC's Papadopoulos said. To convince people to invest > the time and money to switch, Microsoft will have to offer something much > better, he said. > > Linux, boosted by low-cost servers using processors from Intel and Advanced > Micro Devices, now is used on prestigious machines. Thunder, a machine at the > Lawrence Livermore National Laboratory with 512 Linux servers running Red Hat > Enterprise Linux, can perform more than 19 trillion calculations per second, > second only to Japan's Earth Simulator. > > Dozens of machines in a list of the 500 fastest supercomputers run Linux, > including five of the top 10. Only two on the list are identified as Windows > machines. > > One reason Windows has been slow to catch on is that Unix and Linux were bred > to be administered remotely, a necessary feature for managing a cluster with > dozens or hundreds of computers. > > In Windows, "the notion of remote computing is significantly more difficult > than in Unix," Papadopoulos said. "Because Windows was born out of the > desktop, (it is) deeply ingrained in the Microsoft culture that you have > somebody sitting in front of the machine to do work." > > Management is on Microsoft's agenda, though. The company is hiring one > programmer to work on a "graphical and script-based user interface for > efficient job and resource management across large clusters" and another to > create "automated infrastructure to uncover performance and reliability > problems with high performance, large-scale server applications." > > Linux adds another advantage: It's open-source software, meaning that anybody > may see and modify its underlying source code. Most business customers aren't > interested, but high-performance technical computing users need to extract > every bit of performance and track down difficult bugs. > > "The nice thing is that because everything is open, if you have a problem, > you can get at the root of the problem in terms of the software. That moves > things along quite a bit faster," Papadopoulos said. > > That openness also makes it easier to accommodate the multitude of different > technologies used in the high-performance market but not necessarily in the > mainstream computing market, said Brian Stevens, vice president of operating > system development for Linux seller Red Hat. > > Releasing a product > Microsoft declined to share schedule information about the HPC Edition, but > work is already under way. > > For example, a software developer kit for HPC Edition will include support > for the Message Passing Interface, or MPI, widely used software to let > computers in a cluster communicate with one another. > > The Cornell Theory Center's Lifka believes that an early software development > kit for the HPC Edition could arrive as soon as this fall. The center is > helping Microsoft develop and test the new software. > > Microsoft has several upcoming server releases, to which an HPC version of > Windows could be added. Service Pack 1 of Windows Server 2003 is due later > this year, followed by a more substantive upgrade, code-named R2, slated for > 2005. The next major update to Windows, code-named Longhorn, is scheduled to > arrive in server form in 2007. > > According to job postings, Microsoft is adapting MPI to Microsoft's .Net > infrastructure. A key foundation of .Net is the C# programming language and > the Common Language Runtime, or CLR, which lets C# programs run on a > multitude of different systems. > > Lifka said the first phase will use a version of MPI written for a specific > operating system and hardware type. The next foundation will be a version of > MPI for the CLR that will let administrators run the same programs on a wide > variety of different Windows machines--for example, those using Xeon, Opteron > or Itanium processors. > > So far, programs written for the CLR and .Net aren't as fast as those written > for a specific machine, "but we see constant improvement in that," Lifka > added. Another area that needs work is security and easy patch installation, > he said. > > Overall, Lifka is a fan of Windows for high-performance computing. The > biggest reason for his enthusiasm is that it can dovetail easily with other > versions of Windows in a company. > > And companies are more familiar with Windows than Linux, he added. "Moving to > Windows has allowed us to have a greater number and quality of corporate > relationships," Lifka said. > > Microsoft takes a long-term view of the challenge. > > Muglia often discusses technology moving from possible to practical to > seamless, as it matures. High-performance computing on Windows today is in > the possible stage, he said, but the goal is to make it practical. > > "That is something that will happen in the next few years," Muglia said. > "There is an opportunity to make this better." > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Gerry Creager -- gerry.creager@tamu.edu Network Engineering -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578 Page: 979.228.0173 Office: 903A Eller Bldg, TAMU, College Station, TX 77843 From rssr at lncc.br Wed May 26 05:16:24 2004 From: rssr at lncc.br (Renato Silva) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] 32 - 64 Compatibility References: <40B192B1.2020107@csiro.au> <1085407628.5412.88.camel@protein.scalableinformatics.com> Message-ID: <40B48A98.86898D47@lncc.br> Hi Folks I'm contemplating building a small cluster 16 nodes, and some guys ask about the compatibility of a 32 bits with 64 bits machines Is there a easy way to check this compatibility ? Thanks Renato From rsweet at aoes.com Wed May 26 07:14:48 2004 From: rsweet at aoes.com (Ryan Sweet) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Job posting [NL]: HPC Consultant Message-ID: Hello, I'm glad to see that the beowulf.org list is back in operation. We have a contract position open for an HPC consultant. The job is based in Leiden, The Netherlands. Those with permission to work in the EU have priority, but other will be considered. The contract would initially be for 6 months with a possibility for extension or to move to permanent employment after the 6 months are up. Starting date would be ASAP. More info can be found here: http://www.aoes.com/en/jobs/vn0407.html (excerpt below). regards, -Ryan Sweet -- HPC Consultant, 6 month contract: The candidate will work in a team providing UNIX/Linux consulting, focused on engineering/scientific computation, including Beowulf clustering (design, documentation, automation, installation, administration, and support), SMP systems tuning (Solaris/IRIX), application performance analysis, benchmarking, some systems programming, etc.... The work will be project based, with multiple projects running simultaneously during this 6 month contract. The projects will include cluster system engineering, application performance analysis, and development of tools and scripts related to HPC application integration and cluster administration. The work will be based in Leiden, The Netherlands, starting in June or July 2004. The contract will be for 6 months, but could lead to a longer term contract or a permanent position. Profile requirements * expert UNIX administration skills: o TCP/IP networking, packet analysis, IP firewalls, DHCP and TFTP protocols o directory services (NIS/DNS, LDAP) o exposure to multiple UNIX varieties (Solaris, IRIX, Linux, BSD) o systems automation with perl or shell scripting o system performance tuning, capacity planning o knowledge of Linux kernel (2.4) patching, configuration, compilation * at least two years experience with Linux and GNU tools * an understanding of C (and basic C++) or FORTRAN * ability to program perl (proficient in standard modules, perl OO, and regex) and at least one shell * ability to work professionally with customers and colleagues * a high level of competency in written and spoken English * excellent documentation skills Very important requirements are intelligence, professionalism, and the ability to quickly learn a new technology and apply it to commercial solutions. Additionally, the following, while not required, would be a plus: * Beowulf/clustering experience * significant experience with FORTRAN and C programming * experience with parallel programming (MPI/PVM) * experience with MSC.NASTRAN, PATRAN, Tecplot, Maya, or computational fluid dynamics software packages * experience developing with SQL databases and PHP * experience with GLOBUS Toolkit/Grid Computing * experience in a govt. or large academic setting Please send your application and profile to careers@aoes.nl Please indicate the AOES vacancy number VN-0407 in your application. -- Ryan Sweet phone +31(0)71 5795521 fax +31(0)71572 1277 Advanced Operations and Engineering Services AOES Group BV http://www.aoes.com From rgb at phy.duke.edu Wed May 26 09:14:02 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] 32 - 64 Compatibility In-Reply-To: <40B48A98.86898D47@lncc.br> Message-ID: On Wed, 26 May 2004, Renato Silva wrote: > Hi Folks > > I'm contemplating building a small cluster 16 nodes, > and some guys ask about the compatibility of a 32 bits with 64 bits > machines > Is there a easy way to check this compatibility ? Sure. Get machine A (32bit) , get machine B (64bit). Install both, being sure to install all "compatibility libraries" (which hopefully exist and are prebuilt and packaged for your distribution) on B. compile application on A. Attempt to run application on B. Go back and add missing libraries, building and packaging as required, until 32 bit A binary runs on B. Well, ok, I don't know how EASY that all is... but it is straightforward. What is even easier is: cd Src/64 cvs checkout myapp cd myapp make cp myapp ~/bin64 vs cd Src/32 cvs checkout myapp cd myapp make cp myapp ~/bin32 Then all you have to do is write a teeny snippet for e.g. .bashrc that runs uname -p and puts $HOME/bin64 first on your $PATH if it is equal to x86_64, otherwise $HOME/bin32. Or put the uname into your Makefile and create myapp32 and myapp64. And create a shell script called myapp that uses uname -p to execute the correct binary on whichever architecture and just build both versions whenever you are ready to go into production. This is a bit ugly, but it is very easy and it has the virtue of compiling and linking for the actual platform being used; one HOPES that 64 bit libraries and code will outperform 32 bit libraries and code on any 64 bit platform (at least on average) or we all have to ask ourselves a great big why bother sort of question. Eventually one expects mixed environment to disappear as everything becomes 64 bit, but it hasn't happened yet. Back in the bad old days, when our physics department had Sun 3, Sun 4, SGI/Irix, Sun 386i, and maybe some NeXTs, an Ultrix box or two, an AIX loaner (you get the picture) I had a big old conditional tree at the beginning of my rc files righteously ripped off from PVM's aimk that would identify the architecture of whatever I was running on and set some environment variables that I later used to vary path, reset various aliases and so forth so all the platforms behaved as nearly as possibly alike at the shell level and could run any binaries I built per architecture. Linux levelled the field and made this no longer worthwhile, and I think most sites now try to run a single architecture, although that is changing and will likely change some more with x86_64, Itanium, and competitive PPC boxes joining the ubiquitous i386 boxes. PVM was (and likely still is) a past master at this sort of juggling. If you build and install your binaries once per architecture in their PVM arch paths, pvm will parallelize applications transparently across completely different binary architectures. I've seen graphs of some of the early PVM testbed computations where a computation ran on four or five architectures including a Cray, with the Cray doing the vector stuff and the workstation clusters doing non-vector stuff. rgb > > > Thanks > Renato > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb at phy.duke.edu Wed May 26 09:44:59 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Possibly OT In-Reply-To: <20040525230451.6411350e@barton> Message-ID: On Tue, 25 May 2004, Ed wrote: > Hi, Im very new to clustering. I have a perhaps very simple question: > > I have to provide a webhost, but with low spec computers which we have > in abndance. > > Now, I assume that the webservice will use the single IP address > x.x.x.x. > > However, I have not got a clue how to go about this. Should I aim to > have two single computers linked via a serial interface, or should I do > something more adventurous such as link them via a network interface. Am > I asking on a list where Beowulf is a API and only certain programs will > work with Beowulf? This isn't high performance computing (HPC) -- this is high availability computing (HAC). So this is indeed the wrong list for this. Most of the issues that you raise have long since been solved. There are round-robin and/or load balancing daemons that manage connections and farm them out to a cluster of servers. For example, see: http://www.webopedia.com/TERM/R/Round_Robin_DNS.html for RR DNS (which may or may not be the best way for you to proceed -- there are alternative load balancing tools to consider as well). Some of the projects (for linux) are summarized here: http://lcic.org/load_balancing.html LVS is probably where you should start. That would be: http://www.linux-vs.org/ I believe that this is a fairly mature project and in fairly widespread use. I don't use it myself so I couldn't tell you if it is prebuilt in standard distros ready to roll or if you'll have to build your own packages from the GPL sources, but if you're building a serious HA site even the latter shouldn't be too onerous. > > This is probably a question that has been asked many times. But if > Beowulf can control computers and spawn instances of daemons for mail > as mail arrives it would also solve many of the mail problems which we > currently face in the small hosting organisation where I work. Currently > I spend many hours researching this question, however fruitless. Remember, Google is your friend. "load balancing web server linux" turned up a huge amount of information in a few seconds. Google is ALMOST the only "research tool" I use anymore, the exceptions being abstract searches (which tend to be sold services, likely managed through a HA server:-). HTH, rgb > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb at phy.duke.edu Thu May 27 07:19:00 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy Message-ID: Dear Perfect Masters of Grid Computing: Economics is preparing to set up a small pilot cluster at Duke and the following question has come up. Primary tasks: matlab and stata jobs, run either interactively/remote or (more likely) in batch mode. Jobs include both "short" jobs that might take 10-30 minutes run by e.g. 1-2nd year graduate students as part of their coursework and "long" jobs that might take hours to days run by more advanced students, postdocs, faculty. Constraint: matlab requires a license managed by a license manager. There are a finite number of licenses (currently less than the number of CPUs) spread out across the pool of CPUs. Concern: That long running jobs will get into the queue (probably SGE managed queue) and starve the short running jobs for either licenses or CPUs or both. Students won't be able to finish their homework in a timely way because long running jobs de facto hog the resource once they are given a license/CPU. I am NOT an SGE expert, although I've played with it a bit and read a fair bit of the documention. SGE appears to run in FIFO mode, which of course would lead to precisely the sort of resource starvation feared or equal share mode. Equal share mode appears to solve a different resource starvation problem -- that produced by a single user or group saturating the queue with lots of jobs, little or big, so that others submitting after they've loaded the queue have to wait days or weeks to get on. However, it doesn't seem to have anything to do with job >>control<< according to a policy -- stopping a long running job so that a short running job can pass through. It seems like this would be a common problem in shared environments with a highly mixed workload and lots of users (and indeed is the problem addressed by e.g. the kernel scheduler in almost precisely the same context on SMP or UP machines). Recognizing that the license management problem will almost certainly be beyond the scope of any solution without some hacking and human-level policy, are there any well known solutions to this well known problem? Can SGE actually automagically control jobs (stopping and starting jobs as a sort of coarse-grained scheduler to permit high priority jobs to pass through long running low priority jobs)? Is there a way to solve this with job classes or wrapper scripts that is in common use? At your feet, your humble student waits, oh masters of SGE and Grids... rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb at phy.duke.edu Thu May 27 08:03:09 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy In-Reply-To: <40B601AC.9020002@tamu.edu> Message-ID: On Thu, 27 May 2004, Gerry Creager N5JXS wrote: > This is really a first-cut response, with 2 visible possibilities... > > 1. Use 2 license servers, one with 'i' licenses available for short > jobs, and one with 'j' licenses available for longer jobs. For i < j, > starvation of the short jobs shouldn't occur too often, save when > there's too many masters' students trying to get their projects done in > time to graduate and the deadline's tomorrow. > > 2. Priority queuing where short jobs have the nod, and longer jobs are > put aside and required to temporarily relinquish licenses. Liketo to > require programming resources to accomplish this one. > > Good question. Thanks for the suggestions. The lack of even coarse grained kernel-style job control for a cluster continues to be a source of frustration. load balancing queueing systems are getting to be pretty good, but this isn't a problem in load balanced queueing, and a kernel that used load balanced queueing as a scheduler algorithm would be terrible. No, wait! It would be DOS (for a single CPU). With xmlsysd I have access to the data required to implement a queueing system WITH a crude scheduler algorithm with a granularity of (say) order minutes. I've actually hacked out a couple of tries at a simple script-level control system in perl (before per got threads). One would expect that with threads it would be pretty easy to write a script based scheduler that issues STOP and CONT signals to tasks on some sort of RR/priority basis every minute. It wouldn't deal with license starvation, since I don't know how a running matlab task can "temporarily relinquish a license" while it is stopped, but it would manage the problem of being able to use a cluster for a mix of prioritized long and short running jobs without resource-starving the short ones. I have a personal interest in this outside of econ because I am, after all, a bottom feeder in the cluster world. If I could ever arrange it so that my jobs just "got out of the way" when competing jobs were queued on a cluster according to policy, priority, ownership etc. I might be able to wheedle more cycles out of my friends...;-) rgb > > Gerry > > Robert G. Brown wrote: > > Dear Perfect Masters of Grid Computing: > > > > Economics is preparing to set up a small pilot cluster at Duke and the > > following question has come up. > > > > Primary tasks: matlab and stata jobs, run either interactively/remote > > or (more likely) in batch mode. Jobs include both "short" jobs that > > might take 10-30 minutes run by e.g. 1-2nd year graduate students as > > part of their coursework and "long" jobs that might take hours to days > > run by more advanced students, postdocs, faculty. > > > > Constraint: matlab requires a license managed by a license manager. > > There are a finite number of licenses (currently less than the number of > > CPUs) spread out across the pool of CPUs. > > > > Concern: That long running jobs will get into the queue (probably SGE > > managed queue) and starve the short running jobs for either licenses or > > CPUs or both. Students won't be able to finish their homework in a > > timely way because long running jobs de facto hog the resource once they > > are given a license/CPU. > > > > I am NOT an SGE expert, although I've played with it a bit and read a > > fair bit of the documention. SGE appears to run in FIFO mode, which of > > course would lead to precisely the sort of resource starvation feared or > > equal share mode. Equal share mode appears to solve a different > > resource starvation problem -- that produced by a single user or group > > saturating the queue with lots of jobs, little or big, so that others > > submitting after they've loaded the queue have to wait days or weeks to > > get on. However, it doesn't seem to have anything to do with job > > > >>>control<< according to a policy -- stopping a long running job so that > > > > a short running job can pass through. > > > > It seems like this would be a common problem in shared environments with > > a highly mixed workload and lots of users (and indeed is the problem > > addressed by e.g. the kernel scheduler in almost precisely the same > > context on SMP or UP machines). Recognizing that the license management > > problem will almost certainly be beyond the scope of any solution > > without some hacking and human-level policy, are there any well known > > solutions to this well known problem? Can SGE actually automagically > > control jobs (stopping and starting jobs as a sort of coarse-grained > > scheduler to permit high priority jobs to pass through long running low > > priority jobs)? Is there a way to solve this with job classes or > > wrapper scripts that is in common use? > > > > At your feet, your humble student waits, oh masters of SGE and Grids... > > > > rgb > > > > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb at phy.duke.edu Thu May 27 08:09:08 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy In-Reply-To: <40B6036E.1090403@sonsorol.org> Message-ID: On Thu, 27 May 2004, Chris Dagdigian wrote: Thanks, some very useful stuff here, especially the flexlm stuff. > > Here are some personal notes I've collected on license management and > policies under Grid Engine. Perhaps they may be helpful? > > FLEXlm license integration with grid engine 5.3: > http://bioteam.net/dag/sge-flexlm-integration/ > > Grid Engine can do very sophisticated resource allocation above and > beyond FIFO ordering. > > Your need to *stop* a long running job to let a shorter one in is a bit > harder as graceful job premption/supension is something that normally > takes a bit of configuring (both of the app and of Grid Engine). Ya, it isn't just manipulation of the queue. It is the difference between a real scheduler that dispenses time-slices to a mix of tasks, some sleeping, some queued to be run, so that all tasks get time slices fairly whether they complete in a single slice or 10^8 slices, and one that puts a 10^8 slice task on to run to completion if and when it makes it to the top of the queue and never mind the jobs that then have to wait weeks to run. I'll look over the archives and see what I can see, and again, really appreciate the links and comments! rgb > There are other mechanisms in grid engine that people have used to > prevent the 'job starvation' issue. The SGE mailing list > users@gridengine.sunsource.net (I think you must be a member to post) or > the searchable mailing list archives at http://gridengine.sunsource.net > may be very helpful. > > In particular you may find some of the "enterprise edition" policies > such as 'share tree' or 'functional policy' may help. They won't boot > running jobs unless you configure it to but the policies themselves can > help you gain 'fairshare-by-user' or 'fairshare-by-project' style > resource allocation which may meet your needs. > > Also-- > > There is a new feature in Grid Engine 6.0 built into the new "Urgency > scheduling" policy -- a configurable parameter called $WaitTimeUrgency > that causes a jobs' urgency priority to grow the longer it has been > sitting in pending state. I've been told that one of the drivers for > "waitTimeUrgency" was to address pending-job-starvation issues. > > The problem with Urgency Scheduling is that I don't think it has been in > use by many people for all that long. Once 6.0 comes out of beta in a > few weeks there may be better user reports on how Urgency scheduling is > doing in the real world. > > Since documentation is seriously lacking for Grid Engine 6.0 and the Sun > branded version called 'N1GE 6' I've tried to write down what I've > learned about them here: > > http://bioteam.net/dag/gridengine-6-features.html > > > Regards, > Chris > > > > > > Robert G. Brown wrote: > > > Dear Perfect Masters of Grid Computing: > > > > Economics is preparing to set up a small pilot cluster at Duke and the > > following question has come up. > > > > Primary tasks: matlab and stata jobs, run either interactively/remote > > or (more likely) in batch mode. Jobs include both "short" jobs that > > might take 10-30 minutes run by e.g. 1-2nd year graduate students as > > part of their coursework and "long" jobs that might take hours to days > > run by more advanced students, postdocs, faculty. > > > > Constraint: matlab requires a license managed by a license manager. > > There are a finite number of licenses (currently less than the number of > > CPUs) spread out across the pool of CPUs. > > > > Concern: That long running jobs will get into the queue (probably SGE > > managed queue) and starve the short running jobs for either licenses or > > CPUs or both. Students won't be able to finish their homework in a > > timely way because long running jobs de facto hog the resource once they > > are given a license/CPU. > > > > I am NOT an SGE expert, although I've played with it a bit and read a > > fair bit of the documention. SGE appears to run in FIFO mode, which of > > course would lead to precisely the sort of resource starvation feared or > > equal share mode. Equal share mode appears to solve a different > > resource starvation problem -- that produced by a single user or group > > saturating the queue with lots of jobs, little or big, so that others > > submitting after they've loaded the queue have to wait days or weeks to > > get on. However, it doesn't seem to have anything to do with job > > > >>>control<< according to a policy -- stopping a long running job so that > > > > a short running job can pass through. > > > > It seems like this would be a common problem in shared environments with > > a highly mixed workload and lots of users (and indeed is the problem > > addressed by e.g. the kernel scheduler in almost precisely the same > > context on SMP or UP machines). Recognizing that the license management > > problem will almost certainly be beyond the scope of any solution > > without some hacking and human-level policy, are there any well known > > solutions to this well known problem? Can SGE actually automagically > > control jobs (stopping and starting jobs as a sort of coarse-grained > > scheduler to permit high priority jobs to pass through long running low > > priority jobs)? Is there a way to solve this with job classes or > > wrapper scripts that is in common use? > > > > At your feet, your humble student waits, oh masters of SGE and Grids... > > > > rgb > > > > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb at phy.duke.edu Thu May 27 08:13:20 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy In-Reply-To: <40B6044E.7040400@cora.nwra.com> Message-ID: On Thu, 27 May 2004, Orion Poplawski wrote: > You can also setup subordinate queues where the queue with priority will > only accept jobs that will take less than a given time to run. When a > short job gets submitted, a long job on the subordinate queue will be > stopped (SIGSTOP) while the short jobs runs. Your problem here is that > the long job will presumably still hold a license. If matlab has some > kind of checkpointing function, you could tie that into SGE to release > the license. So it WILL manage things with SIGSTOP/SIGCONT. Good, that's what I was hoping. I'll search the docs etc for "subordinate queues" and signals to see if I can figure out how. Excellent. > You can also limit the number of available slots for long running jobs. > > SGE can tie into license management through resource monitors to > determine the number of licenses available. Also excellent. As you note, we'll probably have to screw around with the matlab license manager and issue, but that's a problem with definite bounds and there are a number of ways to try to solve it if we can't find a solution already out there. rgb > > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From lindahl at pathscale.com Thu May 27 15:27:08 2004 From: lindahl at pathscale.com (Greg Lindahl) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Redmond is at it, again In-Reply-To: <40B3BDAE.4090400@charter.net> References: <20040525134414.GU1105@leitl.org> <40B3BDAE.4090400@charter.net> Message-ID: <20040527222708.GH2452@greglaptop.internal.keyresearch.com> On Tue, May 25, 2004 at 05:42:06PM -0400, Jeffrey B. Layton wrote: > In fact, the last quote I got for a 72 node cluster averaged about > $400 a node! Now management has started to compare this price > to Windows 'Doctor, it hurts when I try to buy cluster software from a non-cluster company!' I don't know why you seem to think that only SuSE or Red Hat is the place to go for Linux support. You mention later in your note that you do talk to cluster vendors. I know of several who provide Linux support for a reasonable fee. It would seem that your bean counters would like that approach. I can't figure out from your explaination why you don't buy from them. -- greg (no longer a cluster vendor, and yes, PathScale's compilers do support more than just SLES and RHEL -- our customers require it.) From rgb at phy.duke.edu Fri May 28 10:49:36 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Re: SGE + policy In-Reply-To: <20040527201434.GA41179@piskorski.com> Message-ID: On Thu, 27 May 2004, Andrew Piskorski wrote: > On Thu, May 27, 2004 at 12:01:09PM -0700, beowulf-request@beowulf.org wrote: > > > Date: Thu, 27 May 2004 10:19:00 -0400 (EDT) > > From: "Robert G. Brown" > > > Primary tasks: matlab and stata jobs, run either interactively/remote > > or (more likely) in batch mode. Jobs include both "short" jobs that > > > Constraint: matlab requires a license managed by a license manager. > > Have they considered running Octave rather than Matlab? Depending on > what special-purpose Matlab add-on modules they're using, that might > or might not be practical, but if it is, using Octave should reduce > their cluster and license hassles. (I've never used either of the two > dialects myself though, so have them ask someone who really knows...) They aren't quite there yet. First we build the pilot cluster and make it "work" for their existing work cycle using the tools that they are used to, THEN we start them experimenting with cheaper/better alternatives. The cost scaling of matlab is actually a major strike against it anywhere, IMO. Even if one grants that it is a fabulous program and much better than maple, mathematica, octave (all of which live in similar spaces with widely varying costs) it is hard to manage in a cluster environment. This probably doesn't worry them, because in one sense anybody who uses matlab for doing "serious" computations probably needs therapy, and clusters are often build for doing serious computations. However, this particular cluster is being built as much to support many students doing less serious computations as it is to support work that really should probably be done in c.* or f.*, and in a lot of cases the real expense is teaching people with little or no real programming experience how to convert from a garbage-tolerant interactive environment to a compiled language. rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb at phy.duke.edu Fri May 28 11:44:10 2004 From: rgb at phy.duke.edu (Robert G. Brown) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Re: SGE + policy In-Reply-To: <40B77FFD.6030006@cora.nwra.com> Message-ID: On Fri, 28 May 2004, Orion Poplawski wrote: > Robert G. Brown wrote: > > > > This probably doesn't worry them, because in one sense anybody who uses > > matlab for doing "serious" computations probably needs therapy, and > > clusters are often build for doing serious computations. However, this > > particular cluster is being built as much to support many students doing > > less serious computations as it is to support work that really should > > probably be done in c.* or f.*, and in a lot of cases the real expense > > is teaching people with little or no real programming experience how to > > convert from a garbage-tolerant interactive environment to a compiled > > language. > > > > You can buy a matlab compiler from mathworks. Perhaps this would work > better. There is also some kind of run-time server as well that turns > the code into standalone programs. I've never used either, but we may > soon. Experiences with either would be appreciated. This would be fine if it produces real standalone programs that are portable and can be run anywhere. If the compiler just creates a binary module that can only run on a node with a license, it is less desireable although it still might be advantageous if they run faster that way. We'll look into this as well, thanks, rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From hahn at physics.mcmaster.ca Mon May 31 07:16:50 2004 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] harmonic mitigators vs PFC? In-Reply-To: <20040531022129.22563.qmail@web50108.mail.yahoo.com> Message-ID: > > Somewhere, about 3 or 4 months ago, someone posted a > > link to a very nice > > explanation of managing harmonics in this sort of > > situation. > > this one? ;-) > http://www.mirusinternational.com/pages/faq.html no. this equates "nonlinear load" with "has bad PFC", which is clearly false for PFC switching power supplies. in fact, this WHOLE thread is purely about whether harmonic mitigators make any sense when the load is already corrected to .97 or better. for our local project, we are acquiescing to the mitigators, even though we clearly do not need them. my personal hope is that they'll do *something* useful to us, such as smoothing out the incoming noise/spikes/etc. regards, mark hahn. From kip at camelot.mssm.edu Wed May 26 08:05:33 2004 From: kip at camelot.mssm.edu (Kip Lubliner) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] how useful is bproc, and what does Scyld cost? In-Reply-To: <1085518615.4397.20.camel@pel> References: <20040525193745.GB75378@piskorski.com> <1085518615.4397.20.camel@pel> Message-ID: <40B4B23D.2080407@camelot.mssm.edu> Sean Dilda wrote: >... > >If you're more interesting in having a turn-key cluster solution, I have >some friends who are happy with Rocks (http://www.rocksclusters.org/), >which is free. However, I don't know how well that'll work with the >diskless setup you want. As far as I know, Scyld/clustermatic are the >only ones that'll have a diskless setup "just work". If that's what's >most important to you, then I say give them a try. Otherwise, you may >want to look around. > >Again I'll note, all my info is about a year and a half old. Things may >(or may not) have changed since then. > > I am happily using Rocks myself (version 3.1.0). I can confirm that it doesn't work in a diskless setup. I doubt that it would work with the "1GB for /tmp and swap" setup, either. -- Kip Lubliner Programmer Analyst, Biomathematics, Mount Sinai School of Medicine voice: (212) 241-6902 fax: (212) 241-5851 -- My opinions may have changed, but not the fact that I am right. Ashleigh Brilliant -- From roger at ERC.MsState.Edu Wed May 26 08:13:51 2004 From: roger at ERC.MsState.Edu (Roger L. Smith) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Redmond is at it, again In-Reply-To: <40B3BDAE.4090400@charter.net> References: <20040525134414.GU1105@leitl.org> <40B3BDAE.4090400@charter.net> Message-ID: On Tue, 25 May 2004, Jeffrey B. Layton wrote: > For some people, me being one of them, this is a HUGE threat. Let me > explain. > > Redhat, Suse, etc. have gone to a per-CPU price for license > and support driving up the costs of installing the OS on a cluster. > In fact, the last quote I got for a 72 node cluster averaged about > $400 a node! Amen! I recently talked to RedHat about purchasing RHEL for my systems. Based on my needs for two large clusters and a smattering of desktops, they were asking nearly $10,000 PER YEAR for it, and that was educational pricing! Our university has a major licensing agreement with Novell, and it looks like SLES will be available to me through that agreement in a few months. But when we started trying to decide how up upgrade our RedHat 7.3 installations, we were stymied. We couldn't justify the costs of the commercial distros, I need something a little more stable than Fedora, and I'm just a bit uneasy about converting these large production clusters to any of the other distros. Part of this reluctance is because I'm sure if I have problems with my commercial compilers or other commercial software that I run, they're not going to want to give me support if I'm running some distro that they've never heard of. I'm willing to spend a little money to get the software, but not the rates that they're demanding. _\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_\|/_ | Roger L. Smith Phone: 662-325-3625 | | Sr. Systems Administrator FAX: 662-325-7692 | | roger@ERC.MsState.Edu http://WWW.ERC.MsState.Edu/~roger | | Mississippi State University | |____________________________________ERC__________________________________| From kmurphy at dolphinics.com Wed May 26 08:14:25 2004 From: kmurphy at dolphinics.com (Keith Murphy) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] how useful is bproc, and what does Scyld cost? References: <20040525193745.GB75378@piskorski.com> <1085518615.4397.20.camel@pel> Message-ID: <015e01c44334$2328e6e0$06fea8c0@keithhcmg9ux42> I believe that Linux Labs www.linuxlabs.com has a bproc distro called Nimbus. Good Luck ----- Original Message ----- From: "Sean Dilda" To: "Andrew Piskorski" Cc: Sent: Tuesday, May 25, 2004 1:56 PM Subject: Re: [Beowulf] how useful is bproc, and what does Scyld cost? > There are some pros and cons to bproc (just like everything else). My > experience with bproc is limited to some older versions of Scyld. I've > never used clustermatic. > > In my opinion, if you are already used the tricks to simplify > maintaining a decent sized number machines, then bproc hurts your more > than it helps you. However, if you're not used to using things like > kickstart and yum to quickly add nodes and keep them up to date, then a > bproc system can make your life much easier. > > There are definately some pitfalls to BProc. As it was when I last used > it (late 2002), BProc systems read the binary off the master node, then > caused it to be run on the slave node (I'm simplifying things here) > without copying the binary onto the slave node. This means there's not > really any filesystem you need to keep up on the slave node, which is a > big plus. On the down side, there's no file system on the slave node. > This means that if there are any files your program wants to access, > you're out of luck. You have to know what they are beforehand and > either copy them over, or have them exported by NFS. > > In your case, you're writing a custom app, so its possible that you can > work around the BProc limitations without much trouble. But then again, > your problem may not allow that. BProc's downfalls tend to manifest the > most with 3rd party apps. > > As far as I know, Scyld and Clustermatic are the only distros with > BProc. At one point Progeny was talking about doing some stuff with > BProc, but I don't know what ever came of that. > > Another thing to note is that clustermatic and scyld probably don't have > the same version of bproc, beoboot, etc. Some time back new versions of > the software was coming out of Los Alamos, but Scyld was reluctant to > use those newer versions, and tended to keep hacking on older versions. > So Scyld may have some features clustermatic doesn't have, and vice > versa. > > I don't think I can speak on how much Scyld costs. > > If you're more interesting in having a turn-key cluster solution, I have > some friends who are happy with Rocks (http://www.rocksclusters.org/), > which is free. However, I don't know how well that'll work with the > diskless setup you want. As far as I know, Scyld/clustermatic are the > only ones that'll have a diskless setup "just work". If that's what's > most important to you, then I say give them a try. Otherwise, you may > want to look around. > > Again I'll note, all my info is about a year and a half old. Things may > (or may not) have changed since then. > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > From atp at piskorski.com Wed May 26 08:45:57 2004 From: atp at piskorski.com (Andrew Piskorski) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] how useful is bproc, and what does Scyld cost? In-Reply-To: <015e01c44334$2328e6e0$06fea8c0@keithhcmg9ux42> References: <20040525193745.GB75378@piskorski.com> <1085518615.4397.20.camel@pel> <015e01c44334$2328e6e0$06fea8c0@keithhcmg9ux42> Message-ID: <20040526154556.GA85083@piskorski.com> On Wed, May 26, 2004 at 08:14:25AM -0700, Keith Murphy wrote: > I believe that Linux Labs www.linuxlabs.com has a bproc distro called > Nimbus. http://www.linuxlabs.com/nimbus.html Ah, and they claim to support both Debian and Red Hat. Thanks! Has anyone here used Nimbus? Any comments or lessons learned? Hm, those are also the same folks offering "Clusgres", a clever sonding hack to run PostgreSQL on a cluster using SCI, which I commented on elsewhere back in Feb.: http://openacs.org/forums/message-view?message_id=128060 Unfortunatley the same complaint I had then still applies, they just plain don't have all that much real info on their website. But, Nimbus does sound promising, so I'll contact them. -- Andrew Piskorski http://www.piskorski.com/ From deadline at linux-mag.com Wed May 26 08:45:14 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Redmond is at it, again In-Reply-To: Message-ID: On Tue, 25 May 2004, Robert G. Brown wrote: --snip-- > > So the interesting question is "where's the money" in this move. The > HPC market hasn't proven to be exactly a get-rich-quick proposition for > any of the various groups that have prospected in it. Good question. I believe the move by Microsoft has two logical goals: 1. Enter markets where the return will be good (which is why the article points to financial and cycle scavenging markets -- these markets make sense for them) 2. Help mitigate "Linux Creep". I recall a customer who wanted to call their cluster kudzu (a non-indigenous plant brought from Asia to the southern US that grows over everything.) It is great name because Linux is like kudzu. In some cases , it is growing all over the data center. It came in the "HPC back door". The message that "Gee Linux helps us find oil, build airplanes, search the genome", adds quite a bit of credibility to the whole Linux thing. I think we just got legit. Doug -- ---------------------------------------------------------------- Editor-in-chief ClusterWorld Magazine Desk: 610.865.6061 Cell: 610.390.7765 Redefining High Performance Computing Fax: 610.865.6618 www.clusterworld.com From mwill at penguincomputing.com Wed May 26 10:23:40 2004 From: mwill at penguincomputing.com (Michael Will) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Possibly OT In-Reply-To: <20040525230451.6411350e@barton> References: <20040525230451.6411350e@barton> Message-ID: <200405261023.40475.mwill@penguincomputing.com> Maybe somebody has a cool beowulf setup for this, but in my opinion this is more a task for an LVS (linux virtual server) setup with heartbeat. There are two issues you are trying to solve: One is loadbalancing, can be done with one frontend machine using LVS (linux virtual server) to forward web-sessions to one of several webservers. The other is high availability so that this one machine going down is being replaced by a standby, taking over its IP address. Check out www.ultramonkey.org for a lot of good software and documentation to the topic. I just last week set up a two server HA (active/passive) solution for running mysql and a customer application connecting to it, using a fibre attached nStor for external storage, and it was not too hard. All the information I needed I found on ultramonkey.org. Michael On Tuesday 25 May 2004 03:04 pm, Ed wrote: > I have to provide a webhost, but with low spec computers which we have > in abndance. > > Now, I assume that the webservice will use the single IP address > x.x.x.x... -- Michael Will, Linux Sales Engineer NEWS: We have moved to a larger iceberg :-) NEWS: 300 California St., San Francisco, CA. Tel: 415-954-2822 Toll Free: 888-PENGUIN Fax: 415-954-2899 www.penguincomputing.com From laurenceliew at yahoo.com.sg Thu May 27 01:38:25 2004 From: laurenceliew at yahoo.com.sg (Laurence Liew) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] Redmond is at it, again In-Reply-To: <40B3BDAE.4090400@charter.net> References: <20040525134414.GU1105@leitl.org> <40B3BDAE.4090400@charter.net> Message-ID: <1085647104.5221.209.camel@localhost.localdomain> Hi all > > If you work for a disro company and you are reading this, > > all I can say is, WAKE UP! You are going down the tubes fast > > in the HPC market. If you work for a cluster vendor and you are > > reading this, please push your management hard to adopt at > > least one open distro. We'll pay for support for it, but not using > > the current pricing scheme that the Redhat's, Suse, etc. are > > charging. So what would be a per node price people (edu? commercial?) would pay for the OS with support and updates? Do note that support for HPC clusters requires technically competent and HPC savvy engineers and these do not come cheap.... Based on previous discussions and email.. it seems that USD$50 - USD$100 per node to be about right/acceptable for a cluster OS that is supported with updates and patches and basic support for HPC type questions. Comments? Cheers! Laurence ps. I work for a cluster vendor. On Wed, 2004-05-26 at 05:42, Jeffrey B. Layton wrote: > For some people, me being one of them, this is a HUGE threat. Let me > explain. > > Redhat, Suse, etc. have gone to a per-CPU price for license > and support driving up the costs of installing the OS on a cluster. > In fact, the last quote I got for a 72 node cluster averaged about > $400 a node! Now management has started to compare this price > to Windows and guess what? - Windows is cheaper on a per > node basis. This includes purchasing and support! We already > have a massive license with MS so we get Windows pretty > cheap, but I can get Windows myself cheaper than $400 a node. > We started to talk to management about the support costs, etc. > but they argue that they already have a whole bunch of admins > trained in Windows so support costs should be low (I tried to > explain the difference between admining a cluster and admining > a Windows server). Plus we have an infrastructure to push > updates to Windows machines, so they though updates would > be a no-brainer as well. > > At the ClusterWorld conference I tried to talk to various distro > vendors about pricing, but they weren't listening. I don't mind > paying a relatively small fee to purchase a distro, but paying a > support costs for each CPU is nuts! If I have a problem on one > node, I have a problem on ALL nodes. The distro companies > have yet to understand this. Novell is starting to come around > a bit, but they still aren't there yet. > > We are encouraging cluster vendors to support at least one > open distro. That way we aren't held hostage to these companies > that don't understand HPC. I don't know if we're making any > head way, but I constantly hope we are. We're also trying to > push IT management into "approving" a couple of open distros, > but they are reluctant because they don't see a company behind > them (they insist on the "one throat to choke" model of support). > However, I'm still hopefully. > > If you work for a disro company and you are reading this, > all I can say is, WAKE UP! You are going down the tubes fast > in the HPC market. If you work for a cluster vendor and you are > reading this, please push your management hard to adopt at > least one open distro. We'll pay for support for it, but not using > the current pricing scheme that the Redhat's, Suse, etc. are > charging. > > If you work for a company that sells commercial software for > Linux (e.g. compilers, MPI, etc.), please support more than > RHEL and SLES! Think seriously about supporting an open > distro and also supporting a kernel/glibc combo. > > I'm sure I'm not the only one who feels this way. Let's tell the > distro companies that we won't stand for it! > > Thanks! > > Jeff From gerry.creager at tamu.edu Thu May 27 07:56:44 2004 From: gerry.creager at tamu.edu (Gerry Creager N5JXS) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy In-Reply-To: References: Message-ID: <40B601AC.9020002@tamu.edu> This is really a first-cut response, with 2 visible possibilities... 1. Use 2 license servers, one with 'i' licenses available for short jobs, and one with 'j' licenses available for longer jobs. For i < j, starvation of the short jobs shouldn't occur too often, save when there's too many masters' students trying to get their projects done in time to graduate and the deadline's tomorrow. 2. Priority queuing where short jobs have the nod, and longer jobs are put aside and required to temporarily relinquish licenses. Liketo to require programming resources to accomplish this one. Good question. Gerry Robert G. Brown wrote: > Dear Perfect Masters of Grid Computing: > > Economics is preparing to set up a small pilot cluster at Duke and the > following question has come up. > > Primary tasks: matlab and stata jobs, run either interactively/remote > or (more likely) in batch mode. Jobs include both "short" jobs that > might take 10-30 minutes run by e.g. 1-2nd year graduate students as > part of their coursework and "long" jobs that might take hours to days > run by more advanced students, postdocs, faculty. > > Constraint: matlab requires a license managed by a license manager. > There are a finite number of licenses (currently less than the number of > CPUs) spread out across the pool of CPUs. > > Concern: That long running jobs will get into the queue (probably SGE > managed queue) and starve the short running jobs for either licenses or > CPUs or both. Students won't be able to finish their homework in a > timely way because long running jobs de facto hog the resource once they > are given a license/CPU. > > I am NOT an SGE expert, although I've played with it a bit and read a > fair bit of the documention. SGE appears to run in FIFO mode, which of > course would lead to precisely the sort of resource starvation feared or > equal share mode. Equal share mode appears to solve a different > resource starvation problem -- that produced by a single user or group > saturating the queue with lots of jobs, little or big, so that others > submitting after they've loaded the queue have to wait days or weeks to > get on. However, it doesn't seem to have anything to do with job > >>>control<< according to a policy -- stopping a long running job so that > > a short running job can pass through. > > It seems like this would be a common problem in shared environments with > a highly mixed workload and lots of users (and indeed is the problem > addressed by e.g. the kernel scheduler in almost precisely the same > context on SMP or UP machines). Recognizing that the license management > problem will almost certainly be beyond the scope of any solution > without some hacking and human-level policy, are there any well known > solutions to this well known problem? Can SGE actually automagically > control jobs (stopping and starting jobs as a sort of coarse-grained > scheduler to permit high priority jobs to pass through long running low > priority jobs)? Is there a way to solve this with job classes or > wrapper scripts that is in common use? > > At your feet, your humble student waits, oh masters of SGE and Grids... > > rgb > -- Gerry Creager -- gerry.creager@tamu.edu Network Engineering -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578 Page: 979.228.0173 Office: 903A Eller Bldg, TAMU, College Station, TX 77843 From dag at sonsorol.org Thu May 27 08:04:14 2004 From: dag at sonsorol.org (Chris Dagdigian) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy In-Reply-To: References: Message-ID: <40B6036E.1090403@sonsorol.org> Here are some personal notes I've collected on license management and policies under Grid Engine. Perhaps they may be helpful? FLEXlm license integration with grid engine 5.3: http://bioteam.net/dag/sge-flexlm-integration/ Grid Engine can do very sophisticated resource allocation above and beyond FIFO ordering. Your need to *stop* a long running job to let a shorter one in is a bit harder as graceful job premption/supension is something that normally takes a bit of configuring (both of the app and of Grid Engine). There are other mechanisms in grid engine that people have used to prevent the 'job starvation' issue. The SGE mailing list users@gridengine.sunsource.net (I think you must be a member to post) or the searchable mailing list archives at http://gridengine.sunsource.net may be very helpful. In particular you may find some of the "enterprise edition" policies such as 'share tree' or 'functional policy' may help. They won't boot running jobs unless you configure it to but the policies themselves can help you gain 'fairshare-by-user' or 'fairshare-by-project' style resource allocation which may meet your needs. Also-- There is a new feature in Grid Engine 6.0 built into the new "Urgency scheduling" policy -- a configurable parameter called $WaitTimeUrgency that causes a jobs' urgency priority to grow the longer it has been sitting in pending state. I've been told that one of the drivers for "waitTimeUrgency" was to address pending-job-starvation issues. The problem with Urgency Scheduling is that I don't think it has been in use by many people for all that long. Once 6.0 comes out of beta in a few weeks there may be better user reports on how Urgency scheduling is doing in the real world. Since documentation is seriously lacking for Grid Engine 6.0 and the Sun branded version called 'N1GE 6' I've tried to write down what I've learned about them here: http://bioteam.net/dag/gridengine-6-features.html Regards, Chris Robert G. Brown wrote: > Dear Perfect Masters of Grid Computing: > > Economics is preparing to set up a small pilot cluster at Duke and the > following question has come up. > > Primary tasks: matlab and stata jobs, run either interactively/remote > or (more likely) in batch mode. Jobs include both "short" jobs that > might take 10-30 minutes run by e.g. 1-2nd year graduate students as > part of their coursework and "long" jobs that might take hours to days > run by more advanced students, postdocs, faculty. > > Constraint: matlab requires a license managed by a license manager. > There are a finite number of licenses (currently less than the number of > CPUs) spread out across the pool of CPUs. > > Concern: That long running jobs will get into the queue (probably SGE > managed queue) and starve the short running jobs for either licenses or > CPUs or both. Students won't be able to finish their homework in a > timely way because long running jobs de facto hog the resource once they > are given a license/CPU. > > I am NOT an SGE expert, although I've played with it a bit and read a > fair bit of the documention. SGE appears to run in FIFO mode, which of > course would lead to precisely the sort of resource starvation feared or > equal share mode. Equal share mode appears to solve a different > resource starvation problem -- that produced by a single user or group > saturating the queue with lots of jobs, little or big, so that others > submitting after they've loaded the queue have to wait days or weeks to > get on. However, it doesn't seem to have anything to do with job > >>>control<< according to a policy -- stopping a long running job so that > > a short running job can pass through. > > It seems like this would be a common problem in shared environments with > a highly mixed workload and lots of users (and indeed is the problem > addressed by e.g. the kernel scheduler in almost precisely the same > context on SMP or UP machines). Recognizing that the license management > problem will almost certainly be beyond the scope of any solution > without some hacking and human-level policy, are there any well known > solutions to this well known problem? Can SGE actually automagically > control jobs (stopping and starting jobs as a sort of coarse-grained > scheduler to permit high priority jobs to pass through long running low > priority jobs)? Is there a way to solve this with job classes or > wrapper scripts that is in common use? > > At your feet, your humble student waits, oh masters of SGE and Grids... > > rgb > -- Chris Dagdigian, BioTeam - Independent life science IT & informatics consulting Office: 617-665-6088, Mobile: 617-877-5498, Fax: 425-699-0193 PGP KeyID: 83D4310E iChat/AIM: bioteamdag Web: http://bioteam.net From orion at cora.nwra.com Thu May 27 08:07:58 2004 From: orion at cora.nwra.com (Orion Poplawski) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy In-Reply-To: References: Message-ID: <40B6044E.7040400@cora.nwra.com> Robert G. Brown wrote: > > Primary tasks: matlab and stata jobs, run either interactively/remote > or (more likely) in batch mode. Jobs include both "short" jobs that > might take 10-30 minutes run by e.g. 1-2nd year graduate students as > part of their coursework and "long" jobs that might take hours to days > run by more advanced students, postdocs, faculty. > > Constraint: matlab requires a license managed by a license manager. > There are a finite number of licenses (currently less than the number of > CPUs) spread out across the pool of CPUs. > > Concern: That long running jobs will get into the queue (probably SGE > managed queue) and starve the short running jobs for either licenses or > CPUs or both. Students won't be able to finish their homework in a > timely way because long running jobs de facto hog the resource once they > are given a license/CPU. > > I am NOT an SGE expert, although I've played with it a bit and read a > fair bit of the documention. SGE appears to run in FIFO mode, which of > course would lead to precisely the sort of resource starvation feared or > equal share mode. Equal share mode appears to solve a different > resource starvation problem -- that produced by a single user or group > saturating the queue with lots of jobs, little or big, so that others > submitting after they've loaded the queue have to wait days or weeks to > get on. However, it doesn't seem to have anything to do with job > >>>control<< according to a policy -- stopping a long running job so that > > a short running job can pass through. > > It seems like this would be a common problem in shared environments with > a highly mixed workload and lots of users (and indeed is the problem > addressed by e.g. the kernel scheduler in almost precisely the same > context on SMP or UP machines). Recognizing that the license management > problem will almost certainly be beyond the scope of any solution > without some hacking and human-level policy, are there any well known > solutions to this well known problem? Can SGE actually automagically > control jobs (stopping and starting jobs as a sort of coarse-grained > scheduler to permit high priority jobs to pass through long running low > priority jobs)? Is there a way to solve this with job classes or > wrapper scripts that is in common use? Your biggest problem (as you say) will be licenses. I believe the scheduler tries to evenly allocate the running jobs among the different submitters. So, if you start with 14 empty slots and one person submits 500 jobs, they get filled with those jobs. But if someone else then submits 20 jobs, that person will eventually be given the next 7 slots to run as jobs complete, and will stay split until the second user has no more jobs. You can also setup subordinate queues where the queue with priority will only accept jobs that will take less than a given time to run. When a short job gets submitted, a long job on the subordinate queue will be stopped (SIGSTOP) while the short jobs runs. Your problem here is that the long job will presumably still hold a license. If matlab has some kind of checkpointing function, you could tie that into SGE to release the license. You can also limit the number of available slots for long running jobs. SGE can tie into license management through resource monitors to determine the number of licenses available. -- Orion Poplawski System Administrator 303-415-9701 x222 Colorado Research Associates/NWRA FAX: 303-415-9702 3380 Mitchell Lane, Boulder CO 80301 http://www.co-ra.com From david.n.lombard at intel.com Thu May 27 08:29:59 2004 From: david.n.lombard at intel.com (Lombard, David N) Date: Wed Nov 25 01:03:11 2009 Subject: R: [Beowulf] SGE + policy Message-ID: <187D3A7CAB42A54DB61F1D05F0125722025F5850@orsmsx402.amr.corp.intel.com> [Forwarding to list] From: Robert G. Brown, > Dear Perfect Masters of Grid Computing: Robert, you are one of the few "masters" on this list ;^) > It seems like this would be a common problem in shared environments with > a highly mixed workload and lots of users (and indeed is the problem > addressed by e.g. the kernel scheduler in almost precisely the same > context on SMP or UP machines). Recognizing that the license management > problem will almost certainly be beyond the scope of any solution > without some hacking and human-level policy, are there any well known > solutions to this well known problem? Can SGE actually automagically > control jobs (stopping and starting jobs as a sort of coarse-grained > scheduler to permit high priority jobs to pass through long running low > priority jobs)? Is there a way to solve this with job classes or > wrapper scripts that is in common use? LSF is sometimes used for this very purpose, i.e., to manage access to licensed jobs so that no job fails due to lack of licenses. LSF has "exits" that can handle the license accounting, e.g., checking for available licenses, fully automating the process. For SGE, on which I only have passing familiarity, could you create a matlab-specific queue and limit the number of concurrent jobs? Certainly the various PBS implementations can do this. Alternatively, perhaps a matlab-specific resource could be required by each job and limit it that way? -- David N. Lombard My comments represent my opinions, not those of Intel Corporation. From apseyed at bu.edu Thu May 27 08:16:05 2004 From: apseyed at bu.edu (Patrice Seyed) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy In-Reply-To: Message-ID: <000d01c443fd$87ce5610$26a0299b@smallfry> Dr. Brown, The convention that I've used as suggested by the gurus on the SGE mailing list is the use of the concept of express queues, which uses a resource assignment for "express" and subordinate queuing. SGE usually sets one queue per host (if its dual this needs to be modified slightly). Say your first node has one cpu and is called node-1. Set up a usual queue called "node-1.q" with one job slot, and set it up to be subordinate to "express-1.q" at the 1 job level, and create a queue called "express-1.q" that has one job slot, create a resource called "express" for this queue, and set a soft/hard limit of rt to 2:00. Basically addresses the scenario where a user wants to submit a job that takes less than 2 hours and all the regular queues are full. They can submit their jobs with the "-l express=1" option and the job will go into an express queue belonging to one of the hosts, will suspend the long job in the regular queue until the express job is complete. What makes this work is the restriction the hard limit of 2 hours for this suspension mechanism. I hope this helps. Regarding the license managing you could do something with consumable resources/tracking, also I think that you can use FlexLM. http://bioteam.net/dag/sge-flexlm-integration/ http://gridengine.sunsource.net/project/gridengine/howto/howto.html -Patrice -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Robert G. Brown Sent: Thursday, May 27, 2004 10:19 AM To: Beowulf Mailing List Subject: [Beowulf] SGE + policy Dear Perfect Masters of Grid Computing: Economics is preparing to set up a small pilot cluster at Duke and the following question has come up. Primary tasks: matlab and stata jobs, run either interactively/remote or (more likely) in batch mode. Jobs include both "short" jobs that might take 10-30 minutes run by e.g. 1-2nd year graduate students as part of their coursework and "long" jobs that might take hours to days run by more advanced students, postdocs, faculty. Constraint: matlab requires a license managed by a license manager. There are a finite number of licenses (currently less than the number of CPUs) spread out across the pool of CPUs. Concern: That long running jobs will get into the queue (probably SGE managed queue) and starve the short running jobs for either licenses or CPUs or both. Students won't be able to finish their homework in a timely way because long running jobs de facto hog the resource once they are given a license/CPU. I am NOT an SGE expert, although I've played with it a bit and read a fair bit of the documention. SGE appears to run in FIFO mode, which of course would lead to precisely the sort of resource starvation feared or equal share mode. Equal share mode appears to solve a different resource starvation problem -- that produced by a single user or group saturating the queue with lots of jobs, little or big, so that others submitting after they've loaded the queue have to wait days or weeks to get on. However, it doesn't seem to have anything to do with job >>control<< according to a policy -- stopping a long running job so that a short running job can pass through. It seems like this would be a common problem in shared environments with a highly mixed workload and lots of users (and indeed is the problem addressed by e.g. the kernel scheduler in almost precisely the same context on SMP or UP machines). Recognizing that the license management problem will almost certainly be beyond the scope of any solution without some hacking and human-level policy, are there any well known solutions to this well known problem? Can SGE actually automagically control jobs (stopping and starting jobs as a sort of coarse-grained scheduler to permit high priority jobs to pass through long running low priority jobs)? Is there a way to solve this with job classes or wrapper scripts that is in common use? At your feet, your humble student waits, oh masters of SGE and Grids... rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From landman at scalableinformatics.com Thu May 27 08:49:30 2004 From: landman at scalableinformatics.com (Joe Landman) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy In-Reply-To: References: Message-ID: <40B60E0A.3060608@scalableinformatics.com> Consumable resources are your friends. As is the grid-engine mailing list. This is a bit different than the other license management problems discussed there, so the previous solutions are probably not appropriate, but could be used as the basis for something that is appropriate. Create 2 consumables, one for the short runs, one for the longer runs. Have the sum of the consumable slots equal the number of licenses. Interactive/short users submit to their consumable, and long running users submit to theirs. You really don't want to invoke checkpointing under SGE at this time. SGE (and most schedulers under linux) don't checkpoint, they kill the job. Condor has application libs which do checkpoint, but I have no idea how reliably this works. Moreover, checkpointing a job with an open license request tends to lock that license. If the application itself can checkpoint, you can hook that into the SGE checkpoint mechanism to use that checkpoint rather than the OS feature. This is not a great solution (2 consumables), but it does work, and can be adjusted/tuned on a live cluster trivially. You setup scripts to launch jobs that detect the return code of the Matlab/other job, and send the appropriate return code back to SGE (99 as I remember) to have the job rescheduled if no licenses are available. This of course requires Matlab/other to be able to intelligently let you know if there are no licenses available... Robert G. Brown wrote: >Dear Perfect Masters of Grid Computing: > >Economics is preparing to set up a small pilot cluster at Duke and the >following question has come up. > >Primary tasks: matlab and stata jobs, run either interactively/remote >or (more likely) in batch mode. Jobs include both "short" jobs that >might take 10-30 minutes run by e.g. 1-2nd year graduate students as >part of their coursework and "long" jobs that might take hours to days >run by more advanced students, postdocs, faculty. > >Constraint: matlab requires a license managed by a license manager. >There are a finite number of licenses (currently less than the number of >CPUs) spread out across the pool of CPUs. > >Concern: That long running jobs will get into the queue (probably SGE >managed queue) and starve the short running jobs for either licenses or >CPUs or both. Students won't be able to finish their homework in a >timely way because long running jobs de facto hog the resource once they >are given a license/CPU. > >I am NOT an SGE expert, although I've played with it a bit and read a >fair bit of the documention. SGE appears to run in FIFO mode, which of >course would lead to precisely the sort of resource starvation feared or >equal share mode. Equal share mode appears to solve a different >resource starvation problem -- that produced by a single user or group >saturating the queue with lots of jobs, little or big, so that others >submitting after they've loaded the queue have to wait days or weeks to >get on. However, it doesn't seem to have anything to do with job > > >>>control<< according to a policy -- stopping a long running job so that >>> >>> >a short running job can pass through. > >It seems like this would be a common problem in shared environments with >a highly mixed workload and lots of users (and indeed is the problem >addressed by e.g. the kernel scheduler in almost precisely the same >context on SMP or UP machines). Recognizing that the license management >problem will almost certainly be beyond the scope of any solution >without some hacking and human-level policy, are there any well known >solutions to this well known problem? Can SGE actually automagically >control jobs (stopping and starting jobs as a sort of coarse-grained >scheduler to permit high priority jobs to pass through long running low >priority jobs)? Is there a way to solve this with job classes or >wrapper scripts that is in common use? > >At your feet, your humble student waits, oh masters of SGE and Grids... > > rgb > > > From James.P.Lux at jpl.nasa.gov Thu May 27 09:14:26 2004 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy In-Reply-To: Message-ID: <5.2.0.9.2.20040527090632.017b4250@mailhost4.jpl.nasa.gov> At 10:19 AM 5/27/2004 -0400, Robert G. Brown wrote: >Dear Perfect Masters of Grid Computing: > >Economics is preparing to set up a small pilot cluster at Duke and the >following question has come up. > >Primary tasks: matlab and stata jobs, run either interactively/remote >or (more likely) in batch mode. Jobs include both "short" jobs that >might take 10-30 minutes run by e.g. 1-2nd year graduate students as >part of their coursework and "long" jobs that might take hours to days >run by more advanced students, postdocs, faculty. > >Constraint: matlab requires a license managed by a license manager. >There are a finite number of licenses (currently less than the number of >CPUs) spread out across the pool of CPUs. I can't speak to the interaction between Matlab and SGE, however, I do have some practical experience with competing for shared Matlab licenses (particularly for some toolboxes).. In some ways, what you are dealing with is a common problem in real time systems with phenomena like deadlocks and priority inversion, where a long running low priority task locks out higher priority tasks because of a resource lock. Perhaps a way to deal with the long running resource consumer type job is to implement a requirement that they be periodically checkpointed, stopped, and restarted, allowing a "rebidding" for the limited resource. Good software design practice should inspire designs that support this fairly transparently (i.e. it's foolish to write a program that must run uninterrupted for many hours). If the time granularity is sufficiently fine, then it should work ok. There was some research back in the 70's on optimizing scarce computing resources among various classes of users with different needs(timesharing interactive users: low compute demand, but fast response time needed vs. high compute, but batch oriented). Some successful strategies implemented a form of bidding for resources. The idea is that each "user or consumer" gets a certain regular salary, and they can either explicitly or automatically enter a dynamic market for the resources. The market can have some "allocations or quotas" to prevent things analogous to priority inversions. James Lux, P.E. Spacecraft Telecommunications Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 From gmpc at sanger.ac.uk Thu May 27 09:22:17 2004 From: gmpc at sanger.ac.uk (Guy Coates) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy In-Reply-To: References: Message-ID: > Concern: That long running jobs will get into the queue (probably SGE > managed queue) and starve the short running jobs for either licenses or > CPUs or both. Students won't be able to finish their homework in a > timely way because long running jobs de facto hog the resource once they > are given a license/CPU. You are screwed, but not for the reason you think. This is not a scheduler issue; it's a flexlm limitation. Most queueing systems support pre-emption, when the scheduler will suspend long running jobs to free up job slots/CPUs in favour of short high priority jobs. Unfortunately, as far as flexlm is concerned, your suspended long job has still checked out a license, so you get license starvation, and your short jobs will not be able to start. The only way around this limitation would be to checkpoint and kill the long job, so the license get returned. (Can you checkpoint matlab...?) Cheers, Guy Coates -- Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK Tel: +44 (0)1223 834244 ex 7199 From sean at duke.edu Thu May 27 11:44:17 2004 From: sean at duke.edu (Sean Dilda) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] SGE + policy In-Reply-To: References: Message-ID: <1085683456.4397.40.camel@pel> On Thu, 2004-05-27 at 11:13, Robert G. Brown wrote: > On Thu, 27 May 2004, Orion Poplawski wrote: > > > You can also setup subordinate queues where the queue with priority will > > only accept jobs that will take less than a given time to run. When a > > short job gets submitted, a long job on the subordinate queue will be > > stopped (SIGSTOP) while the short jobs runs. Your problem here is that > > the long job will presumably still hold a license. If matlab has some > > kind of checkpointing function, you could tie that into SGE to release > > the license. > > So it WILL manage things with SIGSTOP/SIGCONT. Good, that's what I was > hoping. I'll search the docs etc for "subordinate queues" and signals > to see if I can figure out how. We were using subordinate queues for a while with the "owned" nodes in the CSEM cluster. Unfortunately, it turns out that subordinate queues and MPI jobs don't play well together, but I don't think Econ is planning on doing much with MPI jobs. One thing you might want to consider is adding a complex to the "high priority" queue. This is just a resource that can be requested in order to get a job submitted into the high priority queue. If you set the complex's requestable flag to FORCED, then it'll only put jobs in those queues if they specifically request the complex. You may also want to look into resource limits. You can place these on the "high priority" queues so that if jobs run for too long, they will be killed. Note, I haven't used resource limits, so I'm not sure how well they work. Based on some of my experience with SGE, I think it might be possible to get around the kill signal from them, but I haven't played with it enough to be certain. Also, if you look at http://gridengine.sunsource.net/, they have a users list for SGE that might also be a good source of info for you. Sean From camm at enhanced.com Thu May 27 12:41:43 2004 From: camm at enhanced.com (Camm Maguire) Date: Wed Nov 25 01:03:11 2009 Subject: [Beowulf] how useful is bproc, and what does Scyld cost? In-Reply-To: <20040525193745.GB75378@piskorski.com> References: <20040525193745.GB75378@piskorski.com> Message-ID: <543c5llkso.fsf@intech19.enhanced.com> Greetings! No direct experience with diskless setups, but all the cluster applications I need work out of the box on Debian. There are even precompiled atlas libraries for several common cpu subarchitectures. We've been running ours for about 7 years -- 1 software install, 3 hardware upgrades, and nothing but 'apt-get upgrade' since. Take care, Andrew Piskorski writes: > I'm tentatively planning a small cluster that might or might not > actually get built. My current plan is somewhere from 5-20 nodes, 1-2 > x86 CPUs per node (exact CPU flavor undecided), gigabit ethernet, and > all nodes either entirely diskless, or using 1 IDE disk solely for > swap and /tmp. > > I would prefer to have as much as posible of the cluster software > infrastructure Just Work, rather than having to spend lots of time > rolling my own. (I will be spending enough time on the custom > software I actually want to RUN on the cluster as is.) I am, of > course, quite willing to select hardware in order to make the software > job easier on myself. > > Since I want to go diskless anyway, so far I am also leaning towards a > bproc based cluster. I only know of two bproc-based cluster > distributions, Scyld and Clustermatic. Scyld is commerical and costs > money, Clustermatic is not and does not. Are there any others? In > particular, are there any Debian based systems that play nicely out of > the box with bproc? > > How much time and effort is Scyld actually going to save me over using > Clustermatic? How much is either going to save me over completely > rolling my own, preferably using Debian rather than the old and > outdated versions of Red Hat that Scyld and Clustermatic seem to use? > Also, are there any major drawbacks or snafus I should worry about in > going down the bproc route? > > Finally, just what DOES Scyld actually cost? Can anyone give me a > rough idea? > > >From Scyld's website, I can't tell whether they charge 50 cents or > $5,000 per node, and the Scyld/Penguin salesman seemed unable to spit > out any kind of ballpark price at all. AFAICT, Scyld seems to expect > you to first actually build your cluster, and then send them your > cluster's complete hardware specs, down to the smallest detail, in > order to get any kind of quote! > > Thanks! > > -- > Andrew Piskorski > http://www.piskorski.com/ > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > > -- Camm Maguire camm@enhanced.com ========================================================================== "The earth is but one country, and mankind its citizens." -- Baha'u'llah From atp at piskorski.com Thu May 27 13:14:34 2004 From: atp at piskorski.com (Andrew Piskorski) Date: Wed Nov 25 01:03:12 2009 Subject: [Beowulf] Re: SGE + policy In-Reply-To: <200405271901.i4RJ04vr004109@bluewest.scyld.com> References: <200405271901.i4RJ04vr004109@bluewest.scyld.com> Message-ID: <20040527201434.GA41179@piskorski.com> On Thu, May 27, 2004 at 12:01:09PM -0700, beowulf-request@beowulf.org wrote: > Date: Thu, 27 May 2004 10:19:00 -0400 (EDT) > From: "Robert G. Brown" > Primary tasks: matlab and stata jobs, run either interactively/remote > or (more likely) in batch mode. Jobs include both "short" jobs that > Constraint: matlab requires a license managed by a license manager. Have they considered running Octave rather than Matlab? Depending on what special-purpose Matlab add-on modules they're using, that might or might not be practical, but if it is, using Octave should reduce their cluster and license hassles. (I've never used either of the two dialects myself though, so have them ask someone who really knows...) -- Andrew Piskorski http://www.piskorski.com/ From hazelsct at debian.org Thu May 27 13:50:20 2004 From: hazelsct at debian.org (Adam C Powell IV) Date: Wed Nov 25 01:03:12 2009 Subject: [Beowulf] how useful is bproc, and what does Scyld cost? In-Reply-To: <543c5llkso.fsf@intech19.enhanced.com> References: <20040525193745.GB75378@piskorski.com> <543c5llkso.fsf@intech19.enhanced.com> Message-ID: <1085691020.3347.33.camel@p4-117-1.mit.edu> There's also pretty complete community-maintained documentation at http://wiki.debian.net/ with the relevant material at http://wiki.debian.net/index.cgi?DebianBeowulf On Thu, 2004-05-27 at 15:41, Camm Maguire wrote: > Greetings! No direct experience with diskless setups, but all the > cluster applications I need work out of the box on Debian. There are > even precompiled atlas libraries for several common cpu > subarchitectures. We've been running ours for about 7 years -- 1 > software install, 3 hardware upgrades, and nothing but 'apt-get > upgrade' since. > > Take care, > > Andrew Piskorski writes: > > > I'm tentatively planning a small cluster that might or might not > > actually get built. My current plan is somewhere from 5-20 nodes, 1-2 > > x86 CPUs per node (exact CPU flavor undecided), gigabit ethernet, and > > all nodes either entirely diskless, or using 1 IDE disk solely for > > swap and /tmp. > > > > I would prefer to have as much as posible of the cluster software > > infrastructure Just Work, rather than having to spend lots of time > > rolling my own. (I will be spending enough time on the custom > > software I actually want to RUN on the cluster as is.) I am, of > > course, quite willing to select hardware in order to make the software > > job easier on myself. > > > > Since I want to go diskless anyway, so far I am also leaning towards a > > bproc based cluster. I only know of two bproc-based cluster > > distributions, Scyld and Clustermatic. Scyld is commerical and costs > > money, Clustermatic is not and does not. Are there any others? In > > particular, are there any Debian based systems that play nicely out of > > the box with bproc? > > > > How much time and effort is Scyld actually going to save me over using > > Clustermatic? How much is either going to save me over completely > > rolling my own, preferably using Debian rather than the old and > > outdated versions of Red Hat that Scyld and Clustermatic seem to use? > > Also, are there any major drawbacks or snafus I should worry about in > > going down the bproc route? > > > > Finally, just what DOES Scyld actually cost? Can anyone give me a > > rough idea? > > > > >From Scyld's website, I can't tell whether they charge 50 cents or > > $5,000 per node, and the Scyld/Penguin salesman seemed unable to spit > > out any kind of ballpark price at all. AFAICT, Scyld seems to expect > > you to first actually build your cluster, and then send them your > > cluster's complete hardware specs, down to the smallest detail, in > > order to get any kind of quote! > > > > Thanks! -Adam P. GPG fingerprint: D54D 1AEE B11C CE9B A02B C5DD 526F 01E8 564E E4B6 Welcome to the best software in the world today cafe! http://lyre.mit.edu/~powell/The_Best_Stuff_In_The_World_Today_Cafe.ogg From laytonjb at charter.net Thu May 27 16:19:27 2004 From: laytonjb at charter.net (Jeffrey B. Layton) Date: Wed Nov 25 01:03:12 2009 Subject: [Beowulf] Redmond is at it, again In-Reply-To: <20040527222708.GH2452@greglaptop.internal.keyresearch.com> References: <20040525134414.GU1105@leitl.org> <40B3BDAE.4090400@charter.net> <20040527222708.GH2452@greglaptop.internal.keyresearch.com> Message-ID: <40B6777F.9080604@charter.net> Greg Lindahl wrote: >On Tue, May 25, 2004 at 05:42:06PM -0400, Jeffrey B. Layton wrote: > > > >>In fact, the last quote I got for a 72 node cluster averaged about >>$400 a node! Now management has started to compare this price >>to Windows >> >> > >'Doctor, it hurts when I try to buy cluster software from a >non-cluster company!' > I love that quote! It would make a good .sig! >I don't know why you seem to think that only SuSE or Red Hat is the >place to go for Linux support. You mention later in your note that you >do talk to cluster vendors. I know of several who provide Linux >support for a reasonable fee. It would seem that your bean counters >would like that approach. I can't figure out from your explaination >why you don't buy from them. > Well the explanation is long and ugly :) The cluster vendors we talk to offer and support Linux at a reasonable price. However, our management folks don't like these cluster vendors because they aren't big enough. They like the large 2 or 3 letter companies or the short name companies :) These companies won't or didn't quote Linux as part of the integrated cluster. They forced us to buy it and they would install it. Not the best choice in my opinion (which I love to point out to management every time I can), but the decision was made by those higher than me who seemingly share one brain cell. Every time I warn management that large does not mean stable and that Enron and Worldcom are perfect examples, they say that the fate of these two companies won't befall the large computer vendors. When I mention to them that Enron and Worldcom said the same thing and even said that they had money in the bank when they went bankrupt (oh, the wonders of creative accounting), they mumble something and then stop talking about it. I also point out that the companies with the most cluster experience and who sell more cluster systems (i.e. nodes) than anyone else is not a big initial or short name company. They DEFINITELY don't like to hear this. When you throw in that these cluster companies are also usually cheaper, management gets down right irate! Then those familiar words come floating out, "Jeff - shut up." That's about the closest I've come to a compliment yet. So, I hope from my rant - part deux - you see my problem. You can also see that rgb has been a bad influence on me in my mailing list posts :) Thanks! Jeff >-- greg (no longer a cluster vendor, and yes, PathScale's compilers do >support more than just SLES and RHEL -- our customers require it.) > From kmoallem at scs.ryerson.ca Thu May 27 18:32:22 2004 From: kmoallem at scs.ryerson.ca (Kaveh Moallemi) Date: Wed Nov 25 01:03:12 2009 Subject: [Beowulf] how useful is bproc, and what does Scyld cost? (fwd) Message-ID: Hello, I don't think that you should need to change distributions just to setup bproc, I found the installation of bproc quite simple. We have a small educational cluster at school and we've been using bproc for several months now. It lends it self really well for a diskless setup and as a bonus LAM/MPI can integrate with bproc so that MPI applications could also run on top of it. I think that you would be pleased with Bproc. BTW, it took us less than a day to convert our "traditional" cluster into a Bproc one. Best of luck, -- ''''' ''''' ''''' ( . . ) ( . . ) ( . . ) ---oo0-(_)-0oo---oo0-(_)-0oo---oo0-(_)-0oo--- Kaveh Moallemi School of Computer Science Ryerson University Toronto, Canada >From: Adam C Powell IV >To: Camm Maguire >CC: Andrew Piskorski , beowulf@beowulf.org,Debian >Beowulf >Subject: Re: [Beowulf] how useful is bproc, and what does Scyld cost? >Date: Thu, 27 May 2004 16:50:20 -0400 > >There's also pretty complete community-maintained documentation at >http://wiki.debian.net/ with the relevant material at >http://wiki.debian.net/index.cgi?DebianBeowulf > >On Thu, 2004-05-27 at 15:41, Camm Maguire wrote: > > Greetings! No direct experience with diskless setups, but all the > > cluster applications I need work out of the box on Debian. There are > > even precompiled atlas libraries for several common cpu > > subarchitectures. We've been running ours for about 7 years -- 1 > > software install, 3 hardware upgrades, and nothing but 'apt-get > > upgrade' since. > > > > Take care, > > > > Andrew Piskorski writes: > > > > > I'm tentatively planning a small cluster that might or might not > > > actually get built. My current plan is somewhere from 5-20 nodes, 1-2 > > > x86 CPUs per node (exact CPU flavor undecided), gigabit ethernet, and > > > all nodes either entirely diskless, or using 1 IDE disk solely for > > > swap and /tmp. > > > > > > I would prefer to have as much as posible of the cluster software > > > infrastructure Just Work, rather than having to spend lots of time > > > rolling my own. (I will be spending enough time on the custom > > > software I actually want to RUN on the cluster as is.) I am, of > > > course, quite willing to select hardware in order to make the software > > > job easier on myself. > > > > > > Since I want to go diskless anyway, so far I am also leaning towards a > > > bproc based cluster. I only know of two bproc-based cluster > > > distributions, Scyld and Clustermatic. Scyld is commerical and costs > > > money, Clustermatic is not and does not. Are there any others? In > > > particular, are there any Debian based systems that play nicely out of > > > the box with bproc? > > > > > > How much time and effort is Scyld actually going to save me over using > > > Clustermatic? How much is either going to save me over completely > > > rolling my own, preferably using Debian rather than the old and > > > outdated versions of Red Hat that Scyld and Clustermatic seem to use? > > > Also, are there any major drawbacks or snafus I should worry about in > > > going down the bproc route? > > > > > > Finally, just what DOES Scyld actually cost? Can anyone give me a > > > rough idea? > > > > > > >From Scyld's website, I can't tell whether they charge 50 cents or > > > $5,000 per node, and the Scyld/Penguin salesman seemed unable to spit > > > out any kind of ballpark price at all. AFAICT, Scyld seems to expect > > > you to first actually build your cluster, and then send them your > > > cluster's complete hardware specs, down to the smallest detail, in > > > order to get any kind of quote! > > > > > > Thanks! > >-Adam P. > >GPG fingerprint: D54D 1AEE B11C CE9B A02B C5DD 526F 01E8 564E E4B6 > >Welcome to the best software in the world today cafe! >http://lyre.mit.edu/~powell/The_Best_Stuff_In_The_World_Today_Cafe.ogg > > >-- >To UNSUBSCRIBE, email to debian-beowulf-REQUEST@lists.debian.org >with a subject of "unsubscribe". Trouble? Contact >listmaster@lists.debian.org > From sila68_2000 at yahoo.com Fri May 28 01:19:18 2004 From: sila68_2000 at yahoo.com (susila bahri) Date: Wed Nov 25 01:03:12 2009 Subject: [Beowulf] ask about beowulf hardware and software Message-ID: <20040528081918.81099.qmail@web20021.mail.yahoo.com> Sir, I am new in using Beowulf machine. Actually, I am studying to make a parallel program in Beowulf cluster machine.But, I want to know hardware configuration and software of the machine(I mean the harware configuration and software of the machine that I am using). What command must I type when I use the machine ? I want to know configuration of the processors, RAM, its Bus etc.Whereas for software,I want to know about description of its parallel libraries, software and programming language.Please help me... Thank's a lot for your kind. Best regards susila Mathematics student in Malaysia ===== SAMPAI JUMPA LAGI __________________________________ Do you Yahoo!? Friends. Fun. Try the all-new Yahoo! Messenger. http://messenger.yahoo.com/ From deadline at linux-mag.com Fri May 28 04:37:55 2004 From: deadline at linux-mag.com (Douglas Eadline, Cluster World Magazine) Date: Wed Nov 25 01:03:12 2009 Subject: [Beowulf] Cluster Urban Legands Message-ID: I have posted an article on ClusterWorld.com called "Cluster Urban Legends". I would be interested in comments or feedback. (You can post them on the site - works like Slashdot) Doug -- ---------------------------------------------------------------- Editor-in-chief ClusterWorld Magazine Desk: 610.865.6061 Fax: 610.865.6618 www.clusterworld.com From daniel.kidger at quadrics.com Fri May 28 05:36:37 2004 From: daniel.kidger at quadrics.com (Dan Kidger) Date: Wed Nov 25 01:03:12 2009 Subject: [Beowulf] Correct Units for quoting Interconnect Bandwidth ? In-Reply-To: References: Message-ID: <200405281336.37554.daniel.kidger@quadrics.com> I have spotted an anomaly when comparing b_eff results with PMB: http://www.hlrs.de/organization/par/services/models/mpi/b_eff/ http://www.pallas.com/e/products/pmb/ The former uses 1.e6 when displaying 'MBytes/s', where that latter appears to use 2^20. (aka 'Mebibytes/s') viz from PMB's Output.c: static double MEGA = 1.0/1048576.0; if( tmax > 0. ) throughput = (Bmark->scale_bw*SCALE*MEGA)*size/tmax; SCALE=1000000 (declare.h) size is in bytes, tmax in usec, and scale_bw=2 if say doing PingPong The result is that PMB figures appear *lower* than equivalent b_eff figures. Do we have a concensus of what units we should be using? So does a 1GB/s interconnect transfer 1 Gigabyte in a second or 1,000,000,000 bytes in a second? (As interconnects move from Kilo though Mega to Giga the gap is of course widening) -- Yours, Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- From orion at cora.nwra.com Fri May 28 11:07:57 2004 From: orion at cora.nwra.com (Orion Poplawski) Date: Wed Nov 25 01:03:12 2009 Subject: [Beowulf] Re: SGE + policy In-Reply-To: References: Message-ID: <40B77FFD.6030006@cora.nwra.com> Robert G. Brown wrote: > > This probably doesn't worry them, because in one sense anybody who uses > matlab for doing "serious" computations probably needs therapy, and > clusters are often build for doing serious computations. However, this > particular cluster is being built as much to support many students doing > less serious computations as it is to support work that really should > probably be done in c.* or f.*, and in a lot of cases the real expense > is teaching people with little or no real programming experience how to > convert from a garbage-tolerant interactive environment to a compiled > language. > You can buy a matlab compiler from mathworks. Perhaps this would work better. There is also some kind of run-time server as well that turns the code into standalone programs. I've never used either, but we may soon. Experiences with either would be appreciated. -- Orion Poplawski System Administrator 303-415-9701 x222 Colorado Research Associates/NWRA FAX: 303-415-9702 3380 Mitchell Lane, Boulder CO 80301 http://www.co-ra.com From sdutta at cfa.harvard.edu Fri May 28 12:56:02 2004 From: sdutta at cfa.harvard.edu (Suvendra Nath Dutta) Date: Wed Nov 25 01:03:12 2009 Subject: [Beowulf] KVM to a compute node Message-ID: <0C646DBD-B0E1-11D8-9C0A-000A95AE26CC@cfa.harvard.edu> What kind of solutions do people go to boot up an individual compute node. We will have nodes with video cards and disks, so they can boot up separately. But I am hoping there is a solution that is better than stringing long KVM wires across the server room. Suvendra. From cousins at limpet.umeoce.maine.edu Fri May 28 13:56:25 2004 From: cousins at limpet.umeoce.maine.edu (Steve Cousins) Date: Wed Nov 25 01:03:12 2009 Subject: Subject: [Beowulf] Re: SGE + policy In-Reply-To: <200405281900.i4SJ05kf015347@bluewest.scyld.com> Message-ID: On Fri, 28 May 2004 beowulf-request@beowulf.org wrote: > On Thu, 27 May 2004, Andrew Piskorski wrote: > > > On Thu, May 27, 2004 at 12:01:09PM -0700, beowulf-request@beowulf.org wrote: > > > > > Date: Thu, 27 May 2004 10:19:00 -0400 (EDT) > > > From: "Robert G. Brown" > > > > > Primary tasks: matlab and stata jobs, run either interactively/remote > > > or (more likely) in batch mode. Jobs include both "short" jobs that > > > > > Constraint: matlab requires a license managed by a license manager. I don't know if the proposed cluster is using dual CPU's but I believe you can run multiple Matlab sessions on one node and use only one license. So if you had two CPU's you could run two sessions (and more probably if you had a bunch of interactive sessions) and use only one license. This would at least drop the number of licenses needed in half. So, depending on what the Matlab licenses cost these days maybe it would be cheaper to get a bunch of 8 CPU Opterons :^o Steve ______________________________________________________________________ Steve Cousins, Ocean Modeling Group Email: cousins@umit.maine.edu Marine Sciences, 208 Libby Hall http://rocky.umeoce.maine.edu Univ. of Maine, Orono, ME 04469 Phone: (207) 581-4302 From liubo1975 at yahoo.co.nz Sun May 30 19:21:29 2004 From: liubo1975 at yahoo.co.nz (Emma) Date: Wed Nov 25 01:03:12 2009 Subject: [Beowulf] harmonic mitigators vs PFC? In-Reply-To: <000a01c44199$5bf39bb0$32a8a8c0@LAPTOP152422> Message-ID: <20040531022129.22563.qmail@web50108.mail.yahoo.com> --- Jim Lux wrote: > > ----- Original Message ----- > From: "Mark Hahn" > To: > Sent: Sunday, May 23, 2004 10:40 AM > Subject: [Beowulf] harmonic mitigators vs PFC? > > > Somewhere, about 3 or 4 months ago, someone posted a > link to a very nice > explanation of managing harmonics in this sort of > situation. > this one? ;-) http://www.mirusinternational.com/pages/faq.html -Bo __________________________________ Do you Yahoo!? Friends. Fun. Try the all-new Yahoo! Messenger. http://messenger.yahoo.com/ From daniel at labtie.mmt.upc.es Mon May 31 12:25:50 2004 From: daniel at labtie.mmt.upc.es (Daniel Fernandez) Date: Wed Nov 25 01:03:12 2009 Subject: [Beowulf] Stable power supplies Message-ID: <1086031550.4975.17.camel@qeldroma.cttc.org> Hi all, We've recently solved our really annoying problem of random hangups, there were two memory modules that started failing after a *long* run period. Moreover there were some badly base polished heatsinks... anyway, all seems running fine at the moment. There is another remaining factor as someone suggested, the power supply, not only the amount of watts is important but also how stable is it running at full load. The voltage lines ( as reported by "sensors" program ) seemed to diverge a bit more than the expected on few nodes ( on the limits as marked by "sensors" ). Power supplies *as is* don't give an accurate information, is there any good source of information about power supplies and testing ? Any reccomendations about manufacturers based on experience would alse be useful. Last point... wondering if results reported by sensors library ( and sensor hardware too ) are as accurate as they seem . Regards. -- Daniel Fernandez Heat And Mass Transfer Center - CTTC www.cttc.upc.edu c/ Colom n?11 UPC Campus Industrials Terrassa , Edifici TR4