Main Page

From UMass GHPCC User Wiki
Jump to: navigation, search

Welcome to the University of Massachusetts Green High Performance Computing Cluster

For questions about, or problems with, GHPCC please email hpcc-support@umassmed.edu

Upcoming Maintenance

  • Regular maintenance is scheduled for the first Wednesday of every month from 9AM through 5PM as needed.
  • 2016: Monday, May 16th - The GHPCC cluster will be taken offline for the MGHPCC data center power maintenance starting at 5pm Mayh 16th. We hope to have the cluster back online and operational on or before 5:00pm on Wednesday, May 18th.
    • During this event the following will be offline and unavailable.
      • All head nodes will be offline
      • All execution nodes will be offline
      • All cluster storage located in the MGHPCC data center will be offline
    • PLEASE NOTE: The cluster scheduler will not dispatch jobs that are predicted to run into the downtime window. Jobs that would still be running at 5pm May 16th if started will instead remain pending and will not be dispatched until after the downtime window is over.

Recent Announcements

We are happy to announce that we have purchased and installed additional cores for our GHPC cluster. We now have in excess of 10,000 cores in the cluster. The breakdown of the cores added include:

  • 3072 new AMD cores – 48 new Nodes: Each AMD node contains 64 cores, and 512GB RAM
  • 960 new Intel cores – 48 new Nodes: Each Intel node contains 20 cores, and 196GB RAM
  • 6 new NVIDIA Tesla K80 GPU nodes, each pair contain Intel CPUs (20 cores) and 256GB RAM

The new AMD nodes were added into the cluster two weeks ago and have been heavily used. There are no changes required during job submissions to access these new nodes.

The new Intel blades will be activated and added to the queues on Friday Feb 27th due to the minor OS changes required:

Please note that the new Intel nodes require more recent version of Redhat than we use for our standard image (the standard image is Redhat 6.4, the new Intel nodes will be running Redhat 6.5). While most jobs will not have any issues, and will not require a resource request, It is possible that programs that run successfully on the standard 6.4 image may have issues running on the 6.5 image, so we have configured a new attribute in LSF (“os”) to allow users to specify if one of the images is required, or forbidden. At the moment we have three values for this attribute: ‘rh6.4’, ‘rh6.5’, and ‘suse11sp2’ which only applies to the SGI UV2 system.

How to specify these new nodes:

  • To specify that the job should only run on the 6.4 image add “-R select[os=rh6.4]” to your bsub parameters.
  • To specify that the job should only run on the 6.5 image add “-R select[os=rh6.5]” to your bsub parameters.


Cluster usage and status information

Usage over the past two months:

Cluster usage over the past two months

Usage over the past day:

As of Sun Dec 4 21:15:02 EST 2016

Current number of available cores: 12776

Current number of cores in use: 10093

Note: This does not mean that submitting a job will result in you being at the end of the line. GHPCC uses Fairshare to provide equal distribution of CPU resources to all users over time. If you have not run a job recently, your chances are good that your job will dispatch quickly. Fast submission also relies on lower resource (memory and CPU) requirements. Jobs with smaller requirements are more likely to be dispatched first.

Known issues

  • The ghpcc-sgi node is running a different distribution (OpenSuSE) of Linux than the blade compute nodes (Red Hat). While GHPCC staff are working to convert this system to make it the same as others, you can direct your jobs to blade systems by including -m blades in your bsub command line:
bsub -q short -m blades -W 2:0 -R "rusage[mem=2048]" myapp

Note that this node is not part of the long or parallel queues, only short and interactive.

  • Your password is locally stored. We are working to change this to use your local campus account password. Most user names will not change, nor will your data be deleted or modified when we perform the cutover. We will have additional announcements as this time nears with more information.

Statements for grant applications

GHPCC

Current Resources (NIH formatted)

Analysis of high throughput sequencing data is performed on a shared high performance computer cluster with 14,376 cores available (Intel and AMD), and 660TBs of high performance EMC Isilon X series storage and 513TBs of Farline storage. The Massachusetts Green High Performance Computing Cluster is located in Holyoke MA and provides computing to the five University of Massachusetts Campuses. The UMass Shared Services Cluster includes equipment (cores and storage) contributed by each UMass campus as well as jointly funded components (LAN, Scheduler Software, IDM services and support staff). The High Performance Computing Cluster (HPCC) consists of the following hardware: an FDR based Infiniband (IB) network and a 10GE network for the storage environment, qty 9 (42) GPU nodes (Intel with 256GB RAM) with two NVIDIA Tesla C2075 - GPU computing processor - Tesla C2075 - 6 GB GDDR5 - PCI Express 2.0 x16 units or K80 GPUs, qty thirteen (13) AMD Opteron(tm) Processor 6380 based Dell chassis with 64 cores / 512GB RAM per blade (48 blades), qty seven (7) AMD (2x AMD Opteron 6278, 2.4GHz, 16C, Turbo CORE, 16M L2/16M L3, 1600Mhz ) based Dell chassis with 64 cores / 512GB RAM per blade (42 blades), qty (3) Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz, QPI, Turbo, 20c, qty nine (9) Intel (Xeon E5-2650 2.00GHz, 20M Cache, 8.0GT/s QPI, Turbo, 8C, 95W, Max Mem 1600MHz) based chassis with 16 cores / 196GB RAM per blade (16 blades), qty two (2) SGI UV200 with 512 Intel (Intel® Xeon® processor E5-4600) cores and 4TBs of fully addressable memory, qty one (1) AMD based Dell chassis with 128 cores Quad-Core AMD Opteron(tm) Processor 2376 and 256GB RAM, qty three (3) Intel (six-core Intel(R) Xeon(R) CPU X5650 @ 2.67GHz ) based Dell chassis with 12 cores / 48GB RAM per blade (16 blades). The HPC environment runs the IBM LSF scheduling software for job management. The Massachusetts Green High Performance Computer Center (MGHPCC) facility has space, power, and cooling capacity to support 680 racks of computing and storage equipment, drawing up to 15MW of power. High speed network connections to the facility are available via dark fiber, the Northern Crossroads, and Internet 2. The MGHPCC facility has been awarded LEED Platinum status.

Generic Resources Review

Analysis of high throughput sequencing data is performed on a shared high performance computer cluster with 14,376 cores available (Intel and AMD), and 660TBs of high performance EMC Isilon X series storage and 513TBs of Farline storage. The Massachusetts Green High Performance Computing Cluster is located in Holyoke MA and provides computing to the five University of Massachusetts Campuses. The UMass Shared Services Cluster includes equipment (cores and storage) contributed by each UMass campus as well as jointly funded components (LAN, Scheduler Software, IDM services and support staff). The HPC environment runs the IBM LSF scheduling software for job management. The Massachusetts Green High Performance Computer Center (MGHPCC) facility has space, power, and cooling capacity to support 680 racks (68 racks for UMASS) of computing and storage equipment, drawing up to 15MW of power. High speed network connections to the facility are available via private dark fiber, the Northern Crossroads, and Internet 2. The MGHPCC facility has been awarded LEED Platinum status.

Slides from the 6/24/2015 Introduction to RNA-Seq

Media:SN_Intro_to_RNA-seq_with_HPC_(60_mins).pptx

Slides from the 6/16/2015 Introduction to High Performance Computing with LSF

Media:SN Intro to High Performance Computing using LSF (120 mins).pptx

Slides from the 6/11/2015 Introduction to Linux shell scripting

Media:SN_Intro_to_Linux_shell_scripting_(120_mins).pptx

Slides from the 6/3/2015 Introduction to R Programming

Media:SN_Intro_to_R_(120_mins).pptx

Slides from the 5/27/2015 Crash course in Python Programming

Media:SN_Intro_to_Python_(120_mins).pptx

Slides from the 5/22/2015 Crash course in Perl Programming

Media:SN_Into_to_Perl_(120_mins).pptx

Slides from the 3/19/2015 Intro to Linux and HPC

Media:SN_Intro_to_Linux_and_HPC_(60_mins).pptx


Wiki Help