Cluster - TORQUE Resource Manager Tutorial

The TORQUE resource manager is a complex piece of software, which deserves some quick tips on how to use the system without having downtime or lost jobs for your users. This page gathers some of the most important information you can know about using Torque and will be further expanded to accommodate all the important topics regarding this management component of computer clusters.

TORQUE is built on the principles of the Portable Batch System, also known as PBS, from which there are two versions OpenPBS(opensource but no longer maintained) and PBS Pro, a paid for PBS software. TORQUE is also open source and derives from OpenPBS, with currently many years of development separating TORQUE from OpenPBS. TORQUE is currently one of the better ways to manager cluster job queues, especially given that Sun Grid Engine is no longer free.

TORQUE PBS Server

The pbs_server process is what controls what nodes are assigned idle jobs from the queue on a machine(node) finishes a calculation and becomes empty. Sometimes it can have some problems, so it is useful to restart the pbs_server process, which usually is installed in /etc/init.d/pbs_server.

Restart PBS Server by issuing, as root(or with sudo):

/etc/init.d/pbs_server restart

The process can be restarted mostly without issue and there are two options for restarting the process, defined in TORQUE's config file:

• delay
• quick

If PBS_SERVER_STOP is set to "quick". In this situation, the running jobs will be let run without interaction until the server is back up. Setting it to "delay", the jobs will be checkpointed, rerun, or pbs_server will wait for the jobs to finish before restarting the service.

If you're trying to sort out blocked resources, it is recommended to use "quick".

OpenWRT, NAT And VPN Connection Issues

If you're having problems with getting VPN connections stuck in the "verifying username and password" step of the connection, you need to load two kernel modules that help the NAT track VPN connections:

• nf_nat_pptp
• nf_conntrack_pptp

These are required for VPN connections to be properly tracked by IPTables. In the OpenWRT router embedded distribution, these two modules are available in the "kmod-ipt-nathelper-extra" package.

On other Linux distributions like Ubuntu, OpenSUSE or Gentoo, these are part of the kernel, so they should already be present.

Drop me a comment below if you're having problems setting it up.

OpenPBS HPC Cluster Software

OpenPBS is an cluster job management tool that is compatible the old PBS resource manager. Torque is commonly used together with Maui as the cluster scheduler, which is what we'll have a look at in this post.
The command you need to manipulate queues is qmgr. To create a new queue we use the following code:

Qmgr: queue create fast

I named this queue fast but you can name it whatever you like. We then set the queue to allow execution of jobs and enable it. We must also start the queue, otherwise the job will be blocked.

Qmgr: set queue fast queue_type = Execution
Qmgr: set queue fast enabled = True
Qmgr: set queue fast started = True

Now we want to add nodes to this new queue:

Qmgr: set queue fast resources_default.neednodes = fast
Qmgr: create node new_node np=8
Qmgr: set node new_node properties = fast

We also define the fast property to both the node and queue to ensure that this node doesn't run on any other queue. Other existing queues need to have resources_default.neednodes set to a different property. We also define the node new_node as having eight processors (np=8).
You can now submit jobs by passing the -q fast parameter to Maui's qsub.

#!/bin/bash
qsub -q fast

Resources

OpenPBS Resource Manager (link)
Maui Sheduler (link)