Raw Notes on Parallel Programming
Parallel programming involves multiple tasks running simultaneously to process data or execute instructions. I’ve been exploring parallel computing for a recent project, and I’ve come across various definitions and distinctions in the literature. These terms often have loose interpretations, so it’s worth noting that each book or source defines them differently.
Definitions
Parallel Computing
This refers to executing different parts of a program simultaneously on multiple processors within the same computer. The aim is to speed up computations by dividing tasks among several CPU cores.
Distributed Computing
A subset of parallel computing but involving multiple computers. In distributed computing, systems typically don’t work on tightly coupled tasks. Unlike clusters, distributed computing often spans machines located in different geographical locations, and these machines can perform independent tasks.
Grid Computing
Grid computing makes use of many separate computers, often loosely connected over a network (usually the Internet). It focuses on tasks that involve independent jobs, with minimal need for inter-node communication. Grid systems often leverage unused computational resources, also known as CPU scavenging, to form a distributed system.
Computer Cluster
A group of interconnected computers that function together as a unified system. Clustered computers typically share fast local area networks and operate more cohesively compared to the loosely connected nodes in distributed systems.
Parallel Virtual Machine (PVM)
PVM is a software tool for creating distributed parallel applications by linking computers across different platforms, whether Unix or Windows. It allows you to treat a collection of machines as a single parallel processor.
Message Passing Interface (MPI)
MPI defines an API that allows multiple computers to communicate efficiently. It’s a cornerstone for high-performance computing (HPC), offering scalability and portability for parallel applications. MPI remains dominant in the HPC world today.
Documentation: Chapter 1. Introduction to Parallel Programming
IBM Redbooks PDF
Amdahl’s Law
Amdahl’s Law describes the limit of performance improvement when a program is parallelized. It states that the speedup of a parallel program is constrained by the sequential portion of the execution. For example, if a program takes 20 hours on a single processor, and 19 hours of that time can be parallelized, the remaining 1 hour will limit the maximum speedup, no matter how many processors are used. In this scenario, the speedup can only go up to 20x.
Beowulf (Computer Cluster)
Beowulf is a type of computer cluster designed for parallel computing. It typically consists of a server node and several client nodes connected over a network, such as Ethernet. What sets Beowulf apart is that it uses commodity hardware and software—standard PCs running Unix-like systems (Linux or Solaris) and open-source parallel tools like PVM and MPI.
The server node controls the cluster and serves files to the client nodes, which generally operate without a direct user interface, often relying on remote logins for interaction. In Beowulf clusters, client nodes don’t necessarily need monitors or keyboards; they are configured and managed entirely by the server.
Key Differences
Unlike a Cluster of Workstations (COW), which is more of a set of independent workstations, Beowulf behaves like a single, cohesive system. The nodes in a Beowulf cluster are treated like simple units (CPU + memory) that can be added to the system with minimal configuration. The focus is on parallelism rather than simply distributing tasks.
Parallel Programming Books
Here are some books I’ve found particularly useful for exploring parallel programming:
- Introduction to Parallel Computing
- Year: 2003
- Read on SafariBooks
This is an excellent starting point for understanding the fundamentals of parallel computing.
- Principles of Concurrent and Distributed Programming
- Year: 2006
- Read on SafariBooks
A solid resource that dives into the theory and application of concurrent programming.
- Server Architectures: Multiprocessors, Clusters, Parallel Systems, Web Servers, and Storage Solutions
- Price: 27 pounds
- Year: 2004
- Read on SafariBooks
Particularly helpful is Part III on optimizing parallel applications, especially Chapter 14, which focuses on optimizing message-passing program
Keep Sharing.