NE255 Lecture 1: Intro to Computational ScienceEngineering

1
NE-255 Lecture 1 Intro to Computational
Science/Engineering

• Fall 08
• Jasmina Vujic
• Department of Nuclear Engineering
• U.C. Berkeley

2
SUMMARY

• Overview of Computational Science
• A Brief History of Computer Technology
• The Modern High Performance Computing Environment
• Basic Computer Architecture
• High Performance Computer Architecture

3
References

• Computational Science Education Project,
  electronic book, U.S. DOE, 1996
  (http//www.phy.ornl.gov/csep/ )
• Greatest Engineering Achievements of the 20th
  Century, National Academy of Engineering
  (http//www.greatachievements.org/ ), 2000
• CS 258, David E. Cooler, Computational Science
  Division, U.C. Berkeley, 1999
• CS 267, Kathy Yelick, Application of Parallel
  Computers, 2004
• CS 152, John Lazzaro and Dave Peterson, 2004

4
Relevant Courses

• Math 128A, Numerical Analysis (solution of
  ordinary differential equations)
• Math 128B, Numerical Analysis (evaluations of
  eigenvalues and eigenvectors, solution of simple
  partial differential equations)
• Math 222A-222B, Partial Differential Equations
  (The theory of boundary value and initial value
  problems, Laplaces equation, heat equation, wave
  equation, Fourier transformations, elliptic
  parabolic and hyperbolic equations)
• Math 228A-228B, Numerical Solutions of Partial
  Differential Equations (Runge-Kutta and
  predictor/corrector methods, boundary value
  problems, finite differences and finite element
  solutions of elliptic equations)

5
Relevant Courses

• E 7, Introduction for Computer Programming for
  Scientists and Engineers (4 units)
• CS 9A, Matlab for Programmers (1 unit),
  self-paced, pass -not pass.
• CS 9C, C for Programmers (1 unit)
• CS 9F, C for Programmers (1 unit)
• CS 9G, JAVA for Programmers (1 unit)

6
Overview of Computational Science

• Three approaches to science and engineering
• EXPERIMENT
• Experimental scientists work by observing how
  nature behaves
• THEORY
• Theoretical scientists use the language of
  mathematics to explain and predict the behavior
  of nature
• COMPUTATION
• Computational scientists use theoretical and
  experimental knowledge to create computer based
  models of aspects of nature

7
Computational Science/Engineering

• Computational Science seeks to gain understanding
  principally through the analysis of mathematical
  models on high performance computers.
• The term computational scientists has been coined
  to describe scientists, engineers and
  mathematicians who apply high performance
  computer technology in innovative and essential
  ways to advance the state of knowledge in their
  respective discipline.
• Thus, we distinguish it from computer science,
  which is the study of computer and computation,
  and theory and experiment, the traditional form
  of science.

8
Solving Computational Problems

• IDENTIFY THE PROBLEM
• POSE THE PROBLEM IN TERMS OF A MATHEMATICAL MODEL
• IDENTIFY A COMPUTATIONAL METHOD FOR SOLVING THE
  MODEL
• IMPLEMENT THE COMPUTATIONAL METHOD ON A COMPUTER
• ASSESS THE ANSWER IN THE CONTEXT OF THE
• Implementation (computer language and
  architecture)
• Method (discrete or continuous)
• Model (symbolic or numerical)
• Visualization and Interpretation
• Experimentation

9
Some Particularly Challenging Computations

• Science
• Global climate modeling
• Astrophysical modeling
• Biology genomics protein folding drug design
• Computational Chemistry
• Computational Material Sciences and Nanosciences
• Engineering
• Crash simulation
• Semiconductor design
• Earthquake and structural modeling
• Computation fluid dynamics (airplane design)
• Combustion (engine design)
• Analysis and design of nuclear reactors
• Business
• Financial and economic modeling
• Transaction processing, web services and search
  engines
• Defense
• Nuclear weapons -- test by simulations
• Cryptography

10
Global Climate Modeling Problem

• Problem is to compute
• f(latitude, longitude, elevation, time) ?
• temperature, pressure,
  humidity, wind velocity
• Approach
• Discretize the domain, e.g., a measurement point
  every 10 km
• Devise an algorithm to predict weather at time
  t1 given t

• Uses
• Predict major events, e.g., El Nino
• Use in setting air emissions standards

Source http//www.epm.ornl.gov/chammp/chammp.html
11
Global Climate Modeling Computation

• One piece is modeling the fluid flow in the
  atmosphere
• Solve Navier-Stokes problem
• Roughly 100 Flops per grid point with 1 minute
  timestep
• Computational requirements
• To match real-time, need 5x 1011 flops in 60
  seconds 8 Gflop/s
• Weather prediction (7 days in 24 hours) ? 56
  Gflop/s
• Climate prediction (50 years in 30 days) ? 4.8
  Tflop/s
• To use in policy negotiations (50 years in 12
  hours) ? 288 Tflop/s
• To double the grid resolution, computation is at
  least 8x
• State of the art models require integration of
  atmosphere, ocean, sea-ice, land models, plus
  possibly carbon cycle, geochemistry and more
• Current models are coarser than this

12
High Resolution Climate Modeling on NERSC-3 P.
Duffy, et al., LLNL
13
Tunnel Vision by Experts

• I think there is a world market for maybe five
  computers.
• Thomas Watson, chairman of IBM, 1943.
• There is no reason for any individual to have a
  computer in their home
• Ken Olson, president and founder of Digital
  Equipment Corporation, 1977.
• 640K of memory ought to be enough for
  anybody.
• Bill Gates, chairman of Microsoft,1981.

Slide source Warfield et al.
14
Solving Computational Problems

• Periods of rapid advancements in the sciences
  have often been sparked by timely technology
  breakthroughs in experimental technique.
• The next epochal period of scientific growth
  maybe unleashed by major design breakthroughs in
  computer architectures and advances in modeling
  approaches, where supercomputers become the
  laboratories to test and advance new theories for
  which no practical experimental apparatus can be
  built. 1990

15
A Brief History of Computer Technology

• THE MECHANICAL ERA (1623 - 1945)
• FIRST GENERATION ELECTRONIC COMPUTERS (1937-1953)
• SECOND GENERATION (1954 - 1962)
• THIRD GENERATION (1963 - 1972)
• FOURTH GENERATION (1972 - 1984)
• FIFTH GENERATION (1984 - 1990)
• SIXTH GENERATION (1990 - )

16
A Brief History of Computer Technology

• THE MECHANICAL ERA (1623 - 1945)
• Difference Engine, 1823, Charles Babbage -
  never completed
• Analytical Engine, 1842, Charles Babbage -
  never completed
• George and Edvard Scheutz, 1853 -won a gold
  medal
• Punch Card Equipment, 1890, Herman Hollerith, the
  1890 census
• Holleriths company become IBM in 1924.

17
A Brief History of Computer Technology

• FIRST GENERATION ELECTRONIC COMPUTERS (1937-1953)
• Sir John Ambrose Fleming invents the vacuum tube
  and diode
• John Atanasoff and Cliford Berry invent first
  electronic computer at Iova State University in
  1939 to solve systems of partial differential
  equations. In 1941, solved 29 equations with 29
  unknowns. The machine was not programmable.
• COLOSSUS, 1943, British military, used to break
  Nazi codes
• ENIAC, 1945, first electronic digital computer,
  used for the design of the hydrogen bomb, U of
  Pennsylvania
• Bardeen, Brattan and Shockley invent the
  transistor at Bell Labs, 1947
• EDVAC, 1950, could store the instructions data
• UNIVAC, 1951 - first commercialization

18
A Brief History of Computer Technology

• SECOND GENERATION (1954 - 1962)
• Gene Amdahl develops the first computer operating
  system for the IBM 704, (1954)
• TRADIC, 1954, Bell Labs and TX-0, 1954, MIT were
  the first machines that used discrete diode and
  transistor technology as electronic switches
  (switching time 0.3 microseconds)
• Reynolds Johnson develops the first disk drive,
  (1955)
• FORTRAN (1957) becomes commercially available
• Jack Kilby of Texas Instruments invents the
  integrated circuit (IC), (1958)
• Seymour Cray of Control Data Corp. develops the
  first transistorized computer, (1958)
• ALGOL (1958), COBOL (1959)
• Silicon chips first appear (1961), first
  minicomputer comes into use (1962)

19
A Brief History of Computer Technology

• THIRD GENERATION (1963 - 1972)
• Integrated circuits, semiconductor memories,
  operating systems, time sharing
• Douglas Englebart, SRI, patents the idea of the
  computer mouse (1963)
• IBM releases its Model 360 computer, which will
  result in 100 billion in sales over its life
  cycle (1964)
• CDC 6600, Seymour Cray - functional parallelism -
  1 MFLOP (1964)
• John Kemeny and Thomas Kurtz develop the BASIC
  computer language. Intel Chairman Gordon Moore
  suggests that IC would double in complexity every
  18 months - Moores Law (1964)
• CDC 7600, Seymour Cray - the first VECTOR
  processor - 10 MFLOPS. PASCAL computer language
  invented. (1969)
• Intel introduces popular 4004 4-bit
  microprocessor, starting the evolution of Intels
  famous line 386, 486, and Pentium processors
  (1971)
• SOLOMON and ILLIAC IV - the first parallel
  computers

20
A Brief History of Computer Technology

• FOURTH GENERATION (1972 - 1984)
• Large-scale and very large scale integration
  (LSI and VLSI - 100,000 devices per chip).
• CRAY 1, CRAY X-MP, CYBER 206, CRAY2
• C programming language, Bell Labs
• UNIX operating system, Bell Labs, UCB
• NSF Supercomputing Centers Sand Diego,
  Urbana, Pittsburgh. Cornell, and Princeton

21
A Brief History of Computer Technology

• FIFTH GENERATION (1984 - 1990)
• Large-scale parallel processing,
• Single-user workstations
• IBM 3090/6 shared memory
• Sequence Balance 8000 - up to 20 processors
  to a single shared-memory module
• iPSC-1 - the hypercube - 128 distributed memory
  processors
• Computer networking, low price of workstations
  and PCs

22
A Brief History of Computer Technology

• SIXTH GENERATION (1990 - )
• Parallel/vector shared/distributed memory
  combinations
• High speed networking
• Virtual supercomputers - networks of
  workstations and supercomputers
• High-performance computing projects

23
A Brief History of Computer Technology
24
Where is Computer Architecture and Engineering?
Application (Netscape)
Operating
Compiler
System (Windows 2K)
Software
Assembler
Instruction Set Architecture
Hardware
I/O system
Processor
Memory
Datapath Control
Digital Design
Circuit Design
transistors

• Coordination of many levels of abstraction

25
Anatomy 5 components of any Computer
Personal Computer
Keyboard, Mouse
Computer
Processor
Memory (where programs, data live
when running)
Devices
Disk (where programs, data live when not
running)
Input
Control (brain)
Datapath (brawn)
Output
Display, Printer
26
Computer Technology - Dramatic Change!

• Processor
• 2X in speed every 1.5 years (since 85) 100X
  performance in last decade.
• Memory
• DRAM capacity 2x / 2 years (since 96) 64x
  size improvement in last decade.
• Disk
• Capacity 2X / 1 year (since 97)
• 250X size in last decade.

27
Technology Trends Microprocessor Capacity
Moores Law
2X transistors/Chip Every 1.5 years Called
Moores Law
Gordon Moore (co-founder of Intel) predicted in
1965 that the transistor density of semiconductor
chips would double roughly every 18 months.
Microprocessors have become smaller, denser, and
more powerful.
Slide source Jack Dongarra
28
Tech. Trends Microprocessor Complexity
2X transistors/Chip Every 1.5 to 2.0 years Called
Moores Law
29
Why do we need better computers?
30
What is Parallel Architecture?

• A parallel computer is a collection of processing
  elements that cooperate to solve large problems
  fast
• Some broad issues
• Resource Allocation
• how large a collection?
• how powerful are the elements?
• how much memory?
• Data access, Communication and Synchronization
• how do the elements cooperate and communicate?
• how are data transmitted between processors?
• what are the abstractions and primitives for
  cooperation?
• Performance and Scalability
• how does it all translate into performance?
• how does it scale?

31
Automatic Parallelism in Modern Machines

• Bit level parallelism within floating point
  operations, etc.
• Instruction level parallelism (ILP) multiple
  instructions execute per clock cycle.
• Memory system parallelism overlap of memory
  operations with computation.
• OS parallelism multiple jobs run in parallel on
  commodity SMPs.
• There are limitations to all of these!
• Thus to achieve high performance, the programmer
  needs to identify, schedule and coordinate
  parallel tasks and data.

32
Application Trends

• Application demand for performance fuels advances
  in hardware, which enables new applns, which...
• Cycle drives exponential increase in
  microprocessor performance
• Drives parallel architecture harder
• most demanding applications
• Range of performance demands
• Need range of system performance with
  progressively increasing cost

33
MeasuringPerformance
34
Speedup

• Speedup (p processors)
• For a fixed problem size (input data set),
  performance 1/time
• Speedup fixed problem (p processors)

35
Improving Real Performance

• Peak Performance is skyrocketing
• In 1990s, peak performance increased 100x in
  2000s, it will increase 1000x
• But efficiency (the performance relative to the
  hardware peak) has declined
• was 40-50 on the vector supercomputers of 1990s
• now as little as 5-10 on parallel supercomputers
  of today
• Close the gap through ...
• Mathematical methods and algorithms that achieve
  high performance on a single processor and scale
  to thousands of processors
• More efficient programming models and tools for
  massively parallel supercomputers

1,000
Peak Performance
100
Performance Gap
Teraflops
10
1
Real Performance
0.1
2000
2004
1996
36
Scientific Computing Demand
37
Engineering Computing Demand

• Large parallel machines a mainstay in many areas
• Petroleum (reservoir analysis)
• Automotive (crash simulation, drag analysis,
  combustion efficiency),
• Aeronautics (airflow analysis, engine efficiency,
  structural mechanics, electromagnetism),
• Computer-aided design
• Pharmaceuticals (molecular modeling)
• Visualization
• in all of the above
• entertainment (films like Toy Story)
• architecture (walk-throughs and rendering)
• Financial modeling (yield and derivative
  analysis)
• Nuclear Reactor Analysis and Design

38
Applications Speech and Image Processing

• Also CAD, Databases, . . .
• 100 processors gets you 10 years, 1000 gets you
  20 !

39
Microprocessor Transistors per Chip

• Growth in transistors per chip

• Increase in clock rate

40
Summary of Application Trends

• Transition to parallel computing has occurred for
  scientific and engineering computing
• In rapid progress in commercial computing
• Database and transactions as well as financial
• Usually smaller-scale, but large-scale systems
  also used
• Desktop also uses multithreaded programs, which
  are a lot like parallel programs
• Demand for improving throughput on sequential
  workloads
• Greatest use of small-scale multiprocessors
• Solid application demand exists and will increase

41
Technology Trends

• Today the natural building-block is also fastest!

42
Consider Scientific Supercomputing

• Proving ground and driver for innovative
  architecture and techniques
• Market smaller relative to commercial as MPs
  become mainstream
• Dominated by vector machines starting in 70s
• Microprocessors have made huge gains in
  floating-point performance
• high clock rates
• pipelined floating point units (e.g.,
  multiply-add every cycle)
• instruction-level parallelism
• effective use of caches (e.g., automatic
  blocking)
• Plus economics
• Large-scale multiprocessors replace vector
  supercomputers

43
Raw Uniprocessor Performance LINPACK
44
Raw Parallel Performance LINPACK

• Even vector Crays became parallel
• X-MP (2-4) Y-MP (8), C-90 (16), T94 (32)
• Since 1993, Cray produces MPPs too (T3D, T3E)

45
Summary Why Parallel Architecture?

• Increasingly attractive
• Economics, technology, architecture, application
  demand
• Increasingly central and mainstream
• Parallelism exploited at many levels
• Instruction-level parallelism
• Multiprocessor servers
• Large-scale multiprocessors (MPPs)
• Focus of this class multiprocessor level of
  parallelism
• Same story from memory system perspective
• Increase bandwidth, reduce average latency with
  many local memories
• Spectrum of parallel architectures make sense
• Different cost, performance and scalability