Cache Optimisations

1
Cache Optimisations
2
??af??? ?p?d?s??
3
SRAM vs DRAM
6-transistor SRAM cell
1-transistor DRAM cell
4
Intel 45nm 6T SRAM cell
5
?e?a???a ???µ??

• ???pe? ?a µe??s??µe t? processor-memory
  performance gap
• ? p??sp??as? ded?µ???? (code data) de? e??a?
  ?µ???µ??f? (principle of locality) .
• S?ed?asµ?? ?e?a???a? µ??µ?? µe ß?s? 2 a????
• ??p???t?ta
• ?a µ????te?a components e??a? p?? ??????a
  (smaller hardware is faster)?

6
?a??de??µa ?e?a???a? ???µ??
7
?as???? ?????e? (?pa??????)?

• block line page
• ? µ????te?? µ???da µetaf???? ded?µ???? µeta?? t??
• ep?p?d?? µ??µ??

8
?as???? ?????e? (?pa??????)?

• hit t? block ß??s?eta? se ??p??a ??s? t??
  e?eta??µe??? ep?p?d?? µ??µ??
• hit rate hits/s???????? p??spe??se?? µ??µ??
• hit time ?????? p??sp??as?? t?? ded?µ????
• miss t? block de? ?p???e? st? e?eta??µe??
  ep?ped? µ??µ??
• miss rate 1 (hit rate)?
• miss penalty (?????? µetaf???? t?? ded?µ????
  e??? block st? s???e???µ??? ep?ped? µ??µ??)
  (?????? ap??t?s?? t?? ded?µ???? ap? t?? CPU)?
• access time ?????? ap??t?s?? t?? 1?? ?????
• transfer time ?????? ap??t?s?? t?? ?p????p??
  ???e??

9
?as???? ?????e? (?pa??????)?

• ??? t?p??et??µe ??a block se ??p??? ep?ped? t??
  ?e?a???a? µ??µ?? (block placement)
• Direct mapped
• ?????te??? ?????? p??sßas??
• ?e?a??te?? miss rate
• Fully associative
• ?????te?? miss rate
• ?e?a??te??? ?????? p??sßas??
• Set associative
• S??d?asµ?? t?? 2 p??????µ????
• ? p?? s?????sµ??? ep?????

10
?as???? ?????e? (?pa??????)?
11
?as???? ?????e? (?pa??????)?

• ??? ß??s???µe ??a block se ??p??? ep?ped? t??
  ?e?a???a? µ??µ?? (block identification)

12
?as???? ?????e? (?pa??????)?

• ???? block a?t??a??st??µe se pe??pt?s? e??? miss
  (block replacement)
• Random, LRU, FIFO
• ?? p???t??? a????????µe ?ta? t? pe??e??µe?? e???
  block µetaß???e?
• write hit write-through vs. write-back
• write miss write-allocate vs. no-write-allocate

13
Cache Misses 3C's

• Compulsory S?µßa????? ?at? t?? p??t? p??sßas? se
  ??a block. ?? block p??pe? ?a ????e? ap?
  ?aµ???te?a ep?peda µ??µ?? ?a? ?a t?p??et??e? st??
  cache (ap??a????ta? ?a? cold start misses ? first
  reference misses). Ta s???ßa??a? a??µa ?a? se µ?a
  ?pe??? cache.
• Capacity ?ta? ? cache de ???? ??a ta ded?µ??a
  ??p??a blocks ap?µa??????ta?. ?ta? ??t????? ?a??
  st? µ????? ????µe capacity miss. ???a? ta misses
  µ?a? Fully Associative Cache (af?? afa???s??µe ta
  compulsory misses).
• Conflict Se µ?a set-associative ? direct-mapped
  cache, p???? blocks ape????????ta? st? ?d?? set.
  ?ts? e?? µp??e? ?a ?p?????? ?de?a sets st??
  ?p????p? cache, ??p??a blocks ap?µa??????ta?.
  ?ta? ??t????? ?a?? st? µ????? ????µe conflict
  miss.

14
Cache Misses 3C's?p???t? Miss Rate (SPEC92)?
15
Cache Misses 3C'sS?et??? Miss Rate (SPEC92)?
100
1-way
80
2-way
4-way
8-way
60
Miss Rate / e?d??
40
Capacity
20
0
2
4
8
1
16
32
64
128
Cache Size (KB)
Compulsory
16
?e?t?st?p???s? ?p?d?s?? t?? Cache

• ?e??s? t?? cache miss penalty
• Multilevel caches, Critical word first, victim
  caches,...
• ?e??s? t?? miss rate
• Block/Cache size, Associativity,
  Pseudoassociative Caches,...
• ?e??s? t?? miss penalty ?a? t?? miss rate µ?s?
  pa?a?????sµ??
• Non-blocking caches, prefetching,...
• ?e??s? t?? hit time
• ?????? caches, trace caches, ...

17
(1) ?e??s? t?? miss penaltyMultilevel Caches

• S?ed?ast??? d???µµa
• ????? cache ?a? ??a t?s? ??????? ?s? ?a? ?
  epe?e??ast??
• ?e???? cache p?? ?a ???? p???? ded?µ??a a??? p???
  p?? a???
• ??s? ?e?a???a µ??µ?? p????? ep?p?d??.
• L1 ????? ?a? ??????? ?ste ? epe?e??ast?? ?a
  µp??e? ?a t?? p??spe??se? se 1-2 ??????? (hit
  time).
• L2 ?e?a??te?? ap? t?? L1. ??? a???, a??? µp??e?
  ?a ??a??p??e? ta pe??ss?te?a ap? ta L1 misses
  µe?????ta? a?t? p?? p??pe? ?a p??spe??s??? t??
  ????a µ??µ?.
• L3
• ...
• Main memory

18
(1) ?e??s? t?? miss penaltyCritical Word First
?a? Early Restart

• ?e? ??e???eta? ?a pe??µ????µe ?a µetafe??e?
  ???????? t? block p??? e?d?p???s??µe t??
  epe?e??ast? ?a s??e??se? t?? e?t??es? t??
  p?????µµat??.
• Critical word first F??t??eta? p??t? ? ???? p??
  ??t?se ? epe?e??ast??. ?? ?p????pe? ???e?? t??
  block µetaf????ta? st?? cache e?? ? epe?e??ast??
  s??e???e? t?? epe?e??as?a.
• Early restart ?? ???e?? t?? block f??t????ta?
  st?? cache µe t?? se???. ?ta? f??t??e? ?
  ??t??µe?? ????, ? epe?e??ast?? s??e???e? t??
  ?e?t?????a t??, e?? ta?t?????a f??t????ta? st??
  cache ?a? ?? ?p????pe? ???e?? t?? block p??
  a?????????.
• ???s?µe? ??a caches µe µe???? µ??e??? cache
  block.
• ???sf????? µ???? ße?t??s? ??a p?????µµata µe
  ????? ?????? t?p???t?ta, af?? pa???s?????? µe????
  p??a??t?ta ?a ??t??? ded?µ??a p?? ß??s???ta? se
  ?e?t?????? ??se?? µ??µ??.

19
(1) ?e??s? t?? miss penalty???te?a??t?ta t??
Read Misses

• ???p???t?s? t?? Read Misses p??? ???????????? ta
  write misses.
• ? write-buffer (FIFO d?µ?) ap????e?e? ta writes
  (t??p???µ??a ded?µ??a) p?? p??pe? ?a ap????e?t???
  sta ep?µe?a ep?peda t?? ?e?a???a? µ??µ??.
• ???a??t?ta RAW hazards!

20
(1) ?e??s? t?? miss penalty???te?a??t?ta t??
Read Misses

• RAW hazards se write-through caches µe
  write-buffers
• O write-buffer ??at? ta p?? p??sfata t??p???µ??a
  ded?µ??a
• ??a ??s? ?a pe??µ????µe ?a ade??se? ?
  write-buffer. ?ts? ?µ?? a????eta? t? miss penalty
• ?e?te?? ??s? e??a? ? ??e???? t?? pe??e??µ???? t??
  buffer se ???e read miss. ?? ta ded?µ??a p??
  ?????µe ?a d?aß?s??µe de? ?p?????? st?? buffer,
  d????µe p??te?a??t?ta st? read miss ?a? t?
  p??????µe st? ep?µe?? ep?ped? t?? ?e?a???a?
  µ??µ??.
• ? te????? a?t? ß???? ?a? se write-back caches.
  ?st? ?t? ??a read miss ?a a?t??atast?se? ??a
  dirty block t?? cache.
• ???? ????af? t?? dirty block st? ep?µe??
  ep?ped? µ??µ?? ? ???p???t?s? t?? miss, a?????s?
  ?a? epa?e?????s? t?? epe?e??ast?
• ???a ?etaf??? t?? dirty block st?? write-buffer
  ? ???p???t?s? t?? miss, a?????s? ?a? epa?e?????s?
  t?? epe?e??ast?-gt ????af? t?? dirty block st?
  ep?µe?? ep?ped? µ??µ??

21
(1) ?e??s? t?? miss penalty???p???s? (merging)
t?? write buffers

• S??d?asµ?? p???ap??? writes se ??a entry t??
  write buffer.
• ?p?d?t???te?? ???s? t?? cache badwidth
  (multiwrites p?? ??????a ap? e???af?? µ??ad????
  ???e?? µe t? se???)?
• ?e??s? t?? stalls p?? ?fe????ta? se full
  write-buffers.

22
(1) ?e??s? t?? miss penaltyVictim Caches

• ???s???? e??? µ????? buffer ??a ap????e?s? t??
  blocks p?? ap?µa??????ta? ap? t?? cache.

• Se ???e miss e??????µe ta pe??e??µe?a t?? victim
  cache p??? s??e??s??µe t?? a?a??t?s? st? ep?µe??
  ep?ped? t?? ?e?a???a? µ??µ??.
• Jouppi 1990 ??? victim cache 4 ??se??
  ap?t??pe? t? 20-95 t?? conflict misses ??a µ?a
  4?? direct mapped cache.

23
(No Transcript)
24
?e?t?st?p???s? ?p?d?s?? t?? Cache

• ?e??s? t?? cache miss penalty
• Multilevel caches, Critical word first, victim
  caches,...
• ?e??s? t?? miss rate
• Block/Cache size, Associativity,
  Pseudoassociative Caches,...
• ?e??s? t?? miss penalty ?a? t?? miss rate µ?s?
  pa?a?????sµ??
• Non-blocking caches, prefetching,...
• ?e??s? t?? hit time
• ?????? caches, trace caches, ...

25
(2) ?e??s? t?? miss rate????s? t?? block size

• ????p???s? t?? t?p???? ??????t?ta? (spatial
  locality).

• ?e??s? t?? compulsory misses
• ?a?t?????a
• ????s? t?? miss penalty
• ???a?? a???s? t?? capacity ?a? conflict misses
• ???se?t??? ep????? t?? block size!

26
(2) ?e??s? t?? miss rate????s? t?? cache size

• ?e??s? t?? capacity misses
• ?e??s? t?? miss rate
• ?e???e?t?µata
• ????s? t?? hit time
• ????s? t?? ?atas?e?ast???? ??st???
• ????p???s? t?? µe????? a???µ?? transistors p??
  ?p???e? p???? sta chips.

27
(2) ?e??s? t?? miss rate????s? t?? ßa?µ??
Associativity

• ????s? t?? ßa?µ?? associativity s??ep??eta?
  µe??s? t?? miss rate.
• ??a?t???
• G?a single processor systems, µ?a 8-way set
  associative cache ??e? p?a?t??? t? ?d?? miss rate
  µe µ?a fully associative cache.
• ??a direct-mapped cache µe size N ??e? t? ?d??
  miss rate µe µ?a 2-way set associative cache µe
  size N/2.
• ?e???e?t?µata
• ????s? t?? hit time
• ????s? t?? ??st???

28
(2) ?e??s? t?? miss ratePseudoassociative Caches

• S??d?asµ?? t??
• Direct-mapped caches ? ????? hit time
• 2-way set associative caches ? ?e??s? t??
  conflict misses
• ?ta? ????µe miss, p??? p??????s??µe sta ep?µe?a
  ep?peda t?? ?e?a???a? µ??µ??, e??????µe a?
  ?p???e? ? d?e????s? p?? ??????µe se µ?a de?te??
  ??s? t?? cache (pseudo-hit).
• ???p???s? ??ast??f? t?? MSB t?? index ??a ?a
  p??spe??s??µe t? pseudo-set.
• ????? ??a ??????? (hit) ?a? ??a p?? a???
  (pseudo-hit) ????? a?a??t?s?? (hit time).

29
(2) ?e??s? t?? miss rateCompiler Optimizations

• ?? p??????µe?e? te?????? apa?t???
  a??a???/p??s???e? st? hardware t?? s?st?µat??.
• ??a??a?t??? ?e?t?st?p???s? t?? software!
• Compiler Optimizations
• Instructions
• ??ad???????s? t?? procedures st? µ??µ? ??a t?
  µe??s? t?? conflict misses
• Data
• Merging arrays
• Loop interchange
• Loop fusion
• Blocking

30
(2) ?e??s? t?? miss rateMerging Arrays

• / Before 2 sequential arrays /
• int valSIZE
• int keySIZE
• / After 1 array of stuctures /
• struct merge
• int val
• int key
• struct merge merged_arraySIZE
• ?e?????ta? ta conflicts µeta?? t?? st???e??? t??
  val ?a? key
• ?e?t??s? t?? ??????? t?p???t?ta? (spatial
  locality)?

31
(2) ?e??s? t?? miss rateLoop interchange

• / Before /
• for (j 0 j lt 100 j j1)?
• for (i 0 i lt 5000 i i1)?
• xij 2 xij
• / After /
• for (i 0 i lt 5000 i i1)?
• for (j 0 j lt 100 j j1)?
• xij 2 xij
• ??????, ? ???e ???? p?? d?aß??eta? ap??e? 100
  ??se?? ap? t?? p??????µe??.
• ?et? t?? a??a??, ? p??sp??as? ???eta? se
  d?ad?????? ??se?? µ??µ??.
• ??aß????ta? µe t? se??? ??e? ?? ???e?? t?? cache
  block
• ?e?t??s? t?? ??????? t?p???t?ta?

32
(2) ?e??s? t?? miss rateBlocking

• ???ßasµa t?? NxN st???e??? t?? z ?a? t?? ?
  st???e??? µ?a? ??aµµ?? t?? y ?a? e???af? t?? N
  st???e??? µ?a? ??aµµ?? t?? x.
• ?a capacity misses e?a?t??ta? ap? t? N ?a? t?
  µ??e??? t?? cache.
• size 3xNxNxsizeof(array_elem) ? 0 capacity
  misses
• S???????? a???µ?? accesses 2N3 N2
• ?d?a ?pe?e??as?a e??? BxB ?p?p??a?a p?? ?a
  ????e? st?? cache

/ Before / for (i 0 i lt N i i1)? for (j
0 j lt N j j1)? r 0 for (k 0 k
lt N k k1) r r yikzkj
xij r
33
(2) ?e??s? t?? miss rateBlocking
/ After / for (jj 0 jj lt N jj jjB)? for
(kk 0 kk lt N kk kkB)? for (i 0 i lt N i
i1)? for (j jj j lt min(jjB-1,N) j
j1)? r 0 for (k kk k lt
min(kkB-1,N) k k1)? r r
yikzkj xij xij r

• B Blocking factor
• ?e??s? t?? capacity misses
• 2N3/B N2
• ?e?t??s? ?a? t?? ???????? ?a? t?? ???????
  t?p???t?ta?

34
?e?t?st?p???s? ?p?d?s?? t?? Cache

• ?e??s? t?? cache miss penalty
• Multilevel caches, Critical word first, victim
  caches,...
• ?e??s? t?? miss rate
• Block/Cache size, Associativity,
  Pseudoassociative Caches,...
• ?e??s? t?? miss penalty ?a? t?? miss rate µ?s?
  pa?a?????sµ??
• Non-blocking caches, prefetching,...
• ?e??s? t?? hit time
• ?????? caches, trace caches, ...

35
(3) ?e??s? miss rate/miss penalty µ?s?
pa?a?????sµ??Multiple Banks

• ??t? ?a ?e????µe t?? cache sa? ??a µ??ad???
  block, t?? d?a????µe se p???ap?? a?e???t?ta
  banks.
• p?. Niagara L2 4 banks
• ???at?t?ta ta?t??????? p??spe??se?? (1 se ???e
  bank)?
• ????? ap?d?s? ?ta? ?? p??spe??se?? µ???????ta?
  a??µesa sta banks. ?p?µ????, t? mapping t??
  d?e????se?? se banks ep??e??e? ?µesa t?? ap?d?s?
  t?? s?st?µat??.
• ??a ap?? ?a? ap?d?t??? mapping e??a? t?
  sequential interleaving
• ?? d?e????se?? a?t?st???????ta? µe t? se??? se
  ???e bank
• ?? ????µe 8 banks, t?te ta blocks ??a ta ?p??a
  block address mod 8 0 ap????e???ta? st? bank 0,
  a?t? ??a ta ?p??a ?s??e? block address mod 8 1
  st? bank 1, ...

36
(3) ?e??s? miss rate/miss penalty µ?s?
pa?a?????sµ??Nonblocking caches

• ?? nonblocking caches ep?t??p??? st?? data caches
  ?a ap?st?????? ded?µ??a (e??p???t?s? cache hits)
  ?s? d?e?pe?a???eta? ??a cache miss.
• ???s? se out-of-order s?st?µata
• ?pa?t???ta? p???ap?? memory banks ??a t??
  pa??????? e??p???t?s? p??spe??se??
• hit under miss ?e??s? t?? effective miss
  penalty ?a??? de? a??????ta? ?a??????e?
  p??spe??se??
• hit under multiple miss / miss under miss
  ?p?p???? µe??s? t?? effective miss penalty
  ep??a??pt??ta? p???ap?? misses
• ????s? t?? p???p????t?ta? t?? cache controller
  ?a??? µp??e? ?a ?p?????? p???ap??? p??spe??se??
  p?? pe??µ????? ?a ??a??p???????

37
(3) ?e??s? miss rate/miss penalty µ?s?
pa?a?????sµ??Nonblocking caches
38
(3) ?e??s? miss rate/miss penalty µ?s?
pa?a?????sµ??Hardware Prefetching

• ?d?a F???? st?? cache a?t? p?? ?a ??t?se? st?
  s????e?a ? epe?e??ast??!
• Instructions
• Se ???e miss f?????µe 2 block. ??t? p?? ??t?se ?
  epe?e??ast?? (ap????e?s? st?? cache) ?a? t?
  aµ?s?? ep?µe?? (?e?t?????). ?? 2? block
  ap????e?eta? se e?a instruction stream buffer.
• Jouppi 1990 Instruction stream buffer µe 16
  blocks ße?t???e? t? hit rate µ?a? 4KB
  direct-mapped instruction cache ?ata 72.
• Data
• ?d?a ?????? ?a? ??a t?? data cache.
• ?p??tas? µe p???ap???? stream buffers, ?p?? ?
  ?a???a? ???e? prefetch µ?a d?af??et??? d?e????s?.
• Palacharla 1994 8 stream buffers µp????? ?a
  µe??s??? ?at? 50-70 ta misses e??? s?st?µat?? µe
  64?? 4-way assoc. caches (Instr. Data)?

39
(3) ?e??s? miss rate/miss penalty µ?s?
pa?a?????sµ?? Software Prefetching

• O compiler e?s??e? ?at?????e? e?t???? (prefetch
  instructions), ?? ?p??e? p???a???? t? µetaf???
  ded?µ???? (data) p??? a?t? ??e?ast??? ap? t?
  p????aµµa.
• ??? e?d??
• Register prefetch F??t?s? ded?µ???? se
  ?ata????t?? (p.?. loads t?? HP PA-RISC)?
• Cache prefetch F??t?s? ded?µ???? st?? cache
  (p.?. MIPS IV, PowerPC, SPARC v9)?
• Nonfaulting/Nonbinding ?e? ep?t??peta? ?a
  p???a??s??? exceptions (p.?. virtual address
  faults)?
• ?p?? ?a? st?? pe??pt?s? t?? hardware prefetching,
  ta s?st?µata a?t? p???p???t??? t? ???s?
  nonblocking caches.

40
?e?t?st?p???s? ?p?d?s?? t?? Cache

• ?e??s? t?? cache miss penalty
• Multilevel caches, Critical word first, victim
  caches,...
• ?e??s? t?? miss rate
• Block/Cache size, Associativity,
  Pseudoassociative Caches,...
• ?e??s? t?? miss penalty ?a? t?? miss rate µ?s?
  pa?a?????sµ??
• Non-blocking caches, prefetching,...
• ?e??s? t?? hit time
• ?????? caches, trace caches, ...

41
(4) ?e??s? hit time?????? ?? ap??? caches

• ???? s?µa?t??? ?d?a?te?a ??a t?? first-level
  caches
• ?e???? ??µµ?t? t?? hit time ap?te?e? ? p??sp??as?
  t?? tag array ?a? ? s?????s? µe t? ?at?????
  ??µµ?t? t?? ??t??µe??? d?e????s??.

• ?????? µ??µe?
• G?????? indexing
• ??p???t?s? ???t? st?? epe?e??ast?
• ?p??? µ??µe? p.?. Direct-mapped
• ?p??????? t?? s?????s?? t?? tag µe t?? ap?st???
  t?? ded?µ????

42
(4) ?e??s? hit time?p?f??? Address Translation

• Virtual Caches ? Virtually Addressed Caches
• ?p?st??? t?? virtual address st?? cache.
• Se ???e a??a?? d?e??as?a? p??pe? ?a ?a?a?????µe
  (flush) t?? cache
• ??st?? ?????? flush compulsory misses
• Aliases ? Synonyms ??af??et???? virtual address
  (p.?. OS ?a? user program) a?t?st???????ta? st??
  ?d?a f?s??? d?e????s? ? ????ap?? a?t???afa t??
  ?d??? block
• Virtual addresses ??a t?? ep????????a ?/? µ???d??
  µe t?? caches
• ??se??
• ???s? e??? process-identifier tag (PID).
  ?pa?te?ta? flush µ??? ?ta? ??a pa??? PID
  ?a?a???s?µ?p??e?ta? ??a µ?a ?a??????a d?e??as?a.
• Aliases
• ?ardware ?at??????? µ??a??sµ?? ?a? ??e????
  e?????ta? µ?a µ??ad??? f?s??? d?e????s? ??a ???e
  block t?? cache
• Software Page coloring ? ?at?????? ep????? t??
  virtual pages/addresses ??a t?? ap?f???
  d?µ??????a? aliases.

43
(4) ?e??s? hit timeVirtually Addressed Caches
44
(4) ?e??s? hit timeTrace Caches

• ????? t??a t? cache block µ?a? Instruction cache
  pe????e? µ?a se??? e?t???? ?p?? a?t?? ??????ta?
  (ap????e???ta?) st?? µ??µ?.
• ??? ap?d?t??? t? block ?a pe????e? µ?a d??aµ???
  se??? e?t???? ?p?? a?t?? e?te????ta? st??
  epe?e??ast?! ? ?race Cache
• ?e???e?t?µata
• ????p???? address mapping
• ??p??e? e?t???? µp??e? ?a ap????e???ta? p???ap???
  f???? ?a??? eµfa?????ta? se p???ap?? traces
  e?a?t?a? d?af??et???? branches. ? ??
  ap?d?t??? ???s? t?? d?a??s?µ?? ?????.

45
Cache Optimizations

• ?e????? MP MR
  HT Complexity
• Multilevel caches 2
• Critical Word First Early Restart 2
• ???te?a??t?ta sta Read Misses 1
• Merging write buffers 1
• Victim Caches 1
• ?e?a??te?? block size - 0
• ?????te?? ßa?µ? Associativity - 1
• Pseudo-associative caches 2
• Compiler Optimizations 0
• Multiple Banks 1
• Nonblocking caches 3
• Hardware Prefetching 2
• Compiler Controlled Prefetching 3
• Small simple caches - 0