|
Sharing pattern
- Problem is with the writes
- Frequently written cache lines exhibit a small number of sharers; so small number of invalidations
- Widely shared data are written infrequently; so large number of invalidations, but rare
- Synchronization variables are notorious: heavily contended locks are widely shared and written in quick succession generating a burst of invalidations; require special solutions such as queue locks or tree barriers
- What about interventions? These are very problematic because in these cases you cannot send the interventions before looking up the directory and any speculative memory lookup would be useless
- For scientific applications interventions are small due to mostly one producer-many consumer pattern; for database workloads these take the lion’s share due to migratory pattern and tend to increase with bigger cache
- Optimizing interventions related to migratory sharing has been a major focus of high-end scalable servers
- AlphaServer GS320 employs few optimizations to quickly resolve races related to migratory hand-off (more later)
- Some academic research looked at destination or owner prediction to speculatively send interventions even before consulting the directory (Martin and Hill 2003, Acacio et al 2002)
- In general, directory provides far better utilization of bandwidth for scalable MPs compared to broadcast
Directory organization
- How to find source of directory information
- Centralized: just access the directory (bandwidth limited)
- Distributed: flat scheme distributes directory with memory and every cache line has a home node where its memory and directory reside
- Hierarchical scheme organizes the processors as the leaves of a logical tree (need not be binary) and an internal node stores the directory entries for the memory lines local to its children; a directory entry essentially tells you which of its children subtrees are caching the line and if some subtree which is not its children is also caching; finding the directory entry of a cache line involves a traversal of the tree until the entry is found (inclusion is maintained between level k and k+1 directory node where the root is at the highest level i.e. in the worst case may have to go to the root to find dir.)
- Format of a directory entry
- Varies a lot: no specific rule
- Memory-based scheme: directory entry is co-located in the home node with the memory line; various organizations can be used; the most popular one is a simple bit vector (with a 128 bytes line, storage overhead for 64 nodes is 6.35%, for 256 nodes 25%, for 1024 nodes 100%); clearly does not scale with P (more later)
- Cache-based scheme: Organize the directory as a distributed linked-list where the sharer nodes form a chain; the cache tag is extended to hold a node number; the home node only knows the id of the first sharer; on a read miss the requester adds itself to the head (involves home and first sharer); on a write miss traverse list and invalidate (essentially serialized chain of messages); advantage: distributes contention and does not make the home node a hot-spot, storage overhead is fixed; but very complex (IEEE SCI standard)
- Lot of research has been done to reduce directory storage overhead
- The trade-off is between preciseness of information and performance
- Normal trick is to have a superset of information e.g., group every two sharers into a cluster and have a bit per cluster: may lead to one useless invalidation per cluster
- We will explore this in detail later
- Memory-based bitvector scheme is very popular: invalidations can be overlapped or multicast
- Cache-based schemes incur serialized message chain for invalidation
- Hierarchical schemes are not used much due to high latency and volume of messages (up and down tree); also root may become a bandwidth bottleneck
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|