1 Memory Resource Controller(Memcg) Implementation Memo.
3 Base Kernel Version: based on 2.6.33-rc7-mm(candidate for 34).
5 Because VM is getting complex (one of reasons is memcg...), memcg's behavior
6 is complex. This is a document for memcg's internal behavior.
7 Please note that implementation details can be changed.
9 (*) Topics on API should be in Documentation/cgroups/memory.txt)
11 0. How to record usage ?
14 page_cgroup ....an object per page.
15 Allocated at boot or memory hotplug. Freed at memory hot removal.
17 swap_cgroup ... an entry per swp_entry.
18 Allocated at swapon(). Freed at swapoff().
20 The page_cgroup has USED bit and double count against a page_cgroup never
21 occurs. swap_cgroup is used only when a charged page is swapped-out.
25 a page/swp_entry may be charged (usage += PAGE_SIZE) at
27 mem_cgroup_try_charge()
30 a page/swp_entry may be uncharged (usage -= PAGE_SIZE) by
32 mem_cgroup_uncharge_page()
33 Called when an anonymous page is fully unmapped. I.e., mapcount goes
34 to 0. If the page is SwapCache, uncharge is delayed until
35 mem_cgroup_uncharge_swapcache().
37 mem_cgroup_uncharge_cache_page()
38 Called when a page-cache is deleted from radix-tree. If the page is
39 SwapCache, uncharge is delayed until mem_cgroup_uncharge_swapcache().
41 mem_cgroup_uncharge_swapcache()
42 Called when SwapCache is removed from radix-tree. The charge itself
43 is moved to swap_cgroup. (If mem+swap controller is disabled, no
44 charge to swap occurs.)
46 mem_cgroup_uncharge_swap()
47 Called when swp_entry's refcnt goes down to 0. A charge against swap
50 mem_cgroup_end_migration(old, new)
51 At success of migration old is uncharged (if necessary), a charge
52 to new page is committed. At failure, charge to old page is committed.
54 3. charge-commit-cancel
55 Memcg pages are charged in two steps:
56 mem_cgroup_try_charge()
57 mem_cgroup_commit_charge() or mem_cgroup_cancel_charge()
59 At try_charge(), there are no flags to say "this page is charged".
60 at this point, usage += PAGE_SIZE.
62 At commit(), the page is associated with the memcg.
64 At cancel(), simply usage -= PAGE_SIZE.
66 Under below explanation, we assume CONFIG_MEM_RES_CTRL_SWAP=y.
69 Anonymous page is newly allocated at
70 - page fault into MAP_ANONYMOUS mapping.
72 It is charged right after it's allocated before doing any page table
73 related operations. Of course, it's uncharged when another page is used
74 for the fault address.
76 At freeing anonymous page (by exit() or munmap()), zap_pte() is called
77 and pages for ptes are freed one by one.(see mm/memory.c). Uncharges
78 are done at page_remove_rmap() when page_mapcount() goes down to 0.
80 Another page freeing is by page-reclaim (vmscan.c) and anonymous
81 pages are swapped out. In this case, the page is marked as
82 PageSwapCache(). uncharge() routine doesn't uncharge the page marked
83 as SwapCache(). It's delayed until __delete_from_swap_cache().
86 At swap-in, the page is taken from swap-cache. There are 2 cases.
88 (a) If the SwapCache is newly allocated and read, it has no charges.
89 (b) If the SwapCache has been mapped by processes, it has been
92 This swap-in is one of the most complicated work. In do_swap_page(),
93 following events occur when pte is unchanged.
95 (1) the page (SwapCache) is looked up.
97 (3) try_charge_swapin()
98 (4) reuse_swap_page() (may call delete_swap_cache())
99 (5) commit_charge_swapin()
102 Considering following situation for example.
104 (A) The page has not been charged before (2) and reuse_swap_page()
105 doesn't call delete_from_swap_cache().
106 (B) The page has not been charged before (2) and reuse_swap_page()
107 calls delete_from_swap_cache().
108 (C) The page has been charged before (2) and reuse_swap_page() doesn't
109 call delete_from_swap_cache().
110 (D) The page has been charged before (2) and reuse_swap_page() calls
111 delete_from_swap_cache().
113 memory.usage/memsw.usage changes to this page/swp_entry will be
116 Before (2) 0/ 1 0/ 1 1/ 1 1/ 1
117 ===========================================
118 (3) +1/+1 +1/+1 +1/+1 +1/+1
120 (5) 0/-1 0/ 0 -1/-1 0/ 0
122 ===========================================
123 Result 1/ 1 1/ 1 1/ 1 1/ 1
125 In any cases, charges to this page should be 1/ 1.
128 At swap-out, typical state transition is below.
130 (a) add to swap cache. (marked as SwapCache)
131 swp_entry's refcnt += 1.
133 swp_entry's refcnt += # of ptes.
134 (c) write back to swap.
135 (d) delete from swap cache. (remove from SwapCache)
136 swp_entry's refcnt -= 1.
139 At (b), the page is marked as SwapCache and not uncharged.
140 At (d), the page is removed from SwapCache and a charge in page_cgroup
141 is moved to swap_cgroup.
143 Finally, at task exit,
144 (e) zap_pte() is called and swp_entry's refcnt -=1 -> 0.
145 Here, a charge in swap_cgroup disappears.
148 Page Cache is charged at
149 - add_to_page_cache_locked().
152 - __remove_from_page_cache().
154 The logic is very clear. (About migration, see below)
155 Note: __remove_from_page_cache() is called by remove_from_page_cache()
156 and __remove_mapping().
158 6. Shmem(tmpfs) Page Cache
159 Memcg's charge/uncharge have special handlers of shmem. The best way
160 to understand shmem's page state transition is to read mm/shmem.c.
161 But brief explanation of the behavior of memcg around shmem will be
162 helpful to understand the logic.
164 Shmem's page (just leaf page, not direct/indirect block) can be on
165 - radix-tree of shmem's inode.
167 - Both on radix-tree and SwapCache. This happens at swap-in
171 - A new page is added to shmem's radix-tree.
172 - A swp page is read. (move a charge from swap_cgroup to page_cgroup)
174 - A page is removed from radix-tree and not SwapCache.
175 - When SwapCache is removed, a charge is moved to swap_cgroup.
176 - When swp_entry's refcnt goes down to 0, a charge in swap_cgroup
180 One of the most complicated functions is page-migration-handler.
181 Memcg has 2 routines. Assume that we are migrating a page's contents
182 from OLDPAGE to NEWPAGE.
184 Usual migration logic is..
185 (a) remove the page from LRU.
186 (b) allocate NEWPAGE (migration target)
187 (c) lock by lock_page().
188 (d) unmap all mappings.
189 (e-1) If necessary, replace entry in radix-tree.
190 (e-2) move contents of a page.
191 (f) map all mappings again.
192 (g) pushback the page to LRU.
193 (-) OLDPAGE will be freed.
195 Before (g), memcg should complete all necessary charge/uncharge to
199 - If OLDPAGE is anonymous, all charges will be dropped at (d) because
200 try_to_unmap() drops all mapcount and the page will not be
203 - If OLDPAGE is SwapCache, charges will be kept at (g) because
204 __delete_from_swap_cache() isn't called at (e-1)
206 - If OLDPAGE is page-cache, charges will be kept at (g) because
207 remove_from_swap_cache() isn't called at (e-1)
209 memcg provides following hooks.
211 - mem_cgroup_prepare_migration(OLDPAGE)
212 Called after (b) to account a charge (usage += PAGE_SIZE) against
213 memcg which OLDPAGE belongs to.
215 - mem_cgroup_end_migration(OLDPAGE, NEWPAGE)
216 Called after (f) before (g).
217 If OLDPAGE is used, commit OLDPAGE again. If OLDPAGE is already
218 charged, a charge by prepare_migration() is automatically canceled.
219 If NEWPAGE is used, commit NEWPAGE and uncharge OLDPAGE.
221 But zap_pte() (by exit or munmap) can be called while migration,
222 we have to check if OLDPAGE/NEWPAGE is a valid page after commit().
225 Each memcg has its own private LRU. Now, its handling is under global
226 VM's control (means that it's handled under global zone->lru_lock).
227 Almost all routines around memcg's LRU is called by global LRU's
228 list management functions under zone->lru_lock().
230 A special function is mem_cgroup_isolate_pages(). This scans
231 memcg's private LRU and call __isolate_lru_page() to extract a page
233 (By __isolate_lru_page(), the page is removed from both of global and
239 Tests for racy cases.
241 9.1 Small limit to memcg.
242 When you do test to do racy case, it's good test to set memcg's limit
243 to be very small rather than GB. Many races found in the test under
245 (Memory behavior under GB and Memory behavior under MB shows very
246 different situation.)
249 Historically, memcg's shmem handling was poor and we saw some amount
250 of troubles here. This is because shmem is page-cache but can be
251 SwapCache. Test with shmem/tmpfs is always good test.
254 For NUMA, migration is an another special case. To do easy test, cpuset
255 is useful. Following is a sample script to do migration.
257 mount -t cgroup -o cpuset none /opt/cpuset
260 echo 1 > /opt/cpuset/01/cpuset.cpus
261 echo 0 > /opt/cpuset/01/cpuset.mems
262 echo 1 > /opt/cpuset/01/cpuset.memory_migrate
264 echo 1 > /opt/cpuset/02/cpuset.cpus
265 echo 1 > /opt/cpuset/02/cpuset.mems
266 echo 1 > /opt/cpuset/02/cpuset.memory_migrate
268 In above set, when you moves a task from 01 to 02, page migration to
269 node 0 to node 1 will occur. Following is a script to migrate all
276 /bin/echo $pid >$2/tasks 2>/dev/null
283 G1_TASK=`cat ${G1}/tasks`
284 G2_TASK=`cat ${G2}/tasks`
285 move_task "${G1_TASK}" ${G2} &
288 memory hotplug test is one of good test.
289 to offline memory, do following.
290 # echo offline > /sys/devices/system/memory/memoryXXX/state
291 (XXX is the place of memory)
292 This is an easy way to test page migration, too.
295 When using hierarchy, mkdir/rmdir test should be done.
296 Use tests like the following.
298 echo 1 >/opt/cgroup/01/memory/use_hierarchy
299 mkdir /opt/cgroup/01/child_a
300 mkdir /opt/cgroup/01/child_b
303 add limit to 01/child_b
304 run jobs under child_a and child_b
306 create/delete following groups at random while jobs are running.
307 /opt/cgroup/01/child_a/child_aa
308 /opt/cgroup/01/child_b/child_bb
309 /opt/cgroup/01/child_c
311 running new jobs in new group is also good.
313 9.6 Mount with other subsystems.
314 Mounting with other subsystems is a good test because there is a
315 race and lock dependency with other cgroup subsystems.
318 # mount -t cgroup none /cgroup -o cpuset,memory,cpu,devices
320 and do task move, mkdir, rmdir etc...under this.
323 Besides management of swap is one of complicated parts of memcg,
324 call path of swap-in at swapoff is not same as usual swap-in path..
325 It's worth to be tested explicitly.
327 For example, test like following is good.
329 # mount -t cgroup none /cgroup -o memory
331 # echo 40M > /cgroup/test/memory.limit_in_bytes
332 # echo 0 > /cgroup/test/tasks
333 Run malloc(100M) program under this. You'll see 60M of swaps.
335 # move all tasks in /cgroup/test to /cgroup
340 Of course, tmpfs v.s. swapoff test should be tested, too.
343 Out-of-memory caused by memcg's limit will kill tasks under
344 the memcg. When hierarchy is used, a task under hierarchy
345 will be killed by the kernel.
346 In this case, panic_on_oom shouldn't be invoked and tasks
347 in other groups shouldn't be killed.
349 It's not difficult to cause OOM under memcg as following.
350 Case A) when you can swapoff
352 #echo 50M > /memory.limit_in_bytes
355 Case B) when you use mem+swap limitation.
356 #echo 50M > memory.limit_in_bytes
357 #echo 50M > memory.memsw.limit_in_bytes
360 9.9 Move charges at task migration
361 Charges associated with a task can be moved along with task migration.
365 #echo $$ >/cgroup/A/tasks
366 run some programs which uses some amount of memory in /cgroup/A.
370 #echo 1 >/cgroup/B/memory.move_charge_at_immigrate
371 #echo "pid of the program running in group A" >/cgroup/B/tasks
373 You can see charges have been moved by reading *.usage_in_bytes or
374 memory.stat of both A and B.
375 See 8.2 of Documentation/cgroups/memory.txt to see what value should be
376 written to move_charge_at_immigrate.
378 9.10 Memory thresholds
379 Memory controller implements memory thresholds using cgroups notification
380 API. You can use tools/cgroup/cgroup_event_listener.c to test it.
382 (Shell-A) Create cgroup and run event listener
384 # ./cgroup_event_listener /cgroup/A/memory.usage_in_bytes 5M
386 (Shell-B) Add task to cgroup and try to allocate and free memory
387 # echo $$ >/cgroup/A/tasks
388 # a="$(dd if=/dev/zero bs=1M count=10)"
391 You will see message from cgroup_event_listener every time you cross
394 Use /cgroup/A/memory.memsw.usage_in_bytes to test memsw thresholds.
396 It's good idea to test root cgroup as well.