so, we must understand that cpu only understands bytes + addresses. so a memory manager exists because what happens is in physical component of ram there are so many sections and each section is defined by an number and at each number in memory there is some value or content and that section number is nothing but what we call addresses. and each of that section has a fixed size of 1 byte which is 8 bits. now a memory manager just is the manager of the memory how it allocates how it stores and all that.
so... yes we are building an memory manager it will be pain but who cares.