From 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 Mon Sep 17 00:00:00 2001 From: Linus Torvalds Date: Sat, 16 Apr 2005 15:20:36 -0700 Subject: Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip! --- Documentation/vm/hugetlbpage.txt | 284 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 284 insertions(+) create mode 100644 Documentation/vm/hugetlbpage.txt (limited to 'Documentation/vm/hugetlbpage.txt') diff --git a/Documentation/vm/hugetlbpage.txt b/Documentation/vm/hugetlbpage.txt new file mode 100644 index 00000000000..1b9bcd1fe98 --- /dev/null +++ b/Documentation/vm/hugetlbpage.txt @@ -0,0 +1,284 @@ + +The intent of this file is to give a brief summary of hugetlbpage support in +the Linux kernel. This support is built on top of multiple page size support +that is provided by most modern architectures. For example, i386 +architecture supports 4K and 4M (2M in PAE mode) page sizes, ia64 +architecture supports multiple page sizes 4K, 8K, 64K, 256K, 1M, 4M, 16M, +256M and ppc64 supports 4K and 16M. A TLB is a cache of virtual-to-physical +translations. Typically this is a very scarce resource on processor. +Operating systems try to make best use of limited number of TLB resources. +This optimization is more critical now as bigger and bigger physical memories +(several GBs) are more readily available. + +Users can use the huge page support in Linux kernel by either using the mmap +system call or standard SYSv shared memory system calls (shmget, shmat). + +First the Linux kernel needs to be built with CONFIG_HUGETLB_PAGE (present +under Processor types and feature) and CONFIG_HUGETLBFS (present under file +system option on config menu) config options. + +The kernel built with hugepage support should show the number of configured +hugepages in the system by running the "cat /proc/meminfo" command. + +/proc/meminfo also provides information about the total number of hugetlb +pages configured in the kernel. It also displays information about the +number of free hugetlb pages at any time. It also displays information about +the configured hugepage size - this is needed for generating the proper +alignment and size of the arguments to the above system calls. + +The output of "cat /proc/meminfo" will have output like: + +..... +HugePages_Total: xxx +HugePages_Free: yyy +Hugepagesize: zzz KB + +/proc/filesystems should also show a filesystem of type "hugetlbfs" configured +in the kernel. + +/proc/sys/vm/nr_hugepages indicates the current number of configured hugetlb +pages in the kernel. Super user can dynamically request more (or free some +pre-configured) hugepages. +The allocation( or deallocation) of hugetlb pages is posible only if there are +enough physically contiguous free pages in system (freeing of hugepages is +possible only if there are enough hugetlb pages free that can be transfered +back to regular memory pool). + +Pages that are used as hugetlb pages are reserved inside the kernel and can +not be used for other purposes. + +Once the kernel with Hugetlb page support is built and running, a user can +use either the mmap system call or shared memory system calls to start using +the huge pages. It is required that the system administrator preallocate +enough memory for huge page purposes. + +Use the following command to dynamically allocate/deallocate hugepages: + + echo 20 > /proc/sys/vm/nr_hugepages + +This command will try to configure 20 hugepages in the system. The success +or failure of allocation depends on the amount of physically contiguous +memory that is preset in system at this time. System administrators may want +to put this command in one of the local rc init file. This will enable the +kernel to request huge pages early in the boot process (when the possibility +of getting physical contiguous pages is still very high). + +If the user applications are going to request hugepages using mmap system +call, then it is required that system administrator mount a file system of +type hugetlbfs: + + mount none /mnt/huge -t hugetlbfs + + +This command mounts a (pseudo) filesystem of type hugetlbfs on the directory +/mnt/huge. Any files created on /mnt/huge uses hugepages. The uid and gid +options sets the owner and group of the root of the file system. By default +the uid and gid of the current process are taken. The mode option sets the +mode of root of file system to value & 0777. This value is given in octal. +By default the value 0755 is picked. The size option sets the maximum value of +memory (huge pages) allowed for that filesystem (/mnt/huge). The size is +rounded down to HPAGE_SIZE. The option nr_inode sets the maximum number of +inodes that /mnt/huge can use. If the size or nr_inode options are not +provided on command line then no limits are set. For size and nr_inodes +options, you can use [G|g]/[M|m]/[K|k] to represent giga/mega/kilo. For +example, size=2K has the same meaning as size=2048. An example is given at +the end of this document. + +read and write system calls are not supported on files that reside on hugetlb +file systems. + +A regular chown, chgrp and chmod commands (with right permissions) could be +used to change the file attributes on hugetlbfs. + +Also, it is important to note that no such mount command is required if the +applications are going to use only shmat/shmget system calls. Users who +wish to use hugetlb page via shared memory segment should be a member of +a supplementary group and system admin needs to configure that gid into +/proc/sys/vm/hugetlb_shm_group. It is possible for same or different +applications to use any combination of mmaps and shm* calls. Though the +mount of filesystem will be required for using mmaps. + +******************************************************************* + +/* + * Example of using hugepage memory in a user application using Sys V shared + * memory system calls. In this example the app is requesting 256MB of + * memory that is backed by huge pages. The application uses the flag + * SHM_HUGETLB in the shmget system call to inform the kernel that it is + * requesting hugepages. + * + * For the ia64 architecture, the Linux kernel reserves Region number 4 for + * hugepages. That means the addresses starting with 0x800000... will need + * to be specified. Specifying a fixed address is not required on ppc64, + * i386 or x86_64. + * + * Note: The default shared memory limit is quite low on many kernels, + * you may need to increase it via: + * + * echo 268435456 > /proc/sys/kernel/shmmax + * + * This will increase the maximum size per shared memory segment to 256MB. + * The other limit that you will hit eventually is shmall which is the + * total amount of shared memory in pages. To set it to 16GB on a system + * with a 4kB pagesize do: + * + * echo 4194304 > /proc/sys/kernel/shmall + */ +#include +#include +#include +#include +#include +#include + +#ifndef SHM_HUGETLB +#define SHM_HUGETLB 04000 +#endif + +#define LENGTH (256UL*1024*1024) + +#define dprintf(x) printf(x) + +/* Only ia64 requires this */ +#ifdef __ia64__ +#define ADDR (void *)(0x8000000000000000UL) +#define SHMAT_FLAGS (SHM_RND) +#else +#define ADDR (void *)(0x0UL) +#define SHMAT_FLAGS (0) +#endif + +int main(void) +{ + int shmid; + unsigned long i; + char *shmaddr; + + if ((shmid = shmget(2, LENGTH, + SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W)) < 0) { + perror("shmget"); + exit(1); + } + printf("shmid: 0x%x\n", shmid); + + shmaddr = shmat(shmid, ADDR, SHMAT_FLAGS); + if (shmaddr == (char *)-1) { + perror("Shared memory attach failure"); + shmctl(shmid, IPC_RMID, NULL); + exit(2); + } + printf("shmaddr: %p\n", shmaddr); + + dprintf("Starting the writes:\n"); + for (i = 0; i < LENGTH; i++) { + shmaddr[i] = (char)(i); + if (!(i % (1024 * 1024))) + dprintf("."); + } + dprintf("\n"); + + dprintf("Starting the Check..."); + for (i = 0; i < LENGTH; i++) + if (shmaddr[i] != (char)i) + printf("\nIndex %lu mismatched\n", i); + dprintf("Done.\n"); + + if (shmdt((const void *)shmaddr) != 0) { + perror("Detach failure"); + shmctl(shmid, IPC_RMID, NULL); + exit(3); + } + + shmctl(shmid, IPC_RMID, NULL); + + return 0; +} + +******************************************************************* + +/* + * Example of using hugepage memory in a user application using the mmap + * system call. Before running this application, make sure that the + * administrator has mounted the hugetlbfs filesystem (on some directory + * like /mnt) using the command mount -t hugetlbfs nodev /mnt. In this + * example, the app is requesting memory of size 256MB that is backed by + * huge pages. + * + * For ia64 architecture, Linux kernel reserves Region number 4 for hugepages. + * That means the addresses starting with 0x800000... will need to be + * specified. Specifying a fixed address is not required on ppc64, i386 + * or x86_64. + */ +#include +#include +#include +#include +#include + +#define FILE_NAME "/mnt/hugepagefile" +#define LENGTH (256UL*1024*1024) +#define PROTECTION (PROT_READ | PROT_WRITE) + +/* Only ia64 requires this */ +#ifdef __ia64__ +#define ADDR (void *)(0x8000000000000000UL) +#define FLAGS (MAP_SHARED | MAP_FIXED) +#else +#define ADDR (void *)(0x0UL) +#define FLAGS (MAP_SHARED) +#endif + +void check_bytes(char *addr) +{ + printf("First hex is %x\n", *((unsigned int *)addr)); +} + +void write_bytes(char *addr) +{ + unsigned long i; + + for (i = 0; i < LENGTH; i++) + *(addr + i) = (char)i; +} + +void read_bytes(char *addr) +{ + unsigned long i; + + check_bytes(addr); + for (i = 0; i < LENGTH; i++) + if (*(addr + i) != (char)i) { + printf("Mismatch at %lu\n", i); + break; + } +} + +int main(void) +{ + void *addr; + int fd; + + fd = open(FILE_NAME, O_CREAT | O_RDWR, 0755); + if (fd < 0) { + perror("Open failed"); + exit(1); + } + + addr = mmap(ADDR, LENGTH, PROTECTION, FLAGS, fd, 0); + if (addr == MAP_FAILED) { + perror("mmap"); + unlink(FILE_NAME); + exit(1); + } + + printf("Returned address is %p\n", addr); + check_bytes(addr); + write_bytes(addr); + read_bytes(addr); + + munmap(addr, LENGTH); + close(fd); + unlink(FILE_NAME); + + return 0; +} -- cgit v1.2.3