Home | Libraries | People | FAQ | More |
Shared memory is the fastest interprocess communication mechanism. The operating system maps a memory segment in the address space of several processes, so that several processes can read and write in that memory segment without calling operating system functions. However, we need some kind of synchronization between processes that read and write shared memory.
Consider what happens when a server process wants to send an HTML file to a client process that resides in the same machine using network mechanisms:
As we can see, there are two copies, one from memory to the network and another one from the network to memory. And those copies are made using operating system calls that normally are expensive. Shared memory avoids this overhead, but we need to synchronize both processes:
Using shared memory, we can avoid two data copies, but we have to synchronize the access to the shared memory segment.
To use shared memory, we have to perform 2 basic steps:
Once the two steps have been successfully completed, the process can start writing to and reading from the address space to send to and receive data from other processes. Now, let's see how can we do this using Boost.Interprocess:
To manage shared memory, you just need to include the following header:
#include <boost/interprocess/shared_memory_object.hpp>
As we've mentioned we have to use the shared_memory_object
class to create, open and destroy shared memory segments that can be mapped
by several processes. We can specify the access mode of that shared memory
object (read only or read-write), just as if it was a file:
using boost::interprocess; shared_memory_object shm_obj (create_only //only create ,"shared_memory" //name ,read_write //read-write mode );
using boost::interprocess; shared_memory_object shm_obj (open_or_create //open or create ,"shared_memory" //name ,read_only //read-only mode );
using boost::interprocess; shared_memory_object shm_obj (open_only //only open ,"shared_memory" //name ,read_write //read-write mode );
When a shared memory object is created, its size is 0. To set the size
of the shared memory, the user must use the truncate
function call, in a shared memory that has been opened with read-write
attributes:
shm_obj.truncate(10000);
As shared memory has kernel or filesystem persistence, the user must explicitly
destroy it. The remove
operation might fail returning false if the shared memory does not exist,
the file is open or the file is still memory mapped by other processes:
using boost::interprocess; shared_memory_object::remove("shared_memory");
For more details regarding shared_memory_object
see the boost::interprocess::shared_memory_object
class reference.
Once created or opened, a process just has to map the shared memory object
in the process' address space. The user can map the whole shared memory
or just part of it. The mapping process is done using the mapped_region
class. The class represents
a memory region that has been mapped from a shared memory or from other
devices that have also mapping capabilities (for example, files). A mapped_region
can be created from any
memory_mappable
object
and as you might imagine, shared_memory_object
is a memory_mappable
object:
using boost::interprocess; std::size_t ShmSize = ... //Map the second half of the memory mapped_region region ( shm //Memory-mappable object , read_write //Access mode , ShmSize/2 //Offset from the beginning of shm , ShmSize-ShmSize/2 //Length of the region ); //Get the address of the region region.get_address(); //Get the size of the region region.get_size();
The user can specify the offset from the mappable object where the mapped region should start and the size of the mapped region. If no offset or size is specified, the whole mappable object (in this case, shared memory) is mapped. If the offset is specified, but not the size, the mapped region covers from the offset until the end of the mappable object.
For more details regarding mapped_region
see the boost::interprocess::mapped_region
class reference.
Let's see a simple example of shared memory use. A server process creates a shared memory object, maps it and initializes all the bytes to a value. After that, a client process opens the shared memory, maps it, and checks that the data is correctly initialized:
#include <boost/interprocess/shared_memory_object.hpp> #include <boost/interprocess/mapped_region.hpp> #include <cstring> #include <cstdlib> #include <string> int main(int argc, char *argv[]) { using namespace boost::interprocess; if(argc == 1){ //Parent process //Remove shared memory on construction and destruction struct shm_remove { shm_remove() { shared_memory_object::remove("MySharedMemory"); } ~shm_remove(){ shared_memory_object::remove("MySharedMemory"); } } remover; //Create a shared memory object. shared_memory_object shm (create_only, "MySharedMemory", read_write); //Set size shm.truncate(1000); //Map the whole shared memory in this process mapped_region region(shm, read_write); //Write all the memory to 1 std::memset(region.get_address(), 1, region.get_size()); //Launch child process std::string s(argv[0]); s += " child "; if(0 != std::system(s.c_str())) return 1; } else{ //Open already created shared memory object. shared_memory_object shm (open_only, "MySharedMemory", read_only); //Map the whole shared memory in this process mapped_region region(shm, read_only); //Check that memory was initialized to 1 char *mem = static_cast<char*>(region.get_address()); for(std::size_t i = 0; i < region.get_size(); ++i) if(*mem++ != 1) return 1; //Error checking memory } return 0; }
Boost.Interprocess provides portable shared memory in terms of POSIX semantics. Some operating systems don't support shared memory as defined by POSIX:
In those platforms, shared memory is emulated with mapped files created in a "boost_interprocess" folder created in a temporary files directory. In Windows platforms, if "Common AppData" key is present in the registry, "boost_interprocess" folder is created in that directory (in XP usually "C:\Documents and Settings\All Users\Application Data" and in Vista "C:\ProgramData"). For Windows platforms without that registry key and Unix systems, shared memory is created in the system temporary files directory ("/tmp" or similar).
Because of this emulation, shared memory has filesystem lifetime in some of those systems.
shared_memory_object
provides a static remove
function to remove a shared memory objects.
This function can fail if the shared memory
objects does not exist or it's opened by another process. Note that this
function is similar to the standard C int
remove(const char *path)
function. In UNIX systems, shared_memory_object::remove
calls shm_unlink
:
boost::interprocess::shared_memory_object
instance to behave as if no shared memory object of this name exists
(that is, trying to open an object with that name will fail and an
object of the same name can be created again).
In Windows operating systems, current version supports an usually acceptable emulation of the UNIX unlink behaviour: the file is renamed with a random name and marked as to be deleted when the last open handle is closed.
Creating a shared memory segment and mapping it can be a bit tedious when
several processes are involved. When processes are related via fork()
operating system call in UNIX systems a simpler method is available using
anonymous shared memory.
This feature has been implemented in UNIX systems mapping the device \dev\zero
or just using the MAP_ANONYMOUS
in a POSIX conformant mmap
system call.
This feature is wrapped in Boost.Interprocess
using the anonymous_shared_memory()
function, which returns a mapped_region
object holding an anonymous
shared memory segment that can be shared by related processes.
Here is an example:
#include <boost/interprocess/anonymous_shared_memory.hpp> #include <boost/interprocess/mapped_region.hpp> #include <iostream> #include <cstring> int main () { using namespace boost::interprocess; try{ //Create an anonymous shared memory segment with size 1000 mapped_region region(anonymous_shared_memory(1000)); //Write all the memory to 1 std::memset(region.get_address(), 1, region.get_size()); //The segment is unmapped when "region" goes out of scope } catch(interprocess_exception &ex){ std::cout << ex.what() << std::endl; return 1; } return 0; }
Once the segment is created, a fork()
call can be used so that region
is used to communicate two related
processes.
Windows operating system also offers shared memory, but the lifetime of this shared memory is very different to kernel or filesystem lifetime. The shared memory is created backed by the pagefile and it's automatically destroyed when the last process attached to the shared memory is destroyed.
Because of this reason, there is no effective way to simulate kernel or filesystem persistence using native windows shared memory and Boost.Interprocess emulates shared memory using memory mapped files. This assures portability between POSIX and Windows operating systems.
However, accessing native windows shared memory is a common request of
Boost.Interprocess users because they
want to access to shared memory created with other process that don't use
Boost.Interprocess. In order to manage
the native windows shared memory Boost.Interprocess
offers the windows_shared_memory
class.
Windows shared memory creation is a bit different from portable shared
memory creation: the size of the segment must be specified when creating
the object and can't be specified through truncate
like with the shared memory object. Take in care that when the last process
attached to a shared memory is destroyed the shared
memory is destroyed so there is no persistency
with native windows shared memory.
Sharing memory between services and user applications is also different.
To share memory between services and user applications the name of the
shared memory must start with the global namespace prefix "Global\\"
. This global namespace
enables processes on multiple client sessions to communicate with a service
application. The server component can create the shared memory in the global
namespace. Then a client session can use the "Global" prefix
to open that memory.
The creation of a shared memory object in the global namespace from a session other than session zero is a privileged operation.
Let's repeat the same example presented for the portable shared memory object: A server process creates a shared memory object, maps it and initializes all the bytes to a value. After that, a client process opens the shared memory, maps it, and checks that the data is correctly initialized. Take in care that if the server exits before the client connects to the shared memory the client connection will fail, because the shared memory segment is destroyed when no proces is attached to the memory.
This is the server process:
#include <boost/interprocess/windows_shared_memory.hpp> #include <boost/interprocess/mapped_region.hpp> #include <cstring> #include <cstdlib> #include <string> int main(int argc, char *argv[]) { using namespace boost::interprocess; if(argc == 1){ //Parent process //Create a native windows shared memory object. windows_shared_memory shm (create_only, "MySharedMemory", read_write, 1000); //Map the whole shared memory in this process mapped_region region(shm, read_write); //Write all the memory to 1 std::memset(region.get_address(), 1, region.get_size()); //Launch child process std::string s(argv[0]); s += " child "; if(0 != std::system(s.c_str())) return 1; //windows_shared_memory is destroyed when the last attached process dies... } else{ //Open already created shared memory object. windows_shared_memory shm (open_only, "MySharedMemory", read_only); //Map the whole shared memory in this process mapped_region region(shm, read_only); //Check that memory was initialized to 1 char *mem = static_cast<char*>(region.get_address()); for(std::size_t i = 0; i < region.get_size(); ++i) if(*mem++ != 1) return 1; //Error checking memory return 0; } return 0; }
As we can see, native windows shared memory needs synchronization to make sure that the shared memory won't be destroyed before the client is launched.
In many UNIX systems, the OS offers another shared memory memory mechanism,
XSI (X/Open System Interfaces) shared memory segments, also known as "System
V" shared memory. This shared memory mechanism is quite popular and
portable, and it's not based in file-mapping semantics, but it uses special
functions (shmget
, shmat
, shmdt
,
shmctl
...).
Unlike POSIX shared memory segments, XSI shared memory segments are not
identified by names but by 'keys' usually created with ftok
.
XSI shared memory segments have kernel lifetime and must be explicitly
removed. XSI shared memory does not support copy-on-write and partial shared
memory mapping but it supports anonymous shared memory.
Boost.Interprocess offers simple (xsi_shared_memory
)
and managed (managed_xsi_shared_memory
)
shared memory classes to ease the use of XSI shared memory. It also wraps
key creation with the simple xsi_key
class.
Let's repeat the same example presented for the portable shared memory object: A server process creates a shared memory object, maps it and initializes all the bytes to a value. After that, a client process opens the shared memory, maps it, and checks that the data is correctly initialized.
This is the server process:
#include <boost/interprocess/xsi_shared_memory.hpp> #include <boost/interprocess/mapped_region.hpp> #include <cstring> #include <cstdlib> #include <string> using namespace boost::interprocess; void remove_old_shared_memory(const xsi_key &key) { try{ xsi_shared_memory xsi(open_only, key); xsi_shared_memory::remove(xsi.get_shmid()); } catch(interprocess_exception &e){ if(e.get_error_code() != not_found_error) throw; } } int main(int argc, char *argv[]) { if(argc == 1){ //Parent process //Build XSI key (ftok based) xsi_key key(argv[0], 1); remove_old_shared_memory(key); //Create a shared memory object. xsi_shared_memory shm (create_only, key, 1000); //Remove shared memory on destruction struct shm_remove { int shmid_; shm_remove(int shmid) : shmid_(shmid){} ~shm_remove(){ xsi_shared_memory::remove(shmid_); } } remover(shm.get_shmid()); //Map the whole shared memory in this process mapped_region region(shm, read_write); //Write all the memory to 1 std::memset(region.get_address(), 1, region.get_size()); //Launch child process std::string s(argv[0]); s += " child "; if(0 != std::system(s.c_str())) return 1; } else{ //Build XSI key (ftok based) xsi_key key(argv[0], 1); //Create a shared memory object. xsi_shared_memory shm (open_only, key); //Map the whole shared memory in this process mapped_region region(shm, read_only); //Check that memory was initialized to 1 char *mem = static_cast<char*>(region.get_address()); for(std::size_t i = 0; i < region.get_size(); ++i) if(*mem++ != 1) return 1; //Error checking memory } return 0; }
File mapping is the association of a file's contents with a portion of the address space of a process. The system creates a file mapping to associate the file and the address space of the process. A mapped region is the portion of address space that the process uses to access the file's contents. A single file mapping can have several mapped regions, so that the user can associate parts of the file with the address space of the process without mapping the entire file in the address space, since the file can be bigger than the whole address space of the process (a 9GB DVD image file in a usual 32 bit systems). Processes read from and write to the file using pointers, just like with dynamic memory. File mapping has the following advantages:
File mapping is not only used for interprocess communication, it can be used also to simplify file usage, so the user does not need to use file-management functions to write the file. The user just writes data to the process memory, and the operating systems dumps the data to the file.
When two processes map the same file in memory, the memory that one process writes is seen by another process, so memory mapped files can be used as an interprocess communication mechanism. We can say that memory-mapped files offer the same interprocess communication services as shared memory with the addition of filesystem persistence. However, as the operating system has to synchronize the file contents with the memory contents, memory-mapped files are not as fast as shared memory.
To use memory-mapped files, we have to perform 2 basic steps:
Once the two steps have been successfully completed, the process can start writing to and reading from the address space to send to and receive data from other processes and synchronize the file's contents with the changes made to the mapped region. Now, let's see how can we do this using Boost.Interprocess:
To manage mapped files, you just need to include the following header:
#include <boost/interprocess/file_mapping.hpp>
First, we have to link a file's contents with the process' address space.
To do this, we have to create a mappable object that represents that file.
This is achieved in Boost.Interprocess
creating a file_mapping
object:
using boost::interprocess; file_mapping m_file ("/usr/home/file" //filename ,read_write //read-write mode );
Now we can use the newly created object to create mapped regions. For more
details regarding this class see the boost::interprocess::file_mapping
class reference.
After creating a file mapping, a process just has to map the shared memory
in the process' address space. The user can map the whole shared memory
or just part of it. The mapping process is done using the mapped_region
class. as we have said
before The class represents a memory region that has been mapped from a
shared memory or from other devices that have also mapping capabilities:
using boost::interprocess; std::size_t FileSize = ... //Map the second half of the file mapped_region region ( m_file //Memory-mappable object , read_write //Access mode , FileSize/2 //Offset from the beginning of shm , FileSize-FileSize/2 //Length of the region ); //Get the address of the region region.get_address(); //Get the size of the region region.get_size();
The user can specify the offset from the file where the mapped region should start and the size of the mapped region. If no offset or size is specified, the whole file is mapped. If the offset is specified, but not the size, the mapped region covers from the offset until the end of the file.
If several processes map the same file, and a process modifies a memory range from a mapped region that is also mapped by other process, the changes are inmedially visible to other processes. However, the file contents on disk are not updated immediately, since that would hurt performance (writing to disk is several times slower than writing to memory). If the user wants to make sure that file's contents have been updated, it can flush a range from the view to disk. When the function returns, the flushing process has started but there is no guarantee that all data has been written to disk:
//Flush the whole region region.flush(); //Flush from an offset until the end of the region region.flush(offset); //Flush a memory range starting on an offset region.flush(offset, size);
Remember that the offset is not an offset on the file, but an offset in the mapped region. If a region covers the second half of a file and flushes the whole region, only the half of the file is guaranteed to have been flushed.
For more details regarding mapped_region
see the boost::interprocess::mapped_region
class reference.
Let's reproduce the same example described in the shared memory section, using memory mapped files. A server process creates a shared memory segment, maps it and initializes all the bytes to a value. After that, a client process opens the shared memory, maps it, and checks that the data is correctly initialized::
#include <boost/interprocess/file_mapping.hpp> #include <boost/interprocess/mapped_region.hpp> #include <iostream> #include <fstream> #include <string> #include <vector> #include <cstring> #include <cstddef> #include <cstdlib> int main(int argc, char *argv[]) { using namespace boost::interprocess; //Define file names const char *FileName = "file.bin"; const std::size_t FileSize = 10000; if(argc == 1){ //Parent process executes this { //Create a file file_mapping::remove(FileName); std::filebuf fbuf; fbuf.open(FileName, std::ios_base::in | std::ios_base::out | std::ios_base::trunc | std::ios_base::binary); //Set the size fbuf.pubseekoff(FileSize-1, std::ios_base::beg); fbuf.sputc(0); } //Remove on exit struct file_remove { file_remove(const char *FileName) : FileName_(FileName) {} ~file_remove(){ file_mapping::remove(FileName_); } const char *FileName_; } remover(FileName); //Create a file mapping file_mapping m_file(FileName, read_write); //Map the whole file with read-write permissions in this process mapped_region region(m_file, read_write); //Get the address of the mapped region void * addr = region.get_address(); std::size_t size = region.get_size(); //Write all the memory to 1 std::memset(addr, 1, size); //Launch child process std::string s(argv[0]); s += " child "; if(0 != std::system(s.c_str())) return 1; } else{ //Child process executes this { //Open the file mapping and map it as read-only file_mapping m_file(FileName, read_only); mapped_region region(m_file, read_only); //Get the address of the mapped region void * addr = region.get_address(); std::size_t size = region.get_size(); //Check that memory was initialized to 1 const char *mem = static_cast<char*>(addr); for(std::size_t i = 0; i < size; ++i) if(*mem++ != 1) return 1; //Error checking memory } { //Now test it reading the file std::filebuf fbuf; fbuf.open(FileName, std::ios_base::in | std::ios_base::binary); //Read it to memory std::vector<char> vect(FileSize, 0); fbuf.sgetn(&vect[0], std::streamsize(vect.size())); //Check that memory was initialized to 1 const char *mem = static_cast<char*>(&vect[0]); for(std::size_t i = 0; i < FileSize; ++i) if(*mem++ != 1) return 1; //Error checking memory } } return 0; }
As we have seen, both shared_memory_object
and file_mapping
objects
can be used to create mapped_region
objects. A mapped region created from a shared memory object or a file
mapping are the same class and this has many advantages.
One can, for example, mix in STL containers mapped regions from shared memory and memory mapped files. Libraries that only depend on mapped regions can be used to work with shared memory or memory mapped files without recompiling them.
In the example we have seen, the file or shared memory contents are mapped to the address space of the process, but the address was chosen by the operating system.
If several processes map the same file/shared memory, the mapping address will be surely different in each process. Since each process might have used its address space in a different way (allocation of more or less dynamic memory, for example), there is no guarantee that the file/shared memory is going to be mapped in the same address.
If two processes map the same object in different addresses, this invalidates the use of pointers in that memory, since the pointer (which is an absolute address) would only make sense for the process that wrote it. The solution for this is to use offsets (distance) between objects instead of pointers: If two objects are placed in the same shared memory segment by one process, the address of each object will be different in another process but the distance between them (in bytes) will be the same.
So the first advice when mapping shared memory and memory mapped files
is to avoid using raw pointers, unless you know what you are doing. Use
offsets between data or relative pointers to obtain pointer functionality
when an object placed in a mapped region wants to point to an object placed
in the same mapped region. Boost.Interprocess
offers a smart pointer called boost::interprocess::offset_ptr
that can be safely placed in shared memory and that can be used to point
to another object placed in the same shared memory / memory mapped file.
The use of relative pointers is less efficient than using raw pointers, so if a user can succeed mapping the same file or shared memory object in the same address in two processes, using raw pointers can be a good idea.
To map an object in a fixed address, the user can specify that address
in the mapped region
's
constructor:
mapped_region region ( shm //Map shared memory , read_write //Map it as read-write , 0 //Map from offset 0 , 0 //Map until the end , (void*)0x3F000000 //Map it exactly there );
However, the user can't map the region in any address, even if the address is not being used. The offset parameter that marks the start of the mapping region is also limited. These limitations are explained in the next section.
As mentioned, the user can't map the memory mappable object at any address and it can specify the offset of the mappable object that is equivalent to the start of the mapping region to an arbitrary value. Most operating systems limit the mapping address and the offset of the mappable object to a multiple of a value called page size. This is due to the fact that the operating system performs mapping operations over whole pages.
If fixed mapping address is used, offset and address parameters should be multiples of that value. This value is, typically, 4KB or 8KB for 32 bit operating systems.
//These might fail because the offset is not a multiple of the page size //and we are using fixed address mapping mapped_region region1( shm //Map shared memory , read_write //Map it as read-write , 1 //Map from offset 1 , 1 //Map 1 byte , (void*)0x3F000000 //Aligned mapping address ); //These might fail because the address is not a multiple of the page size mapped_region region2( shm //Map shared memory , read_write //Map it as read-write , 0 //Map from offset 0 , 1 //Map 1 byte , (void*)0x3F000001 //Not aligned mapping address );
Since the operating system performs mapping operations over whole pages, specifying a mapping size or offset that are not multiple of the page size will waste more resources than necessary. If the user specifies the following 1 byte mapping:
//Map one byte of the shared memory object. //A whole memory page will be used for this. mapped_region region ( shm //Map shared memory , read_write //Map it as read-write , 0 //Map from offset 0 , 1 //Map 1 byte );
The operating system will reserve a whole page that will not be reused
by any other mapping so we are going to waste (page
size - 1) bytes. If we want to use efficiently operating system
resources, we should create regions whose size is a multiple of page size bytes. If the user specifies the following
two mapped regions for a file with which has 2*page_size
bytes:
//Map the first quarter of the file //This will use a whole page mapped_region region1( shm //Map shared memory , read_write //Map it as read-write , 0 //Map from offset 0 , page_size/2 //Map page_size/2 bytes ); //Map the rest of the file //This will use a 2 pages mapped_region region2( shm //Map shared memory , read_write //Map it as read-write , page_size/2 //Map from offset 0 , 3*page_size/2 //Map the rest of the shared memory );
In this example, a half of the page is wasted in the first mapping and another half is wasted in the second because the offset is not a multiple of the page size. The mapping with the minimum resource usage would be to map whole pages:
//Map the whole first half: uses 1 page mapped_region region1( shm //Map shared memory , read_write //Map it as read-write , 0 //Map from offset 0 , page_size //Map a full page_size ); //Map the second half: uses 1 page mapped_region region2( shm //Map shared memory , read_write //Map it as read-write , page_size //Map from offset 0 , page_size //Map the rest );
How can we obtain the page size? The
mapped_region
class has
a static function that returns that value:
//Obtain the page size of the system std::size_t page_size = mapped_region::get_page_size();
The operating system might also limit the number of mapped memory regions per process or per system.
When two processes create a mapped region of the same mappable object, two processes can communicate writing and reading that memory. A process could construct a C++ object in that memory so that the second process can use it. However, a mapped region shared by multiple processes, can't hold any C++ object, because not every class is ready to be a process-shared object, specially, if the mapped region is mapped in different address in each process.
When placing objects in a mapped region and mapping that region in different address in every process, raw pointers are a problem since they are only valid for the process that placed them there. To solve this, Boost.Interprocess offers a special smart pointer that can be used instead of a raw pointer. So user classes containing raw pointers (or Boost smart pointers, that internally own a raw pointer) can't be safely placed in a process shared mapped region. These pointers must be replaced with offset pointers, and these pointers must point only to objects placed in the same mapped region if you want to use these shared objects from different processes.
Of course, a pointer placed in a mapped region shared between processes should only point to an object of that mapped region. Otherwise, the pointer would point to an address that it's only valid one process and other processes may crash when accessing to that address.
References suffer from the same problem as pointers (mainly because they
are implemented as pointers). However, it is not possible to create a fully
workable smart reference currently in C++ (for example, operator
.()
can't be overloaded). Because
of this, if the user wants to put an object in shared memory, the object
can't have any (smart or not) reference as a member.
References will only work if the mapped region is mapped in the same base address in all processes sharing a memory segment. Like pointers, a reference placed in a mapped region should only point to an object of that mapped region.
The virtual table pointer and the virtual table are in the address space of the process that constructs the object, so if we place a class with a virtual function or virtual base class, the virtual pointer placed in shared memory will be invalid for other processes and they will crash.
This problem is very difficult to solve, since each process needs a different virtual table pointer and the object that contains that pointer is shared across many processes. Even if we map the mapped region in the same address in every process, the virtual table can be in a different address in every process. To enable virtual functions for objects shared between processes, deep compiler changes are needed and virtual functions would suffer a performance hit. That's why Boost.Interprocess does not have any plan to support virtual function and virtual inheritance in mapped regions shared between processes.
Static members of classes are global objects shared by all instances of the class. Because of this, static members are implemented as global variables in processes.
When constructing a class with static members, each process has its own copy of the static member, so updating a static member in one process does not change the value of the static member the another process. So be careful with these classes. Static members are not dangerous if they are just constant variables initialized when the process starts, but they don't change at all (for example, when used like enums) and their value is the same for all processes.