Memory Mapped Files

A memory mapped file is a disk file (or a part of it) that has been associated to a segment of the virtual memory space of a process. It’s useful when you need to manipulate files that are too large to fit in memory. Once mapped you can access the file as if it was entirely uploaded in memory. In reality the OS is paging the appropriate part of the file as you read from the mapped file pointer.

Benefits

  • It’s faster to access memory mapped files, than to use direct read and write operations;
  • You can use the data without copying it;
  • Ease of use.

Drawbacks

  • You are limited by the process space. On 32-bit versions of Windows typically the applications have 2 GB addressable memory, meaning that you can not map files bigger than 2 GB.

C++ Implementation

This article discuses sample implementation of a read-only memory mapping in C++. I wrote it in order to use it as a test target in a future article on unit testing. You can use it as a base for your own implementation.

The requirements for the sample implementation were to:

  • support read-only memory mapping on Windows and Linux;
  • use exceptions as an error handling technique;

I don’t know about you, but when I start a new project I like to use forward engineering (i.e. do the design in UML and produce the code from the diagram). I have used the following UML modellers:

My personal favourite definitely is Enterprise Architect, but unfortunately it’s not free. That’s why I used Bouml for this article.

A class diagram is shown bellow:

UML class diagram

UML class diagram

CMemoryMapping

An abstract base class, that defines the interface and hides implementation details from the clients.

Methods:

virtual void open (const char *pszFileName) = 0 throw (std::ios::failure)

Maps a file into memory.
Parameters:
pszFileName – [in] pointer to null-terminated string, containing the name of the file to be mapped.
Preconditions:
pszFileName is a valid pointer.
Throws:
ios::failure exception in case of an error.

const unsigned char* data () const

Returns a pointer to the mapped area.

size_t size () const

Returns the size of the mapped area.

virtual void close () = 0

Releases a previously mapped file.

static CMemoryMapping * instantiate ()

Creates a proper instance (in the heap) – CWindowsMemoryMapping or CPosixMemoryMapping, depending on the OS.

CWindowsMemoryMapping

An implementation of CMemoryMapping that’s using the Windows API.

It’s using the following Windows API functions:

  • GetFileSize() – gets the size of the input file.
  • CreateFileMapping() – opens a file mapping object for the file.
  • MapViewOfFile() – maps the file mapping object into the address space of the process.
  • UnmapViewOfFile() – unmaps the mapped view from the process’s address space.

CPosixMemoryMapping

An implementation of CMemoryMapping that’s using the POSIX API, intended for POSIX-compliant operating systems.

It’s using the following system functions:

  • stat() – gets the size of the input file.
  • mmap() – maps the file into memory.
  • munmap() – releases the mapping.

How to use it?

#include <memory>
#include <iostream>
#include <memmapping/MemoryMapping.h>

using namespace std;

int main(int argc, char *argv[])
{
    auto_ptr<CMemoryMapping>    aptrFile(CMemoryMapping::instantiate());
    try {
        // Map some file by invoking the open method
        aptrFile->open("some_file");

        // Get its size
        size_t nDataLen = aptrFile->size();

        // Get a pointer to the mapped file
        const unsigned char* pbData =  aptrFile->data();

        // Do something with the data
        // ...

        // Unmap the file by invoking the close method (optional)
        aptrFile->close();
    }
    catch(const ios::failure& exError) {
        cerr << "Unable to map the requested file:" << endl << exError.what() << endl;
    }
}

Where to find the code?

You can download the code, project files for MS Visual Studio and Autotools files from here.

virtual void open (const char *pszFileName)=0 throw (std::ios::failure)
Maps a file into memory.
const unsigned char * data () const
Returns a pointer to the mapped area.
size_t size () const
Returns the size of the mapped area.
virtual void close ()=0
Releases a previously mapped file.
You can leave a response, or trackback from your own site.

2 responses to “Memory Mapped Files”

  1. Vinnythepoo says:

    Hi, thanks for your code i was looking for a simple way to wrap mapped file, i have one question tough, suppose i have a data structture with different data members, such as integers, floats,
    how can i stream the data saved in the file using the char pointer ?
    thanks

  2. Vladislav says:

    Hi,
    If the structure contains only primitive data types you can:

    struct TMyStruct
    {
      // Your data here
    };
    
    TMyStruct myStruct;
    auto_ptr<CMemoryMapping>    aptrFile(CMemoryMapping::instantiate());
    
    // Map some file by invoking the open method
    aptrFile->open("some_file");
    
    // Check the size of the data
    assert(sizeof(myStruct) == aptrFile->size());
    
    memcpy(&myStruct,aptrFile->data(),sizeof(myStruct));
    

    Regards,
    Vladislav

Leave a Reply