In the previous post I presented my screencasting application prototype for Sailfish OS. I developed it really fast (in 4 hours) and published it early to get some feedback. The main issue with the app was that it blocked occasionally the whole GUI thread when it was running. This could be observed as a very laggy UI.

The blocking call was Lipstick's screenshot method which I used to grab the "frames". In addition to that, whole grabbing process was pretty slow due to the fact that the image was first written to a file after the dbus call was finished and after that reopened in the app again. I clearly needed a better way to do the frame grabbing. Luckily @LiKimmo pointed me to Mer vncserver sources and I found out about the Linux framebuffer device.

It's always very cool to find useful things from Linux (by Linux I mean the kernel, not the whole GNU/Linux operating system/some distribution) and try out how they work. The framebuffer device provides an abstract interface for the graphics hardware. This means that the developer doesn't need to know anything about the low level hardware stuff to draw something on the screen or to grab the screen contents.

This is how framebuffer device can be used:

  1. Initialize the framebuffer device by opening it
  2. Read two different structures (fb_var_screeninfo and fb_fix_screeninfo) which contain information about the device
  3. Map the device into memory (mmap) according to the size info provided by those structures

After those steps the memory mapped region can be accessed with basic pointer arithmetic. This is very fast, since the frame data is located in RAM region which is now shared and accessible by the app thanks to mmap. However, there are some things to keep in mind while doing this: the size of the screen, possible offsets and size/memory layout of a single pixel.

In SailCast I need to read the raw frame data and save it to a QImage. This occurs in a nested for-loop. The outer loop iterates over image y coordinates and the inner loop over x coordinates. There has to be some calculation inside the loop so that the right pixels are accessed:

QImage img(info.scrinfo.xres, info.scrinfo.yres, QImage::Format_RGBA8888);
int line = info.fix_scrinfo.line_length / 4;

for (unsigned int y = 0; y < info.scrinfo.yres; ++y)
    QRgb *rowData = (QRgb*)img.scanLine(y);
    for (unsigned int x = 0; x < info.scrinfo.xres; ++x)
        rowData[x] = *((unsigned int *)info.fbmmap + 
                       ((x + info.scrinfo.xoffset) + 
                        (y + info.scrinfo.yoffset) * line));

The information of fb_var_screeninfo and fb_fix_screeninfo structures is used to get right offset's for a single pixel. The x and y offsets shift the pixels because the actual frame is inside a larger buffer. The line_length value tells how many bytes there's in a single line in the image. The value of line_length is divided by 4 to get right offset in the memory for single 32-bit pixel (this could be done smarter, because the code currently assumes that single pixel is 32-bit or in other words 4 bytes long). The code iterates over 4 byte chunks because it's then easy to write the 32-bit RGBA value straight to the QImage structure.

This whole nested for loop takes about 12 ms to execute on average. If this loop would be the only thing to do and the operating system scheduler would give all free execution time for the app, SailCast could grab whopping 80 frames per second. Unfortunately that's not the case.

The frame grab loop runs inside a while loop. After the for loop is finished, the image is converted to jpeg format and optionally also rotated according to device orientation. This operation is "slow", because it takes about 30 ms when the jpeg compression is at maximum. If no compression is used, it takes 50 ms. This means that grabbing single frame takes about 40-60 ms which in turn means that the FPS can be something between 16 - 25.

In reality there's also other limiting factors. For example, the frame grab thread is not running all the time because the OS can't give all the execution time for single process. It's also important to remember that Jolla phone is hardware-wise ancient technology so it's actually pretty impressive to get this good performance.

Latest release of SailCast can be grabbed from GitHub. Sample video recorded with VLC: