Most digital cameras can not only store the images as JPEGs, but also ‘raw’, in a format that normally has to be developed before it can be viewed. In fact, the camera does the same thing before saving an image as a JPEG, even those that can’t store the raw itself like smartphones. Almost all digital sensors in cameras from phones up to DSLRs only have one sensor in them which can only detect the brightness of the incoming light. To get a coloured image they have per-pixel colour filters in front of the sensor to capture the red, green or blue light per pixel. The arrangement of these filters can vary, but most resemble the so called Bayer pattern invented by Bryce Bayer in 1976.

Visualising the Bayer pattern

The three colours are arranged in a 2×2 pattern where green is used twice as often as the other colours as the human eye is most sensitive to this colour. The arrangement is as depicted below:

RGGB Bayer Pattern

Note that the pattern doesn’t have to start with red but at least in the cameras for which I could check the raw data it did. Instead of mixing one RGGB block to get one pixel, the missing colours are reconstructed to get a higher resolution out of the sensor. So the sensor above would result in 12 by 10 RGB pixels. So, yes, your 10 megapixels camera only has 5 million green, 2.5 million red and 2.5 million blue sensor elements and 66% of the resulting image are interpolated and were not actually captured!

So how does the data that comes out of the sensor look like? I made the photo below (as raw) and used libraw to decode it. Luckily it comes with a sample called ‘unprocessed_raw’ which can dump the sensor data and which served as a starting point for me.

Converted in Lightroom, no adjustments were made.

What comes out of the ’unprocessed_raw’ sample from libraw just looks black but this is because the sensor data gets saved in a 16 bit grey image but the camera might not have a 16 bit analog to digital converter. So the pixels had to be shifted in this case by 2 places as the camera was only set to 12 bit. Below you can see a small part of the image developed in Adobe Lightroom 4 and the same part of the original sensor data. On the right I coloured the sensor data with the corresponding colours they represent (the colours of the filter for that pixel).

Resized 200%. You might have to click on the image to see the pattern better in full resolution.

The image is quite dark and green is very dominant. To get the true colours out of the raw we would have to rescale the colour channels and find a proper white balance.

Getting the missing colours back

But first let’s see how the missing 66% of the data can be restored. The simplest way to do it is taking the value from the nearest sample of the correct colour: e.g. for a pixel that has already a red value take the blue from the pixel one pixel to the right and one to the left and the green from the pixel below the current pixel. This nearest neighbour filtering leads to the coloured result below (left). Now that the image is very roughly restored, the colours can get adjusted. The raw file stores additional information for that (e.g. the white balance from the camera) but here I just tried to roughly match the output from Lightroom (middle).

Left to right: Nearest neighbour interpolation without and with colour correction, bilinear filtering with colour correction (resized 200%).

A slightly better way to reconstruct the missing colours is to average the surrounding samples instead of just picking one: A simple bilinear filtering. This is shown on the right (also colour adjusted). As you can see, this already improves the highlight on the eye dramatically.

If the same colour adjustments (in fact just a curve in Photoshop) are applied to the coloured Bayer pattern we can visualise it even better:

Left: from Lightroom, right the coloured Bayer pattern. Resized 300%.

We can clearly see the dominance of the red coloured pixels on the rose, the green ones on the leaves and the mixture of all colours on the wooden table.

Results from the pros

Even the bilinear reconstruction is not ideal in any way. There are lots of algorithms to better handle Bayer demosaicing (also written demosaicking if you want to google the details) and as I didn’t want to implement all of them I switched to already available software: The already mentioned Lightroom and RAWTherapee which has the nice feature of letting you choose the algorithm for Bayer demosaicing.

Left to right: Nearest neighbour, bilinear, RAWTherapee ‘fast’ & ‘amaze’, Lightroom. Resized 200%, click the image to see it in full resolution.

Here we can see the leave structure of the rose slightly in the images reconstructed from Lightroom and RAWTherapee with different algorithms. My own quick hack did not reconstruct that but only created a bit of noise. As you can see now, the rose is actually an artificial flower, so the structure is not an artefact. No additional sharpening was used in any image. Below are the same parts of the image with enhanced contrast and sharpened:

Increased contrast and sharpened, click on the image to see it in full resolution.

The visibility of the structure of the fabric is highly dependent of the algorithms used. All images so far were captured with a camera that has an optical low-pass filter which is intended to reduce aliasing that can be introduced by demosaicing the Bayer filter. However, it also reduces the sharpness of the image. The images below were taken with a camera that has 50% more pixel sensors but also has no optical filter anymore. But again, the quality differences were mostly a result of the demosaicing technique chosen.

Same scene, different sensor: 50% higher resolution and no low-pass filter.

The only moiré like pattern I found were in the camera made with the low-pass filter (but with a low quality demosaicing algorithm). So I assume that moiré can be avoided by better algorithms at least as well as with optical filters. ‘Image Demosaicing: A Systematic Survey‘ by Li et al. also shows the great influence of the demosaicing technique to moiré.

Nearest neighbour, bilinear, ‘fast’ and ‘amaze’ from RAWTherapee and Lightroom, 400%. Click on the image to see it in full resolution, otherwise some artefacts might not be visible!

This last example above shows that fine structures can be challenging for demosaicing: The algorithms used by RAWTherapee seem to try to add some sharpening that goes wrong here as can be seen by the few dark and bright pixels. The bilinear reconstruction can hardly preserve the structure from the fabric.

Recap

As it turns out it’s actually not too hard to get access to the raw sensor data of a DSLR and visualise it. A basic reconstruction of the full colour data is hacked together quite quickly but by far no match to the already available implementations (no surprise here). If you own a DSLR I would recommend to shoot in RAW mode as you can see that there are wide differences even in the reconstruction quality of the Bayer pattern (in addition to a higher bit rate and the avoidance of JPEG artefacts!). Who knows what is implemented inside of the camera and what awesome new algorithms will be invented in the future that can improve your already shot images even more?

Other ways to capture an image

While most cameras use the Bayer pattern, this isn’t the only way to capture an image of course:

• You can arrange the colour filter array in a different order.
• You can add other colour filters than just red, green and blue, e.g. add yellow, white or cyan.
• You can capture red, green and blue in one sensor (Foveon).
• You can use three separate sensors – if you can manage to align them precisely, a mechanical problem that gets more and more challenging the higher the resolution gets.
• You can even capture the colours one after another by placing removable colour filters in front of a sensors – sounds impractical? This is used by NASA for some spacecrafts and landers…

To recap, my currently known bugs and issues are these:

• Wrong values from uniform structs (bug id #10518401) - Fixed
• Wrong names for uniforms in uniform blocks (bug id #11335828) - Still exists (this bug was reported in april 2012)
• Rendering to an FBO without fragment data location 0 is not possible (bug id #11826727) – Still exists (this bug was reported in july 2012)
• Uniform block layout qualifier ‘packed’ results in GLSL compile errors (bug id #11884110) – Fixed
• Default layouts of Uniform Blocks can’t be changed at global scope (bug id #11884641) - Fixed
• GLSL noperspective varying interpolation doesn’t work on Intel HD4000 (bug id #12728578) – Fixed
• If a scissor test is active with a larger region than the glViewport and the later is smaller than the framebuffer, the rasterization is not clipped at the viewport. (at least on HD4000). Bugs #12949605 & #13287825 – Still exists
• After a conditional rendering which was not performed (due to no passed samples) subsequent glClear calls will not get executed until the next draw call was issued. (at least on HD4000). Bug #12949224 – Still exists
• Highest OpenGL version available is 3.2 (not a bug, but filed as bugs as (at least) bug id #8225498 and #8233475). - no changes here

In addition to that, MacOS X 10.8.3 adds a few new extensions, for example explicit attribute locations, a feature that was even earlier exposed with a hack.

So, there is still much to do for Apples driver staff…

Why we use FPS

There are two ways to measure how fast an application can render the virtual world: by giving the time it takes to render it or by counting the number of frames that get rendered per second. It seems strange that the later way is more common but it makes sense after you have learned that the brain will interpret a sequence of images as a smooth motion if shown fast enough. As cinema shows us 24 images per second and TV 25 (PAL) or 29.97 (NTSC) any game that renders more frames per second is fast enough, right? Well this isn’t exactly right and I will come to that later but it demonstrates why FPS is a nice way to measure the rendering speed of a whole system (e.g. a game) for the user: He/she has only to compare the number given by a fixed value he learned to be the magic value where a sequence of images begins to feel like a fluid motion.

But instead of memorizing that 30 or 60 FPS is the magic border, you could also memorize 33 or 16 milliseconds as the magic rendering time. As I will show next, talking about milliseconds instead of FPS has some advantages.

Talking about subsystems

FPS doesn’t work anymore then we start talking about parts of the system instead of the whole renderer. Let me give you an example: Let’s say you read this paper about this cool new SSAO algorithm and it claims that with a simple scene you can achieve 200 FPS. Our game targets full 60 FPS, so it sounds like there is plenty of performance left. Now let’s look at the rendering times: the hypothetical SSAO demo took 5 msec to render each frame and we have to render a full game in 16.6 msec. That’s 30% of our budget for one post-processing effect! But maybe it’s not that bad: those 5 msec are needed to render the post-processing effect and a simple scene for which we don’t know the cost of the overhead! If this information would have been given in FPS it would say something like: ‘FPS without post-processing: 2000 FPS, with post-processing: 200 FPS’. But then you would think ‘One order of magnitude slower? No way this would work in my engine!’. But this reveals that the overhead of a simple scene was 0.5 msec and the effect is 4.5 msec. This way it’s much more clear if the effect fits into your budget or not. If the demonstration would use a complex scene the numbers could look like this: Without SSAO: 39.2 FPS, with SSAO: 33.3 FPS. ‘Just 17% overhead, great! My engine is running at 72 FPS, my goal is 60 FPS, so I have enough performance to spare!’. Going to msec reveals that the test application used up 25.5 msec and the post-process (again) 4.5 msec. Your app renders one frame in 13.9 msec and adding 4.5 will give you 18.3 – too bad you are over budget (~55 FPS).

It gets worse when we talk about adding multiple effects: Effect A renders at 200FPS, effect B at 100FPS and effect C at 500FPS so all three combined will render at…? You have to convert to msec/frame to calculate that anyway, so why aren’t we talking about msec per effect to begin with?

The bottom line is this: It’s much more intuitive to handle timings in what they are – the time to calculate something.

Varying rendering speed

Let’s say your engine is too slow, you play around with various settings and this is what you found out: With all effects active your game runs at just 50 FPS, without shadow-map creation its 66.6 FPS so you decide to recreate the SM only every third frame (you’re ok with the resulting artifacts as long as the user has smooth 60 FPS). Even tho you have 60 FPS now, it doesn’t feel smooth at all! Two frames take 15 msec to render and the third one 20! I once worked with a rendering system that didn’t feel as responsive as we expected from constantly (slightly) above 60 FPS. Further investigation revealed that some calculations were only triggered every few frames (one of the shadow maps every second, other systems in other intervals). I plotted the timings per frame and they varied a lot. The distribution of workload over multiple frames was quite bad balanced.

How would you even write down such varying rendering times in FPS? ‘We have an average of 60FPS but N% of the frames have 300FPS while M% have 20-30FPS…’?

Latency

Latency is a big topic: How long does it take your system from the moment you press a button until the resulting action gets displayed on the screen? There are a lot of factors like input latency, rendering time, latency of the driver (by buffering commands for whole frames), additional buffering in your TFT (TVs are even worse as they can perform some post-processing, e.g. 25 to 100Hz upscaling, that’s why they often have a ‘gaming mode’ which switches this of) etc. All of these timings can be given in milliseconds and so should the rendering time! In the context of reducing the whole system latency rendering speeds below 16 msec (and thus above 60FPS) can make sense even if the display only is capable of displaying 60 images per second.

If your game has a high latency from input to the screen or your rendering time varies a lot, your average FPS count will tell me nothing about how smooth the experience is.

tl;dr

Don’t measure your graphics performance in FPS but in milliseconds per frame/effect.

But it doesn’t take long until the number of keys it so large that it becomes confusing. There is also no direct feedback of the currently set values (and a lot of printf doesn’t scale so good as well). So the next step is adding a simple UI. Some use QT for this as it’s a nice GUI toolkit and works cross platform (so did I for many projects). QT is great when your prototype turns into a full application (unless you are building a game) but may require too much glue code for simple prototypes.

More limited but also much simpler to include into existing projects and very simple to set up for simple tasks is AntTweakBar. After writing a couple of lines of glue code, exposing one more variable for testing is a matter of one line of code!

In case you want some haptic feedback or adding a UI is not an option (maybe you don’t want to occlude anything from your game and need the keyboard and mouse for the game control), adding a MIDI device might be an option. rtMidi is a small cross platform library that helps getting the MIDI commands from your device. A simple MIDI controller with nine analog sliders and knobs will set you back 50€. Each control will give you a 7bit value of the current position.

One downside of this setup is the fact that the device only sends the values of the controls when they change, this means that the initial state when starting your app is not reported. So you might want to reset all physical sliders or move each one slightly after your application started to send the current control value to your app.

The video below shows how a small MIDI board is used to adjust the lighting parameters of our beloved Stanford bunny:

Here’s an example:

GLuint query;
glGenQueries(1, &query);
glBeginQuery(GL_SAMPLES_PASSED, query);
// draw bounding box
glEndQuery(GL_SAMPLES_PASSED);
 
glBeginConditionalRender( query, GL_QUERY_WAIT );
// draw actual object: if no sample of the draw call above was
// drawn, all glClear and draw commands will get ignored
glEndConditionalRender();
 
// now everything should get drawn again!
glClear( GL_COLOR_BUFFER_BIT ); // might also get ignored -> BUG
glDrawArrays(...);
glClear( ... ); // now it works again

To be clear, glClear and glClearBuffer commands between glBeginConditionalRender and glEndConditionalRender should get ignored if the query failed, but after glEndConditionalRender everything should be back to normal. In fact at least on the stated configuration (10.8, Intel HD4000) it isn’t, the state of ignoring the glClear calls is stuck until the next draw call gets issued (which works as intended btw.). On 10.7 with a GeForce 9600M it works as intended, so I’m not sure whether this is a bug of the Intel drivers or of 10.8.

The bug is listed under bug id #12949224 and joins the list of other open issues.

OpenGL GLSL Syntax Highlighting for TextWrangler

To install it, download the glsl430.plist (unzip it) and drop it into your ~/Library/Application Support/TextWrangler/Language Modules folder. After restarting TextWrangler, all .glsl .fsh .vsh .gsh .tcsh .tesh .csh .frag .vert and .geo files should get the correct highlighting. In case you prefer other extensions, you can add or remove extensions from the beginning of the plist (just edit it in a text editor) or change the settings in TextWrangler under Preferences->Languages->Custom Extension Mappings.

So I was sick of waiting for better support and wrote a solution: The GLSL syntax checker in QTCreator is build-in, so it has to be deactivated first, then a syntax definition XML file in the format also used for Kate (and probably other KDE apps) can be used to define proper and modern GLSL:

1. Go to Preferences->Environment->MIME-Types in QTCreator and look for all types associated with GLSL (application/x-glsl). The handler can’t be changed, but the file endings – just remove them. Don’t forget the MIME-Types text/x-glsl-es-frag, text/x-glsl-es-geometry, text/x-glsl-es-vert, text/x-glsl-frag and text/x-glsl-vert!
2. Add a new language syntax definition, look at Preferences->Text Editor->Generic Highlighter, you will see a location where QTCreator expects the XML file (~/.config/Nokia/qtcreator/generic-highlighter on OS X and Linux). Place the this glsl.xml file there.
3. Restart QTCreator.

The definition has the following features:

• Support for desktop OpenGL up to 4.3 (with all keywords, build-in functions, constants and variables), as well as OpenGL ES 2.0 and OpenGL ES 3.0 (and WebGL as it uses ES 2 shaders).
• Detects the shader version per file based on the #version definition!
• Unsupported keywords, build-in variables and functions are marked as such based on the detected GLSL version (see gif below).
• Marks the ‘special parameters’ inside of layout() declarations based on the GLSL version (those are not keywords according to the specs so they are not marked outside of a layout() declaration).

See this comparison of the same (non functional) dummy code to get an impression:

Comparing the build-in syntax highlighter with this alternative. Note how keywords get crossed out in case they are not supported by the #version declaration.

This highlighting does not perform a full syntax check and assumes core profile code on GLSL 1.30 and later. The detected file endings are: *.glsl; *.vsh; *.vert; *.tcsh; *.tcs; *.tesh; *.tes; *.gsh; *.geo; *.geom; *.fsh; *.frag; *.csh; *.cs, but this list can easily get changed in line 4 of the XML. The highlighting style can get adjusted beginning at line 1884 (always restart QTCreator after changing the XML).

The current limitations are:

• The shader type itself (Compute Shader, Geometry Shader etc.) does not get detected – it would be possible to create six different XML files with slightly different rules to only mark keywords useful/supported per shader stage (in case the file ending is sufficient to make this distinction).
• Version 130 and later assume core and mark compatibility build-ins as unsupported.
• In #version 300 es for OpenGL ES 3, only one space between the ’300′ and the ‘es’ are allowed.

A description of the XML format can be found here.

The glsl.xml version 4.30 (the version number mimics the latest supported OpenGL version).

Update 12/1/12: In case you prefer TextWrangler/BBEdit, there’s a (simpler) solution for that as well.

Update 12/17/12: A few GLSL 4.20 build-in functions were defined as 4.30 functions (e.g. imageStore).

noperspective varying interpolation bug

Note that in the left screen shot the image looks similar to perspective correct interpolated texture coordinates as the triangles are quite small in screen space but the are in fact interpolated non perspectively.

As @marcofatto noted, the bug does not occur on his NVidia on 10.8, so it seems that the Intel drivers are causing the problems.

The bug has been filed as #12728578 and joins my list of unfixed OpenGL bugs on MacOS 10.8.

AR gaming on a segway.

During the game you had to interact with historical figures and solve quests in a Monkey Island kind of style, but instead of klicking where you want to go you had to go there as the setting was the real city centre of Aachen. As this was intended for tourists to discover the city, we added a Segway for faster travelling – and additional fun ;-)

Of course we build authoring tools, did a user study and tested the image based localisation method from a technical and usability standpoint (you can find the details in the paper). The results summed up: You should look into image based localisation and computer vision if you want to build immersive AR apps, just GPS&compass will never be good enough. The users loved the point&click setting in the real world (and driving a Segway) and even our inexperienced, non-tech-savvy users had no problems with the localisation and gaming metaphors.

Our authoring tool (here with a dummy SanFrancisco setting, the described game was done in Aachen, Germany).

The server for finding our your position wasn’t done in this project but based on the work of our coworkers. On Youtube you can find videos of an older techdemo:

You can find the paper titled “A Framework for Vision-based Mobile AR Applications“ on the MobiVis website for further information.

I will mostly focus on how the programmable parts of the GPUs are designed and less on the remaining fixed-function parts. So let’s take a look at what is needed to build a GPU for current graphics APIs:

We need a processor that can run arbitrary code: our shaders or compute kernels. We will focus on float-point performance as these operations are needed often in graphics. Hardware support for more exotic functions as tan, sin, pow, sqrt etc are also needed. We will end up with some logic to decode the instruction to execute, some registers, a float-point ALU (working with ints is also needed) and a cache. If we copy the design of a multi-core CPU, we would copy&paste the whole thing until the DIE space is filled up. Ok, CPUs are not that simple, but let’s stick with this simplification for a moment.

In reality modern CPU cores are quite complex: they speed up the program execution by reordering independent operations on the fly (out-of-order execution), they try to guess the outcome of a branch before it was evaluated to keep the deep pipelines filled (branch prediction) and implement parallel concepts in a single core: SIMD and simultaneous multithreading. A GPU core is simpler when it comes to execution logic, they are in-order and try less to be clever. While this means that CPUs can handle a single thread better, the GPU design saves a lot of space on the DIE which can be used to add more cores from which parallel applications benefit more. But GPUs also implement the last two tricks (SIMD, simultaneous multithreading), even in more extreme ways.

Why do it once if you can do it twice?

Lets look at them individually: SIMD stands for single instruction, multiple data. One instruction is performed on a vector of operands at a time, that’s why we talk about vector processors. This idea is not new, there have been vector supercomputers in the 70th (CDC Star-100 or Texas Instruments ASC). The trick is, that an instruction only has to be decoded once and multiple ALUs can than perform the work on multiple data elements at once. This design was used for graphics in 1990 with the Intel i750, this chip could store two 8 bit values in one 16 bit register and perform the same operation on both in parallel.

Around the same time, 1989 Intel also introduced a CPU with a new instruction set, the Intel i860 with a VLIW architecture which could also run SIMD like instructions. This chip actually ended up on a ‘graphics card’, the RealityEngine, in 1993 as what we would now call a vertex shader.

Even though these chips weren’t very successful, the technique was added in 1996 into the Pentium MMX which added vector integer operations on 64 bit wide registers (each could hold e.g. 8 8-bit values or 4 16-bit values). AMD introduced 3DNow in 1998 adding float point operations in a similar way. As back then graphics cards were mostly rasterizers and the geometric transformations were still performed by the driver on the CPU, this vector float operations had the potential to speed up 3D graphics. Intel didn’t adapt 3DNow but instead extended its own set of vector operations with float operations, gave it a larger register set (128-bit) and named this SSE in 1999, first implemented on the Pentium 3. The latest iteration on Ivy Bridge has 256-bit wide registers which can operate on 8 float values or 4 double in parallel (it’s also renamed to AVX).

When we have a loop doing the same stuff with a lot of data points and each loop iteration is independent from each other, SIMD can help. Think about blending two images together, each new pixel is the weighed sum of the two input pixels from the two input images. The operations for all pixels are exactly the same, just the data is different. If however inside this loop there is a branch, things get tricky. In case all simultaneously evaluated data points take the same branch, everything is fine. But if even one point branches differently, we have to evaluate two pathes sequentially. This can be done by evaluating both branches and masking out the data points that took the other branch. Of course, the wider our SIMD registers, the higher the performance (given that the CPU can get the data fast enough from memory) but the higher the risk a branch will screw up the parallelism.

Transforming vertices and shading pixels are tasks that do (mostly) the same stuff on different data points so this can get accelerated by SIMD. When you write a shader or a compute kernel, you write the inner part of a loop (over all elements, fragments, vertices…). The compiler can now just merge N shader calls to one stream of SIMD instructions. On a modern Pentium this would mean shading 8 vertices in one go, on a current NVidia it means shading one ‘warp’ of 32 vertices (called ‘wavefront’ over at AMD). Yup, the SIMD width is 32 floats or 1024-bit. This means, that shading 32 fragments is a fast as shading just one! But it also means, that shading just one fragment is as slow as shading 32…

If you have to wait, do something else!

Another technique that we also know from CPUs is simultaneous multithreading, or ‘Hyper-Threading’ as Intel’s marketing guys call it. The basic idea is this: CPUs are fast, really fast, but memory is slow and can’t keep up. This is why data is cached in on-die caches and even quicker registers! But every once in a while the CPU can’t calculate as it has to wait for data. At other times it can calculate but the memory bus is idle. So why not switch from one thread that waits for data to another one that is ready? Well, for once the decision would need slow OS intervention and the switch would need to swap in the data of the other thread … from memory. To make this idea possible, the decision has to be made by the CPU alone and the data has to reside on the CPU the whole time. For this the registers have to be (at least) duplicated and some additional logic has to be added. Note that the ALUs are not doubled so only one thread is actually calculating stuff at any given time but the ALUs can be kept busy even if one thread stalls as it has to wait for memory.

Some chips can hold more than two threads on the CPU like the SPARC T4: 8 threads per core. On a GPU this can be even more extreme: instead of a fixed set of registers they can have large register files and the amount of threads that these can hold is defined by the sum of registers needed by all threads. This number can go in the hundreds.

But there is a difference between simultaneous multithreading of CPUs and (at least some) GPUs: The threads on a CPU are completely independent of each other, e.g. they can belong to different applications. On some GPUs they have to belong to the same shader or kernel. While this looks like a major drawback, it’s actually not that bad: If you think of a 16k triangle mesh, one SIMD thread will evaluate (to stick with the earlier number) 32 vertex shader “threads” simultaneously, so in total 512 such threads have to be started (for 16k vertices) – enough opportunity for lots of simultaneous multi-threading. This limitation can also be exploited in a couple of ways: constants can be placed one time in the register file and don’t have to be copied for each thread. Uniforms can be cached for all threads as those are read-only anyway. Memory and texture access pattern will most likely be similar and this way make best use of the shared (texture) cache. The number of threads that can run in parallel are easy to determine as each thread uses the same number of registers. The threads can even share one instruction cache.

We have the ingredients, lets make a GPU!

Let’s put everything together. At the lowest level, we have a very wide SIMD processor, if it can operate on N floats in parallel, NVidia, AMD and co. would count that as N ‘cores’. Actually, a NVidia Fermi Kepler chip (e.g. in a GeForce 680) has processors that can issue six 32-float wide SIMD instructions at once in a Streaming Multiprocessor (SMX), counting for 192 ‘cores’ at the marketing department. These processors can often run less of the more complex operations as trigonometric functions in parallel as simple float point operations, those have to be done more sequentially in a ‘special function ALU’ (called ‘special function unit’ SFU at NVidia). This is done to save transistors on less used operations and add more ALUs and wider SIMD for more common operations.

Each core has a (texture) cache, a register file and runs multiple threads in parallel with simultaneous multithreading. Fixed-function blocks can also be added here, e.g. texture units or even the fixed-functions of the geometry pipeline (vertex fetch, tessellator, viewport transform etc.) as it’s done on recent NVidia hardware.

Multiple of these processors are then placed on the GPU DIE, together with the remaining parts of the fixed-function pipeline, more cache and the memory controllers. Of course the whole thing also gets a command processor that distributes the workload coming from the host to the various processors. This way it’s easy to build different versions of the chip for the low-end to high-end market: just choose a different number of SIMD processors (and/or you can also change some number of those processors as well, e.g. the SIMD width).

Crunch the numbers on the number crunchers:

We already started to use the NVidia Kepler architecture as an example, so lets complete this: What I called SIMD processor is called SMX or streaming multiprocessor here. It has in fact 192 float point ALUs (‘CUDA cores’) and 32 special function units. Threads are run in batches of 32, called a ‘warp’. The register file holds 64k 32bit values (e.g. float or ints) for all threads. It also has a texture cache, uniform cache, instruction cache and 64kb that can be used as an L1 cache or shared memory (in case it’s used for compute). Also 16 texture units to get in more data. All fixed-functions up to the viewport transformation is handled here as well. To get to the rasterization, we have to look one level up: Two SMX and one shared ‘Raster Engine’ form one ‘Graphics Processing Cluster’ and up to four of those are placed one one DIE together with 512kb L2 cache, memory controller and a ‘GigaThread Engine’ to keep the SMX busy. To sum it up, we get 8 SMX with each 6 32-wide SIMD lanes for a total of 1536 ‘cores’. Each SMX can run a different shader or kernel, so no more as 8 different programs are running at any time but from those thousands of threads doing the same work on different data points.

At AMD, the latest architecture has processors called ‘Graphics Core Next’ (GCN Compute Units): Each has just four SIMD lanes with are just 16 floats wide. Simultaneous multithreading works a bit different here: A GCN is limited to 10 threads per SIMD lane and thus 40 threads per processor (vs. ‘whatever fits in the register file’ on NVidia hardware), but these don’t have to belong to the same shader/kernel making this design more flexible (and in fact a little bit more CPU-like). Each processor has 64kb local memory and 16kb L1 cache. The fixed-function parts for geometry and rasterization are together with a L2 cache ‘globally’ on the DIE. Where or GeForce example has only 8 SMX processors, AMD puts in 32 GCN processors. So here the total is 32 GCNs with 4 16-wide SIMD lanes for a total of 2048 cores.

Just for fun let’s compare this to a (4 core) Core i7 Ivy Bridge: What’s called ‘core’ on a CPU is what I called ‘processor’ on the GPU examples above as individual SIMD ALUs are called ‘cores’ there. So if an i7 would be a GPU, we would count: 4 processors with two threads each, each processor has a 8 float wide SIMD unit. This gives a total of 32 ‘cores’ and a maximum of 64 ‘threads’ in parallel. Still not much, even if taking the higher clock rate into account.

If that’s not enough performance for your graphics needs, you can always put two GPU DIEs on one graphics card and/or install multiple cards to work together…