| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| isp:imagesignalprocessing [2018/05/30 01:33] – [Exposure] Igor Yefmov | isp:imagesignalprocessing [2018/06/04 12:50] (current) – removed Igor Yefmov |
|---|
| ======= Image processing ======= | |
| |
| ====== White Balance ====== | |
| |
| ===== What is white color temperature ===== | |
| A "white" color has different chromatic characteristics, depending on its "temperature". That temperature is traditionally expressed in Kelvin and corresponds to light, emitted by an ideal black body radiator, when heated to that temperature. Think of it as the color of a burning stick of wood at a camp site or a "red-hot iron" that looks more yellow when it actually melts in a furnace. | |
| |
| {{:isp:color-temperature-1500x600.jpg?600|}} | |
| |
| When a scene is illuminated by "white" color there's usually color correction needed to bring the "true" colors out. The "true" in this sense is an approximation of how our brains perceive colors during daylight, which is around ''6000''°K. | |
| |
| For our purposes, we are using individual color channel **gains** to compensate for a given temperature. Lower temperature "white light" needs a lot of blue added to it and very little red and as the temperature climbs up, the amount of added red grows while the added blue goes down. | |
| ===== Color correction for white temperature ===== | |
| For the calibration purposes we have acquired a Philips "Hue White and Color Ambiance A19 LED Starter Kit" that allowed us to test various illumination scenarios for a range of color temperatures. For a given white color temperature setting we have dialed the red and blue gains to make the scene "white" (leaving the green gain at its constant value of ''1024''). | |
| |
| Corrections to the red channel were way more noticeable than those to the blue one so we approximated the blue gains' graph with a single line, described by the formula ''B = 4205.4 - T*0.4087''. | |
| |
| Red channel gains are approximated by two line segments, bordering the ''3500''°K mark: | |
| |
| below ''3500''°K: ''R = 467.6 + T * 0.166'' \\ | |
| above ''3500''°K: ''R = 82 + T * 0.277'' | |
| |
| {{:isp:wb_pig-2.jpg?600|}} | |
| ==== Experimental data and verification ==== | |
| Here is the table with the results of that testing and corresponding calculated values based on the above formulae: | |
| | ::: ^ Manually set ^^ Calculated ^^ | |
| ^ Color, K ^ Red ^ Blue ^ Red ^ Blue ^ | |
| | ''2000'' | ''800'' | ''3388'' | ''800'' | ''3388'' | | |
| | ''2296'' | ''840'' | ''3285'' | ''849'' | ''3267'' | | |
| | ''2592'' | ''913'' | ''3196'' | ''898'' | ''3146'' | | |
| | ''2888'' | ''959'' | ''3078'' | ''948'' | ''3025'' | | |
| | ''3184'' | ''1016'' | ''3005'' | ''997'' | ''2904'' | | |
| | ''3480'' | ''1046'' | ''2858'' | ''1046'' | ''2783'' | | |
| | ''3776'' | ''1119'' | ''2710'' | ''1128'' | ''2662'' | | |
| | ''4072'' | ''1195'' | ''2519'' | ''1210'' | ''2541'' | | |
| | ''4368'' | ''1296'' | ''2460'' | ''1292'' | ''2420'' | | |
| | ''4664'' | ''1370'' | ''2298'' | ''1374'' | ''2299'' | | |
| | ''4961'' | ''1458'' | ''2121'' | ''1456'' | ''2178'' | | |
| | ''5257'' | ''1561'' | ''2033'' | ''1538'' | ''2057'' | | |
| | ''5553'' | ''1620'' | ''1895'' | ''1620'' | ''1936'' | | |
| | ''5849'' | ''1750'' | ''1782'' | ''1702'' | ''1815'' | | |
| | ''6145'' | ''1738'' | ''1694'' | ''1784'' | ''1694'' | | |
| | ''6500'' | ''1738'' | ''1694'' | ''1882'' | ''1549'' | | |
| |
| ===== Practical example ===== | |
| What does all that mean in practice? Suppose you have set up your scene with "white" lamps that are marked as ''4100''°K. Based on the above information you'd know that to get the best "natural" colors out of that scene means setting your red channel gain to ''1217'' and blue gain to ''2529'' while leaving the green gain at its default ''1024'' value. | |
| |
| ===== Automatic white balancing ===== | |
| Quite often end users don't have the patience or the expertise (or both) to properly set up the white balance of the captured scene and prefer to rely on camera "just knowing what to do". | |
| |
| [[https://en.wikipedia.org/wiki/Color_normalization|Many algorithms]] are available today to deal with color normalization, some very sophisticated (and computationally expensive) and others, way less computationally expensive, but so simple that they are easily thrown off by things like a huge green screen background or a bright super-yellow T-shirt. | |
| |
| Below is one of those "simplistic" approaches with a little twist, that makes it an excellent option for real-time image processing on even the most basic of the devices. | |
| ==== Overall idea ==== | |
| The general idea is based on an assumption that if one takes a look around and measures the color of every pixel, then the grand total sum of all those values should be close enough to a pure grey color. This idea has a name and it is the [[https://en.wikipedia.org/wiki/Color_normalization#Grey_world|"grey world"]]. | |
| |
| Unfortunately, the world around is only grey if we are in a more-or-less neutral environment. Such is not the case with large patches of high-contrast colors (like [[https://en.wikipedia.org/wiki/Chroma_key|"green screens"]]) or even a relatively small patches of "highly toxic" colors, like bright yellow. These conditions affect the scene's average color sufficiently enough to throw it off and produce horrible color artifacts instead of normalizing the colors to their perceived values. | |
| |
| To combat the problem of "toxic colors", a simple twist is introduced: ignore all the pixels that are too saturated. Now, what exactly is "too saturated" is up for a debate and is quite subjective. In our experiments, it looked like the threshold value should be chosen in such a way as to end up considering around 25-35% of all the pixels for the grey world calculation. For regular room environments, that means the threshold should be between 20 and 30 but may need adjustment either way if the scene is poorly lit (adjust down), or has lots of highly saturated bright colors (adjust the threshold up). | |
| |
| Once we throw away all the pixels that normally contribute to the normalization going wrong, we proceed with the "usual" grey world color normalization. | |
| ==== Step-by-step algorithm ==== | |
| Below are a few blocks of steps that describe the process in a top-down approach (which is generally easier to comprehend than the bottom-up): | |
| |
| - [[#Collect statistical data set]] | |
| - [[#Calculate color properties]] | |
| - [[#Figure out the adjustment]] and exit, if processing a single frame and the result is "close enough" | |
| - Loop back to step one | |
| |
| === Collect statistical data set === | |
| |
| For each pixel: | |
| - Perform the [[https://en.wikipedia.org/wiki/YUV#HDTV_with_BT.709|RGB->Y'UV]] conversion | |
| - Calculate the pixel's "saturation" using the following formula: | |
| - ''(|U((in range -127..+127))| + |V((in range -127..+127))|) / Y((in range 0..255)) < threshold / 100''((convert threshold from percent value to relative value, i.e. convert, say, ''13%'' to ''0.13'')) | |
| - Of course one should be careful about dividing by ''0'' so first check if the ''Y'' is not ''0'' ;-) | |
| - If the ''Y'' value is **not** ''0'' **and** ''threshold'' comparison yielded ''true'' - add that pixel's ''U'' and ''V'' components to our running totals and increment the total number of "good" pixels by ''1'' | |
| - Optionally collect the same data for ''Y'' value | |
| |
| === Calculate color properties === | |
| |
| Now that we have our data set we can figure out what our picture looks like from the color balance point of view. For that we just calculate the average values of ''U'' and ''V''. The simplest approach is to calculate the [[https://en.wikipedia.org/wiki/Mean#Arithmetic_mean_(AM)|arithmetic mean]] value, but other formulae may provide more accurate results. | |
| |
| At the end of this step we have the following 3 sets of numbers: | |
| - "Main results" are the averages of ''U'' and ''V'' - these tell us how far off the picture is from being considered to be the "grey world" | |
| - Calculate the "average" ''R'', ''G'', and ''B'' values from ''U'' and ''V''. | |
| - Use the ''Y'' from [[#Collect statistical data set|previous step]] (if it was collected) or just go with the value of ''100'' which is hopefully a good representation of a well-adjusted overall brightness | |
| - Calculate the two ratios that represent how far off the ''Red'' and ''Blue'' colors are from the ''Green''((we use a fixed value of ''Green'' and never adjust it for color normalization)). Those ratios are ''R/G'' and ''B/G'' | |
| |
| === Figure out the adjustment === | |
| |
| Now that we have all the information we need to make a decision let's make one! | |
| |
| - If the **absolute** value of average ''U'' is above some arbitrary threshold (we use ''0.1'') then we need to adjust the ''Blue'' channel. In our case we just change the sensor's blue channel gain by multiplying it by the ''B/G'' ratio | |
| - Be careful not to overflow beyond the sensor gain's cap | |
| - If the gain is already at 0 and needs to be increased - make sure you use addition, as multiplication won't help you((or just make sure you never adjust the color channel's gain below, say, ''10'')) ;-) | |
| - Same for the ''Red'' channel, only we use the ''V'' and ''R/G'' values | |
| |
| The ''Blue'' channel gets a bit over-saturated on the sensor we use so a tiny adjustment is introduced: the ''B/G'' ratio is multiplied by ''0.99'' right before it is applied to the new gain's value. | |
| |
| ====== Exposure ====== | |
| FIXME | |
