======= Automatic exposure control =======

In the modern age of digital cameras and rolling shutter sensor designs the variables that affect the luminal quality of the picture have shifted from their traditional set of aperture, shutter speed, film sensitivity (ISO) into Black Level, Gains, and "exposure" (the "exposure" is in quotes because its value doesn't affect the physical blocking of light but rather how the chip reads and re-sets data values in individual sensor elements).

====== Effects of Black Level, Gains, and "Exposure" ======

Below are a few sections that provide a bit of information on what effect each of the 3 types of digital settings have on the output image.

===== Black Level =====

[[https://en.wikipedia.org/wiki/Black_level|Black level]] basically tells the sensor which level of brightness to consider "true black". If you take a look at the brightness histogram of a given image, the value of the Black Level will directly correspond to the gap at the left edge of that histogram, where there are no pixels reported to be of such low brightness value. Increasing the Black Level too high will cause the image to look washed out, while using a too low of a setting causes some grey pixels to be mistakenly read as pitch black.

Remember: the brighter the pixel, the **lower** its read charge. So increasing the Black Level causes more pixels to be considered "brighter than black" and lowering the Black Level value causes more off-black (grey) pixels to be reported as solid black, as in "these pixels were not bright enough to be considered lit".

===== Gains =====

Gains are constant values that are applied to the analog data read from the sensor before running through an ADC and can be either additive or multiplicative.

==== Individual color channel gains ====

There are 3 individual Gain settings for each of the 3 color channels (Red, Green, Blue) and those are applied to pixel data corresponding to that color. Naturally changing these values has a profound effect on the output color.  The offset to increasing gain is that it will increase the noise.

==== Global Gain ====

Global Gain is a Gain that is applied to **every** pixel. Adjusting this setting greatly affects the brightness of the picture while only minimally affecting the color balance. The offset to increasing gain is that it will increase the noise.

===== "Exposure" =====

The timing generator outputs clocks to access the rows of the imaging array, pre-charging and sampling the rows of the array sequentially. In the time between pre-charging and sampling a row, the charge in the pixels decreases with exposure to incident light. This is the exposure time in rolling shutter architecture.

The exposure time is controlled by adjusting the time interval between pre-charging and sampling. After the data of the pixels in the row has been sampled, it is processed through analog circuitry to correct the offset and multiply the data with corresponding Gain. Following analog processing is the ADC which outputs 10, 12, or 14 bits((depends on the sensor chip's model)) of data for each pixel in the array.

====== Automatic exposure control procedure ======

Ignoring the effects of UVC's "Brightness", "Contrast", and "Gamma" controls we are only concentrating on the following sensor chip level adjustments when trying to configure the settings for an optimal picture luminocity: Black level, Global Gain, Exposure.

===== Ideal picture =====

So what do we mean when we set off to adjust the exposure? In short - we are trying to hit these vague goals:
  - No overexposed pixels((however in practice we just limit their number))
  - "Bright enough" overall picture
  - Natural progression from dark to bright

In terms of quantification of the above goals we are striving to:
  - Limit the number of overexposed pixels to less than a given percentage, say ''2%''
  - The global brightness average (arithmetic mean) should be in a comfortable zone around 90-100 (assuming [0..255] range for brightness)
  - Black level should be set to correctly reflect the visual perception of the scene and the image's rendering on a display

===== Exposure step-by-step algorithm =====

The following steps describe the auto-exposure procedure in a top-down approach:
  - [[#Collect statistics]]
  - [[#Decide on action to take]] and based on that:
    - No adjustment needed - loop back to step one
    - [[#Increase the brightness]]
    - [[#Reduce the brightness]]
  - Loop back to step one

==== Collect statistics ====

Pretty straight-forward - just collect the standard histogram data for the "brightness" of the pixel, like a ''Y'' component of the ''Y'UV'' triplet.

For better results the collection should be limited to the central region of the image (where the face usually is) specified by a pair of values that denote the relative vertical and horizontal sizes of that window in percentage points. As an example: a window's horizontal relative size of ''50%'' would mean that the left edge of that window starts at ''25%'' of the total frame's pixel count on the left and continues for ''50%'' of the total frame's pixel count. Both values should be in [''5''..''100''] range.

==== Decide on action to take ====
The decision is based on two factors:
  - A ratio of target brightness to the actual average brightness (e.g. if the current image's average brightness is ''100'' and the target brightness is ''70'', then the ratio is ''0.7'' - meaning we need to dim the picture)
  - How right-hand-side heavy is the histogram. To determine that number we calculate how many right-most bars of the histogram does it take to accumulate the number of pixels that our overexposure tolerance is set to. Given an image with a total of ''2''M pixels and an overexposure tolerance set at ''2%'' if the result is ''240'', then that means that there are **over** ''40,000'' pixels with their brightness at level ''241'' and above.

Once we have the above two factors figured out the decision tree looks as follows:
<code cpp>
enum class ExpCorrAction{stay, inc, dec};
ExpCorrAction _needExpCorrection(int _right, double _adjY){
    if(_right > 60 && _adjY < 1){
        return ExpCorrAction::dec;  // there are pixels at level 100+ and we were asked to dim the image
    }
    if(_right >= 250 && _adjY <= 1){
        return ExpCorrAction::dec;  // there are more overexposed pixels than the threshold and dimming the image
    }
    if(_right < 240 && _adjY > 1){
        return ExpCorrAction::inc;  // there's room to grow and asked to increase brightness
    }
    return ExpCorrAction::stay;
}
</code>

==== Increase the brightness ====

  - If we have not reached the limit on Exposure setting - increase it by multiplying by the target/real brightness ratio
  - Else, if we have not hit the Global Gain limit - increase it by multiplying by the target/real brightness ratio
  - Else (special case!) **only** if the auto-white balance is currently turned **on** and if the brightness adjustment is very high (above ''4'') and if the Red and Blue channels' Gains are not maxed out, then **carefully** increase the Green Gain. By "carefully" I mean the following:
    - Never increase the Green Gain beyond ''1180''
    - Only increase the Green Gain as much as to not cause the overflow for the Gains in Red or Blue channels (take a conservative guess and then lower it by another ''5%'')

==== Reduce the brightness ====

This is pretty much the opposite of the [[#Increase the brightness]] described above with the actions prioritized in the opposite order:
  - First try to decrease the Green Gain by ''10%'', but never below ''1024'' and only if the auto-white balance is currently enabled
  - If the Green Gain is at ''1024'' - decrease the Global Gain by multiplying it by the target/real brightness ratio
  - If the Global Gain is already at ''0'' - decrease the Exposure by multiplying it by the target/real brightness ratio

===== Black Level =====

As the sensor processes input its electrical characteristics could fluctuate (due to, for example, heating up) requiring a manual human intervention to set an appropriate Black level (or an automated "Black level Calibration" procedure, if implemented by the sensor chip).

In order to automate this process one could think of Black Level as the counterpart to the overexposure. The goal is to have the Black Level set at such value as to ensure that there are no more than specified overexposure pixel count that are pitch black.

===== Sample implementation in C++ =====

[[Sample AE implementation in C++]] shows one possible implementation of the approach described above. It does use (without much explanation) some of the external structures, but those should not be hard to comprehend from their usage