Sample AE implementation in C++
The code below provides a sample implementation of the AE algorithm described in Automatic exposure. It relies on a few external dependencies outlined below:
S2R::IImgSensor
from the SUB2r-lib - a conveniency library to access image sensor's chip and control its registersS2R::ImgStats
- image's statistical data, mainly histograms. This structure is not very well documented in here but should be fairly obvious from its usageS2R::enum HIST_TYPE
- a helper enum used as an index into histogram data arrays
The code below conforms to C++17 standard and it is recommended to use the C++17 toolchain as this is the version the SUB2r-lib is targetting.
Helpers
A few short functions for code readability, put into a local anonymous namespace (effectively making these function static
and making sure we don't pollute/collide with the global namespace).
namespace{ constexpr int defectivePixels = 100; // assume there are at least that many DP that will screw our stats int _triggeredLeft(const ImgStats& _stats, PerPart _pp){ auto count = max(defectivePixels, _stats.ae.pixels * _pp); for(int i = 0; i < 256; ++i){ count -= _stats.ae.hist[i]; if(count < 0){ return i; } } return 255; } int _triggeredRight(const ImgStats& _stats, PerPart _pp){ auto count = max(defectivePixels, _stats.ae.pixels * _pp); for(int i = 255; i >= 0; --i){ count -= _stats.ae.hist[i]; if(count < 0){ return i; } } return 0; } }
PerPart
This helper class is to avoid stupid arithmetic errors when mixing various ratios, like absolute ratios, percentages (%), per-milles(‰), parts-per-million (ppm), etc. It also helps with gracefully handling division by zero (which results in defaulting the result of such ratio to 0
).
In addition a few helpful user-defined literals are provided for better code readability when it comes to constants.
This is a super-minimalistic class that doesn't pretend to be production-quality.
namespace{ struct PerPart{ PerPart(long double _val = 0.f) : m_val(_val) {} // 1.0f == 100_pcnt PerPart(unsigned long long int _val, unsigned long long int _div) : PerPart(_div == 0 ? 0.f : static_cast<long double>(_val) / _div) {} operator long double() const { return m_val; } int pcnt() const { return static_cast<int>(round(m_val * 100)); } private: long double m_val = 0.f; }; PerPart operator"" _pcnt(unsigned long long int _val){return PerPart(_val, 100);} PerPart operator"" _pmille(unsigned long long int _val){return PerPart(_val, 1000);} PerPart operator"" _ppm(unsigned long long int _val){return PerPart(_val, 1000 * 1000);} }
AE
Main auto-exposure implementation class.
The unnamed internal struct m_vars
is here to provide an overview of the notable results of calculation and a convenient way to display debug info in the UI part of the program (which is more or less irrelevant for the on-device implementation and therefore is kept to the minimum in this sample implementation).
Primarily the class is used in 3 steps:
- Instantiate an object
- Call its functor
- (Optionally) display the results in UI
Declaration
struct AE{ enum class Action{noop, inc, dec}; // suggested action to take enum class CorrOp{noop, inc_gl, inc_exp, inc_green , dec_gl, dec_exp, dec_green}; // performed correction AE(IImgSensor * _ov, const ImgStats & _stats, const PerPart & _pp, int _aeTgtLuma); // analyze and, if asked and is necessary, adjust sensor's config auto operator()(bool _performCorrection) { _analyzeLH(); _analyzeRH(); if(_performCorrection){ if(m_vars.actionLH != Action::noop) _correctLH(); if(m_vars.actionRH != Action::noop) _correctRH(); } return m_vars; } const auto & details() const {return m_vars;} // diagnostics to display private: void _analyzeLH(); void _analyzeRH(); void _correctLH(); void _correctRH(); void _aeInc(); void _aeDec(); private: using Val = IImgSensor::Value; // shortcut for typing out the full type qualifier IImgSensor & m_ov; // OmniVision sensor interface const PerPart m_hysteresis; // tolerance +/- for how accurate/sensitive the correction should be struct{ int underExposed = 0; // left-most bars on the histogram it takes to cover the N tolerance pixels int overExposed = 255; // right-most bars on the histogram it takes to cover the N tolerance pixels int adjBL = 0; int avgY = 0; // actual arithmetic average of Y in [0..255] double adjY = 1.; // requested brightness adjustment for the image Action actionLH = Action::noop; Action actionRH = Action::noop; CorrOp op = CorrOp::noop; } m_vars; };
Constructor AE::AE
A bulk of (quite simplistic) calculations is performed with data in _stats
and the results are stored in the m_vars
struct:
AE::AE(IImgSensor * _ov, const ImgStats & _stats, const PerPart & _pp, int _tgtLuma, const PerPart & _hysteresis) : m_ov(*_ov) , m_hysteresis(_hysteresis) { m_vars.underExposed = _triggeredLeft(_stats, _pp); m_vars.overExposed = _triggeredRight(_stats, _pp); m_vars.avgY = static_cast<int>(_stats.ae.pixels == 0 ? 0 : _stats.ae.y / _stats.ae.pixels); m_vars.adjY = m_vars.avgY == 0 ? 0 : 1. * _tgtLuma / m_vars.avgY; }
AE::_analyzeLH()
Decide what to do with the “left side of the histogram”, which is controlled by the Black Level
value:
void AE::_analyzeLH() { // try to reduce the BL if the image is severely overexposed { const auto diffLeft = m_vars.underExposed; const auto diffRight = 256 - m_vars.overExposed; if(diffLeft > diffRight * 2){ const int currBL = m_ov[Val::black_level]; if(currBL > 0){ m_vars.adjBL = - max(1, currBL / 10); m_vars.actionLH = Action::dec; }else{ m_vars.adjBL = 0; m_vars.actionLH = Action::noop; } return; } } if(m_vars.underExposed > 2 && m_vars.underExposed < 20){ m_vars.actionLH = Action::noop; // no-op if we are in that sweet spot }else{ // strive to be get into sweet spot const int defaultBL = 16; m_vars.adjBL = (defaultBL - m_vars.underExposed) * 5 / 8; if(m_vars.adjBL > 1){ m_vars.actionLH = Action::inc; }else if(m_vars.adjBL < 1){ m_vars.actionLH = Action::dec; }else{ m_vars.actionLH = Action::noop; } } }
AE::_analyzeRH()
Decide what to do with the “right side of the histogram”, which greatly affects the overall brightness of the image and the amount of overexposure produced by the image sensor:
void AE::_analyzeRH() { m_vars.adjY += (1 - m_vars.adjY) * 5 / 8; // be 62.5% less aggressive than requested to avoid oscillations if(m_vars.overExposed > 60 && m_vars.adjY < 1 - m_hysteresis){ m_vars.actionRH = Action::dec; // plenty of pixels at level 60+ and we were asked to dim the image }else if(m_vars.overExposed > 250 && m_vars.adjY <= 1 - m_hysteresis){ m_vars.actionRH = Action::dec; // too many overexposed pixels and NOT increasing brightness }else if(m_vars.overExposed < 250 && m_vars.adjY > 1 + m_hysteresis){ m_vars.actionRH = Action::inc; // there's room to grow and asked to increase brightness }else{ m_vars.actionRH = Action::noop; } }
AE::_correctLH()
Once the Black Level correction is determined the code to perform that action is quite trivial:
void AE::_correctLH() { auto bl = m_ov[Val::black_level]; bl = std::clamp(bl + m_vars.adjBL, 0, m_ov.getLimit(Val::black_level)); }
AE::_correctRH()
“Right hand side” correction is a bit more involved and is thus split into to distinct functions, one for increasing the brightness and another for decreasing it:
void AE::_correctRH() { switch(m_vars.actionRH){ case Action::inc: _aeInc(); break; case Action::dec: _aeDec(); break; } }
AE::_aeInc()
Increasing the brightness of the image is a 3-step process where first we max out the Exposure
(as it produces the least amount of side-effects), then we are increasing the Global Gain
(which generally increases the noise). And as the absolutely last resort we reluctantly increase the Green Gain
, but only if all the following conditions are true:
- We have determined that to reach the target luminosity we need to increase the brightness at least 4-fold
- AWB is on, since the color representation is going to be severely affected by this change
- We have sufficient buffer for Red and Blue channels' Gain increases
void AE::_aeInc() { const auto limitExp = m_ov.getLimit(Val::exposure); const int setExp = m_ov[Val::exposure]; if(setExp < limitExp){ m_ov[Val::exposure] = min(limitExp, static_cast<int>((setExp ? setExp : 2000) * m_vars.adjY)); m_vars.op = CorrOp::inc_exp; return; } const auto limitGg = m_ov.getLimit(Val::gain_global); const int setGg = m_ov[Val::gain_global]; if(setGg < limitGg){ m_ov[Val::gain_global] = min(limitGg, static_cast<int>((setGg ? setGg : 10) * m_vars.adjY)); m_vars.op = CorrOp::inc_gl; return; } const auto limitRgb = m_ov.getLimit(Val::gain_g); const int setGreen = m_ov[Val::gain_g]; if(m_vars.adjY > 4 && setGreen < limitRgb * 0.95){ const auto safetyBuffer = 0.95; const auto limitRGain = 1180; const int highestRgb = max(m_ov[Val::gain_r], m_ov[Val::gain_b]); const double topMultiplier = std::clamp(m_vars.adjY, 1., safetyBuffer * limitRgb / highestRgb); const auto gainG = static_cast<int>(min(limitRGain, setGreen * topMultiplier)); if(gainG > setGreen){ m_ov[Val::gain_g] = gainG; m_vars.op = CorrOp::inc_green; return; } } m_vars.op = CorrOp::noop; }
AE::_aeDec()
Quite expectedly this function mirrors AE::_aeInc()
. First it tries to bring the Green Gain
back to its default level of 1024
, then reduce the Global Gain
and, lastly, the Exposure
:
void AE::_aeDec() { const int setGreen = m_ov[Val::gain_g]; if(setGreen > 1024){ const auto nextGreen = static_cast<int>(setGreen - (setGreen - 1024) * 10_pcnt); assert(nextGreen >= 1024); m_ov[Val::gain_g] = nextGreen; m_vars.op = CorrOp::dec_green; return; } const int setGg = m_ov[Val::gain_global]; if(setGg > 0){ m_ov[Val::gain_global] = static_cast<int>(setGg * m_vars.adjY); m_vars.op = CorrOp::dec_gl; }else{ m_ov[Val::exposure] *= m_vars.adjY; m_vars.op = CorrOp::dec_exp; if(m_ov[Val::exposure] < 10){ // safety net to make sure we never get to corner case m_ov[Val::exposure] = 10; m_vars.op = CorrOp::noop; } } }
Sample usage
This is a copy-paste of the code used in Aria utility for the POC. This code has minimal comments but should be fairly easy to follow:
namespace{ static CString _corrOp2Txt(AE::CorrOp _op){ using op = AE::CorrOp; switch(_op){ case op::noop: return {}; case op::inc_exp: return L"+++ Increasing Exposure"; case op::inc_gl: return L"+++ Increasing GLOBAL gain"; case op::inc_green: return L"+++ PANIC! Increasing GREEN gain"; case op::dec_exp: return L"--- Reducing Exposure"; case op::dec_gl: return L"--- Reducing GLOBAL gain"; case op::dec_green: return L"--- Reducing GREEN gain"; } return {}; } } void ImgAnalyzerDlg::_aeInfo() { if(((CButton *)(GetDlgItem(IDC_CHECK_ISA_AE)))->GetCheck() != BST_CHECKED){ return; } const auto & stats = m_analyzer.getStats(); const auto correct = ((CButton*)(GetDlgItem(IDC_CHECK_ISA_AE_CORRECT)))->GetCheck() == BST_CHECKED; const auto aeTolerance = PerPart(GetDlgItemInt(IDC_EDIT_ISA_AE_RIGHT, NULL, FALSE), 1000); const int aeTgtLuma = GetDlgItemInt(IDC_EDIT_ISA_AE_BRIGHTNESS, NULL, FALSE); const PerPart aeHysteresis{GetDlgItemInt(IDC_EDIT_ISA_AE_HYSTERESIS, NULL, FALSE), 100}; AE ae{m_fx3.sensor(), stats, aeTolerance, aeTgtLuma, aeHysteresis}; const auto rez = ae(correct); // display the information { const PerPart pxUsed{stats.ae.pixels, stats.total.pixels}; CString msg; msg.AppendFormat(L"Pixels used: %d (%.1f%%)\n", stats.ae.pixels, pxUsed * 100); msg.AppendFormat(L"Under/over-exposed: %d/%d\n", rez.underExposed, rez.overExposed); msg.AppendFormat(L"BL adj.: %d\n", rez.adjBL); msg.AppendFormat(L"Avg Y.: %d\n", rez.avgY); msg.AppendFormat(L"Brightness adj.: %.2f\n", rez.adjY); msg += _corrOp2Txt(ae.details().op); SetDlgItemTextW(IDC_STATIC_ISA_AE, msg); } }
Initial values
Following are the initial defaults for the variables that affect the process of automatic image brightness (Black Level, Exposure, Gains) adjustment:
Name (as seen in ImgAnalyzerDlg::_aeInfo() above) | Value |
---|---|
aeTolerance | 1/1000 |
aeTgtLuma | 90 |
aeHysteresis | 5/100 |
“center window” dimensions | 50% vertically and horizontally, centered at image's center |