OI (optical image)
ISETCam uses the term optical image to describe the the light that has passed through the optics and is incident at the sensor surface. This light is also called the spectral irradiance at the sensor surface. We use 'oi*' to denote the functions that operate on the optical image. As far all ISETCam functions, the basic functions are: oiCreate, oiSet/Get, oiCompute, oiWindow, and oiPlot.
These following code snippet creates a scene and optical image. It then displays the optical image in a window.
scene = sceneCreate;
oi = oiCreate('wvf');
oi = oiCompute(oi,scene);
oiWindow(oi);
Typing 'oi<TAB>' into the Matlab command prompt will bring up a large number of functions. You can find a large number of scripts and tutorials that compute, plot and derive properties of the optical image.
The scene spectral radiance is measured in units of photons/sec/sr/nm/m^2. The optical image spectral irradiance is defined with units of photons/sec/nm/m^2.
The image shown in an oiWindow is, of course, an RGB image. The image is a visual represetation of the oi spectral data. We can address the spectral data in many ways. For example, when we invoke oiPlot to show the data on the 63rd row of an optical image, the following images appear on the screen:
thisRow = 63; oiPlot(oi,'irradiance hline',[1 thisRow]);
The white line in the oiWindow shows which line is plotted - a line through the gray-series of the chart. The height of the mesh shows the spectral irradiance (quanta/sec/nm/m^2). It is plotted as a function of position on the sensor surface (microns) and wavelength (nanometers). For this line the spectral irradiance of the gray patches are mainly scaled copies of one another, from light to dark.
The RGB image shown in the oiWindow is computed from the spectral irradiance by converting the spectral irradiance into an sRGB image:
- The spectral irradiance data are converted to CIE XYZ values (see ieXYZFromPhotons).
- The RGB values in the window are computed from XYZ by assuming the user is viewing a standard sRGB display (xyz2srgb).
To learn more about the OI and optics, see
Up to this point, we have only described optical image calculations for a planar scene; or equivalently, for a scene in which all the objects are far away. For many applications the three-dimensional nature of the scene can be important - perhaps you are interested in depth of field effects (e.g., Bokeh). To accurately model three-dimensional scenes, we developed ISET3d-tiny.
The ISET3d software takes 3D graphics models and multi-element lenses as inputs. It then uses Physically Based Ray Tracing (PBRT) to trace rays in the 3D scene to the spectral irradiance at the sensor surface. The ray tracing correctly handles three-dimensional effects, including occlusions and depth of field.
We have implemented many tools, and added some features in PBRT, to coordinate this ray tracing with ISETCam. The software includes functions and data that enable us to simulate fairly complex optics and sensors. It includes methods to place microlenses at the sensor surface to model advanced camera designs, such as dual pixel autofocus and light field cameras.
These calculations are relatively complex and the models can be advanced. Thus, ISET3d software benefits from advanced hardware (GPUs) as well as the data management of the many graphics assets. For that reason we separated the code into its own separate repository ISET3d. The ISET3d toolbox is used to calculate ISET scenes and optical images, and it works smoothly with both ISETCam and ISETBio.