Pith. sign in

REVIEW 4 major objections 4 minor 79 references

Paleoinspired Vision: From Exploring Colour Vision Evolution to Inspiring Camera Design

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A one-layer opsin model, evolved by segmentation accuracy, reproduces colour-vision transitions and designs task-specific camera filters.

desk verdict Genuinely novel opsin-layer framework with a clean conservation constraint, but the quantitative support is not there yet: single-run tables, RGB-reconstructed HSI, and a trainable C-to-3 layer that confounds the fitness signal. read the letter →

arxiv 2412.19439 v1 pith:F3DPVO3E submitted 2024-12-27 cs.CV

Junjie Zhang , Zhimin Zong , Lin Gu , Shenghan Su , Ziteng Cui , Yan Pu , Zirui Chen , Jing Lu
show 3 more authors
Daisuke Kojima Tatsuya Harada Ruogu Fang
This is my paper · ORCID
classification cs.CV
keywords colourvisionevolutionopsinlayerevolutionaryconservationoptimisationspectralsensitivityhyperspectralimagereconstructionsemanticsegmentationtask-specificcameradesignprimatetrichromacy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Evolutionary hypotheses about colour vision are usually tested through genetic or behavioural experiments. This paper argues that a drastically simplified computational surrogate—a single 'opsin layer' of Gaussian-shaped spectral filters whose only learnable parameter is the peak wavelength $\lambda_{\max}$, driven by segmentation accuracy as a stand-in for fitness—can replay those evolutionary transitions in seconds on a GPU. Using hyperspectral images, including spectra reconstructed from ordinary RGB, the authors reproduce the mammalian loss of two cone opsins, primate trichromacy from gene duplication, the advantage of colour blindness in dim light, the blue-shift of fish rod opsins with depth, and a modest opsin split under simulated bioluminescence. The same optimisation, applied to camera filters, yields task-specific spectral response functions for Martian terrain and cancer-tissue segmentation with small but consistent gains over generic RGB filters. If the spectral reconstructions are faithful, the framework offers a fast, quantitative platform for both evolutionary 'what-if' questions and minimalist camera design.

What carries the argument

The load-bearing machinery is the opsin layer combined with the evolutionary conservation optimisation. The opsin layer is a $1\times1$ convolutional layer with $C$ Gaussian kernels, each kernel acting as a spectral sensitivity function $\psi_c$; because integrating a hyperspectral pixel's radiance against an opsin sensitivity function is the same as a dot product, one convolution maps an $H\times W\times N$ hyperspectral image to an $H\times W\times C$ feature map. The conservative regularisation keeps the Gaussian shape and width $\sigma$ fixed, allows only the peak wavelengths $\lambda_{\max,c}$ to move, and limits each move to $0.5$ nm per epoch, encoding the biology of spectral tuning sites. A MiT-B0 encoder plus a lightweight all-MLP decoder converts those feature maps into segmentation maps, and the segmentation loss supplies the selection pressure. This setup is what converts millions of years of evolution into a few hundred GPU iterations, and it is also the mechanism that lets the authors treat camera filter design as the same optimisation problem.

What would settle it

Train the same opsin-layer pipeline on a dataset with true measured hyperspectral radiance (for example, real HSI of apples on leaves) and compare the converged $\lambda_{\max}$ trajectories with those obtained from RGB-reconstructed spectra; a shift of more than about 10 nm, or failure of a duplicated long-wavelength opsin to split into two distinct peaks, would show that the evolutionary conclusions depend on the reconstruction network rather than on biology.

Watch

Extended reading notes

Core claim

The paper's central claim, stated on its own terms, is that the evolutionary history of vertebrate colour vision is, to a first approximation, a smooth drift of a few Gaussian peak wavelengths under selection, and that this drift can be captured by a single differentiable layer. The opsin layer is a $1\times1$ convolution whose kernels are fixed-width Gaussians $\psi_c(\lambda_i)=\frac{1}{\sqrt{2\pi}\sigma}e^{-(\lambda_i-\lambda_{\max,c})^2/2\sigma^2}$, and the only parameter allowed to change during optimisation is $\lambda_{\max}$, capped at $0.5$ nm per epoch to mimic real spectral tuning sites. Selection is supplied by cross-entropy loss of a MiT-B0 encoder with an all-MLP segmentation decoder, so mIoU plays the role of evolutionary fitness. From ancestral starting points the optimiser moves $\lambda_{\max}$ along trajectories that match known transitions: two mammalian opsins shift under dim light, a duplicated long-wavelength opsin separates into the trichromatic pair, squirrelfish rod opsin shifts from roughly 496 nm at the surface to 481 nm at 70 m depth, and a five-opsin layer develops a maximum 5 nm separation under simulated bioluminescence but not without it. The same loop, applied to camera spectral response functions, shifts filter peaks for Mars segmentation (611.88/522.93/425.90 nm versus a 580/540/425 nm baseline) and for cancer detection, supporting the proposal that task-specific camera filters can be evolved rather than hand-designed.

Load-bearing premise

The load-bearing premise is that segmentation accuracy on hyperspectral images reconstructed from RGB faithfully represents real spectral radiance and can serve as a proxy for evolutionary fitness; if the reconstruction is inaccurate, every optimized $\lambda_{\max}$ is an artifact of the reconstruction network.

Editorial extensions

If this is right

  • The framework gives evolutionary biologists a cheap in silico testbed: hypotheses about opsin tuning under different light environments can be probed in seconds, without stochastic genetic screens.
  • It quantitatively supports the selective story behind primate trichromacy—better fruit detection in leaves—while also predicting conditions under which dichromacy outperforms trichromacy (dim light, khaki-coloured terrain), bearing on why colour blindness persists.
  • It reproduces the depth-dependent blue-shift of squirrelfish rod opsins from roughly 496 nm at the surface to 481 nm at 70 m, and shows a small opsin separation arising under simulated bioluminescence, linking bioluminescence to multiple rod opsin evolution.
  • The same optimisation pipeline transfers directly to camera sensor design: for Mars terrain and cholangiocarcinoma detection, the evolved filter peaks improve segmentation mIoU/IoU over standard RGB-inspired filters, suggesting that application-specific spectral response functions can be manufactured with small iterative filter changes.
  • Under the model's assumptions, hypothetical Martian organisms would be better served by dichromatic than trichromatic or tetrachromatic vision, giving a concrete prediction about the visual systems of alien life under Martian illumination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, the same loop could be applied to other visually guided tasks with known spectral signatures—agricultural weed/crop discrimination, underwater monitoring, or industrial sorting—where a hand-designed filter set is currently the default; the paper does not test these.
  • A direct validation of the weakest link would be to compare converged $\lambda_{\max}$ values on true measured hyperspectral radiance versus RGB-reconstructed spectra, since this would show how much of the reconstructed biology is real.
  • Varying the 0.5 nm-per-epoch cap, or removing the Gaussian shape constraint, would reveal whether the reported evolutionary trajectories are robust biological attractors or artefacts of the regularisation schedule, which the paper keeps fixed.
  • The framework's fitness signal is segmentation accuracy of a specific deep network, not survival or reproduction; using different encoders or tasks as the fitness signal could change which $\lambda_{\max}$ values win, and that sensitivity is not explored.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a computational framework, called evolutionary conservation optimisation, to model colour vision evolution. It introduces an opsin layer whose convolutional kernels are Gaussian approximations of spectral sensitivity functions, with only the centre wavelength λ_max updated under a per-epoch step cap, and uses semantic segmentation mIoU on hyperspectral images as a proxy for evolutionary fitness. The framework is applied to reconstruct several proposed evolutionary transitions: the loss of two cone opsins in mammals, primate trichromacy via gene duplication, retention of colour blindness, blue-shift of squirrelfish rod opsins, and multiple rod opsins under bioluminescence. It is also used to speculate about Martian colour vision and to design task-specific camera spectral filters for Mars and cancer detection. The paper claims that the model quantitatively verifies long-standing biological hypotheses and provides a minimalist camera-design paradigm.

Significance. If the framework were validated, it would provide a fast, in-silico platform for generating quantitative hypotheses about opsin evolution and a practical recipe for task-specific spectral filter design. The opsin-layer parameterisation is simple and the evolutionary conservation constraint is an interesting way to inject biological plausibility into a differentiable pipeline. The paper also connects with a currently active area of minimalist, task-driven camera design. However, the reported experiments do not yet support the central quantitative claims: the fitness signal is confounded by a trainable layer at the front end; several target phenomena are effectively inserted into the experimental setup; there are no error bars or repeated seeds; and key datasets are generated by RGB-to-HSI reconstruction. The manuscript's strengths are the conceptual framework and the breadth of application scenarios, but the evidence as presented is not sufficient to establish the biological verifications or the practical camera-design recommendations.

major comments (4)
  1. [Supplementary Sec. 7.1, Fig. 5; main-text Sec. 3.4] The C-to-3 1×1 convolution layer, placed between the opsin layer and the MiT encoder, is trainable and is trained jointly with the opsin layer, encoder, and decoder. For C=2 and C=3 it can arbitrarily recombine the opsin channels, and for C=1 it can rescale the single channel. Therefore the mIoU values in Secs. 4.2–4.6 and Sec. 5 reflect the joint adaptation of the whole front end, not the opsin spectral sensitivities in isolation. The claim in Sec. 1 that recognition performance on images filtered through specific opsins quantifies the opsin advantage is not supported unless this layer is ablated (e.g., frozen to an identity-like matrix), or controlled by random fixed projections, or by multiple seeds that show the outcome is insensitive to the C-to-3 initialization. Without such an ablation, the biological interpretations of the evolved λ_max values are confounded.
  2. [Sec. 4.5.1, Eq. (4); Sec. 4.5.2, Supp. Sec. 8.1] The blue-shift result follows directly from the input model. In Eq. (4), E(d,λ) = E(0,λ)e^{−Kd(λ)d}, and the diffuse downwelling attenuation Kd(λ) attenuates long wavelengths more strongly with depth, so the ambient spectrum is constructed to shift to shorter wavelengths as d increases. Optimizing a single Gaussian λ_max on such inputs is expected to produce a decreasing λ_max with depth; this is an extraction of the known spectral trend, not an independent verification of the blue-shift hypothesis. Similarly, the multi-rod-opsin experiment (Supp. Sec. 8.1) creates bioluminescence by exempting a specific label region from the diffusion attenuation, thereby injecting a local spectral feature that the multi-kernel optimization can lock onto. The paper should present these as illustrative reconstructions or add null controls (e.g., input spectra without depth-dependent attenuation) to show that the optimization does not simply recover the designed-in trend.
  3. [All result tables (e.g., Tab. 2, Tab. 3, Tab. 4, Tab. 6, Tab. 7)] Every quantitative result is reported as a single mIoU or SR value, with no standard deviation, confidence interval, significance test, or multiple random seeds. Given that some claimed improvements are very small (e.g., Tab. 6: 39.15 vs. 39.20; Tab. 4: 41.01 vs. 40.75), the differences may be within run-to-run variation. The conclusions that one visual system 'outperforms' another, or that a designed filter 'enhances performance,' are not statistically supported. The paper should provide repeated runs with distinct initializations, report mean ± std, and ideally perform paired significance tests for the comparisons that drive the biological and camera-design claims.
  4. [Sec. 4.1.2; Sec. 5.2] The hyperspectral data for MinneApple, VOC2012, and Mars-Seg are reconstructed from RGB using the method of [6]. The optimized λ_max values are therefore attributes of the reconstruction network's estimated spectra, not of the actual scene radiance. This is especially load-bearing for the Mars camera design (Sec. 5.2) and the Martian vision prediction (Sec. 4.6), where the designed filters are trained on synthetic HSI derived from RGB images. The paper should either validate the reconstruction's spectral accuracy on a dataset with measured HSI, or reframe these results as demonstrations on synthetic spectra rather than as recommendations for real Mars-exploration or medical cameras.
minor comments (4)
  1. [Sec. 4.3, Supp. Sec. 7.2] The camouflage score SR is introduced in the main text only by name; it should be defined in the main text or the main-text table should explicitly state that it is the reconstruction fidelity score from the supplementary material. This is currently unclear because Tab. 2 uses SR without defining it.
  2. [Tab. 3 vs. Tab. 2] There appears to be an inconsistency: Tab. 2 lists di-vision SR 0.2195 and tri-vision SR 0.3161, while Tab. 3 reports the 'normal' column as 0.2195 in bright conditions. If 'normal' is meant to be normal trichromatic vision, the value should match the tri-vision value of 0.3161; if it is meant to be dichromatic, the column label is wrong. Please clarify the correspondence between the two tables.
  3. [Sec. 5.3, Tab. 7] The main text says 'As shown in Tab.17' but refers to the 3-filter camera result; the table in the main text is Tab. 7, while Tab. 17 appears in the supplementary material. This cross-reference should be corrected.
  4. [Abstract; Sec. 4.4] There are some typographical errors, including the duplicated 'to' in the abstract ('adaptations to to more effectively spot fruits') and 'clour-blind' in Sec. 4.4. The manuscript should be proofread for such issues.

Circularity Check

4 steps flagged · score 6.0 of 10

Multiple 'evolutionary reconstructions' are fitted consequences of the input conditions and the trainable C-to-3 layer, so the reported lambda_max values do not independently test the biological hypotheses.

  1. fitted input called prediction [Sec. 4.5.1, Eq. 4, Table 5]
    "we simulate the underwater spectrum and intensity of ambient at different depths. E(d, λ) = E(0, λ)e−Kd(λ)d ... we record the λmax that maximize the mIou for segmentation on test set for each depth. The maximum sensitive wavelength λmax has an obvious reduction as the depth increases."

    The attenuation model in Eq. 4 is the input condition: it removes long-wavelength light increasingly with depth. The optimized Gaussian center λmax is selected to maximize mIoU on that blue-shifted data, so the decrease of λmax with depth is a direct consequence of the input spectrum, not an emergent prediction. Presenting this as a 'quantitative proof' that light diffusion causes the blue-shift restates the input as the output.

  2. fitted input called prediction [Sec. 4.5.2, Supplementary Sec. 8.1]
    "We simulate deep-sea environment with bioluminescence by maintaining the region of a specific label not adjusted by the underwater diffusion ... Experimental results in Sec.8.1 reveal that, without bioluminescence, rod opsins show no significant separation, whereas bioluminescence leads to a maximum 5 nm separation across 5 wavelengths."

    The bioluminescence is defined as a spectral region exempt from attenuation, i.e., the input already contains a second spectral niche distinct from the attenuated background. It is not surprising that five free Gaussian centers separate to cover both niches. The claimed 'quantitative evidence supporting the role of bioluminescence' is a consequence of how bioluminescence was inserted into the input, and the paper itself admits the separation is small relative to biological data.

2 more flagged steps
  1. fitted input called prediction [Sec. 4.4, Supplementary Sec. 7.2]
    "we adjust the value of t based on the lighting conditions, as ||Ff g||2 is small in dim light, making the reconstruction criterion for the foreground region more stringent. Specifically, we set t = 0.2 for bright conditions, t = 1.2 for dark(0.1), and t = 1.6 for darker conditions (0.05)."

    The conclusion that colour blindness outperforms normal trichromatic vision under dim light is based on the SR metric whose threshold t is a free parameter changed with each condition. Since t directly controls how easily a foreground pixel is counted as reconstructed, the reported ordering is conditional on arbitrarily chosen values; no robustness analysis is given. The 'quantitative verification' is thus manufactured by the metric's tuning, not by the opsin properties.

  2. other [Supplementary Sec. 7.1, Fig. 5]
    "To ensure compatibility with the MiT encoder, we introduce a 1 × 1 convolutional layer to transform the feature map from H × W × C to H × W × 3 ... In the training stage, we only train the opsin layer and C-to-3 convolution layer (if exist) simultaneously with the encoder and decoder."

    The main text defines evolutionary fitness as 'machine recognition performance on images filtered through specific opsins' (Sec. 1). In the implementation, the encoder sees an arbitrary learned linear combination of the opsin channels (C-to-3), so mIoU used to select λmax is a fitness of the composite front-end, not of the opsin sensitivities alone. The 'predicted' λmax trajectories and camera filters could therefore be produced by the C-to-3 weights and encoder adaptation rather than by any intrinsic advantage of the reported spectral positions.

full rationale

The paper is a self-contained simulation and does not rely on a load-bearing self-citation chain. However, the core experiments that purport to 'reconstruct' or 'quantitatively prove' evolutionary hypotheses are arranged so that the target phenomenon is already present in the input. Blue-shift: the underwater attenuation law (Eq. 4) blue-shifts the illumination, and the optimized λmax shifts accordingly; it is a fitted response to the input spectrum, not a predictive discovery. Bioluminescence: the bioluminescent region is defined as an unattenuated spectral niche, so the separation of multiple rod opsin kernels is a direct consequence of the constructed bimodal environment. Colour-blindness: the camouflage-score threshold t is changed with lighting condition (0.2, 1.2, 1.6), so the reported advantage of dichromacy is a function of that arbitrary choice. In addition, the trainable C-to-3 layer between the opsin layer and the encoder means mIoU measures the whole learned front-end rather than isolated opsin sensitivities, confounding every λmax 'prediction'. The paper even acknowledges that the bioluminescence separation is smaller than biological data. Overall, several derivation steps reduce to fitted parameters or constructed inputs being presented as evolutionary findings, so a partial circularity score of 6 is appropriate.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The paper introduces no new physical entities; the opsin layer is a computational layer, not a postulated biological object. The free parameters listed here are the hand-chosen or unreported constants that shape the optimization and the reported conclusions.

free parameters (5)
  • Gaussian width sigma = not reported
    The spectral sensitivity of each opsin is a Gaussian with a fixed sigma (Eq. 1); sigma controls the bandwidth of the filters and is never specified in the paper, yet it fully determines the shape of the optimizable functions.
  • Camouflage threshold t = 0.2 (bright), 1.2 (dark 0.1), 1.6 (dark 0.05)
    In Sec. 7.2, t is adjusted per lighting condition to make the reconstruction criterion more or less stringent; these hand-set thresholds directly change the SR values that drive the color-blindness conclusions (Sec. 4.4, Table 3).
  • Per-epoch lambda_max step cap = 0.5 nm/epoch
    This cap (Sec. 3.2.2) is an ad hoc modeling choice intended to mimic 5-25 nm mutation shifts; the specific value affects how far the optimization can travel in a fixed number of epochs.
  • Noise factor tau = 0.1 (cones), 0.02 (rods)
    Set in Sec. 4.1.3 from two cited psychophysical papers; the noise model and these values determine the dim-light results in Secs. 4.2, 4.4, and 4.5.
  • Bioluminescent region selection = unspecified label region
    In Supplementary Sec. 8.1, bioluminescence is modeled by keeping a specific label region unaffected by underwater diffusion; the choice of region is not specified and affects whether opsins separate.
assumptions (7)
  • domain assumption Opsin spectral sensitivity is a Gaussian with a single fixed standard deviation sigma.
    Eq. 1; real opsin templates (Govardovskii et al. [22]) are asymmetric and narrower, but the paper only varies lambda_max.
  • ad hoc to paper Only lambda_max can change during evolution; sigma and other kernel weights are fixed.
    Sec. 3.2.2; this is a modeling constraint, not derived from biology.
  • domain assumption Machine segmentation accuracy (mIoU) or reconstruction fidelity (SR) measures evolutionary fitness.
    Sec. 1 ('We quantify this advantage by assessing machine recognition performance'); this is the core proxy that links optimization to natural selection.
  • domain assumption Hyperspectral images reconstructed from RGB via [6] are a faithful representation of environmental spectra.
    Sec. 4.1.2; used for MinneApple, VOC2012, and Mars-Seg, but no validation of the reconstruction accuracy is provided.
  • standard math Underwater ambient light attenuation follows E(d,lambda)=E(0,lambda)*exp(-Kd(lambda)*d).
    Eq. 4, from Sea-thru [1]; this is a published physical model, but its use as the sole driver of blue-shift is an assumption.
  • domain assumption Poisson dark-light noise with variance tau*I models dim-light eyes.
    Eq. 3 and Sec. 4.1.3; a standard approximation, but tau values are taken from human and macaque studies and applied to fish and hypothetical Martians.
  • ad hoc to paper Gene duplication is modeled by duplicating a kernel that then diverges.
    Sec. 4.3 and Fig. 3; the mechanism of duplication is a simulation choice, not a biological process.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Paleoinspired Vision: From Exploring Colour Vision Evolution to Inspiring Camera Design." pith.science (2026). https://pith.science/paper/F3DPVO3E

@misc{pith2026241219439,
  author       = {Pith},
  title        = {Pith review of: Paleoinspired Vision: From Exploring Colour Vision Evolution to Inspiring Camera Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F3DPVO3E}},
  note         = {Machine review of arXiv:2412.19439}
}
read the original abstract

The evolution of colour vision is captivating, as it reveals the adaptive strategies of extinct species while simultaneously inspiring innovations in modern imaging technology. In this study, we present a simplified model of visual transduction in the retina, introducing a novel opsin layer. We quantify evolutionary pressures by measuring machine vision recognition accuracy on colour images shaped by specific opsins. Building on this, we develop an evolutionary conservation optimisation algorithm to reconstruct the spectral sensitivity of opsins, enabling mutation-driven adaptations to to more effectively spot fruits or predators. This model condenses millions of years of evolution within seconds on GPU, providing an experimental framework to test long-standing hypotheses in evolutionary biology , such as vision of early mammals, primate trichromacy from gene duplication, retention of colour blindness, blue-shift of fish rod and multiple rod opsins with bioluminescence. Moreover, the model enables speculative explorations of hypothetical species, such as organisms with eyes adapted to the conditions on Mars. Our findings suggest a minimalist yet effective approach to task-specific camera filter design, optimising the spectral response function to meet application-driven demands. The code will be made publicly available upon acceptance.

Figures

Figures reproduced from arXiv: 2412.19439 by the authors.

Figure 1
Figure 1. The evolution history from vertebrates to primate lineage. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of evolutionary conservation optimisation. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Reconstruction of the gene duplication process in pri [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Blue-shift. Using the processed LIB-HSI dataset [23] adjusted ac￾cording to Eq.4, representing depths of 0m, 10m, 50m, and 70m, we perform a semantic segmentation task by applying the opsin layer with a single kernel to simulate a single rod opsin. Initial parameter λm…
Figure 5
Figure 5. Figure 5: Architecture of the network. Noise Layer The noise layer is applied after each chan￾nel of the opsin layer’s output to incorporate the noise present in opsins. Let the output of the opsin layer for chan￾nel c be Ic; the output of the noise layer is expressed as: I nois…
Figure 6
Figure 6. Figure 6: Training stage. 7.2. Camouflage Score Specifically, we utilize the reconstruction fidelity score pro￾posed by Hala Lamdouar et al. [36] as the metrci to evalua￾tion the ability of the colour vision to recognize camouflage. For the LMS image F output from the opsin laye…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 73 canonical work pages

  1. [6]

    Mask-guided spectral-wise transformer for efficient hyper- spectral image reconstruction

    Yuanhao Cai, Jing Lin, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, and Luc Van Gool. Mask-guided spectral-wise transformer for efficient hyper- spectral image reconstruction. In 2022 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pages 17481–17490, 2022. 2, 5

    work page 2022

  2. [1]

    Sea-thru: A method for removing water from underwater images

    Derya Akkaynak and Tali Treibitz. Sea-thru: A method for removing water from underwater images. In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1682–1691, 2019. 7

    work page 2019

  3. [2]

    A broadband hyperspectral image sensor with high spatio-temporal reso- lution

    Liheng Bian, Zhen Wang, Yuzhe Zhang, Lianjie Li, Yinuo Zhang, Chen Yang, Wen Fang, Jiajun Zhao, Chunli Zhu, Qinghao Meng, Xuan Peng, and Jun Zhang. A broadband hyperspectral image sensor with high spatio-temporal reso- lution. Nature, 635(8037):73–81, 2024. 2

    work page 2024

  4. [3]

    Richard Blackwell

    H. Richard Blackwell. Contrast thresholds of the human eye. J. Opt. Soc. Am., 36(11):624–643, 1946. 5

    work page 1946

  5. [4]

    J. M. Bosten, J. D. Robinson, G. Jordan, and J. D. Mollon. Multidimensional scaling reveals a color dimension unique to ‘color-deficient’ observers. Current Bi- ology, 15(23):R950–R952, 2005. 6

    work page 2005

  6. [5]

    Bowmaker

    James K. Bowmaker. Evolution of colour vision in verte- brates. Eye, 12(3):541–547, 1998. 5, 6

    work page 1998

  7. [7]

    The genetic and evo- lutionary drives behind primate color vision

    Livia S Carvalho, Daniel MA Pessoa, Jessica K Mountford, Wayne IL Davies, and David M Hunt. The genetic and evo- lutionary drives behind primate color vision. Frontiers in ecology and Evolution, 5:34, 2017. 2, 4

    work page 2017

  8. [8]

    Thin on-sensor nanophotonic array cameras

    Praneeth Chakravarthula, Jipeng Sun, Xiao Li, Chenyang Lei, Gene Chou, Mario Bijelic, Johannes Froesch, Arka Ma- jumdar, and Felix Heide. Thin on-sensor nanophotonic array cameras. ACM Transactions on Graphics (TOG) , 42(6):1– 18, 2023. 3

    2023

Show all 79 references
  1. [9]

    Vision transformer adapter for dense predictions, 2023

    Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions, 2023. 5

    2023

  2. [10]

    Neuralink’s ‘breakthrough device’ clear- ance from fda does not mean it has cured blindness

    Devin Coldewey. Neuralink’s ‘breakthrough device’ clear- ance from fda does not mean it has cured blindness. Tech Crunch. 9

  3. [11]

    The differentiable lens: Compound lens search over glass surfaces and materials for object detection

    Geoffroi C ˆot´e, Fahim Mannan, Simon Thibault, Jean- Franc ¸ois Lalonde, and Felix Heide. The differentiable lens: Compound lens search over glass surfaces and materials for object detection. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recogniti...

  4. [12]

    V orobyev D

    M. V orobyev D. Osorio. Colour vision as an adaptation to frugivory in primates. Royal Society, 263:593–599, 1996. 3, 6

    1996

  5. [13]

    W AYNE I. L. DA VIES, SHAUN P. COLLIN, and DA VID M. HUNT. Molecular ecology and adaptation of visual pho- topigments in craniates. Molecular Ecology, 21(13):3121– 3158, 2012. 1, 2, 6

    2012

  6. [14]

    Human eye, 2024

    Hugh Davson and Perkins. Human eye, 2024. 2

    2024

  7. [15]

    The molecular basis of variation in human color vision

    SS Deeb. The molecular basis of variation in human color vision. Clinical Genetics, 67(5):369–377, 2005. 3, 5

    2005

  8. [16]

    Patterson, Przemysław Gorzelak, Samuel Zamora, Philip LeDuc, and Carmel Majidi

    Richard Desatnik, Zach J. Patterson, Przemysław Gorzelak, Samuel Zamora, Philip LeDuc, and Carmel Majidi. Soft robotics informs how an early echinoderm moved. Pro- ceedings of the National Academy of Sciences , 120(46): e2306580120, 2023. 3

    2023

  9. [17]

    Noise and the absolute thresholds of cone and rod vision

    Kristian Donner. Noise and the absolute thresholds of cone and rod vision. Vision Research, 32(5):853–866, 1992. 5

    1992

  10. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  11. [19]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal- network.org/challenges/VOC/voc2012/workshop/index.html. 5

    2012

  12. [20]

    Multiscale vision transformers, 2021

    Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers, 2021. 5

    2021

  13. [21]

    Evolution of colour vision in mammals

    Jacobs GH. Evolution of colour vision in mammals. Philos Trans R Soc Lond B Biol Sci, 2009. 3

    2009

  14. [22]

    GOV ARDOVSKII, NANNA FYHRQUIST, TOM REUTER, DMITRY G

    VICTOR I. GOV ARDOVSKII, NANNA FYHRQUIST, TOM REUTER, DMITRY G. KUZMIN, and KRISTIAN DONNER. In search of the visual pigment template. Visual Neuroscience, 17(4):509–528, 2000. 2, 4

    2000

  15. [23]

    A hyperspectral and rgb dataset for building fac ¸ade 9 segmentation

    Nariman Habili, Ernest Kwan, Weihao Li, Christfried We- bers, Jeremy Oorloff, Mohammad Ali Armin, and Lars Pe- tersson. A hyperspectral and rgb dataset for building fac ¸ade 9 segmentation. In Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedi...

    2022

  16. [24]

    Brent Hawkins, Katrin Henke, and Matthew P

    M. Brent Hawkins, Katrin Henke, and Matthew P. Harris. Latent developmental potential to form limb-like skeletal structures in zebrafish. Cell, 184(4):899–911.e13, 2021. 8

    2021

  17. [25]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 3

    2016

  18. [26]

    Pre-sensor computing with com- pact multilayer optical neural network

    Zheng Huang, Wanxin Shi, Shukai Wu, Yaode Wang, Sigang Yang, and Hongwei Chen. Pre-sensor computing with com- pact multilayer optical neural network. Science Advances, 10(30):eado8516, 2024. 3

    2024

  19. [27]

    Minneapple: A benchmark dataset for apple detection and segmentation,

    Nicolai H ¨ani, Pravakar Roy, and V olkan Isler. Minneapple: A benchmark dataset for apple detection and segmentation,

  20. [28]

    Shu- bin, and Fumiya Iida

    Michael Ishida, Fidji Berio, Valentina Di Santo, Neil H. Shu- bin, and Fumiya Iida. Paleoinspired robotics as an experi- mental approach to the history of life. Science Robotics, 9 (95):eadn1125, 2024. 2, 3

    2024

  21. [29]

    What is the space of spectral sensitivity functions for digital color cameras? In 2013 IEEE Workshop on Applications of Computer Vision (WACV), pages 168–179, 2013

    Jun Jiang, Dengyu Liu, Jinwei Gu, and Sabine S ¨usstrunk. What is the space of spectral sensitivity functions for digital color cameras? In 2013 IEEE Workshop on Applications of Computer Vision (WACV), pages 168–179, 2013. 3

    2013

  22. [30]

    Mollon Kanwaljit S

    John D. Mollon Kanwaljit S. Dulai, Miranda von Dornum and David M. Hunt. The evolution of trichromatic color vi- sion by opsin gene duplication in new world and old world primates. Genome Res, 9(7):629–638, 1999. 2, 6

    1999

  23. [31]

    Hakki Can Karaimer and Michael S. Brown. A software plat- form for manipulating the camera imaging pipeline. In Eu- ropean Conference on Computer Vision (ECCV), 2016. 3

    2016

  24. [32]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. 1

    2017

  25. [33]

    Jeremy Klotz and Shree K. Nayar. Minimalist vision with freeform pixels. In Computer Vision – ECCV 2024 , pages 329–346, Cham, 2025. Springer Nature Switzerland. 3, 4, 9

    2024

  26. [34]

    Camoteacher: Dual-rotation consistency learning for semi-supervised camouflaged object detection, 2024

    Xunfa Lai, Zhiyu Yang, Jie Hu, Shengchuan Zhang, Liujuan Cao, Guannan Jiang, Zhiyu Wang, Songan Zhang, and Ron- grong Ji. Camoteacher: Dual-rotation consistency learning for semi-supervised camouflaged object detection, 2024. 6

    2024

  27. [35]

    T. D. Lamb. Sources of noise in photoreceptor transduction. J. Opt. Soc. Am. A, 4(12):2295–2300, 1987. 5

    1987

  28. [36]

    The making and breaking of camouflage, 2023

    Hala Lamdouar, Weidi Xie, and Andrew Zisserman. The making and breaking of camouflage, 2023. 6, 1

    2023

  29. [37]

    Robotics as a comparative method in ecol- ogy and evolutionary biology

    George V Lauder. Robotics as a comparative method in ecol- ogy and evolutionary biology. Integrative and Comparative Biology, 62(3):721–734, 2022. 3

    2022

  30. [38]

    Spectratrack: megapixel, hundred-fps, and thousand- channel hyperspectral imaging

    Daoyu Li, Jinxuan Wu, Jiajun Zhao, Hanwen Xu, and Liheng Bian. Spectratrack: megapixel, hundred-fps, and thousand- channel hyperspectral imaging. Nature Communications, 15 (1):9459, 2024. 2

    2024

  31. [39]

    Watersplatting: Fast underwater 3d scene reconstruction using gaussian splatting, 2024

    Huapeng Li, Wenxuan Song, Tianao Xu, Alexandre Elsig, and Jonas Kulhanek. Watersplatting: Fast underwater 3d scene reconstruction using gaussian splatting, 2024. 7

    2024

  32. [40]

    A stepwise domain adaptive segmentation net- work with covariate shift alleviation for remote sensing im- agery

    Jiaojiao Li, Shunyao Zi, Rui Song, Yunsong Li, Yinlin Hu, and Qian Du. A stepwise domain adaptive segmentation net- work with covariate shift alleviation for remote sensing im- agery. IEEE Transactions on Geoscience and Remote Sens- ing, 60:1–15, 2022. 5

    2022

  33. [41]

    End-to-end learned single lens design using fast differentiable ray tracing

    Zongling Li, Qingyu Hou, Zhipeng Wang, Fanjiao Tan, Jin Liu, and Wei Zhang. End-to-end learned single lens design using fast differentiable ray tracing. Opt. Lett., 46(21):5453– 5456, 2021. 3

    2021

  34. [42]

    Ellis R. Loew. A third, ultraviolet-sensitive, visual pigment in the tokay gecko (gekko gekko). Vision Research, 34(11): 1427–1431, 1994. The Biology of Ultraviolet Reception. 2, 4

    1994

  35. [43]

    Lucas and Zhexi Luo

    Spencer G. Lucas and Zhexi Luo. Adelobasileus from the upper triassic of west texas: the oldest mammal. Journal of Vertebrate Paleontology, 13(3):309–334, 1993. 2

    1993

  36. [44]

    A jurassic eutherian mammal and divergence of marsupials and placentals

    Zhe-Xi Luo, Chong-Xi Yuan, Qing-Jin Meng, and Qiang Ji. A jurassic eutherian mammal and divergence of marsupials and placentals. Nature, 476(7361):442–445, 2011. 1

    2011

  37. [45]

    Visual Gene Expression Reveals a cone-to-rod Developmental Progression in Deep-Sea Fishes

    Nik Lup ˇse, Fabio Cortesi, Marko Freese, Lasse Marohn, Jan-Dag Pohlmann, Klaus Wysujack, Reinhold Hanel, and Zuzana Musilova. Visual Gene Expression Reveals a cone-to-rod Developmental Progression in Deep-Sea Fishes. Molecular Biology and Evolution, 38(12):5664–5677, 2021. 5

    2021

  38. [46]

    Advancing scientific discovery with the aid of robotics

    Amos Matsiko. Advancing scientific discovery with the aid of robotics. Science Robotics, 9(95):eadt3842, 2024. 3

    2024

  39. [47]

    Measuring the spec- tral response with a set of interference filters

    Christian Mauer and Dietmar Wueller. Measuring the spec- tral response with a set of interference filters. InDigital pho- tography V, pages 235–244. SPIE, 2009. 3

    2009

  40. [48]

    All-optical nonlinear activation function for photonic neural networks

    Mario Miscuglio, Armin Mehrabian, Zibo Hu, Shaimaa I Azzam, Jonathan George, Alexander V Kildishev, Matthew Pelton, and V olker J Sorger. All-optical nonlinear activation function for photonic neural networks. Optical Materials Express, 8(12):3851–3863, 2018. 3

    2018

  41. [49]

    Zuzana Musilova, Fabio Cortesi, Michael Matschiner, Wayne I. L. Davies, Jagdish Suresh Patel, Sara M. Stieb, Fanny de Busserolles, Martin Malmstrøm, Ole K. Tørresen, Celeste J. Brown, Jessica K. Mountford, Reinhold Hanel, Deborah L. Stenkamp, Kjetill S. Jakobsen, Karen L. Car-...

    2019

  42. [50]

    An integrated brain-machine interface platform with thousands of channels

    Elon Musk. An integrated brain-machine interface platform with thousands of channels. J Med Internet Res , 21(10): e16194, 2019. 3, 9

    2019

  43. [51]

    Deeply learned filter response functions for hyperspectral reconstruction

    Shijie Nie, Lin Gu, Yinqiang Zheng, Antony Lam, Nobutaka Ono, and Imari Sato. Deeply learned filter response functions for hyperspectral reconstruction. In 2018 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 4767–4776, 2018. 2, 3, 8

    2018

  44. [52]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  45. [53]

    Wi- dayati, Bambang Suryobroto, Migaku Teramoto, Yusuke 10 Mori, Kunitoshi Nagano, Kazuo Fujita, Hika Kuroshima, and Toshikazu Hasegawa

    Atsuko Saito, Akichika Mikami, Shoji Kawamura, Yoshikazu Ueno, Chihiro Hiramatsu, Kanthi A. Wi- dayati, Bambang Suryobroto, Migaku Teramoto, Yusuke 10 Mori, Kunitoshi Nagano, Kazuo Fujita, Hika Kuroshima, and Toshikazu Hasegawa. Advantage of dichromats over trichromats in disc...

    2005

  46. [54]

    SCHNEEWEIS and J.L

    D.M. SCHNEEWEIS and J.L. SCHNAPF. Noise and light adaptation in rods of the macaque monkey. Visual Neuro- science, 17(5):659–666, 2000. 5

    2000

  47. [55]

    Schott, Rayna C

    Ryan K. Schott, Rayna C. Bell, Ellis R. Loew, Kate N. Thomas, David J. Gower, Jeffrey W. Streicher, and Matthew K. Fujita. Transcriptomic evidence for visual adap- tation during the aquatic to terrestrial metamorphosis in leop- ard frogs. BMC Biology, 20(1):138, 2022. 2, 4

    2022

  48. [56]

    Diver- sity and Evolution of Frog Visual Opsins: Spectral Tuning and Adaptation to Distinct Light Environments

    Ryan K Schott, Matthew K Fujita, Jeffrey W Streicher, David J Gower, Kate N Thomas, Ellis R Loew, Abra- ham G Bamba Kaya, Gabriela B Bittencourt-Silva, C Guill- herme Becker, Diego Cisneros-Heredia, Simon Clulow, Ma- teo Davila, Jr Firneno, Thomas J, C ´elio F B Haddad, Sunita...

    2024

  49. [57]

    I. R. Schwab. The evolution of eyes: major steps. the keeler lecture 2017: centenary of keeler ltd. Eye, 32(2):302–313,

    2017

  50. [58]

    Loen: Lensless opto-electronic neural network empowered ma- chine vision

    Wanxin Shi, Zheng Huang, Honghao Huang, Chengyang Hu, Minghua Chen, Sigang Yang, and Hongwei Chen. Loen: Lensless opto-electronic neural network empowered ma- chine vision. Light: Science & Applications , 11(1):121,

  51. [59]

    Split-aperture 2-in-1 computational cameras

    Zheng Shi, Ilya Chugunov, Mario Bijelic, Geoffroi C ˆot´e, Ji- woon Yeom, Qiang Fu, Hadi Amata, Wolfgang Heidrich, and Felix Heide. Split-aperture 2-in-1 computational cameras. ACM Trans. Graph., 43(4), 2024. 3

    2024

  52. [60]

    Shu, H.-L

    D.-G. Shu, H.-L. Luo, S. Conway Morris, X.-L. Zhang, S.-X. Hu, L. Chen, J. Han, M. Zhu, Y . Li, and L.-Z. Chen. Lower cambrian vertebrates from south china. Nature, 402(6757): 42–46, 1999. 1

    1999

  53. [61]

    Colour blindness may have hidden advan- tages

    Tom Simonite. Colour blindness may have hidden advan- tages. Nature, 2005. 3, 6

    2005

  54. [62]

    End-to-end complex lens design with differen- tiable ray tracing

    Qilin Sun, Congli Wang, Fu Qiang, Dun Xiong, and Heidrich Wolfgang. End-to-end complex lens design with differen- tiable ray tracing. ACM Transactions on Graphics (TOG) , 40(4), 2021. 3

    2021

  55. [63]

    Enlightened experimentation: The new im- perative for innovation, 2001

    Stefan Thomke. Enlightened experimentation: The new im- perative for innovation, 2001. 3, 9

    2001

  56. [64]

    Neural nano-optics for high-quality thin lens imag- ing

    Ethan Tseng, Shane Colburn, James Whitehead, Luocheng Huang, Seung-Hwan Baek, Arka Majumdar, and Felix Heide. Neural nano-optics for high-quality thin lens imag- ing. Nature Communications, 12(1):6493, 2021. 3

    2021

  57. [65]

    Differentiable compound optics and processing pipeline op- timization for end-to-end camera design

    Ethan Tseng, Ali Mosleh, Fahim Mannan, Karl St-Arnaud, Avinash Sharma, Yifan Peng, Alexander Braun, Derek Nowrouzezahrai, Jean-Francois Lalonde, and Felix Heide. Differentiable compound optics and processing pipeline op- timization for end-to-end camera design. ACM Transaction...

    2021

  58. [66]

    Harrison, Greg O’Grady, and Eric J

    Chris Varghese, Ewen M. Harrison, Greg O’Grady, and Eric J. Topol. Artificial intelligence in surgery. Nature Medicine, 30(5):1257–1268, 2024. 8

    2024

  59. [67]

    Ecology and evolution of primate colour vision

    Misha V orobyev. Ecology and evolution of primate colour vision. Clinical and Experimental Optometry, 87(4-5):230– 238, 2004. 2

    2004

  60. [68]

    Inference in artificial intel- ligence with deep optics and photonics

    Gordon Wetzstein, Aydogan Ozcan, Sylvain Gigan, Shanhui Fan, Dirk Englund, Marin Solja ˇci´c, Cornelia Denz, David A B Miller, and Demetri Psaltis. Inference in artificial intel- ligence with deep optics and photonics. Nature, 588(7836): 39—47, 2020. 3

    2020

  61. [69]

    ´Alvarez, and Ping Luo

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jos´e M. ´Alvarez, and Ping Luo. Segformer: Simple and ef- ficient design for semantic segmentation with transformers. CoRR, abs/2105.15203, 2021. 5

    2021 arXiv

  62. [70]

    Condon, and Junyuan Chen

    Chuan Yang, Xian-Hua Li, Maoyan Zhu, Daniel J. Condon, and Junyuan Chen. Geochronological constraint on the cam- brian chengjiang biota, south china. Journal of the Geologi- cal Society, 175(4):659–666, 2018. 1

    2018

  63. [71]

    Molecular evolution of vertebrate visual pigments

    Shozo Yokoyama. Molecular evolution of vertebrate visual pigments. Progress in Retinal and Eye Research, 19(4):385– 419, 2000. 3, 5, 6

    2000

  64. [72]

    The Molecular Ba- sis of Adaptive Evolution of Squirrelfish Rhodopsins.Molec- ular Biology and Evolution, 21(11):2071–2078, 2004

    Shozo Yokoyama and Naomi Takenaka. The Molecular Ba- sis of Adaptive Evolution of Squirrelfish Rhodopsins.Molec- ular Biology and Evolution, 21(11):2071–2078, 2004. 3, 5, 7

    2004

  65. [73]

    Factor space and spectrum for medical hyper- spectral image segmentation

    Boxiang Yun, Qingli Li, Lubov Mitrofanova, Chunhua Zhou, and Yan Wang. Factor space and spectrum for medical hyper- spectral image segmentation. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 , pages 152–162, Cham, 2023. Springer Nature Switzerland. 3

    2023

  66. [74]

    A multidimensional choledoch database and benchmarks for cholangiocarcinoma diagnosis

    Qing Zhang, Qingli Li, Guanzhen Yu, Li Sun, Mei Zhou, and Junhao Chu. A multidimensional choledoch database and benchmarks for cholangiocarcinoma diagnosis. IEEE Access, 7:149414–149421, 2019. 8 11 Paleoinspired Vision: From Exploring Colour Vision Evolution to Inspiring Camer...

    2019

  67. [75]

    Layers after Opsin Layer Figure 5

    Implementation Details 7.1. Layers after Opsin Layer Figure 5. Architecture of the network. Noise Layer The noise layer is applied after each chan- nel of the opsin layer’s output to incorporate the noise present in opsins. Let the output of the opsin layer for chan- nel c be ...

  68. [76]

    to update the parameters of the opsin layer, encoder, and decoder. In training to reconstruct the evolutionary transition to dichromatic vision in mammals (Sec.4.2), we use the Adam optimizer to train the opsin layer with a base learning rate of 2 × 10−2 and the encoder and de...

  69. [77]

    Supplementary Results for Evolution 8.1. Multi Rod Opsins: Result of Bioluminescence We simulate deep-sea environment with bioluminescence by maintaining the region of a specific label not adjusted by the underwater diffusion, representing a simplified model of bioluminescence...

  70. [78]

    In this section, we present ad- ditional potential camera designs to further demonstrate the versatility and broad applicability of our approach

    Supplementary Results for Camera Design In the main text, due to space limitations, we used the max- imum sensitivity wavelengths of human opsins (580, 540 and 425 nm) as a baseline to initiate the training process for the opsin layer, effectively demonstrating the validity of...

  71. [79]

    The opsin layer is integrated into the U-Net architec- ture, and the opsin layer, backbone, and U-Net decoder are 3 Table 16

    and a U-Net decoder, similar to the baseline model in [73]. The opsin layer is integrated into the U-Net architec- ture, and the opsin layer, backbone, and U-Net decoder are 3 Table 16. Camera Specialized for Cancer Detection. Filters λ1max λ2max λ3max λ4max mIoU % R 607.46 - ...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.