Abstract

Division of focal plane (DoFP) polarimeter is widely used in polarization imaging sensors. The periodically arranged micro-polarizers integrated on the focal plane ensure its outstanding real-time performance, but reduce the spatial resolution of output images and further affect the calculation of polarization parameters. In this paper, a four-layer, end-to-end fully convolutional neural network called Fork-Net is proposed, which aims to directly improve the imaging quality of three polarization properties: intensity (i.e., S0), degree of linear polarization (DoLP), and angle of polarization (AoP), rather than focusing on reducing the interpolation error of intensity images of different polarization orientations. The Fork-Net accepts raw mosaic images as input and directly outputs S0, DoLP, and AoP. It is also trained with a customized loss function. The experimental results show that compared with existing methods, the proposed one achieves the highest peak signal-to-noise ratio (PSNR) and prominent visual quality on output images.

© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

Full Article  |  PDF Article
OSA Recommended Articles
Demosaicking DoFP images using Newton’s polynomial interpolation and polarization difference model

Ning Li, Yongqiang Zhao, Quan Pan, and Seong G. Kong
Opt. Express 27(2) 1376-1391 (2019)

Gradient-based interpolation method for division-of-focal-plane polarimeters

Shengkui Gao and Viktor Gruev
Opt. Express 21(1) 1137-1151 (2013)

Image interpolation for division of focal plane polarimeters with intensity correlation

Junchao Zhang, Haibo Luo, Bin Hui, and Zheng Chang
Opt. Express 24(18) 20799-20807 (2016)

References

  • View by:
  • |
  • |
  • |

  1. M. Sarkar, D. S. S. San Segundo Bello, C. van Hoof, and A. Theuwissen, “Integrated polarization analyzing cmos image sensor for material classification,” IEEE Sensors J. 11, 1692–1703 (2011).
    [Crossref]
  2. D. H. Goldstein, “Polarimetric characterization of federal standard paints,” in Polarization Analysis, Measurement, and Remote Sensing III, vol. 4133 (International Society for Optics and Photonics, 2000), pp. 112–124.
    [Crossref]
  3. Y. Aron and Y. Gronau, “Polarization in the lwir: a method to improve target aquisition,” in Infrared Technology and Applications XXXI, vol. 5783 (International Society for Optics and Photonics, 2005), pp. 653–662.
    [Crossref]
  4. B. Huang, T. Liu, H. Hu, J. Han, and M. Yu, “Underwater image recovery considering polarization effects of objects,” Opt. Express 24, 9826–9838 (2016).
    [Crossref] [PubMed]
  5. E. Salomatina-Motts, V. Neel, and A. Yaroslavskaya, “Multimodal polarization system for imaging skin cancer,” Opt. Spectrosc. 107, 884–890 (2009).
    [Crossref]
  6. J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, “Review of passive imaging polarimetry for remote sensing applications,” Appl. Opt. 45, 5453–5469 (2006).
    [Crossref] [PubMed]
  7. R. Perkins and V. Gruev, “Signal-to-noise analysis of stokes parameters in division of focal plane polarimeters,” Opt. Express 18, 25815–25824 (2010).
    [Crossref] [PubMed]
  8. S. Gao and V. Gruev, “Bilinear and bicubic interpolation methods for division of focal plane polarimeters,” Opt. Express 19, 26161–26173 (2011).
    [Crossref]
  9. S. Gao and V. Gruev, “Gradient based interpolation for division of focal plane polarization imaging sensors,” in 2012 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2012), pp. 1855–1858.
    [Crossref]
  10. J. Zhang, H. Luo, B. Hui, and Z. Chang, “Image interpolation for division of focal plane polarimeters with intensity correlation,” Opt. Express 24, 20799–20807 (2016).
    [Crossref] [PubMed]
  11. J. Zhang, W. Ye, A. Ahmed, Z. Qiu, Y. Cao, and X. Zhao, “A novel smoothness-based interpolation algorithm for division of focal plane polarimeters,” in 2017 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2017), pp. 1–4.
  12. J. Zhang, H. Luo, R. Liang, A. Ahmed, X. Zhang, B. Hui, and Z. Chang, “Sparse representation-based demosaicing method for microgrid polarimeter imagery,” Opt. Lett. 43, 3265–3268 (2018).
    [Crossref] [PubMed]
  13. J. Zhang, J. Shao, H. Luo, X. Zhang, B. Hui, Z. Chang, and R. Liang, “Learning a convolutional demosaicing network for microgrid polarimeter imagery,” Opt. Lett. 43, 4534–4537 (2018).
    [Crossref] [PubMed]
  14. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention (MICCAI), (Springer International Publishing, Cham, 2015), pp. 234–241.
  15. C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis Mach. Intell. 38, 295–307 (2016).
    [Crossref]
  16. V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), (2010), pp. 807–814.
  17. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing 13, 600–612 (2004).
    [Crossref] [PubMed]
  18. H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for neural networks for image processing,” arXiv preprint arXiv:1511.08861 (2015).
  19. K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in The IEEE International Conference on Computer Vision (ICCV), (2015), pp. 1026–1034.
  20. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).

2018 (2)

2016 (3)

2011 (2)

S. Gao and V. Gruev, “Bilinear and bicubic interpolation methods for division of focal plane polarimeters,” Opt. Express 19, 26161–26173 (2011).
[Crossref]

M. Sarkar, D. S. S. San Segundo Bello, C. van Hoof, and A. Theuwissen, “Integrated polarization analyzing cmos image sensor for material classification,” IEEE Sensors J. 11, 1692–1703 (2011).
[Crossref]

2010 (1)

2009 (1)

E. Salomatina-Motts, V. Neel, and A. Yaroslavskaya, “Multimodal polarization system for imaging skin cancer,” Opt. Spectrosc. 107, 884–890 (2009).
[Crossref]

2006 (1)

2004 (1)

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing 13, 600–612 (2004).
[Crossref] [PubMed]

Ahmed, A.

J. Zhang, H. Luo, R. Liang, A. Ahmed, X. Zhang, B. Hui, and Z. Chang, “Sparse representation-based demosaicing method for microgrid polarimeter imagery,” Opt. Lett. 43, 3265–3268 (2018).
[Crossref] [PubMed]

J. Zhang, W. Ye, A. Ahmed, Z. Qiu, Y. Cao, and X. Zhao, “A novel smoothness-based interpolation algorithm for division of focal plane polarimeters,” in 2017 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2017), pp. 1–4.

Aron, Y.

Y. Aron and Y. Gronau, “Polarization in the lwir: a method to improve target aquisition,” in Infrared Technology and Applications XXXI, vol. 5783 (International Society for Optics and Photonics, 2005), pp. 653–662.
[Crossref]

Ba, J.

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).

Bovik, A. C.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing 13, 600–612 (2004).
[Crossref] [PubMed]

Brox, T.

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention (MICCAI), (Springer International Publishing, Cham, 2015), pp. 234–241.

Cao, Y.

J. Zhang, W. Ye, A. Ahmed, Z. Qiu, Y. Cao, and X. Zhao, “A novel smoothness-based interpolation algorithm for division of focal plane polarimeters,” in 2017 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2017), pp. 1–4.

Chang, Z.

Chenault, D. B.

Dong, C.

C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis Mach. Intell. 38, 295–307 (2016).
[Crossref]

Fischer, P.

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention (MICCAI), (Springer International Publishing, Cham, 2015), pp. 234–241.

Frosio, I.

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for neural networks for image processing,” arXiv preprint arXiv:1511.08861 (2015).

Gallo, O.

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for neural networks for image processing,” arXiv preprint arXiv:1511.08861 (2015).

Gao, S.

S. Gao and V. Gruev, “Bilinear and bicubic interpolation methods for division of focal plane polarimeters,” Opt. Express 19, 26161–26173 (2011).
[Crossref]

S. Gao and V. Gruev, “Gradient based interpolation for division of focal plane polarization imaging sensors,” in 2012 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2012), pp. 1855–1858.
[Crossref]

Goldstein, D. H.

D. H. Goldstein, “Polarimetric characterization of federal standard paints,” in Polarization Analysis, Measurement, and Remote Sensing III, vol. 4133 (International Society for Optics and Photonics, 2000), pp. 112–124.
[Crossref]

Goldstein, D. L.

Gronau, Y.

Y. Aron and Y. Gronau, “Polarization in the lwir: a method to improve target aquisition,” in Infrared Technology and Applications XXXI, vol. 5783 (International Society for Optics and Photonics, 2005), pp. 653–662.
[Crossref]

Gruev, V.

Han, J.

He, K.

C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis Mach. Intell. 38, 295–307 (2016).
[Crossref]

K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in The IEEE International Conference on Computer Vision (ICCV), (2015), pp. 1026–1034.

Hinton, G. E.

V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), (2010), pp. 807–814.

Hu, H.

Huang, B.

Hui, B.

Kautz, J.

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for neural networks for image processing,” arXiv preprint arXiv:1511.08861 (2015).

Kingma, D. P.

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).

Liang, R.

Liu, T.

Loy, C. C.

C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis Mach. Intell. 38, 295–307 (2016).
[Crossref]

Luo, H.

Nair, V.

V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), (2010), pp. 807–814.

Neel, V.

E. Salomatina-Motts, V. Neel, and A. Yaroslavskaya, “Multimodal polarization system for imaging skin cancer,” Opt. Spectrosc. 107, 884–890 (2009).
[Crossref]

Perkins, R.

Qiu, Z.

J. Zhang, W. Ye, A. Ahmed, Z. Qiu, Y. Cao, and X. Zhao, “A novel smoothness-based interpolation algorithm for division of focal plane polarimeters,” in 2017 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2017), pp. 1–4.

Ren, S.

K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in The IEEE International Conference on Computer Vision (ICCV), (2015), pp. 1026–1034.

Ronneberger, O.

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention (MICCAI), (Springer International Publishing, Cham, 2015), pp. 234–241.

Salomatina-Motts, E.

E. Salomatina-Motts, V. Neel, and A. Yaroslavskaya, “Multimodal polarization system for imaging skin cancer,” Opt. Spectrosc. 107, 884–890 (2009).
[Crossref]

San Segundo Bello, D. S. S.

M. Sarkar, D. S. S. San Segundo Bello, C. van Hoof, and A. Theuwissen, “Integrated polarization analyzing cmos image sensor for material classification,” IEEE Sensors J. 11, 1692–1703 (2011).
[Crossref]

Sarkar, M.

M. Sarkar, D. S. S. San Segundo Bello, C. van Hoof, and A. Theuwissen, “Integrated polarization analyzing cmos image sensor for material classification,” IEEE Sensors J. 11, 1692–1703 (2011).
[Crossref]

Shao, J.

Shaw, J. A.

Sheikh, H. R.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing 13, 600–612 (2004).
[Crossref] [PubMed]

Simoncelli, E. P.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing 13, 600–612 (2004).
[Crossref] [PubMed]

Sun, J.

K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in The IEEE International Conference on Computer Vision (ICCV), (2015), pp. 1026–1034.

Tang, X.

C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis Mach. Intell. 38, 295–307 (2016).
[Crossref]

Theuwissen, A.

M. Sarkar, D. S. S. San Segundo Bello, C. van Hoof, and A. Theuwissen, “Integrated polarization analyzing cmos image sensor for material classification,” IEEE Sensors J. 11, 1692–1703 (2011).
[Crossref]

Tyo, J. S.

van Hoof, C.

M. Sarkar, D. S. S. San Segundo Bello, C. van Hoof, and A. Theuwissen, “Integrated polarization analyzing cmos image sensor for material classification,” IEEE Sensors J. 11, 1692–1703 (2011).
[Crossref]

Wang, Z.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing 13, 600–612 (2004).
[Crossref] [PubMed]

Yaroslavskaya, A.

E. Salomatina-Motts, V. Neel, and A. Yaroslavskaya, “Multimodal polarization system for imaging skin cancer,” Opt. Spectrosc. 107, 884–890 (2009).
[Crossref]

Ye, W.

J. Zhang, W. Ye, A. Ahmed, Z. Qiu, Y. Cao, and X. Zhao, “A novel smoothness-based interpolation algorithm for division of focal plane polarimeters,” in 2017 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2017), pp. 1–4.

Yu, M.

Zhang, J.

Zhang, X.

Zhao, H.

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for neural networks for image processing,” arXiv preprint arXiv:1511.08861 (2015).

Zhao, X.

J. Zhang, W. Ye, A. Ahmed, Z. Qiu, Y. Cao, and X. Zhao, “A novel smoothness-based interpolation algorithm for division of focal plane polarimeters,” in 2017 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2017), pp. 1–4.

Appl. Opt. (1)

IEEE Sensors J. (1)

M. Sarkar, D. S. S. San Segundo Bello, C. van Hoof, and A. Theuwissen, “Integrated polarization analyzing cmos image sensor for material classification,” IEEE Sensors J. 11, 1692–1703 (2011).
[Crossref]

IEEE transactions on image processing (1)

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing 13, 600–612 (2004).
[Crossref] [PubMed]

IEEE Transactions on Pattern Analysis Mach. Intell. (1)

C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis Mach. Intell. 38, 295–307 (2016).
[Crossref]

Opt. Express (4)

Opt. Lett. (2)

Opt. Spectrosc. (1)

E. Salomatina-Motts, V. Neel, and A. Yaroslavskaya, “Multimodal polarization system for imaging skin cancer,” Opt. Spectrosc. 107, 884–890 (2009).
[Crossref]

Other (9)

S. Gao and V. Gruev, “Gradient based interpolation for division of focal plane polarization imaging sensors,” in 2012 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2012), pp. 1855–1858.
[Crossref]

J. Zhang, W. Ye, A. Ahmed, Z. Qiu, Y. Cao, and X. Zhao, “A novel smoothness-based interpolation algorithm for division of focal plane polarimeters,” in 2017 IEEE International Symposium on Circuits and Systems (ISCAS), (IEEE, 2017), pp. 1–4.

O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Medical Image Computing and Computer-Assisted Intervention (MICCAI), (Springer International Publishing, Cham, 2015), pp. 234–241.

V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10), (2010), pp. 807–814.

D. H. Goldstein, “Polarimetric characterization of federal standard paints,” in Polarization Analysis, Measurement, and Remote Sensing III, vol. 4133 (International Society for Optics and Photonics, 2000), pp. 112–124.
[Crossref]

Y. Aron and Y. Gronau, “Polarization in the lwir: a method to improve target aquisition,” in Infrared Technology and Applications XXXI, vol. 5783 (International Society for Optics and Photonics, 2005), pp. 653–662.
[Crossref]

H. Zhao, O. Gallo, I. Frosio, and J. Kautz, “Loss functions for neural networks for image processing,” arXiv preprint arXiv:1511.08861 (2015).

K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in The IEEE International Conference on Computer Vision (ICCV), (2015), pp. 1026–1034.

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014).

Cited By

OSA participates in Crossref's Cited-By Linking service. Citing articles from OSA journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (8)

Fig. 1
Fig. 1 The schematic diagram of an typical DoFP polarimeter.
Fig. 2
Fig. 2 The architecture diagram of proposed Fork-Net. The transparent blocks in the middle actually correspond to feature maps from different layers.
Fig. 3
Fig. 3 Schematic diagram of the polarization image acquisition experimental device.
Fig. 4
Fig. 4 PSNRs vs training epochs curves of S 0, DoLP and AoP for networks with different architectures. (a) Curves of S   0. (b) Curves of DoLP. (c) Curves of AoP.
Fig. 5
Fig. 5 PSNRs vs training epochs curves of S 0, DoLP and AoP for networks with different kernel sizes. (a) Curves of S   0. (b) Curves of DoLP. (c) Curves of AoP.
Fig. 6
Fig. 6 AoP images which indicates the improvement brought by variance constraint term. First row shows the grayscale images and the second row shows the heatmaps of AoP. (a)AoP images output from Fork-Net trained with common MAE loss function. (b) AoP images output from Fork-Net trained with the proposed customized loss function. (c) Original AoP images.
Fig. 7
Fig. 7 Feature maps output from layers alone the DoLP branch. (Due to the limited space, for the three layers in the middle, only four maps are selected to be shown as examples.)
Fig. 8
Fig. 8 S   0, DoLP grayscale images and AoP heatmaps obtained through different methods. Images in the pink boxes show the magnification of local regions.

Tables (3)

Tables Icon

Table 1 PSNRs of Networks with Different Architectures and Loss Functions on Test Set

Tables Icon

Table 2 PSNRs on Test Set and Parameter Numbers of Fork-Nets with Different Kernel Sizes

Tables Icon

Table 3 The PSNRs for Different Methods on Test Set

Equations (6)

Equations on this page are rendered with MathJax. Learn more.

L o s s = 1 N n = 1 N [ 1 W H ( λ 1 Y ^ S 0 ( n ) Y S 0 ( n ) 1 + λ 2 Y ^ D o L P ( n ) Y D o L P ( n ) 1 + λ 3 Y ^ A o P ( n ) Y A o P ( n ) 1 ) λ 4 log C ]
C = 2 σ ^ A o P σ A o P + ( k M A X I ) 2 σ ^ A o P 2 + σ A o P 2 + ( k M A X I ) 2
I n t e n s i t y = 1 2 ( I 0 + I 45 + I 90 + I 135 )
D o L P = ( I 0 I 90 ) 2 + ( I 45 I 135 ) 2 I n t e n s i t y
A o P = 1 2 arctan ( I 45 I 135 I 0 I 90 )
P S N R = 10 log 10 ( M A X I 2 M S E )

Metrics