Journal Paper

Paper Information

Journal: IRANIAN JOURNAL OF ELECTRICAL AND COMPUTER ENGINEERING (IJECE) | Year:0 | Volume:9 | Issue:2 | Start Page:127 | End Page:133

video

sound

Persian Version

View:

24,513

Download:

7,887

Cites:

Information Journal Paper

Title

A COMPLEMENTARY METHOD FOR PREVENTING HIDDEN NEURONS’ SATURATION IN FEED FORWARD NEURAL NETWORKS TRAINING

Writers

MOALEM P. | AYOUGHI S.A.

Pages

 Start Page 127 | End Page 133

Keywords

Abstract

 In feed forward neural networks, hidden layer neurons’ saturation conditions, which are the cause of flat spots on the error surface, is one of the main disadvantages of any conventional gradient descent learning algorithm. In this paper, we propose a novel complementary scheme for the learning based on a suitable combination of anti saturated idden neurons learning process and accelerating methods like the MOMENTUM TERM and the parallel tangent technique. In our proposed method, a NORMALIZED SATURATION CRITERION (NSC) of hidden neurons, which is introduced in this paper, is monitored during learning process. When the NSC is higher than a specified threshold, it means that the algorithm moves towards a flat spot as the hidden neurons fall into saturation condition. In this case, in order to suppress the saturation of hidden neurons, a conventional gradient descent learning method can be accompanied by the proposed complementary gradient descent saturation prevention scheme. When the NSC assumes small values, no saturation detected and the network operates in its normal condition. Therefore, application of a saturation prevention scheme is not recommended. We have evaluated the proposed complementary method in accompaniment to the gradient descent plus momentum and parallel tangent, two conventional improvements on learning methods. We have recorded remarkable improvements in convergence success as well as generalization in some well known benchmarks.

Cites

  • No record.
  • References

  • No record.
  • Related Journal Papers

  • No record.
  • Related Seminar Papers

  • No record.
  • Related Plans

  • No record.





  • Advertising

    مرکز اطلاعات علمی SID
    مرکز اطلاعات علمی SID
    مرکز اطلاعات علمی SID
    مرکز اطلاعات علمی SID
    مرکز اطلاعات علمی SID
    مرکز اطلاعات علمی SID
    مرکز اطلاعات علمی SID
    مرکز اطلاعات علمی SID