Airborne gamma ray spectrometry with its applications such as identification of lithology, facies and depositional environment, depth correlations and core-log integration, mineralogy, geochemistry, cyclo-stratigraphic analysis, environmental monitoring, especially nuclear site surveillance and emergency response recently hanged in the place of honor for its fast, precise and accurate outcomes \cite{Lutter_2018}. The risks of a nuclear power plants incident consequences, require instant measurements of radiation dose or radionuclide pollution within the potential exposure area. Online interpretation of the outcomes of radiometric spectrometry will effectively impress the measures to control or decrease the exposure consequences \cite{Gupta_2012}. AGRS is the most common technique to accelerate the radiometric spectrometry \cite{Grasty_1974}. Utilization advanced computer science procedures, by their numerical optimization, can improve the accuracy of AGRS \cite{Grasty_1974} \cite{_vec_2016}.
The most emphasis focusing theme of this manuscript is the fast optimization of AGRS based on enhancement the updating formula in its training. This optimization was applied on a multilayer neural network, for AGRS, to improve convergence rate and performance by new updating criteria, which are inherited from gamma radiation stochastic properties\cite{Tatsumi_2015}. The proposed ANN was trained by an advanced numerical unconstrained nonlinear optimization, the quasi-Newton method \cite{Price_2018} with novel Enhanced Rezaei-Ashoor-Sarkhosh (ERAS) updating formula. This training technique is an extension for optimization process of quasi-Newton procedures that can solve unconstrained optimization problems such as AGRS. The output of airborne gamma ray spectrometry typically arises of many parameters, such as intensity of a measured radiation, the detector response and the distance between the source and the detector, those all extremely affect the measured spectrum \cite{You_2014} . The proposed updating formula, ERAS, is chosen based on the \cite{Stalter_2012}.
This paper optimizes the training of real-time airborne gamma-ray spectrometry which may be utilized in an automatic environmental radiation surveillance network to be used on board a light Unmanned Aerial Vehicle (UAV)\cite{_vec_2016}. A trained ANN by a quasi-Newton algorithm [4] which uses updates is employed for airborne gamma ray spectrometry evaluating in different altitudes. To resolve the problems of gradient \cite{Amini_2010} , we explored an advanced updating formula ERAS.
Error optimization may provide some numerical challenges in neural network training, due to the numerous parameters and developed extremely powerful approaches of function optimization for neural networks in recent years . This paper utilized an advanced neural network training approach for AGRS, to achieve less convergence rate and much accuracy, based on gamma ray stochastic inherent trait. In this study we used the second-order stochastic Hessian or Hessian-free optimization technique with negative curvature direction detection, for ANN training purpose. Hessian-free is an influential optimization method which has two major modules: First, it models the quadratic optimization problem indirectly, the problem of optimization a convex quadratic function, subject to some linear constraints. This specific form of nonlinear programming or Quadratic Programming (QP) was implemented using Hessian-vector products with the quasi-Newton matrix \cite{Pankratov_2015}; and second, it uses the generalized, truncated or preconditioned conjugate gradient iterations for solving the sub-problems, regarding the generalized dogleg process, where the inaccurate quasi-Newton step is taken asymptotically.
Lately, numerous stochastic quasi-Newton techniques have been offered for large-scale learning machines. Three challenges were mentioned for quasi-Newton methods: 1: the computation of Hessian-vector products in updating formula of training. 2: the difference in their update rate. 3: the applicability range of such algorithms to nonconvex problems. The additivity of stochastic quasi-Newton procedures provides robustness and independency of the quality of the curvature information for AGRS. The proposed stable updating formula may be utilized to perform convergence rate and performance of neural network, which was used and tested in AGRS in this study.
Numerous theoretical \cite{Moslemi_2017} and experimental studies have been done to guide intelligently the direction of photons originated from the object in order to increase SNR. Although the ground radiation measurement to extract the standard radiation map is not pragmatic, due to its circumscriptions such as low speed, long time operation, high cost for limited area and high risk at nuclear accidents. AGRS is the most noteworthy clarification for outdoor measurement and analysis for ground radiation maps, radionuclides type recognition, their density approximation and assessment. The typical methods for acquisition and processing of AGRS data is intensely dependent to survey parameters such as flying height, profile separation, detector volume, energy window width, standards, calibration and lastly the method of the data analysis \cite{Kluso__2010,Lavi_2004}.
Direct comparison of the outcomes of different AGRS assessment is very difficult due to their measurement parameter dependencies. In order to moderate these dependencies, the neural network modeling is applied in this study so far. Thanks to real-time output and ability to model complicated systems such as AGRS, ANNs which usually use to implement tasks such as prediction, segmentation, classification, visualization, evaluation, optimization, decision making and unknown approximate functions.
Adaptive neural networks are appropriate solution for modeling the endlessly varying environment, which is typical for AGRS and they have contributed in meaningful share of nuclear science and engineering aspects such as instrumentation for the detection and measurement of ionizing radiation; particle accelerators and their controls; nuclear medicine and its application; effects of radiation on materials, components, and systems; reactor instrumentation and controls; and measurement of radiation in space. To that effect neural networks with their different learning algorithms and architectures such as Feed-forward, Regulatory feedback, Radial basis function, Recurrent neural network, Physical and etc. have been used in the nuclear science and technologies \cite{Manuel_2013}.
The problem which was considered for this study is the AGRS optimization in different altitudes with different updating formula for ANN. The neural network was trained by different algorithms including quasi-Newton techniques, and then accuracy, validation, convergence speed and computational complexity of each, for AGRS data were reported. The used dataset has been taken by a 3" NaI detector and recorded by a1024 Multi Channel (MC) \cite{peter2018}. Various tests and their outcomes showed that our proposed updating formula, ERAS, is well applicable for airborne gamma ray spectrometry optimization.
The rest of this paper is organized as follows; using of neural network computational machines in AGRS is described in section II. Then, the proposed updating formula for training of ANN on stochastic gamma-ray measurement is presented in section III. The experimental procedures and outcomes are given in section IV, followed by provision of the conclusion in section V.
AGRS and Training
Environmental radioactivity measurement is a technique for mineral exploration, geological mapping and monitoring. Arrangements of rescue operations management and cooperation in nuclear accidents are completely based on the post accidents collected radiation information in the site and around that. The measuring of the radiation dose and exposure information on the ground after such accidents is a high-risk task in terms of the human immunity, safety and radiation protection. AGRS is the only safe and fast solution for gathering necessary data. The common commercial systems for AGRS implements a heavy setup of equipment with a high volume of scintillator crystal. AGRS typically consist of three detectors as different windows for main geochemical elements thorium, uranium and potassium plus on detector for whole concerned spectrum and one for background and space radiation. Such setup usually with more than 100 Kg weight needs a couple of operators. The airborne measurement system then requires a two engine helicopter for safety of flight over cities and demographic areas.
Improving the accuracy of AGRS by employing advanced computer science algorithms recently was reported by some new studies \cite{Pandey_2016}. Radiation monitoring includes the measurement of radiation dose or radionuclide pollution for motives associated to the assessment or control of radiation exposure or radioactive substances and the interpretation of the outcomes \cite{Henrichs_2011}. Radiation monitoring in nuclear events and accidents is a challenging duty. Therefore, to carry out high performance computing in the serious tasks, the advanced radiation data collection approaches are essential.
This manuscript optimized the AGRS based on ANN with a new updating criterion which is inherited from stochastic properties of gamma radiation. The proposed method is independent from detector selection, so that all gamma spectrometer configurations such as NaI, LaBr3 or HPGe detectors can be used. In our study, among all available scintillators, the most commonly used material, sodium iodide was chosen as the detector scintillator. The HPGe detector cannot result high performance in AGRS due to low energy deposition due to its maximum possible size, and the LaBr3 is not still costs effective in compare with NaI(Tl) commercially.
This study optimized the the gamma photon count evaluation based on AGRS data in different altitudes. Eight training approaches have been performed to investigate some parameters including their accuracy and convergence speed of the proposed AGRS. These advanced stochastic quasi-Newton techniques are efficient, robust, and scalable in neural network training and this paper customized them for AGRS application. Finally we have evaluated a new updating criterion, ERAS, which is inherited from stochastic properties of gamma radiation to improve convergence rate and performance of AGRS.
1) Levenberg-Marquardt BackPropagation (LMBP) \cite{Sapna_2012}
2) Scaled Conjugate Gradient BackPropagation (S-CGBP)
\cite{Nayak_2017}
3) Resilient Backpropagation (RBP)
\cite{Saputra_2017}
4) BFGS quasi-Newton backpropagation \cite{Silaban_2017}
5) Conjugate Gradient BackPropagation with Polak-Ribiére updates (CGBP-PR) \cite{Ghani_2017}
6) Conjugate Gradient BackPropagation with Fletcher-Reeves updates (CGBP-FR) \cite{Wanto_2017}
7) Conjugate Gradient BackPropagation with Hestenes–Stiefel updates (CGBP-HS) \cite{Gazi_Sharee_2014}
8) Conjugate Gradient BackPropagation with Dai–Yuan updates (CGBP-DY) \cite{Dai_2013}
9) Conjugate Gradient BackPropagation with Enhanced Rezaei-Ashoor-Sarkhosh updating (CGBP-ERAS)
The AGRS with ERAS Updating Criterion
The new ERAS training of ANN is defined in details for AGRS in this section. The multilayer ANN is trained by some advanced stochastic quasi-Newton techniques, LMBP \cite{Sapna_2012} , S-CGBP \cite{Nayak_2017} , RBP \cite{Saputra_2017} , BFGS \cite{Silaban_2017} , CGBP-PR \cite{Ghani_2017} , CGBP-FR \cite{Wanto_2017} , CGBP-HS \cite{Gazi_Sharee_2014} , CGBP-DY \cite{Dai_2013} and finally optimized for real time stochastic AGRS data with proposed ERAS. In these updating formulas which were used in AGRS, the convolutional and subsampling layers were merged into one layer, which simplifies the network architecture and finally an adaptive update creation [29] incorporated curvature information into stochastic approximation approaches for AGRS data. Noisy curvature estimations that have destructive effects on the robustness of the iterations are the mostly possible outcome of using “classical” quasi-Newton updating methods such as LMBP \cite{Sapna_2012} , S-CGBP \cite{Nayak_2017} , RBP \cite{Saputra_2017} , BFGS \cite{Silaban_2017} , CGBP-PR \cite{Ghani_2017} , CGBP-FR \cite{Wanto_2017} , CGBP-PB, so this paper introduce a new optimized updating formula, ERAS, for AGRS. This new update criterion encountered the problems of the robustness of training’s convergence and excessive computational complexity which were common troubles of traditional quasi-Newton updating methods for AGRS optimization. These complications were committed utilizing on gamma ray stochastic properties, as a part of novelty of this work. The rest of this section dived deep to mathematical description of quasi-Newton updating methods and the proposed optimized updating formula, ERAS.
The Newton's technique is a substitute to the Conjugate Gradient BackPropagation (CGBP) approaches for fast optimization. The basic step of Newton's process is
\(e^{i\pi}+1=0\)
\(x_(k+1)=x_k-A_k^(-1)g_k\) | (1) |
where is the Hessian matrix (second derivatives) of the performance index at the current values of the weights and biases. Newton's technique frequently converges faster than CGBP approaches. Inappropriately, it is complex and expensive to compute the Hessian matrix for ANNs. The quasi-Newton technique is a class of procedures based on Newton's process, which does not need calculation of second derivatives. The approximation of Hessian matrix is updated by quasi-Newton algorithm at each iteration based on a function of the gradient. The most popular update procedures of quasi-Newton technique are the LMBP \cite{Sapna_2012} , S-CGBP \cite{Nayak_2017} , RBP \cite{Saputra_2017} , BFGS \cite{Silaban_2017} , CGBP-PR \cite{Ghani_2017} , CGBP-FR \cite{Wanto_2017} , CGBP-HS \cite{Gazi_Sharee_2014} , CGBP-DY \cite{Dai_2013} update. In this study, these update procedures plus the new proposed update, CGBP-ERAS, are implemented in the ANN training routine for AGRS data.
Training the ANN with free parameters (weights and biases) is equivalent to optimizing a function of independent variables with AGRS data and it can be expressed as the mean squared error (MSE)
where is the number of output neurons, is the number of training patterns, is the number of training iterations, and are the actual and desired response of the -th output neuron due to the i-th counted gamma ray photons, respectively. Let be the -dimensional column vector containing all free parameters (i.e., adaptable weights) of the ANN at the -th iteration
where ‘ ’ denotes the transpose operator.
To optimize the error function in Equ. (2), the following update rule is applied iteratively, starting from an initial weight vector .
where is the weight-update vector, is a search direction, and is the step-length at the -th iteration.
There are various ways for computing the search direction and the step-length, ranging from the simple gradient descent to the more efficient CGBP and quasi-Newton methods. The simplest solution is to take a constant step-length, and set the search direction to the negative gradient, which is the direction of the steepest descent from any given point on the error surface; that
where is the gradient vector of the error function at the -th epoch. The gradient vector is an -dimensional column vector given by
where is the local gradient.
This formulation is often called the steepest descent algorithm or the gradient descent method. In a multilayer ANN network, the gradient vector can be computed very efficiently using quasi-Newton techniques. Gradient descent methods typically work fairly well during the early stages of the optimization process but unfortunately this method behaves poorly with airborne Gamma ray spectrometry data. So, in this study we employ not only the gradient, but also the curvature of the error surface, to minimize the error function of airborne Gamma ray spectrometry data. The rest of this section presents a number of such techniques such as LMBP \cite{Sapna_2012} , S-CGBP \cite{Nayak_2017} , RBP \cite{Saputra_2017} , BFGS \cite{Silaban_2017} , CGBP-PR \cite{Ghani_2017} , CGBP-FR \cite{Wanto_2017} , CGBP-HS \cite{Gazi_Sharee_2014} , CGBP-DY \cite{Dai_2013} and finally our proposed ERAS update procedure. They use batch training, in which any weight update is performed after the presentation of the airborne Gamma ray spectrometry data.
The first strategy can be the local adaptation where the temporal behavior of the partial derivative of the weight is used in the computation of the weight-update. The weight update rule is given by
where “ is the element-by-element product of two column vectors. The vector of adaptive momentum rate is taken as the vector of magnitude with respect to the error in the previous iteration.
The Conjugate Gradient BackPropagation (CGBP) method is another efficient optimization technique; it can minimize a quadratic error function of variables in steps. This method generates a search direction that is mutually conjugate to the previous search directions, with respect to a given positive definite matrix and finds the optimal point in that direction, using a line-search technique. Two search directions and are said to be mutually conjugate with respect to if the following condition is satisfied:
The next search direction is calculated as a linear combination of the previous direction and the current gradient, in such a way that the minimization steps in all previous directions are not interfered with. The next search direction can be determined as follows:
The variable is a scalar chosen so that becomes the -th conjugate direction. There are various ways for computing the scalar : each one generates a distinct nonlinear conjugate gradient method which has its own convergence property and numerical performance. Several formulae for computing have been proposed; the most notable ones are the following:
In this study, we propose a new hybrid CGBP technique by combining the good numerical performance of Polak–Ribière (PR) \cite{Ghani_2017} technique and the wonderful global convergence properties of Fletcher–Reeves (FR) \cite{Wanto_2017} technique. The proposed method is an adapted version of the Hestenes–Stiefel \cite{Gazi_Sharee_2014} and Dai–Yuan \cite{Dai_2013} techniques. The empirical outcomes of proposed algorithm present that this approach overtakes the Polak–Ribière \cite{Ghani_2017} , Fletcher–Reeves \cite{Wanto_2017} , Hestenes–Stiefel \cite{Gazi_Sharee_2014} and Dai–Yuan \cite{Dai_2013} techniques.
The reset of the proposed approach is similar to the other quasi-Newton methods which were established based on Newton’s optimization technique in which the Hessian matrix is replaced by a Hessian approximation to avoid the calculation of the exact Hessian matrix.
The AGRS with ERAS Updating Criterion