Before the popularity of CNN, many feature extraction methods were
proposed. One of the promising concepts is Independent Component
Analysis (ICA). ICA resembles the receptive fields of simple cells in
the visual cortex. ICA on natural images produces phase and
frequency-sensitive decorrelated filters, which resemble oriented Gabor
functions (Figure 15). ICA is different from PCA and cannot be
calculated analytically. ICA requires the minimum mutual information
among the output vectors and is achieved by choosing a nonlinear
activation function as the cumulative density function of the underlying
independent components. ICA
filters
were observed to have more sparsely distributed outputs on natural
images.
FIGURE 15 filters trained through ZCA-whitened natural images, which are
visually the same as the ICA filters
In 2007, Thomas used a biologically plausible method, a feed-forward
path of object detection. It used the Gabor Function to tune the filter
size. It is claimed that the Gabor function is more biologically
inclined and has been proven to be an effective model for simple
cortical cell receptive fields. The model is divided into simple and
complex units. The filter sizes were calculated from 7x7 to 37x37@16 for
simple units and 8x8 for complex units.
Sobel and Prewitt filters are popular gradient-based methods for Edge
detection in the frequency domain. Mathilde et al. proposed
’deep-cluster,’ a novel K-means-based clustering approach for
large-scale, end-to-end training of convolution layers. The paper was
focused on optimization based on K-means, but the K-means clustering was
not a part of the main convolutional architecture. AlexNet was
arbitrarily selected as the backbone architecture and can be replaced
with any similar architecture. The first layer used the Sobel filter as
a feature extractor. The Vertical and Horizontal weight matrices for
Sobel filters are:
\begin{equation}
W_{\text{ver}}=\ \begin{bmatrix}-1&0&1\\
-2&0&2\\
-1&0&1\\
\end{bmatrix}\text{\ and\ }W_{\text{Hor}}=\ \begin{bmatrix}1&2&1\\
0&0&0\\
-1&-2&-1\\
\end{bmatrix}\ \nonumber \\
\end{equation}Most filters cannot get trained over raw images with colors, which is
the primary motivation for applying the Sobel filter. The Sobel filter
is pre-defined, and the values are fixed for edge detection in the
vertical and horizontal directions. As no learning happens for the Sobel
filter, it remains independent of further training. The Prewitt filters
share a similar property to Sobel and could generate a similar result.