Posts Tagged ‘reconstruction system

The problem statements and associated neural network models

The most general blind signal processing (BSP) problem can be formulated as follows. We observe a set of signals from an MIMO (multiple-input multiple-output) non-linear dynamic system (see Fig. 1), where its input signals are generated from a number of independent sources. The objective is to find an inverse neural system (also termed a reconstruction system), to see if it exists and is stable and to estimate the original input signals thereby. This estimation is performed on the basis of only the observed output signals where some a priori knowledge about the system and the stochastic properties of the source signals might be available. Preferably it is required that the inverse system is constructed adaptively so that it has good tracking capability under nonstationary environments. Alternatively, instead of estimating the source signals directly, it sometimes is more convenient to identify the unknown system(in particular when the inverse system does not exist) and then estimate the source signals implicitly by applying a suitable optimization procedure.

Fig. 1. Functional block diagram illustrating a general blind signal processing problem.

A dynamic non-linear system can be described in many different ways. It is sometimes convenient to describe it either as a lumped system or as a distributed system.This means that the system can be described by a set of ordinary differential (or difference) equations or partial differential equations. Alternatively, one may use linear filters described in the frequency domain. It should be emphasized that such a very general problem is in general intractable, and hence some a priori knowledge or assumptions about the system and the source signals are usually necessary.

Four fundamental problems arise:

1. solvability of the problem (in practical terms, this implies the existence of the inverse system and/or identifiability of the system).
2. stability of the inverse model.
3. convergence of the learning algorithm and its speed with the related problem of how to avoid being trapped in local minima;
4. accuracy of the reconstructed source signals.

Although recently many algorithms have been developed which are able to successfully separate source signals, there are still many problems to be studied.

• Development of learning algorithms which work:

1. under nonstationary environments;
2. when the number of the source signals is unknown;
3. when the number of source signals are dynamically changing,

where the properties of the nonstationarities are not known in advance.

• Optimal choices of the nonlinear activation functions when the distributions of sources are unknown and the mixtures contain sub-Gaussian and super-Gaussian sources.
• Influence of additive noises and methods for its cancellation or reduction.
• Global stability and convergence analysis of learning algorithms.
• Statistical efficiency of learning algorithms.
• Optimal strategy for deciding the learning rate parameter, especially in a nonstationary environment.

Our main objective in thispaper is to develop a theoretical framework for solving these problems and to present associated adaptive on-line learning algorithms. One direct approach to solve the problem is as follows.

1. Design suitable neural network models for the inverse(separating) problem.
2. Formulate an appropriate loss function (also termed a contrast, cost, or energy function) such that the global minimization or maximization of this function guarantees the correct separation or deconvolution. Inparticular, this guarantees the statistical independence of the output signals and/or spatial and temporal decorrelation of signals. The loss function should be a function of the parameters of the neural network model.
3. Apply an optimization procedure to derive learning algorithms. There are many optimization techniques based on the stochastic gradient descent algorithm, such as the conjugate gradient algorithm, quasi-Newton method, and so on. We develop a new algorithm called the natural gradient algorithm.

Tags : blind signal processing, BSP, conjugate gradient algorithm, convergence analysis, inverse neural system, lumped system, MIMO, multiple-input multiple-output, natural gradient algorithm, neural network, neural network models, non-linear dynamic system, nonstationary environments, quasi-Newton method, reconstruction system, sub-Gaussian, super-Gaussian, temporal decorrelation of signals