efef
sfe
sf

What to do?

  1. Create working model/protocol to beat the previous performances. Not really useful for publications, but perfect for testing tools.
  2. Create tools and protocols to analyze the workings of LSTMs.
  3. Understand the learning process and its mistakes.
  4. Develop the wacky ideas (e.g., duplet).
https://hackernoon.com/why-coding-multi-agent-systems-is-hard-2064e93e29bb
RdB on deep learning
Possible topics (theory or applications? images or sequences or else?):
Compression of deep nets: analyze critical paths in learned nets (assuming they are very sparse) to reduce the information both for deciphering how conclusions are drawn, see how they fail, improve their performances. Like capsule nets \cite{capsules} that are more compact. Turn the implicit, sparse reasoning in deep nets into rule-based explicit descriptive reasoning.
Distilling deep learning as traditional reasoning (probabilistic? as in Pearl's terms?)

Ricerca di Base

[intro] Since the explosion in popularity of deep neural networks in 2012, the machine learning (AI?) community has been trying to understand the principles of their proficiency, as demonstrated by beating previously established results in application after application. The divergence between empirical methods and theoretical understanding has got so bad that a NIPS presentation was titled "Machine learning has become alchemy" \citep{alchemy}.
[claim] What we critically miss in our empirical applications is insight: the ability to guide the design process (as opposed to blindly test unknown setups) and the confidence of understanding the trained models (what information they extract, how it is channeled into decisions/predictions). A factor compounding these difficulties is that trained models can be rarely compared directly, even the result of a training run will be different (in the value of the net parameters) from another one, although one can theoretically posit a certain kind of equivalence between similarly performing models. Quantitative comparisons are too uninformative (just a number, accuracy) and qualitative comparisons are too subjective (in the case of image features). Luckily, there are many available trained nets out there.
[prior lit] To be fair, the community has not been blind to this conundrum and has provided some tools to do just that (Vedaldi stuff).
[new perspective?] Recently, Geoffrey Hinton introduced a new type of neural network based on capsules. Capsule networks use a different strategy than regular convolutional neural networks: they get rid of the max-pooling layers that Hinton thinks are bad (they destroy locality of features), and are shallow architectures. In rough terms, the capsule network contains a layer of micro-cnns [to be checked] that encapsulate the dynamics of a certain feature (how a digit stretches in width, how a swirl develops, etc) almost exclusively, and whose norm of the response vector signals its presence.
[goal] Analyze trained models of different runs and different architectures (e.g., imagenet, resnet, etc) to find equivalences, information channels. Use it to find ways to encapsulate (compress the structure) and isolate understandable units of reasoning. Modify trained nets not by training them, but by tweaking the connections and the sub-units in them. Forse qsa di piu'...